Next: Debugging. Up: Testing and debugging. Previous: Testing.

Large-scale testing

Textbook: Section 10.10

With large-scale programs (of more than 100,000 lines, built by teams of programmers), it's not appropriate to wait until the program is entirely complete to begin testing.

Unit testing

In unit testing, each piece of the program is thoroughly tested before it is accepted. In Java, the most convenient way to break up a program into pieces will be into its separate classes. For example, for the video store program, you would write individual tests of the various classes (Customer, Store, Video, and Main) before putting them together.

This necessitates writing new classes whose sole purpose is to test others. For example, you might write the following program to test various the checkOut method of the Customer class.

public class CustomerTest {
    public static void main(String[] args) {
        Video[] vids = { new Video("A"), new Video("B"), new Video("C"),
            new Video("D"), new Video("E"), new Video("F") };
        Customer test = new Customer("Me");
        for(int i = 0; i < 5; i++) {
            try {
                test.checkOut(vids[i]);
            } catch(Exception e) {
                System.err.println("Unexpected exception on " + i + ": " + e);
            }
        }
        try {
            test.checkOut(vids[5]);
            System.err.println("Exception not thrown when limit reached");
        } catch(Exception e) {}
        Customer other = new Customer("You");
        try {
            other.checkOut(vids[0]);
            System.err.println("Exception not thrown when video already checked out");
        } catch(Exception e) {}
    }
}

It's not uncommon to have the code for the unit testing to be longer than the code it is meant to test!

Unit testing is problematic when there are dependencies between pieces. For example, there may be different people in charge of the Customer and Video classes. This causes a problem for the person writing the Customer class, as it cannot even be compiled until the Video class is complete.

To get around this problem, the Customer author would write a short stub class, which simply defines non-functional methods that Video is to provide. Then at least the Customer class should be able to be compiled. But this isn't adequate for testing purposes.

Integration testing

This dependency problem is resolved by integration testing. Integration testing requires that you draw a picture of which classes use which other classes, called a dependency graph. For example, for the first Drawer lab, you might draw the following picture.

From this, you could work out an order in which individual classes can be tested. Here, we would have to start out with Rectangle, then move to Drawing, then Canvas, and finally Drawer.

The dependency graph quickly gets much more complex as you add more classes to a program. Here's a dependency graph for the second part of the drawing lab.

And, in fact, that laboratory assignment is constructed with a view toward integration testing: You will successively build in new pieces that are relatively independent of each other.

Regression testing

When a software system is relatively complete, and the designers are engaged in incrementally adding new features, they often use regression testing. In regression testing, the developers build up a large library of tests associated with the program. Preferably, these tests will be automated.

When a developer thinks a feature is complete, the developer submits the modifications. But before they are accepted as valid, all the regression tests in the library are run to test whether the modifications break any existing programs. You don't want to accept a modification if it ends up introducing bugs into the system.

In very large systems, regression testing is often an nightly job, executed every night when the developers aren't using the computers.

Next: Debugging. Up: Testing and debugging. Previous: Testing.