Next: Large-scale testing. Up: Testing and debugging. Previous: None.

Testing

Solid testing requires the following three steps.

1. Generate a test case.
2. Determine what the results should be.
3. Run the program to observe its results.
4. Observe how the program's results differ from expectations.

Testing falls into two categories: white-box testing and black-box testing.

White-box testing

White-box testing involves generating test cases while looking at the code. Generally, you're looking for a large enough set of test cases to hit all the cases in the code.

• Hit all execution paths. Ideally, you'd have enough test cases to exercise every possible way you might hope to go through the program. This, however, is impractical for all but the smallest programs: the number of possible execution paths doubles for every if statement, and loops are even worse. It's just not reasonable.

• Hit all statements. Hitting all the statements of a program is a more realistic scenario. You'd write enough test cases so that every single statement of the program is executed at least once.

For example: Consider the following piece of code to find the maximum of 5 numbers typed by the user.

int max = IO.readInt();

for(int i = 0; i < 4; i++) {
    int q = IO.readInt();
    if(q > max) {
        max = q;
    }
}

1 1 1 1 1     1 1 1 1 5
1 1 1 4 1     1 1 1 4 5
1 1 3 1 1     1 1 3 1 5
1 1 3 4 1     1 1 3 4 5
1 2 1 1 1     1 2 1 1 5
1 2 1 4 1     1 2 1 4 5
1 2 3 1 1     1 2 3 1 5
1 2 3 4 1     1 2 3 4 5

If you settle for the second case, then there is just one case to test: And that case must have the if condition be true at some time.

1 2 1 1 1

Black-box testing

In black-box testing, the tester generates test cases without reference to the source code - that is, the tester is treating the program as a black box, into which the tester cannot look.

Beta testing obviously always involves black-box testing. But even the original software developer does this. In fact, it's probably the primary kind of testing you've been doing on your laboratories: Once you have the program coded, you run it by acting like a regular user.

Good black-box testing will include tests falling into three categories.

• Hit the common cases. This is the easiest one to do: You just try using the program like a normal user, looking for anything that doesn't work right. This is important, since usually (but not always) users will behave normally, and bugs are particularly embarrassing if they are doing nothing odd.

• Hit any boundary cases. Many problems have extremes. Generally you should try tests that exercise these extremes. For example, the video store could hold from 0 to 100 videos. I would certainly want to test whether the selvideo command works when there are 0 videos in the store. If I wanted to be really complete, I'd try filling up the store and adding a 101st video. (Actually, I'd change the constant to 5, recompile, and try adding a 6th video.)

• Try things in odd sequences. Often programmers will assume a user will try to do things in a particular sequence, without thinking about what happens in other cases.

  For example, in the video store program, you might assume that the user always selects a customer before trying to assign a video to the currently selected customer. You ought to try running the case of trying to assign a video to the currently selected customer before any customer has been selected. Chances are good that this would crash the program.

After finding a bug in black-box testing, it's often a good idea to try to prune the test case down to try to determine exactly what's going on. You'd do this before you even begin to try to debug, because the simpler test case will generally illustrate the actual problems better.

Next: Large-scale testing. Up: Testing and debugging. Previous: None.