Testing and Debugging

Chapter 8: Testing and Debugging

Chapter Goals

To learn how to design test harnesses for testing components of your programs in isolation
To understand the principles of test case selection and evaluation
To be able to use assertions to document program assumptions
To become familiar with the debugger

Unit Tests

In a unit test, a function or procedure is compiled outside the program in which it will be used, together with a test harness that feeds arguments to it.
Once you have confidence that the function is working correctly, you can plug it into your program.
Test data can be

entered manually (sqrtest1.cpp),
in a loop (sqrtest2.cpp), or
using random number generation (sqrtest3.cpp).

Unit Tests (`sqrtest1.cpp`)

You need to enter inputs and to force an end of input when you are done, by typing a key such as Ctrl+Z or Ctrl+D.
For each test case, the harness code calls the squareroot function and prints the result.
You can then manually check the computations.

Unit Tests (`sqrtest2.cpp`)

We generate test cases automatically and check the results manually.

Unit Tests (`sqrtest3.cpp`)

Random generation of test cases is useful for testing.

Selecting Test Cases

Test cases:

Positive tests consist of legitimate inputs that you expect the program handles correctly.
For example: 4, 2, 1.5, 1e10, 1.111111111, ...
Boundary cases are also legitimate inputs that you expect the program to handle in a trivial way.
For example: 0
Other boundary cases are also legitimate inputs.
For example: 1e200, 1e-200, 2.222222222222, ...
Negative cases are inputs that you expect the program to fail on.
For example: -1

Keeping test data in a file is smart because you can use it to review every version of the program.
>sqrtest1 < test.in > test.out
When you find a bug, you should record the data that caused the bug, and retest your program with that data after you have tried to fix the bug.
The phenomenon of fixing a bug, only to have it reappear after further program modifications is called cycling. The process of testing against a set of past failures is called regression testing.
Testing the functionality of the program without consideration of its internal structure is called black-box testing.
Testing strategies that look inside a program are called white-box testing.

Test Case Evaluations

Once you have determined what inputs are needed to test the program, you need to decide if the outputs are correct.

Sometimes you can verify output by calculating the correct values by hand.
Sometimes a computation does a lot of work, and it is not practical to do the computation manually.

We could write a test program that verifies that the output values fulfill certain properties.
Here we test the squareroot() function by comparing the square of the returned number with the original input.

Test Case Evaluations (`sqrtest4.cpp`)

Another method is to use a slower but reliable procedure (called an oracle) to verify the results. In this case, we will use pow() as an oracle. (See sqrtest5.cpp)

Test Case Evaluations (`sqrtest5.cpp`)

Assertions (reminder)

Functions often contain implicit assumptions (denominators should be nonzero, salaries should not be negative).
Such illegal values can creep into a program due to an input or processing error - with distressing regularity!

Assertions provide a valuable sanity check.

void raise_salary(Employee& e, double by)
{  assert(e.get_salary() >= 0 );
   assert(by >= -100);
   double new_salary = e.get_salary() * (1 + by / 100);
   e.set_salary(new_salary);
}

If an assertion is not satisfied, the program terminates with a useful error message showing the line number and the code of the failed assertion:
```
assertion failed in file finclac.cpp line 61: by >= -100
```
That is a powerful signal that something went wrong elsewhere and that the program needs further testing.
[futval0.cpp]

Program Traces

To get a printout of the program flow, you can insert trace messages into the beginning and end of every procedure.

It is also useful to print the input parameters when a procedure is entered and to print return values when a function is exited.

string int_name(int n)
{  cout << "Entering digit_name. n = " << n << "\n";
   ...
   cout << "Exiting digit name. Return value = "
        << s << "\n";
   return s;
}

trace.cpp
To get a proper trace, you must locate each function exit point.

Sample output for n = 12305.

Inside int_name. Thousands.
Entering int_name. n = 12
Inside int_name. Teens.
Entering teen_name. n = 12
Exiting teen_name. Return value = twelve
Exiting digit_name. Return value = twelve
Inside int_name. Hundreds.
Entering digit_name. n = 3
Exiting digit_name. Return value = three
Inside int_name. Ones.
Entering digit_name. n = 5
Exiting digit_name. Return value = five
Exiting int_name. Return value = twelve thousand three hundred five

Program Traces (Problems with Trace Messages)

Program traces can be time-consuming to find out which trace message to insert.
If you insert too many messages, you produce a flurry of output that is hard to analyze.
If you insert to few, you may not have enough information to spot the cause of the error.
When you are done with the program, you need to remove all the trace messages.
When you find a new error, you need to stick the print statement back in.
Many profession programmers use a debugger, not trace messages, to locate errors in their code.

The Debugger

Modern development environments contain special programs, called debuggers, that help you locate bugs by letting you follow the execution of a program.
You can stop and restart your program and see the contents of variables whenever your program is temporarily stopped.
Using a debugger is not cost free - it takes time to set up and carry out an effective debugging session.
Larger programs are harder to debug without using a debugger, so learn how to use one is time well spent.

Debuggers vary wildly from one system to another.
If you use an integrated development environment, which contains an editor, compiler, and a debugger, finding and starting the debugger is usually very easy.
On a UNIX system, you must manually build a debug version of your program and invoke the debugger.
You can go a long way with just three commands:
- Run until this line.
- Step to next line.
- Inspect variable.
The exact names for these commands will vary from debugger to debugger, but all debuggers support them.
[prime.cpp]