Testing and Debugging

Chapter 8: Testing and Debugging

Chapter Goals

To learn how to design test harnesses for testing components of your programs in isolation
To understand the principles of test case selection and evaluation
To be able to use assertions to document program assumptions
To become familiar with the debugger
To learn strategies for effective debugging

Unit Tests

In a unit test, a function or procedure is compiled outside the program in which it will be used, together with a test harness that feeds arguments to it.
Once you have confidence that the function is working correctly, you can plug it into your program.
Test data can be entered manually (see sqrtest1.cpp), in a loop (see sqrtest2.cpp), or using random number generation (see sqrtest3.cpp).

Unit Tests (`sqrtest1.cpp`)

Unit Tests (`sqrtest2.cpp`)

Unit Tests (`sqrtest3.cpp`)

Selecting Test Cases

Positive tests consist of legitimate inputs that you expect the program handles correctly.
Boundary cases are also legitimate inputs that you expect the program to handle in a trivial way.
Negative cases are inputs that you expect the program to fail on.
Keeping test data in a file is smart because you can use it to review every version of the program. Feed the file into the program to be tested by using redirection:
```
program < test1.in > test1.out
```
When you find a bug, you should record the data that caused the bug, and retest your program with that data after you have tried to fix the bug.
The phenomenon of fixing a bug, only to have it reappear after further program modifications is called cycling.
The process of testing against a set of past failures is called regression testing.
Testing the functionality of the program without consideration of its internal structure is called black-box testing.
Tasting strategies that look inside a program are called white-box testing.
"Testing can only show the presence of bugs - not their absence." Edsgar Dijkstra.

Test Case Evaluations

Once you have determined what inputs are needed to test the program, you need to decide if the outputs are correct.
Sometimes you can verify output by calculating the correct values by hand.

Sometimes a computation does a lot of work, and it is not practical to do the computation manually.

double squareroot(double a)
{  if (a == 0) return 0;
 
   double xnew = a;
   double xold;

   do
   {  xold = xnew;
      xnew = (xold + a / xold) / 2;
   }
   while (not approx_equal(xnew, xold));
   return xnew;
}

We could write a test program that verifies that the output values fulfill certain properties.
Here we test the squareroot() function by comparing the square of the returned number with the original input. (See sqrtest4.cpp).
Another method is to use a slower but reliable procedure (called an oracle) to verify the results. (In this case, we will use pow() as an oracle). (See sqrtest5.cpp)

Test Case Evaluations (`sqrtest4.cpp`)

Test Case Evaluations (`sqrtest5.cpp`)

Assertions

Functions often contain implicit assumptions (denominators should be nonzero, salaries should not be negative).
Such illegal values can creep into a program due to an input or processing error - with distressing regularity!

Assertions provide a valuable sanity check.

void raise_salary(Employee& e, double by)
{  assert(e.get_salary() >= 0 );
   assert(by >= -100);
   double new_salary = e.get_salary() * (1 + by / 100);
   e.set_salary(new_salary);
}

If an assertion is not satisfied, the program terminates with a useful error message showing the line number and the code of the failed assertion:
```
assertion failed in file finclac.cpp line 61: by >= -100
```

Program Traces

To get a printout of the program flow, you can insert trace messages into the beginning and end of every procedure.

It is also useful to print the input parameters when a procedure is entered and to print return values when a function is exited.

string int_name(int n)
{  cout << "Entering digit_name. n = " << n << "\n";
   ...
   cout << "Exiting digit name. Return value = "
        << s << "\n";
   return s;
}

To get a proper trace, you must locate each function exit point.

Sample output for n = 12305.

Inside int_name. Thousands.
Entering int_name. n = 12
Inside int_name. Teens.
Entering teen_name. n = 12
Exiting teen_name. Return value = twelve
Exiting digit_name. Return value = twelve
Inside int_name. Hundreds.
Entering digit_name. n = 3
Exiting digit_name. Return value = three
Inside int_name. Ones.
Entering digit_name. n = 5
Exiting digit_name. Return value = five
Exiting int_name. Return value = twelve thousand three hundred five

Program Traces (Problems with Trace Messages)

Program traces can be time-consuming to find out which trace message to insert.
If you insert too many messages, you produce a flurry of output that is hard to analyze.
If you insert to few, you many not have enough information to spot the cause of the error.
When you are done with the program, you need to remove all the trace messages.
When you find a new error, you need to stick the print statement back in.
Many profession programmers use a debugger, not trace messages, to locate errors in their code.

The Debugger

Modern development environments contain special programs, called debuggers, that help you locate bugs by letting you follow the execution of a program.
You can stop and restart your program and see the contents of variables whenever your program is temporarily stopped.
Using a debugger is not cost free - it takes time to set up and carry out an effective debugging session.
Larger programs are harder to debug without using a debugger, so learn how to use one is time well spent.

Debuggers vary wildly from one system to another.
If you use an integrated development environment, which contains an editor, compiler, and a debugger, finding and starting the debugger is usually very easy.
On a UNIX system, you must manually build a debug version of your program and invoke the debugger.
You can go a long way with just three commands:
- Run until this line.
- Step to next line.
- Inspect variable.
The exact names for these commands will vary from debugger to debugger, but all debuggers support them.

The "run until this line" command is the most important.
Select a line with the mouse or cursor, then hit a key or select a menu command to run the program to the select line.
It's possible that the program terminates normally without reaching that line, but that could be informative.

The "step to next line" command executes the current line and stops at the next program line.
Once the program has stopped, you can look at the current values of variables.
Some debuggers require you select the variable the choose a command such as "inspect variable."
Some debuggers require you to type the name of the variable into a dialogue box.
Some variables automatically show the values of all variables.

Many debuggers also have "step over" and "step into" commands.
The "step over" command advances to the next program line:

Example: If the current line is

r = future_value(balance, p n);
cout << setw(10) << r;

"step over" advances the program to

r = future_value(balance, p n);
cout << setw(10) << r;

The "step into" command advances into function calls.

Example: if the current line is

r = future_value(balance, p n);
cout << setw(10) << r;

"step into" advances the program to

double future_value(double initial_balance, double p, int n)
{  double b = initial_balance * pow((1 + p /100), n);
   return b;
}

You should not "step into" system functions like setw.
Some debuggers have a "run to end of function" or "step out" command to get out of a function.
Most debuggers can show you a call stack: a listing of all currently pending function calls.
My selecting the function in the middle of the call stack, you can jump to the code line containing that function call.

All debuggers support a navigational approach by inserting breakpoints in the code.
When the program reaches any breakpoint, execution stops.
Breakpoints are particularly useful when you know at which point your program starts doing the wrong thing.
Some debuggers let you set conditional breakpoints - the program stops only when a certain condition, such as n == 0, is met.

When inspecting an object variable, all the fields are displayed.
With some variables you must "open up" the object, usually by clicking on a tree node. Here, we have opened up the name field of the harry object..