Testing and Debugging

Chapter 8: Testing and Debugging

Chapter Goals

To learn how to design test harnesses for testing components of your programs in isolation
To understand the principles of test case selection and evaluation
To be able to use assertions to document program assumptions
To become familiar with the debugger
To learn strategies for effective debugging

Unit Tests

In a unit test, a function or procedure is compiled outside the program in which it will be used, together with a test harness that feeds arguments to it.
Once you have confidence that the function is working correctly, you can plug it into your program.
Test data can be entered manually (see sqrtest1.cpp), in a loop (see sqrtest2.cpp), or using random number generation (see sqrtest3.cpp).

Unit Tests (`sqrtest1.cpp`)

Unit Tests (`sqrtest2.cpp`)

Unit Tests (`sqrtest3.cpp`)

Selecting Test Cases

Positive tests consist of legitimate inputs that you expect the program handles correctly.
Boundary cases are also legitimate inputs that you expect the program to handle in a trivial way.
Negative cases are inputs that you expect the program to fail on.
Keeping test data in a file is smart because you can use it to review every version of the program.
When you find a bug, you should record the data that caused the bug, and retest your program with that data after you have tried to fix the bug.
The phenomenon of fixing a bug, only to have it reappear after further program modifications is called cycling.
The process of testing against a set of past failures is called regression testing.

Test Case Evaluations

Once you have determined what inputs are needed to test the program, you need to decide if the outputs are correct.
Sometimes you can verify output by calculating the correct values by hand.

Sometimes a computation does a lot of work, and it is not practical to do the computation manually.

double squareroot(double a)
{  if (a == 0) return 0;
 
   double xnew = a;
   double xold;

   do
   {  xold = xnew;
      xnew = (xold + a / xold) / 2;
   }
   while (not approx_equal(xnew, xold));
   return xnew;
}

We could write a test program that verifies that the output values fulfill certain properties.
Here we test the squareroot() function by comparing the square of the returned number with the original input. (See sqrtest4.cpp).
Another method is to use a slower but reliable procedure (called an oracle) to verify the results. (In this case, we will use pow() as an oracle). (See sqrtest5.cpp)

Test Case Evaluations (`sqrtest4.cpp`)

Test Case Evaluations (`sqrtest5.cpp`)

Assertions

Functions often contain implicit assumptions (denominators should be nonzero, salaries should not be negative).
Such illegal values can creep into a program due to an input or processing error - with distressing regularity!

Assertions provide a valuable sanity check.

void raise_salary(Employee& e, double by)
{  assert(e.get_salary() >= 0 );
   assert(by >= -100);
   double new_salary = e.get_salary() * (1 + by / 100);
   e.set_salary(new_salary);
}

If an assertion is not satisfied, the program terminates with a useful error message showing the line number and the code of the failed assertion:
```
assertion failed in file finclac.cpp line 61: by >= -100
```

Program Traces

To get a printout of the program flow, you can insert trace messages into the beginning and end of every procedure.

It is also useful to print the input parameters when a procedure is entered and to print return values when a function is exited.

string int_name(int n)
{  cout << "Entering digit_name. n = " << n << "\n";
   ...
   cout << "Exiting digit name. Return value = "
        << s << "\n";
   return s;
}

To get a proper trace, you must locate each function exit point.

Sample output for n = 12305.

Inside int_name. Thousands.
Entering int_name. n = 12
Inside int_name. Teens.
Entering teen_name. n = 12
Exiting teen_name. Return value = twelve
Exiting digit_name. Return value = twelve
Inside int_name. Hundreds.
Entering digit_name. n = 3
Exiting digit_name. Return value = three
Inside int_name. Ones.
Entering digit_name. n = 5
Exiting digit_name. Return value = five
Exiting int_name. Return value = twelve thousand three hundred five

Program Traces (Problems with Trace Messages)

Program traces can be time-consuming to find out which trace message to insert.
If you insert too many messages, you produce a flurry of output that is hard to analyze.
If you insert to few, you many not have enough information to spot the cause of the error.
When you are done with the program, you need to remove all the trace messages.
When you find a new error, you need to stick the print statement back in.
Many profession programmers use a debugger, not trace messages, to locate errors in their code.

The Debugger

Modern development environments contain special programs, called debuggers, that help you locate bugs by letting you follow the execution of a program.
You can stop and restart your program and see the contents of variables whenever your program is temporarily stopped.
Using a debugger is not cost free - it takes time to set up and carry out an effective debugging session.
Larger programs are harder to debug without using a debugger, so learn how to use one is time well spent.

Debuggers vary wildly from one system to another.
If you use an integrated development environment, which contains an editor, compiler, and a debugger, finding and starting the debugger is usually very easy.
On a UNIX system, you must manually build a debug version of your program and invoke the debugger.
You can go a long way with just three commands:
- Run until this line.
- Step to next line.
- Inspect variable.
The exact names for these commands will vary from debugger to debugger, but all debuggers support them.

The "run until this line" command is the most important.
Select a line with the mouse or cursor, then hit a key or select a menu command to run the program to the select line.
It's possible that the program terminates normally without reaching that line, but that could be informative.

The "step to next line" command executes the current line and stops at the next program line.
Once the program has stopped, you can look at the current values of variables.
Some debuggers require you select the variable the choose a command such as "inspect variable."
Some debuggers require you to type the name of the variable into a dialogue box.
Some variables automatically show the values of all variables.

Many debuggers also have "step over" and "step into" commands.
The "step over" command advances to the next program line:

Example: If the current line is

r = future_value(balance, p n);
cout << setw(10) << r;

"step over" advances the program to

r = future_value(balance, p n);
cout << setw(10) << r;

The "step into" command advances into function calls.

Example: if the current line is

r = future_value(balance, p n);
cout << setw(10) << r;

"step into" advances the program to

double future_value(double initial_balance, double p, int n)
{  double b = initial_balance * pow((1 + p /100), n);
   return b;
}

You should not "step into" system functions like setw.
Some debuggers have a "run to end of function" or "step out" command to get out of a function.
Most debuggers can show you a call stack: a listing of all currently pending function calls.
My selecting the function in the middle of the call stack, you can jump to the code line containing that function call.

All debuggers support a navigational approach by inserting breakpoints in the code.
When the program reaches any breakpoint, execution stops.
Breakpoints are particularly useful when you know at which point your program starts doing the wrong thing.
Some debuggers let you set conditional breakpoints - the program stops only when a certain condition, such as n == 0, is met.

When inspecting an object variable, all the fields are displayed.
With some variables you must "open up" the object, usually by clicking on a tree node. Here, we have opened up the name field of the harry object..

Strategies

Reproduce the error.
- What numbers did you enter?
- Where did you click the mouse?
- Run the same program again; type in exactly the same input.
Divide and Conquer
- Step over features using debugger, but don't step into them.
- Locate the last procedure called before failure.
- Step into that procedure, and repeat.

Know what your program should do.
- Does your program do what's expected?
- Before you inspect a variable, decide what value is should have.
  - If value is correct, look elsewhere for the bug.
  - If value is incorrect, double check your computation, then try to find out why your program comes up with a different value.
- Look for off by one errors.
- Look for computation errors (is the formula typed in correctly).
- Make "sign calculations" when the numbers get nasty. ("Should this number be positive or negative?")

Avoid quick fixes - they tend to create problems elsewhere.
Develop a thorough understanding of the problem and the program before you try to fix it.
Sometimes fix and fix only causes the problem to move around. Possibly the program logic is incorrect (does the program need redesign?).

Debugger Limitations

Break points in recursive procedures interrupt the program each time through the procedure.
When using the debugger on a recursive procedure, watch the call stack carefully.
Sometimes the compiler generates faster code by keeping a variable in a processor register rather than reserving a memory location for it.
- The debugger cannot find that variable or displays a wrong value for it.
- You can try turning off all compiler optimizations and recompile.
- You can open a special register window that shows all processor registers (advanced).
Some errors show up when you run the program normally, but go away under the debugger.
- Usually an uninitialized variable is the cause.
- Inspect all variables manually and check that they are initialized.
- Insert print statements if you are desperate.