Chapter 8: Testing and Debugging

Chapter Goals

• To learn how to design test harnesses for testing components of your programs in isolation
• To understand the principles of test case selection and evaluation
• To be able to use assertions to document program assumptions
• To become familiar with the debugger

Unit Tests

• In a unit test, a function or procedure is compiled outside the program in which it will be used, together with a test harness that feeds arguments to it.
• Once you have confidence that the function is working correctly, you can plug it into your program.
• Test data can be entered manually (see sqrtest1.cpp), in a loop (see sqrtest2.cpp), or using random number generation (see sqrtest3.cpp).

Unit Tests (sqrtest1.cpp)

Unit Tests (sqrtest2.cpp)

Unit Tests (sqrtest3.cpp)

Selecting Test Cases

• Positive tests consist of legitimate inputs that you expect the program handles correctly.
  For example: 4, 2, 1.5, 1e10, 1.111111111, ...
• Boundary cases are also legitimate inputs that you expect the program to handle in a trivial way.
  For example: 0, 1e200, 1e-200, 2.222222222222, ...
• Negative cases are inputs that you expect the program to fail on.
  For example: -1
• Keeping test data in a file is smart because you can use it to review every version of the program.
  >sqrtest1 < test.in > test.out
• When you find a bug, you should record the data that caused the bug, and retest your program with that data after you have tried to fix the bug.
• The phenomenon of fixing a bug, only to have it reappear after further program modifications is called cycling.
• The process of testing against a set of past failures is called regression testing.
• Testing the functionality of the program without consideration of its internal structure is called black-box testing.
• Testing strategies that look inside a program are called white-box testing.
  

Test Case Evaluations

• Once you have determined what inputs are needed to test the program, you need to decide if the outputs are correct.
• Sometimes you can verify output by calculating the correct values by hand.
• Sometimes a computation does a lot of work, and it is not practical to do the computation manually.
• We could write a test program that verifies that the output values fulfill certain properties.
• Here we test the squareroot() function by comparing the square of the returned number with the original input. (See sqrtest4.cpp).

  Test Case Evaluations (sqrtest4.cpp)

• Another method is to use a slower but reliable procedure (called an oracle) to verify the results. In this case, we will use pow() as an oracle. (See sqrtest5.cpp)

  Test Case Evaluations (sqrtest5.cpp)

Assertions (reminder)

• Functions often contain implicit assumptions (denominators should be nonzero, salaries should not be negative).
• Such illegal values can creep into a program due to an input or processing error - with distressing regularity!
• Assertions provide a valuable sanity check.

  void raise_salary(Employee& e, double by)
  {  assert(e.get_salary() >= 0 );
     assert(by >= -100);
     double new_salary = e.get_salary() * (1 + by / 100);
     e.set_salary(new_salary);
  }

• If an assertion is not satisfied, the program terminates with a useful error message showing the line number and the code of the failed assertion:

  assertion failed in file finclac.cpp line 61: by >= -100

• That is a powerful signal that something went wrong elsewhere and that the program needs further testing.
  

Program Traces

• To get a printout of the program flow, you can insert trace messages into the beginning and end of every procedure.
• It is also useful to print the input parameters when a procedure is entered and to print return values when a function is exited.

  string int_name(int n)
  {  cout << "Entering digit_name. n = " << n << "\n";
     ...
     cout << "Exiting digit name. Return value = "
          << s << "\n";
     return s;
  }

• trace.cpp
• To get a proper trace, you must locate each function exit point.
• Sample output for n = 12305.

  Inside int_name. Thousands.
  Entering int_name. n = 12
  Inside int_name. Teens.
  Entering teen_name. n = 12
  Exiting teen_name. Return value = twelve
  Exiting digit_name. Return value = twelve
  Inside int_name. Hundreds.
  Entering digit_name. n = 3
  Exiting digit_name. Return value = three
  Inside int_name. Ones.
  Entering digit_name. n = 5
  Exiting digit_name. Return value = five
  Exiting int_name. Return value = twelve thousand three hundred five

Program Traces (Problems with Trace Messages)

• Program traces can be time-consuming to find out which trace message to insert.
• If you insert too many messages, you produce a flurry of output that is hard to analyze.
• If you insert to few, you many not have enough information to spot the cause of the error.
• When you are done with the program, you need to remove all the trace messages.
• When you find a new error, you need to stick the print statement back in.
• Many profession programmers use a debugger, not trace messages, to locate errors in their code.

The Debugger

• Modern development environments contain special programs, called debuggers, that help you locate bugs by letting you follow the execution of a program.
• You can stop and restart your program and see the contents of variables whenever your program is temporarily stopped.
• Using a debugger is not cost free - it takes time to set up and carry out an effective debugging session.
• Larger programs are harder to debug without using a debugger, so learn how to use one is time well spent.

• Debuggers vary wildly from one system to another.
• If you use an integrated development environment, which contains an editor, compiler, and a debugger, finding and starting the debugger is usually very easy.
• On a UNIX system, you must manually build a debug version of your program and invoke the debugger.
• You can go a long way with just three commands:
  • Run until this line.
  • Step to next line.
  • Inspect variable.
• The exact names for these commands will vary from debugger to debugger, but all debuggers support them.