Need Of Testing

Why do software faults occur?
All of the products of software development (specifications, source code, and test documents) are written by human beings. All human beings are prone to make errors regardless of how experienced or skilled they are. So a number of faults in software are inevitable/expected.

Furthermore, many of the computer systems being developed today are so large and complex that it is not generally possible for any one person to understand every aspect of a system. Many people have specialized in particular areas or aspects, for example we have database experts, software design experts, algorithm experts, and network and operating system experts. Also, there are experts in specific areas of the businesses that the systems support. Good documentation (specifications and the like) are necessary to communicate all the relevant information each person needs to complete their parts of a system. However, people are fallible and so make errors causing faults in the specifications. When dealing with one part of a system, a systems designer or programmer may think they understand some other aspect outside of their particular area of expertise but they may not. Assumptions are often made because it is easier and quicker to assume one thing than to find out what the real answer should be.

What do software faults cost?
The cost of software faults that are not detected before a system is put into live operation varies, depending on a number of factors.

Some of the more spectacular software failures are well known. For example the European Space Agency’s Ariane 5 rocket exploded 37 seconds into its launch because of a software fault. This fault cost $7billion. Although it had been tested a series of errors meant that out-of-date test data was used. Another example is the Mariner Space probe. This was meant to be going to Venus but lost its way as a result of a software fault. The FORTRAN program had a full stop instead of a comma in a FOR (looping) statement that the compiler accepted as an assignment statement.

A couple of more recent examples involve Mars probes. In the first example the one organization’s software calculated the distance to landing in inches while another calculated the distance in centimeters. This resulted in the loss of the probe at a cost of $125 million. In the second example, a probe was lost shortly before it should have landed at a cost of $250 million. It turned out to be spurious signals from hardware sensors were interpreted by the software as landing, so antennas were deployed, protective covers removed, etc. This fault had been identified in earlier testing, and changes were made to the hardware and the software. However, there was no final regression test, due to time and budget pressure. (The lack of a final test was also responsible for the Hubble telescope being myopic for its first years of operation.)

One final example is where the expected results were not calculated beforehand and in this instance cost American Airlines $5Omillion. By way of background to this example, airlines never want to fly with empty seats - they would prefer to offer discount seats than to fly empty. However they also do not want to discount the seat when they could receive the full price. Airlines run complex ‘yield management programs’ to achieve the balance right. But these programs often change a lot to increase profitability. One such change was catastrophic for American Airlines. The program gave reasonable -but wrong - results for the number of discount seats! A spokesman said ‘We’re convinced that if we would have done more thorough testing, we would have discovered the problem before the software was ever brought on-line’.

Of course not every software fault causes such huge failure costs (fortunately!). Some failures are simple spelling mistakes or a misalignment in a report.

There is a vast array of different costs but there is a problem, software is not linear. Small faults can have large effects. For example a spelling mistake on a screen title said “ABC SOFTWEAR”, a simple mistake but it cost the company a sale. Another spelling mistake was when the programmer spelt an insurance company’s name wrong — instead of “Union” they spelt it “Onion” — only one letter, but this was a major embarrassment.

The consequences of faults in safety critical systems can be much more serious. The follow examples are all real cases.

· Therac-25 (Canada) - this machine was used in the treatment of cancer patients. It had the capability of delivering both x-rays and gamma rays. The normal dose of x-rays is relatively low while gamma ray dosage is much higher. Switching from one type of ray to the other was a slow process but the operators discovered that if they typed fast it could be made to switch more quickly. However, it only appeared to have switched. Six people died as a result of having too high an x-ray dose. This was a design fault.

· A train driver in Washington DC was killed when an empty train failed to stop in a stabling siding. The train was fully automated; driver just opens and closes doors. Driver had been requesting manual operation, as the trains had been overshooting the stops - this was denied by the humans in “central control” - RISKS Forum, 26 Jan 1998.

· Korean Airlines Flight 801, Boeing 747, 29 survivors (out of 254 passengers and crew). Came in 1000 ft too low. Combination of “normal” factors: severe weather, ground proximity system not switched on (or automatically disabled when landing gear lowered), secondary radar system should have sounded an alarm - either didn’t or wasn’t noticed. - Computer Weekly, 14 Aug 1987.

· Airbus crashes into trees at an air show at Habsham, Germany. Software protected engines by preventing them from being accelerated as fast as the pilot required to avoid hitting some trees after a low fly pass.

The following example is not of a safety critical system, rather a banking system. However, this did contribute to a death.

· An elderly man bought presents for his grandchildren, and as a result became overdrawn for the first time in his life. The banks system automatically sent threatening letters and charged him for being overdrawn. The man committed suicide. Source: client from major UK bank.

Testing is necessary:
Testing is necessary because software is likely to have faults in it and it is better (cheaper, quicker and more expedient) to find and remove these faults before it is put into live operation. Failures that occur during live operation are much more expensive to deal with than failures that occur during testing prior to the release of the software. Of course other consequences of a system failing during live operation include the possibility of the software supplier being sued by the customers!

Testing is also necessary so we can learn about the reliability of the software (that is, how likely it is to fail within a specified time under specified conditions).

Testing is not ‘to prove that the software has no faults’ because it is impossible to prove that software has no faults. Neither is testing done ‘because testing is included in the project plan’. Testing should be included in the project plan, but this is not the reason for testing. (Why is this testing included in the plan - this is the reason for testing.)

Testing is essential in the development of any software system. Testing is needed in order to assess what the system actually does, and how well it does it, in its final environment. A system without testing is merely a paper exercise - it may work or it may not, but there is no way of knowing it without testing it.

Exhaustive testing:
Exhaustive testing is defined as exercising every possible combination of inputs and preconditions. This usually amounts to an impractical number of test cases. For most systems it is relatively easy to identify millions of possible test cases (not necessarily all desirable test cases). For example, a system that accepts an input value corresponding to an employee’s annual salary may be expected to handle figures in the range £0.00 to £999,999.00. Given that every different input value could be made a separate test case this means that we can identify 100,000,000 test cases (the number of ip’s in the range) just from this one input value. Of course, no one would reasonably expect to have to try out every one of these input values but that is what exhaustive testing would require. Exhaustive testing would also require each of these 100,000,000 input values to be tried in different combinations with other inputs and with different preconditions (such as data values in a database). For even simple systems the number of test cases required for exhaustive testing is likely to run into millions of millions of millions.

Exhaustive testing is, therefore, impractical (it is not considered impossible, as it may be achievable for a very small program).

How much testing is enough?
It is possible to do enough testing but determining the how much is enough is difficult. Simply doing what is planned is not sufficient since it leaves the question as to how much should be planned. What is enough testing can only be confirmed by assessing the results of testing. If lots of faults are found with a set of planned tests it is likely that more tests will be required to assure that the required level of software quality is achieved. On the other hand, if very few faults are found with the planned set of tests, then (providing the planned tests can be confirmed as being of good quality) no more tests will be required.

Saying that enough testing is done when the customers or end-users are happy is a bit late, even though it is a good measure of the success of testing. However, this may not be the best test stopping criteria to use if you have very demanding end-users who are never happy!

Why not stop testing when you have proved that the system works? It is not possible to prove that a system works without exhaustive testing (which is totally impractical for real systems).

Have you tested enough when you are confident that the system works correctly? This may be a reasonable test stopping criterion, but we need to understand how well justified that confidence is. It is easy to give yourself false confidence in the quality of a system if you do not do good testing.

Ultimately, the answer to “How much testing is enough?” is “It depends!” (this was first pointed out by Bill Hetzel in his book “The Complete Guide to Software Testing”). It depends on risk, the risk of missing faults, of incurring high failure costs, of losing creditability and market share. All of these suggest that more testing is better. However, it also depends on the risk of missing a market window and the risk of over-testing (doing ineffective testing) which suggest that less testing may be better.

We should use risk to determine where to place the emphasis when testing by prioritizing our test cases. Different criteria can be used to prioritize testing including complexity, criticality, visibility and reliability.

Testing measures software quality:
Testing and quality go hand in hand! Basically we don’t know how good the software is until we have run some tests. Once we have run some good tests we can state how many faults we have found (of each severity level) and may also be able to predict how many faults remain (of each severity level).

Other factors that influence testing:
Other factors that affect our decision on how much testing to perform include possible contractual obligations. For example, a contract between a customer and a software supplier for a bespoke system may require the supplier to achieve 100% statement coverage (coverage measures are discussed in session 3). Similarly, legal requirements may impose a particular degree of thoroughness in testing although it is more likely that any legal requirements will require detailed records to be kept (this could add to the administration costs of the testing).
In some industries (such as the pharmaceutical industry and safety critical industries such as railroad switching and air traffic control) there are standards defined that have the intent of ensuring rigorous testing.