Testing background

If programs show errors out in the field it doesn't give a good impression of the product nor the organisation behind it.
Most errors can be found before hand, although not all will be found or be solved in the right way. The following was found in a book about testing, and validated from experience over the past years. Maybe not all applies to eCS, but in a way they certainly do.

Reasons

A reason for testing is to improve the quality of the program, thereby giving (or keeping up) a good impression about the program as well as the organisation behind it.
There are also other reasons for testing, like the following:

• During the development of a program many specialists participate. All are working with the same specifications. Each contribution should be tested on correct functionality. Later in the development cycle the various components should be integrated into larger parts, which results in the final product at last. Continuous testing during the development cycle can show differences in interpretation of the specifications. Discovered errors during the development cycle should be fixed and incorporated as quick as possible in the specifications, thereby being available for the other specialists. The sooner errors are found the easier (and cheaper) the fix can be made.
• If the programmer is finished with program design and coding, it doesn't mean that the program complies to the specifications. Even before the program just runs (without abnormal terminations) it probably needs some modifications.
• The importance of a good acceptance test can be seen in the following figures. First making mistakes: 64% of the found errors during testing were made in the design phase. During the coding 'only' 36% was made. Then finding the errors: 2/3rd of the design errors were found during the acceptance tests, while only a quarter of the coding errors was found during these tests.
• Developers of a system can be convinced about the good working of their system. The challenge is to express this to the users. They want to have a kind of guarantee that the system works according to their specifications. Testing is usefull here as well. By using the acceptance tests the system is transferred from the developers into the users controlled area. By this a responsibility boundary can be drawn between the developers and the users.
• Software development is mostly done by different people, each with their specific specialisms. Descriptions from analysts are interpreted by designers, which transfer their results on to programmers who code the programs from them. The question is if everybody is understanding the material in the same way. A difference in interpretation is made easily and not everybody will check possible unclear situations. An assumption is made far easier and is most of the times not validated. This will result in delivering something different then was asked for.
• Testing should start as soon as possible, as costs of fixing things become more expensive in later steps of the development cycle.

A good saying that applies here is: trusting is good, but checking is better.
Have confidence in the quality somebody else delivers, but confirm and strenghten this by systematic and critical checks.

In the above mentioned points most developers and users find enough reasons to do testing. In the beginning there is enough enthousiasm, but lack of experience. Most of the times it is not clear how to test. A systematic approach is missing as well as methodologies, techniques and tools.

So we have to start with these. And try to get a complete working set to use. But first have a look at some other general topics.

What is the quality of a program?

The question "Is it a good program?" needs to be answered from the users perspective.
We need to keep an eye open for the balance between:

• does the program deliver what is asked for
• is the program easy to use

This is of course not an objective view, but it is about how the user experiences it. The fact that this is about a subjective experience becomes clear from the fact that different users can have different experiences about the same program. This is dependent on the expectations about the program and the situation they were in before they got this new program.

Is the quality of a program the same as a program without errors?

Quality is something different, and in fact more, then only being errorfree. Having errors as little as possible is one of the conditions for quality. There is a clear relation between the presence of errors and the efforts being needed to use the program. Errors interfere with the working rythm and cause irritation. This asks for more energy from the user and has therefor a negative effect on the experienced quality.

The terms testing and errors go hand in hand. For a lot of people it is not always clear what errors are. Discussions can explode on the question if something is good or bad. Who is right and who as to pay for getting it fixed. Some clarity on this is very welcome. Important in all cases is that the problems are fixed. Otherwise the users are always affected.

What are errors?

Assuming that the specifications describe the functionality that is being asked for, we consider the following situations as errors:

• the program reacts differently then the specifications describe
• the program does less then the specifications describe
• the program does more then the specifications describe

These type of errors are easy to test. The specifications are a solid base to test against to see if something is wrong or right. This is of course only valid if the specifications are complete, clear etc. If there are no specifications then this reference is lacking of course. We then have to look for alternatives, like manuals, problem descriptions, questions of users etc.
But from the users environment there certainly are more things that can be seen as errors. For example:

• not smoothly functioning, e.g. no systemic usage of function keys and a lot of differences in response times.
• not or not completely adjusting to the way of working, e.g. program is working batch wise, while the user is accustomed to transaction wise working.
• interfering with other programs, e.g. by performing one function it blocks other programs.
• working is not checkable, e.g. the outcome looks good, but it cannot be validated.
• not completely solving the users problem, e.g. it delivers some functionality, but other things still have to be done outside the program.
• not or hardly adjustable, e.g. for every little customisation the programmer is needed.

Errors from this list are not always easy to find. They are a realistic probability, but difficult to test.

Accepting that errors will be made, we have to determine answers to the following questions before defining the test approach:

• How many errors can we afford? In programs for controling modern airplanes we can afford us far less errors then in a program calculating the sales figures over last week. This answer is of importance for the broadness and thoroughness of the testing.
• Which amount of time and money is reasonable and needed, looking at the previous point. This is about the needed quality (economically reachable).

It is of course always questionable if such an amount of errors should exist in a program. By prescribing working instructions and the usage of standard components the error level can mostly be reduced significantly.
Despite of having errors a program can be very usefull in reality already. Therefor systems don't have to be errorfree to function well.

Experience shows that unstructured testing (which is happening most of the time) shows less then 50% of the clear errors. A structured testing approach will raise this figure (somtimes significantly).

A score of 100% is an illusion though.

Therefor we need to take care of some things, like backup options, support options etc.

Making errors is not the problem, but neglecting the existence of errors is. If we know that errors exist, they can be looked for and fixed. The testing procedures have to focus on finding errors and getting them fixed.

The question about them all being errors is irrelevant. The essence is that it is about problems.

• Problems which are (correct or not) experienced by users.
• Problems that prevent correct working of the program.
• Problems that need to be fixed.

Talking about found errors is often going into the direction of finding the guilty person. Punishing that person is not relevant for finding the best solution for the problem. Much better is to take care of preventing these kind of problems happening again. Looking for a quilty person to punish is counter-productive most of the times.

What is testing?

In general testing is measuring the quality of a program. Purpose is to make a good quality control possible based on the previous measuring. A number of used definitions are:

• Testing is showing that there a no more errors.
• Testing is proving that the program is doing the things it is supposed to do correctly.

Both definitions say actually the same, although errorfree programs don't exist, meaning that according to these definitions a program is not tested.
Another one:

• Testing is the activity to provide confidence in that the program does what it is supposed to.

How do we measure confidence? Confidence is a by-product for the users by performing tests.
Another try:

• Testing is the activity to improve the existing errors in the program.

This is only valid for the person who is testing his own programs. A test person is only allowed to signal errors, but has to stay away from the code.
Yet another try:

• Testing is the activity to prove the existance of the errors in the program.

This one fails on "the errors", as this means signaling all existing errors. Although this definition is usefull already.

A lot better are the following definitions:
For developers:

• Testing is a dedicated searching for and proving of errors in a program (for as far is this happens economically within the bounderies of needed absence of errors).

For users:

• Testing is the activity to prove the non usability of a program (for as far as this is economically valid and happens effectively enough).

This last definition sounds negative, but appeals to the possibilities of people to form an opinion to the max about the behaviour and results of the program in a real life situation.

Scopes of testing

By defining testing we on purpose didn't say that it is meant to find all errors in a program. This is a target that is not reachable most of the times.
Here is a little example to prove that:

• Say a simple program processes transactions each consisting of 64 possibly present categories. For each transaction this means a 2 to the power 64 = 1,84 x 10 to the power 19 possibilities. If we need one second to create, process, define the outcome and compare the outcome for each transaction we need about 600.000.000.000 years to test all possibilities of the program. Next to that we assumed (incorrectly) that the processing of each transaction is completely independent of previous transactions.

This means that we have to be selective in our testing. Thereby only doing the tests that provide a reasonable large chance to find an error, which could not be found by one of the other test cases. On the other side we also have to select test cases that the presence of errors that will remain unnoticed are minimal.

Most of the times the problem is to define when testing should be stopped. Is this when:

• the available time is over?
• the money is gone?
• requirements are being raised instead of errors?

No, of course not. The only valid reason to stop testing is:

• when all planned test cases are executed with acceptable results.

It is possible to start using a program while it contains erros, the so called known errors. This might be the result of release time pressure and the amount of work to get things fixed. It can even be that some of these are never fixed, because of being extreme situations and fixing is very expensive. Balancing costs of fixing against costs of occuring can lead to a decision of 'leave them in'. The users should be informed about them though. This can result in users not using certain parts of the program or be carefull in using them.

In a large number of projects, even big ones, it turns out that testing isn't performed thoroughly enough. Understandably as testing takes a lot of time and happens after coding the program, when the deadline is nearing rapidly. A lot of projectleaders decide that testing is finished one day before the deadline. The obvious achievement of delivering within time and budget is clearly visible, at the cost of quality control. They accept the risk of a bad program as the bad quality of the program is not proven yet!
Unfortunately it seems that a lot of people consider delivering erroneous programs as normal. Even customers are delighted when a program is delivered on time.

A main reason for this problem is the underestimation of the necessary time needed for testing.
A lot of time has been invested the last couple of years in planning projects in a structured way. Many tools in different areas (planning, coding and controlling) have made developing programs a very well planable and controlable activity. At making a plan most people forget that good testing takes about the same amount of time as good development. A 50/50 balance is normal! A reason for forgetting about testing in the plan is that testing is a cyclic activity. They assume that after testing and fixing possible errors the program is ready. They forget that after a fix, new testing has to be performed which can show new errors.

In general testing is considered a boring and non motivating activity, which somebody wants to get finished as soon as possible. This relates strongly to the nature of testing. Testing has something negative in itself, something sadistical, like "showing how bad this program actually is".

This destructive work seems to be more difficult to most people then the constructive work of analysing, designing and coding. The outcome of testing are most of the times also hard to measure to be stimulating. If many errors are found it is demotivating, because "such a poor program doesn't promise much". If the number of found errors is very low the question "why are we doing all this work?" arises quickly.

Above noted problems requires a professional approach of testing.