Pre-Configuration Management Testing
The two primary test practices conducted prior to configuration management are:
- Peer Reviews are performed to find as many errors as possible in the software before the product enters the integration test. Peer reviews are one of the key performance activities at Level 3 of the Software Engineering Institute’s (SEI) Capability Maturity Model. The SEI accepts two kinds of peer reviews: code walkthroughs, and software inspections (the SEI preferred process, sometimes called Fagan Inspections in reference to Mike Fagan, who developed the process). Software inspections have a well-defined process understood throughout the industry. Done properly, software inspections can remove as much as 87 percent of the life-cycle errors in the software. There is no standard process for walkthroughs, which can have widely differing levels of rigor and effectiveness, and at best will remove about 60 percent of the errors in software.
- Unit Test is conducted by the developer, typically on the individual modules under development. Unit test often requires the use of drivers and stubs because other modules, which are the source of input data or receive the output of the module being tested, are not ready for test.
Post-Configuration Management Testing
Testing conducted after the product is placed under developer configuration control includes all testing beyond unit test. Once the system is under configuration management, a problem discovered during testing is recorded as a trouble report. This testing phase becomes progressively more expensive because it involves integrating more and more modules and functional units as they become available; the system therefore becomes increasingly more complex. Each test requires a documented test plan and procedure, and each problem encountered is recorded on a trouble report. Each proposed fix must be validated against the test procedure during which it was discovered, and must also verify that the code inserted to correct the problem does not cause another problem elsewhere. With each change made to respond to a problem, the associated documentation must be upgraded, the fix must be documented as part of the configuration management process, and the fix must be included in the next system build so that testing is not conducted with patches. The longer it takes to find a problem, the more rework is likely, and the more impact the fix may have on other system modules; therefore, the expense can continue to increase. Thus performing good peer reviews and unit tests is very important.
- Integration Test is a developer test that is successively more complex. It begins by integrating the component parts, which are either the modules that have completed the unit test or COTS products, to form functional elements. The integration test progresses from integration of modules to form entire functional elements, to integration between functional elements, to software-hardware integration testing. Modeling and simulation are often used to provide an operational-like testing environment. An integration test is driven by an integration test plan and a set of integration test procedures. Typically an integration test will have embedded within it a subset of tests indentified as regression tests, which are conducted following a system build. Their objective is to verify that the build process did not create a serious problem that would prevent the system from being properly tested. Often regression tests can be automated.
- Test Data Analysis: When conducting peer reviews, unit tests, integration testing, and system tests, a significant amount of data is collected and metric analysis is conducted to show the condition state of the system. Significant metric data is produced related to such things as defect density, pass-fail data on test procedures, and error trend analysis.
- System Test is an operational-like test of the entire system being developed. Following a successful system test, a determination is made whether the system is ready for acceptance test. After the completed system test, and before the acceptance test, a test readiness review (TRR) may be conducted to assess the readiness of the system to enter the acceptance test.
- Acceptance Test is witnessed by the government. It is the last test before the government formally accepts the system. Similar to the system test, the acceptance test is often a subset of the procedures run during system test.
- Operational Test is performed by an operational unit of the government. It is the final test before the system is declared ready for general distribution to the field.