 |
| |
ISTQB Advanced Level Syllabi
8. Standards & Test Improvement Process
Terms
Capability Maturity Model (CMM), Capability Maturity Model Integration (CMMI), Test Maturity Model (TMM), Test Maturity Model integration (TMMi), Test Process Improvement (TPI).
Support for establishing and improving test processes can come from a variety of sources. This section first considers standards as a useful (sometimes mandatory) source of information for a number of test-related topics. Knowing which standards are available and where they may be applied is considered a learning objective for test managers and testers. Training providers should highlight those specific standards which are particularly relevant to the module being taught.
Once established, a test process should undergo continuous improvement. In section 8.3 generic improvement issues are first covered, followed by an introduction to some specific models which can be used for test process improvement. While test managers will need to understand all of the material in this section, it is also important that test analysts and technical test analysts, as key players in the implementation of improvements, are aware of the improvement models available.
In this, and in the Foundation Level Syllabus, some standards are mentioned. There are standards for
a number of topics related to software such as:
It is not the purpose of this syllabus to list or recommend specific standards. The testers should be aware of how standards are created, and how they should be used within the user’s environment.
Standards can come from different sources:
Some considerations apply when using standards. These are described further in this section.
8.2.1 General Aspects on Standards
8.2.1.1 Sources of Standards
Standards are created by groups of professionals and reflect the collective wisdom. There are two major sources of international standards: IEEE and ISO. National and domain specific standards are also important and available.
Testers should be aware of the standards that apply to their environment and context, whether formal standards (international, national or domain specific) or in-house standards and recommended practices. As standards evolve and change it is necessary to specify the version (publication date) of the standard to ensure that compliance is obtained. When a reference to a standard is specified without the date or version, then the latest version is considered to be referenced.
8.2.1.2 Usefulness of Standards
Standards can be used as an instrument to promote constructive quality assurance, which focuses on minimizing defects introduced rather than finding them through testing (analytical quality assurance). Not all standards are applicable to all projects; the information stated in a standard may be useful for a project, or may hinder it. Following a standard for the sake of following a standard will not help the tester find more defects in a work product [Kaner02]. However standards can provide some reference framework, and provide a basis on which to define test solutions.
8.2.1.3 Coherence & Conflicts
Some standards can lack coherence with other standards, or even provide conflicting definitions. It is up to the standards users to determine the usefulness of the different standards in his/her environment and context.
8.2.2.1 ISO
ISO [www.iso.org] stands for International Standards Organization (recently changed to International Organization for Standardization) and is made up of members representing, for their country, the national body most representative of standardization. This international body has promoted a number of standards useful for software testers, such as:
This list is not exhaustive; other ISO standards may be applicable to a tester's context and environment.
8.2.2.2 IEEE
IEEE [www.ieee.org] is the Institute of Electrical and Electronics Engineer, a professional organization based in the USA. National representatives are available in more than one hundred countries. This organization has proposed a number of standards that are useful for software testers such as:
This list is not exhaustive; other IEEE standards may be applicable to your context and environment.
Many countries have their own specific standards. Some of these standards are applicable and/or useful for software testing. One such British standard is BS-7925-2:1998 "Software testing. Software component testing" that provides information related to many of the test techniques described in this syllabus, including:
BS-7925-2 also provides a process description for Component Testing
8.2.4 Domain Specific Standards
Standards are also present in different technical domains. Some industries tailor other standards for their specific technical domains. Here also are aspects of interest in software testing, software quality and software development.
8.2.4.1 Avionics System
RTCA DO-178B/ED 12B "Software Considerations in Airborne Systems and Equipment Certification" is applicable to software used in civilian aircrafts. This standard also applies to software used to create (or verify) such software used in aircrafts. For avionics software, the United States Federal Aviation Administration (FAA) and the international Joint Aviation Authorities (JAA) prescribe certain structural coverage criteria based on the level of criticality of the software being tested:
| Criticality Level | Potential Impact of Failure | Required Structural | Coverage |
| A Catastrophic | Prevent continued safe flight and landing | Condition determination, Decision, and Statement | |
| B Hazardous / Severe-Major |
Large reduction in safety margins or functional capabilities Crew can not be relied upon to perform their tasks accurately or completely Serious or fatal injuries to a small number of occupants |
Decision and Statement | |
| C Major | Significant reduction in safety margins Significant increase in crew workload Discomfort to occupants possibly including injuries |
Statement | |
| D Minor | Slight reduction of aircraft safety margins of functional capabilities Slight increase in crew workload Some inconvenience to occupants |
None | |
| E No effect | No impact on the capabilities of the aircraft No increase in crew workload |
None |
The appropriate level of structural coverage must be achieved depending on the criticality level of the software that will be certified for use in civilian aviation.
8.2.4.2 Space Industry
Some industries tailor other standards for their specific domain. This is the case for the space industry with the ECSS (European Cooperation on Space Standardization) [www.ecss.org]. Depending on the criticality of the software, the ECSS recommends methods and techniques which are consistent with the ISTQB® Foundation and Advanced Syllabus including:
8.2.4.3 Food & Drug Administration
The FDA also recommends testing strategies and principles which are consistent with the ISTQB® Foundation and Advanced Level Syllabus.
The number of standards available in the different industries is very large. Some are adaptations to specific domains or industries, some are applicable to specific tasks or provide explanation on how to apply one standard.
It is up to the tester to be aware of the different standards (including in-house standards, best practices, etc.) that are applicable within his domain, industry or context. Sometimes the applicable standards are specified, with hierarchical applicability to specific contracts. It is up to the test manager to be aware of the standards that have to be complied with, and ensure that adequate compliance is maintained.
Just as testing is used to improve software, software quality processes are selected and used to improve the process of developing software (and the resulting software deliverables). Process improvement can also be applied to the testing processes. Different ways and methods are available to improve the testing of software and of systems containing software. These methods aim at improving the process (and hence the deliverables) by providing guidelines and areas of improvement. Testing often accounts for a major part of the total project costs. However, only limited attention is given to the test process in the various software process improvement models, such as CMMI (see below for details).
Test improvement models such as the Test Maturity Model (TMM), Systematic Test and Evaluation Process (STEP), Critical Testing Processes (CTP) and Test Process Improvement (TPI) were developed to cover this aspect. TPI and TMM provide a degree of cross-organization metrics that can be used for "benchmark" comparisons. There are many improvement models available in industry today. In addition to those covered in this section, Test Organization Maturity (TOM), Test Improvement Model (TIM) and Software Quality Rank (SQR) should also be considered. There are also a large number of regional models that are in use today. Testing professionals should research all available models to determine the best fit for their situation.
The models presented in this section are not intended to be a recommendation for use but are shown here to provide a representative view of how the models work and what is included within them.
8.3.1 Introduction to Process Improvement
Process improvements are relevant to the software development process as well as for the testing process. Learning from one's own mistakes makes it possible to improve the process organizations are using to develop and test software. The Deming improvement cycle: Plan, Do, Check, Act, has been used for many decades, and is still relevant when testers need to improve the process in use today.
One premise for process improvement is the belief that the quality of a system is highly influenced by the quality of the process used to develop the software. Improved quality in the software industry reduces the need for resources to maintain the software and thus provides more time for creating more and better solutions in the future.
Process models provide a place to start improving, by measuring the organization’s maturity processes with the model. The model also provides a framework for improving the organization’s processes based on the outcome of an assessment.
A Process Assessment leads to a Process Capability Determination, which motivates a Process Improvement. This may invoke a Process Assessment later to measure the effect of the improvement.
8.3.2 Types of Process Improvement
There are two types of models: process reference models and content reference models.
8.4 Improving the Test Process
The IT industry has started to work with test process improvement models as it seeks to reach a higher level of maturity and professionalism. Industry standard models are helping to develop crossorganization metrics and measures that can be used for comparison. The staged models, like TMMi and CMMI provide standards for comparison across different companies and organizations. The continuous models allow an organization to address its highest priority issues with more freedom in the order of implementation. Out of the need for process improvement in the testing industry, several standards have materialized. These include STEP, TMMi, TPI and CTP. These are each discussed further in this section.
All four of these test process assessment models allow an organization to determine where they stand in terms of their current test processes. Once an assessment is performed, TMMi and TPI provide a prescriptive roadmap for improving the test process. STEP and CTP instead provide the organization with means to determine where its greatest process improvement return on investment will come from and leave it to the organization to select the appropriate roadmap.
Once it has been agreed that test processes should be reviewed and improved, the process steps to be adopted for this activity should be as follows:
Initiate
In this activity the confirmation of the stakeholders, the objectives, goals, scope and coverage of the process improvements is agreed. The choice of the process model upon which the improvements will be identified is also made during this activity. The model could be either selected from those published or defined individually.
Before the process improvement activity starts, success criteria should be defined and a method by which they will be measured throughout the improvement activity should be implemented.
Measure
The agreed assessment approach is undertaken culminating in a list of possible process improvements.
Prioritize & Plan
The list of possible process improvements are put in order of priority. The order could be based upon return on investment, risks, alignment to organizational strategy, measurable quantitative or qualitative benefits.
Having established the priority order, a plan for the delivery of the improvements should then be developed and deployed.
Define & Redefine
Based upon the process improvement requirements identified, where new processes are required they are defined, and where existing processes require an update they are redefined and made ready for deployment.
Operate
Once developed, the process improvements are deployed. This could include any training or mentoring required, piloting of processes and ultimately the full deployment of them.
Validate
Having fully deployed the process improvements, it is key that any benefits that were agreed before the improvement(s) were made are validated e.g., benefit realization. It is also important that any success criteria for the process improvement activity have been met.
Evolve
Dependent on the process model used, this stage of the process is where monitoring of the next level of maturity starts and a decision is made to either start the improvement process again, or to stop the activity at this point.
The use of assessment models is a common method which ensures a standardized approach to improving test processes using tried and trusted practices. Test process improvement can however also be accomplished without models by using, for example, analytical approaches and retrospective meetings.
8.5 Improving the Test Process with TMM
The Testing Maturity Model is composed of five levels and is intended to complement CMM. Each of the levels contains defined process areas that must be completely fulfilled before the organization can advance to the next level (i.e. staged representation). TMM provides both a process reference model and a content reference model.
The TMM levels are:
Level 1: Initial
The initial level represents a state where there is no formally documented or structured testing process. Tests are typically developed in an ad hoc way after coding, and testing is seen as the same as debugging. The aim of testing is understood to be proving that the software works.
Level 2: Definition
The second level can be reached by setting testing policy and goals, introducing a test planning process, and implementing basic testing techniques and methods.
Level 3: Integration
The third level is reached when a testing process is integrated into the software development lifecycle, and documented in formal standards, procedures, and methods. There should be a distinct software testing function that can be controlled and monitored.
Level 4: Management & Measurement
Level four is achieved when the testing process is capable of being effectively measured, managed, and adapted to specific projects.
Level 5: Optimization
The final level represents a state of test process maturity, where data from the testing process can be used to help prevent defects, and the focus is on optimizing the established process
To achieve a particular level a number of pre-defined maturity goals and sub-goals must be achieved. These goals are defined in terms of activities, tasks and responsibilities and assessed according to specified "views" for manager, developer/tester and customer/user. For more information on TMM, see [Burnstein03].
The TMMi Foundation [see www.tmmifoundation.org for details] has defined the successor of TMM: TMMi. TMMi is a detailed model for test process improvement based on the TMM framework as developed by the Illinois Institute of Technology and practical experiences of using TMM and positioned as being complementary to the CMMI.
The structure of the TMMi is largely based on the structure of the CMMI (e.g, process areas, generic goals, generic practices, specific goals, specific practices).
8.6 Improving the Test Process with TPI
TPI (Test Process Improvement) uses a continuous representation rather than the staged representation of TMM.
The test process optimization plan as outlined in [Koomen99] involves a set of key areas that are set within the four cornerstones of Lifecycle, Organization, Infrastructure and tools, and Techniques. Key areas can be evaluated at a level between A to D, A being low. It is also possible for very immature key areas not to achieve the initial level A. Some key areas may only be rated at A or B (such as Estimating and Planning) while others (such as Metrics) may be rated at A, B, C or D.
The level achieved for a given key area is assessed by evaluating checkpoints defined in the TPI model. If, for example, all checkpoints for key area "reporting" are answered positively at level A and B, then level B is achieved for this key area.
The TPI model defines dependencies between the various key areas and levels. These dependencies ensure that the test process is developed evenly. It is not possible, for example, to achieve level A in key area "metrics" without also achieving level A in key area "reporting" (i.e. what is the use of taking metrics if they are not reported). The use of dependencies is optional in the TPI model.
TPI is primarily a process reference model.
A Test Maturity Matrix is provided that maps the levels (A, B, C or D) for each key area to an overall test process maturity level. These overall levels are
During a TPI assessment, quantitative metrics and qualitative interviews are used to establish the level of test process maturity.
8.7 Improving the Test Process with CTP (CTP)
As described in [Black02], the basic premise of the Critical Testing Process assessment model is that certain testing processes are critical. These critical processes, if carried out well, will support successful test teams. Conversely, if these activities are carried out poorly, even talented individual testers and test managers are unlikely to be successful. The model identifies twelve critical testing processes.
CTP is primarily a content reference model.
The critical testing processes model is a context-sensitive approach that allows for tailoring the model including:
The critical testing processes model is adaptable within the context of all software development lifecycle models.
Process improvements using CTP begin with an assessment of the existing test process. The assessment identifies which processes are strong and which are weak, and provides prioritized recommendations for improvement based on organizational needs. While the assessments vary depending on the specific context in which they are performed, the following quantitative metrics are commonly examined during a CTP assessment:
The following qualitative factors are commonly evaluated during a CTP assessment:
Once an assessment has identified weak areas, plans for improvement are put into place. Generic improvement plans are provided by the model for each of the critical testing processes, but the assessment team is expect to tailor those heavily.
8.8 Improving the Test Process with STEP
STEP (Systematic Test and Evaluation Process), like CTP and unlike TMMi and TPI, does not require that improvements occur in a specific order.
Basic premises of the methodology include:
STEP is primarily a content reference model.
The STEP methodology is based upon the idea that testing is a lifecycle activity that begins during requirements formulation and continues until retirement of the system. The STEP methodology stresses "test then code" by using a requirements-based testing strategy to ensure that early creation of test cases validates the requirements specification prior to design and coding. The methodology identifies and focuses on improvement of three major phases of testing:
During a STEP assessment, quantitative metrics and qualitative interviews are used. Quantitative metrics include:
Quantitative factors include:
In some cases the STEP assessment model is blended with the TPI maturity model.
8.9 Capability Maturity Model Integration, CMMI
The CMMI can be implemented via two approaches or representations: the staged representation or the continuous one. In the staged representation there are five "levels of maturity", each level building upon the process areas established in the previous levels. In the continuous representation the organization is allowed to concentrate its improvement efforts on its own primary areas of need without regard to predecessor levels.
The staged representation is primarily included in CMMI to ensure commonality with CMM, while the continuous representation is generally considered more flexible.
Within CMMI the process areas of Validation and Verification reference both static and dynamic testing test process.