Software Testing Certification ISTQB ASTQB: ISTQB Syllabi

ISTQB Advanced Level Syllabi

1. Basic Aspects of Software Testing

Terms:
Ethics, measurement, metric, safety critical systems, system of systems, software lifecycle.

1.1 Introduction
This chapter introduces some central testing themes that have general relevance for all testing professionals, whether Test Managers, Test Analysts or Technical Test Analysts. Training providers will explain these general themes in the context of the module being taught, and give relevant examples. For example, in the "Technical Test Analyst" module, the general theme of "metrics and measures" (section 1.4) will use examples of specific technical metrics, such as performance measures.

In section 1.2 the testing process is considered as part of the entire software development lifecycle. This theme builds on the basic concepts introduced in the Foundations Syllabus and pays particular attention to the alignment of the testing process with software development lifecycle models and with other IT-processes.

Systems may take a variety of forms which can influence significantly how testing is approached. In section 1.3 two specific types of system are introduced which all testers must be aware of, systems of systems (sometimes called "multi-systems") and safety critical systems.

Advanced testers face a number of challenges when introducing the different testing aspects described in this syllabus into the context of their own organizations, teams and tasks.

1.2 Testing in the Software Lifecycle

Testing is an integral part of the various software development models such as:

Sequential (waterfall model, V-model and W-model)
Iterative (Rapid Application Development RAD, and Spiral model)
Incremental (evolutionary and Agile methods)

The long-term lifecycle approach to testing should be considered and defined as part of the testing strategy. This includes organization, definition of processes and selection of tools or methods.

Testing processes are not carried out in isolation but interconnected and related to others such as:

Requirements engineering & management
Project management
Configuration- and change management
Software development
Software maintenance
Technical support
Production of technical documentation

Early test planning and the later test execution are related in the sequential software development models. Testing tasks can overlap and/or be concurrent.

Change and configuration management are important supporting tasks to software testing. Without proper change management the impact of changes on the system can not be evaluated. Without configuration management concurrent evolutions may be lost or mis-managed.

Depending on the project context, additional test levels to those defined in the Foundation Level Syllabus can also be defined, such as:

Hardware-software integration testing
System integration testing
Feature interaction testing
Customer Product integration testing

Each test level has the following characteristics:

Test goals
Test scope
Traceability to test basis
Entry and Exit criteria
Test deliverables including reporting
Test techniques
Measurements and metrics
Test tools
Compliance with organization or other standards

Depending on context, goals and scope of each test level can be considered in isolation or at project level (e.g. to avoid unnecessary duplication across different levels of similar tests).

Testing activities must be aligned to the chosen software development lifecycle model, whose nature may be sequential (e.g. Waterfall, V-model, W-model), iterative (e.g. Rapid Application Development RAD, and Spiral model) or incremental (e.g. Evolutionary and Agile methods).

For example, in the V-model, the ISTQB® fundamental test process applied to the system test level could align as follows:

System test planning occurs concurrently with project planning, and test control continues until system test execution and closure are complete.
System test analysis and design occurs concurrent with requirements specification, system and architectural (high-level) design specification, and component (low-level) design specification.
System test environment (e.g. test beds, test rig) implementation might start during system design, though the bulk of it would typically occur concurrently with coding and component test, with work on system test implementation activities stretching often until just days before the start of system test execution.
System test execution begins when the system test entry criteria are all met (or waived), which typically means that at least component testing and often also component integration testing are complete. System test execution continues until system test exit criteria are met.
Evaluation of system test exit criteria and reporting of system test results would occur throughout system test execution, generally with greater frequency and urgency as project deadlines approach.
System test closure activities occur after system test exit criteria are met and system test execution is declared complete, though they can sometimes be delayed until after acceptance testing is over and all project activities are finished.

For each test level, and for any selected combination of software lifecycle and test process, the test manager must perform this alignment during the test planning and/or project planning. For particularly complex projects, such as systems of systems projects (common in the military and large corporations), the test processes must be not only aligned, but also modified according to the project's context (e.g. when it is easier to detect a defect at higher level than at a lower level).

1.3 Specific Systems

1.3.1 Systems of Systems

A system of systems is a set of collaborating components (including hardware, individual software applications and communications), interconnected to achieve a common purpose, without a unique management structure. Characteristics and risks associated with systems of systems include:

Progressive merging of the independent collaborating systems to avoid creating the entire system from scratch. This may be achieved, for example, by integrating COTS systems with only limited additional development.
Technical and organizational complexity (e.g. among the different stakeholders) represent risks for effective management. Different development lifecycle approaches may be adopted for contributing systems which may lead to communication problems among the different teams involved (development, testing, manufacturing, assembly line, users, etc). Overall management of the systems of systems must be able to cope with the inherent technical complexity of combining the different contributing systems, and be able to handle various organizational issues such as outsourcing and offshoring.
Confidentiality and protection of specific know-how, interfaces among different organizations (e.g. governmental and private sector) or regulatory decisions (e.g. prohibition of monopolistic behavior) may mean that a complex system must be considered as a system of systems.
Systems of systems are intrinsically less reliable than individual systems, as any limitation from one (sub)system is automatically applicable to the whole systems of systems.
The high level of technical and functional interoperability required from the individual components in a system of systems makes integration testing critically important and requires well-specified and agreed interfaces.

1.3.1.1 Management & Testing of Systems of Systems

Higher level of complexity for project management and component configuration management are common issues associated with systems of systems. A strong implication of Quality Assurance and defined processes is usually associated with complex systems and systems of systems. Formal development lifecycle, milestones and reviews are often associated with systems of systems.

1.3.1.2 Lifecycle Characteristics for Systems of Systems

Each testing level for a system of systems has the following additional characteristics to those described in section 1.2 Testing in the Software Lifecycle:

Multiple levels of integration and version management
Long duration of project
Formal transfer of information among project members
Non-concurrent evolution of the components, and requirement for regression tests at system of systems level
Maintenance testing due to replacement of individual components resulting from obsolescence or upgrade

Within systems of systems, a testing level must be considered at that level of detail and at higher levels of integration. For example "system testing level" for one element can be considered as "component testing level" for a higher level component.

Usually each individual system (within a system of systems) will go through each level of testing, and then be integrated into a system of systems with the associated extra testing required.

For management issues specific to systems of systems refer to section 3.11.2.

1.3.2 Safety Critical Systems

"Safety critical systems" are those which, if their operation is lost or degraded (e.g. as a result of incorrect or inadvertent operation), can result in catastrophic or critical consequences. The supplier of the safety critical system may be liable for damage or compensation, and testing activities are thus used to reduce that liability. The testing activities provide evidence that the system was adequately tested to avoid catastrophic or critical consequences.

Examples of safety critical systems include aircraft flight control systems, automatic trading systems, nuclear power plant core regulation systems, medical systems, etc.

The following aspects should be implemented in safety critical systems:

Traceability to regulatory requirements and means of compliance
Rigorous approach to development and testing
Safety analysis
Redundant architecture and their qualification
Focus on quality
High level of documentation (depth and breadth of documentation)
Higher degree of auditability.

Section 3.11.3 considers the test management issues related to safety critical systems.

1.3.2.1 Compliance to Regulations

Safety critical systems are frequently subject to governmental, international or sector specific regulations or standards (see also 8.2 Standards Considerations). Those may apply to the development process and organizational structure, or to the product being developed.

To demonstrate compliance of the organizational structure and of the development process, audits and organizational charts may suffice.

To demonstrate compliance to the specific regulations of the developed system (product), it is necessary to show that each of the requirements in these regulations has been covered adequately. In these cases, full traceability from requirement to evidence is necessary to demonstrate compliance. This impacts management, development lifecycle, testing activities and qualification/certification (by a recognized authority) throughout the development process.

1.3.2.2 Safety Critical Systems & Complexity

Many complex systems and systems of systems have safety critical components. Sometimes the safety aspect is not evident at the level of the system (or sub-system) but only at the higher level, where complex systems are implemented (for example mission avionics for aircraft, air traffic control systems).

Example: a router is not a critical system by itself, but may become so when critical information requires it, such as in telemedical services.

Risk management, which reduces the likelihood and/or impact of a risk, is essential to safety critical development and testing context (refer to chapter 3). In addition Failure Mode and Effect Analysis (FMEA) (see section 3.10) and Software Common Cause Failure Analysis are commonly used in such context.

1.4 Metrics & Measurement

A variety of metrics (numbers) and measures (trends, graphs, etc) should be applied throughout the software development life cycle (e.g. planning, coverage, workload, etc). In each case a baseline must be defined, and then progress tracked with relation to this baseline.

Possible aspects that can be covered include:

Planned schedule, coverage, and their evolution over time
Requirements, their evolution and their impact in terms of schedule, resources and tasks
Workload and resource usage, and their evolution over time
Milestones and scoping, and their evolution over time
Costs, actual and planned to completion of the tasks
Risks and mitigation actions, and their evolution over time
Defects found, defect fixed, duration of correction

Usage of metrics enables testers to report data in a consistent way to their management, and enables coherent tracking of progress over time.

Three areas are to be taken into account:

Definition of metrics: a limited set of useful metrics should be defined. Once these metrics have been defined, their interpretation must be agreed upon by all stakeholders, in order to avoid future discussions when metric values evolve. Metrics can be defined according to objectives for a process or task, for components or systems, for individuals or teams. There is often a tendency to define too many metrics, instead of the most pertinent ones.
Tracking of metrics: reporting and merging metrics should be as automated as possible to reduce the time spent in producing the raw metrics values. Variations of data over time for a specific metric may reflect other information than the interpretation agreed upon in the metric definition phase.
Reporting of metrics: the objective is to provide an immediate understanding of the information, for management purpose. Presentations may show a "snapshot" of the metrics at a certain time or show the evolution of the metric(s) over time so that trends can be evaluated.

1.5 Ethics

Involvement in software testing enables individuals to learn confidential and privileged information. A code of ethics is necessary, among other reasons to ensure that the information is not put to inappropriate use. Recognizing the ACM and IEEE code of ethics for engineers, the ISTQB® states the following code of ethics:

PUBLIC- Certified software testers shall act consistently with the public interest.

CLIENT AND EMPLOYER - Certified software testers shall act in a manner that is in the best interests of their client and employer, consistent with the public interest.

PRODUCT - Certified software testers shall ensure that the deliverables they provide (on the products and systems they test) meet the highest professional standards possible.

JUDGMENT- Certified software testers shall maintain integrity and independence in their professional judgment.

MANAGEMENT - Certified software test managers and leaders shall subscribe to and promote an ethical approach to the management of software testing.

PROFESSION - Certified software testers shall advance the integrity and reputation of the profession consistent with the public interest.

COLLEAGUES - Certified software testers shall be fair to and supportive of their colleagues, and promote cooperation with software developers.

SELF - Certified software testers shall participate in lifelong learning regarding the practice of their profession and shall promote an ethical approach to the practice of the profession.