 |
| |
ISTQB Advanced Level Syllabi
4. Test Techniques
Terms:
BS 7925-2, boundary value analysis (BVA), branch testing, cause-effect graphing, classification tree method, condition testing, condition determination testing, control flow analysis, D-D path, data flow analysis, decision table testing, decision testing, defect-based technique, defect taxonomy, dynamic analysis, error guessing, equivalence partitioning, exploratory testing, experienced-based technique, LCSAJ, memory leak, multiple condition testing, pair wise testing, path testing, requirements-based testing, software attacks, specification-based technique, static analysis, statement testing, state transition testing, structure-based technique, test charter, use case testing, wild pointer.
Test design techniques considered in this chapter include the following categories:
These techniques are complementary and may be used as appropriate for any given test activity, regardless of which level of testing is being performed. While any of these techniques could be employed by Test Analysts and Technical Test Analysts, for the purposes of this syllabus, the following division is made based on most common usage:
In addition to these areas, this chapter also includes a discussion of other techniques, such as attacks, static analysis and dynamic analysis. These techniques are normally performed by technical testing analysts.
Note that specification-based techniques can be either functional or non-functional. Non-functional techniques are discussed in the next chapter.
Specification based techniques are a way to derive and select test conditions or test cases based on an analysis of the test basis documentation for a component or system without reference to its internal structure.
Common features of specification-based techniques:
Some techniques also provide coverage criteria, which can be used for measuring the task of test design. Completely fulfilling the coverage criteria does not mean that the entire set of tests is complete, but rather that the model no longer suggests any useful tests based on that technique.
Specification-based tests are often requirements-based tests. Since the requirements specification should specify how the system is to behave, particularly in the area of functionality, deriving tests from the requirements is often part of testing the behavior of the system. The techniques discussed earlier in this subsection can be applied directly to the requirements specification, creating the model for system behavior from the text or graphics of the requirements specification itself, and then the tests follow as described earlier.
In the Advanced Level Syllabus, the following specifications-based test design techniques are considered:
| Name | Technique | Coverage Criteria |
| Equivalence partitioning | ||
| See ISTQB® Foundation Syllabus section 4.3.1 for a description. | Number of covered partitions / total number of partitions. | |
| Boundary value analysis (BVA) | ||
| See ISTQB® Foundation Syllabus section 4.3.2
for a description. Note that BVA can be applied using either 2 or 3 values. The decision which one to apply is most likely to be risk-based. |
Number of distinct boundary value covered / total number of boundary values | |
| Decision table testing and Cause-effect graphing | ||
| See ISTQB® Foundation Syllabus section 4.3.3 for a description of Decision table testing. With decision table testing every combination of conditions, relationships and constraints is tested. In addition to decision tables, a graphical technique using logical notation called causeeffect graphs can also be used. Note next to testing every combination of conditions also collapsed tables are possible. Using full decision tables or collapsed decision table is most likely to be risk-based. [Copeland03] | Number of combination of conditionscovered / maximum number of combination of conditions. | |
| State transition testing | ||
| See ISTQB® Foundation Syllabus section 4.3.4 for a description. | For single transitions, the coverage metrics is the percentage of all valid transitions exercised during test. This is known as 0-switch coverage. For n transitions the coverage measure is the percentage of all valid sequences of n transitions exercised during test. This is known as (n-1) switch coverage. | |
| Classification tree method, orthogonal arrays and all pairs tables | ||
| Factors or variables of interest and the options or
values those can take on are identified, and
singletons, pairs, triples, or even higher-order
combinations of those options or values are
identified. [Copeland03] Classification tree method uses a graphical notation to show the test conditions (classes) and combinations addressed by the test cases [Grochtmann94] |
Depends on the technique applied, e.g.
pair wise is distinct from classification trees. |
|
| Use case testing | ||
| See ISTQB® Foundation Syllabus section 4.3.5 for a description. | No formal criterion applies. | |
Sometimes techniques are combined to create tests. For example, the conditions identified in a decision table might be subjected to equivalence partitioning to discover multiple ways in which a condition might be satisfied. Tests would then cover not only every combination of conditions, but also, for those conditions which were partitioned, additional tests would be generated to cover the equivalence partitions.
A structure-based test design technique, also known as white-box or code-based test techniques, is one in which the code, the data, the architecture or system flow is used as the basis for test design, with tests derived systematically from the structure. The technique determines the elements of the structure which are considered. The technique also provides coverage criteria, which say when test derivation can conclude. These criteria do not mean that the entire set of tests is complete, but rather that the structure under consideration no longer suggests any useful tests based on that technique. Each criterion has to be measured and associated with an objective defined by each project or company.
Common features of structure-based techniques:
In the Advanced Level Syllabus, the following structure-based test design techniques are considered:
| Name | Technique | Coverage Criteria |
| Statement testing | ||
| The executable (non-comment, non-whitespace) statements are identified. | number of statements executed / total number of statements | |
| Decision testing | ||
| The decision statements, such as if/else,
switch/case, for, and while statements, are identified. |
Number of decision outcomes executed / total number of decision outcomes. | |
| Branch testing | ||
| The branches, such as if/else, switch/case, for, and
while statements, are identified. Note that decision and branch testing are the same at 100% coverage, but can be different at lower coverage levels |
Number of branches executed / total number of branches. | |
| Condition testing | ||
| The true/false and case labels are identified. | number of boolean operand values
executed / total number of boolean operand values |
|
| Multiple condition testing | ||
| All possible combinations of true/false conditions are identified. | number of boolean operand value combinations executed / total number of boolean operand value combinations | |
| Condition determination testing | ||
| The possible combinations of true/false conditions that can affect decisions (branches) are identified. |
number of boolean operand values shown to independently affect the decision / total number of boolean operands | |
| LCSAJ (loop testing) | ||
| The possible conditions that control loop iteration
are identified. Linear Code Sequence and Jump
(LCSAJ) is used to test a specific section of the code (a linear sequence of executable code) that
starts at the beginning of a particular control flow
and ends in a jump to another control flow or a
jump to the end of the program. These code
fragments are also known as DD-Paths (for
decision-to-decision path). This technique is used
to define specific test cases and the associated
test data required to exercise the selected control
flow. Designing these tests requires access to a
model of the source code that defines the control
flow jumps. LCSAJ can be used as a basis for code coverage measurement. |
number of executed LCSAJs / total number of LCSAJs. | |
| Path testing | ||
| The unique sequences of statements (paths) are identified. | number of executed paths / total number of paths. | |
One common use for structural coverage criteria is to measure the completeness or incompleteness of a set of tests designed using specification -based, and/or experienced-based techniques. Testing tools called code coverage tools are used to capture the structural coverage achieved by these tests. If important structural coverage gaps are identified (or the coverage mandated by applicable standards is not reached), then tests are added to address those gaps using structural and other techniques.
Analysis of coverage
Dynamic testing using structure-based or other techniques may be conducted to determine if a specific code coverage is attained or not. This is used to provide evidence in the case of critical systems where a specific coverage has to be obtained (see also section 1.3.2 Safety Critical Systems). The results can indicate that some sections of the code are not exercised, and thus lead to the definition of additional test cases to exercise these sections.
4.4 Defect- and Experience-based
A defect-based test design technique is one in which the type of defect sought is used as the basis for test design, with tests derived systematically from what is known about the defect.
The technique also provides coverage criteria which are used to determine when test derivation can conclude. As a practical matter, the coverage criteria for defect-based techniques tend to be less systematic than for behavior-based and structure-based techniques, in that the general rules for coverage are given and the specific decision about what constitutes the limit of useful coverage is discretionary during test design or test execution. The coverage criteria do not mean that the entire set of tests is complete, but rather that defects being considered no longer suggest any useful tests based on that technique.
In the Advanced Level Syllabus, the following defect-based test design technique is discussed:
| Name | Technique | Coverage Criteria |
| Taxonomies (categories & lists of potential defects) | ||
| The tester who uses the taxonomy samples from the list, selecting a potential problem for analysis. Taxonomies can list root cause, defect and failure. Defect taxonomies list most common defects in the software under test. The list is used to design test cases. | Appropriate data defects and types of defects. | |
4.4.2 Experienced-based techniques
There are other test design techniques which consider defect history but do not necessarily have systematic coverage criteria. These are categorized as experienced-based test techniques
Experience-based tests utilize testers' skills and intuition, along with their experience with similar applications or technologies. These tests are effective at finding defects but not as appropriate as other techniques to achieve specific test coverage levels or producing reusable test procedures.
When using dynamic and heuristic approaches, testers tend to use experience-based tests, and testing is more reactive to events than pre-planned approaches. In addition execution and evaluation are concurrent tasks. Some structured approaches to experience-based tests are not entirely dynamic; i.e. the tests are not created entirely at the same time as they execute test.
Note that although some ideas on coverage are presented in the following table, experienced-based techniques do not have formal coverage criteria.
In this syllabus, the following experience-based test design techniques are discussed:
| Name | Technique | Coverage Criteria |
| Error guessing | ||
| The tester uses experience to guess the potential errors that might have been made and determines the methods to uncover the resulting defects. Error guessing is also useful during risk analysis to identify potential failure modes. [Myers97] | When a taxonomy is used, appropriate data faults and types of defects. Without a taxonomy, coverage is limited by the experience of the tester and the time available. | |
| Checklist-based | ||
| The experienced tester uses a high-level list of items to be noted, checked, or remembered, or a set of rules or criteria against which a product has to be verified. These checklists are built based on a set of standards, experience, and other considerations. A user interface standards checklist employed as the basis for testing an application is an example of checklist-based test. | Each item on the checklist has been covered. | |
| Exploratory | ||
| The tester simultaneously learns about the product and its defects, plans the testing work to be done, designs and executes the tests, and reports the results. Good exploratory tests are planned, interactive, and creative. The tester dynamically adjusts test goals during execution and prepares only lightweight documentation. [Veenendaal02] | Charters may be created that specify tasks, objectives, and deliverables, and an exploratory testing session is planned that identifies what to achieve, where to focus, what is in and out of scope, and what resources should be committed. In addition, a set of heuristics about defects and quality may be considered. |
|
| Attacks | ||
| The tester makes a directed and focused evaluation of system quality by attempting to force specific failures to occur. The principle of software attacks, as described in [Whittaker03], is based on the interactions of the software under test with its operating environment. This includes the user interface, operating system with the kernel, the APIs and the file systems. These interactions are based on precise exchanges of data. Any misalignment in one (or more) of the interactions can be the cause of a failure. | The different interfaces of the application being tested. The principal interfaces are the user interface, operating system, kernel, APIs, and file system. |
Defect- and experienced-based techniques apply knowledge of defects and other experiences to increase defect detection, They range from "quick tests" in which the tester has no formally preplanned activities to perform, through pre-planned sessions to scripted sessions. They are almost always useful but have particular value in the following circumstances:
Defect- and experience-based techniques are also useful when used in conjunction with behaviorbased and structure-based techniques, as they fill the gaps in test coverage that result from systematic weaknesses in these techniques.
Static analysis is concerned with testing without actually executing the software under test and may relate to code or the system architecture.
Static analysis is any type of testing or examination that can be done without executing the code. There are a number of static analysis techniques and these are discussed in this section.
4.5.1.1 Control Flow Analysis
Control flow analysis provides information on the logical decision points in the software system and the complexity of their structure. Control flow analysis is described in the ISTQB® Foundation Level Syllabus and in [Beizer95].
4.5.1.2 Data Flow Analysis
Data flow analysis is a structured test technique that tests the paths between where a variable is set to where it is subsequently used. These paths are termed definition-use (du-pairs) or set-use pairs. In this method, test sets are generated to achieve 100% coverage (where possible) for each of these pairs.
This technique, although termed data flow analysis, also considers the flow of the control of the software under test as it pursues the set and use of each variable and may have to traverse the control flow of the software. See also [Beizer95]
4.5.1.3 Compliance to Coding Standards
During static analysis, compliance to coding standards can also be evaluated. Coding standards cover both the architectural aspects and the use (or prohibition of use) of some programming structures. Compliance to coding standards enables the software to be more maintainable and testable. Specific language requirements can also be verified using static testing.
4.5.1.4 Generate code metrics
Code metrics can be generated during static analysis that will contribute to a higher level of maintainability or reliability of the code. Examples of such metrics are:
4.5.2 Static Analysis of Architecture
4.5.2.1 Static Analysis of a Web Site
The architecture of a web site can also be evaluated using static analysis tools. Here the objective is
to check if the tree-like structure of the site is well-balanced or if there is an imbalance that will lead to:
Some specific testing tools that include a web spider engine can also provide, via static analysis, information on the size of the pages and on the time necessary to download it, and on whether the page is present or not (i.e. http error 404). This provides useful information for the developer, the webmaster and the tester.
4.5.2.2 Call Graphs
Call graphs can be used for a number of purposes:
Information on calling modules can also be obtained during dynamic analysis. The information obtained refers to the number of times a module is being called during execution. By merging the information obtained from call graphs during static analysis with the information obtained during dynamic analysis, the tester can focus on modules that are called often and/or repeatedly. If such modules are also complex (see 1.3.2.2 Safety Critical Systems & Complexity) they are prime candidates for detailed and extensive testing.
The principle of dynamic analysis is to analyze an application while it is running. This frequently requires some kind of instrumentation, inserted in the code either manually or automatically.
Defects that are not immediately reproducible can have significant consequences on testing effort and on the ability to release or productively use software. Such defects may be caused by memory leaks, incorrect use of pointers and other corruptions (e.g. of the system stack) [Kaner02]. Due to the nature of these defects, which may include the gradual worsening of system performance or even system crashes, testing strategies must consider the risks associated with such defects and, where appropriate, perform dynamic analysis to reduce them (typically by using tools).
Dynamic analysis is performed while the software is being executed. It may be applied to:
Dynamic analysis may be performed at any test level, but is particularly useful in component and integration testing. Technical and system skills are required to specify the testing objectives of dynamic analysis and, in particular, to analyze results.
A memory leak occurs when the memory (RAM) available to a program is allocated by that program but subsequently not released due to programming errors. The program thereby loses the ability to access this piece of memory and could eventually run out of usable memory.
Memory leaks cause problems which develop over time and may not always be immediately obvious. This may be the case if, for example, the software has been recently installed or the system restarted, which often occurs during testing. For these reasons, the negative affects of memory leaks may often first be noticed when the program is in production.
The symptoms of a memory leak are a steadily worsening of system response time which may ultimately result in system failure. While such failures may be resolved by re-starting (re-booting) the system, this may not always be practical or even possible.
Tools identify areas in the code where memory leaks occur so that they can be corrected. Simple memory monitors can also be used to obtain a general impression of whether available memory is reducing over time, although a follow-up analysis would still be required if this were the case.
There are other kinds of leaks that should be considered. Examples include file handles, semaphores and connection pools for resources.
"Wild" pointers within a program are pointers which are in some way unusable. For example, they may have "lost" the object or function to which they should be pointing or they do not point to the area of memory intended (e.g. beyond the allocated boundaries of an array). When a program uses wild pointers, a variety of consequences may occur:
Note that any changes made to the program’s memory usage (e.g. a new build following a software change) may trigger any of the four consequences listed above. This is particularly critical where initially the program performs as expected despite the use of wild pointers, and then crashes unexpectedly (perhaps even in production) following a software change. It is important to note that such failures are symptoms of an underlying error (i.e. the wild pointer) but not the error itself. (Refer to [Kaner02], "Lesson 74").
Tools may identify wild pointers as they are used by the program, irrespective of their consequences on the program’s execution.
Dynamic analysis may be conducted to analyze program performance. Tools help identify performance bottlenecks and generate a wide range of performance metrics which can be used by the developer to "tune" the system. This is also referred to as "performance profiling".