ISTQB Advanced Level Syllabi
15. Appendix D – Recommendations
As these recommendations apply to various chapters of this syllabus, they have been compiled and merged in one appendix. The items listed below are examinable.
This list provides a number of helpful recommendations to meet testing challenges and builds on the list of seven basic testing principles introduced at the Foundation Level. The list provided in this appendix is not intended to be complete, but instead provides a sample of "lessons learned" which should be considered. Training providers will choose the points from this list which are most relevant to the module being taught
15.1 Recommendations for Industrialization
When implementing tests in an industrialized fashion, one is faced with a number of challenges. Advanced testers should be able to put the different recommendations described in this syllabus in focus within the context of their organization, teams, tasks and software components. The following list provides some areas that have proven to negatively impact performance of test efforts. Please note that the list is not intended to be complete.
- Generate test plans biased toward functional testing
Defects are not limited to functional aspects, or with only a single user. Interaction of multiple users may have impact on the software under test.
- Not enough configuration testing
If multiple types of processors, operating systems, virtual machines, browsers, and various peripherals can be combined into many possible configurations, limiting testing to just a few of these configurations may leave a large number of potential defects undiscovered.
- Putting stress and load testing off to the last minute
The results of stress and load testing may require major changes in the software (up to and including the basic architecture). Since this may require considerable resources to implement, this may be highly negative for the project if these tests are conducted just before the planned introduction into production.
- Not testing the documentation
Users are provided with the software and with documentation. If the documentation does not fit the software, the user will not be able to utilize the full potential of the software, or may even discard the software entirely.
- Not testing installation procedures
Installation procedures, as well as backup and restore procedures, are done a very limited number of times. These procedures are, however, more critical than the software; if the software can not be installed, it will not be used at all.
- Insisting on completing one testing task fully before moving on to the next
Even though some software development lifecycle models suggest the sequential execution of tasks, in practice many tasks often need to be performed (at least partly) concurrently.
- Failing to correctly identify risky areas
Some areas may be identified as risky and may as a result be tested more thoroughly. However, the areas left with minimal or no tests may subsequently turn out to be of higher risk than originally estimated.
- Being too specific about test inputs and procedures
By not giving testers enough scope for their own initiative in terms of defining test inputs and procedures, the tester may not be encouraged to examine areas that may look promising (in terms of possible hidden defects)
- Not noticing and exploring "irrelevant" oddities
Observations or results that may seem irrelevant are often indicators for defects that (like icebergs) are lurking beneath the surface
- Checking that the product does what it’s supposed to do, but not that it doesn’t do what it isn’t supposed to do
By limiting oneself only to what the product is supposed to do, it is possible to miss aspects of the software which it is not supposed to do (additional, undesired functions for example)
- Test suites that are understandable only by their owners
Testers may move to other areas of responsibility. Other testers will then need to read and understand previously specified tests. Failing to provide readable and understandable test specifications may have a negative impact because the test objectives may not be understood or the test may be removed altogether.
- Testing only through the user-visible interface
The interfaces to the software are not limited to the user-interface. Inter-process communications, batch execution and other interrupts also interact with the software, and can generate defects.
- Poor bug reporting and configuration management
Incident reporting and management, as well as configuration management are extremely important to ensure the success of the overall project (which includes development as well as testing). A successful tester may be considered one who gets defects fixed rather than one who finds many defects but fails to report them well enough to be corrected.
- Adding only regression tests
Evolution of test suites over time is not limited to checking that no regression defects occur.
The code will evolve over time and the additional tests will need to be implemented to cover these new functionalities, as well as to check for regressions in other areas of the software.
- Failing to take notes for the next testing effort
The testing tasks do not end when the software is provided to the user or distributed to the market. A new version or release of the software will most likely be produced, so knowledge should be stored and transferred to the testers responsible for the next testing effort.
- Attempting to automate all tests
Automation may seem like a good idea, but automation is a development project on its own.
Not all tests should be automated: some tests are faster to do manually than to automate.
- Expecting to rerun all manual tests
When rerunning manual tests, it is often unrealistic to expect that all tests will be rerun. Testers’ attention span will waver, and testers will tend to focus on particular areas of the software, either consciously or not.
- Using GUI automation tools to reduce test creation cost
A GUI capture/replay tool is an important initial investment and should only be used to support a defined strategy, with all the associated costs understood and evaluated.
- Expecting regression tests to find a high proportion of new bugs
Regression tests generally do not find a large proportion of defects, mostly because they are tests which have already been run (e.g., for a previous version of same software), and defects should have been detected in those previous runs. This does not mean that regression tests should be eliminated altogether, only that the efficiency (capacity to detect new defects) of regression tests is lower than other tests.
- Embracing code coverage with the devotion that only simple numbers can inspire
Code coverage and metrics may seem very interesting from a management point of view, based on the numbers and graphs provided, but numbers can not reflect the efficiency or pertinence of a test. Example: 100% is a nice target for code coverage, but is it a realistic one, is it the adequate one (i.e. is it instruction, condition, or MCDC coverage)?
- Removing tests from a regression test suite just because they don’t add coverage
In a regression test suite, some tests may be (should be) removed, and some others added.
The reason for removal should not be based only on whether the test adds coverage, as pertinence of a test case (the type of defect it checks for) may have no impact on coverage.
Example: coverage of code is not the only type of coverage; tests may have been created for other reasons (such as specific values or sequence of events) than simple coverage.
- Using coverage as a performance goal for testers
Coverage is a measure of completeness, not of performance or efficiency of personnel. Other metrics may be defined to evaluate tester efficiency in the context of their organization and project. The use of such metrics must be very carefully thought through to avoid undesired effects ("dysfunctions").
- Abandoning coverage entirely
Different types of coverage are available (e.g code statements, conditions, modules, functions etc), and the workload to obtain the adequate metrics may be significant. This is however not a reason to abandon coverage metrics altogether, since these may be very useful to the testing task.
Many of these points are dealt with within the context of specific sections of this syllabus.
When introducing testing measures and practices into your organization, it may be useful to consider not just the list of points above, but to also consider additional sources of information such as:
- The ISTQB syllabi (Foundation and Advanced)
- Books included in the reference list of this syllabus
- Reference models such as those covered in section 8.3.
