 |
| |
ISTQB Advanced Level Syllabi
9. Test Tools & Automation
Terms
Debugging tool, dynamic analysis tool, emulator, fault seeding tool, hyperlink test tools, keyword driven testing, performance testing tool, simulator, static analyzer, test execution tool, test management tool, test oracle
This section expands on the Foundation Level Syllabus by first covering a number of general concepts and then discussing some specific tools in further detail.
Even though some of the concepts covered may be more relevant to either test managers, test analysts or technical test analysts, a basic understanding of the concepts is required by all testing professionals. This basic understanding may then be expanded on where appropriate.
Tools can be grouped in a number of different ways, including a grouping according to their actual user, such as test managers, test analysts and technical test analysts. This particular grouping, which reflects the modules of this syllabus, is used for the remaining sections of this chapter. In general, the tools discussed in these sections are primarily relevant to a specific module, although certain tools (e.g., test management tools) may have wider relevance. Where this is the case, examples of the tool’s application in a specific context will be given by the Training Provider.
Test tools can greatly improve the efficiency and accuracy of the test effort, but only if the proper tools are implemented in the proper way. Test tools have to be managed as another aspect of a well-run test organization. Test automation often is assumed to be synonym with test execution, but most manual labor has different forms of test automation, meaning that most areas within test could be automated to some degree if the right tools were present.
Any test tool version, test script or test session should be, as any test basis, under configuration management and bound to a particular software version where it has been used. Any test tool is an important part of the testware and should be managed accordingly, such as:
9.2.1 Cost benefits and Risks of Test Tools and Automation
A cost-benefit analysis should always be performed and show a significant positive return on investment. The main characteristics of a cost-benefit analysis should encompass the following cost items by comparing actual cost for both manual (non-tool usage) and tool-usage in terms of costs (hours translated into cost, direct costs, recurring and non-recurring costs):
Business cases based only on pilot automation projects often miss important costs such as the cost of maintaining, updating and expanding the test scripts when the system changes. Durability of a test case is how long it will remain valid without rework. The time required to implement the first version of automated test scripts is often far beyond the time to run them manually, but will enable the possibility to create many more similar test scripts much faster and easily expand the number of good test cases over time. In addition, significant test coverage and test efficiency improvements will be seen on future uses of the automation after the implementation period. The business case for tool implementation must be based on the long term business case.
On a specific level each test case must be considered to see if it merits automation. Many automation projects are based on implementation of readily available manual test cases without reviewing the actual benefit of the automation of each particular case. It is likely that any given set of test cases (a suite) may contain manual, semi-automated and fully automated tests
In addition to the topics covered in the Foundation Level Syllabus, the following aspects should be considered:
Additional Benefits:
Additional Risks:
Test tools are usually intended to be used for more than one particular project. Depending on the investment and length of a project, it might not give an adequate return on investment within the project, but would on subsequent versions of the software. For example, since the maintenance phase is often test intensive (for every correction a large regression suite must be executed), it can sometimes be cost-beneficial to automate a system that is in the maintenance phase if the system life span can make it economical. For another example it is easy for humans to make mistakes while doing manual testing (such as typing errors), thus the cost-benefit of automating input of data and comparison of output data to data from an oracle (e.g. test result comparison to expected result) in a test suite is beneficial.
For companies that use and are dependent on many test tools (for different phases and purposes) a long-term test tool strategy is advisable to aid in decisions of phase-in and out of different tool versions and tool support. For larger companies with a tool intensive domain, it can be advisable to provide general guidelines for tool acquisition, strategies, tool paradigms or script languages to use.
9.2.3 Integration & Information Interchange Between Tools
Usually there is more than one test tool being used within the test (and development) process. Let’s take an example of a company using a static analysis tool, a test management and reporting tool, a configuration management tool, an incident management tool and a test execution tool all at the same time. It is important to consider if the tools can be integrated and exchange information with each other in a beneficial way. For example, it would be beneficial if all test execution status was reported directly to the test management system to provide immediate updates of progress, and direct traceability from requirements to a particular test case. It is more effort and more error-prone to store test scripts both in a test management database and in the configuration management system. If a tester wants to launch an incident report in the middle of test case execution, the defect tracking and test management systems have to be integrated. While static analysis tools may be separate from the other tools, it would be far more convenient if the tool could report incidents, warnings and feedback directly to the test management system.
Buying a suite of test tools from the same vendor does not automatically mean that the tools work together in this manner, but is a reasonable requirement. All these aspects should be evaluated from the cost of automating the information interchange compared to the risk of tampering with or losing information with sheer manual labor, assuming the organization has the time for the work this will entail.
New concepts like the integrated development environments like Eclipse aim to ease the integration and usage of different tools in the same environment by providing a common interface for development and test tools. A tool vendor can become Eclipse "compliant" by creating a plug-in to the Eclipse framework, making it have the same look and feel as any other tool. This creates a good advantage for the user. Note: this does not automatically mean that even if the user interface is similar, that the tools automatically provide integration and information interchange between themselves.
9.2.4 Automation Languages: Scripts, Script Language
Scripts and script languages are sometimes used to better implement and expand the test conditions and test cases. For example when testing a web application, a script might be used to bypass the user interface to more adequately test the API (application programming interface) itself. Another example would be the case where the testing of a user interface is automated to allow all possible combinations of inputs which would be infeasible with manual testing.
The capability of scripting languages varies widely. Note that scripting languages can range from normal programming languages, to very specific standard notations, e.g. signaling for protocols like TTCN-3.
9.2.5 The Concept of Test Oracles
Test oracles are generally used to determine expected results. As such, they perform the same function as the software under test and so are rarely available. They may be used, however, in situations where an old system is being replaced by a new system with the same functionality, and so the old system can be used as an oracle. Oracles may also be used where performance is an issue for the delivered system. A low performance oracle may be built or used to generate expected results for the functional testing of the high performance software to be delivered.
Every automated tool is software in its own right and may have hardware or software dependencies. A tool should be documented and tested itself regardless of whether it is purchased as-is, adapted or created in house. Some tools are more integrated with the environment, and other tools work better as stand-alone tools.
When the system under test runs on proprietary hardware, operating systems, embedded software or uses non-standard configurations, it may be necessary to create (develop) a tool or adapt a tool to fit the specific environment. It is always advisable to do a cost-benefit analysis that includes initial implementation as well as long-term maintenance.
During deployment of a test automation tool it is not always wise to automate manual test cases as is, but to redefine the test cases for better automation use. This includes formatting the test cases, considering re-use patterns, expanding input by using variables instead of using hard-coded values and utilizing the benefits of the test tool, which has abilities to traverse, repeat and change order with better analysis and reporting facilities. For many test automation tools, programming skills are necessary to create efficient and effective test programs (scripts) and test suites. It is common that large test suites are very difficult to update and manage if not designed with care. Appropriate training in test tools, programming and design techniques is valuable to make sure the full benefits of the tools are leveraged.
Even when manual test cases have been automated, it is important to periodically execute the test cases manually to retain the knowledge of how the test works and to verify correct operation.
When a tool gets into use and the number of test scripts grows, there may be a need to add on features that could be provided by other tools. This is not always possible since tools do not always have open interfaces and sometime use proprietary non-standard scripting languages. It is wise to use tools that have plug-ins to open frameworks or API (Application Programming Interface,. This will guarantee a better future-proofing of the test scripts as testware.
For each type of tool, regardless of the phase in which it is to be used, consider the characteristics listed below. These characteristics can be used both in tool evaluations and when building a tool. In each of these areas, a tool can be weak or strong. A list such as this is useful when comparing the capabilities of similar tools.
9.2.7 Usage of Open Source Test Tools
Tools used to test safety critical systems must be certified to comply with the intended purpose against the corresponding standards. It is not recommended to use open-source tools in safety-critical systems, unless they have obtained the appropriate level of certification.
The quality of open-source software is dependent on the exposure, history and usage of software in question, and should not be assumed to be more (or less) accurate than any commercially available tool.
An assessment of the quality should always be conducted for any test tool to assess the accuracy of the tool. For some tool types it is more easy to confuse a positive evaluation result with a wrong tool execution (e.g. it skipped executions and did not report what was skipped). Careful consideration should be give to license fees. There may also be an expectation that code modifications made to enhance the tool will be shared.
9.2.8 Developing Your Own Test Tool
Many test tools are developed out of the need for an individual tester or designer to speed up their own work. Other reasons for self-developed test tools are the lack of suitable commercial tools or the use of proprietary hardware or test environment. These tools are often efficient to do the task they are supposed to do, but are often very dependent on the person creating the tool. These tools should be documented in a way that they can be maintained by others. It is also important to review the purpose, aim, benefits and possible downside before spreading it across an organization. Often these tools get new requirements and are expanded far beyond the initial use, which might not be beneficial.
9.2.9 Test Tool Classification
In addition to tools being divided into what activity they support (which is the concept used at the Foundation Level), there are other tool classification schemes, including:
And finally, tools can be grouped according to their actual user, such as test managers, test analysts and technical test analysts. This grouping, which reflects the modules of this syllabus, is used for the remaining sections of this chapter. The Foundation Level Syllabus includes a section regarding tools. The sections below are additional aspects of these tools.
This section has the following objectives:
Please refer to the ISTQB® Foundation Level Syllabus section 6 for general information concerning the other tools categories not included in this section
For general information concerning test management tools, please refer to the ISTQB® Foundation Level Syllabus section 6.1.2.
Test management tools should have the ability to track the following information:
Test Management tools are used by test managers, test analysts and technical test analysts.
For general information concerning Test Management tools, please refer to the ISTQB® Foundation Level Syllabus section 6.1.5.
Test Execution tools are mostly used by Test analysts and Technical Test analysts at all levels of testing, to run tests and check the outcome of the tests. The objective of using a Test Execution tool is typically one or more of the following:
Test Execution tools are most often used to automate regression tests.
Test Execution tools work by executing a set of instructions written in a programming language, often called a scripting language. The instructions to the tool are at a very detailed level that specifies individual button presses, key hits and mouse movements. This makes the detailed scripts very susceptible to changes in the software under test (SUT), particularly changes to the graphical user interface (GUI).
The starting point for a script may be a recording (done with capture replay) or a constructed or edited script using existing scripts, templates or keywords. Scripting is a program, and is working exactly like any software. Capturing (or recording) can be used to record an audit trail for non-systematic testing. Most test execution tools include a comparator, the ability to compare an actual result to a stored expected result. The tendency in testing (as in programming) is moving from detailed low-level instruction to more "high-level" languages, here again libraries, macros and sub-programs are utilized. A series of instructions are labelled with one name – in testing called key-word driven or action-word driven. The main advantage is separating instructions from data. It is the same concept as utilizing templates when scripting to minimize user effort.
The main reason why some test tools fail is due to the low skills in programming and understanding that a test tool just solves part of the problems in automating test execution. It is important to note that any test execution automation takes management, effort, skills and attention, e.g. test architecture and configuration management. This also means that test scripts can have defects. The use of testware architecture might give independence from a particular tool vendor. When a tool is procured, there is a tendency to think that the tool’s standards must be followed, for example for the structure and naming conventions of scripts. However, the automating of test set-up can form an interface between your own best way of organizing tests and where the tool needs to find them in order to run them.
9.3.3 Debugging & Troubleshooting Tools
Troubleshooting may employ tools to narrow down the area where a fault occurs. This might also be needed in a system where it is not evident what fault caused the exhibited failure. Troubleshooting tools include traces and simulated environments used to interact with the software or extract information from the system to narrow down the location of the fault.
Programmers reproduce faults and investigate the state of programs by using debugging and tracing tools. Debuggers and traces enable programmers to:
It should be made clear that debugging (and debugging tools) are related to testing but are not testing (or testing tools). Debugging and tracing tools may be used for trouble-shooting purposes by testers to better locate a fault origin and help determine where to send a defect report. Debugging, tracing and troubleshooting tools are mainly used by Technical Test Analysts.
9.3.4 Fault Seeding & Fault Injection Tools
Fault seeding and fault injection are two different techniques that can be used in testing. Fault seeding will utilize a tool similar to a compiler to create single or limited types of code faults in a systematic way. These tools are also often used in conjunction with the mutation test technique and are sometimes called mutation test tools.
Fault injection is aimed at changing the actual interfaces to test components (when source code is not available), but could also be deliberately (re-)injecting a particular fault to check if 1) the software can cope with it (fault tolerance) or 2) to evaluate that a test in a test suite finds the deliberately inserted fault. Fault seeding and fault injection tools are mainly used at the code level by Technical Test Analysts, but it is also possible for a test analyst to manipulate data in a data base or inject faults into the data stream to test the system behavior.
9.3.5 Simulation & Emulation Tools
Simulators are used to support tests where code or other systems are unavailable, expensive or impracticable to use (e.g. testing software to cope with nuclear meltdowns). Some simulators and test harness tools can also support or mimic fault behavior, so error states or error scenarios can be checked. The main risk with using these tools is that resource-related errors like timing issues may not be found which are very important for some type of systems.
Emulators are a particular category of simulators, and consist of software written to mimic the hardware. The advantage of using emulators is that more elaborate testing may be possible. One particular advantage with emulators is that tracing, debugging and time-dependent causes can be recreated, which might be impossible in a real system. Emulators are costly to create, but the advantage of analysis where the system can be run in "slow-motion" are invaluable for some parallel, time dependent and complex systems.
Test Analysts and Technical Test Analysts, depending on the type of emulation required, use these tools.
9.3.6 Static and Dynamic Analysis Tools
For general information concerning test static and dynamic analysis tools, please refer to the ISTQB®
Foundation Level Syllabus section 6.1.6 "Tools for performance and monitoring".
9.3.6.1 Static analysis tools
Static Analysis tools can be used at any time in the software lifecycle and also at all levels/phases of the software development, depending on the measurements provided by the tool.
Static Analysis tools report their findings in terms of warnings. The warnings that are unsubstantiated are called false positives. True positives are real faults that could lead to failures during execution. It can be difficult and time-consuming to discern false from true positives since it requires proper troubleshooting. More recent static analysis tools can use information in the dynamic binding during compilation, and are therefore more powerful in finding real faults with less false positives. Technical Test Analysts use these tools.
9.3.6.2 Dynamic analysis tools
Dynamic Analysis tools provide run-time information on the state of the executing software. These tools are most commonly used to identify unassigned pointers, check pointer arithmetic, monitor the allocation, use and de-allocation of memory to flag memory leaks and highlight other errors difficult to find 'statically'. Memory tools should be re-used at more levels than one in large complex systems, since memory problems are dynamically created. Note that different commercial test tools might be implemented differently, and thus target and report different types of memory or resource (stack, heap) problems. The conclusion is that two different memory tools could identify different problems. Memory tools are particularly useful for some programming languages (C, C++) where memory management is left to the programmer. Technical Test Analysts use these tools.
9.3.7 Keyword-Driven Test Automation
Keywords (sometimes referred to as "Action Words") are mostly (but not exclusively) used to represent high-level business interactions with a system (e.g. "cancel order"). Each keyword is typically used to represent a number of detailed interactions with the system under test. Sequences of keywords (including relevant test data) are used to specify test cases.[Buwalda01]
In test automation a keyword word is implemented as one or more executable test scripts. Tools read test cases written with keywords and call the appropriate test scripts which implement them. The scripts are implemented in a highly modular manner to enable easy mapping to specific keywords.
Programming skills are needed to implement these modular scripts.
The primary advantages of keyword-driven test automation are:
Keyword-based test automation is mostly used by domain experts and Test Analysts.
9.3.8 Performance Testing Tools
For general information concerning test performance tools, please refer to the ISTQB® Foundation Level Syllabus section 6.1.6 "Tools for performance and monitoring"
Performance test tools have two main facilities:
Load generation is performed by implementing a pre-defined operational profile (see section 5.3.3) as a script. The script may initially be captured for a single user (possibly using a capture/replay tool) and then implemented for the specified operational profile using the performance test tool. This implementation must take into account the variation of data per transaction (or sets of transactions).
Performance tools generate a load by simulating large numbers of multiple users ("virtual" users) with specific volumes of input data. In comparison with capture/replay tools, many performance testing scripts reproduce user interaction with the system at the communications protocol level and not by simulating user interaction via a graphical user interface. A limited number of load generation tools can generate the load by driving the application using its user interface.
A wide range of measurements are taken by a performance test tool to enable analysis during or after execution of the test. Typical metrics taken and reports provided include:
Significant factors to consider in the implementation of performance test tools include:
Performance test tools are typically acquired due to the effort required to develop them. It may, however, be appropriate to develop a specific performance tool if technical restrictions prevent a product being used, or if the load profile and facilities to be provided are relatively simple. Performance test tools are typically used by Technical Test Analysts.
Note: performance related defects often have deep ranging impact on the SUT. When performance requirements are imperative, it is often useful to performance test the critical components (via drivers and stubs) instead of waiting for system tests.
Hyperlink test tools are used to scan and check that no broken or missing hyperlinks are present on a web site. Some tools also provide additional information such as a graph of the architecture (arborescence of the site), the speed and size of download (per URL), hits and volumes. These tools may also be helpful for monitoring SLA (Service Level Agreements) compliance. Test Analysts and Technical Test Analysts use these tools.