 |
| |
ISTQB Advanced Level Syllabi
3. Test Management
Terms:
FMEA, level test plan, master test plan, product risk, project risk, risk-based testing, risk analysis, risk identification, risk level, risk management, risk mitigation, risk type, test control, session-based test management, test estimation, test level, test management, test monitoring, test plan, test policy, test point analysis, test scheduling, test strategy, wide band delphi.
This entire chapter covers areas of knowledge required specially for test managers.
3.2 Test Management Documentation
Documentation is often produced as part of test management. While the specific names and scope of the test management documents tend to vary, the following are common types of test management documents found in organizations and on projects:
In some organizations and on some projects, these types of documents may be combined into a single document, the contents of these types of documents may be found in other documents, and some of the contents of these types of documents may be manifested as intuitive, unwritten, or traditional methodologies for testing. Larger and more formal organizations and projects tend to have all of these types of documents as written work products, while smaller and less formal organizations and projects tend to have fewer such written work products. This syllabus describes each of these types of documents separately, though in practice the organizational and project context determines the correct utilization of each type.
The test policy describes the organization’s philosophy toward testing (and possibly quality assurance). It is set down, either in writing or by management direction, laying out the overall objectives about testing that the organization wants to achieve. This policy may be developed by the Information Technology, Research and Development, or Product Development department, but should reflect the organizational values and goals as they relate to testing.
In some cases, the test policy will be complementary to or a component of a broader quality policy. This quality policy describes the overall values and goals of management related to quality.
Where a written test policy exists, it may be a short, high-level document that:
The test policy may address test activities for new development as well as for maintenance. It may also reference a standard for testing terminology to be used throughout the organization.
The test strategy describes the organization’s methods of testing, including product and project risk management, the division of testing into levels, or phases, and the high-level activities associated with testing. The test strategy, and the process and activities described in it, should be consistent with the test policy. It should provide the generic test requirements for the organization or for one or more projects.
As described in the Foundation Syllabus, test strategies (also called test approaches) may be classified based on when test design begins:
Typical strategies (or approaches) include:
Different strategies may be combined. The specific strategy selected should be appropriate to the organization’s needs and means, and organizations may tailor strategies to fit particular operations and projects.
In many instances, a test strategy explains the project and product risks and describes how the test process manages these risks. In such instances, the connection between risks and testing should be explicitly explained, as are options for reducing and managing these risks.
The test strategy may describe the test levels to be carried out. In such cases, it should give high-level guidance on the entry criteria and exit criteria of each level and the relationships among the levels (e.g., division of test coverage objectives).
The test strategy may also describe the following:
Both short and long term test strategies should be defined. This can be done in one or more documents. Different test strategies are suitable for different organizations and projects. For example, where security- or safety-critical applications are involved, a more intensive strategy may be more appropriate than in other cases.
The master test plan describes the application of the test strategy for a particular project, including the particular levels to be carried out and the relationship among those levels. The master test plan should be consistent with the test policy and strategy, and, in specific areas where it is not, should explain those deviations and exceptions. The master test plan should complement the project plan or operations guide in that it should describe the testing effort that is part of the larger project or operation.
While the specific content and structure of the master test plan varies depending on the organization, its documentation standards, and the formality of the project, typical topics for a master test plan include:
On smaller projects or operations, where only one level of testing is formalized, the master test plan and the test plan for that formalized level will often be combined into one document. For example, if system test is the only formalized level, with informal component and integration testing performed by developers and informal acceptance testing performed by customers as part of a beta test process, then the system test plan may include the elements mentioned in this section.
In addition, testing typically depends on other activities in the project. Should these activities not be sufficiently documented, particularly in terms of their influence and relationship with testing, topics related to those activities may be covered in the master test plan (or in the appropriate level test plan). For example, if the configuration management process is not documented, the test plan should specify how test objects are to be delivered to the test team.
The level test plan describes the particular activities to be carried out within each test level, where necessary expanding on the master test plan for the specific level being documented. It provides schedule, task, and milestone details not necessarily covered in the master test plan. In addition, to the extent that different standards and templates apply to specification of tests at different levels, these details would be covered in the level test plan.
On less-formal projects or operations, a single level test plan is often the only test management document which is written. In such situations, some of the informational elements mentioned in sections 3.2.1, 3.2.2 and 3.2.3 could be covered in this document.
3.3 Test Plan Documentation Templates
As mentioned in section 3.2, the specific content and structure of the master test plan varies depending on the organization, its documentation standards, and the formality of the project. Many organizations develop or adapt templates to ensure commonality and readability across projects and operations, and templates are available for test plan documentation.
The IEEE 829 "Standard for Software Testing Documentation" contains test documentation templates and guidance for applying them, including for preparing test plans. It also addresses the related topic of test item transmittal (i.e., release of test items to testing).
Estimation, as a management activity, is the creation of an approximate target for costs and completion dates associated with the activities involved in a particular operation or project. The best estimates:
Estimation of software and system engineering has long been known to be fraught with difficulties, both technical and political, though project management best practices for estimation are well established. Test estimation is the application of these best practices to the testing activities associated with a project or operation.
Test estimation should include all activities involved in the test process, i.e. test planning and control, test analysis and design, test implementation and execution, test evaluation and reporting, and test closure activities. The estimated cost, effort, and, especially, duration of test execution is often of the most interest to management, as test execution is typically on the project critical path. However, test execution estimates tend to be difficult to generate and unreliable when the overall quality of the software is low or unknown. A common practice is also to estimate the number of test cases required. Assumptions made during estimation should always be documented as part of the estimation.
Test estimation should consider all factors that can influence the cost, effort, and duration of the testing activities. These factors include (but are not limited to) the following:
Other factors that may influence the test estimate include complexity of the process, technology, organization, number of stakeholders in the testing, many sub teams, especially geographically separated; significant ramp up, training, and orientation needs; assimilation or development of new tools, techniques, custom hardware, number of testware; requirements for a high degree of detailed test specification, especially to an unfamiliar standard of documentation; complex timing of component arrival, especially for integration testing and test development; and, fragile test data (e.g., data that is time sensitive).
Estimation can be done either bottom-up or top-down. The following techniques can be used in test estimation:
In most cases, the estimate, once prepared, must be delivered to management, along with a justification (see section 3.7). Frequently, some negotiation ensues, often resulting in a reworking of the estimate. Ideally, the final test estimate represents the best-possible balance of organizational and project goals in the areas of quality, schedule, budget, and features.
In general, planning for any set of activities in advance allows for the discovery and management of risks to those activities, the careful and timely coordination with others involved, and a high-quality plan. The same is true of test planning. However, in the case of test planning, additional benefits accrue from advanced planning based on the test estimate, including:
Test scheduling should be done in close co-operation with development, since testing heavily depends on the development (delivery) schedule.
Since, by the time all the information required to complete a test plan arrives, the ability to capitalize on these potential benefits might have been lost, test plans should be developed and issued in draft form as early as possible. As further information arrives, the test plan author (typically a test manager), can add that information to the plan. This iterative approach to test plan creation, release, and review also allows the test plan(s) to serve as a vehicle to promote consensus, communication, and discussion about testing.
3.6 Test Progress Monitoring & Control
There are five primary dimensions upon which test progress is monitored:
Product risks, incidents, tests, and coverage can be and often are measured and reported in specific ways during the project or operation. Preferably these measurements are related to the defined exit criteria as stated in the test plan. Confidence, though measurable through surveys, is usually reported subjectively.
Metrics related to product risks include:
Metrics related to defects include:
Metrics related to tests include:
Metrics related to test coverage include:
These measurements may be reported verbally in narrative form, numerically in tables, or pictorially in graphs and may be used for a number of purposes, including:
Proper ways to gather, analyze, and report these test measures depends on the specific information needs, goals, and abilities of the people who will use those measurements.
When using test results to influence or measure control efforts on the project, the following options should be considered:
Implementing such options typically requires consensus among project or operation stakeholders and consent by project or operation managers.
The way a test report is set up depends largely on the target audience, e.g. project management or business management. For a project manager, having detailed information on defects is likely be of interest; to the business manager, the status of the product risks could be the key reporting issue.
The IEEE 829 "Standard for Software Testing Documentation" provides a template for reporting test summary information.
Based on divergence from the test plan as stated in the test progress report, test control should be performed. Test control is aimed at minimizing the divergence from the test plan. Possible control measures include:
While most organizations consider testing valuable in some sense, few managers, including test managers, can quantify, describe, or articulate that value. In addition, many test managers, test leads, and testers focus on the tactical details of testing (aspects specific to the task or level, while ignoring the larger strategic (higher level) issues related to testing that other project participants, especially managers, care about.
Testing delivers value to the organization, project, and/or operation in both quantitative and qualitative ways:
Test managers and test leads should understand which of these values apply for their organization, project, and/or operation, and be able to communicate about testing in terms of these values.
A well-established method for measuring the quantitative value and efficiency of testing is called cost of quality (or, sometimes, cost of poor quality). Cost of quality involves classifying project or operational costs into four categories:
A portion of the testing budget is a cost of detection, while the remainder is a cost of internal failure. The total costs of detection and internal failure are typically well below the costs of external failure, which makes testing an excellent value. By determining the costs in these four categories, test managers and test leads can create a convincing business case for testing.
3.8 Distributed, Outsourced & Insourced Testing
In many cases, not all of the test effort is carried out by a single test team, composed of fellow employees of the rest of the project team, at a single and same location as the rest of the project team. If the test effort occurs at multiple locations, that test effort may be called distributed. If the test effort is carried out at one or more locations by people who are not fellow employees of the rest of the project team and who are not co-located with the project team, that test effort may be called outsourced. If the test effort is carried out by people who are co-located with the project team but who are not fellow employees, that test effort may be called insourced.
Common across all such test efforts is the need for clear channels of communication and well-defined expectations for missions, tasks, and deliverables. The project team must rely less on informal communication channels like hallway conversations and colleagues spending social time together. Location, time-zone, cultural and language differences make these issues even more critical.
Also common across all such test efforts is the need for alignment of methodologies. If two test groups use different methodologies or the test group uses a different methodology than development or project management, that will result in significant problems, especially during test execution.
For distributed testing, the division of the test work across the multiple locations must be explicit and intelligently decided. Without such guidance, the most competent group may not do the test work they are highly qualified for. Furthermore, the test work as a whole will suffer from gaps (which increase residual quality risk on delivery) and overlap (which reduce efficiency).
Finally, for all such test efforts, it is critical that the entire project team develop and maintain trust that each of the test team(s) will carry out their roles properly in spite of organizational, cultural, language, and geographical boundaries. Lack of trust leads to inefficiencies and delays associated with verifying activities, apportioning blame for problems, and playing organizational politics.
3.9.1 Introduction to Risk-Based Testing
Risk is the possibility of an undesired outcome. Risks exist whenever some problem may occur which would decrease customer, user, participant, or stakeholder perceptions of product quality or project success.
Where the primary effect of the potential problem is on product quality, potential problems are called product risks (or quality risks). Examples include a possible reliability defect (bug) that could cause a system to crash during normal operation. Where the primary effect of the potential problem is on project success, that potential problem is called a project risk (or a planning risk). Examples include a possible staffing shortage that could delay completion of a project.
Not all risks are of equal concern. The level of risk is influenced by different factors:
In risk-based testing, testing is conducted in a manner that responds to risk in three ways:
These three types of responses to risk should occur throughout the lifecycle, not simply at the beginning and end of the project. Specifically, during the project, testers should seek to do the following:
In both cases, actions taken influence how testing responds to risk.
Risk-based testing can be seen as analogous to insurance in many ways. One buys insurance where one is worried about some potential risk, and one ignores risks that one is not worried about. Quantitative analysis similar to risk evaluation that actuaries and other insurance professionals do may be applicable, but typically risk-based testing relies on qualitative analyses.
To be able to properly carry out risk-based testing, testers should be able to identify, analyze and mitigate typical product and project risks related to safety, business and economic concerns, system and data security, and technical and political factors.
Risk management can be thought of as consisting of three primary activities:
These activities are in some sense sequential, but the need for continuous risk management mentioned in the previous and following sections means that, during most of the project, all three types of risk management activity should be used iteratively.
To be most effective, risk management includes all stakeholders on the project, though sometimes project realities result in some stakeholders acting as surrogates for other stakeholders. For example, in mass-market software development, a small sample of potential customers may be asked to help identify potential defects that would impact their use of the software most heavily; in this case the sample of potential customers serves as a surrogate for the entire eventual customer base.
Because of their particular expertise, test analysts should be actively involved in the risk identification and analysis process.
3.9.2.1 Risk Identification
For both product and project risks, testers can identify risks through one or more of the following techniques:
By calling on the broadest possible sample of stakeholders, the risk identification process is most likely to detect the largest possible number of significant risks.
In some approaches to risk identification, the process stops at the identification of the risk itself.
Certain techniques, such as Failure Mode and Effect Analysis (FMEA), require for each potential failure mode to proceed to the identification of the effects of the failure mode on the rest of the system (including upper level systems in case of Systems of Systems), and on the potential users of the system.
Other techniques, such as Hazard Analysis, require an anticipation of the source of the risk.
For a description of the use of Failure Mode Effect, Failure Mode and Effect Analysis and Criticality Analysis, see section 3.10 and [Stamatis95], [Black02], [Craig02], [Gerrard02].
3.9.2.2 Risk Analysis
While risk identification is about identifying as many pertinent risks as possible, risk analysis is the study of these identified risks. Specifically, categorizing each risk and determining the likelihood and impact associated with each risk.
Categorization of risk means placing each risk into an appropriate type. Typical quality risk types are discussed in the ISO 9126 standard of quality characteristics to classify risks. Some organizations have their own set of quality characteristics. Note that, when using checklists as a foundation of risk identification, selection of the type of risk often occurs during the identification.
Determining the level of risk typically involves assessing, for each risk item, the likelihood of occurrence and the impact upon occurrence. The likelihood of occurrence is often interpreted as the likelihood that the potential problem can exist in the system under test. In other words, it arises from technical risk.
Factors influencing technical risk include:
The impact upon occurrence is often interpreted as the severity of the effect on the users, customers, or other stakeholders. In other words, it arises from business risk. Factors influencing business risk include:
Testers can approach establishing the level of risk either quantitatively or qualitatively. If likelihood and impact can be ascertained quantitatively, one can multiply the two values together to calculate the cost of exposure. This is the expected loss associated with a particular risk.
Typically, though, the level of risk can only be ascertained qualitatively. That is, one can speak of likelihood being very high, high, medium, low, or very low, but one cannot say for certainly whether the likelihood is 90%, 75%, 50%, 25%, or 10%. This qualitative approach should not be seen as inferior to quantitative approaches; indeed, when quantitative approaches are used inappropriately, they mislead the stakeholders about the extent to which one actually understands and can manage risk. Informal approaches such as those described in [vanVeenendaal02], [Craig02] and [Black07b] are often qualitative and less rigorous.
Unless risk analysis is based upon extensive and statistically valid risk data (as is the case in the insurance industry), regardless of whether risk analysis is qualitative or quantitative, the risk analysis will be based on the perceived likelihood and impact. This means that personal perceptions and opinions about these factors will influence the determined level of risk. Project managers, programmers, users, business analysts, architects and testers typically have different perceptions and thus possibly different opinions on the level of risk for each risk item. The risk analysis process should include some way of reaching consensus or, in the worst case, establishing through dictate an agreed upon level of risk. Otherwise, risk levels cannot be used as a guide for risk mitigation activities.
3.9.2.3 Risk Mitigation
Once a risk has been identified and analyzed, there are four main possible ways to handle that risk:
The options to select one of these depend on the benefits and opportunities created by each option, as well as the cost and, potentially, additional risks associated with each option.
Mitigation strategies
In most risk-based testing strategies, risk identification, risk analysis, and the establishment of the risk mitigation activities are the foundation of the master test plan and the other test plans. The level of risk associated with each risk item determines the extent of the testing effort (i.e., mitigation action) taken to deal with each risk. Some safety related standards (e.g., FAA DO-178B/ED 12B, IEC 61508), prescribe the test techniques and degree of coverage based on the level of risk.
Project risk mitigation
If project risks are identified, they may need to be communicated to and acted upon by the project manager. Such risks are not always within the power of the testing organization to reduce. However, some project risks can and should be mitigated successfully by the test manager, such as:
Approaches to risk mitigation include early preparation of testware, pre-testing test equipment, pretesting earlier versions of the product, tougher entry criteria to testing, requirements for testability, participation in reviews of earlier project results, participation in problem and change management, and monitoring of the project progress and quality.
Product risk mitigation
When one is talking about a product (quality) risk, then testing is a form of mitigation for such risks. To the extent that one finds defects, testers reduce risk by providing the awareness of defects and opportunities to deal with them before release. To the extent testers do not find defects, testing reduces risk by ensuring that, under certain conditions (i.e., the conditions tested), the system operates correctly.
Product (quality) risks can be mitigated by non-test activities. For example, if it is identified that the requirements are not well written, an efficient solution would be to implement thorough reviews as a mitigating action, as opposed to writing and prioritizing tests that will be run once the badly written software becomes a design and actual code.
The level of risk is also used to prioritize tests. In some cases, all of the highest-risk tests are run before any lower risk tests, and tests are run in strict risk order (often called "depth-first"); in other cases, a sampling approach is used to select a sample of tests across all the identified risks using risk to weight the selection while at the same time ensuring coverage of every risk at least once (often called "breadth-first").
Other risk mitigation actions that can be used by testers include
In [Gerrard02] the concept of test effectiveness is introduced to indicate (as a percentage) how effective testing is expected to be as a risk reduction measure. One would tend not to apply testing to reduce risk where there is a low level of test effectiveness.
Whether risk-based testing proceeds depth-first or breadth-first, it is possible that the time allocated for testing might be consumed without all tests being run. Risk-based testing allows testers to report to management in terms of the remaining level of risk at this point, and allows management to decide whether to extend testing or to transfer the remaining risk onto the users, customers, help desk/technical support, and/or operational staff.
Adjusting testing for further test cycles
If there is time left to test further, any next test cycles should be adapted to a new risk analysis. The main factors here are any totally new or very much changed product risks; unstable or defect-prone areas discovered during the testing; risks from fixed defects; the attempt to concentrate testing on typical faults found during testing; and, potentially under-tested areas (low test coverage). Any new or additional test cycle should be planned using an analysis of such risks as an input. It is also a highly recommended practice have an updated risk sheet at each project milestone.
3.9.3 Risk Management in the Lifecycle
Ideally, risk management occurs throughout the entire lifecycle. If an organization has a test policy document and/or test strategy, then these should describe the general process by which product and project risks are managed in testing, and show how that risk management is integrated into and affects all stages of testing.
Risk identification and risk analysis activities can begin during the initial phase of the project, regardless of the software development lifecycle model followed. Risk mitigation activities can be planned and put into place as part of the overall test planning process. In the master test plan and/or level test plans, both project and product risks can be addressed. The type and level of risk will influence the test levels in which the risks will be addressed, the extent of effort to use in mitigating the risk, the test and other techniques to apply to reduce the risk, and the criteria by which to judge adequacy of risk mitigation.
After planning is complete, risk management, including identification, analysis, and reduction, should be ongoing in the project. This includes identifying new risks, re-assessing the level of existing risks, and evaluating the effectiveness of risk mitigation activities. To take one example, if a risk identification and analysis session occurred based on the requirements specification during the requirements phase, once the design specification is finalized, a re-evaluation of the risks should occur. To take another example, if during testing a component is found to contain considerably more than the expected number of defects, one can conclude that the likelihood of defects in this area was higher than anticipated, and thus adjust the likelihood and overall level of risk upward. This could result in an increase in the amount of testing to be performed against this component.
Once the initial risk identification and analysis activities are complete, and as the reduction activities are carried out, one can measure the degree to which risks have been reduced. This can be done by tracing test cases and discovered defects back to their associated risks. As tests are run and defects found, testers can examine the remaining, residual level of risk. This supports the use of risk-based testing in determining the right moment to release. For an example of reporting test results based on risk coverage, see [Black03].
Test reporting should address risks covered and still open, as well as benefits achieved and not yet achieved.
3.10 Failure Mode and Effects Analysis
The failure mode and effects analysis (FMEA) and a variant including criticality analysis (FMECA) are iterative activities, intended to analyze the effects and criticality of failure modes within a system. The application of these analyses to software is sometimes termed SFMEA and SFMECA where the S stands for Software. In the following sections, only FMEA is used but the information is applicable to the other three methods as well.
Testers must be able to contribute to the creation of the FMEA document. This includes understanding the purpose and application of these documents as well as being able to apply their knowledge to help determine risk factors.
FMEA should be applied:
FMEA should be scheduled as soon as preliminary information is available at a high level, and extended to lower levels as more details become available. The FMEA can be applied at any level of the system or software decomposition depending on the information available and the requirements of a program.
For each critical function, module or component, iteratively:
FMEA provides the following advantages:
The following considerations must be made when applying the FMEA:
3.11.1 Test Management Issues for Exploratory Testing
Session-based test management (SBTM) is a concept for managing exploratory testing. A session is the basic unit of testing work, uninterrupted, and focused on a specific test object with a specific test objective (the test charter). At the end of a single session, a report, typically called a session sheet is produced on the activities performed. SBTM operates within a documented process structure and produces records that complement verification documentation.
A test session can be separated into three stages:
The SBTM session sheet consists of the following:
At the end of each session the test manager holds a debriefing meeting with the team. During debriefing the manager reviews the session sheets, improves the charters, gets feedback from the testers and estimates and plans further sessions.
The agenda for debriefing session is abbreviated PROOF for the following:
3.11.2 Test Management Issues for Systems of Systems
The following issues are associated with the test management of systems of systems:
3.11.3 Test Management Issues for Safety Critical Systems
The following issues are associated with the test management of safety-critical systems:
Because of these attributes, such systems are sometimes called RAMS systems
3.11.4 Other Test Management Issues
Failure to plan for non-functional tests can put the success of an application at considerable risk. Many types of non-functional tests are, however, associated with high costs, which must be balanced against the risks.
There are many different types of non-functional tests, not all of which may be appropriate to a given application.
The following factors can influence the planning and execution of non-functional tests:
3.11.4.1 Stakeholder Requirements
Non-functional requirements are often poorly specified or even non-existent. At the planning stage, testers must be able to obtain expectation levels from affected stakeholders and evaluate the risks that these represent.
It is advisable to obtain multiple viewpoints when capturing requirements. Requirements must be elicited from stakeholders such as customers, users, operations staff and maintenance staff; otherwise some requirements are likely to be missed.
The following essentials need to be considered to improve the testability of non-functional requirements:
3.11.4.2 Required Tooling
Commercial tools or simulators are particularly relevant for performance, efficiency and some security tests. Test planning should include an estimate of the costs and timescales involved for tooling. Where specialist tools are to be used, planning should take account of learning curves for new tools or the cost of hiring external tool specialists.
The development of a complex simulator may represent a development project in its own right and should be planned as such. In particular, the planning for simulators to be used in safety-critical applications should take into account the acceptance testing and possible certification of the simulator by an independent body.
3.11.4.3 Hardware Required
Many non-functional tests require a production-like test environment in order to provide realistic measures. Depending on the size and complexity of the system under test, this can have a significant impact on the planning and funding of the tests. The cost of executing non-functional tests may be so high that only a limited amount of time is available for test execution.
For example, verifying the scalability requirements of a much-visited internet site may require the simulation of hundreds of thousands of virtual users. This may have a significant influence on hardware and tooling costs. Since these costs are typically minimized by renting (e.g. "top-up") licenses for performance tools, the available time for such tests is limited.
Performing usability tests may require the setting up of dedicated labs or conducting widespread questionnaires. These tests are typically performed only once in a development lifecycle.
Many other types of non-functional tests (e.g. security tests, performance tests) require a productionlike environment for execution. Since the cost of such environments may be high, using the production environment itself may be the only practical possibility. The timing of such test executions must be planned carefully and it is quite likely that such tests can only be executed at specific times (e.g. nighttime).
Computers and communication bandwidth should be planned for when efficiency-related tests (e.g. performance, load) are to be performed. Needs depend primarily on the number of virtual users to besimulated and the amount of network traffic they are likely to generate. Failure to account for this may result in unrepresentative performance measurements being taken.
3.11.4.4 Organizational Considerations
Non-functional tests may involve measuring the behavior of several components in a complete system (e.g. servers, databases, networks). If these components are distributed across a number of different sites and organizations, the effort required to plan and co-ordinate the tests may be significant. For example, certain software components may only be available for system testing at particular times of day or year, or organizations may only offer support for testing for a limited number of days. Failing to confirm that system components and staff from other organizations are available "on call" for testing purposes may result in severe disruption to the scheduled tests.
3.11.4.5 Communications Considerations
The ability to specify and run particular types of non-functional tests (in particular efficiency tests) may depend on an ability to modify specific communications protocols for test purposes. Care should be taken at the planning stage to ensure that this is possible (e.g. that tools provide the required compatibility).
3.11.4.6 Data Security Considerations
Specific security measures implemented for a system should be taken into account at the test planning stage to ensure that all testing activities are possible. For example, the use of data encryption may make the creation of test data and the verification of results difficult.
Data protection policies and laws may preclude the generation of virtual users on the basis of production data. Making test data anonymous may be a non-trivial task which must be planned for as part of the test implementation.