ISTQB Advanced (CTAL-TM v3.0) Mock Exam #4 — Questions & Answers
Every question in this mock exam, with the correct answer marked and a written rationale behind each option below — for reading and review, not a timed run.
Chapter 1 · Managing the Test Activities — 26 questions
Four weeks into system testing of a commercial laundry logistics platform, the…
Four weeks into system testing of a commercial laundry logistics platform, the weekly report shows that 58% of the planned test cases have been executed against a planned 80%, and that two of the three highest-rated product risks still have no executed coverage. Which action is a test control measure in the sense of the test monitoring and control activity?
Record the shortfall and the two uncovered risks in the weekly test progress report issued to the steering group.
This is test monitoring and reporting: it makes the deviation visible to stakeholders but does not change anything about how the remaining testing will be carried out.
Re-sequence the remaining execution so that the two uncovered high-risk areas are tested ahead of the lower-rated areas.
Correct. A control directive changes the planned course of the remaining test activities in response to a measured deviation, and re-prioritising execution against product risk is the standard response to uncovered high risks.
Note the shortfall in the material that will feed the lessons learned discussion during test completion for this release.
Lessons learned are gathered in the test completion activity, after execution has finished, so this does nothing to correct the deviation while the release can still be influenced.
Revise the product risk analysis so that the two uncovered areas carry a lower risk level, bringing coverage back in line with the plan.
Re-rating risks to remove a reported deviation adjusts the measurement rather than the work; the risk level should change only when the likelihood or impact themselves have changed.
Test monitoring collects and reports the measurements; test control is the set of corrective directives issued once a deviation is visible. Re-sequencing execution so that the uncovered high risks are addressed first is such a directive: it changes how the remaining work is done. Reporting the shortfall, re-rating the risks to make the chart look better, and deferring the observation to the completion report are, respectively, monitoring, a manipulation of the risk analysis, and a completion activity.
A test manager is closing out the acceptance test of a sheet-music licensing system. Which of the following belong to the test completion activities described by the syllabus?
Restoring the test environment to the state agreed with the parties that share it.
Correct. Returning or shutting down the test environment is listed among the completion activities, because shared environments otherwise carry one project's leftovers into the next.
Archiving the test cases, test data and test environment configuration so that they can be reused by the maintenance team.
Correct. Archiving or handing over the testware is an explicit completion activity, and it is what makes the effort reusable after the project closes.
Negotiating the exit criteria for the acceptance test with the business sponsor and recording them in the plan, so that the basis for closing the test is agreed before execution starts.
Defining and agreeing exit criteria is a planning activity; by the time the test is being closed out the criteria have already been applied.
Evaluating whether the agreed exit criteria have been met and recommending whether to proceed with the release.
Checking results against exit criteria and advising on release readiness sits in test monitoring and control, which runs throughout execution rather than at closure.
Holding a lessons learned session and feeding the agreed improvements into the organisation's process.
Correct. Lessons learned is a named completion activity; without it the information gathered during the project is lost at the moment it becomes usable.
Test completion covers the work that turns a finished test effort into reusable assets and organisational learning: checking which planned deliverables were produced, archiving or handing over the testware, restoring the test environment to an agreed state, and holding the lessons learned discussion. Negotiating the exit criteria and agreeing the entry conditions for the next level belong to planning, and evaluating whether exit criteria have been met is part of monitoring and control.
The CTAL-TM syllabus groups the test manager's work into three test management activities and maps them onto an international standard for test processes. Which statement describes that grouping correctly?
Test strategy, test monitoring and control, and defect management, aligned with the maturity model structure of TMMi.
Defect management is part of managing the product and the test strategy is an output of planning, so neither is one of the three activities; TMMi is a process improvement model, not a process definition.
Test planning, test implementation, and test closure, aligned with the risk management vocabulary of ISO 31000.
Test implementation is a test activity rather than a management activity, and the syllabus aligns the management activities with 29119-2 rather than with a risk management standard.
Test planning, test monitoring and control, and test completion, aligned with the test process model of ISO/IEC/IEEE 29119-2.
Correct. These are the three management activities the syllabus works with, and the alignment is with the ISO/IEC/IEEE 29119-2 test process model.
Test planning, test analysis and design, and test execution, aligned with the test documentation templates of ISO/IEC/IEEE 29119-3.
Analysis, design and execution are test activities carried out within the process rather than management activities, and 29119-3 covers documentation rather than the process model.
The syllabus condenses the seven Foundation-level activities into three management activities, test planning, test monitoring and control, and test completion, and aligns them with the test process model of ISO/IEC/IEEE 29119-2. Test analysis, design, implementation and execution remain in the test process but are not management activities in this model.
A construction-site safety platform is built by five teams working in two-week…
A construction-site safety platform is built by five teams working in two-week iterations, while the regulator still requires a signed sequential acceptance phase before each site goes live. The organisation calls this a hybrid model. According to the syllabus, which motivation for a hybrid model does this situation illustrate?
The model is not hybrid at all, because the presence of iterations makes the whole lifecycle iterative by definition.
A lifecycle that combines iterative development with a sequential phase is precisely what the syllabus calls hybrid, regardless of how much of it is iterative.
The model is a scaled Agile framework, because five teams working in parallel require a scrum-of-scrums arrangement.
Scaling practices such as a scrum-of-scrums describe how teams coordinate; they say nothing about the sequential acceptance phase that makes this lifecycle hybrid.
The organisation is in transition to Agile and the sequential acceptance phase is the part of the old lifecycle not yet converted.
A transition implies the sequential part is expected to disappear once the move completes, whereas here the phase is required by an external authority and will remain.
The mix is fit for purpose, because the context imposes a constraint that the iterative part of the lifecycle cannot absorb.
Correct. The regulator's requirement is a lasting property of the context, and the syllabus treats a deliberate, context-driven mixture as the fit-for-purpose motivation for a hybrid model.
The syllabus names two motivations for hybrid models: an organisation in transition towards Agile that has not finished the move, and an organisation that deliberately keeps a mix because that mix is fit for purpose in its context. A standing regulatory requirement for a sequential acceptance phase is a permanent contextual constraint rather than an unfinished migration, so this is the fit-for-purpose motivation.
The syllabus compares sequential and iterative lifecycles across several aspects, one of which is tools. Which statement describes the difference in the tools aspect?
In a sequential lifecycle tooling tends to be chosen centrally and used in a stable configuration, while in an iterative lifecycle it is integrated into the team's pipeline and maintained by the team.
Correct. The comparison contrasts centrally provisioned, stable tooling with tooling that the delivery team owns and evolves alongside the product.
In a sequential lifecycle the tooling must be commercially licensed, while in an iterative lifecycle the syllabus expects open-source tooling.
The business type of a tool, commercial, open-source or custom, is a separate decision discussed under tool introduction and is not determined by the lifecycle model.
In a sequential lifecycle tooling is owned by the development team, while in an iterative lifecycle a separate test organisation provisions and administers it, so the people who run the tests are rarely the people who selected the tooling.
This reverses the comparison: it is the iterative lifecycle in which the delivery team owns its tooling, and the sequential one in which provisioning is typically central.
In a sequential lifecycle tooling is confined to supporting manual testing, while in an iterative lifecycle it supports automated regression testing.
Both lifecycles use the full range of tool support; what differs is how the tooling is selected, owned and integrated, not whether automation is possible.
In a sequential lifecycle the tooling is typically selected and installed once, by or for a dedicated test organisation, and is used in a comparatively stable configuration across the project. In an iterative lifecycle the tools must support the team's own cadence, which pushes towards tooling integrated into the delivery pipeline and maintained by the team itself. The differences in roles, estimation and metrics are separate aspects of the same comparison.
A dental practice management suite is tested at component, integration, system and acceptance level. The test manager is deciding where responsibility for each activity sits. According to the syllabus, which statement about test management activities across test levels is correct?
The management activities apply from system test onwards, because component and integration testing are development activities rather than test levels.
Component and integration testing are test levels in their own right and are managed as such, even when the people carrying them out are developers.
The management activities are performed once for the project as a whole, and individual test levels simply execute the test cases assigned to them.
Each level needs its own planning, monitoring and completion; treating levels purely as execution units leaves their entry conditions and closure unmanaged.
The same three management activities are applied within each test level, with the scope of each level plan derived from the plan above it.
Correct. Planning, monitoring and control, and completion recur level by level, and each level plan takes its objectives and constraints from the higher-level plan.
The management activities are the test manager's responsibility at every level, and cannot be delegated to the teams performing the lower levels.
Responsibility for managing the lower levels is routinely delegated, with the test manager retaining oversight through the plan hierarchy rather than performing the work.
The management activities recur at every test level, but their scope and owner differ: a level plan is derived from the higher-level plan, and the person managing each level applies planning, monitoring and control, and completion within that level. The activities are not confined to system test, are not performed once for the whole project, and are not delegated wholesale to developers at the lower levels.
A test manager assessing product risks for an esports tournament platform plots each risk on a risk matrix with likelihood on one axis and impact on the other. Which statement about this qualitative approach is correct?
It removes the subjectivity of the assessment, because the position in the matrix is derived from the ratings rather than from opinion, and the derivation follows a fixed rule that every participant in the assessment can check.
The position is only as objective as the likelihood and impact ratings that produced it, and those ratings are themselves judgements made by the participants.
It places each risk in a band formed by its likelihood and impact ratings, which makes the relative ordering of risks visible to stakeholders.
Correct. The matrix is a qualitative instrument: it expresses risk level as a position in a grid of likelihood and impact bands and supports comparison between risks.
It converts each risk into a monetary figure by multiplying the probability of failure by the cost that failure would cause, which allows risks to be ranked against the budget available for mitigation.
That describes cost of exposure, a quantitative technique; a risk matrix deliberately stays with bands rather than currency amounts.
It requires the risks to have been identified in a risk workshop, because other identification techniques do not yield the two ratings, so a register built from interviews would have to be re-assessed in a workshop first.
Any identification technique, including interviews, checklists and retrospectives, can be followed by rating the risk for likelihood and impact.
A risk matrix supports qualitative assessment: risks are placed in bands rather than given numeric values, which makes the relative ordering visible and the discussion with stakeholders tractable. It does not produce a monetary figure, does not remove the subjectivity in the ratings, and is not restricted to risks identified in workshops.
A team has run risk-based testing on four consecutive releases of a charity…
A team has run risk-based testing on four consecutive releases of a charity donation processing system. The risk register has barely changed between releases: the same items are carried forward with the same ratings, and newly introduced payment features are not represented. Which of the named difficulties of risk-based testing does this illustrate?
Keen beginnings, where the risk analysis is performed thoroughly at project start and is then no longer maintained during the project, so the register gradually loses its connection to what the team is actually building.
Keen beginnings describes the decay of risk work within one project after an energetic start, whereas here the register is being carried across whole releases.
Stakeholder churn, where the people who agreed the risk ratings leave and their successors do not accept the assessment, so the ratings have to be renegotiated before they can be used again.
Stakeholder churn concerns changes in who owns the assessment, not the age of the register; nothing in the situation points to the participants changing.
Key risks are being missed, where the identification techniques used do not reach the stakeholders who know about the risk, which leaves whole areas of the product absent from the analysis.
That difficulty is about the coverage of the identification step itself; here the identification step is simply not being repeated at all.
Deja vu, where the same risk register is carried forward between releases and stops reflecting what has actually changed in the product.
Correct. Deja vu is exactly this pattern: the register is reused rather than revisited, so new features arrive untested by the risk analysis.
The syllabus names several recurring difficulties, among them deja vu, where the risk register is reused release after release and stops reflecting the product, and keen beginnings, where risk analysis is done thoroughly at the start of a project and then abandoned. A stale register carried across releases with new functionality missing is the deja vu difficulty.
A test manager is choosing the test approach for the first release of an HVAC building management product. Which statement describes what choosing a test approach means in the syllabus?
Agreeing with the development team which of the known defects will be fixed before the release is allowed to proceed, and recording the remaining ones as the residual risk that the release carries.
Deciding which defects to fix is defect management and release decision work, carried out once defects exist rather than when the approach is chosen.
Tailoring the organisational test strategy to this project, by deciding how testing will be performed given the project's risks and context.
Correct. The approach is the project-level implementation of the organisational strategy, and the tailoring is driven by the project's own risks and constraints.
Writing the organisation's test strategy, so that later projects in the same domain can adopt the same approach without tailoring, which is how the organisation keeps its practice consistent across a whole product line.
The organisational test strategy is written once at organisation level and outlives the project; the approach is what an individual project derives from it.
Producing the detailed schedule of test activities, with dates, dependencies and the people assigned to each activity, so that the effort estimated for each activity can be tracked against what is actually spent.
Scheduling and resource assignment are parts of the test plan that follow from the approach rather than constituting it.
The test approach is the way the organisational test strategy is tailored and implemented for this particular project: it fixes how the testing will be carried out, which techniques, levels and degrees of formality apply, given the project's context and risks. It is neither the organisation-wide strategy itself nor the schedule, and it is not the list of defects to be fixed.
An organisation testing a maritime weather routing service wants to use TPI NEXT rather than TMMi for its improvement programme. Which statement describes the structure of TPI NEXT correctly?
Sixteen process areas derived from CMMI, each of which must be fully satisfied before the next process area can be assessed, giving a strictly sequential improvement path through the model.
TPI NEXT is not derived from CMMI and does not require sequential completion; it is TMMi that is designed to complement CMMI.
Sixteen key areas, each assessed against checkpoints and placed at one of four maturity levels, with the result shown in a maturity matrix.
Correct. This is the TPI NEXT structure, and the per-key-area profile in the maturity matrix is what distinguishes it from a single staged level.
Four key areas aligned with the four areas of competence, each assessed through a skills matrix completed by the test team, which links the maturity of the process directly to the capability of the people.
This mixes up process improvement with the competence model used for team development; TPI NEXT assesses the process, not individual skills.
Five staged maturity levels for the test process as a whole, each containing process areas with specific and generic goals, so that an organisation can state its test process maturity as a single number.
That is the structure of TMMi, which assigns the organisation one overall maturity level rather than a profile across key areas.
TPI NEXT describes the test process through sixteen key areas, each of which is assessed at one of four maturity levels using checkpoints, and the results are shown in a maturity matrix. TMMi, by contrast, defines five staged maturity levels for the process as a whole with process areas inside them. The two models therefore differ in granularity: TPI NEXT gives a per-key-area profile rather than a single overall level.
A test manager applying analytical-based improvement to a laboratory information management system distinguishes between measures, metrics and indicators. Which statement reflects that distinction as the syllabus uses it?
A measure is the raw value observed, a metric is the quantity defined from it, and an indicator relates that quantity to a goal so that it can be interpreted.
Correct. The three terms form a chain from observation to defined quantity to interpretation against a goal, which is what makes an indicator actionable.
A measure is the goal to be reached, a metric is the raw value observed, and an indicator is the technique used to collect the value, so the three terms describe the target, the observation and the collection method in that order.
This scrambles the chain: the goal belongs to the interpretation step, and collection techniques are not what the word indicator denotes.
A measure applies to the product, a metric applies to the process, and an indicator applies to the project as a whole.
Product, process and project are the three categories of metric, which is a different classification from the measure, metric, indicator chain.
A measure is a qualitative judgement, a metric is a quantitative value, and an indicator is the average of several metrics over time, which is why a trend line is usually described as an indicator rather than as a metric.
Measures are not inherently qualitative and an indicator is not an average; an indicator is a metric read against a target or threshold.
A measure is the raw value obtained by measuring something, a metric is the measurement scale or derived quantity defined for a purpose, and an indicator is the metric interpreted against a goal or threshold so that it says something about the process. Root cause analysis techniques such as Ishikawa diagrams are separate instruments used alongside this analysis.
Before buying a test management tool for an archive digitisation programme, the test manager runs the candidate tool against a small but representative part of the real work for two weeks, with the people who will use it. Which good practice for tool introduction does this correspond to?
A pilot, in which the tool is used on real work by the intended users so that its practical fit and the cost of introducing it become visible.
Correct. A pilot exercises the tool in the real working context with the real users, which is what reveals the practical fit and the introduction effort.
A proof of concept, in which the technical feasibility of using the tool in the organisation's environment is established, typically on a small example prepared by the vendor rather than on the organisation's own work.
A proof of concept answers whether the tool can work at all in the technical environment and usually precedes the pilot; here real work and real users are involved.
A tool evaluation against requirements, in which candidate tools are scored against a weighted list of criteria on paper, with the weights agreed in advance by the stakeholders who will use the tool.
Scoring candidates against criteria is a desk exercise carried out earlier; it does not involve running the tool on the organisation's own work.
A retirement assessment, in which the organisation decides whether the incumbent tool should be withdrawn from service, weighing the cost of continued support against the benefit the tool still delivers.
Retirement is the final stage of the tool lifecycle and concerns a tool already in use, not a candidate that has not yet been acquired.
The syllabus distinguishes a proof of concept, which establishes whether the tool can do the job at all in the organisation's technical context, from a pilot, which uses the tool on real work with real users to learn how it behaves in practice and what the introduction will cost. Running representative work with the intended users for a fixed period is a pilot.
A test manager is documenting how the organisation will look after the test automation tool it has just acquired for an ad-tech bidding platform. Which statements about the tool lifecycle and the associated roles are consistent with the syllabus?
The lifecycle continues after acquisition through support and maintenance, evolution, and finally retirement.
Correct. Acquisition is only the first stage; the syllabus treats support and maintenance, evolution and retirement as stages that have to be planned for.
The tool administrator is responsible for installation, configuration, user access and upgrades of the tool.
Correct. The administrator is the operational role, and separating it from the owner keeps day-to-day custody distinct from business accountability.
Evolution of the tool is an unplanned stage that occurs when a defect in the tool has to be worked around.
Evolution covers planned change such as version upgrades and adaptation to new technology, and it is budgeted for rather than treated as an incident.
The tool owner carries business responsibility for the tool, including the business case and the decision to withdraw it.
Correct. The tool owner is the business-side role that answers for the tool's value over its life, which is why the retirement decision sits there.
Once the tool has been introduced successfully, the tool owner role is no longer required and can be discontinued, since the administrator can handle whatever the tool needs from that point on.
The owner role spans the whole lifecycle; dropping it after introduction leaves no one accountable for whether the tool still earns its cost.
The tool lifecycle runs through acquisition, support and maintenance, evolution, and retirement. The tool owner holds business responsibility for the tool across that lifecycle, including the business case and the decision to retire it, while the tool administrator looks after the installation, configuration, access and upgrades. Neither role disappears after acquisition, and evolution is a planned stage rather than an exception.
A test manager reports on the value of the static analysis tool and the defect management tool used on a carbon credit registry. Which pairing of tool and metric is appropriate according to the syllabus?
Static analysis tool: number of licences in use; defect management tool: number of configured workflow statuses.
Both figures describe how the tool is installed rather than what it contributes, so neither supports a statement about the tool's value.
Static analysis tool: defect lead time; defect management tool: defect detection percentage, pairing each tool with the figure that its own reports produce most readily.
This reverses the pairing: a static analysis tool does not resolve defects, and a defect management tool does not detect them.
Static analysis tool: defect detection percentage; defect management tool: defect lead time.
Correct. Detection percentage measures how effective a detection mechanism is, and lead time measures how quickly the managed defect flow resolves items.
Static analysis tool: test case execution rate; defect management tool: requirements coverage achieved.
Execution rate belongs to an execution tool and coverage to a coverage measurement tool, so neither metric matches the tool it is paired with here.
Tool metrics should reflect what the tool is for. Defect detection percentage expresses how many of the defects present were found by a given detection mechanism and is a sensible metric for a defect detection tool such as static analysis. Defect lead time, the time from a defect being reported to it being resolved, speaks to the defect management tool and the process around it. Licence counts and lines of configuration say nothing about value.
A programme test plan for a stadium turnstile access system states the…
A programme test plan for a stadium turnstile access system states the objective: 'The system test shall demonstrate that the turnstile firmware is reliable enough for live operation.' The test manager must restate this as an objective for the system level plan so that it satisfies the S.M.A.R.T. criteria in the plan hierarchy. Which restatement is the best choice?
The system test shall achieve the highest reliability that can be obtained within the time and budget allocated to the test level in the programme plan, measured by the number of reliability test cases that pass within that allocation.
An objective phrased as the best achievable is not measurable: any outcome satisfies it, so it cannot support an exit decision.
The system test shall confirm that the turnstile firmware is reliable enough to be operated at full capacity on match days throughout the coming season, which is the level of assurance the programme plan asks the system test level to provide.
This repeats the vague term reliable and sets a horizon the system test cannot observe, so there is no threshold against which the level can be closed.
The test team shall execute all reliability test cases designed for the turnstile firmware and record the results in the test management tool.
This states an activity to be performed rather than an outcome to be reached, so it does not express what the level is trying to achieve.
By the end of system test entry week 6, the turnstile firmware shall complete 50,000 consecutive simulated entry cycles with no more than two failed reads.
Correct. It names the item, gives a measurable threshold over a defined volume, and fixes the point in the schedule by which it must hold, satisfying each of the S.M.A.R.T. criteria at the level the plan can verify.
A S.M.A.R.T. objective is specific, measurable, achievable, relevant and time-bound. The chosen statement names what is tested, states a measurable threshold and the period it is measured over, and is tied to a point in the schedule. The alternatives either keep the vague term reliable, state a target that the level plan cannot verify within its own scope, or state an activity rather than an objective.
A vertical farm climate control programme has a master test plan covering three releases, a level plan for system test of each release, and an iteration plan for each two-week iteration. The test manager must place the following objectives in the right plan. Which two belong in the iteration plan?
System test for a release is not closed while any high-rated product risk has no executed test coverage.
This is an exit criterion for the system test level as a whole, so it belongs in the level plan rather than in any single iteration plan.
The regulator receives a signed test completion report for each release before the corresponding site goes live, and the report is compiled from the evidence produced during the iterations.
This is a release-level commitment to an external party spanning the whole programme, so it belongs in the master test plan.
All acceptance criteria of the stories accepted into iteration 7 are covered by at least one executed test before the iteration review.
Correct. The scope is the content of one iteration and the deadline is the iteration review, so the objective can only live in the iteration plan.
The automated regression suite runs to completion on the integration environment at least once within each iteration.
Correct. The commitment is framed per iteration and is verified inside the iteration, which places it in the iteration plan.
A climate simulation environment comparable to the production glasshouse is available for all three releases.
An environment commitment spanning three releases is a cross-release concern and belongs in the master test plan.
The plan hierarchy puts long-lived, cross-release commitments in the master plan, level-wide objectives in the level plan, and objectives whose scope and horizon are a single iteration in the iteration plan. Objectives tied to the stories in one iteration and to the regression suite executed within that iteration belong in the iteration plan; the agreement with the regulator, the cross-release environment strategy and the level-wide exit criterion do not.
A test manager facilitates a retrospective for a prosthetics manufacturing…
A test manager facilitates a retrospective for a prosthetics manufacturing workshop system. The session opens with a reminder that the discussion is not about blame, the group then lists five improvements it would like, and only afterwards does the facilitator show the defect and cycle-time data from the iteration. The group keeps the five improvements it had already named, picks two, and closes. Which statement best describes what went wrong against the five-step retrospective process?
Just two of the five proposed improvements were selected, whereas the process expects all derived improvements to become actions.
Deciding on improvement actions explicitly involves selecting a feasible subset; carrying every proposal forward is what makes retrospective actions go unfinished.
The facilitator opened with a reminder about blame, which belongs to the closing step rather than to the introduction.
Setting the ground rules, including the absence of blame, is part of the introduction; it is what makes the data collection step honest.
Improvements were derived before the data was collected, so the data had no opportunity to shape what the group proposed.
Correct. Collect data precedes derive improvements precisely so that the proposals answer what actually happened, and showing the data afterwards leaves the anchoring in place.
The session was closed immediately after deciding on actions, without a separate step to communicate the outcome to stakeholders.
Closing the retrospective after the decision step is the prescribed sequence; communication of the agreed actions is part of closing rather than a missing sixth step.
The five steps run in a fixed order: introduce the session, collect data, derive improvements from that data, decide on improvement actions, and close. Here improvements were derived before the data was collected, so the data could not influence what was proposed; presenting it afterwards does not repair the sequence because the group had already anchored on its list.
A retrospective for a subtitling workflow product has the following agenda items. Which two belong to the 'derive improvements' step of the five-step process?
Assigning an owner and a target date to each of the two changes the group has selected.
Owners and dates are attached in the decide on improvement actions step, after the candidates have been derived and chosen.
Reminding the participants that the session looks at the process rather than at individuals.
Ground rules are stated in the introduction step, which prepares the session rather than producing improvements.
Collecting from each participant the three events in the iteration that cost them the most time, written on cards so that they can be grouped later in the session.
Gathering observations from participants is the collect data step, which has to be complete before improvements are derived.
Proposing concrete changes to the way test data is requested, each one tied to one of the identified causes.
Correct. Candidate changes derived from the identified causes are the output of this step, before any selection is made.
Grouping the collected observations into likely causes and naming which part of the process each cause sits in.
Correct. Turning raw observations into causes is the analytical work that the derive improvements step exists for.
Deriving improvements means working from the collected data to candidate changes: grouping observations into causes and proposing concrete changes that would address them. Stating the ground rules belongs to the introduction, gathering the raw observations belongs to collect data, and assigning owners and dates belongs to deciding on improvement actions.
A test manager is defining the test scope for the release of a canal lock…
A test manager is defining the test scope for the release of a canal lock control system. The lock gates are driven by a supplier's programmable controller delivered under a contract that places all verification of the controller with the supplier. The programme's highest-rated product risk is that the gate fails to stop at the open position when the water level sensor reports an out-of-range value, which involves both the supplier's controller and the operator console built in-house. The supplier will provide a test report but will not accept additional test requests within this release. What is the most appropriate scope decision?
Escalate to the sponsor and hold the release decision until the contract is amended so that additional supplier test requests become possible.
Escalation may be worthwhile in parallel, but it is not a scope decision and it leaves the test manager without a plan for the release that is currently being tested.
Keep the interface behaviour in scope, exercise it from the console side with simulated out-of-range sensor values, and record the supplier-internal part as residual risk.
Correct. The risk lives at the interface, which the in-house side can exercise, and the part that genuinely cannot be covered is reported as residual risk rather than silently dropped.
Re-test the supplier's controller in full within the in-house system test, so that the programme does not depend on the supplier's own verification evidence.
The programme has neither the access nor the mandate to repeat the supplier's internal verification, and duplicating it would consume the budget needed for the interface that is actually reachable.
Place the whole risk out of scope for this release, since contractual responsibility for the controller lies with the supplier and the supplier will issue a report, which the programme can attach to its own test completion report as supplier evidence.
Contractual responsibility allocates work, not risk; leaving the highest-rated product risk untested because of a contract boundary is exactly the failure the risk analysis exists to prevent.
Scope definition has to follow the risk, not the contract boundary. The risk spans the interface between the supplier's controller and the in-house console, so the test manager keeps the interface behaviour in scope and tests it from the in-house side with simulated out-of-range sensor values, while taking the supplier's report as an input and recording the part that cannot be covered as residual risk. Removing the risk from scope because one component is contracted out leaves the highest risk unaddressed, and duplicating the supplier's internal verification is both out of reach and not where the risk sits.
Two test levels feed the same quality gate for an aquarium life-support…
Two test levels feed the same quality gate for an aquarium life-support monitoring system. Integration test reports all its exit criteria met. System test reports 94% of planned cases executed, all high-rated risks covered, and two open defects: one of high severity in the alarm escalation path with a workaround that requires an operator to be present, and one of low severity in a report layout. The gate criterion reads: no open high-severity defects and at least 95% execution. The release is needed for a regulatory inspection in four days. What should the test manager do with the gate?
Reject the release outright and schedule a further two weeks of system test to reach the execution threshold before the inspection.
The gate decision is not the test manager's to make unilaterally, and a schedule that misses the inspection imposes a business consequence the accountable stakeholder has not weighed.
Report the gate as not passed, state that the alarm defect is only tolerable while an operator is continuously present, and put the release decision to the accountable stakeholder.
Correct. The test manager applies the criteria, makes the operational condition attached to the workaround visible, and leaves the decision with the party that carries the consequence.
Pass the gate, because the integration level met all its criteria and the system level is within one percentage point of the execution threshold, so taken together the two levels put the programme substantially where the plan expected it to be at this point.
Averaging across levels and waiving a high-severity breach turns the gate into a formality; the execution shortfall is also not the only criterion that failed.
Pass the gate conditionally and record the alarm defect as residual risk, since a documented workaround exists for it.
A workaround that depends on a person being present is not equivalent to a fix, and recording it as residual risk without the stakeholder's decision transfers an operational commitment they have not accepted.
Quality gate management means applying the criteria and, where they are not met, making the deviation and its consequences explicit so that the accountable party decides. Here two criteria are breached. The alarm escalation defect matters because its workaround assumes continuous operator presence, which is an operational condition rather than a fix, so the right move is to report the gate as not passed, state the condition under which the risk would be acceptable, and put the decision to the accountable stakeholder.
A risk-based system test of a hydrogen refuelling network has eleven risk areas…
A risk-based system test of a hydrogen refuelling network has eleven risk areas rated from very high to low. The sponsor has asked for a weekly statement of residual risk level that is meaningful at any point during the six-week execution window, because a go-live decision may be brought forward at short notice. Which execution order serves that requirement best, and why?
Breadth-first, because it distributes the test effort evenly and therefore reduces the overall residual risk faster than depth-first.
Even distribution ignores the rating: effort spent on low-rated areas reduces residual risk more slowly than the same effort spent on the highest-rated ones.
Depth-first, because completing the highest risk areas first means the residual risk at any cut-off is confined to the lower-rated areas.
Correct. Depth-first makes the residual statement interpretable at every point: what remains uncovered is, by construction, what was rated least important.
Depth-first, because working steadily down the ordered list means the lower-rated risk areas are still reached well before the end of the six-week window.
Depth-first says nothing about whether the lower areas will be reached at all; that is precisely the trade-off the sponsor is being asked to accept.
Breadth-first, because touching every risk area early means no area is entirely without evidence if the decision is brought forward.
Shallow coverage everywhere produces evidence that is too thin to support a claim about any area, so the residual risk statement becomes uniformly uncertain rather than conservative.
Depth-first execution works through the highest risk areas to completion before moving down the list, so at any moment the areas already finished are the most important ones and the residual risk statement is both meaningful and conservative. Breadth-first spreads shallow coverage across all areas first, which gives a broad but uniformly thin picture and makes it hard to say that any area is adequately covered if the decision is brought forward.
A bicycle manufacturing line includes a press whose control software has been…
A bicycle manufacturing line includes a press whose control software has been assigned a safety integrity level under IEC 61508. Analysis shows one residual hazard: if two guard sensors fail in the same cycle, the press can close while the frame jig is still being loaded. The techniques prescribed for the assigned integrity level have all been applied and the hazard remains credible, though very unlikely. Testing cannot reduce it further. Which risk treatment should the test manager recommend and how should it be reported?
Transfer the risk by extending the manufacturer's liability insurance to cover injuries arising from an unexpected press closure.
Risk transfer moves the financial consequence but leaves the hazard to the operator unchanged, which is not an acceptable treatment for a safety hazard of this kind.
Accept the risk without further measures and omit it from the report, since every technique prescribed for the integrity level has already been applied, and the integrity level is the standard's own statement of what is sufficient for equipment of this class.
Applying the prescribed techniques discharges the method obligation but not the duty to report what remains; silent acceptance removes the decision from the party that carries it.
Apply additional test techniques beyond those prescribed for the integrity level until the hazard can no longer be demonstrated in the laboratory, and record the additional techniques as an improvement to the organisation's safety test approach.
The analysis already states that testing cannot reduce the hazard further, and failing to demonstrate it is not evidence that it has been removed.
Propose a contingency measure outside the software, such as an independent mechanical interlock, and report the remaining residual risk explicitly for acceptance.
Correct. With testing exhausted, a contingency measure that does not depend on the control software is the appropriate treatment, and the residual part still has to be accepted knowingly.
Testing is the main mitigation measure, but when it has been exhausted the remaining options are a contingency plan, transferring the risk, or accepting it. Here a physical interlock that stops the press independently of the control software is a contingency measure outside the software, and it is proposed together with an explicit statement of the remaining residual risk so that the accountable party accepts what is left. Inventing techniques beyond those prescribed, insuring the hazard, or suppressing it because the prescribed techniques were applied are all inadequate responses.
A test manager analyses the organisational test strategy against a new project…
A test manager analyses the organisational test strategy against a new project: a music licensing settlement engine that must reconcile with four external collecting societies, each of which exposes a different file interface and releases test data only in quarterly windows. The team is experienced, the domain is well understood, the delivery model is the organisation's standard two-week iteration, and the budget is normal. Which two of the seven factors for defining the project test strategy most constrain this project?
The domain, because settlement between rights holders is a well understood part of the organisation's business.
A well understood domain is explicitly the easy case; the factor binds when the domain is unfamiliar or heavily regulated in ways the organisation has not met before.
The SDLC model, because the two-week iteration cadence is the organisation's standard delivery model.
The lifecycle model is described as the organisation's normal one, so it imposes no constraint beyond what the strategy already assumes.
Interfaces with other systems, because four external parties each expose a different file interface that has to be exercised.
Correct. Heterogeneous external interfaces drive the levels, environments and stubs the strategy must provide for, and they cannot be simplified away.
Availability of test data, because the collecting societies release usable data only in quarterly windows.
Correct. A quarterly data window dictates when meaningful testing can happen and forces the strategy to plan around those dates.
Test resources, because the team is experienced and the budget is at the usual level for a project of this size, which determines how much of the strategy can actually be carried out.
Experienced staff and a normal budget are the baseline the strategy is written for; they are not a constraint that has to be designed around.
The seven factors are the domain, organisational goals, project goals and type, test resources, the SDLC model, interfaces with other systems, and the availability of test data. In this project the four heterogeneous external interfaces and the quarterly release windows for test data are the binding constraints; the domain, the lifecycle model and the team's experience are all described as ordinary.
The organisational test strategy of a publishing group states that every system…
The organisational test strategy of a publishing group states that every system test shall run against a full copy of production data refreshed weekly. A new project builds a rights clearance service for a jurisdiction whose data protection rules forbid moving author contact data outside the production environment. The test manager must analyse the strategy against the project and recommend what the project test strategy should say. Which recommendation is best?
Follow the organisational strategy as written and apply for an exemption from the data protection rule for the duration of the project, on the grounds that the copy stays inside the organisation's own test environment and is deleted at the end of the project.
The rule is a legal constraint on the project, not an internal policy to be waived, and planning around an exemption that may not be granted leaves the project without a strategy.
Defer the project test strategy until the organisational test strategy has been amended to cover jurisdictions with this restriction.
The organisational strategy is revised on its own cycle, and a project may not suspend its planning while waiting; tailoring with a recorded deviation is the mechanism for exactly this case.
Keep the intent of production-like data but replace the mechanism with masked or generated data matching the production profile, and record the deviation and its reason in the project test strategy.
Correct. Tailoring means preserving what the organisational strategy is trying to achieve while substituting a mechanism that is lawful in this context, with the deviation documented.
Drop the production-like data requirement for this project and rely on test data designed from the requirements alone.
This discards the intent rather than the mechanism: data designed only from requirements misses the distribution and edge cases that production-like data exists to expose.
The organisational strategy expresses an intent, realistic production-like data, through a mechanism, a full weekly copy, that this project cannot use. The analysis should preserve the intent and change the mechanism: generate or mask data that reproduces the production profile for the fields that matter, state the deviation and its reason in the project test strategy, and feed the case back so the organisational strategy can accommodate it. Ignoring the rule, abandoning realistic data, or waiting for the organisational strategy to be rewritten all fail the project.
A test manager prepares the cost side of the business case for a commercial test…
A test manager prepares the cost side of the business case for a commercial test execution tool for a flight simulator training records system. The figures are: licences 24,000 per year; initial installation and integration with the pipeline 18,000, one off; training for eight testers 9,600, one off; tool administration 0.2 of a full-time post, costed at 14,000 per year; and a delay of one month to the regression automation project because the two senior testers will be occupied with the introduction, valued at 11,000. Which classification of these amounts is correct?
Non-recurring 27,600; recurring 38,000 per year; opportunity cost 11,000.
Correct. Installation and initial training are the one-off costs, licences and administration repeat annually, and the deferred automation project is the value forgone.
Non-recurring 18,000; recurring 47,600 per year; opportunity cost 11,000, treating the training of the eight testers as an annual refresher cost.
This treats the initial training as a recurring cost, but the 9,600 is stated as training for the eight testers at introduction rather than an annual programme.
Non-recurring 27,600; recurring 38,000 per year; opportunity cost nil, because the delay affects an internal project rather than a customer commitment.
An opportunity cost is precisely the value of the internal alternative that is given up, so an internally borne delay is what the category is for.
Non-recurring 38,600; recurring 24,000 per year; opportunity cost 11,000, treating the administration share of a post as part of the initial set-up.
This moves tool administration into the one-off bucket, yet 0.2 of a post costed per year is the clearest example of a recurring cost in the list.
Non-recurring costs are incurred once to bring the tool into service: installation and integration, 18,000, and initial training, 9,600, giving 27,600. Recurring costs repeat every period: licences, 24,000, and administration, 14,000, giving 38,000 per year. The deferred regression automation is an opportunity cost, the value of what the organisation gives up by committing the two testers to the introduction, 11,000.
Eighteen months after an antique auction house introduced a test automation…
Eighteen months after an antique auction house introduced a test automation tool, the predicted benefits have not appeared: the suite runs nightly but roughly a third of the failures each morning turn out to be caused by the suite itself, no one has updated the scripts for the two interface changes made last year, and the business case was signed off by a project manager who has since moved on. Which analysis of the ROI shortfall is best supported?
The licence cost was underestimated in the business case, which is why the realised return is below the predicted return, and the annual renewal has risen faster than the figure assumed when the tool was acquired.
Nothing indicates the licence figure was wrong, and a cost error would not explain why a third of the failures are produced by the suite itself.
The tool owner role lapsed, so the maintenance the business case assumed was never funded, and unmaintained scripts produce false failures that erode the benefit.
Correct. The benefit in the business case was conditional on maintained assets, and the role responsible for defending that condition over the tool's life no longer exists.
The testers lack the skills to use the tool, which is why the interface changes were not reflected in the scripts, and the training given at introduction covered the recording features rather than script maintenance.
The scripts were written and run successfully for a year and a half, so capability is not the constraint; what is missing is the allocated time and accountability to maintain them.
The wrong business type of tool was chosen, and an open-source tool would not have shown this shortfall.
Scripts decay for the same reason whichever business type the tool has; the open-source option changes the licence cost, not the maintenance obligation.
The syllabus attaches the ROI risks of a tool to the absence of ongoing ownership and maintenance: a tool whose assets are not maintained produces false failures, loses trust and therefore stops delivering the benefit that was costed. The tool owner role exists to carry the business case through the tool's life; when it lapses, the maintenance effort that the recurring costs were meant to fund is simply not spent. The failure is therefore one of ownership and maintenance rather than of tool selection, licence spend or staff ability.
Chapter 2 · Managing the Product — 15 questions
A monthly report for a scuba dive-centre booking platform contains four figures…
A monthly report for a scuba dive-centre booking platform contains four figures: the number of test cases implemented against those planned, the number of defects found in the booking module per thousand lines of changed code, the mean time from a defect being reported to it being assigned, and the proportion of the test budget consumed. Which figure is a product metric in the syllabus classification?
The number of defects found in the booking module per thousand lines of changed code.
Correct. Defect density expresses a quality property of the item under test, which is what makes it a product metric.
The number of test cases implemented against the number planned for this month.
Progress against plan describes how the project is advancing rather than a property of the product, so this is a project metric.
The proportion of the allocated test budget that has been consumed by the end of the month.
Budget consumption is a project metric: it tracks the use of project resources and says nothing about the quality of the product.
The mean time from a defect being reported to it being assigned to a developer.
This measures how quickly the defect management process moves an item along, which places it among the process metrics.
The syllabus separates project metrics, which describe progress against the plan and consumption of project resources, product metrics, which describe a property of the item under test, and process metrics, which describe how the development or test process behaves. Defect density in a module is a property of the product; implementation progress and budget consumption are project metrics, and the time to assign a reported defect describes the defect management process.
The weekly report for an orchestral tour scheduling system shows that 61% of planned test cases have passed. The test manager adds the same figure for each of the previous five weeks. What does adding the earlier figures provide that the single value does not?
An explanation of why the pass rate is at 61%, because the earlier values identify the cause of the shortfall.
A trend shows that something changed and roughly when, but the cause has to be established separately, for instance by root cause analysis.
A more accurate value for this week, because averaging six weeks of measurements reduces the measurement error.
The current week's figure is unchanged by the earlier ones, and the point of a trend is the movement over time rather than a smoothed single number.
A trend, which shows the direction and rate of change and therefore supports a forecast of when the criterion will be met.
Correct. The series turns a snapshot into a trend, and it is the direction and rate of change that allow a forecast and a timely control decision.
A different metric, because a series of pass rates measures test effectiveness whereas a single pass rate measures test progress.
The measurement is the same in both cases; presenting it as a series changes how it is read, not what it measures.
A single value is a snapshot: it states where the project stands at one moment. A series of the same measurement over time is a trend, which shows the direction and rate of change and therefore supports a forecast and an earlier control decision. The series does not make the measurement more accurate, does not change what is being measured, and does not by itself explain the cause of the movement.
A test manager estimating the test effort for a seed bank registry lists the activities the estimate must cover. Which set of activities reflects what the syllabus expects an estimate of testing to include?
Test execution and defect reporting, because analysis and design are carried out by the business analysts who write the requirements, and their effort is already accounted for in the development estimate for the release.
Even where analysts contribute, test analysis and design are test activities that consume test effort and belong in the estimate.
Test execution and the effort needed to fix the defects that execution will reveal, estimated from historical defect counts, so that the estimate covers everything that happens between the start and the end of the test phase.
Fixing defects is development effort; the test estimate covers retesting the fixes rather than making them.
First execution of each planned test case, plus a contingency percentage applied to the total to absorb everything else.
A blanket contingency hides the activities instead of estimating them, which is why re-execution and environment work routinely overrun.
Test analysis and design, implementation including test data and environment preparation, execution, retesting and regression testing, and reporting and completion.
Correct. The estimate has to span the whole test process, and retesting, regression and environment preparation are the parts most often left out.
Estimating what activities testing will involve means covering the whole test process, not just execution: analysis and design of tests, implementation including test data and environment preparation, execution, the retesting and regression testing that follow defect fixes, and the reporting and completion work. Limiting the estimate to first-pass execution is the classic omission the syllabus warns about.
A test manager reviews what pushed the test effort on a film post-production…
A test manager reviews what pushed the test effort on a film post-production pipeline above the estimate. The syllabus groups the factors that influence test effort into product, development process, people, test results and test context. Which two of the following belong to the 'test results' group?
The experience of the testers with the domain and with the tools in use.
Skill and experience are properties of the team, which the syllabus places in the people group.
The proportion of fixes that fail retest and have to be re-reported and tested again.
Correct. Failed fixes are a direct product of the test results and multiply the execution effort for the same scope.
The volatility of the requirements for the colour grading module during the project.
Requirement stability is a characteristic of the product being tested and belongs to the product group.
The number and severity of the defects found, which drive how much retesting and regression testing is needed.
Correct. Defect counts and their severity are results of testing, and the rework they generate is the main way results feed back into effort.
The maturity of the development process and the discipline of the configuration management in place, both of which shape how much rework reaches the test team.
These describe how the software is produced, which is the development process group.
The test results group covers influences that arise from what testing finds: the number and severity of the defects detected, and the amount of rework, retesting and regression testing they cause. Team experience belongs to people, the stability of the requirements belongs to the product, and the maturity of the development process belongs to the development process group.
A test manager reviews the defect workflow proposed for an amateur radio licensing registry. Which property of the proposed workflow would conflict with the good practices the syllabus lists for a defect lifecycle?
The workflow gives the status IN PROGRESS three outgoing transitions, to RESOLVED, to CLARIFICATION and to DEFERRED.
Several outgoing transitions from a non-terminal status is the recommended shape; it is a single outgoing transition that would be the defect.
The workflow requires a reason to be recorded whenever an item moves from RESOLVED back to IN PROGRESS.
Recording the reason for a re-opening is good practice, since the re-opened rate is one of the indicators used to judge the quality of debugging.
The workflow assigns the transition into RESOLVED to the developer and the transition out of it to the tester.
Different roles on adjacent transitions is exactly the segregation the good practices call for.
The workflow has two terminal statuses, CLOSED and REJECTED, with no transition out of either.
Correct. The good practice is a single terminal status, because two of them make the counts of finished items ambiguous and split the reporting.
Among the good practices are: exactly one terminal status, so that it is unambiguous when an item is finished; more than one outgoing transition from each non-terminal status, so that an item is never stuck on a single path; and different roles on adjacent statuses, so that no one hands work to themselves. A workflow with two terminal statuses breaks the first of these.
A defect management committee has been set up for a national theatre box-office system, and a triage meeting is held twice a week. Which description of the two matches the syllabus?
The committee owns the defect management process and settles disputed rules; the triage meeting assesses and prioritises individual defect reports.
Correct. One body governs the process, the other applies it to the items currently in the flow, and keeping them apart stops rule changes being made case by case.
The committee assesses and prioritises individual defect reports; the triage meeting reviews the defect management process every quarter, so that changes to the process are made on a fixed cycle rather than case by case.
This swaps the two: the recurring per-item work is the triage meeting, and process governance is the committee's standing task.
The committee and the triage meeting are two names for the same body, distinguished simply by how often it convenes.
They differ in purpose, not just in frequency: governance of the process against assessment of individual reports.
The committee decides which defects the developers will fix next; the triage meeting records the decisions in the defect management tool, which keeps the audit trail of prioritisation decisions complete and attributable.
Recording decisions is administrative work, typically the defect manager's, and the committee's role is the process rather than the queue.
The defect management committee is the standing body that owns the defect management process itself: who may use which statuses, what a report must contain, how disputes about severity or priority are settled. The triage meeting is the recurring working session where individual reports are assessed and prioritised. The defect manager administers the process and the data rather than deciding the fixes alone.
Defect statistics for a prison visitor-booking system show that 68% of the…
Defect statistics for a prison visitor-booking system show that 68% of the defects found in system test originate in requirements, and that almost none of them were found by the requirements reviews that were held. Which process improvement action follows most directly from a phase containment analysis of this data?
Increase the number of system test cases covering the requirements areas where most of the defects were found.
More system testing finds the same defects later and at higher cost, which is the opposite of what phase containment analysis is used to achieve.
Reassign the requirements reviews from the business analysts to the test team, so that the reviewers are independent of the authors, since the independence of the reviewer is a known factor in review effectiveness.
Independence may help, but the data points at the effectiveness of the review technique rather than at who holds the pen, and moving the work does not change how it is done.
Introduce a defect taxonomy so that future requirements defects can be classified more consistently when they are reported, which will make the next phase containment analysis easier to carry out.
A taxonomy improves the classification of what is reported, but on its own it adds no detection capability in the phase where the defects arise.
Change how the requirements reviews are conducted, for example by introducing reading techniques derived from the types of defect that are escaping.
Correct. Phase containment points at the gap between the introducing and the detecting phase, and here the detection step exists but is ineffective, so the review technique is what has to change.
Phase containment analysis compares the phase in which a defect was introduced with the phase in which it was detected, and the improvement action it points to is to strengthen detection in the phase where the defects originate. Here the reviews already exist but are not catching the defects, so the action is to change how the reviews are conducted, for example by introducing a checklist or scenario-based reading derived from the escaping defect types.
During system test of a canal-boat hire system a tester observes that a booking…
During system test of a canal-boat hire system a tester observes that a booking made for 29 February is rejected with a date error. The tester raises a defect report. The developer investigates and finds that the application is correct: the test database was seeded with a non-leap year calendar table, so the date genuinely does not exist in the configured data. How should this be handled, using the terms and the workflow of the syllabus?
Move the report to the rejected status recording the test-data cause, correct the seed data, and treat the original observation as a false-positive test result.
Correct. The anomaly was real but the test object was not at fault, so the report is rejected with its cause recorded and the faulty test data is fixed.
Keep the report open as a defect against the application, since the behaviour differed from what the tester expected when the test was designed, and that expectation was derived from a requirement the application is supposed to satisfy.
A difference from expectation is an anomaly, and once investigation shows the test object behaved correctly the item is not a defect in that object.
Close the report as resolved and record the seed data correction as the fix that resolved it.
Resolved and closed are for items that were defects in the test object; using them here would overstate the defect count and distort the statistics.
Move the report to the deferred status, since the behaviour will not be seen again until the next leap year, which gives the team almost four years to decide whether the date handling needs to change.
Deferring implies a genuine defect whose fix is postponed, whereas here there is nothing in the application to fix.
What the tester saw was an anomaly: an observed difference from expectation whose cause was not yet known. Investigation showed the application behaved correctly and the discrepancy came from the test data, so the report does not describe a defect in the test object. The item is moved to the rejected status with the cause recorded, and the real action is to correct the test data; a result that looked like a failure but was caused by the test setup is a false-positive test result, not evidence of a defect.
An Agile team building a honey traceability application does not normally raise a defect report for something it finds and fixes within the same iteration. Applying the situations in which the syllabus says a defect report should nevertheless be created, which two of the following warrant a report?
A defect found in the harvest-weight calculation that was accepted two iterations ago and is already in use by the pilot apiaries.
Correct. A defect in previously accepted functionality concerns work the team has already declared done, so it has to be visible rather than quietly corrected.
A suggestion from the product owner that the colour of the batch status badge should be changed.
A change of opinion about intended behaviour is a change request rather than a defect report.
An observation made during a pairing session about code that has not yet been committed to the repository.
Work in progress that has not been committed is not yet a delivered item; feedback at this point is part of writing the code.
A defect in this iteration's story that the developer fixes within the hour, before the story is reviewed, with the tester confirming the correction at the next build.
This is the ordinary within-iteration correction for which the syllabus accepts that no report is needed.
A defect in this iteration's story that cannot be fixed before the iteration ends and will be carried into the next one.
Correct. Anything crossing the iteration boundary needs a record, otherwise the carried work disappears from the team's view and from the statistics.
The syllabus names situations in which an Agile team should still produce a defect report even though it works in short iterations: when the defect will not be fixed within the iteration, when it was found in functionality accepted in an earlier iteration, when it must be visible to people outside the team, when a regulatory or contractual obligation requires traceability, and when the data is needed for process improvement. A defect found and fixed within the hour on this iteration's story, and a comment made during a pairing session on code not yet committed, do not meet those conditions.
A test manager on a vineyard logistics programme wants two things from the…
A test manager on a vineyard logistics programme wants two things from the defect data next year: an indication of how well debugging is being done, and an indication of how well defect reports are being written. Applying the principle that only data which will be used should be collected, which two fields should be added to the defect report template?
The name of the team to which the person reporting the defect belongs.
Attributing reports to teams invites comparison between people rather than measurement of the two process properties in question.
A counter or flag recording that the item was re-opened after having been marked resolved.
Correct. Re-opening is the observable consequence of an inadequate fix, so the re-opened rate is the usable indicator of debugging quality.
A structured rejection reason, with values distinguishing not reproducible, duplicate and working as specified.
Correct. Aggregating rejection reasons separates poorly written reports from genuine disagreements about intended behaviour.
A free-text comment in which the developer explains what was wrong with the report, so that the tester receives specific feedback on each rejected item.
Free text cannot be aggregated across hundreds of reports, so it does not yield an indicator even though it may help in an individual case.
The severity of the defect as assessed by the tester at the time of reporting.
Severity is already collected for prioritisation; adding it again serves neither of the two questions being asked.
The quality of debugging shows in how often a fix fails, so a count or flag of re-openings after a fix is the field that supports it. The quality of the reports shows in how often they are rejected as not reproducible or as duplicates, so a structured rejection reason is the field that supports that. Severity is already needed for other purposes, a free-text note cannot be aggregated, and the reporter's team says nothing about either question on its own.
A defect taxonomy for a dialysis clinic scheduling system has the top-level…
A defect taxonomy for a dialysis clinic scheduling system has the top-level categories data handling, user interface, interface to external systems, performance and security. A new report arrives: when two schedulers edit the same patient slot at the same time, the second save overwrites the first without warning and the audit trail shows only the later change. Where should it be classified, and why does the classification matter?
User interface, because the schedulers encountered the behaviour while editing the slot on the scheduling screen.
Classifying by where a defect was observed rather than by what is wrong makes the clusters follow the test approach instead of the product's weaknesses.
Interface to external systems, because concurrency problems arise from the way the database server handles simultaneous sessions, and the database is a separate product supplied and operated by a different vendor team.
The category refers to interfaces with other systems, not to the internal database, and the defect is in how the application handles concurrent edits.
Security, because the audit trail is incomplete and audit trails are a security control in a clinical system.
The incomplete trail is a consequence of the lost update rather than a separate access-control weakness, and splitting it out would obscure the single underlying cause.
Data handling, because the defect is a lost concurrent update with an incomplete audit trail; classifying by the screen where it appeared would hide a data cluster inside the interface category.
Correct. The taxonomy is meant to group by the nature of the defect so that clusters point at real weaknesses, and a lost update is a data handling problem wherever it surfaces.
The behaviour is a concurrent update that silently discards data and leaves the audit trail incomplete, which is a data handling defect regardless of where the user noticed it. Classification matters because the taxonomy is used to aggregate defects into clusters that direct improvement effort, so assigning the item to the screen on which it appeared would hide a data-handling weakness inside the user interface category.
Four weeks of system test data for a waste-to-energy plant control system read…
Four weeks of system test data for a waste-to-energy plant control system read as follows. Week 1: 78 cases executed, 14 defects found, risk coverage 22%. Week 2: 96 executed, 17 found, risk coverage 44%. Week 3: 104 executed, 16 found, risk coverage 61%. Week 4: 112 executed, 15 found, risk coverage 64%. The remaining exit criterion is 90% risk coverage. The sponsor asks whether testing can stop. Which analysis of the combined metrics is best supported?
Testing can stop: the number of defects found has fallen for two consecutive weeks, which indicates that the product has stabilised.
Fifteen to seventeen defects a week is a flat rate rather than a decline, and a stable find rate is evidence that more defects remain to be found.
Testing can stop once one more week has been executed at the current rate, because risk coverage has grown by about 14 points a week on average.
The average across four weeks hides the shape: coverage grew by 22, 17 and then only 3 points, so extrapolating the average overstates what another week would deliver.
Testing cannot stop: the find rate is flat rather than falling and risk coverage has stalled 26 points short, so execution should be redirected at the uncovered risk areas.
Correct. A flat find rate gives no saturation signal, and the stall in risk coverage while execution continues shows the effort is no longer going where the exit criterion is measured.
Testing cannot stop, and the right response is to lower the exit criterion from 90% to the 64% already achieved, since the remaining areas are evidently expensive, and the sponsor can be asked to accept the revised figure in the next weekly report.
Changing the criterion to match the achieved figure removes the measurement rather than the risk, and nothing in the data shows the remaining areas were even attempted.
Execution is steady and the defect find rate is flat rather than falling, so there is no evidence of saturation. The decisive signal is risk coverage: it grew quickly in the first three weeks and then almost stopped, moving only three points in week 4 while execution continued. That combination says effort is now going into areas that add little risk coverage, so testing cannot stop at 64% against a 90% criterion, and the response is to redirect execution at the uncovered risks rather than to extend the current pattern.
A dashboard for a fishing-quota administration system shows requirements…
A dashboard for a fishing-quota administration system shows requirements coverage rising from 71% to 88% over three weeks, while risk coverage falls from 58% to 51% over the same period. No requirements were added or removed and no risk ratings were changed. What does this combination most likely indicate, and what should the test manager do about the report?
One of the two measures must be miscalculated, so the test manager should remove risk coverage from the dashboard until the calculation has been verified, since two coverage figures derived from the same executed tests would be expected to move in the same direction.
The two measures have different denominators and can legitimately diverge; dropping the one that contradicts the comfortable figure removes exactly the signal the report exists to carry.
The product is improving, because requirements coverage is the measure tied to the contract and it has risen by 17 points.
Requirements coverage says how much of the specification has been exercised, not how much of the risk has been addressed, so it cannot support a claim about the product overall.
Risk coverage has fallen because the risk analysis has become stale, so the register should be re-rated to restore consistency between the two measures, and the re-rating can be carried out in the same session as the weekly risk review.
The ratings are stated to be unchanged, and re-rating to make two charts agree adjusts the measurement rather than investigating what happened.
Execution has moved into low-risk requirements while coverage of some risk areas has lapsed, so the report must show both measures together with that explanation.
Correct. The divergence is informative in itself, and presenting either figure alone would let the reader draw the opposite conclusion from the same data.
Two coverage measures can only move in opposite directions if the denominator of one has grown relative to what has been tested, or if the executed tests are concentrated in requirements that carry little risk. With requirements and ratings unchanged, the explanation is that newly executed tests are covering low-risk requirements while some risk areas previously counted as covered have lost that status, for example because tests there now fail. The report should present the two measures together with that explanation, since either figure alone misleads.
A hybrid programme for a courtroom transcription service runs a sequential…
A hybrid programme for a courtroom transcription service runs a sequential integration phase with an external supplier and iterative development in-house. The test manager must estimate the test effort for the next release. Three years of comparable in-house iteration data exist and are of good quality; the supplier integration is new, no comparable data exists, and two specialists who know that domain are available but are in different time zones and hard to convene. Which estimation arrangement best fits the selection factors?
Expert-based estimation for both parts, because the presence of an external supplier makes the whole release too complex for historical data to apply, and keeping to a single technique makes the two parts of the estimate comparable when they are added together.
Discarding three years of good in-house data for the part it actually describes trades a reliable basis for an opinion.
Planning poker for both parts, since the programme is partly Agile and a single technique keeps the estimate consistent across the release.
Planning poker needs the participants together and relies on comparable past items, neither of which holds for a first-time supplier integration.
Metric-based estimation for the in-house iterative part, and Wideband Delphi for the supplier integration, since no data exists there and the experts cannot easily be convened.
Correct. The two parts have different data and expert situations, and Delphi's iterative written rounds suit experts who cannot be brought into one room.
Metric-based estimation for both parts, extrapolating the supplier integration from the in-house iteration data adjusted by a complexity factor.
There is no comparable data for the supplier integration, so the extrapolation would rest on a factor that is itself a guess.
The selection factors are the tolerable estimation error, data availability, expert availability, knowledge of modelling, and time constraints, and the syllabus pairs low complexity with metric-based estimation, high complexity with expert-based, sequential contexts with Wideband Delphi and Agile contexts with planning poker. Here the two parts differ, so the estimate should too: metric-based estimation for the in-house iterative part where good data exists, and an expert-based technique for the supplier integration where it does not, with Wideband Delphi chosen over a meeting-bound method because the experts are hard to convene.
A test manager has produced a single-point estimate of 40 person-days for…
A test manager has produced a single-point estimate of 40 person-days for testing a glacier monitoring data pipeline. The sponsor's concern is not the figure itself but that previous estimates for comparable work have been wrong in both directions by large margins, and the sponsor wants to understand the range before committing. Which response addresses that concern, and what does it not address?
Apply a fixed contingency of 30% to the 40 person-days, since the historical errors have been large in both directions.
A flat uplift raises the single figure without describing the spread, and it quietly asserts that the error is one-sided when it is not.
Convene a Wideband Delphi session, because a consensus figure reached by several independent estimators carries uncertainty that one estimator cannot express, drawing on the experience of everyone who has estimated comparable work before.
Delphi addresses one-sided bias by combining independent judgements, but it still delivers a consensus point rather than the range the sponsor asked for.
Re-estimate with planning poker, because the disagreement between estimators will reveal the range that the single-point estimate concealed, and the spread of the cards played is itself a statement of the uncertainty.
Planning poker is a relative-sizing technique for comparable backlog items and converges on one value; the visible disagreement is a by-product rather than a stated range.
Produce optimistic, most likely and pessimistic values to give an expected value and a spread, which answers the range question but would not correct a one-sided bias.
Correct. Three-point estimation quantifies dispersion, which is what the sponsor asked about, and the caveat is accurate because a systematic bias shifts all three points alike.
Three-point estimation exists to express uncertainty: optimistic, most likely and pessimistic values produce an expected value and a spread, which is precisely what a sponsor asking about range needs. It does not remove bias: if the estimator is systematically optimistic, all three points move together, so a technique that brings several independent judgements to bear, such as Wideband Delphi, is what addresses a consistent directional error. Here the error is in both directions, which is dispersion rather than bias.
Chapter 3 · Managing the Team — 9 questions
A test manager building a profile for a vacancy on a cable-car control team…
A test manager building a profile for a vacancy on a cable-car control team lists four requirements: knowledge of the ropeway domain, the ability to derive test cases with boundary value analysis, the ability to explain a risk assessment to an operations manager, and the self-discipline to keep working through a long regression cycle. Using the four areas of competence, which mapping is correct?
Domain knowledge methodological, boundary value analysis professional, explaining the assessment personal, self-discipline social, taking technique knowledge to be the subject matter of the profession.
This swaps the first pair and the second pair: techniques are methodological and domain knowledge is professional, while communication is social and self-management is personal.
Domain knowledge professional, boundary value analysis methodological, explaining the assessment social, self-discipline personal.
Correct. Each requirement falls into exactly one area, and this is the mapping the syllabus uses.
Domain knowledge professional, boundary value analysis professional, explaining the assessment social, self-discipline social, grouping the four requirements into two areas rather than four.
Technique knowledge is methodological rather than professional, and self-discipline is a personal rather than a social competence.
Domain knowledge personal, boundary value analysis methodological, explaining the assessment professional, self-discipline personal, treating the ability to explain an assessment as part of the profession.
Domain knowledge is not a personal attribute, and explaining an assessment to a stakeholder is a social competence rather than a professional one.
The four areas are professional competence, which covers domain and subject knowledge, methodological competence, which covers the techniques and methods of the discipline, social competence, which covers working with and communicating to others, and personal competence, which covers self-management. Domain knowledge is professional, boundary value analysis is methodological, explaining an assessment to a stakeholder is social, and self-discipline is personal.
A test manager for a brewery bottling-line system uses both a skills matrix and a team role model such as Belbin. What is the difference in what the two instruments tell the manager?
The matrix shows the behaviours people bring to the group; the role model shows which technical competences the team is missing, so each instrument answers the question the other one is usually asked.
This reverses the two: it is the matrix that records competences and the role model that describes behaviour.
The matrix measures individual performance for appraisal purposes; the role model measures the same thing at team level.
Neither instrument is an appraisal device, and using a skills matrix as one discourages honest self-assessment and destroys its value.
The matrix shows which competences the team holds and where they are missing; the role model describes the behaviours people bring to working together.
Correct. One instrument measures capability against what the work needs, the other describes group behaviour, and they are used for different decisions.
The matrix lists the training each person has completed; the role model lists the training each person should complete next, so taken together the two of them form the team's development record.
A skills matrix records competence however it was acquired, not course attendance, and a role model has nothing to say about training plans.
A skills matrix records which competences each team member has and at what level, so it shows where the team is short of a capability it needs. A team role model describes the behavioural roles people tend to take in a group, so it says something about how the team will work together. The two answer different questions and neither replaces the other; a role model is not a measure of competence and a skills matrix says nothing about team dynamics.
Two members of a test team working on a dog-grooming salon booking product need…
Two members of a test team working on a dog-grooming salon booking product need to learn a new API testing tool. One is paired for three weeks with a colleague who uses it daily, working on real tasks. The other is assigned an experienced tester from another team who meets her fortnightly to discuss her development and how she is applying what she learns. Which skill development approaches do these describe?
The first is self-study and the second is peer learning, because in each case the tester drives her own learning with occasional support.
Neither case is unsupported self-study, and the second involves someone deliberately more experienced rather than a peer.
The first is training on the job and the second is mentoring or coaching.
Correct. Learning while doing the real work is training on the job, and a periodic developmental relationship is mentoring or coaching.
The first is mentoring or coaching and the second is training on the job, because the pairing has the more experienced colleague doing the work alongside her.
The pairing on real tasks is the on-the-job case; the fortnightly developmental conversation does not involve doing the work together.
The first is peer learning and the second is formal training and education, since in both cases the tester learns from someone outside her own role.
Peer learning involves colleagues at a similar level learning together, and a fortnightly one-to-one with an experienced tester is not a formal course.
Training on the job means learning by doing the real work, typically alongside someone who already has the skill, which is the first case. Mentoring or coaching is a longer-running relationship with someone who supports a person's development and reflection rather than performing the task with them, which is the second. Self-study and formal training and education are distinguished by the absence of that personal relationship.
A test team on a gene-bank registry has been together for a year. Members…
A test team on a gene-bank registry has been together for a year. Members resolve disagreements among themselves, pick up each other's work without being asked, and the test manager's involvement is now mostly removing obstacles outside the team. Which Tuckman phase is this, and what does the model suggest about the manager's behaviour?
Adjourning, and the manager should begin planning how the team's knowledge will be handed over when it disbands, so that nothing is lost when the members move on to other products.
Adjourning applies when the team is about to be dissolved, and nothing in the description suggests the work is ending.
Storming, and the manager should intervene in the disagreements to prevent them from damaging the team's productivity, since unresolved conflict about roles and approach is what holds a team at this stage.
Storming is marked by unresolved conflict about roles and approach, whereas this team settles its disagreements without the manager.
Performing, and the manager should delegate and concentrate on removing obstacles outside the team rather than directing the work.
Correct. At performing the team manages its own work, and the manager's value lies in what only the manager can reach, which is outside the team.
Norming, and the manager should now set out explicit working agreements so that the team's informal arrangements are written down, which is what stabilises the way the members work with one another.
Working agreements are established at norming; a team that already resolves disagreements and redistributes work by itself has moved past that point.
This is the performing phase: the team is productive and largely self-organising, and the manager's useful contribution shifts from directing the work to delegating and clearing external impediments. Treating a performing team as if it were still forming, by reasserting direction, is the behaviour the model warns against.
A hearing-aid fitting system incurred the following costs last year: 60,000 on…
A hearing-aid fitting system incurred the following costs last year: 60,000 on testing before release, 25,000 on training testers in the fitting domain, 90,000 reworking defects found before release, and 150,000 on recalls and support calls after defects reached audiologists. Using the four cost of quality categories, which classification is correct?
Prevention 25,000; appraisal 60,000; internal failure 90,000; external failure 150,000.
Correct. Training prevents defects, testing appraises, pre-release rework is internal failure and post-release recall is external failure.
Prevention 85,000; appraisal nil; internal failure 90,000; external failure 150,000, combining the two pre-release investments into a single prevention figure.
Combining testing with training treats detection as prevention and leaves the appraisal category empty, which removes the distinction the model exists to draw.
Prevention 60,000; appraisal 25,000; internal failure 90,000; external failure 150,000, on the basis that testing is the activity that keeps defects away from the customer.
This swaps the first two: testing is the activity that detects defects and therefore appraisal, while training is what reduces the chance of creating them.
Prevention 25,000; appraisal 60,000; internal failure 240,000; external failure nil, counting the recall and support spend as rework on defects found in the product.
Recall and support costs arise after the product reached the audiologists, which is the defining condition for external failure.
Prevention costs are incurred to stop defects arising, which here is the domain training. Appraisal costs are incurred to find defects, which is the testing before release. Internal failure costs arise from defects found before delivery, which is the pre-release rework. External failure costs arise from defects that reached the customer, which is the recall and support spend.
A test manager is writing the benefits section of a business case for keeping a dedicated test team on a civil-registry product. Which two of the following are qualitative benefits of testing in the sense the syllabus uses?
Improved standing with the supervisory authority that audits the registry.
Correct. Reputation and trust with an external authority are qualitative benefits in the syllabus sense.
A fall in the mean time to repair a reported defect from nine days to four.
A measured change in a process time is quantitative, however it is subsequently used in the argument.
A test team of six people dedicated to the product for the next financial year.
This is the resource the business case asks for, that is an input, rather than a benefit the investment produces.
Greater confidence among the stakeholders who must take the release decision.
Correct. Confidence in a decision is a genuine benefit that cannot be reduced to a figure, which is what makes it qualitative.
A reduction of 40,000 per year in the cost of reworking defects found after release.
An avoided cost expressed in currency is a quantitative benefit.
Quantitative benefits can be expressed in figures, such as avoided failure costs or reduced support volume. Qualitative benefits are real but not reducible to a number: confidence in the release decision, the reputation of the organisation with its users, and the regulator's trust. Reduced rework cost and a shorter mean time to repair are quantitative, and the number of testers is an input rather than a benefit.
Across two releases of a ferry-terminal passenger system the cost of quality…
Across two releases of a ferry-terminal passenger system the cost of quality moved as follows. Release 1: prevention 20,000, appraisal 120,000, internal failure 140,000, external failure 180,000. Release 2: prevention 90,000, appraisal 130,000, internal failure 95,000, external failure 60,000. The scope and size of the two releases were comparable. Which conclusion is best supported by this data?
The extra 80,000 spent on prevention and appraisal was accompanied by a 165,000 fall in failure costs, so the total cost of quality fell by 85,000.
Correct. The arithmetic holds and the comparison of the two categories is the argument the cost of quality model is used to make.
No conclusion can be drawn, because the cost of quality categories are not comparable between two different releases, since the way costs are allocated to the four categories differs from one release team to another.
The releases are stated to be comparable in scope and size, which is the condition under which the comparison is meaningful.
Appraisal spending is what drove the improvement, since external failure costs fell by two thirds while appraisal rose, and appraisal is the category most directly concerned with finding defects before delivery.
Appraisal rose by only 10,000 against 70,000 for prevention, so attributing the movement to appraisal does not follow from the figures.
The extra 80,000 spent on prevention and appraisal was accompanied by a 120,000 fall in failure costs, so the total cost of quality fell by 40,000, taking into account the movement in the external failure category.
This counts only the external failure movement and omits the 45,000 fall in internal failure costs, understating both figures.
Total cost of quality fell from 460,000 to 375,000. Prevention rose by 70,000 and appraisal by 10,000, a combined increase of 80,000, while failure costs fell by 45,000 internally and 120,000 externally, a combined fall of 165,000. The data therefore supports the argument that the additional prevention spend was more than repaid by the reduction in failure costs, with the largest single movement in the external failure category.
A test team has maintained a monolithic claims-free driving discount engine for…
A test team has maintained a monolithic claims-free driving discount engine for six years, testing it through its user interface at the end of each quarterly release. The product is being rebuilt as a set of services with contract-based interfaces, delivered continuously, in a domain the team knows well. The test manager must decide which skills to develop first. Analysing the required skills against the activities of the test process and the context drivers, which choice is best justified?
Designing tests at the level of the service contracts and automating them inside the delivery pipeline, since the architecture and lifecycle changed while the domain did not.
Correct. Mapping the changed context drivers onto the test process activities points at interface-level design and pipeline execution, and the unchanged domain means professional competence needs no investment.
Running a team role assessment first, so that the manager knows which members will adapt best before any technical training is arranged.
A role model describes behaviour in the group and does not identify the competence gap created by a change of architecture and lifecycle.
Waiting until the service decomposition is complete, so that the skills needed can be derived from the finished architecture rather than from a plan, which avoids investing in training for service interfaces that may still be redesigned.
The affected activities are already identifiable from the direction of the change, and in continuous delivery the skills are needed as the first services arrive.
Deepening the team's knowledge of the insurance discount rules, since the services will implement the same business logic as the monolith.
The domain is explicitly unchanged and the team already knows it well, so this investment addresses the one driver that did not move.
The context drivers named by the syllabus are the domain, the architecture and technology, and the lifecycle. Here the domain is unchanged, so professional competence is already in place; what changes is the architecture, services with contract interfaces, and the lifecycle, continuous delivery. The test process activities most affected are analysis and design at interface level and implementation and execution inside a pipeline. The priority is therefore methodological skill in designing and automating interface-level tests, not domain training, not a role model exercise, and not deferring until the architecture is finished.
The test team for a cheese-maturing warehouse system consists of four testers…
The test team for a cheese-maturing warehouse system consists of four testers strong in manual system testing of the warehouse domain. The organisation will move the product onto a sensor-driven platform within a year, with telemetry arriving continuously and acceptance decisions based on time-series data. One tester has begun learning the telemetry stack in her own time. Analysing the competence gap against the test process activities, which two actions are most justified?
Send all four testers on an advanced course in warehouse logistics so that their domain knowledge is current for the new platform, since the maturing rules themselves will carry over to the new platform unchanged.
The domain is the one driver that has not changed, so this invests where there is no gap.
Replace two of the four testers with engineers experienced in telemetry platforms before the migration begins.
This trades away the domain knowledge that remains valuable and that takes longest to rebuild, in order to buy a skill that can be developed.
Postpone any development until the platform is in place, so that training is based on the system as actually delivered.
The skills are needed for analysis and design work that starts before delivery, and postponing leaves the team unable to contribute during the migration.
Bring in a specialist to work alongside the team on test analysis and design for the time-series acceptance decisions.
Correct. The analysis and design activity is where the gap bites hardest, and an external specialist working alongside the team transfers the method rather than just producing the tests.
Pair the tester who has begun learning the telemetry stack with the others on real tasks, so that the new capability spreads through the team.
Correct. Training on the job around a colleague who already has the skill is the fastest route for a small team and it keeps the domain knowledge in place.
The gap is methodological and technical rather than professional: the team knows the domain but not how to analyse, design and evaluate tests over continuous time-series data. Pairing the tester who has started learning with the rest spreads the new capability through training on the job, and bringing in a specialist for the test analysis and design activity closes the part of the gap the team cannot bridge from inside. Replacing the team, waiting for the platform or sending everyone on a domain course would each address the wrong driver.