ISTQB Specialist (CT-GenAI v1.1) Mock Exam #3 — Questions & Answers
Every question in this mock exam, with the correct answer marked and a written rationale behind each option below — for reading and review, not a timed run.
Chapter 1 · Introduction to GenAI for Software Testing — 7 questions
Which description of a system prompt in a large language model tool is correct?
Instructions the tool supplies ahead of the user's message, setting the assistant's role, rules and output conventions for that session.
Correct. The system prompt is tool-side configuration that frames every user message in the session.
The first message a user types in a new conversation, which the tool then stores and replays at the start of every later conversation with that user.
Incorrect. That describes a saved user instruction or profile; the system prompt is set by the tool, not typed by the user.
A diagnostic request sent to the provider's platform that returns the model version, latency and token usage of the preceding generation.
Incorrect. Usage and version metadata come from the API response or a dashboard, not from a prompt that shapes behaviour.
A block of text the provider appends after each answer so downstream systems can label the content as machine-generated.
Incorrect. That would be an output marker or watermark; a system prompt acts before generation, not after it.
A system prompt is configuration the tool itself places before the user's message. It fixes the role the assistant plays, the rules it must follow and the shape of its output, and it stays in force for the exchange even though the user does not see or type it. Two tools built on the same model can behave differently because their system prompts differ, which is why a prompt alone does not describe the configuration that produced an answer.
A tester asks an assistant to write out sixty regression test cases in full. The reply is well formed but stops in the middle of case thirty-four, and repeating the request produces the same truncation at roughly the same point. What is the most likely explanation?
The input the tester supplied exceeded the model's context window, so the later part of the request was discarded before generation began.
Incorrect. A context overflow degrades what the model takes in; here the request was short and the output is what stops.
The response reached the maximum number of output tokens the tool allows per reply, so generation stopped before the last cases were written.
Correct. A reproducible cut-off part-way through a long reply is the signature of the per-response output limit.
The assistant ran out of material because the request did not carry enough detail to support more than about thirty distinct regression test cases.
Incorrect. A model short of material produces thin or repetitive cases; it does not stop in the middle of writing one.
The provider throttled the request part-way through because the account had reached its rate limit for the current billing period.
Incorrect. A rate limit refuses or delays a request; it does not deliver a well formed partial answer twice in a row.
Input length and output length are two separate limits. The context window bounds what the model can take in; a separate cap bounds how much it may write in a single reply. An answer that is correct in form but stops mid-item, reproducibly and at roughly the same place, points at the output cap rather than at the request. The practical fix is to ask for the cases in smaller batches, or to raise the output limit where the tool exposes it.
Before adopting a hosted assistant for test work, a team reads the provider's model card and accompanying documentation. Which TWO statements about what that documentation can settle are correct?
It reports how the model will perform on the team's own requirements documents, because benchmark scores transfer directly to any domain of similar difficulty.
Incorrect. Benchmarks are measured on public data sets; performance on the team's own material has to be measured by the team.
It states the model's intended uses, known limitations and the benchmarks the provider ran, which narrows down where the model is a reasonable fit.
Correct. Intended use and stated limitations are exactly what a model card is written to communicate.
It lists the training data item by item, so a team can check whether its own confidential documents were used during pre-training of the model.
Incorrect. Model cards summarise data sources at a high level; they do not provide an item-level inventory to search.
It describes the configuration the provider ships, so a team still has to measure the model on its own artefacts before relying on it for test work.
Correct. The documentation sets expectations; evidence for the team's own context comes from the team's own evaluation.
It records which prompts the team's testers have sent, which gives the organisation an audit trail of how the model is being used in practice.
Incorrect. Usage logging is a property of the deployment and the contract, not something a model card provides.
A model card is the provider's own account of intended use, known limitations and the benchmarks it chose to run. That is useful for ruling a model in or out, and it describes the shipped configuration, but published benchmark scores are measured on public data sets and do not carry over to a team's own requirements, defect reports or test artefacts. Nor does a model card enumerate training data or record anything about how a particular organisation uses the model.
A tester gets well-structured output from a prompt in a vendor's chat application. A colleague sends the identical prompt through the same vendor's API and gets noticeably different, less structured output. Both are using the same published model name. What does this most likely show?
The API returns the model's raw internal representation, which the chat application converts into readable prose before it is displayed.
Incorrect. Both surfaces return generated text; the API does not expose an internal representation that needs conversion.
The chat application runs a larger version of the model than the one the API exposes under the same published name and version string.
Incorrect. A shared model name denotes the same weights; the difference here comes from what surrounds the prompt.
The two surfaces apply different system prompts and default settings, so the user's prompt is not the whole of what the model receives.
Correct. The surrounding configuration differs, which is why the same prompt text behaves differently in each place.
The API applies stricter content filtering than the chat application, and that filtering removes the structure from any answer longer than a few paragraphs.
Incorrect. Content filtering blocks or redacts material; it does not selectively strip headings and lists from long answers.
A prompt is only part of what reaches the model. The chat application adds its own system prompt and ships defaults for sampling, output length and formatting; the API sends what the caller specifies and little else. Reproducing a result therefore means recording the surface and its settings alongside the prompt text, not the prompt text alone.
A team pastes a 400-page specification into every request so that the assistant has the full picture, then asks one short question per request. Both the monthly bill and the response times rise sharply. What is the most accurate explanation?
Response time and cost depend on the number of requests rather than on their size, so the rise has to come from the number of questions the team is asking.
Incorrect. Both are driven mainly by tokens processed; a few very large requests can cost more than many small ones.
The assistant stores the specification after the first request, so the cost of the later requests reflects storage rather than processing.
Incorrect. A stateless request keeps nothing between calls, so there is no stored copy to be charged for.
Long inputs force the provider to route the request to a larger model, and it is that routing decision the team is being billed for.
Incorrect. The model is selected by the caller; input length does not silently promote a request to a different model.
Each request is processed from scratch, so the whole specification is read again every time and both cost and latency scale with the tokens sent.
Correct. The document is re-read on every call, which is what makes the large input expensive and slow.
Each request to a stateless model is processed on its own: the whole input is read again, and both the price and the time to first token grow with the number of tokens sent. Pasting a large document on every call therefore pays for it on every call. The usual remedies are to select the relevant sections, to summarise once and reuse the summary with care, or to put the document behind retrieval so that only the matching passages are sent.
A tester asks a general assistant how the team's in-house automation framework, called Harbour, handles retries. The answer is detailed and confident, and describes configuration options the framework does not have. Which explanation fits best?
The model produced text statistically consistent with the name and the question, drawing on public frameworks rather than on the team's code.
Correct. With no access to the framework, the model generates what such an answer usually looks like.
The model retrieved an outdated copy of the team's own repository from its training data and described an earlier release of the framework's retry handling.
Incorrect. A private repository is not part of the training data, and a model does not retrieve documents from it on demand.
The assistant misread the question and answered about retry handling in a different tool whose name it treated as a close spelling match.
Incorrect. The answer is not about another named tool; it is generic framework material fitted to the name given.
The answer is a template the provider returns whenever a named product cannot be identified, with the product name substituted into the text.
Incorrect. There is no such fallback template; the text is generated afresh like any other answer.
A general model has not seen the team's code. Given a plausible-sounding product name and a question about retries, it produces text that is statistically consistent with how automation frameworks in general describe retry configuration. The output reads as specific knowledge because fluency and specificity are properties of the generation, not evidence that a source was consulted. Questions about internal artefacts need those artefacts supplied in the prompt or made available through retrieval.
A tester adds the instruction "state your confidence as a percentage" to a prompt that asks an assistant to classify 200 defect reports. The model returns a confidence figure alongside every classification. How should those figures be treated?
As a calibrated probability produced by the model's scoring layer, which makes them suitable for setting an automatic acceptance threshold.
Incorrect. The figure is generated text, not a calibrated score exposed by the model.
As generated text like the rest of the answer, useful as a prompt to take a closer look but not as a measured property of that classification.
Correct. A self-reported figure can direct attention, but it carries no measurement behind it.
As a measure of how much of the defect report the model was able to read, with lower figures marking the reports that exceeded the input limit.
Incorrect. Truncation is a property of the input length and is reported by the tool, not by a confidence figure.
As an indication of how often the same classification would be produced on repeated runs, which can replace running the classification twice.
Incorrect. Run-to-run stability has to be measured by repeating the run; the model does not estimate it.
A stated confidence is produced by the same generative process as the classification itself. It is text that looks like a measurement, not a calibrated probability derived from the model's internal scores, and it tends to be high even where the classification is wrong. A low figure is still worth using as a signal to look at that report first, but acceptance thresholds and sampling plans need evidence from an evaluation set, not from the model's own self-report.
Chapter 2 · Prompt Engineering for Effective Software Testing — 11 questions
Two prompts for deriving tests from a requirement are compared. The first asks…
Two prompts for deriving tests from a requirement are compared. The first asks for "comprehensive test cases". The second supplies the requirement text, names the coverage criterion to reach, and asks for each case to be listed against the part of the requirement it exercises. Why is the second prompt better for a tester?
Because naming a criterion makes the model apply the corresponding formal algorithm internally, so the resulting set is complete by construction.
Incorrect. Naming a technique shapes the output, it does not make the model execute that technique reliably.
Because the word comprehensive is interpreted by the model as a request for the largest number of distinct cases it can fit into one reply.
Incorrect. The weakness of the vague prompt is that the result cannot be checked, not that it produces too many cases.
Because the output can be checked against a stated target, so a reviewer can tell which parts of the requirement were covered and which were left out.
Correct. A named criterion and an explicit mapping make the result verifiable by a human reviewer.
Because a prompt that includes the requirement text is processed through a different retrieval mechanism than one that merely refers to the document by name.
Incorrect. Including the text matters because the model has the content, not because a different mechanism is used.
"Comprehensive" is not a property the output can be measured against, so a reviewer has nothing to compare the result with. Naming the criterion and asking for each case to be mapped to the part of the requirement it exercises turns the answer into something checkable: the reviewer reads the mapping, sees which parts of the requirement appear and which do not, and can judge the set. The prompt does not make the model apply the technique correctly, but it makes a failure to do so visible.
A team adds a second step to its workflow: after an assistant generates test cases, a separate prompt asks a model to check each case against the requirement and flag the ones that do not follow from it. What is the correct way to treat the flags that come back?
As a verification step that satisfies the review requirement, because the checking prompt was written independently of the generating prompt.
Incorrect. A separately written prompt is still a generation, and it provides no independent evidence about the case.
As a ranking of the cases by quality, since a model asked to judge output in its own domain produces a consistent ordering from one run to the next.
Incorrect. Such verdicts vary between runs and are not a stable quality ordering.
As a defect count for the generating prompt, which the team can track over time instead of reviewing the generated cases themselves.
Incorrect. The count is only as sound as the verdicts behind it, so it cannot replace reviewing the cases.
As a triage aid that shows a reviewer where to look first, while the judgement about each flagged case still rests with that reviewer.
Correct. The flags order the reviewer's attention; they do not discharge the review.
A checking prompt is another generation. It can be useful: it reads every case, it is cheap, and the cases it flags are a sensible place for a reviewer to start. What it cannot do is stand in for the review, because the same failure modes that produce an unfounded test case also produce an unfounded verdict about one, and an unflagged case has not been verified by anything. Treat the flags as triage, and keep the human judgement on the artefact.
A tester wants an assistant to draft exploratory testing charters for a newly built payments screen. Which TWO elements, included in the prompt, most improve how usable the resulting charters are?
A statement of the mission and scope of each charter, so that a charter says what is to be explored, with what, and where it stops.
Correct. Mission and scope are what make a charter actionable and time-boxed.
A request that each charter list the exact steps to follow and the expected result at every step, so it can be executed without judgement.
Incorrect. That produces scripted test cases; exploratory charters deliberately leave the detail to the tester.
A request for an estimated pass rate per charter, so the team can predict how many defects the exploratory session will reveal.
Incorrect. The model has no basis for such an estimate, and a charter is not run to a pass rate.
The known risks and the recent changes for that screen, so that the charters concentrate on the areas where defects are most likely.
Correct. Risk and change information is what steers exploration towards productive areas.
An instruction to produce one charter per requirement in the specification, so that the number of charters matches the number of requirements.
Incorrect. Charter count follows from risk and time available, not from a one-to-one mapping to requirements.
A charter is useful when it says what is to be explored, with which resources, and where it stops, and when it points at an area where defects are plausible. Supplying the mission and scope, and the known risks and recent changes for that screen, gives the model what it needs to produce charters a tester can actually run. Turning charters into scripted steps defeats their purpose, one charter per requirement is an arbitrary count, and a predicted defect yield is not something the model can know.
An assistant is asked to rewrite 40 existing manual test cases so that a repeated set-up becomes one shared precondition and the varying inputs become a data table. The result is shorter and easier to read. What must the tester check before the originals are replaced?
That every input, step and expected result from the originals still appears somewhere in the new form, by comparing the two sets case by case.
Correct. The risk in a condensing rewrite is silent loss, and only a comparison exposes it.
That the new cases are written in the same tense and voice as the rest of the suite, which is what makes a rewritten suite maintainable over time.
Incorrect. Style consistency is worth having, but it says nothing about whether coverage survived the rewrite.
That the data table is sorted in the order the cases were originally numbered, so the mapping back to the old identifiers stays obvious.
Incorrect. Ordering helps navigation; it does not establish that the content is complete.
That the assistant used the same model version as the one that generated the original cases, since mixing versions makes a suite inconsistent.
Incorrect. The originals were written by testers, and in any case the version does not substitute for checking the result.
A rewrite that shortens a suite can also lose part of it: a step that applied to only three of the forty cases, an expected result that differed in one row, a precondition that was not in fact common. Because the model produces a plausible result either way, the check has to be a comparison of the two sets, not an impression of the new one. Readability, ordering and model version are secondary to the question of whether anything was silently dropped.
A test analyst wants generated test conditions that can be reviewed against the specification without re-reading the whole document. Which TWO instructions in the prompt serve that purpose best?
Ask the model to assign each condition a priority from one to five, so that the review can start with the highest-priority items.
Incorrect. A generated priority has no basis the reviewer can check and does not reveal unsupported conditions.
Ask for the sentence or clause identifier of the specification that each condition is derived from, quoted beside the condition itself.
Correct. A pointer back to the source is what lets a reviewer verify an item quickly.
Ask for the conditions to be returned in the same order as the headings of the specification, so the reviewer can read them from top to bottom.
Incorrect. Ordering eases navigation but says nothing about whether a condition follows from the text.
Ask the model to confirm at the end that every condition was derived from the supplied text, so the reviewer has that assurance on record.
Incorrect. The confirmation is generated like everything else and carries no evidence.
Ask for a separate list of the conditions the model could not tie to any part of the supplied text, together with the reason for each.
Correct. Separating the unsupported items is what stops them passing as derived conditions.
Reviewability comes from being able to follow each generated item back to the text it came from, and from being told where the model had nothing to follow. Asking for the clause identifier beside each condition gives the reviewer a short path to the source; asking for a separate list of conditions with no identifiable basis makes the unsupported material visible instead of leaving it mixed in. Priorities, ordering and a closing assurance are all generated as freely as the conditions themselves and add nothing a reviewer can check.
A team keeps the prompts it uses for test design in a repository beside the test code, with review on every change and a tag for each release. Which argument best justifies that effort?
It allows the prompts to be executed automatically in the pipeline, which is what makes generated test cases repeatable between releases.
Incorrect. Storing a prompt does not run it, and repeatability also depends on the model and its settings.
The prompt is part of how the tests were produced, so a change to it is a change to the method and belongs under the same review as the tests.
Correct. The prompt is a methodological artefact and is managed like one.
It lets the team prove to an auditor that no tester has written a prompt outside the agreed set, because the repository records every prompt that is sent.
Incorrect. A repository holds the agreed prompts; it does not capture what individuals type into a chat window.
It reduces the cost of each request, since prompts stored in a repository are sent to the provider in a compressed, pre-tokenised form.
Incorrect. Where a prompt is stored has no bearing on how it is transmitted or billed.
When tests are produced with the help of a model, the prompt is part of the method that produced them, in the same way a test design technique or a checklist would be. Changing it changes what the team gets, so it deserves the review and the history that any other part of the method gets, and a tagged version lets the team see which prompt produced the suite shipped in a given release. The repository does not execute prompts, log what testers actually send, or change what a request costs.
A checkout flow is parameterised by browser (Chrome, Firefox, Safari), payment…
A checkout flow is parameterised by browser (Chrome, Firefox, Safari), payment method (card, wallet, bank transfer, voucher), currency (EUR, USD, GBP) and account state (logged in, guest). That is 72 combinations and the team has time for about 15. The tester's first prompt was "give me the most important checkout combinations to test". Which refinement is best?
Ask for all 72 combinations as a table, then read through them and delete the rows that look redundant until about fifteen remain.
Incorrect. This moves the whole selection problem back to the tester with no criterion to apply.
Ask the model to pick the 15 combinations it considers most important and to state the likely business impact of each one in a sentence.
Incorrect. The model has no information about business impact here, so the justifications would be invented.
Supply the four parameters and their values, ask for a set covering every pair of values at least once, and ask what it leaves untested.
Correct. A stated pairwise goal is checkable against the output, and the residual gap is made explicit.
Ask for 15 of the 72 combinations drawn at random, so that the selection is unbiased and can be repeated later on with a different seed value.
Incorrect. A random sample of 15 leaves many value pairs untouched and gives no coverage argument.
The weakness of the original prompt is that "most important" gives the model no criterion and the tester no way to judge the answer. Naming the parameters and their values and asking for a set in which every pair of values appears at least once states a coverage goal that can be checked against the returned table, and it lands near the 15 cases the window allows. Asking what the set leaves untested is the second half of the refinement: it makes the residual risk from three-way and higher interactions explicit instead of hidden.
A regression suite has 180 cases and the smoke window before a release is 40 minutes, enough for roughly 25 of them. The tester has the list of modules changed in this release and the defect history of the last six releases. Which prompt is most likely to produce a selection the team can defend?
Ask the assistant to select the 25 most critical regression cases from the suite, since criticality is the accepted basis for a smoke selection.
Incorrect. Criticality is the right idea but the model has no information with which to judge it here.
Ask for the 25 cases with the shortest execution time, so that the widest part of the suite fits inside the 40-minute window.
Incorrect. Fitting the window is not the goal; the fastest cases may cover none of the changed modules.
Ask the assistant to rank all 180 cases by how often each has failed historically, and take the top 25 from the ranking it returns.
Incorrect. Past failure rate is one input among several and ignores what changed in this release.
Supply the case list, the changed modules and the defect history, state the ranking criteria, and ask for the evidence behind each case.
Correct. Stated criteria plus the supporting data make the ranking reviewable.
A selection is defensible when someone else can see why each case is in it. The model has no access to the change list or the defect history unless they are supplied, and no notion of what this team counts as risk unless the criteria are stated. Supplying both and asking for the evidence behind each case turns the answer into a ranking a reviewer can argue with. Asking for "the most critical" cases, or ranking on execution time alone, produces a list with nothing behind it.
A user story has 12 acceptance criteria and an assistant has produced 20 test cases from it. Before accepting the set, the tester wants to know what is not covered. Which TWO prompt steps best serve that?
Ask the assistant to state the percentage of the story that the 20 cases cover, so the figure can be recorded in the test report.
Incorrect. A generated percentage has no derivation behind it and hides which criteria are missing.
Ask for 20 further cases, on the basis that doubling the number of cases raises the chance that each criterion is reached at least once.
Incorrect. More cases without a mapping leaves the tester with the same unanswered question.
Ask for a matrix mapping each acceptance criterion to the test cases that exercise it, using the criterion identifiers from the story.
Correct. The matrix makes the claimed coverage explicit and checkable against the story.
Ask for an explicit list of the criteria that no case exercises, with an instruction to leave it empty rather than invent coverage.
Correct. Naming the uncovered criteria is the direct answer to the question being asked.
Ask the assistant to re-read the story and confirm that the existing 20 cases are sufficient for the acceptance criteria as written.
Incorrect. A yes or no verdict is generated text; it shows nothing the tester can verify.
Finding the gap means forcing the mapping into the open. A matrix keyed on the criterion identifiers shows which criteria have cases against them, and an explicit list of criteria with no case shows the rest; adding the instruction to leave that list empty rather than invent coverage removes the incentive to fill it. A coverage percentage is a generated number with no derivation, more cases do not establish which criteria are reached, and asking the model to confirm sufficiency returns an opinion rather than a mapping.
A tester asks an assistant to produce expected results for 30 invoice totals…
A tester asks an assistant to produce expected results for 30 invoice totals. The specification gives the VAT rate and the line-item prices but says nothing about rounding. The assistant returns 30 totals, all rounded half up at two decimals, with no comment on the choice. How should the prompt be refined?
Require the model to state any rule it applied that the specification does not give, and to list those as open questions rather than settle them.
Correct. Surfacing the assumption turns an invented rule into a question for the business analyst.
Require the model to round half up at two decimals, since that is the convention most finance systems use for invoice totals in practice.
Incorrect. Fixing the convention in the prompt hides the specification gap instead of exposing it.
Require the model to produce the totals twice with differently worded prompts and keep only the values that agree across both runs.
Incorrect. Both runs would apply the same plausible convention, so agreement proves nothing here.
Require the model to show the intermediate line totals and the VAT amount for each invoice, so the tester can see which lines contributed to each figure.
Incorrect. The intermediate figures are useful but still leave the rounding decision unstated.
The problem is not the arithmetic but the silent decision: the specification is incomplete, and the model closed the gap with a plausible convention instead of reporting it. Expected results become a usable oracle when every assumption the model had to make is visible, so the tester can take the open point to the business analyst rather than discover it from a failed test. Pinning the convention in the prompt would hide the same gap; repetition and intermediate figures do not reveal it either.
A non-functional requirement reads: 95th-percentile response time under 2 seconds for the search endpoint at 500 concurrent users, with a month-end peak of three times normal volume. A tester's first prompt was "suggest performance test scenarios". Which refinement produces the most usable result?
Ask for a list of the popular performance testing tools together with the scripts each one would need for a search endpoint of this kind.
Incorrect. Tool choice is a separate decision and does not produce the load profile the requirement implies.
Supply the requirement, ask for load, stress and peak profiles with the user counts, ramp-up and duration, plus the measurement to record.
Correct. The profiles and the measurement are derived from the numbers the requirement states.
Ask the assistant to estimate how many servers the search endpoint needs in order to meet a 2-second response time at 500 concurrent users.
Incorrect. That is a sizing estimate the model cannot ground, and it is not a test design.
Ask for 20 search queries that are likely to be slow, so that the performance test exercises the worst cases the endpoint can encounter.
Incorrect. Query selection matters, but without a load profile there is nothing to run them under.
The requirement already contains the numbers a load profile needs: the concurrency to sustain, the percentile and threshold to measure against, and a peak multiple to drive the stress and peak profiles. A prompt that supplies them and asks for user counts, ramp-up, duration and the measurement to record returns something a tester can turn into a script and compare with the requirement. Tool lists, server sizing and a guess at slow queries all sidestep the profile that the requirement implies.
Chapter 3 · Managing Risks of GenAI in Software Testing — 10 questions
In a tool built on a large language model, what is an output guardrail?
A limit on how many requests a user may send in a given period, applied by the provider to protect the capacity of the service.
Incorrect. That is rate limiting, which is about load rather than about the content of an answer.
A fixed set of approved prompts that users choose from, so that no free-text instruction ever reaches the underlying model.
Incorrect. Restricting input is an input-side control; a guardrail here acts on what is generated.
A check applied to the generated text before it is shown, which blocks or alters output that breaks the rules set for the tool.
Correct. The guardrail inspects the generated answer and intervenes before it reaches the user.
A record of each generated answer kept for audit purposes, so that output which later proves wrong can be traced to its request.
Incorrect. That describes logging, which records what happened rather than preventing it.
A guardrail is a control that sits between the model and the user. After the model generates an answer, the guardrail inspects it against the rules set for the tool, and blocks, redacts or rewrites what breaks them. It is a mitigation rather than a fix: it reduces the chance that unwanted output reaches a user, but it does not make the generation itself more reliable.
A security officer asks about the data residency commitment offered by a cloud assistant. What does data residency mean?
The length of time the provider keeps a submitted prompt before deleting it from its systems and from its backups.
Incorrect. That is the retention period, which is a separate commitment from location.
The guarantee that submitted data is encrypted while it travels between the customer's network and the provider's service.
Incorrect. That is encryption in transit and says nothing about where the data comes to rest.
The set of staff at the provider who are permitted to read customer data when investigating a support request.
Incorrect. That is the provider's access control, not the geography of the data.
The geographic location, usually a country or region, where the submitted data is stored and processed by the provider.
Correct. Residency identifies the jurisdiction in which the data sits and is handled.
Data residency is about place: the country or region in which submitted data is stored and processed. It matters because the applicable law follows the location, which is why organisations under regional data-protection rules ask for it before sending anything to a hosted service. Retention periods, encryption in transit and provider access controls are separate commitments that have to be checked on their own.
In the context of using generative AI for test work, what does keeping a human in the loop mean?
A person reviews and approves each generated artefact before it is used, and remains answerable for the result.
Correct. Review before use, with accountability staying with the person, is the substance of the idea.
A person is available to answer questions from the model whenever it reports that it lacks information.
Incorrect. The model does not reliably report what it lacks, and availability is not review.
A person monitors the provider's service status and intervenes when the assistant becomes slow or unavailable.
Incorrect. That is operational monitoring and has nothing to do with the content of the output.
A person writes every prompt by hand rather than reusing a stored template, so each request reflects current intent.
Incorrect. How the prompt is written does not address what happens to the output afterwards.
Keeping a human in the loop means a person reviews and approves what the model produces before it is used as a test artefact, and carries the responsibility for the outcome. The point is accountability: the model contributes drafts, the person decides what enters the suite, the test plan or the defect report, and that decision is where the quality of the result is established.
A team uses an assistant to generate automation helper code. Legal asks what should happen before such code is merged into the product repository. Which answer is best?
Nothing more than the usual code review, because code produced by a model is new text and therefore carries no licence obligations of its own.
Incorrect. Output can reproduce licensed material, so novelty cannot be assumed.
Run the same licence and provenance scan the team applies to third-party code, since generated code can reproduce material from elsewhere.
Correct. The existing provenance control is the right one to extend to generated code.
Record the prompt in the commit message, so that anyone questioning the authorship of the code can see how it was produced.
Incorrect. Documenting the origin is useful but does not detect licensed material in the result.
Ask the provider for written confirmation that the output of each request is free of third-party material before the merge.
Incorrect. Providers do not certify individual outputs, so this does not produce a usable control.
Generated code is produced from patterns learned across public repositories, and it can reproduce recognisable fragments of code that carries licence conditions. That makes it a provenance question of the same kind the team already handles for third-party dependencies, so the existing licence and provenance scanning applies to it. Recording the prompt documents how the code arose but establishes nothing about what it contains, and no provider undertakes to certify each output.
A generated test fixture that turned out to contain a real customer record was committed to a public repository three days ago. Which TWO actions belong in the immediate response?
Report it through the organisation's data incident process, because the exposure is a disclosure and not just a repository mistake.
Correct. Disclosure of personal data triggers the incident process, which decides on notification.
Delete the fixture file in a new commit and close the issue, because the current state of the repository no longer carries the data in it.
Incorrect. The commit history still contains the record and remains publicly reachable.
Change the prompt so that future fixtures are generated without personal data, and treat that change as the remediation.
Incorrect. That is a sensible preventive step but leaves the current exposure unaddressed.
Ask the assistant to confirm whether the record it produced corresponds to a real customer, and act on the answer it gives.
Incorrect. The model cannot check this, and the answer would be another generation.
Remove the record from the repository history as well as from the current files, since the commit remains reachable otherwise.
Correct. Only rewriting the history removes the data from what the public can retrieve.
Personal data in a public repository is a disclosure, so the organisation's incident process applies and the decision about notification is not the team's to make informally. Technically, deleting the file in a new commit leaves the record reachable in the history, so the history has to be rewritten and any mirrors and caches dealt with. Changing the prompt prevents a repeat but does nothing about the exposure, and asking the model whether the record is real produces a guess.
A test organisation has built a prompt library and a fine-tuned model around one provider, and now wants to roll that setup out to four more teams. Which risk deserves assessment before the roll-out?
That the provider's model will become slower as more teams use it, since the account shares a single capacity allocation.
Incorrect. Capacity is a contractual and operational matter, and it is readily addressed.
That the four new teams will write prompts of lower quality, which would reduce the measured benefit across the whole organisation.
Incorrect. Quality varies between teams in any practice; it is a training matter, not the structural risk here.
That the investment is tied to one provider, so a change in price, terms or model behaviour would be costly to move away from.
Correct. Scaling deepens the dependency, which is what makes the exit cost worth assessing first.
That a larger number of users makes it harder to keep the prompt library in a single repository under one review process.
Incorrect. This is an ordinary scaling inconvenience rather than a risk to the investment.
Prompts tuned to one model's behaviour, and a fine-tuned variant that exists only on one platform, are investments that do not transfer. Scaling multiplies them, so a later change of price, licence terms, model behaviour or availability becomes correspondingly expensive to absorb. Assessing that exposure before the roll-out usually means keeping prompts portable, recording what each depends on, and knowing what moving would cost.
A team's draft rule says "no production data in the assistant". A tester then pastes an unreleased product specification, which contains no personal data, into a public assistant. What does this show about the rule?
The rule is adequate and the tester simply broke it, since an unreleased specification is a form of production data.
Incorrect. A specification is not production data under any usual reading, which is exactly the gap.
The rule should be replaced by a technical control, because written rules are not a workable basis for handling confidential material.
Incorrect. Technical controls help, but the defect here is the scope of the rule rather than its form.
The rule is correct but needs an exception process, so that a tester with a legitimate need can obtain approval in advance.
Incorrect. An exception process addresses approved deviations, not material the rule never mentions.
The rule names one data type and misses classification, so confidential material with no personal data falls outside it.
Correct. Framing the rule on classification covers the sensitive material the data-type wording omits.
The rule is written around one category of data and therefore says nothing about material that is sensitive for another reason. An unreleased specification is commercially confidential even though no personal data appears in it, and the same applies to architecture notes, contract terms and security findings. A workable rule refers to the organisation's classification of the document, so that every class of sensitive material is covered rather than one.
A team proposes that the assistant which drafts a test plan should also carry out the first review of that plan, in order to save reviewer time. Which TWO objections are sound?
The review would take about the same amount of time as the drafting step, so the saving the team expects does not materialise in practice.
Incorrect. The machine review is cheap; the problem is what it fails to catch, not what it costs.
An error that comes from the model's own reading of the material is likely to survive a review performed by the same model.
Correct. The failure modes are correlated, so the review misses what matters most.
The review adds a step without adding an independent check, so the plan reaches the human reviewer with unearned credibility.
Correct. An apparently reviewed artefact invites a lighter human reading than it deserves.
A model is unable to evaluate text at all, so any review step performed by one returns a result chosen at random.
Incorrect. Model review has some value; the objection is to its lack of independence here.
The provider's terms forbid using one generated artefact as the input to a second request within the same account.
Incorrect. No such restriction applies, and the objection does not rest on contractual terms.
A review is worth having when it can fail differently from the work it examines. A model reviewing its own output shares the reading of the material that produced the error, so the errors most worth catching are the ones most likely to survive. Worse, a plan that has been through a review step arrives at the human reviewer looking checked, which encourages a lighter reading than it would otherwise get. The objection is about independence, not about cost or about models being incapable of evaluation.
At a release sign-off, a test manager presents "900 test cases executed, 640 of them generated this sprint" as evidence of coverage. Why is that a weak argument?
A count says nothing about what the cases exercise, and generated cases may duplicate one another or test nothing in scope.
Correct. Volume is not coverage, and generated sets are especially liable to duplication.
Generated cases execute faster than written ones, so the figure overstates the effort the team actually spent on the release.
Incorrect. Execution time is unrelated to how the case was produced, and effort is not the claim.
The figure mixes two populations, and a sign-off should quote generated and manually written cases as two separate totals in the report.
Incorrect. Splitting the total gives two counts rather than one and still shows no coverage.
A proportion of generated cases above half is an accepted warning sign, and 640 of 900 is well beyond that threshold.
Incorrect. There is no such accepted threshold; the proportion is not what makes the argument weak.
Coverage is a statement about what has been exercised, and a count of cases is not one. Generated sets are particularly prone to near-duplicates and to cases that restate a requirement without testing it, so a large number can coexist with sizeable gaps. The evidence a sign-off needs is the mapping from requirements or risks to the cases that exercise them, together with what the review of those cases found.
A test team supports a public-sector tender response. Three facts hold: the bid…
A test team supports a public-sector tender response. Three facts hold: the bid documents are commercially confidential until award; the assistant the team wants to use is a public consumer product with no contract in place; and the audit regime requires every test artefact to name its author. One tester has already pasted part of the bid into the assistant. Which set of controls is the right response?
Instruct the team to stop using the assistant entirely for the duration of the tender, and revisit the question once the award is published.
Incorrect. Stopping use addresses the future but leaves the disclosure that has already happened unhandled.
Treat the paste as an incident, move to a contracted tool with confidentiality terms, and record a named reviewer for every generated artefact.
Correct. Each of the three facts is answered by one of the three controls.
Allow continued use with a rule that the bid text is paraphrased before it is pasted, and name the assistant as the author of generated artefacts.
Incorrect. Paraphrased confidential content is still confidential, and a tool cannot be an author.
Allow continued use, since no personal data is involved, and add a note to the test plan stating that parts of it were produced with an assistant.
Incorrect. Commercial confidentiality applies independently of personal data, and a blanket note is not attribution.
Each fact drives one control. Confidential material has left the organisation with no contract behind it, so the paste is an incident to be reported rather than a lapse to be noted. Future use needs a tool whose terms cover confidentiality and retention, which the consumer product does not. And because the audit regime requires a named author, generated material has to be attributed to the person who reviewed and accepted it. Paraphrasing does not make confidential content safe to send, and no tool can be named as an author.
Chapter 4 · LLM-Powered Solutions for Software Testing — 5 questions
A retrieval-based assistant answers questions about the team's test policy. Since the policy was revised last month it has been confidently quoting the superseded rule. What is the most likely cause, and the right fix?
The model's training data predates the revision, so the fix is to wait for the provider's next model release, which will include the new wording.
Incorrect. In a retrieval setup the answer comes from the supplied passages, not from training data.
The chunk size in the index is too large, so the revised passage is diluted; the fix is to re-chunk the whole policy document.
Incorrect. Chunking affects which passage matches, but the revised text is not in the index at all.
The index still holds the old document, so retrieval returns the superseded passage; the fix is to re-index after each change.
Correct. A stale index serves stale passages, and freshness is maintained by re-indexing.
The prompt does not state the current date, so the fix is to include the date in every question asked about the policy.
Incorrect. A date in the prompt cannot conjure a passage that the index does not contain.
In a retrieval setup the model answers from the passages it is given, so an answer that quotes an old rule usually means the old passage is what was retrieved. The index was built before the revision and has not been rebuilt, which makes this a freshness problem in the retrieval layer rather than a defect in the model. Re-indexing on change, and knowing how stale the index may be, is part of operating such a system.
A retrieval assistant answers questions about the test strategy and appends a citation to a document section under each answer. What do those citations establish?
That the answer was composed from the cited section and from nothing else, since retrieval restricts the model to the supplied passages.
Incorrect. The model still draws on what it learned in training alongside the retrieved text.
That the cited section has been reviewed and approved, because an approval step governs what is admitted to the index.
Incorrect. What enters the index is a configuration choice and says nothing about review status.
That the answer would be reproduced word for word by anyone who read the cited section for themselves.
Incorrect. The answer is a generated paraphrase, not a quotation of the source.
That the section was retrieved for the question; whether the answer follows from it still has to be checked by opening the section.
Correct. The citation records retrieval and makes verification cheap; it is not verification itself.
A citation records which passage the retrieval step selected and handed to the model. It does not establish that the generated answer follows from that passage: the model can overstate it, combine it with material from elsewhere, or state something the passage does not contain. The citation is valuable precisely because it makes checking cheap, so the discipline is to open the section when the answer matters rather than to treat the reference as a verdict.
An assistant built on retrieval over the team's documentation gives a poor answer. Which TWO observations point at the retrieval step rather than at the model?
The answer is fluent and well structured but reaches a conclusion that the cited passages do not support in any way.
Incorrect. Correct passages and an unsupported conclusion point at the generation step.
The same question asked twice in a row returns two differently worded answers drawn from the same cited passages.
Incorrect. Wording variation across runs is ordinary generation behaviour.
The passages shown with the answer do not contain the fact asked about, although the document set does contain it.
Correct. The material exists but was not retrieved, which locates the failure in the search.
The answer is cut off part-way through the third paragraph when the question asks for a long explanation.
Incorrect. Truncation is an output length limit and has nothing to do with retrieval.
Pasting the relevant passage straight into the prompt produces a correct answer to the same question as before.
Correct. The model handles the material once it has it, so the gap was in getting it there.
Retrieval and generation fail in different ways, and the question is which stage was starved. If the passages handed to the model do not contain the fact, although the document set does, the search did not find it. If supplying the passage by hand fixes the answer, the model could use it once it had it. An unsupported conclusion drawn from correct passages, run-to-run wording differences and a truncated reply all sit on the generation side.
A team maintains a set of question and expected-answer pairs and runs it after every change to the prompts or to the index of its retrieval assistant. What does that practice mainly protect against?
A change that improves one kind of question while quietly breaking answers that worked before and that nobody rechecks.
Correct. The set makes silent regressions from a prompt or index change visible.
The provider replacing the underlying model without giving notice, which the set would detect before any request has been sent.
Incorrect. The set only reveals a change once it is run, which is after requests are sent.
Users asking questions that the documentation does not answer at all, which the evaluation set identifies as being out of scope.
Incorrect. Scope is defined by what goes into the index, not by a regression set.
The index growing beyond the size that the retrieval component is able to search within an acceptable response time.
Incorrect. That is a performance concern, measured by latency rather than by answer quality.
Changes to a prompt, a chunking rule or an index are rarely improvements across the board: a wording that helps one class of question often degrades another, and because each answer looks plausible on its own, the loss goes unnoticed. A fixed set of question and answer pairs turns that into a regression check, in the same way a test suite does for code. It does not police the provider, scope or index size.
An internal retrieval assistant is built over a document store that also holds HR and legal material which most testers are not entitled to read. What is the right way to keep those documents from reaching the wrong user?
Add an instruction to the system prompt telling the model not to disclose HR or legal material to users outside those departments.
Incorrect. The content has already been supplied to the model, and such instructions can be circumvented.
Apply the user's own permissions at the retrieval step, so that documents the user is not entitled to read are not passed on.
Correct. Enforcing access where documents are selected removes the exposure entirely.
Run an output filter that removes any answer mentioning HR or legal topics before it is displayed to the person who asked.
Incorrect. Topic filtering is coarse, blocks legitimate answers and still lets paraphrases through.
Index the restricted documents with a marker in the text, so the model can recognise them and decline to quote their content.
Incorrect. The marker relies on the model choosing to honour it, which is not an access control.
Confidentiality in a retrieval system is enforced where the documents are selected, not where the answer is written. If the search runs under the asking user's own permissions, restricted material is not handed to the model and so cannot appear in an answer. Instructions in a system prompt, output filters and markers in the text are all attempts to persuade or police a generation that has already been given the content, and each of them can be worked around.
Chapter 5 · Deploying and Integrating GenAI in Test Organizations — 7 questions
Some organisations set up a small cross-team group when introducing generative AI into testing. What is the purpose of such a group?
To write the prompts that the test teams then use, so that individual testers are not required to write prompts themselves.
Incorrect. Teams write prompts for their own context; the group shares what works rather than replacing them.
To negotiate the commercial terms with model providers on behalf of every department in the organisation.
Incorrect. Procurement may be involved, but that is not what the group is set up to do.
To agree the rules, approved tools and shared practices, and to keep them current as teams gain experience.
Correct. The group settles the cross-team decisions and revises them as practice develops.
To approve each individual request before it is sent, so that no unreviewed prompt reaches an external provider.
Incorrect. Per-request approval is unworkable and is not the role such a group takes.
The group exists so that the decisions which should hold across teams are made once: which tools are cleared, what the rules for confidential material are, what good practice looks like, and how all of that is revised as experience accumulates. It coordinates rather than executes, which is why it neither writes the teams' prompts for them nor sits in the path of individual requests.
An organisation publishes an approved-tool list for generative AI. What does such a list do?
It ranks the available assistants by measured quality, so that teams can pick the best performer for each kind of activity.
Incorrect. Approval is about assessment and clearance, not about a performance ranking.
It records which tools each team is currently using, so that spending can be reported centrally each quarter.
Incorrect. That is a usage or licence inventory, which serves a different purpose.
It states which prompts are permitted, so that testers can select one rather than composing their own.
Incorrect. A prompt library is a separate artefact from a list of cleared tools.
It names the tools assessed and cleared for use, so a tester knows what may be used without further approval.
Correct. The list communicates what has been cleared and makes anything else an exception.
An approved-tool list records which assistants have been through the organisation's assessment, covering contract terms, data handling and suitability, and have been cleared for use. Its practical value is that a tester can act without seeking approval each time, and that anything outside the list is visible as an exception. It is neither a quality ranking, nor a usage inventory, nor a catalogue of permitted prompts.
The syllabus describes how a tester's work changes once generative AI is in regular use. Which statement describes that change?
More of the work goes into specifying what is wanted and verifying what comes back, and less into producing the first draft.
Correct. Specification and verification grow as drafting shrinks.
The work moves away from test design and towards the administration of the tool and the management of its licences.
Incorrect. Administration is a side task; test design judgement becomes more important rather than less.
The work becomes largely unchanged, since the assistant operates as an additional execution environment.
Incorrect. An assistant drafts artefacts; it is not an execution environment.
The work shifts towards writing more detailed test scripts, because generated cases tend to require longer step descriptions.
Incorrect. Generated cases do not need longer steps, and script length is not the change described.
Drafting is the part a model can take over, so the effort moves to the two ends of the activity: stating precisely what is wanted, with the context and constraints that make the request answerable, and judging what comes back against the test basis. Both ends demand more test design knowledge and more domain knowledge than before, not less, which is why the change is described as a shift in emphasis rather than a reduction in skill.
Alongside its approved-tool list, an organisation defines an exception process. What is that process for?
For recording which testers have completed the mandatory training before they are given access to any of the approved tools.
Incorrect. That is a training record, which is managed separately from exceptions.
For a team needing a tool outside the list to request a reviewed, time-limited approval instead of proceeding unofficially.
Correct. The process makes a deviation visible, reviewed and bounded.
For reporting a generated artefact that turned out to be wrong, so that the tool in question can be reassessed afterwards.
Incorrect. That is feedback or incident reporting rather than an exception to the list.
For granting a team permission to skip the review of generated artefacts when a release deadline is close.
Incorrect. No exception process waives the review of generated material.
An approved list is always behind what teams want to try, and a list with no way around it produces quiet workarounds instead of compliance. The exception process gives a team a route to use something outside the list: it states the need, the risk is reviewed, and the approval is bounded in time and scope. The value is that the deviation is visible and reviewed rather than hidden, which is also why it never covers skipping review.
A safety-related project requires that test cases be reviewed by someone other than their author. The team wants to use an assistant to draft the cases. What follows for the review?
The requirement is met, because the author of a generated case is the model and the tester reviewing it is a different party.
Incorrect. A tool is not an author; accountability rests with the person who accepted the case.
The requirement cannot be met while an assistant is used, so generated cases are not admissible in such a project at all.
Incorrect. Drafting with an assistant is compatible with the standard once the review is independent.
The tester who prompted and accepted the cases counts as their author, so a second person is still needed for the review.
Correct. Authorship follows accountability, so the independent review remains a human one.
The requirement applies to the prompt rather than to the cases, so a second person should review the prompt that produced them.
Incorrect. The artefact under review is the test case; reviewing the prompt does not replace that.
Authorship in such a standard attaches to the person who is accountable for the artefact, and that is the tester who framed the request and accepted the result, not the tool that produced the draft. The independent review therefore still has to come from a second person. Nothing here rules the assistant out: it can draft, and the team gains the time the drafting would have taken, as long as the review stays where the standard puts it.
After a year of using generative AI in testing, a head of quality reviews what the team now needs from its people. Which TWO expectations are consistent with the way the work has changed?
Test design and domain knowledge matter more, since judging a generated artefact means knowing what a right one looks like.
Correct. Evaluation demands the knowledge that drafting used to exercise.
Specifying a request precisely and evaluating what comes back become skills worth developing and worth hiring for.
Correct. These are the activities that grow once drafting is delegated.
Testers need less knowledge of test techniques, since the assistant applies the techniques on the team's behalf anyway.
Incorrect. The model imitates technique output without reliably applying it, so the tester must recognise failures.
The team needs at least one member able to retrain the provider's model on the organisation's own defect history.
Incorrect. Few teams retrain models, and nothing in the change makes that a core skill.
Writing test cases by hand becomes the main skill to assess at interview, since generated cases are no longer being used at all.
Incorrect. The premise is wrong, and hand-writing speed is not what distinguishes good work here.
When drafting is cheap, the scarce skills are the ones at either end of it. Judging whether a generated test case, charter or defect report is any good requires knowing the techniques and the domain well enough to see what is missing, so that knowledge becomes more valuable rather than less. Alongside it sits the ability to state a request precisely and to evaluate the answer. Nothing in the change argues for retraining models in-house or for treating hand-written cases as the interview criterion.
After a successful year in one test team, an organisation wants a second team to reach the same result. What makes that result most likely to transfer?
Giving the second team the same tool licences, since the benefit came from having access to the assistant in the first place.
Incorrect. Access is where the first team started, not what it learned.
Moving two members of the first team into the second one permanently, so the practice travels with the people.
Incorrect. This spreads the knowledge by weakening its source and leaves it undocumented.
Setting the second team the same productivity target, so that it has a clear measure to work towards from the start.
Incorrect. A target without a method tells the team what to achieve but not how.
Handing over the documented prompts, the workflow they sit in and the review step, not just the tool the first team used.
Correct. The practice, written down, is what actually carries the result across.
What the first team gained was not access to a model but a way of working: prompts that proved reliable for particular activities, the point in the process where each is used, and the review that catches what the model gets wrong. Those are the transferable parts, and they transfer only if they have been written down. Licences alone reproduce the starting conditions rather than the practice, moving people drains the first team, and a target without a method is pressure rather than help.