ISTQB Specialist (CT-GenAI v1.1) Mock Exam #1 — Questions & Answers
Every question in this mock exam, with the correct answer marked and a written rationale behind each option below — for reading and review, not a timed run.
Chapter 1 · Introduction to GenAI for Software Testing — 7 questions
Which statement describes what a foundation model is?
A rule-based engine that stores hand-written response templates and selects one of them according to the keywords found in the user's input.
This describes a classic template or chatbot engine. Its answers are authored in advance by people, whereas a foundation model generates text it was never given verbatim.
A model trained on one labelled dataset for a single defined task, such as sorting incoming support tickets into three product categories, and retrained from scratch whenever a fourth category is added.
This describes a task-specific supervised model, right down to the retraining cost. It is trained for one purpose and cannot be redirected to another task by prompting, which is exactly the property a foundation model has.
A benchmark suite that vendors use to compare the response quality of competing language models against a fixed, published set of prompts.
This describes an evaluation harness, which measures models rather than being one. Benchmarks are an input to model selection, not a category of model.
A large model trained on a broad, mostly unlabelled corpus, which can then be adapted to many downstream tasks without being retrained from scratch.
Correct. The defining traits are the breadth of the training corpus and the fact that one trained model serves many tasks through prompting, retrieval or light fine-tuning.
A foundation model is trained once on a very broad corpus and is then adapted, by prompting, retrieval or fine-tuning, to many different downstream tasks. That is what separates it from a model built for one labelled task, from a template engine, and from an evaluation suite.
A tester asks an LLM how many times the letter "r" appears in a short product name. The model answers with the wrong number, even though it summarises a ten-page specification for the same product accurately. Which property of the model best explains the difference?
The model reads text as tokens rather than as individual characters, so character-level counting is not something its input representation supports reliably.
Correct. A token often covers several characters, and the mapping from tokens back to letters is not part of what the model computes over, which is why letter counting is weak while meaning-level tasks are strong.
The product name exceeded the model's context window, so the start of the input was truncated before the model had a chance to read the whole string.
Context windows are measured in thousands of tokens and a product name is a handful. The far longer specification in the same question was processed without truncation, which rules this out.
The temperature setting was too low, which makes the model repeat the most frequent answer seen during training instead of computing the answer for this input.
Temperature controls how much randomness is applied when sampling the next token. A low value makes output more repeatable, but it does not turn a counting task into a lookup of training data.
Counting is an arithmetic operation, and the model delegates all arithmetic to an external calculator service that was unreachable when this request was made.
A plain language model performs no such delegation by itself. Tool use of that kind has to be built around the model, and its absence would produce an error or a tool-call attempt, not a confident wrong count.
Text reaches the model as tokens, not as a sequence of characters, so character-level questions fall outside what the input representation exposes. Semantic tasks such as summarising work on exactly the representation the model does have.
A test team stores the exact prompt that produced a set of test cases so that a colleague can reproduce the result later. Running the same prompt against the same model, the colleague receives a noticeably different set of test cases. What should the team conclude about working with generative AI?
Generated output is not reproducible from the prompt, so the generated test cases have to be stored and version-controlled as artefacts in their own right rather than regenerated on demand.
Correct. Because generation samples from a distribution, the prompt records intent but not result. Treating the output as the artefact keeps the test basis stable and reviewable.
The prompt was too short to constrain the model; a longer prompt fixing the output format and the number of cases would have made the two runs return an identical set, because constraints of that kind leave the model no freedom.
A more constrained prompt narrows the variation and is good practice, but it does not remove sampling. Two runs of a well-specified prompt can still differ in wording and in which cases are included.
The colleague worked from a different account, and providers assign each account its own separately fine-tuned copy of the model with its own wording habits.
Accounts share the same deployed model. Differences in output between two accounts come from sampling or from configuration such as temperature, not from a private copy of the model per account.
The provider updated the model between the two runs, which is the usual explanation when two requests made within a few minutes of each other return different text.
Model versions change on a scale of weeks or months, not minutes, and providers normally pin a named version. Sampling is the ordinary explanation for run-to-run variation.
Generation involves sampling, so the same prompt can yield different output on different runs. The prompt is therefore not a reproducible substitute for the artefact: the generated test cases themselves have to be stored, reviewed and placed under configuration management.
A test manager is briefing the team before they start using a general-purpose LLM in their daily work. Which TWO statements about what such a model produces should be part of that briefing?
Output is generated as a plausible continuation of the prompt, so how fluent and confident an answer reads is independent of whether it is factually correct.
Correct. Fluency comes from the language model objective, correctness does not follow from it, and this is why a reviewer cannot use the quality of the prose as a signal.
The model keeps the documents it was trained on and returns the matching passage for each request, which makes it usable as a lookup over that material.
Training adjusts model weights rather than building a document store, so there is no passage to return. Retrieval over a document set is a separate component that has to be added around the model.
The model has a training cut-off, so anything released after that date has to be supplied in the prompt or fetched at run time by a retrieval step.
Correct. Without supplied context the model answers from what it learned, which is why questions about a recent release or a new internal process need the material provided alongside the question.
The model checks each answer against an internal correctness test before responding and withholds the response when that check does not pass.
There is no such internal verifier in a plain language model. Checking an answer against a source is work that has to be built around the model or done by a reviewer.
The model applies the organisation's own coding and documentation standards by default, because standards of that kind were part of the general training corpus.
General training gives the model common industry conventions, not one organisation's internal standards. Those have to be stated in the prompt or enforced by review.
Two properties shape everything else a tester does with an LLM: output is generated as plausible continuation text rather than retrieved from a store of verified facts, and the model's knowledge stops at its training cut-off. Both push the team towards grounding the model in supplied material and reviewing what comes back.
A tester pastes a 200-page requirements specification into one chat message and asks for test conditions covering the whole document. The reply covers the opening chapters in detail and leaves most of the later chapters untouched. Which explanation fits best, and what follows from it?
The later chapters use tables and numbered lists, and the model converts those structures to plain text before reading them, which strips their content out of the answer.
Tables and lists are flattened to text, but flattening keeps the wording. Structure is lost, content is not, so this would change how the material is phrased rather than whether it appears.
The input ran past the length the model can attend to usefully, so the later material was dropped or weakly weighted, and the work has to be split into sections small enough to process one at a time.
Correct. Attention over a very long input degrades towards the far end even without hard truncation, and splitting the source is the standard countermeasure in a test-analysis workflow.
Test conditions come from functional requirements, and the later chapters of a specification describe non-functional aspects, which fall outside what a request for test conditions asks the model to produce.
Test conditions are derived from non-functional requirements as well, and nothing in the request restricted the scope. The position of the omitted material, not its type, is what points at the cause.
A single request returns a fixed number of items regardless of how much input it was given, so the list was cut short once that number was reached.
There is a limit on how much text the model emits, but it is not a fixed item count, and a response limit would truncate the list at the end rather than skew coverage towards the opening chapters.
Very long inputs run past what the model can attend to usefully, and material at the far end of a long input gets weak treatment even when it is not truncated outright. The practical response is to split the document into portions the model handles well and to work section by section.
A team asks an LLM to write test scripts against a library API that changed three months ago. The generated scripts call a method that was removed in that change. What is the most appropriate way to deal with this?
Ask the model to confirm that the method exists before it generates the script, and treat a positive confirmation as evidence that the script is current.
The confirmation is generated from the same outdated knowledge as the script, so it restates the error with added confidence. Self-confirmation is not an independent check.
Raise the temperature so that the model explores less frequent completions, which improves the chance of it reaching the newer method name somewhere in the space of candidates it treats as plausible for this position in the code.
Temperature changes how the sampler picks among candidates the model already considers likely. A method the model has not seen is not among them, so more randomness produces more varied wrong calls.
Provide the current API documentation in the prompt or through a retrieval step, because the change happened after the training cut-off and the model otherwise falls back on the interface it learned.
Correct. Supplying the current material is the direct remedy for a knowledge cut-off, and it also gives the reviewer something concrete to check the generated calls against.
Repeat the request several times and keep whichever version compiles, since a failure to compile identifies the outdated calls at no extra cost.
Compilation catches a removed method but says nothing about a method whose signature or behaviour changed, and repeating a request grounded in old knowledge tends to reproduce the same old calls.
The model answers from what it learned up to its training cut-off, so a change made after that date is invisible to it. Putting the current interface documentation into the prompt, or retrieving it at run time, is what moves the model onto the current API.
Why is it misleading to describe a hallucination as a defect in the model that the vendor could remove with a patch?
Hallucinations appear when the prompt contains spelling or grammar mistakes that the model has to guess its way through, so the problem sits in the prompt and is corrected by writing prompts more carefully.
A clean, well-written prompt produces unsupported content just as readily. Prompt quality changes how often it happens, not whether the mechanism that causes it is present.
A hallucination shows up as malformed or truncated output, so it is caught by the same validation that catches formatting problems in a generated document before anyone reads it.
Unsupported content arrives in well-formed, fluent prose, which is precisely what makes it hard to spot. Format validation passes it through untouched.
Vendors record hallucinations as enhancement requests rather than as defects, which places them outside the defect management process and therefore beyond the reach of any patch the vendor might ship in a future release of the model.
How a vendor classifies an issue in its tracker is a process detail and says nothing about whether the behaviour can be engineered away. The reason lies in how generation works.
Generation predicts plausible continuation text instead of looking facts up, so confident but unsupported content follows from the mechanism itself and is managed by grounding and review rather than removed.
Correct. Because plausibility and factual support are different properties, the behaviour is inherent. Retrieval, supplied context and human review reduce the exposure rather than eliminate the cause.
Hallucination is a consequence of the generative mechanism: the model produces the most plausible continuation, and plausibility is not the same as support in a source. It is therefore managed, by grounding the model in supplied material and by reviewing output, rather than fixed.
Chapter 2 · Prompt Engineering for Effective Software Testing — 11 questions
Before asking for test conditions for a new requirement, a tester puts three worked examples of correctly written test conditions from the team's existing suite into the prompt. Which technique is this, and what does it mainly achieve?
Few-shot prompting: the examples show the model the format, granularity and vocabulary expected, so the generated conditions match the ones the team already maintains.
Correct. A handful of in-prompt examples is the definition of few-shot, and matching an existing house style is the effect that makes it valuable in test analysis.
Chain-of-thought prompting: the examples make the model set out its intermediate reasoning before answering, which raises accuracy on tasks that take several steps to solve.
Chain-of-thought is about asking for the reasoning steps, not about showing finished examples. It would help on a derivation task, but it is not what adding sample outputs does.
Fine-tuning: the examples enter the model's training data for this project, so later sessions reproduce the same style without the examples being repeated.
Fine-tuning changes model weights through a separate training run. Content placed in a prompt affects that request only and is gone when the session ends.
Retrieval-augmented generation: the examples serve as the retrieved context that grounds the answer in the team's own documents.
Retrieval-augmented generation fetches the context automatically from a document store at request time. Here the tester chose and pasted the examples, so nothing was retrieved.
Supplying a few worked examples in the prompt is few-shot prompting. The examples communicate format, granularity and vocabulary far more efficiently than a description of them, which is why the generated items come back looking like the team's own.
A prompt opens with "You are a test analyst reviewing an acceptance criterion for testability." What does assigning a role in this way mainly change about the response?
It sets the perspective, vocabulary and level of detail the answer is written from, which moves the content towards the concerns that role would raise.
Correct. Role framing conditions the response on a body of language associated with that role, so a review comes back in terms of testability, ambiguity and measurability rather than in general terms.
It confines the model to the part of its training data written by people holding that job title, which cuts down the amount of off-topic material in the answer.
Training data is not partitioned by author role and cannot be selected from at request time. The role influences the style and focus of generation, not which data the model may draw on.
It raises the confidence threshold the model applies, so statements it cannot support from its own knowledge are left out of the answer.
There is no confidence threshold of that kind to raise. A role framing can make an answer sound more authoritative without making it better supported, which is a trap rather than a safeguard.
It gives the session access to role-specific tools such as the requirements repository, which the model then queries while composing its answer.
Tool access is configured around the model by whoever builds the integration. Naming a role in the prompt grants nothing, however the role is described.
A role sets the perspective the answer is written from: which concerns are treated as relevant, what vocabulary is used and how much detail is appropriate. It steers the content of the answer; it does not give the model new knowledge, new certainty or new access.
A team is turning an ad-hoc prompt into a reusable template for generating test cases from a user story. Which TWO additions do the most to make the output usable without rework?
A request for the model to rate its own confidence in each generated case, so that the reviewer can move quickly past the cases it rated highly.
A self-assigned confidence score is generated text like the rest of the answer and correlates poorly with correctness. Using it to decide what to skip concentrates review effort in the wrong place.
A statement of the required output structure and fields, so the generated cases can go into the test management tool without being restructured by hand.
Correct. Format is the cheapest thing to specify and the most expensive thing to repair afterwards, and a fixed field list also makes it obvious when the model has left something out.
An instruction to be creative and to avoid reusing phrasing from the user story, which broadens the variety of the generated test cases and keeps the set from reading as a paraphrase of the acceptance criteria.
Pushing the wording away from the user story makes traceability harder and invites cases about behaviour the story does not describe. Variety that is not anchored in the test basis costs review time.
The acceptance criteria and the relevant business rules pasted into the prompt, so the model works from the project's own material rather than from general assumptions.
Correct. Supplying the test basis is what turns a generic answer into one about this system, and it gives the reviewer a source to check the generated cases against.
A note that a senior tester will review the output, which raises the care the model applies when it chooses its wording.
Telling the model who reads the result changes tone at best. The model has no separate careful mode to switch into, and the claim invites the team to expect quality it has not asked for concretely.
The two things that reliably cut rework are telling the model what shape the answer must have and giving it the project's own material to work from. Self-rated confidence, instructions to be creative and notes about who will review the output add words without changing what the model can do.
A tester adds "work through this step by step before giving your answer" to a prompt that asks the model to derive equivalence partitions from a set of business rules. Which statement best describes what this does and what it does not do?
It lengthens the response and slows generation without changing the result, so accuracy on multi-step tasks stays level with what a direct answer would have produced and the extra text only costs review time.
The response does get longer, but on tasks that decompose into stages the step-by-step form generally improves the result, which is the reason the technique is used at all.
It usually improves the result on tasks that take several stages and makes the reasoning reviewable, but the printed steps are generated text rather than a record of how the answer was computed.
Correct. The steps are useful to a reviewer because they can be checked against the rules, and treating them as an audit trail is the mistake the technique invites.
It makes the response reproducible from one run to the next, because the model has to follow the steps it stated at the outset, although it adds nothing on questions that take a single step to answer.
Sampling still applies to the steps themselves, so two runs can take different routes and reach different partitions. The observation about single-step questions is fair, but the claim about reproducibility is not.
It lets the model hand each stated step to an external reasoning engine and read the result back, which is why the improvement is largest on tasks that involve arithmetic or counting.
No external engine is involved unless one has been wired in deliberately. The improvement comes from the model conditioning on its own intermediate output as it generates.
Asking for the intermediate steps usually helps on tasks that need several stages and gives the reviewer something to check. What it does not give is an audit trail: the printed steps are generated in the same way as the rest of the answer, so they explain the answer rather than record how it was produced.
A tester asks a model for fifty rows of synthetic customer records to load into a test environment. The reply is a readable prose description of fifty fictional customers, one paragraph each. Which change to the prompt addresses the problem most directly?
Ask for the records in batches of ten, because a long answer tends to drift away from the requested shape and back towards prose as the model works through the list.
Smaller batches help with truncation and with quality drift over a long list, but the first paragraph was already prose. The shape was never specified, so every batch would come back the same way.
Increase the number of records requested, since a larger volume moves the model towards a compact tabular form instead of full sentences.
Volume does not select a format. A request for five hundred prose descriptions returns five hundred prose descriptions, or a truncated list of them.
Add an instruction that the data has to be realistic, which leads the model to imitate the layout of a production export.
Realism is about the values, not the layout, and it is a separate concern worth stating. It does not tell the model which columns to emit or how to delimit them.
State the exact output structure: the field names, their order, the separator between fields, and an instruction to return the data rows and nothing else.
Correct. Format instructions are what the model acts on directly, and the closing instruction to return nothing else removes the commentary that otherwise has to be stripped before loading.
The model produced valid content in the wrong shape because the prompt described the content and left the shape open. Stating the field names, their order, the delimiter and the instruction to return nothing else is what makes the output loadable.
Instead of one prompt asking for regression tests for a checkout flow, a team…
Instead of one prompt asking for regression tests for a checkout flow, a team runs a chain: extract the business rules from the specification, then derive test conditions from the extracted rules, then write test cases for each condition, with a short review after every step. Which TWO benefits does the chain give over the single prompt?
Each step produces a small, checkable output, so a mistake is caught where it appears instead of being carried forward into a long final answer.
Correct. A wrong business rule extracted in step one would otherwise reappear silently in every test case built on it, and a reviewer reading forty finished cases is unlikely to trace it back.
The model keeps what it worked out at each step, so it is better calibrated to this product by the time the team reaches the next feature.
Nothing is retained between requests unless it is passed back in the prompt. Any carry-over across features comes from the team reusing its own saved artefacts, not from the model learning.
The specification is read once in the first step, so the later steps run without the test basis in context and stay cheaper to execute as the chain gets longer and the artefacts accumulate.
Dropping the test basis after step one is how detail gets lost: derived conditions still need checking against the source, and the later steps commonly need the original wording in context.
The total volume of text processed falls, because each step sees a fraction of the material that the single combined prompt would have needed.
Chaining normally processes more text overall, since instructions and shared context are repeated in each call. The gain is in reviewability, not in volume.
The intermediate artefacts are reusable: the extracted rules and the test conditions feed review, risk analysis and traceability without being regenerated.
Correct. The chain produces the same work products a manual test analysis would, which is what lets the result be maintained and traced rather than treated as one disposable answer.
Chaining buys two things: each step produces a small output a reviewer can actually check, so an error is caught where it appears, and the intermediate artefacts have value of their own in review, risk analysis and traceability. It does not teach the model anything between steps, and it usually costs more tokens rather than fewer.
A tester has to derive test conditions for this requirement of a travel…
A tester has to derive test conditions for this requirement of a travel insurance quote service: a quote is offered when the traveller is between 18 and 75 years old inclusive, the trip starts no earlier than tomorrow and lasts at most 90 days, and the destination is not on the excluded-countries list; travellers aged 66 to 75 are offered a quote with a medical declaration; trips longer than 30 days carry a long-stay loading. The first prompt was "Give me test conditions for this insurance requirement" and it returned eight generic conditions such as "check age validation". Which replacement prompt is the most effective for this task?
Paste the requirement and ask the model to act as an experienced insurance tester and produce a professional and complete set of test conditions covering the whole quote service.
Role framing and the words professional and complete give the model nothing to act on. Completeness is defined by the rules and their boundaries, and the prompt never says which those are or how the answer should be laid out.
Paste the requirement, ask for exactly twenty test conditions, and ask the model to rank them by how likely each one is to reveal a defect in the quote service so that the weakest can be dropped before review.
A fixed count is arbitrary against a rule set of this size and pushes the model to pad or truncate. The likelihood ranking is generated text with nothing behind it, and it invites the team to skip the lower half.
Paste the requirement, ask for test conditions covering each rule and each boundary of the age, duration and start-date rules, ask for one condition per line, and ask each line to name the rule it comes from.
Correct. It names the coverage target, which is what makes the answer checkable; it fixes the output shape; and the rule reference lets the reviewer spot a rule with no conditions and a condition with no rule.
Paste the requirement and ask the model to write the automated test scripts directly, on the grounds that the test conditions can be read back out of the finished scripts afterwards.
Jumping to scripts skips the level where coverage is decided, so gaps become invisible: a rule the model overlooked leaves no trace in the scripts. Reading conditions back out is also slower than deriving them first.
The weakness of the first prompt is that it says nothing about coverage or form. The effective replacement pastes the requirement, names the coverage wanted (each rule plus the boundaries of the age, duration and start-date rules), fixes the output shape, and asks for the rule each condition traces to so the result can be reviewed against the requirement.
An order discount rule reads: orders below 100.00 EUR receive no discount…
An order discount rule reads: orders below 100.00 EUR receive no discount; orders from 100.00 to 499.99 EUR receive 5%; orders of 500.00 EUR and above receive 10%. Amounts are stored to two decimal places. A tester prompts a model for the boundary values to test and receives the set 99, 100, 101, 499, 500, 501. What is the most appropriate next step?
Reply that amounts carry two decimal places, ask for the boundaries restated at that precision, and then check the returned set against the rule yourself, expecting 99.99, 100.00, 499.99 and 500.00.
Correct. The precision of the value range was missing from the prompt, and adding it is the direct fix; verifying the returned set against the rule is what keeps the responsibility for coverage with the tester.
Accept the set: two-value boundary analysis calls for one value on each side of each boundary, and the six integers supplied cover both sides of both boundaries in the discount rule, with a third value inside each partition.
Two-value analysis needs the boundary and its nearest neighbour in the actual value range. With two decimal places the nearest neighbour below 100.00 is 99.99, so 99 leaves the interesting values untested.
Ask the model to regenerate the answer with a lower temperature setting, which makes it stay closer to the most likely continuation and so removes the rounding it applied to the monetary amounts.
Temperature affects how varied the sampling is, not the numeric precision of the content. A low-temperature run is likely to return the same integers with more consistency.
Request three-value boundary analysis instead, which brings the missing decimal values into the set because it covers one additional point on each side of every boundary in the rule.
Three-value analysis adds a further point on one side of the boundary, but it inherits the same assumption about precision. Without stating the two decimal places, it returns 98, 99, 100 and repeats the error.
The model returned integers for a field held to two decimals, so the values just below each boundary are wrong: the value immediately below 100.00 is 99.99, not 99, and the value immediately below 500.00 is 499.99, not 499. The fix is to restate the precision in a follow-up prompt and then check the returned set against the rule rather than accepting it.
A tester needs 500 rows of test data for a loyalty scheme. Each row holds a…
A tester needs 500 rows of test data for a loyalty scheme. Each row holds a member reference, a join date, a tier and a points balance. Tier follows the balance: Bronze below 1000 points, Silver from 1000 to 4999, Gold from 5000 upwards. About one member in ten has to have joined within the last thirty days so that the new-member journey can be exercised. The data will be loaded into a shared test environment that an offshore team also uses. Which prompt fits this task best?
Ask for 500 realistic loyalty members of the kind a real retailer would have, and say that the tier values should follow the usual thresholds used across the industry.
Neither the output shape nor the tier thresholds are stated, so the rows arrive in an arbitrary format with tiers the model has invented, and the ten per cent of recent joiners the test needs is missing.
Paste an extract of the production member table as a worked example, ask for 500 further rows in the same style so the distribution of balances and join dates matches production, and note that the generated rows must not reuse any member reference that appears in the extract.
This sends real member records to an external service and then spreads them through a shared test environment, and avoiding duplicate references does nothing about that. Distribution realism can be described in words instead, which is what the requirement asks for anyway.
Ask for 50 rows with the four fields and the tier rules, then repeat the identical request ten times over to build up the 500 rows that are needed.
Batching is sensible, but repeating an identical prompt produces overlapping member references and no control over the overall share of recent joiners, so the two properties the data set is judged on are left to chance.
Ask for 500 delimited rows with the four fields named and ordered, state the tier thresholds as rules each row must satisfy and the share of join dates inside the last thirty days, and require every value to be invented rather than taken from a real person.
Correct. Format, consistency rules and the recent-joiner share are all stated as checkable constraints, and the instruction about invented values keeps real member data out of a shared environment.
The prompt has to carry the three things that make the data loadable and safe: a fixed output shape with named fields, the consistency rules the rows must satisfy and the required share of recent join dates, and an explicit instruction that the values are invented. Seeding the prompt with a production extract to make the data realistic puts real member data into a shared environment and into a third-party service.
A team uses a model to review user stories before refinement. The prompt is…
A team uses a model to review user stories before refinement. The prompt is "Review this user story and tell me what is wrong with it", followed by: "As a returning customer I want to reuse a saved card so that checkout is faster. Acceptance criteria: the saved card is shown at checkout; the customer can pay with it; expired cards are handled." The model returns fluent but generic observations such as "the story could be more specific" and "consider adding non-functional requirements". Which TWO changes to the prompt would improve the review most?
Ask the model to rewrite the story in the form it considers correct, and treat the rewritten version as the outcome of the review.
A rewrite hides which problems were found and silently invents detail the business has not agreed. The review output the team needs is a list of issues to discuss, not a replacement story.
Ask the model to give each issue a severity from one to five so that refinement can concentrate on the three highest-scoring issues.
The severity figures are generated alongside the issues and carry no assessment behind them. Using them to decide what to discuss lets the model set the agenda for a conversation it is not part of.
Paste the team's definition of ready and ask for the story to be checked against each item on it, with a verdict and a short reason per item.
Correct. A named criterion replaces the model's idea of a good story with the team's, and a per-item verdict makes an omission visible: a missing item stands out where a free-form list would simply not mention it.
Mention that the story was written by a junior business analyst who is new to the domain, so that the review is carried out with appropriate thoroughness and pitched at the right level.
Describing the author changes the tone of the write-up at most, and it can bias the review towards fault-finding. Thoroughness comes from the criterion the review is run against.
Require every issue raised to quote the exact wording from the story that prompted it, so each point can be confirmed against the text.
Correct. A quotation anchors the issue in the story, and an issue that cannot be anchored, such as a complaint about a missing criterion that is in fact present, becomes visible immediately.
The review is generic because the prompt supplies no criterion to review against and no way to check what comes back. Giving the model the team's own definition of ready turns the review into a checklist with a verdict per item, and requiring a quotation for every issue makes each point verifiable against the story instead of plausible-sounding.
A tester has these session notes: "checkout, promo code SUMMER20 entered at the…
A tester has these session notes: "checkout, promo code SUMMER20 entered at the payment step, spinner for about forty seconds then the page reloads with an empty basket, seen twice on staging build 4.12.3 in Chrome 141 and once in Firefox, logged-in user with three items". They prompt a model to "turn this into a defect report" and receive a polished report that also states a severity, names a database timeout as the cause and lists six reproduction steps, none of which the tester recorded. Which change to the prompt deals with this most directly?
Ask for the report in a formal tone appropriate to a regulated project, which discourages the model from adding speculative detail such as a suspected root cause it cannot support.
Tone and invention are independent: a formal register makes an invented cause read more authoritative rather than less. The instruction gives the model nothing to withhold.
State the fields of the team's defect report template and instruct the model to use only the information supplied, marking any field the notes do not cover as unknown.
Correct. The template tells the model what has to be filled, and the instruction to stay inside the supplied facts turns a missing severity or cause into an explicit unknown that the tester then completes deliberately.
Ask for three alternative versions of the report and keep whichever of them contains the fewest details that the notes do not support, discarding the other two.
Comparing three versions still leaves the tester reading each line against the notes, and all three are generated under the same instruction, so all three invent. It multiplies the review work rather than removing the cause.
Supply the notes as a numbered list rather than as free text, so that the model treats each note as a separate observation instead of merging them into a single narrative of the session.
Structuring the input helps the model keep the observations distinct, which is worth doing, but nothing in it stops the model from filling an empty report field with a plausible value.
The model filled the gaps in a report template with plausible content because nothing told it what the template was or that it had to stay inside the supplied facts. Naming the fields and instructing the model to mark as unknown anything the notes do not cover turns the gaps into visible gaps instead of invented content.
Chapter 3 · Managing Risks of GenAI in Software Testing — 10 questions
Which statement defines a hallucination in the output of a generative model?
Content the model declines to produce because a safety filter blocked part of the request, which leaves the answer visibly incomplete.
A blocked request produces a refusal or a gap the reader can see. A hallucination is the opposite case: the answer looks complete and gives the reader nothing to notice.
Content that is factually correct but written in a register that does not suit the audience the answer was prepared for.
This describes a tone mismatch, which is a writing issue. The defining property of a hallucination is the absence of factual support, not the way the content is phrased.
Content presented as fact, in the same fluent and confident register as the rest of the answer, that has no support in the supplied sources or in reality.
Correct. Both halves matter: the content is unsupported, and it is indistinguishable in style from the supported content around it, which is why it survives a quick read.
Content that reproduces a passage from the prompt word for word instead of being restated in the model's own wording, so the answer adds nothing the reader did not already have.
Quoting the prompt back is an unhelpful answer, and sometimes a sign the prompt was ambiguous, but the quoted material is supported by definition since it came from the prompt.
A hallucination is content stated as fact that has no support in the supplied sources or in reality, delivered in the same fluent, confident register as the parts of the answer that are correct. That combination is what makes it costly: nothing in the form of the text marks it out.
What is the NIST AI Risk Management Framework?
A binding regulation that defines the conformity assessment an AI system has to pass before it may be placed on a market.
This describes the shape of AI legislation rather than the framework, which carries no legal force and imposes no market-entry procedure on anyone.
A certification scheme under which an accredited body audits an organisation's AI management system against stated requirements and issues a certificate of compliance for a fixed period.
Auditable management system requirements of that kind are what a management system standard provides. The framework is guidance an organisation applies to itself.
A technical test standard that fixes the metrics and the sample sizes used when measuring the accuracy of a machine learning model.
The framework discusses measurement as one of its functions but prescribes no metric and no sample size. Those belong to test standards and to the evaluation literature.
Voluntary guidance that organises AI risk work into the four functions Govern, Map, Measure and Manage, applied across the life cycle of an AI system.
Correct. The four core functions are the structure of the framework, and its voluntary nature is what distinguishes it from regulation and from a certifiable standard.
The NIST AI Risk Management Framework is voluntary guidance that organises AI risk work into four core functions, Govern, Map, Measure and Manage, applied across the life cycle of an AI system. It is neither a law, nor a certification scheme, nor a measurement standard.
What does ISO/IEC 42001 specify?
A catalogue of prompt engineering techniques with guidance on the situations in which each technique suits a given testing activity.
Technique catalogues of that kind come from syllabi and practitioner literature. A management system standard is about how an organisation governs its work, not about how a tester writes a prompt.
Requirements for an artificial intelligence management system, so that an organisation can establish, operate and improve its governance of AI and be audited against them.
Correct. It is a management system standard in the same family as the information security one, which is why it speaks about policy, roles and continual improvement rather than about model behaviour.
The metrics for assessing the output quality of a generative model, together with the thresholds a model has to reach before it may be used in production for a given class of task.
The standard requires an organisation to define and apply suitable evaluation, but it sets no metric and no pass threshold of its own. Those are decided per use case.
The content and structure of the documentation a supplier hands over with an AI component when an acceptance phase closes.
Handover documentation may well be one output of a compliant management system, but prescribing its table of contents is not what the standard does.
ISO/IEC 42001 sets out requirements for an artificial intelligence management system: the policies, roles, controls and improvement cycle an organisation puts in place to govern its use of AI. Because they are requirements, an organisation can be audited and certified against them.
To get help writing integration tests, a tester pastes an extract of the customer database and the internal architecture document into a public chat assistant. Which risk does this create that no control on the tester's own laptop can address?
The material has left the organisation and now falls under the provider's terms, where it may be retained or used for further training, and the organisation can neither withdraw it nor account for it.
Correct. Disclosure is irreversible and it takes customer data and internal design out of the organisation's control, which is why usage rules address what may be pasted rather than how the answer is used.
The extract may be too large to process in one request, so part of the architecture document is silently left out and the answer is written against an incomplete picture of the system under test.
Truncation produces an incomplete answer, which the tester can notice and work around by splitting the input. It says nothing about what happens to the material that was sent.
The assistant may return integration tests that do not compile against the current code base, costing the tester the time needed to rewrite them by hand and delaying the sprint they were planned for by a day or more.
This is a quality problem with a local remedy: the tester reads the output, finds it does not build and fixes it. Nothing has been lost outside the organisation.
The assistant may answer in a language other than the one the architecture document uses, which makes the generated tests harder for the rest of the team to review and maintain afterwards.
A language mismatch is fixed by asking again in the required language. It is a usability annoyance rather than a consequence of sending confidential material to a third party.
The decisive point is that the material has left the organisation. From that moment it is governed by the provider's terms, it may be retained or used for further training, and the organisation can neither withdraw it nor account for where it went. The other options describe inconvenience, not exposure.
A team wants to reduce the chance that unsupported content reaches its test suite through model-generated test cases. Which TWO measures address this most directly?
Ask the model to state at the end of each answer how confident it is in what it produced, and accept without further checking the answers that it marks as confident.
The confidence statement is generated by the same process as the content it comments on. Using it as a filter concentrates the team's trust exactly where the model is most fluent.
Supply the source material the model has to work from and require every statement in the answer to be traceable back to a specific part of it.
Correct. Grounding gives the model something to draw on instead of its own priors, and the traceability requirement turns an unsupported statement into one the reviewer can spot because it has nothing to point at.
Move to a larger model, since the tendency to produce unsupported content diminishes as the number of parameters grows and the training corpus widens.
Larger models are often more accurate, but the mechanism that produces confident unsupported text is unchanged, and a stronger model can make such content harder to spot rather than rarer in practice.
Run the same prompt twice and keep whichever of the two answers the model itself judges to be the better one when it is shown both and asked to choose.
Two samples from the same model can share the same error, and asking the model to pick between them adds another generated judgement rather than an independent check against the test basis.
Review the generated test cases against the test basis before they enter the suite, treating what comes back from the model as a draft rather than a result.
Correct. Review is the control that catches what grounding misses, and framing the output as a draft keeps responsibility for coverage and correctness with the tester.
Two measures work on the cause: giving the model the source material and requiring each claim to be traceable to it, and reviewing the output against the test basis before it is accepted. Self-reported confidence, model size and self-comparison all rely on the model to judge its own output, which is the thing being questioned.
A team proposes to stop reviewing model-generated test cases whenever the model was given the acceptance criteria, arguing that grounding the model in those criteria makes the review step unnecessary. Which argument best answers this proposal?
The review would indeed be unnecessary if the acceptance criteria were complete, so the effort now spent reading generated cases is better invested in making the criteria themselves complete and unambiguous.
Improving the criteria is worth doing and it does help, but even complete criteria leave the question of whether the generated cases cover them, which is what the review answers.
The review can be dropped once the team has measured the model's accuracy on a sample of a hundred generated cases, found the resulting error rate acceptable for the risk level of the product and recorded that measurement.
A measured rate on past samples does not tell the team which cases in the next batch are the faulty ones, and the rate itself shifts with the feature, the prompt and the model version.
The review has to stay because the generated cases need reformatting into the layout the test management tool expects, and someone has to check the fields have landed in the right columns before they are stored.
Reformatting is a mechanical step that can be automated, and it says nothing about whether the content is right. Defending review on those grounds concedes the substantive point.
Grounding lowers how often the output departs from the criteria but leaves the tester accountable for coverage and correctness, and a comparison of the cases against the criteria is what shows whether one was missed.
Correct. A criterion with no test case generated for it leaves no trace in the output, so the omission is invisible unless someone checks the set against the source.
Grounding reduces how often the output departs from the criteria; it does not transfer accountability for coverage and correctness. Whether a criterion was missed, or a case drifted from what the criterion says, is visible only when someone compares the generated cases with the criteria.
A regulated financial services company is drafting the rules under which its testers may use a commercial LLM service. Which TWO provisions belong in those rules?
A stated list of what may not be sent to the service, covering customer data, credentials and material held under a third-party confidentiality obligation.
Correct. Disclosure cannot be undone, so the rule has to bite before the material is sent, and a concrete list is something a tester can apply in the moment without asking.
A rule permitting the service for test activities but not for development activities, on the grounds that generated code carries more risk than generated test cases.
The confidentiality and accuracy risks follow the material and the use, not the job title of the person. A generated test case that silently misses a regulatory rule is no safer than generated code.
A requirement that generated material entering a deliverable is reviewed and recorded as generated, so its origin is known when the deliverable is later audited.
Correct. In a regulated setting the question asked afterwards is where a given artefact came from and who checked it, and that is answerable only if the origin was recorded at the time.
A rule that every answer is cross-checked against a second commercial service, with the answer accepted when the two services agree.
Two models trained on overlapping material agree on plausible-sounding errors quite readily, so agreement is weak evidence. The check that matters is against the project's own test basis.
A requirement to record how many prompts each tester sends per week, so that the team can report the productivity gained from the tool.
Prompt counts measure activity rather than outcome and invite people to game them. Whatever their value to an adoption report, they are not a risk control.
The two provisions that carry weight are a clear boundary on what may leave the organisation and a requirement that generated material entering a deliverable is reviewed and marked as generated. Splitting usage by activity, cross-checking one service against another and counting prompts per tester address neither confidentiality nor accountability.
A team generates test data for a name and address form by asking a model for a thousand realistic customer records. The names that come back cluster in one language and script, and every address follows a single country's format. What is the testing consequence?
The set cannot be used for performance testing, because values with little variety compress well in storage and in transit and therefore understate how much capacity the real records will occupy.
Compression behaviour is a marginal effect and concerns volume testing at most. It says nothing about the functional coverage the data set provides, which is the issue here.
The set breaches data protection rules, because generated names are likely to coincide with the names of real people living in that country.
A coincidental match with a common name does not make the record personal data about that person, since nothing in it was derived from them. The coverage gap is the real finding.
The set exercises a narrow slice of the input domain, so defects in handling other scripts, name lengths and address formats stay hidden, and the ranges to cover have to be stated in the prompt.
Correct. The model reproduces what was frequent in its training data, and a form of this kind fails on exactly the inputs the set is missing, such as diacritics, single-word names and postcodes of a different shape.
The set has to be regenerated at a higher temperature, which is the setting that governs how much variety the generated values carry.
Temperature adds sampling variation around the same dominant patterns, so a warmer run yields more unusual spellings within the same language and format. Coverage comes from naming the ranges required.
The generated set reflects what dominated the training data, so it exercises a narrow slice of the input domain. Defects in handling other scripts, longer or single-word names, and other address formats are not exposed by this data, and the ranges to be covered have to be stated in the prompt rather than left to the model.
Six months after introducing a model to draft test cases, a team observes that…
Six months after introducing a model to draft test cases, a team observes that reviewers approve generated cases noticeably faster and raise fewer comments on them than on cases drafted by colleagues, although the two sets are of comparable quality when examined closely. What does this describe, and how should the team respond?
Automation bias: output is trusted more readily because it came from a tool, so the review needs a stated criterion and a sample checked in depth rather than an instruction to review carefully.
Correct. The bias is well documented and strengthens with familiarity, and the counter is structural: something concrete to check against and a deliberate deep check on part of the batch.
A real improvement in quality: fewer comments show that the generated cases are better written than the ones colleagues draft, so the review effort spent on them can safely be reduced further.
The premise rules this out, since the two sets hold up equally well under close examination. The difference sits in how the cases are reviewed rather than in how they were written.
Reviewer fatigue caused by the larger volume of cases the tool produces each sprint, addressed by generating fewer cases per sprint so that every case still receives its proper share of attention.
Fatigue would slow reviews of both sets equally. A gap that appears between generated and human-drafted cases points at how the source of the case is perceived, not at the workload.
A conflict of interest, because the reviewers are the people who pushed for the tool and have an interest in the trial being recorded as a success when the adoption is reviewed.
That motive would have to be established rather than assumed, and the effect appears in teams that were sceptical about the tool as well. The ordinary explanation is the bias itself.
Treating output as more trustworthy because it came from a tool is automation bias, and it grows the longer the tool is used. Telling reviewers to be careful does not counter it; giving the review a stated criterion and checking a sample in depth does.
A health-tech company is eight weeks into a pilot in which testers draft test…
A health-tech company is eight weeks into a pilot in which testers draft test cases from user stories using a hosted model through the vendor's public web interface. Three events have been reported. A tester pasted a user story containing a de-identified but re-identifiable patient case. A generated test case naming a competitor's product was copied unchanged into a client-facing test plan. A third tester found that generated cases for a dosage calculation read convincingly while omitting two of the seven documented dosage rules. The pilot lead has to propose controls for the remaining six weeks. Which set responds to all three events?
Narrow the pilot to two experienced testers, brief them on all three events in detail, ask them to log any further event as it happens together with the prompt that produced it, and hold a review at the end of the six weeks so that the roll-out decision rests on a complete picture of what went wrong, how often and in which activity.
Logging observes the problems rather than preventing them, and experience is no protection against a silently omitted rule. The patient data would still reach the vendor and the omission would still be invisible in the output.
Store every prompt and every answer in the pilot log, review the log at the end of each week against the three events already seen, and add a rule allowing generated content for exploratory work while forbidding its use in documented test cases for the rest of the pilot.
A weekly log review comes after the material has already been sent and the test plan already issued, and banning documented use removes the part of the pilot that was meant to be evaluated while leaving the disclosure route open.
Pause the pilot for a week, run a prompt engineering session for the whole team covering the three events, and resume with a shared prompt library written jointly so that prompts are consistent between testers and produce output that can be compared across the team.
Better prompts help with the omitted rules and with little else. A prompt library does not govern what may be pasted into a public interface, and it does not put a review step in front of client-facing documents.
Move usage to an enterprise workspace covered by a no-training agreement with a stated list of what may not be pasted, require every generated artefact to be reviewed against its source before it leaves the team, and require the dosage rules to be supplied in the prompt with a verdict per rule.
Correct. The three controls map one to one onto the three events, and the per-rule verdict is what turns an omission into something a reviewer can see rather than something that leaves no trace.
Each event needs a different control: the disclosure needs a governed workspace and a stated boundary on what may be pasted, the unchecked reuse needs a review before generated material leaves the team, and the silent omission needs the rules supplied in the prompt with a verdict per rule so a missing rule becomes visible. A set that addresses only one of the three leaves the other two live.
Chapter 4 · LLM-Powered Solutions for Software Testing — 5 questions
A test team wants a model to answer questions about its own regression suite and its internal test policy. Which property of a retrieval-augmented setup makes it suited to this?
The team's documents are absorbed into the model's weights while the index is built, so the model keeps answering from them even when the document store is unreachable or taken down for maintenance.
Indexing builds a searchable store beside the model and leaves the weights untouched. If the store is unavailable, retrieval returns nothing and the model falls back on its general knowledge.
The relevant passages from the team's own documents are fetched at request time and put into the prompt, so the answer comes from current internal material and the passage behind it can be shown.
Correct. Freshness and attributability are the two properties that matter for questions about internal policy, and both follow from supplying the material at request time rather than relying on training.
The retrieval step checks each generated sentence against the document store after the answer is composed and removes any sentence that it cannot confirm in an indexed passage.
Retrieval runs before generation and supplies context. Verifying output against sources afterwards is a separate mechanism that has to be built deliberately, and it is not what retrieval does.
The model is limited to answering questions whose wording appears in the indexed documents, which stops it from answering anything the team's own documentation does not already cover.
Nothing in retrieval constrains what the model may answer. When the search returns nothing useful, the model still produces an answer, which is why the setup needs its own handling for that case.
Retrieval fetches the relevant passages from the team's own documents at request time and places them in the prompt. The answer is therefore generated from current internal material, and the passage used can be shown alongside it so the reader can check the answer against the source.
A team is deciding between fine-tuning a model on its material and building retrieval over its document store. Which consideration points towards retrieval?
The team wants the model to adopt a house writing style and a fixed output structure consistently across every answer it produces, whatever the question was and whichever tester happened to ask it.
A stable behaviour applied to every answer is the classic case for fine-tuning, or at minimum for a system prompt. Retrieval supplies facts and leaves the way they are expressed largely untouched.
The team wants to keep each request small and predictable in cost, because retrieval adds the fetched passages to the prompt whereas a fine-tuned model answers without any added context.
The observation is accurate but it argues the other way: it is a cost of retrieval and an argument in favour of fine-tuning. Retrieval is chosen despite the larger prompts, not because of them.
The material changes often and answers have to reflect the current version, since a revised document is available as soon as it is indexed while fine-tuning would need another training run.
Correct. Update cost is the practical dividing line: reindexing a changed document takes minutes, whereas teaching a fine-tuned model about that change means repeating the training and the evaluation.
The team needs the model to work without reaching any internal service at all, so that testers travelling to client sites can keep using it while they are offline.
Retrieval depends on the document store being reachable at request time, so an offline requirement points away from it. The knowledge would have to live in the model itself.
Retrieval suits material that changes, because an updated document is available to the next request as soon as it is indexed. Fine-tuning suits a stable requirement about form or behaviour, since changing what a fine-tuned model knows means running the training again.
Which TWO statements describe an agent-based LLM solution as distinct from a single request to a model?
The model is given tools it may call and chooses during execution which ones to use and in what order, so the sequence of steps is not laid down in advance.
Correct. Deciding the next step at run time is the defining feature, and it is why two runs of the same task can take different routes and touch different systems.
The solution runs without a prompt, deriving its goal from the state of the systems it is connected to rather than from an instruction.
An agent still starts from a stated goal. What it does without instruction is choose the intermediate steps; where the goal comes from is unchanged.
Each step is carried out by a separate model fine-tuned for that step alone, and it is this specialisation, rather than the general model, that gives the sequence its reliability.
Agent solutions commonly use one general model across all the steps. Specialised models can be wired in, but nothing in the pattern requires it and reliability does not come from there.
Because the steps are chosen at run time, the solution needs a stated boundary on what the tools may do and a record of the calls that were made.
Correct. A tool that can write or delete turns an unexpected step into an unexpected change, and without a call log there is no way to reconstruct what happened during a run.
The agent retains the results of earlier runs in its weights, so a repeated task is completed more quickly on each subsequent run.
Weights are unchanged by running the agent. Any memory across runs is stored outside the model, in a database or a file the agent reads back.
What makes a solution agentic is that the model is given tools and decides at run time which to call and in what order, so the sequence of steps is not fixed by the designer. Because the steps are chosen during execution, the solution needs a boundary on what those tools may do and a record of the calls that were actually made.
A team has built an internal assistant that drafts test cases for its product. Why does it need a repeatable evaluation set rather than relying on how satisfied the users say they are?
Satisfaction is collected after the fact and from whoever happens to answer, whereas an evaluation set is measured before the assistant is built, at the point where design changes are still inexpensive.
An evaluation set is run repeatedly over the life of the assistant, not once at the start. Its value lies precisely in being available after every later change.
An evaluation set is required by the model provider's terms of use before an internal assistant may be rolled out beyond a handful of pilot users.
Providers set terms about content and usage, not about the evaluation practices of the customer's internal tooling. The reason for the set is engineering, not contractual.
Satisfaction surveys produce a single overall figure, so an evaluation set is needed to split that figure into separate scores for accuracy and for style.
A survey can perfectly well ask several questions. The gap is not the number of dimensions but the absence of a fixed, repeatable input against which two versions can be compared.
The model version, the prompts and the retrieved content all change over time, and a fixed set is what shows whether a given change improved or degraded the output rather than only how it felt.
Correct. Without a stable baseline a regression caused by a provider's model update or a prompt edit is indistinguishable from ordinary variation in the answers people happen to see.
The model version, the prompts and the retrieved content all change over time, and each change can improve one kind of answer while degrading another. A fixed evaluation set run before and after a change is what separates a real improvement from a shift in impression.
In a retrieval setup built over a team's test documentation, what is the role of embeddings?
They represent passages and the query as vectors whose closeness reflects similarity of meaning, so a passage can be found even when it shares none of the query's wording.
Correct. Matching on meaning rather than on words is what makes retrieval useful over documentation where the reader's vocabulary and the author's rarely coincide.
They compress each document into a shorter summary that fits inside the prompt, so that many more of the team's documents can be supplied to the model within the same context limit.
Summarisation is a separate technique that produces readable text. An embedding is a numeric vector, and nothing readable can be recovered from it to put in a prompt.
They record which passages have been retrieved most often across past queries, so that the passages the team has found useful before are ranked higher in every subsequent search of the document store.
Usage-based ranking is a feedback mechanism some systems add on top. It has nothing to do with how an embedding is computed, which depends only on the text itself.
They convert each document into the internal format the model was trained on, without which the model would be unable to read the retrieved passage at all.
A model reads text directly once it is tokenised, and retrieval passes the retrieved passages through as text. Embeddings serve the search step, not the reading step.
Embeddings turn passages and the incoming query into vectors whose closeness reflects similarity of meaning. That is what lets a search for "how long do we keep attempt data" find a passage headed "retention period for test results", which a keyword search would miss.
Chapter 5 · Deploying and Integrating GenAI in Test Organizations — 7 questions
Which statement describes shadow AI?
Use of an AI tool that runs entirely on the employee's own machine, so that none of the material worked on reaches an external provider or leaves the corporate network.
Where the model runs is a separate question from whether its use is sanctioned. A locally hosted tool adopted without approval is still shadow AI, and an approved cloud service is not.
Use of AI tools by employees outside the organisation's approved list and governance, typically through personal accounts and without the organisation's knowledge.
Correct. The defining element is that the organisation cannot see the usage, so it can neither assess the exposure nor apply its own rules to it.
Use of an AI model whose internal workings the supplier declines to document, leaving the organisation with no insight into how its output is produced.
This describes opacity, which is the subject of explainability and transparency. It concerns a property of the model rather than the governance status of its use.
Use of an AI tool approved for one department that is then extended to a second department under the terms of the same contract.
Extending an approved tool to another team may need a fresh assessment, but the tool and its use remain inside the governance process, which is what shadow AI sits outside of.
Shadow AI is the use of AI tools outside the organisation's approved list and governance, usually through personal accounts and without the organisation knowing. It matters because the exposure is real while the organisation has no visibility of it.
What is the purpose of an organisational policy on the use of generative AI?
To describe the technical architecture of the organisation's AI services, including which models are deployed, how requests are routed between them and where the logs of those requests are kept.
An architecture description serves the people building and running the services. A policy addresses what employees may do, which is a different document with a different audience.
To set out the training curriculum an employee completes before a licence for an AI tool is issued to them.
Training requirements may well be referenced by a policy, but a curriculum is a supporting artefact. It does not say what may be done with the tool once the licence is in hand.
To state which tools may be used, for which work and with which data, and who is accountable for the output, so that individual employees are not deciding these questions case by case.
Correct. Consistency and accountability are what a policy provides, and both are lost when the decision is left to whoever happens to be at the keyboard.
To record the productivity improvements the organisation expects from AI tools, so that the benefits can be reported to management each quarter.
Expected benefits belong to the business case for adoption. A policy that reads as a set of targets gives an employee nothing to apply when deciding whether to paste a document.
The policy answers, once and for the whole organisation, which tools may be used, for which work, with which material, and who is accountable for the output. Without it each employee decides those questions alone, in the moment, with no way to be consistent.
What is the purpose of a pilot phase when a test organisation introduces generative AI?
To use the tool on real work within a limited scope, so that the benefits, the risks and the working practices can be assessed before a wider commitment is made.
Correct. Real work and a limited scope are both essential: without real work the assessment says nothing, and without a limit the organisation has committed before it has learned anything.
To compare the commercial products available on the market against one another under controlled conditions, so that the strongest of them can be selected and licensed for the organisation.
Tool selection usually happens before a pilot and answers a narrower question. A pilot is about whether and how the chosen tool works in this organisation's own setting.
To train the model on the organisation's own material so that it produces answers suited to the organisation before general use begins.
Adapting a model is a technical activity that may or may not form part of the adoption. A pilot can be run with a model that is never trained on anything of the organisation's.
To collect the requirements the organisation will put to the supplier when the contract is negotiated after the roll-out has been approved.
Requirements gathering may be one useful by-product, but framing the pilot this way assumes the roll-out decision has been taken, which is the question the pilot exists to inform.
A pilot puts the tool to work on real tasks within a limited scope, so the organisation can judge the benefits, the risks and the working practices it needs before committing more widely. It is an evaluation in real conditions rather than a procurement exercise or a preparation step.
When a test team adopts generative AI, what does change management address?
The controlled handling of modifications to configuration items, so that every change to a test artefact is recorded, assessed for impact and traceable back to the request that caused it.
This describes change control within configuration management. The wording is close, which is why the two are confused, but it governs artefacts rather than the people adopting a new way of working.
The scheduling of tool upgrades and model version changes, so that the team is not interrupted by an update during a release window.
Scheduling upgrades is release and operations planning. It is worth doing, and a model version change does need managing, but it is not what the term covers.
The approval route by which a proposed amendment to the test strategy is reviewed and signed off by the test manager before it takes effect.
An approval workflow for a document is a governance step inside the test process. It says nothing about how a team is brought through a change in how it works.
The people side of the adoption: roles, expectations, concerns and the agreed way of working, so that the change is understood and carried by the team rather than imposed on it.
Correct. Adoption failures in this area are usually about trust, unclear responsibility for output and unstated expectations, none of which a tool decision addresses.
Change management in an adoption is about the people: the roles that shift, the expectations that have to be set, the concerns that have to be heard and the way of working that has to be agreed. It is a different subject from change control over configuration items, which governs modifications to artefacts.
Three months into adoption, a test manager wants a metric that says how well the use of generative AI is going in the team. Which choice is the most informative?
The number of prompts each tester sends per week, since it is easy to collect from the tool itself, needs no extra reporting from the team and shows how far the tool has spread into the daily work.
Prompt volume rises when the tool works well and when it works badly and has to be asked repeatedly. It also responds to being measured, which makes it worse still as a signal.
The share of generated artefacts that pass review without substantive change, read together with the effort the review took, because it speaks to whether the output is actually usable.
Correct. The pair matters: a high pass rate bought with heavy review effort is not a gain, and the combination is hard to improve without the output genuinely getting better.
The reduction in the number of test cases the team writes by hand, because that measures how much of the drafting work the tool has taken over from the testers themselves.
A case generated and then substantially rewritten counts as taken over by this metric. It also rewards generating more cases, which is not the same as testing better.
The proportion of the team holding an AI-related certification, since capability is the binding constraint during the early months of an adoption of this kind.
Certification says something about what people have studied rather than about what the adoption is producing. It is an input measure and it moves slowly, so it answers a different question.
A useful adoption metric has to say something about the quality of what the tool produces and what it costs to use, not only about how much it is used. The share of generated artefacts that survive review without substantive change, read together with the review effort, does that; prompt counts and hand-written totals measure activity and can be moved without any improvement.
A survey finds that most testers in an organisation already use a consumer AI assistant on their own accounts, although none of them is on the approved list. Which TWO actions reduce this shadow AI most effectively?
Block access to consumer AI sites on the corporate network, which stops the traffic at the point where it leaves the organisation and makes any remaining usage visible in the proxy logs.
A block on the corporate network moves the same work to a phone or a home laptop, where the organisation sees nothing at all. It removes the visibility rather than the exposure.
Add a clause to the employment contract making unapproved AI use a disciplinary matter, so that the consequence is clear to everyone.
A sanction deters people from admitting what they use rather than from using it, which drives the practice further out of sight and makes the next survey less accurate.
Provide an approved tool that is at least as convenient as the unapproved one, because usage settles on whatever is quickest to reach.
Correct. Shadow AI is mostly a convenience problem, and an approved route that costs an extra login and a slower model loses to the browser tab people already have open.
Ask each team to report every quarter which AI tools its members use, so that the organisation builds a picture of actual usage over time.
Quarterly reporting describes the situation months later and depends on people volunteering what they know is unapproved. It informs the problem without changing it.
Make the route for approving a new tool short and visible, so that a tester who asks for one gets an answer in days rather than months.
Correct. A slow or invisible approval process is what teaches people not to ask, and shortening it is what keeps the next new tool inside the governance process.
Shadow AI grows where the sanctioned path is slower or less convenient than the unsanctioned one. Providing an approved tool that is at least as easy to reach, and making the approval route for new tools quick and visible, remove the reason people go around the process. Blocking, contract clauses and quarterly self-reporting either push the usage onto personal devices or measure it after the fact.
A team compares two ways of working. In the first, testers copy text between a…
A team compares two ways of working. In the first, testers copy text between a chat window and the test management tool. In the second, the same model is reached from inside the test management tool, with the user story and the existing test cases already supplied as context. What is the main testing argument for the second?
The integrated route runs a different and more capable model than the chat window does, and it is that difference in model quality, rather than anything about the workflow itself, which produces the better test cases.
The premise states it is the same model. Where a request is sent from does not change what the model can do.
The integrated route costs less per request, because model providers charge a lower rate for traffic that arrives through a partner integration than for traffic from a chat interface.
Pricing depends on the commercial arrangement rather than on the route, and in any case cost per request is not a testing argument about the quality of the work.
The context the model needs is supplied automatically and the result lands where it is reviewed, linked and stored, so the manual copying where material is truncated, unlinked or misfiled disappears.
Correct. Both halves are testing arguments: coverage improves when the model reliably sees the story and the existing cases, and traceability survives when the output is created in place instead of pasted in.
The integrated route makes the review step unnecessary, because the tool records automatically which user story each generated test case was produced from.
Recording the origin supports traceability and says nothing about whether the case is correct or whether the story is fully covered. Review is needed on both routes.
Integration changes two things that matter: the context the model needs arrives automatically rather than depending on what a tester remembered to paste, and the result lands where it is reviewed, linked and stored. The manual copying in between is where material gets truncated, loses its links to the story, or ends up attached to the wrong item.