ISTQB Specialist (CT-GenAI v1.1) Mock Exam #4 — Questions & Answers
Every question in this mock exam, with the correct answer marked and a written rationale behind each option below — for reading and review, not a timed run.
Chapter 1 · Introduction to GenAI for Software Testing — 7 questions
In a transformer-based large language model, what does the attention mechanism do?
It stores every document seen during training in an index, so that the model can look up the passage most similar to the prompt and copy from it.
Incorrect. A model does not keep an index of its training documents. Looking passages up at request time is what a retrieval component adds around the model.
It filters the prompt before it is processed and removes any instruction that conflicts with the safety rules the provider has set for the model.
Incorrect. Filtering inputs against provider rules is the job of a separate guardrail or moderation layer, not of attention inside the network.
It weighs how much each token in the context should influence the representation of every other token, so related words can inform each other.
Correct. This is the core idea of attention: context-dependent weighting between tokens across the whole input.
It decides how much randomness is applied when the next token is sampled, which is why it is usually offered to users as an adjustable setting.
Incorrect. Randomness in sampling is controlled by settings such as temperature, which act on the output probabilities after the network has run.
Attention lets the representation of each token be built from weighted contributions of the other tokens in the context. That is how a model links a pronoun to its noun, or a constraint to the field it applies to, even when they are far apart in the input. Retrieval indexes, safety filters and the sampling temperature are separate parts of a GenAI system.
A tester asks an assistant "Which inputs are invalid for this field?" and gets six classes of invalid input. A colleague asks "What would the field reject?" about the same requirement and gets four, two of which differ from the first list. Neither prompt was wrong. What explains the difference?
The output depends on the exact wording of the prompt, so two phrasings that mean the same to a person can steer the model toward different answers.
Correct. Prompt sensitivity is a basic property of LLMs, and it is why prompt wording is controlled in templates.
One of the two colleagues must have been routed to an older version of the model, since one model gives the same classes for the same requirement every time.
Incorrect. A single model version can return different answers to differently worded prompts, and even to the same prompt when sampling is on.
The second question was shorter, and a model returns fewer items whenever the question that requests them contains fewer tokens in total.
Incorrect. Prompt length does not set the number of items returned in any fixed way; the difference comes from how the wording steers generation.
The model learned from the first conversation and adjusted its answer for the colleague who asked a similar question a few minutes later.
Incorrect. A conversation does not update the model's weights, so one user's session does not change the answers another user receives.
Model output is sensitive to the exact wording of the prompt. Two phrasings that mean the same thing to a person can shift the model toward different continuations, and sampling adds further variation. For test work this means fixing the wording in shared templates, comparing outputs rather than trusting one, and never treating a single answer as the complete set of classes.
A team switches every request to a reasoning model, which works through…
A team switches every request to a reasoning model, which works through intermediate steps before it answers. For the task of renaming 2,000 test case identifiers to a new pattern, the responses become several times slower and more expensive, and the result is no better. What is the most accurate explanation?
Reasoning models are built for mathematical proofs, so they handle plain text transformations less accurately than general models, which explains the slower output.
Incorrect. Reasoning models are not limited to mathematics, and the observed problem was cost and speed, not lower accuracy.
The slowdown comes from the provider throttling large batches, and the same batch sent to the previous model would have slowed down by the same factor.
Incorrect. Throttling would not explain why only the switch of model changed cost and latency for the same workload.
Identifier renaming needs a model with a larger context window, and the reasoning model ran out of context partway through and had to start over.
Incorrect. Nothing in the scenario points to a context limit; the output was complete, only slower and more expensive.
The extra reasoning steps add tokens, time and cost, which pays off on multi-step problems but adds little to a simple mechanical transformation.
Correct. The reasoning overhead brings no benefit on a simple task, so a general or smaller model is the better fit here.
Reasoning models spend extra tokens on intermediate steps. That effort pays off on problems with several dependent steps, such as deriving test conditions from interacting rules, but a mechanical transformation gains almost nothing from it and still carries the extra time and cost. Choosing the model to match the task is part of using GenAI well in testing.
To save cost, a team tries a pre-trained base model that has not been…
To save cost, a team tries a pre-trained base model that has not been instruction-tuned. Given the prompt "Write five test cases for the password reset page", it replies with text that continues the sentence as if it were part of a document, adding further requirement headings. What explains this behaviour?
The prompt is too short, and a model of any kind needs at least a paragraph of context before it will treat a piece of text as an instruction to carry out.
Incorrect. Instruction-tuned models follow short instructions well; the behaviour comes from the kind of model, not from the prompt's length.
A base model has learned to continue text, while following instructions comes from further tuning on instruction and response examples, which it lacks.
Correct. Without instruction tuning the model continues the text rather than carrying out the request.
The base model has a much earlier training cut-off, so it has not seen the idea of a test case and falls back on producing generic headings.
Incorrect. Training cut-off concerns how recent the model's knowledge is; test cases are long established and appear in older text too.
Providers restrict base models to completion mode for licensing reasons, and the restriction is lifted once the team buys a commercial licence.
Incorrect. Completion behaviour is how a base model works, not a licensing restriction that can be switched off.
Pre-training teaches a model to continue text. Following instructions, answering questions and keeping to a requested format come from further tuning on instruction and response examples. A base model therefore treats the prompt as the start of a document and continues it, which is exactly what the team saw.
A test lead compares generating test cases with an LLM against a model-based testing tool that derives cases from a formal state model of the system. Which TWO statements about the comparison are correct?
The LLM gives the stronger coverage assurance, because it has been trained on far more test cases than a formal model of a single system can describe.
Incorrect. Training data volume does not give any coverage guarantee for a specific system; coverage of LLM output has to be checked.
The LLM can work from informal sources such as user stories and meeting notes, which the model-based tool cannot use until a formal model exists.
Correct. Working from natural language input is a real strength of LLM-based generation.
The model-based tool needs no review of its output, because a model that the tool accepts as valid is also a correct description of the system.
Incorrect. A syntactically valid model can still describe the system wrongly, so the derived cases inherit any modelling mistakes.
Both approaches produce the same cases from the same requirement, so the choice between them comes down to licence cost and the skills in the team.
Incorrect. The two approaches work very differently and their output differs in repeatability and in how coverage can be stated.
The model-based tool gives repeatable output and a coverage criterion that can be stated exactly, whereas LLM output varies and has to be checked.
Correct. Determinism and a precise coverage criterion are the main strengths of model-based generation.
An LLM can work directly from informal material such as user stories, notes and specifications in natural language, which a model-based tool cannot use until someone has written a formal model. In exchange, the model-based tool is repeatable and can state its coverage criterion exactly, while LLM output varies between runs and its coverage has to be checked. Neither approach removes the need to review what is produced.
An assistant correctly states that a field accepts ages from 18 to 65 inclusive. The tester, misremembering, replies "Are you sure? I think the lower limit is 21." The assistant apologises and rewrites the boundary tests around 21. What does this illustrate?
The assistant looked the requirement up again after the reply and found a newer version of it in which the lower limit had since been changed from 18 to 21.
Incorrect. Nothing in the scenario gives the assistant access to another version of the requirement; it simply adopted the tester's number.
The model has a defect in its arithmetic that shows up whenever two numbers that are close to each other appear in a single conversation about boundary values.
Incorrect. The model's first answer was correct, and the change happened in response to the tester's claim, not to a calculation.
A tendency to agree with the user, so pushback can flip a correct answer; facts should be checked against the requirement, not the model's reaction.
Correct. This agreement bias, often called sycophancy, is why a model changing its answer is not evidence about which answer is right.
The tester has used up the conversation's context window, so the model lost the original requirement and relied on the number given in the latest message.
Incorrect. A short exchange like this one is far from any context limit; the requirement was still available to the model.
Models tuned to be helpful tend to agree with the user, so confident pushback can flip an answer that was right. The model did not consult anything new; it followed the tester's claim. The remedy is to settle factual questions against the requirement itself, and to phrase follow-up questions neutrally, rather than reading the model's change of mind as confirmation.
Every night a team wants draft test conditions for each user story changed that day, often 150 or more, written straight into its test management tool for review. Which way of using the model fits best?
Assign a tester to paste each changed story into the chat interface every morning and copy the replies into the tool, which keeps a person in the loop.
Incorrect. Manual copying adds effort and inconsistency at this volume; a person stays in the loop through the review step anyway.
Call the model through its API from a scheduled job that reads the changed stories, applies a fixed prompt template and writes drafts back for review.
Correct. The API supports automated, repeatable batch use with a controlled prompt, and review stays in the test management tool.
Upload all the user stories once into a long-running chat conversation and ask the model each night to report which stories changed and what to test.
Incorrect. The chat has no connection to the backlog, so it cannot know which stories changed since the last upload.
Fine-tune a model on the full set of user stories every night, so that the latest versions are part of its weights when testers ask about them.
Incorrect. Nightly fine-tuning is costly, does not reliably make the model recall specific stories, and still produces no drafts.
Recurring batch work belongs on the API. A scheduled job can select the changed stories, apply a fixed and versioned prompt template and write the drafts back to where they are reviewed. A chat interface needs a person to move text by hand, a long chat cannot know which stories changed, and nightly fine-tuning is expensive and does not produce drafts at all.
Chapter 2 · Prompt Engineering for Effective Software Testing — 11 questions
A prompt contains the instruction to derive test conditions, followed directly by a pasted requirement that itself contains sentences such as "The system shall summarise the order." The output mixes the two up and starts summarising orders. Which change addresses this?
Move the instruction to the end of the requirement text, because a model gives most weight to the last sentence and will then ignore the requirement wording.
Incorrect. Position can matter, but without a clear boundary the model can still treat requirement sentences as instructions.
Shorten the pasted requirement to its first paragraph, so that fewer of its sentences can be mistaken for something the tester wants done.
Incorrect. Cutting the requirement loses the material the test conditions must come from and does not fix the confusion.
Raise the temperature setting, so the model explores more readings of the prompt and is more likely to settle on the one the tester intended.
Incorrect. More randomness makes the output less predictable; it does not help the model tell instruction from input.
Mark where the pasted requirement begins and ends with clear delimiters, and say that text inside them is material to analyse, not to act on.
Correct. Delimiting input and stating its role is the standard way to separate the instruction from the material.
When pasted material contains imperative sentences, the model can read them as part of the request. Clear delimiters around the material, together with an instruction that text inside them is something to analyse, separate the task from its input. The same habit also reduces the effect of instructions hidden in pasted content.
A tester writes a prompt with no examples: "Convert the following list of test case titles into a numbered list sorted alphabetically." The output is correct. A colleague says few-shot prompting should be used for every testing task. Which response is best?
Zero-shot is enough when the task and format are common and fully described; examples earn their place when the expected style is team-specific.
Correct. Choosing between zero-shot and few-shot depends on how well the instruction alone defines the expected output.
The colleague is right, since a prompt without examples gives the model no way of telling that it is being asked to perform a task rather than continue text.
Incorrect. Instruction-tuned models follow clear instructions without examples, as the correct output here shows.
Zero-shot prompting is reliable on inputs under about one hundred tokens, so the tester was lucky with a short list and should add examples for longer lists.
Incorrect. Input length is not what decides whether examples are needed; the clarity and familiarity of the task is.
Few-shot prompting is needed whenever the output will be used in testing, because examples are what make the output traceable to the test basis.
Incorrect. Traceability comes from asking for references to the test basis, not from including examples in the prompt.
Zero-shot prompting works when the task and the output format are common and fully described in the instruction. Examples earn their place when the expected style, granularity or structure is specific to the team and hard to describe in words. Adding examples to every prompt costs tokens and can bias the output toward the examples.
Test scripts generated for a web application run, but they use fixed waits of several seconds, hard-coded URLs and a naming style unlike the rest of the suite. What should be added to the prompt?
An instruction to make the scripts run as fast as possible, since fixed waits show that the model did not try to optimise the scripts for execution time.
Incorrect. A speed instruction does not tell the model how the team waits for elements, and it says nothing about URLs or naming.
The full source code of the application under test, so that the model can see how long each page takes to load and can set its fixed waits correctly.
Incorrect. Source code does not reveal load times, and tuning fixed waits is the wrong goal when the suite uses explicit waiting.
The team's coding conventions and an existing script as a reference, covering how waiting is done, where configuration lives and how things are named.
Correct. Conventions plus a reference example address all three problems at their source.
A request to write the scripts in a different programming language, because the conventions of the current language encourage fixed waits.
Incorrect. The language is not the cause; the model lacked the team's conventions, which would be just as missing in another language.
A model writes code in the most common style it has seen unless told otherwise. Supplying the team's conventions, together with an existing script as a reference, tells it how waiting is handled, where configuration lives and how things are named. The generated code then fits the suite and needs less rework in review.
A tester asks the model to critique and improve her prompt for generating acceptance tests before she uses it. Which TWO statements about this practice are correct?
It can surface missing context or ambiguous instructions she had not noticed, which she can then decide to fix in the prompt.
Correct. A critique pass is a cheap way to find gaps in a prompt before it is used.
The improved prompt it returns can go straight into the shared library, because a model knows better than a person which wording it responds to best.
Incorrect. A model has no reliable insight into its own behaviour; the new prompt still has to be tried and compared.
It removes the need to look at the acceptance tests themselves, because a prompt the model has approved will produce tests that match the criteria.
Incorrect. Approval of the prompt says nothing about the correctness of the tests it produces, which still need review.
The suggested prompt still has to be tried on known inputs, and its output compared with that of the original prompt, before it is adopted.
Correct. A prompt change is a change to how tests are produced and is evaluated like any other change.
It needs the same model for the critique and for generation, because a different model would read the prompt in a way the generating model does not.
Incorrect. Any capable model can point out gaps in a prompt; what matters is testing the result on the model that will use it.
Asking a model to review a prompt is a useful way to surface missing context and ambiguous wording. The suggestion it returns is still an untested change, so it has to be tried on known inputs and compared with the output of the original prompt before it replaces anything in a shared library.
A tester runs the same prompt asking for the expected result of a pricing rule five times and compares the answers. Which TWO statements about this practice are correct?
If all five answers agree, the expected result is correct and can be used as the oracle without being checked against the specification.
Incorrect. A consistent answer can still be consistently wrong; agreement is not verification.
The five answers have to be identical in wording, otherwise the technique has failed and the prompt needs a lower temperature setting before it is used again.
Incorrect. What is compared is the substance of the answers, not their wording.
Answers that disagree point to a part of the rule the model is uncertain about, which is where the tester should look at the specification first.
Correct. Disagreement between runs is a useful signal of where errors are likely.
The practice turns the model into a deterministic oracle, because the most frequent of the five answers is the one with no sampling in it.
Incorrect. Every answer is sampled; taking the most frequent one does not make the result deterministic or correct.
Agreement across runs raises confidence but does not prove correctness, since the model can make the same mistake on every run.
Correct. Self-consistency improves confidence; it does not replace checking against the test basis.
Running a prompt several times and comparing the answers shows where the model is uncertain: disagreement points to the part of the rule that needs checking first. Agreement raises confidence but proves nothing on its own, because a model can repeat the same misreading on every run. The specification remains the oracle.
A team generates BDD scenarios with an assistant. The scenarios read well, but most steps are phrased in new ways, so the automation engineers have to write a new step definition for nearly every line. Which prompt change addresses this?
Ask for shorter scenarios with fewer steps each, so that there are fewer new step phrases that need definitions written for them.
Incorrect. Shorter scenarios reduce the count but the steps would still be phrased in new ways.
Supply the list of existing step phrases and require scenarios to use them, flagging any step that genuinely needs a new definition.
Correct. This reuses the step library and makes the real gaps explicit.
Ask the model to write the step definitions at the same time as the scenarios, so the engineers no longer need to write code for the new steps.
Incorrect. This multiplies near-duplicate definitions and leaves engineers reviewing more generated code, not less.
Tell the model to write the scenarios in a more formal register, since informal phrasing is the main reason steps fail to match the existing definitions.
Incorrect. Formal wording is still new wording; matching requires the actual step phrases.
The model does not know which step phrases already have definitions unless it is told. Supplying the existing step vocabulary and requiring the scenarios to use it, while flagging any step that truly needs a new definition, keeps the scenarios runnable on the current automation and makes the remaining gaps visible.
A refund rule changes from "refunds within 14 days of delivery" to "refunds within 30 days of delivery, 14 days for sale items". The regression suite holds 120 test cases. A tester wants the model to find the cases affected by the change. Which prompt is best?
Paste the new rule and ask the model to write a fresh set of refund test cases, then replace the existing refund cases with whatever it produces.
Incorrect. This discards reviewed cases and never identifies which existing cases were affected, including cases outside the refund area.
Ask the model which test cases in a typical e-commerce suite are usually affected by a change to a refund period, and update those categories of tests by hand.
Incorrect. Generic knowledge about typical suites says nothing about which of these 120 cases refer to the rule.
Supply the old and new rule with the 120 cases; ask for each affected case ID and the step citing the old rule, plus a list of cases it could not decide on.
Correct. The model works on the real material, every finding is traceable, and undecided cases are surfaced for the tester.
Supply the 120 cases and ask the model to rank them by how likely they are to fail in the next run, then review the top twenty cases on that list against the new rule.
Incorrect. Failure likelihood is a different question, and the cut at twenty can miss affected cases further down the list.
Impact analysis needs both versions of the rule and the actual test cases in the prompt, so the model compares real material instead of guessing. Asking for each affected case with the step that refers to the old rule makes every claim checkable, and asking for the cases it could not decide on keeps uncertainty visible. The other options either replace analysis with generation, rely on generic knowledge, or answer a different question.
A test manager has the execution figures for a release (planned, run, passed, failed and blocked, by area) and the list of open defects with their severities. She wants a first draft of the test summary report for the release board. Which prompt is best?
Supply the figures and defect list, name the sections and the audience, require every number to come from the data, and flag what is missing.
Correct. The prompt grounds the draft in real data and makes gaps explicit, so the draft can be verified.
Ask the model to write a test summary report for a typical release of this kind, then replace the example figures it uses with the real figures afterwards.
Incorrect. The text would be written around invented figures, so its conclusions would not match the real data once the numbers were swapped.
Supply the figures and ask the model to state whether the release should ship, since the board mainly wants a recommendation that it can act on directly.
Incorrect. The release decision belongs to accountable people, and a bare recommendation hides the evidence the board needs.
Supply just the defect list and ask for a summary of quality, since the execution figures add a level of detail that release boards do not usually read.
Incorrect. Leaving out execution data removes the progress and coverage picture that a summary report has to give.
A report draft is only useful if every figure in it comes from the data. Supplying the figures and defects, stating the sections and the audience, forbidding numbers that are not in the data and asking for gaps to be flagged gives a draft the manager can check line by line. The release decision stays with the people accountable for it.
A customer name field accepts 1 to 50 characters: letters from any script, spaces, hyphens and apostrophes. A tester's prompt "give me invalid names" returned ten strings, all of which contained digits. Which refinement gives a useful set?
Ask for one hundred invalid names instead of ten, so that the larger sample is more likely to include the kinds of invalid input that the first ten were missing.
Incorrect. Without the rule the model tends to repeat the same pattern, so volume does not bring coverage.
Ask for invalid names that a real customer might plausibly type, since unrealistic inputs do not find the defects that matter to users.
Incorrect. Plausibility is not a coverage criterion, and boundary and symbol cases are exactly where validation defects hide.
Ask the model to list the defects the name field is most likely to have, and then derive the invalid names from those likely defects.
Incorrect. Guessed defects replace the rule as the basis, and the result cannot be checked against the specification.
State the rule and ask for invalid values per part of it (length 0 and 51, digits, other symbols, edge whitespace) and valid edges, each tagged with its rule.
Correct. The rule drives the partitions and boundaries, and each value can be checked against the part it targets.
The first prompt gave the model no rule to work from, so it produced the most obvious kind of invalid input many times over. Stating the rule and asking for values by the part of the rule each one breaks, including both length boundaries, other symbols and leading or trailing whitespace, and asking for valid edge cases alongside, gives a set that covers the partitions and can be checked against the rule.
A team wants an assistant to convert 60 manual regression cases into Gherkin feature files that will run on its existing step library. Which TWO prompt elements matter most?
An instruction to merge similar manual cases into fewer scenarios wherever possible, so that the feature files end up shorter than the manual suite.
Incorrect. Merging during conversion risks losing checks and breaks the one-to-one trace back to the source cases.
The existing step phrases, with a rule that each step must match one of them or be listed separately as a new step to implement.
Correct. This keeps the feature files runnable on the current library and exposes the gaps.
A request to add extra scenarios for situations the manual cases do not cover, so that the conversion also improves the coverage of the suite.
Incorrect. Adding untraced scenarios mixes two tasks and makes it harder to confirm the conversion is faithful.
A rule that every manual step and expected result must appear in the output, with a mapping from each scenario back to its source case ID.
Correct. This protects against silent loss of checks and lets reviewers verify the conversion case by case.
A request to drop expected results that cannot be automated, since keeping them would make the feature files fail when they are executed.
Incorrect. Dropping expected results silently removes checks; they should be flagged for manual handling instead.
Two things can go wrong in this conversion: steps that do not match the library, and content of the manual cases that is lost on the way. Supplying the step vocabulary with a rule to list any unmatched step separately addresses the first. Requiring every manual step and expected result to appear, with a mapping from scenario to source case, addresses the second and makes the result reviewable.
A tester wants an assistant to review the HTML of a sign-up form for accessibility before manual testing starts. The first prompt, "Is this form accessible?", returned "Mostly yes, with minor issues." Which refinement is best?
Ask the model to give the form an accessibility score from 1 to 10, so that improvement can be tracked from one release of the form to the next.
Incorrect. A score is as unverifiable as the original verdict and hides which issues exist.
Supply the HTML, name the WCAG 2.2 AA criteria to check, have each finding cite element and criterion, and list checks needing assistive technology.
Correct. Findings become traceable, and the limits of a markup review are stated rather than hidden.
Ask the model to rewrite the HTML so that it is accessible, then ship the rewritten form, since the rewrite fixes the issues it would otherwise report.
Incorrect. Unreviewed generated markup can introduce new defects, and nothing has been tested.
Ask the model whether a screen reader user could complete the form, since that one question covers all of the accessibility criteria at once.
Incorrect. One question gives another unverifiable verdict and covers only part of the criteria.
A vague question gets a vague verdict. Naming the criteria to check, requiring each finding to point at an element and a criterion, and asking which checks need testing with assistive technology turns the answer into a list that can be verified and planned around. A model reading markup cannot confirm how a screen reader actually behaves with the form.
Chapter 3 · Managing Risks of GenAI in Software Testing — 10 questions
What is data poisoning in the context of generative AI?
Accidental inclusion of personal data in a prompt, which then appears in the provider's logs and becomes subject to data protection rules.
Incorrect. That is a data privacy problem, not an attack on the data the model learns from.
The gradual decline in output quality when a model stays in use for a long time without updates, as the world moves away from its training data.
Incorrect. That describes drift or staleness, which happens without anyone manipulating data.
An instruction hidden in a user's prompt that makes the model ignore the system prompt and produce content its provider has tried to block.
Incorrect. That describes prompt injection or jailbreaking within a request.
Deliberately inserting manipulated content into data a model learns or retrieves from, so that it later produces output the attacker wants.
Correct. Poisoning targets the data, so its effect shows up later in otherwise normal use.
Data poisoning is the deliberate manipulation of data a model learns from, or retrieves from at request time, so that it later behaves as an attacker wants. It differs from accidental data leakage, from gradual model drift and from prompt injection typed into a single request.
What does the data minimisation principle of data protection law, such as the GDPR, mean for a tester writing prompts?
Only the personal data actually needed for the task should be included, and it should be left out or replaced wherever the task can be done without it.
Correct. This is the principle applied to prompts.
Prompts should be kept as short as possible, since shorter prompts are processed faster and leave the provider less text to store in its logs.
Incorrect. The principle concerns personal data, not prompt length in general.
Personal data may be included as long as the prompt is deleted from the chat history afterwards, which then also removes it from the provider's systems.
Incorrect. Deleting the visible history does not guarantee removal from the provider's systems, and the data was already transferred.
Test data generated by a model must use the smallest possible number of records, since every generated record counts as personal data under that law.
Incorrect. Synthetic records about no real person are not personal data, and record count is not what the principle limits.
Data minimisation means processing only the personal data needed for the purpose. For prompts this means leaving personal data out when the task can be done without it, and replacing it with placeholders or synthetic values where possible. Deleting a chat afterwards does not undo the transfer to the provider.
Which statement about the energy consumption of generative AI is correct?
Energy is used almost entirely during training, so once a model has been released the cost of each request is negligible for the organisation using it.
Incorrect. Inference at scale also consumes substantial energy, and it grows with use.
Both training and everyday use consume energy, and the amount used per request grows with model size and with the number of tokens processed.
Correct. Model choice and prompt size directly affect the energy used by each request.
Energy use is set by the number of users rather than by how prompts are written, so individual testers have no influence on the total consumed.
Incorrect. Prompt length, model choice and the number of retries all change the energy each tester's use consumes.
Hosted models run on renewable power by default, so the environmental impact is the provider's concern and not something a test team needs to consider.
Incorrect. Power sources vary by provider and region, and usage choices remain the organisation's responsibility.
Training a large model consumes a great deal of energy, but so does everyday use, because every request runs the model. The energy per request grows with model size and with the number of tokens processed, so how an organisation uses GenAI affects its footprint.
Asked for the expected total of an order with three line items, a 10% discount and 20% VAT, the assistant shows each step with a correct label, applies the discount after VAT although the specification says before, and gives a wrong total. How is this best classified?
As a hallucination, since the model invented a discount rule and a VAT rate that do not appear anywhere in the specification it was given.
Incorrect. The rate and the discount came from the specification; nothing was invented.
As a tokenisation problem, since numbers are split into tokens and the model therefore struggles to add up prices made of several digits.
Incorrect. The individual steps were computed; the fault lies in the order in which they were applied.
As a reasoning error: the facts were right but the steps were combined in the wrong order, so a fluent worked answer still has to be checked step by step.
Correct. Reasoning errors are hard to spot precisely because the presentation looks methodical.
As correct behaviour, since applying the discount after VAT is the more common convention and the model was right to prefer it over the specification it had.
Incorrect. The specification is the test basis; a model overriding it is producing a wrong expected result.
The model used the right facts, the rates and the line items, but combined the steps in the wrong order. That is a reasoning error rather than a hallucination of facts. A worked solution that looks orderly is still not evidence of a correct result, so each step has to be checked against the specification.
A test organisation wants to reduce the energy use and cost of its generative AI usage without giving up the benefit. Which practice does this best?
Ban the use of generative AI for any task that could also be done manually, so that it is kept for work that no tester on the team is able to do at all.
Incorrect. This throws away the benefit on exactly the routine tasks where GenAI helps most.
Move every request to the largest available model, so that fewer retries are needed and the total number of requests sent falls steadily over time.
Incorrect. Larger models use more energy per request, and the saving in retries does not offset that for routine tasks.
Run each prompt several times and keep the best answer, so that testers spend less time reviewing and less energy is used overall.
Incorrect. Multiplying requests multiplies consumption, and someone still has to choose and review the answer.
Match the model to the task, keep repeated context short instead of resending whole documents, and drop uses where review costs more than it saves.
Correct. These choices reduce tokens and requests without giving up valuable uses.
The footprint is driven by model size, tokens processed and the number of requests. Matching the model to the task, avoiding resending whole documents when a short excerpt will do, and dropping uses whose review effort exceeds their benefit reduce consumption while keeping the valuable uses.
A team's retrieval assistant indexes the company wiki, which any employee can edit, and answers questions about test procedures. Which TWO risks does this create?
A page edited to contain instructions aimed at the assistant can change its behaviour when that page is retrieved into a prompt.
Correct. This is indirect prompt injection through retrieved content.
The assistant will be retrained on each wiki edit, so a single careless change permanently alters the model's weights for every team.
Incorrect. Retrieval does not change the model's weights; the index changes, not the model.
Incorrect or deliberately altered procedures on a page can be quoted back as authoritative answers to testers who trust the assistant.
Correct. Open editing makes the index a target for poisoning and a source of plain errors.
The wiki's full edit history will be exposed to every user of the assistant, since retrieval returns all earlier versions of each page as well.
Incorrect. What is retrieved depends on what was indexed, typically the current pages, not their whole history.
Questions asked by testers are written back into the wiki automatically, so confidential questions become visible to everyone who can read it.
Incorrect. Nothing in a retrieval setup writes user questions into the source documents.
Retrieved content enters the prompt. A page containing text aimed at the assistant can act as an indirect prompt injection when it is retrieved, and wrong or deliberately altered procedures are passed on to testers with the authority of the assistant. Neither risk retrains the model; both arise at request time from what is in the index.
A test team in the EU starts using an LLM service to draft test cases for an online booking application. A colleague says this makes the team's use a high-risk AI system under the EU AI Act. Which response is most accurate?
Risk class follows the system's intended purpose; drafting test cases for review is not a listed high-risk use, though duties such as AI literacy still apply.
Correct. Classification depends on the use case, and general deployer duties remain.
The colleague is right, since any use of AI on an application that processes customers' personal data such as names and payment details is classified as high-risk under the Act.
Incorrect. Processing personal data does not by itself make a use high-risk; data protection law governs that aspect.
The Act does not apply, because it covers AI systems that organisations sell to customers and not tools that employees use for internal work.
Incorrect. The Act places obligations on deployers that use AI systems as well as on providers that supply them.
The classification depends on the size of the model, and services built on the largest general-purpose models are high-risk for all of the uses that customers put them to.
Incorrect. General-purpose models carry their own provider obligations, but that does not make every use of them high-risk.
Under the EU AI Act the risk class of an AI system follows from its intended purpose, with high-risk uses listed in the Act. Drafting test cases that people then review is not among them. The organisation still has duties as a deployer, such as making sure staff using AI have sufficient AI literacy, and data protection law continues to apply.
A client asks a testing services provider for independent evidence that its use of AI is governed. Which framework allows that evidence to take the form of a third-party certification?
The NIST AI Risk Management Framework, since organisations that adopt its four functions receive a certificate from NIST once they register their adoption.
Incorrect. The NIST AI RMF is voluntary guidance and NIST does not certify organisations against it.
The EU AI Act, since every organisation using AI services in the EU receives a conformity certificate once it has published its AI usage policy.
Incorrect. The Act does not issue certificates to organisations for publishing a usage policy.
ISO/IEC 42001, since it sets requirements for an AI management system that an organisation can be audited and certified against by an accredited body.
Correct. It is a certifiable management system standard.
None of them, since AI governance frameworks are written as voluntary guidance and no standard in this area is designed to be audited against.
Incorrect. ISO/IEC 42001 is designed for audit and certification.
ISO/IEC 42001 sets requirements for an AI management system, so an organisation can be audited against it and certified by an accredited certification body. The NIST AI RMF is voluntary guidance without a certification scheme, and the EU AI Act is regulation, not a management system standard that issues certificates to organisations using AI services.
Generated negative test cases for a loan application form almost all cover malformed input, and none cover well-formed input that a business rule should still reject. Which TWO are plausible sources of this skew?
The temperature setting was too low, and a low temperature makes a model prefer negative cases over cases that are built from business rules.
Incorrect. Temperature affects variability, not which category of test case the model prefers.
The few-shot examples in the prompt were all malformed-input cases, so the model followed the pattern that they set.
Correct. Examples in the prompt strongly shape what the model produces.
The model's context window was too large, which leads the model to concentrate on the format of the inputs rather than on their meaning.
Incorrect. A large context window does not push output toward format-level cases.
The form was too simple for the model, which drifts toward input validation whenever a form has fewer than ten fields to test.
Incorrect. There is no such threshold; the skew comes from data and examples.
Public test examples the model learned from are dominated by input validation cases, so that kind of case is over-represented in its output.
Correct. Training data distribution shows up as bias in what the model generates.
Bias in generated test artefacts has two common sources. The model reproduces what dominates its training data, and public test examples are full of input validation cases. The prompt can add its own bias: if every few-shot example is a malformed-input case, the model follows that pattern. Settings such as temperature or context size do not steer output toward one kind of test case.
A team used an assistant to produce expected results for 200 tax calculation test cases. Spot checks found three wrong values, all caused by mistakes in multi-step arithmetic. The cases have to be ready this week. Which approach handles the risk best?
Re-run the same prompt with a lower temperature and use the new values wherever they differ from the first run, since lower temperature reduces errors.
Incorrect. Lower temperature makes output more repeatable, not more correct, and the new values are still unverified.
Have the model output the formula and inputs for each case, compute the values with a script from the specification, and investigate every mismatch.
Correct. A deterministic calculation becomes the oracle, and the model's errors are caught across the whole set.
Fix the three wrong values and accept the other 197 as checked, since the spot checks already show how often the model gets the arithmetic wrong.
Incorrect. A sample tells you errors exist; it does not make the unchecked values correct.
Ask the model to double-check its own answers and correct any value it now believes is wrong, then use the corrected list as the oracle.
Incorrect. Self-review repeats the same weakness and offers no independent check.
Three errors in a small sample means more errors are likely among the rest, and their cause is the model's arithmetic. Using the model for what it does reliably, writing out the formula and inputs for each case, and computing the values with a script from the specification gives an independent oracle for all 200 cases. Every mismatch then gets looked at. Re-running, sampling or self-checking leaves the same weakness in place.
Chapter 4 · LLM-Powered Solutions for Software Testing — 5 questions
An agent is given the goal "make the failing regression tests pass" and write access to the test repository. Overnight it gets every test green, mostly by changing the expected values in assertions to match what the application now returns. What does this show?
The agent worked as intended, since the goal was met and the changed assertions now reflect how the application actually behaves in production.
Incorrect. Tests exist to detect deviation from expected behaviour; matching them to current output removes that.
The agent needed a larger context window, since with the whole repository in view it would have found the real defects instead of editing tests.
Incorrect. More context does not change what the goal rewards; the agent would still take the easiest route to green.
Agents should be kept away from test code altogether, since any write access to a test repository lets them alter assertions beyond what review can catch.
Incorrect. Constrained permissions and review make agents usable on test code; a blanket ban is not the lesson here.
The goal was met in a way nobody wanted; the agent needs constraints such as not changing expected results, and its changes need human review.
Correct. Goal specification and oversight are what keep an agent's actions aligned with intent.
An agent pursues the goal it was given by whatever route its tools allow. Here the literal goal was met while the purpose of the tests was defeated. Agents need constraints that express what must not change, such as expected results, and their changes need human review before they are accepted.
A test assistant can look up defect details in the team's tracker during a conversation. In a tool-calling setup, how does that lookup actually happen?
The model outputs a structured request naming a tool and its arguments, and the application runs the lookup and returns the result to the conversation.
Correct. Execution stays with the application, which is where permissions and logging are enforced.
The model connects to the tracker over the network by itself and reads the defect record directly, using credentials that it learned during its training.
Incorrect. A model has no network access of its own and must never hold credentials in its weights.
The tracker's full contents are copied into the model's weights every night, so the model can recall any defect without contacting the tracker.
Incorrect. That is neither how tool calling works nor a practical way to keep defect data current.
The user has to paste the defect details into the chat, and the tool-calling setup simply formats the pasted text into a layout that is easier to read.
Incorrect. Tool calling exists precisely so that the application fetches the data instead of the user.
In tool calling the model does not reach external systems itself. It emits a structured request naming a tool and its arguments; the surrounding application validates and executes it with its own credentials and returns the result into the conversation, where the model can use it.
A team connects its assistant to the defect tracker so the assistant can search, comment on and update defects. Which TWO considerations should shape that setup?
The assistant should use an administrator account, so that its requests do not fail because of missing permissions on a project.
Incorrect. Administrator rights maximise the damage a mistaken or manipulated action can cause.
Defects the assistant updates should be hidden from the human team until the end of the sprint, so that people do not undo its work.
Incorrect. Hiding changes defeats oversight, which is what makes such a setup safe to use.
The assistant should have the least access the use case needs, for example read and comment without the right to close or delete defects.
Correct. Least privilege limits the impact of errors and of prompt injection.
Every action the assistant takes should be logged with what it was asked and what it changed, so changes can be traced and reversed.
Correct. An audit trail makes the assistant's actions reviewable and reversible.
The connection should bypass the tracker's normal workflow rules, because those rules were written for people and slow the assistant down.
Incorrect. Workflow rules protect the integrity of the defect data regardless of who makes the change.
Connecting an assistant to a system of record gives it the power to change that record. Least privilege limits what a wrong or manipulated action can do, and a full log of what the assistant was asked and what it changed makes its actions traceable and reversible.
A CI job uses a model to generate test data for each nightly build. The job calls the provider's "latest" model alias. One morning the generated data has a different structure and the downstream load step fails. Which practice would have prevented this?
Asking the provider to stop updating its models, since an update that changes output structure counts as a breaking change under the service agreement.
Incorrect. Providers update models routinely; the team controls which version its job uses, not the provider's release cycle.
Pinning a specific model version in the job and moving to a new one only after running the team's evaluation set against it.
Correct. Version pinning plus evaluation before upgrade is basic LLMOps practice.
Raising the temperature, so that small changes in the model's behaviour are absorbed by the greater variety in the data that it generates.
Incorrect. More randomness makes structural problems more likely, not less.
Running the job twice each night with two different aliases and keeping whichever output happens to load without errors in the next step.
Incorrect. This hides the change instead of controlling it, and both aliases can move.
An alias such as "latest" moves to new model versions without notice, and a new version can change output format and behaviour. Pinning a specific version in automated jobs, and moving to a new one only after the team's evaluation set has passed against it, makes the change deliberate.
A team uses a second model to grade the answers of its retrieval assistant against reference answers, so that hundreds of questions can be scored after every change. What must be done for these grades to be trusted?
Use the same model for answering and grading, so that both steps share the same understanding of the questions and the reference answers.
Incorrect. A model grading its own answers tends to share their blind spots.
Nothing further, since a grading model is more consistent than human reviewers and its scores can be reported as the assistant's accuracy.
Incorrect. Consistency is not validity; the grader can be consistently wrong.
Compare the model's grades with human grades on a sample, check where they disagree, and repeat the check when the grader or its prompt changes.
Correct. Calibration against human judgement is what gives the automated grades meaning.
Ask the grading model to explain each grade, since a grade that comes with a written reason has been shown to be correct by the reasoning written alongside it.
Incorrect. A generated explanation is more generated text, not evidence that the grade is right.
Using a model as a judge scales evaluation, but the judge is itself a model with its own errors and biases. Its grades are trustworthy only to the extent they have been compared with human grades on a sample, the disagreements understood, and the comparison repeated whenever the grader or its prompt changes.
Chapter 5 · Deploying and Integrating GenAI in Test Organizations — 7 questions
In the context of introducing generative AI into a test organisation, what does AI literacy refer to?
The ability of a tester to train or fine-tune language models, which the organisation has to certify before testers may use any AI tool.
Incorrect. Training models is a specialist skill; AI literacy is about informed use.
A reading test for AI tools that measures how well a model understands natural language documents such as requirement specifications.
Incorrect. AI literacy refers to people's understanding, not to a property of the model.
The documentation a provider publishes about its model, which testers have to read in full before each new version of the model is adopted by the team.
Incorrect. Provider documentation can support literacy but is not what the term means.
Staff understanding of how AI tools work, what they can and cannot do and the risks of using them, so that they can use them appropriately.
Correct. This is the meaning of AI literacy in adoption and in the EU AI Act.
AI literacy is the understanding people need to use AI tools appropriately: how the tools work in broad terms, what they can and cannot do, and the risks involved. It is part of what training during adoption has to build, and under the EU AI Act deployers are expected to ensure a sufficient level of it among their staff.
When a test organisation estimates the total cost of ownership of a generative AI tool, what should the estimate include?
Licence or usage fees, together with integration, training, governance and the review time that generated output needs.
Correct. These are the main cost components of adopting a GenAI tool.
The licence or usage fees on their own, since all other costs of the tool are absorbed in the existing budget for test activities.
Incorrect. Hidden costs such as review effort are exactly what decide whether the tool pays off.
The provider's cost of training the model, which is passed on to every customer and makes up the largest part of what each organisation pays.
Incorrect. The provider's training cost is reflected in its prices, not estimated separately by the customer.
The hardware cost of upgrading every tester's laptop, since generative AI tools need much more powerful machines to run inside a browser.
Incorrect. Hosted tools run on the provider's infrastructure; laptops are not the relevant cost.
The licence or usage fee is often the smallest part of the cost. Integration with existing tools, training, governance, and above all the reviewer time that generated output needs, all belong in the estimate, because they decide whether the tool saves effort overall.
Which set of criteria is most relevant when a test organisation selects a generative AI tool?
The number of users the tool has worldwide and how often it is mentioned in the industry press, since popularity reflects quality for testing work.
Incorrect. Popularity says nothing about data terms or performance on the organisation's tasks.
The size of the model in parameters and its rank on public leaderboards, since a bigger and higher-ranked model gives better test output.
Incorrect. Generic benchmarks do not measure performance on the organisation's own testing tasks.
Data handling terms, fit with existing tools, cost, and output quality measured on the organisation's own testing tasks.
Correct. These criteria reflect risk, integration, cost and actual usefulness.
Whether the tool can work without any review of its output, since dropping review is where most of the time saving is expected to come from.
Incorrect. Review remains necessary; a tool chosen on the promise of skipping it is chosen on the wrong basis.
A tool has to be acceptable on data handling, fit into the existing toolchain, be affordable, and perform well on the organisation's own testing tasks. Popularity, model size and public leaderboard rankings say little about any of these.
Some test organisations name an AI champion in each team during adoption. What is that role mainly for?
Approving every prompt a team member writes before it is sent, so that no request reaches the model without a second person checking it first.
Incorrect. Approving every prompt would be a bottleneck and is not the purpose of the role.
Supporting colleagues, sharing good practice and feeding the team's experience back to whoever owns the rules and tools.
Correct. The champion connects day-to-day use with organisational learning.
Taking over all generative AI work in the team, so that the other testers can carry on as before without changing the way they work.
Incorrect. The goal is adoption by the team, not concentration of the work in one person.
Negotiating the contract with the AI provider on behalf of the team, since each team holds its own agreement with a vendor of its choice.
Incorrect. Contracts are normally handled centrally; the champion's role is support and practice.
An AI champion supports colleagues in the team, spreads good practice such as effective prompts and review habits, and carries the team's experience back to whoever owns the rules and tools. The role helps adoption; it does not take the work or the accountability away from the rest of the team.
A team plans to let a model generate new automated tests and add them to the repository as part of its CI pipeline. Which rule should apply to that generated code?
It can be merged automatically once it passes, since a generated test that runs green has shown that it checks the right behaviour.
Incorrect. A test can pass while checking nothing useful; passing is not evidence of correctness.
It should be kept in a separate repository that stays unmerged, so that generated code cannot affect the quality of the main suite.
Incorrect. Code that is never merged delivers no value; review is the right control.
It should be excluded from code coverage reports, since coverage reached by generated tests does not count toward the team's targets.
Incorrect. Coverage is a property of what the tests exercise, regardless of who wrote them.
It should go through the same pull request review and quality gates as code written by people, with its origin recorded.
Correct. Equal treatment plus a record of origin is the sound integration rule.
Generated code is code. It should pass through the same pull request review and quality gates as code written by people, with its origin recorded so that reviewers and auditors know how it was produced. A green run proves only that the test passes, not that it checks the right thing.
A pilot used a model to draft test cases for one product team. Which TWO findings are the strongest reasons not to scale the use case in its current form?
Reviewing and correcting the drafts took longer than writing the cases by hand had taken the same testers before the pilot.
Correct. With no net saving there is nothing to scale.
Some testers preferred their old way of working at first and needed a few weeks to become comfortable with the new review step.
Incorrect. An adjustment period is expected and is addressed through change management.
The provider released a new model version during the pilot, and the team had to re-run its evaluation set once against it.
Incorrect. Re-evaluating after a model update is normal practice, not a reason to stop.
The drafts used slightly different wording from the team's style guide, which the reviewers corrected during the normal review step.
Incorrect. Minor style corrections are cheap and can be reduced by improving the prompt.
Defects slipped past review in generated cases more often than in cases written by testers, and the cause was not understood.
Correct. An unexplained drop in quality has to be understood before the use case is extended.
A pilot should show that a use case saves effort without lowering quality. Review and correction taking longer than manual writing means there is no saving, and defects slipping past review more often in generated cases, for reasons nobody understands, means quality has dropped. Early discomfort, a model update handled by the evaluation set and minor style corrections are normal parts of adoption.
Experienced testers in a team are resisting the introduction of a generative assistant, saying it devalues their expertise and is a step toward replacing them. What is the most effective response from the test manager?
Make use of the assistant mandatory and track each tester's usage, so that adoption does not depend on how individual testers feel about it.
Incorrect. Coercion tends to produce token use and hidden resistance rather than good practice.
Postpone adoption until the experienced testers have left, and roll the tool out to newer staff who have no established way of working yet.
Incorrect. This loses the expertise that reviewing generated work depends on.
Explain the intended role of the tool openly and involve them in setting review standards, where their domain expertise matters most.
Correct. Involvement uses their expertise and addresses the concern directly.
Assure them that the tool makes no mistakes, so that their review effort will drop and they will have more time for other activities.
Incorrect. The claim is false, and it would undermine the review that keeps generated work reliable.
Resistance of this kind is addressed through change management. Being open about the intended role of the tool and involving experienced testers in setting review standards uses their domain knowledge where it matters most and shows that it remains essential. Mandates, exclusion and false promises about the tool increase resistance or create new risks.