Precision, Recall and F1 for Testers: What Actually Matters
A tester-focused guide to precision, recall and F1 score: how to read a confusion matrix, when each metric matters, and why 94% accuracy can still be a bad model.
Precision, recall and F1 are the three metrics a tester uses to judge whether a classification model is actually good — not just numerically impressive. Precision measures how many of the model's positive predictions were correct. Recall measures how many of the real positive cases the model actually found. F1 combines the two into a single balanced score. If you test AI systems, these three numbers tell you far more than a single accuracy figure ever will.
This is one of the most heavily examined topics in the ISTQB CT-AI (Certified Tester AI Testing) certification, and one of the most practical skills for anyone testing machine-learning features. This guide explains each metric from a tester's point of view, with a worked example you can reproduce.
What is a confusion matrix?
A confusion matrix is a small table that compares what the model predicted against what was actually true, for a classification problem. Every prediction falls into one of four cells:
True Positive (TP) — predicted positive, and it really was positive.
False Positive (FP) — predicted positive, but it was actually negative (a false alarm, Type I error).
False Negative (FN) — predicted negative, but it was actually positive (a miss, Type II error).
True Negative (TN) — predicted negative, and it really was negative.
Every metric below is just a different ratio of these four numbers. Learn the four cells and the formulas stop being something to memorise.
What is precision?
Precision answers: when the model says "positive", how often is it right? The formula is precision = TP / (TP + FP). Low precision means the model raises a lot of false alarms. Precision matters most when acting on a false positive is expensive or annoying — for example, a spam filter that quarantines real invoices, or a defect classifier that floods developers with false warnings until they stop trusting it.
What is recall?
Recall answers: of all the cases that were really positive, how many did the model catch? The formula is recall = TP / (TP + FN). Low recall means the model misses real cases. Recall matters most when a miss is dangerous or costly — disease screening, fraud detection, or safety-critical defect detection, where letting a real positive slip through is far worse than a false alarm.
Precision vs recall: the trade-off
Precision and recall usually pull against each other. Lower the model's decision threshold so it predicts "positive" more readily and you catch more real cases (recall up) but also raise more false alarms (precision down). Raise the threshold and precision improves while recall drops. There is no single "correct" balance — it is a business decision driven by the cost of a false alarm versus the cost of a miss. As a tester, your job is to make that trade-off explicit and testable, not to chase one number.
What is F1 score?
F1 collapses precision and recall into one number so models are easier to compare. The formula is F1 = 2 x (precision x recall) / (precision + recall). It is the harmonic mean, not the simple average, which means F1 stays low if either metric is low. A model with 95% precision but 10% recall does not get a comfortable 52% — its F1 is about 18%. That property is exactly why F1 is useful on imbalanced data: it refuses to reward a model that is good at one half of the job and terrible at the other.
A worked example
Suppose a defect-prediction model reviews 1,000 code changes. It flags 100 as defective; 80 of those really were defective (TP) and 20 were clean (FP). It passed 900 changes, but 40 of those actually contained defects (FN); the remaining 860 were genuinely clean (TN). Then:
Precision = 80 / (80 + 20) = 0.80 (80%).
Recall = 80 / (80 + 40) = 0.667 (about 67%).
F1 = 2 x (0.80 x 0.667) / (0.80 + 0.667) = 0.727 (about 73%).
Accuracy = (80 + 860) / 1,000 = 0.94 (94%).
The headline 94% accuracy looks excellent. But recall is only 67% — the model quietly misses one defect in three. Whether that is acceptable depends entirely on what a missed defect costs. This gap is the whole reason testers report precision and recall, not accuracy alone.
Why accuracy can lie: the accuracy paradox
On imbalanced datasets accuracy becomes actively misleading. Imagine a fraud detector where only 1% of transactions are fraudulent. A lazy model that predicts "not fraud" every single time scores 99% accuracy — and catches zero fraud. Its recall is 0% and its F1 is 0%. This is the accuracy paradox, and it is why a tester should always ask about class balance before trusting an accuracy figure. When classes are imbalanced, precision, recall and F1 are the metrics that tell the truth.
What testers actually need to do
Testing an ML classifier is not about reproducing these formulas from memory — it is about asking the right questions. Confirm which class counts as "positive" and why. Check whether the dataset is balanced. Establish which error is worse for this product, a false positive or a false negative, and confirm the reported metric reflects that priority. Insist on a baseline so a number has meaning. And watch for a suspiciously high accuracy sitting next to a low recall — that pairing is a classic sign of a model that looks good on a slide and fails in production.
These metrics sit inside the broader machine-learning testing picture covered in our ISTQB CT-AI certification guide, and the CT-AI v2.0 syllabus reorganised where they live — see what changed in the 2026 syllabus. A companion article on testing ML models for data, bias and quality goes deeper on the data side. To practise computing precision and recall from a confusion matrix under exam conditions, try the CT-AI mock exams, where every answer carries a full rationale. Definitions follow the official ISTQB glossary.
Frequently asked
Precision asks how many of the items the model flagged as positive were actually positive (TP / (TP + FP)). Recall asks how many of the real positives the model actually caught (TP / (TP + FN)). Precision punishes false alarms; recall punishes misses.
F1 is the harmonic mean of precision and recall, from 0 to 1. There is no universal threshold — it depends on the domain and the cost of errors. Judge it against a baseline and against the business cost of false positives versus false negatives rather than a fixed number.
On imbalanced data accuracy is misleading. If only 1% of cases are positive, a model that always predicts negative scores 99% accuracy while catching zero real positives. Precision, recall and F1 expose that failure; accuracy hides it.
Prioritise recall when a miss is expensive or dangerous — disease screening, fraud detection, safety-critical defect detection. Prioritise precision when a false alarm is costly or erodes trust, such as spam filtering that must not hide real email.
Yes. The CT-AI syllabus covers the confusion matrix, accuracy, precision, recall and F1 in its machine-learning functional-performance section, and expects you to compute them from a given confusion matrix.
F1 = 2 x (precision x recall) / (precision + recall). Because it is a harmonic mean, F1 stays low if either precision or recall is low, so it rewards models that balance both.
Part of the ExamCaliber editorial team. Every ExamCaliber question and rationale is written and reviewed by hand against the current syllabus — never scraped from exam dumps.