Testing ML Models: Data, Bias and Quality Characteristics

Testing a machine learning model means checking three things ordinary software testing ignores: the quality of the data the model learned from, whether it has absorbed bias, and how it behaves against AI-specific quality characteristics such as robustness and explainability. Because an ML model derives its behaviour from data rather than hand-written rules, most model defects trace back to the data, not the code.

Why is testing an ML model different?

In traditional software the logic is written by developers, so you test the code against a specification. An ML model has no explicit rule for each output — it infers patterns from examples. There is no single correct answer to assert, behaviour is probabilistic, and the same model can quietly degrade as real-world data shifts. Testing therefore moves from does the code match the spec to does the model perform well enough, fairly enough, and reliably enough on data it has never seen. Terms like robustness and drift used here follow the ISTQB glossary.

Start with data quality

Data quality determines model quality: garbage in, garbage out applies twice, because bad data both produces bad outputs and teaches the model bad patterns. Before judging a model, test the dataset across these dimensions:

• Accuracy — do values reflect reality, with correct labels and no corrupted records?

• Completeness — are there missing values or under-represented groups?

• Consistency — same format and meaning across sources (units, encodings, date formats)?

• Uniqueness — duplicates that let the model memorise records or skew class balance?

• Timeliness — is the data recent enough to represent the current problem?

• Relevance — do the features actually relate to what you are predicting?

Split the data: training, validation and test sets

An ML dataset is split into a training set (the model learns from it), a validation set (used to tune hyperparameters), and a test set (held back to estimate real-world performance). The golden rule: the test set must stay unseen during training. If test or validation information leaks into training — a defect called data leakage — the metrics look excellent in the lab and collapse in production. A typical split is around 60/20/20 or 80/10/10, but the priority is that the test set is representative and isolated.

Detect and test for bias

Bias means the model systematically favours or disadvantages certain groups or outcomes. It usually enters through the data. Watch for:

• Sample (selection) bias — the training data does not represent the real population, for example only daytime photos.

• Label bias — historical human labels encode past prejudice, so the model learns to reproduce it.

• Algorithmic bias — feature or optimisation choices amplify a small imbalance in the data.

• Class imbalance — one class dominates, so a lazy model scores high accuracy while failing the rare class.

Test for bias by evaluating metrics per subgroup, not just overall. A model with 94% overall accuracy can still be far worse for a minority group — the accuracy paradox. Our guide to precision, recall and F1 for testers explains why a single accuracy figure hides these failures.

AI-specific quality characteristics

The ISTQB CT-AI syllabus extends classic quality characteristics with ones that only matter for AI systems. When you plan tests for a model, allocate coverage to:

• Robustness and adaptability — does performance hold up under noisy, adversarial or slightly shifted inputs?

• Autonomy and self-learning — does a model that keeps learning stay within safe limits?

• Explainability, interpretability and transparency — can stakeholders understand why a decision was made?

• Fairness and freedom from bias — are outcomes equitable across groups?

• Safety and side effects — does the model avoid harmful or unintended behaviour while optimising its objective?

Techniques for testing ML models

Because you rarely have an exact expected output, ML testing leans on techniques that check relationships rather than single answers:

• Metamorphic testing — apply a transformation that should not change the result (rotate an image, rephrase a sentence) and verify the output stays consistent.

• Back-to-back testing — compare the new model against a previous version or a reference model on the same inputs.

• A/B testing and canary releases — expose a small share of live traffic to the new model and compare outcomes before full rollout.

• Adversarial testing — craft inputs designed to fool the model to probe its robustness.

• Performance metrics — measure precision, recall, F1 and the confusion matrix on the held-out test set.

Common ML defects to watch for

Two failure modes dominate. Overfitting: the model memorises the training data — say 99% training accuracy — but scores far lower on the test set, say 72%; it learned noise, not the pattern. Drift: the world changes after deployment, so concept drift (the relationship between inputs and outputs shifts) or data drift (the input distribution shifts) quietly erodes accuracy. Overfitting is caught before release with a clean test set; drift is caught after release with continuous monitoring.

Testing ML models is where AI testing becomes concrete, and it is a core part of the ISTQB CT-AI certification. To practise data-quality, bias and metric questions under exam conditions, try the ExamCaliber CT-AI mock exam.