Article 10 of the AI Act sets requirements for the data used to train, validate, and test your model. It is not long — about two pages — but it asks for something most technical files do not contain: evidence that your validation data is representative, that you assessed it for bias, and that you documented its statistical properties including gaps and shortcomings.
In plain terms: "96% accuracy on our test set" is no longer sufficient evidence. The regulation wants to know 96% on whom, measured where, and under what conditions.
Why the standard validation workflow falls short
Most AI medical device teams validate their model the way any competent ML team does — hold out a test set, compute sensitivity and specificity, report the result. That workflow produces one number. The regulation asks a different question: does the model work equally well across the conditions it will encounter in clinical use?
A chest CT algorithm might perform at 97% sensitivity overall. But break the results down by scanner manufacturer, acquisition protocol, and patient demographics, and the picture often changes. Performance on scans from one hospital may look very different from performance on scans from another. The overall number hides that variation. The technical file needs to show it.
This is not a theoretical concern. The Team-NB questionnaire on AI in medical devices — 189 questions, published November 2024, already used in audits — asks explicitly for stratified performance across clinically relevant subgroups. The AI Act's own Annex IV requires the technical documentation to include "the metrics used to measure accuracy, robustness, and compliance with other relevant requirements" along with "potentially discriminatory impacts." One aggregate metric does not cover either requirement.
Three things to check in your current technical file
Is your validation performance reported per operating condition, or only as one number?
If your device processes data from multiple scanner types, imaging protocols, or clinical settings, does the technical file show performance separately for each? A Notified Body auditor following the Team-NB questionnaire will ask for this breakdown. If it does not exist yet, producing it from your existing validation data is technically straightforward — but it needs to be done before the audit, not during it.
Do you have a documented description of your dataset's statistical properties?
Article 10(2) requires that training, validation, and test datasets "shall be subject to data governance and management practices appropriate for the intended purpose." In practice this means a documented characterisation — what the dataset contains, where it came from, which populations and conditions it covers, and which ones it does not. "We used 10,000 images from three clinical sites" is a start. The regulation wants to know which sites, what the demographic distribution was, what inclusion and exclusion criteria were applied, and what gaps remain after accounting for all of that.
Have you assessed your data for completeness and potential bias?
Article 10(2)(f) requires examination "in view of possible biases that are likely to affect the health and safety of persons." This is not asking whether your model is "fair" in the abstract. It is asking whether you looked at the data with specific failure modes in mind — underrepresented subgroups, known confounders, site-specific acquisition artefacts — and documented what you found. If you found nothing, document that too. The auditor's concern is not that bias exists, but that nobody checked.
The gap is not technical — it is documentary
None of this requires new technology. The model, the data, and the results already exist inside your validation pipeline. What is missing in most technical files is the structured analysis and the documentation that connects it to the specific regulatory requirements Article 10 imposes.
That gap is closeable. Stratified performance analysis is a known statistical procedure. Dataset characterisation is a defined deliverable. Bias assessment against a specific intended use population is a documented method. The challenge is not building new capability — it is producing the evidence in a form that a Notified Body reviewer can evaluate independently, with a traceable audit trail, before the December 2027 deadline leaves no time to remediate what the analysis finds.