Annex 22: validating an AI model before it touches a GMP decision

Draft Annex 22 is short, and it is deliberately conservative. It was published for consultation on 7 July 2025 as a companion to the revised Annex 11, and it governs AI models used in the manufacture of medicines and active substances. Its starting point is simple: if a model's output affects patient safety, product quality or data integrity, the model has to earn its place with the same rigour as any other validated system, plus some requirements that only make sense for machine learning.

Which models are in, and which are out

The draft draws a firm line. For critical GMP applications it expects models that are static, meaning they do not keep learning once in use, and whose output is deterministic, meaning the same input always gives the same output. Models that adapt during use, models with probabilistic output, and generative AI including large language models fall outside its scope for critical use.

For non-critical uses, the draft suggests its principles can still be applied, provided a suitably qualified person reviews each output. That is the "human in the loop" model, and it is where most companies will start.

The evidence the draft expects

1. Intended use, written down and owned

Describe what the model does, the data it takes in, the full range of inputs it will meet in operation, and its known limitations. Subject matter experts from the process, not only the data scientists, approve it. Everything downstream is tested against this description, so vague wording here becomes untestable claims later.

2. Acceptance criteria before any testing

Choose the metrics that matter for the task, such as sensitivity, specificity, accuracy or F1 score, and set their thresholds before testing starts. The draft expects the model to perform at least as well as the process it replaces. If today's manual visual inspection catches a given proportion of defects, that is the floor.

3. Test data that the model has never seen

Test data must represent the full intended input space, be labelled correctly, and be kept independent of the data used to train and tune the model. In practice that means access controls and an audit trail on the test set itself, so you can prove the developers did not see it.

4. Test execution against the plan

Run the test, record the results against the pre-set criteria, and investigate every failure. A model that passes on average but fails on one product variant has not passed.

5. Explainability and confidence

For critical applications the draft asks that you can show which features drove a result, using techniques such as feature attribution or heat maps, and that the model reports a confidence score. Low-confidence results should be flagged as undecided and routed to a person rather than accepted.

6. Operation: the model stays validated

Change control covers the model, its configuration and the system around it. Performance is monitored in use, and so is the input data: if the materials, equipment or images drift away from what the model was tested on, you need to know before the results do.

How this lands alongside Annex 11 and Part 11

Annex 22 sits on top of Annex 11, not instead of it. The model runs inside a computerised system that still needs access control, audit trails, backup and periodic review. In the United States there is no equivalent GMP text yet; FDA's January 2025 draft guidance on AI credibility is the nearest reference point, and we compare the two in Part 11 and AI.