Validation of AI Systems in regulated Industries: Research needs, regulatory framework and Outlook

When AI itself becomes a compliance topic

When artificial intelligence (AI) can take on regulatory tasks, contributes to GxP and IT compliance, and is increasingly being integrated into medical devices, the following questions inevitably arise: How can AI itself be validated? How can IT compliance of AI applications be established and demonstrated?

Thomas Pauly, Practice Manager, DHC GmbH

Why Traditional Validation Methods Reach Their Limits

Established regulatory methods for software system validation have existed for years, based for example on FDA guidelines, GxP regulations, or IEC 62304 for medical devices. These approaches work well for systems whose behavior is deterministic, reproducible, and traceable.

AI-based systems, however, are data-driven and probabilistic. They change their performance through new training data, potentially even from processes they themselves influence. For such self-learning systems, classical Computerized System Validation (CSV) methods reach their limits.

New Requirements for Validation and Compliance

The validation of AI requires an integrated consideration of

data quality and provenance,
model architecture and training processes, as well as
the application context in which the AI is embedded.

While existing standards such as GAMP 5 Second Edition contain initial approaches to validating machine learning systems, they are not yet sufficient for fully adaptive systems. This creates the risk that regulation becomes a brake on innovation.

This applies both to the use of AI in medical devices (“AI as a medical device”) and AI-based systems supporting business, production, or quality processes.

Missing Standards as Innovation Barrier

In these areas, digitalization is advancing rapidly, yet there is a lack of suitable methods and experience to demonstrably prove IT compliance. Missing regulatory guidance and the resulting uncertainty can become an obstacle to innovation capability and business success, particularly in economically significant, often mid-sized industries such as pharmaceuticals, biotechnology and medical technology. Against this background, there is a need for methodological and technological development in the validation of AI systems.

The overarching framework for further development is set in particular by current European regulations on data processing and artificial intelligence. These include the Data Act (June 2023), the Data Governance Act (DGA) (September 2023) and the EU AI Act, which aims to be the world’s first comprehensive regulation of artificial intelligence. Future approaches to AI validation must fit within this legal framework. In parallel, there are international initiatives developing industry-specific regulations for AI, for example from the EMA or the FDA.

However, these have so far focused primarily on the regulatory requirements for the development of AI-based systems. For their operation, and consequently also for the question of validation and IT compliance, no definitive answers exist to date. Open questions include:

How does AI influence the inspection practices of regulatory authorities?
What adjustments are needed for existing CSV methods?
How can the required compliance of AI systems be demonstrated?

These questions highlight the existing gap: there is no reliable regulatory framework that systematically connects CSV and AI validation. Likewise, standardized methods and technological tools for verifiably testing AI systems under current regulations are missing.

The upcoming Annex 22:
A step toward closing the gap?

The draft Annex 22 “Artificial Intelligence” is intended to close precisely this gap in the future. It complements Annex 11 “Computerised Systems” and addresses the validation of AI models in GMP-critical applications. Its scope covers machine learning models that acquire their functionality through training with data, not through hard-coded rules. Annex 22 stipulates that only static models, i.e. models with frozen parameters, may be used in critical GMP applications, since these parameters for the manufacturing processes (CPPs) and the corresponding attributes in quality control (CQAs) are usually defined in the medicinal product’s marketing authorization dossier and may not be changed. Dynamic or generative models that continue to evolve after their release are therefore explicitly excluded in this context. Furthermore, the draft describes requirements for test data, explainability, confidence levels, change control and human-in-the-loop principles. The goal is to ensure that AI models are only deployed where their performance, stability and traceability are compatible with GMP requirements.

This creates, for the first time, a framework that enables the use of AI in regulated environments without jeopardizing the fundamental principles of patient safety, product quality and data integrity.

Conclusion

The validation of AI systems stands at the intersection of technological innovation and regulatory responsibility. It is currently one of the most dynamic and at the same time least standardized topics in the life sciences sector. Between classical CSV methods and modern, data-driven systems, there is a gap that requires new methods, new ways of thinking, and new tools.

With the EU AI Act, a legal framework is being created for the first time that addresses the use of AI across all sectors. Annex 22 “Artificial Intelligence” concretizes this framework for GMP-regulated environments and creates the basis for traceable, risk-based validation of AI models. This sets regulatory guardrails that enable innovation without jeopardizing the fundamental principles of patient safety, product quality, and data integrity.

For companies, this means: They should engage with the new requirements early, adapt their validation strategies, and build internal competencies for assessing AI systems. Only those who actively shape this transformation can ensure that artificial intelligence does not become a compliance risk but rather an innovation driver in regulated industries.

→ More about AI system validation