AI Validation in GxP Systems: Why Business Requirements Aren't Enough
“The system shall compare documents”
“The system shall analyse KPIs”
“The system shall flag delays”
“The system shall suggest follow-up actions”
They sound like AI requirements.
But they aren’t.
They are business requirements—they describe what the business wants to achieve, not how the AI is expected to behave.
At first glance, this may seem like a minor distinction. However, for regulated GxP systems, it has a significant impact on design, development, risk assessment, and validation.
Validation demonstrates that a system is fit for its intended use. To achieve that, we need requirements that clearly define how an AI-enabled capability is expected to function—not just the outcome it should deliver.
As AI capabilities become increasingly embedded in regulated systems, this distinction becomes even more important.
Business Requirements Define the "What"
Business requirements capture the business objective.
For example:
Improve efficiency.
Reduce manual review effort.
Identify inconsistencies between records.
Support users in making informed decisions.
These requirements explain what the organisation wants to achieve, but they don’t define how the AI is expected to operate.
Why Business Requirements Are Not Enough
Once development begins, the project team needs much more detail.
Questions such as these must be answered:
- What information will the AI analyse?
- What inputs are required?
- What output is expected?
- When should recommendations be generated?
- What logic or criteria will be applied?
- Can users accept, reject, or modify AI recommendations?
- What happens if insufficient information is available?
- How should errors be handled?
Without these details, developers cannot implement the capability consistently, and validation teams cannot objectively verify whether it behaves as intended.
This is where Functional Requirements and AI-specific Requirements become essential.
The AI Validation Requirements Framework
For AI-enabled GxP systems, I recommend thinking about requirements as a structured hierarchy.
Layer | Purpose |
Business Requirement | Defines the business objective. |
Functional Requirement | Defines how the system should perform the required functionality. |
AI Requirement | Defines how the AI capability should behave, including intended use, inputs, outputs, decision logic, human oversight, limitations, and acceptance criteria. |
Risk Assessment | Evaluates risks associated with the AI capability. |
Validation Testing | Verifies that the AI capability is fit for its intended use. |
Objective Evidence | Demonstrates that validation activities have been successfully completed. |
Each layer builds upon the previous one. If the Functional or AI Requirements are missing, the remaining validation activities become difficult to justify.
A Practical Example
Consider the following business requirement:
“The system shall compare documents.”
The business objective appears straightforward.
However, from a design and validation perspective, many important questions remain unanswered.
For example:
- Which documents can be compared?
- What content should the AI compare (text, tables, metadata, attachments)?
- How are differences identified?
- Should the AI detect only textual changes, or also interpret meaning and context?
- How should the differences be presented to the user?
- Should the AI summarise the changes or only highlight them?
- Can users review and verify the identified differences?
- What happens if the AI cannot confidently determine whether content has changed?
- How should large or complex documents be handled?
The business requirement tells us what the system should do.
The Functional and AI Requirements define how the capability is expected to behave.
Those details become the foundation for design, risk assessment, testing, and validation.
Why This Matters for Validation
Without detailed Functional and AI Requirements, several validation activities become significantly more difficult.
Risk Assessment
How can AI-specific risks such as incorrect comparisons, missed differences, hallucinated summaries, inconsistent outputs, or excessive user reliance be assessed if expected behaviour has never been defined?
Validation Testing
Validation requires objective evidence.
Without defined behaviour, it becomes difficult to determine:
- What should be tested?
- What constitutes a successful outcome?
- How should incorrect responses be evaluated?
- What acceptance criteria should be applied?
Traceability
A complete traceability chain should demonstrate the relationship between:
Business Requirement → Functional Requirement → AI Requirement → Risk Assessment → Validation Test → Objective Evidence
If the Functional and AI Requirements are missing, this chain becomes incomplete.
Human Oversight Remains Essential
Current regulatory expectations continue to emphasise appropriate human oversight for AI-enabled GxP systems.
AI may assist users by analysing information or identifying differences, but responsibility for regulated decisions remains with qualified personnel.
Validation should therefore demonstrate not only that AI provides appropriate results, but also that users can review, understand, verify, and appropriately act upon those results.
Practical Recommendations
When documenting AI-enabled capabilities:
- Start with the Business Requirement.
- Develop detailed Functional Requirements describing system behaviour.
- Define AI-specific Requirements, including intended use, inputs, outputs, decision logic, human oversight, limitations, and acceptance criteria.
- Perform an AI-focused risk assessment.
- Design validation tests based on defined acceptance criteria.
- Maintain complete traceability from requirements through validation evidence.
This structured approach supports effective design, development, risk assessment, and validation while aligning with the principles of GAMP® 5 Second Edition, FDA Computer Software Assurance (CSA), and EU GMP Annex 11.
Final Thoughts
AI doesn’t reduce the need for good requirements—it increases it.
As AI becomes embedded in regulated systems, the quality of validation will increasingly depend on the quality of the requirements that define AI behaviour.
Remember this simple principle:
Business Requirements tell us what the business needs.
Functional Requirements tell us how the system works.
AI Requirements tell us how the AI is expected to behave.
That distinction may appear subtle, but it is fundamental to effective AI validation.
