AI in Auditing, without Loss of Control: Requirements for subject-specific Audit Software - And why "theHuman in the Loop" must not remain an empty Promise
Initial situation: Flood of data meets sampling obligation
The digitalisation of pharmaceutical quality management systems has multiplied the volume of auditable data over the past ten years. Where a single file of batch records used to be sufficient, there are now hundreds of SOPs, risk assessments, validation reports, ebatch record extracts and supplier qualifications - usually as PDFs, Word files, scanned images or Excel sheets. Auditors face a structural dilemma: regulatory expectations ("systematic, independent, documented, with sufficient objective evidence") meet a volume of data that can, in practice, no longer be reviewed qualitatively within the agreed audit time.
This is precisely where AI supported audit tools come in. David Abraham's GMP Journal article outlines such a framework - document review, checklist creation, non-conformity detection, report generation - while warning of two dangers: the data protection risk of open cloud LLMs (large language models) such as ChatGPT, Gemini or Copilot, and a "checklist oriented tunnel vision" that confuses superficial conformity with genuine assessment.
Both risks are real. However, they can be mitigated through the right tool architecture, not by foregoing AI altogether. The following contribution formulates a product neutral catalogue of requirements: which properties must subject-specific, auditable software have in order to create a productive AI workflow that measurably increases compliance while placing the professional judgement of the auditors at its centre?
From standards catalogue to individual interview
A robust audit does not begin with the model, but with the right questionnaire. Suitable audit software should therefore be able to build on standard catalogues of relevant standards and guidelines (including EU GMP, ICH Q7/Q9/Q10, ISO 13485, ISO 9001, GDP, user specific supplier questionnaires). From these catalogues, an interview tailored to the specific audit should be assembled. Requirements for this composition:
- Hierarchical structure with sufficient depth (e.g. chapter -> sub chapter -> main question -> follow up questions). Main questions are typically framed as yes/partly/no questions; follow up questions are free text for further detail.
- Criticality defined for each individual question, so that, in advance, the impact of a topic on the product or process of the audited company can be set - an important lever for steering effort and depth, particularly for commissioned audits.
- Modular composition of entire chapter trees or individual questions from one or more regulations - for example, when a supplier audit must combine GMP requirements with ISO 9001 process requirements.
- Resubmission of open CAPAs from previous audits of the same auditee, so that unresolved actions are automatically embedded in the next questionnaire.
Recommendation
Copenhagen, Denmark25 August 2026
Raw Data - Understanding, Defining and Managing
The decisive point: AI should only come into play after the questionnaire has been composed - not before. This structurally excludes the anti pattern rightly criticised by David Abraham: "LLM conjures a checklist out of thin air". The assessment framework remains standard based, auditable and versionable.
The AI layer in interview mode
During audit execution, the software should offer a clearly delineated AI panel directly adjacent to the questionnaire, essentially consisting of two components:
1. A file manager for the proof provided by the auditee (attachments to the audit).
2. AI integration for local or cloud based inference.
Dual deployment: local or sovereign cloud
An auditable solution should support two complementary modes of operation:
| Mode | Endpoint (example) | Use case |
| Local | On prem runtime (e.g. Ollama at http://localhost:11434) with open weight models such as Mistral, Gemma, Qwen or GPT OSS | Maximum data sovereignty; auditor's notebook or on prem server, e.g. for sensitive supplier or authority audits |
| Sovereign cloud | Server side proxy with bearer token to a contractually bound provider | Higher model quality where a robust data protection agreement with the cloud provider exists |
Both modes must be switchable per audit. Uncontrolled default routing to open consumer services (such as the public web interfaces of ChatGPT or Gemini) must be blocked at system level. While this prevents obvious data leakage, a look at the regulatory details of cloud usage reveals deeper hurdles.
The data protection reality
Many software providers argue that the data protection risk is solved once the cloud provider contractually assures that submitted prompts will not be used for training purposes. From a data protection (GDPR) and regulatory (GxP) perspective, however, this is far too narrow. Preventing model training merely protects intellectual property from flowing into public model outputs - the fundamental data protection issue at infrastructure level remains unresolved:
1. The problem of data transmission (Article 28 GDPR)
Even without model training, audit documents (which contain countless personal data such as employee names, signatures and IDs) are transmitted to third party servers, processed there and often stored for up to 30 days as part of abuse monitoring. This absolutely requires a comprehensive data processing agreement and a prior data protection impact assessment (DPIA).
2. The geopolitics of the US CLOUD Act
Where commercial cloud interfaces from US providers are used (even if servers are located in Europe), these companies are subject to the US CLOUD Act. US authorities can, in principle, compel access to sensitive data. A truly "sovereign cloud" in the audit context therefore requires European infrastructure without any reference to US law, along with a strict zero data retention protocol (deletion of all prompts immediately after processing in volatile RAM).
3. The validation dilemma (Annex 11)
In the GMP environment, the rule is: what is not validated does not exist. Commercial cloud models are modified through continuous provider updates in the background (model drift). A reproducible audit result can therefore hardly be guaranteed. For critical GxP processes, local deployment is the only viable route, since only here can the model be "frozen", qualified and validated.
Processing of proofs: OCR and text conversion
Before an LLM can evaluate proofs, they must be converted into a textually reliable form. Suitable audit software provides deterministic preprocessing for this:
- Text files (.txt, .md, .json, .csv, .xml, .html) are loaded directly.
- Image files (.jpg, .png, .webp ...) are base64 encoded and passed to a vision capable model.
- PDF documents are split into images page by page and processed in configurable batches, so as not to exceed the model's context window.
- Text from Word documents (.docx) is extracted.
A page image index is then created – an explicit mapping table that tells the LLM which image number corresponds to which page of which document. This index is the basis for clean referencing (see Section 4) and clearly differentiates a subject-specific solution from simply "dragging a PDF into a chat".
Specialised OCR profiling (e.g. temperature: 0, constant seed, top_p: 1, top_k: 1, an appropriately chosen repeat_penalty and repeat_last_n, strict stop sequences) ensures that text extraction is as deterministic as possible. Converted texts should be stored as .txt files alongside the original, versioned into the audit file system and reused on the next call – no duplicate processing, full traceability.
Token budgeting: no silent truncation
The input (prompt) feeds the AI with data. The processing (inference) uses a model's learned knowledge to analyse the input and generate an output.
Before each inference call, the software should count the tokens of the overall prompt locally (e.g. using a tokenizer corresponding to the model) and compare this with the configured context window (num_ctx). If the prompt exceeds the budget, the call must not be executed; instead, the user must be notified. This avoids the common real world phenomenon where the model has silently "seen" only part of the proofs yet still produces a plausible sounding but incomplete answer.
The two AI workflows in auditing
It is useful to divide AI functionality into two clearly separated steps, each of which can be validated independently. Neither should ever write directly into the interview; instead, each produces a proposal that is only adopted after confirmation.
Workflow A – chapter coverage (coverage analysis)
Question to the AI: "Which chapters of my interview are substantively covered by the available documents – and by what percentage?"
The model receives the reduced interview JSON and all prepared proofs, and returns the interview with two additional fields at each chapter level:
- Coverage: percentage of substantive coverage, also for chapters without their own questions (aggregated from sub chapters).
- Documents: list of specific proofs that clearly answer questions in this chapter.
The output answers a central question for preparation: "Where is it worthwhile to go deeper in the audit conversation – and where can I keep it briefer because the documentation is clear?" The auditor no longer allocates scarce audit time based on gut feeling, but according to a data driven heat map.
Workflow B – pre-answering questions
Question to the AI: "Which main questions can be clearly answered with yes, partly or no based on the documents – and which follow up questions are substantiated by specific text passages?"
The model fills:
- The options field of the main question with a numerical code, e.g. 2 = yes, 1 = partly, 0 = no, -3 = deferred (= insufficient data).
- The answers to follow up questions – with a mandatory, structured reference:
Source: [file name], page [N], quotation: '[short text excerpt from the document]'
The prompt should include at least three explicit countermeasures against hallucinations:
1. Page image index as a table directly before the audit questions (interview) – the model must look up the correct page.
2. Prohibition of generic statements ("Do not simply write 'page 1' for everything").
3. Obligation to quote verbatim from the document ("Quote the REAL text from the image, not invented examples").
Main questions with the answer "no" require a substantive justification; questions without suitable proofs must be set to "deferred" – with a machine readable indication of what is missing. This is precisely what David Abraham means by "reasonable limits between automated support and professional judgement": the AI must not guess; it must explicitly admit when it cannot decide.
Recommendation
Copenhagen, Denmark26-28 August 2026
Data Integrity Master Class
Auditor in the loop: confirmation instead of automation
After each workflow, the result should be displayed as a preview alongside the original interview. Only after a deliberate confirmation is the proposal adopted - and even then not by overwriting, but via a deep merge with the original catalogue. All existing annotations, statistical aggregates, tags and the source catalogue history remain intact. The AI supplements; it does not replace.
This is the technical implementation of the principle that David Abraham emphasises in Section 4 of his article: the use of AI in audits should always be balanced by the professional judgement of the auditors. In practice, this means that the AI's answer is never the audit result, but its preparation - verified, countersigned and corrected where necessary.
Audit context as part of the prompt
A frequently underestimated lever for answer quality is the contextualisation of the audit itself. Structured audit metadata should be provided to the model with every call, for example:
**Audit context:**
- Audit: Supplier qualification, primary packaging materials 2026 Q2
- Auditee: Example GmbH, Plant South
- Location: 70xxx Stuttgart, DE
- Programme: GMP / ICH Q7
This context measurably influences how the proofs are assessed - for example, a cleanroom monitoring report will be evaluated differently in the context of "primary packaging materials" than in the context of "API manufacture, Grade A". This enforces a risk based assessment standard instead of letting the model respond "in a vacuum".
From findings to report: automated report generation
After completion of the site audit, the finished interview - together with the report parameters (audit atmosphere, audit location, audit objective, classification of the audit result, justification in the event of reclassification, attachments) - should be passed to a server side reporting function. The result is a fully formatted AuditReport.docx (optionally also .pdf), including:
- Cover page and audit metadata in the chosen language,
- Chapter tree with rating for each main question,
- Follow up questions with answer details and machine verified references from the pre answering workflow,
- Automatically generated CAPA list,
- Atmosphere rating and justification for any reclassification of the overall result.
The same mechanism can be used to generate an audit agenda in advance. Reports can therefore be delivered, on the day of the audit, in consistently evidence backed quality - shortening the "reporting and follow up" phase outlined in the GMP Journal article. The actual follow up naturally remains dependent on the scope of findings and their remediation.
How a subject-specific solution addresses the risks named in the GMP Journal
| Risk | Architectural response |
| Data protection breaches via public LLMs | Local inference (on premises) as the safest standard. For cloud use: exclusive reliance on European cloud proxies with a strict zero data retention protocol (no caching, no logging). |
| Disclosure of sensitive information | Proofs leave the audit workspace only where the chosen mode permits this; conversion results are persisted in the audit file system, not with the model provider. |
| Distorted or incorrect results | Strict referencing requirement in the prompt (file + page + verbatim quotation); page image index to prevent generic page references; hallucination prevention via stop sequences and low temperature in OCR paths; local token counting to prevent silent truncation. |
| IP issues | Standards catalogues are licensed and versioned within the application; the model does not receive the entire catalogue as "training fodder", but only a reduced JSON view of the current interview. |
| Security gaps | Web applications should be delivered as PWAs with service worker, CSP capable and without API keys in the client bundle (cloud keys held via local proxy or locally encrypted). |
| Checklist tunnel vision | Two phase workflow: first surface the coverage map (to know where coverage is thin), then apply pre answering. AI answers must never be adopted automatically; proposals must be confirmed manually; original interview fields are preserved via deep merge. |
| Blurring of the boundary between AI <--> professional judgement | Clear answer option codes (including "deferred") force the model to declare knowledge gaps explicitly instead of smoothing them over plausibly. |
| Insufficient validity (Annex 11) | Exclusion of uncontrolled, drifting consumer APIs. Preference for local, "frozen" open weight models that allow reproducible software qualification. Cloud hosting is only permissible in the GMP context where it involves a dedicated private cloud instance (e.g. Azure OpenAI Enterprise) that guarantees a zero data retention protocol and in which the model is permanently frozen under a specific version identifier (pinned model version), in order to meet the software validation requirements of Annex 11. |
What AI in auditing does not replace
It would be negligent to sell AI as an "audit autopilot". Three limitations remain - and should intentionally not be removed by auditable software:
1. Observation in the audit.
AI does not see dirty airlocks, unlabelled reagent containers or unsafe operators. The presence of the auditor on site remains indispensable for critical products or processes - software merely frees up time for more observation.
2. Interview dynamics.
How an answer is given - hesitation, evasiveness, seeking reassurance from a superior - can be an indicator of weaknesses. This is captured on site and manually entered into the follow up question; pre-answers from AI serve only as a structural framework.
3. Risk based judgement.
Whether 70 per cent coverage in a chapter is "sufficient" depends on the overall risk of the audit. This decision is made by the auditors - not by the model.
Well designed software actively supports these limitations: fields that are protected or that are none of the AI's business (statistics, tags, aggregates) should be excluded from the deep merge and not overwritten by AI answers.
Conclusion: AI as a sampling amplifier, not an audit replacement
The central advance that a subject-specific AI integration brings to pharmaceutical auditing can be summarised in three sentences:
1. The auditor can review more proofs, more thoroughly, in the same amount of time – the quality of sampling improves structurally. This also increases the explanatory power regarding the audited company's compliance.
2. Every AI statement is traceable back to file, page and quotation – hallucinations are not excluded, but are immediately made visible.
3. Professional judgement remains with the auditors, as adopting each proposal is a conscious act, not a default.
A solution designed in this way therefore meets both: the regulatory expectation of an objective, evidence based audit process, and the requirement for professional judgement enshrined in ISO 19011 and ICH Q9. AI in auditing is not a replacement for auditors, but a structural amplifier – provided the tool is purpose built for this role and not cobbled together from a generic chat frontend.
Anyone who wishes to use AI seriously in GMP audits should assess their tool against exactly these criteria: data sovereignty, referencing obligation, controlled adoption of results, standards based catalogues, documented audit trail. The discussion as to whether AI belongs in audits is therefore over – it already does. The only open question is how it is allowed in.
Sources / Further reading
" David Abraham: Use of Artificial Intelligence (AI) in Audits, GMP Journal Issue 46, Apr/May 2026.
" ICH Q9(R1) - Quality Risk Management.
" ISO 19011:2018 - Guidelines for auditing management systems.
" EU GMP Guide, Annex 11 - Computerised Systems.
About the Authors
Dr Klaus Bonadt is a graduate mathematician and epidemiologist, CTO of iduneo GmbH, and co-shareholder of ilytics GmbH, where he is responsible for backend architecture and AI-powered solutions for GMP audits.
Peter Riebel holds a degree in civil engineering informatics and is a co-shareholder of ilytics GmbH, where he leads marketing, front-end development, and the integration of AI into the Audit Analyzer for GMP audits
* Referring to the GMP Journal article "Use of Artificial Intelligence (AI) in Audits" (David Abraham, QRS-Associates / ECA GMP-Auditor Association) – with a focus on concrete architectural and process requirements for AI supported audit tools.
