Questions and Answers to Cloud Computing in a GxP Environment - Part 3
The trend in the pharmaceutical industry is also moving towards cloud computing. Financial but also organizational advantages speak for the cloud. At the same time, however, potential dangers and regulatory restrictions should also be taken into account. Nine experts from the pharmaceutical industry and regulatory authorities answer a comprehensive catalog of questions from the following GxP-relevant topics:
- Basics of Cloud Computing Technology
- Regulations and Expectations of Inspectors
- Customer-Supplier-Relationship
- Requirements for Cloud Service Providers (CSP)
- Requirements for Supplier Evaluation and Supplier Audits
- Requirements for Qualifcation / Validation
The experts:
Frank Behnisch, CSL Behring GmbH, Marburg
Klaus Feuerhelm, Local GMP Inspectorate/Regierungspräsidium Tübingen
Oliver Herrmann, Q-FINITY Quality Management, Dillingen
Eberhard Kwiatkowski, PharmAdvantageIT GmbH, Neuschoo
Stefan Münch, Körber Pharma Consulting, Karlsruhe Yves Samson, Kereon AG, Basel
Dr. Wolfgang Schumacher, formerly F. Hoffmann-La Roche AG, Basel
Dr. Arno Terhechte, Local GMP Inspectorate / Bezirksregierung Münster
Sieghard Wagner, Chemgineering Germany GmbH, Stuttgart
14. Regulatory Basics and Inspectors' Expectations
What should the assessment of "cloud suppliers" include from an authority's point of view?
Substantial guidance for evaluating a CSP from an authority perspective is provided by EFG 11's Votum V1100202 "Anforderungen an die Aufbewahrung elektronischer Daten" (Electronic Data Retention Requirements; Document is only available in German).
According to Votum V1100202, the requirements for qualifying the IT infrastructure (IAAS, PAAS), validating the application (SAAS) and ensuring availability, readability and integrity must be fulfilled by an (internal or external) service provider. Crucial is the indication that a risk to patients and/or the quality of the drug product is to be eliminated. According to the Votum, an assessment should include the following points in particular:
- Service Level Agreement (SLA)
- Qualification and validation have been verified or can be verified as part of a continuous assessment of the CSP in terms of quality attributes
- The deletion of the data after termination of the business relationship with the CSP is ensured
- The relocation of the data or the application back or to another CSP is possible
In the case of an applied and appropriate QMS it can be assumed that operative controls defined in the QMS will be carried out and that results not compliant with the specifications will be addressed within the meaning of deviations / OOS.
The RU is nevertheless required to continuously evaluate compliance when implementing the QMS. Chapter 7 of the EU Guidelines to Good Manufacturing Practice specifies: "The Contract Giver is ultimately responsible to ensure processes are in place to assure the control of outsourced activities." Type and extent can be defined on the basis of risk, and they are influenced by the experiences from the continuous monitoring of the service provider.
15. Qualification/Validation Requirements
Special considerations for validation of SaaS; who is accountable?
SaaS means "Software as a Service" and describes one of several service models of Cloud Service Providers (CSP). SaaS means that the CSP provides and manages the complete application including infrastructure and platform. The regulated company pays a subscription fee but does not have to invest in server hardware and software development. Thus, it pays for provision and operation only, whereas the CSP takes care of IT administration and further services like maintenance and updates of the solution.
However, to be frank: Accountability cannot be delegated! The regulated company is still fully responsible for the regular use of the application and its implementation by the CSP. This responsibility can be realized by qualifying the CSP and validating the provided SaaS solution(s).
As for any other software application, the initial step is to write down the requirements, e.g. as a user requirements specification (URS). The URS defines the application's purpose and can be used as a baseline for the evaluation of different CSPs and its applications, often complemented by commercial aspects. CSPs on the short list should be qualified, ranging from filling in a questionnaire to conducting a multi-day on-site audit, depending on the application's risk and the data to be processed.
Recommendation
Vienna, Austria11-14 June 2024
Computerised System Validation
Therefore, transparency of the CSP as well as the customer / supplier relationship and the collaboration method will significantly impact the validation process. Basically, SaaS should be considered a "black box" solution that is going to be validated like any other type of software. However, the following aspects require special attention:
- Documentation provided by the CSP / supplier
This aspect gains weight as it does not affect the application alone, but includes infrastructure, operating system etc. installed, implemented, and operated by the CSP. - Supplier activities (towards validation) and quality of the application
Both can be verified by reviewing the CSPs documentation. - Data security, privacy, and protection GDPR regulations apply. Regulated companies need to understand how and where its data is being stored and processed and how multi-tenant systems segregate and protect their data.
- Update strategy / deployment
Besides the application's quality at the time of (initial) evaluation and assessment, regulated companies need to understand the processes and methods for configuration management, change control, error correction, and deployment, all contributing to maintain high quality in a validated state. Typically, the regulated company is not involved in and therefore has no control over scope and time of updates. - Exit strategy
Finally, another important aspect that should ideally be considered during evaluation is the exit strategy: SaaS solutions are convenient but may lead to reliance ("vendor lock-in"). Keep in mind that the CSP operates the application, but additionally stores and manages your data.
Generally, validation of SaaS follows the same principles as traditional computerized system validation (CSV). However, SaaS introduces new risks and shifts the focus, as the CSP's / supplier's activities take a larger role.
16. Qualification/Validation Requirements
What are the consequences of the different service models (IaaS / PaaS / SaaS / XaaS) for supplier management and the related qualification / validation?
As described in "What's the meaning of IaaS / PaaS / SaaS / XaaS?" (see GMP Journal Issue 35, October/November 2022), the major differences in service models regarding qualification and validation lie in the shift between the supplier's (Cloud Service Provider, CSP) and the user's (regulated company) responsibility. Special cases like HPCaaS or AIaaS (see "What's the meaning of IaaS / PaaS / SaaS / XaaS?") are handled like SaaS to keep it simple, i.e., we are looking at IaaS, PaaS, and SaaS only.
It is important to note that the responsibility of the regulated company regarding patient safety, product quality, and - especially noteworthy in the context of cloud computing - data integrity cannot be delegated to a CSP or any other service provider!
However, cloud models are perfect examples for leveraging: "Build on top of your service provider's outputs and achievements!". Thus, the qualification and validation activities to be performed by the regulated company are geared by the service model (plus the usual suspects like GxP relevance, risk, novelty, complexity etc.):
- IaaS: The CSP provides the infrastructure only. As configuration and operation of all other layers are handled by the regulated company, specific additional measures are not required. To minimize risk, we recommend requesting a minimum availability of the infrastructure and define a plan B to continue operation in the case it is "down" (the latter is recommended for infrastructure operated by the regulated company, too).
- PaaS: The CSP provides the infrastructure and installs the operating system. As operating systems fall into GAMP software category 1 and are - at least for all common operating systems - considered to be of low risk, the recommendations for IaaS apply here as well. As an additional risk control, the operating system's configuration should be specified and verified. Furthermore, the regulated company should understand how and how often the CSP updates the operating system and when and how the regulated company is being notified. The update procedure should be defined in an SLA.
- SaaS: The CSP provides the infrastructure, the operating system, and the application. Here, the CSP needs to specify and verify the application according to the regulated company's requirements and standards. Furthermore, some aspects of configuration control stay with the CSP. This model requires solid trust, often underpinned by a risk-based assessment and evaluation of the CSP, typically supported by an audit. The overall risk as well as the assessment scope can be reduced if the CSP provides relevant documentation like certificates or White Papers. Risks identified as result of the assessment should be mitigated by commensurate control measures, e.g. SLAs. The CSP's update strategy is even important for SaaS (than for PaaS), as changes and updates may have wider impact and carry more risk. Therefore, understanding the CSP's update strategy and being notified about any planned and upcoming changes is even more relevant.
Recommendation
Vienna, Austria11 June 2024
Computerised System Validation: Leveraging Suppliers
No matter whether IaaS, PaaS, or SaaS: The roles and responsibilities of both, CSP and the regulated company, should be clearly defined and any procedures for updates and change control should be contractually agreed.
17. Qualification/Validation Requirements
Can an automated deployment chain replace an IQ? If so, what information must the deployment chain provide?
From Annex 11, there is a requirement to qualify IT infrastructure. The way infrastructure is provided (e.g. as IaaS) also influences the process infrastructure is qualified and how it is demonstrated - keyword "documented evidence" - in order to meet the requirements and specifications.
"Traditional paper-based infrastructure qualification" was based on defined hardware and specified structures that were verified. Virtualization, automation and monitoring tools have evolved infrastructure deployment into a dynamic process, and static qualification is transforming into continuous process control, again automated via monitoring tools.
The components of infrastructure qualification such as requirement / specification, design, configuration, deployment and verification remain as the foundation. The qualification process and the accompanying monitoring merge and, if necessary, also use the same tools (DevOps).
An automated deployment chain linked with permanent monitoring would be suitable as proof of infrastructure qualification. The suitability of the tools must be demonstrated. It must also be guaranteed that errors are detected, reported and corrected.
Although infrastructure is undeniably a critical component in the context of cloud services, the information on its qualification provided by the CSP and required by the RU (Regulated User) is rudimentary and only in a few cases corresponds to a qualification of an on-premise solution. Also, the engagement of a global CSP alone is not a sufficient proof of a qualified IT infrastructure.
18. Qualification/Validation requirements
What is the value of a "validation" performed by a CSP on its own for the services it provides?
For SaaS applications offered by the cloud service provider for GMP, a full validation of the base system is usually intended. In this case, the CSP carries out all the necessary sub-steps of the validation (from URS to UAT) itself and documents them in accordance with the current regulations. The pharmaceutical entrepreneur can refer to these validation documents for the standard system, but must carry out all steps of system adaptation (customizing) on top of the standard itself. The user acceptance test (UAT) must always be performed by the pharmaceutical company itself and cannot be transferred to the CSP. Of course, system test documents created by the CSP that are available can be used for this purpose. Specifically developed MES applications (Manufacturing Execution System), where the "standard" made available by the provider is only a small part of the system, are of course excluded from this procedure.
Some providers already demonstrate a clear overview of who provides which validation documents on their homepage. Nevertheless, it makes sense to clearly define in the SLA (Service Level Agreement) which documents will be provided by the CSP in case of health authority inspections and which costs will be charged by the CSP. It is quite common for CSPs to charge the usual daily rates for the provision of personnel "on duty" during an authority inspection, even though no questions were addressed to the CSP.
19. Requirements for the Cloud Service Provider (CSP)
A non-(GXP-)qualified PAAS could change the versions of some of its generic microservices used by the application to be deployed as a GXP SAAS. Changing the versions of such generic microservices could be beyond the control of the SAAS provider. What would be required to make this scenario GXP-compliant?
Not qualifying a GxP-relevant IT infrastructure platform (= PaaS) contradicts basic GxP compliance requirements: Using such an IT infrastructure platform in the context of an application that requires validation (= SaaS), the IT infrastructure concerned must be qualified1. Thus, the (trivial) answer to the question of how to make the scenario presented GxP-compliant is simply that the concerned platform-as-service must be qualified.
Given that the GxP world is not ideal either, one could speculate about workarounds to counter such non-compliance. E.g. by reducing the degree of risk for a transitional period with interim measures, or generally becoming more resilient to such weaknesses by means of an appropriate process design. The following is certainly an incomplete list of possible countermeasures:
- The example of the massive security vulnerability in the Java library log4j has shown how important it is to have knowledge and control over the software components used. Consequently, a complete overview of the building blocks or libraries contained in the application should be kept as part of the validation. In the case vulnerabilities or security breaches become known such a software bill of materials (SBOM) will allow a targeted measures.
- In general it is a good idea to be aware of the change management of the cloud service provider, regardless of whether or not the services are subject to validation/qualification. Active change management includes information about planned updates or (important) patches that are made available to the customers in a timely manner. For its part, the regulated user must assess such information in a controlled process (e.g. for relevance) and, if necessary, schedule its own measures to safeguard the announced change (e.g., regression tests). A prerequisite, of course, is that the regulated user has the necessary know-how and resources to contribute to change management. One might think about of a complete shift of change management to the side of the regulated operator. This will generally fail due to the lack of knowledge about which software components are used for the platform service.
- If no information about upcoming platform changes is known, either as regular, planned updates or as part of patch management, the only control measures available to the regulated user are active monitoring or close-meshed regression tests:
- Active monitoring of the application allows to detect anomalies, limitations, or performance changes. This requires that monitoring is carried out within a tight time frame and, above all, that suitably qualified employees (IT personnel or key users) evaluate the monitoring data in order to initiate appropriate measures if necessary.
- However, the effectiveness of such a monitoring process stands and falls with the existence of meaningful key figures that allow an objective assessment of the "health status" of the GxP-relevant application.
- Regression tests can provide information about whether the existing (and validated) system functionality is still unchanged. In the scenario assumed here, however, the constraints for such tests are extremely unfavorable, since it is not known when system changes will occur, nor what, if anything, has been changed. This means that effective regression tests would have to be scheduled and performed with high frequency in order to detect impaired or missing system functions in time. Therefore, the cost-benefit ratio for this workaround is very bad.
About the Author
Dr Andreas Mangel organises and conducts courses and conferences for the ECA Academy in the areas sterile production and computer validation.
Noten:
1 EU GMP Annex 11 [Principle]: The application should be validated; IT infrastructure should be qualified.
