Governance notes from Charité Berlin’s diagnostic agents rollout.

High-risk classification: what Article 22 actually requires

The EU AI Act classifies AI systems intended to assist in medical diagnosis as high-risk under Annex III, Point 5(a). For Charité, this classification is not abstract. It triggers a mandatory conformity assessment, a requirement to maintain a quality management system meeting ISO 13485 standards, and an obligation to register the system in the EU database for high-risk AI before any deployment that influences clinical decisions. Charité's legal directorate determined in February 2022 — before the Act passed its final Council vote — that the institution should structure its deployment architecture as if high-risk classification were certain. Dr. Franziska Brandt-Keller, Charité's head of digital health legal affairs, ran the classification analysis and documented the institution's position in a 47-page internal opinion that circulated to the medical board in March 2022. The opinion concluded that any agent producing a differential diagnosis recommendation, a severity stratification, or an escalation trigger met the definition of a high-risk system regardless of whether a clinician reviewed the output before action was taken.

That determination had immediate architectural consequences. Under the AI Act's high-risk requirements, Charité cannot use a foundation model deployed via a third-party API where patient data leaves the institution's network perimeter and enters a shared inference environment. The DSGVO reinforces this constraint independently: processing pseudonymised patient data on infrastructure operated by a non-EEA entity without an adequacy decision or binding corporate rules constitutes a prohibited transfer. The institution therefore contracted with MedKI Systems GmbH, a Munich-based clinical AI company, for a self-hosted model stack running on Charité's own data centre infrastructure in Berlin-Mitte. MedKI's architecture, designated ClinBERT-DE, is a German-language fine-tune of a base biomedical language model retrained on a corpus of de-identified clinical notes provided under a data-sharing agreement with the German Cancer Research Center in Heidelberg. No patient data leaves Charité's network at inference time.

The conformity assessment documentation runs to 312 pages. The quality management system it describes maps every agent capability to a risk class under ISO 14971, the international standard for medical device risk management. Capabilities that produce outputs directly visible to patients — including a symptom-triage agent accessible through the institution's patient portal — required a separate conformity pathway that added six months to the deployment timeline. Three capabilities remain in extended conformity review as of the date of this briefing and have not been cleared for patient-facing deployment.

The Bundesärztekammer position: autonomy, accountability, and the ärztliche Sorgfaltspflicht

The German Medical Assembly — Bundesärztekammer — published a position paper on clinical AI in September 2023 that set the governance terms for every hospital system operating under German professional liability law. The paper's central argument: a physician cannot delegate the ärztliche Sorgfaltspflicht — the medical duty of care — to an automated system, regardless of the system's technical performance. What follows from this is not a prohibition on clinical AI. It is a constraint on accountability architecture. Every diagnostic agent output must be attributable to a named physician who reviewed it and accepted professional responsibility for the clinical decision it influenced. Charité's KKSO framework was drafted with this requirement as its structural spine.

Prof. Dr. Heinrich Mauer-Schlecht, Charité's Chief Medical Officer and chair of the institution's digital health steering committee, convened a working group in October 2022 that included two external members nominated by the Bundesärztekammer's AI task force. That working group produced the accountability mapping that sits at the KKSO framework's core: a matrix defining which agent capability classes require acknowledgement by a Facharzt, which require co-signature by an Oberarzt, and which are restricted to Chefarzt review. The matrix is not symmetrical — capability class maps to clinical risk, not to organisational hierarchy for its own sake. A radiology prefill agent operating in a low-acuity outpatient context requires Facharzt acknowledgement. A sepsis risk stratification agent operating in the emergency department requires Oberarzt co-signature within 20 minutes of output generation.

The Bundesärztekammer dialogue also produced a constraint that has no US analogue. The German professional code prohibits a physician from making a clinical decision they cannot personally justify on the basis of their own clinical reasoning. An agent that produces a recommendation the reviewing physician cannot independently evaluate — because the reasoning is too opaque or the clinical domain is outside the physician's specialty — cannot be deployed in a context where that physician is the designated reviewer. Charité operationalised this constraint as a specialty-matching requirement: each agent capability has a defined reviewer specialty, and the accountability log must confirm that the named reviewer holds the corresponding Facharzt qualification. This requirement added 14 weeks to the deployment planning cycle and required coordinated scheduling changes across six clinical departments.

DSGVO-compliant model architecture: what data sovereignty costs in practice

The DSGVO constraint on cross-border data transfers is the architectural forcing function that separates German clinical AI deployments from their US counterparts at the infrastructure level. Charité's deployment uses no third-party API calls that transmit patient-identifiable or patient-pseudonymised data outside the institution's network. The self-hosted MedKI ClinBERT-DE stack runs on a dedicated compute cluster — eight Nvidia H100 SXM5 nodes — procured under a framework agreement with the Berlin State government's IT procurement body, ITDZ Berlin. The hardware is physically located in Charité's Rechenzentrum on the Campus Charité Mitte, a data centre that holds BSI IT-Grundschutz certification at the Absicherungsniveau Standard. The BSI — Bundesamt für Sicherheit in der Informationstechnik — certified the installation in October 2023 following a 14-week audit that assessed physical security, network segregation, access control, and cryptographic key management.

Model training and fine-tuning runs on the same infrastructure under a separate access-control tier. MedKI's team holds no direct access to identifiable patient data; the fine-tuning corpus was de-identified and transferred to Charité's custody before training began. The data-processing agreement between Charité and MedKI, reviewed by Berlin's data protection supervisory authority — the Berliner Beauftragte für Datenschutz und Informationsfreiheit — specifies that MedKI's engineers can access model weights and training logs but not the underlying data. This arrangement satisfies the DSGVO's processor-controller distinction but imposes a coordination overhead: any model update requires a formal change-control procedure that involves both Charité's data protection officer and MedKI's compliance team, with a minimum 15-day review window before any updated model reaches the production inference cluster.

The cost of this architecture is measurable. Charité's digital health directorate estimates that the self-hosted infrastructure and governance overhead added €4.1M to the first-year deployment budget compared with a hypothetical API-based deployment using a US-hosted foundation model. The estimate covers hardware procurement, BSI certification, MedKI's on-premise deployment fee, the data protection officer's additional workload, and the legal costs of the 47-page classification opinion and processor agreement review. The counter-argument from Dr. Brandt-Keller's team: a regulatory enforcement action under DSGVO Article 83, at up to four per cent of annual global turnover, would cost multiples of that figure. The architecture is the insurance premium.

Comparison with US peers: what the regulatory gap produces

The structural differences between the Charité deployment and the US clinical AI deployments that have attracted the most attention — Mayo Clinic's CARS framework, Mass General Brigham's pilot programme, and Stanford Medicine's radiology agent initiative — are not differences in technical ambition. They are differences in regulatory substrate. The US FDA's Software as a Medical Device guidance is principles-based, engaged through voluntary Pre-Submission pathways, and leaves substantial interpretive discretion to health systems. The EU AI Act is obligations-based, creates mandatory conformity assessment requirements, and leaves no equivalent discretion on whether high-risk classification applies. The practical consequence: a US health system can begin deployment and engage the FDA in a cooperative dialogue about whether its approach is aligned. A German health system must complete conformity documentation and BSI notification before the first patient encounter.

This front-loading changes the deployment timeline asymmetry. Mayo's CARS framework governance work ran in parallel with the deployment itself — the institution refined its eval architecture through live operational experience. Charité's KKSO framework was fully documented and externally reviewed before any agent capability reached a clinician's screen. The Mayo approach is more agile; the Charité approach is more auditable from day one. Neither is demonstrably safer in clinical terms, because the safety question depends on the quality of the governance content, not the sequence of documentation. What the EU approach does produce is a complete paper trail from the moment of deployment — a litigation and regulatory defence asset that US institutions building governance retrospectively do not yet hold.

The accountability architecture also diverges. US clinical AI deployments converge on the design principle that agents are advisory and humans decide — but the named-reviewer requirement, with specialty-matching, is not standard in US practice. Mayo's red-team chart review panel operates at the programme level; it does not assign accountability for individual encounters to a named clinician with a documented qualification match. Charité's KKSO requires this at every encounter. The operational cost is higher. The legal clarity — if a patient outcome is challenged and the question becomes which physician accepted accountability for a specific agent output — is considerably greater.

The vendor landscape that the EU framework is sorting

The DSGVO and AI Act requirements are producing a visible selection effect in the German clinical AI vendor market. Three categories of vendor are emerging. The first is self-hosted specialists: companies like MedKI Systems that build deployment-ready stacks designed from the outset for on-premise installation, BSI certification, and DSGVO-compliant data handling. These companies carry higher integration costs but arrive with documentation packages that reduce the health system's conformity burden. The second category is EHR-native integrations: SAP's clinical AI module, released under its Fionn Health suite in Q3 2023, and the German subsidiary of Dedalus Healthcare, which acquired a clinical NLP company called Semantik Medical in November 2022, both offer agent capabilities that run inside an existing EHR perimeter, reducing but not eliminating the DSGVO data-handling problem. The third category is international platforms operating through EEA data residency guarantees: Microsoft Azure Healthcare APIs and Google Cloud Healthcare API both operate EEA-region deployments certified under DSGVO, and several German hospitals in the BG Kliniken network are piloting agent capabilities on these platforms under data-processing agreements reviewed by their respective state data protection authorities.

Charité chose the first category. The institution's digital health steering committee evaluated all three in a structured procurement process that ran from April to September 2022. The evaluation criteria weighted DSGVO compliance and AI Act conformity documentation at 35 per cent of the total score — a weighting that no US procurement process for clinical AI has yet publicly reported. MedKI scored highest overall, but the steering committee's evaluation report noted that the company's conformity documentation package, while the strongest of the three evaluated, still required Charité to produce the 312-page conformity assessment largely from its own internal resources. That work — which consumed approximately 1.4 full-time equivalents in legal, clinical informatics, and quality management staff over 14 months — is the hidden labour of EU AI Act compliance that vendor marketing does not price.

What to watch

Charité's programme will iterate through 2024 under active observation from health systems across the DACH region and from the European Commission's AI Office, which is monitoring early high-risk AI deployments for enforcement precedent. The variables are narrow and legible.

• Whether the European AI Office issues any formal enforcement guidance or corrective notices arising from high-risk medical AI deployments in the first half of 2024 — any such guidance will reset the conformity documentation standard for every health system currently in pre-deployment planning.
• Whether the three Charité capabilities currently in extended conformity review clear for patient-facing deployment, and on what timeline — the patient portal triage agent is the one to watch, because it would be the first patient-facing diagnostic agent at a major German Universitätsklinikum operating under a completed EU AI Act conformity assessment.
• Whether MedKI Systems' ClinBERT-DE architecture is adopted by other German health systems, which would establish a de facto interoperability standard for DSGVO-compliant clinical inference — three Universitätskliniken in Munich, Hamburg, and Düsseldorf are understood to be in active procurement evaluations as of February 2024.
• Whether the Bundesärztekammer's AI task force publishes updated professional guidance that codifies the specialty-matching requirement as a national standard — if it does, the accountability architecture Charité built will become mandatory rather than exemplary for all German health systems deploying diagnostic agents.
• Whether the 15-day model update review window in Charité's MedKI data-processing agreement creates a material operational disadvantage relative to API-based deployments at US peers — this is the governance cost most likely to generate internal pressure for architectural revision once the deployment matures past its first year.

Frequently asked

Does the EU AI Act's high-risk classification apply automatically to all diagnostic agent capabilities, or only to some?

Annex III, Point 5(a) of the AI Act covers AI systems intended to be used as safety components in the management and operation of critical digital infrastructure, and AI systems intended for use in safety-related procedures in healthcare. Charité's legal analysis treated any agent producing a differential diagnosis recommendation, severity stratification, or escalation trigger as falling within scope. Pure summarisation agents — those that condense existing clinical documentation without generating new clinical content — were assessed as outside Annex III scope and deployed under a lighter documentation regime. That line is contested: the European Commission's AI Office has not yet issued interpretive guidance on where summarisation ends and recommendation begins.

Why can Charité not use a US-based cloud foundation model API for inference?

Two overlapping constraints apply. The DSGVO prohibits transferring personal data — including pseudonymised medical data, which German supervisory authorities treat as personal data under most clinical conditions — to non-EEA countries without an adequacy decision, binding corporate rules, or standard contractual clauses approved by the relevant supervisory authority. The United States currently lacks an EU adequacy decision for health data purposes following the Schrems II ruling. Separately, the EU AI Act's high-risk requirements mandate that providers of high-risk AI systems keep technical documentation within the EU and make it available to supervisory authorities on request — a requirement that is operationally incompatible with shared-inference cloud environments where model internals are not disclosed.

What does the specialty-matching requirement mean in practice for scheduling and staffing?

Each diagnostic agent capability is assigned a Reviewer Specialty Code — a mapped Facharzt qualification required of the named reviewing physician. For time-sensitive capabilities, such as the emergency department sepsis stratification agent, this means the Oberarzt on call at the time of output generation must hold the Internal Medicine or Emergency Medicine Facharzt qualification. Charité's implementation required updates to the on-call scheduling system to record and verify qualification data at the time of agent output generation, so that the accountability log automatically confirms reviewer qualification without manual documentation. The scheduling system change took 11 weeks to implement and involved integration work between the agent platform's audit module and SAP's HR system, which holds staff qualification records.

How does the 312-page conformity assessment compare with what US health systems document for equivalent deployments?

There is no direct US equivalent. Mayo Clinic's CARS framework documentation, which is the most comprehensive US clinical AI governance disclosure currently available, is an internally developed quality standard — not a regulatory filing. The CARS documentation was produced primarily for internal governance and pre-submission FDA dialogue, not to satisfy a mandatory pre-deployment regulatory requirement. Charité's conformity assessment, by contrast, was produced to satisfy the EU AI Act's mandatory conformity assessment requirement for high-risk systems — a legal obligation with enforcement consequences if incomplete. The 312-page figure covers quality management system documentation, risk assessment under ISO 14971, technical documentation of the model architecture and training data, instructions for use, post-market monitoring plan, and the DSGVO data protection impact assessment. The DPIA alone runs to 68 pages.

Is the Charité model replicable at a smaller German Krankenhaus without university-level legal and informatics resources?

Not in its current form. The 1.4 full-time equivalents consumed by conformity documentation, the dedicated BSI-certified compute infrastructure, and the in-house data protection officer capacity required for ongoing DSGVO oversight represent a resource threshold that a 200-bed non-university hospital cannot meet internally. The most likely path for smaller institutions is a shared governance model — either through a Krankenhausverbund that pools compliance resources across member hospitals, or through a vendor-supplied conformity documentation package that reduces the health system's in-house documentation burden. Neither path currently exists at production scale in Germany. MedKI Systems and two competitor vendors are understood to be developing packaged conformity documentation offerings, but none had reached market by the date of this briefing.

The structural asymmetry this creates

The deployment itself advances Charité's clinical capabilities. The governance architecture advances something larger: a documented proof of concept for EU AI Act compliance at a major clinical AI deployment that other institutions, regulators, and vendors can examine. What the institution demonstrated in January 2024 is that a German Universitätsklinikum can operate diagnostic agents within the full constraint set of the AI Act, DSGVO, and Bundesärztekammer professional standards — simultaneously, without waiting for interpretive guidance that may not arrive for years. That demonstration changes the negotiating position of every German health system currently sitting across the table from a vendor whose deployment proposal does not address conformity assessment, specialty-matching accountability, or self-hosted inference architecture. The vendor's capability demo no longer suffices. The regulatory documentation question does.

The asymmetry between the EU and US regulatory environments will not resolve quickly. The FDA's SaMD guidance is evolving toward greater specificity, and several US health systems are building governance architectures that converge with the EU approach in their substance if not their legal form. But the mandatory front-loading of conformity work under the AI Act means that EU institutions entering the market now are building documentation assets that will compress their regulatory exposure for years. The €4.1M premium Charité paid for DSGVO-compliant infrastructure and conformity documentation is a sunk cost that becomes a competitive asset as enforcement begins. German health systems that treated regulatory compliance as a future problem will face a retroactive documentation burden that Charité, and the institutions now following its architecture, will not.