AI transparency & human oversight
Human oversight is usually written about as a principle. In the EU AI Act it is a design requirement — a list of things a person must be able to do, which the interface is responsible for making possible. That list is the spine of every AI page in this system.
Overview
This system has covered algorithmic output for a while, but only one shape of it: a model emits a score or a flag about one input, and a clinician accepts or rejects it. Three newer shapes break that assumption — a mark on an image, a paragraph of generated prose, and a sequence of actions taken without asking — and each fails differently.
What they share is that they are harder to check than a number, and in two cases the output actively discourages checking. This page holds the regulatory citations the pattern pages then reference, so those pages can cite rather than assert.
Everything here is a reading of published text, given so you can check it — see How to read a claim, where this is Standard class. Whether the AI Act applies to your device, in what role, and from when, depends on its classification, your position in the supply chain, and transition dates. Verify against the current consolidated Regulation and take your own advice.
Oversight is buildable
Article 14 lists five things a person must be able to do. Each maps onto something the interface either provides or fails to.
Both regimes apply
A medical device that is or contains AI answers to the AI Act and the MDR. Neither substitutes for the other.
Say that it is a machine
Machine-generated content is disclosed as such, at the point of use and in the record — not only in a manual.
Two regimes, both applying
Under AI Act Article 6(1) an AI system is high-risk where it is intended as a safety component of a product covered by the Union harmonisation legislation in Annex I — which includes the MDR and IVDR — and that product requires third-party conformity assessment. In practice most AI in a Class IIa or above device lands there.
| MDR / IVDR asks | AI Act adds | |
|---|---|---|
| Risk | ISO 14971 risk management across the lifecycle | Art. 9 — a risk management system specific to the AI, including reasonably foreseeable misuse |
| Data | Clinical evidence for the intended purpose | Art. 10 — governance of training, validation and testing data, including representativeness and bias examination |
| Records | Traceability in the technical file | Art. 12 — automatic logging of events over the system's lifetime — the obligation an unattended sequence would have to be reconstructed from, if this system covered one yet |
| Information | Annex I §23 labelling and instructions for use | Art. 13 — instructions giving capabilities, limitations of performance, and the human oversight measures |
| Oversight | IEC 62366-1 usability engineering; use error | Art. 14 — oversight designed into the system. See below |
| Performance | Clinical performance for the intended purpose | Art. 15 — accuracy, robustness and cybersecurity, with accuracy metrics declared in the instructions |
| Disclosure | — | Art. 50 — where applicable, that content is AI-generated, and machine-readable marking of synthetic output |
Article 14, read as a specification
This is the part worth reading closely. Article 14 requires high-risk systems to be designed so that a natural person can do five specific things. Each is an interface obligation, and each already has — or now needs — a page here.
| A person must be able to… | Which means, on screen | Where it lives |
|---|---|---|
| Understand the capacities and limitations | The scope of what the model was evaluated on, stated with the output — not only in a manual nobody opens mid-shift. | Confidence disclosure · Autonomous result |
| Remain aware of automation bias — the tendency to over-rely on output | Design that works against reliance: evidence before conclusion, symmetric agree/disagree, and output that does not read as already-verified. | Confidence · Reader paradigm · Composed output |
| Correctly interpret the output | An output whose form matches what it is. A probability is a claim; a heatmap is not a reason; a fluent paragraph is not a verified one. | Attribution · Composed output · Derived measurement |
| Decide not to use it, or disregard the output | Declining must cost no more than accepting. Where disagreeing is slower than agreeing, the system has removed the choice while appearing to offer it. | Clinician override · Editing the draft |
| Intervene or interrupt — a stop | A live, reachable stop that needs no reading, and leaves a defined state behind. | Therapy recommendation |
An earlier version of this system declined to cover conversational and generative tools on the grounds that the design rules were not settled, and that any page would be inventing rules it could not defend. Article 14 is the answer to that objection. "Remain aware of automation bias" and "decide not to use it" are not aspirations — they are things a design either affords or prevents, and they can be argued against a published text rather than against taste.
What changes with the shape of the output
The same oversight obligation lands differently depending on what the model emits. This is the reason the new pattern pages exist rather than a single "AI" page.
| Output | How it resists checking | The design answer |
|---|---|---|
| A score | Barely — a number announces itself as a claim, and the reader knows to weigh it. | Calibrate it, scope it, never dress it as a verdict. |
| A mark on an image | It directs the eye. Once seen it cannot be unseen, and attention spent where the mark is, is attention not spent elsewhere. | Control when it appears, make it removable, describe it in text, and never let a heatmap stand in for a reason. |
| A paragraph of prose | Actively. Fluency reads as authority; the better the writing, the less it is checked. And what was omitted is invisible by construction. | Ground every clinical statement, mark what is ungrounded, keep the register honest, and make signing an authorship act. |
| A sequence of actions | It has already happened. Errors compound step to step, and nobody was watching the middle. | State the plan before the first step, log every step, keep a stop live throughout — and, for now, not build one: this is the empty cell the register holds open deliberately. |
Saying it came from a machine
Article 50 requires, for the systems it covers, that people are informed they are interacting with an AI system and that generated content is disclosed as such, with synthetic output marked in a machine-readable way. For clinical software the practical reading is unglamorous and strict:
- At the point of use. A draft says it was machine-drafted, on the draft — not in a settings screen or a footer.
- In the record. The fact survives the sign-off. A note that was generated and then approved is not the same artefact as one a clinician wrote, and the record should be able to tell them apart years later — the same instinct as version stamps on every output.
- Not as an apology. "AI-generated" is a provenance statement, not a disclaimer, and it does not substitute for saying what the tool can and cannot do.
- Never as a badge alone. An "AI" chip on a worklist row tells a clinician nothing actionable — see Reprioritisation, where the movement is stated instead.
What this page does not settle
- Whether the AI Act applies to your device, in which role, and from when. Classification, supply-chain position and transition dates all bear on it.
- How to demonstrate conformity. This is a design system; the evidence is yours.
- Model development. Data governance, bias examination and robustness are real obligations and are not interface problems.
- Whether a given tool is a medical device at all. A tool that documents an encounter without informing a clinical decision may not be — see MDCG 2019-11 on qualification. This system covers the case where the output is relied on.
Do's and don'ts
Treat Article 14's five capabilities as a checklist against a specific screen: can this user, here, understand the limits, resist over-reliance, interpret the output, decline it, and stop it?
Answering "no" to any of them is a design finding, and it is testable in formative evaluation.
Satisfy "human oversight" by putting a clinician's name on the output and calling them accountable.
Naming a supervisor is not designing oversight. If declining costs more than accepting, the choice was removed while appearing to be offered.
State the capabilities and limitations with the output, in the words a clinician would use.
Article 13 puts them in the instructions; Article 14 requires the person to actually understand them. A manual satisfies the first and rarely the second.
Rely on training to close a gap the interface creates.
"Users are trained not to over-rely on it" is the automation-bias equivalent of a dismissible safety warning.
Disclose machine authorship on the artefact, and keep it in the record after sign-off.
A note that was generated and then approved is a different artefact from one a clinician wrote. Someone reconstructing a decision needs to know which.
Let the disclosure disappear once a human signs.
Sign-off changes who is accountable. It does not change where the words came from.
Match the design answer to how the output resists checking — a mark, a paragraph and an action fail differently.
One "AI guidelines" page covering all of them would be too general to act on, which is why this system splits them.
Reuse the confidence-score playbook for generated prose.
A probability beside a paragraph implies the paragraph was scored. It was not — see Composed output.
Accessibility
- Machine-authorship disclosure is text in the accessible name of the block it describes, never a visual tint or an icon. A user who cannot see the styling must still know who wrote it.
- Limitations travel with the output in reading order, not in a sidebar a screen-reader user reaches afterwards.
- The stop is reachable by keyboard from anywhere in the flow it stops, and is never behind a menu.
- Overlays and marks carry text descriptions — location and extent — because an outline conveys nothing non-visually.
- Never colour alone to distinguish generated from human content, or grounded from ungrounded statements.
- Readable at 320 px (SC 1.4.10) — oversight controls are the last thing that should be reflowed off-screen.
Outcomes of use
What this contributes to, in the terms of Usability & context of use. These are attributes believed to contribute to an outcome; the outcome itself is settled by observing real use in a specified context, not by this page.
- Effectiveness — reliance on an output matches that output's actual reliability. Over-reliance and reflexive dismissal are both failures, and Article 14 names the first one explicitly.
- Efficiency — oversight that costs less than the assistance saves. Oversight designed as ceremony gets performed as ceremony.
- Satisfaction — a clinician who feels responsible for an output rather than exposed by it. Accountability without the ability to intervene is the fastest way to lose a user's trust in a tool they are obliged to use.
Clinical safety notes
Trace these in your risk file (ISO 14971) and usability engineering file (IEC 62366-1).
- Capabilities and limitations stated with the output. Mitigates: use outside the evaluated population, and reliance the evidence does not support.
- Automation bias designed against, not trained against. Mitigates: over-reliance surviving every countermeasure that depends on the user remembering.
- Declining costs no more than accepting. Mitigates: an oversight control that exists formally and is unusable in practice.
- A live stop with a defined state after it. Mitigates: an intervention that leaves the system in an unknown condition.
- Machine authorship disclosed and retained after sign-off. Mitigates: a generated artefact indistinguishable from a human one during incident reconstruction.
- Automatic logging of events. Mitigates: an unattended sequence that cannot be reconstructed — AI Act Art. 12.
- Design answers matched to output shape. Mitigates: a control appropriate to a score being applied to prose or to an action, where it does nothing.
Related
- Confidence disclosure — the same obligations for a calibrated score.
- Composed output — why prose needs a different answer from a number.
- Attribution — interpreting an output correctly when it looks like a reason.
- Reference applications — why the far end of the autonomy axis is deliberately left empty.
- Device identification panel — where Art. 13 information meets MDR labelling.
- Reference applications — the rejection this page overturned, and why.
- Status & limitations — how to read the regulatory claims here.