Attribution & the explanation that isn't
A saliency, heat or attention overlay shows where a model's response concentrated. Every reader alive will read it as the reason for the finding. Those are different claims, only one of them is true, and the gap between them is this whole page.
Overview
Attribution methods answer a narrow engineering question: which parts of the input moved the output most. That is useful for the people who built the model. Rendered over anatomy and placed beside a finding, it answers a different question entirely — why did it say that? — and it answers it wrongly, because it does not contain that information.
The failure is not that overlays are inaccurate. It is that they are persuasive about something they do not describe. A reader who sees a bright region over a lesion concludes the model found the lesion for the reason they would have. Nothing in the overlay supports that, and the impression is very hard to dislodge.
A model may show where it looked. It may never present where it looked as why it concluded.
The corollary does the practical work: a finding must be statable in words, with a location, without the overlay. If the overlay is the only way to know what was found, there is no finding — there is a picture.
Where, not why
An overlay is a pointer. It carries no reasoning, and the interface never lets it stand in for any.
The finding survives without it
Words and a location come first. The overlay is a convenience for finding the place, not the content of the claim.
No quantity it does not have
No intensity ramp, no percentages, no colour scale. A gradient reads as a measured surface, and none was measured.
What a mark is allowed to claim
Three things get drawn on an image and they make three different claims. Conflating them is the most common error in this area, and it is usually made by rendering all three the same way.
| What is drawn | The claim it may make | The claim it may not |
|---|---|---|
| A detection mark box or outline |
"Something the model was trained to detect is here." A locatable, checkable assertion. | That it is correct, that it is the only one, or that its extent is a measurement. |
| A boundary segmentation contour |
"This is the extent the model assigned." Becomes a measurement the moment a number is derived — see Derived measurement. | That the boundary is where a human would have drawn it. |
| An attribution overlay saliency, heat, attention |
"The response concentrated here." Nothing else. It is an engineering diagnostic that happens to be renderable. | That this is why. Also: that brighter means more certain, that the highlighted tissue is abnormal, or that unhighlighted tissue was assessed. |
Where it looked is not why it concluded
mark 1 · outline only
Cover the image and the finding is still complete: what, where, how big, from what. That is the test.
- The words come first in the reading order, not the picture. A finding rendered only as pixels cannot be quoted, dictated, escalated by telephone, or read by a clinician using magnification.
- Never label an overlay "explanation", "reasoning", "why" or "because". Label it what it is — where the response concentrated.
- Never offer it as the answer to a challenge. When a clinician asks why the model flagged something, an overlay is not a response; the honest response is the model's scope, its performance characteristics and its version.
- Unhighlighted is not "assessed and clear". Absence of attribution says nothing about absence of disease, and the interface never implies otherwise.
- Attribution is not evidence in the record. It does not enter the report and is not stored as the justification for a decision — the same boundary Reprioritisation draws.
Rendering: outline, never ramp
A continuous intensity ramp reads as a measured surface over anatomy — a per-pixel quantity with units, of the kind a perfusion or dose map genuinely produces. An attribution method produces nothing of the sort. Rendering one as a smooth heat field asserts a precision and a physical meaning that do not exist, and it does so in the visual language of quantities that do.
This is the direct heir of Confidence disclosure's rule that a score is never a colour scale. Same failure, one dimension up.
A boundary and nothing else. It says "here", which is all it knows, and the tissue underneath stays visible.
A ramp. It obscures the tissue it points at, implies brighter-is-more, and looks exactly like a perfusion map — a thing with units.
- Outline, not fill. An overlay that obscures the anatomy has removed the evidence in order to comment on it.
- Never an alarm hue. Marks are not priorities; borrowing the IEC palette dilutes it everywhere else — see Colour.
- Never colour alone to distinguish mark types. Shape, dash pattern and a text label carry it.
- No animation on appearance (SC 2.3.1), and no pulsing to draw attention — the mark is already the attention.
- Removable in one action, and off by default in the paradigms that require it. When the marks appear is settled by Reader paradigm; this page only governs what they claim.
A finding has an address
When the referent is one of hundreds of thousands of comparable units — slices in a study, beats in a Holter, regions of a whole-slide image — a finding exists and the reader has found it are different states, and only one of them is visible on screen.
- The address is stated in the units the reader navigates in — series and slice, not an internal index. A coordinate nobody can type into their own viewer is not an address.
- Navigating there is not assessing it. The system records that a reader was taken to a location, never that they evaluated it — the same distinction as Pending work's refusal of a "seen" state.
- State the bound of what was searched. "3 of 4 series analysed; the non-contrast series was not" is a different claim from silence, and only one of them is honest — see Autonomous result.
- Never imply the remainder was cleared. An absence of marks over 299 slices means the model produced no output there, which is not the same as nothing being there.
States
| State | Rendering |
|---|---|
| Marks available, not shown | Stated, with a control to show them. Silence is ambiguous: nothing-found and not-yet-shown are different facts. |
| Marks shown | Outline plus a text finding. The finding stands alone if the overlay is dismissed. |
| Nothing marked | Stated as "no regions marked", scoped to what the model detects — never "normal", and never "no abnormality". |
| Analysis failed or partial | Which series were and were not analysed. Distinct from nothing-marked, which is the distinction Key–value pair exists to protect. |
| Outside validated scope | No marks and a reason — the paediatric case, the unusual protocol. Withhold rather than extrapolate, per Confidence disclosure. |
| Reader dismissed the marks | Recorded. The knowledge does not go away when the pixels do, and the record should reflect what was on screen — see Clinician override. |
Who holds each rule
An attribution map is an overlay on an image, and this package ships no image viewer. Almost every rule here is about something the package cannot see.
| Rule | Held by | Where |
|---|---|---|
| Never an alarm hue | Status chip | If the overlay’s legend is a chip, it cannot reach the alarm palette without an alarm tone. The overlay itself is drawn by you. Asserted: StatusChip: an alarm tone maps to the governed hue, not a chosen colour |
| The words come first in the reading order | The implementer | Document order on the review screen. |
| Never label an overlay “explanation”, “reasoning”, “why” or “because” | Spatial finding | The control that shows the marks has a constant label — Show where the response concentrated — and the component’s code contains none of those words. What your viewer labels its own chrome is still yours; the control the reader uses is not. Asserted: SpatialFinding: marks start hidden, and the control never says explain |
| Never offer it as the answer to a challenge | The implementer | Interaction design: what the disagree action reveals. |
| Unhighlighted is not “assessed and clear” | Spatial finding | The none state requires a scope and renders “No regions marked for … This says nothing about anything else.” Neither normal nor no abnormality is in the code.Asserted: SpatialFinding: nothing-marked is scoped, and never 'normal' |
| Attribution is not evidence in the record | The implementer | What gets written to the Event log is your choice; the log will faithfully record a heatmap if you tell it to. |
| Outline, not fill; no animation on appearance; never colour alone | The implementer | Canvas rendering, all of it yours. |
| Removable in one action, and off by default | Spatial finding | A Switch that starts off, with no prop to start it on. Hiding the overlay pixels when it moves is your viewer’s job; onShownChange is the moment.Asserted: SpatialFinding: marks start hidden, and the control never says explain |
| The address is stated in the units the reader navigates in | Spatial finding | address is required and is series, slice and the size of the series. The numbers come from your viewer; that they are the only address expressible does not.Asserted: SpatialFinding: the address is in the reader's units, and complete without the image |
| Navigating there is not assessing it | Spatial finding | onNavigate is the only callback and receives the address. Nothing here can be wired to a “seen” or “assessed” event — the same objection as tabs, held by the type this time.Asserted: SpatialFinding: navigating is the only callback; nothing records assessment |
A pattern page has no component underneath it, so a rule stated here is held by whoever builds against it unless a component already holds it. Of these 10 rules, 1 are held by a component this package ships, and 9 are the implementer’s. One held, nine not. This page describes rules for a component this package does not have, which is stated on it and is worth restating here.
Do's and don'ts
Where the response concentrated — a diagnostic view of the model, not a reason for the finding.
Names the overlay as what it is. A reader can decide how much weight to give it, which is the whole point of Article 14's "interpret the output correctly".
Explanation — why the model flagged this study.
It is not an explanation and does not contain one. The label is the entire failure: everything downstream follows from the reader believing it.
Region marked, left upper quadrant · series 4, slice 47 of 300 · 14 mm across the mark
Statable in words with a location. It survives a phone call, a printout and a screen reader.
See highlighted region.
The overlay is the finding. Dismiss it and there is nothing left to act on, quote or escalate — which means there was never a finding, only a picture.
No regions marked in the 4 series analysed. The model detects intracranial haemorrhage only.
Scoped to what the model looks for, and bounded by what it actually processed.
No abnormality detected.
A claim about the whole study made by a model that looks for one thing. The unhighlighted 299 slices were not cleared; they were not asked about.
Why did it flag this? → Model scope, performance characteristics and version.
The honest answer to a challenge. None of it is an overlay, and all of it is checkable.
Why did it flag this? → Show the heatmap.
Answers a question about reasoning with a picture of activation. The reader leaves satisfied and knows nothing more than before.
Accessibility
- Every mark has a text equivalent giving type, location and extent. An outline conveys nothing to a screen reader, and "see highlighted region" conveys nothing to anyone not looking at the pixels.
- The finding precedes the image in reading order, so the assertion is reached before the illustration of it.
- Marks are distinguishable without colour — dash pattern, shape and label (SC 1.4.1).
- Non-text contrast ≥ 3:1 for mark outlines against the image beneath them (SC 1.4.11), which is why they are white on a dark field and never a mid-grey.
- Toggling marks never traps focus and returns focus to the control that invoked it.
- No flashing on appearance (SC 2.3.1).
- The image field does not theme. A diagnostic image is not inverted because the surrounding interface is dark, and the demos here follow that.
- Reflow to 320 px (SC 1.4.10) — below the width at which a mark is legible, the text finding replaces the image rather than the image shrinking.
Outcomes of use
What this contributes to, in the terms of Usability & context of use. These are attributes believed to contribute to an outcome; the outcome itself is settled by observing real use in a specified context, not by this page.
- Effectiveness — the reader's belief about why the model responded matches what is actually known, which is very little. Over-trust founded on a persuasive picture is the failure this page exists to prevent.
- Efficiency — an address that takes the reader to the right slice, against the minutes otherwise spent hunting for what the software meant.
- Satisfaction — a tool that answers "why did it say that?" honestly. A product that responds with a heatmap trains its users to stop asking, which reads as trust and is not.
Clinical safety notes
Trace these in your risk file (ISO 14971) and usability engineering file (IEC 62366-1). Each bullet says what checks it: a component that holds the rule, a test in your verification file, deployment configuration, post-market surveillance, or a design review where no test can see it. A control checked by nothing is a sentence.
- An attribution overlay is never labelled or offered as an explanation. (held by Spatial finding) Mitigates: reliance on a reason the output does not contain.
- The finding is statable in words with a location. (held by Spatial finding) Mitigates: a finding that cannot be quoted, escalated, printed or read non-visually.
- No continuous intensity ramp. (design review; no test can see it) Mitigates: a per-pixel quantity implied where none was measured.
- Outline rather than fill. (design review; no test can see it) Mitigates: the overlay obscuring the evidence it comments on.
- Nothing-marked is scoped to what the model detects. (held by Spatial finding) Mitigates: a single-condition model read as a clearance of the whole study.
- Unanalysed and unmarked rendered differently. (a test in your verification file) Mitigates: a failed or partial analysis read as a negative one.
- Address given in the reader's own navigation units. (held by Spatial finding) Mitigates: a finding the reader cannot locate.
- Navigation recorded, assessment not inferred. (held by Spatial finding) Mitigates: "the reader was shown it" being mistaken for "the reader evaluated it".
- Marks excluded from the report and the record of justification. (a test in your verification file) Mitigates: a detection aid's output becoming a diagnostic claim.
Related
- Reader paradigm — when marks appear; this page governs what they claim.
- Derived measurement — what happens when a boundary becomes a number.
- Confidence disclosure — the same refusal of a colour scale, one dimension down.
- AI transparency — "correctly interpret the output" as an Article 14 obligation.
- ECG review — annotation over a waveform, where this system first argued the ordering.
- ForeCall — the reference application, and why there is no imaging viewer here.