Human Oversight Is a System Requirement, Not a UI Feature
Holding
Human oversight is ineffective if it is reduced to an approval button. In clinical AI, oversight must give a qualified person enough information, time, authority, and workflow position to intervene.
Authority
The EU AI Act identifies human oversight as an obligation for high-risk systems. FDA digital health materials also frame software oversight through intended use and patient risk. The engineering duty is to translate oversight into system behavior.
Issue
A clinician cannot supervise a model whose rationale, input completeness, uncertainty, and limitations are hidden. Nor can oversight function if the alert appears after the clinical decision has already been made or if local workflow makes override impractical.
Resolution
The system should identify the responsible human role, disclose the model's intended use, show relevant inputs and missingness, expose uncertainty, support override, and log the final decision. Escalation should be triggered for out-of-distribution cases, conflicting signals, or low-confidence outputs.
Evidence Package
The evidence should include role definitions, user instructions, usability testing, override logs, escalation rules, training records, and review of cases where human and model decisions diverged. Oversight exists only when the record shows that intervention was possible and meaningful.