Clinical AI Readiness Assessment
The artefact an AI product manager owns before a model is trained: a go/no-go that scores six weighted risk dimensions, applies the Cures Act CDS carve-out test and IMDRF categorisation, and gates on blockers no score can override.
Six dimensions, two statutory tests, one gate.
-
Step 01 — Instrument
Thirty-four questions across six risk dimensions, each answered on a fixed four-point ordinal scale (0–3). Questions carry an internal weight of 1–3; dimensions carry an external weight summing to 100. A dimension sub-score is the weighted answer total over the weighted maximum of the questions actually answered, expressed 0–100.
-
Step 02 — Cures Act CDS test
The four conditions of the 21st Century Cures Act clinical decision support carve-out, codified at FD&C Act §520(o)(1)(E), are evaluated as conjunctive gates. A function sits outside the device definition under this provision only if all four hold; the other subparagraphs of §520(o)(1) carve out other software and are not tested here. Any failure returns likely SaMD — regulatory pathway required, naming the condition that failed. FDA's September 2022 CDS guidance factors — directive output, and time-critical decision-making defeating independent review — are applied as the conservative reading. Where a condition passes only on a contested reading, the pass is reported as narrow rather than as a clean hold.
-
Step 03 — IMDRF categorisation
The IMDRF SaMD risk framework (N12) matrix is applied independently of the US device determination: significance of the information to the healthcare decision (treat or diagnose / drive clinical management / inform clinical management) crossed with state of the healthcare situation (critical / serious / non-serious), returning Category I–IV.
-
Step 04 — Blocker gating
Seven hard blockers are declared in advance as conditions over answers, not as score thresholds. Several are compound — the safety blocker requires autonomous action and irreversibility and high severity together. If any fires, the verdict is DO NOT PROCEED irrespective of the weighted total.
-
Step 05 — Banding and floors
Absent a blocker, the weighted total bands to PROCEED (≥78), PROCEED WITH CONDITIONS (58–77) or REDESIGN REQUIRED (<58). Two floors then apply: any dimension below 55 caps the verdict at PROCEED WITH CONDITIONS; any dimension below 40 caps it at REDESIGN REQUIRED. A strong average cannot bury a collapsed dimension.
-
Step 06 — Register and stability
Every failing answer emits a risk register row with an authored, question-specific mitigation and a computed severity index. A seeded sensitivity analysis (mulberry32, fixed seed) then perturbs every answered question by −1/0/+1 across 500 runs and reports how often the verdict survives assessor disagreement.
Assess a use case.
Loaded with a sepsis early-warning use case. Two further worked examples are provided. All three are answered realistically; two of the three land on a verdict their total score alone would not predict.
Demonstration only. Not validated for clinical or compliance use. Synthetic records. No real patient data is used anywhere in this tool. The worked examples are illustrative constructions, not descriptions of any real programme, product or vendor. This instrument is not a regulatory determination, not a certified compliance control, and not a medical device. It does not substitute for qualified regulatory counsel, an institutional review board, a privacy office, or the accountable clinical governance body — each of which must reach its own conclusion.
A weighted total is a measure of how much of a programme is in good order. A blocker is a statement that a specific thing is absent — a lawful basis for the data, a human between the model and an irreversible act, an owner for a regulated function. Those are not quantities that other good work can substitute for. Averaging lets a strong data posture pay for a missing kill switch, and it lets a well-run programme carry a disqualifying defect at a comfortable-looking 73/100. So the blockers are evaluated first and independently, and the total is reported alongside the verdict rather than producing it. The care-management example below is exactly this case: it is the highest-scoring of the three worked examples and the only one that must not ship.
| Dimension | Weight | Sub-score | Profile | Answered | Contribution |
|---|
Weights are not equal by design. Patient safety and regulatory posture together carry half the instrument because a failure in either is discovered late, reaches a patient or a regulator, and cannot be retrofitted. Monitoring carries the smallest single weight but has the widest reach — it is the dimension that catches the other five after go-live.
All four conditions must hold for the software function to sit outside the device definition under this provision. These are conjunctive: one failure is sufficient to place the function inside device regulation.
A row is emitted for any answer scoring 0 or 1, and for any answer scoring 2 on a critical (weight-3) question. Severity index = (3 − answer) × question weight × (dimension weight ÷ 10). Mitigations are authored per question, not generated from a template. Rows marked inherent characterise the hazard the use case carries rather than a gap in the plan — they are excluded from the conditions named in the executive summary, because you cannot design away the severity of sepsis, only the operating point at which you meet it.
| Sev | Idx | Risk | Dimension | Recommended mitigation |
|---|
Ordinal judgements are not measurements. Five hundred perturbed copies of this assessment are scored, each answered question independently shifted by −1, 0 or +1 level (clamped to 0–3) using a fixed seed, so the figures below are reproducible on every load. If the verdict is fragile, it should be read as a range rather than a decision.
| Verdict under perturbation | Runs | Share | Profile |
|---|
| Dimension | Question | Answer recorded | Score |
|---|
What it catches, and what it cannot.
What it catches is the failure a scoring average hides: a programme 73% in good order and still disqualified. It encodes the Cures Act carve-out's four conjunctive conditions and the IMDRF N12 matrix, and gates on seven pre-declared blockers rather than the total.
What it misses is most of what a determination requires. FDA's 2022 CDS guidance reads these conditions more narrowly than the statute alone suggests; where a reading is contested — is encounter audio a signal from a signal acquisition system? — the tool does not resolve it. The borderline answer passes the gate, but the pass is reported as narrow and the determination says so, because the conservative reading would fail that condition and only counsel can settle which reading applies. It has no view on state law, the EU AI Act, 42 CFR Part 2, or payer contracts. Two honest assessors will differ by a level on most questions, which is why stability is reported.
A real deployment adds what this cannot produce: a counsel determination memo, an IRB submission, subgroup validation on local data, and a signed governance record.
A human stays in the loop at three points. The regulatory determination belongs to qualified counsel. The severity judgements belong to the accountable clinician. The decision to accept residual risk belongs to a named executive — never to a number.