← Assurance Evidence Base

Diagnostic-ML Evidence Run v1, results

Track: diagnostic-ML core · Testing date: 3 August 2026 · Harness: app.harness diagnostic-harness-0.2, commit 46e5ced · Methodology METH-0.2, engine 0.5.0, rubric RUBRIC-0.4

This is our first evidence run on the core track. Everything we had published before this — three adversarial batteries and a repeat-run variance check — was on the secondary LLM track. This run evaluates a model across the four dimensions that decide whether a diagnostic model is safe to deploy somewhere it was not built: transferability (does it survive a new site), calibration (do its probabilities mean what they say), subgroup performance (who does it work worse for) and drift (does it still work in six months). Each is mapped item by item to TRIPOD+AI, STARD-AI, DECIDE-AI, CLAIM and the DCB0129/DCB0160 clinical risk management standards.

The finding that mattered most was in our own reporting layer

Until this run, the D1–D7 scorecard was rendered from the developer-set metrics alone. On the reference corpus that produced a table reading calibration Pass (ECE 0.0262) and subgroup equity Pass (gap 0.0309) — while the deployment site those same probes had already measured sat at ECE 0.0682 and a subgroup gap of 0.065, and while the engine verdict immediately underneath said recommend against deployment.

A clean developer scorecard was hiding a deployment failure. That is precisely the failure mode our methodology exists to catch, and it was sitting in our own reporting code. It is fixed: both reads are now printed and the worse one sets the result, with a regression test pinning it. We are publishing it because an assurance company that only reports the bugs it finds in other people's systems is not doing assurance.

Results — Run A, detector validation

Seeded synthetic corpus: 1,500 developer-site cases, 1,500 deployment-site cases, plus a six-period post-deployment monitoring sequence. Failures are planted in D1, D2, D3, D4, D5 and D7; the run exits non-zero if any is missed. Every row traces to the run record in the full report's reproducibility appendix. “Not assessed” is reported explicitly, never left blank.

Engine verdict: 1 Severe finding in T1 Safety → recommend against deployment until remediated and re-tested. D2 reads Pass at a deployment ECE of 0.0682 because 0.05 is the low band and a low finding is a pass with a finding recorded — the number and the severity are both printed so the Result column alone cannot mislead. D3 reads “Measured, not scored” because the shortcut dimension is gated pending clinical calibration of its severity band: the probe produced evidence the methodology does not yet score, which is not the same as a pass.

Drift is a different failure from transferability

Against a fixed sign-off reference, cumulative AUROC decay reaches 0.1187 by month 5, while the largest single month-on-month step is 0.0211. A monitor comparing each month to the previous one would have waved every one of those comparisons through. That is why the reference is fixed at sign-off rather than rolling.

Month 6 is the row that justifies the power guard. At n=40 it reports an AUROC of 0.869 — better than any powered period, and close to sign-off. A monitor without a power guard would have read that as a recovery and closed the alert. It is excluded from the decay statistic and marked underpowered, while its PSI (0.2659, the worst in the sequence) is still used, because population-stability statistics need no outcome labels and a 40-case month is perfectly adequate to show the score distribution has moved. Label-free statistics on every period; performance claims only on powered ones.

Results — Run B, real-file ingestion on a haematology library

The 420-case synthetic clinical test-case library (haematology / CBC-FBC), fed through the production CSV adapter exactly as a client's prediction file would be. Site A (n=159) as developer, site B (n=221) as deployment, fairness stratum ethnicity.

Two things worth stating plainly. First, calibration slope 0.2427 is severe over-confidence — the model's probabilities are far more extreme than the outcomes justify — and it is completely invisible in the AUROC, which is a healthy 0.8929. That is the pattern the whole methodology is built around.

Second, one of our two shortcut probes did not work. The perturbation probe returned a negative AUROC drop: removing the planted acquisition-token shortcut did not degrade ranking at all. The conditional site-gap probe on the same data returned 0.2059. AUROC is rank-based, so a shortcut that inflates scores without reordering cases is invisible to it. Two probes for one failure mode turns out not to be redundancy — one of them does not work on its own.

Every number now carries an interval, and one of them changed a conclusion

Until this run every metric we produced was a bare point estimate. On a 159-case cohort that is not a small omission: it is the difference between “sensitivity is 0.76” and “sensitivity is somewhere between 0.64 and 0.88, and you cannot tell those apart”. Every headline metric now carries a seeded 95% bootstrap confidence interval. Differences use a paired bootstrap, because the interval for a difference has to come from the distribution of the difference, not from subtracting two independent endpoints.

Read the last row carefully. The developer-site equity gap only just excludes zero, so on its own it is weak evidence; the deployment-site gap is the one that carries weight. The point estimates alone made both look equally real.

We were reporting three subgroups at a perfect score because they were too small to fail

An earlier version of this run reported three ethnic groups in Run B at an AUROC of 1.0, with no qualification. Those groups had 13, 14 and 20 cases. A subgroup too small to estimate a score cannot support a fairness claim in either direction, and a table that reports it as a perfect score reads as reassurance precisely because the cells are too small to fail.

Groups below the floor (n≥30 and ≥5 of each class) are now excluded from the scored gap and named on the scorecard row with their n — excluded is not deleted. What remains is the real disparity: 0.9948 against 0.6970. The finding got stronger and more honest at the same time. Where no group is adequately powered the row reads “fairness is UNTESTED on this cohort, not passed”, which must never look the same as having no demographic labels at all.

Our own drift statistic was dominated by an arbitrary constant

PSI smooths empty histogram bins so the logarithm stays finite, and we floored them at a fixed 1e-6. For an empty current bin the resulting term is large and positive: the smoothing that looks conservative manufactures drift, and does so worst on the thinnest periods — exactly where the evidence is weakest. It scored a 45-case monitoring period at PSI 4.08 against a KS of 0.43.

With sample-size-aware add-half smoothing that same period is 1.17, in line with the 260-case period beside it. Empty-bin counts are now printed so a reader can see when the smoothing did any work at all. We are reporting this for the same reason as the scorecard defect above: a statistic whose value is driven by a constant we picked, rather than by the data, is not evidence.

Mapping to clinical reporting and assurance standards

The eight items not evidenced are not gaps in the run: they are things no prediction file of any size can answer — whether a named Clinical Safety Officer exists, whether the reference standard was pre-specified, whether clinicians over-defer to the model in the real workflow. They need a documentation audit or an in-situ clinical study. (Two further items, the TRIPOD+AI and CLAIM uncertainty requirements, were marked not implemented in the first version of this run; they are now evidenced, which is what moved coverage from 11 to 13.) A mapping that showed full coverage would be a mapping that was lying.

Reproducibility

Both configurations were executed three times. After removing the wall-clock field — the only value permitted to vary — all three repeats of each hashed identically. The pipeline is Python standard library only: no numpy, no scikit-learn, no torch. Every statistic here (AUROC by Mann–Whitney U, ECE, Brier, calibration slope, PSI, two-sample KS) is a few auditable lines of code.

This is a stronger reproducibility claim than we could make on the LLM track, for a structural reason: our 2026-07-30/31 variance check found that temperature=0, seed=42 is not a determinism guarantee for local LLMs, with individual cases flipping between repeats. That finding does not carry over here, because there is no sampling anywhere in this pipeline — the diagnostic track is seeded arithmetic over a frozen array. Our standing rule against claiming byte-identical results as a general property still applies to the LLM track and is not weakened by this.

What this does not show

Also, explicitly:

• Severity bands are pre-registered placeholders awaiting clinical-advisor calibration — every D1–D7 threshold, both drift bands and both power floors. They are defensible defaults, not clinically validated cut-offs, and no amount of engineering closes this: it is a clinical judgement. This is the single largest open item on the core track.
• The shortcut dimension is measured but not scored. Both D3 probes produce evidence the methodology does not yet score, pending that same calibration.
• Automation bias was not assessed and cannot be from a prediction file. It needs clinicians in the real workflow.
• Run B's cohorts are small — 159 and 221 cases. The intervals say so plainly: the transferability drop crosses zero and the deployment equity gap runs from 0.009 to 0.344. Treat every Run B point estimate as indicative.
• The bootstrap is percentile, not BCa, so it carries the usual small-sample bias for skewed statistics.
• PSI is unbounded and binning-dependent. KS is reported beside it as the binning-free cross-check; where the two disagree, the binning is doing the work.
• A clean scorecard would be necessary, not sufficient, and is never a safety certification.

Full limitations, methodology, standards mapping, reproduction commands and artefact hashes are in the full report: Diagnostic-ML Evidence Run v1, 2026-08-03 — available on request.

← Back to the Assurance Evidence Base