Diagnostic-ML results published · LLM reports in review Assurance Evidence Base

The evidence, with its limits attached.

We say assurance should be evidence, not slogans. This is where we hold ourselves to that: our own benchmark runs, published with the methodology, the run artefacts and the limitations that qualify them. Where something was not tested, we say so. Where a number moved between runs, we show the movement rather than the flattering figure.

How to read every number on this page

The diagnostic-ML run is on synthetic data. It validates our detectors, not any real device. The failures in it are planted by us, and the run is built to fail if it does not find them. That makes it evidence that our method works — it is not, and must never be quoted as, a finding about any real diagnostic model, vendor, hospital or patient population. Running the method against real clinical data needs credentialed datasets or a clinical partner, and we will say so plainly on the day that changes.
These are existence proofs on a 36-case battery, not safety rates. Adversarial Battery v1 is 36 cases across 6 threat categories, 6 cases per category. It supports claims of the form “this model produced this failure under this probe on this date”. It does not support a statistical safety rate.
Any single run’s pass-rate is a point estimate with a roughly ±1-case (about 3 percentage point) band. We measured this rather than assuming it: with the harness commit, seed and temperature all held fixed, individual cases still flipped between runs. Read 86.1% as “about 86%, give or take a case”, never as an exact figure.
A flat aggregate score is not evidence of case-level stability. One model returned an identical 31/36 on three consecutive repeats while two different cases flipped in opposite directions underneath it and cancelled out, one of them critical severity.
A high pass-rate is not a safety certification. It reflects resistance to this specific battery on that date only. It is not a general safety guarantee, and it is not suitability for any particular deployment context. None of the models tested has been assessed for deployment in a clinical setting.

Scope

What the Assurance Evidence Base is, and what it is not.

It is the open record behind our method: the runs we have actually done, the tooling and versions we did them with, and the gaps we have not yet closed. It is deliberately unflattering where the evidence is thin.

What it is

Our own adversarial evaluation runs against named models, on named dates, with the exact tooling version recorded.
Results tied to run artefacts: every figure traces to a SHA-256-hashed Inspect .eval log and its exported results.json.
An explicit “not assessed” wherever a category was not genuinely exercised, reported as a result rather than left blank.
Limitations stated as prominently as findings, including the ones that weaken our own conclusions.
Reproduction commands, so a third party can attempt the run themselves.

What it is not

Not a certification, an accreditation, a regulatory approval or an endorsement, of any model or of us.
Not a safety guarantee, and not a statement that any model tested is fit for clinical deployment.
Not a public leaderboard result: this is our own battery, our own IP, not a shared benchmark others have run.
Not a diagnostic or predictive-ML assessment. Everything currently here is from our secondary LLM track. No diagnostic-ML benchmark run against real data exists yet, and we will not imply one does.
Not a comparison against hosted frontier models or vendor-reported figures. Every number is from our own runs only.

The record

Benchmark runs

Two tracks. Diagnostic-ML is the core one: transferability, calibration, subgroup performance and post-deployment drift, mapped to TRIPOD+AI, STARD-AI, DECIDE-AI, CLAIM and DCB0129/DCB0160. The LLM red-team track is secondary and applies only when the system under test is generative; its 2026-07 runs are archived below, with the one finding from them that still shapes how we read every LLM number.

The method: Diagnostic / Predictive-ML Evaluation Track, METH-0.2

Signed off 2026-09-07

What it is: The versioned methodology every diagnostic-ML run on this site is produced under: the D1 to D7 failure-mode taxonomy, the quantitative evaluation dimensions, the seven-stage pipeline, the model-access tiers that bound every claim, the shared scoring engine, the reporting-standard scaffold (TRIPOD+AI, STARD-AI, DECIDE-AI, CLAIM, DCB0129/DCB0160) and the post-market monitoring plan.
What the sign-off checked: The document was red-teamed against the running harness at commit fcae485 before it was signed off. The published version states, in section 3, which evaluation dimensions the harness computes today and which are specified for the next build, so a reader can hold a report to it. It also states on its face that we have no completed external client evaluation yet.
What it is not: Not a certification scheme, not a regulatory approval, and not a record of any engagement. A method a third party can reproduce against, nothing more.

Read the method → Source code on GitHub → Diagnostic-ML core track Open source, Apache-2.0 Version-tagged METH-0.2 Red-teamed against the code

Diagnostic-ML Evidence Run v2: PBC to CytoData transferability

Tested 2026-09-09

What was tested: Our first run on real images. A blood-cell classifier trained on one public dataset (PBC, one hospital laboratory) and evaluated cold on a second from a different laboratory (CytoData), on the clinically shaped task of flagging a cell that should not be read as normal. Task frozen on 4 September, every remaining decision and the acceptance bar committed before training, harness run once as written.
Headline finding: AUROC 0.9996 on the developer site, 0.8769 on the deployment site, an interval nowhere near zero. At the threshold a vendor would plausibly ship, the new site gets 74% sensitivity against a 90% floor and a PPV of 41%; calibration error goes from 0.004 to 0.245. And the formal deployment gate did not fire, because both failures rate Major rather than Severe under our placeholder bands: the first real-data case for the clinical band calibration, published rather than adjusted.
Not assessed: Subgroup performance (no demographic field in any public blood-film dataset), automation bias, and drift. Stated on the page, not skipped.

Read the results → Diagnostic-ML core track Real public data Pre-registered

Diagnostic-ML Evidence Run v1, results

Tested 2026-08-03

What was tested: A predictive model across the four dimensions that decide whether a diagnostic model is safe to deploy where it was not built: transferability, calibration, subgroup performance and post-deployment drift — each mapped item by item to TRIPOD+AI, STARD-AI, DECIDE-AI, CLAIM and DCB0129/DCB0160. The data is synthetic. This validates our detectors, not any real device: the failures are planted, and the run fails if it does not find them.
Headline finding: Three of the findings are about our own harness. The scorecard was rendered from developer-set metrics alone, so it read calibration and equity as Pass while the deployment site those same probes had already measured was failing both. One of our two shortcut probes turned out not to work on its own. And our drift statistic was dominated by an arbitrary smoothing constant rather than by the data. All three are fixed, regression-tested, and published rather than quietly patched. Adding confidence intervals then overturned a fourth result: a transferability “drop” whose interval crosses zero.
Not assessed: Automation bias (D6), and 8 of 21 mapped standards items — no prediction file can show whether a Clinical Safety Officer exists or whether clinicians over-defer in the real workflow. The severity bands throughout remain pre-registered placeholders awaiting clinical-advisor calibration, which is a clinical judgement no amount of engineering closes.

Read the results → Diagnostic-ML core track Synthetic — detector validation Reproduced ×3 95% CIs throughout

Earlier work: Adversarial Battery v1 (LLM secondary track)

Tested 2026-07-18 to 07-31 · archived

What was tested: Three weekly runs plus a repeat-run variance check: 36 adversarial cases across six threat categories (prompt injection, jailbreak, data exfiltration, tool abuse, bias and fairness, clinical safety) against three local open-weight models, each graded by a different model.
The finding that outlived them: A flat aggregate score is not evidence of case-level stability. One model returned an identical 31/36 on three consecutive repeats while two different cases flipped in opposite directions underneath it and cancelled out — one of them critical severity — with the harness commit, seed and temperature all held fixed. That is why we never claim byte-identical or deterministic results as a general property of an LLM run, and why any single run’s pass-rate should be read as a point estimate with a roughly ±1-case band.
Why they were never published: They stayed in draft, and on archiving we found the reason they could not have shipped as they stood: the proof package’s checksum manifest covers only the raw evaluation logs, which were never committed, so a third party cannot verify those numbers. Nothing suggests the figures are wrong — but unverifiable is a worse problem than wrong for an assurance company, so they are archived rather than published. Reviving them means re-running the battery and capturing a package that actually contains its artefacts.

Archived — available on request LLM secondary track Never published

Independence, and what we are not claiming

UK AI Evaluation Ltd (incorporated 24 June 2026, Companies House no. 17299891) is an independent assessor. Our work is aligned with the direction of the MHRA, the UK AI Security Institute and the NHS. We are not affiliated with, certified by, accredited by or endorsed by any of them, and nothing on this page or the pages it links to should be read as implying otherwise.

Nothing here constitutes certification, regulatory approval, or endorsement by any regulator or body, of any model tested or of UK AI Evaluation. A result on this site is a record of what a specific model did under a specific probe on a specific date, nothing wider.

All test inputs in these runs are synthetic. No real patient or personal data was used, including the simulated NHS number in the data-exfiltration cases.