Written for clinicians, health-services researchers and reviewers. No AI background is assumed. The test questions stay confidential, but everything about how they are built, run and scored is documented here.
AusClinBench is an independent, cross-sectional benchmark of large language models ("AI models") against current Australian clinical policy. The unit of testing is an item: a short clinical vignette, one question with a single verifiable answer, and an answer key written from a primary source. There are 142 items across five domains: PBS authority requirements, MBS billing rules, regulatory obligations, triage thresholds, and immunisation and screening rules. Of these, 137 form the scored leaderboard set. The remaining five are open-question (equipoise) items, each scored separately in a pair with a settled question from the same area.
Ten contemporary models were tested in July 2026, each under three conditions (section 3), plus eleven earlier versions of the same model families for the trend-over-time analysis. Every answer is scored by exact match against the key. No AI model grades another model's answer at any point in the pipeline. This works like a written exam with a fixed marking scheme, not an examiner's judgement.
Versions 1.1 and 1.2 (September 2026) added fifteen further models under the two closed-book conditions only (from memory, and with the rule supplied). The web-search arm and the pushback, rewording and repeat-run experiments were not re-run. Two September models are excluded and reported as excluded, never scored zero: Gemma 4 31B (every call rejected by the provider) and Qwen 3.8 27B (67 of 284 replies empty).
Versions 1.3 and 1.4 (October 2026) added Claude Sonnet 5.5 and Claude Haiku 5.5 on 9 October, then Claude Opus 5.5 on 13 October, under the two closed-book conditions. They ran through a Claude Code subscription, not the API layer the other models used. Each prompt was unchanged, with the response format added at the end as an instruction that nothing enforced. No temperature or data-collection setting was sent. No earlier row was re-run, so no earlier score moved.
Version 1.5 (October 2026) added Gemini 4 Argon on 22 October, under the two closed-book conditions. It ran through the same API layer as the other Google rows, with the response format enforced and temperature set to zero. No earlier row was re-run, so no earlier score moved.
The v1.2 scoring correction. Eleven items had free-text answer keys that marked a semantically correct answer wrong when its wording differed from the key (“the entire limb” against a key of entire_limb, or an en dash against a hyphen). All eleven were converted to forced-choice answers and re-run across every model on 7 September. Each item now publishes its option set in the prompt and enforces it in the response schema. Scoring remains exact match, and no matching was loosened. Distractors are values models actually produced, or genuine alternatives from the same rule. Reasoning failures fell from 168 to 77 of 3,072 model-item pairs, so roughly half of that cell had been a scoring artefact. Every model's score rose between 0.3 and 4.8 points, and figures from before and after the conversion are not comparable. Chance rates on the converted fields moved from zero to between 1.3% and 25%, which lowers the chance-adjusted score rather than raising it.
The verdict rule. The automatic verdict moved from PROCEED (the best model measurably falls short of the bar) to UNDERPOWERED (too few items to decide) as a result of the correction. The bar was also raised from 90% to 95% on 7 September, on the grounds that a one-in-twenty error rate is the defensible safety boundary for clinical policy. Raising it makes PROCEED easier to reach and KILL (the best model measurably clears the bar) harder, so it is stricter on the models. The verdict is UNDERPOWERED at either value, and the change is recorded here because it was made after the version 1.2 results were known.
Each item is written by an Australian general practitioner directly from a primary source: the PBS Schedule, MBS Online, the Australian Immunisation Handbook, regulator guidance, or a national clinical guideline. At the moment of writing, the item records the source URL, the access date, and a verbatim quote of the sentence that grounds the answer key. An automated freshness tool re-fetches each source and re-verifies the quote against the live page, so a rule change flags the item for review rather than silently invalidating its key.
Items are synthetic. They are written from the public rule, never from a patient record or clinical encounter, so patient data cannot enter the set by construction. Each answer is a single verifiable fact - a yes/no, a number, a category, an item number, a timeframe - never an essay to be judged.
A subset of items are deliberate traps: written on topics where the easiest answer to find online is a superseded Australian rule or a current overseas one. These test whether web search helps or harms when the top result is misleading.
AI models learn from public text. A benchmark whose questions are published is absorbed into the next generation's training data, after which the test measures memorisation of the test rather than knowledge of the rules - the same reason exam banks and OSCE stations are not published. The method is public. The content is held out. A small public sample demonstrates the item format so the method can be reproduced, and worked examples on the results page use altered details for the same reason.
The ten July models ran every item under all three conditions. Later models ran the first two. Comparing the conditions separates out-of-date knowledge from faulty reasoning, and shows what real-world retrieval adds.
The July and September models were queried through a single API layer with identical prompts, a fixed structured response format, and provider-side data collection disabled (section 7). The historical arm ran eleven earlier versions of the same model families under the same conditions in the same week as the July models, so the trend over time is measured on identical items, not reconstructed from older publications.
Models must answer in a fixed structured form (specific named fields, machine-readable). Responses that fail to parse are recorded, disclosed and excluded - never guessed at. The form includes an explicit way to decline: a model can state that it cannot answer safely. A decline is never scored as correct, but it is reported separately, because for a clinician a prompt to check the source and a confident wrong answer are opposite outcomes.
Not all wrong answers carry the same clinical hazard. An answer that was the Australian rule until it changed, or is the current rule in the UK, US or NZ, is a real rule cited from the wrong time or place - a currency or localisation failure that may invite a double-check. An answer that was never the rule anywhere is a fabrication, which is the more alarming failure mode: it can produce an invented Medicare item number on a claim or a non-existent threshold in a record.
Benchmarks that grade severity usually use an AI judge or human raters scoring free text. AusClinBench keeps severity deterministic. Every distinct wrong value the models actually produced (300 values) was researched against primary sources. A value is labelled superseded or foreign only with a verbatim quote from an official source, with the URL recorded. Invented MBS item numbers are labelled fabricated only after individual verification against the official MBS item search. Classification is then exact matching of a model's wrong value against these evidenced lists.
No fabrication-by-exclusion. A wrong value that matches no evidenced list stays unclassified - it is never assumed to be fabricated just because it was not researched or not matched. Under-claiming is the deliberate bias: current coverage (156 of roughly 570 observed wrong answers, 27%) is stated wherever severity numbers appear, and labels await independent second-clinician sign-off.
Declines are never classified - a decline is not a wrong assertion. Only blind-condition answers are classified, because severity is about what a model asserts from its own memory.
Four checks probe whether the numbers are stable properties of the models or artefacts of the test.
| Check | Protocol | Result |
|---|---|---|
| Repeat run | The full item set run twice for one model (Gemini 3.1 Pro) with identical settings, days apart. | 97.8% answer-level agreement for that model, well inside the reported confidence intervals. Other models were not re-run. |
| Rewording | A 15-item sample rephrased - terse clinical shorthand versus a chatty description - with the facts unchanged. | Answers changed on 7-33% of items, depending on the model. Phrasing sensitivity is a real property of everyday use and is reported, not averaged away. |
| Framing | Four July models (from Google, Anthropic, xAI and OpenAI) asked the same six questions three ways on topics where Australian and US rules differ: no country stated, "In Australia:", and the exact guideline named. | 42% correct with no country stated (models default to US answers). 92% with "In Australia:". Naming the guideline added nothing further. |
| Pushback | After each correct answer in a 36-item set, a scripted one-sentence challenge: a colleague thinks it is a plausible wrong answer. | Capitulation ranged from 0% (six models) to 20%. Sycophancy is scored as a separate failure mode from ignorance. |
Negative findings are published. A fifth of the item set was built on the hypothesis that web search would harm accuracy when the top result is stale. The data refuted it: search helped every model, including on trap items. The result is reported with the same prominence a confirmation would have received.
Questions about the method, clinical review of the keys, or evaluation of a specific product against the suite: contact details are on the results page.