CLINICAL TECHNOLOGY, PUT TO THE TEST
AI scribes, under the microscope.
We test AI medical scribes on the same recorded consultations. Explore how they compare on transcription, clinical notes and speed—then look closer at the cases behind the numbers.
Every consultation has an inventory of every fact said that relates to the patient's conditions, symptoms or signs, the standard Weiner et al. (2020) used to audit doctors' notes. The inventories were drafted by AI from each reference transcript alone, before any note was graded, and every fact quotes the line it comes from.
Local preview — Hanah notes regenerated on 11 October 2026 with GPT-6.1 Sol, low reasoning: 50 notes across 10 consultations, five stored transcript variants each. Two consultations await new Hanah notes. WER and transcription evidence retain their original capture dates; other scribes retain their existing results. New notes were graded blind by Sonnet 5.5 medium, followed by an unblinded agent audit; provisional, without independent clinician validation. Hanah timings are estimates: prompted-note wait uses measured API generation duration + 2 seconds readiness allowance; time to a finalised note adds estimated review and correction time. Both exclude transcription finalisation and app overhead, pending a live rerun.
Benchmark · 22 September 2026
AI scribes against a doctor writing the note
6 scribes, 12 consultations, Hanah note preview on 10 cases; other results retain their original dates. The doctor column comes from published studies, not our grading.
Swipe or scroll the table to see every scribe →
| Measure | Doctor-written note (published studies)* | Hanah | Heidi | CliniScripts | Lyrebird | Freed | PatientNotes |
|---|---|---|---|---|---|---|---|
| Missed, partial or invented facts about a condition, symptom or sign (Weiner et al.), before any review. Lower is better. | |||||||
| As a share of a doctor writing it (100%). Lower is better. | |||||||
| Work to fix the note, as a share of writing it by hand. Lower is better. | |||||||
| Notes with no missing, partial or invented facts. Higher is better. |
- * Doctor-written note (published studies): Published figures, not graded by us. Errors: Weiner et al. 2020, 105 physician notes compared with covert recordings of the visit: 5.0 errors per note under their standard (any error about a condition, symptom or sign; ours counts the same way), 1.72 documented things that did not happen (the hallucination flags row), 6.1 errors in all, and only 10% of notes error-free; per-run figures are per note × 12. Time: Apathy et al. 2023, a median 8.1 minutes of active note-writing per visit across 203,728 ambulatory physicians. Editing effort is 100% by definition: the doctor writes the whole note. Not like-for-like, in the doctors' favour: our inventory counts every relevant fact and partial captures, more strictly than Weiner et al.; graded our way, six doctors' notes came out at 26 errors each. A like-for-like baseline needs clinicians to write notes for our own consultations, graded blind with the scribes; we have not done that yet.
Every measure · 22 September 2026
6 AI scribes, 12 consultations
Notes graded blind, with provisional agent audit for the new Hanah results. Each figure is the average across runs; select any cell to see every run.
Local preview — Hanah notes regenerated on 11 October 2026 with GPT-6.1 Sol, low reasoning: 50 notes across 10 consultations, five stored transcript variants each. Two consultations await new Hanah notes. WER and transcription evidence retain their original capture dates; other scribes retain their existing results. New notes were graded blind by Sonnet 5.5 medium, followed by an unblinded agent audit; provisional, without independent clinician validation. Hanah timings are estimates: prompted-note wait uses measured API generation duration + 2 seconds readiness allowance; time to a finalised note adds estimated review and correction time. Both exclude transcription finalisation and app overhead, pending a live rerun.
Swipe or scroll the table to see every scribe →
| Measure | Doctor-written note (published studies)* | Hanah1 | Heidi | CliniScripts2 | Lyrebird3 | Freed4 | PatientNotes |
|---|---|---|---|---|---|---|---|
| Wait from pressing stop to the graded note, plus review time: how long until the note can be finalised if it is reviewed straight away. | 8m 6s |
||||||
| Share of notes that captured every fact in the consultation's inventory in full with no hallucination flags, so nothing needed editing. | 10% |
||||||
| 0% means no editing required; 100% means writing the full note by hand. Fixes are weighted by the work they take: partial fact 1, missing fact 2, hallucination flag 3, and +2 when a fact is captured wrongly. | 100% |
||||||
| Estimated clinician time to proofread and fix one note (190 wpm reading, 5 s to recall and 40 wpm to type each missing or partial fact, 12 s to verify and delete each hallucination flag, including a wrong fact before it is retyped). | — |
||||||
| The single slowest note to review and fix across every case and run. | — |
||||||
| Time to write the note yourself (8.1 min, the median in Apathy et al. 2023) minus review time. | — |
||||||
| How much quicker reviewing and fixing a note is than writing it yourself (8.1 min, the median in Apathy et al. 2023). Negative means slower. | — |
||||||
| Relevant facts the note missed or captured only partly, plus every hallucination flag (a wrong detail counts twice, as in Weiner et al.), against an inventory of every fact said that relates to the patient's conditions, symptoms or signs (Weiner et al. 2020). Total per run, across 12 cases. The flat count is in Hallucination flags. | 60.0 |
||||||
| Note errors (Weiner et al. standard) per 100 notes, times 70%: in Biro et al. (2025), GPs reviewing AI drafts left each planted error uncorrected 65–75% of the time. An estimate of what a typical review would let through, across 12 cases. | 500 |
||||||
| Compared with the 5.0 errors per note Weiner et al. (2020) found in physicians' notes checked against recordings of the visit. Our test counts more strictly than theirs, so this favours the doctors. 100% means no errors; below 0% means more errors than the doctors. Across 11 cases. | 0% |
||||||
| Every clinical element the note documents that the consultation does not support, including wrong values, sides, doses and timings: Weiner et al.’s (2020) errors of commission. Paraphrase and clinical terms for what was said do not count. Average per note, across 12 cases. | 1.72 |
||||||
| Missing items, partial items and hallucination flags a clinician would need to fix, per note. | 6.06 |
||||||
| Share of each consultation's fact inventory captured by the note (partial counts as half). | — |
||||||
| Mis-heard words that change clinical meaning, rated blind, across 12 cases. | — |
||||||
| Pooled word error rate per run against the human reference. Every word counts equally, including fillers. | — |
||||||
| Median time from pressing stop to the note written to our instruction, per run. | — |
||||||
| Average network data per consultation, per run. | — |
||||||
| Physical memory footprint (macOS `footprint`, matching Activity Monitor's Memory column) sampled every minute during one continuous 50-minute stress-test consultation — separate from the 12-case suite above. Every scribe recorded this visit at the same time on one machine, so absolute values reflect shared contention between them, not isolated real-world usage. Compare scribes to each other here, not to a device's spec sheet. | — |
||||||
| CPU used per minute during the same continuous 50-minute stress-test consultation, measured as cumulative CPU-seconds consumed divided by wall-clock time (can exceed 100% when a scribe uses more than one core). Every scribe ran concurrently on one machine. | — |
- * Doctor-written note (published studies): Published figures, not graded by us. Errors: Weiner et al. 2020, 105 physician notes compared with covert recordings of the visit: 5.0 errors per note under their standard (any error about a condition, symptom or sign; ours counts the same way), 1.72 documented things that did not happen (the hallucination flags row), 6.1 errors in all, and only 10% of notes error-free; per-run figures are per note × 12. Time: Apathy et al. 2023, a median 8.1 minutes of active note-writing per visit across 203,728 ambulatory physicians. Editing effort is 100% by definition: the doctor writes the whole note. Not like-for-like, in the doctors' favour: our inventory counts every relevant fact and partial captures, more strictly than Weiner et al.; graded our way, six doctors' notes came out at 26 errors each. A like-for-like baseline needs clinicians to write notes for our own consultations, graded blind with the scribes; we have not done that yet.
- Local preview — Hanah notes regenerated on 11 October 2026 with GPT-6.1 Sol, low reasoning: 50 notes across 10 consultations, five stored transcript variants each. Two consultations await new Hanah notes. WER and transcription evidence retain their original capture dates; other scribes retain their existing results. New notes were graded blind by Sonnet 5.5 medium, followed by an unblinded agent audit; provisional, without independent clinician validation. Hanah timings are estimates: prompted-note wait uses measured API generation duration + 2 seconds readiness allowance; time to a finalised note adds estimated review and correction time. Both exclude transcription finalisation and app overhead, pending a live rerun.
- Local preview — Hanah notes regenerated on 11 October 2026 with GPT-6.1 Sol, low reasoning: 50 notes across 10 consultations, five stored transcript variants each. Two consultations await new Hanah notes. WER and transcription evidence retain their original capture dates; other scribes retain their existing results. New notes were graded blind by Sonnet 5.5 medium, followed by an unblinded agent audit; provisional, without independent clinician validation. Hanah stop-to-note and finalisation timings are unavailable pending a live rerun.
- One note is counted as a product failure: PatientNotes day5 in run 2.
- Avg RAM and Avg CPU come from a separate stress test — one continuous 50-minute myotherapy consultation, every scribe recording at the same time on one machine — not from the 12-case suite above; absolute values reflect that shared contention.
- Platform stability over repeated runs (for information, not ranked): capturing the heart failure and TIA consultations took Lyrebird 17 attempts for 10 usable runs. Seven failed on Lyrebird’s side: a note-generation error blaming “poor internet” while five other scribes generated notes on the same connection, a closed server connection, four loads not usable within 30 seconds, and a note that stopped partway through Subjective but was shown as finished. Those captures were repeated. No other scribe had an app-side failure on these consultations.
- 1 Hanah: Local preview — Hanah notes regenerated on 11 October 2026 with GPT-6.1 Sol, low reasoning: 50 notes across 10 consultations, five stored transcript variants each. Two consultations await new Hanah notes. WER and transcription evidence retain their original capture dates; other scribes retain their existing results. New notes were graded blind by Sonnet 5.5 medium, followed by an unblinded agent audit; provisional, without independent clinician validation. Hanah timings are estimates: prompted-note wait uses measured API generation duration + 2 seconds readiness allowance; time to a finalised note adds estimated review and correction time. Both exclude transcription finalisation and app overhead, pending a live rerun..
- 2 CliniScripts: Notes graded on default template.
- 3 Lyrebird: 11 cases (no French).
- 4 Freed: Notes graded on default template.
Download everything: one JSON file with every run’s scores, transcripts, notes and jury reasons, or every run’s scores as CSV.
Face-off
Pick two scribes
Each rope is pulled toward the scribe that did better on that measure. The further the ribbon moves from the centre line, the bigger the gap. The shaded band is where it would land between each scribe’s best and worst run: a wide band means inconsistent results. Small figures are the run range.
GO ONE LEVEL DEEPER
Results by clinical case
Each consultation compares the scribes on the same audio across every graded run. Open one to read each run's transcript and note, with every error marked.
English consultations
Six English consultation tests.
Language-specific consultations
Separate comparisons with capability exclusions and translation-aware interpretation.
Physician dictation
Scored against verbatim references built from the publisher’s answer keys. Spoken dictation commands are not scored.
Extended-session stress test
A 50-minute synthetic fixture with background distractors. Included in the overall results.