METHODS / VERSIONED, NOT MYSTERIOUS

Evaluation methods and limitations

Local preview — Hanah notes regenerated on 11 October 2026 with GPT-6.1 Sol, low reasoning: 50 notes across 10 consultations, five stored transcript variants each. Two consultations await new Hanah notes. WER and transcription evidence retain their original capture dates; other scribes retain their existing results. New notes were graded blind by Sonnet 5.5 medium, followed by an unblinded agent audit; provisional, without independent clinician validation. Hanah timings are estimates: prompted-note wait uses measured API generation duration + 2 seconds readiness allowance; time to a finalised note adds estimated review and correction time. Both exclude transcription finalisation and app overhead, pending a live rerun.

Every number needs a definition. Every comparison needs the same inputs. Here is what this edition measures, and what it cannot establish.

User experience

We don't test user experience. From what we've seen AI scribes are much the same from one product to the next: press record, hold the consultation, get a note. What sets them apart is what they are bought for, so that is what we measure, objectively: how accurately each scribe hears the consultation, how accurate and complete its note is, and how long it takes to get to a note you can finalise.

The actual product experience

We replay the same audio through visible Chromium browsers controlled by Playwright. Each capture has an isolated copy of a signed-in profile. Audio is prepared with the shared harness as mono PCM at 44.1 kHz, with one second of leading silence and ten seconds of trailing silence. This is microphone input to the product; it does not prescribe the codec the product sends to its servers.

The harness records visible transcript changes, final transcripts, generated notes, UI events and whole-session network totals. It does not record video. Products that require a language selection are configured before recording; unsupported combinations are excluded.

Multiple runs per scenario

Due to variations between runs on the same scenario data we ran each scenario multiple times. Every scribe ran each consultation it took part in five times. Every figure is the average across runs, and every cell opens a chart with one dot per run.

Lyrebird does not support the French case and is scored on the cases it supports. One capture failed inside the product (PatientNotes); that run is averaged over the cases that completed.

Final transcripts

Word error rate (WER) is substitutions plus deletions plus insertions, divided by reference words, pooled across every case in a run. Lower is better. We wait for the product's finalisation controls and transcript stability before capturing the text.

Equivalent numbers such as "ten" and "10" count as the same word, and the case pages don't highlight them. Contractions remain literal differences. Hindi uses a documented script-equivalence policy.

WER treats every word alike, so we also count significant transcript errors. Each transcript is compared with the reference, differences that are only fillers or function words are dropped, and a blind rater judges every remaining difference: could it change diagnosis, treatment, follow-up, safety or the record? Wrong drugs, doses, numbers, sides and flipped negations usually are. Spelling, paraphrase and dictation commands written out are not. An identical difference gets the same rating wherever it appears.

Corrected Source Material

Where a source transcript didn't match what was said in the recording, we corrected it, and every scribe is scored against the same corrected text. Spoken commands such as "period" and "next paragraph" are removed before scoring.

Grading the notes blind

Every note was graded by a jury of AI reviewers (Claude subagents) at least 5 times. Notes for the same consultation were shuffled into packets labelled A to G, with identical context for every note.

Each consultation has an inventory of every fact said that relates to the patient's medical conditions, symptoms or signs, the standard Weiner et al. (2020) used when they compared physicians' notes with recordings of the visit. Each fact is one clinical point: symptoms and their details, pertinent negatives, history, medications, examination findings said aloud, the assessment and the plan. That is 12 to 45 facts per consultation, 358 in all. Diagnoses count only when the clinician said them; a scribe should not diagnose. Immaterial circumstance, such as where a medicine was bought, is left out; clinically meaningful detail, such as the colour of vomit or the side of pain, stays. The inventories were drafted by AI from the reference transcript alone, without seeing any note, and every fact quotes the line it comes from.

The jury marks each fact full, partial or missing. A separate blind pass then flags every clinical element the note documents that the consultation does not support: a symptom, negative, history item, medication, dose, finding, diagnosis or plan item that was never said, or was said differently (wrong value, side, dose, timing or person, or a possibility written as fact). These are Weiner et al.’s errors of commission, and each is a hallucination flag, with its reasoning. Paraphrase and clinical terms for what was said do not count. Every missed or partial fact is a note error, and so is every hallucination flag. A detail written wrongly counts twice, as in Weiner et al.: once as the correct fact missing, once as the flag. There is no separate severity rating: by construction, every fact on the inventory matters.

Until 2026-09-25 notes were graded against a checklist of 13 to 16 key facts per consultation, with a separate rater judging each error's significance. The checklist capped how many omissions a note could have, so we replaced both with the full inventory. Every scribe's error count rose two to three times; the order barely moved. Significant transcript errors still use a significance rating. These are AI reviews. They are not independent clinician validation and do not establish clinical safety.

Review time

Review time estimates how long a clinician would take to make each note safe to finalise:

• reading the whole note at 190 words a minute;
• 5 seconds to recall each missing or partial fact, then typing it at 40 words a minute (half of it for a partial fact);
• 12 seconds to check and delete each hallucination flag. A flag that gets an inventory fact wrong also costs the 12 seconds, on top of retyping the fact.

Formatting fixes are not counted, because every scribe follows formatting instructions well. Waiting for the note is not counted either, because clinicians usually review notes later in the day. Time to a finalised note, below, counts the wait.

Time saved

Time saved compares review time with writing the same note yourself. Writing it yourself is taken as 8.1 minutes, the median active note-writing time per visit across 203,728 US ambulatory physicians in Apathy et al. (2023). Published studies put it at 6–10 minutes depending on specialty and workflow; we use the median, the lower of the two central figures.

Time saved is 8.1 minutes minus the note's review time. It is the same baseline for every scribe, so a scribe saves more time only by needing less review.

Faster than writing it yourself

Faster than writing it yourself puts time saved as a share of the time to write the note yourself: 1 minus review time divided by writing time. 60% means reviewing and fixing the note takes 40% as long as writing it. A negative figure would mean the note takes longer to fix than to write. Each note is scored on its own, then averaged across a run.

Time to a finalised note

Time to a finalised note is the wait from pressing stop until the graded note is ready, plus its review time. It is the case where a clinician reviews each note straight after the consultation. If the gap between patients is longer than this, the note is finalised before the next patient comes in. For scribes graded on their default note, the wait is timed to the default note. Like stop to prompted note, it uses production runs only.

Notes ready to finalise as written

A note is ready to finalise as written when the jury marked every inventory fact as fully captured and found no hallucination flags, so nothing needs editing. Formatting is not counted, as in review time. Each scribe's figure is the share of its notes that met this.

Editing effort

Editing effort shows how much of the note the clinician still has to fix, whatever its length. 0% means no editing required. 100% means writing the full note by hand.

Each fix is weighted by the work it takes:

• partial fact, 1: it only needs amending;
• missing fact, 2: it has to be recalled and written;
• hallucination flag, 3: it has to be spotted, checked and deleted;
• fact captured wrongly, 2 more: the wrong version has to be spotted and checked before it is fixed.

A note's weights are added up and divided by the score for writing it yourself, which is 2 for every fact on the consultation's inventory. Hallucinations can push a note above 100%.

Errors reaching the record

This is an estimate of how many note errors (Weiner et al. standard) would still be in the record after a clinician reviews the note. It multiplies each scribe's note errors per note by 70%.

The 70% comes from Biro et al. (2025). Twenty primary-care GPs reviewed AI-drafted replies to patient messages, four of which had an error planted in them. Each error went uncorrected by 13 to 15 of the 20 GPs, so 65–75% of reviewers missed it. We use the middle, 14 of 20.

It is an estimate, not a measurement. Biro et al. studied replies to patient messages, not consultation notes, and review habits vary. The same rate is applied to every scribe, so it does not change their order or the gaps between them; it turns errors in the draft into what a typical review would let through. The figure is pooled over every graded note; the range is the same figure for each run.

The doctor-written note

The doctor column shows a doctor writing the note by hand, from published studies rather than our own grading.

• Errors: Weiner et al. (2020) compared 105 physicians' notes with covert recordings of standardised-patient visits. They found 6.1 errors per note, 5.0 of which met their standard (they called these clinically significant); 72% were omissions, and 1.7 per note were things documented that did not happen. 90% of notes had at least one error. Our inventories use the same standard for which facts count. Per-run figures in the table are per note × 11 consultations.
• Time: Apathy et al. (2023) measured a median 8.1 minutes of active note-writing per visit across 203,728 US ambulatory physicians (mean 9.1). The same 8.1 minutes is the writing-it-yourself baseline for time saved, and the headline table shows each scribe's time to a finalised note as a share of it.
• Ready to finalise: Weiner et al. found at least one error in 90% of notes, so 10% of doctor-written notes would pass as written. Editing effort is 100% by definition: the doctor writes the whole note.

These figures are not like-for-like, and the difference favours the doctors. The scribes are graded against our fact inventories, one clinical point per fact and partial captures counted as errors; Weiner et al. counted coarser elements in real clinic notes. Our test is the stricter of the two: graded our way, six doctors' notes (below) came out at 26 errors each, against Weiner et al.’s 5.0. One difference is deliberate: Weiner et al. left out documented examination findings, because a recording cannot show whether a doctor examined the patient. That reason does not apply to a scribe, which only knows what was said, so an examination finding in a scribe's note that was never said is counted as a hallucination flag. The honest comparison would have independent clinicians write notes for our twelve consultations and grade them blind alongside the scribes. We have not done that yet; until we do, read the doctor column as a published reference point, not a result of this benchmark.

"Fewer errors than doctor-written notes" is 100% minus each scribe's errors per note as a share of Weiner et al.’s 5.0, pooled over every graded note; 100% means none. The range is the same figure for each run.

This compares the scribe's note before review with the doctor's note as filed, which nobody reviews. Review is not a reliable safety net for the scribe's errors either: in Biro et al. (2025), 20 primary-care physicians reviewed AI-drafted replies to patient messages, and each seeded error went uncorrected by 13 to 15 of them. 35–45% of the drafts with errors were sent unedited.

Our own check. Eight of the twelve consultations come from PriMock57, which publishes the note the consulting doctor wrote at the time (seven Babylon GPs, 10-minute mock video consultations, notes typed live and released unreviewed). For the first six we mixed each doctor's note into that case's scribe notes and graded it blind against the same inventory. The doctors' notes averaged 26 errors each, mostly omissions, and none was error-free. On those same six consultations the scribes averaged 4.3 (Hanah) to 19.7 (PatientNotes). It is six brief notes, so we do not publish it as a doctor error rate; it is the like-for-like check that every scribe's notes had fewer errors than the doctors' own on the same consultations.

Face-off

The face-off compares two scribes on every measure. Under the score, four takeaways each use a single measure, so no gap is counted twice: word error rate (transcription), hallucination flags, note errors (Weiner et al. standard, note accuracy) and time to a completed note. Each gap is how much less of it the better scribe has: (worse − better) ÷ worse.

"Weighted in a scribe's favour" is the plain average of gaps, equally weighted, with a tie counted as 0%. It is an approximate summary: the choice of four measures and equal weights is based on summarised outcomes, and a large gap counts for at most 100%.

The face-off opens on the two leading scribes by this weighting: every scribe is compared with every other, the two with the most head-to-head wins are shown, and the summed margin breaks a tie.

Notes and timing

Every scribe that takes a note instruction was given this one, through its own UI:

Write SOAP notes using the headings Subjective, Objective, Assessment, and Plan. Use bullet points in every section and concise clinical shorthand.

Freed and CliniScripts don't take an instruction this way, so they are graded on their default note. "Stop to prompted note" still times the prompted note where one exists.

Stop to prompted note is the median, per run, of the time from pressing stop to the requested note being ready. Completion gates include short UI stability waits, and captures share a machine and internet connection, so this is not an isolated server-latency test.

Bandwidth bill

Chromium NetLog totals include app loading, recording, finalisation, notes and browser shutdown, including WebSocket transport. Socket and UDP byte events are counted once; decrypted HTTP and WebSocket payloads are not added again. This is not ISP-billed traffic: packet headers and TCP retransmissions are not included. Missing or invalid logs are unavailable, never zero.

Reproducibility

Download everything as one JSON file: every run's scores, transcript, error marks, note, fact verdicts and jury reasoning for all 12 consultations. The scores alone are also a CSV. Source recordings can be played or downloaded from each consultation page. Authentication profiles and raw network logs are not published.