RESULTS · Published
The best AI scribe for urology
Hanah was the best AI scribe we tested on a urinary symptoms consultation. Lyrebird and Heidi were close to each other in second and third.
The test
The consultation is a GP video appointment from the PriMock57 set: six days of painful urination, foul-smelling urine, blood in the urine and central lower abdominal pain, in a patient with an antibiotic allergy. Six AI scribes recorded it five times each, and every note was graded blind against the 42 clinical facts said in the consultation. We rank scribes by time to a finalised note, which is the wait after pressing stop plus the time a clinician needs to review the note and fix what is wrong. The recording, transcripts, notes and grading reasons are on the test page.
Results
| Rank | Scribe | Time to a finalised note | Note errors per note | Coverage | Hallucination flags per note | Word error rate |
|---|---|---|---|---|---|---|
| 1 | Hanah | 2 min 25 s | 5.0 | 93% | 1.2 | 8.8% |
| 2 | Lyrebird | 3 min 34 s+1 min 09 s | 10.4+5.4 | 88%−5% | 3.2+2.0 | 10.6%+1.8% |
| 3 | Heidi | 3 min 39 s+1 min 14 s | 11.4+6.4 | 78%−16% | 0.8 | 10.4%+1.6% |
| 4 | CliniScripts | 4 min 17 s+1 min 52 s | 12.2+7.2 | 75%−18% | 0.6 | 8.8% |
A Hanah note was ready to finalise 2 minutes 25 seconds after the consultation ended, on average. It captured 93% of the facts, with 5.0 note errors per note. Reviewing and fixing it took 74% less time than writing the note yourself.
Lyrebird averaged 3 minutes 34 seconds and Heidi 3 minutes 39 seconds. Heidi's quickest run, at 2 minutes 52 seconds, was faster than Hanah's slowest at 2 minutes 57 seconds, which puts it within the margin of error on time. Hanah was quicker on average and had less than half the note errors of either, 5.0 against 10.4 for Lyrebird and 11.4 for Heidi. Lyrebird had the second highest coverage at 88%.
Hanah had the lowest word error rate on this consultation at 8.8%, level with CliniScripts. Hanah's transcripts had no mis-heard words that change clinical meaning in any run.
Findings
- The doctor said this infection can be treated without testing the urine. Not one of the 30 notes from the six scribes recorded it.
- The patient takes mebeverine. Every other scribe mis-heard the drug name in at least one run, and one heard it as "heparin" and listed heparin, a blood thinner, among the patient's medications in every note.
- The prescription was sent to the patient's nominated pharmacy, to start as soon as possible. Hanah recorded this in full in every run, and Heidi recorded part of it in every run.
Figures in this article come from the ScribeStandard benchmark. See the full results, each test and how we test.