RESULTS · Published
The best AI scribe for dermatology
Hanah was the best AI scribe we tested on an eczema consultation. Heidi came second, within the margin of error on time, and Lyrebird was third.
The test
The consultation is a GP video appointment from the PriMock57 set: four days of an itchy, sore red rash on the chest, hands and inner elbows that was keeping the patient awake at night. Six AI scribes recorded it five times each, and every note was graded blind against the 39 clinical facts said in the consultation. We rank scribes by time to a finalised note, which is the wait after pressing stop plus the time a clinician needs to review the note and fix what is wrong. The recording, transcripts, notes and grading reasons are on the test page.
Results
| Rank | Scribe | Time to a finalised note | Note errors per note | Coverage | Hallucination flags per note | Word error rate |
|---|---|---|---|---|---|---|
| 1 | Hanah | 2 min 46 s | 4.6 | 93% | 1.0 | 14.3% |
| 2 | Heidi | 3 min 17 s+32 s | 8.0+3.4 | 84%−9% | 0.4 | 16.9%+2.5% |
| 3 | Lyrebird | 4 min 02 s+1 min 17 s | 12.0+7.4 | 86%−7% | 3.8+2.8 | 18.7%+4.4% |
| 4 | Freed | 4 min 44 s+1 min 58 s | 6.6+2.0 | 89%−4% | 0.6 | 20.3%+5.9% |
| 5 | CliniScripts | 4 min 58 s+2 min 12 s | 12.8+8.2 | 74%−18% | 1.0 | 14.9%+0.6% |
A Hanah note was ready to finalise 2 minutes 46 seconds after the consultation ended, on average. It captured 93% of the facts, with 4.6 note errors and 1.0 hallucination flags per note. Reviewing and fixing it took 71% less time than writing the note yourself.
Heidi averaged 3 minutes 17 seconds. Its quickest run, at 2 minutes 41 seconds, was faster than Hanah's slowest at 2 minutes 59 seconds, which puts the two within the margin of error on time. Hanah was quicker on average, captured more of the consultation (93% against 84%) and had fewer note errors (4.6 against 8.0). Heidi had the fewest hallucination flags on this consultation at 0.4 per note.
Lyrebird was third at 4 minutes 2 seconds and Freed fourth at 4 minutes 44 seconds. Freed had the second fewest note errors at 6.6.
Hanah also had the lowest word error rate on this consultation at 14.3%, with CliniScripts close at 14.9%.
Findings
- The rash affected the chest, hands and inner elbows. Lyrebird heard "elbows" as "eyebrows" in all five transcripts.
- The patient said the rash was not on the face. Hanah recorded this in every run, Heidi in one of five and Lyrebird in none.
- The patient could not go hiking because of the rash. Hanah and Lyrebird recorded this in every run, and Heidi never recorded it in full.
- Most of Hanah's hallucination flags in this test were routine negatives, such as "bowels normal", for questions the doctor never asked.
Figures in this article come from the ScribeStandard benchmark. See the full results, each test and how we test.