RESULTS · Published
The best AI scribe for mental health
Hanah was the best AI scribe we tested on a low mood consultation. Heidi and Lyrebird were close behind, and all three are within the margin of error on time to a finalised note. Hanah was quickest on average and had the fewest note errors.
The test
The consultation is a GP video appointment from the PriMock57 set: a patient with two months of low energy, poor sleep and reduced appetite since starting a new job, with a family history of suicide. Six AI scribes recorded it five times each, and every note was graded blind against the 26 clinical facts said in the consultation. We rank scribes by time to a finalised note, which is the wait after pressing stop plus the time a clinician needs to review the note and fix what is wrong. The recording, transcripts, notes and grading reasons are on the test page.
Results
| Rank | Scribe | Time to a finalised note | Note errors per note | Coverage | Hallucination flags per note | Word error rate |
|---|---|---|---|---|---|---|
| 1 | Hanah | 2 min 26 s | 4.2 | 92% | 1.4 | 13.7% |
| 2 | Heidi | 2 min 45 s+19 s | 6.8+2.6 | 80%−12% | 0.8 | 13.2% |
| 3 | Lyrebird | 3 min 02 s+36 s | 8.0+3.8 | 84%−8% | 2.8+1.4 | 17.8%+4.1% |
| 4 | CliniScripts | 4 min 40 s+2 min 15 s | 13.2+9.0 | 65%−27% | 3.2+1.8 | 13.3% |
A Hanah note was ready to finalise 2 minutes 26 seconds after the consultation ended, on average. It captured 92% of the facts, with 4.2 note errors per note. Reviewing and fixing it took 77% less time than writing the note yourself.
Heidi averaged 2 minutes 45 seconds and Lyrebird 3 minutes 2 seconds. Hanah's slowest run took 2 minutes 52 seconds, while Heidi's quickest took 2 minutes 27 seconds and Lyrebird's 2 minutes 26 seconds, which is why we treat the three as close. Heidi had 6.8 note errors per note and Lyrebird 8.0. Heidi had the fewest hallucination flags at 0.8 per note, with Hanah at 1.4.
Heidi also had the lowest word error rate on this consultation at 13.2%, with CliniScripts at 13.3% and Hanah at 13.7%. Lyrebird produced its note fastest after pressing stop, though its longer review put it third overall.
Findings
- The patient said they had no previous mental health diagnoses. Not one of the 30 notes from the six scribes recorded this in full.
- The patient said there was no other family history of medical or psychiatric conditions, including depression. Hanah recorded this in every run, Heidi in two of five, Lyrebird in one and CliniScripts in none.
- After some back and forth, the patient said there had been no change in their morning erections. Hanah recorded this correctly in every run. Heidi and Lyrebird did in three of five.
Figures in this article come from the ScribeStandard benchmark. See the full results, each test and how we test.