Researchers in software programming, engineering, and tutorials express concerns regarding the accuracy of transcriptions generated by OpenAI’s Whisper, highlighting potential issues.
While it’s not uncommon for discussions to revolve around creative liberties taken, it’s astonishing that accuracy becomes an issue in transcription, where one would reasonably expect the transcript to closely mirror the original audio.
As substitute researchers informed the AP, Whisper has been accused of injecting everything from racial remarks to unfounded medical treatments into transcripts, raising concerns about its credibility and reliability. However, this adoption of Whisper could prove to have devastating consequences in healthcare settings?
A University of Michigan researcher studying public conferences has found hallucinations in approximately 80% of analyzed audio transcriptions. A machine learning engineer spent more than 100 hours reviewing Whisper transcriptions and discovered that over half of them contained hallucinations. The developer reported finding hallucinations in nearly all the 26,000 transcriptions generated using Whisper.
An OpenAI representative emphasized that the company is “continuously striving to improve the reliability of its models, including reducing hallucinations,” while also noting that its usage policies prohibit applying Whisper in certain high-stakes decision-making scenarios.
“We express our gratitude to researchers for generously sharing their research results.”
