Domestic AI speech recognition models showed half the error than their foreign counterparts

Domestic AI models for speech recognition when performing Russian-language tasks made half as many errors as foreign neural networks OpenAI and Nvidia, MWS AI (part of MTS Web Services) told RIA Novosti on the eve of the Digital Solutions forum.

“According to the testing results, Russian models made half as many errors as foreign solutions from OpenAI and Nvidia, and demonstrated sufficient accuracy for use in major specialized business scenarios in key sectors of the Russian economy. Testing was carried out on more than 12 thousand labeled audio recordings with a total duration of about 38 hours. The sample included speech without significant interference, as well as recordings with acoustic noise and background speech,” the service said.

The neural networks were compared by WER (Word Error Rate), an indicator that reflects the number of errors in word recognition relative to the number of words in the reference transcript.

It is noted that according to the testing results, the average WER for GigaAM from Sber was 7.4% - about 7.4 errors for every 100 words of the reference transcript. This is more than half as much as Nvidia Parakeet's 15.8%, and more than three times less than Nvidia's Nemotron at 25.5%.

However, in recordings with acoustic noise and background speech, the gap is more pronounced: GigaAM's WER was 18%, Nvidia Parakeet - 33%, and OpenAI's Whisper - 62%. In second place in terms of accuracy was another Russian development - Vosk from Alpha Cephei.