Model evaluation · Reporting

Open ASR adds Egyptian and Nigerian English. What should you test before switching APIs?

The new private splits broaden speech-recognition evaluation. Use grouped scores to shortlist models, then measure errors and correction work on your own audio.

Two private Liva AI spontaneous conversation splits: 1.34 hours Egyptian-accented English and 1.55 hours Nigerian-accented English. Audio durations, not model scores.
Published audio hours for the two Liva AI private splits, from the November 6, 2026 Open ASR Leaderboard announcement. Durations are transcribed directly, not inferred from scores. AI API Cost · original dataset illustration · Source ↗

Two new splits, not two country-level rankings

On November 6, the Open ASR Leaderboard announced two private Liva AI evaluation splits: 1.34 hours of Egyptian-accented English and 1.55 hours of Nigerian-accented English. Both use spontaneous conversations. If you buy transcription for calls or voice notes, the useful change is wider evaluation coverage, not evidence that a particular API now wins on your users' audio.

The new clips feed the Private tab's grouped comparisons. The leaderboard documents that it does not expose individual private split scores. A non-US aggregate includes several other accents; it does not tell you the Nigerian or Egyptian result on its own. The sets are English speech, not evidence for transcription quality in Egyptian Arabic or Nigerian Pidgin.

Keep the denominator visible

Word error rate, or WER, counts substitutions, insertions and deletions against reference text, divided by the number of reference words. An aggregate can hide a costly group even when the calculation is correct. In the hypothetical table, the pooled score is 5.6% because Group A contributes nine times as many words as Group B, which has 20% WER. Those labels do not refer to either new Liva split.

The private leaderboard uses macro-averages for its accent-group comparisons, not the pooled calculation shown here. For your own test, report the score and number of words for each relevant group, and label how you combine results. If a group is small, show that limitation rather than trusting a single rank.

How pooled WER can hide a difficult group

Scroll sideways to read all columns.

SampleReference wordsWord editsWER
Group A900364.0%
Group B1002020.0%
Pooled1,000565.6%
Hypothetical reference words and edit counts, not benchmark data, country-group results, or model measurements. WER (%) = edits ÷ reference words × 100. Pooled row uses summed counts, not the leaderboard’s private macro-average. Method checked November 7, 2026.Source 1 ↗

Test the transcript you need to ship

Freeze a small, consented evaluation set across the accents, recording channels and noise conditions your application actually sees. Keep the same clips and reference transcripts across providers. Choose the normalization rule before the run: the leaderboard normalizes text for WER, including case and punctuation. If those details matter for captions, add a separate formatting check rather than treating normalized WER as a finished-caption score.

For support audio, a missed booking reference can matter more than a filler word. Track errors on names, amounts or identifiers alongside WER. Keep examples of clips that require manual correction so you can tell whether a model fails on the audio itself or on a detail your downstream workflow requires.

Compare the bill after applying your acceptance rule

During an approved trial, record the provider's actual billed usage and the time spent reviewing or correcting outputs. Report accepted audio minutes, including which groups failed review, rather than comparing invoice totals alone. The new leaderboard splits can help choose candidates; only the matched workload test can tell you whether a lower rate reduces the cost of delivering transcripts you can use.

Sources & verification

Provider documentation supports the facts referenced below. Our explanations and recommendations are editorial interpretation.

  1. Hugging Face: Egyptian and Nigerian English join the Open ASR Leaderboard ↗Checked Nov 7, 2026
  2. Hugging Face: Adding private data to the Open ASR Leaderboard ↗Checked Nov 7, 2026
  3. Hugging Face and Voice Arena: Open ASR Leaderboard adds its first Global South language ↗Checked Nov 7, 2026
  4. Open ASR Leaderboard: metrics and Private data documentation ↗Checked Nov 7, 2026

Corrections: hello@aiapicost.com

Keep reading