Voice Arena and Hugging Face have added two new evaluation sets to the Open ASR Leaderboard: Monsoon en-IN for Indian English and Monsoon hi-IN for Hindi, making Hindi the first Indic language — and the first Global South language — on a multilingual leaderboard tab that until now covered only European languages. Hindi is spoken by more than half a billion people. Each set ships as a public split for self-scoring and a private split withheld to limit benchmark-specific optimisation.

The four splits are speaker-disjoint and together comprise 4,888 speakers, with 12 speaker attributes recorded per clip. The Indian English public set contains 5.62 hours of audio from 1,444 speakers; the Indian English private set contains 5.58 hours from 1,405 speakers. The Hindi public set holds 1.33 hours from 468 speakers, and the Hindi private set 4.47 hours from 1,571 speakers. All clips are segmented from unscripted dual-channel spontaneous conversations so each clip carries a single speaker.

The collection was designed to vary along nine axes: geography, age, gender, vocabulary, devices, acoustic environments, speech type, speech rate, and the existence of multiple valid transcripts for the same audio. Contributors were recruited across hundreds of districts rather than recorded at length in fewer places, and they used their own handsets and connections, indoors and outdoors, rather than supplied hardware in a quiet room. Prompts pushed contributors toward opinion, disagreement, narration and recall — the registers where named entities, numbers and unrehearsed phrasing appear. Age and gender are recorded per speaker and verified.

The English sets use standard string references with the leaderboard’s normaliser collapsing most spelling variation. Because Hindi spelling variation cannot be resolved by a fixed normaliser mapping, the Hindi sets ship a lattice: for each span of the transcript, a list of spellings accepted as correct. Beyond the reported table fields, each clip also records occupation, education, marital status, income band, handset brand, current city and years in the current district.

Why Speaker Metadata Changes ASR Scoring

The core argument of the release is that a single aggregate word error rate (WER) can be right on average and wrong for a particular population. The Hugging Face announcement cites prior research showing ASR error rates are unevenly distributed: Racial Disparities in Automated Speech Recognition found commercial systems roughly twice as bad for Black speakers as for white speakers, and Quantifying Bias in Automatic Speech Recognition found further differences by gender, age and accent. None of that is visible on a traditional leaderboard, the announcement notes, because test sets record what was said and almost nothing about who said it.

Benchmarks shape what gets built. A model that scores well on the Open ASR Leaderboard gets adopted and iterated on, while capabilities the leaderboard does not measure tend not to improve. Adding Hindi — spoken by over half a billion people — creates a market signal for Indic ASR quality where none existed on this leaderboard. The speaker-attribute metadata makes it possible to ask whether a model that performs well overall fails for particular ages, regions, devices or income bands.

The Leaderboard’s Push for Trustworthy Metrics

Recent work on the Open ASR Leaderboard has focused on making the single WER number harder to game, according to the announcement: held-out private splits, benchmark-fitting analysis that quantifies how much models reproduce reference transcripts rather than transcribe from audio alone, and closing gaps in normalisers so correct predictions and variants are not penalised. The Monsoon sets extend that effort, with the public/private split design continuing the anti-contamination approach.

The new sets are deliberately small in hours and large in speakers. The ten largest contributors account for between 2.8% and 6.8% of total duration depending on the split, and more than half of all speakers appear exactly once — a mean of roughly 1.5 segments per speaker. The Indian English public set draws on 428 native districts across 30 states and union territories, and the splits span 18 to 25 device manufacturers and hundreds of current cities. The announcement states these are claims about variance rather than volume: a result on Monsoon is an average over hundreds of distinct voices.

Small Hours, Unproven Outcomes

Several open questions remain. The Hindi public split contains only 1.33 hours of audio across 468 speakers, and it is not yet clear how stable model rankings are on a set of that size, even with speaker diversity as a design goal. The announcement does not report baseline results from existing ASR models on either set, so how current leaderboard leaders perform on Indian English and Hindi is unknown from this release.

The effectiveness of the Hindi lattice approach — accepting multiple valid spellings per transcript span — against traditional normalisation has not been demonstrated with published comparisons. The benchmark-fitting and normalisation analyses that made the leaderboard’s WER harder to game were built for its existing sets; whether they carry over unchanged to the new languages is not stated. Whether leaderboard participants will use the 12 speaker attributes to publish disaggregated results, rather than a single score, also remains to be seen.

Rankings, Results and More Languages

The public splits are available now for self-scoring, and the private splits will be used for leaderboard evaluation, following the existing held-out-split methodology. The natural next milestones are the first published model rankings on Monsoon en-IN and Monsoon hi-IN, disaggregated performance breakdowns by the recorded speaker attributes, and benchmark-fitting analysis applied to the new sets. The announcement says the multiple-valid-transcripts property is “the subject of a later section” in the source material, suggesting further methodological documentation is forthcoming. Whether additional Indic and Global South languages follow depends on whether the collection pipeline is extended; nothing further is confirmed.

Where I land

I think this is a genuinely meaningful addition rather than a token gesture. The strongest design decision is not the Hindi coverage itself but the speaker metadata: documented bias research has shown aggregate WER hides population-level failures for years, and a leaderboard that records who spoke finally makes those failures measurable at scale. The private-split and lattice methodology also fits the leaderboard’s existing anti-gaming direction.

The strongest counterargument is size. A 1.33-hour public Hindi split may be too small to produce stable rankings, and if model scores swing on noise, the leaderboard could misallocate attention rather than guide it. There is also a risk that participants simply report the aggregate number and ignore the twelve attributes that carry the real value.

What would change my assessment: published evidence that rankings on the small Hindi public split are stable across random speaker resamples, and early disaggregated results showing participants actually using the speaker attributes. If those appear, this becomes a template for other under-served languages; if not, it risks being a well-designed set that the field flattens into one number anyway.

Key Questions

What exactly was added to the Open ASR Leaderboard?

Two evaluation sets: Monsoon en-IN (Indian English) and Monsoon hi-IN (Hindi). Each has a public split available for self-scoring and a private split withheld by the organisers to limit benchmark-specific optimisation. Hindi is the first Global South language on the leaderboard’s multilingual tab.

How big are the new evaluation sets?

Roughly 12 hours of English (5.62 h public, 5.58 h private) and about 5.8 hours of Hindi (1.33 h public, 4.47 h private), drawn from 4,888 speakers in total across speaker-disjoint splits. The organisers describe the design as optimised for speaker variance rather than volume.

Why do the Hindi sets use a lattice instead of standard references?

Hindi has far more orthographic variation than English, and the announcement states no normaliser can resolve it because the variants are not a fixed mapping between two conventions. The lattice lists, for each transcript span, the spellings accepted as correct.

What metadata is recorded per speaker?

Twelve attributes, including age, gender, district and state, occupation, education, marital status, income band, handset brand, current city and years in the current district. Age and gender are verified per speaker.

Can anyone run their model on these sets?

Yes, on the public splits, which are released for self-scoring. The private splits are withheld so official leaderboard scores cannot be optimised against the exact test audio.

Source: Hugging Face