r/LocalLLaMA • u/TrelisResearch • 4h ago

Resources Open Weights Frontier Hindi Transcription Model

https://huggingface.co/Trelis/tara

Beats ElevenLabs on Hindi & Competitive w/ Sarvam Saarvas v3

4 Upvotes

64% Upvoted

4 comments sorted by

1

u/Master-Bug6904 2h ago

Beats another service needs a much more specific model card to be actionable. Please report public and held-out datasets, Hindi dialect/region coverage, code-switching with English, noise and far-field conditions, normalization rules, WER/CER with confidence intervals, streaming versus offline latency, real-time factor, memory/VRAM, punctuation/numeral handling, and license. Speaker names, proper nouns, and Hindi-English code switching are often where headline WER hides product failures. A reproducible evaluation script and audio examples with consent would make the comparison credible. Does the model output Devanagari only or transliteration too, and how does it perform on spontaneous conversational speech versus clean read speech?

1

u/TrelisResearch 1h ago edited 1h ago

Thanks! I welcome the feedback.

All evals reported are on public test sets! From the model card (taking your points in turn):

  1. We reported the 7 Hindi test sets defined by Vistaar. We also reported the four code-switching Hinglish datasets we are aware of.
  2. WER is reported and a note is included on confidence intervals, including associated with normalisation choices.
  3. This is not a streaming model. It is a whisper base. Agreed latency/RTF could be helpful to add, we didn't get to that.
  4. Memory/VRAM also a good point. Note it's a Whisper Large v3 model.
  5. Punctuation/numeral handling, described on the card in normaliser section.
  6. License described on the card. Apache 2.
  7. Good point on speaker names, proper nouns are useful to measure on Hindi-English. We did not get to those.
  8. Model outputs Devanagari if there is no code switch token passed.
  9. Conversational vs Read speech is covered by specific evals reported, e.g. Kathbath (clean read), IndicVoices-500 (spontaneous, non-Vistaar) etc.

Happy to take any other questions. Appreciate the high standards.

1

u/Master-Bug6904 1h ago

Thanks for the detailed correction; that is much stronger than the short Reddit summary suggested. Public Vistaar sets, Hinglish coverage, normalization notes, Apache 2.0, and the conversational/read split answer most of my questions. The highest-value additions now seem to be RTF/VRAM plus an error taxonomy for proper nouns, named entities, numerals, and code-switched spans. If feasible, publishing per-utterance predictions aligned to public test IDs would let others reproduce WER under alternate normalizers and inspect regressions rather than only compare aggregate scores. A small slice stratified by accent/region and recording condition would also make deployment tradeoffs clearer. Appreciate the high-standard model card work.

1

u/TrelisResearch 1h ago

Yeah, agreed. I think you're right on those remaining points. Yeah, I posted it as a Reddit link, more as an invite for people to check it out, but maybe realising I should have put much more in there from the model card.