How to Train a Hindi TTS Model Slightly Better Than Veena

This is a follow-up to Part 4, which reverse-engineered Veena. Veena is a Llama-3.2-3B base predicting SNAC codec tokens, with a vocabulary extension of 28,704 tokens (7 codebooks × 4,096 SNAC entries plus control tokens). To produce a model that is slightly better than Veena on Hindi you do not need a new architecture. You need to beat it on data, codec and alignment.

The levers below are listed in order of impact. The biggest wins come from the first three.

1. Data (biggest lever by far)

This is roughly 70 percent of the win.

Volume target

Aim for 2,000 to 10,000 hours of Hindi speech paired with clean transcripts. Veena was trained on a few thousand hours of Indic data. Match that at a minimum. Two to three times that volume for Hindi specifically is a safer bet.

Sources

• IndicVoices (AI4Bharat, around 16,000 hours across Indic languages, with a large Hindi slice).
• Shrutilipi (AI4Bharat, around 6,400 hours of aligned Hindi).
• Kathbath and Vaani (IISc, Google), IndicTTS (IIT Madras).
• Hindi YouTube: audiobooks, lectures, podcasts and news. Force-align with WhisperX or MMS, then filter by WER.
• Commercial: ULCA, SPRING Lab, Reverie — if budget allows.
• Hindi film dubbing tracks are great for emotion diversity, but licensing is messy.

Quality filters you must apply

• Signal: SNR above 20 dB, no music or overlapped speech (VAD plus a music detector).
• Alignment: ASR round-trip WER under 10 percent after forced alignment.
• Speaker diversity: at least 500 unique speakers. De-duplicate near-duplicates by speaker embedding.
• Prosody balance: do not let news-anchor style dominate. Oversample conversational, emotional, child and elderly voices.

Code-mixing is critical for Hindi

Real-world Hindi is heavily code-mixed with English. Include Hinglish data explicitly and keep both Devanagari and romanised transcripts. Veena handles this reasonably well. A carefully balanced dataset mix is where you can clearly beat it.

Emotion and style labels

Weakly label around 20 percent of the corpus with emotion tags (neutral, happy, sad, angry, questioning) using an audio emotion classifier. This teaches conditional generation of tokens like `<laugh>` and the emotion specials from Part 4.

2. Codec choice (second biggest lever)

SNAC — Veena's choice — is good but not the ceiling in late 2025 or 2026.

Recommended: switch to Mimi or XCodec2. A lower token rate means the 3 B Llama sees more seconds of audio per training example and generates faster at inference. This single change is often audibly better than SNAC at the same model size.

Trade-off: you lose compatibility with Veena's inference stack and have to redo the vocabulary-expansion math. For example, Mimi has 8 codebooks of 2,048 entries, so about 16,384 added tokens.

3. Base LM

Keep Llama-3.2-3B. It is the sweet spot for voice: fast enough for streaming, strong enough for prosody.

Alternatives for a small lift without going bigger: Qwen2.5-3B (stronger Hindi in its text pretraining) or Gemma-2-2B (very efficient). Going to 7–8 B gives clearly better prosody but kills real-time inference on single consumer GPUs. Not worth it for "slightly better".

4. Tokenizer and text front-end

Veena uses the stock Llama tokenizer, which fragments Devanagari badly, often many bytes per character. Wins available here:

• Extend the text tokenizer with Hindi BPE merges learned on a large Hindi corpus such as IndicCorp v2. Sequences shrink two to three times, which is faster and better for long-range prosody.
• Normalise numbers, dates, English loan-words and abbreviations before tokenisation. A good Hindi text normaliser (indic-nlp-library plus custom rules for currency, dates and phone numbers) gives an audible lift.
• Decide the romanised-Hindi policy up front. Either normalise all input to Devanagari pre-tokenisation, or train on both scripts explicitly.

5. Training recipe

A three-stage, standard LM-TTS recipe:

• Audio continued-pretraining. Take Llama-3.2-3B, extend the vocabulary, train on (text, audio_token) pairs with next-token prediction. About one epoch over all audio data. Most of the compute lives here.
• Supervised fine-tuning on a curated subset with speaker ids, emotion tags and high-quality studio recordings. This teaches the control tokens.
• Preference tuning (this is where you actually beat Veena): DPO or KTO on human preference pairs. Generate two completions per prompt and have Hindi native speakers rank them for naturalness and pronunciation. Five thousand to twenty thousand preference pairs give an audible lift over pure SFT. Veena, from public information, does not do this step.

Compute ballpark: continued-pretraining a 3 B model on about 5,000 hours of audio tokens is roughly 2,000 to 4,000 H100-hours, which is about $5,000 to $12,000 on rented GPUs. SFT and DPO are cheap afterwards, around 200 H100-hours together.

6. Evaluation (so you can prove you are better)

Build the evaluation harness before training, not after.

• Intelligibility: Hindi WER using a strong Hindi ASR (IndicWav2Vec, Whisper-large-v3) back-transcribing generated audio against the prompt text. Lower is better.
• Speaker similarity: cosine similarity on ECAPA-TDNN embeddings versus a reference voice.
• Naturalness: comparative MOS with Hindi native raters, A/B versus Veena on 100 to 200 held-out prompts covering news, conversation, emotion, code-mixing, numbers and dates, and long sentences.
• Prosody: F0 contour correlation versus natural speech, and pause-placement accuracy at punctuation.
• Hindi-specific stress tests: retroflex consonants (ट/ठ/ड/ढ), nasalisation (ं/ँ), schwa deletion (कमल as "kamal" not "kamala"), geminate consonants and Hinglish code-switch boundaries.

Minimum viable project plan

If you only make one change, make this one: switch SNAC → Mimi and add DPO on human preferences. Everything else is table stakes.

Where this fits in a real product

A better Hindi voice model is only useful inside a voice stack that can run it at low latency over normal phone lines. Part 6 covers that side — serving Veena (or your Veena successor) behind OpenAI Realtime and Twilio, with an end-to-end turn under one second.