Low-Latency Hindi Voice AI in Production: Running Veena Behind OpenAI Realtime and Twilio

This is the production side of the Hindi voice AI series. Part 4 opened up Maya Research's Veena model. Part 5 was the training recipe. This post is about the operational side: how to run Veena-quality Hindi TTS inside a real AI phone agent built on OpenAI Realtime and Twilio, and keep the end-to-end turn under one second.

If you want the simpler market overview of the stack this fits into, see Part 3 of the main Voice AI guide.

The latency budget — where milliseconds live

A voice turn's end-to-end latency is measured from the moment the user stops speaking to the moment the user hears the agent's first audio byte. The thresholds callers actually feel:

• At or below 800 ms the conversation feels natural.
• Around 1,200 ms it starts to feel sluggish.
• Above 1,800 ms it feels broken.

There is a hidden cost for India users: 150 to 220 ms of network RTT per round-trip to OpenAI's US-East Realtime endpoint. Moving inference to Mumbai is a bigger latency win than any model change.

Three paths, ranked by latency and effort

Path A — Hybrid "Veena TTS override" (recommended first move)

Expected end-to-end: 900 to 1,200 ms. Effort: one to two weeks.

Twilio μ-law 8k
   │
   ▼
OpenAI Realtime (ASR + LLM + VAD, modalities=["text"])
   │  streamed text
   ▼
Sentence / clause buffer
   │
   ▼
Veena TTS (self-hosted, Mumbai region, vLLM)
   │  24 kHz PCM
   ▼
μ-law 8k → Twilio

This path reuses an existing OpenAI Realtime integration. Only one small file of new code is needed: a Veena TTS provider that implements the existing TTS interface. You still pay the OpenAI → US RTT, but the Hindi voice-quality complaint goes away immediately.

Path B — India-local STT + OpenAI LLM + Veena TTS

Expected end-to-end: 700 to 950 ms. Effort: three to four weeks.

Twilio μ-law 8k
   │
   ▼
Silero VAD (Node, local, free)
   │
   ▼
Sarvam Saarika Flash streaming STT (Mumbai)
   │  partial text
   ▼
GPT-4o-mini or Gemini 2.5 Flash (streaming chat completions)
   │  streamed tokens
   ▼
Clause buffer
   │
   ▼
Veena TTS (self-hosted, Mumbai)
   │  24 kHz PCM → μ-law 8k → Twilio

Everything on the audio path is Indian-hosted, which cuts 150 to 200 ms of OpenAI US RTT. Sarvam Saarika is specifically Hindi and Indic-tuned, so transcription is better than Whisper or Realtime for Hindi. The LLM stays on a frontier vendor — cheap, fast, no retraining.

Path C — Fully cascaded, open-weights only

Expected end-to-end: 800 to 1,100 ms. Effort: six to ten weeks. Not recommended as a first move.

Worth it only if you want to drop OpenAI entirely for cost or sovereignty reasons, or if you plan to fine-tune the LLM on your own Hindi call data.

Latency levers that actually matter

Inside Path A, these are the levers that shave 200 to 400 ms end-to-end if done right:

• Co-locate Veena with the WebSocket server in Mumbai (ap-south-1, E2E Networks, Yotta or Oracle Mumbai). Saves 100 to 200 ms of RTT per TTS chunk.
• Serve Veena on vLLM with continuous batching, not plain `transformers.generate`. Roughly 2–3× throughput and ~40 percent lower time-to-first-audio.
• Stream TTS on first clause, not first sentence. Split on `, । : —` with a minimum word count of 6. Saves 150 to 300 ms on long responses.
• Pre-warm Veena. First inference after cold start is 2 to 5 seconds. Keep at least one GPU replica warm during business hours. Scale to zero only overnight.
• WebSocket between the call server and Veena, not HTTP. Avoids a TCP+TLS handshake per call. Saves 50 to 150 ms per turn.
• Decode SNAC in frames of 7 tokens (one SNAC frame is about 86 ms of audio) and emit μ-law immediately. Do not wait for a full-sentence buffer at the TTS server.
• Pre-generate the greeting as a static audio file. Do not re-synthesise "नमस्ते! मैं AI सहायक हूं…" on every call. Saves 300 to 500 ms on connect.
• Reduce the system prompt size. Realtime pre-processes the full system prompt into audio-aware context, so every token costs.

Barge-in — the detail everyone gets wrong

Barge-in has to be truly immediate. When VAD fires on "user is speaking" the call server has to do four things, atomically:

• Clear Twilio's 20 ms frame buffer with a `clear` event plus a `mark`.
• Cancel the TTS stream in flight.
• Send `response.cancel` to OpenAI Realtime.
• Send `conversation.item.truncate` so the model's own transcript matches what the caller actually heard.

If any one of these is missing, callers hear 100 to 300 ms of agent voice after they start talking. That single bug is the most common reason a technically fine voice agent still feels off.

A matching detail: add an echo grace window of about 450 ms at the start of agent speech during which VAD is ignored. Phone lines echo outbound TTS back into the inbound track. Without the grace window, the agent interrupts itself.

GPU and hosting choices for Veena in India

• GPU: an L4 with 24 GB or an A10G is enough for Veena at bf16. A100 is overkill unless you batch many concurrent calls per GPU. H100 only pays off above about 8 concurrent calls per card.
• Region: Mumbai (ap-south-1 or Indian providers like E2E, Yotta, Oracle Mumbai). This saves more latency than any model change.
• Cost: an always-on L4 in Mumbai is roughly $250–600 per month and serves 4 to 8 concurrent calls comfortably.
• Reliability: run two replicas across two availability zones with health checks. Fall back to a commercial Hindi TTS (for example ElevenLabs Flash v2.5) if the primary degrades. Veena has no vendor SLA by itself.

Why cascaded still wins over pure speech-to-speech

Speech-to-speech models are great demos. In production for Indian voice AI, cascaded (or hybrid cascaded) still wins. The structural reasons:

• Cost: S2S is about 10× cascaded at scale because audio context accumulates every turn and the model re-charges for the full history each turn.
• Telephony: PSTN is 8 kHz narrowband. S2S's wideband acoustic advantage largely disappears over phone networks — which is exactly where production Indian voice AI lives.
• Multilingual coverage: Realtime APIs are English-first. Cascaded lets you pick a Hindi-strong STT (Sarvam Saarika) and a Hindi-strong TTS (Veena) independently.
• Observability: compliance, moderation, QA and the training-data flywheel all need text at every boundary. Cascaded gives this for free.
• LLM swap cadence: frontier LLMs ship every 2 to 4 months. S2S models retrain every 6 to 12 months. Cascaded lets you ride the fast curve.

Published surveys (a16z Voice AI 2025, Deepgram State of Voice AI 2025, LiveKit, Inworld, Gradium, Soniox, RTCLeague) are directionally consistent: cascaded is the default for enterprise voice agents, with hybrid as the common choice for language-sensitive products. In India specifically, cascaded or hybrid share is 80 to 90 percent.

Rollout plan

Do not attempt Path C (fully open stack) until Paths A and B are shipped and measured. The LLM alone (Sarvam-M, Krutrim, Nemotron) is a three-month project before it matches GPT-4o-mini on Hindi reasoning and tool-calling.

Risks worth calling out

• Autoregressive TTS has non-deterministic latency per sentence. A 10-token output can take 150 ms, a 50-token output 600 ms. Track p95 latency, not just the median.
• Veena is open-weight with no vendor SLA. Plan for redundancy and a commercial-TTS fallback.
• OpenAI pricing and model deprecation have shifted several times. The architecture should allow dropping Realtime for Sarvam Saarika plus GPT-4o-mini chat completions if needed.
• Every real Hindi call is training data for a future fine-tuned Veena variant. Design the logging schema with retraining in mind from day one.

The short version

The fastest path to good Hindi voice AI in production: keep OpenAI Realtime for ASR, LLM and VAD, override the TTS with self-hosted Veena on vLLM in Mumbai, decode SNAC in sliding windows, split on clauses, pre-warm the GPU, and fix barge-in properly. That gets you a sub-1,000 ms Hindi phone agent on existing infrastructure, with a clear upgrade path to a fully Indian-hosted stack once you have call data to measure against.