Real-Time Speech Recognition with Whisper.rn: Amba's On-Device Architecture

Introduction

Amba is a gamified English reading and pronunciation coach inspired by the Venda word for speak. The core promise is simple: read a passage aloud, see the words light up like karaoke as you speak, and get an accuracy score in real time. Everything happens on-device.

Delivering that experience forced us to make hard trade-offs about latency, model size, and privacy. This article walks through the architecture we landed on, the problems we hit, and the numbers that matter.

Why On-Device Whisper?

The obvious alternative is a cloud speech API. OpenAI's hosted Whisper and Google's Speech-to-Text are excellent, but they fail Amba's product constraints in three ways:

Constraint	Cloud Speech API	On-Device Whisper.rn
Latency	400–900ms round trip + jitter	80–200ms per chunk, local
Privacy	Audio leaves the device	Audio never leaves the device
Cost	Pay per minute	Zero marginal cost
Offline	Not available	Fully offline

For a reading coach where a child may be practicing in a school without reliable connectivity, offline is not a nice-to-have — it is the product.

Architecture Overview

The pipeline has four stages:

1. Capture — the microphone buffers 1024-sample frames at 16kHz mono.
2. Chunking — audio is accumulated into ~1.2 second windows with a sliding overlap.
3. Transcription — Whisper.rn decodes each window, returning text with per-word timestamps.
4. Scoring — we align predicted words against the target text and compute a pronunciation confidence.

type WhisperChunk = {
  text: string;
  words: Array<{
    word: string;
    start: number; // seconds into the chunk
    end: number;
    confidence: number;
  }>;
};

The sliding window approach is what makes the experience feel live. Whisper's large models expect longer context, but the small tiny and base models handle short windows well enough for word-level alignment.

Streaming Word Timestamps

The karaoke highlight effect depends on knowing exactly when each word was spoken. Whisper.rn exposes word-level timestamps when you pass word_timestamps: true. We map those relative timestamps into the global reading timeline:

function mapChunkToTimeline(
  chunk: WhisperChunk,
  chunkStart: number,
): Array<WordEvent> {
  return chunk.words.map((w) => ({
    text: w.word,
    start: chunkStart + w.start,
    end: chunkStart + w.end,
    confidence: w.confidence,
  }));
}

A subtle but important detail: we add a 150ms lookahead offset to each start time. Without it, the highlight visibly lags the user's speech by roughly one frame, which feels sluggish on the karaoke effect.

Accuracy Scoring

Raw Whisper confidence is useful but noisy for a reading coach. We blend three signals into a single 0–100 score per word:

• Acoustic confidence from the Whisper model.
• Phonetic distance between the predicted word and the target, using a Levenshtein edit distance over phonemes.
• Timing alignment — words spoken far from their expected position are penalized.

function wordScore(
  acoustic: number,
  phonetic: number,
  timingPenalty: number,
): number {
  const raw = acoustic * 0.5 + phonetic * 0.35 + (1 - timingPenalty) * 0.15;
  return Math.max(0, Math.min(100, Math.round(raw * 100)));
}

We validated the scoring against 40 hand-labelled recordings from early testers and tuned the weights until the ranking of "good" vs "needs practice" readings matched human judgment in 92% of cases.

Performance Results

Measured on a mid-range Android device (Snapdragon 7 Gen 1):

• Model warm start: 180ms
• First word highlight: 480ms after the user starts speaking
• Chunk decode: 35–75ms per 1.2s window
• Battery: ~2% per 10 minutes of active reading

The biggest win came from pre-warming the model when the reading session screen appears, rather than when the user taps the microphone.

Privacy as a Feature

Because transcription runs entirely on-device, no audio ever leaves the device. We surface this in the product: a persistent "On-device processing" indicator and a privacy section in the App Store listing. In practice, "runs offline" has become one of the top reasons users mention in reviews.

What We Learned

• Small models beat big models when the UI needs sub-second feedback. The base model with word timestamps outperforms the small model for this use case once latency is factored in.
• Tune lookahead per device tier. Flagship devices tolerate 50ms offsets; budget devices need closer to 200ms.
• Quantize for cold start. Quantized base cut model load time by 40% with negligible scoring drift.

Conclusion

On-device Whisper gave Amba a privacy-first, offline-capable, real-time scoring loop that a cloud API simply cannot match. The karaoke effect that users love is a direct result of treating latency as the primary architectural constraint, not an afterthought.

If you are building a voice-first mobile product, start with the device constraints first and let the architecture fall out of them.