How speech recognition are designed

01 DESIGN STARTS WITH THE DATA

The first decision in designing a speech recognition system is not which model to use but which data to train on. A system trained on read speech from a single dialect will fail on conversational speech from another. A system trained on clean studio audio will degrade in a moving car. The training corpus determines the accent distribution, the noise conditions, the speaking styles, and the vocabulary that the system can handle — everything else is a refinement of what the data already contains.

Designing the corpus means specifying the speaker demographics, the recording conditions, the domain (medical dictation, meeting transcription, voice search), and the ratio of clean to noisy samples. Data augmentation — adding synthetic noise, speed perturbation, or SpecAugment time and frequency masking — extends the effective diversity of a fixed corpus without collecting new audio. The choice of augmentation is a design decision: aggressive masking improves robustness but can hurt clean-condition accuracy, and the designer must decide which failure mode matters more for the target use case.

02 CHOOSING THE FRONT END

The front end converts the raw waveform into the representation the model consumes. The classic choice — MFCCs or log-mel filter banks, computed on 25-millisecond windows with 10-millisecond advance — remains common because it is cheap, well understood, and compatible with decades of published models. The designer selects the number of mel bands (typically 40 to 80), whether to apply mean normalisation, and whether to append delta features.

Recent end-to-end systems bypass hand-engineered features and feed the raw waveform or the log-mel spectrogram directly to a neural network, letting the first layers learn an appropriate representation. This adds parameters and training cost but removes a set of manual choices and can discover features that MFCCs do not represent. The trade-off is between a stable, interpretable, low-compute front end and a learned, higher-compute one that may extract more signal at the price of more data and less transparency.

Frame stacking and subsampling reduce the temporal resolution before the model sees the features. A transformer operating on 10-millisecond frames processes 100 frames per second of audio; subsampling by a factor of 4 reduces this to 25, cutting compute by a factor of 16 in the attention layers. The designer must decide how much temporal resolution the model actually needs: phoneme-level distinctions may require 10-millisecond resolution, but word-level transcription can tolerate coarser frames.

03 MODEL ARCHITECTURE DECISIONS

The model architecture is the most visible design choice and the one that has changed most rapidly. A hybrid DNN-HMM system uses a neural network to estimate phoneme posterior probabilities and an HMM to enforce sequence structure; a CTC-based system uses a recurrent or convolutional network trained with a loss function that marginalises over all valid alignments; an attention-based encoder-decoder learns to align and transcribe simultaneously; a transformer replaces recurrent layers with self-attention, trading sequential computation for parallel computation and longer-range context.

Each choice has consequences beyond accuracy. Recurrent networks are harder to parallelise on training hardware, so they train slower on GPUs than transformers, even at comparable parameter counts. Transformers require more memory for attention matrices at long sequence lengths, so a system that transcribes hour-long audio must either chunk the input or use memory-efficient attention variants. The designer must match the architecture not only to the target task but to the hardware budget and the inference latency target.

Model size is a design variable, not a fixed property. A 1-billion-parameter transformer may achieve the best benchmark accuracy but cannot run on a phone; a 10-million-parameter distilled version may sacrifice 2 percentage points of word error rate but runs in real time on a mobile CPU. Quantisation — representing weights in 8-bit integers instead of 32-bit floats — reduces memory and compute by a factor of four with a sub-percent accuracy loss. The designer decides where on this continuum the system sits.

04 TRAINING OBJECTIVES AND LOSS FUNCTIONS

The training objective encodes what the system is rewarded for. Connectionist temporal classification (CTC) maximises the probability of the target label sequence summed over all possible alignments with the input frames; it requires no pre-segmented data but assumes that the output labels are conditionally independent given the input, which is a strong assumption for language. Attention-based losses learn the alignment jointly with the transcription and do not require the independence assumption, but can produce misalignments on long inputs unless regularised.

The transducer loss — used in systems such as RNN-T — combines a prediction network that models label dependencies with a transcription network that models acoustic input, and a joint network that fuses them. It does not require pre-segmented data, handles streaming naturally, and does not assume conditional independence of outputs. This makes it a common choice for real-time on-device dictation, where the system must produce text as the user speaks, not after the full utterance is available.

Multi-task objectives add auxiliary losses — language identification, voice activity detection, punctuation prediction — to the primary transcription loss. These do not directly improve word error rate but reduce the number of separate components in the pipeline and can improve robustness by sharing representations across related tasks. The designer decides which auxiliary tasks to include, how to weight them, and whether to train jointly or fine-tune sequentially.

05 EVALUATION AND METRICS

A speech recognition system is evaluated primarily by its word error rate (WER): the minimum number of word insertions, deletions, and substitutions needed to transform the hypothesis into the reference, divided by the number of reference words. WER is a necessary metric but an insufficient one. A system with a 5 percent WER on a general-domain benchmark may have a 20 percent WER on accented speech or in noisy conditions, and the distribution of errors matters as much as the mean.

Designing the evaluation suite means selecting test sets that represent the deployment conditions: the accent distribution, the noise levels, the speaking styles, the vocabulary. A system evaluated only on clean read speech will look good in benchmarks and fail in the field. A robust evaluation includes adversarial conditions — babble noise, reverberant rooms, code-switching between languages — and reports performance per condition, not just an aggregate.

Latency and throughput are evaluated alongside accuracy. Real-time factor — the ratio of processing time to audio duration — must be below 1.0 for real-time use. For streaming systems, the end-of-utterance latency — the delay between the user stopping and the final text appearing — is a user-facing metric that can matter more than WER. A system that is 1 percent more accurate but 500 milliseconds slower may feel worse to a user dictating a message.

06 DEPLOYMENT CONSTRAINTS

Where the model runs determines many earlier choices. A cloud-based system can be large, expensive, and power-hungry because it runs on server GPUs; the user pays for it in latency and privacy. An on-device system must be small, fast, and private, running on a mobile CPU or NPU with limited memory and power. A keyword-spotting system on a smart speaker runs continuously and must consume milliwatts, so it is a tiny model trained to detect a handful of wake words, not a full transcription engine.

The cloud-device split is itself a design decision. A hybrid system may run keyword spotting on-device to wake a larger cloud-based transcription engine, streaming audio only after the wake word. This preserves privacy — raw audio does not leave the device until triggered — while retaining the accuracy of a large model. The designer decides where the boundary sits, what the on-device model does, and what the cloud model does.

Streaming versus batch is a deployment constraint that ripples back to architecture. A batch transcription system can process the full utterance, use bidirectional attention, and revise earlier hypotheses after seeing later context. A streaming system must commit to each word as it is produced, use causal attention or unidirectional recurrence, and accept that earlier decisions cannot be revised. The accuracy gap between streaming and batch is a design tax on real-time systems.

07 FAILURE MODES AND ROBUSTNESS DESIGN

A well-designed system is defined not only by its accuracy but by how it fails. Confident but wrong errors — where the system produces a fluent but acoustically unjustified transcription — are the hardest for users to detect and correct. Designing against them means calibrating confidence scores, reporting uncertainty, and knowing when to abstain or ask for clarification rather than committing to a likely-wrong hypothesis.

Domain shift is a persistent design challenge. A system trained on general-domain speech may fail on medical dictation because the vocabulary differs, even if the acoustic conditions are similar. The designer addresses this with domain-specific fine-tuning, domain-adaptive language models, or domain-specific vocabulary injection — adding pronunciations for technical terms that the base lexicon does not contain.

Adversarial robustness — the system's response to deliberately manipulated audio — is an emerging design concern. Audio containing imperceptible perturbations can cause a system to transcribe text that a human listener does not hear, a vulnerability for systems that act on transcribed commands. Designing for robustness means training on adversarially perturbed examples, smoothing the decision boundary, or requiring secondary confirmation for security-relevant commands.