How speech recognition work

01 CAPTURING THE ACOUSTIC SIGNAL

Speech recognition begins where hearing begins: with a pressure wave. When a person speaks, their vocal tract shapes fluctuations in air pressure that propagate outward. A microphone converts those fluctuations into a continuous electrical signal, sampled tens of thousands of times per second. At 16 kHz — a common rate for telephony-grade speech — the system records 16,000 amplitude values per second, and at 44.1 kHz, used for consumer audio, it records 44,100.

The raw waveform is dense and noisy. It contains not only the linguistic content of interest but also room reverberation, microphone colouration, breath sounds, and background noise. The first task of any speech recognition system is to discard what does not carry linguistic information while preserving what does. The biological system does this in the cochlea, where the basilar membrane performs a frequency decomposition; the engineering system does it with a digital filter bank that mimics that decomposition.

02 EXTRACTING SPECTRAL FEATURES

From the raw waveform the system computes a sequence of feature vectors, each summarising a short window of audio — typically 25 milliseconds of sound, advanced every 10 milliseconds. The dominant representation for decades was the mel-frequency cepstral coefficient (MFCC) vector, which applies a perceptually motivated frequency warp, takes a logarithm of the energy in each band, and applies a discrete cosine transform. The result is a compact set of roughly 13 to 40 numbers per frame that capture the spectral shape most relevant to speech.

MFCCs work because human hearing is roughly logarithmic in frequency. Two tones separated by an octave at low frequencies are perceived as similarly spaced as two tones separated by an octave at high frequencies, even though the absolute frequency difference is much larger in the latter case. The mel scale encodes this perceptual geometry, and the cepstral transform decorrelates the log-spectral bands so that the downstream model can treat them as approximately independent. Later systems replaced MFCCs with log-mel filter banks or even raw waveform learning, but the principle persisted: compress the waveform into a frame-by-frame representation that the decoder can operate on.

Delta and acceleration coefficients — the first and second temporal differences of each feature — are appended to capture how the spectrum is changing, not just its instantaneous shape. A single frame in isolation is ambiguous; the trajectory of features across frames carries the phonetic information that distinguishes a /b/ from a /p/ or a vowel from its neighbour.

03 MODELLING PHONEMES WITH HIDDEN MARKOV MODELS

After feature extraction, the system faces a sequential inference problem. It has a stream of feature vectors and must determine which sequence of phonemes — the smallest units of sound that distinguish meaning — most likely produced them. For decades the dominant tool was the hidden Markov model (HMM), a statistical framework in which an unobserved chain of states emits observable feature vectors according to probability distributions.

An HMM treats speech as a doubly stochastic process. A hidden state sequence transitions according to one set of probabilities, and each state emits an acoustic observation according to another. The states are typically tied to phoneme subunits — the entry, hold, and exit of a phone — and the emission distributions are Gaussian mixtures or, later, neural network posteriors. The Viterbi algorithm searches for the single most probable state path given the observations, and that path maps to a phoneme sequence, which maps to words via a pronunciation lexicon.

The strength of HMMs is their principled handling of temporal variability. A speaker may elongate a vowel or truncate a consonant; the model accommodates this by allowing state durations to vary. Their weakness is a strong conditional independence assumption: the probability of the next state depends only on the current state, not on the longer history. This makes HMMs poor at capturing the long-range coarticulation effects that real speech exhibits.

04 DECODING WITH LANGUAGE MODELS

The acoustic model alone cannot resolve ambiguity. The phoneme sequence /k/, /ae/, /t/ could be "cat" or "kat" or a fragment of a longer word. The system needs a language model — a statistical description of which word sequences are probable in the target language — to adjudicate. The classic n-gram language model estimates the probability of the next word given the previous n−1 words, using counts from a large text corpus.

Decoding is the process of searching the joint space of acoustic and language model probabilities to find the word sequence that maximises the combined score. This search is over an exponentially large space, so practical decoders use pruning heuristics — beam search, for example — to keep only the most promising partial hypotheses. The decoder consults the lexicon to expand words into phoneme sequences, the acoustic model to score each phoneme against the features, and the language model to score the word sequence, fusing all three into a single hypothesis.

The language model encodes syntactic and semantic regularities that the acoustic signal cannot. "Recognise speech" and "wreck a nice beach" produce nearly identical acoustics; the language model resolves the pun because it knows that the former is a far more probable phrase in the context of a speech recognition article than the latter.

05 NEURAL NETWORKS AND THE HYBRID ERA

In the 2010s, deep neural networks replaced Gaussian mixture models as the acoustic-model component inside HMM systems. A feed-forward or recurrent network trained on pairs of features and phoneme labels produced sharper emission probabilities than the mixtures they replaced, and word error rates dropped by roughly 30 percent overnight in the systems that adopted them. This was the hybrid DNN-HMM architecture: neural networks for acoustic scoring, HMMs for sequence structure, and n-gram language models for word-level context.

Recurrent neural networks — LSTMs and GRUs — improved on feed-forward nets by maintaining a hidden state that carried information across frames, letting the model condition its current prediction on a longer acoustic context. Bidirectional versions read the sequence both forward and backward, capturing future as well as past context. Connectionist temporal classification (CTC) provided a training objective that let these networks learn alignments between input frames and output labels without pre-segmented training data, a significant practical advance.

06 END-TO-END MODELS AND ATTENTION

The hybrid architecture required training three separate components — acoustic model, lexicon, and language model — and aligning their outputs. End-to-end models collapse this pipeline into a single neural network that maps features directly to text. The transformer architecture, introduced in 2017, uses self-attention to let every position in the input sequence attend to every other position, capturing long-range dependencies that HMMs and recurrent networks modelled poorly.

Systems such as OpenAI's Whisper, trained on 680,000 hours of multilingual data, use encoder-decoder transformers to map log-mel spectrograms directly to token sequences. The encoder extracts representations from the audio; the decoder generates text tokens autoregressively, conditioned on the encoder output and previously generated tokens. This unified training reduces the engineering surface: there is no separate lexicon, no separately trained language model, and no hand-engineered alignment. The cost is that the model must learn all of these mappings from data, which requires very large training sets to avoid overfitting narrow domains.

End-to-end models also changed how the systems fail. HMM-based systems produced characteristically "confident but wrong" errors when the language model dominated a weak acoustic signal; transformer models can produce fluent but acoustically unjustified outputs — a failure mode that is harder to detect because the output looks plausible at the language level even when the acoustic evidence is absent.

07 ERROR CORRECTION AND ADAPTATION

Even the best system errs. A second pass — a rescoring or error-correction stage — takes the n-best list of hypotheses from the decoder and re-ranks them with a stronger language model, a neural rescoring network, or a semantic consistency check. This does not change the acoustic evidence; it changes how the system weighs competing interpretations of that evidence.

Speaker adaptation tunes the model to an individual voice. Early systems used maximum likelihood linear regression to shift Gaussian means; modern systems fine-tune a portion of a neural network on a small enrolment set. The gain is real but bounded: adaptation reduces the gap between a speaker's accent or vocal tract characteristics and the model's training distribution, but it cannot recover phonetic information that the front end discarded.

Environmental adaptation addresses the mismatch between training and deployment conditions. A model trained on clean studio audio degrades in a noisy car; a model trained on a single microphone degrades when the user switches to a headset. Multi-condition training, data augmentation with synthetic noise, and robust feature extraction all address this, and the tension between robustness and accuracy remains the central engineering trade-off.