The science behind speech recognition

01 SPEECH IS A PRESSURE WAVE

Speech recognition begins with physics. When a speaker vocalises, their vocal folds vibrate, creating periodic pulses of air pressure. These pulses propagate through the vocal tract — the cavity formed by the throat, mouth, and nasal passages — where the shape of the cavity reshapes the spectrum of the pressure wave. The wave then radiates through the air, reaches a microphone, and is converted into an electrical signal. Every stage of this chain is governed by acoustic physics, and the science of speech recognition is the science of inverting that chain: recovering the vocal tract shape and the excitation from the pressure waveform recorded at the microphone.

The pressure wave is a longitudinal wave — air molecules oscillate along the direction of propagation. Its amplitude corresponds to loudness, its frequency to pitch, and its spectral shape to timbre. For speech, the fundamental frequency of voicing typically ranges from 80 to 350 Hz, depending on the speaker's vocal fold length and tension. The higher-frequency content — the formants, the fricatives, the bursts — carries most of the phonetic information that distinguishes one sound from another.

The wave is continuous, but a digital recognizer operates on discrete samples. The sampling theorem of Nyquist and Shannon states that a signal band-limited to B Hz can be perfectly reconstructed from samples taken at a rate of at least 2B samples per second. Speech recognition systems typically sample at 16 kHz, preserving frequencies up to 8 kHz, which covers most of the phonetically relevant energy. Higher-fidelity systems sample at 44.1 kHz, but the additional bandwidth contributes less to recognition accuracy than it does to perceived audio quality.

02 THE SOURCE-FILTER MODEL OF SPEECH PRODUCTION

The dominant scientific framework for understanding speech is the source-filter model, formulated by Gunnar Fant in 1960. The model separates speech production into two independent components: a sound source and a filter. For voiced speech, the source is the quasi-periodic vibration of the vocal folds; for unvoiced speech, the source is turbulent noise produced at a constriction in the vocal tract. The filter is the vocal tract itself, whose resonant frequencies — the formants — are determined by its shape.

The vocal tract can be modelled as a concatenation of acoustic tubes of varying cross-section. The resonant frequencies of this tube depend on its length and the configuration of constrictions and openings. A vowel such as /a/ is produced with a relatively open tract and a low first formant; a vowel such as /i/ is produced with a constriction near the palate and a high second formant. The formant frequencies are the acoustic fingerprint of the phoneme, and the recognizer's task is to estimate them from the observed spectrum.

The source-filter model is a simplification — the source and filter are not perfectly independent, and the vocal tract is not a lossless tube — but it captures the essential separability of speech: the excitation carries pitch and voicing, and the filter carries phonetic identity. This separability is what lets a speech recognizer distinguish a whispered /a/ from a whispered /i/ even when the voiced excitation is absent, and what lets a speech synthesiser change the pitch of a vowel without changing its identity.

03 THE COCHLEAR FREQUENCY ANALYSIS

The biological system that decodes speech is the cochlea, a fluid-filled coiled tube in the inner ear. The basilar membrane inside the cochlea varies in stiffness and width along its length, and this variation makes it a frequency analyser: a pressure wave entering the cochlea causes the membrane to vibrate maximally at a location determined by its frequency. High frequencies peak near the base; low frequencies peak near the apex. Hair cells at each location transduce the mechanical vibration into neural firing, and the brain receives a frequency-decomposed representation of the sound.

This decomposition is non-linear. The cochlea resolves low frequencies more finely than high ones: two tones at 100 and 200 Hz are perceived as more widely separated than two tones at 3100 and 3200 Hz, even though the absolute difference is 100 Hz in both cases. This non-linearity is captured by the mel scale, a perceptual frequency unit that compresses high frequencies. The mel scale is the basis for the mel-frequency filter bank used in speech feature extraction: the system divides the spectrum into bands that are narrow at low frequencies and wide at high ones, matching the resolution of the cochlea.

The cochlea also exhibits active amplification. The outer hair cells, which receive more efferent than afferent connections, mechanically amplify quiet sounds and sharpen frequency tuning. This active process gives the ear its remarkable dynamic range — roughly 120 decibels — and its sensitivity to faint signals in noise. A speech recognizer does not replicate this active amplification directly, but the spectral subtraction and noise-reduction techniques used in front-end processing serve an analogous function: enhancing the signal-to-noise ratio before the decoder operates.

04 PHONETIC FEATURES AND COARTICULATION

The phonetic content of speech is not encoded in isolated static snapshots but in dynamic trajectories of the spectrum. A phoneme is not a single spectral shape but a pattern of change — a transition into and out of a target. The formants move as the vocal tract transitions from one configuration to the next, and the recognizer must track these movements, not just measure the formants at their target values.

Coarticulation complicates this. The vocal tract does not reach each phoneme target in isolation; each phoneme is shaped by its neighbours because the articulators — tongue, lips, jaw — move continuously and cannot instantaneously switch configurations. A /t/ before an /u/ is produced with a tongue position already anticipating the /u/, and the resulting acoustic signal differs from the same /t/ before an /a/. The science of acoustic phonetics documents these coarticulatory effects, and a recognizer must model them: the acoustic model does not treat each phoneme as a fixed template but as a context-dependent unit whose acoustic realisation varies with its neighbours.

This is why speech recognition models use triphones — phoneme models conditioned on left and right context — rather than monophones. The triphone captures coarticulation by letting the model learn a different acoustic distribution for the same phoneme in different phonetic environments. The cost is a multiplication of the number of model parameters, which is managed through parameter tying: acoustically similar triphones share parameters, so the model generalises across contexts that produce similar coarticulatory effects.

05 PROBABILISTIC INFERENCE AND BAYES

Speech recognition is fundamentally a problem of probabilistic inference under uncertainty. The system observes a sequence of acoustic features and must infer the word sequence that produced them. This is an inverse problem: the forward mapping from words to acoustics goes through vocal production, room acoustics, and microphone recording, and each stage adds noise and variability. The inverse mapping — from acoustics back to words — is many-to-one: many word sequences could produce the same acoustic signal.

Bayes' theorem provides the framework. The posterior probability of a word sequence given the acoustics is proportional to the product of the acoustic likelihood — how probable the acoustics are given that word sequence — and the language prior — how probable the word sequence is in the language. The recognizer searches for the word sequence that maximises this posterior, and the architecture of the system is a decomposition of this search: the acoustic model estimates the likelihood, the language model estimates the prior, and the decoder searches the joint space.

The hidden Markov model was the dominant framework for this inference for decades because it provides a tractable approximation: the hidden state sequence is the phoneme sequence, the observations are the feature vectors, and the Viterbi algorithm finds the most probable state path efficiently. The Markov assumption — that the next state depends only on the current state — is a simplification that makes the inference tractable at the cost of ignoring long-range dependencies. Modern neural models relax this assumption: a transformer's self-attention lets every output token attend to every input frame, capturing dependencies that an HMM must discard.

06 THE INFORMATION-THEORETIC LIMIT

How much information does a speech signal carry, and how much can a recognizer recover? Shannon's information theory provides the tools. The mutual information between the spoken word sequence and the observed acoustic signal is the upper bound on what any recognizer can extract: no decoder can recover information that the signal does not contain. Noise, reverberation, and channel distortion reduce this mutual information, and the recognizer's accuracy is bounded by the resulting channel capacity.

The speech rate of a typical speaker is about 150 words per minute, or 2.5 words per second. At roughly 5 bits per word for a vocabulary of 30,000 words, the information rate is about 12.5 bits per second. This is far below the raw channel capacity of a 16 kHz sampled signal — the channel can transmit thousands of bits per second — which means the recognizer's task is not bandwidth-limited but noise-limited. The signal contains far more capacity than the linguistic message requires, but the noise and variability of the speech chain consume the surplus.

This margin explains why speech recognition can tolerate significant degradation and still succeed: the linguistic information is highly redundant, and a recognizer that recovers even a fraction of the acoustic detail can still decode the message. It also explains why the hard cases are not those with the most speech but those with the most noise: a clean studio recording at 100 words per minute is easier than a noisy car cabin at the same rate, because the mutual information between the message and the signal is higher in the first case.

07 WHAT THE SCIENCE DOES NOT EXPLAIN

The physical and statistical sciences explain how a pressure wave becomes a word sequence, but they do not fully explain meaning. A recognizer transcribes "recognise speech" or "wreck a nice beach" with equal fidelity when the acoustics are ambiguous; choosing the correct interpretation requires semantic context that goes beyond the signal. The science of speech recognition addresses the acoustic-to-text mapping; the science of natural language understanding addresses the text-to-meaning mapping, and the two are separate fields even when they are combined in a single product.

The science also does not fully predict human performance. A human listener understands speech in noise that defeats current recognizers, adapts to a new accent in seconds, and fills in masked phonemes from context without conscious effort. The cochlea, the auditory pathway, and the language centres of the brain implement computations that the engineering approximations capture only partially. The gap between human and machine recognition is narrowing but remains, and it is a gap of robustness and adaptation, not of peak accuracy on clean benchmarks.

Finally, the science does not settle the question of whether speech recognition should be modelled on human processing or engineered from first principles. The cochlear model inspired the mel scale, but the transformer — the architecture that drove the largest accuracy gains — was inspired by attention mechanisms in machine translation, not in biology. The field progresses by borrowing from the science of speech where it helps and from the science of computation where it helps, and the synthesis is an engineering discipline as much as a scientific one.