Large Language Models Explained: How LLMs Actually Work Under the Hood

01What Is a Large Language Model?

A large language model is a neural network trained on massive quantities of text to predict the next token in a sequence. That simple objective, applied at sufficient scale, produces systems that can write essays, answer questions, translate languages, and write code. The key word is scale: what makes a language model large is not just its parameter count but the volume and diversity of its training data.

Modern LLMs contain anywhere from billions to trillions of parameters, trained on trillions of words of text from books, websites, and code repositories. The result is a statistical model of language that captures grammar, facts, reasoning patterns, and stylistic conventions with remarkable fidelity.

02Tokenization: How LLMs Read Text

Before a model can process text, it must be broken into tokens. Tokenization is the process of splitting text into subword units, typically using algorithms like Byte-Pair Encoding (BPE) or SentencePiece. A token might be a whole word, a partial word, or even a single character, depending on frequency.

The choice of tokenizer affects everything downstream. A model trained with a tokenizer that splits common words into many subword tokens will be less efficient at processing those words. The tokenizer effectively defines the vocabulary of the model, and its design is a critical engineering decision.

03Transformer Architecture and Attention Mechanisms

The Transformer architecture, introduced in 2017, is the foundation of every modern LLM. Its key innovation is the self-attention mechanism, which allows the model to weigh the relevance of different tokens in a sequence when making predictions. Instead of processing text sequentially like earlier recurrent networks, Transformers process all tokens simultaneously, making them highly parallelizable.

Self-attention works by computing three vectors for each token: a query, a key, and a value. The attention score between two tokens is the dot product of the query of one and the key of the other, normalized by a softmax function. This allows the model to focus on relevant context regardless of position in the sequence.

04Training: Pretraining, Fine-Tuning, and RLHF

LLM training happens in stages. First, pretraining: the model learns to predict the next token across trillions of words of internet text, acquiring broad language competence and factual knowledge. This stage consumes the vast majority of training compute.

Next comes fine-tuning on curated datasets for specific tasks. Finally, Reinforcement Learning from Human Feedback (RLHF) aligns the model with human preferences. Human raters compare model outputs, and a reward model is trained to predict which outputs are preferred. The language model is then optimized against this reward signal.

05How LLMs Generate Text: Prediction and Sampling

Text generation is autoregressive: the model predicts one token at a time, appends it to the input, and predicts the next. At each step, the model outputs a probability distribution over its entire vocabulary. The simplest approach is greedy decoding, always picking the most likely token, but this produces repetitive text.

More sophisticated approaches like top-k sampling, nucleus sampling (top-p), and temperature scaling introduce controlled randomness. Temperature controls how sharply the distribution is peaked: low temperature makes the model more deterministic, high temperature makes it more creative but also more prone to errors.

06Scaling Laws and Why Size Matters

The scaling laws discovered by Kaplan et al. at OpenAI in 2020 showed that model performance improves predictably as a power law of parameter count, data size, and compute. Crucially, the laws showed that larger models are more sample-efficient: they reach the same performance with less data per parameter.

Chinchilla (DeepMind, 2022) refined these findings, showing that many models were undertrained for their size. The optimal ratio is approximately 20 tokens per parameter. This insight led to a shift toward training smaller models on more data, improving efficiency without sacrificing capability.

07Limitations: Hallucinations, Bias, and Context Windows

LLMs hallucinate: they generate confident-sounding statements that are factually wrong. This is not a bug but a feature of how they work. The model does not know whether something is true; it knows whether it sounds plausible based on its training distribution. Mitigating hallucinations requires external verification, retrieval-augmented generation, or human review.

Bias is inherited from training data. If the internet contains stereotypes, the model will reproduce them. Context windows limit how much text the model can consider at once, though recent models have expanded this from a few thousand tokens to over a million. Each limitation is being actively addressed, but none has been fully solved.