How Large Language Models Work: Inside the AI Powering Modern Chatbots

01 The Token Problem

Before a language model can think, it must read. But machines do not read words the way humans do. They read tokens: fragments of text that might be a whole word, a syllable, or even a single character. The first step in any large language model pipeline is tokenization, the process of chopping raw text into units the model can process. A word like "unbelievable" might become three tokens: "un", "believ", and "able". The choice of tokenization scheme directly affects how many tokens a model can consume, how efficiently it represents language, and how well it handles languages other than English.

Modern tokenizers like byte-pair encoding, used by GPT-3 and its successors, build their vocabularies by starting with individual bytes and iteratively merging the most frequent pairs until a target vocabulary size is reached. A typical vocabulary contains 50,000 to 100,000 tokens. Each token maps to an integer index, which the model uses to look up a learned embedding vector. These vectors, typically 4,096 to 12,288 dimensions depending on the model, are the model's actual input. Everything downstream, attention, feed-forward layers, output projection, operates on these dense numerical representations.

02 Attention Is All You Need

The architectural breakthrough that made large language models possible was the transformer, introduced by Vaswani et al. in 2017. Before transformers, natural language processing relied on recurrent neural networks that processed text one word at a time, carrying a hidden state forward. This sequential processing created a bottleneck: long-range dependencies between distant words were hard to learn, and training could not be parallelized across the sequence.

The transformer solved both problems with self-attention. Instead of processing words one at a time, self-attention lets every token attend to every other token simultaneously. For each token, the model computes query, key, and value vectors through learned linear projections. The dot product of a query with all keys produces attention scores, which are normalized via softmax into weights. The token's output is then a weighted sum of all value vectors. Multi-head attention runs this process in parallel across multiple learned projection sets, letting the model capture different relationship types simultaneously.

The result is an architecture that can be trained in parallel across the entire input sequence, scaling efficiently to thousands of GPUs and billions of parameters. Every major language model in production today, from GPT-4 to Claude to Gemini to Llama, is a transformer descendant.

03 The Training Pipeline

Training a large language model happens in stages. The first is pretraining, where the model learns to predict the next token across trillions of tokens scraped from the internet, books, code repositories, and licensed datasets. The objective is simple: given a sequence of tokens, predict the next one. The model is penalized by cross-entropy loss, and gradients flow backward through billions of parameters via backpropagation. This phase consumes the vast majority of compute, often thousands of GPU-years, and produces what practitioners call a base model.

A base model is a powerful next-token predictor but not a useful assistant. It will happily continue a prompt with more web text, complete a half-finished sentence with plausible fiction, or generate code that almost compiles. To turn it into a chatbot, researchers apply instruction tuning, also called supervised fine-tuning, where the model is trained on curated examples of instruction-response pairs. This teaches the model to follow directions rather than simply continue text.

The third stage is alignment, typically through reinforcement learning from human feedback. Human annotators rank model responses by quality, a reward model is trained on these rankings, and the language model is optimized against the reward model using proximal policy optimization or similar algorithms. This stage is what makes models refuse harmful requests, produce balanced answers, and maintain a consistent persona.

04 Inference: Generating Text One Token at a Time

Once trained, a language model generates text autoregressively. Given an input prompt, it computes a probability distribution over the entire vocabulary for the next token, samples or greedily selects one token, appends it to the sequence, and repeats. This sequential generation is why large language models are inherently slower at generation than at comprehension: processing the prompt can be parallelized, but each new token depends on all previous ones.

The computational cost of inference grows with model size. A 175-billion-parameter model like GPT-3 requires roughly 350 GB of memory just to store its weights in 16-bit precision, far exceeding what any single GPU can hold. Production systems use techniques like pipeline parallelism, tensor parallelism, and KV-cache optimization to distribute the model across multiple accelerators and avoid recomputing attention for previous tokens. Quantization, reducing weights from 16-bit to 8-bit or 4-bit, can cut memory requirements dramatically at a modest cost in output quality.

05 Hallucinations and Reliability

Large language models do not have beliefs, knowledge stores, or fact-checking mechanisms in the way a search engine might. They generate text by sampling from a learned probability distribution over tokens. When that distribution assigns high probability to factually correct continuations, the output is accurate. When it does not, the model produces fluent, confident, and entirely fabricated statements. This phenomenon, called hallucination, is not a bug but a feature of next-token prediction under a loss function that rewards plausibility over truth.

Researchers have developed several mitigations. Retrieval-augmented generation grounds model outputs in externally retrieved documents, reducing but not eliminating hallucination. Constitutional AI and other alignment techniques train models to express uncertainty and defer to evidence. Chain-of-thought prompting encourages models to reason step by step, which can improve accuracy on mathematical and logical tasks. None of these approaches fully solves the reliability problem, and production deployments must assume some baseline hallucination rate.

06 The Economics of Scale

The dominant strategy in language model development has been scaling: more parameters, more data, more compute. OpenAI's GPT-2 had 1.5 billion parameters. GPT-3 had 175 billion. GPT-4 is estimated at over a trillion parameters using a mixture-of-experts architecture. Each generation costs orders of magnitude more to train. GPT-3's training run was estimated at around 4.6 million dollars. GPT-4 is estimated at over 60 million. The forthcoming generation of models is expected to cross 100 million dollars in training cost.

This economic reality has consequences. Only a handful of organizations can afford to train frontier models from scratch. The open-source community, led by Meta's Llama releases, Mistral, and the Allen Institute, provides capable smaller models that can be fine-tuned cheaply, but the frontier keeps moving. The question facing the industry is whether scaling laws will plateau before the cost curve becomes unsustainable, or whether algorithmic improvements will make smaller models more efficient, democratizing access.

07 Limits and Open Problems

Despite their commercial success, large language models have well-documented limitations. They struggle with multi-step reasoning that requires maintaining intermediate state across long contexts. They exhibit inconsistency: the same prompt can produce different answers on different runs. They inherit biases from their training data, which overrepresents English, Western perspectives, and internet-connected populations. They cannot reliably perform arithmetic without external tools, and their spatial reasoning is often poor despite fluent descriptions of spatial concepts.

The open problems are significant. Context length, once capped at 2,000 tokens, has expanded to 128,000 and beyond, but attention costs grow quadratically with sequence length in naive implementations. Efficient attention mechanisms like flash attention and sparse attention reduce but do not eliminate this cost. Multimodal integration, combining text with images, audio, and video, is an active frontier. And the fundamental question of whether next-token prediction can ever lead to genuine reasoning, or whether it is an elaborate pattern-matching trick, remains contested.