How ChatGPT Works: Inside the Architecture of a Large Language Model

01 From Text to Tokens: The First Translation Layer

Before a language model can process language, it must convert human text into numbers. This step, called tokenization, splits input text into discrete units called tokens. ChatGPT uses a byte-pair encoding (BPE) tokenizer that represents common words as single tokens and rare words as sequences of sub-word fragments. The word "hamburger" might become two tokens, "ham" and "burger," while the word "the" is a single token. The tokenizer for GPT-4 has a vocabulary of approximately 100,000 tokens, each mapped to a unique integer index.

Tokens are not the same as words. A single English word averages about 1.3 tokens, but tokenization varies significantly across languages. Chinese characters may require multiple tokens each, while common English phrases compress efficiently. This variability has real consequences: the cost of running a model depends on token count, not word count, and languages with less efficient tokenization face higher per-word inference costs.

02 The Transformer: A Revolution in Sequence Processing

The transformer architecture, introduced in the 2017 paper "Attention Is All You Need" by Vaswani and colleagues at Google, replaced the recurrent neural networks that previously dominated natural language processing. The key innovation was the self-attention mechanism, which allows every token in a sequence to attend to every other token simultaneously, rather than processing them one at a time. This parallelization made training on massive datasets computationally feasible for the first time.

A transformer model consists of stacked layers, each containing a multi-head self-attention sub-layer and a feed-forward neural network sub-layer. In ChatGPT-class models, the architecture uses decoder-only transformers, meaning the model predicts the next token in a sequence based on all preceding tokens. A typical large model may have 96 or more such layers, each with hidden dimensions of thousands of parameters, producing a total parameter count in the hundreds of billions.

03 Self-Attention: The Core Mechanism

Self-attention is the operation that gives transformers their power. For each token in the input, the model computes three vectors: a query, a key, and a value. The query vector represents what the token is looking for, the key vector represents what each token offers, and the value vector is the information that gets passed along. The attention score between two tokens is the dot product of the query and key vectors, scaled by the square root of the dimension size, then normalized through a softmax function to produce weights summing to one.

Multi-head attention runs this process in parallel across multiple heads, each learning different attention patterns. One head might learn to attend to syntactic relationships like subject-verb agreement, while another captures semantic similarity. The outputs of all heads are concatenated and projected back to the model's hidden dimension. With 96 heads operating in parallel across 96 layers, a single forward pass involves billions of dot-product computations, which is why GPU acceleration is essential for practical inference.

04 Pre-Training: Learning the Shape of Language

The first and most computationally expensive stage of training a model like ChatGPT is pre-training. The model is exposed to trillions of tokens of text drawn from web pages, books, code repositories, and other sources. The training objective is simple: predict the next token given all preceding tokens. Through this autoregressive objective, the model learns grammar, factual associations, reasoning patterns, and stylistic conventions without any explicit supervision.

Pre-training a large model requires thousands of GPUs running for weeks or months. The computational cost scales roughly with the product of the model's parameter count and the number of training tokens. A model with 175 billion parameters trained on 500 billion tokens requires approximately 3.14 times 10 to the 23rd floating-point operations, a figure that informs the massive data center investments behind frontier model development. The resulting base model can generate fluent text but has no concept of conversation, safety, or instruction following.

05 Fine-Tuning and RLHF: Teaching the Model to Chat

After pre-training, the model undergoes supervised fine-tuning on thousands of human-written conversation examples. Human annotators craft prompts and ideal responses, teaching the model the format and style of helpful dialogue. This stage is relatively cheap in compute terms but expensive in human labor, as each example must be carefully crafted to demonstrate the desired behavior.

Reinforcement Learning from Human Feedback (RLHF) is the final alignment stage. Human raters compare multiple model outputs for the same prompt and rank them by quality. These rankings train a separate reward model that predicts human preference scores. The language model is then optimized using proximal policy optimization (PPO) to maximize the reward model's scores, iteratively shifting its output distribution toward responses that humans rate favorably. This process is repeated for multiple rounds, gradually improving the model's helpfulness, honesty, and safety.

06 Inference: Generating One Token at a Time

When a user sends a prompt to ChatGPT, the model processes the entire conversation history through its transformer layers to produce a probability distribution over the vocabulary for the next token. The model samples from this distribution using a temperature parameter that controls randomness: low temperature produces focused, deterministic outputs, while higher temperature yields more creative but less predictable responses. Each generated token is appended to the sequence, and the model runs another forward pass to predict the next one.

This autoregressive generation is inherently sequential: each token depends on all preceding tokens. Techniques like key-value caching, which stores the intermediate representations of previously processed tokens, reduce the computational cost of each subsequent step from quadratic in sequence length to linear. Speculative decoding, where a smaller draft model proposes tokens that the large model verifies in parallel, can further reduce latency by 2 to 3 times for certain workloads.

07 Limitations and the Path Forward

Despite their fluency, large language models have well-documented limitations. They can produce confident-sounding but factually incorrect statements, a phenomenon called hallucination. They inherit biases from their training data, and their knowledge is bounded by the cutoff date of their training corpus. The self-attention mechanism has a quadratic memory cost in sequence length, which limits the context window size, though techniques like sparse attention and rotary position embeddings have pushed practical context windows to hundreds of thousands of tokens.

The frontier of language model development in 2026 is shifting toward multimodal architectures that process text, images, and audio within a unified transformer framework, and toward agentic systems that use language models as reasoning engines to plan and execute multi-step tasks. The transformer architecture that powers ChatGPT remains the foundation, but the applications being built on top of it are expanding far beyond conversational text generation.