From Data to Deployment: The Five-Stage Pipeline of Building an AI Model

01 Stage One: Data Collection and Curation

Every AI model begins with data, and the quality of that data determines the ceiling of what the model can learn. The collection phase involves identifying relevant sources, acquiring raw text, images, or tabular records, and establishing licensing and provenance. For large language models, training corpora are assembled from web crawls, book repositories, code repositories, and curated datasets. The Common Crawl archive, for instance, contains petabytes of web text accumulated since 2007 and forms the backbone of many frontier model training sets.

Data curation is where most projects encounter their first major cost. Raw web data is noisy, duplicated, and occasionally toxic. Filtering pipelines must remove machine-generated text, boilerplate navigation elements, and personally identifiable information. Teams at companies like OpenAI and Anthropic have invested heavily in deduplication and quality filtering, recognizing that a model trained on repeated low-quality text will reproduce those patterns at inference time.

02 Stage Two: Data Preparation and Feature Engineering

Once data is collected, it must be transformed into a format the model can consume. For text models, this means tokenization: converting strings into integer sequences from a fixed vocabulary. The choice of tokenizer affects everything from context length to multilingual coverage. Byte-pair encoding, used by GPT-family models, and SentencePiece, used by Gemini and Claude, represent two dominant approaches with different tradeoffs in vocabulary size and sequence length.

For non-text modalities, preparation involves normalization, augmentation, and labeling. Image models benefit from random crops, flips, and color jittering that expand the effective training set. Audio models convert waveforms to spectrograms. Supervised learning tasks require human-labeled examples, and the cost of labeling scales linearly with dataset size. A 100,000-image classification dataset at five dollars per label costs half a million dollars before a single parameter is trained.

03 Stage Three: Model Training

Training is the phase most people associate with AI, and it is where the majority of compute cost accumulates. A transformer model learns by processing input sequences, generating predictions, computing the difference between predictions and targets, and adjusting its parameters via backpropagation. Each pass through the training data is called an epoch, and modern models may train for dozens of epochs over trillions of tokens.

The computational cost of training scales with the product of model parameters, dataset size, and number of training steps. A 70-billion-parameter model trained on 2 trillion tokens might require approximately 1.4 million GPU-hours on NVIDIA H100 accelerators. At current cloud pricing of roughly two dollars per GPU-hour, that translates to 2.8 million dollars in compute alone, before accounting for storage, networking, and engineering overhead. Frontier models at the trillion-parameter scale can cost tens of millions of dollars per training run.

04 Stage Four: Evaluation and Validation

A trained model must be evaluated against held-out data it has never seen, to measure generalization rather than memorization. Standard benchmarks like MMLU for knowledge, HumanEval for code generation, and GSM8K for mathematical reasoning provide comparable scores across models. However, benchmark performance is a necessary but insufficient signal. Models can achieve high scores through dataset contamination, where test examples inadvertently appear in training data.

Human evaluation remains the gold standard for subjective qualities like helpfulness, harmlessness, and honesty. Reinforcement learning from human feedback, the technique that transformed GPT-3 into ChatGPT, uses human raters to compare model outputs and trains a reward model to approximate those preferences. This stage is labor-intensive and culturally sensitive: raters in different regions may disagree on what constitutes a helpful response.

05 Stage Five: Deployment and Monitoring

Serving a model in production introduces challenges that do not appear in research environments. Inference latency must meet user expectations, typically under one second for chatbot responses. Throughput must handle concurrent requests, which for popular APIs can reach thousands per second. Model weights, which for frontier models exceed 100 gigabytes, must be distributed across multiple accelerators using tensor parallelism or pipeline parallelism.

Monitoring detects model drift, where the statistical properties of input data change over time and degrade output quality. A model trained on 2023 web data may produce increasingly stale or inaccurate responses as the world evolves. Production ML systems implement automated retraining pipelines that periodically refresh model weights on new data, along with guardrails that filter harmful outputs and rate limits that prevent abuse.

06 The Hidden Complexity of Iteration

In practice, the five-stage pipeline is not linear. Evaluation reveals weaknesses that send teams back to data collection. Deployment surfaces edge cases that require additional training data. The most successful AI organizations treat model development as a continuous loop rather than a waterfall, with telemetry from production informing the next training cycle. This MLOps discipline, borrowing from DevOps, is what separates organizations that ship reliable AI products from those that produce impressive demos.

The cost structure also favors iteration. A team that trains a 7-billion-parameter model for 100,000 dollars can afford to try ten variations. A team training a trillion-parameter model at 60 million dollars per run gets fewer attempts, raising the stakes of each iteration and favoring organizations with deep infrastructure budgets. This economic reality is one reason the frontier model landscape remains concentrated among a small number of well-capitalized labs.