How recommendation systems are designed

01 THE PROBLEM OF CHOICE

A recommendation system is an information filtering system that ranks items from a large catalogue according to its estimate of what a particular user will value. The problem exists because choice has become vast and attention has not. A streaming service carries tens of thousands of titles; an online marketplace lists hundreds of millions of products; a social platform processes billions of posts per day. No human can survey these catalogues. The recommender's job is to reduce a space of millions to a ranked list of dozens, and to do it in milliseconds.

The design challenge is not merely prediction. A system that always recommends the most popular item is trivially accurate for the average user but useless for any individual whose taste diverges from the mean. A system that recommends only things the user has already seen is perfectly safe but pointless. The design space is governed by competing objectives: accuracy, diversity, novelty, serendipity, and fairness, none of which can be maximised simultaneously without trade-offs.

02 CONTENT-BASED FILTERING

The earliest approach to recommendation is content-based filtering. The system maintains a profile of each user's preferences and a representation of each item's attributes, then recommends items whose attributes match the user profile. If a user watches many action films, the system recommends more action films. The technique is as old as the catalogue itself, and it remains the first layer of most production systems because it works without data from other users.

The limitation is that content-based filtering can only recommend within the boundaries of what the user has already shown interest in. It cannot surprise. If the attribute representation is shallow, as it often is, the system will recommend items that share surface features but differ in substance. A user who enjoyed a thoughtful science fiction film may be recommended every film tagged with the same genre label, regardless of quality or tone.

03 COLLABORATIVE FILTERING

Collaborative filtering solves the narrowness of content-based filtering by drawing on the behaviour of many users rather than the attributes of items. The core idea is simple: if user A and user B have similar tastes, then items that A likes but B has not yet seen are candidates for B's recommendations. The approach does not need to understand what the items are, only how users interact with them.

Collaborative filtering comes in two forms. Neighbourhood methods find users who are similar to the target user and aggregate their preferences. Model-based methods learn a compact representation of user preferences from the entire interaction matrix. Both forms face the cold start problem: a new user has no interaction history, and a new item has no interactions from any user. Until enough data accumulates, neither user nor item can be meaningfully matched, and the system must fall back to content-based or popularity-based recommendations.

04 MATRIX FACTORIZATION AND THE NETFLIX PRIZE

The Netflix Prize, launched in 2006, transformed the field. Netflix offered one million dollars to any team that could improve its recommendation accuracy by ten percent over its existing system, measured by root mean squared error on a held-out dataset of over 100 million ratings. The competition ran for nearly three years and was won by a coalition called BellKor's Pragmatic Chaos in 2009.

The technique that emerged as dominant was matrix factorization. The user-item interaction matrix, which is almost entirely empty, is decomposed into two much smaller matrices: one representing each user as a vector of latent factors, and one representing each item as a vector of the same latent factors. A predicted rating is the dot product of the two vectors. The latent factors are not interpretable by humans; they are discovered by gradient descent optimisation. Typical factor dimensions range from 20 to 200, enough to capture genre, mood, style, and other qualities that drive preference but resist labelling.

The practical insight was that matrix factorization generalises better than neighbourhood methods. A user who has rated only a few items can still be placed in the latent factor space, and items similar in that space are good recommendations even if no other user has rated both. The Netflix Prize demonstrated that this approach could reduce prediction error by more than ten percent, and matrix factorization became the backbone of production recommenders for the next decade.

05 DEEP LEARNING AND NEURAL RANKING

The transition from matrix factorization to deep learning was driven by two developments: the availability of richer signals beyond explicit ratings, and the capacity of neural networks to combine heterogeneous features. A user's behaviour is not limited to star ratings; it includes clicks, watch time, scroll depth, search queries, and session context. A deep neural network can ingest all of these alongside item metadata and produce a personalised ranking in a single forward pass.

YouTube's recommendation system, described in a 2016 paper by Covington, Adams, and Sargin, exemplifies the architecture. It splits recommendation into two stages: a candidate generation network that retrieves a few hundred items from a catalogue of millions, and a ranking network that scores those candidates with a richer feature set. The two-stage design is now standard across the industry because it balances recall, the ability to find relevant items, against precision, the ability to order them well, without requiring a single network to score every item in the catalogue for every user request.

The Wide and Deep architecture, published by Google researchers in 2016, further refined the approach by combining a linear model that memorises frequent feature co-occurrences with a deep network that generalises across sparse features. The combination allows the system to exploit known associations, such as users who liked a specific franchise also liking its sequel, while also generalising to novel combinations.

06 OBJECTIVES BEYOND ACCURACY

Accuracy, measured as the system's ability to predict what a user will click or rate highly, is necessary but not sufficient. A system optimised only for click prediction will converge on the most engaging content, which may be sensational, repetitive, or polarising. Production recommenders are designed with multiple objectives that are combined into a single ranking score, typically as a weighted sum.

Diversity ensures that the recommended list is not dominated by one type of item. Novelty biases the system toward items the user has not encountered. Serendipity goes further, favouring items that are unexpected but satisfying, which differs from novelty because a novel but boring recommendation is not serendipitous. Fairness objectives ensure that the system does not systematically disadvantage certain creators or certain types of content. These objectives often conflict: maximising diversity reduces accuracy on average, and maximising novelty can reduce short-term engagement.

The weights assigned to each objective are among the most consequential design decisions in a recommendation system. They determine whether the system amplifies popular content or surfaces niche material, whether it reinforces existing preferences or broadens them, and whether it optimises for immediate engagement or long-term satisfaction. These weights are not purely technical; they encode values about what good recommendation means.

07 THE FILTER BUBBLE AND ITS DISCONTENTS

The concept of the filter bubble, introduced by Eli Pariser in 2011, describes a condition in which recommendation systems narrow the information environment of each user by feeding them content that confirms their existing preferences. The concern is that personalised filtering reduces exposure to challenging or diverse viewpoints, creating isolated information ecosystems that can deepen polarisation.

The empirical evidence is mixed. Studies have found that recommendation systems do increase the concentration of content consumed, but the effect is smaller than often assumed, and it interacts with other factors such as user agency and platform design. The filter bubble is not a property of any single algorithm but of the system that includes the algorithm, the user interface, and the user's own choices. A system that shows a personalised feed with no option to explore outside it produces a different information environment from one that personalises the default but allows easy deviation.

The design question is not whether to personalise but how to calibrate the degree and visibility of personalisation. A recommender that always defaults to the personalised feed is architecturally different from one that exposes the user to a broader catalogue and only personalises when asked. The trade-off between relevance and discovery is a design choice, not a technical inevitability.

08 THE ARCHITECTURE OF ATTENTION

A recommendation system is an architecture of attention. It decides which items from a catalogue of millions reach a user's screen, and in doing so it shapes what that user knows, wants, and believes. The engineering choices, from the selection of a loss function to the weighting of diversity against accuracy, determine the shape of the information environment that billions of people inhabit.

The systems in production today are not monolithic algorithms but layered pipelines that combine content-based filtering, collaborative models, neural ranking, and business rules. The design is iterative: each component is tuned against offline metrics and online experiments, and the system evolves as the catalogue, the user base, and the objectives change. The recommender is never finished because the world it models is never static.

The frontier of recommendation design is moving toward generative and conversational systems that do not merely rank existing items but construct new ones or explain their choices in natural language. Whether this represents an improvement depends on whether the additional flexibility is used to increase user agency or to further optimise engagement. The technology does not determine the answer, but it constrains the space of possible answers. The design of recommendation systems is, ultimately, the design of the interface between human attention and the information that competes for it.