How AI Sees Images: From Pixels to Concepts

01An image is a grid, not a picture

To a computer, a photograph begins as numbers: rows and columns of pixels, with each location carrying intensity or colour channels. Nothing in that raw array says “cat,” “road,” or “face.” The first task is to transform a dense grid into a representation that preserves useful structure while discarding irrelevant variation.

That is why computer vision is more than image processing. Brightening or blurring changes pixels; vision tries to infer objects, geometry, motion, or a scene that a system can act on.

02Convolution looks locally before it looks globally

Convolutional neural networks introduced a powerful bias: nearby pixels usually belong to nearby visual structure. A small filter slides across the image and responds to repeated patterns. Early layers can detect edges and colour transitions; later layers combine those responses into textures, parts, and whole-object evidence.

Shared weights make the same detector reusable at different positions. Pooling and deeper receptive fields then let the network trade exact location for broader context. The result is not a little human eye inside the machine—it is a learned hierarchy of numerical feature maps.

03ImageNet made the learning problem measurable

The ImageNet project assembled a scale of labelled data that earlier vision systems did not have. More than 14 million images were annotated across more than 20,000 categories, with bounding boxes available for at least one million. The annual ILSVRC contest narrowed the comparison to 1,000 classes and made progress visible.

Data was the multiplier. A model can discover useful filters only when training examples expose it to enough variation: different lighting, poses, backgrounds, camera angles, and contexts.

04AlexNet changed the default recipe

AlexNet’s 2012 result was a sharp break from hand-engineered pipelines. Its 15.3% top-five error rate beat the runner-up by more than 10.8 percentage points. The model had 60 million parameters and 650,000 neurons, and GPUs made its depth practical.

The lesson was not simply “use a bigger network.” It was the combination of a convolutional architecture, a large labelled dataset, nonlinear activations, regularization, and hardware that could run the training loop fast enough.

05CLIP aligns two kinds of evidence

The Computerphile video focuses on CLIP, a system trained to bring images and their text descriptions into a shared embedding space. Instead of asking only “which of these fixed labels is present?”, the model can compare an image with many text prompts and choose the closest meaning.

That alignment is a bridge between recognition and language. It also explains why prompt wording matters: the model is not reading a caption in the human sense; it is measuring relationships in a learned vector space.

06Seeing is not understanding

Vision models can be brittle. A small crop, unusual viewpoint, shortcut in the background, or adversarial perturbation can change the answer while leaving the human interpretation stable. A high score on a benchmark therefore measures performance on a distribution, not an inner guarantee of common sense.

Interpretation also remains difficult. Feature maps can reveal what activates, but they do not automatically reveal why a decision was made or whether the model relied on a spurious correlation.

07The useful abstraction is a world model

The arc from pixels to concepts is a sequence of compressions: measurements become features, features become objects, objects become relationships, and relationships become a prediction or action. Each compression is useful because it removes detail; each can also remove the wrong detail.

N43’s takeaway is practical: ask what data a vision model saw, what invariances its architecture encourages, and what evidence its output actually uses. “AI sees” is shorthand for a pipeline of learned representations—not a miniature human observer.