Vision language models: how AI learned to see and understand images

01From text-only to multimodal

For most of the last decade, the systems that could genuinely reason about language were blind. A large language model ingests text and predicts text, and everything it knows about the visual world arrives secondhand, compressed into words. That works until a user uploads a photo, a chart, a whiteboard sketch — and suddenly the model has nothing to say. Vision language models, usually shortened to VLMs, close that gap. A VLM is a system that can jointly interpret and generate information from both images and text, extending the capabilities of text-only LLMs into genuine multimodal learning.

The lineage is older than the current boom. Early image captioning systems from the 2010s chained handcrafted visual features into rule-based sentence templates. By mid-decade, convolutional encoders fed recurrent decoders to write captions about pictures of baseball games and brunch tables. The pivotal moment arrived in 2021, when OpenAI published CLIP — a model trained on roughly 400 million image-text pairs, using contrastive learning to place images and their descriptions into the same mathematical space. CLIP was never designed to chat, but its general-purpose image representations became the visual cortex for the VLMs that followed.

From 2022 onward, the modern recipe consolidated: a pretrained image encoder, an off-the-shelf language model, and a learned connector stitching the two together. Systems like DeepMind's Flamingo, the open-source LLaVA, Salesforce's InstructBLIP, and Microsoft's Kosmos proved the pattern worked. When GPT-4V shipped in 2023 and multimodal input became a standard feature of ChatGPT, Gemini, Claude, and Copilot, VLMs moved from research curiosity to default interface. The video explainer from IBM Technology that this article accompanies walks through the same arc — the field went from describing images to reasoning about them in under five years.

02How vision encoders turn pixels into tokens

The core trick is translation. Language models only consume one currency: tokens. So before an image can be reasoned about, it must be converted into something that looks like language. A vision encoder — typically a vision transformer, or ViT — chops an image into a grid of fixed-size patches, commonly 16 by 16 pixels. Each patch gets flattened, projected, and assigned an embedding vector, exactly as a tokenizer would slice a sentence into words. The result is a sequence of visual tokens: a 224-pixel-square image becomes 196 of them.

These patch embeddings are then compared against a pretrained vocabulary of visual concepts, so every patch carries both what it contains and where it sits in the grid. Position information is injected so the model knows a cat above a table is not a table above a cat. The encoder — very often a CLIP-style contrastive model, or a successor like SigLIP — outputs a compact set of vectors summarizing the whole scene. In many production VLMs, further compression collapses hundreds of patch vectors into a handful of high-level summary vectors, which keeps the language model's context window from drowning in picture-data.

The payoff is uniformity. Once an image is a sequence of embeddings, the language model cannot tell the difference between a photograph and a paragraph. Both are just vectors. A question about the image is literal: the picture tokens sit in the context window beside the question tokens, and the model answers the way it answers any text query — by attending across everything at once.

03Alignment between image and language

Encoders produce vectors and language models expect vectors, but the two vector dialects do not match out of the box. The image encoder was trained on captions and contrastive objectives; the LLM was trained on web text. Between them sits a projector — a comparatively small module whose only job is translation. LLaVA popularized the simplest approach: a single linear layer that maps each visual embedding into the LLM's word-embedding space, followed by supervised fine-tuning on image-question-answer triples. Other architectures use cross-attention layers that let the text stream peek at image features as they flow past, which Flamingo demonstrated with gated modules added to a frozen language model.

Training a VLM is a staging exercise. The encoder and language backbone are usually frozen first, and only the connector learns, using curated image-caption data. Then instruction-tuning data — real questions, chart reads, OCR requests — teaches the assembled system to behave like a helpful assistant rather than a caption machine. The choice of data matters more than the choice of architecture: a VLM is, in a real sense, an alignment layer between two sensory modalities, and sloppy pairing data produces a model that confidently describes things that are not in the picture.

04What VLMs can and cannot do

The capability list is now long. A competent VLM can caption a photo, answer questions about a diagram, extract text from a screenshot, read a scatter plot's axes, identify a manufacturing defect against a reference photo, and reason about spatial relationships — left, above, overlapping. On standard benchmarks, models score above 90 percent on image-text retrieval and push past 80 percent on visual question answering, numbers unthinkable for captioning systems of a decade ago.

The failure list is equally instructive. VLMs remain brittle at precise counting — ask how many pills are in a photo and the error rate climbs as the number grows. Fine-grained spatial reasoning, like whether one object slightly occludes another, still trips models trained mostly on captions, because captions rarely bother to mention occlusion. Dense text — a full page in one screenshot — overwhelms patch tokenization, and specialized OCR still wins. And VLMs inherit the hallucination problem from their language half: when visual evidence is thin, the model fills gaps with statistically plausible text, which can be fluent and wrong in equal measure.

Practical users learn to route around the weaknesses. When a task needs reliable counting, ask the model to detect and list objects instead of naming a total. When a document needs perfect transcription, run OCR first and hand the VLM the text. The mature way to treat a VLM is as a strong generalist that occasionally needs a specialist double-checking its work — the same posture sensible engineers take with its text-only parent.

05Real-world applications

Adoption has spread far beyond chatbots with photo upload. In healthcare, VLMs draft radiology report impressions and structure pathology slides for review, always with a physician signing off. Industrial operators run VLM inspection on production lines, where a camera plus a model flags solder defects or missing fasteners without weeks of bespoke training data. Accessibility is possibly the most consequential domain: point a camera at anything and ask what it is, and a VLM answers — reading signs for blind users, menus in foreign restaurants, forms in unfamiliar bureaucracies.

Elsewhere, document intelligence is the quiet workhorse — invoices, contracts, insurance claims, intake forms, anything where unstructured pixels must become structured fields. E-commerce platforms use VLMs for automated product tagging, visual search, and moderation of listing photos. In each case the economics are the same: a system that reads pixels and text with one model replaces pipelines that used to chain OCR, classifiers, and rule engines, each with its own failure modes.

06The frontier: video and embodied AI

Images are single frames; the world is not. The research frontier has moved to video, where a model must track objects across time, remember that the person who left the frame three seconds ago is the same person who returns, and answer questions about events rather than appearances. Video multiplies the token problem brutally — every frame is another grid of patches — so progress depends on temporal compression tricks and smarter token selection. Early video-language systems already answer natural questions about clips, but long-form understanding remains open.

The other frontier is embodied. A robot equipped with a camera and a VLM gains something earlier navigation stacks never had: a model that shares a vocabulary with the human giving instructions. Say pick up the red mug by the sink and the system decomposes the sentence, grounds red, mug, and sink in the camera feed, and plans a motion. This is why VLMs are becoming the perception-and-language layer of general-purpose robots, and why every serious robotics lab now treats multimodal models as core infrastructure. The same loop that let a chatbot look at a photograph is starting to let machines look at the room — and act on what they see. That is the arc the IBM Technology video traces, and it is a long way from finished.