Live demo · Multimodality

How a multimodal LLM sees text and images

A vision-language model does one surprising thing: it turns a picture and a sentence into the same kind of object - a single sequence of vectors - then reasons over both at once. Step through the pipeline below to watch it happen.

Everything here runs in your browser and is derived from the real pixels and the text you type. The exact numbers are illustrative - a trained model learns them from data - but the shapes and steps are how real models like CLIP, LLaVA and GPT-4o actually work.

Text prompt

Image

text tokens image patches the model computing

The two inputs

One image, one sentence - two completely different kinds of data.

Raw text

image: 300×300 px text: 0 chars

What's happening A model can't do math on letters or pixels directly. Both inputs have to become numbers in the same space before the transformer can touch them. That conversion is the whole trick of multimodality: a vision encoder handles the image, a text tokenizer handles the sentence, and a projector stitches them into one stream.

Real systems CLIP LLaVA GPT-4o Gemini Qwen-VL

Tokenize & patchify

Text splits into tokens. The image splits into a grid of patches.

Text → tokens

Image → patches

What's happening Text is broken into sub-word tokens by a BPE or SentencePiece tokenizer (a word can be several tokens; ▁ marks a word start). The image is cut into fixed patches - here a 6×6 grid - exactly like a Vision Transformer (ViT). Real encoders use small patches: CLIP ViT-L/14 uses 14px patches, giving 256 patches for a 224px image. Each patch is flattened and treated as its own "visual word."

Techniques ViT BPE / SentencePiece tiktoken patch embedding SigLIP DINOv2

One shared sequence of vectors

Every token and every patch becomes a vector - and they all line up together.

value−+ each row = one token/patch · each column = one embedding dimension

The key idea This is where the two modalities merge. A linear projection maps each patch and token to the model's hidden width (d_model, e.g. 4096 in LLaMA-7B), and a positional encoding (learned, sinusoidal, or RoPE) is added so order is preserved. In a VLM this alignment is done by a small adapter: LLaVA uses a 2-layer MLP projector, BLIP-2 a Q-Former, Flamingo a Perceiver Resampler. From here on, the model doesn't care which row came from pixels and which from letters - it's all just vectors.

Techniques linear projection RoPE MLP projector (LLaVA) Q-Former (BLIP-2) Perceiver (Flamingo)

Attention, routing & activations

The transformer lets a word look at the pixels - then fires neurons.

Cross-modal attention - pick a word, watch the image light up

Top attended patches

Mixture-of-Experts routing - a gate sends the token to 2 of 4 experts

Feed-forward activations (GELU) - which neurons fire for this token

What's happening Self-attention (softmax(QKᵀ/√d)·V, run in parallel multi-heads) lets every position pull information from every other - including a text token reading the image patches it cares about (that's the heatmap, a form of cross-attention). In a Mixture-of-Experts layer a router sends each token to its top-2 of N experts (as in Mixtral 8×7B); the feed-forward net then fires activations through GELU or SwiGLU. Stack this block ~30-80× (LLaMA-7B: 32, GPT-3: 96), with RMSNorm and residuals between.

Techniques scaled dot-product attention multi-head cross-attention MoE top-2 (Mixtral) SwiGLU / GELU RMSNorm

Predict the next token

All that computation collapses into one probability distribution.

P(next token | image + text)

What's happening A final RMSNorm and the LM head (the unembedding matrix) project the last vector to one logit per vocabulary entry (~32k-128k tokens), then softmax turns logits into probabilities. A sampler picks one token - controlled by temperature and top-p / top-k - appends it, and runs the entire pipeline again. That autoregressive loop is how it "describes the image" or answers your question, one token at a time.

Techniques LM head / unembedding softmax temperature top-p / top-k autoregressive decoding