0180
Multimodal
Early models handled only text. A multimodal model takes images, text, and sometimes audio at once, and understands them in a single shared representation.
An image is cut into patches and each patch is turned into an embedding, exactly as words are turned into token embeddings. Both kinds of embedding are projected into one shared space, so a picture of a cat and the words "a cat" land near each other. From there the same transformer attends across everything, letting the model answer questions about an image or describe what it sees.
text & image → one space
Check yourself
A cat photo and the text "a cat" are totally different inputs. How does the model treat them as related?
Go deeper (free): Hugging Face — Vision-language models ↗