Embeddings

An embedding maps each word to a vector — a list of numbers, often hundreds long. This lesson flattens that to two dimensions so it fits on screen, but the idea holds: position encodes meaning. The model is not told these positions; it learns them from data, by noticing which words appear in similar contexts. Words used the same way drift close together. Because meaning becomes geometry, you can do arithmetic with it — the famous example is king minus man plus woman landing near queen. Embeddings are the first thing that happens inside a language model: every word is turned into its vector before attention and the later layers ever see it. This is exactly the numeric form the Attention and Language Models lessons assume.

This lesson places each word using two numbers, an x and a y, so it fits on a flat screen. Real models use far more. Each word becomes a list of hundreds of numbers — a common size is 768, written 768d, from early models like BERT and GPT-1. Larger models use 1024, 4096, or more. Why so many? With only two axes you can express maybe how animal-like or how food-like a word is, and words get crushed together. Hundreds of axes give the model room to capture tense, formality, gender, sentiment, topic, and many subtler patterns at once — each dimension a separate shade of meaning it discovers from data. The trade-off is you cannot picture it: humans see 2D and 3D, not 768. The maths is identical though, distance still means similarity. When tools show a 768d space as a 2D picture, they squash it down with techniques like PCA, t-SNE, or UMAP, which is essentially the flattened view you see here.