kNN & SVMs
Two classic classifiers that need no neural network. k-Nearest Neighbours barely trains at all: it memorises every example, and to classify a new point it finds the k closest ones and takes a majority vote — small k reacts to every local quirk (overfits), large k smooths toward the global majority (underfits). Support Vector Machines take the opposite view: forget the crowd, find the single dividing line with the widest possible margin, positioned only by the few border points — the support vectors.
Drag the query point around. Dashed lines mark its k nearest neighbours; the vote decides its colour. Slide k and watch borderline predictions flip at k=1 but hold steady at k=9.
The query point takes the majority vote of its k nearest neighbours. Drag it toward the border and watch small k flip erratically while larger k stays stable.
How it works
kNN and SVMs sit at two extremes of a spectrum. kNN is a lazy learner: zero work at training time, all work at prediction time, and it needs the entire dataset in memory forever — costly when data is large. SVMs are eager: heavy work up front to find the maximum-margin boundary, then prediction is a single dot product. The kernel trick extends SVMs to curved boundaries by implicitly mapping points into a higher-dimensional space where a straight line suffices. Before deep learning took over, kernel SVMs were the state of the art for many tasks, and both remain strong baselines on small tabular datasets today.
Variants & real-world flavors
kNN's behaviour hinges on the distance metric: Euclidean for continuous features, Manhattan when outlier dimensions should count less, cosine for text and embeddings (that variant powers vector search). Naive kNN checks every point, so large datasets use spatial indexes — kd-trees and ball trees exactly, or approximate nearest-neighbour methods (HNSW, the engine inside vector databases) at scale. Weighted kNN counts closer neighbours more. SVMs come in flavors via the kernel: linear for high-dimensional sparse data like text, RBF for smooth curved boundaries (the everyday default), polynomial for feature interactions — plus a soft-margin C parameter that trades violations against margin width.
Check yourself
Near the boundary between the two clusters, why does k=1 give a different answer almost every pixel you drag, while k=9 barely changes?
Go deeper (free): StatQuest — k-nearest neighbors (video) ↗