Video & Audio Generation
Video and audio generators are diffusion models with one extra ingredient: time. An image model denoises a single frame; a video model must denoise many frames that agree with each other. The fix is temporal attention — while each frame is being denoised, it attends to neighbouring frames, so a ball in frame 3 lands where frame 2 left it. Audio works the same way on a spectrogram: a picture of sound over time, denoised with the same machinery, then converted back to a waveform.
Watch six frames play as a loop. Toggle temporal attention off and each frame is denoised independently — every still looks plausible, but the ball teleports between frames and the video flickers.
Check yourself
Why can a model that makes perfect single images still produce unwatchable video?
Go deeper (free): Lilian Weng — Video diffusion models ↗