Text-to-Video

Text-to-video extends diffusion across time as well as space.

A video is a sequence of frames, and generating each independently gives a flickering mess.

The hard problem is consistency, since the same object must persist between frames.

Temporal attention layers let frames attend to one another.

That is how a coat stays the same colour from one second to the next.

Some models generate keyframes first and interpolate between them.

Others treat the video as a single volume of space and time.

Cost scales roughly with the number of frames, so a few seconds is expensive.

Training data is scarcer than for images and much heavier to process.

Captions for video are also poorer, since describing motion is harder than describing a scene.

Quality improved remarkably fast, from seconds of warping shapes to convincing short clips.

Long coherent video remains difficult, since drift accumulates over time.

Physics is approximate, and objects pass through each other or change size.

Control is limited, and asking for a specific camera move is unreliable.

The obvious misuse risk is why watermarking and provenance work matters here.