TracksStudyAbout
Sign in
Start studying
Module 12 · Generative AI Module contents
Module 12  ·  Card 7 / 18intermediate

Schedulers and Sampling Steps

Diffusion removes noise gradually, and the scheduler decides how much comes off at each step.

Fewer steps means a faster image, and below a threshold the result visibly falls apart.

Different schedulers reach usable quality at different step counts, which is mostly why they exist.

Early samplers needed hundreds of steps; modern ones produce good images in twenty or fewer.

Distilled models push it to a handful of steps by learning to take much larger jumps.

Back Next
Learn in depth ↓
Module 12 · Generative AI · 18 concepts
In depth  /  15 lines

Schedulers and Sampling Steps, in more detail

A diffusion model is trained to predict the noise present in a partially noisy image.

Generation runs that prediction repeatedly, starting from pure noise and cleaning it up in stages.

The scheduler, also called the sampler, decides how that sequence of stages is arranged.

It sets the noise level at each step and how the prediction is turned into the next image.

The number of steps is the most visible knob, because it maps directly onto generation time.

Each step is one full pass through the network, so twenty steps costs twenty passes.

Cut the steps too far and the output degrades in a characteristic way: smeared, incoherent, half-formed.

The original formulation needed hundreds of steps, which made generation slow and expensive.

DDIM reframed the process deterministically and produced usable images in a few dozen steps.

Later solvers treat denoising as a differential equation and use better numerical integration.

Those reach comparable quality in fewer steps, which is the practical reason to choose one.

Schedulers also differ in whether they inject randomness at each step or stay deterministic.

Deterministic samplers give reproducible output from a seed, which matters for iterating on a prompt.

Stochastic ones can produce more varied and sometimes more detailed results at higher step counts.

Distillation goes furthest, training a model to reproduce many steps in one, so a few steps suffice.

Deep dive

What is underneath Schedulers and Sampling Steps

The scheduler is a solver, not a setting

A trained diffusion model predicts noise at a given timestep. It does not say how to walk from pure noise to an image, and the scheduler is that walk.

dxdt=f(x,t)\frac{d\mathbf{x}}{dt} = f(\mathbf{x}, t)

The reverse process can be written as a differential equation, so sampling is numerical integration and a scheduler is a choice of integrator.

That reframing is why the number of steps became a dial rather than a property of the model: a better solver reaches the same solution in fewer, larger steps.

DDPM, the original walk

xt−1=1αt(xt−βt1−αˉtϵθ)+σtz\mathbf{x}_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left( \mathbf{x}_t - \frac{\beta_t}{\sqrt{1 - \bar{\alpha}_t}} \boldsymbol{\epsilon}_{\theta} \right) + \sigma_t \mathbf{z}

Each step removes a little predicted noise and adds a little fresh noise, following the same schedule the forward process used.

T=1000T = 1000

It is stochastic, so two runs from the same starting noise give different images, and it needs roughly as many steps as it was trained with.

DDIM, and why it is faster

xt−1=αˉt−1 x^0+1−αˉt−1 ϵθ\mathbf{x}_{t-1} = \sqrt{\bar{\alpha}_{t-1}} \, \hat{\mathbf{x}}_0 + \sqrt{1 - \bar{\alpha}_{t-1}} \, \boldsymbol{\epsilon}_{\theta}
x^0=xt−1−αˉt ϵθαˉt\hat{\mathbf{x}}_0 = \frac{\mathbf{x}_t - \sqrt{1 - \bar{\alpha}_t} \, \boldsymbol{\epsilon}_{\theta}}{\sqrt{\bar{\alpha}_t}}

DDIM estimates the fully clean image at every step and then re-noises it to the next level down, adding no randomness.

σt=0\sigma_t = 0

Being deterministic, it can skip timesteps: any subsequence of the training schedule is a valid path, so 50 steps out of 1,000 works.

It also makes the noise a latent code. The same seed gives the same image, and interpolating two seeds interpolates the images, which stochastic sampling cannot do.

The higher order solvers

DPM-Solver takes second and third order steps.

Euler's method is a first-order approximation. Using two or three network evaluations to estimate the curvature lets each step travel further without drifting.

20 steps≈quality of 200 first-order steps20 \text{ steps} \approx \text{quality of } 200 \text{ first-order steps}

That is where the current defaults come from, and it is why modern image tools produce a good image in 20 to 30 steps rather than a thousand.

One sampling run, in order

  1. Choose the number of steps and select that many timesteps from the training schedule.
  2. Start from noise, scaled by the scheduler's expected initial variance.
  3. Evaluate the network, once or several times depending on the solver order.
  4. Take the step the solver prescribes, moving to the next timestep.
  5. Repeat, and decode the final latent through the autoencoder if the model works in latent space.

What to expect from the step count

quality(n) saturates\text{quality}(n) \text{ saturates}

Gains are steep to about 20 steps, marginal to 50, and invisible past 100 on a modern solver.

Doubling the steps doubles the latency and the cost exactly, since every step is one or more full passes through the network.

guidance×steps\text{guidance} \times \text{steps}

Guidance multiplies that again, because classifier-free guidance evaluates the network twice per step unless the two passes are batched.

A step count tuned for one scheduler does not transfer to another, which is why comparisons across tools are usually comparing solvers rather than models.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
What does the scheduler control in a diffusion model?
Generative AI

18 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close