CS492D: Diffusion Models and Their

Applications
Denoising Diffusion Implicit Models 2 /
Classifier-Free Guidance

LECTURE 6
MINHYUK SUNG
Fall 2024
KAIST

CS492(D): Diffusion Models and Their Applications (Fall 2024)

 RECAP: DDPM vs. DDIM
𝛼! 1 − 𝛼*!"# 𝛼*!"#𝛽! 1 − 𝛼*!"#
𝑞 𝒙!"#|𝒙! , 𝒙$ = 𝒩 𝒙! + 𝒙$ , 𝛽! 𝐈
1 − 𝛼*! 1 − 𝛼*! 1 − 𝛼*!

For each time step 𝑡 = 𝑇, … , 1, repeat:

#
1. Compute 𝒙$|! = 𝒙! − 1 − 𝛼*! 𝛆1 ( (𝒙! , 𝑡)
&!
'

'! #"&
'!"# & !"# )!
'
2. Compute 𝜇5 = 𝒙! + 𝒙$|! .
#"&'! #"&
'!

3. Sample 𝒛! ~𝒩 𝟎, 𝐈 .

#"&
'!"#
4. Compute 𝒙!"# = 𝜇5 + 𝛽! 𝒛! .
#"&
'!

CS492(D): Diffusion Models and Their Applications (Fall 2024) 2

 RECAP: DDPM vs. DDIM
𝒙! − 𝛼*! 𝒙$
𝑞* 𝒙!"# 𝒙! , 𝒙$ = 𝒩 𝛼*!"#𝒙$ + 1 − 𝛼*!"# − 𝜎!+ ; , 𝜎!+𝐈
1 − 𝛼*!

For each time step 𝑡 = 𝑇, … , 1, repeat:

#
1. Compute 𝒙$|! = 𝒙! − 1 − 𝛼*! 𝛆1 ( (𝒙! , 𝑡)
&!
'

& ! 𝒙$|!
𝒙! " '
2. Compute 𝜇5 = 𝛼*!"#𝒙$|! + 1 − 𝛼*!"# − 𝜎!+ ; .
#"&
'!
3. Sample 𝒛! ~𝒩 𝟎, 𝐈 .
4. Compute 𝒙!"# = 𝜇5 + 𝜎! 𝒛! .

CS492(D): Diffusion Models and Their Applications (Fall 2024) 3

 RECAP: DDPM vs. DDIM

Q. What is 𝜇! with respect to 𝒙! and 𝛆! for both DDPM and

DDIM cases?

CS492(D): Diffusion Models and Their Applications (Fall 2024) 4

 RECAP: DDPM vs. DDIM
1 1 − 𝛼! 1 − 𝛼*!"#
𝑞 𝒙!"#|𝒙! , 𝒙$ = 𝒩 𝐱! − 𝛆! , 𝛽! 𝐈
𝛼! 1 − 𝛼*! 1 − 𝛼*!

For each time step 𝑡 = 𝑇, … , 1, repeat:

# #"'!
1. Compute 𝜇5 = 𝐱! − 𝛆1 (𝒙 , 𝑡) .
'! '! ( !
#"&

2. Sample 𝒛! ~𝒩 𝟎, 𝐈 .

#"&
'!"#
3. Compute 𝒙!"# = 𝜇5 + #"&
'!
𝛽! 𝒛! .

CS492(D): Diffusion Models and Their Applications (Fall 2024) 5

 RECAP: DDPM vs. DDIM
𝒙! − 1 − 𝛼*! 𝛆!
𝑞* 𝒙!"# 𝒙! , 𝒙$ = 𝒩 𝜇5 = 𝛼*!"# + 1 − 𝛼*!"# − 𝜎!+ ; 𝛆! , 𝜎!+𝐈
𝛼*!

For each time step 𝑡 = 𝑇, … , 1, repeat:

'! 𝛆- & (𝒙! ,!)
𝒙! " #"&
1. Compute 𝜇5 = 𝛼*!"# + 1 − 𝛼*!"# − 𝜎!+ ; 𝛆1 ( (𝒙! , 𝑡).
&!
'
2. Sample 𝒛! ~𝒩 𝟎, 𝐈 .
3. Compute 𝒙!"# = 𝜇5 + 𝜎! 𝒛! .

CS492(D): Diffusion Models and Their Applications (Fall 2024) 6

 RECAP: DDIM

Q. Check out that DDPM is a special case of DDIM when

" "
1 − 𝛼)!#$
𝜎! = 𝜎!! = 𝛽! .
1 − 𝛼)!

CS492(D): Diffusion Models and Their Applications (Fall 2024) 7

 RECAP: DDIM
Controlling Stochasticity

% !"#
$#&
Let 𝜎! = 𝜂 𝛽! .
$#%
&!

• 𝜂 = 0: Deterministic process.
For the same 𝒙 ' ~𝒩 𝟎, 𝐈 , we always obtain the same 𝒙( .

• 𝜂 = 1: Same as DDPM.

CS492(D): Diffusion Models and Their Applications (Fall 2024) 8

 RECAP: DDIM

Q. What is the maximum 𝜎! ?

CS492(D): Diffusion Models and Their Applications (Fall 2024) 9

 RECAP: DDIM
𝒙! − 1 − 𝛼*! 𝛆!
𝑞* 𝒙!"# 𝒙! , 𝒙$ = 𝒩 𝜇5 = 𝛼*!"# + 1 − 𝛼*!"# − 𝜎!+ ; 𝛆! , 𝜎!+𝐈
𝛼*!

A. 1 − 𝛼#!"# − 𝜎!$ must be non-negative.

The maximum 𝜎! is 1 − 𝛼#!"# .

CS492(D): Diffusion Models and Their Applications (Fall 2024) 10

 RECAP: Accelerating Sampling Process
The reverse process for a sub-sequence time steps 𝜏 =
[𝜏$ , 𝜏" , … , 𝜏) ]:

𝑝* 𝒙+ = 𝑝 𝐱 ' : 𝑝* 𝐱 +$"# |𝐱 +$
!,$

CS492(D): Diffusion Models and Their Applications (Fall 2024) 11

 RECAP: Accelerating Sampling Process
• As smaller time steps are used, the quality of the generated
data can worsen.

• However, quality degradation is mitigated when the DDIM

reverse process becomes more deterministic.

CS492(D): Diffusion Models and Their Applications (Fall 2024) 12

 RECAP: DDIM Summary
• DDPM is a special case of DDIM.

• No need to retrain the noise predictor.

• The reverse process becomes deterministic when 𝜎! = 0.

• Quality degradation with fewer time steps is mitigated

when 𝜎! is small.

CS492(D): Diffusion Models and Their Applications (Fall 2024) 13

 Song et al., Denoising Diffusion Implicit Models, ICLR 2021.

RECAP: Accelerating Sampling Process

• When 𝜂 = 0 (deterministic),
• the quality does not get bad too much even when S = 10.
• The quality when S = 1000 is even better than DDPM!

CS492(D): Diffusion Models and Their Applications (Fall 2024) 14

 Assignment 2
• Assignment 2 is about changing DDPM to DDIM.

• It builds on your solution from Assignment 1. You can start

Assignment 2 when you finish Assignment 1!

CS492(D): Diffusion Models and Their Applications (Fall 2024) 15

 Classifier-Free Guidance (CFG)

Ho and Salimans, Classifier-Free Diffusion Guidance, NeurIPS 2021 Workshop.

CS492(D): Diffusion Models and Their Applications (Fall 2024) 16

 Ho and Salimans, Classifier-Free Diffusion Guidance, NeurIPS 2021 Workshop.

Classifier-Free Guidance (CFG)

How to make a conditional diffusion model?

Dog

CS492(D): Diffusion Models and Their Applications (Fall 2024) 17

 Classifier-Free Guidance (CFG)
Modify the noise prediction network to take the condition
(label) 𝑦 as an additional input:

𝛆> * 𝒙! , 𝑡 → 𝛆> * 𝒙! , 𝑦, 𝑡

CS492(D): Diffusion Models and Their Applications (Fall 2024) 18

 Classifier-Free Guidance (CFG)
We can jointly train the conditional and unconditional
diffusion models by introducing a null label 𝜙.

𝛆> * 𝒙! , 𝜙, 𝑡 : Noise prediction without condition.

CS492(D): Diffusion Models and Their Applications (Fall 2024) 19

 Classifier-Free Guidance (CFG)
How can we enhance the conditioning?

We can extrapolate the conditional noise 𝛆> * 𝒙! , 𝑦, 𝑡 from

the null-condition noise 𝛆> * 𝒙! , 𝜙, 𝑡 :

𝛆! * 𝒙! , 𝑦, 𝑡 = 1 + 𝜔 𝛆> * 𝒙! , 𝑦, 𝑡 − 𝜔>𝛆* 𝒙! , 𝜙, 𝑡

where 𝜔 ≥ 0.

CS492(D): Diffusion Models and Their Applications (Fall 2024) 20

 Classifier-Free Guidance (CFG)
How can we enhance the conditioning?

We can extrapolate the conditional noise 𝛆> * 𝒙! , 𝑦, 𝑡 from

the null-condition noise 𝛆> * 𝒙! , 𝜙, 𝑡 :

𝛆! * 𝒙! , 𝑦, 𝑡 = 1 + 𝜔 𝛆> * 𝒙! , 𝑦, 𝑡 − 𝜔>𝛆* 𝒙! , 𝜙, 𝑡

where 𝜔 ≥ 0.

CS492(D): Diffusion Models and Their Applications (Fall 2024) 21

 Meng et al., On Distillation of Guided Diffusion Models, CVPR 2023.

Classifier-Free Guidance (CFG)

CS492(D): Diffusion Models and Their Applications (Fall 2024) 22

 Classifier-Free Guidance (CFG)
• (+) Very easy to implement.

• (+) Versatile; can be used not only for labels but also for
any additional information (e.g., text descriptions).

• (−) The noise predictor needs to be evaluated twice in the

generation process.

CS492(D): Diffusion Models and Their Applications (Fall 2024) 23

 Assignment 2
Implement CFG based on the DDIM implementation.

Wildlife

Cat

Dog

CS492(D): Diffusion Models and Their Applications (Fall 2024) 24