Most text encoder blocks declare the guider as an expected component and decide from it whether to produce negative embeddings. We've started moving away from that — WanAnimate2TextEncoderStep doesn't declare one — but the codebase hasn't converged, so I want to write the rule down.
The guider should be optional for a text encoder block, so it behaves sensibly in both capacities:
Standalone — the block should not be aware of the guider at all. If a user passes a negative_prompt, they get negative_prompt_embeds back. Today they may not: the block asks the guider, and if that pipeline's guider spec happens to default to disabled, the negative prompt is silently dropped. That's not intuitive — you asked for something and got nothing back, for a reason belonging to a component you never used (see #13505).
Combined into the pipeline, with the denoiser — here the block should be aware of the guider setting: with the guider disabled it should not produce negative embeddings (nothing would consume them), and it should warn when a negative_prompt was supplied, so the user learns why it had no effect instead of silently getting nothing back.
wan_animate_2 is closest to this today, but it encodes the negative prompt anyway rather than warning and skipping, so even that one needs a pass:
# src/diffusers/modular_pipelines/wan_animate_2/encoders.py
block_state.negative_prompt_embeds = None
if components.requires_unconditional_embeds or block_state.negative_prompt is not None:
...Note ComponentSpec has no notion of an optional component today — declaring it means the block expects it — so part of this is deciding whether we formalize that, or keep the wan_animate_2 shape (don't declare it; consult requires_unconditional_embeds, which is False when no guider exists).
State of the codebase, so we know the size of the pass:
- declares a
guider spec, reads components.requires_unconditional_embeds — ernie_image, flux2, helios, hunyuan_video1_5, krea2, ltx, qwenimage, wan, z_image
- declares a
guider spec, reads it directly (components.guider.num_conditions > 1) — anima, stable_diffusion_xl
- no guider in the text encoder —
wan_animate_2 (by design), ltx2 and stable_diffusion_3 (denoise-only)
plus requires_unconditional_embeds copy-pasted into ten ModularPipeline subclasses with identical bodies (flux2 adds an is_distilled early return). Once settled it goes into .ai/modular.md — this came up writing those docs in #14452.
Opened by Claude (Opus 5) on behalf of @yiyixuxu.
Most text encoder blocks declare the guider as an expected component and decide from it whether to produce negative embeddings. We've started moving away from that —
WanAnimate2TextEncoderStepdoesn't declare one — but the codebase hasn't converged, so I want to write the rule down.The guider should be optional for a text encoder block, so it behaves sensibly in both capacities:
Standalone — the block should not be aware of the guider at all. If a user passes a
negative_prompt, they getnegative_prompt_embedsback. Today they may not: the block asks the guider, and if that pipeline's guider spec happens to default to disabled, the negative prompt is silently dropped. That's not intuitive — you asked for something and got nothing back, for a reason belonging to a component you never used (see #13505).Combined into the pipeline, with the denoiser — here the block should be aware of the guider setting: with the guider disabled it should not produce negative embeddings (nothing would consume them), and it should warn when a
negative_promptwas supplied, so the user learns why it had no effect instead of silently getting nothing back.wan_animate_2is closest to this today, but it encodes the negative prompt anyway rather than warning and skipping, so even that one needs a pass:Note
ComponentSpechas no notion of an optional component today — declaring it means the block expects it — so part of this is deciding whether we formalize that, or keep the wan_animate_2 shape (don't declare it; consultrequires_unconditional_embeds, which isFalsewhen no guider exists).State of the codebase, so we know the size of the pass:
guiderspec, readscomponents.requires_unconditional_embeds—ernie_image,flux2,helios,hunyuan_video1_5,krea2,ltx,qwenimage,wan,z_imageguiderspec, reads it directly (components.guider.num_conditions > 1) —anima,stable_diffusion_xlwan_animate_2(by design),ltx2andstable_diffusion_3(denoise-only)plus
requires_unconditional_embedscopy-pasted into tenModularPipelinesubclasses with identical bodies (flux2adds anis_distilledearly return). Once settled it goes into.ai/modular.md— this came up writing those docs in #14452.