Add First Block Cache support for Flux 2 - #14359
Conversation
Fixes huggingface#14280 `Flux2TransformerBlock` and `Flux2SingleTransformerBlock` were never registered in `TransformerBlockRegistry`, so `enable_cache(FirstBlockCacheConfig(...))` raised `ValueError: Model class ... not registered`. Registering them is not enough on its own: the model forward did the txt/img `cat` once between the two block loops, so the FBCache head block (double, 2-tuple of separate streams) and tail block (single, bare concatenated tensor) had incompatible outputs and the residual bridge failed with `TypeError: unsupported operand type(s) for -: 'Tensor' and 'list'`. - Register both Flux 2 block classes with the same indices as Flux v1 (`return_hidden_states_index=1`, `return_encoder_hidden_states_index=0`). - Move the `cat`/`split` into `Flux2SingleTransformerBlock.forward`, matching `FluxSingleTransformerBlock`. Every Flux 2 block now takes `(hidden_states, encoder_hidden_states)` and returns `(encoder_hidden_states, hidden_states)`. The block already carried `split_hidden_states` / `text_seq_len` parameters for this; nothing ever passed them, so they are removed. - Add `TestFlux2TransformerFBCCache` and `TestFlux2KleinPipelineFirstBlockCache`.
|
Rebased onto current main (6f2010e). The only conflict was in class TestFlux2KleinPipelineFirstBlockCache(Flux2KleinPipelineTesterConfig, FirstBlockCacheTesterMixin):
"""First Block Cache tests for the Flux2 Klein pipeline."""The import moved from Re-ran on the rebased tree (8x A40, sm_86):
The one |
Fixes #14280
Flux2TransformerBlockandFlux2SingleTransformerBlockwere never registered inTransformerBlockRegistry, soenable_cache(FirstBlockCacheConfig(...))raisedValueError: Model class ... not registered. Registering them is not enough on its own: the modelforward did the txt/img
catonce between the two block loops, so the FBCache head block (double,2-tuple of separate streams) and tail block (single, bare concatenated tensor) had incompatible
outputs and the residual bridge failed with
TypeError: unsupported operand type(s) for -: 'Tensor' and 'list'.Changes:
(
return_hidden_states_index=1,return_encoder_hidden_states_index=0).cat/splitintoFlux2SingleTransformerBlock.forward, matchingFluxSingleTransformerBlock. Every Flux 2 block now takes(hidden_states, encoder_hidden_states)and returns(encoder_hidden_states, hidden_states). The block alreadycarried
split_hidden_states/text_seq_lenparameters for this; nothing ever passed them, sothey are removed.
TestFlux2TransformerFBCCacheandFirstBlockCacheTesterMixinonFlux2KleinPipelineFastTests.This is direction (A) from the issue. Direction (B) would put one model's double/single boundary
into the generic
apply_first_block_cache.Verified on 8x A40 (sm_86), torch 2.13.0+cu126, bf16, with dummy models:
Tensor - listbridge error, sothe model change is load-bearing.
kv_cache_mode="extract",kv_cache_mode="cached". The Klein KV-cache path including ref-token modulation blending isunchanged.
cat: 288.0 / 288.7 ms per forward with the patch vs 287.6 / 287.7 mswithout, on an idle A40 at 8 double + 48 single blocks, dim 3072, 1024 image + 512 text tokens.
Noise floor.
tests/models/transformers/test_models_transformer_flux2.pyhas an identical failure set beforeand after (10
torch.compiletests that die in the inductor C++ backend on this box, unrelated).tests/hooks/and the Flux v1 cache tests pass.Not verified: no pretrained FLUX.2 checkpoint was loaded, so there is no image-quality check, and no
FBCache speedup numbers. All
torch.compiletests fail on this machine for an unrelated toolchainreason, before and after, so the compiled path is untested.
Flux2Pipeline(non-Klein) still doesnot open a
cache_context, so FBCache raisesNo context is setthere; that is a separate gap andI left it alone.
Reported by @sqhuang, whose issue also worked out the root cause and both candidate directions.