From 7c99742b9cb656581ca58c8e3d18bbdcc78fd55a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 15 Aug 2026 15:43:42 +0000 Subject: [PATCH 1/5] spec(LTX25-IC-LORA): the adapter this port could not read, and the second blocker the refusal did not name (#923) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The row was scoped as "port ICLoraPipeline so video-to-video works". Reading the tree first says that is two pieces of work, not one, and only the first is what the reference refusal blames. `ltx2_video.cpp:1336-1346` refuses reference conditioning because the IC-LoRA's `downscale_factor` and `temporal_scale_factor` live in the adapter's metadata and nothing here reads it. True, and this row closes it. But thirty lines earlier the last-frame refusal documents the other cause: every APPENDING conditioning item is blocked on token-append machinery the phase loop does not have, and `VideoConditionByReferenceLatent` appends. So closing the metadata half and lifting the refusal would ship a wrong render. §6 sizes that second blocker and §7 owes it; the DiT turns out not to be in the way, the fixed-width phase loop is, and the same machinery is what blocks the last-frame keyframe arm. Two findings worth the spec being written before the code. The tree already dequantizes FP8 and NVFP4 to bf16 inside one function, so a single fusion hook serves every arm and no quantizer is needed — the tree has none. And the aggregation dtype is bfloat16 in all four of upstream's fuse rules, which is precisely the too-wide-dtype trap a token gate cannot see, so it gets its own mutation. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 1 + .agents/specs/ltx25-ic-lora.md | 317 +++++++++++++++++++++++++++++++++ 2 files changed, 318 insertions(+) create mode 100644 .agents/specs/ltx25-ic-lora.md diff --git a/.agents/issue-index.md b/.agents/issue-index.md index bc2b285ad..30195521e 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -239,3 +239,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#907](https://github.com/mudler/vllm.cpp/issues/907) | `BACKEND-CUDA-COMP-CORE` | Five tests are red on dgx.casa (GB10, sm_121a) at `main`, proven PRE-EXISTING by a control build at `10b8bbdaa`: `test_capi` (SIGSEGV in an ABI v8 custom-logits-processor case, plausibly [#547](https://github.com/mudler/vllm.cpp/issues/547) or [#844](https://github.com/mudler/vllm.cpp/issues/844)), `test_cuda_ops` 439/440, `test_linear_method` 83/85, `test_ops_gdn` 4899/4900 ([#614](https://github.com/mudler/vllm.cpp/issues/614)), `test_qwen3_5_gdn_spec_routing` 119/123. Three of the five had no issue at all, which is why this exists | bug | | [#912](https://github.com/mudler/vllm.cpp/issues/912) | `ENG-EXPERT-STREAM` | Stream routed experts from NVMe so a model larger than device memory runs. Target `Qwen/Qwen3.8-2.4T-A95B`, REGISTERED against `Qwen3_5MoeForCausalLM` and blocked only on capacity: 370 GiB at UD-Q1_0 against 128 GB of unified memory on GB10. The only one of the three offload rows that helps on a unified-memory host, because `ENG-WEIGHT-OFFLOAD` and `ENG-HYBRID-PLACEMENT` both move bytes inside one physical pool. Cheaper than the spec assumed: on the GGUF path the mmap'd file already IS the bank and the per-expert slicer landed 2026-07-22 ([#824](https://github.com/mudler/vllm.cpp/issues/824)) | feature | | [#911](https://github.com/mudler/vllm.cpp/issues/911) | — | A `file:line` anchor into a file the row is ITSELF editing is stale by default, and spec BODIES are checked by nothing: `check-agent-record.py`'s `MATRIX_PATHS` (`:521`, `:529-530`) covers the five matrices, `feature-matrix.md` and `specs/model-family-inventory.md`, leaving 4772 line-carrying citations across 315 `.agents/specs/*.md` unexamined (positive control: 2314 line-less `.cpp` mentions match the same shape). `ltx25-prompt-adaln.md` shipped EIGHT stale repo-local anchors across two repair commits, moved by its own `020381676` and by `98f8e046d` (#658), then SEVEN more that were correct at `00613767d` and wrong at the merge of `origin/main`, because `0785cfc4d` (#882) added 70 lines to `ltx2_video.cpp` and 306 to `test_ltx2_video.cpp` ahead of every one. The obvious checker is a TAUTOLOGY — reading the span out of the file it validates reports 27 of 27 fresh on the same tree where reading the spans against their CLAIMS finds seven stale. Remedy is already in use and unwritten: `path:NN @ ` for a historical claim, claim-sourced uniqueness re-derivation for a live one, re-run after the merge. Narrower than [#632](https://github.com/mudler/vllm.cpp/issues/632) on surface and sharper on mechanism. Listed under `## Owed` in [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md) | bug | +| [#923](https://github.com/mudler/vllm.cpp/issues/923) | `LTX25-IC-LORA` | Port upstream `ICLoraPipeline` (`ltx-pipelines/ic_lora.py` @ `fd4ded7f`): read an IC-LoRA adapter and its `__metadata__`, fuse the delta into the DiT at load, and expose it through the `ltx-2.5` load extras and `ltx2-gen --lora`. The conditioning half already landed — `Ltx2ConditionVideoByReference` (`ltx2_conditioning.cpp:221`) and `Ltx2ConvVideoEncode` are ported and gated, and `ref_video_dir` reaches the engine as a dir of `frame_%06d.ppm` — so the gap is the adapter path the reference refusal names at `ltx2_video.cpp:1341-1343`. The tree's `include/vllm/lora/` is NOT this mechanism: it is vLLM's runtime punica brick (f32, slot-indexed, `LinearMethodBase`), where LTX fuses at LOAD (`loader/fuse_loras.py:119-150`), so this row does not route through it. One hook after `MaterializeDitTensor` (`ltx2_loader.cpp:424-499`) serves the F32, BF16, FP8 and NVFP4 arms at once, because both quantized branches already `return vt::DType::kBF16`. Retires the `kLoraFusion` marker, which [#691](https://github.com/mudler/vllm.cpp/issues/691) predicted would go false exactly here | feature | diff --git a/.agents/specs/ltx25-ic-lora.md b/.agents/specs/ltx25-ic-lora.md new file mode 100644 index 000000000..941098e60 --- /dev/null +++ b/.agents/specs/ltx25-ic-lora.md @@ -0,0 +1,317 @@ +# LTX-2.5 IC-LoRA — the adapter this port could not read, and the second blocker the refusal did not name + +**Row:** `LTX25-IC-LORA` (a row of the `#644` full-port campaign). +**Issue:** [#923](https://github.com/mudler/vllm.cpp/issues/923). +**Branch:** `row/LTX25-IC-LORA`. +**Parent spec:** [`ltx-2-5.md`](ltx-2-5.md) — operator-owned, NOT edited by this row. +**Upstream root (primary):** Lightricks/LTX-2 @ `fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca`, +verified at the working checkout `/home/mudler/_git/LTX-2` with a clean tree. +**Entry point ported from:** `packages/ltx-pipelines/src/ltx_pipelines/ic_lora.py`. + +vLLM is not the reference here and cannot be: vLLM-Omni carries no LTX-2.5, and +`ICLoraPipeline` has no vLLM analogue at all. The oracle is the `ltx-2` pin +recorded in [`../oracles/`](../oracles/), used as the secondary oracle the rule +allows where vLLM implements nothing. + +--- + +## 0. What is claimed, and what is not + +This row builds **the LoRA adapter path**: reading an IC-LoRA safetensors file, +reading its metadata, and fusing its delta into the DiT at load, on every dtype +arm the loader materializes. That is the thing the reference refusal names as +absent, and it is the shared prerequisite for `ic_lora.py`, `hdr_ic_lora.py` and +`dubit.py` alike. + +It does **not** ship video-to-video. Three statements up front so nothing is +discovered later. + +1. **The reference-video arm has a SECOND blocker, and the brief for this row did + not account for it.** The refusal at `ltx2_video.cpp:1336-1346` names one + cause — the LoRA metadata is unread — and that cause is true and is closed + here. But `ltx2_video.cpp:1305-1335`, thirty lines earlier, documents the + other one: every *appending* conditioning item is blocked on token-append + machinery the engine does not have. `VideoConditionByReferenceLatent` appends + (`reference_video_cond.py:97-100`, ported at `ltx2_conditioning.cpp:265` via + `AppendTokens`), so the reference arm needs it too. §6 sizes it and §7 records + it as owed. Closing only the metadata half and lifting the refusal would ship + a wrong render. +2. **No render-quality claim, and no real-weights claim.** The evidence here is + numeric parity against upstream's fusion arithmetic plus mutation gates on + synthetic fixtures. No IC-LoRA checkpoint was fused on real weights, because + this row ran without GPU authority (`dgx.casa` was under a long render). +3. **No speed claim.** The parent spec's `PENDING` speed axis is untouched. + +## 1. The gap, re-verified against the current tree + +Re-derived at `95b7366`, not read out of the record. + +| claim | check | result | +|---|---|---| +| the reference conditioning MATH landed | `Ltx2ConditionVideoByReference`, `src/vllm/model_executor/models/ltx2_conditioning.cpp:221` | present, gated by `test_ltx2_vae` | +| the video VAE ENCODER landed and is reachable | `Ltx2ConvVideoEncode`, called at `src/vllm/multimodal/ltx2_video.cpp:1645` | present, reached | +| `ref_video_dir` reaches the engine | `src/capi/vllm_c.cpp:1642`, `include/vllm/multimodal/video_engine.h:96` | present; a DIR of `frame_%06d.ppm`, so no codec is involved | +| arbitrary safetensors `__metadata__` is readable | `SafetensorsFile::Metadata()`, `include/vllm/model_executor/model_loader/safetensors_reader.h:55-57` | present, returns the whole map | +| anything reads a LoRA file | `grep -rn "lora_A\|lora_B" src include` | **only `include/vllm/lora/` — see below** | +| anything fuses a LoRA into LTX weights | `grep -rn kLoraFusion src include` | **only the refusal, `ltx2_pipeline.cpp:1178`** | + +**The tree's existing `include/vllm/lora/` is not this mechanism and this row does +not route through it.** It is vLLM's runtime *punica* subsystem — f32, +slot-indexed, per-token, hanging off `LinearMethodBase` for text decode +(`punica.h:15-24`, `layers.h:26-28`). LTX does not apply an adapter at runtime; +it **fuses at load** (`loader/fuse_loras.py:119-150`). Mirroring vLLM's LoRA here +would mirror the wrong upstream. Recorded as a deliberate divergence from the +shared-seam preference, with the reason: the seam cannot express the behaviour, +which is exactly the case AGENTS.md allows extending or bypassing for. + +## 2. Upstream anchors + +Every anchor below was re-derived at the final tree and asserted unique; §5.3 +records the method and the result. + +### 2.1 The pipeline + +| what | anchor | +|---|---| +| `ICLoraPipeline.__init__` | `ic_lora.py:71-173` | +| stage 1 takes the LoRAs, **stage 2 takes none** | `ic_lora.py:104-125` (`loras=tuple(loras)` at `:108` against `loras=()` at `:119`) | +| reference scale factors read from LoRA metadata | `ic_lora.py:150-173` | +| conflicting factors across LoRAs raise | `ic_lora.py:158-163` and `:167-172` | +| `__call__` | `ic_lora.py:175-349` | +| `conditioning_attention_strength` range check | `ic_lora.py:230-233` | +| reference appended LAST, after the image conditionings | `ic_lora.py:377-402` | +| the CLI's `--lora` | `utils/args.py:600-611` | + +### 2.2 The adapter format and its fusion + +| what | anchor | +|---|---| +| `LoraPathStrengthAndSDOps(path, strength, sd_ops)` | `loader/primitives.py:160-167` | +| key shape `.lora_A.weight` / `.lora_B.weight` → `.weight` | `loader/fuse_loras.py:183-186` and `:196-198` | +| the delta: `sum((B * strength) @ A)` | `loader/fuse_loras.py:99-116` | +| **aggregation dtype is bfloat16** | `loader/fuse_loras.py:71` (`bf16_fuse_rule`) | +| the bf16 fuse: `deltas.add_(weight)` then cast to the weight dtype | `loader/fuse_loras.py:61-68` | +| a LoRA naming a key the model lacks is SKIPPED, not an error | `loader/fuse_loras.py:135-137` | +| metadata `reference_downscale_factor`, default 1 | `iclora_utils.py:30-38` | +| metadata `reference_temporal_scale_factor`, default 1 | `iclora_utils.py:41-49` | + +### 2.3 The quantized arms + +Upstream carries a fuse rule per policy, and **all four aggregate in bfloat16**: + +| arm | rule | anchor | +|---|---|---| +| bf16 | add, cast back | `loader/fuse_loras.py:61-71` | +| fp8 scaled-mm | dequant by `weight_scale`, add in f32, **re-quantize** with a fresh scale | `quantization/fp8_scaled_mm.py:167-189` | +| fp8 cast | fused add-round (stochastic on CUDA+Triton) | `quantization/fp8_cast.py:204-239` | +| NVFP4 | dequant to bf16, add, **re-quantize**, emit weight + both scales | `quantization/nvfp4/fuse.py:13-50` | + +## 3. Design + +### 3.1 The insertion point, and why every arm is served by one of them + +Both quantized arms in this tree **dequantize to bf16 inside one function** before +anything else sees a byte. `MaterializeDitTensor` +(`src/vllm/model_executor/models/ltx2_loader.cpp:424-499`) has four branches — +F32 memcpy, BF16 memcpy, `DequantFp8ToBf16` at `:461-469`, and +`Ltx2DequantNvfp4ToBf16` at `:470-498` — and the last two both `return +vt::DType::kBF16`. The header states the policy outright: "The default +materialization is **bf16**, which is the checkpoint's own model dtype" +(`ltx2_loader.h:76-81`). + +So the fusion hook goes **immediately after `MaterializeDitTensor` returns**, and +one hook covers FP8, NVFP4, BF16 and F32. It is reached from both callers: +`Ltx2LoadDitFromSafetensors` (`ltx2_loader.cpp:614-621`) and +`Ltx2StreamDitToDevice` (`:667-684`). On the device arm it runs *before* +`backend.Copy`, which preserves that arm's stated invariant that one host buffer +is live at a time (`:668-670`). + +**This is a deliberate divergence from upstream's per-arm rules, and it is forced +by an existing design rather than chosen here.** Upstream re-quantizes because it +keeps FP8/NVFP4 weights resident for its quantized kernels; this tree does not +keep them, so there is nothing to re-quantize into. The consequence is recorded +rather than hidden: our fused weight **skips upstream's lossy quantize round +trip** and is therefore slightly more precise than upstream's on the FP8 and +NVFP4 arms. It costs no extra bytes — the weight was already bf16 — so this is +not the too-wide-dtype failure that rule guards against. The tree has no FP8 or +NVFP4 *quantizer* at all (`grep` for one returns nothing; only `DequantFp8ToBf16` +and `Ltx2DequantNvfp4ToBf16` exist), so mirroring the round trip is not available +to be chosen. + +### 3.2 The memory format + +Mirrored deliberately, per `.agents/porting.md` §"Mirror the memory format". + +| ask | upstream answer | here | +|---|---|---| +| what dtype does the delta accumulate in? | **bfloat16** — every one of the four `FuseRule`s sets `aggregation_dtype=torch.bfloat16` | bf16 | +| is the first product rounded differently from the rest? | yes: `matmul(B * strength, A).to(dtype)` first, `addmm_(B, A, alpha=strength)` after (`fuse_loras.py:110-116`) | mirrored | +| what dtype is the fused weight stored as? | the weight's own dtype | the materialized dtype, unchanged | + +An f32 accumulator would be the exact defect that rule exists for: the tokens +would still match and every golden would still pass. It is bf16 here on purpose, +and §5.2 mutates that choice to prove the gate can see it. + +### 3.3 Surface + +- Load extras `lora_path` and `lora_strength` on the `ltx-2.5` family, mirroring + upstream's `(path, strength)` pair. LoRAs are a **constructor** argument + upstream (`ic_lora.py:104-114`), not a `__call__` argument, so a load extra is + the faithful shape and a per-generation field would not be. +- `--lora PATH [STRENGTH]` on `ltx2-gen`, mirroring `utils/args.py:600-611` and + its `DEFAULT_LORA_STRENGTH`. +- No ABI change. Both extras ride the existing `extra_keys`/`extra_values` + parallel arrays, which `include/vllm.h:926-927` records as existing for exactly + this. +- **Exactly one LoRA**, with a second refused by name. Upstream's own `dubit.py` + enforces the same (`dubit.py:364-365`) and `hdr_ic_lora.py` takes exactly one + (`hdr_ic_lora.py:271-272`). N-LoRA fusion is recorded as owed in §7 rather than + half-built; the conflict-detection loop this row ports (`ic_lora.py:155-173`) + is what N-LoRA needs and is written to take a list already. + +### 3.4 `kLoraFusion` stops being a marker + +`Ltx2UnportedPipelineFeature::kLoraFusion` is classified as a +declared-out-of-scope marker and its message asserts `DECLARED, NOT REQUESTABLE` +(`ltx2_pipeline.cpp:1178-1181`). Once `lora_path` exists that sentence is false. +Issue [#691](https://github.com/mudler/vllm.cpp/issues/691) records that the +ledger test gates the message **text** and not the property, and predicts this +exact divergence in its own words. So the enumerator is **removed**, not +reclassified: there is no longer an unported LoRA-fusion feature to name, and a +refusal for a served capability is worse than no refusal. + +## 4. Risks + +| risk | mitigation | +|---|---| +| the fused delta lands on the wrong tensor because LoRA keys carry a ComfyUI prefix the contract strips | normalize onto `Ltx2TensorSpec::name` explicitly and **refuse by name** when a LoRA names a module the contract lacks after normalization, rather than silently skipping — see §4.1 for where this deliberately departs from upstream | +| a LoRA that touches nothing at all fuses green and renders identically | the adapter load refuses when **zero** contract tensors matched; a no-op LoRA is a user error, not a successful load | +| editing `ltx2_video.cpp` above line 1100 shifts the gated READER ANCHORS | the new readers go below the existing ones where possible; the anchors are re-derived at the final tree and the gate is run | +| `docs/FEATURES.md` is a keyed record other agents are editing concurrently | reapply by key and diff unrelated keys byte-for-byte before commit | + +### 4.1 One deliberate departure + +Upstream **skips** a LoRA key whose target weight is absent +(`fuse_loras.py:135-137`, `if original_weight is None: continue`). It can afford +that: its state dict is the whole model. Here the contract is a fixed enumerated +set with unported modules already stripped, so a skip would silently absorb both +a genuinely-inapplicable key *and* a misnamed one, and the second is the failure +this project keeps paying for. The load therefore refuses by name on an unmatched +key. This is a divergence, it is argued here and in the commit, and it is the +narrower behaviour. + +## 5. Tests and gates + +### 5.1 Ported from upstream — there is nothing to port, and that is measured + +The rule is to port the upstream tests in the same change. **The pinned upstream +ships none, anywhere in the repository**, so the obligation is discharged as +not-applicable rather than skipped. Measured at +`fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca`, with a positive control so a null +result cannot be a wrong search term: + +| probe | result | +|---|---| +| `find . -name 'test_*.py' -o -name '*_test.py'` | **0** | +| `find . -type d -name 'test*'` | **0** | +| `find . -name conftest.py` | **0** | +| `grep -rl "import pytest\|import unittest" --include='*.py' .` | **0** | +| *control:* `find . -name '*.py'` | **280** | + +The control is what makes the zeros admissible: the same traversal that found no +tests found 280 Python files. So the fusion arithmetic is gated against +upstream's expression read from source (`fuse_loras.py:99-116`) and executed by +hand, which is stated as what it is — a source-derived value gate, not a ported +test. + +### 5.2 Written here, each RED first + +| case | proves | +|---|---| +| the fused weight equals `W + (B*strength) @ A` | the arithmetic, against values computed from the upstream expression | +| the accumulator is bf16, not f32 | a rank-2 adapter whose f32 and bf16 accumulations differ in the stored bf16 result; **this is the memory-format gate** | +| `strength` scales the delta linearly | strength is read, not ignored | +| a LoRA naming an absent module refuses by name | §4.1 | +| a LoRA matching zero tensors refuses by name | §4 | +| the metadata factors are read, absent ⇒ 1 | `iclora_utils.py:30-49` | +| a second `lora_path` refuses by name | §3.3 | +| **the FP8 arm fuses** — an FP8 fixture plus a LoRA changes the materialized weight | that the hook is after dequant, on the arm most users run | +| **the NVFP4 arm fuses** | the same for NVFP4 | + +### 5.3 Reachability + +The production entry point is `vllm_video_engine_load` → `LoadVideoEngine` → +`Ltx2VideoEngine::Load` → `Ltx2LoadDitFromSafetensors`. The smallest failing test +enters **there**, through `LoadVideoEngine` with a `lora_path` load extra on the +engine fixture, and asserts the rendered conditioning digest moves. The +reachability mutation deletes the fusion call site in `ltx2_loader.cpp` and +re-runs that case; a green gate would be the finding. + +### 5.4 Gate + +``` +cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DVLLM_CPP_CUDA=OFF +cmake --build build -j6 +ctest --test-dir build -j4 --output-on-failure +``` + +Known-red on `main` at the base SHA `95b7366`, proven pre-existing by running +`scripts/agent-preflight.sh` on a byte-for-byte unmodified tree +(`git status --porcelain` empty) before any edit: the seven-checker `#873` family +— `check-release-binary-contract`, `check-release-workflow`, +`check-test-registration`, `test_check_release_binary_contract`, +`test_release_manifest`, `test_release_pipeline`, `test_check_test_registration`. + +## 6. The second blocker, sized + +Recorded here so the next agent does not re-derive it. + +Serving a reference video needs the engine's phase loop to carry a **grown** token +sequence and trim it back. What is and is not in the way: + +- **The DiT is not in the way.** `Ltx2ModalityInput::tokens` is a per-call field, + and the DiT already accepts a self-attention strength mask + (`ltx2.h:458-462`, implemented `ltx2_dit.cpp:588-593`, mirroring + `_prepare_self_attention_mask`, `transformer_args.py:208-237`). +- **The conditioning item is not in the way.** It is ported and gated. +- **`Ltx2LatentState` having no attention-mask field is not in the way for the + DEFAULT arm.** At `conditioning_attention_strength == 1.0` with no mask, + upstream computes `attn_mask = None` (`iclora_utils.py:157-160`) and + `update_attention_mask` returns `None` (`mask_utils.py:141-143`), so the + default arm needs no mask at all. +- **The phase loop is in the way.** One `Ltx2VideoTokenCount(vshape, 1)` feeds + the sigma schedule, the `Ltx2ModalityInput` and `Ltx2VideoUnpatchify` + (`ltx2_video.cpp:1305-1322` documents this for the keyframe arm, and it is the + same obstruction). It must instead carry `state.tokens` through denoise and + trim to the target count before unpatchify, mirroring `clear_conditioning` + (`ltx_core/tools.py:88-105`). + +That work is **shared with the last-frame keyframe arm**, which is blocked on the +identical machinery. It is therefore its own row rather than a tail of this one. + +## 7. Owed + +Each is owed by this row and named in the commit and pull request bodies. + +| owed | issue | +|---|---| +| token-append grow-and-trim in the phase loop; until it lands, the reference-video and reference-image arms stay refused, with the refusal rewritten to name this cause instead of the metadata one this row closed | filed as part of this row's delivery, see the pull request body | +| the `conditioning_attention_mask` / `conditioning_attention_strength < 1.0` arm, which needs `Ltx2LatentState` to carry a mask and `build_attention_mask`'s block structure (`mask_utils.py:170-243`) | same | +| N-LoRA fusion (more than one adapter) | same | +| GGUF k-quant LoRA fusion — not applicable rather than owed: the LTX-2.5 DiT ships FP8 and NVFP4, and no GGUF LTX DiT exists to fuse into | n/a | +| a real-weights IC-LoRA fusion measurement | blocked on GPU authority; this row had none | + +## 8. Stop conditions + +- Stop and report `NEEDS_DECISION` if closing the metadata half would require + lifting the reference refusal before the token-append machinery exists. **This + fired**; §0.1 and §6 are the result, and the refusal is rewritten rather than + lifted. +- Stop if `docs/FEATURES.md` cannot be reapplied by key with unrelated keys + byte-identical. +- Do not use the GPU. `dgx.casa` was under a long render for this row's duration. + +## Now + +`ACTIVE` — the adapter path is implemented and gated; the reference arm stays +refused on the cause named in §6. From 84c03c0e06ae49a4bd06037409fc12a908e98194 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 15 Aug 2026 16:51:46 +0000 Subject: [PATCH 2/5] feat(LTX25-IC-LORA): read the IC-LoRA adapter, fuse it into every arm, and stop blaming the metadata (#923) The reference-video refusal said this port could not read an IC-LoRA's `downscale_factor` and `temporal_scale_factor`. That was true. It is now false, and the refusal is rewritten rather than deleted, because a SECOND cause it never named still holds the arm shut. What lands is the adapter path: reading `.lora_A.weight` / `.lora_B.weight` pairs, resolving them onto the DiT contract through upstream's ComfyUI prefix strip, reading the file's `__metadata__`, and fusing `(B * strength) @ A` into the weights at load. Reachable through the `lora_path` / `lora_strength` load extras and `ltx2-gen --lora PATH [STRENGTH]`, mirroring upstream's own constructor-argument shape: `DiffusionStage.from_checkpoint` takes the LoRAs, so a per-generation field would promise something fusion cannot do. One hook serves the F32, BF16, FP8 and NVFP4 arms, because both quantized branches of `MaterializeDitTensor` already return bf16. That is a deliberate divergence: upstream re-quantizes after adding the delta to keep its packed weights resident, this tree keeps bf16 and carries no FP8 or NVFP4 quantizer to round-trip with. Our fused weight therefore skips upstream's lossy round trip at no extra bytes, and both quantized arms are gated against the loader's own unfused output rather than against an independent dequantization. The accumulator is bfloat16, which is what all four upstream fuse rules declare. An f32 accumulator would be MORE precise and no golden could ever fail on it, so it has its own case, built so the two orders separate by a whole bf16 step instead of by a rounding that a bf16 store puts straight back. The first version of that case compared rank-1 values and passed under both arms; the comment on it now says so. An adapter naming a module the contract does not bind REFUSES here where upstream skips it. Upstream can afford a skip because its state dict is the whole model; this contract is a fixed enumerated set, so a skip would absorb a misnamed key and an inapplicable one alike. `kLoraFusion` is retired, not reclassified. It carried DECLARED, NOT REQUESTABLE - an assertion that no load extra asks for LoRA fusion - and one now does. #691 predicted this drift in its own words and records that the ledger test gates the message text rather than the property; the compiler caught it here, which is weaker than what #691 asks for and does not close it. Reachability is proven on the rendered pixels, not on `last_conditioning()`: the conditioning trace is filled before the denoise loop, so it cannot see a fused weight, and a first attempt comparing it found every arm identical for that reason rather than because the LoRA did nothing. Two defects found in this row's own work and fixed here rather than filed. The reader kept one key per target, so a malformed A factor was reported under the B factor's name - "refuse BY NAME" defeated by the message itself - and the case that would have caught it checked only the dtype. Both are now per-side and mutation-proven. `docs/FEATURES.md` and `docs/USAGE.md` ride with the change that made them stale, which is also what `check-public-doc-tables` and the doc checkpoint require. FEATURES had disagreed with itself: `:165` said LoRA was `refused` while `:337` correctly said `Declared, not requestable`. Both keys move together. Each edit was applied by key with an asserted single match and the unrelated keys then proved byte-identical - FEATURES 366 of 368 lines untouched, USAGE 3117. Owed and named in the pull request body: #930 for the token-append grow-and-trim that the reference and last-frame arms both wait on, and #932 for the sub-1.0 attention-strength arm and N-adapter fusion. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 2 + .agents/specs/ltx25-ic-lora.md | 16 +- CMakeLists.txt | 4 + docs/FEATURES.md | 4 +- docs/USAGE.md | 36 +- examples/ltx2_gen/main.cpp | 14 + .../vllm/model_executor/models/ltx2_loader.h | 27 + .../vllm/model_executor/models/ltx2_lora.h | 186 ++++++ .../model_executor/models/ltx2_pipeline.h | 24 +- include/vllm/multimodal/ltx2_video.h | 16 + .../model_executor/models/ltx2_loader.cpp | 71 ++ src/vllm/model_executor/models/ltx2_lora.cpp | 374 +++++++++++ .../model_executor/models/ltx2_pipeline.cpp | 4 - src/vllm/multimodal/ltx2_video.cpp | 86 ++- tests/CMakeLists.txt | 1 + tests/vllm/models/test_ltx2_loader.cpp | 214 ++++++ tests/vllm/models/test_ltx2_lora.cpp | 607 ++++++++++++++++++ tests/vllm/models/test_ltx2_pipeline.cpp | 8 +- tests/vllm/multimodal/test_ltx2_video.cpp | 265 +++++++- 19 files changed, 1918 insertions(+), 41 deletions(-) create mode 100644 include/vllm/model_executor/models/ltx2_lora.h create mode 100644 src/vllm/model_executor/models/ltx2_lora.cpp create mode 100644 tests/vllm/models/test_ltx2_lora.cpp diff --git a/.agents/issue-index.md b/.agents/issue-index.md index 30195521e..a2943ba7c 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -240,3 +240,5 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#912](https://github.com/mudler/vllm.cpp/issues/912) | `ENG-EXPERT-STREAM` | Stream routed experts from NVMe so a model larger than device memory runs. Target `Qwen/Qwen3.8-2.4T-A95B`, REGISTERED against `Qwen3_5MoeForCausalLM` and blocked only on capacity: 370 GiB at UD-Q1_0 against 128 GB of unified memory on GB10. The only one of the three offload rows that helps on a unified-memory host, because `ENG-WEIGHT-OFFLOAD` and `ENG-HYBRID-PLACEMENT` both move bytes inside one physical pool. Cheaper than the spec assumed: on the GGUF path the mmap'd file already IS the bank and the per-expert slicer landed 2026-07-22 ([#824](https://github.com/mudler/vllm.cpp/issues/824)) | feature | | [#911](https://github.com/mudler/vllm.cpp/issues/911) | — | A `file:line` anchor into a file the row is ITSELF editing is stale by default, and spec BODIES are checked by nothing: `check-agent-record.py`'s `MATRIX_PATHS` (`:521`, `:529-530`) covers the five matrices, `feature-matrix.md` and `specs/model-family-inventory.md`, leaving 4772 line-carrying citations across 315 `.agents/specs/*.md` unexamined (positive control: 2314 line-less `.cpp` mentions match the same shape). `ltx25-prompt-adaln.md` shipped EIGHT stale repo-local anchors across two repair commits, moved by its own `020381676` and by `98f8e046d` (#658), then SEVEN more that were correct at `00613767d` and wrong at the merge of `origin/main`, because `0785cfc4d` (#882) added 70 lines to `ltx2_video.cpp` and 306 to `test_ltx2_video.cpp` ahead of every one. The obvious checker is a TAUTOLOGY — reading the span out of the file it validates reports 27 of 27 fresh on the same tree where reading the spans against their CLAIMS finds seven stale. Remedy is already in use and unwritten: `path:NN @ ` for a historical claim, claim-sourced uniqueness re-derivation for a live one, re-run after the merge. Narrower than [#632](https://github.com/mudler/vllm.cpp/issues/632) on surface and sharper on mechanism. Listed under `## Owed` in [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md) | bug | | [#923](https://github.com/mudler/vllm.cpp/issues/923) | `LTX25-IC-LORA` | Port upstream `ICLoraPipeline` (`ltx-pipelines/ic_lora.py` @ `fd4ded7f`): read an IC-LoRA adapter and its `__metadata__`, fuse the delta into the DiT at load, and expose it through the `ltx-2.5` load extras and `ltx2-gen --lora`. The conditioning half already landed — `Ltx2ConditionVideoByReference` (`ltx2_conditioning.cpp:221`) and `Ltx2ConvVideoEncode` are ported and gated, and `ref_video_dir` reaches the engine as a dir of `frame_%06d.ppm` — so the gap is the adapter path the reference refusal names at `ltx2_video.cpp:1341-1343`. The tree's `include/vllm/lora/` is NOT this mechanism: it is vLLM's runtime punica brick (f32, slot-indexed, `LinearMethodBase`), where LTX fuses at LOAD (`loader/fuse_loras.py:119-150`), so this row does not route through it. One hook after `MaterializeDitTensor` (`ltx2_loader.cpp:424-499`) serves the F32, BF16, FP8 and NVFP4 arms at once, because both quantized branches already `return vt::DType::kBF16`. Retires the `kLoraFusion` marker, which [#691](https://github.com/mudler/vllm.cpp/issues/691) predicted would go false exactly here | feature | +| [#930](https://github.com/mudler/vllm.cpp/issues/930) | — | LTX-2.5's engine phase loop is fixed at one `Ltx2VideoTokenCount(vshape, 1)` feeding the sigma schedule, the `Ltx2ModalityInput` and `Ltx2VideoUnpatchify`, so no APPENDING conditioning item can be served — and that single obstruction is what refuses BOTH the reference-image/video arms and the last-frame keyframe. Found while porting `ICLoraPipeline` ([#923](https://github.com/mudler/vllm.cpp/issues/923)), which closed the reason the reference refusal used to give and rewrote it onto this one. Much smaller than the two refusals imply: the conditioning item is ported and gated (`ltx2_conditioning.cpp:221`), the VAE encoder is reached, `Ltx2ModalityInput::tokens` is already per-call, and the DiT already accepts a self-attention strength mask — so the work is carrying the grown `state.tokens` through denoise and trimming back before unpatchify (`clear_conditioning`, `ltx_core/tools.py:88-105`). Listed under `## Owed` in [`ltx25-ic-lora.md`](specs/ltx25-ic-lora.md) | feature | +| [#932](https://github.com/mudler/vllm.cpp/issues/932) | — | Two arms of upstream `ICLoraPipeline` that [#923](https://github.com/mudler/vllm.cpp/issues/923) deliberately did not build. (1) `conditioning_attention_strength < 1.0` and `conditioning_attention_mask`: only the DEFAULT third branch of `iclora_utils.py:151-160` is served, which is why the row could ship at all — at strength 1.0 with no mask upstream computes no attention mask anywhere. The other two need `Ltx2LatentState` to carry a mask and `build_attention_mask`'s block structure (`mask_utils.py:170-243`); the DiT side already accepts one. Blocked behind [#930](https://github.com/mudler/vllm.cpp/issues/930). (2) N-adapter fusion, which needs upstream's SECOND rounding pattern — `addmm_(B, A, alpha=strength)` at `fuse_loras.py:115`, which rounds differently from the first product and which #923 REFUSES rather than guesses. Listed under `## Owed` in [`ltx25-ic-lora.md`](specs/ltx25-ic-lora.md) | feature | diff --git a/.agents/specs/ltx25-ic-lora.md b/.agents/specs/ltx25-ic-lora.md index 941098e60..601cd8271 100644 --- a/.agents/specs/ltx25-ic-lora.md +++ b/.agents/specs/ltx25-ic-lora.md @@ -33,7 +33,7 @@ discovered later. other one: every *appending* conditioning item is blocked on token-append machinery the engine does not have. `VideoConditionByReferenceLatent` appends (`reference_video_cond.py:97-100`, ported at `ltx2_conditioning.cpp:265` via - `AppendTokens`), so the reference arm needs it too. §6 sizes it and §7 records + `AppendTokens`), so the reference arm needs it too. §6 sizes it and `## Owed` records it as owed. Closing only the metadata half and lifting the refusal would ship a wrong render. 2. **No render-quality claim, and no real-weights claim.** The evidence here is @@ -165,7 +165,7 @@ and §5.2 mutates that choice to prove the gate can see it. this. - **Exactly one LoRA**, with a second refused by name. Upstream's own `dubit.py` enforces the same (`dubit.py:364-365`) and `hdr_ic_lora.py` takes exactly one - (`hdr_ic_lora.py:271-272`). N-LoRA fusion is recorded as owed in §7 rather than + (`hdr_ic_lora.py:271-272`). N-LoRA fusion is recorded under `## Owed` rather than half-built; the conflict-detection loop this row ports (`ic_lora.py:155-173`) is what N-LoRA needs and is written to take a list already. @@ -289,17 +289,17 @@ sequence and trim it back. What is and is not in the way: That work is **shared with the last-frame keyframe arm**, which is blocked on the identical machinery. It is therefore its own row rather than a tail of this one. -## 7. Owed +## Owed Each is owed by this row and named in the commit and pull request bodies. | owed | issue | |---|---| -| token-append grow-and-trim in the phase loop; until it lands, the reference-video and reference-image arms stay refused, with the refusal rewritten to name this cause instead of the metadata one this row closed | filed as part of this row's delivery, see the pull request body | -| the `conditioning_attention_mask` / `conditioning_attention_strength < 1.0` arm, which needs `Ltx2LatentState` to carry a mask and `build_attention_mask`'s block structure (`mask_utils.py:170-243`) | same | -| N-LoRA fusion (more than one adapter) | same | -| GGUF k-quant LoRA fusion — not applicable rather than owed: the LTX-2.5 DiT ships FP8 and NVFP4, and no GGUF LTX DiT exists to fuse into | n/a | -| a real-weights IC-LoRA fusion measurement | blocked on GPU authority; this row had none | +| token-append grow-and-trim in the phase loop. Until it lands the reference-video and reference-image arms stay refused, with the refusal rewritten to name this cause instead of the metadata one this row closed. Shared with the last-frame keyframe arm, which is why it is its own row | [#930](https://github.com/mudler/vllm.cpp/issues/930) | +| the `conditioning_attention_mask` / `conditioning_attention_strength < 1.0` arm, which needs `Ltx2LatentState` to carry a mask and `build_attention_mask`'s block structure (`mask_utils.py:170-243`) | [#932](https://github.com/mudler/vllm.cpp/issues/932) | +| N-adapter fusion, which additionally needs upstream's SECOND rounding pattern (`addmm_` with `alpha`, `fuse_loras.py:115`) that this row refuses rather than guesses | [#932](https://github.com/mudler/vllm.cpp/issues/932) | +| GGUF k-quant LoRA fusion — **not applicable** rather than owed: the LTX-2.5 DiT ships FP8 and NVFP4, and no GGUF LTX DiT exists to fuse into | n/a | +| a real-weights IC-LoRA fusion measurement | blocked on GPU authority; `dgx.casa` was under a long render for this row's duration and this row had no GPU authority | ## 8. Stop conditions diff --git a/CMakeLists.txt b/CMakeLists.txt index 8b27c5e84..165913be9 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -863,6 +863,10 @@ add_library(vllm STATIC # FP8 and torchao-NVFP4 DiT arms, the torchao-NVFP4 text encoder, and the # load-time device staging GB10's ATS penalty makes the default. src/vllm/model_executor/models/ltx2_loader.cpp + # LTX-2.5 (ROW LTX25-IC-LORA, issue #923): reading an IC-LoRA adapter and its + # `__metadata__`, and the bf16 delta the loader fuses into every arm. Its own + # TU for the same reason as the two above. + src/vllm/model_executor/models/ltx2_lora.cpp src/vllm/entrypoints/openai/video_api.cpp src/vllm/entrypoints/openai/speech_api.cpp src/vllm/entrypoints/openai/speech_api_synthesize.cpp diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 7eb1b22aa..3ec850f18 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -162,7 +162,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | Voxtral audio (`VoxtralForConditionalGeneration`) | Voxtral-Mini-3B-2507 | near-tie-robust 16/16 vs vLLM 0.25.0 | decode 0.97x (beats vLLM); encoder FORWARD 15.90x of vLLM's whole TTFT (pin 46.02 ms), or 2.89x with opt-in `VT_WHISPER_ENC_FA2=1` (costs 3 near-tie divergences vs 0). Not a TTFT ratio. Pending | | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending | | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams | -| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCs, cond, pipeline, quant loaders gated, reduced dims. Prompt AdaLN host+dev; Gemma-4->xattn FIXTURE-gated. Img chain PPM->resize->encode->place->noise. Temporal x2 ups gated, UNDRIVEN. Render OWED | `ltx-2.5`/`ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +24 GB tower. FP8/torchao/NVFP4; kf abs-pos ported; BOTH DiTs load, NO `allow_unported`. IMAGE cond SERVED `crf=0`; DiffVAE/LoRA/keyframe/ref refused. Speed PENDING | +| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCs, cond, pipeline, quant loaders gated, reduced dims. Prompt AdaLN host+dev; Gemma-4->xattn FIXTURE-gated. Img chain PPM->resize->encode->place->noise. Temporal x2 ups gated, UNDRIVEN. Render OWED | `ltx-2.5`/`ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +24 GB tower. FP8/torchao/NVFP4; kf abs-pos ported; BOTH DiTs load, NO `allow_unported`. IMAGE cond SERVED `crf=0`; IC-LoRA fusion SERVED (`--lora`, all four dtype arms, adapter metadata read); DiffVAE/keyframe/ref refused, ref now on token-APPEND not on the metadata. Speed PENDING | | MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, diffusers lane) | MiniMax-Music3 (8.6B Qwen3 LLM + 0.646B RVQ decoder + 2.4B fp32 DiT + DAC Flow-VAE); diffusers arm, ~28.5 GB | `ACTIVE`. Loader 1413/1413; AR, acoustic and the 8.6B LM forward all gated vs real weights; `SpeechRegistry` + `vllm_speech_*` v20 + `/v1/audio/speech`; GGUF Q4_K depth decoder value-gated. No composed request observed | Not measured. The denominator will be SGLang-Omni in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) | | LTX-2.5 tiled + streaming Conv VAE decode | LTX-2.5 video VAE | gated vs executed upstream `ltx_core` @ `fd4ded7f` (`test_ltx2_tiling` 10/10, 915 assertions); one-tile and untiled-spatial controls BIT-EXACT vs untiled on both causality arms; an untiled frames axis is REFUSED | Streams temporal chunks through upstream's AUTO layout (768/64 px, 80/24 frames); above one tile the pixel volume is never materialized. NO-OP below 768px and 81 frames; 81-120 IS tiled, differing 6.70% of range | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | @@ -334,7 +334,7 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the | Muse Glimmer 30B (Meta) | Text gated at **reduced depth 4/52** only; vision wired but never reference-checked | [spec](../.agents/specs/muse-glimmer.md) / [#268](https://github.com/mudler/vllm.cpp/issues/268). Full depth, multi-step decode, image/video, server path and parser scoping open. vLLM speed OPEN GAP; llama.cpp bar #333 | | LTX-2.5 AUTO duration (the duration head) | Brick ported, never constructed | `duration_head_path` is REFUSED by name rather than accepted-and-ignored ([#611](https://github.com/mudler/vllm.cpp/issues/611)); supplying a head cannot load one. Give `num_frames` or `duration` | | LTX-2.5 arms a request CAN reach | Refused by name at the call site | The spatiotemporal latent upsampler (both flags set). Supplying that checkpoint names that arm, not the temporal one. The temporal-only x2 arm is ported, not refused | -| LTX-2.5 arms nothing can request | Declared, not requestable | LoRA fusion, `int8-convrot` (ComfyUI-only), single-node multi-GPU, `BetaScheduler` (upstream selects no scheduler either). No flag or extra asks for these. `multishot` was RETIRED: no such entry point exists upstream | +| LTX-2.5 arms nothing can request | Declared, not requestable | `int8-convrot` (ComfyUI-only), single-node multi-GPU, `BetaScheduler` (upstream selects no scheduler either). No flag or extra asks for these. `multishot` was RETIRED: no such entry point exists upstream, and `kLoraFusion` was RETIRED because it came TRUE - `lora_path` requests it | | Multi-GPU execution | Hardware-blocked | TP proven equal to tp=1 on CPU; no 2-GPU box to run it | | LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server | | Multimodal over HTTP | Image request path wired; forward + codec pending | `ROAD-V1-MM` W1-W3 landed. Open: no mm-forward on `Request.mm_features`; no image codec. Video/audio/multi-image now **refuse** with HTTP 400 rather than drop ([#686](https://github.com/mudler/vllm.cpp/issues/686)) | diff --git a/docs/USAGE.md b/docs/USAGE.md index 0772f1eb0..5e0823a38 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -507,8 +507,10 @@ the token-APPEND machinery — a keyframe is appended to the sequence with its o positions and a rebuilt attention mask, then trimmed back off, and this engine's phase loop is fixed at the target grid's token count — while the served first-frame arm only REPLACES tokens that already exist; the reference arms need -the IC-LoRA's scale factors, which live in LoRA metadata this project does not -read; reference audio additionally needs the AUDIO VAE's encoder key filter, +that SAME token-append machinery, because a reference latent is appended too +(their old reason - the IC-LoRA scale factors living in adapter metadata this +project could not read - was closed on 2026-08-15 by `--lora`, which reads +them); reference audio additionally needs the AUDIO VAE's encoder key filter, which is not built. (Until 2026-08-13 this said a last-frame keyframe needs the DiT's unported `keyframes_abs_pos_embedding`. That was wrong: a supplied keyframe is appended unmarked, so the embedding never applies to it. Where the embedding @@ -628,6 +630,29 @@ own CRF 18 and refuses — see the out-of-distribution note above. half resolution). Omitting all three renders the recipe default, which is 1024x1536 at 121 frames and is a much larger request than it looks. +`--lora ic-lora.safetensors [STRENGTH]` fuses an IC-LoRA adapter into the DiT at +load, mirroring upstream's `--lora PATH [STRENGTH]` +(`ltx-pipelines/utils/args.py:600-611`). The strength is optional and defaults to +1.0. It is a LOAD-time flag, not a per-request one, because the adapter is fused +into the weights and cannot vary between generations - upstream takes it as a +`DiffusionStage.from_checkpoint` constructor argument for the same reason +(`ic_lora.py:104-114`). + +The adapter is a safetensors file of `.lora_A.weight` / `.lora_B.weight` pairs, +with or without ComfyUI's `diffusion_model.` prefix. It works on every arm the +DiT loads - bf16, FP8 and NVFP4 alike - because those are all dequantized to +bf16 before the delta is added. Three things REFUSE by name rather than +proceeding quietly: an adapter naming a module this port does not bind (upstream +would skip it, and a skip cannot be told apart from a typo), an adapter that +fuses into nothing at all, and a second `--lora`, since only one adapter is +accepted so far. + +Supplying an adapter also reads its `reference_downscale_factor` and +`reference_temporal_scale_factor` metadata (`iclora_utils.py:30-49`). Those are +what a reference video needs, and reading them was what the reference refusal +used to say was missing - it now names the token-append machinery instead, which +is the cause that actually remains. + `--upsampler` is what the distilled recipe's second phase needs. Without it that phase refuses rather than skipping: its three-step refinement is what makes the upscaled latent valid, and decoding the half-resolution latent instead would hand @@ -1989,8 +2014,9 @@ spatiotemporal upsampler is the arm with `spatial_upsample` AND **ported** and is not refused; nothing shipped drives it yet, so it is gated rather than served. Four more are recorded as out of scope but are **not requestable**, so no flag or extra can -reach them: LoRA fusion, `int8-convrot`, single-node multi-GPU, and -`BetaScheduler`. Their messages +reach them: `int8-convrot`, single-node multi-GPU, and +`BetaScheduler`. (LoRA fusion was in that list until 2026-08-15 and is now +SERVED - see `--lora` above - so its marker was retired rather than moved.) Their messages say `DECLARED, NOT REQUESTABLE` so the two kinds are not confused. `BetaScheduler` is in that group rather than the reachable one because upstream selects it nowhere: every `ltx-pipelines` entry point hard-codes @@ -2071,6 +2097,8 @@ batch-wide payload by rows. The second argument is the requested row count; each row keeps the source tensor's independent `num_tokens_per_position` width. +(That brick is the TEXT decode path and is a different mechanism from LTX-2.5's +IC-LoRA, which fuses into the weights at load and IS served - see `--lora`.) The LoRA adapter headers ([`lora/lora_weights.h`](../include/vllm/lora/lora_weights.h), [`lora/punica.h`](../include/vllm/lora/punica.h), [`lora/layers.h`](../include/vllm/lora/layers.h)) are present but **not yet wired diff --git a/examples/ltx2_gen/main.cpp b/examples/ltx2_gen/main.cpp index 8aa7b0a31..8c395fd7a 100644 --- a/examples/ltx2_gen/main.cpp +++ b/examples/ltx2_gen/main.cpp @@ -98,6 +98,7 @@ const char* Need(int argc, char** argv, int i, const char* flag) { " [--model-version 2.5] [--pipeline-kind distilled_two_stage]\n" " [--upsampler ] phase 2 needs it\n" " [--max-phase N] [--allow-unported]\n" + " [--lora [STRENGTH]] fused at load; 1.0\n" " [--prompt-valid-rows N] how many embed rows are real tokens\n" " [--frames N] [--width N] [--height N] [--seed N]\n" " [--first-frame ] [--image-crf 0]\n" @@ -188,6 +189,19 @@ int main(int argc, char** argv) { else if (f == "--prompt-valid-rows") SetExtra("prompt_embeds_valid_rows", Need(argc, argv, ++i, f.c_str())); else if (f == "--allow-unported") SetExtra("allow_unported_modules", "1"); + // IC-LoRA (row LTX25-IC-LORA, issue #923), mirroring upstream's + // `--lora PATH [STRENGTH]` (ltx-pipelines utils/args.py:600-611): the + // strength is optional and defaults to upstream's DEFAULT_LORA_STRENGTH. + // LOAD extras, because upstream fuses the adapter into the weights at + // construction (ic_lora.py:104-114) and it cannot vary per request. + else if (f == "--lora") { + SetExtra("lora_path", Need(argc, argv, ++i, f.c_str())); + // The optional second word. Consumed only when it does not look like the + // next flag, which is how upstream's `nargs="+"` LoraAction disambiguates. + if (i + 1 < argc && argv[i + 1][0] != '-') { + SetExtra("lora_strength", argv[++i]); + } + } // Image conditioning (row LTX25-IMAGE-COND, issue #644). `--first-frame` is // a binary PPM; `--image-crf` is the PER-GENERATION extra, so it rides // vp.extra_* rather than mp.extra_*. Only 0 is served, and it is NOT diff --git a/include/vllm/model_executor/models/ltx2_loader.h b/include/vllm/model_executor/models/ltx2_loader.h index 7e0b52b4c..ab96b7f0a 100644 --- a/include/vllm/model_executor/models/ltx2_loader.h +++ b/include/vllm/model_executor/models/ltx2_loader.h @@ -177,6 +177,7 @@ #include "vllm/model_executor/models/ltx2_connector.h" #include "vllm/model_executor/models/ltx2_text_encoder.h" #include "vllm/model_executor/models/ltx2_upsampler.h" +#include "vllm/model_executor/models/ltx2_lora.h" #include "vllm/model_executor/models/ltx2_video_vae.h" #include "vt/backend.h" #include "vt/dtype.h" @@ -374,6 +375,19 @@ struct Ltx2DitLoadOptions { // Widen the bf16 materialization to f32 for `Ltx2DitForward`, whose gate is // f32 by declaration. Doubles the footprint; see the DTYPE note above. bool widen_to_f32 = false; + // IC-LoRA adapters to FUSE INTO the weights as they are materialized + // (ltx-core loader/fuse_loras.py:119-150; `DiffusionStage.from_checkpoint` + // takes them as a constructor argument, ic_lora.py:104-114, which is why they + // live on the LOAD options rather than on a generation request). + // + // Fusion happens immediately after `MaterializeDitTensor` and therefore serves + // the F32, BF16, FP8 and NVFP4 arms with one code path: both quantized + // branches dequantize to bf16 before returning. On the streaming arm it runs + // before the device copy, so the "one host buffer live at a time" invariant is + // unchanged. See `ltx2_lora.h` for the arithmetic and the dtype argument. + // + // Exactly one adapter is accepted; a second refuses by name. + std::vector loras; }; // A host buffer owned by a loaded checkpoint. Pointer-stable: the views index @@ -395,6 +409,19 @@ struct Ltx2DitCheckpoint { // Module prefixes present in the file and outside the L2 contract, in header // order, deduplicated. Empty means the file and the contract agree. std::vector unported; + // How many contract tensors an IC-LoRA delta was fused into. Zero when no + // adapter was given; never zero WITH one, because a LoRA that fused into + // nothing is refused at load. + int64_t lora_fused_tensors = 0; + // The reference scale factors the fused adapter declares in its own + // `__metadata__` (iclora_utils.py:30-49), resolved by + // `Ltx2ResolveLoraReferenceFactors`. Both 1 with no adapter, which is also + // upstream's default for an adapter that declares neither. + // + // These are what `Ltx2ConditionVideoByReference` takes and what the reference + // refusal used to name as unreadable. They are read here so that the refusal + // can name the cause that ACTUALLY remains. + Ltx2LoraReferenceFactors lora_reference; Ltx2DitWeights weights; std::map views; // Host-resident buffers (`Ltx2LoadDitFromSafetensors`). Pointer-stable. diff --git a/include/vllm/model_executor/models/ltx2_lora.h b/include/vllm/model_executor/models/ltx2_lora.h new file mode 100644 index 000000000..4d6805e8f --- /dev/null +++ b/include/vllm/model_executor/models/ltx2_lora.h @@ -0,0 +1,186 @@ +// LTX-2.5 IC-LoRA — reading an adapter and FUSING it into the DiT at load. +// +// Ported from Lightricks/LTX-2 @ fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca: +// +// ltx-core loader/primitives.py:160-167 LoraPathStrengthAndSDOps +// ltx-core loader/fuse_loras.py:99-116 aggregate_lora_products +// ltx-core loader/fuse_loras.py:119-150 fuse_lora_weights +// ltx-core loader/fuse_loras.py:183-204 the key shape and the products +// ltx-core loader/sd_ops.py:135-137 LTXV_LORA_COMFY_RENAMING_MAP +// ltx-pipelines iclora_utils.py:30-49 the reference-scale metadata +// ltx-pipelines ic_lora.py:150-173 resolving factors across LoRAs +// +// ─── THIS IS NOT `include/vllm/lora/` ──────────────────────────────────────── +// +// That subsystem is vLLM's runtime punica path: f32, slot-indexed, applied per +// token through `LinearMethodBase` (`punica.h:15-24`, `layers.h:26-28`). LTX +// does not apply an adapter at runtime. It FUSES the delta into the weights at +// load and the adapter then has no runtime existence at all +// (`fuse_loras.py:119-150`). Mirroring vLLM's LoRA here would mirror the wrong +// upstream, so this is a separate path by decision rather than by oversight. +// The row's spec §1 records the argument. +// +// ─── THE ARITHMETIC, AND THE DTYPE THAT CARRIES IT ─────────────────────────── +// +// delta = sum over adapters of (B * strength) @ A +// W' = W + delta +// +// `B` is [out, rank], `A` is [rank, in], `W` is [out, in] row-major. +// +// **The accumulator is bfloat16, not f32.** All four of upstream's fuse rules +// set `aggregation_dtype=torch.bfloat16` — bf16 (`fuse_loras.py:71`), fp8-cast +// (`quantization/fp8_cast.py:239`), fp8-scaled-mm +// (`quantization/fp8_scaled_mm.py:189`) and NVFP4 +// (`quantization/nvfp4/fuse.py:50`). An f32 accumulator here would be the exact +// defect AGENTS.md names: every token would still match and every golden would +// still pass while the path carried a precision upstream does not have. +// `test_ltx2_lora` mutates this deliberately, because nothing else can see it. +// +// Upstream's own docstring records that the FIRST product rounds differently +// from the rest — `matmul(B * strength, A)` against `addmm_(B, A, alpha=...)` +// (`fuse_loras.py:103-107`) — and that pattern is preserved here. Only the +// first form is reachable: this row fuses exactly ONE adapter and refuses a +// second by name (see `Ltx2ResolveLoraReferenceFactors`), so implementing the +// second form would be landing a branch nothing can select. +// +// ─── WHERE IT IS APPLIED, AND WHY ONE PLACE SERVES EVERY ARM ───────────────── +// +// `MaterializeDitTensor` (ltx2_loader.cpp) has four dtype branches, and BOTH +// quantized ones — `DequantFp8ToBf16` and `Ltx2DequantNvfp4ToBf16` — return +// `vt::DType::kBF16`. By the time any caller sees a byte, an FP8 or NVFP4 +// checkpoint is already bf16. So a single hook immediately after that call +// serves F32, BF16, FP8 and NVFP4 at once. +// +// That is a DELIBERATE DIVERGENCE from upstream's per-arm rules, and it is +// forced rather than chosen. Upstream re-quantizes after adding the delta +// because it keeps FP8/NVFP4 weights resident for its quantized kernels; this +// tree keeps bf16 (ltx2_loader.h's DTYPE note), so there is nothing to +// re-quantize into, and the tree carries no FP8 or NVFP4 quantizer to do it +// with. The consequence is that our fused weight SKIPS upstream's lossy +// round trip and is slightly more precise on those two arms. It costs no extra +// bytes — the weight was already bf16 — so this is not the too-wide-dtype +// failure the rule above guards against. +#pragma once + +#include +#include +#include +#include + +#include "vt/dtype.h" + +namespace vllm { + +class SafetensorsFile; + +// LoraPathStrengthAndSDOps (loader/primitives.py:160-167). `sd_ops` has no +// field here because the only map upstream ever passes for LTX is +// LTXV_LORA_COMFY_RENAMING_MAP (`sd_ops.py:135-137`, used at `args.py:172`), +// whose entire content is stripping a `diffusion_model.` prefix — so it is +// applied unconditionally by `Ltx2LoraContractName` rather than carried. +struct Ltx2LoraSpec { + std::string path; + // DEFAULT_LORA_STRENGTH (utils/args.py, quoted into --lora's help at :605-608). + double strength = 1.0; +}; + +// The reference scale factors an IC-LoRA was TRAINED with, read from the +// adapter's own `__metadata__` (iclora_utils.py:30-49). Both default to 1, which +// is upstream's default for an absent key. +// +// These are what `Ltx2ConditionVideoByReference` needs and what the reference +// refusal in `ltx2_video.cpp` named as unreadable. +struct Ltx2LoraReferenceFactors { + int64_t downscale = 1; + int64_t temporal = 1; +}; + +// One `A`/`B` factor pair, already resolved onto the DiT contract's tensor name. +struct Ltx2LoraFactorPair { + // The CONTRACT name this pair fuses into, e.g. + // "transformer_blocks.0.attn1.to_q.weight". + std::string target; + int64_t out_features = 0; + int64_t rank = 0; + int64_t in_features = 0; + // [out, rank] and [rank, in], row-major, bf16 bit patterns. Held bf16 because + // that is the dtype the products are formed in (`fuse_loras.py:202-203` casts + // both to the rule's `aggregation_dtype` before the matmul). + std::vector b; + std::vector a; +}; + +// One loaded adapter file. +class Ltx2LoraAdapter { + public: + // Read `spec.path`, resolve every `.lora_A.weight` / `.lora_B.weight` pair + // onto a contract name, and keep the file's whole `__metadata__`. + // + // `contract` is the set of tensor names the DiT actually binds. A pair whose + // target is not in it REFUSES BY NAME. Upstream instead SKIPS it + // (`fuse_loras.py:135-137`, `if original_weight is None: continue`) because + // its state dict is the whole model and a skip is unambiguous there. Here the + // contract is a fixed enumerated set with unported modules already stripped, + // so a skip would absorb a misnamed key and a genuinely inapplicable one + // alike — and the first is the failure this project keeps paying for. The + // row's spec §4.1 argues the divergence; it is the narrower behaviour. + static Ltx2LoraAdapter Open(const Ltx2LoraSpec& spec, + const std::vector& contract); + + const std::string& path() const { return path_; } + double strength() const { return strength_; } + const std::map& metadata() const { return metadata_; } + + // The pair targeting `name`, or nullptr. One target at most: a duplicate is + // refused at Open. + const Ltx2LoraFactorPair* Find(const std::string& name) const; + + private: + std::string path_; + double strength_ = 1.0; + std::map metadata_; + std::vector pairs_; +}; + +// `diffusion_model..lora_A.weight` -> `.weight`, mirroring +// LTXV_LORA_COMFY_RENAMING_MAP's prefix strip (`sd_ops.py:136`) composed with +// the `.lora_{A,B}.weight` -> `.weight` rewrite `_affected_weight_keys` performs +// (`fuse_loras.py:185-186`). Returns false when `key` is not a LoRA factor. +bool Ltx2LoraContractName(const std::string& key, std::string* out_target, + bool* out_is_a); + +// One `__metadata__` integer factor, defaulting to 1 when the key is absent +// (iclora_utils.py:35, 46). +// +// Upstream swallows EVERY failure and returns 1 (`iclora_utils.py:36-38`, +// `except Exception: ... return 1`). A malformed value is refused by name here +// instead, because the two factors silently reverting to 1 place the reference +// plausibly and wrongly — which is the failure mode the reference refusal was +// written to prevent in the first place. Absent still means 1; only malformed +// differs. +int64_t Ltx2ReadLoraMetadataFactor(const std::map& metadata, + const std::string& key, const std::string& path); + +// Resolve the reference factors across the adapters, mirroring ic_lora.py:150-173 +// including its conflict errors (`:158-163`, `:167-172`). +// +// Refuses more than one adapter by name. Upstream's own `dubit.py` enforces +// exactly one (`dubit.py:364-365`) and `hdr_ic_lora.py` takes exactly one +// (`hdr_ic_lora.py:271-272`); N-adapter fusion is owed by the row rather than +// half-built, and this signature already takes the list it will need. +Ltx2LoraReferenceFactors Ltx2ResolveLoraReferenceFactors( + const std::vector& adapters); + +// Add every adapter's delta for `target` into `buffer`, which holds `rows*cols` +// elements of `dtype` as produced by `MaterializeDitTensor`. Returns true when +// anything was fused. +// +// This is `fuse_lora_weights` (`fuse_loras.py:119-150`) collapsed onto one +// already-materialized tensor: the aggregation is bf16, the add is upstream's +// `deltas.add_(weight)` and the store is its `.to(dtype=weight.dtype)` +// (`fuse_loras.py:67-68`). +bool Ltx2FuseLoraIntoTensor(const std::vector& adapters, + const std::string& target, vt::DType dtype, int64_t rows, + int64_t cols, uint8_t* buffer, size_t buffer_bytes); + +} // namespace vllm diff --git a/include/vllm/model_executor/models/ltx2_pipeline.h b/include/vllm/model_executor/models/ltx2_pipeline.h index 3e87cfeff..46109ce52 100644 --- a/include/vllm/model_executor/models/ltx2_pipeline.h +++ b/include/vllm/model_executor/models/ltx2_pipeline.h @@ -621,12 +621,24 @@ Ltx2PipelineRecipe ResolveLtx2PipelineRecipe(const std::string& pipeline_kind, // DECLARED-OUT-OF-SCOPE MARKER — no request field, load extra or CLI flag asks // for it, so nothing outside the ledger test reaches it. It is a record of what // upstream HAS and this port does NOT, which is worth keeping; calling it a -// refusal is what was wrong. `kBetaScheduler`, `kLoraFusion`, `kInt8ConvRot` -// and `kMultiGpuParallelism` are markers, and their messages say so. +// refusal is what was wrong. `kBetaScheduler`, `kInt8ConvRot` and +// `kMultiGpuParallelism` are markers, and their messages say so. // -// TWO ENUMERATORS WERE RETIRED on 2026-08-13, recorded here because the -// retirement IS the record — a reader who finds them in git history needs to know -// they did not simply move: +// THREE ENUMERATORS HAVE BEEN RETIRED, recorded here because the retirement IS +// the record — a reader who finds them in git history needs to know they did not +// simply move. Two on 2026-08-13 because they were wrong, one on 2026-08-15 +// because it came true: +// +// `kLoraFusion` — RETIRED 2026-08-15 by row LTX25-IC-LORA (#923) because it +// came TRUE. It said LoRA fusion was out of scope and carried the +// DECLARED, NOT REQUESTABLE marker, which asserted that no request field or +// load extra asks for it. The `lora_path` / `lora_strength` load extras now +// do, and `Ltx2DitLoadOptions::loras` fuses the delta into every arm, so the +// marker's own sentence had become false. #691 predicted this exact drift in +// its own words — the ledger test gates the message TEXT and not the property, +// so nothing here would have caught it — and that is why the enumerator is +// REMOVED rather than reclassified: there is no longer an unported LoRA-fusion +// feature to name, and a refusal for a served capability is worse than none. // // `kMultishot` — FABRICATED. It refused "multishot generation" and cited // "ltx-pipelines multishot entry points". No such entry point, symbol or string @@ -667,8 +679,6 @@ enum class Ltx2UnportedPipelineFeature { // has zero hits for the name. Mirroring that means no // scheduler-kind field here either, so nothing reaches the // refusal — `Ltx2Schedule`, which holds it, has no caller. - kLoraFusion, // ltx-core loader/primitives.py:160 (LoraPathStrengthAndSDOps), - // fused by loader/fuse_loras.py kInt8ConvRot, // ComfyUI-ecosystem quantization, and NOT an LTX-2 arm: the four // inference kinds upstream defines are fp8-cast / fp8-scaled-mm / // nvfp4-cast / nvfp4-prequant (quantization_factory.py:23-26). diff --git a/include/vllm/multimodal/ltx2_video.h b/include/vllm/multimodal/ltx2_video.h index f0d4f20b3..eb3c544b3 100644 --- a/include/vllm/multimodal/ltx2_video.h +++ b/include/vllm/multimodal/ltx2_video.h @@ -201,6 +201,22 @@ inline constexpr char kLtx2AllowUnportedExtra[] = "allow_unported_modules"; // say so rather than skipping the phase silently. inline constexpr char kLtx2MaxPhaseExtra[] = "max_phase"; +// An IC-LoRA adapter to FUSE into the DiT, and its strength +// (ltx-core loader/primitives.py:160-167 `LoraPathStrengthAndSDOps`, fused by +// loader/fuse_loras.py:119-150; the CLI pair is ltx-pipelines utils/args.py:600-611). +// +// LOAD extras rather than generation fields, because upstream takes the LoRAs as +// a `DiffusionStage.from_checkpoint` CONSTRUCTOR argument (ic_lora.py:104-114) +// and fuses them into the weights, so the adapter is a property of the loaded +// model and cannot vary per request. A per-generation field would promise +// something the mechanism cannot do. +// +// `lora_strength` absent is 1.0, upstream's DEFAULT_LORA_STRENGTH. Supplying a +// strength without a path refuses, because a strength alone is a request that +// silently did nothing. +inline constexpr char kLtx2LoraPathExtra[] = "lora_path"; +inline constexpr char kLtx2LoraStrengthExtra[] = "lora_strength"; + // How many of the supplied prompt-embeds rows are REAL tokens; the rest are // padding. Absent means every row is real. // diff --git a/src/vllm/model_executor/models/ltx2_loader.cpp b/src/vllm/model_executor/models/ltx2_loader.cpp index dfdfc88ac..a606cd1ca 100644 --- a/src/vllm/model_executor/models/ltx2_loader.cpp +++ b/src/vllm/model_executor/models/ltx2_loader.cpp @@ -499,6 +499,61 @@ vt::DType MaterializeDitTensor(const SafetensorsFile& file, const DitPlan& plan, Fail("'" + spec.name + "' has dtype " + t.dtype + ", which this loader does not read"); } +// --- IC-LoRA fusion, the three helpers the two load arms share --------------- +// +// Kept here rather than in `ltx2_lora.cpp` because they are about the LOADER's +// contract — `Ltx2TensorSpec`, the enumerated name set, the "did anything +// happen" check — and `ltx2_lora.cpp` is about the adapter format. + +// Open every requested adapter against the contract this load will bind, and +// resolve the reference factors, whose conflict and arity refusals fire here +// rather than at first use (ic_lora.py:150-173 resolves them in __init__). +std::vector OpenDitLoras(const Ltx2DitLoadOptions& options, + const std::vector& contract) { + if (options.loras.empty()) return {}; + std::vector names; + names.reserve(contract.size()); + for (const Ltx2TensorSpec& spec : contract) names.push_back(spec.name); + + std::vector out; + out.reserve(options.loras.size()); + for (const Ltx2LoraSpec& spec : options.loras) { + out.push_back(Ltx2LoraAdapter::Open(spec, names)); + } + // The caller resolves the reference factors immediately, and that call is what + // refuses more than one adapter and refuses conflicting metadata — before any + // tensor is materialized. + return out; +} + +bool FuseLorasInto(const std::vector& loras, const Ltx2TensorSpec& spec, + vt::DType dtype, std::vector& buffer) { + if (loras.empty()) return false; + // A LoRA factor pair is rank 2 and so is its target. A contract tensor of any + // other rank cannot be a LoRA target, and `Ltx2LoraAdapter::Open` has already + // refused a pair naming a name outside the contract, so this is a shape + // filter rather than a silent skip of a possible target. + if (spec.shape.size() != 2) return false; + return Ltx2FuseLoraIntoTensor(loras, spec.name, dtype, spec.shape[0], spec.shape[1], + buffer.data(), buffer.size()); +} + +// A LoRA that fused into NOTHING loads green and renders identically to no LoRA +// at all. That is a user error — a wrong file, or one trained against another +// model — and it must not read as success. +void CheckLorasWereApplied(const std::vector& loras, int64_t fused) { + if (loras.empty() || fused > 0) return; + std::string paths; + for (const Ltx2LoraAdapter& lora : loras) { + paths += std::string(paths.empty() ? "" : ", ") + "'" + lora.path() + "'"; + } + Fail("the adapter(s) " + paths + + " fused into ZERO tensors of this checkpoint. Every A/B pair named a tensor the " + "contract binds, so the delta was computed for none of them — which means the " + "render would be byte-identical to loading no adapter, while reporting success. " + "Refusing instead."); +} + // The families the file carries and NOTHING IN THIS PORT reads. // // The two `*_embeddings_connector` families are outside the DiT contract and are @@ -615,13 +670,21 @@ Ltx2DitCheckpoint Ltx2LoadDitFromSafetensors(const SafetensorsFile& file, RefuseUnported(out.unported); } + const std::vector loras = OpenDitLoras(options, contract); + out.lora_reference = Ltx2ResolveLoraReferenceFactors(loras); + int64_t fused = 0; for (const Ltx2TensorSpec& spec : contract) { auto buffer = std::make_shared(); buffer->dtype = MaterializeDitTensor(file, plan, spec, buffer->bytes); + // AFTER materialize, so one hook serves F32, BF16, FP8 and NVFP4 alike: + // both quantized branches above have already returned bf16. + if (FuseLorasInto(loras, spec, buffer->dtype, buffer->bytes)) ++fused; out.views[spec.name] = MakeView(buffer->bytes.data(), buffer->dtype, vt::Device{}, spec.shape); out.storage.push_back(std::move(buffer)); } + CheckLorasWereApplied(loras, fused); + out.lora_fused_tensors = fused; if (options.widen_to_f32) Ltx2WidenDitToF32(out); out.weights = BindLtx2DitWeights(out.params, out.views); @@ -668,6 +731,9 @@ Ltx2DitCheckpoint Ltx2StreamDitToDevice(vt::Queue& queue, const SafetensorsFile& RefuseUnported(out.unported); } + const std::vector loras = OpenDitLoras(options, contract); + out.lora_reference = Ltx2ResolveLoraReferenceFactors(loras); + int64_t fused = 0; vt::Backend& backend = vt::GetBackend(queue.device.type); for (const Ltx2TensorSpec& spec : contract) { // ONE tensor's host buffer is live at a time: it is dequantized, uploaded, @@ -675,6 +741,9 @@ Ltx2DitCheckpoint Ltx2StreamDitToDevice(vt::Queue& queue, const SafetensorsFile& // the device copy plus one tensor rather than two whole models. std::vector host; const vt::DType dtype = MaterializeDitTensor(file, plan, spec, host); + // Fused BEFORE the upload, on the host buffer that is already live, so the + // invariant above is unchanged: no second device allocation, no round trip. + if (FuseLorasInto(loras, spec, dtype, host)) ++fused; void* device = backend.Alloc(host.size()); backend.Copy(queue, device, host.data(), host.size()); backend.Synchronize(queue); // `host` dies at the end of this iteration @@ -685,6 +754,8 @@ Ltx2DitCheckpoint Ltx2StreamDitToDevice(vt::Queue& queue, const SafetensorsFile& out.device_storage.emplace_back(device, [&backend](void* p) { backend.Free(p); }); load_stats::AddDeviceUpload(host.size()); } + CheckLorasWereApplied(loras, fused); + out.lora_fused_tensors = fused; out.weights = BindLtx2DitWeights(out.params, out.views); return out; } diff --git a/src/vllm/model_executor/models/ltx2_lora.cpp b/src/vllm/model_executor/models/ltx2_lora.cpp new file mode 100644 index 000000000..3ee0b8a44 --- /dev/null +++ b/src/vllm/model_executor/models/ltx2_lora.cpp @@ -0,0 +1,374 @@ +// LTX-2.5 IC-LoRA adapter reading and fusion. See ltx2_lora.h for the upstream +// anchors, the dtype argument, and the two deliberate divergences. +// +// A separate translation unit from `ltx2_loader.cpp` for the reason that file's +// siblings already record: `ltx2_loader.cpp` is 1500 lines that several rows +// edit concurrently, and a new family does not need to lock it. +#include "vllm/model_executor/models/ltx2_lora.h" + +#include +#include +#include +#include +#include +#include + +#include "vllm/model_executor/model_loader/safetensors_reader.h" +#include "vt/dtype.h" + +namespace vllm { +namespace { + +[[noreturn]] void Fail(const std::string& what) { + throw std::runtime_error("ltx2 lora: " + what); +} + +std::string ShapeText(const std::vector& shape) { + std::string out = "["; + for (size_t i = 0; i < shape.size(); ++i) { + out += (i != 0 ? ", " : "") + std::to_string(shape[i]); + } + return out + "]"; +} + +// LTXV_LORA_COMFY_RENAMING_MAP is a single prefix strip (`sd_ops.py:136`). +constexpr const char* kComfyPrefix = "diffusion_model."; +constexpr const char* kLoraASuffix = ".lora_A.weight"; +constexpr const char* kLoraBSuffix = ".lora_B.weight"; + +bool HasSuffix(const std::string& s, const char* suffix) { + const size_t n = std::strlen(suffix); + return s.size() >= n && s.compare(s.size() - n, n, suffix) == 0; +} + +// Read a rank-2 tensor into bf16 bit patterns. The products are formed in the +// rule's aggregation dtype and upstream casts both factors to it before the +// matmul (`fuse_loras.py:202-203`), so bf16 is where they are held. +std::vector ReadFactorAsBf16(const std::string& key, const StTensor& t, + const std::string& path) { + if (t.shape.size() != 2) { + Fail("'" + key + "' in '" + path + "' is rank " + std::to_string(t.shape.size()) + + " " + ShapeText(t.shape) + "; a LoRA factor is rank 2"); + } + int64_t numel = t.shape[0] * t.shape[1]; + if (numel <= 0) { + Fail("'" + key + "' in '" + path + "' is empty " + ShapeText(t.shape)); + } + std::vector out(static_cast(numel)); + if (t.dtype == "BF16") { + if (t.nbytes != out.size() * sizeof(uint16_t)) { + Fail("'" + key + "' in '" + path + "' declares " + std::to_string(t.nbytes) + + " BF16 bytes but its shape " + ShapeText(t.shape) + " needs " + + std::to_string(out.size() * sizeof(uint16_t))); + } + std::memcpy(out.data(), t.data, t.nbytes); + return out; + } + if (t.dtype == "F32") { + if (t.nbytes != out.size() * sizeof(float)) { + Fail("'" + key + "' in '" + path + "' declares " + std::to_string(t.nbytes) + + " F32 bytes but its shape " + ShapeText(t.shape) + " needs " + + std::to_string(out.size() * sizeof(float))); + } + // Narrowed to bf16 on purpose: this is the cast upstream performs at + // `fuse_loras.py:202-203`, not a shortcut. An f32 factor kept f32 would + // widen the accumulator through the back door. + for (size_t i = 0; i < out.size(); ++i) { + float v = 0.0F; + std::memcpy(&v, t.data + i * sizeof(float), sizeof(float)); + out[i] = vt::F32ToBF16(v); + } + return out; + } + if (t.dtype == "F16") { + Fail("'" + key + "' in '" + path + + "' is F16. Upstream casts LoRA factors to the fuse rule's aggregation dtype " + "(fuse_loras.py:202-203) and every LTX rule aggregates in BF16, so an F16 " + "adapter is readable in principle — but no shipped LTX IC-LoRA is F16, so this " + "port has never seen one and will not guess at its scaling. File an issue with " + "the adapter."); + } + Fail("'" + key + "' in '" + path + "' has dtype " + t.dtype + + ", which this reader does not read. LoRA factors are BF16 or F32."); +} + +} // namespace + +bool Ltx2LoraContractName(const std::string& key, std::string* out_target, + bool* out_is_a) { + bool is_a = false; + const char* suffix = nullptr; + if (HasSuffix(key, kLoraASuffix)) { + is_a = true; + suffix = kLoraASuffix; + } else if (HasSuffix(key, kLoraBSuffix)) { + suffix = kLoraBSuffix; + } else { + return false; + } + std::string module = key.substr(0, key.size() - std::strlen(suffix)); + // LTXV_LORA_COMFY_RENAMING_MAP: strip `diffusion_model.` when present + // (`sd_ops.py:136`). The DiT contract's own names are already stripped of + // `model.diffusion_model.` by the loader's plan, so the two meet. + const size_t plen = std::strlen(kComfyPrefix); + if (module.size() > plen && module.compare(0, plen, kComfyPrefix) == 0) { + module = module.substr(plen); + } + if (module.empty()) return false; + if (out_target != nullptr) *out_target = module + ".weight"; + if (out_is_a != nullptr) *out_is_a = is_a; + return true; +} + +int64_t Ltx2ReadLoraMetadataFactor(const std::map& metadata, + const std::string& key, const std::string& path) { + const auto it = metadata.find(key); + // Absent is 1, which is upstream's default (`iclora_utils.py:35, 46`). + if (it == metadata.end()) return 1; + const std::string& raw = it->second; + size_t consumed = 0; + long long value = 0; + try { + value = std::stoll(raw, &consumed); + } catch (const std::exception&) { + consumed = 0; + } + if (consumed != raw.size() || value < 1) { + Fail("'" + path + "' carries metadata " + key + "='" + raw + + "', which is not a positive integer. Upstream swallows this and silently returns " + "1 (iclora_utils.py:36-38); refusing instead, because a factor that reverts to 1 " + "places the reference plausibly and WRONGLY, and no output check can see that."); + } + return static_cast(value); +} + +Ltx2LoraAdapter Ltx2LoraAdapter::Open(const Ltx2LoraSpec& spec, + const std::vector& contract) { + if (spec.path.empty()) Fail("an adapter path is empty"); + const std::set known(contract.begin(), contract.end()); + + Ltx2LoraAdapter out; + out.path_ = spec.path; + out.strength_ = spec.strength; + + const SafetensorsFile file = SafetensorsFile::Open(spec.path); + out.metadata_ = file.Metadata(); + + // Gather the A and B halves by target, so a pair missing its other half is + // reported as the pair it is rather than as two unrelated tensors. + std::map a_of; + std::map b_of; + // The A and B keys are tracked SEPARATELY. One map keyed by target held + // whichever key was seen last, so a malformed A factor was reported under the + // B factor's name -- which is precisely the "refuse BY NAME" property these + // messages exist to have, defeated by the message itself. + std::map a_key_of; + std::map b_key_of; + for (const std::string& key : file.Names()) { + std::string target; + bool is_a = false; + if (!Ltx2LoraContractName(key, &target, &is_a)) continue; + auto& side = is_a ? a_of : b_of; + if (side.count(target) != 0) { + Fail("'" + spec.path + "' carries two " + std::string(is_a ? "A" : "B") + + " factors for '" + target + "'"); + } + side[target] = &file.Get(key); + (is_a ? a_key_of : b_key_of)[target] = key; + } + + if (a_of.empty() && b_of.empty()) { + Fail("'" + spec.path + + "' carries no `.lora_A.weight` / `.lora_B.weight` pair at all, so it is not a " + "LoRA adapter. Refusing rather than loading a model with no delta and reporting " + "success."); + } + + for (const auto& kv : a_of) { + const std::string& target = kv.first; + const auto b_it = b_of.find(target); + if (b_it == b_of.end()) { + Fail("'" + spec.path + "' has an A factor for '" + target + + "' with no matching B factor"); + } + if (known.count(target) == 0) { + // The divergence argued in ltx2_lora.h and the row's spec §4.1: upstream + // skips (`fuse_loras.py:135-137`), this refuses. + Fail("'" + spec.path + "' targets '" + target + + "', which the DiT contract does not bind. Upstream would SKIP this key " + "(fuse_loras.py:135-137) because its state dict is the whole model; here the " + "contract is a fixed enumerated set, so a skip would absorb a MISNAMED key and " + "an inapplicable one alike. If this adapter was trained against a module this " + "port does not carry, that is the thing to report."); + } + const StTensor& a = *kv.second; + const StTensor& b = *b_it->second; + + Ltx2LoraFactorPair pair; + pair.target = target; + // A is [rank, in], B is [out, rank] (`fuse_loras.py:196-198` pairs them for + // a [out, in] weight). + pair.rank = a.shape.size() == 2 ? a.shape[0] : 0; + pair.in_features = a.shape.size() == 2 ? a.shape[1] : 0; + pair.out_features = b.shape.size() == 2 ? b.shape[0] : 0; + pair.a = ReadFactorAsBf16(a_key_of[target], a, spec.path); + pair.b = ReadFactorAsBf16(b_key_of[target], b, spec.path); + if (b.shape.size() != 2 || b.shape[1] != pair.rank) { + Fail("'" + spec.path + "' pairs A " + ShapeText(a.shape) + " with B " + + ShapeText(b.shape) + " for '" + target + + "'; B's second dimension must be A's first (the rank)"); + } + out.pairs_.push_back(std::move(pair)); + } + for (const auto& kv : b_of) { + if (a_of.count(kv.first) == 0) { + Fail("'" + spec.path + "' has a B factor for '" + kv.first + + "' with no matching A factor"); + } + } + return out; +} + +const Ltx2LoraFactorPair* Ltx2LoraAdapter::Find(const std::string& name) const { + for (const Ltx2LoraFactorPair& p : pairs_) { + if (p.target == name) return &p; + } + return nullptr; +} + +Ltx2LoraReferenceFactors Ltx2ResolveLoraReferenceFactors( + const std::vector& adapters) { + Ltx2LoraReferenceFactors out; + if (adapters.empty()) return out; + if (adapters.size() > 1) { + Fail("this port fuses exactly ONE adapter and " + std::to_string(adapters.size()) + + " were given. Upstream's own dubit.py enforces the same (dubit.py:364-365) and " + "hdr_ic_lora.py takes exactly one (hdr_ic_lora.py:271-272); ic_lora.py does " + "accept a list, and N-adapter fusion is recorded as owed by row LTX25-IC-LORA " + "rather than half-built."); + } + // ic_lora.py:150-173. The conflict branches are kept even though one adapter + // cannot trip them, because they are the loop N-adapter support will use and + // removing them would have to reinvent them. + for (const Ltx2LoraAdapter& lora : adapters) { + const int64_t scale = + Ltx2ReadLoraMetadataFactor(lora.metadata(), "reference_downscale_factor", lora.path()); + if (scale != 1) { + if (out.downscale != 1 && out.downscale != scale) { + Fail("conflicting reference_downscale_factor values in LoRAs: already have " + + std::to_string(out.downscale) + ", but " + lora.path() + " specifies " + + std::to_string(scale) + ". Cannot combine LoRAs with different reference scales."); + } + out.downscale = scale; + } + const int64_t temporal = Ltx2ReadLoraMetadataFactor( + lora.metadata(), "reference_temporal_scale_factor", lora.path()); + if (temporal != 1) { + if (out.temporal != 1 && out.temporal != temporal) { + Fail("conflicting reference_temporal_scale_factor values in LoRAs: already have " + + std::to_string(out.temporal) + ", but " + lora.path() + " specifies " + + std::to_string(temporal) + + ". Cannot combine LoRAs with different temporal scales."); + } + out.temporal = temporal; + } + } + return out; +} + +bool Ltx2FuseLoraIntoTensor(const std::vector& adapters, + const std::string& target, vt::DType dtype, int64_t rows, + int64_t cols, uint8_t* buffer, size_t buffer_bytes) { + if (adapters.empty()) return false; + + // The aggregator. BF16 BY DECLARATION — see the dtype note in ltx2_lora.h. + // `has_delta` rather than an empty vector, because a delta of exactly zero is + // a legitimate (if useless) adapter and must still count as fused. + std::vector agg; + bool has_delta = false; + + for (const Ltx2LoraAdapter& lora : adapters) { + const Ltx2LoraFactorPair* pair = lora.Find(target); + if (pair == nullptr) continue; + if (pair->out_features != rows || pair->in_features != cols) { + Fail("'" + lora.path() + "' targets '" + target + "' with a [" + + std::to_string(pair->out_features) + ", " + std::to_string(pair->in_features) + + "] delta, but that tensor is [" + std::to_string(rows) + ", " + + std::to_string(cols) + "] in this checkpoint"); + } + if (has_delta) { + // Unreachable: Ltx2ResolveLoraReferenceFactors refuses more than one + // adapter, so no second product can arrive. Upstream's second form is + // `addmm_(B, A, alpha=strength)`, which rounds differently from the first + // (`fuse_loras.py:103-116`); implementing it here would land a branch + // nothing can select, so it refuses instead of guessing. + Fail("two adapters both target '" + target + + "'. Upstream aggregates them with a SECOND rounding pattern " + "(addmm_ with alpha, fuse_loras.py:115) that this port does not implement, " + "because only one adapter is accepted. Recorded as owed by row LTX25-IC-LORA."); + } + agg.assign(static_cast(rows) * static_cast(cols), 0); + + // `(B * strength) @ A`, upstream's FIRST product form (`fuse_loras.py:113`). + // `B * strength` is a bf16 tensor times a Python float and stays bf16, so + // the scaled factor is ROUNDED TO BF16 BEFORE the matmul — not folded into + // the accumulation. That rounding is the pattern the docstring at + // `fuse_loras.py:103-107` says is preserved, so it is preserved here. + std::vector bs(pair->b.size()); + for (size_t i = 0; i < bs.size(); ++i) { + bs[i] = vt::F32ToBF16(vt::BF16ToF32(pair->b[i]) * static_cast(lora.strength())); + } + // torch's bf16 matmul accumulates in f32 and stores bf16; mirrored. + for (int64_t o = 0; o < rows; ++o) { + const uint16_t* brow = bs.data() + static_cast(o) * static_cast(pair->rank); + for (int64_t i = 0; i < cols; ++i) { + float acc = 0.0F; + for (int64_t k = 0; k < pair->rank; ++k) { + acc += vt::BF16ToF32(brow[k]) * + vt::BF16ToF32(pair->a[static_cast(k) * static_cast(cols) + + static_cast(i)]); + } + agg[static_cast(o) * static_cast(cols) + static_cast(i)] = + vt::F32ToBF16(acc); + } + } + has_delta = true; + } + if (!has_delta) return false; + + const size_t numel = static_cast(rows) * static_cast(cols); + // `deltas.add_(weight)` then `.to(dtype=weight.dtype)` (`fuse_loras.py:67-68`). + // The add happens IN THE AGGREGATOR'S dtype, which is bf16 even when the + // target is f32 — that is upstream's in-place semantics, not an approximation + // of them, and it is why the f32 branch below rounds through bf16. + if (dtype == vt::DType::kBF16) { + if (buffer_bytes != numel * sizeof(uint16_t)) { + Fail("'" + target + "' is " + std::to_string(buffer_bytes) + + " bf16 bytes but its shape needs " + std::to_string(numel * sizeof(uint16_t))); + } + auto* w = reinterpret_cast(buffer); + for (size_t i = 0; i < numel; ++i) { + w[i] = vt::F32ToBF16(vt::BF16ToF32(agg[i]) + vt::BF16ToF32(w[i])); + } + return true; + } + if (dtype == vt::DType::kF32) { + if (buffer_bytes != numel * sizeof(float)) { + Fail("'" + target + "' is " + std::to_string(buffer_bytes) + + " f32 bytes but its shape needs " + std::to_string(numel * sizeof(float))); + } + for (size_t i = 0; i < numel; ++i) { + float w = 0.0F; + std::memcpy(&w, buffer + i * sizeof(float), sizeof(float)); + // add_ on the bf16 aggregator rounds the SUM to bf16 before `.to(f32)`. + const float sum = vt::BF16ToF32(vt::F32ToBF16(vt::BF16ToF32(agg[i]) + w)); + std::memcpy(buffer + i * sizeof(float), &sum, sizeof(float)); + } + return true; + } + Fail("'" + target + "' materialized as a dtype this fuser does not write. FP8 and NVFP4 " + "never reach here: MaterializeDitTensor dequantizes both to BF16 before returning " + "(ltx2_loader.cpp, the F8_E4M3 and U8 branches both `return vt::DType::kBF16`)."); +} + +} // namespace vllm diff --git a/src/vllm/model_executor/models/ltx2_pipeline.cpp b/src/vllm/model_executor/models/ltx2_pipeline.cpp index a54f87763..454c1dd82 100644 --- a/src/vllm/model_executor/models/ltx2_pipeline.cpp +++ b/src/vllm/model_executor/models/ltx2_pipeline.cpp @@ -1175,10 +1175,6 @@ void Ltx2RefuseUnportedPipelineFeature(Ltx2UnportedPipelineFeature feature) { "every ltx-pipelines entry point hard-codes LTX2Scheduler() — so mirroring upstream " "means this port has no scheduler-kind field either." + marker + owed); - case Ltx2UnportedPipelineFeature::kLoraFusion: - Refuse("ltx2: LoRA fusion (ltx-core loader/primitives.py:160 LoraPathStrengthAndSDOps, " - "fused by loader/fuse_loras.py) is out of scope." + - marker + owed); case Ltx2UnportedPipelineFeature::kInt8ConvRot: // VERIFIED UNREACHABLE so nobody re-audits it, and stated as UNREACHABLE rather // than ABSENT because absent is what the first version of this message claimed diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index 56d0c634b..057dda3d5 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -257,6 +257,24 @@ int64_t ExtraInt(const std::map& extras, const std::st } } +// The IC-LoRA strength (utils/args.py:600-611). Upstream's `LoraAction` parses +// it as a plain float and applies no range clamp, so neither does this: a +// negative or >1 strength is a legitimate, if unusual, request that upstream +// honours, and refusing it here would diverge. What IS refused is a value that +// is not a number at all, which upstream's `float()` would raise on too. +double ParseLoraStrength(const std::string& raw) { + try { + size_t consumed = 0; + const double value = std::stod(raw, &consumed); + if (consumed != raw.size()) throw std::invalid_argument("trailing"); + if (!std::isfinite(value)) throw std::invalid_argument("non-finite"); + return value; + } catch (const std::exception&) { + Fail("the extra '" + std::string(kLtx2LoraStrengthExtra) + "' is '" + raw + + "', which is not a finite number"); + } +} + // The one key this family DEFINES and does not SERVE. `Ltx2DurationPredict` is // ported and gated as a brick (`ltx2_duration_head.h`), but nothing here // constructs one, so a supplied path names a file the engine never opens. @@ -281,12 +299,13 @@ constexpr char kLtx2DurationHeadPathExtra[] = "duration_head_path"; // they are no longer trusted: the list below is derived from this file on every // run and compared, and the failure prints the replacement to paste in. // READER ANCHORS (derived and gated by test_ltx2_video): -// 690 745 841 857 859 929 954 1059 1100 +// 709 719 720 782 878 894 896 966 991 1096 1137 const char* const kKnownLoadExtras[] = { kLtx2AudioPromptEmbedsExtra, kLtx2PipelineKindExtra, kLtx2ModelVersionExtra, kLtx2AllowUnportedExtra, kLtx2MaxPhaseExtra, kLtx2DitConfigPathExtra, kLtx2PromptValidRowsExtra, kLtx2EncoderConfigPathExtra, "upsampler_path", kLtx2DurationHeadPathExtra, + kLtx2LoraPathExtra, kLtx2LoraStrengthExtra, }; // FNV-1a over the raw bytes of a float buffer — the `Ltx2ConditioningTrace` @@ -693,6 +712,24 @@ std::unique_ptr Ltx2VideoEngine::Load(const VideoModelParams& p // `Ltx2StreamDitToDevice` dequantizes and uploads ONE TENSOR AT A TIME, so peak // residency is the device copy plus one tensor rather than two whole models. dit_options.widen_to_f32 = !im.on_device; + // The IC-LoRA adapter, fused into the weights as they are materialized. This + // is the production call site for the whole `ltx2_lora.h` family: deleting it + // makes the adapter unreachable, which is what the reachability mutation in + // the row's spec §5.3 checks. + const std::string lora_path = VideoExtra(params.extras, kLtx2LoraPathExtra); + const std::string lora_strength = VideoExtra(params.extras, kLtx2LoraStrengthExtra); + if (lora_path.empty() && !lora_strength.empty()) { + Fail("'" + std::string(kLtx2LoraStrengthExtra) + "' was given without '" + + std::string(kLtx2LoraPathExtra) + + "'. A strength with no adapter fuses nothing, and silently doing nothing is what " + "this refusal exists to prevent."); + } + if (!lora_path.empty()) { + Ltx2LoraSpec spec; + spec.path = lora_path; + if (!lora_strength.empty()) spec.strength = ParseLoraStrength(lora_strength); + dit_options.loras.push_back(std::move(spec)); + } im.dit = im.on_device ? Ltx2StreamDitToDevice(*im.queue, dit_file, dit_options) : Ltx2LoadDitFromSafetensors(dit_file, dit_options); @@ -1334,16 +1371,45 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { "applied on every render. It was never what blocks a last-frame keyframe."); } if (!gen.ref_image_paths.empty() || !gen.ref_video_dir.empty()) { + // WHAT CHANGED, AND WHAT DID NOT. This refusal used to blame the IC-LoRA + // metadata: the `downscale_factor` and `temporal_scale_factor` that + // `Ltx2ConditionVideoByReference` needs live in the adapter's own + // `__metadata__` (iclora_utils.py:30-49) and nothing here read it. Row + // LTX25-IC-LORA (#923) closed that — supply `lora_path` and the factors are + // read, resolved and carried on the checkpoint — so blaming it now would be + // a refusal asserting something the tree no longer has. + // + // The arm is still refused, because there was a SECOND cause the old message + // did not name, and it is the one the LAST-FRAME refusal above already + // spells out in full: `VideoConditionByReferenceLatent` APPENDS tokens + // (reference_video_cond.py:97-100, ported at ltx2_conditioning.cpp via + // AppendTokens), and this phase loop is fixed at the target grid's token + // count — one `Ltx2VideoTokenCount(vshape, 1)` feeds the sigma schedule, the + // `Ltx2ModalityInput` handed to the DiT, and `Ltx2VideoUnpatchify`. Serving + // it means growing that sequence through the DiT and trimming it back + // (`clear_conditioning`, ltx_core/tools.py:88-105). That machinery is shared + // with the last-frame keyframe arm and is owed by its own row. + std::string factors = "no adapter was supplied, so none were read"; + if (im.dit.lora_fused_tensors > 0) { + factors = "the supplied adapter declares downscale=" + + std::to_string(im.dit.lora_reference.downscale) + + " temporal=" + std::to_string(im.dit.lora_reference.temporal) + + ", fused into " + std::to_string(im.dit.lora_fused_tensors) + " tensors"; + } Fail( - "reference-image / reference-video conditioning is not served. The encoder and the " - "placement are both here — `Ltx2ConditionVideoByReference` is ported and gated — but " - "it takes a `downscale_factor` and a `temporal_scale_factor` that must match what the " - "IC-LoRA was TRAINED with (conditioning/types/reference_video_cond.py:36-37, applied at " - ":65-77), and " - "upstream carries those in the LoRA's own metadata, which this project does not read. " - "A guessed pair places the reference plausibly and wrongly, which no output check can " - "see, so it is refused instead. Use first_frame_ppm / first_frame_path for " - "image-to-video."); + "reference-image / reference-video conditioning is not served. What is missing is " + "the TOKEN-APPEND machinery, and NOT the IC-LoRA metadata, which this engine now " + "reads (" + factors + + "). `Ltx2ConvVideoEncode` and `Ltx2ConditionVideoByReference` are both ported and " + "gated, and the adapter's `downscale_factor` / `temporal_scale_factor` are resolved " + "at load (iclora_utils.py:30-49). The gap is that " + "`VideoConditionByReferenceLatent.apply_to` APPENDS tokens to the sequence " + "(conditioning/types/reference_video_cond.py:97-100) and this phase loop is fixed at " + "the target grid's token count, so the appended tokens have nowhere to go and " + "`clear_conditioning` (ltx_core/tools.py:88-105) has nothing to trim. That is the " + "SAME obstruction the last-frame keyframe arm names above, and it is owed by its own " + "row rather than by LTX25-IC-LORA, which closed the metadata half. Use " + "first_frame_ppm / first_frame_path for image-to-video."); } if (!gen.ref_audio_path.empty() || !gen.ref_audio_wav.empty()) { Fail( diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index f58d12c2f..408af3dda 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -329,6 +329,7 @@ target_compile_definitions(test_ltx2_pipeline PRIVATE # own manifests, a few hundred of their own bytes against torch, and synthetic # files for whole-model materialization. All three .inc files live beside the # test source. +vllm_cpp_add_test(test_ltx2_lora vllm/models/test_ltx2_lora.cpp) vllm_cpp_add_test(test_ltx2_loader vllm/models/test_ltx2_loader.cpp) target_include_directories(test_ltx2_loader PRIVATE ${CMAKE_CURRENT_SOURCE_DIR}/vllm/models) vllm_cpp_add_test(test_video_api vllm/entrypoints/openai/test_video_api.cpp) diff --git a/tests/vllm/models/test_ltx2_loader.cpp b/tests/vllm/models/test_ltx2_loader.cpp index cb6ff2e64..5999b6911 100644 --- a/tests/vllm/models/test_ltx2_loader.cpp +++ b/tests/vllm/models/test_ltx2_loader.cpp @@ -2156,3 +2156,217 @@ TEST_CASE("ltx2 loader: BindLtx2DitWeights names the tensor it is missing") { CHECK(what.find(dropped) != std::string::npos); std::remove(path.c_str()); } + +// ───────────────────────────────────────────────────────────────────────────── +// IC-LoRA fusion on the QUANTIZED arms (row LTX25-IC-LORA, issue #923) +// ───────────────────────────────────────────────────────────────────────────── +// +// WHY THESE TWO CASES EXIST AND ARE NOT REDUNDANT WITH test_ltx2_lora. That +// suite gates the fusion arithmetic against a bf16 buffer it hands over itself. +// It cannot show that an FP8 or NVFP4 CHECKPOINT reaches the same code, and the +// design claim this row makes is precisely that one hook serves every arm +// because `MaterializeDitTensor`'s `F8_E4M3` and `U8` branches both +// `return vt::DType::kBF16` before anything else sees a byte. +// +// If that claim were wrong — if either quantized branch returned its packed +// dtype — the fuser's final `Fail` would fire and these cases would throw. So +// they are the gate on the "one hook serves four arms" design, not a second +// copy of the arithmetic gate. +namespace { + +// Write a one-target IC-LoRA whose delta is a known constant, against the same +// contract `BuildSyntheticDit` writes. +std::string WriteLoraFor(const Ltx2DitParams& p, const std::string& target, float scale, + const std::string& path) { + std::vector shape; + for (const Ltx2TensorSpec& spec : vllm::EnumerateLtx2DitTensors(p)) { + if (spec.name == target) shape = spec.shape; + } + REQUIRE_MESSAGE(shape.size() == 2, "LoRA target '", target, "' is not rank 2"); + const int64_t rank = 2; + const std::string module = target.substr(0, target.size() - std::string(".weight").size()); + + std::vector entries; + { + StEntry a; + a.name = "diffusion_model." + module + ".lora_A.weight"; + a.dtype = "BF16"; + a.shape = {rank, shape[1]}; + for (int64_t i = 0; i < rank * shape[1]; ++i) { + const uint16_t v = vt::F32ToBF16(1.0F); + a.bytes.append(reinterpret_cast(&v), sizeof(v)); + } + entries.push_back(std::move(a)); + + StEntry b; + b.name = "diffusion_model." + module + ".lora_B.weight"; + b.dtype = "BF16"; + b.shape = {shape[0], rank}; + for (int64_t i = 0; i < shape[0] * rank; ++i) { + const uint16_t v = vt::F32ToBF16(scale); + b.bytes.append(reinterpret_cast(&v), sizeof(v)); + } + entries.push_back(std::move(b)); + } + WriteSafetensors(entries, path); + return path; +} + +// The delta every case above produces: rank terms of `1.0 * scale`, so a +// uniform `rank * scale` on every element of the target. +constexpr float kLoraScale = 0.5F; +constexpr float kExpectedDelta = 2 * kLoraScale; + +void CheckArmFuses(Ltx2DitQuant quant, const char* tag) { + const Ltx2DitParams p = TinyParams(); + const SyntheticDit syn = BuildSyntheticDit(p, quant, {}); + const std::string dit_path = TmpPath((std::string("lora_") + tag).c_str()); + WriteSafetensors(syn.entries, dit_path); + + // A target that is quantized on BOTH arms: a rank-2 non-table weight. + const std::string target = "transformer_blocks.0.attn1.to_q.weight"; + const std::string lora_path = + WriteLoraFor(p, target, kLoraScale, TmpPath((std::string("lora_a_") + tag).c_str())); + + const SafetensorsFile file = SafetensorsFile::Open(dit_path); + + // The control: the same checkpoint with no adapter, so the delta is measured + // against what this loader actually produces rather than against an + // independently computed dequantization. + const vllm::Ltx2DitCheckpoint plain = vllm::Ltx2LoadDitFromSafetensors(file); + REQUIRE(plain.lora_fused_tensors == 0); + + vllm::Ltx2DitLoadOptions options; + vllm::Ltx2LoraSpec spec; + spec.path = lora_path; + spec.strength = 1.0; + options.loras.push_back(spec); + const vllm::Ltx2DitCheckpoint fused = vllm::Ltx2LoadDitFromSafetensors(file, options); + + INFO("arm = ", std::string(tag)); + // Exactly one contract tensor was touched. + CHECK(fused.lora_fused_tensors == 1); + // No adapter metadata, so both factors are upstream's default of 1. + CHECK(fused.lora_reference.downscale == 1); + CHECK(fused.lora_reference.temporal == 1); + + const vt::Tensor& before = plain.views.at(target); + const vt::Tensor& after = fused.views.at(target); + REQUIRE(before.dtype == vt::DType::kBF16); + REQUIRE(after.dtype == vt::DType::kBF16); + REQUIRE(before.Numel() == after.Numel()); + + // THE VALUE CLAIM: every element moved by exactly the delta. Checked + // element-wise rather than as a norm, because a norm cannot see a delta + // applied to the wrong half of the tensor. + const uint16_t* b0 = before.Ptr(); + const uint16_t* a0 = after.Ptr(); + int64_t moved = 0; + for (int64_t i = 0; i < before.Numel(); ++i) { + // Rounded to bf16, because the STORE is bf16: comparing against an f32 sum + // fails on the last mantissa bit for every element whose sum is not exactly + // representable, which is most of them. + const float want = vt::BF16ToF32(vt::F32ToBF16(vt::BF16ToF32(b0[i]) + kExpectedDelta)); + CHECK(vt::BF16ToF32(a0[i]) == doctest::Approx(want)); + if (a0[i] != b0[i]) ++moved; + } + // And it is not vacuous: the delta is non-zero, so the bytes must actually + // differ. A fuser that returned early would pass the Approx loop above only + // if `kExpectedDelta` were 0. + CHECK(moved == before.Numel()); + + // A tensor NO adapter targets is byte-identical, which is what shows the hook + // is keyed on the name rather than applied to everything materialized. + const std::string untouched = "transformer_blocks.0.attn1.to_k.weight"; + const vt::Tensor& u0 = plain.views.at(untouched); + const vt::Tensor& u1 = fused.views.at(untouched); + REQUIRE(u0.Numel() == u1.Numel()); + int64_t untouched_moved = 0; + for (int64_t i = 0; i < u0.Numel(); ++i) { + if (u0.Ptr()[i] != u1.Ptr()[i]) ++untouched_moved; + } + CHECK(untouched_moved == 0); + + std::remove(dit_path.c_str()); + std::remove(lora_path.c_str()); +} + +} // namespace + +TEST_CASE("ltx2 loader: an IC-LoRA fuses into the FP8 arm") { + // The arm most users run. `MaterializeDitTensor`'s F8_E4M3 branch calls + // DequantFp8ToBf16 and returns kBF16, so the delta lands on a dequantized + // weight and no FP8 quantizer is needed — which is the whole reason this port + // can serve every arm with one hook (ltx2_lora.h, the WHERE IT IS APPLIED + // note). + CheckArmFuses(Ltx2DitQuant::kFp8, "fp8"); +} + +TEST_CASE("ltx2 loader: an IC-LoRA fuses into the NVFP4 arm") { + // Same claim, the other quantized branch: U8-packed NVFP4 with its two scale + // sidecars, dequantized by Ltx2DequantNvfp4ToBf16 which also returns kBF16. + // + // Upstream RE-QUANTIZES here (quantization/nvfp4/fuse.py:40-47) because it + // keeps the packed weight resident for its NVFP4 kernels. This tree keeps + // bf16 (ltx2_loader.h's DTYPE note), so there is nothing to re-quantize into + // and no NVFP4 quantizer in the tree to do it with. The divergence is + // deliberate and recorded in the row's spec §3.1: our fused weight SKIPS + // upstream's lossy round trip, at no extra bytes. + CheckArmFuses(Ltx2DitQuant::kNvfp4, "nvfp4"); +} + +TEST_CASE("ltx2 loader: an adapter that fuses into NOTHING refuses rather than loading green") { + // Reachable because the contract carries rank-1 and rank-3 tensors a LoRA + // pair can legitimately name but never fuse into. A load that reported + // success here would render byte-identically to no adapter at all. + const Ltx2DitParams p = TinyParams(); + const SyntheticDit syn = BuildSyntheticDit(p, Ltx2DitQuant::kFp8, {}); + const std::string dit_path = TmpPath("lora_none"); + WriteSafetensors(syn.entries, dit_path); + + // A non-rank-2 contract tensor: the scale-shift table is [rows, inner] rank 2, + // so pick a genuinely rank-1 one. + std::string rank1; + for (const Ltx2TensorSpec& spec : vllm::EnumerateLtx2DitTensors(p)) { + const bool is_weight = spec.name.size() > 7 && + spec.name.compare(spec.name.size() - 7, 7, ".weight") == 0; + if (spec.shape.size() == 1 && is_weight) { + rank1 = spec.name; + break; + } + } + REQUIRE_MESSAGE(!rank1.empty(), "the contract has no rank-1 tensor to build this case on"); + + const std::string module = rank1.substr(0, rank1.size() - std::string(".weight").size()); + std::vector entries; + for (const char* side : {".lora_A.weight", ".lora_B.weight"}) { + StEntry e; + e.name = "diffusion_model." + module + side; + e.dtype = "BF16"; + e.shape = {1, 1}; + const uint16_t v = vt::F32ToBF16(1.0F); + e.bytes.append(reinterpret_cast(&v), sizeof(v)); + entries.push_back(std::move(e)); + } + const std::string lora_path = TmpPath("lora_none_a"); + WriteSafetensors(entries, lora_path); + + const SafetensorsFile file = SafetensorsFile::Open(dit_path); + vllm::Ltx2DitLoadOptions options; + vllm::Ltx2LoraSpec spec; + spec.path = lora_path; + options.loras.push_back(spec); + + std::string what; + try { + (void)vllm::Ltx2LoadDitFromSafetensors(file, options); + } catch (const std::exception& e) { + what = e.what(); + } + INFO("what: ", what); + CHECK(what.find("fused into ZERO tensors") != std::string::npos); + CHECK(what.find(lora_path) != std::string::npos); + + std::remove(dit_path.c_str()); + std::remove(lora_path.c_str()); +} diff --git a/tests/vllm/models/test_ltx2_lora.cpp b/tests/vllm/models/test_ltx2_lora.cpp new file mode 100644 index 000000000..f54dc64f8 --- /dev/null +++ b/tests/vllm/models/test_ltx2_lora.cpp @@ -0,0 +1,607 @@ +// LTX-2.5 IC-LoRA — the adapter format, its metadata, and the fusion arithmetic. +// +// Row LTX25-IC-LORA, .agents/specs/ltx25-ic-lora.md, issue #923. +// +// WHAT GATES WHAT, because two of these look alike and are not: +// +// * THE ARITHMETIC. `W + (B * strength) @ A`, checked against values computed +// from upstream's own expression (`ltx-core loader/fuse_loras.py:99-116`) at +// Lightricks/LTX-2 @ fd4ded7f2d88d3da713abcdd4ad41ecc4a9314ca. There is no +// upstream test to port — the pinned repository ships NONE, measured with a +// positive control in the row's spec §5.1 — so this is a source-derived value +// gate and is labelled as one rather than called a ported test. +// +// * THE ACCUMULATOR'S DTYPE. Upstream aggregates in BF16 in all four of its +// fuse rules. A token gate cannot see an accumulator that is too wide, and +// neither can the arithmetic case above at ordinary magnitudes — so one case +// here is built specifically so that f32 and bf16 accumulation DISAGREE in +// the stored result, and it is the only thing standing between this port and +// a silently more precise path than the one it mirrors. +// +// The quantized arms are gated in test_ltx2_loader.cpp, where the synthetic FP8 +// and NVFP4 DiT builders live, and reachability in test_ltx2_video.cpp, where +// the engine fixture does. +#include + +#include +#include +#include +#include +#include +#include +#include + +#include + +#include "vllm/model_executor/models/ltx2_lora.h" +#include "vt/dtype.h" + +namespace { + +// ── a minimal safetensors writer ───────────────────────────────────────────── +// +// Local rather than shared: a LoRA file is two rank-2 BF16 tensors and a +// `__metadata__` object, and the fixtures that build whole DiTs are far heavier +// than that. Bytes are laid out exactly as the reader expects, so a malformed +// case here is malformed for the reason the case names. + +struct LoraEntry { + std::string name; + std::string dtype; // "BF16" | "F32" | "U8" + std::vector shape; + std::vector values; +}; + +std::string TempPath(const char* tag) { + static int counter = 0; + return std::string("/tmp/ltx2_lora_test_") + tag + "_" + std::to_string(getpid()) + "_" + + std::to_string(++counter) + ".safetensors"; +} + +void WriteLoraFile(const std::vector& entries, + const std::map& metadata, + const std::string& path) { + std::string header = "{"; + bool first = true; + if (!metadata.empty()) { + header += "\"__metadata__\":{"; + bool mfirst = true; + for (const auto& kv : metadata) { + if (!mfirst) header += ","; + mfirst = false; + header += "\"" + kv.first + "\":\"" + kv.second + "\""; + } + header += "}"; + first = false; + } + std::string payload; + for (const LoraEntry& e : entries) { + const size_t at = payload.size(); + size_t bytes = 0; + if (e.dtype == "F32") { + bytes = e.values.size() * sizeof(float); + payload.resize(at + bytes); + std::memcpy(&payload[at], e.values.data(), bytes); + } else if (e.dtype == "BF16") { + bytes = e.values.size() * sizeof(uint16_t); + for (const float v : e.values) { + const uint16_t b = vt::F32ToBF16(v); + payload.append(reinterpret_cast(&b), sizeof(b)); + } + } else { + bytes = e.values.size(); + for (const float v : e.values) payload.push_back(static_cast(v)); + } + if (!first) header += ","; + first = false; + header += "\"" + e.name + "\":{\"dtype\":\"" + e.dtype + "\",\"shape\":["; + for (size_t i = 0; i < e.shape.size(); ++i) { + header += (i != 0 ? "," : "") + std::to_string(e.shape[i]); + } + header += "],\"data_offsets\":[" + std::to_string(at) + "," + + std::to_string(at + bytes) + "]}"; + } + header += "}"; + while (header.size() % 8 != 0) header += " "; + const uint64_t n = header.size(); + std::string file(reinterpret_cast(&n), sizeof(n)); + file += header; + file += payload; + FILE* f = std::fopen(path.c_str(), "wb"); + REQUIRE(f != nullptr); + std::fwrite(file.data(), 1, file.size(), f); + std::fclose(f); +} + +// The one target every simple case uses. A real contract name, so the +// unknown-target refusal is genuinely testing name resolution and not a typo. +const char* const kTarget = "transformer_blocks.0.attn1.to_q.weight"; +const char* const kModule = "transformer_blocks.0.attn1.to_q"; + +// Write a rank-`rank` adapter for `kModule`, with `b` [out, rank] and `a` +// [rank, in] given row-major. +std::string WriteAdapter(int64_t out_features, int64_t rank, int64_t in_features, + const std::vector& b, const std::vector& a, + const std::map& metadata = {}, + const std::string& module = kModule, + const std::string& prefix = "diffusion_model.") { + const std::string path = TempPath("adapter"); + WriteLoraFile( + { + {prefix + module + ".lora_A.weight", "BF16", {rank, in_features}, a}, + {prefix + module + ".lora_B.weight", "BF16", {out_features, rank}, b}, + }, + metadata, path); + return path; +} + +std::string Caught(const std::function& fn) { + try { + fn(); + } catch (const std::exception& e) { + return e.what(); + } + return ""; +} + +bool Mentions(const std::string& haystack, const std::string& needle) { + return haystack.find(needle) != std::string::npos; +} + +std::vector ContractWith(const std::string& name) { return {name}; } + +// Fuse one adapter into a bf16 buffer and read the result back as floats. +std::vector FuseBf16(const std::vector& adapters, + const std::string& target, int64_t rows, int64_t cols, + const std::vector& weight, bool* out_fused = nullptr) { + std::vector buffer(weight.size()); + for (size_t i = 0; i < weight.size(); ++i) buffer[i] = vt::F32ToBF16(weight[i]); + const bool fused = vllm::Ltx2FuseLoraIntoTensor( + adapters, target, vt::DType::kBF16, rows, cols, + reinterpret_cast(buffer.data()), buffer.size() * sizeof(uint16_t)); + if (out_fused != nullptr) *out_fused = fused; + std::vector out(buffer.size()); + for (size_t i = 0; i < buffer.size(); ++i) out[i] = vt::BF16ToF32(buffer[i]); + return out; +} + +} // namespace + +// ───────────────────────────────────────────────────────────────────────────── +// The key shape +// ───────────────────────────────────────────────────────────────────────────── + +TEST_CASE("ltx2 lora: a factor key resolves onto the contract name") { + std::string target; + bool is_a = false; + + // LTXV_LORA_COMFY_RENAMING_MAP strips `diffusion_model.` (sd_ops.py:136), and + // `_affected_weight_keys` rewrites the suffix (fuse_loras.py:185-186). + CHECK(vllm::Ltx2LoraContractName("diffusion_model." + std::string(kModule) + + ".lora_A.weight", + &target, &is_a)); + CHECK(target == kTarget); + CHECK(is_a); + + CHECK(vllm::Ltx2LoraContractName("diffusion_model." + std::string(kModule) + + ".lora_B.weight", + &target, &is_a)); + CHECK(target == kTarget); + CHECK_FALSE(is_a); + + // The prefix is OPTIONAL: a PEFT-style adapter without it resolves the same. + CHECK(vllm::Ltx2LoraContractName(std::string(kModule) + ".lora_A.weight", &target, &is_a)); + CHECK(target == kTarget); + + // Anything that is not a factor is not one. `.weight` alone is the TARGET, and + // reading it as a factor would fuse a weight into itself. + CHECK_FALSE(vllm::Ltx2LoraContractName(kTarget, &target, &is_a)); + CHECK_FALSE(vllm::Ltx2LoraContractName("diffusion_model.x.lora_A.bias", &target, &is_a)); + CHECK_FALSE(vllm::Ltx2LoraContractName("", &target, &is_a)); +} + +// ───────────────────────────────────────────────────────────────────────────── +// The arithmetic +// ───────────────────────────────────────────────────────────────────────────── + +TEST_CASE("ltx2 lora: the fused weight is W + (B * strength) @ A") { + // out=2, rank=2, in=2. Chosen so every product is exactly representable in + // bf16, which makes this case about the FORMULA and leaves the rounding + // question entirely to the dtype case below. + // + // B = [[1, 2], A = [[1, 0], B @ A = [[1, 2], + // [0, 1]] [0, 1]] [0, 1]] + const std::vector b = {1, 2, 0, 1}; + const std::vector a = {1, 0, 0, 1}; + const std::vector w = {10, 20, 30, 40}; + + SUBCASE("strength 1.0") { + const std::string path = WriteAdapter(2, 2, 2, b, a); + vllm::Ltx2LoraSpec spec; + spec.path = path; + spec.strength = 1.0; + std::vector adapters; + adapters.push_back(vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget))); + + bool fused = false; + const std::vector got = FuseBf16(adapters, kTarget, 2, 2, w, &fused); + CHECK(fused); + CHECK(got[0] == doctest::Approx(11.0)); // 10 + 1 + CHECK(got[1] == doctest::Approx(22.0)); // 20 + 2 + CHECK(got[2] == doctest::Approx(30.0)); // 30 + 0 + CHECK(got[3] == doctest::Approx(41.0)); // 40 + 1 + std::remove(path.c_str()); + } + + SUBCASE("strength scales the delta and NOT the weight") { + const std::string path = WriteAdapter(2, 2, 2, b, a); + vllm::Ltx2LoraSpec spec; + spec.path = path; + spec.strength = 0.5; + std::vector adapters; + adapters.push_back(vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget))); + + const std::vector got = FuseBf16(adapters, kTarget, 2, 2, w); + // Half the delta, all of the weight. A strength that scaled the sum would + // give 5.5 / 11 / 15 / 20.5 and is what this distinguishes. + CHECK(got[0] == doctest::Approx(10.5)); + CHECK(got[1] == doctest::Approx(21.0)); + CHECK(got[2] == doctest::Approx(30.0)); + CHECK(got[3] == doctest::Approx(40.5)); + std::remove(path.c_str()); + } + + SUBCASE("strength 0 is a no-op on the values but still counts as fused") { + const std::string path = WriteAdapter(2, 2, 2, b, a); + vllm::Ltx2LoraSpec spec; + spec.path = path; + spec.strength = 0.0; + std::vector adapters; + adapters.push_back(vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget))); + + bool fused = false; + const std::vector got = FuseBf16(adapters, kTarget, 2, 2, w, &fused); + // `fused` is about whether a delta was COMPUTED, not whether it was + // non-zero: a zero-strength adapter is a legitimate request, and reporting + // it as "fused nothing" would trip the load-time refusal for the wrong + // reason. + CHECK(fused); + for (size_t i = 0; i < w.size(); ++i) CHECK(got[i] == doctest::Approx(w[i])); + std::remove(path.c_str()); + } +} + +TEST_CASE("ltx2 lora: a tensor no adapter targets is left alone") { + const std::string path = WriteAdapter(2, 2, 2, {1, 2, 0, 1}, {1, 0, 0, 1}); + vllm::Ltx2LoraSpec spec; + spec.path = path; + std::vector adapters; + adapters.push_back(vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget))); + + const std::vector w = {10, 20, 30, 40}; + bool fused = true; + const std::vector got = + FuseBf16(adapters, "transformer_blocks.0.attn1.to_k.weight", 2, 2, w, &fused); + CHECK_FALSE(fused); + for (size_t i = 0; i < w.size(); ++i) CHECK(got[i] == doctest::Approx(w[i])); + std::remove(path.c_str()); +} + +// ───────────────────────────────────────────────────────────────────────────── +// THE DTYPE GATE +// ───────────────────────────────────────────────────────────────────────────── + +TEST_CASE("ltx2 lora: the delta accumulates in BF16, not f32") { + // WHY THIS CASE EXISTS. Upstream sets `aggregation_dtype=torch.bfloat16` in + // every one of its four fuse rules (fuse_loras.py:71, fp8_cast.py:239, + // fp8_scaled_mm.py:189, nvfp4/fuse.py:50). Accumulating in f32 instead would + // be MORE precise, so no golden and no token comparison could ever fail on it + // - which is exactly why the choice needs a gate of its own. + // + // THE CONSTRUCTION, and why the obvious one does not work. `B * strength` is + // rounded to bf16 BEFORE the matmul (fuse_loras.py:113: a bf16 tensor times a + // Python float stays bf16), so the two arms differ by that rounding. But the + // RESULT is stored bf16 too, and at rank 1 the final store rounds the + // difference straight back out again - a first version of this case did + // exactly that and passed under BOTH arms. That is the coverage hole this + // comment exists to stop being reopened. + // + // So the per-term error is ACCUMULATED until it exceeds the bf16 step of the + // sum. With B = A = 1 and rank 192: + // + // strength = 1 + 2^-8, and bf16(1 * strength) = 1.0 exactly - 2^-8 is half + // of bf16's 2^-7 step at 1.0, so round-to-nearest-EVEN takes it down. + // upstream: 192 terms of 1.0 = 192.0, stored bf16 -> 192.0 + // f32 fold: 192 * (1 + 2^-8) = 192.75, stored bf16 -> 193.0 + // + // One bf16 step apart in the stored result, so the store cannot hide it. + const int64_t kRank = 192; + const double strength = 1.0 + 1.0 / 256.0; + const std::vector b(static_cast(kRank), 1.0F); + const std::vector a(static_cast(kRank), 1.0F); + + const std::string path = WriteAdapter(1, kRank, 1, b, a); + vllm::Ltx2LoraSpec spec; + spec.path = path; + spec.strength = strength; + std::vector adapters; + adapters.push_back(vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget))); + + const std::vector got = FuseBf16(adapters, kTarget, 1, 1, {0.0F}); + + // The bf16 answer, which is upstream's. + CHECK(got[0] == doctest::Approx(192.0)); + // Checked on the BIT PATTERN too, because `doctest::Approx` carries a relative + // epsilon and these two are only one part in 192 apart. + CHECK(vt::F32ToBF16(got[0]) == vt::F32ToBF16(192.0F)); + // The f32 answer, which it must NOT be. + CHECK(vt::F32ToBF16(got[0]) != vt::F32ToBF16(193.0F)); + std::remove(path.c_str()); +} + +TEST_CASE("ltx2 lora: the reference factors come from the adapter's metadata") { + SUBCASE("absent is 1, which is upstream's default") { + const std::string path = WriteAdapter(2, 2, 2, {1, 2, 0, 1}, {1, 0, 0, 1}); + vllm::Ltx2LoraSpec spec; + spec.path = path; + std::vector adapters; + adapters.push_back(vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget))); + const vllm::Ltx2LoraReferenceFactors f = vllm::Ltx2ResolveLoraReferenceFactors(adapters); + CHECK(f.downscale == 1); + CHECK(f.temporal == 1); + std::remove(path.c_str()); + } + + SUBCASE("declared values are read") { + const std::string path = + WriteAdapter(2, 2, 2, {1, 2, 0, 1}, {1, 0, 0, 1}, + {{"reference_downscale_factor", "2"}, + {"reference_temporal_scale_factor", "4"}}); + vllm::Ltx2LoraSpec spec; + spec.path = path; + std::vector adapters; + adapters.push_back(vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget))); + const vllm::Ltx2LoraReferenceFactors f = vllm::Ltx2ResolveLoraReferenceFactors(adapters); + // These are the two numbers the reference refusal named as unreadable. + CHECK(f.downscale == 2); + CHECK(f.temporal == 4); + std::remove(path.c_str()); + } + + SUBCASE("a malformed value REFUSES rather than silently reverting to 1") { + // The one place this port deliberately diverges from upstream, which + // swallows every exception and returns 1 (iclora_utils.py:36-38). A factor + // that reverts to 1 places the reference plausibly and wrongly, and no + // output check can see that. + for (const char* bad : {"two", "0", "-3", "2.5", ""}) { + const std::string path = WriteAdapter(2, 2, 2, {1, 2, 0, 1}, {1, 0, 0, 1}, + {{"reference_downscale_factor", bad}}); + vllm::Ltx2LoraSpec spec; + spec.path = path; + std::vector adapters; + adapters.push_back(vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget))); + const std::string err = + Caught([&] { (void)vllm::Ltx2ResolveLoraReferenceFactors(adapters); }); + INFO("value = '", bad, "' error = ", err); + CHECK(Mentions(err, "reference_downscale_factor")); + CHECK(Mentions(err, "not a positive integer")); + std::remove(path.c_str()); + } + } +} + +// ───────────────────────────────────────────────────────────────────────────── +// The refusals +// ───────────────────────────────────────────────────────────────────────────── + +TEST_CASE("ltx2 lora: an adapter naming a module the contract lacks refuses BY NAME") { + // Upstream SKIPS this key (fuse_loras.py:135-137). Here it refuses, because + // the contract is a fixed enumerated set and a skip would absorb a misnamed + // key and an inapplicable one alike. The row's spec §4.1 argues it. + const std::string path = WriteAdapter(2, 2, 2, {1, 2, 0, 1}, {1, 0, 0, 1}, {}, + "transformer_blocks.0.not_a_real_module"); + vllm::Ltx2LoraSpec spec; + spec.path = path; + const std::string err = Caught([&] { + (void)vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget)); + }); + INFO("error = ", err); + // BY NAME: the offending target is in the message, not just "a key". + CHECK(Mentions(err, "transformer_blocks.0.not_a_real_module.weight")); + CHECK(Mentions(err, "does not bind")); + // And it says what upstream would have done, so the divergence is visible to + // whoever is reading the refusal rather than only to whoever reads the spec. + CHECK(Mentions(err, "fuse_loras.py:135-137")); + std::remove(path.c_str()); +} + +TEST_CASE("ltx2 lora: a file that is not an adapter refuses rather than loading nothing") { + const std::string path = TempPath("empty"); + WriteLoraFile({{"some.weight", "BF16", {2, 2}, {1, 2, 3, 4}}}, {}, path); + vllm::Ltx2LoraSpec spec; + spec.path = path; + const std::string err = + Caught([&] { (void)vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget)); }); + INFO("error = ", err); + CHECK(Mentions(err, "no `.lora_A.weight`")); + std::remove(path.c_str()); +} + +TEST_CASE("ltx2 lora: a half pair refuses, naming the side that is missing") { + SUBCASE("A with no B") { + const std::string path = TempPath("half_a"); + WriteLoraFile({{std::string(kModule) + ".lora_A.weight", "BF16", {1, 2}, {1, 1}}}, {}, + path); + vllm::Ltx2LoraSpec spec; + spec.path = path; + const std::string err = + Caught([&] { (void)vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget)); }); + INFO("error = ", err); + CHECK(Mentions(err, "no matching B factor")); + std::remove(path.c_str()); + } + SUBCASE("B with no A") { + const std::string path = TempPath("half_b"); + WriteLoraFile({{std::string(kModule) + ".lora_B.weight", "BF16", {2, 1}, {1, 1}}}, {}, + path); + vllm::Ltx2LoraSpec spec; + spec.path = path; + const std::string err = + Caught([&] { (void)vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget)); }); + INFO("error = ", err); + CHECK(Mentions(err, "no matching A factor")); + std::remove(path.c_str()); + } +} + +TEST_CASE("ltx2 lora: a delta whose shape disagrees with the target refuses") { + // out=2 in=2 in the adapter, against a [4, 4] tensor. Fusing regardless would + // read past the delta and write a corrupt weight that nothing downstream can + // attribute. + const std::string path = WriteAdapter(2, 2, 2, {1, 2, 0, 1}, {1, 0, 0, 1}); + vllm::Ltx2LoraSpec spec; + spec.path = path; + std::vector adapters; + adapters.push_back(vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget))); + + std::vector buffer(16, 0); + const std::string err = Caught([&] { + (void)vllm::Ltx2FuseLoraIntoTensor(adapters, kTarget, vt::DType::kBF16, 4, 4, + reinterpret_cast(buffer.data()), + buffer.size() * sizeof(uint16_t)); + }); + INFO("error = ", err); + CHECK(Mentions(err, "[2, 2] delta")); + CHECK(Mentions(err, "[4, 4]")); + std::remove(path.c_str()); +} + +TEST_CASE("ltx2 lora: more than one adapter refuses BY NAME") { + // Upstream's ic_lora.py accepts a list; dubit.py:364-365 enforces exactly one + // and hdr_ic_lora.py:271-272 takes exactly one. N-adapter fusion is recorded + // as owed by the row rather than half-built, and the refusal says so. + const std::string a = WriteAdapter(2, 2, 2, {1, 2, 0, 1}, {1, 0, 0, 1}); + const std::string b = WriteAdapter(2, 2, 2, {1, 2, 0, 1}, {1, 0, 0, 1}); + std::vector adapters; + for (const std::string& p : {a, b}) { + vllm::Ltx2LoraSpec spec; + spec.path = p; + adapters.push_back(vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget))); + } + const std::string err = + Caught([&] { (void)vllm::Ltx2ResolveLoraReferenceFactors(adapters); }); + INFO("error = ", err); + CHECK(Mentions(err, "exactly ONE adapter")); + CHECK(Mentions(err, "LTX25-IC-LORA")); + std::remove(a.c_str()); + std::remove(b.c_str()); +} + +TEST_CASE("ltx2 lora: an unreadable factor dtype refuses, naming the RIGHT factor") { + // THE KEY NAME IS THE POINT, not just the dtype. A first version of this + // reader kept ONE key per target, so it held whichever of A/B appeared last + // in the header and reported a malformed A factor under the B factor's name. + // "Refuse BY NAME" defeated by the message itself, and a case that checked + // only for "U8" passed straight through it. So each subcase makes exactly one + // side unreadable and asserts the message names THAT side and not the other. + SUBCASE("the A factor is unreadable") { + const std::string path = TempPath("u8_a"); + WriteLoraFile( + { + {std::string(kModule) + ".lora_A.weight", "U8", {1, 2}, {1, 2}}, + {std::string(kModule) + ".lora_B.weight", "BF16", {2, 1}, {1, 1}}, + }, + {}, path); + vllm::Ltx2LoraSpec spec; + spec.path = path; + const std::string err = + Caught([&] { (void)vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget)); }); + INFO("error = ", err); + CHECK(Mentions(err, "U8")); + CHECK(Mentions(err, "BF16 or F32")); + CHECK(Mentions(err, ".lora_A.weight")); + CHECK_FALSE(Mentions(err, ".lora_B.weight")); + std::remove(path.c_str()); + } + SUBCASE("the B factor is unreadable") { + const std::string path = TempPath("u8_b"); + WriteLoraFile( + { + {std::string(kModule) + ".lora_A.weight", "BF16", {1, 2}, {1, 1}}, + {std::string(kModule) + ".lora_B.weight", "U8", {2, 1}, {1, 2}}, + }, + {}, path); + vllm::Ltx2LoraSpec spec; + spec.path = path; + const std::string err = + Caught([&] { (void)vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget)); }); + INFO("error = ", err); + CHECK(Mentions(err, "U8")); + CHECK(Mentions(err, ".lora_B.weight")); + CHECK_FALSE(Mentions(err, ".lora_A.weight")); + std::remove(path.c_str()); + } +} + +TEST_CASE("ltx2 lora: an F32 adapter is NARROWED to bf16, not kept f32") { + // fuse_loras.py:202-203 casts both factors to the rule's aggregation dtype + // before the matmul. An F32 adapter held at F32 would widen the accumulator + // through the back door - the same defect the dtype case above gates, arriving + // by a different route, so it uses the same accumulate-until-it-shows shape. + // + // B[k] = 1 + 2^-8 in F32, A[k] = 1, rank 192, strength 1. + // narrowed to bf16: 192 terms of 1.0 = 192.0 + // kept f32: 192 * (1 + 2^-8) = 192.75 -> bf16 -> 193.0 + const int64_t kRank = 192; + const float unrepresentable = 1.0F + 1.0F / 256.0F; + const std::string path = TempPath("f32_adapter"); + WriteLoraFile( + { + {std::string(kModule) + ".lora_A.weight", + "F32", + {kRank, 1}, + std::vector(static_cast(kRank), 1.0F)}, + {std::string(kModule) + ".lora_B.weight", + "F32", + {1, kRank}, + std::vector(static_cast(kRank), unrepresentable)}, + }, + {}, path); + vllm::Ltx2LoraSpec spec; + spec.path = path; + spec.strength = 1.0; + std::vector adapters; + adapters.push_back(vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget))); + + const std::vector got = FuseBf16(adapters, kTarget, 1, 1, {0.0F}); + CHECK(vt::F32ToBF16(got[0]) == vt::F32ToBF16(192.0F)); + CHECK(vt::F32ToBF16(got[0]) != vt::F32ToBF16(193.0F)); + std::remove(path.c_str()); +} + +TEST_CASE("ltx2 lora: the f32 target branch rounds through the bf16 accumulator") { + // The scale_shift tables are the only F32 tensors in the contract. Upstream's + // `_bf16_fuse` does `deltas.add_(weight)` IN PLACE on the bf16 aggregator and + // only then casts to the weight's dtype (fuse_loras.py:67-68) — so an f32 + // target still rounds the SUM through bf16. Mirrored rather than "improved". + const std::string path = WriteAdapter(1, 1, 1, {1.0F}, {1.0F}); + vllm::Ltx2LoraSpec spec; + spec.path = path; + std::vector adapters; + adapters.push_back(vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget))); + + // 1 + 2^-9 in f32; adding a delta of 1.0 gives 2.001953125, which bf16 cannot + // hold and rounds to 2.0. + float weight = 1.0F + 1.0F / 512.0F; + std::vector buffer(sizeof(float)); + std::memcpy(buffer.data(), &weight, sizeof(float)); + const bool fused = vllm::Ltx2FuseLoraIntoTensor(adapters, kTarget, vt::DType::kF32, 1, 1, + buffer.data(), buffer.size()); + CHECK(fused); + float got = 0.0F; + std::memcpy(&got, buffer.data(), sizeof(float)); + CHECK(got == doctest::Approx(2.0)); + // Not 2.001953125: that is what an f32-throughout add would produce. + CHECK(got != doctest::Approx(weight + 1.0F)); + std::remove(path.c_str()); +} diff --git a/tests/vllm/models/test_ltx2_pipeline.cpp b/tests/vllm/models/test_ltx2_pipeline.cpp index b01b8fc3a..00b141351 100644 --- a/tests/vllm/models/test_ltx2_pipeline.cpp +++ b/tests/vllm/models/test_ltx2_pipeline.cpp @@ -1266,7 +1266,13 @@ TEST_CASE("ltx2 every out-of-scope feature is refused BY NAME") { // what this port has, which is the defect this row closes. const std::vector> markers = { {vllm::Ltx2UnportedPipelineFeature::kBetaScheduler, "BetaScheduler"}, - {vllm::Ltx2UnportedPipelineFeature::kLoraFusion, "LoRA"}, + // `kLoraFusion` WAS here and is RETIRED (row LTX25-IC-LORA, #923). It is + // not moved to `reachable`: it is gone. The `lora_path` load extra now + // fuses an adapter, so there is no unported LoRA-fusion feature left to + // name, and #691's prediction — that this list gates message TEXT and + // would not notice the property going false — is why the enumerator was + // deleted rather than reclassified. The compiler is what caught it, which + // is a weaker guarantee than #691 asks for and does not close #691. {vllm::Ltx2UnportedPipelineFeature::kInt8ConvRot, "int8-convrot"}, {vllm::Ltx2UnportedPipelineFeature::kMultiGpuParallelism, "multi-GPU"}, }; diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index c997b61b6..bf50cab7d 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -697,6 +697,10 @@ TEST_CASE("ltx2 video: every accepted load extra is READ by something") { vllm::multimodal::kLtx2MaxPhaseExtra, vllm::multimodal::kLtx2DitConfigPathExtra, vllm::multimodal::kLtx2PromptValidRowsExtra, vllm::multimodal::kLtx2EncoderConfigPathExtra, "upsampler_path", + // Row LTX25-IC-LORA (#923): the IC-LoRA adapter and its strength. Both + // have readers -- `lora_path` builds an `Ltx2LoraSpec` and `lora_strength` + // is parsed into it -- so they belong here and not in `refused`. + vllm::multimodal::kLtx2LoraPathExtra, vllm::multimodal::kLtx2LoraStrengthExtra, }; // The keys the family defines and does NOT serve. Growing this list is a // deliberate act; growing it silently is the defect #611 records. @@ -724,7 +728,7 @@ TEST_CASE("ltx2 video: every accepted load extra is READ by something") { // Every name this row inventoried is still accepted... for (const std::string& key : served) CHECK(listing.find(key) != std::string::npos); for (const std::string& key : refused) CHECK(listing.find(key) != std::string::npos); - // ...and there is no ELEVENTH name that this inventory has never seen. The + // ...and there is no THIRTEENTH name that this inventory has never seen. The // separator is ", ", so the count is one more than the separators. size_t names = 1; for (size_t at = listing.find(", "); at != std::string::npos; at = listing.find(", ", at + 2)) { @@ -827,13 +831,14 @@ TEST_CASE("ltx2 video: the recorded reader anchors are the ones in the source") } REQUIRE(array_end > array_line); - // The nine SERVED keys, by the token each is spelled with in the source. Order + // The eleven SERVED keys, by the token each is spelled with in the source. Order // is irrelevant — the comparison is on the sorted set — so this list is not a // second place the anchors live. const std::vector served_tokens = { "kLtx2AudioPromptEmbedsExtra", "kLtx2PipelineKindExtra", "kLtx2ModelVersionExtra", "kLtx2AllowUnportedExtra", "kLtx2MaxPhaseExtra", "kLtx2DitConfigPathExtra", "kLtx2PromptValidRowsExtra", "kLtx2EncoderConfigPathExtra", "\"upsampler_path\"", + "kLtx2LoraPathExtra", "kLtx2LoraStrengthExtra", }; std::vector derived; for (const std::string& token : served_tokens) { @@ -1184,14 +1189,27 @@ TEST_CASE("ltx2 video: keyframe and reference conditioning is refused BY WHAT IS CHECK(msg.find("#658") != std::string::npos); } } - SUBCASE("a reference video names the IC-LoRA metadata this project does not read") { + SUBCASE("a reference video names TOKEN-APPEND, and no longer blames the LoRA metadata") { + // REPLACED, not relaxed (row LTX25-IC-LORA, #923). This subcase used to + // assert the refusal blamed the unread IC-LoRA metadata. That reason was + // true and is now false: `lora_path` reads the adapter's + // `reference_downscale_factor` / `reference_temporal_scale_factor` + // (iclora_utils.py:30-49) and carries them on the checkpoint. The arm is + // still refused, on the cause the message did not previously name. const std::string msg = refusal("a reference video", [](vllm::multimodal::VideoGenParams& g, const Workspace& w) { g.ref_video_dir = w.root; }); INFO(msg); - CHECK(msg.find("temporal_scale_factor") != std::string::npos); - CHECK(msg.find("LoRA") != std::string::npos); + // The cause that actually remains, and the upstream symbol that has it. + CHECK(msg.find("TOKEN-APPEND") != std::string::npos); + CHECK(msg.find("reference_video_cond.py") != std::string::npos); + CHECK(msg.find("clear_conditioning") != std::string::npos); + // And the retired reason must NOT come back as the thing that is missing. + // A refusal that names a blocker the tree no longer has is the defect row + // LTX25-RETIRE-DEAD-ARMS exists to retire, arriving from the other side. + CHECK(msg.find("which this project does not read") == std::string::npos); + CHECK(msg.find("NOT the IC-LoRA metadata") != std::string::npos); } SUBCASE("reference audio names the AUDIO encoder, which this row did not build") { const std::string msg = refusal("reference audio", @@ -2701,3 +2719,240 @@ TEST_CASE("ltx2 video: a trace for a render that never completed says so") { // ...and yet no render came out of it. This is the whole assertion. CHECK_FALSE(trace.completed); } + +// ───────────────────────────────────────────────────────────────────────────── +// IC-LoRA reachability (row LTX25-IC-LORA, issue #923) +// ───────────────────────────────────────────────────────────────────────────── +// +// THE QUESTION THIS ANSWERS is not "does the fuser work" — test_ltx2_lora +// answers that by calling it. It is "does anything a USER can do reach it", +// which a unit test constructing an `Ltx2LoraAdapter` by hand cannot establish +// (.agents/reachability.md). +// +// So this enters through the production entry point, `LoadVideoEngine`, with a +// `lora_path` LOAD EXTRA — the same path `vllm_video_engine_load` and +// `ltx2-gen --lora` take — and asserts the RENDER moves. The reachability +// mutation is deleting the `dit_options.loras.push_back` call site in +// `ltx2_video.cpp`; that leaves the fuser and its whole unit suite green and +// REDs the cases below, which is the difference between measuring a class and +// measuring a capability. +namespace { + +// Write an IC-LoRA adapter targeting one REAL tensor of the reduced DiT +// contract, with its shape derived from the contract rather than hard-coded, so +// a fixture geometry change cannot leave this silently targeting nothing. +std::string WriteFixtureLora(const std::string& path, const std::string& target, + float scale, + const std::map& metadata = {}) { + const vllm::Ltx2DitParams params = ltx2_fixture::ReducedDitParams(); + std::vector shape; + for (const vllm::Ltx2TensorSpec& spec : vllm::EnumerateLtx2DitTensors(params)) { + if (spec.name == target) shape = spec.shape; + } + REQUIRE_MESSAGE(shape.size() == 2, + "the fixture LoRA target '", target, + "' is not a rank-2 tensor of the reduced DiT contract"); + const int64_t out_features = shape[0]; + const int64_t in_features = shape[1]; + const int64_t rank = 2; + + // B [out, rank] and A [rank, in], both constant, so the delta is a uniform + // `scale * rank` on every element — large enough that the render cannot be + // numerically indistinguishable from the unfused one. + std::vector entries = { + {"diffusion_model." + target.substr(0, target.size() - std::string(".weight").size()) + + ".lora_A.weight", + "BF16", + {rank, in_features}, + std::vector(static_cast(rank * in_features), 1.0F), + {}}, + {"diffusion_model." + target.substr(0, target.size() - std::string(".weight").size()) + + ".lora_B.weight", + "BF16", + {out_features, rank}, + std::vector(static_cast(out_features * rank), scale), + {}}, + }; + std::string metadata_json; + if (!metadata.empty()) { + metadata_json = "{"; + bool first = true; + for (const auto& kv : metadata) { + if (!first) metadata_json += ","; + first = false; + metadata_json += "\"" + kv.first + "\":\"" + kv.second + "\""; + } + metadata_json += "}"; + } + ltx2_fixture::WriteSafetensors(entries, metadata_json, path); + return path; +} + +// The target every case uses: the first block's query projection, which every +// render must read. +const char* const kFixtureLoraTarget = "transformer_blocks.0.attn1.to_q.weight"; + +} // namespace + +TEST_CASE("ltx2 video: an IC-LoRA supplied through the LOAD EXTRA reaches the PIXELS") { + // THE WITNESS IS THE RENDERED ARTIFACT, not `last_conditioning()`. The + // conditioning trace is filled BEFORE the denoise loop runs, so it is a + // function of the prompt and the conditioning items and cannot see a fused + // weight at all — a first version of this case compared `video_digest` and + // found every arm identical, which reads exactly like "the LoRA does nothing" + // and was in fact "the instrument cannot see it". `RenderBytes` takes the + // decoded output, which is downstream of the DiT weights. + Workspace ws; + + const std::string lora = + WriteFixtureLora(ws.root + "/ic.safetensors", kFixtureLoraTarget, 1.0F); + vllm::multimodal::VideoModelParams fused = ConditioningParams(ws.paths); + fused.extras[vllm::multimodal::kLtx2LoraPathExtra] = lora; + + const std::string plain = RenderBytes(ConditioningParams(ws.paths), ws.root + "/plain"); + const std::string with_lora = RenderBytes(fused, ws.root + "/fused"); + REQUIRE(plain.size() == with_lora.size()); + REQUIRE(plain.size() > 0); + + size_t differing = 0; + for (size_t i = 0; i < plain.size(); ++i) { + if (plain[i] != with_lora[i]) ++differing; + } + MESSAGE("the IC-LoRA moves " << differing << " of " << plain.size() << " artifact bytes"); + // THE REACHABILITY CLAIM. Every byte of the REQUEST is identical; the only + // difference is the `lora_path` LOAD EXTRA. Deleting the + // `dit_options.loras.push_back` call site in ltx2_video.cpp leaves the whole + // of test_ltx2_lora green and REDs this, which is the difference between + // measuring a class and measuring a capability (.agents/reachability.md). + // + // Strictly greater than zero and no count floor above it: a count-based + // tolerance would bound nothing. + CHECK(differing > 0); +} + +TEST_CASE("ltx2 video: the IC-LoRA strength reaches the PIXELS, and 0 is a no-op") { + Workspace ws; + const std::string lora = + WriteFixtureLora(ws.root + "/ic.safetensors", kFixtureLoraTarget, 1.0F); + + const auto render = [&](const char* strength, const char* out) { + vllm::multimodal::VideoModelParams mp = ConditioningParams(ws.paths); + mp.extras[vllm::multimodal::kLtx2LoraPathExtra] = lora; + if (strength != nullptr) { + mp.extras[vllm::multimodal::kLtx2LoraStrengthExtra] = strength; + } + return RenderBytes(mp, std::string(ws.root) + "/" + out); + }; + + // A no-adapter control, so "strength 0 renders the base model" is asserted + // against the base model rather than against itself. + const std::string baseline = RenderBytes(ConditioningParams(ws.paths), ws.root + "/plain"); + const std::string full = render(nullptr, "full"); + const std::string half = render("0.5", "half"); + const std::string zero = render("0.0", "zero"); + + // Strength 0 fuses a zero delta, so the weights are the base model's again. + // This is what proves the strength is READ rather than accepted and dropped: + // an implementation that ignored it would give `zero == full != baseline`. + CHECK(zero == baseline); + CHECK(full != baseline); + CHECK(half != full); + CHECK(half != baseline); +} + +TEST_CASE("ltx2 video: the IC-LoRA load extras refuse by name on misuse") { + Workspace ws; + + SUBCASE("a strength with no adapter refuses rather than doing nothing") { + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras[vllm::multimodal::kLtx2LoraStrengthExtra] = "0.5"; + try { + (void)vllm::multimodal::LoadVideoEngine(mp); + FAIL("a strength with no adapter must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("lora_strength") != std::string::npos); + CHECK(msg.find("lora_path") != std::string::npos); + } + } + + SUBCASE("a non-numeric strength refuses BY NAME") { + const std::string lora = WriteFixtureLora(ws.root + "/ic.safetensors", kFixtureLoraTarget, + 1.0F); + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras[vllm::multimodal::kLtx2LoraPathExtra] = lora; + mp.extras[vllm::multimodal::kLtx2LoraStrengthExtra] = "strong"; + try { + (void)vllm::multimodal::LoadVideoEngine(mp); + FAIL("a non-numeric strength must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("lora_strength") != std::string::npos); + CHECK(msg.find("not a finite number") != std::string::npos); + } + } + + SUBCASE("an adapter naming a module the DiT does not bind refuses BY NAME") { + // The divergence from upstream's silent skip (fuse_loras.py:135-137), + // observed through the PRODUCTION load rather than through the reader. + const std::string path = ws.root + "/bad.safetensors"; + ltx2_fixture::WriteSafetensors( + { + {"diffusion_model.transformer_blocks.0.not_a_module.lora_A.weight", + "BF16", + {2, 4}, + std::vector(8, 1.0F), + {}}, + {"diffusion_model.transformer_blocks.0.not_a_module.lora_B.weight", + "BF16", + {4, 2}, + std::vector(8, 1.0F), + {}}, + }, + std::string(), path); + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras[vllm::multimodal::kLtx2LoraPathExtra] = path; + try { + (void)vllm::multimodal::LoadVideoEngine(mp); + FAIL("an adapter targeting an unbound module must be refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + CHECK(msg.find("transformer_blocks.0.not_a_module.weight") != std::string::npos); + CHECK(msg.find("does not bind") != std::string::npos); + } + } +} + +TEST_CASE("ltx2 video: the IC-LoRA reference factors are read from the adapter's metadata") { + // The two numbers the reference refusal used to name as unreadable. Read + // through the PRODUCTION load, and reported back out through the refusal + // itself, which is the only user-visible surface that carries them today. + Workspace ws; + const std::string lora = + WriteFixtureLora(ws.root + "/ic.safetensors", kFixtureLoraTarget, 1.0F, + {{"reference_downscale_factor", "2"}, + {"reference_temporal_scale_factor", "4"}}); + vllm::multimodal::VideoModelParams mp = FixtureParams(ws.paths); + mp.extras[vllm::multimodal::kLtx2LoraPathExtra] = lora; + const std::unique_ptr engine = + vllm::multimodal::LoadVideoEngine(mp); + + vllm::multimodal::VideoGenParams gen = FixtureGen(ws.root + "/ref"); + gen.ref_video_dir = ws.root; + try { + (void)engine->Generate(gen); + FAIL("the reference-video arm is still refused"); + } catch (const std::exception& e) { + const std::string msg = e.what(); + INFO(msg); + // The factors the adapter declared, echoed back — so this asserts the READ + // happened, not merely that a refusal fired. + CHECK(msg.find("downscale=2") != std::string::npos); + CHECK(msg.find("temporal=4") != std::string::npos); + // And the refusal still names the cause that genuinely remains. + CHECK(msg.find("TOKEN-APPEND") != std::string::npos); + } +} From 64f53752d414c3ca3864853fa27719716d9fcc20 Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 15 Aug 2026 17:00:09 +0000 Subject: [PATCH 3/5] spec(LTX25-IC-LORA): record what hdr_ic_lora and dubit would need from this seam (#923) Both were read rather than ported, so the adapter path would not have to be rewritten when their rows arrive. The finding is that neither would touch it. Both fuse exactly one adapter at load - `dubit.py:364-365` refuses any other count and `hdr_ic_lora.py:271-272` hard-codes one - so the N-adapter work this row owes is not on their path. Everything else they want lands on the pipeline and the conditioning state: per-tile conditioning slices, a frozen audio modality, a text context loaded from a file instead of encoded. One place the seam WAS shaped for them: `metadata()` returns the whole `__metadata__` map rather than the two factors `ic_lora.py` happens to read, because `hdr_ic_lora` reads `hdr_transform` and `use_hdr_transform` instead and does not use the `iclora_utils` readers at all. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/specs/ltx25-ic-lora.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/.agents/specs/ltx25-ic-lora.md b/.agents/specs/ltx25-ic-lora.md index 601cd8271..16ff97265 100644 --- a/.agents/specs/ltx25-ic-lora.md +++ b/.agents/specs/ltx25-ic-lora.md @@ -289,6 +289,35 @@ sequence and trim it back. What is and is not in the way: That work is **shared with the last-frame keyframe arm**, which is blocked on the identical machinery. It is therefore its own row rather than a tail of this one. +## 6.1 What `hdr_ic_lora` and `dubit` would need from this seam + +Both were read (not ported) so the adapter path would not have to be rewritten +for them. Neither is in scope; this records what they would ask for, so a later +row can tell an extension from a redesign. + +**What this row already gives them.** Both fuse a LoRA at load, and both take +exactly one — `dubit.py:364-365` refuses any other count and +`hdr_ic_lora.py:271-272` hard-codes a single adapter at strength 1.0. So +`Ltx2LoraSpec` + `Ltx2DitLoadOptions::loras` is the right shape for both as it +stands, and neither needs the N-adapter work owed below. + +**What they would need added.** + +| need | who | why the current shape does not cover it | +|---|---|---| +| the raw `__metadata__` map, not just two typed factors | `hdr_ic_lora` | it reads `hdr_transform` and `use_hdr_transform` (`hdr_ic_lora.py:201-208`) and does NOT use `iclora_utils`' readers at all. `Ltx2LoraAdapter::metadata()` already returns the whole map, so this is **already covered** — which is why the accessor exposes the map rather than the two ints | +| per-stage adapter sets | both | `ic_lora.py:115-119` gives stage 2 `loras=()`, `hdr_ic_lora.py:293-312` gives both stages the same adapter, and `dubit` reuses ONE stage for both. Nothing here hard-codes "stage 2 has no LoRA" — the adapter is a load option on one DiT — so a two-DiT pipeline would carry two `Ltx2DitLoadOptions`. No change to this file | +| an audio conditioning item with negative RoPE positions | `dubit` | `AudioConditionByReferenceLatent` is ported (`ltx2_conditioning.h`), but `dubit.py:335-354` shifts positions by `-aud_dur - 0.04`, and the audio VAE **encoder** has no load path here at all (the reference-audio refusal says so). Unrelated to the LoRA seam | +| a frozen modality and cross-stage latent carry | `dubit` | `ModalitySpec{frozen, noise_scale, initial_latent}` (`dubit.py:321-327`). A pipeline concern, not an adapter one | +| tiled diffusion, per-tile seeds, conditioning slicing | `hdr_ic_lora` | `hdr_ic_lora.py:485-596`. Its stage 2 is a phase LIST with per-phase tiling and a `use_ic_lora` flag, and it SLICES the reference conditioning per tile, dividing the slice indices by `reference_downscale_factor` (`:567-568`). That factor is the one this row reads, so the seam feeds it correctly; the slicing itself is pipeline work | +| a text-context provider that is not the Gemma tower | `hdr_ic_lora` | it loads `video_context` / `audio_context` from a safetensors file and runs no text encoder (`hdr_ic_lora.py:274-281`) | + +**The conclusion that matters:** nothing either file needs would change the +adapter reader or the fusion hook. Their requirements land on the pipeline and +the conditioning state, not here. The one place the seam was deliberately shaped +for them is `metadata()` returning the whole map rather than the two factors +`ic_lora.py` happens to want. + ## Owed Each is owed by this row and named in the commit and pull request bodies. From 019b9559d6d3f00626c1c06716bcbd1a7cab753a Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sat, 15 Aug 2026 17:31:24 +0000 Subject: [PATCH 4/5] docs(LTX25-IC-LORA): bring the two edited FEATURES rows under their ENTRY budgets (#923) The pre-push hook refused: line 165 was 654 chars against a 600-char ENTRY budget and line 337 was 281 against 220. Both are rows this row itself widened. Shortened MY rows rather than deleting anyone else's to make room, which is the distinction AGENTS.md draws - the budget is per ENTRY and the page has none, so a shared-file budget's "remove another entry to add one" failure mode does not apply here and must not be imitated. The forensics the rows no longer carry are in `.agents/specs/ltx25-ic-lora.md`, which is where they belong. Re-proved after the trim: 366 of 368 FEATURES lines byte-identical, only 165 and 337 moved. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- docs/FEATURES.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/FEATURES.md b/docs/FEATURES.md index 3ec850f18..f96a70ff3 100644 --- a/docs/FEATURES.md +++ b/docs/FEATURES.md @@ -162,7 +162,7 @@ in `ltx2_text_encoder.cpp` is the call that would have to change. | Voxtral audio (`VoxtralForConditionalGeneration`) | Voxtral-Mini-3B-2507 | near-tie-robust 16/16 vs vLLM 0.25.0 | decode 0.97x (beats vLLM); encoder FORWARD 15.90x of vLLM's whole TTFT (pin 46.02 ms), or 2.89x with opt-in `VT_WHISPER_ENC_FA2=1` (costs 3 near-tie divergences vs 0). Not a TTFT ratio. Pending | | Whisper audio encoder | openai/whisper-small; whisper-large-v3 (Voxtral cfg) | encoder tower 77/77; large-v3 tower 203/203 | pending | | MiniMax-H3 DiT (`MiniMaxH3DiTModel`, vllm-omni lane) | MiniMax-H3 (33.1B video+audio) | portable 79/79; all three modalities COHERENT on Q4_K_M (§8.20); PRUNED ckpts run, Q8_0 seam 0.9941 (§8.21); ref2va grid was NVFP4 quant error, §8.9 REFUTED; GGUF/NVFP4/bf16 shards stream | FP4/Marlin landed; speed pending; no bf16 render yet. Render from the Q4_K_M GGUF, not the NVFP4 arm. Krea 2 text-to-image (roadmap C11) is scoped to reuse these DiT seams | -| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCs, cond, pipeline, quant loaders gated, reduced dims. Prompt AdaLN host+dev; Gemma-4->xattn FIXTURE-gated. Img chain PPM->resize->encode->place->noise. Temporal x2 ups gated, UNDRIVEN. Render OWED | `ltx-2.5`/`ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +24 GB tower. FP8/torchao/NVFP4; kf abs-pos ported; BOTH DiTs load, NO `allow_unported`. IMAGE cond SERVED `crf=0`; IC-LoRA fusion SERVED (`--lora`, all four dtype arms, adapter metadata read); DiffVAE/keyframe/ref refused, ref now on token-APPEND not on the metadata. Speed PENDING | +| LTX-2.5 DiT (`LTX2VideoTransformer3DModel`, Lightricks lane) | LTX-2.5 (21.00B video+audio) | `SPIKE`. DiT, VAEs+ENCs, cond, pipeline, quant loaders gated, reduced dims. Prompt AdaLN host+dev; Gemma-4->xattn FIXTURE-gated. Img chain PPM->resize->encode->place->noise. Temporal x2 ups gated, UNDRIVEN. Render OWED | `ltx-2.5`/`ltx2-gen`. ~29 GB NVFP4/GB10, FP8 ~44 GB, +24 GB tower. FP8/torchao/NVFP4; kf abs-pos ported; BOTH DiTs load, NO `allow_unported`. IMAGE `crf=0`+IC-LoRA `--lora` SERVED; DiffVAE/kf/ref refused. Speed PENDING | | MiniMax-Music3 (`MiniMaxMusic3ForConditionalGeneration`, diffusers lane) | MiniMax-Music3 (8.6B Qwen3 LLM + 0.646B RVQ decoder + 2.4B fp32 DiT + DAC Flow-VAE); diffusers arm, ~28.5 GB | `ACTIVE`. Loader 1413/1413; AR, acoustic and the 8.6B LM forward all gated vs real weights; `SpeechRegistry` + `vllm_speech_*` v20 + `/v1/audio/speech`; GGUF Q4_K depth decoder value-gated. No composed request observed | Not measured. The denominator will be SGLang-Omni in its production configuration (both CUDA graphs, compiled DIT and DAV, batched seeded sampling) | | LTX-2.5 tiled + streaming Conv VAE decode | LTX-2.5 video VAE | gated vs executed upstream `ltx_core` @ `fd4ded7f` (`test_ltx2_tiling` 10/10, 915 assertions); one-tile and untiled-spatial controls BIT-EXACT vs untiled on both causality arms; an untiled frames axis is REFUSED | Streams temporal chunks through upstream's AUTO layout (768/64 px, 80/24 frames); above one tile the pixel volume is never materialized. NO-OP below 768px and 81 frames; 81-120 IS tiled, differing 6.70% of range | | MTP speculator | Qwen3.6-27B, Qwen3.6-35B-A3B | token-identical to vLLM `mtp` at c1 | ~4% faster c1; +16% output tput (MoE) | @@ -334,7 +334,7 @@ CPU elementwise GEMM (f32/f16/bf16) runs AVX2 and AVX-512 tiers on x86 where the | Muse Glimmer 30B (Meta) | Text gated at **reduced depth 4/52** only; vision wired but never reference-checked | [spec](../.agents/specs/muse-glimmer.md) / [#268](https://github.com/mudler/vllm.cpp/issues/268). Full depth, multi-step decode, image/video, server path and parser scoping open. vLLM speed OPEN GAP; llama.cpp bar #333 | | LTX-2.5 AUTO duration (the duration head) | Brick ported, never constructed | `duration_head_path` is REFUSED by name rather than accepted-and-ignored ([#611](https://github.com/mudler/vllm.cpp/issues/611)); supplying a head cannot load one. Give `num_frames` or `duration` | | LTX-2.5 arms a request CAN reach | Refused by name at the call site | The spatiotemporal latent upsampler (both flags set). Supplying that checkpoint names that arm, not the temporal one. The temporal-only x2 arm is ported, not refused | -| LTX-2.5 arms nothing can request | Declared, not requestable | `int8-convrot` (ComfyUI-only), single-node multi-GPU, `BetaScheduler` (upstream selects no scheduler either). No flag or extra asks for these. `multishot` was RETIRED: no such entry point exists upstream, and `kLoraFusion` was RETIRED because it came TRUE - `lora_path` requests it | +| LTX-2.5 arms nothing can request | Declared, not requestable | `int8-convrot` (ComfyUI-only), single-node multi-GPU, `BetaScheduler`. No flag or extra asks for these. `multishot` and `kLoraFusion` were RETIRED | | Multi-GPU execution | Hardware-blocked | TP proven equal to tp=1 on CPU; no 2-GPU box to run it | | LoRA end to end | CPU brick landed | Unwired standalone; not usable through the server | | Multimodal over HTTP | Image request path wired; forward + codec pending | `ROAD-V1-MM` W1-W3 landed. Open: no mm-forward on `Request.mm_features`; no image codec. Video/audio/multi-image now **refuse** with HTTP 400 rather than drop ([#686](https://github.com/mudler/vllm.cpp/issues/686)) | From f727cfd8555f23b65ffcb8aa58f146af8429d17b Mon Sep 17 00:00:00 2001 From: Ettore Di Giacinto Date: Sun, 16 Aug 2026 02:39:11 +0000 Subject: [PATCH 5/5] fix(LTX25-IC-LORA): the reference refusal blamed a seam that had already landed, and its test could not tell (#975) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A fresh review returned FAIL on three findings, and the largest of them is that the refusal this row rewrote was FALSE before it merged. ## The refusal named a cause the tree had closed Row `LTX25-IC-LORA` moved the reference-video refusal off the IC-LoRA metadata, which it had genuinely closed, and onto the token-APPEND machinery. That was accurate on 2026-08-15. Row `LTX25-TOKEN-APPEND` landed the seam in `c7cb59fbb` on 2026-08-16, before this branch merged, so both reasons the message had ever given were false at once. `#964` left the wording byte-identical only because this pull request was open and its cause was then still true. The determination was made again from the merged tree rather than inherited. The attention-strength wrapper is not in the way either: on the DEFAULT arm upstream sets `attn_mask = None` at `conditioning_attention_strength >= 1.0` with no latent mask (`iclora_utils.py:159-160`) and applies `ConditioningItemAttentionStrengthWrapper` only `if attn_mask is not None` (`:168-169`), so #932 is not the blocker for the default case. TWO causes remain, and the refusal now names both. 1. The reference CLIP has no pixel path. Upstream reads it at `height // scale` by `width // scale` (`iclora_utils.py:116-117`), refuses a target the factor does not divide (`:112-115`), keeps frame 0 then every Nth frame (`temporal_subsample`, `:87-89`, called at `:144`) and encodes the whole clip (`:145-148`). This engine's only pixel-to-latent route encodes exactly one frame at the phase's own resolution, and nothing reads `ref_video_dir`. 2. The reference item is a STAGE-1 item and stage 2 must run UNFUSED. `ic_lora.py:108` gives stage 1 `loras=tuple(loras)` and the reference conditioning (`:269-278`, `:377-402`); `:119` gives stage 2 `loras=()` and `:314-321` gives it `combined_image_conditionings` with no reference item. This engine holds ONE DiT, fused at load, that every phase runs. Serving the arm is therefore its own row, filed as #975. Piece 2 changes how the engine holds its weights, not how it conditions, and a second resident DiT is a memory decision this row cannot take on its own. ## The test could not have caught any of that, and now can The case asserted five SUBSTRINGS. Two were upstream symbol names, present in the pinned checkout whatever this engine can do, and three were literals the message declared about itself. None could go red when the ENGINE changed. The reviewer proved it: replacing the local-cause sentence with a self-declared falsehood, keeping all five substrings, left `test_ltx2_video` at 44 cases / 914 assertions / SUCCESS. The case now MEASURES the engine first. It renders with and without an appending conditioning item, reads `video_tokens` — the one trace field written inside the phase loop — and requires the grown count to exceed the plain one and both renders to come back at the target frame count. Only then does it constrain the message, and the property it asserts is positional rather than lexical: every occurrence of a closed cause must sit after the `WHAT IS *NOT* THE REASON` marker, because recording a ruled-out cause is the message's own convention and must stay possible. ## The bf16 headline was only two thirds gated The aggregation dtype binds three roundings, and only two had cases: `B * strength` (`fuse_loras.py:113`) and `deltas.add_(weight)` (`:67-68`). The matmul RESULT's own `.to(dtype=dtype)` had none — widening only it left `test_ltx2_lora` at 13/13 and `test_ltx2_loader` at 31/31. The new case puts `acc = 1 + 2^-8` exactly on a bf16 tie and adds `w = 2^-9`, so the ported order stores 1.0 and an f32 accumulator stores 1.0078125, one bf16 step apart in the STORED result where the final rounding cannot absorb it. ## Records `.agents/issue-index.md` carried a SECOND `#930` row describing that blocker as open. #930 is closed, and main already carries the authoritative row, so the duplicate is dropped before it lands — the append-only rule protects rows that exist on `main`, and this one never did. The `#932` row's "blocked behind #930" is corrected, and #975 is appended. FOLLOWING_AGENTS_PROTOCOL Issue: https://github.com/mudler/vllm.cpp/issues/975 Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code] --- .agents/issue-index.md | 4 +- .agents/specs/ltx25-ic-lora.md | 107 ++++++++++++++-------- src/vllm/multimodal/ltx2_video.cpp | 101 +++++++++++++------- tests/vllm/models/test_ltx2_lora.cpp | 48 ++++++++++ tests/vllm/multimodal/test_ltx2_video.cpp | 107 ++++++++++++++++++---- 5 files changed, 280 insertions(+), 87 deletions(-) diff --git a/.agents/issue-index.md b/.agents/issue-index.md index fcc6601fa..559034791 100644 --- a/.agents/issue-index.md +++ b/.agents/issue-index.md @@ -241,8 +241,7 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#912](https://github.com/mudler/vllm.cpp/issues/912) | `ENG-EXPERT-STREAM` | Stream routed experts from NVMe so a model larger than device memory runs. Target `Qwen/Qwen3.8-2.4T-A95B`, REGISTERED against `Qwen3_5MoeForCausalLM` and blocked only on capacity: 370 GiB at UD-Q1_0 against 128 GB of unified memory on GB10. The only one of the three offload rows that helps on a unified-memory host, because `ENG-WEIGHT-OFFLOAD` and `ENG-HYBRID-PLACEMENT` both move bytes inside one physical pool. Cheaper than the spec assumed: on the GGUF path the mmap'd file already IS the bank and the per-expert slicer landed 2026-07-22 ([#824](https://github.com/mudler/vllm.cpp/issues/824)) | feature | | [#911](https://github.com/mudler/vllm.cpp/issues/911) | — | A `file:line` anchor into a file the row is ITSELF editing is stale by default, and spec BODIES are checked by nothing: `check-agent-record.py`'s `MATRIX_PATHS` (`:521`, `:529-530`) covers the five matrices, `feature-matrix.md` and `specs/model-family-inventory.md`, leaving 4772 line-carrying citations across 315 `.agents/specs/*.md` unexamined (positive control: 2314 line-less `.cpp` mentions match the same shape). `ltx25-prompt-adaln.md` shipped EIGHT stale repo-local anchors across two repair commits, moved by its own `020381676` and by `98f8e046d` (#658), then SEVEN more that were correct at `00613767d` and wrong at the merge of `origin/main`, because `0785cfc4d` (#882) added 70 lines to `ltx2_video.cpp` and 306 to `test_ltx2_video.cpp` ahead of every one. The obvious checker is a TAUTOLOGY — reading the span out of the file it validates reports 27 of 27 fresh on the same tree where reading the spans against their CLAIMS finds seven stale. Remedy is already in use and unwritten: `path:NN @ ` for a historical claim, claim-sourced uniqueness re-derivation for a live one, re-run after the merge. Narrower than [#632](https://github.com/mudler/vllm.cpp/issues/632) on surface and sharper on mechanism. Listed under `## Owed` in [`ltx25-prompt-adaln.md`](specs/ltx25-prompt-adaln.md) | bug | | [#923](https://github.com/mudler/vllm.cpp/issues/923) | `LTX25-IC-LORA` | Port upstream `ICLoraPipeline` (`ltx-pipelines/ic_lora.py` @ `fd4ded7f`): read an IC-LoRA adapter and its `__metadata__`, fuse the delta into the DiT at load, and expose it through the `ltx-2.5` load extras and `ltx2-gen --lora`. The conditioning half already landed — `Ltx2ConditionVideoByReference` (`ltx2_conditioning.cpp:221`) and `Ltx2ConvVideoEncode` are ported and gated, and `ref_video_dir` reaches the engine as a dir of `frame_%06d.ppm` — so the gap is the adapter path the reference refusal names at `ltx2_video.cpp:1341-1343`. The tree's `include/vllm/lora/` is NOT this mechanism: it is vLLM's runtime punica brick (f32, slot-indexed, `LinearMethodBase`), where LTX fuses at LOAD (`loader/fuse_loras.py:119-150`), so this row does not route through it. One hook after `MaterializeDitTensor` (`ltx2_loader.cpp:424-499`) serves the F32, BF16, FP8 and NVFP4 arms at once, because both quantized branches already `return vt::DType::kBF16`. Retires the `kLoraFusion` marker, which [#691](https://github.com/mudler/vllm.cpp/issues/691) predicted would go false exactly here | feature | -| [#930](https://github.com/mudler/vllm.cpp/issues/930) | — | LTX-2.5's engine phase loop is fixed at one `Ltx2VideoTokenCount(vshape, 1)` feeding the sigma schedule, the `Ltx2ModalityInput` and `Ltx2VideoUnpatchify`, so no APPENDING conditioning item can be served — and that single obstruction is what refuses BOTH the reference-image/video arms and the last-frame keyframe. Found while porting `ICLoraPipeline` ([#923](https://github.com/mudler/vllm.cpp/issues/923)), which closed the reason the reference refusal used to give and rewrote it onto this one. Much smaller than the two refusals imply: the conditioning item is ported and gated (`ltx2_conditioning.cpp:221`), the VAE encoder is reached, `Ltx2ModalityInput::tokens` is already per-call, and the DiT already accepts a self-attention strength mask — so the work is carrying the grown `state.tokens` through denoise and trimming back before unpatchify (`clear_conditioning`, `ltx_core/tools.py:88-105`). Listed under `## Owed` in [`ltx25-ic-lora.md`](specs/ltx25-ic-lora.md) | feature | -| [#932](https://github.com/mudler/vllm.cpp/issues/932) | — | Two arms of upstream `ICLoraPipeline` that [#923](https://github.com/mudler/vllm.cpp/issues/923) deliberately did not build. (1) `conditioning_attention_strength < 1.0` and `conditioning_attention_mask`: only the DEFAULT third branch of `iclora_utils.py:151-160` is served, which is why the row could ship at all — at strength 1.0 with no mask upstream computes no attention mask anywhere. The other two need `Ltx2LatentState` to carry a mask and `build_attention_mask`'s block structure (`mask_utils.py:170-243`); the DiT side already accepts one. Blocked behind [#930](https://github.com/mudler/vllm.cpp/issues/930). (2) N-adapter fusion, which needs upstream's SECOND rounding pattern — `addmm_(B, A, alpha=strength)` at `fuse_loras.py:115`, which rounds differently from the first product and which #923 REFUSES rather than guesses. Listed under `## Owed` in [`ltx25-ic-lora.md`](specs/ltx25-ic-lora.md) | feature | +| [#932](https://github.com/mudler/vllm.cpp/issues/932) | — | Two arms of upstream `ICLoraPipeline` that [#923](https://github.com/mudler/vllm.cpp/issues/923) deliberately did not build. (1) `conditioning_attention_strength < 1.0` and `conditioning_attention_mask`: only the DEFAULT third branch of `iclora_utils.py:151-160` is served, which is why the row could ship at all — at strength 1.0 with no mask upstream computes no attention mask anywhere. The other two need `Ltx2LatentState` to carry a mask and `build_attention_mask`'s block structure (`mask_utils.py:170-243`); the DiT side already accepts one. NOT blocked behind [#930](https://github.com/mudler/vllm.cpp/issues/930), which closed in `c7cb59fbb`; the remaining reference-arm blockers are [#975](https://github.com/mudler/vllm.cpp/issues/975). (2) N-adapter fusion, which needs upstream's SECOND rounding pattern — `addmm_(B, A, alpha=strength)` at `fuse_loras.py:115`, which rounds differently from the first product and which #923 REFUSES rather than guesses. Listed under `## Owed` in [`ltx25-ic-lora.md`](specs/ltx25-ic-lora.md) | feature | | [#919](https://github.com/mudler/vllm.cpp/issues/919) | `LTX25-RESOLUTION-ENVELOPE` | `vllm_video_generate` integer-divides `width`/`height` into the latent grid (`ltx2_video.cpp:1456-1463 @ 5a0ffe9e3`) with no divisibility check, so a 100x100 request on the distilled two-stage recipe silently renders 96x96. The only geometry guard in the LTX path is a LOWER bound (`ltx2_video.cpp:1464-1471 @ 5a0ffe9e3`). Every repo-local anchor in this row is SHA-pinned because the fix edits the files it cites, and inserts lines above both spans (#911). Upstream hard-validates and raises at the top of a pipeline `__call__` — `assert_resolution` (`ltx-pipelines utils/helpers.py:540-551` @ `fd4ded7f2`), 64 for two-stage and 32 for one-stage, NINE invocations including `ti2vid_two_stages.py:184` and `ti2vid_two_stages_hq.py:199` — so mirroring means refusing, not flooring. Nine, not the 21 lines a grep for the name returns (9 invocations + 1 definition + 10 imports + 1 `__all__` string), and not every pipeline: 13 pipeline `__call__`s take a resolution and the three `*_mgpu.py` variants plus `hdr_ic_lora.py:352` skip the guard. `docs/USAGE.md:626-629 @ 5a0ffe9e3` already documents the rule as though it were enforced. Frames are the OPPOSITE answer: upstream floors an explicit `num_frames` exactly as we do (`ltx_core/types.py:113`) and validates it nowhere, so that half is a doc correction | bug | | [#921](https://github.com/mudler/vllm.cpp/issues/921) | — | The res_2s DENOISING LOOP (`ltx-pipelines utils/samplers.py:206-447` @ `fd4ded7f2`) is unported, so `TI2VidTwoStagesHQPipeline` cannot be served. What exists is one substep's SDE arithmetic: `Ltx2Res2sSdeCoeff`/`Ltx2Res2sStep` (`ltx2_pipeline.cpp:307-360 @ 5a0ffe9e3`, the two functions in full) mirror `Res2sDiffusionStep` (`diffusion_steps.py:118-190`) and are gated. Absent are the `phi`/`get_res2s_coefficients` exponential integrator (`utils/res2s.py:4-62`), the SECOND transformer evaluation per step at `sub_sigma = sqrt(sigma * sigma_next)` (`samplers.py:315` and `samplers.py:380-386`, spelt out because a bare `:NN` after a res2s.py citation reads as res2s.py) against our once-per-step loop (`ltx2_video.cpp:1735 @ 5a0ffe9e3`, with its single forward at `ltx2_video.cpp:1813-1817 @ 5a0ffe9e3`), the bong anchor refinement (`samplers.py:357-364`), and any `Ltx2StepperKind` enumerator to select it. The sampler IS the HQ variant, so this arm must refuse by name rather than substitute Euler and render something plausible that is quietly not HQ. Listed under `## Owed` in [`ltx25-resolution-envelope.md`](specs/ltx25-resolution-envelope.md) | feature | | [#922](https://github.com/mudler/vllm.cpp/issues/922) | `LTX25-A2V-AUDIO-INPUT` | LTX-2.5 audio-to-video (`A2VidPipelineTwoStage`, `a2vid_two_stage.py:53,143`) is absent: `vllm_video_params` carries no field or extra that accepts a driving waveform and `ltx2-gen` has no `--audio-path`, so nothing turns a file on disk into the audio latent the DiT's audio stream consumes. Distinct from reference-audio conditioning, which is correctly refused by name at `ltx2_video.cpp:1348-1355 @ 5a0ffe9e`; the two share one blocking dependency, the audio VAE ENCODER load path (`ltx2_loader.cpp:1295-1300` materializes `audio_vae.decoder.` only). The analysis half is already ported and unreached — `Ltx2AudioEncoderForward` (`ltx2_audio_vae.cpp:1114`), `Ltx2WaveformToLogMel` (`:1019`), `Ltx2SlaneyMelFilterbank` (`:970`) — and the engine applies ONE `phase.noise_scale` to both streams (`ltx2_video.cpp:1708-1712 @ 5a0ffe9e`) where upstream's `ModalitySpec` carries `noise_scale` and `frozen` per modality (`utils/types.py:99-112`). Spec [`ltx25-a2v-audio-input.md`](specs/ltx25-a2v-audio-input.md) | feature | @@ -261,3 +260,4 @@ rather than merged. `scripts/check-agent-record.py` gates both. | [#953](https://github.com/mudler/vllm.cpp/issues/953) | `MODEL-MUSIC-minimax-music3-mini-max-music3-for-conditional-generation` | `POST /v1/audio/speech` silently DROPPED five keys that SGLang-Omni, serving this same model on this same route, refuses BY NAME: `temperature`, `top_p`, `top_k`, `repetition_penalty` (`request_builders.py:14-19,109-114` — this model's AR stage has ONE sampler, a fixed top-50 draw, `encoders.py:48,94-103`, so the knobs can be neither honoured nor honestly ignored) and `max_new_tokens` (`request_builders.py:56-68` — upstream's LENGTH spelling, counted in 25 Hz FRAMES rather than seconds, so a 250-frame request silently became the family's 60 s default). The identical class as [#925](https://github.com/mudler/vllm.cpp/issues/925), which cost four multi-hour runs. FIXED IN FLOW while sweeping [#672](https://github.com/mudler/vllm.cpp/issues/672) for upstream parity: all five refused by name, RED first in `test_speech_api.cpp`, two mutations both firing | bug | | [#965](https://github.com/mudler/vllm.cpp/issues/965) | `MODEL-MUSIC-minimax-music3-mini-max-music3-for-conditional-generation` | `windows-msvc-cpu`/`windows-msvc-vulkan` fail on EVERY open pull request with `C4456: declaration of 'loaded' hides previous local declaration` at `server_main.cpp:1315` — the speech engine's `loaded` nested inside the text engine's `loaded` at `:1025`, both already on `main`, and the ONLY warning in the job. **It is NOT [#645](https://github.com/mudler/vllm.cpp/issues/645)**, which is the `M_PI` regression in three LTX2 sources: a second cause hiding behind a known-red name, which is why "known-red" needs a MATCHED-ARM check and not a label. Confirmed pre-existing against three unrelated PRs (#956, #950, #939) that all fail identically. Invisible on `main` because `windows-msvc-*` are PR-only ([#584](https://github.com/mudler/vllm.cpp/issues/584)), so it presents to each author in turn as a red their own diff caused. FIXED IN FLOW while landing [#672](https://github.com/mudler/vllm.cpp/issues/672): the inner declaration is renamed, no detector weakened and no warning suppressed | bug | | [#968](https://github.com/mudler/vllm.cpp/issues/968) | `MODEL-DIFFUSION-LTX25` | `windows-msvc-cpu`/`windows-msvc-vulkan` red on every branch based on `c7cb59fbb`: `C4244: conversion from 'const double' to 'float'` raised inside MSVC's ``/`` from the two narrowing `positions.assign` calls [#964](https://github.com/mudler/vllm.cpp/pull/964) added at `ltx2_video.cpp:203,214`, where `StreamState::positions` and `Ltx2LatentState::positions` differ in element type. NOT fixed in flow: `#964`'s own comment at `:129-132` reasons that "double -> float -> double reproduces the bits", so the narrowing is DELIBERATE and a silencing cast is a claim about that reasoning rather than a formatting fix — it belongs to the LTX-2.5 lane. MATCHED-ARM evidence, by grepping each job log: #966 and #951 (both on `c7cb59fbb`) hit it, #967/#956/#950/#939/#938 (all pre-`c7cb59fbb`) do not. It was INVISIBLE until [#965](https://github.com/mudler/vllm.cpp/issues/965) removed the `C4456` shadow that failed the same jobs first — two independent causes stacked behind one habitually-red name, and neither was [#645](https://github.com/mudler/vllm.cpp/issues/645) | bug | +| [#975](https://github.com/mudler/vllm.cpp/issues/975) | `LTX25-IC-LORA` | The reference-image / reference-video arm of `ltx-2.5` is still refused, and BOTH reasons the refusal ever gave are now false: the IC-LoRA metadata ([#923](https://github.com/mudler/vllm.cpp/issues/923) reads it at load, `iclora_utils.py:30-49`) and the token-APPEND machinery ([#930](https://github.com/mudler/vllm.cpp/issues/930) built it in `c7cb59fbb`; the LAST-frame keyframe is SERVED on it). Two causes remain. (1) The reference CLIP has no pixel path: upstream reads it at `height // scale` by `width // scale` (`iclora_utils.py:116-117`), refuses a target the factor does not divide (`:112-115`), keeps frame 0 then every Nth frame (`temporal_subsample`, `:87-89`, called at `:144`) and encodes the clip (`:145-148`), while this engine's only pixel-to-latent route encodes ONE frame at the phase's own resolution and nothing reads `ref_video_dir` (`src/vllm/multimodal/video_engine.cpp:375`). `Ltx2ConvVideoEncode` already takes a `frame_count`, so the encoder is not the gap. (2) The reference item is a STAGE-1 item and stage 2 must run UNFUSED: `ic_lora.py:108` gives stage 1 `loras=tuple(loras)` and the reference conditioning (`:269-278`, `:377-402`), `:119` gives stage 2 `loras=()` and `:314-321` gives it `combined_image_conditionings` with no reference item — and this engine holds ONE DiT, fused at load, that every phase runs. `Ltx2LatentState` having no attention-mask field is NOT the reason either: the default arm builds no mask (`iclora_utils.py:159-160`, `:168-169`). Listed under `## Owed` in [`ltx25-ic-lora.md`](specs/ltx25-ic-lora.md) | feature | diff --git a/.agents/specs/ltx25-ic-lora.md b/.agents/specs/ltx25-ic-lora.md index 16ff97265..9793101e9 100644 --- a/.agents/specs/ltx25-ic-lora.md +++ b/.agents/specs/ltx25-ic-lora.md @@ -26,16 +26,16 @@ absent, and it is the shared prerequisite for `ic_lora.py`, `hdr_ic_lora.py` and It does **not** ship video-to-video. Three statements up front so nothing is discovered later. -1. **The reference-video arm has a SECOND blocker, and the brief for this row did - not account for it.** The refusal at `ltx2_video.cpp:1336-1346` names one - cause — the LoRA metadata is unread — and that cause is true and is closed - here. But `ltx2_video.cpp:1305-1335`, thirty lines earlier, documents the - other one: every *appending* conditioning item is blocked on token-append - machinery the engine does not have. `VideoConditionByReferenceLatent` appends - (`reference_video_cond.py:97-100`, ported at `ltx2_conditioning.cpp:265` via - `AppendTokens`), so the reference arm needs it too. §6 sizes it and `## Owed` records - it as owed. Closing only the metadata half and lifting the refusal would ship - a wrong render. +1. **The reference-video arm has TWO further blockers, and neither is the one + this spec first named.** The refusal named one cause — the LoRA metadata is + unread — and that cause was true and is closed here. This spec then rewrote it + onto the token-APPEND machinery, which was true on 2026-08-15 and false on + 2026-08-16: row `LTX25-TOKEN-APPEND` (#930) landed that seam in `c7cb59fbb` + and the LAST-frame keyframe is now served on it. §6 is rewritten against the + current tree and now names what genuinely remains — the reference clip's own + pixel path, and the stage split that gives stage 2 no adapter — recorded as + [#975](https://github.com/mudler/vllm.cpp/issues/975). Closing the metadata + half and lifting the refusal would still ship a wrong render. 2. **No render-quality claim, and no real-weights claim.** The evidence here is numeric parity against upstream's fusion arithmetic plus mutation gates on synthetic fixtures. No IC-LoRA checkpoint was fused on real weights, because @@ -234,6 +234,8 @@ test. | a LoRA naming an absent module refuses by name | §4.1 | | a LoRA matching zero tensors refuses by name | §4 | | the metadata factors are read, absent ⇒ 1 | `iclora_utils.py:30-49` | +| **the matmul RESULT is rounded to bf16 before the weight is added** — rank 2, `acc = 1 + 2^-8` on a bf16 tie, `w = 2^-9`, so a bf16 aggregator stores 1.0 and an f32 one stores 1.0078125 | upstream's `.to(dtype=aggregation_dtype)` at `fuse_loras.py:113`. This is the THIRD rounding the aggregation dtype binds; widening only it left `test_ltx2_lora` 13/13 and `test_ltx2_loader` 31/31, which is what made the case necessary | +| **the reference refusal may not name a seam this engine has** — the case measures `video_tokens` with and without an appending item, and only then forbids the message from blaming the loop | the repair for the review finding that five substring assertions could not go red when the ENGINE changed | | a second `lora_path` refuses by name | §3.3 | | **the FP8 arm fuses** — an FP8 fixture plus a LoRA changes the materialized weight | that the hook is after dequant, on the arm most users run | | **the NVFP4 arm fuses** | the same for NVFP4 | @@ -262,32 +264,59 @@ Known-red on `main` at the base SHA `95b7366`, proven pre-existing by running `check-test-registration`, `test_check_release_binary_contract`, `test_release_manifest`, `test_release_pipeline`, `test_check_test_registration`. -## 6. The second blocker, sized +## 6. What still blocks reference video, re-derived at `c90e3fc02` -Recorded here so the next agent does not re-derive it. +**This section was wrong once and is rewritten rather than patched.** Its first +version named the token-APPEND machinery, which row `LTX25-TOKEN-APPEND` (#930) +then landed in `c7cb59fbb`. The determination below is re-derived against the +merged tree, not read out of the earlier record, and it is +[#975](https://github.com/mudler/vllm.cpp/issues/975). -Serving a reference video needs the engine's phase loop to carry a **grown** token -sequence and trim it back. What is and is not in the way: +**Ruled OUT, with what ruled each one out.** - **The DiT is not in the way.** `Ltx2ModalityInput::tokens` is a per-call field, - and the DiT already accepts a self-attention strength mask - (`ltx2.h:458-462`, implemented `ltx2_dit.cpp:588-593`, mirroring - `_prepare_self_attention_mask`, `transformer_args.py:208-237`). -- **The conditioning item is not in the way.** It is ported and gated. + and the DiT already accepts a self-attention strength mask. +- **The conditioning item is not in the way.** `Ltx2ConditionVideoByReference` + (`ltx2_conditioning.cpp:221`) is ported and gated. +- **The phase loop is no longer in the way.** `c7cb59fbb` binds a `target_tokens` + local, grows `video.tokens` past it on an appending item, carries the grown + count through denoise, and trims back through `Ltx2ClearConditioning` + (`ltx_core/tools.py:88-117`) before unpatchify. The LAST-frame keyframe arm is + SERVED on exactly that machinery, which is the executable proof it exists. - **`Ltx2LatentState` having no attention-mask field is not in the way for the - DEFAULT arm.** At `conditioning_attention_strength == 1.0` with no mask, - upstream computes `attn_mask = None` (`iclora_utils.py:157-160`) and - `update_attention_mask` returns `None` (`mask_utils.py:141-143`), so the - default arm needs no mask at all. -- **The phase loop is in the way.** One `Ltx2VideoTokenCount(vshape, 1)` feeds - the sigma schedule, the `Ltx2ModalityInput` and `Ltx2VideoUnpatchify` - (`ltx2_video.cpp:1305-1322` documents this for the keyframe arm, and it is the - same obstruction). It must instead carry `state.tokens` through denoise and - trim to the target count before unpatchify, mirroring `clear_conditioning` - (`ltx_core/tools.py:88-105`). - -That work is **shared with the last-frame keyframe arm**, which is blocked on the -identical machinery. It is therefore its own row rather than a tail of this one. + DEFAULT arm.** At `conditioning_attention_strength >= 1.0` with no latent mask, + upstream sets `attn_mask = None` (`iclora_utils.py:159-160`) and applies + `ConditioningItemAttentionStrengthWrapper` only `if attn_mask is not None` + (`:168-169`). The sub-1.0 arm is #932 and is not what blocks this one. + +**What IS in the way.** + +1. **The reference clip has no pixel path.** Upstream reads the reference at + `height // scale` by `width // scale` (`iclora_utils.py:116-117`), refuses a + target either axis of which the factor does not divide (`:112-115`), keeps + frame 0 and then every Nth frame (`temporal_subsample`, `:87-89`, called at + `:144`), and encodes the whole clip (`:145-148`). This engine's only + pixel-to-latent route is `Ltx2LoadImageAndPreprocess` followed by + `Ltx2ConvVideoEncode` at `frame_count = 1` and the phase's OWN height and + width, and it refuses an encode returning more than one latent frame. Nothing + anywhere reads `ref_video_dir` (`src/vllm/multimodal/video_engine.cpp:375`), + which is a directory of `frame_%06d.ppm`. `Ltx2ConvVideoEncode` already takes + a `frame_count`, so the encoder is not the gap; the reader, the resize target + and the subsample are. +2. **The reference item is a STAGE-1 item, and stage 2 must run UNFUSED.** + `ICLoraPipeline` builds two `DiffusionStage`s from one checkpoint and gives + stage 1 `loras=tuple(loras)` (`ic_lora.py:108`) and stage 2 `loras=()` + (`:119`); stage 1 takes `_create_conditionings`, which appends the reference + item (`:269-278`, `:377-402`), and stage 2 takes plain + `combined_image_conditionings` with no reference item (`:314-321`). This + engine holds ONE `Ltx2Dit`, fused at load, that every phase of the recipe + runs. Serving the arm on the two-phase distilled recipe therefore needs a + second unfused DiT or a phase-scoped adapter, and serving it on phase 0 alone + is upstream's `skip_stage_2` (`:302-308`), a different request. + +Piece 2 changes how the engine HOLDS its DiT, not how it conditions, so this is +its own row rather than a tail of this one. It also carries a memory decision a +conditioning change does not: a second resident DiT is ~21 B parameters. ## 6.1 What `hdr_ic_lora` and `dubit` would need from this seam @@ -324,7 +353,7 @@ Each is owed by this row and named in the commit and pull request bodies. | owed | issue | |---|---| -| token-append grow-and-trim in the phase loop. Until it lands the reference-video and reference-image arms stay refused, with the refusal rewritten to name this cause instead of the metadata one this row closed. Shared with the last-frame keyframe arm, which is why it is its own row | [#930](https://github.com/mudler/vllm.cpp/issues/930) | +| serving the reference-image and reference-video arms: the reference CLIP's own pixel path (read, resize to `height // scale`, temporal subsample, multi-frame encode) and the stage split that gives stage 2 no adapter. §6 derives both. Token-append is NOT part of it any more — #930 landed in `c7cb59fbb` | [#975](https://github.com/mudler/vllm.cpp/issues/975) | | the `conditioning_attention_mask` / `conditioning_attention_strength < 1.0` arm, which needs `Ltx2LatentState` to carry a mask and `build_attention_mask`'s block structure (`mask_utils.py:170-243`) | [#932](https://github.com/mudler/vllm.cpp/issues/932) | | N-adapter fusion, which additionally needs upstream's SECOND rounding pattern (`addmm_` with `alpha`, `fuse_loras.py:115`) that this row refuses rather than guesses | [#932](https://github.com/mudler/vllm.cpp/issues/932) | | GGUF k-quant LoRA fusion — **not applicable** rather than owed: the LTX-2.5 DiT ships FP8 and NVFP4, and no GGUF LTX DiT exists to fuse into | n/a | @@ -333,9 +362,14 @@ Each is owed by this row and named in the commit and pull request bodies. ## 8. Stop conditions - Stop and report `NEEDS_DECISION` if closing the metadata half would require - lifting the reference refusal before the token-append machinery exists. **This - fired**; §0.1 and §6 are the result, and the refusal is rewritten rather than - lifted. + lifting the reference refusal before the machinery it needs exists. **This + fired twice.** The first time it named token-append; §6's first version was the + result. The second time, in review repair, that cause had itself landed + (`c7cb59fbb`) and the determination had to be made again from the tree. §6 is + rewritten, the refusal is rewritten onto the two causes that remain, and + serving the arm is reported as `NEEDS_DECISION` under + [#975](https://github.com/mudler/vllm.cpp/issues/975) because a second resident + DiT is a change to how the engine holds its weights, not a conditioning change. - Stop if `docs/FEATURES.md` cannot be reapplied by key with unrelated keys byte-identical. - Do not use the GPU. `dgx.casa` was under a long render for this row's duration. @@ -343,4 +377,5 @@ Each is owed by this row and named in the commit and pull request bodies. ## Now `ACTIVE` — the adapter path is implemented and gated; the reference arm stays -refused on the cause named in §6. +refused on the two causes §6 derives, which are #975 and are neither of the two +this refusal has previously given. diff --git a/src/vllm/multimodal/ltx2_video.cpp b/src/vllm/multimodal/ltx2_video.cpp index 67f40becc..da1566671 100644 --- a/src/vllm/multimodal/ltx2_video.cpp +++ b/src/vllm/multimodal/ltx2_video.cpp @@ -1417,15 +1417,28 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { // silently ignored renders an unconditioned clip that looks like the feature // not working. // - // THESE MESSAGES ARE WRITTEN TO BE RE-CHECKABLE, and the count is now SIX + // THESE MESSAGES ARE WRITTEN TO BE RE-CHECKABLE, and the count is now SEVEN // refusals in this campaign whose stated reason turned out to be false or - // stale. Two of the six stood right here. The first said no encoder weights + // stale. Three of the seven stood right here. The first said no encoder weights // could be materialized — true when written, and what this row fixed. The // second replaced it and blamed `keyframes_abs_pos_embedding`, which was // verifiably NOT the blocker at the pin (see the last-frame message below for // the three anchors that refute it), and a test had been written to assert // that wrong reason by name. // + // THE SEVENTH IS THE REFERENCE REFUSAL BELOW, and it is worth reading closely + // because it went stale in under a day and for the same reason as the second: + // a test asserted the reason by NAME, so nothing could notice. Row + // LTX25-IC-LORA (#923) rewrote it onto the token-append machinery on + // 2026-08-15 — accurate that day — and row LTX25-TOKEN-APPEND (#930) landed + // that machinery in `c7cb59fbb` on 2026-08-16, before #923 merged. Both + // reasons the message had ever given were then false at once, and its five + // assertions were four upstream symbol names plus a literal the message + // declared about itself, none of which can go red when the ENGINE changes + // underneath. The repair is in the test, not only here: the case now measures + // that the loop grows and trims and then forbids the message from stating it + // as a cause rather than as a ruled-out one. + // // So: name the exact symbol or upstream `file:line` that would have to change // for the refusal to become false, never a category — and where a plausible // reason has already been ruled OUT, say so and cite what ruled it out, so the @@ -1450,24 +1463,32 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { const bool wants_last_frame = !gen.last_frame_path.empty(); const bool wants_image = wants_first_frame || wants_last_frame; if (!gen.ref_image_paths.empty() || !gen.ref_video_dir.empty()) { - // WHAT CHANGED, AND WHAT DID NOT. This refusal used to blame the IC-LoRA - // metadata: the `downscale_factor` and `temporal_scale_factor` that - // `Ltx2ConditionVideoByReference` needs live in the adapter's own - // `__metadata__` (iclora_utils.py:30-49) and nothing here read it. Row - // LTX25-IC-LORA (#923) closed that — supply `lora_path` and the factors are - // read, resolved and carried on the checkpoint — so blaming it now would be - // a refusal asserting something the tree no longer has. + // TWO CAUSES REMAIN, AND NEITHER IS ONE THIS MESSAGE HAS EVER GIVEN. The + // message names both, and then names the three ruled-out reasons with what + // ruled each one out, because a reader who arrives here in a month should + // re-check the claim rather than re-derive the refutation for a third time. + // + // 1. THE REFERENCE CLIP HAS NO PIXEL PATH. Upstream resolves the reference + // at `height // scale` by `width // scale` (iclora_utils.py:116-117), + // refuses a target either axis of which the factor does not divide + // (:112-115), keeps frame 0 and then every Nth frame (`temporal_subsample`, + // :87-89, called at :144), and encodes the whole clip (:145-148). This + // engine's only pixel-to-latent route is `Ltx2LoadImageAndPreprocess` + // followed by `Ltx2ConvVideoEncode` at `frame_count = 1` and the phase's + // OWN height and width, and it refuses an encode that returns more than + // one latent frame. Nothing anywhere reads `ref_video_dir`, which is a + // directory of `frame_%06d.ppm`. // - // The arm is still refused, because there was a SECOND cause the old message - // did not name, and it is the one the LAST-FRAME refusal above already - // spells out in full: `VideoConditionByReferenceLatent` APPENDS tokens - // (reference_video_cond.py:97-100, ported at ltx2_conditioning.cpp via - // AppendTokens), and this phase loop is fixed at the target grid's token - // count — one `Ltx2VideoTokenCount(vshape, 1)` feeds the sigma schedule, the - // `Ltx2ModalityInput` handed to the DiT, and `Ltx2VideoUnpatchify`. Serving - // it means growing that sequence through the DiT and trimming it back - // (`clear_conditioning`, ltx_core/tools.py:88-105). That machinery is shared - // with the last-frame keyframe arm and is owed by its own row. + // 2. THE REFERENCE ITEM BELONGS TO STAGE 1, AND STAGE 2 MUST RUN UNFUSED. + // `ICLoraPipeline` builds two `DiffusionStage`s from the same checkpoint + // and gives stage 1 `loras=tuple(loras)` (ic_lora.py:108) and stage 2 + // `loras=()` (:119); stage 1 takes `_create_conditionings`, which appends + // the reference item (:269-278, :377-402), and stage 2 takes plain + // `combined_image_conditionings` with no reference item at all + // (:314-321). This engine holds ONE `Ltx2Dit`, fused at load, that every + // phase of the recipe runs. Serving the arm on a two-phase recipe needs a + // second unfused DiT or a phase-scoped adapter, and serving it on one + // phase only is upstream's `skip_stage_2` (:302-308), a different request. std::string factors = "no adapter was supplied, so none were read"; if (im.dit.lora_fused_tensors > 0) { factors = "the supplied adapter declares downscale=" + @@ -1476,19 +1497,35 @@ VideoResult Ltx2VideoEngine::Generate(const VideoGenParams& gen) { ", fused into " + std::to_string(im.dit.lora_fused_tensors) + " tensors"; } Fail( - "reference-image / reference-video conditioning is not served. What is missing is " - "the TOKEN-APPEND machinery, and NOT the IC-LoRA metadata, which this engine now " - "reads (" + factors + - "). `Ltx2ConvVideoEncode` and `Ltx2ConditionVideoByReference` are both ported and " - "gated, and the adapter's `downscale_factor` / `temporal_scale_factor` are resolved " - "at load (iclora_utils.py:30-49). The gap is that " - "`VideoConditionByReferenceLatent.apply_to` APPENDS tokens to the sequence " - "(conditioning/types/reference_video_cond.py:97-100) and this phase loop is fixed at " - "the target grid's token count, so the appended tokens have nowhere to go and " - "`clear_conditioning` (ltx_core/tools.py:88-105) has nothing to trim. That is the " - "SAME obstruction the last-frame keyframe arm names above, and it is owed by its own " - "row rather than by LTX25-IC-LORA, which closed the metadata half. Use " - "first_frame_ppm / first_frame_path for image-to-video."); + "reference-image / reference-video conditioning is not served. TWO things are " + "missing. FIRST, the reference CLIP has no pixel path: upstream reads it at " + "`height // reference_downscale_factor` by `width // reference_downscale_factor` " + "(iclora_utils.py:116-117), refuses a target the factor does not divide (:112-115), " + "keeps frame 0 and then every Nth frame (`temporal_subsample`, :87-89, called at " + ":144) and encodes the whole clip (:145-148), while this engine's only " + "pixel-to-latent route encodes exactly ONE frame at the phase's own resolution and " + "nothing reads `ref_video_dir` at all. SECOND, the reference item is a STAGE-1 item " + "and stage 2 must run with NO adapter: `ICLoraPipeline` gives stage 1 " + "`loras=tuple(loras)` (ic_lora.py:108) and the reference conditioning (:269-278), " + "and gives stage 2 `loras=()` (:119) and `combined_image_conditionings` with no " + "reference item (:314-321) — and this engine holds one DiT, fused at load, that " + "every phase runs. WHAT IS *NOT* THE REASON, because this refusal has now given two " + "reasons that later became false: (a) the IC-LoRA METADATA. Row LTX25-IC-LORA (#923) " + "closed that; supply `lora_path` and the factors are read at load " + "(iclora_utils.py:30-49) — right now, " + factors + + ". (b) the TOKEN-APPEND machinery. This message blamed it on 2026-08-15 and row " + "LTX25-TOKEN-APPEND (#930) landed it in `c7cb59fbb` the next day: the phase loop " + "now binds a `target_tokens` local, grows `video.tokens` past it on an appending " + "item, carries the grown count through denoise, and trims back through " + "`Ltx2ClearConditioning` (ltx_core/tools.py:88-117) before unpatchify. The " + "last-frame keyframe arm is SERVED on exactly that machinery, which is the " + "executable proof it exists. (c) `Ltx2LatentState` carrying no attention-mask " + "field. On the DEFAULT arm upstream builds no mask: at " + "`conditioning_attention_strength >= 1.0` with no latent mask `attn_mask` is None " + "(iclora_utils.py:159-160) and `ConditioningItemAttentionStrengthWrapper` is " + "applied only `if attn_mask is not None` (:168-169). The sub-1.0 arm is owed by " + "#932, and it is not what blocks this one. Use first_frame_ppm / first_frame_path " + "for image-to-video, and last_frame_path for a closing keyframe."); } if (!gen.ref_audio_path.empty() || !gen.ref_audio_wav.empty()) { Fail( diff --git a/tests/vllm/models/test_ltx2_lora.cpp b/tests/vllm/models/test_ltx2_lora.cpp index f54dc64f8..05fdbacca 100644 --- a/tests/vllm/models/test_ltx2_lora.cpp +++ b/tests/vllm/models/test_ltx2_lora.cpp @@ -579,6 +579,54 @@ TEST_CASE("ltx2 lora: an F32 adapter is NARROWED to bf16, not kept f32") { std::remove(path.c_str()); } +TEST_CASE("ltx2 lora: the matmul RESULT is rounded to bf16 before the weight is added") { + // THE THIRD ROUNDING, and the one the two cases above cannot see. Upstream's + // aggregation dtype binds three separate places, not one: + // + // 1. `B * strength` -> bf16 (fuse_loras.py:113; gated above by the + // strength case and the F32-adapter case) + // 2. the MATMUL RESULT -> bf16 (fuse_loras.py:113's `.to(dtype=dtype)`, + // which is `aggregation_dtype`) <- THIS CASE + // 3. `deltas.add_(weight)` -> bf16 (fuse_loras.py:67-68; gated below) + // + // MEASURED: widening only (2) - keeping the f32 accumulator and adding the + // weight to it before the single store - left `test_ltx2_lora` at 13/13 and + // `test_ltx2_loader` at 31/31. Both of the other roundings survived that + // mutation, which is why neither of their cases moved. + // + // THE CONSTRUCTION. Rank 2, with the two products chosen so the SUM sits + // exactly on a bf16 tie and the weight then pushes the two arms to different + // sides of the next one: + // + // acc = 1.0 * 1.0 + 2^-8 * 1.0 = 1.00390625 (exact in f32) + // w = 2^-9 = 0.001953125 (exact in bf16) + // + // ported: bf16(acc) = 1.0 by ties-to-even (2^-8 is half of the 2^-7 step at + // 1.0, and 1.0's mantissa is the even one), then + // bf16(1.0 + 2^-9) = 1.0 + // f32 acc: bf16(1.00390625 + 0.001953125) = bf16(1.005859375) = 1.0078125 + // + // One bf16 step apart in the STORED result, so the final store cannot absorb + // it - which is exactly how a first attempt at this case would fail. + const float kHalfStep = 1.0F / 256.0F; // 2^-8 + const float kQuarterStep = 1.0F / 512.0F; // 2^-9 + const std::string path = WriteAdapter(/*out_features=*/1, /*rank=*/2, /*in_features=*/1, + /*b=*/{1.0F, kHalfStep}, /*a=*/{1.0F, 1.0F}); + vllm::Ltx2LoraSpec spec; + spec.path = path; + spec.strength = 1.0; + std::vector adapters; + adapters.push_back(vllm::Ltx2LoraAdapter::Open(spec, ContractWith(kTarget))); + + const std::vector got = FuseBf16(adapters, kTarget, 1, 1, {kQuarterStep}); + CHECK(vt::F32ToBF16(got[0]) == vt::F32ToBF16(1.0F)); + CHECK(vt::F32ToBF16(got[0]) != vt::F32ToBF16(1.0F + 4.0F * kQuarterStep)); + // Stated as the value an f32 accumulator would produce, so a reader can see + // which number this case is separating 1.0 from. + CHECK(vt::BF16ToF32(vt::F32ToBF16(1.0F + kHalfStep + kQuarterStep)) == doctest::Approx(1.0078125)); + std::remove(path.c_str()); +} + TEST_CASE("ltx2 lora: the f32 target branch rounds through the bf16 accumulator") { // The scale_shift tables are the only F32 tensors in the contract. Upstream's // `_bf16_fuse` does `deltas.add_(weight)` IN PLACE on the bf16 aggregator and diff --git a/tests/vllm/multimodal/test_ltx2_video.cpp b/tests/vllm/multimodal/test_ltx2_video.cpp index 97250786b..d09e7c102 100644 --- a/tests/vllm/multimodal/test_ltx2_video.cpp +++ b/tests/vllm/multimodal/test_ltx2_video.cpp @@ -1436,27 +1436,95 @@ TEST_CASE("ltx2 video: keyframe and reference conditioning is refused BY WHAT IS const vllm::multimodal::VideoResult result = engine->Generate(gen); CHECK(result.frame_count == 9); } - SUBCASE("a reference video names TOKEN-APPEND, and no longer blames the LoRA metadata") { - // REPLACED, not relaxed (row LTX25-IC-LORA, #923). This subcase used to - // assert the refusal blamed the unread IC-LoRA metadata. That reason was - // true and is now false: `lora_path` reads the adapter's - // `reference_downscale_factor` / `reference_temporal_scale_factor` - // (iclora_utils.py:30-49) and carries them on the checkpoint. The arm is - // still refused, on the cause the message did not previously name. + SUBCASE("a reference video may not blame a seam THIS ENGINE demonstrably has") { + // WHAT THIS CASE USED TO DO, AND WHY THAT WAS THE DEFECT. It asserted five + // SUBSTRINGS of the refusal: `reference_video_cond.py`, `clear_conditioning`, + // `TOKEN-APPEND`, `NOT the IC-LoRA metadata`, and the absence of one retired + // phrase. Two of those five are UPSTREAM symbol names, which are present in + // the pinned checkout whatever this engine can do, and the other three are + // literals the message declares about itself. So not one of them could go + // red when the ENGINE changed — and the engine did change, twice, in two + // days: #923 made the metadata readable and #930 (`c7cb59fbb`) built the + // token-append seam. A reviewer replaced the local-cause sentence with a + // self-declared falsehood, kept all five substrings, and the whole suite + // stayed green. + // + // SO THIS CASE MEASURES THE ENGINE FIRST and only then constrains the + // message. The measurement is the same instrument the token-append row + // gates itself with: `video_tokens` is written INSIDE the phase loop, so it + // can observe what the loop does, unlike every field filled before denoise. + const vllm::multimodal::VideoModelParams cond_params = ConditioningParams(ws.paths); + const std::string kf = ws.root + "/append_witness.ppm"; + WriteBytes(kf, ConditioningPpm(20, 28, 31)); + + auto tokens_of = [&](const std::string& tag, const std::string& keyframe) { + const std::unique_ptr own = + vllm::multimodal::LoadVideoEngine(cond_params); + auto* ltx2 = dynamic_cast(own.get()); + REQUIRE(ltx2 != nullptr); + vllm::multimodal::VideoGenParams g = FixtureGen(ws.root + "/" + tag); + if (!keyframe.empty()) { + g.last_frame_path = keyframe; + g.extras[vllm::multimodal::kLtx2ImageCrfExtra] = "0"; + } + const vllm::multimodal::VideoResult result = own->Generate(g); + // THE TRIM, observed from outside: the volume handed to unpatchify is the + // target grid, so the clip comes back at the requested length whether or + // not anything was appended. + CHECK(result.frame_count == 9); + return ltx2->last_conditioning().video_tokens; + }; + + const int64_t plain = tokens_of("ref_witness_plain", ""); + const int64_t grown = tokens_of("ref_witness_grown", kf); + // THE GROWTH. Both numbers are measured; pinning either to a literal would + // pass on a build that never grew anything. + REQUIRE_MESSAGE(grown > plain, + "this engine's phase loop did not grow its token sequence for an appending " + "conditioning item (" + << grown << " against a target of " << plain + << "), so the rest of this case cannot say what the refusal may claim"); + const std::string msg = refusal("a reference video", [](vllm::multimodal::VideoGenParams& g, const Workspace& w) { g.ref_video_dir = w.root; }); INFO(msg); - // The cause that actually remains, and the upstream symbol that has it. - CHECK(msg.find("TOKEN-APPEND") != std::string::npos); - CHECK(msg.find("reference_video_cond.py") != std::string::npos); - CHECK(msg.find("clear_conditioning") != std::string::npos); - // And the retired reason must NOT come back as the thing that is missing. - // A refusal that names a blocker the tree no longer has is the defect row - // LTX25-RETIRE-DEAD-ARMS exists to retire, arriving from the other side. + + // BECAUSE THE TWO MEASUREMENTS ABOVE HOLD, the refusal may not CLAIM the + // phase loop. It may still MENTION it — the message's own convention is to + // record a ruled-out cause under `WHAT IS *NOT* THE REASON` so the next + // reader re-checks rather than re-derives — so the property asserted here is + // positional: every occurrence of a closed cause sits after that marker. + // + // That is what makes this case red for the mutation that motivated it. + // Restoring the pre-repair message leaves no marker at all AND puts + // `TOKEN-APPEND` in the first sentence, so both halves fire. + const size_t ruled_out = msg.find("WHAT IS *NOT* THE REASON"); + REQUIRE_MESSAGE(ruled_out != std::string::npos, + "the refusal carries no `WHAT IS *NOT* THE REASON` section, so a cause this " + "engine has already closed cannot be told apart from one it still has"); + for (const char* closed : {"TOKEN-APPEND", "fixed at the target grid's token count", + "nowhere to go", "nothing to trim"}) { + const size_t at = msg.find(closed); + const bool only_as_ruled_out = (at == std::string::npos) || (at > ruled_out); + CHECK_MESSAGE(only_as_ruled_out, + "the refusal states '" + << std::string(closed) + << "' as a cause rather than as a ruled-out one, and this case has just " + "MEASURED that the loop grows (" + << plain << " -> " << grown << ") and trims back to the target grid"); + } + // The metadata half, same shape: the factors printed are READ from the + // adapter at load, so this asserts the read happened rather than asserting + // a sentence about it. `factors` says "no adapter was supplied" here. + CHECK(msg.find("no adapter was supplied") != std::string::npos); CHECK(msg.find("which this project does not read") == std::string::npos); - CHECK(msg.find("NOT the IC-LoRA metadata") != std::string::npos); + // And the two causes that DO remain are named, by the upstream anchors a + // reader can go and check. + CHECK(msg.find("iclora_utils.py:116-117") != std::string::npos); + CHECK(msg.find("ic_lora.py:108") != std::string::npos); + CHECK(msg.find("ref_video_dir") != std::string::npos); } SUBCASE("reference audio names the AUDIO encoder, which this row did not build") { const std::string msg = refusal("reference audio", @@ -3372,8 +3440,13 @@ TEST_CASE("ltx2 video: the IC-LoRA reference factors are read from the adapter's // happened, not merely that a refusal fired. CHECK(msg.find("downscale=2") != std::string::npos); CHECK(msg.find("temporal=4") != std::string::npos); - // And the refusal still names the cause that genuinely remains. - CHECK(msg.find("TOKEN-APPEND") != std::string::npos); + CHECK(msg.find("fused into") != std::string::npos); + // And it does NOT reintroduce the reason this row closed. The cause that + // genuinely remains is gated by "a reference video may not blame a seam THIS + // ENGINE demonstrably has", which measures the engine before it reads the + // message; asserting the remaining cause by NAME here as well would be the + // second copy of the mistake that case exists to correct. + CHECK(msg.find("which this project does not read") == std::string::npos); } }