mirror of
https://github.com/huggingface/diffusers.git
synced 2026-10-24 20:59:27 +03:00
remove-lora-utils-tests
679 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Kieran Didi
|
5378870419 |
Add RFD4-Proteina (#15088)
* Add RFD4-Proteina Adds the RFD4-Proteina pipeline with its transformer, autoencoder and per-modality schedulers, a LoRA loader mixin, a checkpoint conversion script, tests and documentation. * Address review: pass tensors instead of dicts, name the pipeline example inputs prepare_latents returns a tuple and the sampling loop works on named tensors. The pipeline output has bb_ca, local_latents and clean_* fields instead of dicts, and the callback tensors are named. The pipeline properties move below __init__, the transformer return annotation names its tuple, LoRA moves from the loaders page to the pipeline docs, and the pipeline tests spell out their helpers. * Remove the constructor options that are constant for the released checkpoint The transformer always recycles the trunk, updates the pair representation with triangle multiplication (without bias) and aligns the chirality feature, as the released checkpoint does. The converter now checks that a checkpoint has exactly these settings instead of carrying them as options. * Spell out the scheduler test arguments and the docs examples, default the autoencoder to its released features The scheduler test helper takes its options by name, and the usage and batching examples pass the five required inputs by name. The autoencoder's default encoder features now match the released checkpoint, and the decoder moves its allowed-residue table to the token types' device. * Follow the repository's test and docs checks Restore the LoRA mixin on the loaders page, which utils/check_support_list.py requires for every LoRA loader mixin, and keep the mention in the pipeline docs. Compose the attention and torch.compile testers on the models, drop the model-level LoRA tests that the first pull request should not carry, and make the pipeline's backbone-noising step public as a lifecycle method. * Keep the pipeline test helpers at the top of the test class * Drop the training-only detach calls and make the noise helpers static methods * Remove num_inference_steps and skip the unused encode for inverse folding The number of steps is fixed by the released recipe and is the length of recycle_depths. Inverse folding starts both modalities at time zero, so its latents are noise and the encoded input was only consuming random draws. --------- Co-authored-by: YiYi Xu <yixu310@gmail.com> |
||
muhammadali-afridi
|
de988b7ee7 |
Add Bria fibo-2 (#14991)
* Add Bria fibo-2: BriaFibo2Pipeline and BriaFibo2Transformer2DModel
fibo-2 is Bria's 8.1B flow-matching DiT, built on the Z-Image block and
conditioned on Qwen3-VL hidden states two ways: a Perceiver resampler turns
them into gist tokens that share the stream with the image tokens, and five
blocks add gated cross-attention to their own bundle of Qwen3-VL layers. It
uses the Flux 2 VAE.
One pipeline class covers both of the model's jobs and both checkpoints:
text-to-image generation, instruction editing of up to five images with an
optional mask, the turbo checkpoint (4 steps, no guidance) and the guided one.
- BriaFibo2Transformer2DModel, with images to edit passed as context latents
- BriaFibo2Pipeline, including:
- captions cleaned and serialized the way training did, for generation and
edits, in every caption form fibo-2 was trained with: a full caption, a
short description, or a short description with its text_render list
- images of the caption's text, which Qwen3-VL sees beside the prompt:
drawn the way training drew them, by default (`render_text_images`), or
passed as `text_images`
- masked edits, the region greyed out through a binary mask as in training
- a conversion script from the original checkpoint layout
- model and pipeline tests, and docs
Verified against Bria's reference implementation on an A100 and an H100.
* Apply RoPE with explicit casting instead of an autocast block
The rotation already runs in float32 and casts back to the input's dtype, and
none of its ops are autocast ops, so the autocast block changed nothing.
* Mark the full-graph compile tests as expected failures instead of skipping them
Both fail with torch._dynamo.exc.Unsupported, on the torch.device context in
which RopeEmbedder, copied from Z-Image, builds its frequency table. Strict, so
they show up once Dynamo can trace it.
* Test the pipeline through its public call only
Removes the tests of private prompt and text-image helpers. The caption
cleaning test now checks that a caption renders like its trained form, and
the text images test draws its strips with the public draw_text_image.
* Annotate the transformer's tuple return as tuple[torch.Tensor]
* Make _get_qwen_prompt_embeds a staticmethod that takes the text encoder and processor
The chat template, the sequence length and the Qwen3-VL layers become
arguments, defaulting to module constants; encode_prompt passes the
pipeline's own. The edit and text prompt helpers become classmethods.
Embeddings are unchanged.
* Point to the release checkpoints, briaai/Fibo-2.0-dev and briaai/Fibo-2.0-turbo
The docs and the example load Fibo-2.0-dev and show the turbo checkpoint's
4-step edits without guidance. Text-image fonts download from Fibo-2.0-dev,
and the conversion script's usage converts a local original checkpoint.
* Parse prompts in one method, take fonts as an argument, and size and mask images in BriaFibo2ImageProcessor
- `_parse_prompt` replaces `_clean_caption`, `_normalize_caption`, `_split_edit_prompt`, `_get_generation_prompt` and
`_get_caption`. `_text_render_entries`, `_get_text_prompt` and `_draw_text_images` are inlined.
- The pipeline no longer fetches fonts: `__call__` takes `fonts`, and the docs show how to get the ones training
used from briaai/Fibo-2.0-dev. A text that needs them warns without them.
- `BriaFibo2ImageProcessor` holds the trained sizes, the cover-and-crop and the masking. The masking stays before the
crop, at the image's own size, where training greyed the images.
The prompts and images Qwen3-VL sees, the VAE's inputs and the outputs are byte-identical to before.
* Prepare the images to edit and the mask in BriaFibo2ImageProcessor.preprocess, as Flux3's image processor does
* Point to the turbo checkpoint's new name, briaai/Fibo-2.0-edit-turbo
* Show a working example for each use in the docs, and call the model Fibo 2.0, as its model cards do
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Linoy Tsaban <57615435+linoytsaban@users.noreply.github.com>
|
||
Alexander Ivanov
|
aea7b70be0 |
[feat] add KVAE-Audio autoencoder (#14384)
* [feat] add KVAE-Audio autoencoder * [fix] non-contiguous tensor into in_proj in AutoencoderKLKVAEAudio * [chore] sync KVAE-Audio with upstream test and annotation changes * [refactor] dedicated attention class and processor for AutoencoderKLKVAEAudio * [refactor] remove the runtime sample_rate argument from AutoencoderKLKVAEAudio * [refactor] always apply attention in AutoencoderKLKVAEAudio, drop use_attn * [refactor] drop the derived latent_dim default from AutoencoderKLKVAEAudio * [docs] fix encoder_rates docstring in AutoencoderKLKVAEAudio * [refactor] rename Snake1d to KVAEAudioSnake1d in AutoencoderKLKVAEAudio * [tests] skip layerwise casting training for AutoencoderKLKVAEAudio |
||
YiYi Xu
|
4df4a5b357 |
FLUX.3: match the released BFL reference image sizing (#15073)
* FLUX.3: match the released BFL reference image sizing A single reference is kept whole: downscaled so its padded canvas fits under 4 MP, reflect-padded to the token grid (`resize_mode="fill"` of the FLUX.3 image processor), and cropped back on output. Several references are fitted under 1 MP each. The default output size follows the first reference. --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
YiYi Xu
|
b800f828f3 |
[docs] FLUX.3: modular-only notes, lazy loading, callbacks (#15069)
* [docs] FLUX.3: modular-only notes, lazy loading, callbacks Add a tips callout on what differs from DiffusionPipeline (lazy loading, outputs, guider, offloading, editing blocks), move the prompting advice into the upsampling section with the pop-the-step example, and add a per-step callbacks section with a block that takes callback_on_step_end. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * [docs] FLUX.3: callback block walks all denoising loops Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * [docs] modular: add "Coming from DiffusionPipeline" to the quickstart and move per-step callbacks to the callbacks guide Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * [docs] FLUX.3: show the upsampled caption; clarify the index note in the quickstart Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * [docs] modular: make "Coming from DiffusionPipeline" its own page Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
YiYi Xu
|
d42ebf9c6e |
FLUX.3 prompt upsampler block (#15058)
* adding FLUX.3 prompt upsampler block |
||
Stephen Batifol
|
af4271c3e1 |
[docs] FLUX.3: prompt upsampler required, ours or any LLM (#15059)
* docs(flux3): the prompt upsampler is required, ours or any LLM * docs(flux3): every example goes through upsample() * docs(flux3): strongly advise the prompt upsampler * docs(flux3): the model is FLUX 3 Image Klein |
||
Sayak Paul
|
7ecb1e1b4f |
Flux3 image (#15046)
* Bump the actions group with 17 updates (#12) Bumps the actions group with 17 updates: | Package | From | To | | --- | --- | --- | | [actions/checkout](https://github.com/actions/checkout) | `6` | `7` | | [actions/upload-artifact](https://github.com/actions/upload-artifact) | `4.6.2` | `7.0.1` | | [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) | `3.12.0` | `4.2.0` | | [docker/login-action](https://github.com/docker/login-action) | `3.7.0` | `4.6.0` | | [docker/build-push-action](https://github.com/docker/build-push-action) | `6.19.2` | `7.3.0` | | [huggingface/doc-builder/.github/workflows/build_main_documentation.yml](https://github.com/huggingface/doc-builder) | `e60a538eea9817ab312196d0d233604b01697265` | `af49b8dc3859c3d80c7220c8c0ba146431132e49` | | [actions/setup-python](https://github.com/actions/setup-python) | `5.6.0` | `7.0.0` | | [huggingface/doc-builder/.github/workflows/build_pr_documentation.yml](https://github.com/huggingface/doc-builder) | `e60a538eea9817ab312196d0d233604b01697265` | `af49b8dc3859c3d80c7220c8c0ba146431132e49` | | [anthropics/claude-code-action](https://github.com/anthropics/claude-code-action) | `1.0.155` | `1.0.193` | | [huggingface/security-workflows/.github/workflows/codeql-reusable.yml](https://github.com/huggingface/security-workflows) | `1.1.0` | `1.2.0` | | [actions/download-artifact](https://github.com/actions/download-artifact) | `4.3.0` | `8.0.1` | | [actions/labeler](https://github.com/actions/labeler) | `5.0.0` | `7.0.0` | | [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) | `1.13.0` | `1.14.2` | | [huggingface/tailscale-action](https://github.com/huggingface/tailscale-action) | `7d53c9737e53934c30290b5524d1c9b4a7c98c8a` | `07d36e4c3b68707bd3fe070dbb96f03f993a6efd` | | [trufflesecurity/trufflehog](https://github.com/trufflesecurity/trufflehog) | `3.94.2` | `3.97.0` | | [crate-ci/typos](https://github.com/crate-ci/typos) | `1.42.1` | `1.49.0` | | [huggingface/doc-builder/.github/workflows/upload_pr_documentation.yml](https://github.com/huggingface/doc-builder) | `9ad2de8582b56c017cb530c1165116d40433f1c6` | `af49b8dc3859c3d80c7220c8c0ba146431132e49` | Updates `actions/checkout` from 6 to 7 - [Release notes](https://github.com/actions/checkout/releases) - [Commits](https://github.com/actions/checkout/compare/v6...v7) Updates `actions/upload-artifact` from 4.6.2 to 7.0.1 - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/v4.6.2...v7.0.1) Updates `docker/setup-buildx-action` from 3.12.0 to 4.2.0 - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits]( |
||
Akshan Krithick
|
62cde2b011 |
add modular pipeline for qwen-image 2.1 (#14976)
* add modular pipeline for qwen-image 2.1 * split out a turbo blockset and drop the guider from the text encoders * Update tests/modular_pipelines/qwenimage21/test_modular_pipeline_qwenimage21_turbo.py * Update tests/modular_pipelines/qwenimage21/test_modular_pipeline_qwenimage21.py --------- Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com> |
||
Junsong Chen
|
1628cb1c33 |
Add SANA-WM camera-controlled image-to-video pipeline (#13881)
* feat(sana-wm): add diffusers-style SANA-WM camera-controlled I2V pipeline
Adds the public SANA-WM bidirectional camera-controlled image-to-video
model as a first-class diffusers pipeline + transformer. Layout mirrors
``sana_video``: the model lives under ``src/diffusers/models/transformers/``
as a near-single-file (kernels split off so the ``@triton.jit`` decorators
don't drown the model body); the pipeline lives under
``src/diffusers/pipelines/sana_wm/``.
Files added:
src/diffusers/models/transformers/
├── transformer_sana_wm.py # SanaWMTransformer3DModel + blocks + helpers
└── transformer_sana_wm_kernels.py # fused Triton kernels + camera math
src/diffusers/pipelines/sana_wm/
├── __init__.py
├── pipeline_sana_wm.py
├── pipeline_output.py
├── refiner.py
└── cam_utils.py
Pipeline architecture:
* Stage 1: 1600M ``SanaWMTransformer3DModel`` DiT with bidirectional
GDN-Triton linear attention + UCPE camera-control branch, LTX-style
flow-matching Euler scheduler with per-token timesteps.
* Stage 2: LTX-2 sink-bidirectional Euler refiner (3 distilled sigma
steps, reuses diffusers' ``LTX2VideoTransformer3DModel`` +
``LTX2TextConnectors`` + Gemma-3 text encoder).
* Decode through the LTX-2 VAE (``AutoencoderKLLTX2Video``).
One-line usage:
pipe = SanaWMPipeline.from_pretrained(
"Efficient-Large-Model/SANA-WM_bidirectional-diffusers",
torch_dtype=torch.bfloat16,
).to("cuda")
out = pipe(image=img, prompt="...", action="w-80,jw-40,w-40",
intrinsics=[fx, fy, cx, cy])
End-to-end smoke test (stage-1 + refiner + VAE decode) passes on H100.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(sana-wm): align pipeline with merged sana_video style; fix mp4 export
transformer_sana_wm.py:
* License header switched to the "HuggingFace Team and SANA-WM Authors"
style used by merged sana_video.
* Imports rewritten in stdlib -> third-party -> diffusers order; use
diffusers `from ...utils import logging` instead of stdlib `logging`.
* Fix 9 `Optional[X]` annotations written as `X or None` (Python's `or`
short-circuits and silently returns `X`).
* Fix two `assert (cond, msg)` tuple-asserts in PatchEmbedMS3D.forward
that always pass (SyntaxWarning at import time).
* Remove duplicate `__all__` declarations (the second silently overwrote
the first).
* Remove dead `reset_bn` (imports a nonexistent `packages.apps.utils`,
would crash on call).
* Remove the duplicate `logger = logging.getLogger(__name__)` further
down in the file.
transformer_sana_wm_kernels.py:
* License header normalized; collapse three duplicate triton/torch import
blocks into one.
pipeline_sana_wm.py:
* License header normalized.
* `_decode_latents` now returns `(T, H, W, 3)` float in [0, 1], matching
the diffusers convention used by `VideoProcessor`. Returning uint8
silently broke `export_to_video`: it does `frame * 255` assuming float
input, so uint8 overflows to `(-x) mod 256` and inverts colors.
* `__call__` converts to PIL/uint8 only when `output_type="pil"`.
* Intrinsics argument now accepts (4,), (F, 4), (3, 3), and (F, 3, 3)
forms (auto-extracts fx, fy, cx, cy from a 3x3 K) and auto-trims to
`num_frames` when a longer-than-needed trajectory is passed.
* Inline `retrieve_timesteps` with the standard `# Copied from
diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.retrieve_timesteps`
marker, matching merged sana_video.
* Docstrings + EXAMPLE_DOC_STRING updated to reflect the new return type.
pipeline_output.py:
* Update `frames` field docstring to describe the new float [0, 1] return.
refiner.py, cam_utils.py, scripts/sana_wm/convert_sana_wm_to_diffusers.py:
* License headers normalized.
Docs:
* New `docs/source/en/api/pipelines/sana_wm.md` and
`docs/source/en/api/models/sana_wm_transformer3d.md`, modeled on
sana_video.md / sana_video_transformer3d.md, wired into
`docs/source/en/_toctree.yml` under Models and Pipelines.
5s end-to-end smoke test (81 frames @ 16fps, 30 stage-1 steps + 3-step
LTX-2 refiner) passes on 1x H100 80GB with `enable_model_cpu_offload`.
Round-trip diff vs raw float frames is 2.06/255 mean (h264 lossy noise),
confirming the export_to_video fix.
* feat(sana-wm): port chunk-causal AR refiner mode (RefinerChunkRunner + KV cache hooks)
The first cleanup pass only kept the legacy single-shot refiner path. That
path is what the model was *not* trained on — its docstring even says
"feeding the full sequence at once is out-of-distribution" — and its cost
is O(T^2) attention over the full latent volume, which made longer videos
unusable (~21 min per refiner step at 321 frames on an H100).
Port the chunk-causal AR mode from the upstream reference so the refiner
matches the training contract:
* `refine_latents` now defaults to `block_size=3, kv_max_frames=11`
(the canonical AR recipe). Pass `block_size=None` to fall back to the
legacy single-shot path.
* New `_refine_latents_ar` + `_RefinerChunkRunner` orchestrate the sliding
window: pre-capture pre-RoPE sink K/V on `z_sana[:source_sink_frames]`
at sigma=0, then for each `block_size`-frame chunk run a 3-step Euler
with prefix `{sink_k_pre, sink_v, sink_pe, history_k, history_v}` and
capture post-RoPE K/V to feed the next window. History is bounded to
`kv_max_frames - source_sink_frames` so per-block compute is constant.
* New `_predict_x0_active_block` runs the transformer on the active block
only (Q from active, K/V from prefix+active).
* New `_capture_block_kv` runs sigma=0 forward with a pre_rope/post_rope
capture flag set on each `attn1`.
* New `_forward_video_only_with_rope` takes a pre-built RoPE so each block
can use absolute frame positions in the source video.
* `_streaming_self_attention` extended with the `_kv_cache_capture`,
`_tf_capture_kv`, `_tf_kv_prefix` hook contract that AR mode uses to
inject and capture K/V on each block.
* New helpers: `_build_rotary_emb_for_absolute_positions`,
`_set_kv_prefix_on_blocks`, `_clear_kv_prefix_on_blocks`,
`_set_capture_flag_on_blocks`, `_collect_captured_kv_from_blocks`.
* `_encode_prompt` now also moves the Gemma-3 text encoder back to CPU
after producing the embeds — otherwise it stays resident through the
entire AR loop and gates how much GPU memory the refiner transformer
has left.
Module-level docstring updated to document both modes; existing
single-shot path preserved verbatim.
* feat(sana-wm): block-level checkpoint for AR refiner (resume after preemption)
The AR refiner is expensive (~3-5 min per block) and the refinement loop
ran end-to-end has no in-progress state to recover, so a SLURM preemption
mid-refinement loses all progress. With the canonical
``block_size=3, kv_max_frames=11`` setup, refining a 50s video is 34
blocks of work that has to make it through without preemption on a
backfill queue.
Add per-block atomic checkpointing:
* ``SanaWMLTX2Refiner.refine_latents(checkpoint_dir=Path)`` and
``_refine_latents_ar`` accept a directory. After each completed AR
block, the AR loop writes ``checkpoint_dir/state.pt`` atomically
(tmp + os.replace).
* The payload is ``{block_idx_done, n_blocks, sink_size, block_size,
output_shape, output, runner_state}``. ``runner_state`` is a CPU snapshot
of the runner's ``_sink_kv_pre``, ``_history_kv_post``,
``_history_frames`` and ``torch.Generator`` state.
* On entry, if ``state.pt`` exists with a compatible shape signature, the
AR loop loads the persisted output tensor + runner state and resumes
from ``block_idx_done + 1`` instead of recomputing from scratch.
* ``SanaWMPipeline.__call__(refiner_checkpoint_dir=...)`` plumbs the
directory through to the refiner.
Checkpoint size: ~output_volume + sink_KV (~360MB for 50 layers) +
rolling history KV (~3-4GB at full capacity) — saved once per block,
total per-block save overhead ~10s on lustre.
* test(sana-wm): add CPU unit tests + slow GPU integration stub
* CPU unit tests for cam_utils helpers (action DSL → c2w, intrinsics
rescale-for-crop, resize+center-crop, snap_num_frames 8k+1 rounding).
* Public-surface registration tests (top-level diffusers symbols,
SanaWMPipelineOutput dataclass shape, refiner signature has AR defaults
+ checkpoint_dir, pipeline __call__ accepts c2w/action/intrinsics/
refiner_checkpoint_dir).
* @slow @require_torch_accelerator integration stub for an end-to-end I2V
against the public checkpoint, currently @unittest.skip — wires up the
nightly GPU path without exploding regular CI.
SanaWMTransformer3DModel has hardcoded depth/hidden_size/num_heads inside
its inner SanaMSVideoCamCtrl (not exposed through register_to_config), so
the usual PipelineTesterMixin small-config fast tests aren't applicable
without a transformer refactor (followup PR).
* feat(sana-wm): make triton optional + auto-fallback to pure-PyTorch attention
`transformer_sana_wm_kernels.py` previously did a hard `import triton`
at the top of the file. That blocked importing the SANA-WM transformer
on any environment without Triton (CPU-only, ROCm without Triton,
older Triton, etc.), even though the model has pure-PyTorch attention
classes for every `*Triton` variant.
Make Triton optional and have the dispatcher transparently fall back:
* Wrap `import triton` / `import triton.language as tl` in try/except.
When unavailable, install a shim where `@triton.jit` is a no-op so
the kernel function definitions still load (they just aren't compiled
by Triton). Module-level `triton.X` / `tl.X` lookups return a
self-shimming sentinel so signature parsing doesn't blow up either.
* Add `is_triton_available()` + `_require_triton(entry_point)`. The four
Triton-backed entry points called by the model (`fused_qk_inv_rms`,
`fused_bigdn_func`, `cam_prep_func`, `cam_scan_bidi_chunkwise`) now
raise a clear RuntimeError on a Triton-less host with a hint to use
the pure-PyTorch attention variants — but the dispatcher does this
automatically (see below) so users shouldn't ever see it.
* Delete the leftover duplicate `import torch / triton / triton.language`
block at line 262 (left over from the upstream port).
* Register `BidirectionalGDNUCPESinglePathLiteLA` in `ATTENTION_BLOCKS`
so the fallback chain can find it.
* New `_resolve_attention_block(name, role)` walks the requested class's
MRO at dispatch time. If Triton isn't usable AND the requested class
name ends in `Triton`, route to the closest registered non-`Triton`
ancestor (BidirectionalGDNUCPESinglePathLiteLABothTriton ->
BidirectionalGDNUCPESinglePathLiteLA, etc.) and log a one-shot warning.
* Rewire both `SanaVideoMSCamCtrlBlock` dispatch sites to use
`_resolve_attention_block` for the GDN+UCPE camera branch and the main
attention branch (the `BidirectionalSoftmaxUCPESinglePathLiteLA` branch
doesn't use Triton at all so it stays hard-coded).
Tests:
* `test_kernels_module_imports_with_triton_hidden` — reloads the kernels
module with `sys.modules['triton'] = None` and verifies the module
imports, `is_triton_available()` is False, and the pure-PyTorch helpers
remain callable.
* `test_resolve_attention_block_cpu_fallback` — on a CPU-only host, the
three `*Triton` attn types resolve to the correct non-Triton ancestor.
* `test_triton_entry_point_raises_clean_error_without_triton` — verifies
the `_require_triton` guard yields a RuntimeError that mentions Triton.
* fix(sana-wm): make optional deps lazy + register transformer in __init__
Three CI checks were failing on the PR:
1. `check_code_quality` (43 ruff errors): mix of unused imports / import
sorting / E731 lambdas (auto-fixable) plus a handful of F821 dead-code
references inherited from the upstream research codebase (`xformers.*`
inside `if _xformers_available:` blocks, an undefined `BlockHook` type
annotation, two `x_sa`/`mlp_out` references in a block forward whose
live assignment was already overridden by subclasses). Ran `ruff check
--fix --unsafe-fixes` + `ruff format`, fixed the type annotation
manually, and added targeted `# noqa: F821` markers on the conditionally
unreachable lines.
2. `check_torch_dependencies`: `transformer_sana_wm.py` hard-imported
`einops`, `fla`, `timm`, `termcolor`. The minimum-deps CI environment
doesn't have them, and diffusers' lazy loader rewrites `ModuleNotFoundError`
as `RuntimeError` so `test_pipeline_imports` blew up. Wrapped each of
the four optional imports in a try/except shim — `rearrange`/
`ShortConvolution`/`DropPath`/`Attention_`/`Mlp` become placeholders
that raise a clear `ImportError` on construction, `colored` falls back
to plain text. Class bodies that subclass these still parse at module
load, so `import diffusers.models.transformers.transformer_sana_wm`
succeeds anywhere. Same treatment for the kernels file's
`from einops import rearrange, repeat`.
3. `build_pr_documentation`: doc-builder imported `SanaWMTransformer3DModel`
from `diffusers.models.transformers` (not the diffusers top level) and
that subpackage's `__init__.py` was missing the entry. Added the import.
* style(sana-wm): apply make style + fix-copies
* `doc-builder style src/diffusers docs/source --max_len 119` rewraps
docstrings in the six SANA-WM files (transformer, kernels, pipeline,
refiner, output, cam_utils) to the repo-wide 119-column limit. No
behaviour change — purely whitespace inside docstrings.
* `make fix-copies` regenerates `dummy_pt_objects.py` and
`dummy_torch_and_transformers_objects.py` to add `DummyObject` stubs
for the three new public classes (`SanaWMTransformer3DModel`,
`SanaWMPipeline`, `SanaWMLTX2Refiner`), so `from diffusers import …`
gives the standard "missing backend" message on installs without
torch / transformers.
Verified: `make quality` passes (ruff check, ruff format check,
doc-builder style check_only, check_doc_toc). Test suite still
15 passed / 1 skipped.
* refactor(sana-wm): drop einops dependency, inline with torch ops
Per @dg845's review on `transformer_sana_wm.py:44`. The 9 call sites all
match well-known patterns that have one-liner torch equivalents:
rearrange(s, "b d -> (b d)") -> s.reshape(b * d)
rearrange(x, "(b d) d2 -> b (d d2)", ...) -> x.reshape(b, d * d2)
rearrange(R, "b t h w i j -> b t h w j i") -> R.transpose(-1, -2)
repeat(x, "b h w c -> b t h w c", t=T) -> x.unsqueeze(1).expand(-1, T, -1, -1, -1)
repeat(x, "b t c -> b t h w c", h=H, w=W) -> x[:, :, None, None, :].expand(-1, -1, H, W, -1)
repeat(x, "... -> b ...", b=B) -> x.unsqueeze(0).expand(B, *x.shape)
repeat(x, "H W C -> B T H W C", B, T) -> x[None, None].expand(B, T, -1, -1, -1)
repeat(x, "B H W C -> B T H W C", T) -> x.unsqueeze(1).expand(-1, T, -1, -1, -1)
Each replacement is bit-identical to the einops original — verified
against a fresh `einops` install on random tensors before swapping. The
optional `from einops import ...` shim block is gone from both
`transformer_sana_wm.py` and `transformer_sana_wm_kernels.py`.
* refactor(sana-wm): remove dead code per @dg845's review
Drop 23 unused symbols (1451 lines) from `transformer_sana_wm.py` that
aren't reachable from the public SANA-WM checkpoint's
`SanaMSVideoCamCtrl` -> `SanaVideoMSCamCtrlBlock` -> GDN/UCPE attention
path. Each was verified to have zero call sites outside of its own
definition (or only within other-deleted items).
Classes:
* `SanaMS`, `SanaMSBlock` — alternative `Sana` subclass + block, not
used by the SANA-WM checkpoint (which goes through `SanaMSVideoCamCtrl`).
* `ChunkCausalAttention`, `CachedCausalAttention`,
`ChunkedLiteLAReLURope`, `LiteLAReLURope` — chunk-causal / cached
attention variants and their common base; SANA-WM uses the bidi GDN
path. `LiteLAReLURope` had only the three (now-deleted) subclasses
referencing it.
* `PAGCFGIdentitySelfAttnProcessorLiteLA`,
`PAGIdentitySelfAttnProcessorLiteLA`, `SelfAttnProcessorLiteLA`,
`SelfAttnProcessorLiteLAReLURope` — PAG processors; we don't expose
PAG in the SANA-WM pipeline.
* `ChunkGLUMBConvTemp`, `CachedGLUMBConvTemp`, `MBConvPreGLU` —
alternative FFN/conv blocks; the checkpoint uses `GLUMBConvTemp`.
* `MaskFinalLayer`, `DecoderLayer` — alternative final layers; the
checkpoint uses `T2IFinalLayer`.
* `LabelEmbedder`, `CaptionEmbedderDoubleBr` — alternative embedders;
the checkpoint uses `CaptionEmbedder`.
Helpers:
* `set_grad_checkpoint`, `prepare_prompt_ar`, `resize_and_crop_tensor`,
`generate_temporal_head_mask_mod`, `is_chunk_causal_request`,
`get_chunk_index_from_config` — training-only or
chunk-causal-mode helpers with no inference call sites.
Verified after the diff:
* `make quality` clean.
* CPU test suite 15 passed / 1 skipped (slow GPU integration).
* `import diffusers ; SanaWMPipeline / SanaWMTransformer3DModel /
SanaWMLTX2Refiner` resolve identically.
* fix(sana-wm): defer transformers import + document mask/return_dict
Two PR-CI failures, both ours:
* `check_torch_dependencies`: line 33 hard-imported `transformers`, which
isn't present in the minimum-deps environment. Move the lone
`AutoModelForCausalLM` use site inside `initialize_gemma_params` (a
training-only helper).
* `check_repository_consistency`: `SanaWMTransformer3DModel.forward`'s
docstring was missing entries for `mask` and `return_dict`. Added.
* refactor(sana-wm): address review feedback on SanaWMPipeline
Per @dg845's review comments on pipeline_sana_wm.py:
* Read the VAE spatial/temporal strides from the ``vae`` component
(``vae_spatial_compression_ratio`` / ``vae_temporal_compression_ratio``)
instead of hardcoding 32/8, mirroring LTX2Pipeline.
* Inline the stage-1 sampling loop into ``__call__`` and factor the noise
init into a standard ``prepare_latents`` method.
* Use ``self.scheduler`` for the flow-matching Euler steps instead of
constructing a new scheduler per call.
* Drive the sampling loop with ``self.progress_bar`` (respects
``set_progress_bar_config``) instead of a bare tqdm.
* Move input validation/normalization into a ``check_inputs`` method.
* Move first-frame resize+center-crop, [-1, 1] normalization and the
intrinsics rescale into a ``SanaWMImageProcessor(VaeImageProcessor)``
subclass (new ``image_processor.py``).
* Add ``generator`` as a ``__call__`` argument (``seed`` kept as a
convenience shortcut).
* Post-process decoded latents with ``VideoProcessor.postprocess_video``
rather than a hand-rolled conversion; drop the ``.cpu()`` casts on the
``output_type="latent"`` path.
* Move the pipeline README into the docs (docs/.../sana_wm.md); delete
the in-package README.
* refactor(sana-wm): make SanaWMLTX2Refiner a standalone DiffusionPipeline
Per @dg845's review: the refiner has components (transformer, connectors,
text encoder, tokenizer) and runs a denoising loop, so it fits better as a
pipeline than a ModelMixin.
* SanaWMLTX2Refiner now subclasses DiffusionPipeline and registers its
components via ``register_modules`` (dropping the bespoke
``from_pretrained`` / ``save_pretrained``); standard load/save now handle
the ``refiner/`` subfolder.
* Add a ``FlowMatchEulerDiscreteScheduler`` component (shift=1.0) and drive
the Euler steps through ``scheduler.step`` / ``scheduler.scale_noise``
(single-shot and per-AR-block) instead of hand-rolled updates. Numerically
equivalent to the previous flow-matching update.
* Rename the entry point ``refine_latents`` -> ``__call__``; add a ``device``
arg so the parent can hand it the execution device without a bulk move.
* SanaWMPipeline: keep the refiner as an optional nested component; free the
parent's GPU weights before running it (it manages its own sub-module
placement) and bring the VAE back for decode.
* Conversion script emits the new refiner layout (model_index.json +
scheduler/ + tokenizer/); test asserts the refiner is a DiffusionPipeline
with the canonical AR ``__call__`` defaults.
Validated end-to-end on 1xH100 (stage-1 + refiner on the official demo,
coherent video output).
* docs(sana-wm): document the refiner __call__ `device` arg
Fixes the check_repository_consistency failure — `SanaWMLTX2Refiner.__call__`
gained a `device` parameter but its docstring wasn't updated.
* refactor(sana-wm): clean up dead code + reuse shared utils in the DiT
Addresses several of @dg845's transformer review comments (the safe,
output-preserving subset — GPU smoke gives byte-identical output):
* Reuse diffusers' shared `FP32LayerNorm` (models/normalization.py) and
`get_1d_rotary_pos_embed` (models/embeddings.py); delete the local copies.
* Remove dead inference code paths:
- the `if self.diagonal_mask is not None:` flex-attention block
(`diagonal_mask` is always `None`) + the now-unused
`create_block_mask_cached` helper and `create_block_mask` import;
- the `SANA_FSDP2_BLOCK_TIMING` block-timing/profiling scaffolding;
- the `save_qkv` / `qkv_store_buffer` visualization hooks (never
enabled at inference), at both the attention and model level.
* Collapse `SanaVideoMSCamCtrlBlock.forward_frame_aware` into `forward`
(the pipeline/refiner always pass >=3D timesteps, so the non-frame-aware
branch was dead — it even referenced undefined locals).
* Drop the 191-line `SanaMSVideoCamCtrl.load_state_dict` shape-remapping
override — it's never reached by the shipped convert/inference flow
(`nn.Module.load_state_dict` on the wrapper doesn't call it), and the
release checkpoint already ships correctly-shaped weights.
* Delete the unused `PatchEmbedMS` module.
Net -503 lines; no numerics change (stage-1 + refiner smoke output mean
identical to the pre-refactor run).
* refactor(sana-wm): merge the 3 DiT classes into one SanaWMTransformer3DModel
Per @dg845's review: `SanaWMTransformer3DModel` was a thin wrapper over
`SanaMSVideoCamCtrl`, which in turn subclassed `Sana` and overwrote most of
it. Fold all three into a single `SanaWMTransformer3DModel(ModelMixin,
ConfigMixin)`:
* The `@register_to_config` __init__ signature is unchanged (so config.json
is identical); the body builds the modules directly on `self` instead of
a nested `self._inner`.
* Only the surviving `Sana.__init__` pieces are kept (t_embedder,
cfg_embedder, attention_y_norm, config attrs, initialize_weights); the
parts the subclass overwrote are gone.
* `forward` takes the diffusers signature (hidden_states / timestep /
encoder_hidden_states / encoder_attention_mask / return_dict) and returns
`Transformer2DModelOutput`, folding in the old wrapper's arg-renaming.
* Deleted now-unreachable code: `class Sana`, `class SanaMSVideoCamCtrl`,
`class SanaBlock`, `class PatchEmbed` (the video model uses
`PatchEmbedMS3D`), the `add_inner_prefix` helper, and the dead
`sincos`/`flux_rope` pos-embed branches in forward (release uses
`wan_rope`). Net -468 lines.
This drops the `_inner.` state-dict prefix, so the conversion script no
longer adds it. State-dict is otherwise identical: the merged model's
`state_dict()` has exactly the same 871 param keys as before (verified),
and a stage-1 + refiner GPU smoke on the public checkpoint gives
byte-identical output (frame mean 0.5560, matching the pre-merge run).
Addresses the class-merge review comment and removes the unused
`SanaBlock`/`PatchEmbed` modules.
* refactor(sana-wm): drop the no-op custom grad-checkpoint wrapper
`auto_grad_checkpoint` gated on a `grad_checkpointing` attribute that was
never set, so it always fell through to `module(*args, **kwargs)` — i.e. a
no-op. Call the transformer blocks directly instead and delete the unused
`auto_grad_checkpoint` / `checkpoint_sequential` helpers and the
`torch.utils.checkpoint` import.
(Inference is unchanged — the block never read the `use_reentrant` kwarg
the wrapper passed. Full training-time gradient checkpointing via the
standard `_gradient_checkpointing_func` would need the block's dynamic
kwargs — camera_conditions / prope_fns / chunk_index — threaded through,
so it's left as a follow-up for this inference-focused release.)
* refactor(sana-wm): drop the fla-core and termcolor dependencies
Per @dg845 / @sayakpaul: the SANA-WM transformer required `fla-core` to be
constructible at all (and fla's `ShortConvolution` cannot even run on CPU —
its dispatch does `torch.cpu.device(...)`, which doesn't exist).
`ShortConvolution` was only ever used as a depthwise *causal* conv1d with
`activation=None`, so it is replaced by a ~20-line self-contained PyTorch
module with the same parameter layout (`weight` of shape `(C, 1, K)`, no
bias) and the same `(output, cache)` return signature. No `fla-core` and no
`kernels` dependency is needed now, and the model can be built and run on
CPU. Verified on an H100 in bf16 at the real config (hidden 2240, kernel 4):
`max|Δ|` vs fla is exactly `0.0` for every shape tested, and the state dict
keeps the identical 871 keys.
Also:
* Use the existing `is_timm_available()` utility for the timm imports. The
`else` branch keeps a placeholder class rather than raising at module
scope, because several layers subclass these symbols and this module is
imported eagerly by `diffusers.models` — a module-level raise would break
plain `import diffusers` (and the `check_torch_dependencies` CI job) when
timm isn't installed. The error is instead raised on construction.
* Drop `termcolor` entirely: replace the `self.logger = print` +
`colored(...)` pattern with the module-level `logger`, so there is no
optional dependency left to gate.
* Only attempt the optional null-embedding load when `null_embed_path` is
actually set (it is unset for the public checkpoint, so this previously
logged a spurious failure on every construction) and load it with
`weights_only=True`.
* fix(sana-wm): address review nits in the pipeline / refiner / cam utils
* refiner: load the AR resume checkpoint with `weights_only=True`. The
payload is only tensors / ints / tuples / dicts plus the generator's uint8
state, so it round-trips safely under the restricted unpickler.
* refiner: fix `_tokens_per_frame` to divide by `patch_size_t` rather than
multiply — `_pack_latents` emits `(T // patch_size_t) * (H // p) * (W // p)`
tokens, so one latent frame contributes `(H // p) * (W // p) / patch_size_t`.
No-op for LTX-2 (`patch_size_t=1`) but the history trimming would have kept
too many tokens otherwise.
* pipeline: replace `torch.cuda.empty_cache()` with diffusers'
backend-agnostic `empty_device_cache(device.type)`.
* pipeline: extract the offload probe into `_model_cpu_offload_active()`,
matching the `hasattr(self, "_all_hooks") and len(self._all_hooks) > 0`
idiom `DiffusionPipeline` uses internally, instead of a bare truthiness
check on the attribute.
* pipeline_output: add the missing Apache-2.0 license header.
* cam_utils: drop the unnecessary `+ 1e-6` when normalizing the forward /
right vectors — the branch is only taken when the norm is already > 0, so
the epsilon just introduced a small systematic bias.
* refactor(sana-wm): strip research-repo code from the DiT, fix model conventions
From the `self-review` skill run against `.ai/models.md` / `.ai/AGENTS.md`.
`transformer_sana_wm.py`: 6555 -> 4592 lines.
Conventions:
* `_no_split_modules` was `["blocks"]` — an attribute name, but accelerate
matches on class name, so it matched nothing and `device_map="auto"` could
split a block across devices and crash. Now `["SanaVideoMSCamCtrlBlock"]`.
* Add `_repeated_blocks` (enables `compile_repeated_blocks()`) and
`_skip_layerwise_casting_patterns`. `_keep_in_fp32_modules` is deliberately
left unset with a note: the blocks apply the timestep modulation inline, so
keeping `scale_shift_table` / `t_embedder` in fp32 upcasts the hidden states
and feeds fp32 activations to bf16 weights (caught by a GPU smoke run).
* Expose `num_layers` / `hidden_size` / `num_attention_heads` / `patch_size`
through `register_to_config` instead of hardcoding the release architecture,
so a tiny variant can be built for tests. Defaults are the released values,
so `config.json` and the state dict are unchanged.
* `torch.float64` -> `torch.float32` on the live RoPE paths (gotcha 5),
`torch.empty` -> `torch.zeros` for parameter init (gotcha 6), and stop
reading `self.proj.weight.dtype` to cast activations (gotcha 4).
* `WanRotaryPosEmbed.freqs` is a non-persistent buffer instead of a plain
attribute reassigned inside `forward` (which broke `.to()` and compile).
Dead code (`AGENTS.md`: "delete training-time code paths, experimental flags,
and ablation branches entirely — only keep the inference path"):
* All weight-init / transfer-learning helpers — `from_pretrained` overwrites
them, and they also printed ~20 lines of noise on every construction.
* `CaptionEmbedder.initialize_gemma_params` (fetched `google/gemma-2b-it`
out-of-band at runtime, and referenced an attribute that never exists),
`token_drop`, and the training branch of its forward.
* The cam-debug statistics apparatus, the `save_block_output` hooks (whose
`get_block_output` was never defined), and `block_hook`.
* Both unreachable recurrence variants, `_maybe_drop_cam_branch`, the xformers
branches (`_xformers_available` was a literal `False`, defined twice, and
`xformers` was never imported), 3 env-var escape hatches, 9 unused classes,
12 unused module-level helpers, and the sincos family — which also removes
the NumPy import, satisfying the no-NumPy-in-forward rule.
* Inline the `fp32_attention` mechanism: it was set on every submodule via
`model.apply` and read at 18 sites, so it was folded to the shipped
always-on behaviour rather than deleted.
Ephemeral comments (commit SHAs from a private tree, references to files that
don't exist in diffusers, FSDP2 rationale, stale docstrings) removed.
State dict is unchanged: 871/871 keys match the released checkpoint.
* fix(sana-wm): honour scheduler shift, use randn_tensor, drop cluster scaffolding
From the `self-review` skill run against `.ai/pipelines.md` / `.ai/testing.md`.
`flow_shift` was a dead knob. `__call__` set `self.scheduler.config.shift`,
but `FlowMatchEulerDiscreteScheduler.set_timesteps` reads `self.shift` (i.e.
`self._shift`), so the documented argument had no effect and every run used
the checkpoint's `shift=9.8`. It also half-mutated a `FrozenDict` — the freeze
guard checks a name-mangled attribute, so the assignment silently desynced
`config.shift` from `config["shift"]` on a shared component. Removed the
argument so the scheduler owns the shift (`pipelines.md` gotcha 3), which also
drops a per-call mutation of a registered component (gotcha 7).
Other pipeline fixes:
* `torch.randn` -> `randn_tensor` in `prepare_latents` and the refiner
(gotcha 10). `__call__` advertises `generator: Generator | list[Generator]`,
but `torch.randn` raises on a generator list and on a CPU generator with a
CUDA device, so that path could not work.
* Delete the SLURM preemption/resume feature (`checkpoint_dir`,
`_atomic_save_state`, `_capture_state` / `_restore_state`) and the
single-shot refiner path the docstring itself called a debugging fallback —
roughly 270 lines of research-cluster scaffolding.
* `_empty_cuda_cache` (CUDA-only) -> `empty_device_cache`;
`@torch.inference_mode()` -> `@torch.no_grad()` and removed from inner
helpers the decorator already covers (gotcha 2).
* Remove dead `_callback_tensor_inputs` (no `callback_on_step_end` exists),
`_exclude_from_cpu_offload` (a no-op — the base class already skips
non-`nn.Module` components), `_kv_max_frames`, and `latents.detach()`.
Tests: drop the `@slow` integration stub — `testing.md` says integration and
slow tests don't belong in the initial PR.
`refiner.py`: 1286 -> 1017 lines.
* refactor(sana-wm): drop the timm dependency
Per @yiyixuxu's review. The three timm imports were all trivially replaceable:
* `Mlp` — both local subclasses fully overrode `forward` and only inherited
`fc1` / `act` / `drop1` / `fc2` / `drop2`, so it is now a self-contained
module and the redundant `class Mlp(Mlp)` wrapper is gone.
* `Attention` — `GDN` inherited only `qkv` and `proj` from it (it already
defines its own `q_norm` / `k_norm`), so those two layers are declared
inline and `GDN` subclasses `nn.Module`.
* `DropPath` — `drop_path` was hardcoded to `0.0`, so this was always
`nn.Identity()`. Removed along with the rest of the plumbing.
No `dummy_timm_objects.py` is needed since the dependency is gone rather than
optional, and `setup.py` is untouched (other models still use timm).
State dict is unchanged (871/871 keys match the released checkpoint) and the
end-to-end GPU smoke gives byte-identical output (frame mean 0.5554).
* refactor(sana-wm): drop the Triton kernels and apply review feedback
Per @yiyixuxu's inline review. Removes `transformer_sana_wm_kernels.py`
entirely (3234 lines) — the Triton kernels will be published to
`kernels-community` and wired back in a follow-up PR.
* Delete the three `*Triton` attention classes. Each only overrode `forward`
to call a fused kernel; their pure-PyTorch parents (`BidirectionalGDN`,
`BidirectionalGDNUCPESinglePathLiteLA`) compute the same result. The
Triton-availability probe and the MRO-walking fallback go with them.
The released `config.json` still names the `*Triton` variants, so
`ATTENTION_BLOCKS` maps those strings onto the pure-PyTorch classes.
* Move the nine pure-PyTorch camera-math helpers that were reachable from
the model into `transformer_sana_wm.py`; drop the rest with the kernels
file (they were only reachable from the deleted Triton path).
Other review items:
* Use `get_activation()` from `..activations`; delete the local activation
and normalization registries (`build_norm` was only ever called with
`None`).
* Delete `GLUMBConv` and make `GLUMBConvTemp` self-contained. `ConvLayer` is
slimmed and renamed `SanaWMConvLayer`, but has to stay a module — the
checkpoint keys are `mlp.inverted_conv.conv.weight`, so collapsing it to a
bare `nn.Conv2d` would drop the `.conv` level.
* Remove the closure-returning helpers the review flagged as hard to read
and compile-hostile: `prepare_prope_fns` and friends now return plain
tensors via `_prepare_ucpe_ray_transforms` + `_apply_ucpe_transform`, and
the `_register_block` decorator factory is an explicit dict update.
* Remove all seven `@torch.compile` decorators — users can reach for
`compile_repeated_blocks()` instead.
* Drop `partial` (pass `chunk_size` at the call site), `DWMlp`, the tuple
helpers, `get_same_padding`, `modulate`, an unused `apply_rotary_emb`,
and the validation of internal-only arguments.
* Tests: drop `SanaWMTritonFallbackTests` along with the mechanism it covered.
State dict is unchanged (871/871 keys) and the end-to-end GPU smoke passes
with correct video output.
* refactor(sana-wm): inline the modulation helper, drop the backward-only shim
Two more items from @yiyixuxu's inline review:
* Inline `t2i_modulate` at its four call sites.
* Delete `_IdentityForwardContiguousBackward` / `_contiguous_backward`. It is
the identity in forward and only exists to hand a contiguous gradient to
the backward pass, so it is dead weight in an inference-only port.
State dict unchanged (871/871 keys); GPU smoke output matches the previous
run exactly (frame mean 0.5560).
* Update src/diffusers/models/transformers/transformer_sana_wm.py
Co-authored-by: YiYi Xu <yixu310@gmail.com>
* Update src/diffusers/models/transformers/transformer_sana_wm.py
Co-authored-by: YiYi Xu <yixu310@gmail.com>
* style(sana-wm): drop the blank line left by removing the camera-branch init
* refactor(sana-wm): address @dg845's review round
Transformer:
* Stop stashing patch-grid shape on `self` during `forward` (`self.f/h/w`) and
thread it through as locals; `unpatchify` now takes it explicitly.
* Replace the 7 `assert`s with `ValueError`s.
* Drop `attn_drop`/`proj_drop` from `MultiHeadCrossAttention` (training-only,
and `attn_drop` was never applied) plus a stale training-era banner comment.
Pipeline / refiner:
* Don't mutate the components handed to the pipeline. The VAE tiling +
framewise settings move to the docs, `padding_side="right"` is passed per
tokenizer call, and the `.eval()` calls are gone (no `self.training`
branches remain, and `from_pretrained` already returns eval-mode modules).
* Gate `cam_utils`' optional imports on `is_torchvision_available()` and a new
`is_pi3_available()` helper.
* Annotate `SanaWMLTX2Refiner.__init__` and inline `_refine_latents_ar` into
its single caller.
Conversion script:
* Move to `scripts/` alongside the other Sana converters.
* Raise on missing/unexpected keys instead of printing, so a bad mapping can't
silently emit a broken transformer.
State dict unchanged (871/871 keys). GPU smoke on the public checkpoint is
byte-identical to the previous run (frame mean 0.5560), including with the VAE
settings applied by the caller rather than the pipeline.
* refactor(sana-wm): rename WanRotaryPosEmbed to SanaWMRotaryPosEmbed
The name implied this was Wan's rotary embedding, but it isn't: the per-axis
split is configurable through `fhw_dim`, and the frequencies stay complex in a
single `freqs` buffer instead of being split into real cos/sin buffers. So it
can't carry a `# Copied from`. Renamed, with a docstring recording why.
The buffer is `persistent=False`, so the state dict is unchanged (871/871).
* test(sana-wm): add transformer model tests, migrate pipeline tests to pytest
* `tests/models/transformers/test_models_transformer_sana_wm.py` — generated
with `utils/generate_model_tests.py` and filled in, following
`test_models_transformer_sana_video.py`. The tiny config sets
`softmax_every_n=2` so one block exercises the GDN camera branch and the
other the softmax variant. Dummy inputs supply the conditioning the forward
requires: `encoder_attention_mask`, `(B, F, 20)` camera conditions, and
`chunk_plucker`.
* `tests/pipelines/sana_wm/test_sana_wm.py` — rewritten in the pytest style of
`tests/pipelines/sana_video/test_sana_video.py`: no `unittest`, bare
asserts, module-level imports, and `parametrize` in place of loop-style
cases (15 test functions become 31 cases).
`AttentionTesterMixin` is skipped because the model calls
`F.scaled_dot_product_attention` directly rather than going through a
diffusers attention processor.
* fix(sana-wm): make the transformer layerwise-casting and offload safe
* `TimestepEmbedder.dtype` used `next(self.parameters()).dtype`, which reports
the storage dtype under layerwise casting rather than the compute dtype. Use
`get_parameter_dtype`, which is layerwise-casting aware. (The model-level
`self.dtype` already routes through it via `ModelMixin`.)
* Stop reading submodule `.weight`/`.bias` inside `forward`. Group-offload
hooks fire on a module's `forward`, so reading its parameters directly
leaves them offloaded and trips a device mismatch. The frame-gate and
output-gate helpers now call their submodules, and the fused camera QKV
projection becomes three `q/k/v` calls -- algebraically the same as one
GEMM over the concatenated weights.
* Don't mutate the tokenizer in `SanaWMLTX2Refiner._encode_prompt`; pass
`padding_side="left"` per call, matching `SanaWMPipeline`.
* Add a `torch.compile` test with `recompile_limit=2` -- the repeated block
compiles once per attention variant.
GPU smoke unchanged (frame mean 0.5560).
* refactor(sana-wm): explicit arguments instead of **kwargs, generator instead of seed
Replace the `**kwargs` bag threaded through model -> block -> attention with
explicit keyword arguments. Only three runtime keys were ever read at the
leaves (`frame_valid_mask`, `precomputed_gates`, `ucpe_ray_transforms`); the
rest were forwarded and silently swallowed. `camera_embedding`,
`chunk_index`, `chunk_index_global` and `chunk_split_strategy` turned out to
be pure dead plumbing -- written into the per-block kwargs dicts and never
read by any attention or MLP forward -- so they are gone.
The pipeline also grows the standard diffusers arguments:
* `prompt_embeds` / `prompt_attention_mask` / `negative_prompt_embeds` /
`negative_prompt_attention_mask` on `encode_prompt` and `__call__`.
* `seed` / `refiner_seed` are replaced by `generator` / `refiner_generator`.
`generator=torch.Generator(device).manual_seed(42)` reproduces exactly what
`seed=42` used to build, so results are unchanged.
State dict unchanged (871/871). CPU old-vs-new equality on a tiny config over
both attention variants is exact (`torch.equal`, max diff 0.0), and the GPU
smoke on the public checkpoint still gives frame mean 0.5560.
* refactor(sana-wm): drop the unreachable chunk-split strategies
@yiyixuxu was right that the model only ever runs uniform chunking; my earlier
reply defending the strategies was wrong. The two sites that actually chunk
both call `normalize_chunk_index(None, T, chunk_size)` with three positional
arguments, so `chunk_split_strategy` always took its `"uniform"` default. The
configured `first_chunk_plus_one` reached the per-block kwargs dict and was
then swallowed by the attention forwards' `**kwargs` without ever being read.
Flattening that bag into explicit arguments is what surfaced it.
Also note the `chunk_size` those call sites use is `chunk_gdn_chunk_size`, a
different attribute from the `chunk_size` that was being threaded through.
So `chunk_index_from_chunk_size`, `normalize_chunk_index`,
`is_uniform_chunking` and `compute_chunk_sizes` are gone, the uniform
boundaries are inlined at both call sites, and `chunk_split_strategy` is
dropped from the model and block constructors.
The released `config.json` still carries the key; loading is unaffected
(`extract_init_dict` ignores it) but it logs an "not expected and will be
ignored" warning, so it should come out of the checkpoint config on the next
export.
State dict unchanged (871/871); GPU smoke still frame mean 0.5560, which
confirms those branches were never taken.
* refactor(sana-wm): use the shared RMSNorm
@yiyixuxu asked whether the shared `RMSNorm` could be used here and my earlier
reply overstated the obstacles. Re-checking each one:
* `scale_factor` is not a blocker. `attention_y_norm.weight` is one of the 871
checkpoint keys, so `from_pretrained` overwrites whatever the constructor
initialised -- exactly the same reasoning that removed the other
hand-written inits in this PR. It only ever affected from-scratch models.
* `norm_dim` is not a blocker either; it is always the default `-1`.
* The numerics do differ: ours ran the normalisation and the weight multiply
in fp32, while the shared class computes only the variance in fp32 and does
the scaling and weight multiply in the input/weight dtype.
Measured rather than argued: the end-to-end GPU smoke on the public
checkpoint moves from frame mean 0.5560 to 0.5561, and the decoded video is
visually identical. That is well inside bf16 noise, so the local class is not
worth keeping.
The now-unused `y_norm_scale_factor` config argument goes too. As with
`chunk_split_strategy`, the released `config.json` still carries it and will
log an "not expected and will be ignored" warning until the checkpoint config
is re-exported.
* refactor(sana-wm): migrate MultiHeadCrossAttention to the diffusers attention API
* Split the attention maths into a `SanaWMCrossAttnProcessor` and make
`MultiHeadCrossAttention` an `AttentionModuleMixin`, so the processor can be
swapped and the standard tooling applies.
* Use `dispatch_attention_fn` instead of calling
`F.scaled_dot_product_attention` directly, which lets the model pick up
alternative attention backends. This also drops the two transposes, since
the dispatcher takes `(batch, seq, heads, dim)`.
* Build a boolean padding mask rather than an additive float one. The old
`(1 - mask) * -10000.0` form isn't supported by the varlen backends.
Projection names are unchanged (`q_linear`, `kv_linear`, `proj`, `q_norm`,
`k_norm`), so the state dict stays at 871/871 keys. The GPU smoke on the
public checkpoint is unchanged at frame mean 0.5561, so `-inf` masking makes
no difference against the old `-10000.0` at this precision.
* feat(sana-wm): give the refiner its own transformer model
Per @yiyixuxu and @dg845: the stage-2 refiner no longer reaches into
`LTX2VideoTransformer3DModel`. It used to drive AR refinement by setting
attributes on LTX-2's submodules (`attn._tf_kv_prefix`, `_kv_cache_capture`,
`_tf_capture_kv`), which meant a second `forward` implementation smuggled in
from the pipeline.
New `transformer_sana_wm_refiner.py`, following the `transformer_wan_vace.py`
pattern of importing the reusable pieces rather than duplicating them:
* `SanaWMRefinerKVLayerCache` / `SanaWMRefinerKVCache`, modelled on
`Flux2KVCache`, holding the per-layer sink and history K/V.
* `SanaWMLTX2RefinerTransformerBlock`, which subclasses
`LTX2VideoTransformerBlock` and overrides only `forward`, so the submodule
structure stays byte-identical.
* `SanaWMLTX2RefinerTransformer3DModel`, taking `kv_cache` / `kv_cache_mode`
explicitly through `forward`.
`refiner.py` drops from 982 to 638 lines: the four `*_kv_prefix_on_blocks` /
`*_captured_kv_*` helpers, `_forward_video_block`, `_streaming_self_attention`
and `_forward_video_only_with_rope` are all gone, along with a dead
`n_context_tokens > 0` branch that neither call site could reach.
The state dict is unchanged at 3510 keys, identical to
`LTX2VideoTransformer3DModel` at the same config, so only the class name in
the checkpoint metadata has to change. CPU equivalence against the old path
is exact across four AR blocks -- covering sink injection, history growth and
history trimming -- with a negative control confirming the cache path is
actually exercised. The GPU smoke on the public checkpoint is unchanged at
frame mean 0.5561.
* refactor(sana-wm): run the refiner as a separate pipeline
Per @dg845, the two stages are no longer nested. `SanaWMPipeline` produces
stage-1 latents and `SanaWMLTX2Refiner` consumes them:
latents = SanaWMPipeline(...)
video = SanaWMLTX2Refiner(...)
The refiner takes an optional `vae` -- pass the base pipeline's so the weights
are shared, the way SDXL shares components between base and refiner -- and
decodes to video when given one. The manual device juggling in
`SanaWMPipeline.__call__` is gone with the nesting that forced it; each
pipeline now manages its own placement through the standard offload hooks.
Un-nesting turned up a real bug: `torch_dtype` never reached the nested
sub-pipeline, so `SanaWMPipeline.from_pretrained(..., torch_dtype=bfloat16)`
silently ran the whole of stage 2 in float32 -- transformer, text encoder and
connectors alike. That is where the "~87 GB of refiner weights" in the old
comment came from. Loaded on its own the refiner honours the requested dtype,
which halves its footprint; the GPU smoke moves from frame mean 0.5561 to
0.5548 and the decoded video is visually identical.
The conversion script writes the refiner to its own output directory
(`--dst-refiner`, default `<dst>-refiner`) rather than a `refiner/`
subfolder. `DiffusionPipeline.from_pretrained` has no `subfolder` argument, so
a nested folder silently loads the *base* pipeline's same-named component
folders instead -- exactly the kind of wrong-weights failure that only
surfaces because `connectors/` happens not to exist at the top level.
* Apply style fixes
* docs(sana-wm): document the refiner's output_type argument
`check_forward_call_docstrings` flagged `SanaWMLTX2Refiner.__call__` as
missing an entry for `output_type`, which the un-nesting added.
* refactor(sana-wm): support only what the released checkpoint uses
@yiyixuxu asked whether we can support just the released configuration and
flatten what's left. We can — the checkpoint only ever instantiates two
attention classes, so the other five in the chain were dead weight.
* Collapse `GDN -> BidirectionalGDN`, `GDN -> _GDNUCPEBase`,
`BidirectionalGDNUCPELiteLA -> ...PostUCPERenorm -> ...SinglePathLiteLA`
into a self-contained `BidirectionalGDNUCPESinglePathLiteLA` plus
`_SoftmaxUCPESinglePathLiteLA`. Inheritance depth goes 5 -> 1.
* Blocks take the class, not a string: `SanaVideoMSCamCtrlBlock` gains an
`attn_cls` argument, and the model picks per layer with a plain loop. That
removes the `ATTENTION_BLOCKS` registry, `_resolve_attention_block`,
`_GDN_TO_SOFTMAX_CAMCTRL`, `_inject_softmax_layers`, the `attn_type` /
`camctrl_type` validation, and the unreachable no-camera branch.
* `attn_type` and `camctrl_type` leave the config signature. As with
`chunk_split_strategy` and `y_norm_scale_factor`, the released
`config.json` still carries them and loading logs a benign "not expected"
warning until the checkpoint is re-exported.
* Also drop the zero-init on `GLUMBConvTemp.t_conv` (overwritten by
`from_pretrained`) and inline the one-line `restore_shape` helper. The
model docs lose a paragraph describing an `_inner.` prefix and an
`add_inner_prefix` helper that no longer exist.
3908 -> 3153 lines. State dict unchanged at 871/871 keys with identical
ordering, and 15 GDN + 5 softmax blocks as before. Old-vs-new CPU equality is
exact (`torch.equal`, max diff 0.0) on both a tiny config and the released
geometry, across a `softmax_every_n` sweep and on the backward pass, with
negative controls confirming the comparison isn't vacuous. The GPU smoke is
byte-identical too: refined latents mean 0.035024 / std 0.887467, frame mean
0.5548.
* refactor(sana-wm): make the softmax attention class standalone
Drops the last bit of inheritance between the two attention classes, per
@yiyixuxu. `_SoftmaxUCPESinglePathLiteLA` now subclasses `nn.Module`
directly.
It turned out to be a simplification rather than duplication: of the parent's
thirteen methods it only ever called three (`_apply_output_gate`,
`_prepare_frame_valid_masks`, `_stabilize_cam_transforms`), and the other ten
were dead for it. It is also always built with `conv_kernel_size=0`, so the
whole short-convolution half of the inherited `__init__` -- `conv_q/k/v`,
the `*_cam` variants, `kernel_func` -- was permanently `None` or unused and
is now gone.
State dict unchanged: 871/871 keys in the same order, and the softmax blocks
still emit the same 24 keys each. Old-vs-new equality is exact
(`torch.equal`, max abs diff 0.0) on two input configurations, with negative
controls from 0.109 to 14.62 confirming the comparison isn't vacuous; the
inlined `__init__` even consumes the RNG stream identically, so seeded
from-scratch weights are bit-for-bit the same. GPU smoke unchanged at
refined-latent mean 0.035024 / std 0.887467.
* feat(sana-wm): export the refiner weights in bfloat16
The refiner's Gemma-3 text encoder was the one component still stored in
float32, at 45.4 GB. Since the pipeline is loaded with
`torch_dtype=torch.bfloat16`, those weights were cast on every load anyway,
so storing them cast changes nothing and halves the download:
text_encoder 45.4 GB -> 22.7 GB
refiner total 84 GB -> 61 GB
The VAE stays in float32, as the docs recommend. Verified rather than
assumed: the GPU smoke is bit-identical (refined latents mean 0.035024 / std
0.887467, frame mean 0.5548), and it now runs in 124s instead of 384s because
there is far less to load.
* refactor(sana-wm): pipeline clean-up from @yiyixuxu's review
`SanaWMLTX2Refiner`:
* Dissolve `_RefinerChunkRunner` (191 lines) into `__call__`, so the AR loop
reads top-to-bottom like `pipeline_wan_animate.py`, and swap the bare
`tqdm` for `self.progress_bar(...)`.
* Actually support `enable_model_cpu_offload()`. The docs already advertised
it, but three things blocked it: the `device` default resolved through
`next(self.transformer.parameters()).device`, which is `cpu` under offload;
`_encode_prompt` called the inner Gemma backbone directly, bypassing the
hook that `cpu_offload_with_hook` installs on the top-level module; and
`_decode_latents` read `self.vae.device`. All three now go through
`self._execution_device`, and the manual `.to(...)` juggling is gone.
* TODO on the `LTX2TextConnectors` import pointing at
huggingface/diffusers#14749.
Two argument removals fall out: `progress=` (use
`set_progress_bar_config(disable=True)`, which is what every other pipeline
does) and `device=`.
`SanaWMPipeline`:
* `check_inputs` only validates now; the camera-trajectory normalisation
moves to `prepare_camera_trajectory`.
* Add the standard `guidance_scale` / `do_classifier_free_guidance` /
`num_timesteps` / `current_timestep` / `interrupt` properties.
* `return_dict=False` returned only `(frames,)`, silently dropping `c2w` and
`latent`. It now returns the same three values either way -- the flag picks
the container, not the contents.
* Take a `PIL.Image`, not a path, and drop the `_model_cpu_offload_active`
helper that had no callers left after the un-nesting.
The Pi3X intrinsics estimator leaves `diffusers` entirely: it pulled in an
extra dependency and fetched a second checkpoint without the user ever
calling `from_pretrained`. It is packaged as a remote modular block hosted
alongside the checkpoint, and the docs show `ModularPipeline.from_pretrained`
instead. `is_pi3_available` goes with it.
Equivalence for the refiner rewrite is exact across six configurations --
varying sink size, block size, KV window and seed so history growth and
trimming both fire -- `torch.equal` with max abs diff 0.0, negative controls
from 1.9e-2 to 5.55. GPU smoke on the public checkpoint is unchanged at
refined-latent mean 0.035024 / std 0.887467.
* refactor(sana-wm): model-side clean-up from @yiyixuxu's review
* Split each attention class's main and camera branches into their own
modules, so `forward` is a flat list of layer calls. The branch modules
deliberately hold no parameters: if they owned `qkv` / `q_proj_cam` /
... PyTorch would prefix by attribute name and rename 390 of the 871
checkpoint keys. (`state_dict` duplicates a module aliased in two places
rather than de-duplicating it, so aliasing isn't a way out either.) The
compute is split; the ownership isn't.
* `transformer_sana_wm_refiner.py` is self-contained -- nine `# Copied from`
markers replace the `transformer_ltx2` import, at whole-function,
whole-class and method-level granularity as each piece required. That costs
+831 lines, since the copied block `__init__` drags in both attention
processors.
* The refiner's streaming self-attention is a real attention processor,
`SanaWMLTX2RefinerAttnProcessor`, and the block calls `self.attn1(...)`. The
processor swap happens in the model `__init__` after the blocks are built,
because `attn1` is constructed inside the `# Copied from` body.
* Inline the KV-cache mode strings, drop `plucker_proj`'s zero-init.
State dicts unchanged: 871/871 and 3510/3510, ordered lists identical.
Equivalence is exact -- max abs diff 0.0 -- against the real released
checkpoint for the attention split, and across five refiner configurations
driven through two AR blocks, with negative controls throughout.
`check_copies` passes. GPU smoke unchanged at refined-latent mean 0.035024 /
std 0.887467.
* test(sana-wm): cover the GDN branch, which was silently dead in these tests
`SanaWMTemporalShortConvolution` allocates its weight with `torch.zeros` and
nothing initialises it any more -- the hand-written init came out earlier in
this PR because `from_pretrained` overwrites it. That is true for real
checkpoints, but a from-scratch model gets a zero `conv_k`, so the GDN key is
identically zero and the whole branch returns zeros. Every test in this file
builds a random model, so they were all passing while exercising none of the
GDN maths.
Two tests: one pins the zero-init as deliberate rather than accidental, the
other fills the short convolutions with noise and asserts the output moves.
Checked that the second one bites -- forcing the short convolution to return
zeros makes it fail.
* [SANA-WM] Address review: reuse diffusers embeddings, grad checkpointing, callbacks, offload fixes
- Reuse Timesteps/TimestepEmbedding; drop unused cfg_embedder / y_embedding / model_max_length
- Rename Mlp/PatchEmbed/CaptionEmbedder with SanaWM prefix
- Support gradient checkpointing in SanaWMTransformer3DModel
- Keep complex RoPE freqs out of buffers so Module.to(dtype) doesn't drop the imaginary part
- Disable autocast around camera geometry (torch.cross)
- Pipeline: callback_on_step_end, Literal output_type, _execution_device, maybe_free_model_hooks
- Fix output docstring, bump copyright to 2026
- Port tests to new pytest mixins, add refiner transformer tests
* [SANA-WM] Drop cam_utils and registration tests
* [SANA-WM] Add return type annotations to forward methods
---------
Co-authored-by: HaoyiZhu <hyizhu1108@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
Co-authored-by: YiYi Xu <yixu310@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
|
||
Steven Liu
|
122b1e11fd |
[docs] Community methods (#14973)
* freeu * cachedit |
||
|
Sayak Paul
|
bc5e3bd9c5 |
Kandinsky6 (#14949)
* implement Kandinsky 6 - add TI2VA and SR pipelines - additonally implement PiflowScheduler and MMAudioVAE * update docs for Kandinsky 6 * remove redundant bigvgan code and manual checkpoint loading inside model code * add missing _no_split_modules * use attention processor in SR * fix unguarded dependencies - removed: einops, pydantic - lazy: torchvision, av, librosa * remove dead manual weight loading code; flattent MMAudioVAE structure * move attention processors to transformer files * remove duplicate code in K6 latent_upscaler * address 1st batch of comments * patch transformer k6 * patch SR * patch pipeline style * replace TimestepEmbedding and fix batching * refactor ti2va dit and pipeline * refactor piflow scheduler * refactor SR * patch sr transformer autocast * major refactor * untrack notebooks directory Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * patch mmaudio VAE * patch mmaudio VAE * fix inits * fix flex attn * refactoring * remove __all__ * separate MMAudio VAE and Vocoder * fix naming * refactor checkpointing, magcache, transformer, sr vae for K6 * refactor * address apply_scale_shift_norm problem * update docs * apply style chanes * address review feedback on transformer, SR VAE, and flex attention Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * address comments * fix critical bugs * Revert checkpoint-breaking module renames from |
||
naykun
|
da1d3829cf |
Configure fixed sampling sigmas on QwenImage21Pipeline (#14950)
* feat(qwenimage21): configure sampling sigmas on the pipeline * docs(qwenimage21): clarify default and runtime sampling sigmas |
||
|
Sayak Paul
|
cff9dafbbf |
[modular] Echo team joy future academy jd echo by @Gelercatty (#14941)
* Add Echo modular pipeline * Address Echo modular pipeline review feedback * Inline Echo input list normalization * Align Echo latent boundaries and batch expansion * Polish Echo documentation per review * Load official Echo checkpoint and separate block tests * consolidate testing. * formatting --------- Co-authored-by: Yanwen Ma <gelercat117@gmail.com> |
||
Dhruv Nair
|
578c9b2c66 |
Add single file support for Krea 2 (#14914)
update |
||
|
Dhruv Nair
|
acbabca385 |
Consolidate torch device backend dispatch (#14792)
* introduce TorchDeviceBackend * update --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
|
Steven Liu
|
4295ee3ec5 |
[docs] Adapters (#14842)
* lora * legacy adapters * ip adapter * controlnet * feedback |
||
|
Dhruv Nair
|
f2edf6ea30 |
Add single file support for Qwen 2.1 (#14836)
update |
||
yzhautouskay
|
344d6e7300 |
Add SeaCache support for Cosmos3 pipelines (#14663)
* Add optional FP32 sampling state for Cosmos3 * Add cache execution contexts to Cosmos3 pipelines * Add SeaCache support for Cosmos3 * Cap consecutive SeaCache reuses * Align SeaCache defaults with Cosmos3 inference * Enable SeaCache by default for Cosmos3 * Document SeaCache for Cosmos3 * Use FP32 sampling state by default for Cosmos3 * Fix SeaCache reconfiguration, transfer state isolation, and FP32 sampling * Remove SeaCache stats and ablation instrumentation * Move SeaCache to transformer-level and disable by default * Align SeaCache and tests with model-level cache patterns * Make Cosmos3 sampling state always FP32 * Add pipe definition to the docs snippet * Remove redundant hook registry cache resets * Use raw vision latents for SeaCache indicators * Make Cosmos3 SeaCache safe for regional compilation * Fix SeaCache with Cosmos3 model parallelism * Support single-stream models in SeaCache * Add Wan T2V support to SeaCache * pass denoising step info to cache hooks via cache_context Replaces SeaCache's pipeline-state callbacks: the pipeline attaches the loop coordinates (step_index, sigma, num_inference_steps) to the denoiser call with `cache_context(name, ...)`, carried by a frozen CacheContext dataclass and stored on each stateful hook's StateManager (the registry no longer holds context). Co-Authored-By: Dhruv Nair <dhruv.nair@gmail.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Fix Wan cache metadata and Cosmos modular cleanup Pass denoising step metadata to Wan T2V conditional and unconditional cache contexts. Restore stateful cache reset after Cosmos modular pipeline calls, restore the required Literal import, and update regression tests and documentation. * Refactor Cosmos3 SeaCache decoder hook * Apply CI formatting updates * Fix distilled system prompt fallback * Preserve Cosmos3 fp32 sampling state --------- Co-authored-by: yiyixuxu <yixu310@gmail.com> Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
|
naykun
|
6256aa7666 |
Add Qwen-Image 2.1 (#14804)
* feat: add Qwen-Image 2.1 pipeline with block-causal attention and KV cache New classes: - QwenImage21Transformer2DModel: single-stream transformer with block-causal attention and t=0 modulation for the text and condition-image prefix - AutoencoderKLQwenImage21: 64-channel VAE (z_dim=64, decoder_base_dim=144) - QwenImage21Pipeline: text-to-image and image-conditioned generation Attention: - Block-causal: the joint text/image sequence is causal while each image block (condition and target) stays internally bidirectional. Built as a compiled flex_attention BlockMask, which keeps the score matrix block-sparse and makes 2048x2048 feasible. - flex_attention is optional. Without it the mask is approximated by a two-pass prefill (prefix causally, then the target image over the cached prefix). Exact for text and for the target image, approximate for condition images. KV cache: - The text and condition-image prefix is modulated from t=0, so its activations do not change across denoising steps and its keys and values are cached after the first step. Later steps only recompute the target image's tokens. Also: separate text-to-image and image-conditioned prompt templates with image-pad token downsampling, and plain classifier-free guidance. Includes model tests, docs, and full registration. * fix style * refactor: address PR review feedback for Qwen-Image 2.1 - Remove `causal_block` config flag (always on for released checkpoint) - Split attention into QwenImage21FlexAttnProcessor and QwenImage21SDPAAttnProcessor - Replace two-pass approximate SDPA prefill with exact multi-pass prefill (each image block gets bidirectional attention, text gets causal mask) - Refactor KV cache to QwenImage21KVCache/QwenImage21KVLayerCache classes with explicit kv_cache_mode="extract"/"cached"/"extend" - Lazy-compile flex_attention on first use (fixes OOM on uncompiled path) - Fix edit pipeline: remove broken _downsample_image_pad_tokens, add mm_token_type_ids for transformers 5.x, auto-convert RGB to RGBA - Use @apply_lora_scale decorator, extract _IMG_TOKENS_PER_SLOT constant - Add # Copied from markers for retrieve_latents and _encode_vae_image - Fix mutable default feat_idx=[0] in all 8 VAE forward methods - Update docs: add usage snippet, remove stale causal_block references - Delete examples/qwenimage21/ (snippet moved to docs) * fix: KV cache pinned the whole prefill sequence at batch size 1 The "extract" branch stored the prefix as `key[:, cache_write_slice].contiguous()`. At batch size 1 that slice already counts as contiguous, because PyTorch ignores size-1 dimensions in the check, so `contiguous()` returned the same view and the cache pinned the full prefill K/V for every step of the denoising loop: 8.0 GiB of resident memory at 2048x2048 across 32 layers. At batch size 2 and above the slice is not contiguous, the copy happens, and the leak disappears, so no test caught it. Store a `clone()` instead, and assert in the tests that the cached prefix owns its storage. Measured at 2048x2048 on one H100, bf16, batch 1, 20 steps: peak memory for a full pipeline call drops from 64.5 GiB to 56.5 GiB. * move the block-causal segmentation into the SDPA processor Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WpU4nuugzCw8T1c4tL6N2A * refactor: let the processor pick the block-causal path Builds on the previous commit by @yiyixuxu, which moved the segmentation into the processor. - Drop the internal `torch.compile(flex_attention)`. The convention is that the caller compiles, so `QwenImage21FlexAttnProcessor` goes through `dispatch_attention_fn(..., backend="flex")` and warns once when it finds an uncompiled `flex_attention` — that falls back to a dense fp32 score matrix and runs out of memory at high resolution. - `_attention_backend` is `None` on the flex processor. It is what `set_attention_backend()` sets and only applies to the cached decode steps; the prefill needs the flex kernel for its `BlockMask` and is not configurable. The processor raises from `__init__` when flex_attention is unavailable. - Pad the sequence axis directly instead of transposing around `F.pad`, so the padded tensors stay contiguous, which the compiled flex kernel requires. - Rename `QwenImage21SDPAAttnProcessor` to `QwenImage21AttnProcessor` and make it the default. Once compiled, flex is 1.5% faster end to end at 2048x2048 (31.8s vs 32.3s over 20 steps) and 3.8% faster with two condition images, in exchange for 37s of compilation; uncompiled it cannot render 2048x2048 at all. A default that only works when the caller compiles is the wrong trade, so flex is documented as the opt-in path instead. - Derive the prefix segment boundaries once per forward rather than once per layer. They only depend on `image_ids` and `prefix_len`, so the per-layer version repeated the same `tolist()` device sync 32 times. `forward` passes down whichever representation the installed processors read, and builds neither for a processor that does not need it. - Replace `test_non_flex_backend_rejected_when_causal`, which no longer describes the intended behaviour, with a check that `set_attention_backend` only affects decode. Measured on one H100, bf16, batch 1, 20 steps at 2048x2048: the default path runs out of the box in 32.7s at 56.5 GiB peak, and `set_attn_processor(QwenImage21AttnProcessor())` now works at all — it used to hand the flex `BlockMask` to a non-flex kernel and raise. * fix: VAE class defaults did not describe the released model `scale_factor_spatial` was 8 while the encoder applies four spatial downsamples: `encode` takes a 1024x1024 image to a (1, 64, 1, 64, 64) latent, so the ratio is 16. Every tile-to-latent conversion divides by it, so tiling silently produced a wrong-shaped latent — a 2048x2048 encode came out as 168x168 instead of 128x128. The in/out channel defaults were 3 while this VAE takes four channels, so the class could not be instantiated from its own defaults. Add the model test file that was missing, covering the ratio against the architecture and the shape of a tiled encode. Tile values are not compared: each tile starts the causal convolution feature cache fresh, which is a property of the tiling implementation rather than of these defaults. * address review feedback: copies, pipeline, docs VAE: - Mark the classes that are byte-identical to their Wan originals with `# Copied from` (`DupUp3D`, `WanUpsample`, `WanRMS_norm`, `WanAttentionBlock`, `patchify`, `unpatchify`). Adopting the upstream `RMS_norm.forward` in the process also picks up a fix we had missed: it normalizes in fp32 for fp16/bf16/fp8 inputs. - Drop the `non_linearity` argument, which was always "silu", from the blocks that take it, and validate `AvgDown3D`'s channel divisibility with a `ValueError` before the fields are assigned rather than with an `assert` after. Pipeline: - Take `calculate_dimensions` verbatim from the edit pipeline so it can carry a `# Copied from`, which is what fixes `check_repository_consistency`: the marker on `_encode_vae_image` was one blank line out of sync. - Move the `QwenImage21KVCache` import to the top of the module. Docs: - Apply @stevhliu's suggestions. `models.autoencoders.vae.AutoencoderKLOutput` does not exist, so that autodoc reference was broken; the module is `autoencoder_kl`. - Describe the two attention processors instead of the old "with and without the flex backend" split, and add the snippet for opting into flex, which has to be compiled. `make style` also reflowed a few docstrings from the previous commit. * address review feedback: validate before doing work The `kv_cache` checks in the transformer's `forward` ran after the input projections, the joint sequence build, the rotary embeddings and the modulation, so a bad `kv_cache_mode` was only reported once that work had been done. They now run first. Following the same point through the pipeline turned up a check that could never fire: `check_inputs` warns when `height` and `width` are not divisible by `vae_scale_factor * 2`, but the rounding happened before the call, so the values it saw were always divisible. It now runs before the rounding, and `output_resolution=1000` warns and yields 992x992. * fix: num_images_per_prompt > 1 raised in the pipeline `encode_prompt` expands `prompt_embeds` and its mask to `batch_size * num_images_per_prompt`, but returns `image_pad_mask` unexpanded, and the target slots appended to that mask were sized from `latents`, which is expanded. Any call with `num_images_per_prompt > 1` therefore died in the concatenation: RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 1 but got size 2 for tensor number 1 in the list. Size the slots from each mask's own batch instead. The transformer reads the layout from row 0 because samples share it, so the mask does not need expanding. Verified end to end: `num_images_per_prompt=2`, a list of two prompts, both together, and each of those with classifier-free guidance and with a condition image. * Use the recommended sampling defaults: 40 steps, no guidance Qwen-Image 2.1 is meant to be sampled in 40 steps without classifier-free guidance, so `num_inference_steps` defaults to 40 and `true_cfg_scale` to 1.0. The other QwenImage pipelines default to 50 and 4.0, which is why this differs from its siblings. It also removes a warning from every default call: `true_cfg_scale=4.0` with no negative prompt took the "guidance is not enabled" branch. Passing a `negative_prompt` without raising `true_cfg_scale` still warns, which is the case worth warning about. The docs and the example docstring rely on the defaults now instead of passing a step count, and the docs state the recommendation. * docs: complete the forward and __call__ docstrings `utils/check_forward_call_docstrings.py` on main checks that every argument in a forward/__call__ signature has a docstring entry and that a non-None return type has a Returns section. It landed after this branch's base, so it only started running here once the copy check stopped failing ahead of it. Add the missing entries: `sample_posterior` and `generator` plus Returns on the VAE's forward, `attention_kwargs` and `return_dict` plus Returns on the transformer's, and the four embedding arguments and `callback_on_step_end_tensor_inputs` on the pipeline's __call__. * add pipeline tests for qwenimage 2.1 (#6) * add more copied froms (#5) * docs: describe multiple condition images, and inline the flex warning The pipeline page now has a section on passing several condition images, which is where @sayakpaul asked for it, and the flex section emphasises that the processor wants a compiled model. `_warn_if_flex_attention_is_uncompiled()` is inlined at its only call site, as requested. A class-level flag keeps it to one warning per process: the default processor is constructed per attention module, so 32 instances would otherwise each warn, and the logger has no `warning_once`. * fix-copies: drop the AvgDown3D marker `AvgDown3D` validates its channel divisibility with a `ValueError` before assigning its fields, where Wan still asserts after, so the copy is not consistent and `check_copies` fails on it. The other ten markers from #5 are fine and stay. * fix: feed the transformer the pre-norm text hidden state The transformer was trained on the last decoder layer's output of the text encoder, before the encoder's final RMSNorm. Up to transformers 4.x that is what `hidden_states[-1]` holds. From transformers 5.0 the output capturing ties that entry to `last_hidden_state`, so it comes back normalized instead, and the transformer reads something a third of the way off — visible first as garbled text in the rendered image. Neutralize the final norm for the encoder call with a forward hook that returns the module's input, so `hidden_states[-1]` is the layer output on either version. Nothing else is touched: the weights stay untouched, which matters because they are on `meta` under offloading, and no version check is needed. On the released checkpoint the prompt embeddings now match the pre-norm value exactly and the rendered image is pixel-identical to it, where before the whole image shifted by 5.35/255 on average. * fix: match the checkpoint's text conditioning, and repair the unexercised paths Aligned with the text encoder the checkpoint was trained with: - The image marker is `<image1>`, `<image2>`, … as in training, not `Picture 1: `. The two tokenize to different lengths (4 tokens against 5), so the conditioning the transformer read was a sequence that never occurred in training, and the rendered image moves 4.02/255 on average. With the marker corrected the output is pixel-identical to the training template. This also retires the `random.choice` over four spellings of that word, which ran on the global RNG and left image-conditioned generation irreproducible from `generator`. - Condition images reach the vision encoder with their alpha composited over white, as in training. The VAE still reads all four channels. - The processor pads on the left, as in training. The joint sequence is re-padded on the right either way, so this only changes the positions the encoder itself sees for a batch of prompts of different lengths. - An empty prompt becomes a space. Qwen has no bos token, so the encoder would otherwise have nothing to read. Prompt embeddings: - The 2D prompt mask was repeated with `repeat(1, n, 1)`, which prepends an axis and tiles the rows where the 3D embeddings interleave theirs. With more than one prompt and more than one image per prompt, each sample was denoised against another prompt's padding. - Supplying `prompt_embeds` raised: `image_pad_mask` only comes from `encode_prompt`'s own encoding, and the target slots appended to it were sized from `latents`. It is synthesized for text embeddings now, and required when the embeddings cover condition images. Supplying embeddings without a mask raised too. - `has_neg_prompt` no longer requires `negative_prompt_embeds_mask`, so a caller who passes `encode_prompt`'s own output back in keeps guidance. An unpadded prompt returns `None` for the mask, the pattern `pipeline_qwenimage.py` uses, since a mask that carries no information costs the backends that reject one. Condition images: - A tensor or ndarray `image` raised on `image.size`. They are normalized to PIL up front; a latents tensor is rejected with a message, because the text encoder has to see the image, and a per-prompt nested list with another, because one flat set applies to the whole batch. That also removes the half-wired path where a latents tensor was silently dropped from both the prompt and the latents. - A list of prompts with a condition image raised a bare `StopIteration`: every prompt's template repeats the placeholders, but the processor was handed one set of images. Denoising and validation: - Interrupting on the first step used to `continue`, skipping the step that prefills the KV cache and leaving the next one to decode from an empty one. It breaks out now. - `kv_cache_mode` without a `kv_cache` is rejected instead of failing later on a shape. - `prepare_latents` checks the generator list before spending a VAE encode per image. - The VAE's image convolution names its limitation instead of asserting: it folds the single frame away and has no temporal context, so it cannot take a feature cache. Removed: `_downsample_image_pad_tokens`, `_max_length`, the `_drop_idx_ti2i` alias, and the KV cache's `is_populated` and `clear()`, none of which anything reached. Training collapses each run of `<|image_pad|>` to one token; skipping that is equivalent here, since those positions are overwritten by the VAE latents either way. The text encoder hook now points at huggingface/transformers#48087, which lets the config untie `hidden_states[-1]` from 5.18 and will make the hook unnecessary. * fix callback test (#2) * Add a TODO at the text encoder hook It can be replaced with `tie_last_hidden_states=False` in the text encoder's config once huggingface/transformers#48087 ships in a stable release. --------- Co-authored-by: yiyixuxu <yixu310@gmail.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
陈志谦
|
7221eef457 |
docs: pool eight small doc fixes into one PR (#14757)
* docs: fix latent -> latents in LTX latent upsampler docstrings __call__ takes 'latents' (the docstring said 'latent', and its own description says 'Input latents to normalize'). * docs: fix 'promt' typo in kandinsky5 default prompt templates The default system template sent to the Qwen text encoder misspelled 'prompt engineer' as 'promt engineer' (4 pipelines). Note: this changes the token sequence of the default template, so default outputs will differ numerically from before / from the upstream Kandinsky5 reference. If exact parity with the reference repo is preferred, the reference itself would need the same fix. * docs: fix docstring parameter names that do not match signatures - flux controlnet encode_prompt documented a clip_skip parameter that does not exist (copy-paste from SD-style docs; flux has no CLIP layer-skipping) - hunyuan_video1_5 pipelines documented num_images_per_prompt while the parameter is num_videos_per_prompt - kandinsky5 _encode_prompt_qwen/_encode_prompt_clip documented num_videos_per_prompt, which neither method accepts * docs(zh): fix English anchors pointing at translated Chinese headings speed-memory-optims.md linked ./memory#model-offloading and #group-offloading while the zh memory page translates those headings (模型卸载 / 组卸载); using_peft_for_inference.md linked #hotswapping while the zh heading is 热切换. Repointed at the Chinese slugs. * docs: fix PAG anchor and the StableDiffusionPipelineSafe row in the SD overview - pag.md linked #pagappliedlayers; the heading '### pag_applied_layers' keeps its underscores (#pag_applied_layers) - the SD overview table linked ./stable_diffusion_safe, a doc page that no longer exists anywhere under docs/source; the row now names the deprecated pipeline as plain text instead of a dead link * docs: fix dead Masked Im2Im TOC anchors in community pipelines README The TOC rows linked #stable-diffusion-masked-im2im / #stable-diffusion-xl-masked-im2im; the actual headings are 'Masked Im2Im Stable Diffusion Pipeline' and 'Masked Im2Im Stable Diffusion Pipeline XL' (lines 3184/3219). * docs(examples): drop duplicated 'with' in T5/Gemma sequence-length help strings 16 example scripts' --max_sequence_length help read 'to use with with the T5 text encoder' (or the Gemma variant); surfaced in --help output. * fix(links): keep the pag_applied_layers anchor in doc-builder form doc-builder slugifies headings by dropping underscores, so the heading '### pag_applied_layers' anchors as #pagappliedlayers — the form the check-links CI expects. GitHub-style underscores broke it. Signed-off-by: simpleqt <89645338+simpleqt@users.noreply.github.com> * fix(kandinsky5): recompute prompt_template_encode_start_idx after the promt fix Correcting 'promt' -> 'prompt' in the system templates shifts each template's token count by one (verified with the Qwen2.5-VL tokenizer: 41->40, 55->54, 129->128, 129->128). Without this, the hardcoded start indices slice the encoded prompt one token off. --------- Signed-off-by: simpleqt <89645338+simpleqt@users.noreply.github.com> |
||
Yiming Zhao
|
759164b7ad |
[Cosmos3] Mixed W8A8/W8A16 denoising for ModelOpt FP8 checkpoints (#14664)
* Add opt-in mixed W8A8/W8A16 denoising for Cosmos3 ModelOpt FP8 checkpoints. Keep native ModelOpt GEMM on middle steps and dequant-linear W8A16 on the first/last steps so CFG cond/uncond share one precision per scheduler step. * Read Cosmos3 mixed-precision schedule from the checkpoint policy. Enable first/last W8A16 only when transformer/config.json declares diffusion_step_policy, so distilled FP8 stays native W8A8 instead of inheriting a hardcoded 3+3 window. * Harden Cosmos3 mixed-precision loading and document official Hub fp8 schedules. Read the checkpoint runtime policy from on-disk transformer/config.json when the live ModelOpt config omits it, fail closed on incomplete policies, and allow FP32 activations on the W8A16 path. * Address review: simplify FP8 mixed docs, no-op non-ModelOpt backends, drop unit tests. W8A8/W8A16 is explained without listing call-site overrides; TorchAO and other quantizers keep their native forwards; focused tests are removed until Hub fp8 usage is clearer. * Document FP8 mixed default vs none trade-offs without ModelOpt restore boilerplate. Keep serialized restore in the ModelOpt guide and spell out that all-W8A8 is faster but can flicker on multi-step video. * Apply style fixes * Regenerate Cosmos modular auto docstrings for mixed-precision inputs. --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
Alex Artyomov
|
82e309a2b1 |
Add LTX-2.5 DFR pipeline (keyframe slots, spatial detailing, tiled temporal rounds) (#14567)
* Add LTX-2.5 DFR pipeline with generated keyframe slots Ports DFRPipeline from the Lightricks reference. Stage 1 generates video plus single-pixel-frame keyframe slots at a fraction of the requested resolution on a VAE-aligned segment grid; both are spatially latent-upsampled and stage 2 re-denoises at twice that resolution with the slots re-attached and an optional spatial detailing IC-LoRA active for that stage only. Optional temporal x2/x4 refine rounds tile the canvas at shared keyframes and densify with ancestral Euler. With spatial_upscalings=2 a full-resolution detailing epilogue follows the rounds. The transformer already stored keyframes_abs_pos_embedding for load/save; this wires it into the forward through a new video_keyframes_mask argument, which only a keyframes-aware pipeline passes, so other pipelines are unaffected. The epilogue denoises the whole canvas in one loop and tiles the transformer call inside it, so every Euler step steps a canvas whose tiles have already agreed on their overlaps. Spatial tiles blend under a trapezoidal mask, since neither side of a height or width border holds a known answer. Temporal tiles are cut on the keyframe seams the last refine round stitched on: both windows reproduce a shared keyframe there, so the later one drops its run-up under a rectangular mask rather than averaging it. Conditionings are attached once on the whole canvas and filtered per tile at the token level, and a keyframe two windows share is one token they both read. The epilogue is handed its keyframes rather than asked to generate them. Each carry plane is decoded on its own -- the VAE is causal, so a stacked decode would bleed neighbours -- then Lanczos-stretched x2 in RGB and encoded again at the output resolution, and pinned fully clean. Only the video latent is spatially upsampled. Conditioning fps is snapped to 60 above 30 rather than merely capped there, at every stage. RoPE time is pixel_frame / fps, and the transformer is trained around 24/25/30 and 60; a temporal round taking 24 fps to 48 lands between those, and it shows as stutter at the latent borders. Playback fps is unchanged, so 24 fps with one round still ships 48 fps. A condition's index is read on the canvas num_frames asks for, and the moment it names is carried onto each refine round's longer canvas by scaling its pixel position by 2**round. The scaled position does not generally land on a latent boundary, so it travels as a pixel index rather than through the public latent index; a keyframe conditioning is appended as extra tokens instead of being spliced into the base grid, so it does not need to. height and width must be divisible by 2**spatial_upscalings times the VAE's spatial compression ratio, which makes 4K 3840x2176 rather than 3840x2160. That rule is checked ahead of the looser one every LTX-2 pipeline applies, so the error names the divisor a DFR caller actually has to satisfy. Four details are easy to get wrong, and each is covered by a test after showing up as a visible seam at a tile handover: - The ancestral step injects noise into every token, so the conditioning blend has to be re-applied afterwards. Skipping it lets the strength-0.95 anchor keyframes erode over the schedule, and those anchors are the only thing pinning adjacent tiles onto the same content. - Velocity is converted to x0 with each token's own noise level, not the scalar schedule sigma: a token held at strength s sits at (1 - s) * sigma. - Each tile draws its ancestral noise from a generator seeded seed + 1000 * round + tile, kept separate from the main generator so the draws do not consume state the next tile's initial noising reads. - Two tiles invent the slot that falls in the later one's dropped lead-in. The stitch keeps the earlier tile's frames there, so the earlier tile's copy is the one the canvas holds, and the one the next round must anchor on. * Split the DFR pipeline into composable stages Addresses the review on #14567: use the same compose pattern as the other LTX two-stage pipelines, get ancestral Euler from an existing scheduler, and put the temporal rounds in their own pipeline so the schedule is not switched mid-call. `LTX2DFRPipeline.__call__` is now one denoise pass at `height` x `width`. Callers compose stage 1, `LTX2LatentUpsamplePipeline`, stage 2 and each temporal round, and a documented recipe is the copy-paste 1080p path. The recipe knobs (`spatial_upscalings`, `temporal_upscalings`, `detailing_lora_adapter_name`) and the required upsampler components are gone; `height`/`width` are this pass, not the final output. `ancestral_euler_step` is replaced by `LTXEulerAncestralRFScheduler.step` plus a re-application of the conditioning blend, which ancestral noise would otherwise erode on the strength-0.95 seam anchors. The new `LTX2DFRTemporalRefinePipeline` owns that scheduler and one round; stage 1, stage 2 and the epilogue stay on `FlowMatchEulerDiscreteScheduler`. It refuses any other scheduler rather than silently taking a deterministic step and returning a softer canvas. Pack/unpack, `prepare_latents`, `denoise` and `encode_conditions` move to `LTX2DFRCoreMixin` so neither pipeline subclasses the other. Public latents are raw on both sides of every boundary, `output_type="latent"` returns the untrimmed canvas so a slot on the pad is not dropped, and `trim_canvas` does the trim before decode. `LTX2DFRPipelineOutput` carries `keyframes` and `keyframe_positions`, which cannot be re-derived after a round. Verified against the pre-split implementation: bit-exact on dummy components in fp32 and bf16, with and without the IC-LoRA reference, over one and two rounds; and within one bf16 ulp on the real checkpoint, where the only difference is that the split normalizes upsampled latents in fp32 rather than bf16. Also in this pass: - `__call__` takes `video_tiles` (the `epilogue_tiles` layout) instead of a resolved token plan. Resolving one needs the RoPE coordinates that only exist once `prepare_latents` has run, so a caller could not build the plan at all. - `rebuild_epilogue_keyframes` is public and takes and returns raw latents. The composed epilogue needs it, so it was public API in practice while named private, and its normalized return forced callers into `_denormalize_latents`. - Drop prompt enhancement and `num_videos_per_prompt` from the temporal pipeline. Enhancement belongs to stage 1 -- re-running it would denoise the canvas under a different prompt than the one that generated it -- and the batch is set by the incoming latent canvas, so `num_videos_per_prompt > 1` only ever raised. - Fix the docs recipe: `requested_frames` counted latent frames where `trim_canvas` wants pixel frames, truncating a 241-frame render to 25; the three pipelines share components, so place them together instead of offloading one and leaving `temporal_latent_upsampler` off the device; and the detailing IC-LoRA is applied at 0.5, the strength the reference hardcodes. - Pass `crf=0` on the conditions in the refine-round test. The default CRF sends the image through H.264 re-compression, which needs PyAV, so the test failed on any environment without it while testing nothing about re-compression. * Drop LTX2DFRCoreMixin in favor of # Copied from Each DFR pipeline now owns pack/unpack, prepare_latents, and denoise so the classes stay self-contained. Shared helpers copy from LTX2Pipeline / LTX2ConditionPipeline, and DFR-specific methods copy from LTX2DFRPipeline. dfr_core.py keeps only the constants and canvas helpers. * inline _finalize_output Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * remove dfr_core.py Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * make return_dict=False return a tuple of 4, consistent with diffusers convention; update docs and pipeline output accordingly Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * remove the dfr_layout test suite Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Steven Liu
|
7a8c9da2a9 |
[CI] Repair broken links (#14627)
fix Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
dg845
|
c969cf2f7e |
Move LTX2Guidance Guider to guiders/ Folder (#14558)
* Move LTX2Guidance from /modular_pipelines/ltx2/ to guiders/ to allow LTX-2 guiders to be correctly loaded from checkpoint * Add regression test for LTX-2 guiders declared as from_pretrained components Covers the round trip this move fixes: a checkpoint that declares `guider` and `audio_guider` in `modular_model_index.json` must record a resolvable (library, class_name) pair and reload its non-default guidance scales. The existing `test_modular_index_consistency` skips components without a `pretrained_model_name_or_path`, so `from_config` guiders were uncovered. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Migrate LTX2Guidance docs to guider docs after move * Improve LTX2Guidance docs and add arg docstrings --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
kaixuanliu
|
b9bc22bd57 |
make examples in docs device agnostic (#14414)
* docs: annotate hardcoded cuda device with alternatives in English docs * docs: annotate hardcoded device_map="cuda" with alternatives * update Signed-off-by: kaixuanliu <kaixuan.liu@intel.com> --------- Signed-off-by: kaixuanliu <kaixuan.liu@intel.com> |
||
Shivam Shrirao
|
06e0f2a81c |
Fibo Edit: multi-reference conditioning and batching (#14566)
* Fibo Edit: multi-reference conditioning, batching, timesteps * docs: simplify Fibo Edit page, fix EXAMPLE_DOC_STRING to the VLM prompt flow * docs: point Fibo Edit at the Gemini prompt-to-JSON block and the 1.5 base/turbo ids briaai/FIBO-edit-prompt-to-JSON is retired in favor of briaai/FIBO-edit-gemini-prompt-to-JSON, which handles multiple reference images and masks. Also corrects the checkpoint ids to their canonical briaai/Fibo-Edit-1.5-base and briaai/Fibo-Edit-1.5-turbo spellings. * apply review feedback on reference handling and the attention mask - collapse the mask branch to `if not attention_mask.all():` in both Fibo pipelines - encode references via `self.image_processor.preprocess(...)`, dropping the manual numpy normalization and the `_vae_safe_size` helper - normalize `image` into a reference list once in `__call__`, pass it to `check_inputs` * make style * make style |
||
|
dg845
|
9841a24724 |
[docs] LTX-2.5 Modular Pipeline Docs (#14450)
* LTX-2.5 modular pipeline init docs * Update LTX-2.5 modular docs now that the modular pipeline automatic downloads the Gemma 4 prompt enhancer and processor * fix some modular docs typos * fix more typos * Update docs/source/en/api/pipelines/ltx2.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Remove unused transformers imports from LTX-2.5 modular examples --------- Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> |
||
iridescentWen
|
2f7e0154a9 |
docs: fix 14 typos in docs, comments and docstrings (#14541)
docs: fix typos in docs, comments and docstrings
Prose, comments and docstrings only. No executable line changed.
docs/:
- optimization/memory.md: maxmium -> maximum; heigh -> height (in the
channels_last layout tuple, which read "(batch size, heigh, width,
channels)")
- api/pipelines/bria_fibo.md: proffesional -> professional
- api/pipelines/wan.md: involed -> involved
src/ comments and docstrings:
- guiders/magnitude_aware_guidance.py: supression -> suppression
- models/attention.py: spliting -> splitting
- models/attention_dispatch.py: abritrary -> arbitrary (x2)
- models/transformers/transformer_ltx2.py: timstamps -> timestamps (x2)
- pipelines/ltx2/pipeline_ltx2{,_image2video}.py: corrct -> correct
- pipelines/deprecated/unidiffuser/modeling_uvit.py: tbe -> the
- pipelines/visualcloze/pipeline_visualcloze_{combined,generation}.py:
"whe they are lists" -> "when they are lists" in a user-facing
ValueError message
Deliberately not touched:
- pipeline_kandinsky*.py "promt"/"scren" live inside prompt_template,
which is fed to the text encoder verbatim. Changing them would change
generation output.
- ANE (Apple Neural Engine), MoT (Mixture-of-Transformers): initialisms.
- "racoon": appears in prompt examples whose reference images were
generated with that exact prompt.
- "Shuting Wang": a paper author's name.
|
||
|
Steven Liu
|
0eff7f0664 |
[docs] Clean up links (#14535)
* docs * more cleaning |
||
Atharva Joshi
|
4e0466f3e5 |
fix: image preprocessing for cosmos3 (#14519)
fix: preserve Cosmos3 conditioning image aspect ratio |
||
Jingya HUANG
|
7d2e86a85a |
[core] Support tensor parallelism for model inference (CUDA, Neuron) (#13718)
* draft:add neuron as a legit backend
* feat: neuron-specific changes in the pipeline
* tests: eager tests
* draft: start with tp for flux2
* fix: style
* fix:apr_02 beta
* feat:add wan
* fix:pixart
* fix: rewrite flux swiglu activation to avoid gather op in neuron IR
* test: pixart compile mode on neuron
* cleanup & fix style
* merge: another change
* review: cleanup+suggestions
* fix: CIs style
* tests: add test units for tp
* fix: in case of text-encoder(s) on CPU
* review:cleanup+add test
* fix: style
* doc: remove it for now
* Add from_single_file support for SkyReelsV2 and ChronoEdit transformers (#13946)
SkyReels-V2 and ChronoEdit are both built on Wan, and their transformers have
the same keys as WanTransformer3DModel, so they reuse
convert_wan_transformer_to_diffusers (like WanVACE / WanAnimate). This lets the
community GGUF builds load directly.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* multi-GPU VAE Fix for Cosmos 3 (#13924)
fix(cosmos3): pin VAE latent norm buffers to encode output device
Under sharded placement (device_map="balanced"), vae.encode() runs on the
VAE's own device while the mean/inv_std buffers were pinned to x.device,
causing a cross-device RuntimeError. Compute raw_mu first, then pin the
normalization buffers to its device so all tensors share one device.
Co-authored-by: Atharva Joshi <atjoshi@smc521ge-0036.ipp2a2.colossus.nvidia.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* docs: fix repeated word typo in set_timesteps docstring (#13876)
* docs: fix repeated word typo in set_timesteps docstring
Removed the duplicate word "schedule" from the docstring for the sigmas argument in EulerDiscreteScheduler.set_timesteps.
* Update scheduling_euler_discrete.py
* Apply style fixes
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* clean some stuff to simplify code.
* clean more to remove permutation related shenanigans.
* revert: put torch.chunk back
* Update docs/source/en/training/distributed_inference.md
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* Address Sayak's TP review nits (docstrings + stale comments)
- _modeling_parallel: fix stale comment claiming attention processors read
tp_degree at runtime (no longer true after the PackedColwise/Rowwise
refactor; processors are TP-agnostic via head_dim).
- tensor_parallel: single backticks instead of double in docstrings, add
input/output examples to _blocks_to_block_sizes and _resolve_tp_plan, and
drop the stale "applies the Flux2 fused-weight permutations" line from
apply_tensor_parallel (permuters were removed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix stale permuter reference in Flux2ParallelSelfAttnProcessor comment
The proportional QKV/MLP slice is now guaranteed by PackedColwiseParallel
block-by-block sharding, not the removed fused-weight permuter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Derive TP backend inside apply_tensor_parallel from the mesh device type
Drops the `backend` parameter: apply_tensor_parallel already holds the TP
mesh, so it derives neuron-vs-default from `config._mesh.device_type` itself.
The caller no longer computes it. Documents why torch_device can't be used
(Neuron reports as "cpu" via torch's accelerator API). Removes the now-unused
is_torch_neuronx_available import from modeling_utils.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Simplify TP backend derivation to the mesh device type alone
Drop the redundant is_torch_neuronx_available() guard: a DeviceMesh("neuron", ...)
only exists when running on Neuron, so tp_mesh.device_type is the single source
of truth.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove incorrect "Neuron reports as cpu" rationale from TP backend docstring
Verified on the current native torch_neuronx drop that Neuron does surface as
the torch accelerator (torch._C._get_accelerator().type == "neuron",
get_device() == "neuron"); the old "reports as cpu" claim was stale. The
mesh-device-type derivation is unchanged — it's the device being sharded onto.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drop stale "XLA" labels from Neuron device/runtime comments
Neuron is a native torch backend now (not XLA-routed): remove the "(XLA)"
parenthetical from the randn-on-CPU note in torch_utils and the "XLA runtime"
mention from the Neuron TP test docstring. Behavior unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* make style: reflow tensor_parallel docstrings
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Revert "revert: put torch.chunk back"
This reverts commit
|
||
Buffett Liu
|
360bef8074 |
Feat -- Stable Audio 3 (#14119)
* Stable Audio 3 First Version Commit, require model dict loading on GPUs * Fixing bugs for sa3 VAE * Make quality fix * Fix the unit test failure * Fix SA3 pipeline noise: learned text-padding embedding + no cross-attn mask The diffusers Stable Audio 3 pipeline produced noise instead of music because the cross-attention conditioning was built differently from the reference: - The reference text conditioner uses padding_mode="learned": padded text positions (~245 of 256 for a short prompt) are filled with a trained `padding_embedding`, and the DiT attends to *all* positions (its cross-attention mask is intentionally disabled). Our pipeline instead zeroed padded positions and masked them out, wiping ~95% of the conditioning signal. Changes: - transformer_stable_audio3: add learned `prompt_padding_embedding`; in forward, replace masked text positions with it (in cond_token_dim space, before to_cond_embed) then attend to the full context, matching the reference. - pipeline_stable_audio_3: stop zeroing padded positions (the DiT now handles them); default `silence_padding_duration` to 6.0 (reference headroom default). - convert_..._to_diffusers: convert `conditioner.conditioners.prompt.padding_embedding` into the DiT. - scheduling_ping_pong: pin schedule endpoints (sigmas[0]=1.0, sigmas[-1]=0.0) to match the reference LogSNRShift endpoint preservation. - run_..._inference: coerce float16->float32 on CPU. - tests: update transformer state-dict expectations (523 tensors, add prompt_padding_embedding) and ping-pong sigma endpoint assertions. Verified against the reference with identical noise + identical conditioning: full 8-step ping-pong trajectories agree to 4e-6 (final latent). Removed the dev-only parity scripts (verify_*_dit_parity, verify_*_vae_parity) from the tree. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Style fix * Autopipeline available * Adding document description for testing examples. * Solving fix for serge review bot suggestion * Design review comments fix * CI/CD Fix for ping pong scheduler md file * Fix doc build: remove autodoc for nonexistent StableAudio3EulerSchedulerOutput * Fix doc build: qualify non-exported autodoc references with module paths * Fix details in scheduler and some nits * suggestion fix * fix-copies: propagate prepare_latents dtype fix to audio2audio/inpaint pipelines --------- Co-authored-by: Buffett Liu <jengliu@linkedin.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> |
||
|
iridescentWen
|
59c47446f2 |
docs: fix typos in Kandinsky 5.0 video docs (#14345)
- `qualty` -> `quality` - `infered` -> `inferred` (x3) - `bakend` -> `backend` (x3, in the trailing `# <--- Set attention bakend to Flex` comments; the actual `set_attention_backend` calls were already spelled correctly) - `wothout` -> `without` Prose and comments only, no code or behavior changes. |
||
Kashif Rasul
|
d6bfaa71b8 |
Fix batched DiffusionGemma adaptive stopping (#14386)
* Fix DiffusionGemma stopping * Test batched stopping * Style DiffusionGemma docs * Address DiffusionGemma review * Update DiffusionGemma review --------- Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> |
||
apolinário
|
8c3dcd6da1 |
Add LoRA loading support for MiniMax-H3 (#14408)
* Add LoRA support for MiniMax-H3 * Address review: Copied-from loader, LoRA badge, docstring caveats, drop generic scale test * Move the MiniMax-H3 LoRA tests to tests/lora * Support the remaining published MiniMax-H3 LoRA layouts Three more published H3 LoRAs did not load. Two were key-layout gaps and the third was the reason neither was noticed: a layout that reaches no module at all used to return without an exception. - musubi-tuner writes one flattened `lora_unet_` name per module. It is un-flattened against the H3 module vocabulary, since `qkv_proj`, `token_refiner`, `final_layer` and the output heads carry underscores that are not path separators, and then goes through the existing kohya path. - one producer publishes its own converter's output: diffusers module names with peft's `.default.` infix left in and no component prefix. The infix is dropped and the prefix added. - a state dict that filters to nothing in both partitions now warns instead of loading as a silent no-op, in the wording `load_lora_adapter` uses for the same situation. A fourth loaded, but at the wrong strength. One producer ships no `.alpha` scalars and records the alpha it trained with in the file's own `__metadata__` instead, under `alpha`. Its 8-step turbo LoRA pairs that entry's 8 with rank 128, an effective scale of 0.0625, so synthesizing `alpha == rank` applied the adapter 16x too strong. That entry is now read as the uniform network alpha. Per-module scalars still win when a file carries both, since the converter has already folded them into the weights, and a non-numeric value is warned about and ignored. `__metadata__` only exists on a file, so `_fetch_state_dict` hands it back on request. * Simplify the MiniMax-H3 LoRA comments * Trim the MiniMax-H3 LoRA tests to loading and effectiveness * Reference the alpha convention in the MiniMax-H3 LoRA docstring * Drop a redundant format comment * Test only diffusers-written state dicts * Rely on get_peft_kwargs for alpha-less files * Drop the peft version guards after the main cleanup --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
|
apolinário
|
2da7040be1 |
Add MiniMax Music 3 (#14456)
* Add MiniMax Music 3 transformer, pipeline sub-models, and conversion script
* Add MiniMax Music 3 pipeline, tests, and docs
* Wire example docstring, point diffusers-format references at MiniMaxAI/MiniMax-Music-3
* Use MiniMaxAI/MiniMax-Music3 as the official diffusers weights repo id
* Apply doc-builder style
* Use dtype in examples, demonstrate the structured prompt format
* Use soundfile in examples like the other audio pipelines
* Convert to a modular pipeline and move models under src/diffusers/models
Per review: the conditioner moves to models/condition_embedders, the RVQ depth
decoder and vocoder to models/transformers and models/autoencoders, and the
standard pipeline is replaced by MiniMaxMusic3ModularPipeline (anima-style
blocks, helios-style chunk loop, guider-abstracted CFG with a zeros
unconditional branch). Index configs now reference all components under the
diffusers library. Also applies the transformer review suggestions
(Transformer2DModelOutput, inlined rotary forward, explicit processor default).
* Point modular index loading specs at the Hub repo in the converter
* Apply doc-builder style to the modular blocks docstrings
* Address review-bot comments: overlap comment accuracy, single pipeline-level progress bar, zero-frame duration guard, import ordering, docstring cross-reference
* Support CPU offloading in the AR step: trigger offload hooks before submodule calls (minimax_h3 workaround), clear error when the AR models cannot co-reside
* Docs: PR-install tip and memory-usage section with measured offloading numbers
* Address review comments on the block assembly
Take the individual models rather than `components` in `_generate_depth_codes` and
`_embed_audio_frame`; `num_codebooks` / `audio_vocab_size` come off the depth decoder's
config, so the two pipeline properties that existed only for them are gone.
Assemble the blockset as three top-level children, each owning models and runnable on its
own — generate the semantics once and denoise it with different settings:
semantic_generator MiniMaxMusic3SemanticGenerationStep [tokenize, generate]
denoise MiniMaxMusic3CoreDenoiseStep [prepare_chunks, denoise]
decode MiniMaxMusic3VocoderDecodeStep
`MiniMaxMusic3TextEncoderStep` is renamed `MiniMaxMusic3TokenizeStep` (it only tokenizes)
and the autoregressive leaf `MiniMaxMusic3AutoregressiveStep`, freeing its old name for
the sequential.
Point the tests at `hf-internal-testing/tiny-minimax-music3-modular-pipe`; the previous
fixture repo's component specs pointed at a local path, so nothing loaded off that machine.
Drop the PR-branch install tip from the docs.
20s and 60s generations are bit-identical to before these changes.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: yiyixuxu <yixu310@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
||
|
YiYi Xu
|
3a2f35d4ef |
Wan-Animate-2 (authored by @kelseyee) (#14413)
* support wan-animate-2 --------- Co-authored-by: kelseyee <971704395@qq.com> Co-authored-by: yiyi@huggingface.co <yiyi@huggingface.co> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: sayakpaul <spsayakpaul@gmail.com> |
||
|
Sayak Paul
|
7564fb016d |
Ltx 2.5 (#14447)
* Add LTX-2.4 support to the LTX2 transformer, pipelines, and conversion script LTX-2.4 reuses the existing LTX2 model and pipeline classes. The delta over 2.3 is a small set of additive config flags plus a Gemma 3 -> Gemma 4 text encoder swap, so no new model or pipeline classes are introduced. - transformer_ltx2.py: add `ff_bias`/`audio_ff_bias` (2.4's video FFN drops its bias) and `use_prompt_adaln_single` (toggles timestep-dependent prompt cross-attention modulation; when off, cross-attention K/V becomes timestep-independent and cacheable across denoising steps for a given prompt). Both default to their 2.3 behavior. The flag is read back from `self.config` rather than mirrored onto an instance attribute, and all three new constructor args are documented. - pipeline_ltx2.py / pipeline_ltx2_image2video.py: add an optional `prompt_enhancer` component to both pipelines (previously T2V-only). LTX-2.4's fine-tuned text encoder is conditioning-only, so enhancement uses a separate off-the-shelf google/gemma-4-E2B-it checkpoint with its own message format and decoding recipe -- unlike LTX-2.0/2.3, where one checkpoint serves both roles. `enhance_prompt()` resolves the format and `.generate` kwargs from whichever model is active, and frees a dedicated enhancer from GPU memory right after use (guarded so it does not interfere with accelerate's offload hooks), mirroring the existing text_encoder handling. Both `__call__`s gain `enable_prompt_enhancement: bool | None = None`, which resolves to `True` when a dedicated `prompt_enhancer` is configured (LTX-2.4) or when `system_prompt` was passed explicitly (matching prior LTX-2.0/2.3 behavior exactly), and `False` otherwise. Explicit `False` disables enhancement even on 2.4. When enabled with no `system_prompt` on a 2.4 pipeline, the matching default system prompt is injected. `max_new_tokens`/`seed` keep their literal defaults (512/10) unconditionally: greedy decoding consumes no randomness, so `seed` is inert for the dedicated-enhancer case, and 512 tokens comfortably covers the target caption length. No public API or default changes were needed for LTX-2.0/2.3. Validation of the enhancement arguments happens in `check_inputs`, so an unsatisfiable request fails before the prompt is encoded rather than partway through generation. Text encoder, tokenizer, processor and enhancer type hints name the concrete Gemma classes they accept. - utils.py: add a `PromptEnhancementConfig` dataclass plus `GEMMA3_PROMPT_ENHANCEMENT_CONFIG`/`GEMMA4_PROMPT_ENHANCEMENT_CONFIG` as the single source of truth for each model's message prefix and `.generate` kwargs, shared by both pipelines. Add the validated "capstyle_plus" LTX2_4_T2V/I2V_DEFAULT_SYSTEM_PROMPT strings, marked `docstyle-ignore` so `doc-builder style` cannot re-wrap them -- the prompts must stay byte-for-byte identical to the reference, newlines included. - convert_ltx2_to_diffusers.py: add a "2.4" branch to all five get_ltx2_*_config functions. Transformer/VAE/vocoder configs are structurally identical to 2.3 (verified against the checkpoint's own safetensors metadata); only `ff_bias=False` differs. Connector `caption_channels`/`text_proj_in_factor` are now derived from the live Gemma text config rather than hardcoded, since 2.4's text encoder is not pinned to a single checkpoint the way Gemma-3-12B is for 2.0/2.3. Swap `Gemma3ForConditionalGeneration`/`Gemma3Processor` for `AutoModelForImageTextToText`/`AutoProcessor` so one path covers Gemma 3 and Gemma 4. Raise a clear error when --version 2.4 is requested without pointing --text_encoder_model_id at a Gemma 4 (gemma4_unified) checkpoint, and add --prompt_enhancer_model_id, required whenever --version 2.4 is combined with --add_processor, since falling back to --text_encoder_model_id (correct for 2.0/2.3) would pair 2.4 with the wrong enhancement model. Also fixes a pre-existing vocoder class-selection bug that only checked for "2.3", and `processor` never being passed into the --full_pipeline LTX2Pipeline(...) construction. - docs/source/en/api/pipelines/ltx2.md: add an "LTX-2.4" section covering what carries over from 2.3 unchanged (guidance recommendations, aside from a different STG block index) and what does not (a single-stage checkpoint only, with no two-stage or distilled workflow yet), plus the corrected prompt-enhancement recipe and its enabled-by-default behavior for both LTX2Pipeline and LTX2ImageToVideoPipeline. * Fix resolution-dependent timestep shift being a no-op in all LTX2 pipelines Every LTX2Pipeline variant's `mu = calculate_shift(...)` call passed the scheduler's `max_image_seq_len` config value as the `image_seq_len` argument instead of the current generation's actual packed sequence length. Since calculate_shift's formula is a line through (base_seq_len, base_shift) and (max_seq_len, max_shift), passing image_seq_len == max_seq_len always evaluates to exactly max_shift -- so `mu` was pinned to a constant (2.05 for the LTX-2.4 scheduler config) regardless of height/width/num_frames, even though `use_dynamic_shifting: true` is set specifically to make this resolution-dependent. Found while benchmarking LTX-2.4 diffusers output against the reference pipeline with bit-identical starting noise: the reference computes this shift from the real video token count (ltx_core's LTX2Scheduler.execute), which diverges substantially from the constant diffusers was using at any resolution other than exactly the checkpoint's max-anchor token count. For a 768x512, 121-frame video (6144 tokens), the reference lands on mu ~= 2.78 versus diffusers' constant 2.05. Fixed by passing the current call's actual packed video latent length (`latents.shape[1]`) as `image_seq_len`, matching the pattern already used correctly in pipeline_flux.py (the source this was copied from). Applies to all 5 LTX2 pipeline variants, each with their own independent `__call__` (not linked via `# Copied from` for this method). * Add LTX-2.4 duration head for automatic num_frames prediction LTX-2.4 checkpoints ship a small regression head (~1.9M params) that predicts the natural duration of the shot implied by a caption, from the same text connector output the transformer is conditioned on. With it converted, a caller can let the model choose the video length instead of picking `num_frames`. - duration_head.py: `LTX2DurationHead` (a `ModelMixin` optional pipeline component) plus the `LTX2AutoDuration` request object. Modality-specific projections map the video and audio connector streams into a shared pooler dim, learnable modality embeddings tag them, one learnable query cross-attends the concatenation, and a small MLP regresses a log-duration. `forward` returns seconds as a tensor; `predict_num_frames` clamps to bounds and snaps to the VAE's causal temporal grid. The attention pooler uses explicit to_q/to_k/to_v/to_out with `dispatch_attention_fn` rather than `torch.nn.MultiheadAttention`, which both reference implementations use because that is the layout the checkpoint ships. `nn.MultiheadAttention` is documented in diffusers as breaking `enable_sequential_cpu_offload`; the split form also gets backend dispatch. Two details are load-bearing for numerical parity: the GELU must be tanh-approximated (the exact GELU gives different numbers against the JAX-trained head), and the clamp must precede the grid snap (a clamped frame count is not necessarily grid-aligned). Where narrow bounds convert to a frame window containing no grid point -- at 24 fps [1.0s, 1.02s] rounds to [24, 24], and 24 is not 8k + 1 -- the nearest grid point is used and a warning logged, rather than refusing to generate over a rounding artifact. The output MLP's config argument is `mlp_hidden_dim`, not the reference's `mlp_hidden`: the submodule keeps the checkpoint's `mlp_hidden` name, and `ModelMixin.__getattr__` resolves config keys ahead of submodules, so the two colliding would shadow the `nn.Linear` with an `int`. - pipeline_ltx2.py / pipeline_ltx2_image2video.py: `num_frames` becomes `int | LTX2AutoDuration | None`. Omitting it auto-predicts when the checkpoint ships a head and keeps the legacy 121 otherwise, mirroring the reference -- whose CLI also defaults to auto-prediction -- and matching the resolution these pipelines already do for `enable_prompt_enhancement`, which likewise switches on the presence of an optional LTX-2.4 component. Pre-2.4 pipelines have no head, so nothing changes for them. The prediction runs immediately after the connectors and before `num_frames` is first read. Only the positive half of the CFG-concatenated batch is used, and rows past the first are `num_videos_per_prompt` duplicates. Auto-duration is rejected from `check_inputs` -- before prompt enhancement and encoding, so a bad request costs nothing -- when there is no head, and when more than one prompt is supplied: the batch carries a single temporal dimension, so prompts with different natural lengths cannot share one frame count. Batched prompts with an explicit integer `num_frames` are unaffected. - convert_ltx2_to_diffusers.py: `--duration_head` (with `--duration_head_prefix`) alongside the other per-component flags. The head's keys sit at the checkpoint top level rather than under the DiT prefix, so the existing prefix helper handles them. The checkpoint's fused `in_proj_weight` is split into separate q/k/v, and dimensions are read back from weight shapes since checkpoint metadata carries no duration_head config; only `num_pooler_heads` is not recoverable and is fixed at 4. Pre-2.4 checkpoints yield no keys and are skipped rather than failing. Verified against ltx_core on the real 2.4 checkpoint: fed the reference's recorded connector tokens, the converted head predicts 12.9375s (video only), 2.671875s (audio only) and 3.515625s (both) -- bit-identical to the reference -- and the same frame counts through the clamp/snap path at 24/25/30/8 fps. The snapping arithmetic matches the reference `seconds_to_clamped_num_frames` across 260 combinations of duration, frame rate and bounds. * Add LTX-2 modular video and audio decoder blocks Add the first blocks of the LTX-2 modular pipeline under modular_pipelines/ltx2/decoders.py: - LTX2VaeDecoderStep: unpacks and decodes video latents (or returns latents for output_type="latent"), applying the optional decode-time noise on normalized latents before denormalizing, matching the standard LTX2Pipeline decode stage. - LTX2AudioDecoderStep: unpacks and decodes audio latents into a waveform via the audio VAE and vocoder in a single block. Pack/unpack/denormalize helpers are redefined at module level rather than imported, since modular blocks must not import from diffusers.pipelines.* (the vocoder class is imported from the pipelines path for now, flagged for relocation to models/). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Add LTX-2 modular pipeline (t2v + i2v) and parity harnesses Add the LTX-2 modular pipeline package under modular_pipelines/ltx2/, covering the joint video+audio text-to-video and image-to-video workflows for LTX-2.4: - encoders.py: dedicated Gemma-4 prompt enhancer (t2v/i2v), Gemma text encoder, text connectors, and the i2v image VAE encoder. - before_denoise.py: text-input expansion, flow-match timesteps (with a deep-copied audio scheduler), video/audio latent prep, and RoPE coords. - denoise.py: the joint video+audio denoise loop with manual guidance (CFG + spatio-temporal + modality-isolation), shared across t2v/i2v. - decoders.py: video VAE decode and audio VAE + vocoder decode. - modular_blocks_ltx2.py: LTX2Blocks (t2v), LTX2ImageToVideoBlocks (i2v), and LTX2AutoBlocks (both, default), plus the auto/conditional wrappers. - modular_pipeline.py: LTX2ModularPipeline with the compression-ratio and patch-size properties the blocks read. Wire up lazy imports and register the pipeline (top-level diffusers exports, modular_pipelines __init__, MODULAR_PIPELINE_MAPPING, and dummy objects). Also add integrations/ (temporary, for-visibility) parity harnesses that compare the modular t2v/i2v blocksets against the standard LTX-2 pipelines by sharing the same loaded components. This directory is meant to be removed before the final integration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Add script to create tiny LTX-2.4 test pipeline ckpt (under assumption that it inherits all configs from LTX-2.3) * make fix-copies * Run modular connector once on CFG-concatenated batch for bitwise parity Run the LTX-2 text connector once on the CFG-concatenated `[uncond, cond]` batch (as `LTX2Pipeline` does) instead of once per branch, then split the outputs back into uncond/cond. The connector is applied per batch element, so both forms are mathematically equivalent, but its GEMM/attention kernels round identically for a given row only at batch >= 2; running the branches separately diverged from the standard pipeline by ~1e-6 at `num_videos_per_prompt=1`. The modular path is now bitwise-identical to the standard pipeline at any batch size. Also extend the T2V/I2V parity harnesses with a `--num_videos_per_prompt` argument and a `--check_tensor_stats` flag for per-output min/mean/std/max. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Use tiny LTX2VocoderWithBWE in the LTX-2.4 test checkpoint Swap the placeholder LTX2Vocoder for a scaled-down LTX2VocoderWithBWE that mirrors LTX-2.3's vocoder (snakebeta + antialiasing, no final activation/bias, 16kHz -> 48kHz bandwidth extension) while keeping the same in/out channel shapes. Dimensions are reduced but the shape invariants the two-stage forward requires are preserved: in_channels = audio_vae.output_channels * mel_bins, bwe_in_channels = out_channels * num_mel_channels, filter_length == window_length, and prod(bwe_upsample_factors) == (output_sr // input_sr) * hop_length so the BWE residual and the resampled stage-1 skip line up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Match LTX-2.4 transformer flags in the test checkpoint Only `ff_bias=False` is a transformer-level delta from LTX-2.3 per the authoritative LTX-2.4 config in `scripts/convert_ltx2_to_diffusers.py`; `audio_ff_bias` and `use_prompt_adaln_single` keep their `True` defaults. The test checkpoint was incorrectly overriding both to `False`, so drop those two overrides. T2V/I2V modular-vs-standard parity remains bitwise. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Add CPU offload flag to parity scripts to be able to load full checkpoints * Support dynamic timestep shift and duration head in LTX-2 modular pipeline Track two recent additions to the standard LTX-2.4 pipelines in the modular blocks (t2v + i2v): - Resolution-aware timestep shift: LTX2SetTimestepsStep now computes `mu` from the actual packed video sequence length (derived from height/width/num_frames and the transformer patch sizes) instead of a constant, matching the standard pipeline's `latents.shape[1]`-based shift. Uses the compute-from-dims approach (like LTX-1 / Flux2), so no block reordering is needed. - Optional duration head: add LTX2DurationStep, which predicts a concrete `num_frames` from the connector text conditioning via the `duration_head` component when `num_frames` is an `LTX2AutoDuration` request, and re-emits it. Wrapped in the LTX2AutoDurationStep conditional (skipped for an integer `num_frames`) and wired into all three blocksets after the connector step, so `num_frames` is resolved before the shift and latent prep run. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Add duration predictor support for tiny checkpoint and parity scripts * Refactor LTX-2 modular guidance into video/audio guiders Replace the manual multi-term guidance in `LTX2LoopDenoiser` with a `LTX2Guidance` guider (in `guider.py`), instantiated once as the video `guider` and once as the `audio_guider`. Each combines CFG + spatio-temporal guidance (STG) + modality-isolation via the delta formulation in x0 space; the denoiser owns a `plan_guidance_passes` union plan across the two guiders, runs each transformer pass, converts velocity->x0, and delegates the per-modality combine to the guiders. Guidance scales are now guider config, not `__call__` kwargs. Parity harnesses updated to configure the guiders accordingly. Parity investigation (not yet resolved): - The refactor runs every guidance pass as its own single-batch transformer forward, whereas the standard `LTX2Pipeline` batches the cond+uncond CFG pair into one forward and runs STG/modality-isolation as separate single-batch passes. STG and modality already match (single-batch in both). - The change is mathematically equivalent, not a logic bug: in fp32 the denoised latents match to ~8e-6 mean abs diff (sparse outliers up to ~3.5e-4), and disabling STG does not move the diff. - But GPU matmul is not batch-invariant, so cond computed alone differs from cond computed inside a batch-of-2. Negligible in fp32 (~1e-6/op); ~1e-2/op in bf16, where amplification by the CFG delta and accumulation over sampler steps drives the modular vs. standard bf16 latents to ~10% mean-relative divergence. - Net: numerical, but the modular pipeline does NOT reproduce the standard pipeline bitwise in bf16 (the real inference dtype). Restoring parity would require re-batching the cond+uncond pair into a single forward to match the reference execution, keeping STG/modality single-batch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Batch cond+uncond CFG forward to mirror standard LTX-2 pipeline Restore fp32 bitwise parity with `LTX2Pipeline` in the modular denoiser by running the cond+uncond CFG pair as a single batched transformer forward (`torch.cat([latents] * 2)` + `.chunk(2)`), keeping STG and modality-isolation as separate single-batch conditional forwards -- matching the reference op-for-op (batch sizes, repeated coords, cache-context names). `plan_guidance_passes` now emits forward-groups (`identifiers` / `conditioning` aligned lists + `flags` + `cache_context`) instead of one entry per pass; the denoiser runs each group once and chunks the CFG forward back into its `[uncond, cond]` identifiers. The `LTX2Guidance` combine is unchanged -- only how the four x0 tensors are obtained changed. Parity results: - fp32: bitwise (0.0 max abs diff), verified at full-checkpoint scale including under CPU offload. This is the authoritative parity gate. - bf16: the previous single-batch-per-pass design diverged ~10% mean-relative from the standard pipeline (GPU matmul is not batch-invariant: cond alone vs. cond in a batch-of-2). Batching the CFG pair removes that gap; a smaller ~1% (tiny) / ~5% (full) bf16 gap remains. It is not a logic difference (fp32 is bitwise across scale and offload); it is a bf16-kernel effect -- coarser mantissa amplifying non-associative accumulation order, plus bf16 using different kernels than fp32 (tensor-core GEMM algorithm selection, fused attention). bf16 is therefore a close-but-not-bitwise check, not a gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Drive LTX-2 modular guidance through the guider API (per-pass inputs) Replace the batched-CFG denoiser with one that runs each guidance pass as its own single-batch transformer forward, driven end-to-end through the standard guider API. `LTX2Guidance.prepare_inputs(guider_inputs)` builds one identifier- tagged batch per active pass from a dict whose values are 4-tuples indexed by pass [cond, uncond, stg, modality]; the per-pass model flags (`spatio_temporal_guidance_blocks`, `isolate_modalities`) ride in those tuples alongside the encoder inputs, so a pass fully describes its own forward. The denoiser unions both guiders' passes by identifier, runs each once (storing video+audio x0 on the batch), and combines each modality via its guider's `forward`/`__call__`, filtered to that guider's active passes so the batch count matches `num_conditions`. Removes the bespoke `plan_guidance_passes` union helper and the empty-dict `prepare_inputs_from_block_state` call: the plan is now expressed as the `guider_inputs` tuples + `active_predictions()`, so guidance logic lives behind the guider API rather than in the denoiser. Parity trade vs. the previous batched-CFG design: - Batched CFG matched the standard pipeline op-for-op and was fp32-bitwise. Running every pass single-batch is mathematically equivalent but, since GPU matmul is not batch-invariant, cond computed alone differs from cond inside a batch-of-2: ~1e-4 mean-relative in fp32 on a full checkpoint (sparse outliers up to ~3.5e-4 max), ~10% mean-relative in bf16. - This is numerical, not a logic difference, and fp32-within-tolerance (not bitwise) is the modular-ecosystem norm. The trade buys end-to-end guider-API usage (swappable within the LTX-2 guidance family, per-pass flags carried the same way as encoder inputs) at the cost of the bitwise guarantee. Parity harnesses: gate on magnitude-aware stats (mean abs diff relative to mean magnitude, plus a loose max-abs ceiling) instead of assert_close's near-bitwise fp32 defaults, which no single-batch design can clear. fp32 is the authoritative gate (1e-3/1e-3); bf16 is a loose sanity check (0.15/0.5). `--atol`/`--rtol` become `--mean_rel_tol`/`--max_abs_tol`; motivation documented in-file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * Drive LTX-2 modular guidance via prepare_inputs_from_block_state Move `LTX2LoopDenoiser` onto the standard Wan/Z-Image guider idiom: the denoiser now owns a `guider_input_fields` map (transformer arg -> per-pass block-state attribute names, indexed [cond, uncond, stg, modality]) and calls `guider.prepare_inputs_from_block_state(block_state, guider_input_fields)` instead of hand-building a literal `guider_inputs` dict in `__call__`. This lifts the cond/uncond/stg/modality field mapping to a construction-time arg (swappable per workflow) and resolves the connector_*->encoder_hidden_states name mismatch via the map keys. The two per-pass model flags (`spatio_temporal_guidance_blocks`, `isolate_modalities`) are pass-identity constants, not block-state conditioning, so they can't ride the name-referenced field map; the denoiser sets them on each batch by identifier after preparation, via a `pass_flags.get(identifier, (None, False))` lookup. The plain-conditional default keeps this correct for any guider that emits a subset of passes (e.g. a swapped-in `ClassifierFreeGuidance` -> just pred_cond/pred_uncond gets no STG and no modality isolation). `LTX2Guidance` gains `prepare_inputs_from_block_state` (names, via the base `_prepare_batch_from_block_state` helper); `prepare_inputs` (literals) is retained so both halves of the guider data-prep API are implemented. Pass structure and numerics are unchanged (still four single-batch forwards), so the fp32/bf16 parity story is untouched; user-confirmed parity OK. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Route LTX-2 batch-invariant denoiser kwargs via denoiser_input_fields Tag the three upstream-produced, batch-invariant transformer kwargs (`audio_num_frames` from the audio-latents step; `video_coords` / `audio_coords` from the coords step) with `kwargs_type="denoiser_input_fields"`, and have `LTX2LoopDenoiser` collect them from `block_state.denoiser_input_fields` filtered against the transformer's forward signature (à la qwenimage/cosmos3) instead of listing them as explicit inputs. This drops three explicit `InputParam`s in favor of one `denoiser_input_fields` template input. Per-pass conditioning (cond/uncond/stg/modality) stays on the guider field map -- the tag only delivers a flat dict, so it can't do the cond/uncond split or the connector_*->encoder_hidden_states rename. The locally-computed latent dims (num_frames/height/width/fps) are still supplied in-denoiser: they aren't upstream outputs and their names would clash with the pixel-space values in state. Parity unchanged (validated T2V + multi-frame I2V): the tagged values reach the transformer identically; the first-forward capture shows them bitwise-equal. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Fix LTX-2 parity harnesses: guidance overrides reach both pipelines The CLI guidance flags (`--guidance_scale`, etc.) only fed the modular guiders; the standard pipeline was always called with the hardcoded GUIDANCE dict. So `--guidance_scale 1.0` disabled CFG on the modular side only, comparing standard-with-full-CFG (batch-2) against modular-no-CFG (batch-1) -- an apples-to-oranges run that manifested as a huge (but spurious) I2V mismatch. Add `_resolve_guidance(args)` (CLI overrides on top of GUIDANCE) and drive BOTH the standard call and `_make_guiders` from the one resolved dict, in both the t2v and i2v harnesses. With CFG correctly disabled on both sides, multi-frame I2V is bitwise-ish (~5e-6 mean-rel), confirming the default-guidance ~8e-3 divergence is the documented cond/uncond batch-invariance (amplified by I2V's per-token masked timestep + clean anchor frame), not a logic bug. Also: - i2v: loosen the fp32 gate to (2e-2, 1.5e-1) to fit that amplified-but-numerical multi-frame divergence, with a comment pointing at the CFG-off run as the tight bug-catching gate. - i2v: add `--debug_forward`, which diffs the first transformer forward (inputs + outputs) between the two runs via a forward hook -- the diagnostic that pinpointed the batch-shape mismatch above. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Document denoiser_input_fields standalone-run caveat for LTX-2 denoiser `audio_num_frames` / `video_coords` / `audio_coords` reach `LTX2LoopDenoiser` via the `denoiser_input_fields` tag, not as named inputs. Note in the docstring that a standalone run (without the upstream tagging blocks) must pass them through `denoiser_input_fields={...}`; plain named kwargs are silently ignored (modular.md's kwargs_type standalone gotcha). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address LTX-2 modular self-review: fix dead enhancer trigger + docstrings Blocking fix: `enable_prompt_enhancement` was declared only as a `block_trigger_inputs` entry / `select_block` param, never as an `InputParam`, so it was not an accepted pipeline input -- `pipe(enable_prompt_enhancement=True)` was dropped as "unexpected" and the prompt enhancer could never run. Declare it on both enhancer sub-blocks so it reaches `select_block` (mirrors how the `image` trigger is declared). Also from the self-review: - Regenerate the modular auto-docstrings: the guidance knobs (guidance_scale, stg_scale, ...) that moved onto the guider no longer show as block inputs, and the enhancer trigger now appears. - Add descriptions to the 16 `InputParam`s that rendered as "TODO: Add description." in the generated docstrings (conversion checklist requires none). - Drop the defensive `getattr(tokenizer, "padding_side", "left")` (a declared tokenizer always has it; gotcha #7). - Rewrite two ephemeral comments (the connector-output "reconcile when denoise.py is written" NOTE, now resolved; the guider "batched-CFG variant in git history" pointer) into standing rationale. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Fix stale I2V parity-harness --help text for the loosened fp32 gate The `--mean_rel_tol` / `--max_abs_tol` help strings still advertised the old fp32 defaults (1e-3, 1e-3); the I2V gate is now (2e-2, 1.5e-1). Update the help to match the actual DTYPE_TOLERANCES so `--help` doesn't misreport the gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * make style and make quality * Add the LTX-2.4 diffusion VAE decoder with a native-parity harness * Add a converter for the LTX-2.4 diffusion VAE decoder * Initialise use_slicing and document the diffusion VAE decisions * Record real-weight parity against the reference decoder * Add model-level tests for the LTX-2.4 diffusion VAE decoder The generated test file needed three decisions rather than fill-ins: the class had no forward() for the mixins to call, the decoder denoises so forward has to take a generator for any output comparison to mean anything, and MemoryTesterMixin.test_group_offloading reused one inputs_dict across four forwards without re-seeding it. The last is fixed in the mixin, reusing the helper test_group_offloading_with_disk already had for the same reason, now module-level and reading the signature off the class (offloading replaces model.forward with a *args wrapper). * Fold the diffusion VAE converter into convert_ltx2_to_diffusers.py --diffusion_vae now sits beside --vae instead of shipping a second script. It also drops the standalone script's --encoder-vae: the encoder is in the same checkpoint and goes through the conv VAE's own rename rules, with its config pinned in get_ltx2_diffusion_video_vae_config. Output verified bitwise identical to the standalone script's on the rc2 checkpoint, 491/491 tensors and the same config.json. * Make the diffusion decoder work through LTX2Pipeline The documented usage -- LTX2Pipeline.from_pretrained(repo, vae=diffusion_vae) -- had never been run: every real-weight run so far called vae.decode() directly. Through the pipeline it raised on vae.config.timestep_conditioning, which this decoder has no reason to carry, and would have bound the positional decode timestep to the decoder's generator. The decode step now branches on the decoder type, skipping the decode_timestep pre-noising and passing the generator instead, which also makes decoding seed-reproducible; two full runs from one seed agree bitwise. The conv path's own statements are unchanged. * Make the diffusion decoder work through the modular decode step Same fix as the standard pipeline, in LTX2VaeDecoderStep: skip the decode_timestep pre-noising and pass the generator instead of a timestep. Verified on rc2 weights by injecting the decoder into LTX2Blocks().init_pipeline() with update_components, the route the modular parity harness already uses -- two runs at 320x448x17 from one seed agree bitwise, so no modular_model_index.json repo was needed to run this. * Note that the pipeline decode block predates the LTX-2.4 integration PR * Address diffusers self-review on the diffusion VAE decoder Four things the project's own rules catch: - models.md forbids unconditional torch.float64 in a model (MPS/NPU/Neuron cannot run it). The RoPE frequency base now goes through maybe_adjust_dtype_for_device, the helper flux/flux2/wan use for exactly this. Measured cost of the downcast where it applies: 1.5e-08 on the frequencies, 1e-06 on the angles. - rope_dim_split was threaded through five classes but was never a config entry, so it was always None and always fell back to the default split. Removed, and the default-split helper inlined into its one caller. - FromOriginalModelMixin was declared with no SINGLE_FILE_LOADABLE_CLASSES entry, so from_single_file raised rather than working. Dropped; single-file support can come with its own mapping later. - set_attention_backend() overwrites the processor's _attention_backend, which handed the neighborhood BlockMask to backends that cannot read it. Guarded like AnyFlowCausalAttnProcessor does. Also runs make fix-copies, which was missing the dummy object for the new class. * Add the API doc page for AutoencoderKLLTX2VideoDiffusionDecoder * Mark the parity harness transient and drop the notes doc from the diff Matches the header the other integrations/ harnesses on this branch already carry, and states that the file imports the native reference package. The notes doc goes to the PR description instead: the self-review skill is explicit that it never ships in the diff. * Accept the diffusion decoder in the image-to-video pipeline too Same branch as the text-to-video pipeline: skip the decode_timestep pre-noising and pass the generator instead of a timestep. The remaining LTX2 pipelines that share this decode block (condition, hdr_lora, ic_lora, latent_upsample) are deliberately left for later, so the surface stays the two pipelines this PR actually exercises. * Reorganize LTX-2 modular blocks into coarser component-aligned phases Regroup the top-level LTX-2 block assemblies so each direct child of LTX2AutoBlocks is a coherent, component-aligned phase, mirroring Flux 2's [text_encoder, vae_encoder, denoise, decode] shape: - New LTX2TextConditioningStep wraps [text_encoder, text_input, connectors] as one "text_encoder" phase (LTX-2's text conditioning is genuinely a three-step model chain; text_input must precede connectors for batch-invariant parity, so the whole chain lives here rather than pushing text_input into denoise). - New LTX2DecoderStep wraps [video_decode, audio_decode] as one "decode" phase. - duration stays a top-level child: it is component-aligned (duration_head) and a ConditionalPipelineBlocks that self-skips for concrete num_frames, so it is a cleanly optional phase alongside prompt_enhancer / vae_encoder -- keeping it out of the (mandatory) denoise block. Top-level children become: t2v: [text_encoder, duration, denoise, decode] i2v: [text_encoder, duration, vae_encoder, denoise, decode] auto: [prompt_enhancer, text_encoder, duration, vae_encoder, denoise, decode] Also reorder the class definitions to follow this flow, keeping the core-denoise trio (LTX2CoreDenoiseStep, LTX2Image2VideoCoreDenoiseStep, LTX2AutoCoreDenoiseStep) contiguous instead of split by LTX2AutoVaeEncoderStep. Pure re-grouping: no block bodies change and intermediate outputs still propagate, so t2v/i2v parity is bit-identical to before (user-confirmed at 17 frames). Docstrings regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Enable LTX-2 prompt enhancement by default when the enhancer is loaded Mirror the standard pipeline's tri-state prompt-enhancement default (pipeline_ltx2.py:559-560), where `enable_prompt_enhancement` defaults to auto-on when a `prompt_enhancer` component is present (LTX-2.4) and off otherwise, and wire the enhancer into the t2v/i2v assemblies. - Both enhancer sub-blocks: default `enable_prompt_enhancement=None` (tri-state). Resolve `None` against component presence inside `__call__` -- select_block cannot see components, so the gate dispatches for None/True and the sub-block decides. `None` + no enhancer -> pass the prompt through unchanged (the Cosmos3 optional-component pattern); explicit `True` + no enhancer -> a clear ValueError instead of the previous cryptic `NoneType` crash from within the enhance helper. - LTX2AutoPromptEnhancerStep.select_block: skip only on an explicit `False`. - Add LTX2AutoPromptEnhancerStep to LTX2Blocks and LTX2ImageToVideoBlocks so all three assemblies share the prompt_enhancer-first shape (previously only LTX2AutoBlocks had it; passing the flag to the others was silently ignored). - Parity harnesses: force `enable_prompt_enhancement=False` on the modular run too (the modular sets now carry an enhancer), and fix the now-stale "contains no enhancer block" docstrings. Only the dedicated-enhancer (LTX-2.4) path is wired; the standard pipeline's text_encoder-as-enhancer fallback for LTX-2.0/2.3 remains a follow-up. Parity unchanged (both pipelines run with enhancement disabled in the harnesses). Docstrings regenerated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Rename LTX-2.4 product identifiers to LTX-2.5. Update conversion --version, model ids, system-prompt constant names, and docs/comments to the public 2.5 name. Behavior is unchanged in this commit. * Add image-conditioning H.264 CRF matching Lightricks training. Resolve CRF 18 for LTX-2.5 (Gemma 4) and 33 for earlier generations, and apply the PyAV re-compress path for I2V and single-frame Condition/IC keyframes. * Align LTX-2 pipelines with the Lightricks reference behavior. Extract shared prompt-enhancement and check_inputs mixins; enable Condition duration+enhancement and IC enhancement (HDR remains encode-only); match enhancement recipe and LTX-2.3/2.5 sampling defaults. * Update LTX-2 docs for 2.5 naming, defaults, and opt-in enhancement. Document LTX_2_3/2.5 sampling defaults, STG block 28, and enable_prompt_enhancement=True as the opt-in path. * Replace LTX2AutoDuration with min_seconds/max_seconds call args. Drop the dataclass wrapper and expose duration bounds directly on the T2V, I2V, and Condition pipelines. * Align LTX-2 modular blocks with the merged standard-pipeline changes The standard LTX-2 pipelines moved to shared `LTX2PromptEnhancementMixin` / `LTX2CheckInputsMixin`, dropped `LTX2AutoDuration`, adopted the Lightricks reference defaults, and gained H.264 CRF re-compression of image conditionings. Port all of that to the modular blocks. - Prompt enhancement: re-derive `_enhance_prompt` from the new mixin -- the `user prompt:` / `User Raw Input Prompt:` templates, 896px long-side image prep, left-padding to a multiple of 8 for Flash Attention, `clean_response` on the decoded output, and `max_new_tokens=None` resolving to the Gemma-4 budget of 600. - Revert `enable_prompt_enhancement` to a plain `False` opt-in, matching the reference pipelines (the tri-state auto-on default is gone upstream). The friendly error for `True` with no enhancer loaded is kept. - Auto-duration: trigger on an omitted `num_frames` instead of an `LTX2AutoDuration` sentinel, with `min_seconds`/`max_seconds` as call args. `LTX2DurationStep` raises when no `duration_head` is loaded rather than silently falling back to 121 frames. - Every `num_frames` InputParam default becomes `None`: input defaults are seeded into the pipeline state before any block runs, so a default of 121 would leave the auto-duration branch permanently unreachable. - Defaults sweep: `num_inference_steps` 40 -> 30, `use_cross_timestep` False -> True, and the video/audio guider ComponentSpec configs updated to the reference guidance stack. - `LTX2VaeEncoderStep` gains `image_crf` and re-compresses the conditioning image before preprocess, declaring `text_encoder` only to resolve the model default CRF. - Rename LTX-2.4 -> LTX-2.5 throughout, following the upstream product rename. Parity harnesses updated for the new duration arguments, plus an `--image_crf` flag on the i2v harness defaulting to 0 so the comparison isolates block logic from the PyAV codec round-trip. Verified against a real checkpoint: t2v, i2v at `--image_crf 0`, and i2v at `--image_crf -1` all pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Evaluate the decoder's SwiGLU in token tiles Decode-only peak VRAM at 121 frames and 512x768 was 24.6 GiB, and 17.0 GiB of that was one transient inside the first stage-5 block: `w_gate(x)` and `w_up(x)` are both hidden-width and their product makes a third, so the whole video's worth of hidden activations was live three times over. At that size each one is 2,973,696 tokens x 1024 x 2 bytes = 5.67 GiB. The MLP is pointwise across tokens, so tiling it is exact -- it changes only how many hidden-width elements exist at once. Measured on a full-size decode: peak 24.8 -> 19.4 GiB (-22%), wall clock 2.79 -> 2.84 s (+1.6%), output bit-identical (max abs diff 0.000e+00). That also puts the diffusion decoder below the conv decoder, which needs 23.8 GiB at the same size. Tile size 16384 tokens is the reference decoder's own default (`DEFAULT_SWIGLU_TILE_SIZE`). Bounding a token count rather than a number of tiles keeps peak memory independent of resolution. This is the first of the two transients the reference's default decode mode avoids; the other is the full-volume QKV projection plus RoPE's fp32 upcast, which it addresses by chunking attention along width. That one is not bit-exact (it reassociates NATTEN's accumulation), so it is left for a follow-up rather than changed here. * Declare _no_split_modules on the diffusion decoder Both block types close over a residual add that combines outputs from different children, which is the case models.md says has to stay co-located, so neither can be split across devices. Unskips three memory tests that were gated on the attribute: the file goes from 35 passed / 7 skipped to 38 passed / 4 skipped, and the three (test_cpu_offload, test_disk_offload_with_safetensors, test_disk_offload_without_safetensors) pass. * Match the encoder defaults to the published LTX-2.4 config block_out_channels was (256, 512, 1024, 2048) and layers_per_block was (4, 6, 6, 2, 2); both were inherited from the 2.0 conv VAE. The published vae_diffusion/config.json has (256, 512, 1024, 1024) and (4, 6, 4, 2, 2). Real loads were unaffected because the config sets these explicitly, but the defaults described a model that does not exist. All 27 __init__ defaults that appear in the published config now match it. * Cover the token-tiled SwiGLU path in the tests The dummy video is 9x48x48, so its stage-5 grid is 5184 tokens against a 16384-token tile size: every other test in the file takes the untiled branch and the tiling loop ran nowhere in CI. This shrinks the tile size to force ~41 tiles and requires torch.equal against the untiled evaluation, since the MLP is pointwise across tokens and tiling must be exact rather than close. Checked the test is not vacuous: sabotaging the tiled branch alone (silu on w_up instead of w_gate) fails it at 8.83e-01. * Let the parity harness run against the LTX-2.5 reference too The harness only spoke LTX-2.4's decoder API and died on `forward_pre_diffusion` against 2.5, so it could not check the release the port is now targeted at. Three shims absorb the differences: * 2.5 selects a decode pathway by swapping `__class__` on every block, so an as-constructed decoder has no `forward_combined`. Install the COMBINED pathway explicitly, uncompiled, since that is the pathway this port implements. * stages 1-4 split into `forward_stages_1_to_3` + `forward_stage_4`, so the chunked pathway can defer stage 4. * the trailing NATTEN ghost pad moved out to the caller while its crop stayed behind `forward_stage_4(pad_trailing=True)`; the two have to be paired or the context loses real frames (3 of 17 survive at this config, which is how the mismatch first showed up). Same numbers against both references -- context 5.367e-04, step 7.337e-04 -- which is the check that the shims are semantically right and not just shape-compatible. * Add LTX-2.X modular condition blocks Ports LTX2ConditionPipeline to the modular pipeline system, including the LTX-2.5 additions (per-condition CRF, conditions-sourced prompt enhancement, auto-duration, the Lightricks reference defaults). New blocks: - encoders.py: LTX2ConditionEncoderStep (resize/center-crop, single-frame H.264 re-compression at the model CRF, temporal trim, VAE encode) and LTX2ConditionPromptEnhancerStep (grounds the rewrite in the first PIL frame found in `conditions`, text-only otherwise). - before_denoise.py: LTX2ConditionPrepareLatentsStep (first-frame overwrite + keyframe token append with their own RoPE coords, emitting conditioning_mask / clean_latents / appended_coords / base_token_count), LTX2ConditionSetTimestepsStep, LTX2ConditionPrepareAudioLatentsStep, LTX2ConditionPrepareCoordsStep, and the _prepare_keyframe_coords helper. - denoise.py: LTX2ConditionLoopBeforeDenoiser (mask-scaled per-token video timestep), LTX2ConditionLoopAfterDenoiser (x0 blend against the clean condition latents, then a full-sequence scheduler step), LTX2ConditionDenoiseStep. - decoders.py: LTX2TrimConditionTokensStep. - modular_blocks_ltx2.py: LTX2ConditionAutoPromptEnhancerStep, LTX2ConditionCoreDenoiseStep, LTX2ConditionDecoderStep, LTX2ConditionBlocks. LTX2LoopDenoiser is reused unchanged: the two LTX2Guidance guiders already express the whole CFG + STG + modality-isolation stack the condition pipeline uses, and the transformer takes the longer packed sequence with the base num_frames/height/width exactly as the standard pipeline does. The standard pipeline's CFG batch duplication (conditioning mask, coords) has no modular counterpart, since each guidance pass is its own single-batch forward. Block ordering: prepare-latents runs *before* set-timesteps, unlike the text-to-video and image-to-video core denoise steps. The resolution-aware shift `mu` is computed from the packed latent sequence length, which here includes the appended keyframe tokens, so the latents must exist first. This mirrors LTX2ConditionPipeline (its section 4 precedes section 5). The text-to-video and image-to-video blocksets are unaffected: with no appended tokens the grid-derived sequence length already equals latents.shape[1]. Two subtleties the parity harness surfaced: - The condition pipeline samples audio noise directly in packed shape [B, L, C * M], while the text-to-video and image-to-video pipelines sample unpacked [B, C, L, M] and pack afterwards. Both draw the same number of values from the generator but lay them out differently, so reusing the shared LTX2PrepareAudioLatentsStep silently desynchronizes the audio noise and, through the joint attention, the video too (1.4 mean-rel on audio, 0.13 on video before the fix). Hence a standalone condition audio block. - `noise_scale` means different things across workflows: 0.0 for text-to-video/image-to-video, `None -> sigmas[0] or 1.0` for conditions. A blockset keeps the first non-None default across its blocks, so a 0.0 would have shadowed the condition resolution and left the latents unnoised. Both condition-path blocks declare None; prepare-latents resolves once and writes the value back for the audio step. Keyframe tokens, masks and coords are expanded to the generation batch (the pattern the in-context pipeline already uses for reference tokens). The standard pipeline builds them at batch 1 and cats them onto a batch-B sequence, so it raises for num_videos_per_prompt > 1 with any condition at latent index > 0; the modular blocks handle that case. Verification: integrations/ltx2_condition_parity.py (transient, removed before merge) compares LTX2ConditionBlocks against LTX2ConditionPipeline on shared component objects. Eight cases on a tiny checkpoint -- first-frame only, keyframe only, both plus a multi-frame video condition, two keyframes, no conditions, --crf -1, --predict_duration, num_videos_per_prompt=2 -- all pass at 3e-7 to 1.3e-6 mean-rel. On a full checkpoint at 768x512x17 with two conditions through the CRF path: video 8.7e-04 mean-rel / 1.1e-02 max-abs, audio 1.5e-04 / 2.1e-03, in line with the image-to-video figures at the same frame count. make style, make quality and make fix-copies are clean; auto-docstrings regenerated with no TODO placeholders. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Add LTX-2.X modular in-context (IC-LoRA) blocks Ports LTX2InContextPipeline to the modular pipeline system on top of the condition blocks, and fixes two bugs in the standard in-context pipeline that the parity harness surfaced. New blocks: - encoders.py: LTX2ReferenceEncoderStep (preprocess, VAE-encode and pack each reference video, compute the coords that map its tokens into the target space, and build the per-token cross-attention strengths), plus the _downsample_mask_to_latent helper. - before_denoise.py: LTX2InContextPrepareLatentsStep (frame conditions as in the condition step, then the reference tokens appended last) and LTX2BuildVideoSelfAttentionMaskStep. - modular_blocks_ltx2.py: LTX2AutoReferenceEncoderStep, LTX2AutoBuildVideoSelfAttentionMaskStep, LTX2InContextCoreDenoiseStep, LTX2InContextBlocks. LTX2ConditionDenoiseStep is reused unchanged: with the blend gate fixed below, reference tokens are pinned by exactly the same x0 blend as frame conditions, matching the reference implementation where VideoConditionByReferenceLatent and VideoConditionByKeyframeIndex produce the same (denoise_mask, clean_latent) pair and post_process_latent runs unconditionally. The condition encoder, timesteps, audio-latents, coords and decoder blocks are shared as well. video_self_attention_mask reaches the transformer through the denoiser_input_fields tag, so LTX2LoopDenoiser needed no change. `reference_conditions` is optional, matching LTX2InContextPipeline: IC-LoRAs that carry their behavior in the adapter weights (camera control, style) take no reference video, which is the shape of the standard pipeline's own docstring example. The reference encoder and the attention-mask step are both skipped when it is absent. There is no duration step (the in-context pipeline ships without a duration_head), so LTX2ConditionEncoderStep now raises a concise error if `num_frames` is still None by the time the conditions are encoded. Per-reference token counts come from the encoder rather than an equal split of the total, so references of differing lengths (a reference video shorter than num_frames) get the right strengths. Standard-pipeline fixes in pipeline_ltx2_ic_lora.py: - The x0 blend was gated on `has_conditions` alone, so with reference conditions and no frame conditions the reference tokens were seeded clean and given a zeroed timestep but never re-pinned. Their x0 prediction error, divided by sigma each step, then accumulated and drifted them away from the encoded reference. Now gated on `has_conditions or num_ref_tokens > 0`, in line with the surrounding gates and the reference implementation. - video_self_attention_mask was rebound in the loop rather than expanded into a per-pass local. The main pass expanded it to the CFG-doubled batch, after which the STG pass's expand() to `latents.shape[0]` raised "RuntimeError: The expanded size of the tensor (1) must match the existing size (2)". Any run combining an attention mask with STG or modality guidance -- i.e. the default guidance settings -- died on the first step. Also removed the unused has_appended_tokens local and replaced the "# - TODO" placeholder above the attention-mask construction with the block structure, which the reference confirms. Verification: integrations/ltx2_in_context_parity.py (transient, removed before merge). Twelve cases on a tiny checkpoint -- reference only, reference at partial strength, scalar attention strength, pixel-space attention mask, downscale factor 2, references plus first-frame and keyframe conditions, two references, the CRF path, and four reference-free variants -- all pass at 3.4e-7 to 1.3e-6 mean-rel. The seven-case condition suite still passes. On a full checkpoint at 768x512x17 with a reference plus two conditions: video 4.4e-05 mean-rel / 5.6e-04 max-abs, audio 3.2e-05 / 2.4e-04. Known limitation, not addressed here: transformer_ltx2.py converts the multiplicative [0, 1] self-attention mask to an additive bias linearly, as (1 - mask) * -10000, whereas the reference uses log space (bias = log(mask), finfo.min at zero). Intermediate strengths therefore saturate to "fully masked", making conditioning_attention_strength effectively binary and spatially varying masks inert beyond their zero/non-zero pattern. Parity is unaffected -- both pipelines feed the same mask into the same transformer -- but the feature does not currently work as documented. make style, make quality and make fix-copies are clean; auto-docstrings regenerated with no TODO placeholders. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Make the diffusion decoder a decode pipeline instead of a VAE Addresses the review: a full diffusion model should not be an AutoencoderKL that hides its denoising behind `.decode()`, and it should not have to be threaded into every LTX-2 pipeline as a `vae`. AutoencoderKLLTX2VideoDiffusionDecoder becomes LTX2VideoDiffusionDecoderModel, a plain ModelMixin with no encoder. Encoding stays with AutoencoderKLLTX2Video, whose latent space this consumes unchanged, so latents remain interchangeable between the two decoders. Dropping the encoder also removes the converter's awkwardness of sourcing encoder weights and the real block_out_channels from the conv `vae/` folder: ten encoder-only config keys are gone. LTX2VideoDiffusionDecodePipeline drives it, shaped like LTX2LatentUpsamplePipeline: run any LTX-2 pipeline with output_type="latent", then decode. `vae` is an optional component, since it is only ever consulted for the latent statistics, and the decoder carries its own `latents_mean` / `latents_std` buffers to fall back on. That means a decode-only workflow never loads a second autoencoder. The `vae=` path is removed accordingly: the isinstance branches in LTX2Pipeline, LTX2ImageToVideoPipeline and the modular decode step are gone, along with the fixtures that built a diffusion decoder to pass as `vae`. The model's slicing/tiling test class went with them, since enable_tiling was an AutoencoderMixin concern and this is no longer an autoencoder. Tests: 37 passed / 3 skipped on the model, 83 passed / 2 skipped across the t2v and i2v pipelines. Still to come: mapping the 2.5 checkpoint to a blockset that selects the diffusion decode block, and the KandinskyCombinedPipeline pattern for running the two-step flow by default in the standard API. * Add a modular decode block for the LTX-2 diffusion decoder The previous commit removed the `isinstance(vae, ...)` branch that let the diffusion decoder stand in as the modular `vae`, which was the right call twice over: it is what the review objected to, and modular.md says a new case gets a new block rather than a branch inside an existing one. But removing it without a replacement left modular with no route to the diffusion decoder at all. LTX2DiffusionVaeDecoderStep is that replacement. It declares the decoder as its own `diffusion_decoder` component rather than borrowing the `vae` slot, and passes a generator instead of a decode timestep, since this decoder draws the noise it denoises. `LTX2VaeDecoderStep` is untouched, so the convolutional path is unaffected. What is still missing is the wiring: a blockset using this block, and an `_ltx2_map_fn` so a checkpoint that ships the diffusion decoder selects it automatically instead of the user choosing. That lands in modular_blocks_ltx2.py, which #12 is reorganizing, so it needs sequencing against that PR rather than racing it. * Test the diffusion decode pipeline, and give it a video-only output The decode pipeline had no coverage: it was never actually run. Writing the tests immediately turned up a bug, which is that it returned `LTX2PipelineOutput`, whose `audio` field is required. This pipeline decodes video only, so it now returns a video-only `LTX2VideoDecodeOutput` instead of passing a meaningless `audio=None`. Four tests: decode with no `vae` (the fallback to the decoder's own latent statistics), decode with a `vae` supplied (its statistics take precedence, checked by making the two disagree rather than asserting a shape), reproducibility under a seeded generator plus divergence under a different seed, and `denormalize=False` actually skipping the statistics. The decoder's stat buffers are set to non-trivial values in the fixture so a run that silently skipped denormalization could not pass by coincidence. * Fix the decode example: output_type="latent" is already denormalized The example in the docs and the pipeline docstring both showed `output_type="latent"` feeding straight into the decode pipeline on its defaults. That path denormalizes twice: `LTX2Pipeline` applies the latent statistics before returning latents (pipeline_ltx2.py), and the decode pipeline applies them again because `denormalize` defaults to True. Every channel ends up scaled by its own std a second time, so e.g. channel 0 (std 0.238) comes out roughly 4x too small. Passing `denormalize=False` in the example and saying so in both docstrings. Caught by running the flow end to end on real rc4 weights, which is also where the two decoders agree at 0.9978 once the statistics are applied only once. Whether the *default* should flip, or `output_type="latent"` should stop denormalizing instead, is a contract question across both pipelines and is still open. * Fix IC-LoRA video self-attention mask grouping The video self-attention mask deviated from the reference implementation (`ltx_core.conditioning.mask_utils.build_attention_mask`) in two ways. Both predate the modular blocks and applied to `LTX2InContextPipeline` as well; the modular block had ported the same construction. 1. Keyframe conditions cross-attended with reference tokens. `num_noisy_tokens` was derived as `latents.shape[1] - num_ref_tokens`, which folds the appended keyframe tokens into the noisy group, so they received the reference cross mask. The reference takes `num_noisy_tokens` from the target latent shape -- generated-video tokens only -- and tracks the extras offset separately, so keyframe <-> reference is 0.0. Split the two quantities apart: `_build_video_self_attention_mask` now takes `num_prefix_tokens` alongside `num_noisy_tokens`. 2. Multiple reference conditions fully attended to each other. All references were passed as a single attention group with a 1.0 self-block. The reference wraps each `LTX2ReferenceCondition` in its own `ConditioningItemAttentionStrengthWrapper`, building the mask once per item, so reference_i <-> reference_j is 0.0. The call site now splits `ref_cross_mask` into one group per reference before building the mask. Threading per-reference token counts out of `_encode_reference_conditions` also removes the equal-split assumption in `prepare_latents` and `prepare_reference_latents`, which assigned the wrong per-token strengths when references encoded to different token counts (reachable when a reference video is shorter than `num_frames`). API impact ---------- `prepare_latents` returns a 7-tuple instead of a 6-tuple, adding `ref_token_counts` (`list[int]`) as the new final element. Elements 1-6 keep the types they have on `main`; in particular `ref_cross_mask` remains the concatenated `[1, num_ref_tokens]` tensor. Callers that unpack the tuple fail immediately with a `ValueError` rather than silently receiving a changed type. `_encode_reference_conditions` (private) likewise gains `reference_token_counts` as a fourth return element. `prepare_reference_latents` keeps its documented 4-tuple unchanged. Verification ------------ Correctness against the reference. Both the standard helper and the modular block were checked for exact tensor equality (`torch.equal`) against `build_attention_mask`, driven the way `ic_lora.py` drives it -- one `ConditioningItemAttentionStrengthWrapper` per reference, so the reference builds iteratively with `existing_mask` carried forward. Five layouts, all matching, exercised through the same concatenate-then-split path the call site uses: - references only, 1 reference - keyframes + 1 reference (exercises fix 1) - keyframes + 2 references (exercises fixes 1 and 2) - 2 references of unequal length (exercises fix 2 and the token counts) - no keyframes, 3 references (exercises fix 2) Standard/modular parity on a real checkpoint, 512x512x17, fp32, via `integrations/ltx2_in_context_parity.py`. Video latents (1, 128, 3, 16, 16), audio latents (1, 8, 18, 16); gates are mean-rel 2e-2 and max-abs 1.5e-1: --reference 1.0 9 --condition 2 0.5 9 --conditioning_attention_strength 0.5 video: max abs 4.822e-05, mean rel 7.851e-06 audio: max abs 1.875e-04, mean rel 1.773e-05 --reference 1.0 9 --reference 0.8 9 --conditioning_attention_strength 0.5 video: max abs 1.953e-04, mean rel 9.101e-05 audio: max abs 9.251e-05, mean rel 1.026e-05 The first case is the layout where `num_prefix_tokens != num_noisy_tokens`; a wrong prefix/base split would have placed the group blocks at the wrong offsets with valid shapes, surfacing as a parity mismatch. The returned video shape also unpacks from exactly 768 base tokens (3 x 16 x 16), confirming `base_token_count` is the quantity now fed as `num_noisy_tokens`. Scope of the parity runs: both sides share this construction, so parity shows they did not desynchronize, not that either matches the reference -- that rests on the equality test above. Both references in the second case are 9 frames and so encode to equal token counts, meaning the equal-split and per-count paths coincide; the token-count change is not discriminated by these runs. A reference pair of differing lengths (e.g. `--reference 1.0 9 --reference 0.8 17`, giving 512 vs 768 tokens) would separate them, and would have failed parity before this commit since the modular block already used per-reference counts while the standard pipeline split evenly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Add LTX25AutoBlocks and LTX25ModularPipeline Following the review: LTX-2.5 is a new checkpoint, so it gets its own blockset and pipeline rather than a config flag inside the LTX-2 ones. LTX25AutoBlocks subclasses LTX2AutoBlocks and overrides exactly one entry, `video_decode`, to LTX2DiffusionVaeDecoderStep. Every other leaf block is reused, so the two blocksets have identical block_names and cannot drift apart. That makes the diffusion decoder the default for LTX-2.5, which matches what the checkpoint ships natively, without the user having to pass a decoder at all. Going the other way stays a one-line swap rather than a second official blockset: blocks.sub_blocks["video_decode"] = LTX2VaeDecoderStep() LTX25ModularPipeline carries `default_blocks_name = "LTX25AutoBlocks"` and a new "ltx2.5" key in MODULAR_PIPELINE_MAPPING, so a repo shipping `modular_model_index.json` routes straight to the new blockset. The condition and in-context workflows belong in this blockset too, but those blocks live in the modular condition/IC PR, so folding them in belongs there rather than as a competing change here. * Drop the encoder half in the diffusion decoder converter The converter still assembled the whole VAE checkpoint, encoder included, and handed it to a strict `load_state_dict`. That was correct while the class was an autoencoder; after it became decoder-only the ~84 `encoder.*` keys have nowhere to land and the load raises. Nothing caught it because no test converts anything and every other check loads pre-converted weights, where unexpected keys are only warned about. Now the encoder is discarded rather than remapped, keeping the `decoder.` half and the per-channel statistics that become `latents_mean` / `latents_std`. `strict=True` stays, since it is what guards against a botched rename rule. Verified by actually running it: pulled the 395 `vae.*` tensors out of the native rc4 checkpoint by byte range, converted, and diffed against the already-published folder. 407/407 parameters, max abs difference 0.000e+00, so the fix changes nothing about the weights it produces. Also added a regression test, since the absence of any converter coverage is why this slipped. * Share the neighborhood attention mask and parameterise the AdaLN chunk count Two of the resolved review threads. Both are internal to the decoder modules and are unaffected by the pipeline-vs-model question still open on the thread above them. Build the FlexAttention mask once per stage instead of once per block. The mask depends only on the grid and the kernel, both fixed within a stage, so 24 builds collapse to 5; measured at 768x512x121 that is 582ms to 132ms. build_block_mask returns None for NATTEN, which encodes the window in its kernel. That is not merely an optimisation: create_block_mask materialises O(N^2) uncompiled, so a 69x64x96 stage would ask for 167 GiB and stage 5 for 8.2 TiB, and the mask must never be built for a path that would not read it. Make the AdaLN chunk count a constructor argument, num_chunks on the shared AdaLN and num_mod_params on the diffusion block, with the decoder passing shared_adaln.num_chunks through so the two cannot drift. num_mod_params is already the name LTX2AdaLayerNormSingle uses. The block's forward still destructures exactly seven chunks and reads four of them, which is the reference's shape. Decoding the test fixture is bitwise identical to before (max abs diff 0.000e+00). 39 passed, 4 skipped. * Export the LTX-2.5 modular classes from the top-level namespace `ModularPipeline.from_pretrained` resolves a repo's `_class_name` with `getattr(importlib.import_module("diffusers"), class_name)`, so a class that is only reachable from `diffusers.modular_pipelines.ltx2` cannot be selected by a checkpoint. `LTX25ModularPipeline` and `LTX25AutoBlocks` were exported there but not from `diffusers` itself, which left the whole checkpoint-to-blockset mapping dead: loading a repo whose index names them raised `AttributeError: module diffusers has no attribute LTX25ModularPipeline`. Exported alongside `LTX2ModularPipeline` / `LTX2AutoBlocks`, which is how the loader already finds those. Verified against a real repo: `ModularPipeline.from_pretrained(...)` now resolves LTX25ModularPipeline -> LTX25AutoBlocks -> LTX2DiffusionVaeDecoderStep, with the decoder supplied by the repo index and nothing passed by hand. Found by loading an actual checkpoint rather than constructing the blocks directly; calling `LTX25AutoBlocks().init_pipeline(repo)` goes blocks-to-pipeline and never exercises the lookup. * Name the diffusion decoder LTX-2.5 and save it to diffusion_decoder/ Two naming fixes, one of them load-bearing. The converter wrote the decoder to a `vae_diffusion/` subfolder, a leftover from when the class was an autoencoder. The folder name matters now: `DiffusionPipeline.from_pretrained` resolves each component from the subfolder named after it, so `LTX2VideoDiffusionDecodePipeline` (component name `diffusion_decoder`, same as the modular `ComponentSpec`) can never load itself out of a repo whose folder is called `vae_diffusion`. The converter now saves to `diffusion_decoder/`, matching the layout the LTX-2.5 weight repos already use, and the docs example loads from that name. The docstrings also called it the LTX-2.4 decoder. The diffusion decoder ships with LTX-2.5; the converter handles those checkpoints under the `--version 2.4` architecture family, which the version guard's message now spells out instead of misattributing the feature. Verified by re-running the converter on the rc4 distilled checkpoint after the change: the output lands in `diffusion_decoder/` and is bit-identical (sha256) to the pre-change conversion, so this renames and redocuments without touching what is produced. The converter regression test passes. * Move the LTX-2 text-input expansion into the core denoise step `LTX2TextInputStep` expanded the prompt embeddings to the effective batch size inside `LTX2TextConditioningStep`, so running the text-conditioning stage on its own baked `num_videos_per_prompt` into its outputs. Feeding them to `LTX2CoreDenoiseStep` with a different `num_videos_per_prompt` meant re-running the text stage or reshaping its outputs by hand. `LTX2TextInputStep` now expands the connector conditioning (the tensors the denoiser actually reads through `guider_input_fields`) instead of the raw Gemma embeds, and runs as the first block of both core denoise steps. The raw `prompt_embeds` expansion is dropped -- it only existed to feed the connector at the expanded batch. `batch_size` / `dtype` move up to `LTX2TextEncoderStep`, since `LTX2DurationStep` needs `batch_size` before the denoise stage. Two behavior notes: - The connectors now run at one row per prompt rather than `num_videos_per_prompt` rows. They are applied per batch element, so this is mathematically identical (verified to ~3.6e-7, float32 rounding) and N times cheaper, but not bitwise identical to `LTX2Pipeline` when `num_videos_per_prompt > 1`. - Expansion uses `repeat_interleave` for the masks as well as the embeds. The previous code (inherited from the standard pipeline) interleaved the embeds but tiled the masks, which misaligned them whenever both `batch_size > 1` and `num_videos_per_prompt > 1`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Describe the connector inputs on LTX2TextInputStep `LTX2TextInputStep` declares the connector conditioning tensors and now runs first in the denoise sequence, so its declarations -- not `LTX2LoopDenoiser`'s -- are the ones the docstring generator picks up. They carried no `description`, which left `TODO: Add description.` placeholders on the three core denoise blocks. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Build the condition and in-context workflows from LTX2AutoBlocks `LTX2AutoBlocks` covered text-to-video and image-to-video only; the condition and in-context (IC-LoRA) workflows were reachable solely through their own blocksets. Fold them into the auto blockset so all four are selected from the inputs, the way t2v and i2v already were. - Merge `LTX2ConditionAutoPromptEnhancerStep` into `LTX2AutoPromptEnhancerStep`, which now picks the condition, image or text enhancer off `conditions` / `image`. - Add `LTX2AutoConditionEncoderStep`, triggered by `conditions` or `reference_conditions` so a reference-only in-context run still gets the empty per-condition lists `LTX2InContextPrepareLatentsStep` requires. - Extend `LTX2AutoCoreDenoiseStep` to four branches and add `LTX2AutoDecoderStep`, which trims the appended keyframe/reference tokens whenever `base_token_count` is present. Two fixes this exposed: - `noise_scale` means an interpolation factor (t2v/i2v, 0.0) in one workflow and an initial noise level (condition/in-context, `None` -> `sigmas[0]` or 1.0) in another. A blockset keeps the first non-`None` default across its blocks, so sharing one blockset published 0.0 and silently zeroed the noise for condition workflows that omitted the argument. `LTX2PrepareLatentsStep` and `LTX2PrepareAudioLatentsStep` now declare `None` and resolve to 0.0 in the block, mirroring what the condition steps already do. - `LTX2AutoBuildVideoSelfAttentionMaskStep.select_block` compared `conditioning_attention_strength` against 1.0, but `get_execution_blocks` passes unset trigger inputs as an explicit `None`, so resolving any in-context workflow raised `TypeError`. The parity harnesses passed `noise_scale=None` explicitly, which took the argument-present path and hid the shadowing bug; `--omit_noise_scale` drops the key entirely so the default is exercised. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Run the text-input step in the condition and in-context denoise steps `LTX2TextInputStep` is the only block that expands the connector text conditioning by `num_videos_per_prompt`, and it was missing from `LTX2ConditionCoreDenoiseStep` and `LTX2InContextCoreDenoiseStep`. The prepare-latents/audio/coords steps all build their tensors at `batch_size * num_videos_per_prompt`, so the video latents arrived at the transformer at batch B*N while `encoder_hidden_states` was still at B — a shape error in cross-attention for any `num_videos_per_prompt > 1`. The step goes at the head of both sequences, before prepare-latents, so the prepare-latents-before-set-timesteps ordering these workflows need (`mu` is read off the full packed sequence length) is untouched. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Broadcast keyframe and reference conditioning across the generation batch Frame conditions are VAE-encoded at batch 1, but `latents`, `conditioning_mask` and `clean_latents` are built at `batch_size * num_videos_per_prompt`. Appending a keyframe condition (latent index > 0) therefore failed in `prepare_latents` with "Sizes of tensors must match except in dimension 1" whenever the batch was greater than 1. First-frame conditions were unaffected because they are applied by indexed assignment, which broadcasts. Expand the keyframe tokens, mask and coords to `batch_size` before the concatenation, and do the same for the IC-LoRA reference coords, which feed the same `appended_coords` block that is later concatenated onto batch-`batch_size` `video_coords`. This is what the modular LTX-2 blocks already do, and matches the `[B, 3, ...]` shape both docstrings promise. Also picks up a pending `make style` docstring rewrap in the same file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat: use natten for processing. (#15) * feat: use natten for processing. * address dhruv's comments. * remove unneeded test * remove integration file * make vae properties accessible without loading it. (#17) * Address review: replace LTX2 mixins with Copied-from pipeline methods. Inline check_inputs/enhance_prompt on LTX2Pipeline and sync I2V/Condition/InContext via # Copied from, and update docs for shared 2.5 defaults. (cherry picked from commit c183923cdd9c951f9694f9f6cd903714330cff03) * Point the modular enhance-prompt comment at LTX2Pipeline `LTX2PromptEnhancementMixin` was inlined into the pipelines in c183923cd, so the mirrored-from reference no longer names an existing class. * Remove the integrations/ dev scripts The LTX-2 parity harnesses and the tiny-checkpoint builder were local development scaffolding and are not part of the shipped library. * moular tests (#19) * Keep LTX-2.5.1 keyframes_abs_pos_embedding through convert. (#20) Distilled 2.5.1 monoliths ship this (1, inner_dim) buffer with use_keyframes_abs_pos_embedding=True; store it on the transformer for load/save without wiring it into the regular distilled forward yet. * remove test_ltx2_keyframes_abs_pos_embedding_load_and_forward * Update the LTX-2.5 docs: two stage generation, distilled default, repo id (#21) * feat: implement tiling for the diffusion decoder. (#22) * add a note on tiling (#23) * fix documentation. --------- Co-authored-by: aartyomov <aartyomov@lightricks.com> Co-authored-by: Daniel Gu <daniel.gu@huggingface.co> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: linoytsaban <linoy.tsaban@gmail.com> Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> Co-authored-by: DN6 <dhruv.nair@gmail.com> Co-authored-by: Linoy Tsaban <57615435+linoytsaban@users.noreply.github.com> |
||
|
Sayak Paul
|
53e5b00d99 |
[tests] implement todos in the modular tests for H3 followup (#14404)
* remove doc placeholder. * fix remaining tests * empty * empty |
||
|
apolinário
|
9c6a68c32b | Point the MiniMax-H3 docs install note at main (#14401) | ||
|
apolinário
|
f53d552036 |
Add MiniMax-H3 (#14355)
* Add MiniMax-H3 * Keep the MiniMax-H3 VAEs in float32 under torch_dtype casts * Make MiniMax-H3 modular only * Assert the VAE float32 pin as a positive contract * Point the modular tests at the hf-internal-testing tiny repo * Document performance recipes per hardware class * Slim hardware loading snippets and PR install note * Use pinnable int8 config and freeze quantized components * Scope load time quantization and disable low cpu mem usage under streamed offload * Minimax h3 follow up (review & refactor) (#14371) review & refactor * Regenerate the modular auto docstrings and restyle the H3 docstrings `make modular-autodoctrings` and `make quality` both failed on CI. The nine `# auto_docstring` blocks in `modular_blocks_minimax_h3.py` were stale, and the docstrings of eleven files had not been through `doc-builder style`. Nothing but docstrings and comments changes here. One trap worth recording: `utils/modular_auto_docstring.py` shells out to `ruff` and degrades silently when it is not on `PATH` ("Warning: tool not found ... Skipping formatting"), which makes it rewrite 42 unrelated pipelines instead of the one that is stale. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Fix the H3 fast tests against the refactored state contract `condition_latents` is the VAE encoder block's own output — a list of one latent per condition — and the after-denoise step unpacks `latents` and drops the conditioning rows, so neither is meaningful once a request has come back. The end-to-end tests no longer reach for them; a new standalone test runs `MiniMaxH3KeyframeVaeEncoderStep` on its own, which is where the list is real. Three other tests were failing for reasons of their own: - `test_check_inputs_references` was missing its `@parametrize` and errored at collection. Added four cases that hit real validation. - The duration ceiling had drifted between the two blocks that check it. `before_denoise` warned and reassigned before validating, so it reported `got 362` — a count the caller never passed, right after warning it had rounded their 346. It now validates first, like `before_encoder` already did, and both messages read `got 346 (rounded up to 362)`. - `height` without `width` raised `TypeError: unsupported operand type(s) for %: 'NoneType' and 'int'` on `t2va`, and died inside the resize on `fl2va`. Both blocks now raise the same error `MiniMaxH3Ref2VASetupStep` already raised. 58 passed -> 71 passed, 5 failed, no errors. What is left: `output_type="latent"` has no opt-out in the video decode block, `reference_image_short_edge` is a constructor argument so it does not survive save/reload, an image reference is PIL-only where the test expects three layouts, and `test_float16_inference`, which fails on the base branch too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Stop claiming `num_inference_steps` is optional in the H3 tests The block declares it `InputParam.template("num_inference_steps", required=True)` and `expected_workflow_defaults` lists it under `required_inputs` for all three workflows, so `optional_params` said the opposite in the same file. It was inherited from the mixin's default set and never examined. Nothing enforces the requirement today: `_check_inputs` substitutes the declared default before it tests `required`, and the template carries `default=50`, so omitting the input runs 50 steps rather than raising. That is not specific to H3 — 14 call sites on main are in the same position — and it is tracked in #14388. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Fix the two-card recipe, and stop accepting `output_type="latent"` The two-card section did not run, on this branch or on the base one. A pipeline resolves a single execution device for all of its components, so the documented shape — one pipeline whose conditioner is `device_map`ped to the second card and whose denoiser is moved to the first — built the rotary positions on `cuda:1` and handed them to a transformer on `cuda:0`: transformer_minimax_h3.py, in Rope.forward RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0! Reproduced against `abc5e9bf7` too, so it is not something the refactor broke; the section went in as a plausible recipe that was never executed. The split is therefore between pipelines rather than inside one: pop the text encoder block into a conditioner of its own, give each pipeline a `ComponentsManager` pinned to a card, and pass the state from one call to the other. The managers' hooks are what align `prompt_embeds` onto the denoiser's card; placing the components by hand instead works too, but then that one tensor has to be moved explicitly. Verified end to end against the tiny fixture. `output_type="latent"` is separate. It is not an output format — a request that wants latents runs a pipeline without the decode blocks — and it used to run the whole VAE decode before dying inside `postprocess_video` with "latent does not exist", which does not say what to do instead. The video decode block now rejects anything it cannot postprocess, before decoding. `test_output_type` covers the three formats it does accept, and the rejection is a `test_check_inputs` case. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Skip the workflow tests whose tiny repositories cannot serve them `test_from_pretrained_workflow`, `test_load_components_workflow` and `test_unload_components` build a pipeline from `pretrained_model_name_or_path` and compare it against the test class's own blocks. Three fixtures cannot answer that, in three different ways, and none of them is about the pipelines' code: - `tiny-anima-modular-pipe` has no `modular_model_index.json`. Anima assembles its dummy components in `get_pipeline` and never loads from a repository, so the path it declares had never been exercised. - `tiny-flux2-klein-modular` names `Flux2KleinBaseAutoBlocks` while its `_class_name` and `is_distilled` both say distilled. `from_pretrained` honours `_blocks_class_name`, so it builds base blocks, which declare a `guider` the distilled ones do not. - `tiny-qwenimage-edit-modular` names `QwenImageModularPipeline`, so `from_pretrained` falls back to `QwenImageAutoBlocks`, which declares no `image_conditioned` workflow. Skipped with a TODO each, since the fixes are Hub-side. The klein and qwenimage ones are a single key in `modular_model_index.json`; Anima needs a repository published. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Declare `reference_image_short_edge` as pipeline config `MiniMaxH3Ref2VASetupStep` resolves three pieces of released-checkpoint geometry — the canvas short edge, the canvas area cap, and the separate short edge an image reference is encoded at — and declared only the first two as `ConfigSpec`s. The third was a constructor argument, so it did not survive a save and reload: the reloaded pipeline rebuilt the block from the repository, took the 2048 default instead of the 64 the tests configure, resized reference images at a different resolution, and landed 0.398 away from the original output. All three now sit together, which is also what lets the test fixture stop reaching into the block tree to swap in a configured copy — the shrunken geometry is one `update_components` call next to the canvas rule it belongs with, and `test_reference_image_geometry` exercises the released default rather than constructing the block with an explicit 2048. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Accept a reference image in the three layouts a video reference takes `MiniMaxH3VideoReference.frames` takes a list of images, a channels-last array or a channels-first tensor; `MiniMaxH3ImageReference.image` took a PIL image only, and the two array layouts failed on `entry.image.size` — an element count on numpy, a bound method on torch — with `TypeError: 'int' object is not subscriptable`. Passing an array through as-is would have been worse than the crash. The resize would silently drop to `F.interpolate`'s nearest-neighbour where a PIL image gets LANCZOS, and the VAE encode does `np.array(image).permute(2, 0, 1)`, which fixes the axes of an image and scrambles those of a channels-first tensor. So the layouts are normalized onto a PIL image before any of that, through the processor component the block already declares: `pt_to_numpy` for the channel axis and `numpy_to_pil` for the array. Neither converts dtype, and `numpy_to_pil` scales by 255, so `uint8` is normalized on the original object first — after `pt_to_numpy` a `uint8` tensor is floats over `[0, 255]` and a later dtype check would miss it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Read the model's constants off the pipeline rather than the module `before_denoise.py` imported `MINIMAX_H3_AUDIO_CHANNELS`, `MINIMAX_H3_AUDIO_TAG` and `MINIMAX_H3_VIDEO_TAG` and read them inside its layout builders, even though two of the three were already exposed as pipeline properties and used that way everywhere else. They are arguments now, passed from `components.*` at the two `__call__` sites; `audio_tag` needed the property `text_tag` and `video_tag` already had, which is presumably why it was the one still reaching for the global. Same treatment where a helper cannot see a pipeline: `resolve_canvas_size` and `audio_latent_num_frames` take the constants as default arguments instead of closing over them, and `_normalize_video_condition` takes the rate it resamples onto, so `before_encoder.py` imports none of them at all. What is left of the constants is their definitions, the properties that expose them, and default argument values. Two helpers with a single caller each are inlined at it: the two rotary-time sums, which are a parity contract — the reference sums the same series pairwise in one place and sequentially in the other, and the orders differ in the last ulp from 16 latent frames onwards. Both inlined sums are bit-identical to the functions they replace across 1..399 latent frames, and the two still disagree on 373 of those counts. `_frame_position_grid` stays, and `build_packed_sequence` now calls it rather than repeating its five lines. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Reject a reference video too short for the conditioner to read The conditioner reads a reference video at 2 fps and Qwen3-VL merges the sampled frames in groups of two, so anything under 13 frames at 24 fps samples down to a single frame and dies inside transformers: image_processing_glm4v.py ValueError: t:1 must be larger than temporal_factor:2 which names neither the reference nor the rate that produced it, and only fires on the versions carrying that check — it passed locally and failed on CI. `_sample_video_condition_frames` now rejects it where the sampling happens, with the bound derived from the rates rather than hardcoded: "must run at least 13 frames at 24 fps (0.54 seconds), got 4". `test_reference_media_layouts` was building exactly such a reference, four frames, which is what surfaced this. It uses a second of video now, like `test_reference_combinations` already did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Drop the padded-layout attention mask `forward` built a boolean attention mask whenever a row carried a negative modality tag, reproducing the reference implementation's `cu_seqlens = [0, used, S]` split of the padding tail it adds for FlashAttention. Nothing in this port ever produces such a row: both packers partition `[0, sequence_length)` and write only the text, audio and video tags, so the branch was unreachable and the mask was never built. The `clamp(min=0)` that kept a `-1` from indexing the AdaLN table backwards goes with it. Removing it is not only dead-code removal. `if bool(is_pad.any())` is a data-dependent branch, which has no fullgraph representation, and it was the reason whole-model `fullgraph=True` compilation and `torch.export` were skipped: 4 passed, 1 skipped tests -k "compile or export or aot" Both now run. The model tests go from 40 passed / 4 skipped to 41 passed / 2 skipped, against the same 8 pre-existing memory-offload failures. `attention_mask` stays on the processor, attention and block signatures — it is the signature every other processor in diffusers has, and a custom one may want it — but the model itself never passes anything but `None`, so attention runs unmasked over the one document and every backend stays available. Two of the four documentation examples have been re-run against this and come back bit-identical, frames and audio: `t2va` at the full 768x1344 canvas and `fl2va`. The two `ref2va` ones are still to run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Apply suggestions from code review Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> --------- Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> |
||
Álvaro Somoza
|
8b33b47332 |
[quantization] SDNQ core loading (#14277)
* initial * fix code quality * dependency test fix * apply review suggestions * add extras and version check * fix gating and prefer local cache for modular * simplify transformers registration and make the version warning silenceable * remove pipeline level test * only model level testing * layers to quantizze, change em dashes * env flag for transformers --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
|
Steven Liu
|
351de3a6fe |
[docs] deprecate torch_dtype for dtype (#14313)
* docs
* update
* Revert "docs"
This reverts commit
|
||
|
Atharva Joshi
|
6ad357395d |
Cosmos3 Model Parallelism (#14054)
* feat(cosmos3): multi-GPU inference — context + tensor parallelism
Cosmos 3 cannot use diffusers' declarative `_cp_plan` CP path: it is grouped-query
attention (the shared Ulysses kernel assumes K/V share the query head count), its
understanding (causal) and generation (full) streams are separate packed sequences
(gen attends to cat(und, gen)), and per-pathway lengths are ragged. The model carries
no parallelism logic -- it exposes only small, CP-agnostic seams; all sharding lives
outside it, in a reusable example module.
Model (transformer_cosmos3.py): adds two default-None `forward` seams -- `_cp_shard_fn`
(shards und/gen + rotary before the decoder layers) and `_cp_gather_fn` (gathers/unpads
after the final norm) -- and extracts `Cosmos3AttnProcessor._run_attention` as an
override point. The non-parallel path is unchanged.
Helpers (examples/cosmos3/cosmos_parallel.py): one importable module, two orthogonal
and composable axes:
* Context parallelism (Ulysses) -- `enable_cosmos3_context_parallel`. Shards the
sequence; brackets the two attention pathways with all-to-all (DTensor redistribute),
repeats GQA KV heads, pads ragged lengths and masks padded generation keys.
* Tensor parallelism (Megatron) -- `enable_cosmos3_tensor_parallel`. Column/row-shards
the attention + MLP weights so a checkpoint that does not fit one GPU (Super, ~120 GB)
loads across several; weights load to CPU then shard layer by layer.
Both expand KV heads to the query-head count and call SDPA with enable_gqa=False so it
dispatches to the flash kernel; enable_gqa=True forces the math path, which materializes
the full [S, S] score matrix and OOMs on long videos. A dense `Cosmos3FlashAttnProcessor`
(`enable_cosmos3_flash_attention`) provides the same for TP without CP.
CLI (examples/cosmos3/inference_cosmos3.py): imports these helpers, so any modality
(text-to-image/video, image-to-video, sound, action) runs single- or multi-GPU via
`--tp-degree` / `--cp-degree` (their product must equal --nproc_per_node). Single-GPU
behavior is unchanged.
Docs + example README updated. Verified: CP attention core is bit-exact vs non-CP in
fp32 (max|d|=0), and a full 36-layer forward matches CP-on vs CP-off to ~1e-6 in fp32
(bf16 differs only by floating-point rounding).
* refactor(cosmos3): make CP/TP attention processors standalone
Address review feedback on PR #14054: the parallel attention processors no
longer subclass Cosmos3AttnProcessor, so the model file needs no override seam.
- transformer_cosmos3.py: revert Cosmos3AttnProcessor to inline the attention
in __call__ (remove the _run_attention seam); restores it to its base version.
- cosmos_parallel.py: Cosmos3CPAttnProcessor and Cosmos3FlashAttnProcessor each
get their own full __call__, sharing a _project_qkv_with_rope prologue helper.
Verified behavior-preserving on 4x RTX PRO 6000: cp_unit_test (fp32) passes at
1e-4; cp_numeric_check is byte-identical to the pre-refactor code; the end-to-end
CLI passes in CP-only, TP-only, and TP+CP modes.
* fix(cosmos3): support modular multi-GPU inference
|
||
Cedric Perauer
|
623213d29f |
Krea2 Modular Pipeline Support, Documentation, Test Cases (#14083)
* towards krea2 modular * add test cases * self-review fixes: test model id to hf internal, guider change * remove pnd * modular autodoc * modular autodoc * Update src/diffusers/modular_pipelines/krea2/modular_blocks_krea2.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Update src/diffusers/modular_pipelines/krea2/before_denoise.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Update docs/source/en/api/pipelines/krea2.md Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * input params, dim multiple of scale factor * patch * docs for krea2 turbo modular * docs fix * Update tests/modular_pipelines/krea2/test_modular_pipeline_krea2.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Update src/diffusers/modular_pipelines/krea2/before_denoise.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * modular blocks krea2 turbo fix * Update docs/source/en/api/pipelines/krea2.md Co-authored-by: YiYi Xu <yixu310@gmail.com> * Update src/diffusers/modular_pipelines/krea2/before_denoise.py Co-authored-by: YiYi Xu <yixu310@gmail.com> * fix default height/width and inference time steps * Apply suggestions from code review Co-authored-by: YiYi Xu <yixu310@gmail.com> * Regenerate stale krea2 auto docstrings Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
Chenyang Zhu
|
9f6fc2c136 |
Add ace step lora support (#14193)
* feat: add LoRA support to AceStepPipeline Add standard diffusers LoRA API (load_lora_weights, set_adapters, fuse_lora, save_lora_weights) for AceStepPipeline, targeting the transformer (AceStepTransformer1DModel) only. Includes automatic conversion of ACE-Step-1.5 original PEFT format LoRA weights (q_proj/k_proj/v_proj/o_proj naming) to diffusers format (to_q/to_k/to_v/to_out.0 naming) during loading. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use HuggingFace Hub ID for tokenizer in AceStep LoRA test and apply ruff formatting - Replace hardcoded local path with "Qwen/Qwen3-Embedding-0.6B" so tests run on CI - Apply ruff formatting fix (slice spacing) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: override test_lora_fuse_nan instead of skipping it The base test hardcodes block attribute names (transformer_blocks, blocks, etc.) but AceStep uses 'layers'. Override with AceStep-specific block access path instead of skipping. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add joint_attention_kwargs to AceStepPipeline for per-step LoRA scale - Add @apply_lora_scale("joint_attention_kwargs") decorator and joint_attention_kwargs param to AceStepTransformer1DModel.forward - Add joint_attention_kwargs param to AceStepPipeline.__call__ and wire it through to all three transformer calls - Un-skip 4 LoRA scale tests that now pass (39 passed, 13 skipped) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: override test_lora_fuse_nan instead of skipping it The base test hardcodes block attribute names (transformer_blocks, blocks, etc.) but AceStep uses 'layers'. Override with AceStep-specific block access path instead of skipping. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add joint_attention_kwargs to AceStepPipeline for per-step LoRA scale - Add @apply_lora_scale("joint_attention_kwargs") decorator and joint_attention_kwargs param to AceStepTransformer1DModel.forward - Add joint_attention_kwargs param to AceStepPipeline.__call__ and wire it through to all three transformer calls - Un-skip 4 LoRA scale tests that now pass (39 passed, 13 skipped) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: use attention_kwargs instead of joint_attention_kwargs, use torch_device in test - Rename joint_attention_kwargs to attention_kwargs in transformer forward and pipeline __call__ for consistency with the diffusers naming convention for new pipelines - Add docstring for the attention_kwargs parameter - Use torch_device instead of hardcoded "cpu" in test_lora_fuse_nan Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Apply style fixes * docs: document ACE-Step attention kwargs * docs: document ACE-Step LoRA loader mixin --------- Co-authored-by: zcy <zhuchenyang.zcy@alibaba-inc.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
yzhautouskay
|
8eac6ad1c4 |
Cosmos3 Distilled support (#14177)
* Cosmos3 distilled support (dedicated modular pipeline + blocks) * Address review comments * Refactor unit test to use ModularPipelineTesterMixin * Docs fix * Tests refactor * Pass generator to SDE scheduler for reproducibility * Apply suggestion from @yiyixuxu * Distilled text encoder step bugfix --------- Co-authored-by: YiYi Xu <yixu310@gmail.com> |
||
|
yzhautouskay
|
e29a9ce631 |
Transfer support for Cosmos3 ModularPipeline (#14150)
* Add transfer support to the Cosmos3 modular pipeline * Convention matches fixes * Align Cosmos3 transfer blocks with modular conventions * Move Cosmos3TransferSetupStep to before_encoder.py * Decouple Cosmos3 transfer blocks from task-pipeline internals * Temporarily disable standalone transfer workflow * Auto docstring run fix * Apply style fixes --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
Vinh H. Pham
|
60ec6f7242 |
Add Nunchaku Lite single-file quantization (#14100)
* Add Nunchaku Lite single-file quantization * Support config-backed Nunchaku Lite loading * Remove Nunchaku runtime manifest metadata loading * Simplify Nunchaku compact config loading * Add Nunchaku Lite quantization tests * Document Nunchaku Lite checkpoint loading * Refine Nunchaku Lite quantization docs * Remove unused Nunchaku smooth factor original weights * Update docs/source/en/quantization/nunchaku.md Co-authored-by: sergereview[bot] <283583894+sergereview[bot]@users.noreply.github.com> * Update src/diffusers/quantizers/nunchaku/utils.py Co-authored-by: sergereview[bot] <283583894+sergereview[bot]@users.noreply.github.com> * Update src/diffusers/quantizers/nunchaku/nunchaku_quantizer.py Co-authored-by: sergereview[bot] <283583894+sergereview[bot]@users.noreply.github.com> * Address Nunchaku Lite review feedback * Update Nunchaku Lite quantizer * Rename Nunchaku Lite fp4 precision to nvfp4 * Allow torch_dtype to override Nunchaku compute dtype * Keep Nunchaku replacement modules on meta during loading * Require kernels when importing Nunchaku runtime utils * Use kernels requirement decorators for Nunchaku tests * Validate Nunchaku CUDA capability requirements * Apply suggestions from code review Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * docs: update nunchaku quantization guide * test: remove nunchaku quantization tests * fix: reject hopper for nunchaku lite * Apply suggestion from @rootonchair * docs: update nunchaku lite compile example * Apply style fixes * Apply suggestions from code review Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> Co-authored-by: Vinh H. Pham <phamvinh257@gmail.com> * update doc to be more reasonable * fix import that requires kernels --------- Co-authored-by: sergereview[bot] <283583894+sergereview[bot]@users.noreply.github.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> |