* refactor a* pipeline tests.
* [tests] fix expected slice in the sparsectrl dict/tuple test
The `test_dict_tuple_outputs_equivalent` override ported in e10f8d08e carried a
nine-value expected slice that is not the one the pre-migration file used, so CI
failed on CPU with `Shape mismatch: actual torch.Size([6]) vs expected
torch.Size([9])`.
Six is correct: the base test slices `output_tuple[0, -3:, -3:, -1, -1]` and this
pipeline's dummy inputs use `num_frames=2`, so the frame axis yields 2 rather
than 3. Restored the original values, unchanged since #14120.
The literal is only read when `torch_device == "cpu"`, which is why local GPU
runs did not catch it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor lumina2 pipeline tests to the new mixin structure
* move lumina2 lora tests to the pipeline-level mixins
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Drop callback tensor inputs that are never bound in the denoising loop
LEditsPPPipelineStableDiffusion allows prompt_embeds, LEditsPPPipelineStableDiffusionXL
allows negative_add_time_ids and StableDiffusionXLControlNetUnionInpaintPipeline allows
masked_image_latents, but none of those names exist as locals in the respective __call__,
so callback_kwargs[k] = locals()[k] raises KeyError as soon as a user requests them.
Add a test_callback_inputs fast test to both LEdits++ test classes, which are plain
unittest.TestCase and therefore never ran the PipelineTesterMixin callback tests.
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* refactor zimage and krea2 pipeline tests
* remove group offloading skip and propaget the refactors to the rest of zimage
* add negative prompt
---------
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* refactor deepfloyd if img2img pipeline tests to the new mixin structure
* drop float16 tolerance override
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* Move LTX2Guidance from /modular_pipelines/ltx2/ to guiders/ to allow LTX-2 guiders to be correctly loaded from checkpoint
* Add regression test for LTX-2 guiders declared as from_pretrained components
Covers the round trip this move fixes: a checkpoint that declares `guider` and
`audio_guider` in `modular_model_index.json` must record a resolvable
(library, class_name) pair and reload its non-default guidance scales. The
existing `test_modular_index_consistency` skips components without a
`pretrained_model_name_or_path`, so `from_config` guiders were uncovered.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Migrate LTX2Guidance docs to guider docs after move
* Improve LTX2Guidance docs and add arg docstrings
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* skip hunyuan framepack group offloading tests
* exclude only the SigLIP image encoder from leaf-level group offloading
`test_pipeline_level_group_offloading_inference` was skipped outright for
HunyuanVideoFramepack because `image_encoder` is a `SiglipVisionModel`, whose
attention pooling head wraps a `torch.nn.MultiheadAttention`. That hands
`self.out_proj.weight` to `torch.nn.functional.multi_head_attention_forward`
instead of calling `self.out_proj`, so the leaf-level onload hook on `out_proj`
never fires and its weights stay on the offload device.
Add a `group_offloading_leaf_level_exclude_modules` knob to the old-style
`PipelineTesterMixin` and the new-style `BasePipelineTesterConfig` (empty by
default, so no behavior change elsewhere), pass it through to
`enable_group_offload(exclude_modules=...)` in both implementations of the test,
and set it to `["image_encoder"]` for framepack instead of skipping. Block-level
offloading is unaffected — the whole head is onloaded as one unmatched module —
hence the level in the name.
The test now passes and covers leaf-level offloading of the transformer, VAE and
both text encoders. The VAE is coverage nothing else provided:
`test_group_offloading_inference` deliberately excludes `vae` and
`image_encoder`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* docs: note how to handle a component that breaks leaf-level offloading
Records the rule behind `group_offloading_leaf_level_exclude_modules`: leaf-level
offloading hooks only the supported leaf types and onloads each on its own
forward, so any code that reads a leaf's `.weight` instead of calling the leaf
bypasses that hook.
Routes the fix by who owns the component. A diffusers model declares the gap with
`_supports_group_offloading = False` on the `ModelMixin` subclass, which both
offload mixins honor. A third-party component that can't be annotated goes in
`group_offloading_leaf_level_exclude_modules`, which keeps offload coverage for
every other component — where a hand-written skip would drop it for the whole
pipeline, the VAE included, since the component-scoped
`test_group_offloading_inference` deliberately excludes it.
`torch.nn.MultiheadAttention` is called out as the common instance rather than as
the definition, with `HunyuanDiTAttentionPool` as a case that fails the same way
with no MHA module involved, so the guidance still applies when a future
component fails for a different reason.
Also notes that a failure should be reproduced before a skip or exclusion is
added: of the five pipelines currently skipping the pipeline-level test, only
framepack and motif_video still fail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Drop old style test info and simplify leaf level group offloading testing.md note.
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Fibo Edit: multi-reference conditioning, batching, timesteps
* docs: simplify Fibo Edit page, fix EXAMPLE_DOC_STRING to the VLM prompt flow
* docs: point Fibo Edit at the Gemini prompt-to-JSON block and the 1.5 base/turbo ids
briaai/FIBO-edit-prompt-to-JSON is retired in favor of
briaai/FIBO-edit-gemini-prompt-to-JSON, which handles multiple reference
images and masks. Also corrects the checkpoint ids to their canonical
briaai/Fibo-Edit-1.5-base and briaai/Fibo-Edit-1.5-turbo spellings.
* apply review feedback on reference handling and the attention mask
- collapse the mask branch to `if not attention_mask.all():` in both Fibo pipelines
- encode references via `self.image_processor.preprocess(...)`, dropping the manual
numpy normalization and the `_vae_safe_size` helper
- normalize `image` into a reference list once in `__call__`, pass it to `check_inputs`
* make style
* make style
TestAutoencoderVidTokSlicingTiling overrode test_enable_disable_tiling and
test_enable_disable_slicing with copies that ran their three forward passes
outside torch.no_grad(). The retained autograd graphs pushed each test to a
10.9 GiB peak on a 14.74 GiB T4, so with any memory left over from earlier
files in the same xdist worker the class OOM'd -- taking
test_forward_with_norm_groups down with it.
The mixin already implements both tests correctly (no_grad, plus torch.allclose
for the disable-comparison instead of the `.all() == .all()` idiom the copies
used, which compares two booleans and always passes). Drop the overrides and
inherit them.
Peak memory on a T4, measured per test:
before 10937.3 MiB
after 1666.5 MiB
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor bria pipeline tests to the new mixin structure
* add lora tester class
* remove lora tester class
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* restructure guides as skill references
* reorganize AGENTS.md and references
* clean up
* clean up
* clean up
* fix AGENTS.md references and add check_ai to Makefile
* clean up
* update paths
* clean up
* draft:add neuron as a legit backend
* feat: neuron-specific changes in the pipeline
* tests: eager tests
* draft: start with tp for flux2
* fix: style
* fix:apr_02 beta
* feat:add wan
* fix:pixart
* fix: rewrite flux swiglu activation to avoid gather op in neuron IR
* test: pixart compile mode on neuron
* cleanup & fix style
* merge: another change
* review: cleanup+suggestions
* fix: CIs style
* tests: add test units for tp
* fix: in case of text-encoder(s) on CPU
* review:cleanup+add test
* fix: style
* doc: remove it for now
* Add from_single_file support for SkyReelsV2 and ChronoEdit transformers (#13946)
SkyReels-V2 and ChronoEdit are both built on Wan, and their transformers have
the same keys as WanTransformer3DModel, so they reuse
convert_wan_transformer_to_diffusers (like WanVACE / WanAnimate). This lets the
community GGUF builds load directly.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* multi-GPU VAE Fix for Cosmos 3 (#13924)
fix(cosmos3): pin VAE latent norm buffers to encode output device
Under sharded placement (device_map="balanced"), vae.encode() runs on the
VAE's own device while the mean/inv_std buffers were pinned to x.device,
causing a cross-device RuntimeError. Compute raw_mu first, then pin the
normalization buffers to its device so all tensors share one device.
Co-authored-by: Atharva Joshi <atjoshi@smc521ge-0036.ipp2a2.colossus.nvidia.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* docs: fix repeated word typo in set_timesteps docstring (#13876)
* docs: fix repeated word typo in set_timesteps docstring
Removed the duplicate word "schedule" from the docstring for the sigmas argument in EulerDiscreteScheduler.set_timesteps.
* Update scheduling_euler_discrete.py
* Apply style fixes
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* clean some stuff to simplify code.
* clean more to remove permutation related shenanigans.
* revert: put torch.chunk back
* Update docs/source/en/training/distributed_inference.md
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* Address Sayak's TP review nits (docstrings + stale comments)
- _modeling_parallel: fix stale comment claiming attention processors read
tp_degree at runtime (no longer true after the PackedColwise/Rowwise
refactor; processors are TP-agnostic via head_dim).
- tensor_parallel: single backticks instead of double in docstrings, add
input/output examples to _blocks_to_block_sizes and _resolve_tp_plan, and
drop the stale "applies the Flux2 fused-weight permutations" line from
apply_tensor_parallel (permuters were removed).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix stale permuter reference in Flux2ParallelSelfAttnProcessor comment
The proportional QKV/MLP slice is now guaranteed by PackedColwiseParallel
block-by-block sharding, not the removed fused-weight permuter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Derive TP backend inside apply_tensor_parallel from the mesh device type
Drops the `backend` parameter: apply_tensor_parallel already holds the TP
mesh, so it derives neuron-vs-default from `config._mesh.device_type` itself.
The caller no longer computes it. Documents why torch_device can't be used
(Neuron reports as "cpu" via torch's accelerator API). Removes the now-unused
is_torch_neuronx_available import from modeling_utils.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Simplify TP backend derivation to the mesh device type alone
Drop the redundant is_torch_neuronx_available() guard: a DeviceMesh("neuron", ...)
only exists when running on Neuron, so tp_mesh.device_type is the single source
of truth.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Remove incorrect "Neuron reports as cpu" rationale from TP backend docstring
Verified on the current native torch_neuronx drop that Neuron does surface as
the torch accelerator (torch._C._get_accelerator().type == "neuron",
get_device() == "neuron"); the old "reports as cpu" claim was stale. The
mesh-device-type derivation is unchanged — it's the device being sharded onto.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drop stale "XLA" labels from Neuron device/runtime comments
Neuron is a native torch backend now (not XLA-routed): remove the "(XLA)"
parenthetical from the randn-on-CPU note in torch_utils and the "XLA runtime"
mention from the Neuron TP test docstring. Behavior unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* make style: reflow tensor_parallel docstrings
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Revert "revert: put torch.chunk back"
This reverts commit 7ea75f771a.
* fix: sharding for neuron + validate flux 1
* fix: change for check_repository_consistency
* feat: support qwen image as well
* Update docs/source/en/training/distributed_inference.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* review: keep bria and nucleusmoe out
* review: keep bria and nucleusmoe out
* review: keep bria and nucleusmoe out
* review: address agent review
* review: apply suggestions
* Update docs/source/en/training/distributed_inference.md
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* review: change example tp size to 4
* review: add flux tp test
* review: address comments
* review: improve the doc
* review:apply qwen rotary_emb per device
* review:remove redundant device def after PR #14383
* review: doc title suggestion
* review: restore copy comment
* review: remove cp divisibility explanation from doc on parallelism guide
* fix style
---------
Co-authored-by: Hz_Zhang <47402297+HaozheZhang6@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Atharva Joshi <atjoshi@nvidia.com>
Co-authored-by: Atharva Joshi <atjoshi@smc521ge-0036.ipp2a2.colossus.nvidia.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Ramkumar R <ramkumarashvanth09@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Stable Audio 3 First Version Commit, require model dict loading on GPUs
* Fixing bugs for sa3 VAE
* Make quality fix
* Fix the unit test failure
* Fix SA3 pipeline noise: learned text-padding embedding + no cross-attn mask
The diffusers Stable Audio 3 pipeline produced noise instead of music because
the cross-attention conditioning was built differently from the reference:
- The reference text conditioner uses padding_mode="learned": padded text
positions (~245 of 256 for a short prompt) are filled with a trained
`padding_embedding`, and the DiT attends to *all* positions (its
cross-attention mask is intentionally disabled). Our pipeline instead zeroed
padded positions and masked them out, wiping ~95% of the conditioning signal.
Changes:
- transformer_stable_audio3: add learned `prompt_padding_embedding`; in forward,
replace masked text positions with it (in cond_token_dim space, before
to_cond_embed) then attend to the full context, matching the reference.
- pipeline_stable_audio_3: stop zeroing padded positions (the DiT now handles
them); default `silence_padding_duration` to 6.0 (reference headroom default).
- convert_..._to_diffusers: convert
`conditioner.conditioners.prompt.padding_embedding` into the DiT.
- scheduling_ping_pong: pin schedule endpoints (sigmas[0]=1.0, sigmas[-1]=0.0)
to match the reference LogSNRShift endpoint preservation.
- run_..._inference: coerce float16->float32 on CPU.
- tests: update transformer state-dict expectations (523 tensors, add
prompt_padding_embedding) and ping-pong sigma endpoint assertions.
Verified against the reference with identical noise + identical conditioning:
full 8-step ping-pong trajectories agree to 4e-6 (final latent). Removed the
dev-only parity scripts (verify_*_dit_parity, verify_*_vae_parity) from the tree.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Style fix
* Autopipeline available
* Adding document description for testing examples.
* Solving fix for serge review bot suggestion
* Design review comments fix
* CI/CD Fix for ping pong scheduler md file
* Fix doc build: remove autodoc for nonexistent StableAudio3EulerSchedulerOutput
* Fix doc build: qualify non-exported autodoc references with module paths
* Fix details in scheduler and some nits
* suggestion fix
* fix-copies: propagate prepare_latents dtype fix to audio2audio/inpaint pipelines
---------
Co-authored-by: Buffett Liu <jengliu@linkedin.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* [CLI] Clean up and improvements to `diffusers-cli`
Modular pipelines:
- `custom_blocks` also writes `modular_model_index.json`, so a packaged repo is
loadable by `ModularPipeline.from_pretrained` (and runnable by `run`), not just
importable as blocks.
- Add `--cpu-offload auto`, which offloads through the pipeline's
`ComponentsManager`, plus `--offload-margin` to set its `memory_reserve_margin`.
- Add `--workflow` passthrough; warn and ignore it on standard pipelines.
- Route `model_index.json` repos whose `_class_name` is modular to the modular path.
Output saving:
- Resolve every media field on the output object (`images`, `frames`, `audios`,
`audio`) instead of the first match, keeping the batch dimension. Fixes video
output being misrouted through the image branch and LTX2's audio being dropped.
- Fix the `output_type="pt"` video transpose: `postprocess_video` returns
`(B, F, C, H, W)`, not `(B, C, F, H, W)`.
- Save `(B, C, samples)` waveforms as multi-channel wavs (StableAudio).
- Write each video's frames to `<stem>-frames/<NNNN>.png`. Frames need no video
backend, so they replace the `.pt` fallback when `export_to_video` fails.
- Report frame directories as a single path so hundreds of frames don't flood the
terminal; `--push-to` and the sandbox download preserve subfolder structure.
- Raise on unrecognized payloads instead of dumping JSON/pickle.
Remote:
- Add `imageio`, `imageio-ffmpeg`, and `peft` to the default sandbox deps.
- Raise sandbox `start_timeout` to 600s; the 120s default expires while a cold node
pulls the pytorch base image.
* Apply suggestion from @sayakpaul
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* update
* update
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
