mirror of
https://github.com/huggingface/diffusers.git
synced 2026-10-24 20:59:27 +03:00
5d10b4de3b65b1debb8a94de3a0e3bfa51fc628a
1777 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Dhruv Nair
|
25b85c1d0b |
[CI] Refactor Chronoedit, PRX, EasyAnimate, Ovis transformer tests (#13347)
* update * update --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
|
Dhruv Nair
|
d6c360d2d2 |
[CI] Refactor Bria Transformer Tests (#13341)
* update * update * update * update --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
|
Dhruv Nair
|
3b80bcf4ad |
[CI] Refactor LTX Transformer Tests (#13254)
* update * update --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
Sayak Paul
|
c33abfcc8d |
[tests] fix anyflow tests (#13855)
* fix anyflow tests * [tests] fix anyflow tests layerwise casting (#13863) Fix AnyFlow FAR causal transformer training layerwise / mixed precision tests --------- Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> |
||
|
sayakpaul
|
b3ec48148c |
Revert "fix kvae gradient checkpointing tests"
This reverts commit
|
||
|
sayakpaul
|
52ecdb07eb | fix kvae gradient checkpointing tests | ||
apolinário
|
9b0818cf87 |
Add Ideogram 4 (#13859)
* Add Ideogram 4 Adds the Ideogram 4 text-to-image model: transformer, standard pipeline, modular pipeline, docs, and tests. Checkpoint: ideogram-ai/ideogram-4-nf4 Co-Authored-By: YiYi Xu <yiyi@huggingface.co> * Use split q/k/v projections in Ideogram4 attention Replace the fused `qkv`/`o` linears with canonical `to_q`/`to_k`/`to_v`/`to_out` projections, matching the standard diffusers attention layout and the split checkpoint format. Mathematically equivalent to the fused form (q/k/v are contiguous row-slices of the fused weight). Drops the now-inapplicable fuse/unfuse overrides. --------- Co-authored-by: Jin <jin.li@ideogram.ai> Co-authored-by: YiYi Xu <yiyi@huggingface.co> |
||
|
Dhruv Nair
|
f34de4330e |
[CI] Fix torch_device import in AutoencoderTesterMixin (#13852)
update |
||
Wang, Yi
|
525d483add |
updatge the test marigold to make it pass in xpu (#13856)
Signed-off-by: Wang, Yi <yi.a.wang@intel.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
Akshan Krithick
|
4c77dcdbac |
refactor autoencoder tests (vq, kvae_video, oobleck, consistency_decoder, tiny, vidtok) (#13849)
* refactor vq tests * refactor autoencoder_kl_kvae_video tests * refactor autoencoder_oobleck tests * refactor consistency_decoder_vae tests * refactor autoencoder_tiny tests * refactor autoencoder_vidtok tests * remove unused base_precision and test_outputs_equivalence skips --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
|
Akshan Krithick
|
334ef1aa1f |
refactor autoencoder_kl_cogvideox tests (#13840)
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
|
Akshan Krithick
|
2875896e39 |
refactor autoencoder_hunyuan_video tests (#13835)
* refactor autoencoder_hunyuan_video tests * remove unused base_precision and test_outputs_equivalence skip --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
|
Akshan Krithick
|
5385b2ae19 |
refactor autoencoder_magvit tests (#13834)
* refactor autoencoder_magvit tests * remove unused base_precision --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
Jingya HUANG
|
1d61993458 |
[Neuron] Add AWS Neuron (Trainium/Inferentia) as an officially supported device (#13289)
* draft:add neuron as a legit backend * feat: neuron-specific changes in the pipeline * tests: eager tests * fix: style * fix:apr_02 beta * cleanup: remove tp part, for another pr * fix: restore ring_anything to ContextParallelConfig after over-aggressive TP cleanup The previous cleanup commit removed TensorParallelConfig (correct) but also accidentally removed ring_anything from ContextParallelConfig (incorrect). ring_anything is a context-parallel feature referenced in context_parallel.py and must remain in the config. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * removal: style fix * tests:sdxl + flux2 * tests: simplify * tests: simplify * fix style * fix style * fix style for doc-builder * review: address comments * review:apply suggestion for the fix of index_for_timtestep * review: stronger guard on image slice * Apply suggestions from code review Co-authored-by: YiYi Xu <yixu310@gmail.com> * fix: when set_begin_index not implemented for scheduler * review:add maybe_adjust_dtype_for_device and apply to all models with downcasting needs * fix: dependency * Update src/diffusers/pipelines/stable_diffusion_xl/pipeline_stable_diffusion_xl.py Co-authored-by: YiYi Xu <yixu310@gmail.com> * review: apply maybe_adjust_dtype_for_device in pixart pipe * review: update .ai/models.md --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: YiYi Xu <yixu310@gmail.com> |
||
|
Sayak Paul
|
ed0711878c |
[tests] refactor caching tests. (#13235)
* refactor magcache tests. * include taylorseer in the caching mixin. * up * add back magcache and migrate to pytest --------- Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> |
||
|
Sayak Paul
|
2fa9b93c4d |
[tests] migrate group offloading tests to pytest (#13234)
* migrate group offloading tests to pytest * fix tests. --------- Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> |
||
|
Akshan Krithick
|
6dbf6e0651 |
refactor autoencoder tests (asymmetric_kl, ltx_video) (#13845)
* refactor asymmetric_autoencoder_kl tests * refactor autoencoder_ltx_video tests --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
rmatif
|
c943837899 |
Add Anima modular pipeline (#13732)
* Add Anima pipeline * Fix empty Anima negative prompts * Fix Anima registration * Clean up Anima conditioner * Refactor Anima to modular * Use modular loader in Anima docs * Move Anima text conditioner * Apply style fixes * Address Anima review nits * Fix Anima rotary autocast * Clean up Anima LoRA checks * Update Anima model id * Document Anima LoRA loader * Document Anima conditioner forward --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: YiYi Xu <yixu310@gmail.com> |
||
|
Akshan Krithick
|
e3fc602e36 |
refactor sana transformer tests (#13826)
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
Guian Fang
|
d94ad81b87 |
[AnyFlow] FAR: standalone causal-mask builder + torch.compile follow-up (#13792)
* [AnyFlow] FAR: standalone causal-mask builder + torch.compile follow-up
Follow-up to #13745. Extracts FAR mask construction to a module-level
helper and adds an `attention_mask` forward kwarg so
AnyFlowFARTransformer3DModel can be wrapped in
`torch.compile(fullgraph=True)`. The pipeline pre-builds the mask during
KV-cache prefill so users get end-to-end fullgraph compile.
* Public method `AnyFlowFARTransformer3DModel.build_attention_mask(...)`
(modes: "train", "cache") plus private module-level helper
`_build_anyflow_far_causal_block_mask(...)`.
* `_build_freqs` cache lookup/write bypassed under
`torch.compiler.is_compiling()` to avoid a Dynamo guard recompile on
the second compiled call (applied in bidi source; synced to FAR via
`# Copied from`).
* `TestAnyFlowFARTransformer3DCompile(TorchCompileTesterMixin)` —
recompilation_and_graph_break, repeated_blocks, and group_offloading
pass on H200; AOT is `@pytest.mark.skip`'d (torch.export rejects
BlockMask as a pytree input).
* Base `get_dummy_inputs` omits `attention_mask` so every non-compile
test class exercises the in-forward fallback; the compile class
overrides to inject a pre-built mask.
* Bit-exact: pre-built path vs internal-build fallback max|Δ|=0.0e+00.
* [AnyFlow] docs: full author list, repo demo examples, slimmer pipeline page
* Full author list and NVIDIA → NUS → MIT institution order; TL;DR +
abstract + Available Models bullets.
* Rewritten pipeline-selection tip describing both pipelines symmetrically.
* T2V / I2V / V2V examples now use the canonical 81-frame setup and the
demo prompts / conditioning assets shipped under
`NVlabs/AnyFlow/assets/evaluation/` (linked via raw.githubusercontent.com).
* Drop the inline "Optimizing Memory" and "torch.compile" sections — those
notes will live in the NVlabs/AnyFlow repo's own performance guide rather
than the diffusers pipeline reference.
* Sync zh user guide and the two model-API stubs.
* [AnyFlow] FAR: move chunk_partition default into transformer config
- AnyFlowFARTransformer3DModel.__init__ now accepts chunk_partition via
@register_to_config (default (1, 3, 3, 3, 3, 3, 3, 2) for the released
81-frame checkpoints, matching the field on Hub).
- AnyFlowFARPipeline.__call__ no longer requires chunk_partition; defaults
to self.transformer.config.chunk_partition. Per-call override still
supported for V2V / non-default num_frames.
- Drop the AnyFlowFARPipeline.default_chunk_partition class attribute.
- Update docs (en pipelines/models, zh using-diffusers) and the conversion
script to match.
* [AnyFlow] FAR pipeline: fix `timesteps` shadowing across chunks
Inside the per-chunk rollout loop, the local variable `timesteps` was
reassigned to `self.scheduler.timesteps` after `set_timesteps()`. On the
next chunk iteration the same name was passed back into
`set_timesteps(timesteps=...)`, where a non-None value enters the
*custom-schedule* branch — `apply_shift` re-runs on already-shifted
values, double-shifting the schedule for every chunk after the first.
Concretely, with `shift=5` and `num_inference_steps=4`:
- chunk 0 timesteps: [1000, 937.5, 833.3, 625] (correct)
- chunk 1+ timesteps: [1000, 986.8, 961.3, 892.9] (double-shifted)
The later steps drift toward `t=1000` instead of toward `t=0`, the
flow-map model is conditioned on the wrong source sigma, and the chunk
KV cache accumulates errors that show up as artifacts in later video
frames.
Fix: rebind the cached schedule to a fresh local name
(`scheduler_timesteps`) so the outer-scope `timesteps` kwarg (the
user-provided custom schedule, when any) stays untouched across chunks.
Layer-by-layer verification against the NVlabs reference implementation
on H200 (elephant prompt, seed 0, 4 NFE, 81 frames):
- chunk 0 inference: bit-exact (0.0 mean diff)
- chunk 1 step 0: 0.194 → 0.014 (-93%)
- chunk 7 last step: 0.564 → 0.274 (-51%)
* [AnyFlow] FAR: doc-builder line wrap for chunk_partition docstrings
Pure rewrap to satisfy `doc-builder style --max_len 119`. Two docstrings
introduced in
|
||
|
Wang, Yi
|
5ec96c3be9 |
This PR updates the Stable Diffusion IP-Adapter integration (#13810)
* This PR updates the Stable Diffusion IP-Adapter integration tests to keep them passing across PyTorch versions. These cases pass on PyTorch 2.11, but fail on PyTorch 2.12 because torch.randn no longer produces the same values for the same seed, which changes the generated latents and breaks the expected image slices. Signed-off-by: Wang, Yi <yi.a.wang@intel.com> * adjust xpu expectation Signed-off-by: Wang, Yi <yi.a.wang@intel.com> --------- Signed-off-by: Wang, Yi <yi.a.wang@intel.com> Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> |
||
|
Guian Fang
|
e39aecff57 |
Add AnyFlow Any-Step Video Diffusion Pipelines (Bidirectional + FAR Causal) (#13745)
* [Pipelines] AnyFlow: scaffold pipelines/anyflow + register all top-level imports
This is the lazy-loader scaffolding only. Body files (pipeline_anyflow.py,
pipeline_anyflow_causal.py, transformer_anyflow.py,
scheduling_flow_map_euler_discrete.py) come in subsequent commits.
* [Schedulers] AnyFlow: add FlowMapEulerDiscreteScheduler
The flow-map scheduler advances samples from timestep t to caller-provided
target r in a single Euler step, supporting any-step sampling on flow-map-
distilled checkpoints. It is a general-purpose scheduler — not specific to the
AnyFlow checkpoints.
Tests: 12 standalone tests covering instantiation, set_timesteps endpoints,
shift identity/monotonicity, step shape preservation, zero-interval identity,
one-shot sampling, train weight schemes, scale_noise endpoints.
Docs: api/schedulers/flow_map_euler_discrete.md
* [Models] AnyFlow: add AnyFlowTransformer3DModel
A 3D DiT extending the v0.35.1 Wan2.1 backbone with two config-toggled modules:
* FAR causal blocks (init_far_model=True): block-sparse causal attention via
flex_attention + compressed-frame patch embedding for frame-level
autoregressive generation (Gu et al., 2025, arXiv:2503.19325).
* Dual-timestep flow-map embedding (init_flowmap_model=True): adds a delta
timestep embedder enabling flow-map sampling z_t -> z_r over arbitrary
intervals (AnyFlow).
With both flags off, the model reduces to stock Wan2.1.
The class is intentionally self-contained rather than annotated with
'# Copied from diffusers.models.transformers.transformer_wan' because upstream
Wan has been refactored extensively since v0.35.1 (new WanAttention class,
different processor architecture).
Tests: 9 unit tests covering construction in 3 modes, bidi forward shape and
determinism, return_dict variants, save/load round-trip with and without
init_far_model, gradient checkpointing toggle.
Docs: api/models/anyflow_transformer3d.md
* [Pipelines] AnyFlow: add AnyFlowPipeline and AnyFlowCausalPipeline
* AnyFlowPipeline (pipeline_anyflow.py, ~590 LOC): bidirectional T2V using
flow-map sampling. Loads checkpoints from nvidia/AnyFlow-Wan2.1-T2V-{1.3B,14B}.
* AnyFlowCausalPipeline (pipeline_anyflow_causal.py, ~700 LOC): FAR-based
causal pipeline supporting T2V/I2V/TV2V via task_type kwarg. Loads checkpoints
from nvidia/AnyFlow-FAR-Wan2.1-{1.3B,14B}-Diffusers.
Both pipelines reuse stock WanLoraLoaderMixin, AutoencoderKLWan, UMT5EncoderModel,
and AutoTokenizer from upstream. The transformer is the AnyFlowTransformer3DModel
introduced in the previous commit. The scheduler is FlowMapEulerDiscreteScheduler.
Tests:
* tests/pipelines/anyflow/test_anyflow.py: PipelineTesterMixin fast tests +
slow integration test against nvidia/AnyFlow-Wan2.1-T2V-1.3B-Diffusers.
* tests/pipelines/anyflow/test_anyflow_causal.py: same structure for FAR variant.
Reference slices for slow integration tests are deferred to Phase 7
(Final quality pass) where the user runs them on a real GPU.
* [Docs] AnyFlow: add main pipeline documentation page
Modeled on the Helios pipeline doc (PR #13208). Sections: paper link + abstract,
supported checkpoints table, memory/speed optimization tabs, T2V/I2V/TV2V
examples for both bidirectional and causal variants, autodoc trailers.
* [Auto/Scripts] AnyFlow: register AutoPipelineForText2Video + add conversion script
* Register AnyFlowPipeline in AUTO_TEXT2VIDEO_PIPELINES_MAPPING.
* AnyFlowCausalPipeline is intentionally NOT registered for AutoPipeline because
its task switch (t2v / i2v / tv2v) is too rich for a single auto-resolve key.
* scripts/convert_anyflow_to_diffusers.py: convert .pt training checkpoints
(with 'ema' state dict) into a diffusers save_pretrained layout. Supports all
4 released NVIDIA AnyFlow variants. Replaces the omegaconf-based config in the
upstream repo with argparse to match other diffusers conversion scripts.
* [Quality] AnyFlow: ruff-format + regenerated dummy stubs
* ruff format pass on all 5 source files (long lines + trailing comma fixes)
* check_dummies.py --fix_and_overwrite regenerated:
- dummy_pt_objects.py: AnyFlowTransformer3DModel + FlowMapEulerDiscreteScheduler
- dummy_torch_and_transformers_objects.py: AnyFlowPipeline + AnyFlowCausalPipeline
Local fast tests: 21/21 passed
- 12 scheduler tests (FlowMapEulerDiscreteScheduler)
- 9 transformer tests (AnyFlowTransformer3DModel construction + bidi forward + save/load)
The pipeline fast tests in tests/pipelines/anyflow/ require a local dev install
that matches the diffusers main branch's transformers >= compatibility floor.
The reference slices for slow integration tests (real GPU + 1.3B/14B
checkpoints) are intentionally left as TODO stubs to be captured by the user
on a real GPU machine before opening the PR.
* [AnyFlow] address review feedback: bug fixes + DMD wording + EN/ZH tutorials
Critical bug fixes (verified against precision-validation review):
* pipeline_anyflow.py / pipeline_anyflow_causal.py: replace hardcoded
transformer_dtype = torch.bfloat16 with self.transformer.dtype, so
pipe.to("cpu") and PipelineTesterMixin save/load tests do not crash on a
dtype mismatch in the patch_embedding conv3d.
* transformer_anyflow.py: drop the duplicate `base = base = ...` assignment in
_build_causal_mask (was a copy-paste typo carried over from FAR-Dev).
* transformer_anyflow.py: drop unused `q_is_context` / `k_is_context` locals
and the `# noqa: F841` markers that were silencing the dead-store warning.
* transformer_anyflow.py: remove `CacheMixin` from the inheritance list — the
pipeline manages KV cache directly, the mixin's interface is unused.
* transformer_anyflow.py: guard the module-level `torch.compile(flex_attention)`
with try/except so the file imports cleanly on CPU CI / no-Triton machines.
* convert_anyflow_to_diffusers.py: replace ad-hoc print warnings with the
stdlib logger (warning_once-style) and a module-level basicConfig.
Documentation accuracy:
* AnyFlowCausalPipeline class docstring + main pipeline doc + EN/ZH tutorial:
drop the fictitious `task_type` / `image` / `video` arguments and document
the real API: pass `context_sequence={"raw": tensor}` (or `{"latent": ...}`)
to switch between T2V (None) / I2V (1-frame) / TV2V (4n+1-frame) modes.
* Pipeline class docstrings + main doc: explicitly describe AnyFlow's
two-stage LoRA distillation including DMD reverse-divergence supervision
with Flow-Map backward simulation in stage 2 (was previously implicit).
* training_rollout: add detailed docstring explaining its role as the
3-segment Flow-Map backward simulation entry point used during DMD training.
* Long-form tutorial doc `using-diffusers/anyflow.md` (EN, 239 LOC) and
Chinese mirror `docs/source/zh/using-diffusers/anyflow.md` (224 LOC) added
and registered in both `_toctree.yml` files.
Tests:
* Skip `test_attention_slicing_forward_pass` in both pipeline test classes
with a clear rationale (custom attention processor does not support slicing).
* All 21 standalone tests still pass (12 scheduler + 9 transformer).
Quality gates:
* `ruff check` clean across all AnyFlow files.
* `ruff format --check` reports 6 files already formatted.
* `python utils/check_copies.py` reports no diff.
Out of scope for this commit (deferred until reviewer feedback):
* Splitting AnyFlowTransformer3DModel into bidi + causal subclasses
* Unifying _forward_inference / _forward_cache return types
* Migrating model tests from plain unittest to BaseModelTesterConfig + mixins
* HF model card / config.json metadata updates on the nvidia/* repos
(push to Hub manually before opening the PR)
* [AnyFlow] rename Causal->FAR + explicit forward signature + dataclass output
Round 2 of review feedback. Three groups of changes; transformer state-dict
keys, module hierarchy, and tensor flow are unchanged so the H200 bit-exact
validation remains valid.
A. Pipeline rename (mechanical, no behavior change):
* Class: AnyFlowCausalPipeline -> AnyFlowFARPipeline (Causal in diffusers
usually means an attention mask; AnyFlow's variant is FAR autoregressive,
so the FAR name is more specific and matches the paper).
* File: pipeline_anyflow_causal.py -> pipeline_anyflow_far.py (git mv).
* Test file: test_anyflow_causal.py -> test_anyflow_far.py (git mv).
* All references updated in src/, tests/, docs/, scripts/, plus stale
anyflowcausalpipeline anchor links in tutorial markdown.
B. Pipeline test bug fixes (closes 19 fast-test failures reported by
precision-validation reviewer):
* pipeline_anyflow.py / pipeline_anyflow_far.py: __call__ now sets
self._num_timesteps = num_inference_steps before the rollout, so the
PipelineTesterMixin callback tests can read pipe.num_timesteps.
* tests/pipelines/anyflow/test_anyflow_far.py: drop the fictitious
task_type="t2v" kwarg that crashed every causal fast test (the FAR
pipeline selects mode via context_sequence, not a task_type arg).
C. Transformer architecture cleanups (review-driven, no tensor changes):
* Replace forward(*args, **kwargs) dispatcher with an explicit signature
listing every supported kwarg (hidden_states, timestep, r_timestep,
encoder_hidden_states, encoder_hidden_states_image, chunk_partition,
clean_hidden_states, clean_timestep, kv_cache, kv_cache_flag, is_causal,
attention_kwargs, return_dict). Helps IDE / type-checker / torch.compile
tracing.
* Drop SimpleNamespace returns. Add AnyFlowFARTransformerOutput
(BaseOutput dataclass with sample + kv_cache fields) for the two causal
paths that need to also propagate kv_cache (_forward_inference and the
newly return_dict-aware _forward_cache). _forward_train and
_forward_bidirection now consistently return Transformer2DModelOutput.
Pipeline call sites already use return_dict=False with positional
unpacking, so the fix is transparent there.
Out of scope (deferred until canonical-org HF metadata sync):
* Splitting AnyFlowTransformer3DModel into a bidi class plus an
AnyFlowFARTransformer3DModel subclass — touches register_to_config keys
and would require updating model_index.json on every released checkpoint.
* Promoting chunk_partition from register_to_config to a forward-time
argument (same reason).
* Renaming training_rollout to _denoise — would break callers in the
FAR-Dev on-policy trainer that produced the released checkpoints.
Local fast tests: 21/21 still pass (12 scheduler + 9 transformer).
ruff check, ruff format, and check_copies.py are all clean.
* [AnyFlow] wire callback_on_step_end through inference_range + add chunk_partition to FAR fast-test fixture
Two root causes for the 19 remaining PipelineTesterMixin failures, identified
by the H200 reviewer:
1. callback_on_step_end was accepted by __call__ but never invoked. Both
pipelines pass it through to training_rollout (and FAR additionally through
inference()), and inference_range now fires it after scheduler.step in
the standard inference branch:
if callback_on_step_end is not None:
callback_kwargs = {k: locals()[k] for k in callback_on_step_end_tensor_inputs}
callback_outputs = callback_on_step_end(self, i, t, callback_kwargs)
latents = callback_outputs.pop("latents", latents)
prompt_embeds = ...
negative_prompt_embeds = ...
`nonlocal prompt_embeds, negative_prompt_embeds` lets the callback rewrite
the closure-captured embeddings, matching upstream WanPipeline semantics.
The 3-segment grad_timestep training rollout does not invoke the callback;
it is intentionally training-only.
2. tests/pipelines/anyflow/test_anyflow_far.py::get_dummy_components built
the dummy transformer without a `chunk_partition`, leaving it None on the
model config and crashing the pipeline at `sum(self.transformer.config.chunk_partition)`.
Set `chunk_partition=[1, 1, 1]` in the fixture (3 chunks of 1 latent frame
each, matching the test's num_frames=9 -> 3 latent frames).
Local fast tests: 21/21 still pass.
ruff check, ruff format, and check_copies.py are all clean.
* [AnyFlow] Phase 2: split transformer + drop chunk_partition from config + rename helpers
Major architectural refactor that aligns the integration with diffusers conventions
ahead of the canonical-org Hub upload. State-dict keys, module hierarchy, and
tensor flow are unchanged so the H200 bit-exact validation remains valid; only
the on-disk transformer/config.json fields move.
Changes:
1. **Sibling transformer classes** replace the flag-driven single class:
* AnyFlowTransformer3DModel — bidirectional only. Drops compressed_patch_size /
full_chunk_limit / init_far_model / init_flowmap_model / chunk_partition
kwargs (always-on for AnyFlow distilled checkpoints).
* AnyFlowFARTransformer3DModel — adds far_patch_embedding + the 3 FAR forward
paths (train / cache-prefill / autoregressive inference).
* AnyFlowTimeTextImageEmbedding (the legacy single-time embedder used only by
the old setup_flowmap_model bootstrap) is removed; both classes now build
AnyFlowDualTimestepTextImageEmbedding directly in __init__.
* setup_flowmap_model / setup_far_model methods are removed; weight warm-start
for far_patch_embedding (trilinear interpolation from patch_embedding) moves
into AnyFlowFARTransformer3DModel.__init__.
2. **chunk_partition** is no longer a model config field. The FAR pipeline owns
the schedule:
* AnyFlowFARPipeline.default_chunk_partition = [1, 3, 3, 3, 3, 3, 3, 2]
matches the released 81-frame NVIDIA checkpoints.
* AnyFlowFARPipeline.__call__ / _denoise_rollout accept a chunk_partition
argument that overrides the default for non-default num_frames.
3. **training_rollout -> _denoise_rollout** rename across both pipelines and all
English / Chinese docs that referenced it. Signals the method is internal to
the pipeline driver, not a public training API.
4. **Conversion script + tests + docs + registries**:
* scripts/convert_anyflow_to_diffusers.py: VARIANTS dict picks the right
transformer class per variant; init_far_model / init_flowmap_model /
chunk_partition kwargs are removed from the from_pretrained call.
* Transformer test file split into AnyFlowTransformer3DModelTest and
AnyFlowFARTransformer3DModelTest classes.
* Pipeline test fixtures use the right class and pass chunk_partition via
get_dummy_inputs (3-frame schedule [1, 1, 1] for the 9-frame test).
* New docs page docs/source/en/api/models/anyflow_far_transformer3d.md;
anyflow_transformer3d.md rewritten for the bidi-only class.
* AnyFlowFARTransformer3DModel registered in src/diffusers/__init__.py,
src/diffusers/models/__init__.py, models/transformers/__init__.py and the
dummy_pt_objects.py stubs.
* docs/source/en/_toctree.yml: new entry for the FAR transformer page.
5. **Cleanups**:
* Pipeline __call__ no longer passes is_causal=False to the bidi forward (the
bidi class doesn't accept it).
* Pipeline class docstrings drop stale references to init_*_model flags.
Local tests: 22/22 pass (12 scheduler + 10 transformer covering both classes).
ruff check / format / check_copies clean.
Hub artifacts (model_index.json, transformer/config.json, scheduler config) need
to be regenerated for the released checkpoints; the HF update guide will be
delivered separately.
* [AnyFlow] Phase 3: convention compliance against .ai/AGENTS.md + .ai/models.md
Hard violations (per official diffusers guidelines):
* drop einops dependency — replace 25+ rearrange() calls with native
permute/reshape/unflatten in transformer + both pipelines
* device-gate torch.float64 — apply_rotary_emb and AnyFlowRotaryPosEmbed now
fall back to float32 / complex64 on MPS / NPU; freqs are lazily rebuilt
per-device via _build_freqs (matches transformer_wan / transformer_flux
pattern)
* migrate attention to dispatch_attention_fn — replace direct
F.scaled_dot_product_attention calls with dispatch_attention_fn (works
with sage / flash / native backends); introduce AnyFlowAttention(
AttentionModuleMixin) with _default_processor_cls / _available_processors;
rename processors to AnyFlowAttnProcessor / AnyFlowCrossAttnProcessor and
declare _attention_backend / _parallel_config class attrs
* drop dead config fields — qk_norm and added_kv_proj_dim are pruned from
both transformer __init__ signatures and AnyFlowTransformerBlock;
AnyFlowAttention is hardcoded to rms-norm-across-heads (the only scheme
the released checkpoints use) and has no add_k_proj path (T2V only)
* add _repeated_blocks = ["AnyFlowTransformerBlock"] to both transformer
classes for compile_repeated_blocks() support (matches Wan)
* annotate prepare_latents with `# Copied from diffusers.pipelines.wan.
pipeline_wan.WanPipeline.prepare_latents`; the pipeline-side rearrange
to (B, T, C, H, W) layout is moved to the call site
State-dict keys are preserved (legacy Attention had identical to_q / to_k /
to_v / to_out / norm_q / norm_k naming), so existing AnyFlow checkpoints load
bit-exactly into the new AnyFlowAttention class.
The HF Hub config-update guide is updated correspondingly: transformer/
config.json now drops qk_norm and added_kv_proj_dim alongside the previous
init_far_model / init_flowmap_model / chunk_partition removals.
22 fast CPU tests still pass; ruff format / ruff check / check_copies all
clean.
* [AnyFlow] FAR fast-test compat: rope 0-dim guard + flex_attention CPU/head-dim fallbacks + KV-cache dtype + num_timesteps
Phase 3 migrated bidi + cross-attention to dispatch_attention_fn but the FAR
causal path still calls flex_attention directly, which has hard requirements
(CPU compile, head_dim >= 16) that fail on PipelineTesterMixin's tiny dummy
components. Real ckpts (head_dim=128, CUDA) never hit these branches; bit-exact
numerical equivalence with FAR-Dev preserved on all 4 released ckpts (forward
0.00e+00, backward kernel-nondet only, ratio 1.000).
Code fixes:
1. AnyFlowRotaryPosEmbed._forward_compressed_frame / _forward_full_frame now
short-circuit to an empty tensor when num_frames / height / width is 0.
PipelineTesterMixin's dummy VAE has scale_factor_spatial=8, so a 16x16 raw
spatial input becomes a 2x2 latent which then floors to 0 against
compressed_patch_size=(1, 4, 4); the original
`freqs[:0].view(0, k, 1, -1)` reshape was ambiguous in that regime.
2. flex_attention dispatch: split the module-load
`torch.compile(flex_attention, dynamic=True)` into `_flex_attention_eager`
(always available) plus `_flex_attention_compiled`, with a tiny wrapper
that picks compiled for CUDA tensors and eager for CPU. Avoids
torch._inductor C++ codegen failures that broke fast tests after
`pipe.to("cpu")`. CUDA performance unchanged (L10 benchmark: 0.0% delta on
bidi 1.3B fwd, 0.0% delta on FAR causal 1.3B fwd).
3. AnyFlowAttnProcessor (FAR causal branch): when head_dim < 16
(flex_attention's hard minimum) zero-pad q/k/v's last dim to 16 and pass
`scale=1/sqrt(original_head_dim)` to flex_attention. Padded value rows
contribute 0, so trimming the output back is mathematically equivalent.
Released ckpts use head_dim=128 so the branch is never taken in production.
4. pipeline_anyflow_far.encode_kv_cache: replace the hardcoded
`latents.to(torch.bfloat16)` with `self.transformer.dtype`. The hardcoded
bf16 crashed conv3d on dummy fp32 components ("Input type (BFloat16) and
bias type (float) should be the same"); real bf16 ckpts are unaffected.
5. pipeline_anyflow_far._denoise_rollout sets
`self._num_timesteps = (len(chunk_partition) - num_context_chunks) * num_inference_steps`
before the chunk loop, so PipelineTesterMixin.test_callback_cfg's
`pipe.num_timesteps`-based assertion matches the actual number of callback
fires (chunks * NFE) instead of the previous hardcoded num_inference_steps.
Tests:
* test_callback_inputs cannot pass without changing FAR's chunk-wise output
semantics — it zeroes latents on the final step and asserts the *entire*
output buffer is zero, but only the active chunk's slice is overwritten in
a chunk-wise rollout. Marked `@unittest.skip` with a detailed rationale;
callback functionality itself is still covered by test_callback_cfg.
* Full pytest run on tests/pipelines/anyflow/ +
tests/models/transformers/test_models_transformer_anyflow.py +
tests/schedulers/test_scheduler_flow_map_euler_discrete.py: 81 passed,
0 failed, 11 skipped.
Quality gates:
* `ruff check` and `ruff format --check` clean across all AnyFlow files.
* `python utils/check_copies.py` clean.
* `python utils/check_dummies.py` clean.
* [AnyFlow] docs/code: paper-release tidy-up
User-facing alignment with the official HF Hub model card and the day-of-announcement
materials at https://huggingface.co/collections/nvidia/anyflow.
* Fill in the arXiv identifier 2605.13724 (5 paper links + 2 BibTeX entries).
* Rename TV2V → V2V across docs + pipeline_anyflow{,_far}.py so the diffusers
copy uses the same Video-to-Video terminology as the official model card.
* Add the [nvidia/anyflow](https://huggingface.co/collections/nvidia/anyflow)
HF collection link to the three tutorial intros.
* Drop the temporary "guyuchao/* staging" tip from the EN tutorial / API page
/ ZH tutorial — the nvidia/AnyFlow-*-Diffusers repos are now live.
* Wire up NVlabs/AnyFlow (training code) and nvlabs.github.io/AnyFlow (project
page) in place of the prior <github-org> / <project-page-url> placeholders.
* Cite the authors (Yuchao Gu, Guian Fang et al.) and NUS ShowLab × NVIDIA
affiliation in the main tutorial, API pipeline page, and both transformer
model pages; BibTeX uses the standard `and others` to elide the full list
until the next pass.
Working tree, CI gates, and tests after the change:
ruff format --check ✓
ruff check ✓
python utils/check_copies.py ✓
python utils/check_dummies.py ✓
pytest tests/models + tests/schedulers (22 fast) ✓
No production code logic changes — only docstring wording inside pipeline
files (TV2V → V2V).
* [AnyFlow] docs: drop in official BibTeX (full author list)
Replace the placeholder ``@article{gu2026anyflow, author = {Gu, Yuchao and
Fang, Guian and others}, ...}`` block in both the English and Chinese
tutorials with the canonical ``@misc{gu2026anyflowanystepvideodiffusion,
...}`` form from arxiv.org/abs/2605.13724, which lists all seven authors:
Yuchao Gu, Guian Fang, Yuxin Jiang, Weijia Mao, Song Han, Han Cai,
Mike Zheng Shou.
Docs-only.
* [AnyFlow] align with diffusers conventions + drop training-only code
Scheduler
- FlowMapEulerDiscreteScheduler.step now returns a
FlowMapEulerDiscreteSchedulerOutput dataclass (or tuple with return_dict=False)
and uses the conventional positional order (model_output, timestep, sample,
r_timestep).
- Drop training-only helpers: adaptive_weighting, set_train_weight,
get_train_weight, linear_timesteps_weights, and the weight_type config field.
- Add scale_model_input no-op for API parity; raise ValueError on missing
r_timestep.
Transformer
- Remove gate_track debug write inside
AnyFlowDualTimestepTextImageEmbedding.forward_timestep.
- Compile flex_attention lazily on first CUDA call instead of at import time.
- Replace assert with ValueError in build_block_mask.
- Resolve <arxiv-id> placeholders to 2605.13724.
Pipelines (AnyFlowPipeline + AnyFlowFARPipeline)
- Add EXAMPLE_DOC_STRING + @replace_example_docstring and full __call__
docstrings covering every argument.
- Move use_mean_velocity from __init__ to __call__ so save/load round-trips.
- Drop _denoise_rollout's grad_timestep branch (DMD on-policy training rollout),
the inner inference_range closure, and the redundant negative-prompt concat.
- Replace asserts with ValueError; wire show_progress to tqdm; rename inference
-> _inference; remove dead current_timestep property.
- Update scheduler.step call sites to the new signature.
- Trim class docstrings to inference-only language.
Pipeline output
- Add Apache 2.0 license header; switch to relative import.
Auto pipeline / conversion script
- Register AnyFlowFARPipeline in AUTO_IMAGE2VIDEO_PIPELINES_MAPPING and
AUTO_VIDEO2VIDEO_PIPELINES_MAPPING.
- Document the weights_only=False requirement in the conversion script.
Tests
- Scheduler tests use the new step signature and verify the Output dataclass
contract.
- Drop the four obsolete training-weight tests; drop weight_type kwarg from
pipeline test fixtures; remove internal milestone names from TODO comments.
Docs
- Resolve <arxiv-id> in the scheduler docs page.
- Trim DMD / on-policy distillation language in EN/ZH tutorials and the
pipelines page; the paper abstract quote is preserved verbatim.
* [AnyFlow] split FAR causal transformer into transformer_anyflow_far.py
Per @dg845's review on #13745: extract FAR causal modules into a dedicated
sibling file so each transformer variant reads in isolation. Shared submodules
are duplicated via `# Copied from` so `make fix-copies` keeps both in sync.
- `transformer_anyflow.py`: bidi-only. `AnyFlowAttnProcessor` no longer carries
the flex/KV-cache branch (was: dispatch in one branch, bare flex_attention in
the other); `AnyFlowRotaryPosEmbed` drops the compressed-frame helpers and
the `is_causal` arg; `AnyFlowDualTimestepTextImageEmbedding` drops its causal
branch. `AnyFlowTransformerBlock` keeps a single class with a new
`is_causal: bool = False` ctor flag that selects the self-attn processor —
the forward path is identical in both modes, only the processor differs.
- `transformer_anyflow_far.py`: new. Contains `AnyFlowFARTransformerOutput`,
`AnyFlowCausalAttnProcessor` (routed through `dispatch_attention_fn(backend=
"flex")` with a clear ValueError when a non-flex backend is configured; the
BlockMask is consumed only by the flex backend in `_native_flex_attention`),
`AnyFlowDualTimestepTextImageEmbeddingCausal`, `AnyFlowCausalRotaryPosEmbed`,
`AnyFlowFARTransformer3DModel`, and `# Copied from` clones of the shared
shared `AnyFlowAttention`/`AnyFlowCrossAttnProcessor`/`AnyFlowImageEmbedding`/
`AnyFlowTransformerBlock`/`AnyFlowAttnProcessor` modules.
Verified bit-exact against the pre-refactor branch on H200 (float32):
- bidi: L2 = 0.000e+00, max|Δ| = 0.000e+00
- FAR : L2 = 4.772e-06, max|Δ| = 3.576e-07
The FAR delta is fp32 accumulation noise from the dispatch path permuting
(B,L,H,D) ↔ (B,H,L,D) around the same `flex_attention` kernel.
Addresses review comments at transformer_anyflow.py:215, :261, :450, :622,
:671, :958.
* [AnyFlow] pipeline cleanup: video_processor, encode_video, inline rollout, kwarg rename
Per @dg845's review on #13745, applied to both bidi `AnyFlowPipeline` and
causal `AnyFlowFARPipeline`:
- Use `self.video_processor.preprocess_video(...)` instead of the manual
`* 2 - 1` normalize.
- Merge `vae_encode` + `encode_latents` + `_normalize_latents` into a single
`encode_video` method, mirroring `WanImageToVideoPipeline.encode_image`'s
flat structure.
- Inline `_denoise_rollout` into `AnyFlowPipeline.__call__`. For the FAR
pipeline, inline both `_denoise_rollout` and `_inference` as a nested loop
(outer over chunks, inner over denoising steps), mirroring
`WanAnimatePipeline.__call__`. `encode_kv_cache` is intentionally kept as a
method — it is one transformer call with a different `kv_cache_flag` mode
(cache-write), and inlining it would interleave two distinct forward
semantics in the same loop body and lose readability.
- Rename `context_sequence` → `video` (pixel-space) + `video_latents`
(pre-encoded), matching `WanVideoToVideoPipeline`. For the FAR pipeline,
the old `{"raw"/"latent"}` dict form is replaced by the two kwargs.
Mutually-exclusive validation raises `ValueError`.
Addresses review comments at pipeline_anyflow.py:358, :372, :393, :473 and
pipeline_anyflow_far.py:395, :489, :675.
* [AnyFlow] scheduler: N-length timesteps + step defaults r_timestep
Per @dg845's review on #13745:
- `set_timesteps(N)` now produces `N` timesteps backed by an internal
`sigmas[N+1]` linspace, matching `FlowMatchEulerDiscreteScheduler.set_
timesteps`. The final sigma (== 0) is the implicit r-endpoint of the last
step; the pipeline rollouts iterate `for i, t in enumerate(timesteps)`
without the old `[:-1]` slicing.
- `step(r_timestep=None)` now defaults to the next timestep on the schedule
(resolved via fp-tolerant `argmin` over `sigmas[:-1]`), instead of raising.
Any-step sampling is preserved when `r_timestep` is explicit. The raise
stays only for the case where the caller passes a `timestep` value that
isn't on the schedule and provides no `r_timestep` — there's no sensible
default in that case.
- Build sigmas in float64 on CPU then move to the target device, with a
float32 downcast for MPS / NPU (float64 isn't supported on those backends).
Pipeline rollout loops updated to compute `r = sigmas[i + 1] * num_train_
timesteps` for the model's `r_timestep` input and pass `r_timestep=None` to
`scheduler.step` (which resolves it from the schedule internally).
Addresses review comments at scheduling_flow_map_euler_discrete.py:107 and
:148.
* [AnyFlow] tests: regenerate via generate_model_tests.py; split bidi/FAR files
Per @dg845's review on #13745: replaced the hand-rolled transformer tests
with the standard mixin-based suite produced by `utils/generate_model_tests
.py`, and split the FAR causal model tests into their own file to mirror the
transformer file split.
- `tests/models/transformers/test_models_transformer_anyflow.py`: regenerated
bidi suite. Pulls in `ModelTesterMixin`, `MemoryTesterMixin`,
`TrainingTesterMixin`, `AttentionTesterMixin`, `TorchCompileTesterMixin` via
`BaseModelTesterConfig`, with `get_init_dict()` / `get_dummy_inputs()`
filled in for the small bidi config used in CI.
- `tests/models/transformers/test_models_transformer_anyflow_far.py`: new.
Same mixin set (TorchCompile is intentionally skipped — FAR's
`_build_causal_mask` uses `flex_attention.create_block_mask(_compile=False)`
which conflicts with the standard compile tester's assumptions; the bidi
file covers compile, FAR is bit-exact-validated end-to-end on H200 via the
pipeline replay). Also carries an `AnyFlowCausalAttnProcessor` smoke test
that exercises the backend gate (non-flex backends must raise) and asserts
the `AnyFlowFARTransformerOutput` dataclass exposes the expected fields.
Addresses review comments at test_models_transformer_anyflow.py:71 and :128.
* [AnyFlow] docs: update for video / video_latents kwarg rename
Following the pipeline kwarg refactor in e9d50b2, sweep the user-facing docs
to reflect the new API:
- `docs/source/en/api/pipelines/anyflow.md`: T2V / I2V / V2V code examples now
use `video=` instead of `context_sequence={"raw": ...}`. The "Generation
with AnyFlow (FAR Causal)" intro describes the new mutually-exclusive
`video` / `video_latents` selector.
- `docs/source/en/using-diffusers/anyflow.md`: the scenario selector table,
the "Image-to-video and video-to-video" walkthrough, and the closing note
about pre-encoded latents are all updated. `vae_encode` references are
replaced with `encode_video`.
* [AnyFlow] tests: skip FAR training tests on CPU (flex backward); align scheduler tests with N-length timesteps
- TestAnyFlowFARTransformer3DTraining: skip test_training / test_training_with_ema /
test_gradient_checkpointing_equivalence on CPU. FAR causal self-attn uses
torch.nn.attention.flex_attention whose backward kernel is GPU-only.
- test_scheduler_flow_map_euler_discrete: assert timesteps is N-length (not N+1) and
the sigma=0 r-endpoint lives in self.sigmas[-1]; test_step_one_shot_sampling now
exercises r_timestep=None (resolved from sigmas) since N=1 has no timesteps[1].
* [AnyFlow] docs: complete forward() Args: sections for check_forward_call_docstrings
main #13758 added utils/check_forward_call_docstrings.py which requires every signature
arg to appear as its own `name (...):` entry under Args:. Expand the bidi and FAR
transformer forward docstrings to list each parameter individually.
* [AnyFlow] apply 5/21 review suggestions (A: 1-click)
FAR transformer:
- AnyFlowCausalAttnProcessor: default _attention_backend = 'flex' (was None);
remove None from _SUPPORTED_BACKENDS. None previously fell through to SDPA
which silently ignored the BlockMask; failing loudly is the right default.
- dispatch_attention_fn call: read self._attention_backend instead of hardcoded
'flex', so '_native_flex' selection works.
- _build_freqs / _forward_full_frame: add '# Copied from' to bidi RoPE.
Pipelines:
- bidi + FAR docstrings: video shape (B, C, T, H, W) -> (B, T, C, H, W) to
match VideoProcessor.preprocess_video.
- FAR EXAMPLE_DOC_STRING: single-frame I2V tensor wrap uses unsqueeze(1) for the
T axis instead of unsqueeze(2).
- FAR encode_video: drop duplicated @torch.no_grad() decorator.
Tests:
- test_anyflow / test_anyflow_far: lift the test_save_load_optional_components
skip (the test actually passes).
- FAR processor smoke test: assert default backend is 'flex' (was 'None').
* [AnyFlow] apply 5/21 review suggestions (B: refactors)
Pipelines:
- check_inputs accepts video / video_latents and raises early on:
(a) mutual exclusion (was checked late in __call__);
(b) FAR's (num_frames - 1) % 4 == 0 constraint.
__call__ no longer carries duplicate validation.
- FAR pipeline: drop the show_progress kwarg and replace the single tqdm with
nested progress bars in the LLaDA-2 pattern — outer 'Chunks' (position=0)
and per-chunk inner 'Inference Steps' (position=1, leave=False) — both
picking up DiffusionPipeline._progress_bar_config (so set_progress_bar_config
controls them, including disable=None).
Scheduler:
- step() resolves source and target sigmas by indexing self.sigmas via the new
index_for_timestep(), instead of dividing the input timesteps by
num_train_timesteps. This keeps the math correct for any future schedule
whose timestep/sigma relationship is non-linear. For an off-schedule
r_timestep the code falls back to r / num_train_timesteps, so explicit
any-step sampling outside the schedule still works (and t off-schedule with
r=None still raises a clear ValueError, as before).
Numerical equivalence: for the shipped linspace+shift schedule the two
formulations are bit-identical (verified: max abs diff = 0.0 over an N=8,
shift=5 schedule).
* [AnyFlow] apply Claude bot review (5/21): 8 findings beyond dg845's list
Finding #1 — attention_kwargs plumbing:
Both transformers now decorate forward() with @apply_lora_scale('attention_kwargs')
(matches Wan); pipelines forward attention_kwargs to the transformer + encode_kv_cache,
and the unused parameter is dropped from the inner _forward_train / _forward_cache /
_forward_inference signatures. Pipeline docstrings updated to the standard wording.
Finding #2 — naming:
Rename far_cfg -> layout_cfg in the bidi transformer (the bidi path is not FAR; the
FAR transformer keeps far_cfg, which is accurate there).
Finding #3 — scheduler state machine:
Add _step_index, _begin_index, step_index property, begin_index property,
set_begin_index(), _init_step_index(). step() lazily initializes and advances the
counter so downstream callbacks / composable schedulers can observe rollout progress.
Sigma resolution remains a pure function of (timestep, r_timestep) — calling step()
twice with identical args still returns identical prev_sample (idempotent).
Finding #4 — redundant @torch.no_grad():
Drop the redundant decorators on bidi pipeline's encode_video and FAR pipeline's
encode_kv_cache (callers are already in __call__'s no-grad scope).
Finding #5 — dead code:
Remove the unreachable temb.ndim == 2 else branch from the bidi transformer's
output-norm path (condition_embedder.forward always returns a 3D temb).
Finding #6 — private rename:
forward_far_patchify[_inference] -> _forward_far_patchify[_inference] (only called
internally by _forward_train / _forward_cache / _forward_inference).
Finding #7 — pipeline comment numbering:
Bidi + FAR pipelines renumber steps so the # 4. slot is no longer skipped.
Finding #8 — mask-mod comment numbering:
_build_causal_mask numbered comments now run 1) 2) 3) ... (was 1) 3) 4) ...).
Tests:
- New test_step_index_advances + test_set_begin_index_anchors_step_index in the
scheduler test file exercise the new state machine.
- All existing pipeline / transformer / scheduler tests still pass (85 passed,
85 skipped on CPU).
Bit-exact: 8-step rollout vs the previous formulation, max abs diff = 0.0 (the new
sigma-lookup is byte-identical to t/num_train_timesteps on this schedule).
* [AnyFlow] scheduler: honour off-schedule any-step in _init_step_index; drop dead _resolve_next_timestep
Audit caught two issues in the previous scheduler commit:
1. The new state machine raised in _init_step_index whenever the first timestep
wasn't on the active schedule, contradicting the documented contract that
step() falls back to t/num_train_timesteps for off-schedule any-step
sampling. The fall-back numerics were intact but they were unreachable —
the init check fired first.
Fix: _init_step_index now initializes _step_index to 0 when the timestep is
off-schedule (still a valid observable counter for callbacks). step()'s
sigma resolution is untouched, so on-schedule rollouts stay bit-exact and
off-schedule any-step sampling actually runs again. Regression test:
test_step_off_schedule_anystep_supported.
2. _resolve_next_timestep had no remaining callers after the step() rewrite
inlined the same lookup. Removed (private helper, no external API).
* [AnyFlow] docs: align user guides with video shape + kwarg fixes
- en api/pipelines/anyflow.md: video shape (B, C, T, H, W) -> (B, T, C, H, W);
example tensor wrap uses unsqueeze(0).unsqueeze(1) and permute(0, 3, 1, 2)
to match VideoProcessor.preprocess_video's 5D contract.
- zh using-diffusers/anyflow.md: same shape fixes; also flip the I2V / V2V
examples from the obsolete context_sequence={...} dict to the current
video= / video_latents= kwargs; helper to_video_tensor returns (1, T, C, H, W);
add a note about mutual exclusion.
* [AnyFlow] tests: drop @slow integration test scaffolds for initial PR
.ai/skills/model-integration/SKILL.md is explicit: 'No integration / slow
tests in the initial PR — don't add anything gated on @slow / RUN_SLOW=1
yet.' Our two integration test classes were shape-only assertions with TODOs
for a future numeric reference, so dropping them loses no actual coverage —
the relevant rollouts are covered by H200 bit-exact replay outside the
pytest suite. Can land a follow-up PR after merge with proper numeric
reference slices once the maintainer is comfortable enabling slow tests.
* Apply style fixes
* [AnyFlow] apply 5/22 dg845 review: comment cleanups + custom sigmas/timesteps schedule
dg845 third pass — 7 of 9 comments applied; the 8th (custom sigmas/timesteps support)
matches FlowMatchEulerDiscreteScheduler conventions; the 9th (_build_causal_mask
refactor) is explicitly marked non-blocking and deferred to a follow-up that also
re-enables TorchCompileTesterMixin.
Comment cleanups:
- transformer_anyflow.py:704 temb output-norm comment: drop redundant 'no ndim==2 branch'.
- pipeline_anyflow.py:550 denoise loop comment: '# 6. Denoising loop'.
- pipeline_anyflow_far.py:684 denoise loop comment: '# 8. Denoising loop (outer over
chunks, inner over timesteps).'.
- pipeline_anyflow_far.py:702 drop trailing inline comment on `timesteps = scheduler.timesteps`.
- scheduling_flow_map_euler_discrete.py: clearer wording on the off-schedule `r_timestep`
error.
Custom schedule support:
- FlowMapEulerDiscreteScheduler.set_timesteps gains `sigmas` and `timesteps` kwargs
mirroring FlowMatchEulerDiscreteScheduler. Default behaviour is unchanged
(linspace + shift); the validation + length-N → length-N+1 terminal-0 append are
shared with the default path so on-schedule rollouts stay bit-exact.
- AnyFlowPipeline.__call__ and AnyFlowFARPipeline.__call__ accept `sigmas` and
`timesteps` kwargs, override num_inference_steps from their length, and forward
to set_timesteps (matches LTX2Pipeline pattern).
- New scheduler tests: test_set_timesteps_custom_sigmas and
test_set_timesteps_custom_timesteps cover both override paths.
Dtype skip on save/load:
- TestAnyFlowTransformer3D and TestAnyFlowFARTransformer3D now skip
test_from_save_pretrained_dtype_inference (parametrized over fp16/bf16), mirroring
WanTransformer3DModel's skip — the test's tolerance requirements are too high for
meaningful signal under AnyFlow's flow-map mixed-precision sampling.
* [AnyFlow] docs: apply hf-doc-builder line wrap (max_len 119)
CI doc-builder style check flagged 3 files with docstring lines >119 chars.
Ran 'doc-builder style src/diffusers docs/source --max_len 119' to autoformat;
content unchanged, line wrapping only.
* [AnyFlow] apply 5/22 follow-up review: new_zeros terminal sigma + cleanup
dg845 blocking suggestion (r3287274209):
- scheduling_flow_map_euler_discrete.py:185 — use `working_sigmas.new_zeros(1)`
instead of `torch.zeros(1, dtype=...)` so the appended terminal sigma inherits
both device and dtype from working_sigmas. The current working_sigmas always
starts on CPU so the device mismatch is latent, but new_zeros is the correct
defensive pattern and matches how the published FAR test fixtures run on CUDA.
Claude bot final-review follow-ups:
- transformer_anyflow_far.py: drop three stale `# step 3: generate attention mask`
comments left over from the original numbered-step structure (bot #6).
- pipeline_anyflow_far.py: annotate `encode_video` with
`# Copied from diffusers.pipelines.anyflow.pipeline_anyflow.AnyFlowPipeline.encode_video`
and align docstring + inline comment so `make fix-copies` keeps them in sync (bot #3).
Skipped (not real / judgment-call):
- bot #2 (private rename of `_forward_far_patchify*`) — already done in 84605d5;
bot was looking at a stale snapshot.
- bot #4 (check_inputs `# Copied from`) — FAR's check_inputs has an extra
`(num_frames - 1) % 4 == 0` constraint that doesn't map onto the bidi version,
so a clean `# Copied from` link would require restructuring. Bot called it a
consistency nit; leaving as-is.
- bot #5 (`encode_kv_cache` → `_encode_kv_cache`) — bot itself flagged this as
judgment-call territory; the helper is a coherent operation that advanced
inference callers may want to invoke directly.
---------
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
|
||
jiqing-feng
|
a362223ddc |
Fix OOM in WanAnimate BitsAndBytes Training Test (#13777)
reduce input size for tests Signed-off-by: jiqing-feng <jiqing.feng@intel.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
|
jiqing-feng
|
907c0c2c76 |
Implement _dequantize for TorchAO quantizer (#13538)
* Implement _dequantize for TorchAO quantizer - Add _dequantize() method in TorchAoHfQuantizer that dequantizes TorchAOBaseTensor weights back to standard nn.Parameter - Fix _verify_if_layer_quantized to check isinstance(weight, TorchAOBaseTensor) so dequantized layers are correctly detected as non-quantized * enable dequantize for TorchAO tester mixin Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * check dequantize Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix dequantize: clear is_quantized flag and cast dtype after dequantize * fix Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * fix error report Signed-off-by: jiqing-feng <jiqing.feng@intel.com> --------- Signed-off-by: jiqing-feng <jiqing.feng@intel.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
|
jiqing-feng
|
2f4a7177f0 |
Fix training gradient underflow in quantization tests (#13539)
* Fix training gradient underflow in quantization tests Change autocast dtype from float16 to bfloat16 in _test_quantization_training. Float16's limited dynamic range causes gradients to underflow to zero when passing through quantized tensor subclass operations. * fix autocast dtype check Signed-off-by: jiqing-feng <jiqing.feng@intel.com> --------- Signed-off-by: jiqing-feng <jiqing.feng@intel.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
|
Sayak Paul
|
0ad0a32c67 |
fix lfs pointer rejection problems for hub tests (#13733)
* fix lfs pointer rejection problems for hub tests * fix more * Delete .claude directory |
||
Kashif Rasul
|
79de3064dd |
[LLADA2] Fix llada2 review #13598 (#13698)
* [LLaDA2] address review findings from #13598 Fixes the six in-scope issues raised in the llada2 model/pipeline review: 1. Carry tokenizer `attention_mask` through `_prepare_input_ids` and add an `attention_mask` arg to `__call__` for pre-tokenized inputs. The runtime mask now reflects prompt padding and zeros out the block-aligned tail past `prompt_length + gen_length` instead of treating those positions as valid context. 2. Thread the per-call `block_length` into `BlockRefinementScheduler.set_timesteps` so the transfer schedule matches the requested block size (previously the scheduler only read its constructor default). 3. Drop `x0`/`x0_p`/`confidence` from `_callback_tensor_inputs` (never bound locals) and bind `sampled_tokens`, `sampled_probs`, `editing_transfer_index`, `active_block` so all advertised callback keys resolve. 4. Allow EOS exactly at index `prompt_length` (the first generated position) to mark a row finished. 5. Freeze rows that have already emitted EOS so subsequent block refinement doesn't extend them, and trim per-row at decode (previously gated on batch_size==1) so post-EOS positions don't leak into decoded text. 6. Stop calling `self.set_progress_bar_config(...)` from inside `__call__`; build a local config dict for the inner block bar so user-supplied flags (in particular `disable=True`) survive the call. Adds regression tests pinning each of the six fixes. * fix formatting * undo changes * set block_length to optional and use scheduler's default --------- Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> |
||
Ricardo-M-L
|
de5fcf6fe3 |
fix(randn_tensor): compare device.type, not torch.device, when suppressing MPS info log (#13508)
* fix(randn_tensor): compare device.type, not torch.device to str, when suppressing MPS info log
When a CPU generator is passed with an MPS target, randn_tensor intentionally skips
the 'generator was on cpu, tensor will be moved to <device>' info log — MPS doesn't
support device-side generators, so the suggestion to create one on MPS would be
misleading. The guard was written as `if device != "mps"`, but a few lines
earlier `device` is coerced to a `torch.device` object, and
`torch.device("mps") == "mps"` is False (torch.device's __eq__ with a string
returns NotImplemented, falling back to identity — they're different types).
Result: the guard is effectively always True, so MPS users get the spurious log
whenever they pass a CPU generator — the opposite of the documented intent.
Fix: compare `device.type` (a str) against "mps". Added a regression test in
tests/others/test_utils.py that exercises both the MPS and non-MPS paths via
`assertLogs` on the diffusers logger.
* refactor: use CaptureLogger instead of assertLogs in randn_tensor test
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: YiYi Xu <yixu310@gmail.com>
|
||
|
Sayak Paul
|
68a4847768 |
[tests] fix lora checkpoint serialization issues (#13676)
* fix save load text encoder lora inference * fix rest * fix one more. * fix test_set_adapters_match_attention_kwargs * fix styling issues. |
||
|
Sayak Paul
|
5fd27277b8 |
[tests] fix bitsandbytes compile tests for flux. (#13750)
fix bitsandbytes compile tests for flux. |
||
|
Sayak Paul
|
e2dae06ed5 |
[tests] Fix controlnet tests (#13736)
* add serge reviewer to enable claude for inline reviews. * remove local settings * fix controlnet test failures. * remove serge reviewer workflow from the mix |
||
dg845
|
037efdae1d |
Add LTX-2.X IC LoRA and HDR Pipelines (#13572)
* LTX-2.X IC LoRA + HDR IC LoRA pipeline implementation draft from Claude * Refactor HDR export to accept custom tone-mapping functions * Apply parity fixes + refactor + allow HDR LoRA pipeline to accept connector embeddings * Change LTX2ConditionPipeline default __call__ parameters to match the suggested params for the LTX-2.3 model * Improve IC LoRA example and fix some LTX2ICLoraPipeline bugs * Improve HDR IC LoRA example * Clean up the code a bit * make style and make quality * Rename LTX2ICLoraPipeline to LTX2InContextPipeline and LTX2HDRLoraPipeline to LTX2HDRPipeline * Improve LTX2InContextPipeline and LTX2HDRPipeline docstrings * Add export function to directly convert HDR tensors to .mp4 files * Clean up the code/docstrings some more * Move new video_self_attention_mask LTX-2.X transformer arg to end to preserved positional arg ordering * make fix-copies * Simplify HDR export functions to only export to mp4 * Revert LTX2ConditionPipeline default __call__ values back to previous (LTX-2.0) defaults * Inline simple_tone_map and linear_to_srgb into encoder_hdr_tensor_to_mp4 for HDR export * Apply suggestions from review * Refactor HDR video processor to handle output_type in postprocess_hdr_video * Rewrite simple_tone_map as function rather than lambda since otherwise ruff complains * make style and make quality * Refactor HDR pipeline to use audio_scheduler as an optional component * Add initial LTX2ConditionPipeline tests * make style and make quality * Add initial LTX2InContextPipeline tests * Add initial LTX2HDRPipeline tests * Try to fix failing tests for new LTX-2.X pipelines * Propagate suggestions from HDR pipeline to IC and condition pipelines --------- Co-authored-by: YiYi Xu <yixu310@gmail.com> |
||
Wai Ting Cheung
|
fc77592427 |
feat: Add Motif-Video model and pipelines (#13551)
* feat: add Motif Video T2V and I2V pipelines with AdaptiveProjectedGuidance support Add complete Motif Video implementation to diffusers: New Models: - Add MotifVideoTransformer3DModel with T5Gemma2Encoder for multimodal conditioning - Supports text-to-video and image-to-video generation with vision tower integration New Pipelines: - Add MotifVideoPipeline for text-to-video generation - Default resolution: 736x1280, 121 frames, 25 fps - Supports classifier-free guidance and AdaptiveProjectedGuidance - Add MotifVideoImage2VideoPipeline for image-to-video generation - First frame conditioning with vision encoder - Same defaults as T2V pipeline Enhanced Guidance: - Update AdaptiveProjectedGuidance with normalization_dims parameter - Support "spatial" normalization for 5D tensors (per-frame spatial normalization) - Support custom dimension lists for flexible normalization - Update AdaptiveProjectedMixGuidance with same parameter Documentation & Tests: - Add comprehensive API documentation for transformer and pipelines - Add test suites for both T2V and I2V pipelines - Register all new components in __init__ files - Add dummy objects for torch and transformers backends Total: 18 files changed, 3416 insertions(+), 2 deletions(-) * Remove linear quadratic * Remove musicldm * Update docstring * Address vision_encoder comment * Add copy source in I2V pippeline * Refactor _get_prompt_embeds Co-authored-by: Beomgyu Kim <beomgyu.kim@motiftech.io> * Fix a typo * Refactor MotifVideo transformer to use diffusers Attention conventions - Use default Attention class with custom MotifVideoAttnProcessor2_0 - Inline cross-attention in transformer blocks - Use dispatch_attention_fn for backend support - Inherit AttentionMixin for attn_processors/set_attn_processor - Move TransformerBlockRegistry to _helpers.py - Add _repeated_blocks for regional compilation * Use base classes for scheduler and guider * Implement MotifVideoAttention * Update style and quality * Fix a typo * Fix a typo * Fix a typo * Update year * Address rope dtype * Update docstring and remove frame_rate * Address unused sigmas * Add available processors * Address copy from comment * Remove torch.no_grad() * Remove use_attention_mask * Address inline cross-attention * Address compute dtype * Remove unused variables * Merge main APG into this branch and update documentation * Refactor cross attention processor * Remove unused timestep * Inline create_attention_mask * Make guider required * Address encode_prompt comment * Address preprocess_video comment * Use T5Gemma2Encoder in test cases * Address None feature_extractor * Address output type * Renable skipped tests * Update style and quality * Generate standard transformer test case * Add model test case * Remove guider in documentation * Implement cross_attn layer * Remove prepare_negative_prompt * Address latent is None * Clean up feature_extractor * Fix prepare_latents * Remove transformers assertion * Fix style and quality * Fix python utils/check_copies.py --fix_and_overwrite python utils/check_dummies.py --fix_and_overwrite outputs * Add dropout rate to text config * Skip tests requiring guidance_scale * Fix encode_prompt in test cases * Fix test_cpu_offload_forward_pass_twice * Update tests/pipelines/motif_video/test_motif_video.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Update tests/pipelines/motif_video/test_motif_video.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Update tests/pipelines/motif_video/test_motif_video.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Update tests/pipelines/motif_video/test_motif_video_image2video.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Address test_attention_slicing_forward_pass comment * Update tests/pipelines/motif_video/test_motif_video_image2video.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Update tests/pipelines/motif_video/test_motif_video_image2video.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Update tests/pipelines/motif_video/test_motif_video_image2video.py Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> * Skip I2V test cases * Fix style and quality * Add docs to toctree * Fix docs location in toctree and add link in overview * Inline gradient checkpointing * Add _keep_in_fp32_modules for timestep_embedder * Address num_decoder_layers comment * Address guider is not None comment * Remove _keep_in_fp32_modules * Address parameter_dtype comment --------- Co-authored-by: Ken Cheung <ken.cheung@motiftech.io> Co-authored-by: Beomgyu Kim <beomgyu.kim@motiftech.io> Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> Co-authored-by: YiYi Xu <yixu310@gmail.com> |
||
|
Sayak Paul
|
8f14cdefc5 |
[tests] refactor ltx2 autoencoder tests to use latest mixins (#13739)
* refactor ltx2 autoencoder tests to use latest mixins * fix more. * fix tests * is_flaky |
||
|
Sayak Paul
|
adff1cae9f | fix autoencoder memory tests (#13734) | ||
|
Sayak Paul
|
cbdedbaf03 |
[tests] add attention backend tests. (#13174)
* add attention backend tests. * remove existing tests/others/test_attention_backends.py file * modify generate_model_tests.py * remove native. * account for _keep_in_fp32_modules * don't skip when exception is raised. * use is_kernels_available() * mark with compile. * move rtol and atol to methods as defaults. * Apply suggestions from code review Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * up * up |
||
|
Sayak Paul
|
a59f359c8f |
[tests] refactor autoencoderkl tests (#13368)
* refactor autoencoderkl tests * fix tests * confirm coverage * up |
||
Cheung Ka Wai
|
72ea12132e |
add SP support for flash_varlen_hub backend (#13479)
* add mask support for flash backend * fix test case * refactor test * add protection * fix comment * update according to suggestion * revert change * fix according to claude review * add test converage for QwenImage * add SP support and fix non-contiguous mask for flash_varlen kernel * revert change * Update tests/models/testing_utils/parallelism.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * Update tests/models/testing_utils/parallelism.py Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * drop `_padded_to_unpad` * follow `if _parallel_config is None` pattern * rename `attn_mask_2d` * move check to the top * make comment clear * move non-contiguous-attention-mask as default dummy data * revert and update --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
|
jiqing-feng
|
d773308ca7 |
Reduce WanAnimate TorchAO test input sizes to prevent OOM (#13541)
Shrink dummy inputs to avoid OOM on devices without FlashAttention. Reduce hidden_states spatial from 64x64 to 16x16 and frames from 21 to 5, bringing self-attention sequence length from 21,504 to 320. Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
MQ
|
10302496a6 |
[feat] JoyAI-JoyImage-Edit support (#13444)
* [feat] JoyAI-JoyImage-Edit support * [fix] remove rearrange * [refactor] two pass when do cfg * [refactor] remove repa, use wantimetextembeding, refactor modulate code * [refactor] Joyimage Attention refactor * remove vae tiling and autocast * [fix] remove einops from setup.py * [refactor] Refactor JoyImageEditPipeline to use explicit arguments instead of namespace and remove _build_arg * [fix] remove deprecated method decode_latents * [refactor] refactor the image pre-processing logic into a separate VaeImageProcessor subclass * [refactor] add JoyImageAttention to align with Attention + AttnProcessor design and update conversion script for new weight key mapping (e.g. img_attn_qkv -> attn.img_attn_qkv) * [refactor] simplify bucket logic in JoyImageEditImageProcessor by replacing runtime generation with precomputed lookup tables * [fix] remove leftover training-only parameters * [fix] add layerwise casting and fp32 module patterns to JoyImageTransformer3DModel. Reference WanTransformer3DModel to fix layer casting errors during inference. * [test] add JoyImageEditPipeline fast tests and JoyImageEditTransformer3DModel model tests * [fix] fix some pipeline args to support batch inference * [fix] duplicate images to match batch size when fewer images than prompts in JoyImageEditPipeline * [fix] remove no longer used config parameters * Apply style fixes * [fix] remove unused dataclass and rewrite helpers as inline functions * [fix] make dummy objects for JoyImageEdit * [fix] allow test_torch_compile_repeated_blocks to pass * [fix] add examples on JoyImageEditPipeline * fix code style issues with ruff and black * Apply style fixes * [fix] change default num_inference_steps to 40 * [fix] use forward hook to extract pre-norm hidden states for transformers 5.x compatibility * [fix] change the assert to ValueError in pipeline * [fix] rename JoyImageTransformer3DModel to JoyImageEditTransformer3DModel, clean up anything about the alias * [fix] support gradient checkpointing * [refactor] simplify RoPE utilities, inline helpers, copy WanTimeTextImageEmbedding locally and remove unused parameters * [fix] remove _get_text_encoder_ckpt and qwen_processor * [fix] change nn.RMSNorm to FP32LayerNorm * [fix] small fixes for suggestions given by Claude * [refactor] build model using from _pretained instead of config * [refactor] auto-wrap prompt and support text-to-image in JoyImage Edit pipeline * make style, make quality and make fix-copies * [test] small fix to use vocab_size=1024 * [refactor] separate encode_prompt_multiple_images from encode_prompt, support prompt_embeds/prompt_embesd_mask/num_images_per_prompt in edit mode * [test] fix CI: use strict=False for xfail and add @require_torch_accelerator to group offloading test * [refactor] separate image_latents from latents in prepare_latents to align with flux2 * make style --------- Co-authored-by: zhangmaoquan.1 <zhangmaoquan.1@jd.com> Co-authored-by: huangfeice <huangfeice@gmail.com> Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: YiYi Xu <yixu310@gmail.com> |
||
Alan Ponnachan
|
7b107d3836 |
feat: Add Modular Pipeline for Stable Diffusion 3 (SD3) (#13324)
* initial architecture * add blocks to various inits * styling * push tiny-sd3-modular to hub and fix the tests * rename modules * guidance refactoring * Apply style fixes * set default height and width * - skip layer refactoring - add autodocstring to assembled blocks * add description and run autostring script * styling * add descriptions for outputparams and styling * 1. fix imports 2. refactored encoders and inputs 3. refactored for more flat structure 4. styling * fix dtype * Apply style fixes * fix ci failures * resolve review points * Apply style fixes * minor nits * Apply style fixes * Apply suggestion from @yiyixuxu * run make fix-copies --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: YiYi Xu <yixu310@gmail.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> |
||
|
jiqing-feng
|
8ee10d8536 |
Enable TorchAO int4wo quantization tests on XPU (#13537)
* Enable TorchAO int4wo quantization tests on XPU - Remove _int4wo_skip marker that restricted int4wo tests to CUDA only - Add XPU-specific int4_packing_format='plain_int32' for Int4WeightOnlyConfig * add xpu to not skip Signed-off-by: jiqing-feng <jiqing.feng@intel.com> * Apply style fixes --------- Signed-off-by: jiqing-feng <jiqing.feng@intel.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> |
||
|
Sayak Paul
|
cff1e39f1b |
[tests] fix lora tests involving clip. (#13675)
fix lora tests involving clip. |
||
|
Sayak Paul
|
2b9572d6ee |
[core] remove txt_seq_lens from qwen transformer. (#13674)
* remove txt_seq_lens from qwen transformer. * remove unneeded test |
||
Gong Junmin
|
1a8a17b71b |
Add ACE-Step pipeline for text-to-music generation (#13095)
* Add ACE-Step pipeline for text-to-music generation
Rebased on origin/main from the original pr-13095 branch (3 commits squashed).
- AceStepDiTModel: Diffusion Transformer with RoPE, GQA, sliding window,
AdaLN timestep conditioning, and cross-attention.
- AceStepConditionEncoder: fuses text / lyric / timbre into a single
cross-attention sequence.
- AceStepPipeline: text2music / cover / repaint / extract / lego / complete.
- Conversion script for the original checkpoint layout.
- Docs + tests.
* Fix ACE-Step pipeline audio quality and auto-detect turbo/base/sft variants
The PR's original inference produced low-quality audio on turbo because the
pipeline (a) mangled the SFT prompt format, (b) applied classifier-free guidance
with the wrong unconditional embedding (empty-string encoded vs. the learned
`null_condition_emb`), and (c) hardcoded turbo defaults even when loading a
base/SFT checkpoint.
Changes:
* Converter preserves `null_condition_emb` (stored under the condition encoder)
and propagates `is_turbo`/`model_version` into the transformer config so the
pipeline can route per-variant defaults.
* `AceStepConditionEncoder` registers `null_condition_emb` as a learned
parameter matching the original module.
* Pipeline auto-detects variant via `is_turbo`/`model_version` and picks
defaults that match `acestep/inference.py`:
* turbo: steps=8, shift=3.0, guidance_scale=1.0 (no CFG)
* base/SFT: steps=27, shift=1.0, guidance_scale=7.0
* Base/SFT timestep schedule uses the linear+shift transform from
`acestep/models/base/modeling_acestep_v15_base.py`; turbo still uses the
hardcoded 8-step `SHIFT_TIMESTEPS` table.
* CFG reuses the learned `null_condition_emb` and batches the
conditional+unconditional forwards into a single transformer call.
* `SFT_GEN_PROMPT` matches the newline layout in `acestep/constants.py` so the
text encoder sees the same prompt distribution it was trained on.
DiT parity vs. the original ACE-Step 1.5 turbo DiT is bit-identical
(max_abs=0.0 in fp32 eager/SDPA across 4 seed/shape cases) — see
scripts/dit_parity_test.py.
* Add ACE-Step parity test scripts
Two developer-facing parity harnesses live under scripts/:
* dit_parity_test.py — loads the same converted turbo weights into the
original AceStepDiTModel and the diffusers AceStepDiTModel, drives
identical (hidden_states, timestep, timestep_r, encoder_hidden_states,
context_latents) inputs, and asserts max-abs-diff ≤ 1e-5 in fp32
eager/SDPA. Currently passes bit-identical (max_abs=0) across four
shape/seed cases including batched + odd-length paths.
* audio_parity_jieyue.py — full end-to-end audio parity. Given the same
JSON example, runs both the original ACE-Step 1.5 pipeline and the
diffusers AceStepPipeline at matched seed/precision (bf16 + FA2 by
default) and saves side-by-side .wav files for listening verification.
Supports text2music / cover / repaint × turbo / base / sft via a
--matrix mode that writes 18 wavs named
{variant}_{task}_{official,diffusers}.wav.
* Route SFT parity to acestep-v15-sft checkpoint
On jieyue the release tree has a dedicated SFT checkpoint at
checkpoints/acestep-v15-sft with its own modeling_acestep_v15_base.py
shipped under acestep/models/sft/. Point the SFT row of the parity matrix
at that checkpoint / module so we're testing the actual SFT weights, not
the plain base ones.
* audio_parity_jieyue: fix doubled 'acestep-' in cache path; --converted-root flag
Previously the converted-pipeline cache dir was
`/tmp/acestep-<variant>-diffusers` but <variant> already starts with
"acestep-", giving `/tmp/acestep-acestep-v15-turbo-diffusers`. Drop the
prefix.
On jieyue the overlay rootfs (including /tmp) only has a few GB free; a
full turbo conversion needs ~5 GB per variant. Add --converted-root (env
ACESTEP_CONVERTED_ROOT) so the cache can live on vepfs.
* audio_parity_jieyue: two-phase matrix bootstraps cover/repaint from text2music
The ACE-Step release bundle on jieyue doesn't ship sample .wav/.mp3
files, so matrix mode had no default --src-audio and would skip
cover/repaint entirely. Run text2music first for every variant, then
reuse the TURBO official text2music output as the shared source for the
cover/repaint rows. Users can still override with --src-audio.
* audio_parity_jieyue: seed the diffusers generator on the pipeline device
The ORIGINAL ACE-Step pipeline seeds on the execution device
(`torch.Generator(device=device).manual_seed(seed)`), i.e. the CUDA RNG
stream when running on GPU. Previously the parity harness seeded the
diffusers side with a CPU generator, so even though the seed integer
matched, the two sides drew different noise from the outset and the
final outputs were essentially uncorrelated. Use the execution-device
generator on both sides for a fair comparison.
* Fix ACE-Step pipeline: switch to APG guidance + peak normalization
Two issues found after the first jieyue audio parity run:
1. The original base/SFT pipeline uses APG (Adaptive Projected Guidance,
acestep/models/common/apg_guidance.py) with a stateful momentum
buffer and norm/projection steps — NOT vanilla CFG. Using vanilla CFG
produced uncorrelated outputs vs. the reference (pearson ~0.0 on
20 s samples); this PR ports `_apg_forward` + `_APGMomentumBuffer`
and plugs them into the denoising loop when `guidance_scale > 1`.
Momentum is instantiated once per pipeline call (persists across
denoising steps) to match the reference semantics.
2. The post-VAE "anti-clipping normalization" in this pipeline was
`audio /= std * 5` with a `std<1 -> std=1` guard. The original
post-processing in
acestep/core/generation/handler/generate_music_decode.py is simple
peak normalization: `if audio.abs().max() > 1: audio /= peak`. The
std-based proxy both (a) let clips with peak < 1 leak through
unchanged (over-quiet) and (b) failed to bring clipping peaks to
exactly 1 in a bunch of base/SFT cases (observed max=1.000, std=0.200
repeatedly in the first parity run). Switch to peak normalization on
both sides.
Tested via scripts/audio_parity_jieyue.py on A800; re-run pending to
confirm the base/SFT correlation improvements.
* Fix ACE-Step chunk mask values to match the original pipeline
The DiT receives `context_latents = concat(src_latents, chunk_mask)` on the
channel dim, and was trained with chunk_mask values drawn from the three
sentinels documented in acestep/inference.py:
2.0 -> model-decided (default for text2music / cover / full-generation)
1.0 -> keep this latent frame from src_latents (repaint preserved region)
0.0 -> explicitly repaint this frame (only inside the repaint window)
Previously _build_chunk_mask returned all-1.0 for text2music (and cover /
lego), and an inverted 0/1 mask for repaint (1 inside the window, 0 outside).
Either case puts context_latents out of distribution. Switch text2music /
cover to the 2.0 sentinel and flip the repaint mask so it's 1.0 outside /
0.0 inside. Update the repaint src_latents zero-out to multiply by the new
mask (was `1 - chunk_mask`) so the zero region still lines up with the
repaint window.
* Add direct invoker for ACE-Step generate_music (ground truth)
Our earlier audio_parity_jieyue.py reconstructs the original pipeline by
calling AceStepConditionGenerationModel.generate_audio() directly, which
silently skips a lot of the real handler plumbing (conditioning masks,
silence-latent tiling, cover/repaint pre-processing, etc.). That made the
'official' wavs we saved sound wrong — flat, drone-like, not music.
This new script calls acestep.inference.generate_music end-to-end through
the real AceStepHandler, with LM + CoT explicitly disabled so we still have
a deterministic comparison. Use it to generate the ground-truth 'official'
wav for a given JSON example, then separately run the diffusers pipeline
with the same inputs and diff the two.
* run_official_generate_music: call initialize_service to bind a DiT variant
AceStepHandler() is a shell — you have to call handler.initialize_service(
project_root=..., config_path=..., device=..., use_flash_attention=..., ...)
before generate_music will work. Mirror what cli.py does at the equivalent
spot (around cli.py:1400).
* Fix silence-reference for ACE-Step timbre encoder
The root cause for the flat / drone-like outputs I was seeing (including
in my 'official' reconstruction): when no reference_audio is provided the
pipeline was feeding literal zeros to the timbre encoder. The real
handler feeds a slice of the learned `silence_latent` tensor.
The handler also transposes silence_latent on load (see
acestep/core/generation/handler/init_service_loader.py:214:
self.silence_latent = torch.load(...).transpose(1, 2)
) converting [1, 64, 15000] -> [1, 15000, 64] so that
`silence_latent[:, :750, :]` yields the expected [1, 750, 64] shape.
Changes:
* Converter: load silence_latent.pt, transpose to [1, T, C], bake into
the condition_encoder safetensors under key `silence_latent`.
(Also keeps the raw .pt file at the pipeline root for debugging.)
* AceStepConditionEncoder: register `silence_latent` as a persistent
buffer so from_pretrained loads it alongside the trained weights.
* Pipeline: when reference_audio is None, slice
`condition_encoder.silence_latent[:, :timbre_fix_frame, :]` and
broadcast across the batch instead of zeros. Emits a loud warning
(and falls back to zeros) if the buffer is all-zero — that means the
checkpoint was produced by an older converter and should be rebuilt.
* audio_parity_jieyue.py: the reference path now matches the handler's
silence-latent slicing.
Without this fix, every variant/task combo produced drone-like audio
even when my numeric DiT-forward parity claimed they were identical.
* Fix three more ACE-Step pipeline bugs I found by dumping real inputs
Instrumented the live generate_audio call in the real ACE-Step handler and
observed the exact tensors it sees — my diffusers pipeline was wrong in
three independent ways:
1. src_latents for text2music should be silence_latent tiled to
latent_length, NOT zeros. The handler fills no-target cases from
silence_latent_tiled (observed std=0.96). Zeros are OOD for the DiT
context_latents concat and produce drone-like outputs.
2. chunk_mask values cap at 1.0 (not 2.0). The handler starts with a
bool tensor (True inside the generate span, False outside); the
chunk_mask_modes=auto -> 2.0 override does NOT take effect because
the underlying tensor is bool, so setting entry = 2.0 casts to True.
After the later .to(dtype) float cast, the DiT sees 1.0/0.0 — exactly
what I observed in the captured tensor (unique values = [True]).
3. Default shift is 1.0 for ALL variants, including turbo. I was
defaulting turbo to shift=3.0 which picks a different SHIFT_TIMESTEPS
table (the 8-step schedule is keyed by shift, not variant).
Also:
* Added _silence_latent_tiled() helper that slices / tiles the learned
silence_latent (now loaded as a buffer on the condition encoder) to
the requested latent length.
* Repaint path now substitutes silence_latent (not raw zeros) inside
the repaint window — matches conditioning_masks.py.
* audio_parity_jieyue.py mirrors the same src/chunk/shift choices on
its 'original' leg for apples-to-apples parity once the buggy
reconstruction is removed from the picture.
* Add peak+loudness post-normalization to AceStepPipeline
The real pipeline normalizes audio in two stages (see
acestep/audio_utils.py:72 normalize_audio + generate_music_decode.py):
1. if peak > 1: audio /= peak (anti-clip)
2. audio *= target_amp / peak (target_amp = 10 ** (-1/20) ~ 0.891)
Step 2 is loudness normalization to -1 dBFS. Without it diffusers outputs
had peak=1.0 vs the real 0.891 — same music content (pearson was ~0.86
already), just 1.12x louder. Add step 2 after the existing anti-clip step.
* Match acestep/inference.py inference_steps=8 for ALL variants
GenerationParams.inference_steps default is 8 — turbo AND base/SFT. I had
base/SFT defaulting to 27 here, so every base/SFT parity run was comparing
a 27-step diffusers trajectory against an 8-step real trajectory. Different
number of denoising steps means different audio even at fixed seed.
This likely explains the lower base/SFT correlation in my earlier jieyue
runs (turbo was 0.86, base/SFT were 0.32-0.34). Aligning step counts
should bring base/SFT closer to turbo parity.
* Address PR #13095 review: rename classes + reuse diffusers primitives
Response to dg845's PR comments batch 1+2. DiT parity harness still bit-identical
(max_abs=0 on fp32 / SDPA across 4 shape cases).
Transformer file:
* Rename AceStepDiTModel -> AceStepTransformer1DModel (alias kept).
* Rename AceStepDiTLayer -> AceStepTransformerBlock (alias kept).
* Inherit AttentionMixin + CacheMixin on the DiT model.
* Swap in diffusers.models.normalization.RMSNorm for the hand-rolled
AceStepRMSNorm (weight-key-compatible).
* Swap the hand-rolled rotary embedding + apply_rotary for diffusers'
get_1d_rotary_pos_embed + apply_rotary_emb (use_real_unbind_dim=-2 to
match the cat-half convention ACE-Step inherits from Qwen3).
* Use get_timestep_embedding with flip_sin_to_cos=True — keeps the
(cos, sin) ordering of the original sinusoidal. State-dict-compatible.
* Drop max_position_embeddings arg from DiT config (RoPE computes freqs
per call based on seq_len); converter drops it.
* Gradient-checkpoint call now takes just the layer module (matches the
Flux2 idiom).
Pipeline modeling file (pipelines/ace_step/modeling_ace_step.py):
* Moved _pack_sequences + AceStepEncoderLayer here — they aren't used
by the DiT, so they shouldn't live in the transformer file.
* AceStepLyricEncoder + AceStepTimbreEncoder set
_supports_gradient_checkpointing = True and wrap encoder-layer calls
through the checkpointing func when enabled.
* Use diffusers RMSNorm + the RoPE helper from the transformer file
(shared single implementation).
Converter (scripts/convert_ace_step_to_diffusers.py):
* model_index.json now carries AceStepTransformer1DModel.
* Drop max_position_embeddings / use_sliding_window from the emitted
configs.
No numerical regressions: scripts/dit_parity_test.py PASSES with
max_abs=0.0 on fp32/SDPA across short, long, batched, and
padding-path shape variants.
* Address PR #13095 review: pipeline polish + converter HF-hub support
Response to dg845 review comments on the pipeline side. DiT parity still
bit-identical (max_abs=0 across 4 shape cases).
Pipeline (pipelines/ace_step/pipeline_ace_step.py):
* Add `sample_rate` + `latents_per_second` properties sourced from the
VAE config so the pipeline no longer hardcodes 48000 / 25 / 1920.
Propagates through prepare_latents, chunk_mask window math, and the
audio-duration round-trip.
* Add `do_classifier_free_guidance` property (matches LTX2 et al.).
* Add `check_inputs(...)` called from `__call__` before allocating noise.
Validates prompt type, lyrics type, task_type, step count, guidance
scale, shift, cfg interval bounds and repaint window ordering.
* Add `callback_on_step_end` + `callback_on_step_end_tensor_inputs` —
the modern callback form. The legacy `callback` / `callback_steps`
pair is kept for back-compat. Setting `pipe._interrupt = True` inside
the callback stops the loop early.
* Expose `encode_audio(audio)` as a public helper that wraps the tiled
VAE encode + (B, T, D) transpose the pipeline performs internally.
Converter (scripts/convert_ace_step_to_diffusers.py):
* Accept a Hugging Face Hub repo id for `--checkpoint_dir`; resolves it
via `huggingface_hub.snapshot_download` when the argument isn't a
local path.
Exports:
* Register `AceStepTransformer1DModel` in the top-level __init__,
models/__init__, models/transformers/__init__, and dummy_pt_objects so
`from diffusers import AceStepTransformer1DModel` works and the
pipeline loader resolves the new class name from model_index.json.
Deferred for a follow-up (commented inline in the PR): full
`Attention + AttnProcessor + dispatch_attention_fn` refactor and
`FlowMatchEulerDiscreteScheduler` migration — both would benefit from a
dedicated parity re-run and review.
* Fix stale ACE-Step 1.0-era docs / class names in the 1.5 integration
Docs and docstrings still carried a mix of 1.0 paper title, non-existent
`ACE-Step/ACE-Step-v1-5-turbo` hub id, `shift=3.0` turbo default, and
the old `AceStepDiTModel` class name. Cleaned up to match the actual
1.5 release:
* pipelines/ace_step.md: correct citation title ("ACE-Step 1.5: Pushing
the Boundaries of Open-Source Music Generation"), correct repo
(`ace-step/ACE-Step-1.5`), new variants table with real HF ids
(`Ace-Step1.5` / `acestep-v15-base` / `acestep-v15-sft`) and their
per-variant step/CFG defaults, drop the wrong `shift=3.0` tip.
* models/ace_step_transformer.md: page renamed to
`AceStepTransformer1DModel` with a short 1.5-specific description;
`AceStepDiTModel` noted as a backwards-compat alias.
* pipeline_ace_step.py: import, docstring, `Args`, and `__init__`
annotation reference `AceStepTransformer1DModel`; example model id
now `ACE-Step/Ace-Step1.5`; `_variant_defaults` docstring and the
`__call__` variant-fallback comment no longer claim `shift=3.0` /
`27 steps` — real defaults are 8 steps / shift=1.0 across all
variants, guidance=1.0 (turbo) vs 7.0 (base+sft).
* Address PR #13095 review: VAE tiling on AutoencoderOobleck + Timesteps class
Two more deferred review threads from dg845 addressed:
* Move tiled encode/decode onto AutoencoderOobleck
(https://github.com/huggingface/diffusers/pull/13095#discussion_r2785513647).
AutoencoderOobleck now carries `use_tiling` + `tile_sample_min_length` /
`tile_sample_overlap` / `tile_latent_min_length` / `tile_latent_overlap`
attributes and private `_tiled_encode` / `_tiled_decode` methods; the
existing `encode` / `_decode` dispatch to them when tiling is enabled and
the input exceeds the threshold. `AutoencoderMixin.enable_tiling()` is
already inherited.
AceStepPipeline's private `_tiled_encode` / `_tiled_decode` and the
`use_tiled_decode` `__call__` arg are gone; `__init__` now calls
`self.vae.enable_tiling()` so the long-audio memory behaviour is preserved
by default. Users can opt out with `pipe.vae.disable_tiling()`.
Note: the VAE-side tiling concatenates encoder features (h) and samples
the posterior once, instead of the old per-tile `.sample()` calls. This
is the standard diffusers pattern; numerically differs only in the
structure of the noise across tile boundaries.
* Use the Timesteps nn.Module for the sinusoid
(https://github.com/huggingface/diffusers/pull/13095#discussion_r2785420234).
`AceStepTimestepEmbedding` wraps `Timesteps(in_channels, flip_sin_to_cos=
True, downscale_freq_shift=0)` instead of calling `get_timestep_embedding`
directly — reviewer asked for the Module form.
* Address PR #13095 review: refactor AceStepAttention to Attention + AttnProcessor
Splits the monolithic AceStepAttention into the diffusers standard
Attention + AttnProcessor layout:
- AceStepAttention (torch.nn.Module, AttentionModuleMixin) holds the
to_q/to_k/to_v/to_out projections and norm_q/norm_k RMSNorms.
- AceStepAttnProcessor2_0 runs the attention dispatch through
dispatch_attention_fn so users can pick flash / sage / native backends
via model.set_attention_backend(...) or the attention_backend context
manager.
GQA (Q has 16 heads / K,V have 8) is preserved by passing enable_gqa=True
to dispatch_attention_fn instead of repeat_interleave; fusion is disabled
(_supports_qkv_fusion = False) because Q and K,V have different output
sizes.
The converter is updated to rename the six attention sub-keys
(q_proj -> to_q, k_proj -> to_k, v_proj -> to_v, o_proj -> to_out.0,
q_norm -> norm_q, k_norm -> norm_k) on both the DiT decoder path and the
condition encoder path, since AceStepLyricEncoder / AceStepTimbreEncoder
share the same AceStepAttention class.
Addresses review comments r2785433213 and r2785450463.
* Address PR #13095 review: migrate to FlowMatchEulerDiscreteScheduler
Replace the hand-rolled flow-matching Euler loop with
`FlowMatchEulerDiscreteScheduler`. ACE-Step still computes its own shifted /
turbo sigma schedule via `_get_timestep_schedule`, but now passes it to
`scheduler.set_timesteps(sigmas=...)` and delegates the ODE step to
`scheduler.step()`. The scheduler is configured with `num_train_timesteps=1`
and `shift=1.0` so `scheduler.timesteps` stays in `[0, 1]` (the convention the
DiT was trained on) and the scheduler doesn't re-shift already-shifted sigmas.
The scheduler's appended terminal `sigma=0` reproduces the old loop's
final-step "project to x0" case exactly: `prev = x + (0 - t_curr) * v`.
Parity on jieyue (seed=42, bf16 + flash-attn, turbo text2music, 8 steps):
waveform Pearson = 0.999999
spectral Pearson = 1.000000
max |diff| = 2.5e-3 (fp32 step-math vs previous bf16 step-math)
fp32 Euler-loop A/B against the hand-rolled path: max |diff| = 3.6e-7.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Address PR #13095 review: move DiT tests + drop stale test kwargs
- Move the DiT transformer tests out of the pipeline test file into a new
tests/models/transformers/test_models_transformer_ace_step.py that follows
the standard BaseModelTesterConfig + ModelTesterMixin scaffold (matches
test_models_transformer_longcat_audio_dit.py).
- Drop `max_position_embeddings` from the remaining AceStepDiTModel and
AceStepConditionEncoder test fixtures — neither constructor accepts that
argument anymore.
- Drop `use_sliding_window` from the same fixtures — also no longer a
constructor argument (the actual `sliding_window` int kwarg is kept).
- Wire `FlowMatchEulerDiscreteScheduler(num_train_timesteps=1, shift=1.0)`
into `get_dummy_components()` now that the pipeline requires it.
Resolves https://github.com/huggingface/diffusers/pull/13095#discussion_r3115653554,
r3115664850, r3115673059, r3115676580, r3115680700.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Address PR #13095 review from dg845 (2026-04-23)
Fixes 5 review threads + style:
1. Converter now builds `AceStepPipeline` in memory and calls
`save_pretrained`. Previously the hand-written `model_index.json` was
missing the `scheduler` entry — fresh converter output couldn't be loaded
by `AceStepPipeline.from_pretrained` (r3127767785). This also makes the
converter robust to future `__init__` signature changes.
2. `latent_length` uses `math.ceil(...)` instead of `int(...)` so non-integer
products (e.g. `latents_per_second=2.0, audio_duration=0.4 → 0.8`) round up
to `1` instead of truncating to `0` and crashing shape checks (r3127790939).
3. Add `_callback_tensor_inputs = ["latents"]` on `AceStepPipeline` so the
standard diffusers callback tests pick up the right tensor (r3127795954).
4. `AceStepConditionEncoder.silence_latent` no longer hard-codes the channel
dim to 64. The placeholder buffer now uses the `timbre_hidden_dim`
constructor argument, so smaller test configs with `timbre_hidden_dim != 64`
load without shape errors (r3127812932).
5. Revert `self.vae.enable_tiling()` from `AceStepPipeline.__init__`. Users can
call `pipe.vae.enable_tiling()` themselves for long-form generation; that
matches the opt-in convention used by the rest of diffusers (r3127777296).
6. `ruff check --fix` + `ruff format` over all ACE-Step sources (the style fix
dg845 asked for via `@bot /style`).
Also: converter now accepts sharded `model.safetensors.index.json` layouts
alongside the single-file `model.safetensors`, so the 5B XL turbo variant
converts without a pre-processing step.
Parity on jieyue (seed=42, bf16 + flash-attn, turbo text2music 160s, fresh
converter output loaded via `from_pretrained`):
waveform Pearson = 0.999954
spectral Pearson = 0.999977
max |a-b| bf16 = 4.3e-02 (dominated by the VAE tiling default flip)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Address PR #13095 review from yiyixuxu (2026-04-23)
Code-level (22 threads):
1. Delete 3 dev/parity scripts (`scripts/audio_parity_jieyue.py`,
`scripts/dit_parity_test.py`, `scripts/run_official_generate_music.py`)
that shouldn't have been committed.
2. Rename `AutoencoderOobleck._encode_one` → `_encode` to match the convention
used by other diffusers VAEs.
3. Delete the hard-coded `SHIFT_TIMESTEPS` / `VALID_SHIFTS` table in
`pipeline_ace_step.py`: the per-shift turbo schedules are recovered
exactly by `linspace(1, 0, N+1)[:-1]` plus the flow-match shift formula
that the non-turbo branch already uses, so a single code path covers both.
4. Drop the backwards-compat `AceStepDiTModel` / `AceStepDiTLayer` aliases
and every reference (top-level `__init__`, `models/__init__`,
`transformers/__init__`, dummy objects, tests, docs toctree, model card).
`AceStepTransformer1DModel` is the only exported name now.
5. Remove the unused `attention_mask` / `encoder_attention_mask` args from
`AceStepTransformer1DModel.forward`; the model rebuilds its masks from
the sequence shape and never consumed them.
6. In the DiT forward and both encoders, pass `None` instead of an all-zero
`full_attn_mask` / `encoder_4d_mask` to non-sliding attention layers — SDPA
dispatches to a faster kernel when the mask is None.
7. Inline the shared `_run_encoder_layers` helper directly into
`AceStepLyricEncoder.forward` / `AceStepTimbreEncoder.forward` so layer
calls are visible at the forward boundary (diffusers style).
8. Move `is_turbo` / `sample_rate` / `latents_per_second` from `@property`s
that re-read module configs each call to cached attributes populated in
`__init__` (Flux2-style), with a default-ACE-Step fallback when
`self.vae` is offloaded. Drop the now-unused `SAMPLE_RATE = 48000`
module-level constant and the three property definitions.
9. Warn + coerce `guidance_scale` to 1.0 on turbo (guidance-distilled)
checkpoints, following `pipeline_flux2_klein`. Prevents over-guided
audio when users forward their base/sft CFG settings to a turbo pipe.
10. Remove the `logger.warning(...)` paths that triggered on
`silence_latent` missing/zero — those only fired for author-side
unconverted checkpoints and tests; end users always load converted
weights where the buffer is baked in.
11. Drop the redundant `with torch.no_grad():` wrappers inside
`encode_prompt` — the pipeline's `__call__` runs under `torch.no_grad`
already.
12. Strip "reviewer comment on PR #13095" attribution comments from three
docstrings (here and everywhere).
Parity on jieyue (seed=42, bf16 + flash-attn, XL turbo 160s text2music):
waveform Pearson = 0.9747
spectral Pearson = 0.9895
The shift comes from full-attention layers switching `attn_mask=0_tensor` →
`attn_mask=None`, which dispatches to a different SDPA kernel on bf16. The
two outputs are algebraically equivalent for fp32 eager; on bf16+FA the
delta is dominated by kernel-level ULPs, well within the sampler-noise
band (ear-check on the 160s example confirms no audible regression).
Still open — AudioTokenizer/Detokenizer (deferred) + APG guider follow-up
(dims differ from `diffusers.guiders.adaptive_projected_guidance`, not a
drop-in; worth a separate PR).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Address ACE-Step audio token and APG review
* Fix ACE-Step docs CI
* Address ACE-Step pipeline cleanup review
* Fix ACE-Step flash attention sliding windows
* Add ACE-Step callback properties
* Address ACE-Step final review comments
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: YiYi Xu <yixu310@gmail.com>
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
|
||
|
Akshan Krithick
|
b231a6a896 |
Add Ernie-Image modular pipeline (#13498)
* Add Ernie-Image modular pipeline * Address review * Fix alphabetical ordering and generator type_hint * Address review * Add height,width as outputs of prompt enhancer --------- Co-authored-by: YiYi Xu <yixu310@gmail.com> |
||
|
Sayak Paul
|
f7fd76adcd |
[attention backends] fix ring CP for flash and flash 3 (#13182)
* tests: add cp backend and attention backend tests. * up * up * up * fix ring for flash and flash_3 * generate. * Apply suggestions from code review Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com> * up * up --------- Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com> |
||
hlky
|
a37f6f8394 |
Improve trust_remote_code (#13448)
* Robust trust check for custom_pipeline parameter of DiffusionPipeline.from_pretrained method * test_custom_components_from_local_dir * Apply style fixes * fix * Update src/diffusers/utils/dynamic_modules_utils.py Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com> * Adjust tests and allow community pipeline * DIFFUSERS_DISABLE_REMOTE_CODE --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com> |
||
|
Sayak Paul
|
7c88e5fe21 |
[tests] fix group offloading with disk tests (#13491)
fix group offloading with disk tests |
||
|
Dhruv Nair
|
50987b1290 |
[CI] Fix BnB tests (#13481)
* update * update * update |