* [docs] FLUX.3: modular-only notes, lazy loading, callbacks
Add a tips callout on what differs from DiffusionPipeline (lazy loading,
outputs, guider, offloading, editing blocks), move the prompting advice
into the upsampling section with the pop-the-step example, and add a
per-step callbacks section with a block that takes callback_on_step_end.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* [docs] FLUX.3: callback block walks all denoising loops
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* [docs] modular: add "Coming from DiffusionPipeline" to the quickstart and move per-step callbacks to the callbacks guide
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* [docs] FLUX.3: show the upsampled caption; clarify the index note in the quickstart
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
* [docs] modular: make "Coming from DiffusionPipeline" its own page
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Modular Diffusers is no longer experimental: remove the import-time warning
and the docstring warnings from ModularPipeline, the block base classes, and
every model's modular pipeline, and drop it from the docs overview.
ComponentsManager and the Mellon integration are still moving, so they keep
a warning β added one to the ComponentsManager docs page and to
mellon_node_utils, which had none.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* Add MiniMax-H3
* Keep the MiniMax-H3 VAEs in float32 under torch_dtype casts
* Make MiniMax-H3 modular only
* Assert the VAE float32 pin as a positive contract
* Point the modular tests at the hf-internal-testing tiny repo
* Document performance recipes per hardware class
* Slim hardware loading snippets and PR install note
* Use pinnable int8 config and freeze quantized components
* Scope load time quantization and disable low cpu mem usage under streamed offload
* Minimax h3 follow up (review & refactor) (#14371)
review & refactor
* Regenerate the modular auto docstrings and restyle the H3 docstrings
`make modular-autodoctrings` and `make quality` both failed on CI. The nine
`# auto_docstring` blocks in `modular_blocks_minimax_h3.py` were stale, and the
docstrings of eleven files had not been through `doc-builder style`.
Nothing but docstrings and comments changes here.
One trap worth recording: `utils/modular_auto_docstring.py` shells out to
`ruff` and degrades silently when it is not on `PATH` ("Warning: tool not
found ... Skipping formatting"), which makes it rewrite 42 unrelated pipelines
instead of the one that is stale.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Fix the H3 fast tests against the refactored state contract
`condition_latents` is the VAE encoder block's own output β a list of one latent
per condition β and the after-denoise step unpacks `latents` and drops the
conditioning rows, so neither is meaningful once a request has come back. The
end-to-end tests no longer reach for them; a new standalone test runs
`MiniMaxH3KeyframeVaeEncoderStep` on its own, which is where the list is real.
Three other tests were failing for reasons of their own:
- `test_check_inputs_references` was missing its `@parametrize` and errored at
collection. Added four cases that hit real validation.
- The duration ceiling had drifted between the two blocks that check it.
`before_denoise` warned and reassigned before validating, so it reported `got
362` β a count the caller never passed, right after warning it had rounded
their 346. It now validates first, like `before_encoder` already did, and both
messages read `got 346 (rounded up to 362)`.
- `height` without `width` raised `TypeError: unsupported operand type(s) for %:
'NoneType' and 'int'` on `t2va`, and died inside the resize on `fl2va`. Both
blocks now raise the same error `MiniMaxH3Ref2VASetupStep` already raised.
58 passed -> 71 passed, 5 failed, no errors. What is left: `output_type="latent"`
has no opt-out in the video decode block, `reference_image_short_edge` is a
constructor argument so it does not survive save/reload, an image reference is
PIL-only where the test expects three layouts, and `test_float16_inference`,
which fails on the base branch too.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Stop claiming `num_inference_steps` is optional in the H3 tests
The block declares it `InputParam.template("num_inference_steps", required=True)`
and `expected_workflow_defaults` lists it under `required_inputs` for all three
workflows, so `optional_params` said the opposite in the same file. It was
inherited from the mixin's default set and never examined.
Nothing enforces the requirement today: `_check_inputs` substitutes the declared
default before it tests `required`, and the template carries `default=50`, so
omitting the input runs 50 steps rather than raising. That is not specific to
H3 β 14 call sites on main are in the same position β and it is tracked in #14388.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Fix the two-card recipe, and stop accepting `output_type="latent"`
The two-card section did not run, on this branch or on the base one. A pipeline
resolves a single execution device for all of its components, so the documented
shape β one pipeline whose conditioner is `device_map`ped to the second card and
whose denoiser is moved to the first β built the rotary positions on `cuda:1` and
handed them to a transformer on `cuda:0`:
transformer_minimax_h3.py, in Rope.forward
RuntimeError: Expected all tensors to be on the same device, but found at
least two devices, cuda:1 and cuda:0!
Reproduced against `abc5e9bf7` too, so it is not something the refactor broke;
the section went in as a plausible recipe that was never executed.
The split is therefore between pipelines rather than inside one: pop the text
encoder block into a conditioner of its own, give each pipeline a
`ComponentsManager` pinned to a card, and pass the state from one call to the
other. The managers' hooks are what align `prompt_embeds` onto the denoiser's
card; placing the components by hand instead works too, but then that one tensor
has to be moved explicitly. Verified end to end against the tiny fixture.
`output_type="latent"` is separate. It is not an output format β a request that
wants latents runs a pipeline without the decode blocks β and it used to run the
whole VAE decode before dying inside `postprocess_video` with "latent does not
exist", which does not say what to do instead. The video decode block now rejects
anything it cannot postprocess, before decoding. `test_output_type` covers the
three formats it does accept, and the rejection is a `test_check_inputs` case.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Skip the workflow tests whose tiny repositories cannot serve them
`test_from_pretrained_workflow`, `test_load_components_workflow` and
`test_unload_components` build a pipeline from `pretrained_model_name_or_path`
and compare it against the test class's own blocks. Three fixtures cannot answer
that, in three different ways, and none of them is about the pipelines' code:
- `tiny-anima-modular-pipe` has no `modular_model_index.json`. Anima assembles
its dummy components in `get_pipeline` and never loads from a repository, so
the path it declares had never been exercised.
- `tiny-flux2-klein-modular` names `Flux2KleinBaseAutoBlocks` while its
`_class_name` and `is_distilled` both say distilled. `from_pretrained` honours
`_blocks_class_name`, so it builds base blocks, which declare a `guider` the
distilled ones do not.
- `tiny-qwenimage-edit-modular` names `QwenImageModularPipeline`, so
`from_pretrained` falls back to `QwenImageAutoBlocks`, which declares no
`image_conditioned` workflow.
Skipped with a TODO each, since the fixes are Hub-side. The klein and qwenimage
ones are a single key in `modular_model_index.json`; Anima needs a repository
published.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Declare `reference_image_short_edge` as pipeline config
`MiniMaxH3Ref2VASetupStep` resolves three pieces of released-checkpoint geometry
β the canvas short edge, the canvas area cap, and the separate short edge an
image reference is encoded at β and declared only the first two as `ConfigSpec`s.
The third was a constructor argument, so it did not survive a save and reload:
the reloaded pipeline rebuilt the block from the repository, took the 2048
default instead of the 64 the tests configure, resized reference images at a
different resolution, and landed 0.398 away from the original output.
All three now sit together, which is also what lets the test fixture stop
reaching into the block tree to swap in a configured copy β the shrunken
geometry is one `update_components` call next to the canvas rule it belongs
with, and `test_reference_image_geometry` exercises the released default rather
than constructing the block with an explicit 2048.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Accept a reference image in the three layouts a video reference takes
`MiniMaxH3VideoReference.frames` takes a list of images, a channels-last array
or a channels-first tensor; `MiniMaxH3ImageReference.image` took a PIL image
only, and the two array layouts failed on `entry.image.size` β an element count
on numpy, a bound method on torch β with `TypeError: 'int' object is not
subscriptable`.
Passing an array through as-is would have been worse than the crash. The resize
would silently drop to `F.interpolate`'s nearest-neighbour where a PIL image
gets LANCZOS, and the VAE encode does `np.array(image).permute(2, 0, 1)`, which
fixes the axes of an image and scrambles those of a channels-first tensor. So
the layouts are normalized onto a PIL image before any of that, through the
processor component the block already declares: `pt_to_numpy` for the channel
axis and `numpy_to_pil` for the array. Neither converts dtype, and
`numpy_to_pil` scales by 255, so `uint8` is normalized on the original object
first β after `pt_to_numpy` a `uint8` tensor is floats over `[0, 255]` and a
later dtype check would miss it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Read the model's constants off the pipeline rather than the module
`before_denoise.py` imported `MINIMAX_H3_AUDIO_CHANNELS`, `MINIMAX_H3_AUDIO_TAG`
and `MINIMAX_H3_VIDEO_TAG` and read them inside its layout builders, even though
two of the three were already exposed as pipeline properties and used that way
everywhere else. They are arguments now, passed from `components.*` at the two
`__call__` sites; `audio_tag` needed the property `text_tag` and `video_tag`
already had, which is presumably why it was the one still reaching for the
global.
Same treatment where a helper cannot see a pipeline: `resolve_canvas_size` and
`audio_latent_num_frames` take the constants as default arguments instead of
closing over them, and `_normalize_video_condition` takes the rate it resamples
onto, so `before_encoder.py` imports none of them at all. What is left of the
constants is their definitions, the properties that expose them, and default
argument values.
Two helpers with a single caller each are inlined at it: the two rotary-time
sums, which are a parity contract β the reference sums the same series pairwise
in one place and sequentially in the other, and the orders differ in the last
ulp from 16 latent frames onwards. Both inlined sums are bit-identical to the
functions they replace across 1..399 latent frames, and the two still disagree
on 373 of those counts. `_frame_position_grid` stays, and `build_packed_sequence`
now calls it rather than repeating its five lines.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Reject a reference video too short for the conditioner to read
The conditioner reads a reference video at 2 fps and Qwen3-VL merges the sampled
frames in groups of two, so anything under 13 frames at 24 fps samples down to a
single frame and dies inside transformers:
image_processing_glm4v.py
ValueError: t:1 must be larger than temporal_factor:2
which names neither the reference nor the rate that produced it, and only fires
on the versions carrying that check β it passed locally and failed on CI.
`_sample_video_condition_frames` now rejects it where the sampling happens, with
the bound derived from the rates rather than hardcoded: "must run at least 13
frames at 24 fps (0.54 seconds), got 4".
`test_reference_media_layouts` was building exactly such a reference, four
frames, which is what surfaced this. It uses a second of video now, like
`test_reference_combinations` already did.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Drop the padded-layout attention mask
`forward` built a boolean attention mask whenever a row carried a negative
modality tag, reproducing the reference implementation's `cu_seqlens = [0, used,
S]` split of the padding tail it adds for FlashAttention. Nothing in this port
ever produces such a row: both packers partition `[0, sequence_length)` and write
only the text, audio and video tags, so the branch was unreachable and the mask
was never built. The `clamp(min=0)` that kept a `-1` from indexing the AdaLN
table backwards goes with it.
Removing it is not only dead-code removal. `if bool(is_pad.any())` is a
data-dependent branch, which has no fullgraph representation, and it was the
reason whole-model `fullgraph=True` compilation and `torch.export` were skipped:
4 passed, 1 skipped tests -k "compile or export or aot"
Both now run. The model tests go from 40 passed / 4 skipped to 41 passed /
2 skipped, against the same 8 pre-existing memory-offload failures.
`attention_mask` stays on the processor, attention and block signatures β it is
the signature every other processor in diffusers has, and a custom one may want
it β but the model itself never passes anything but `None`, so attention runs
unmasked over the one document and every backend stays available.
Two of the four documentation examples have been re-run against this and come
back bit-identical, frames and audio: `t2va` at the full 768x1344 canvas and
`fl2va`. The two `ref2va` ones are still to run.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Apply suggestions from code review
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
---------
Co-authored-by: YiYi Xu <yixu310@gmail.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* up
* up up
* update outputs
* style
* add modular_auto_docstring!
* more auto docstring
* style
* up up up
* more more
* up
* address feedbacks
* add TODO in the description for empty docstring
* refactor based on dhruv's feedback: remove the class method
* add template method
* up
* up up up
* apply auto docstring
* make style
* rmove space in make docstring
* Apply suggestions from code review
* revert change in z
* fix
* Apply style fixes
* include auto-docstring check in the modular ci. (#13004)
* initial support: workflow
* up up
* treeat loop sequential pipeline blocks as leaf
* update qwen image docstring note
* add workflow support for sdxl
* add a test suit
* add test for qwen-image
* refactor flux a bit, seperate modular_blocks into modular_blocks_flux and modular_blocks_flux_kontext + support workflow
* refactor flux2: seperate blocks for klein_base + workflow
* qwen: remove import support for stuff other than the default blocks
* add workflow support for wan
* sdxl: remove some imports:
* refactor z
* update flux2 auto core denoise
* add workflow test for z and flux2
* Apply suggestions from code review
* Apply suggestions from code review
* add test for flux
* add workflow test for flux
* add test for flux-klein
* sdxl: modular_blocks.py -> modular_blocks_stable_diffusion_xl.py
* style
* up
* add auto docstring
* workflow_names -> available_workflows
* fix workflow test for klein base
* Apply suggestions from code review
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
* fix workflow tests
* qwen: edit -> image_conditioned to be consistent with flux kontext/2 such
* remove Optional
* update type hints
* update guider update_components
* fix more
* update docstring auto again
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: yiyi@huggingface.co <yiyi@ip-26-0-160-103.ec2.internal>
Co-authored-by: yiyi@huggingface.co <yiyi@ip-26-0-161-123.ec2.internal>
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
* add metadata field to input/output param
* refactor mellonparam: move the template outside, add metaclass, define some generic template for custom node
* add from_custom_block
* style
* up up fix
* add mellon guide
* add to toctree
* style
* add mellon_types
* style
* mellon_type -> inpnt_types + output_types
* update doc
* add quant info to components manager
* fix more
* up up
* fix components manager
* update custom block guide
* update
* style
* add a warn for mellon and add new guides to overview
* Apply suggestions from code review
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/en/modular_diffusers/mellon.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* more update on custom block guide
* Update docs/source/en/modular_diffusers/mellon.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* a few mamual
* apply suggestion: turn into bullets
* support define mellon meta with MellonParam directly, and update doc
* add the video
---------
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: yiyi@huggingface.co <yiyi@ip-26-0-160-103.ec2.internal>
* add a real quick start guide
* Update docs/source/en/modular_diffusers/quickstart.md
* update a bit more
* fix
* Apply suggestions from code review
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/en/modular_diffusers/quickstart.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/en/modular_diffusers/quickstart.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* update more
* Apply suggestions from code review
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* address more feedbacks: move components amnager earlier, explain blocks vs sub-blocks etc
* more
* remove the link to mellon guide, not exist in this PR yet
---------
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
