* Basic implementation of request scheduling
* Basic editing in SD and Flux Pipelines
* Small Fix
* Fix
* Update for more pipelines
* Add examples/server-async
* Add examples/server-async
* Updated RequestScopedPipeline to handle a single tokenizer lock to avoid race conditions
* Fix
* Fix _TokenizerLockWrapper
* Fix _TokenizerLockWrapper
* Delete _TokenizerLockWrapper
* Fix tokenizer
* Update examples/server-async
* Fix server-async
* Optimizations in examples/server-async
* We keep the implementation simple in examples/server-async
* Update examples/server-async/README.md
* Update examples/server-async/README.md for changes to tokenizer locks and backward-compatible retrieve_timesteps
* The changes to the diffusers core have been undone and all logic is being moved to exmaples/server-async
* Update examples/server-async/utils/*
* Fix BaseAsyncScheduler
* Rollback in the core of the diffusers
* Update examples/server-async/README.md
* Complete rollback of diffusers core files
* Simple implementation of an asynchronous server compatible with SD3-3.5 and Flux Pipelines
* Update examples/server-async/README.md
* Fixed import errors in 'examples/server-async/serverasync.py'
* Flux Pipeline Discard
* Update examples/server-async/README.md
* Apply style fixes
* Add thread-safe wrappers for components in pipeline
Refactor requestscopedpipeline.py to add thread-safe wrappers for tokenizer, VAE, and image processor. Introduce locking mechanisms to ensure thread safety during concurrent access.
* Add wrappers.py
* Apply style fixes
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Store vae.config.scaling_factor to prevent missing attr reference
In sdxl advanced dreambooth training script
vae.config.scaling_factor becomes inaccessible after: del vae
when: --cache_latents, and no --validation_prompt
Co-authored-by: Teriks <Teriks@users.noreply.github.com>
Co-authored-by: Linoy Tsaban <57615435+linoytsaban@users.noreply.github.com>
Co-authored-by: YiYi Xu <yixu310@gmail.com>
* 3 files
* add conditoinal pipeline
* refactor qwen modular
* add layered
* up up
* u p
* add to import
* more refacotr, make layer work
* clean up a bit git add src
* more
* style
* style
* Adding torchao version guard for floatx usage
Summary: TorchAO removing floatx support, added version guard in quantization_config.py
* Adding torchao version guard for floatx usage
Summary: TorchAO removing floatx support, added version guard in quantization_config.py
Altered tests in test_torchao.py to version guard floatx
Created new test to verify version guard of floatx support
* Adding torchao version guard for floatx usage
Summary: TorchAO removing floatx support, added version guard in quantization_config.py
Altered tests in test_torchao.py to version guard floatx
Created new test to verify version guard of floatx support
* Adding torchao version guard for floatx usage
Summary: TorchAO removing floatx support, added version guard in quantization_config.py
Altered tests in test_torchao.py to version guard floatx
Created new test to verify version guard of floatx support
* Adding torchao version guard for floatx usage
Summary: TorchAO removing floatx support, added version guard in quantization_config.py
Altered tests in test_torchao.py to version guard floatx
Created new test to verify version guard of floatx support
* Adding torchao version guard for floatx usage
Summary: TorchAO removing floatx support, added version guard in quantization_config.py
Altered tests in test_torchao.py to version guard floatx
Created new test to verify version guard of floatx support
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* Initial LTX 2.0 transformer implementation
* Add tests for LTX 2 transformer model
* Get LTX 2 transformer tests working
* Rename LTX 2 compile test class to have LTX2
* Remove RoPE debug print statements
* Get LTX 2 transformer compile tests passing
* Fix LTX 2 transformer shape errors
* Initial script to convert LTX 2 transformer to diffusers
* Add more LTX 2 transformer audio arguments
* Allow LTX 2 transformer to be loaded from local path for conversion
* Improve dummy inputs and add test for LTX 2 transformer consistency
* Fix LTX 2 transformer bugs so consistency test passes
* Initial implementation of LTX 2.0 video VAE
* Explicitly specify temporal and spatial VAE scale factors when converting
* Add initial LTX 2.0 video VAE tests
* Add initial LTX 2.0 video VAE tests (part 2)
* Get diffusers implementation on par with official LTX 2.0 video VAE implementation
* Initial LTX 2.0 vocoder implementation
* Use RMSNorm implementation closer to original for LTX 2.0 video VAE
* start audio decoder.
* init registration.
* up
* simplify and clean up
* up
* Initial LTX 2.0 text encoder implementation
* Rough initial LTX 2.0 pipeline implementation
* up
* up
* up
* up
* Add imports for LTX 2.0 Audio VAE
* Conversion script for LTX 2.0 Audio VAE Decoder
* Add Audio VAE logic to T2V pipeline
* Duplicate scheduler for audio latents
* Support num_videos_per_prompt for prompt embeddings
* LTX 2.0 scheduler and full pipeline conversion
* Add script to test full LTX2Pipeline T2V inference
* Fix pipeline return bugs
* Add LTX 2 text encoder and vocoder to ltx2 subdirectory __init__
* Fix more bugs in LTX2Pipeline.__call__
* Improve CPU offload support
* Fix pipeline audio VAE decoding dtype bug
* Fix video shape error in full pipeline test script
* Get LTX 2 T2V pipeline to produce reasonable outputs
* Make LTX 2.0 scheduler more consistent with original code
* Fix typo when applying scheduler fix in T2V inference script
* Refactor Audio VAE to be simpler and remove helpers (#7)
* remove resolve causality axes stuff.
* remove a bunch of helpers.
* remove adjust output shape helper.
* remove the use of audiolatentshape.
* move normalization and patchify out of pipeline.
* fix
* up
* up
* Remove unpatchify and patchify ops before audio latents denormalization (#9)
---------
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* Add support for I2V (#8)
* start i2v.
* up
* up
* up
* up
* up
* remove uniform strategy code.
* remove unneeded code.
* Denormalize audio latents in I2V pipeline (analogous to T2V change) (#11)
* test i2v.
* Move Video and Audio Text Encoder Connectors to Transformer (#12)
* Denormalize audio latents in I2V pipeline (analogous to T2V change)
* Initial refactor to put video and audio text encoder connectors in transformer
* Get LTX 2 transformer tests working after connector refactor
* precompute run_connectors,.
* fixes
* Address review comments
* Calculate RoPE double precisions freqs using torch instead of np
* Further simplify LTX 2 RoPE freq calc
* Make connectors a separate module (#18)
* remove text_encoder.py
* address yiyi's comments.
* up
* up
* up
* up
---------
Co-authored-by: sayakpaul <spsayakpaul@gmail.com>
* up (#19)
* address initial feedback from lightricks team (#16)
* cross_attn_timestep_scale_multiplier to 1000
* implement split rope type.
* up
* propagate rope_type to rope embed classes as well.
* up
* When using split RoPE, make sure that the output dtype is same as input dtype
* Fix apply split RoPE shape error when reshaping x to 4D
* Add export_utils file for exporting LTX 2.0 videos with audio
* Tests for T2V and I2V (#6)
* add ltx2 pipeline tests.
* up
* up
* up
* up
* remove content
* style
* Denormalize audio latents in I2V pipeline (analogous to T2V change)
* Initial refactor to put video and audio text encoder connectors in transformer
* Get LTX 2 transformer tests working after connector refactor
* up
* up
* i2v tests.
* up
* Address review comments
* Calculate RoPE double precisions freqs using torch instead of np
* Further simplify LTX 2 RoPE freq calc
* revert unneded changes.
* up
* up
* update to split style rope.
* up
---------
Co-authored-by: Daniel Gu <dgu8957@gmail.com>
* up
* use export util funcs.
* Point original checkpoint to LTX 2.0 official checkpoint
* Allow the I2V pipeline to accept image URLs
* make style and make quality
* remove function map.
* remove args.
* update docs.
* update doc entries.
* disable ltx2_consistency test
* Simplify LTX 2 RoPE forward by removing coords is None logic
* make style and make quality
* Support LTX 2.0 audio VAE encoder
* Apply suggestions from code review
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* Remove print statement in audio VAE
* up
* Fix bug when calculating audio RoPE coords
* Ltx 2 latent upsample pipeline (#12922)
* Initial implementation of LTX 2.0 latent upsampling pipeline
* Add new LTX 2.0 spatial latent upsampler logic
* Add test script for LTX 2.0 latent upsampling
* Add option to enable VAE tiling in upsampling test script
* Get latent upsampler working with video latents
* Fix typo in BlurDownsample
* Add latent upsample pipeline docstring and example
* Remove deprecated pipeline VAE slicing/tiling methods
* make style and make quality
* When returning latents, return unpacked and denormalized latents for T2V and I2V
* Add model_cpu_offload_seq for latent upsampling pipeline
---------
Co-authored-by: Daniel Gu <dgu8957@gmail.com>
* Fix latent upsampler filename in LTX 2 conversion script
* Add latent upsample pipeline to LTX 2 docs
* Add dummy objects for LTX 2 latent upsample pipeline
* Set default FPS to official LTX 2 ckpt default of 24.0
* Set default CFG scale to official LTX 2 ckpt default of 4.0
* Update LTX 2 pipeline example docstrings
* make style and make quality
* Remove LTX 2 test scripts
* Fix LTX 2 upsample pipeline example docstring
* Add logic to convert and save a LTX 2 upsampling pipeline
* Document LTX2VideoTransformer3DModel forward pass
---------
Co-authored-by: sayakpaul <spsayakpaul@gmail.com>
* docs: add comprehensive docstrings and refine type hints for EDM scheduler methods and config parameters.
* refactor: Add type hints to DPM-Solver scheduler methods.
* feat: Add transformer cache context for conditional and unconditional predictions for skyreels-v2 pipes.
* docs: Remove SkyReels-V2 FLF2V model link and add contributor attribution.
* LTX Video 0.9.8 long multi prompt
* Further align comfyui
- Added the “LTXEulerAncestralRFScheduler” scheduler, aligned with [sample_euler_ancestral_RF](7d6103325e/comfy/k_diffusion/sampling.py (L234))
- Updated the LTXI2VLongMultiPromptPipeline.from_pretrained() method:
- Now uses LTXEulerAncestralRFScheduler by default, for better compatibility with the ComfyUI LTXV workflow.
- Changed the default value of cond_strength from 1.0 to 0.5, aligning with ComfyUI’s default.
- Optimized cross-window overlap blending: moved the latent-space guidance injection to before the UNet and after each step, aligned with[KSamplerX0Inpaint]([ComfyUI/comfy/samplers.py at master · comfyanonymous/ComfyUI](https://github.com/comfyanonymous/ComfyUI/blob/master/comfy/samplers.py#L391))
- Adjusted the default value of skip_steps_sigma_threshold to 1.
* align with diffusers contribute rule
* Add new pipelines and update imports
* Enhance LTXI2VLongMultiPromptPipeline with noise rescaling
Refactor LTXI2VLongMultiPromptPipeline to improve documentation and add noise rescaling functionality.
* Clean up comments in scheduling_ltx_euler_ancestral_rf.py
Removed design notes and limitations from the implementation.
* Enhance video generation example with scheduler
Updated LTXI2VLongMultiPromptPipeline example to include LTXEulerAncestralRFScheduler for ComfyUI parity.
* clean up
* style
* copies
* import ltx scheduler
* copies
* fix
* fix more
* up up
* up up up
* up upup
* Apply suggestions from code review
* Update docs/source/en/api/pipelines/ltx_video.md
* Update docs/source/en/api/pipelines/ltx_video.md
---------
Co-authored-by: yiyixuxu <yixu310@gmail.com>
* [Flux.1] improve pos embed for ascend npu by setting it back to npu computation.
* [Flux.2] improve pos embed for ascend npu by setting it back to npu computation.
* [LongCat-Image] improve pos embed for ascend npu by setting it back to npu computation.
* [Ovis-Image] improve pos embed for ascend npu by setting it back to npu computation.
* Remove unused import of is_torch_npu_available
---------
Co-authored-by: zhangtao <zhangtao529@huawei.com>
* Community Pipeline: Add z-image differential img2img
* add pipeline for z-image differential img2img diffusion examples : run make style , make quality, and fix white spaces in example doc string.
---------
Co-authored-by: r4inm4ker <jefri.yeh@gmail.com>
* fix torchao quantizer for new torchao versions
Summary:
`torchao==0.16.0` (not yet released) has some bc-breaking changes, this
PR fixes the diffusers repo with those changes. Specifics on the
changes:
1. `UInt4Tensor` is removed: https://github.com/pytorch/ao/pull/3536
2. old float8 tensors v1 are removed: https://github.com/pytorch/ao/pull/3510
In this PR:
1. move the logger variable up (not sure why it was in the middle of the
file before) to get better error messages
2. gate the old torchao objects by torchao version
Test Plan:
import diffusers objects with new versions of torchao works:
```bash
> python -c "import torchao; print(torchao.__version__); from diffusers import StableDiffusionPipeline"
0.16.0.dev20251229+cu129
```
Reviewers:
Subscribers:
Tasks:
Tags:
* Apply style fixes
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Add z-image-omni-base implementation
* Merged into one transformer for Z-Image.
* Fix bugs for controlnet after merging the main branch new feature.
* Fix for auto_pipeline, Add Styling.
* Refactor noise handling and modulation
- Add select_per_token function for per-token value selection
- Separate adaptive modulation logic
- Cleanify t_noisy/clean variable naming
- Move image_noise_mask handler from forward to pipeline
* Styling & Formatting.
* Rewrite code with more non-forward func & clean forward.
1.Change to one forward with shorter code with omni code (None).
2.Split out non-forward funcs: _build_unified_sequence, _prepare_sequence, patchify, pad.
* Styling & Formatting.
* Manual check fix-copies in controlnet, Add select_per_token, _patchify_image, _pad_with_ids; Styling.
* Add Import in pipeline __init__.py.
---------
Co-authored-by: Jerry Qilong Wu <xinglong.wql@alibaba-inc.com>
Co-authored-by: YiYi Xu <yixu310@gmail.com>
Use `T5Tokenizer` instead of `MT5Tokenizer`
Given that the `MT5Tokenizer` in `transformers` is just a "re-export" of
`T5Tokenizer` as per
https://github.com/huggingface/transformers/blob/v4.57.3/src/transformers/models/mt5/tokenization_mt5.py
)on latest available stable Transformers i.e., v4.57.3), this commit
updates the imports to point to `T5Tokenizer` instead, so that those
still work with Transformers v5.0.0rc0 onwards.
* move node registry to mellon
* up
* fix
* modula rpipeline update: filter out none for input_names, fix default blocks for pipe.init() and allow user pass additional kwargs_type in a dict
* qwen modular refactor, unpack before decode
* update mellon node config, adding* to required_inputs and required_model_inputs
* modularpipeline.from_pretrained: error out if no config found
* add a component_names property to modular blocks to be consistent!
* flux image_encoder -> vae_encoder
* controlnet_bundle
* refator MellonNodeConfig MellonPipelineConfig
* refactor & simplify mellon utils
* vae_image_encoder -> vae_encoder
* mellon config save keep key order
* style + copies
* add kwargs input for zimage
* cosmos predict2.5 base: convert chkpt & pipeline
- New scheduler: scheduling_flow_unipc_multistep.py
- Changes to TransformerCosmos for text embeddings via crossattn_proj
* scheduler cleanup
* simplify inference pipeline
* cleanup scheduler + tests
* Basic tests for flow unipc
* working b2b inference
* Rename everything
* Tests for pipeline present, but not working (predict2 also not working)
* docstring update
* wrapper pipelines + make style
* remove unnecessary files
* UniPCMultistep: support use_karras_sigmas=True and use_flow_sigmas=True
* use UniPCMultistepScheduler + fix tests for pipeline
* Remove FlowUniPCMultistepScheduler
* UniPCMultistepScheduler for use_flow_sigmas=True & use_karras_sigmas=True
* num_inference_steps=36 due to bug in scheduler used by predict2.5
* Address comments
* make style + make fix-copies
* fix tests + remove references to old pipelines
* address comments
* add revision in from_pretrained call
* fix tests
