* feat: implement three RAE encoders(dinov2, siglip2, mae)
* feat: finish first version of autoencoder_rae
* fix formatting
* make fix-copies
* initial doc
* fix latent_mean / latent_var init types to accept config-friendly inputs
* use mean and std convention
* cleanup
* add rae to diffusers script
* use imports
* use attention
* remove unneeded class
* example traiing script
* input and ground truth sizes have to be the same
* fix argument
* move loss to training script
* cleanup
* simplify mixins
* fix training script
* fix entrypoint for instantiating the AutoencoderRAE
* added encoder_image_size config
* undo last change
* fixes from pretrained weights
* cleanups
* address reviews
* fix train script to use pretrained
* fix conversion script review
* latebt normalization buffers are now always registered with no-op defaults
* Update examples/research_projects/autoencoder_rae/README.md
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* Update src/diffusers/models/autoencoders/autoencoder_rae.py
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* use image url
* Encoder is frozen
* fix slow test
* remove config
* use ModelTesterMixin and AutoencoderTesterMixin
* make quality
* strip final layernorm when converting
* _strip_final_layernorm_affine for training script
* fix test
* add dispatch forward and update conversion script
* update training script
* error out as soon as possible and add comments
* Update src/diffusers/models/autoencoders/autoencoder_rae.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* use buffer
* inline
* Update src/diffusers/models/autoencoders/autoencoder_rae.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* remove optional
* _noising takes a generator
* Update src/diffusers/models/autoencoders/autoencoder_rae.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* fix api
* rename
* remove unittest
* use randn_tensor
* fix device map on multigpu
* check if the key is missing in the original state dict and only then add to the allow_missing set
* remove initialize_weights
---------
Co-authored-by: wangyuqi <wangyuqi@MBP-FJDQNJTWYN-0208.local>
Co-authored-by: Kashif Rasul <kashif.rasul@gmail.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* Fix Helios paper link in documentation
Updated the link to the Helios paper for accuracy.
* Fix reference link in HeliosTransformer3DModel documentation
Updated the reference link for the Helios Transformer model paper.
* Update Helios research paper link in documentation
* Update Helios research paper link in documentation
* LTX2 condition pipeline initial commit
* Fix pipeline import error
* Implement LTX-2-style general image conditioning
* Blend denoising output and clean latents in sample space instead of velocity space
* make style and make quality
* make fix-copies
* Rename LTX2VideoCondition image to frames
* Update LTX2ConditionPipeline example
* Remove support for image and video in __call__
* Put latent_idx_from_index logic inline
* Improve comment on using the conditioning mask in denoising loop
* Apply suggestions from code review
Co-authored-by: Álvaro Somoza <asomoza@users.noreply.github.com>
* make fix-copies
* Migrate to Python 3.9+ style type annotations without explicit typing imports
* Forward kwargs from preprocess/postprocess_video to preprocess/postprocess resp.
* Center crop LTX-2 conditions following original code
* Duplicate video and audio position ids if using CFG
* make style and make quality
* Remove unused index_type arg to preprocess_conditions
* Add # Copied from for _normalize_latents
* Fix _normalize_latents # Copied from statement
* Add LTX-2 condition pipeline docs
* Remove TODOs
* Support only unpacked latents (5D for video, 4D for audio)
* Remove # Copied from for prepare_audio_latents
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Álvaro Somoza <asomoza@users.noreply.github.com>
* Implement synchronous onload for offloaded parameters
Add fallback synchronous onload for conditionally-executed modules.
* add test for new code path about group-offloading
* Update tests/hooks/test_group_offloading.py
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* use unittest.skipIf and update the comment
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* Fix LTX-2 image-to-video generation failure in two stages generation
In LTX-2's two-stage image-to-video generation task, specifically after
the upsampling step, a shape mismatch occurs between the `latents` and
the `conditioning_mask`, which causes an error in function
`_create_noised_state`.
Fix it by creating the `conditioning_mask` based on the shape of the
`latents`.
* Add unit test for LTX-2 i2v two stages inference with upsampler
* Downscaling the upsampler in LTX-2 image-to-video unit test
* Apply style fixes
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Fix Flash Attention 3 interface compatibility for new FA3 versions
Newer versions of flash-attn (after Dao-AILab/flash-attention@ed20940)
no longer return lse by default from flash_attn_3_func. The function
now returns just the output tensor unless return_attn_probs=True is
passed.
Updated _wrapped_flash_attn_3 and _flash_varlen_attention_3 to pass
return_attn_probs and handle both old (always tuple) and new (tensor
or tuple) return formats gracefully.
Fixes#12022
* Simplify _wrapped_flash_attn_3 return unpacking
Since return_attn_probs=True is always passed, the result is
guaranteed to be a tuple. Remove the unnecessary isinstance guard.
Z-Image uses CFG formula `pred = pos + scale * (pos - neg)` where
`guidance_scale = 0` means no guidance. The threshold should be `> 0`
instead of `> 1` to match this formula.
Co-authored-by: Hezlich2 <typretypre@gmail.com>
* fix: graceful fallback when attention backends fail to import
## Problem
External attention backends (flash_attn, xformers, sageattention, etc.) may be
installed but fail to import at runtime due to ABI mismatches. For example,
when `flash_attn` is compiled against PyTorch 2.4 but used with PyTorch 2.8,
the import fails with:
```
OSError: .../flash_attn_2_cuda.cpython-311-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEab
```
The current code uses `importlib.util.find_spec()` to check if packages exist,
but this only verifies the package is installed—not that it can actually be
imported. When the import fails, diffusers crashes instead of falling back to
native PyTorch attention.
## Solution
Wrap all external attention backend imports in try-except blocks that catch
`ImportError` and `OSError`. On failure:
1. Log a warning message explaining the issue
2. Set the corresponding `_CAN_USE_*` flag to `False`
3. Set the imported functions to `None`
This allows diffusers to gracefully degrade to PyTorch's native SDPA
(scaled_dot_product_attention) instead of crashing.
## Affected backends
- flash_attn (Flash Attention)
- flash_attn_3 (Flash Attention 3)
- aiter (AMD Instinct)
- sageattention (SageAttention)
- flex_attention (PyTorch Flex Attention)
- torch_npu (Huawei NPU)
- torch_xla (TPU/XLA)
- xformers (Meta xFormers)
## Testing
Tested with PyTorch 2.8.0 and flash_attn 2.7.4.post1 (compiled for PyTorch 2.4).
Before: crashes on import. After: logs warning and uses native attention.
* address review: use single logger and catch RuntimeError
- Move logger to module level instead of creating per-backend loggers
- Add RuntimeError to exception list alongside ImportError and OSError
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Apply style fixes
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* switch to transformers main again./
* more
* up
* up
* fix group offloading.
* attributes
* up
* up
* tie embedding issue.
* fix t5 stuff for more.
* matrix configuration to see differences between 4.57.3 and main failures.
* change qwen expected slice because of how init is handled in v5.
* same stuff.
* up
* up
* Revert "up"
This reverts commit 515dd06db5.
* Revert "up"
This reverts commit 5274ffdd7f.
* up
* up
* fix with peft_format.
* just keep main for easier debugging.
* remove torchvision.
* empty
* up
* up with skyreelsv2 fixes.
* fix skyreels type annotation.
* up
* up
* fix variant loading issues.
* more fixes.
* fix dduf
* fix
* fix
* fix
* more fixes
* fixes
* up
* up
* fix dduf test
* up
* more
* update
* hopefully ,final?
* one last breath
* always install from main
* up
* audioldm tests
* up
* fix PRX tests.
* up
* kandinsky fixes
* qwen fixes.
* prx
* hidream
[gguf] Convert to plain tensor earlier in dequantize_gguf_tensor
Once dequantize_gguf_tensor fetches the quant_type attributed from the
GGUFParamter tensor subclass, there is no further need of running the
actual dequantize operations on the Tensor subclass, we can just convert
to plain tensor right away.
This not only makes PyTorch eager faster, but reduces torch.compile
tracer compile time from 36 seconds to 10 seconds, because there is lot
less code to trace now.
* Guard ftfy import with is_ftfy_available
* Remove xfail for PRX pipeline tests as they appear to work on transformers>4.57.1
* make style and make quality
* Pin setuptools version for dependencies which explicitly depend on pkg_resources
* Revert setuptools pin as k-diffusion pipelines are now deprecated
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* support device type device_maps to work with offloading.
* add tests.
* fix tests
* skip tests where it's not supported.
* empty
* up
* up
* fix allegro.
