* add ltx2 pipeline tests.
* up
* up
* up
* up
* remove content
* style
* Denormalize audio latents in I2V pipeline (analogous to T2V change)
* Initial refactor to put video and audio text encoder connectors in transformer
* Get LTX 2 transformer tests working after connector refactor
* up
* up
* i2v tests.
* up
* Address review comments
* Calculate RoPE double precisions freqs using torch instead of np
* Further simplify LTX 2 RoPE freq calc
* revert unneded changes.
* up
* up
* update to split style rope.
* up
---------
Co-authored-by: Daniel Gu <dgu8957@gmail.com>
* Denormalize audio latents in I2V pipeline (analogous to T2V change)
* Initial refactor to put video and audio text encoder connectors in transformer
* Get LTX 2 transformer tests working after connector refactor
* precompute run_connectors,.
* fixes
* Address review comments
* Calculate RoPE double precisions freqs using torch instead of np
* Further simplify LTX 2 RoPE freq calc
* Make connectors a separate module (#18)
* remove text_encoder.py
* address yiyi's comments.
* up
* up
* up
* up
---------
Co-authored-by: sayakpaul <spsayakpaul@gmail.com>
* remove resolve causality axes stuff.
* remove a bunch of helpers.
* remove adjust output shape helper.
* remove the use of audiolatentshape.
* move normalization and patchify out of pipeline.
* fix
* up
* up
* Remove unpatchify and patchify ops before audio latents denormalization (#9)
---------
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* cosmos predict2.5 base: convert chkpt & pipeline
- New scheduler: scheduling_flow_unipc_multistep.py
- Changes to TransformerCosmos for text embeddings via crossattn_proj
* scheduler cleanup
* simplify inference pipeline
* cleanup scheduler + tests
* Basic tests for flow unipc
* working b2b inference
* Rename everything
* Tests for pipeline present, but not working (predict2 also not working)
* docstring update
* wrapper pipelines + make style
* remove unnecessary files
* UniPCMultistep: support use_karras_sigmas=True and use_flow_sigmas=True
* use UniPCMultistepScheduler + fix tests for pipeline
* Remove FlowUniPCMultistepScheduler
* UniPCMultistepScheduler for use_flow_sigmas=True & use_karras_sigmas=True
* num_inference_steps=36 due to bug in scheduler used by predict2.5
* Address comments
* make style + make fix-copies
* fix tests + remove references to old pipelines
* address comments
* add revision in from_pretrained call
* fix tests
* add vae
* Initial commit for Flux 2 Transformer implementation
* add pipeline part
* small edits to the pipeline and conversion
* update conversion script
* fix
* up up
* finish pipeline
* Remove Flux IP Adapter logic for now
* Remove deprecated 3D id logic
* Remove ControlNet logic for now
* Add link to ViT-22B paper as reference for parallel transformer blocks such as the Flux 2 single stream block
* update pipeline
* Don't use biases for input projs and output AdaNorm
* up
* Remove bias for double stream block text QKV projections
* Add script to convert Flux 2 transformer to diffusers
* make style and make quality
* fix a few things.
* allow sft files to go.
* fix image processor
* fix batch
* style a bit
* Fix some bugs in Flux 2 transformer implementation
* Fix dummy input preparation and fix some test bugs
* fix dtype casting in timestep guidance module.
* resolve conflicts.,
* remove ip adapter stuff.
* Fix Flux 2 transformer consistency test
* Fix bug in Flux2TransformerBlock (double stream block)
* Get remaining Flux 2 transformer tests passing
* make style; make quality; make fix-copies
* remove stuff.
* fix type annotaton.
* remove unneeded stuff from tests
* tests
* up
* up
* add sf support
* Remove unused IP Adapter and ControlNet logic from transformer (#9)
* copied from
* Apply suggestions from code review
Co-authored-by: YiYi Xu <yixu310@gmail.com>
Co-authored-by: apolinário <joaopaulo.passos@gmail.com>
* up
* up
* up
* up
* up
* Refactor Flux2Attention into separate classes for double stream and single stream attention
* Add _supports_qkv_fusion to AttentionModuleMixin to allow subclasses to disable QKV fusion
* Have Flux2ParallelSelfAttention inherit from AttentionModuleMixin with _supports_qkv_fusion=False
* Log debug message when calling fuse_projections on a AttentionModuleMixin subclass that does not support QKV fusion
* Address review comments
* Update src/diffusers/pipelines/flux2/pipeline_flux2.py
Co-authored-by: YiYi Xu <yixu310@gmail.com>
* up
* Remove maybe_allow_in_graph decorators for Flux 2 transformer blocks (#12)
* up
* support ostris loras. (#13)
* up
* update schdule
* up
* up (#17)
* add training scripts (#16)
* add training scripts
Co-authored-by: Linoy Tsaban <linoytsaban@gmail.com>
* model cpu offload in validation.
* add flux.2 readme
* add img2img and tests
* cpu offload in log validation
* Apply suggestions from code review
* fix
* up
* fixes
* remove i2i training tests for now.
---------
Co-authored-by: Linoy Tsaban <linoytsaban@gmail.com>
Co-authored-by: linoytsaban <linoy@huggingface.co>
* up
---------
Co-authored-by: yiyixuxu <yixu310@gmail.com>
Co-authored-by: Daniel Gu <dgu8957@gmail.com>
Co-authored-by: yiyi@huggingface.co <yiyi@ip-10-53-87-203.ec2.internal>
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
Co-authored-by: apolinário <joaopaulo.passos@gmail.com>
Co-authored-by: yiyi@huggingface.co <yiyi@ip-26-0-160-103.ec2.internal>
Co-authored-by: Linoy Tsaban <linoytsaban@gmail.com>
Co-authored-by: linoytsaban <linoy@huggingface.co>
The function load_model_dict_into_meta was moved from modeling_utils.py to
model_loading_utils.py but the imports in the conversion scripts were not
updated, causing ImportError when running these scripts.
This fixes the import in 6 conversion scripts:
- scripts/convert_sd3_to_diffusers.py
- scripts/convert_stable_cascade_lite.py
- scripts/convert_stable_cascade.py
- scripts/convert_stable_audio.py
- scripts/convert_sana_to_diffusers.py
- scripts/convert_sana_controlnet_to_diffusers.py
Fixes#12606
* rename photon to prx
* rename photon into prx
* Revert .gitignore to state before commit b7fb0fe9d6
* rename photon to prx
* rename photon into prx
* Revert .gitignore to state before commit b7fb0fe9d6
* make fix-copies
* Add Photon model and pipeline support
This commit adds support for the Photon image generation model:
- PhotonTransformer2DModel: Core transformer architecture
- PhotonPipeline: Text-to-image generation pipeline
- Attention processor updates for Photon-specific attention mechanism
- Conversion script for loading Photon checkpoints
- Documentation and tests
* just store the T5Gemma encoder
* enhance_vae_properties if vae is provided only
* remove autocast for text encoder forwad
* BF16 example
* conditioned CFG
* remove enhance vae and use vae.config directly when possible
* move PhotonAttnProcessor2_0 in transformer_photon
* remove einops dependency and now inherits from AttentionMixin
* unify the structure of the forward block
* update doc
* update doc
* fix T5Gemma loading from hub
* fix timestep shift
* remove lora support from doc
* Rename EmbedND for PhotoEmbedND
* remove modulation dataclass
* put _attn_forward and _ffn_forward logic in PhotonBlock's forward
* renam LastLayer for FinalLayer
* remove lora related code
* rename vae_spatial_compression_ratio for vae_scale_factor
* support prompt_embeds in call
* move xattention conditionning out computation out of the denoising loop
* add negative prompts
* Use _import_structure for lazy loading
* make quality + style
* add pipeline test + corresponding fixes
* utility function that determines the default resolution given the VAE
* Refactor PhotonAttention to match Flux pattern
* built-in RMSNorm
* Revert accidental .gitignore change
* parameter names match the standard diffusers conventions
* renaming and remove unecessary attributes setting
* Update docs/source/en/api/pipelines/photon.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* quantization example
* added doc to toctree
* Update docs/source/en/api/pipelines/photon.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/en/api/pipelines/photon.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* Update docs/source/en/api/pipelines/photon.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
* use dispatch_attention_fn for multiple attention backend support
* naming changes
* make fix copy
* Update docs/source/en/api/pipelines/photon.md
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* Add PhotonTransformer2DModel to TYPE_CHECKING imports
* make fix-copies
* Use Tuple instead of tuple
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* restrict the version of transformers
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* Update tests/pipelines/photon/test_pipeline_photon.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* Update tests/pipelines/photon/test_pipeline_photon.py
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
* change | for Optional
* fix nits.
* use typing Dict
---------
Co-authored-by: davidb <davidb@worker-10.soperator-worker-svc.soperator.svc.cluster.local>
Co-authored-by: David Briand <david@photoroom.com>
Co-authored-by: davidb <davidb@worker-8.soperator-worker-svc.soperator.svc.cluster.local>
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
Co-authored-by: sayakpaul <spsayakpaul@gmail.com>
