Aryan
cbc2ec8f44
AnimateDiff prompt travel ( #9231 )
...
* update
* implement prompt interpolation
* make style
* resnet memory optimizations
* more memory optimizations; todo: refactor
* update
* update animatediff controlnet with latest changes
* refactor chunked inference changes
* remove print statements
* undo memory optimization changes
* update docstrings
* fix tests
* fix pia tests
* apply suggestions from review
* add tests
* update comment
2024-08-28 14:48:12 +05:30
Sayak Paul
2d9ccf39b5
[Core] fuse_qkv_projection() to Flux ( #9185 )
...
* start fusing flux.
* test
* finish fusion
* fix-copues
2024-08-23 10:54:13 +05:30
zR
960c149c77
Cogvideox-5B Model adapter change ( #9203 )
...
* draft of embedding
---------
Co-authored-by: Aryan <aryan@huggingface.co >
2024-08-22 16:03:29 -10:00
Aryan
0ec64fe9fc
[tests] fix broken xformers tests ( #9206 )
...
* fix xformers tests
* remove unnecessary modifications to cogvideox tests
* update
2024-08-22 15:17:47 +05:30
Sayak Paul
5090b09d48
[Flux LoRA] support parsing alpha from a flux lora state dict. ( #9236 )
...
* support parsing alpha from a flux lora state dict.
* conditional import.
* fix breaking changes.
* safeguard alpha.
* fix
2024-08-22 07:01:52 +05:30
YiYi Xu
c291617518
Flux followup ( #9074 )
...
* refactor rotary embeds
* adding jsmidt as co-author of this PR for https://github.com/huggingface/diffusers/pull/9133
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
Co-authored-by: Joseph Smidt <josephsmidt@gmail.com >
2024-08-21 08:44:58 -10:00
satani99
9003d75f20
Add StableDiffusionXLControlNetPAGImg2ImgPipeline ( #8990 )
...
* Added pad controlnet sdxl img2img pipeline
---------
Co-authored-by: YiYi Xu <yixu310@gmail.com >
2024-08-21 07:24:22 -10:00
YiYi Xu
214372aa99
fix a regression in is_safetensors_compatible ( #9234 )
...
fix
2024-08-21 18:56:55 +05:30
Vinh H. Pham
867e0c919e
StableDiffusionLatentUpscalePipeline - positive/negative prompt embeds support ( #8947 )
...
* make latent upscaler accept prompt embeds
---------
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
Co-authored-by: YiYi Xu <yixu310@gmail.com >
2024-08-20 18:00:55 -10:00
Dhruv Nair
940b8e0358
[CI] Multiple Slow Test fixes. ( #9198 )
...
* update
* update
* update
* update
2024-08-19 13:31:09 +05:30
Dhruv Nair
b2add10d13
Update is_safetensors_compatible check ( #8991 )
...
* update
* update
* update
* update
* update
2024-08-19 11:35:22 +05:30
M Saqlain
ba4348d9a7
[Tests] Improve transformers model test suite coverage - Lumina ( #8987 )
...
* Added test suite for lumina
* Fixed failing tests
* Improved code quality
* Added function docstrings
* Improved formatting
2024-08-19 08:29:03 +05:30
Sayak Paul
f848febacd
feat: allow sharding for auraflow. ( #8853 )
2024-08-18 08:47:26 +05:30
Sayak Paul
39b87b14b5
feat: allow flux transformer to be sharded during inference ( #9159 )
...
* feat: support sharding for flux.
* tests
2024-08-16 10:00:51 +05:30
Aryan
a85b34e7fd
[refactor] CogVideoX followups + tiled decoding support ( #9150 )
...
* refactor context parallel cache; update torch compile time benchmark
* add tiling support
* make style
* remove num_frames % 8 == 0 requirement
* update default num_frames to original value
* add explanations + refactor
* update torch compile example
* update docs
* update
* clean up if-statements
* address review comments
* add test for vae tiling
* update docs
* update docs
* update docstrings
* add modeling test for cogvideox transformer
* make style
2024-08-14 03:53:21 +05:30
ηε₯ε
5ffbe14c32
[FLUX] Support ControlNet ( #9126 )
...
* cnt model
* cnt model
* cnt model
* fix Loader "Copied"
* format
* txt_ids for multiple images
* add test and format
* typo
* Update pipeline_flux_controlnet.py
* remove
* make quality
* fix copy
* Update src/diffusers/pipelines/flux/pipeline_flux_controlnet.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* Update src/diffusers/pipelines/flux/pipeline_flux_controlnet.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* Update src/diffusers/pipelines/flux/pipeline_flux_controlnet.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* Update src/diffusers/pipelines/flux/pipeline_flux_controlnet.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* Update src/diffusers/models/controlnet_flux.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* fix
* make copies
* test
* bs
---------
Co-authored-by: haofanwang <haofanwang.ai@gmail.com >
Co-authored-by: haofanwang <haofan@HaofandeMBP.lan >
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
2024-08-13 18:17:40 +05:30
ζιιΉ
cc0513091a
Support SD3 controlnet inpainting ( #9099 )
...
* add controlnet inpainting pipeline
* [SD3] add controlnet inpaint example
* update example and fix code style
* fix code style with ruff
* Update controlnet_sd3.md : add control inpaint pipeline
* Update docs/source/en/api/pipelines/controlnet_sd3.md
Co-authored-by: Aryan <contact.aryanvs@gmail.com >
* Update docs/source/en/api/pipelines/controlnet_sd3.md
Co-authored-by: Aryan <contact.aryanvs@gmail.com >
* Update docs/source/en/api/pipelines/controlnet_sd3.md
Co-authored-by: Aryan <contact.aryanvs@gmail.com >
* Update src/diffusers/pipelines/controlnet_sd3/pipeline_stable_diffusion_3_controlnet_inpainting.py
Co-authored-by: Aryan <contact.aryanvs@gmail.com >
* Update __init__.py : add sd3 control pipelines
* Update pipeline : add new param doc & check input reference.
* fix typo
* make style & make quality
* add unittest for sd3 controlnet inpaint
---------
Co-authored-by: ιΉεΎ <linjinpeng.ljp@alibaba-inc.com >
Co-authored-by: Aryan <contact.aryanvs@gmail.com >
2024-08-13 17:30:46 +05:30
zR
2dad462d9b
Add CogVideoX text-to-video generation model ( #9082 )
...
* add CogVideoX
---------
Co-authored-by: Aryan <aryan@huggingface.co >
Co-authored-by: sayakpaul <spsayakpaul@gmail.com >
Co-authored-by: Aryan <contact.aryanvs@gmail.com >
Co-authored-by: yiyixuxu <yixu310@gmail.com >
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
2024-08-06 21:23:57 -10:00
Aryan
16a93f1a25
[core] FreeNoise ( #8948 )
...
* initial work draft for freenoise; needs massive cleanup
* fix freeinit bug
* add animatediff controlnet implementation
* revert attention changes
* add freenoise
* remove old helper functions
* add decode batch size param to all pipelines
* make style
* fix copied from comments
* make fix-copies
* make style
* copy animatediff controlnet implementation from #8972
* add experimental support for num_frames not perfectly fitting context length, ocntext stride
* make unet motion model lora work again based on #8995
* copy load video utils from #8972
* copied from AnimateDiff::prepare_latents
* address the case where last batch of frames does not match length of indices in prepare latents
* decode_batch_size->vae_batch_size; batch vae encode support in animatediff vid2vid
* revert sparsectrl and sdxl freenoise changes
* revert pia
* add freenoise tests
* make fix-copies
* improve docstrings
* add freenoise tests to animatediff controlnet
* update tests
* Update src/diffusers/models/unets/unet_motion_model.py
* add freenoise to animatediff pag
* address review comments
* make style
* update tests
* make fix-copies
* fix error message
* remove copied from comment
* fix imports in tests
* update
---------
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
2024-08-07 10:35:18 +05:30
Γlvaro Somoza
39e1f7eaa4
[Kolors] Add PAG ( #8934 )
...
* txt2img pag added
* autopipe added, fixed case
* style
* apply suggestions
* added fast tests, added todo tests
* revert dummy objects for kolors
* fix pag dummies
* fix test imports
* update pag tests
* add kolor pag to docs
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-08-07 09:29:52 +05:30
Marc Sun
e4325606db
Fix loading sharded checkpoints when we have variants ( #9061 )
...
* Fix loading sharded checkpoint when we have variant
* add test
* remote print
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-08-06 13:38:44 -10:00
Ahn Donghoon (μλν / suno)
926daa30f9
add PAG support for Stable Diffusion 3 ( #8861 )
...
add pag sd3
---------
Co-authored-by: HyoungwonCho <jhw9811@korea.ac.kr >
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
Co-authored-by: crepejung00 <jaewoojung00@naver.com >
Co-authored-by: YiYi Xu <yixu310@gmail.com >
Co-authored-by: Aryan <contact.aryanvs@gmail.com >
Co-authored-by: Aryan <aryan@huggingface.co >
2024-08-06 09:11:35 -10:00
Vinh H. Pham
87e50a2f1d
[Tests] Improve transformers model test suite coverage - Hunyuan DiT ( #8916 )
...
* add hunyuan model test
* apply suggestions
* reduce dims further
* reduce dims further
* run make style
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-08-06 12:59:30 +05:30
Sayak Paul
52f1378e64
[Core] add QKV fusion to AuraFlow and PixArt Sigma ( #8952 )
...
* add fusion support to pixart
* add to auraflow.
* add tests
* apply review feedback.
* add back args and kwargs
* style
2024-08-05 14:09:37 -10:00
Tolga CangΓΆz
3dc97bd148
Update CLIPFeatureExtractor to CLIPImageProcessor and DPTFeatureExtractor to DPTImageProcessor ( #9002 )
...
* fix: update `CLIPFeatureExtractor` to `CLIPImageProcessor` in codebase
* `make style && make quality`
* Update `DPTFeatureExtractor` to `DPTImageProcessor` in codebase
* `make style`
---------
Co-authored-by: Aryan <aryan@huggingface.co >
2024-08-05 09:20:29 -10:00
YiYi Xu
bc3c73ad0b
add sentencepiece as a soft dependency ( #9065 )
...
* add sentencepiece as soft dependency for kolors
* up
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-08-05 08:04:51 -10:00
Aryan
b7058d142c
PAG variant for HunyuanDiT, PAG refactor ( #8936 )
...
* copy hunyuandit pipeline
* pag variant of hunyuan dit
* add tests
* update docs
* make style
* make fix-copies
* Update src/diffusers/pipelines/pag/pag_utils.py
* remove incorrect copied from
* remove pag hunyuan attn procs to resolve conflicts
* add pag attn procs again
* new implementation for pag_utils
* revert pag changes
* add pag refactor back; update pixart sigma
* update pixart pag tests
* apply suggestions from review
Co-Authored-By: yixu310@gmail.com
* make style
* update docs, fix tests
* fix tests
* fix test_components_function since list not accepted as valid __init__ param
* apply patch to fix broken tests
Co-Authored-By: Sayak Paul <spsayakpaul@gmail.com >
* make style
* fix hunyuan tests
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-08-05 17:56:09 +05:30
Vinh H. Pham
e1d508ae92
[Tests] Improve transformers model test suite coverage - Latte ( #8919 )
...
* add LatteTransformer3DModel model test
* change patch_size to 1
* reduce req len
* reduce channel dims
* increase num_layers
* reduce dims further
* run make style
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
Co-authored-by: Aryan <aryan@huggingface.co >
2024-08-05 17:13:03 +05:30
Sayak Paul
fc6a91e383
[FLUX] support LoRA ( #9057 )
...
* feat: lora support for Flux.
add tests
fix imports
major fixes.
* fix
fixes
final fixes?
* fix
* remove is_peft_available.
2024-08-05 10:24:05 +05:30
Sayak Paul
0e460675e2
[Flux] allow tests to run ( #9050 )
...
* fix tests
* fix
* float64 skip
* remove sample_size.
* remove
* remove more
* default_sample_size.
* credit black forest for flux model.
* skip
* fix: tests
* remove OriginalModelMixin
* add transformer model test
* add: transformer model tests
2024-08-02 11:49:59 +05:30
Sayak Paul
7b98c4cc67
[Core] Add PAG support for PixArtSigma ( #8921 )
...
* feat: add pixart sigma pag.
* inits.
* fixes
* fix
* remove print.
* copy paste methods to the pixart pag mixin
* fix-copies
* add documentation.
* add tests.
* remove correction file.
* remove pag_applied_layers
* empty
2024-08-02 07:12:41 +05:30
Sayak Paul
27637a5402
Flux pipeline ( #9043 )
...
add flux!
Signed-off-by: Adrien <adrien@huggingface.co >
Co-authored-by: Adrien <adrien.69740@gmail.com >
Co-authored-by: Anatoly Belikov <abelikov@singularitynet.io >
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
Co-authored-by: yiyixuxu <yixu310@gmail.com >
2024-08-01 11:30:52 -10:00
YiYi Xu
95a7832879
fix load sharded checkpoint from a subfolder (local path) ( #8913 )
...
fix
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-08-01 20:15:42 +05:30
Aryan
05b706c003
PAG variant for AnimateDiff ( #8789 )
...
* add animatediff pag pipeline
* remove unnecessary print
* make fix-copies
* fix ip-adapter bug
* update docs
* add fast tests and fix bugs
* update
* update
* address review comments
* update ip adapter single test expected slice
* implement test_from_pipe_consistent_config; fix expected slice values
* LoraLoaderMixin->StableDiffusionLoraLoaderMixin; add latest freeinit test
2024-08-01 12:39:39 +05:30
Yoach Lacombe
ea1b4ea7ca
Fix Stable Audio repository id ( #9016 )
...
Fix Stable Audio repo id
2024-07-30 23:17:44 +05:30
Aryan
e5b94b4c57
[core] Move community AnimateDiff ControlNet to core ( #8972 )
...
* add animatediff controlnet to core
* make style; remove unused method
* fix copied from comment
* add tests
* changes to make tests work
* add utility function to load videos
* update docs
* update pipeline example
* make style
* update docs with example
* address review comments
* add latest freeinit test from #8969
* LoraLoaderMixin -> StableDiffusionLoraLoaderMixin
* fix docs
* Update src/diffusers/utils/loading_utils.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* fix: variable out of scope
---------
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
2024-07-30 17:10:37 +05:30
Yoach Lacombe
69e72b1dd1
Stable Audio integration ( #8716 )
...
* WIP modeling code and pipeline
* add custom attention processor + custom activation + add to init
* correct ProjectionModel forward
* add stable audio to __initèè
* add autoencoder and update pipeline and modeling code
* add half Rope
* add partial rotary v2
* add temporary modfis to scheduler
* add EDM DPM Solver
* remove TODOs
* clean GLU
* remove att.group_norm to attn processor
* revert back src/diffusers/schedulers/scheduling_dpmsolver_multistep.py
* refactor GLU -> SwiGLU
* remove redundant args
* add channel multiples in autoencoder docstrings
* changes in docsrtings and copyright headers
* clean pipeline
* further cleaning
* remove peft and lora and fromoriginalmodel
* Delete src/diffusers/pipelines/stable_audio/diffusers.code-workspace
* make style
* dummy models
* fix copied from
* add fast oobleck tests
* add brownian tree
* oobleck autoencoder slow tests
* remove TODO
* fast stable audio pipeline tests
* add slow tests
* make style
* add first version of docs
* wrap is_torchsde_available to the scheduler
* fix slow test
* test with input waveform
* add input waveform
* remove some todos
* create stableaudio gaussian projection + make style
* add pipeline to toctree
* fix copied from
* make quality
* refactor timestep_features->time_proj
* refactor joint_attention_kwargs->cross_attention_kwargs
* remove forward_chunk
* move StableAudioDitModel to transformers folder
* correct convert + remove partial rotary embed
* apply suggestions from yiyixuxu -> removing attn.kv_heads
* remove temb
* remove cross_attention_kwargs
* further removal of cross_attention_kwargs
* remove text encoder autocast to fp16
* continue removing autocast
* make style
* refactor how text and audio are embedded
* add paper
* update example code
* make style
* unify projection model forward + fix device placement
* make style
* remove fuse qkv
* apply suggestions from review
* Update src/diffusers/pipelines/stable_audio/pipeline_stable_audio.py
Co-authored-by: YiYi Xu <yixu310@gmail.com >
* make style
* smaller models in fast tests
* pass sequential offloading fast tests
* add docs for vae and autoencoder
* make style and update example
* remove useless import
* add cosine scheduler
* dummy classes
* cosine scheduler docs
* better description of scheduler
---------
Co-authored-by: YiYi Xu <yixu310@gmail.com >
2024-07-30 15:29:06 +05:30
Γlvaro Somoza
73acebb8cf
[Kolors] Add IP Adapter ( #8901 )
...
* initial draft
* apply suggestions
* fix failing test
* added ipa to img2img
* add docs
* apply suggestions
2024-07-26 14:25:44 -04:00
Aryan
5c53ca5ed8
[core] AnimateDiff SparseCtrl ( #8897 )
...
* initial sparse control model draft
* remove unnecessary implementation
* copy animatediff pipeline
* remove deprecated callbacks
* update
* update pipeline implementation progress
* make style
* make fix-copies
* update progress
* add partially working pipeline
* remove debug prints
* add model docs
* dummy objects
* improve motion lora conversion script
* fix bugs
* update docstrings
* remove unnecessary model params; docs
* address review comment
* add copied from to zero_module
* copy animatediff test
* add fast tests
* update docs
* update
* update pipeline docs
* fix expected slice values
* fix license
* remove get_down_block usage
* remove temporal_double_self_attention from get_down_block
* update
* update docs with org and documentation images
* make from_unet work in sparsecontrolnetmodel
* add latest freeinit test from #8969
* make fix-copies
* LoraLoaderMixin -> StableDiffsuionLoraLoaderMixin
2024-07-26 17:46:05 +05:30
Aryan
57a021d5e4
[fix] FreeInit step index out of bounds ( #8969 )
...
* fix step index out of bounds
* add test for free_init with different schedulers
* add test to vid2vid and pia
2024-07-26 16:45:55 +05:30
Sayak Paul
d87fe95f90
[Chore] add LoraLoaderMixin to the inits ( #8981 )
...
* introduce to promote reusability.
* up
* add more tests
* up
* remove comments.
* fix fuse_nan test
* clarify the scope of fuse_lora and unfuse_lora
* remove space
* rewrite fuse_lora a bit.
* feedback
* copy over load_lora_into_text_encoder.
* address dhruv's feedback.
* fix-copies
* fix issubclass.
* num_fused_loras
* fix
* fix
* remove mapping
* up
* fix
* style
* fix-copies
* change to SD3TransformerLoRALoadersMixin
* Apply suggestions from code review
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* up
* handle wuerstchen
* up
* move lora to lora_pipeline.py
* up
* fix-copies
* fix documentation.
* comment set_adapters().
* fix-copies
* fix set_adapters() at the model level.
* fix?
* fix
* loraloadermixin.
---------
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
2024-07-26 08:59:33 +05:30
YiYi Xu
62863bb1ea
Revert "[LoRA] introduce LoraBaseMixin to promote reusability." ( #8976 )
...
Revert "[LoRA] introduce LoraBaseMixin to promote reusability. (#8774 )"
This reverts commit 527430d0a4
2024-07-25 09:10:35 -10:00
Sayak Paul
527430d0a4
[LoRA] introduce LoraBaseMixin to promote reusability. ( #8774 )
...
* introduce to promote reusability.
* up
* add more tests
* up
* remove comments.
* fix fuse_nan test
* clarify the scope of fuse_lora and unfuse_lora
* remove space
* rewrite fuse_lora a bit.
* feedback
* copy over load_lora_into_text_encoder.
* address dhruv's feedback.
* fix-copies
* fix issubclass.
* num_fused_loras
* fix
* fix
* remove mapping
* up
* fix
* style
* fix-copies
* change to SD3TransformerLoRALoadersMixin
* Apply suggestions from code review
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* up
* handle wuerstchen
* up
* move lora to lora_pipeline.py
* up
* fix-copies
* fix documentation.
* comment set_adapters().
* fix-copies
* fix set_adapters() at the model level.
* fix?
* fix
---------
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
2024-07-25 21:40:58 +05:30
Aryan
3ae0ee88d3
[tests] speed up animatediff tests ( #8846 )
...
* speed up animatediff tests
* fix pia test_ip_adapter_single
* fix tests/pipelines/pia/test_pia.py::PIAPipelineFastTests::test_dict_tuple_outputs_equivalent
* update
* fix ip adapter tests
* skip test_from_pipe_consistent_config tests
* fix prompt_embeds test
* update test_from_pipe_consistent_config tests
* fix expected_slice values
* remove temporal_norm_num_groups from UpBlockMotion
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
2024-07-25 17:35:43 +05:30
Sayak Paul
d8bcb33f4b
[Tests] fix slices of 26 tests (first half) ( #8959 )
...
* check for assertions.
* update with correct slices.
* okay
* style
* get it ready
* update
* update
* update
---------
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
2024-07-25 14:56:49 +05:30
Dhruv Nair
93983b6780
[CI] Skip flaky download tests in PR CI ( #8945 )
...
update
2024-07-24 09:25:06 +05:30
Sayak Paul
41b705f42d
remove residual i from auraflow. ( #8949 )
...
* remove residual i.
* rename to aura_flow in pipeline test
2024-07-24 07:31:54 +05:30
Sayak Paul
50d21f7c6a
[Core] fix QKV fusion for attention ( #8829 )
...
* start debugging the problem,
* start
* fix
* fix
* fix imports.
* handle hunyuan
* remove residuals.
* add a check for making sure there's appropriate procs.
* add more rigor to the tests.
* fix test
* remove redundant check
* fix-copies
* move check_qkv_fusion_matches_attn_procs_length and check_qkv_fusion_processors_exist.
2024-07-24 06:52:19 +05:30
Vinh H. Pham
7a95f8d9d8
[Tests] Improve transformers model test suite coverage - Temporal Transformer ( #8932 )
...
* add test for temporal transformer
* remove unused variable
* fix code quality
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-07-23 15:36:30 +05:30
Aritra Roy Gosthipaty
8b21feed42
[Tests] reduce the model size in the audioldm2 fast test ( #7846 )
...
* chore: initial model size reduction
* chore: fixing expected values for failing tests
* requested edits
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-07-23 14:34:07 +05:30
