Junsong Chen
5a196e3d46
[Sana] Add Sana, including SanaPipeline, SanaPAGPipeline, LinearAttentionProcessor, Flow-based DPM-sovler and so on. ( #9982 )
...
* first add a script for DC-AE;
* DC-AE init
* replace triton with custom implementation
* 1. rename file and remove un-used codes;
* no longer rely on omegaconf and dataclass
* replace custom activation with diffuers activation
* remove dc_ae attention in attention_processor.py
* iinherit from ModelMixin
* inherit from ConfigMixin
* dc-ae reduce to one file
* update downsample and upsample
* clean code
* support DecoderOutput
* remove get_same_padding and val2tuple
* remove autocast and some assert
* update ResBlock
* remove contents within super().__init__
* Update src/diffusers/models/autoencoders/dc_ae.py
Co-authored-by: YiYi Xu <yixu310@gmail.com >
* remove opsequential
* update other blocks to support the removal of build_norm
* remove build encoder/decoder project in/out
* remove inheritance of RMSNorm2d from LayerNorm
* remove reset_parameters for RMSNorm2d
Co-authored-by: YiYi Xu <yixu310@gmail.com >
* remove device and dtype in RMSNorm2d __init__
Co-authored-by: YiYi Xu <yixu310@gmail.com >
* Update src/diffusers/models/autoencoders/dc_ae.py
Co-authored-by: YiYi Xu <yixu310@gmail.com >
* Update src/diffusers/models/autoencoders/dc_ae.py
Co-authored-by: YiYi Xu <yixu310@gmail.com >
* Update src/diffusers/models/autoencoders/dc_ae.py
Co-authored-by: YiYi Xu <yixu310@gmail.com >
* remove op_list & build_block
* remove build_stage_main
* change file name to autoencoder_dc
* move LiteMLA to attention.py
* align with other vae decode output;
* add DC-AE into init files;
* update
* make quality && make style;
* quick push before dgx disappears again
* update
* make style
* update
* update
* fix
* refactor
* refactor
* refactor
* update
* possibly change to nn.Linear
* refactor
* make fix-copies
* replace vae with ae
* replace get_block_from_block_type to get_block
* replace downsample_block_type from Conv to conv for consistency
* add scaling factors
* incorporate changes for all checkpoints
* make style
* move mla to attention processor file; split qkv conv to linears
* refactor
* add tests
* from original file loader
* add docs
* add standard autoencoder methods
* combine attention processor
* fix tests
* update
* minor fix
* minor fix
* minor fix & in/out shortcut rename
* minor fix
* make style
* fix paper link
* update docs
* update single file loading
* make style
* remove single file loading support; todo for DN6
* Apply suggestions from code review
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* add abstract
* 1. add DCAE into diffusers;
2. make style and make quality;
* add DCAE_HF into diffusers;
* bug fixed;
* add SanaPipeline, SanaTransformer2D into diffusers;
* add sanaLinearAttnProcessor2_0;
* first update for SanaTransformer;
* first update for SanaPipeline;
* first success run SanaPipeline;
* model output finally match with original model with the same intput;
* code update;
* code update;
* add a flow dpm-solver scripts
* 🎉 [important update]
1. Integrate flow-dpm-sovler into diffusers;
2. finally run successfully on both `FlowMatchEulerDiscreteScheduler` and `FlowDPMSolverMultistepScheduler`;
* 🎉 🔧 [important update & fix huge bugs!!]
1. add SanaPAGPipeline & several related Sana linear attention operators;
2. `SanaTransformer2DModel` not supports multi-resolution input;
2. fix the multi-scale HW bugs in SanaPipeline and SanaPAGPipeline;
3. fix the flow-dpm-solver set_timestep() init `model_output` and `lower_order_nums` bugs;
* remove prints;
* add convert sana official checkpoint to diffusers format Safetensor.
* Update src/diffusers/models/transformers/sana_transformer_2d.py
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update src/diffusers/models/transformers/sana_transformer_2d.py
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update src/diffusers/models/transformers/sana_transformer_2d.py
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update src/diffusers/pipelines/pag/pipeline_pag_sana.py
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update src/diffusers/models/transformers/sana_transformer_2d.py
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update src/diffusers/models/transformers/sana_transformer_2d.py
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update src/diffusers/pipelines/sana/pipeline_sana.py
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update src/diffusers/pipelines/sana/pipeline_sana.py
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* update Sana for DC-AE's recent commit;
* make style && make quality
* Add StableDiffusion3PAGImg2Img Pipeline + Fix SD3 Unconditional PAG (#9932 )
* fix progress bar updates in SD 1.5 PAG Img2Img pipeline
---------
Co-authored-by: Vinh H. Pham <phamvinh257@gmail.com >
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
* make the vae can be None in `__init__` of `SanaPipeline`
* Update src/diffusers/models/transformers/sana_transformer_2d.py
Co-authored-by: hlky <hlky@hlky.ac >
* change the ae related code due to the latest update of DCAE branch;
* change the ae related code due to the latest update of DCAE branch;
* 1. change code based on AutoencoderDC;
2. fix the bug of new GLUMBConv;
3. run success;
* update for solving conversation.
* 1. fix bugs and run convert script success;
2. Downloading ckpt from hub automatically;
* make style && make quality;
* 1. remove un-unsed parameters in init;
2. code update;
* remove test file
* refactor; add docs; add tests; update conversion script
* make style
* make fix-copies
* refactor
* udpate pipelines
* pag tests and refactor
* remove sana pag conversion script
* handle weight casting in conversion script
* update conversion script
* add a processor
* 1. add bf16 pth file path;
2. add complex human instruct in pipeline;
* fix fast \tests
* change gemma-2-2b-it ckpt to a non-gated repo;
* fix the pth path bug in conversion script;
* change grad ckpt to original; make style
* fix the complex_human_instruct bug and typo;
* remove dpmsolver flow scheduler
* apply review suggestions
* change the `FlowMatchEulerDiscreteScheduler` to default `DPMSolverMultistepScheduler` with flow matching scheduler.
* fix the tokenizer.padding_side='right' bug;
* update docs
* make fix-copies
* fix imports
* fix docs
* add integration test
* update docs
* update examples
* fix convert_model_output in schedulers
* fix failing tests
---------
Co-authored-by: Junyu Chen <chenjydl2003@gmail.com >
Co-authored-by: YiYi Xu <yixu310@gmail.com >
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
Co-authored-by: chenjy2003 <70215701+chenjy2003@users.noreply.github.com >
Co-authored-by: Aryan <aryan@huggingface.co >
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
Co-authored-by: hlky <hlky@hlky.ac >
2024-12-16 02:16:56 +05:30
Aryan
22c4f079b1
Test error raised when loading normal and expanding loras together in Flux ( #10188 )
...
* add test for expanding lora and normal lora error
* Update tests/lora/test_lora_layers_flux.py
* fix things.
* Update src/diffusers/loaders/peft.py
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-12-15 21:46:21 +05:30
Aryan
96c376a5ff
[core] LTX Video ( #10021 )
...
* transformer
* make style & make fix-copies
* transformer
* add transformer tests
* 80% vae
* make style
* make fix-copies
* fix
* undo cogvideox changes
* update
* update
* match vae
* add docs
* t2v pipeline working; scheduler needs to be checked
* docs
* add pipeline test
* update
* update
* make fix-copies
* Apply suggestions from code review
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* update
* copy t2v to i2v pipeline
* update
* apply review suggestions
* update
* make style
* remove framewise encoding/decoding
* pack/unpack latents
* image2video
* update
* make fix-copies
* update
* update
* rope scale fix
* debug layerwise code
* remove debug
* Apply suggestions from code review
Co-authored-by: YiYi Xu <yixu310@gmail.com >
* propagate precision changes to i2v pipeline
* remove downcast
* address review comments
* fix comment
* address review comments
* [Single File] LTX support for loading original weights (#10135 )
* from original file mixin for ltx
* undo config mapping fn changes
* update
* add single file to pipelines
* update docs
* Update src/diffusers/models/autoencoders/autoencoder_kl_ltx.py
* Update src/diffusers/models/autoencoders/autoencoder_kl_ltx.py
* rename classes based on ltx review
* point to original repository for inference
* make style
* resolve conflicts correctly
---------
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
Co-authored-by: YiYi Xu <yixu310@gmail.com >
2024-12-12 16:21:28 +05:30
Sayak Paul
a6a18cff5e
[LoRA] add a test to ensure set_adapters() and attn kwargs outs match ( #10110 )
...
* add a test to ensure set_adapters() and attn kwargs outs match
* remove print
* fix
* Apply suggestions from code review
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com >
* assertFalse.
---------
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com >
2024-12-12 12:52:50 +05:30
Canva
7db9463e52
Add support for XFormers in SD3 ( #8583 )
...
* Add support for XFormers in SD3
* sd3 xformers test
* sd3 xformers quality
* sd3 xformers update
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
Co-authored-by: YiYi Xu <yixu310@gmail.com >
2024-12-12 12:05:39 +05:30
Dhruv Nair
ad40e26515
[Single File] Add single file support for AutoencoderDC ( #10183 )
...
* update
* update
* update
2024-12-11 16:57:36 +05:30
Darshil Jariwala
65b98b5da4
Add PAG Support for Stable Diffusion Inpaint Pipeline ( #9386 )
...
* using sd inpaint pipeline and sdxl pag inpaint pipeline to add changes
* using sd inpaint pipeline and sdxl pag inpaint pipeline to add changes
* finished the call function
* added auto pipeline
* merging diffusers
* ready to test
* ready to test
* added copied from and removed unnecessary tests
* make style changes
* doc changes
* updating example doc string
* style fix
* init
* adding imports
* quality
* Update src/diffusers/pipelines/pag/pipeline_pag_sd_inpaint.py
* make
* Update tests/pipelines/pag/test_pag_sd_inpaint.py
* slice and size
* slice
---------
Co-authored-by: Darshil Jariwala <darshiljariwala@Darshils-MacBook-Air.local >
Co-authored-by: Darshil Jariwala <jariwala.darshil2002@gmail.com >
Co-authored-by: YiYi Xu <yixu310@gmail.com >
Co-authored-by: hlky <hlky@hlky.ac >
2024-12-10 21:06:31 +00:00
Aryan
49a9143479
Flux Control LoRA ( #9999 )
...
* update
---------
Co-authored-by: yiyixuxu <yixu310@gmail.com >
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-12-10 09:08:13 -10:00
Yu Zheng
6131a93b96
support sd3.5 for controlnet example ( #9860 )
...
* support sd3.5 in controlnet
---------
Co-authored-by: YiYi Xu <yixu310@gmail.com >
2024-12-06 10:59:27 -10:00
Sayak Paul
fa3a9100be
[LoRA] depcrecate save_attn_procs(). ( #10126 )
...
depcrecate save_attn_procs().
2024-12-06 10:38:57 -10:00
Junsong Chen
cd892041e2
[DC-AE] Add the official Deep Compression Autoencoder code(32x,64x,128x compression ratio); ( #9708 )
...
* first add a script for DC-AE;
* DC-AE init
* replace triton with custom implementation
* 1. rename file and remove un-used codes;
* no longer rely on omegaconf and dataclass
* replace custom activation with diffuers activation
* remove dc_ae attention in attention_processor.py
* iinherit from ModelMixin
* inherit from ConfigMixin
* dc-ae reduce to one file
* update downsample and upsample
* clean code
* support DecoderOutput
* remove get_same_padding and val2tuple
* remove autocast and some assert
* update ResBlock
* remove contents within super().__init__
* Update src/diffusers/models/autoencoders/dc_ae.py
Co-authored-by: YiYi Xu <yixu310@gmail.com >
* remove opsequential
* update other blocks to support the removal of build_norm
* remove build encoder/decoder project in/out
* remove inheritance of RMSNorm2d from LayerNorm
* remove reset_parameters for RMSNorm2d
Co-authored-by: YiYi Xu <yixu310@gmail.com >
* remove device and dtype in RMSNorm2d __init__
Co-authored-by: YiYi Xu <yixu310@gmail.com >
* Update src/diffusers/models/autoencoders/dc_ae.py
Co-authored-by: YiYi Xu <yixu310@gmail.com >
* Update src/diffusers/models/autoencoders/dc_ae.py
Co-authored-by: YiYi Xu <yixu310@gmail.com >
* Update src/diffusers/models/autoencoders/dc_ae.py
Co-authored-by: YiYi Xu <yixu310@gmail.com >
* remove op_list & build_block
* remove build_stage_main
* change file name to autoencoder_dc
* move LiteMLA to attention.py
* align with other vae decode output;
* add DC-AE into init files;
* update
* make quality && make style;
* quick push before dgx disappears again
* update
* make style
* update
* update
* fix
* refactor
* refactor
* refactor
* update
* possibly change to nn.Linear
* refactor
* make fix-copies
* replace vae with ae
* replace get_block_from_block_type to get_block
* replace downsample_block_type from Conv to conv for consistency
* add scaling factors
* incorporate changes for all checkpoints
* make style
* move mla to attention processor file; split qkv conv to linears
* refactor
* add tests
* from original file loader
* add docs
* add standard autoencoder methods
* combine attention processor
* fix tests
* update
* minor fix
* minor fix
* minor fix & in/out shortcut rename
* minor fix
* make style
* fix paper link
* update docs
* update single file loading
* make style
* remove single file loading support; todo for DN6
* Apply suggestions from code review
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* add abstract
---------
Co-authored-by: Junyu Chen <chenjydl2003@gmail.com >
Co-authored-by: YiYi Xu <yixu310@gmail.com >
Co-authored-by: chenjy2003 <70215701+chenjy2003@users.noreply.github.com >
Co-authored-by: Aryan <aryan@huggingface.co >
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
2024-12-07 01:01:51 +05:30
Sayak Paul
65ab1052b8
[Tests] xfail incompatible SD configs. ( #10127 )
...
* xfail incompatible SD configs.
* fix
2024-12-05 15:11:52 +05:30
Sayak Paul
40fc389c44
[Tests] fix condition argument in xfail. ( #10099 )
...
* fix condition argument in xfail.
* revert init changes.
2024-12-05 10:13:45 +05:30
Sayak Paul
e8da75dff5
[bitsandbytes] allow directly CUDA placements of pipelines loaded with bnb components ( #9840 )
...
* allow device placement when using bnb quantization.
* warning.
* tests
* fixes
* docs.
* require accelerate version.
* remove print.
* revert to()
* tests
* fixes
* fix: missing AutoencoderKL lora adapter (#9807 )
* fix: missing AutoencoderKL lora adapter
* fix
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
* fixes
* fix condition test
* updates
* updates
* remove is_offloaded.
* fixes
* better
* empty
---------
Co-authored-by: Emmanuel Benazera <emmanuel.benazera@jolibrain.com >
2024-12-04 22:27:43 +05:30
Sayak Paul
c1926cef6b
[tests] refactor vae tests ( #9808 )
...
* add: autoencoderkl tests
* autoencodertiny.
* fix
* asymmetric autoencoder.
* more
* integration tests for stable audio decoder.
* consistency decoder vae tests
* remove grad check from consistency decoder.
* cog
* bye test_models_vae.py
* fix
* fix
* remove allegro
* fixes
* fixes
* fixes
---------
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
2024-12-04 15:58:36 +05:30
hlky
cfdeebd4a8
Test skip_guidance_layers in SD3 pipeline ( #10102 )
...
* Test `skip_guidance_layers` in pipelines
* Move to test_pipeline_stable_diffusion_3
2024-12-03 14:28:31 -10:00
Benjamin Paine
63b631f383
Add StableDiffusion3PAGImg2Img Pipeline + Fix SD3 Unconditional PAG ( #9932 )
...
* fix progress bar updates in SD 1.5 PAG Img2Img pipeline
---------
Co-authored-by: Vinh H. Pham <phamvinh257@gmail.com >
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-12-02 21:39:47 -10:00
Emmanuel Benazera
963ffca434
fix: missing AutoencoderKL lora adapter ( #9807 )
...
* fix: missing AutoencoderKL lora adapter
* fix
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-12-02 17:10:20 -10:00
Sayak Paul
827b6c25f9
[CI] Add quantization ( #9832 )
...
* add quantization to nightly CI.
* prep.
* fix lib name.
* remove deps that are not needed.
* fix slice.
2024-12-02 14:53:43 +05:30
Fanli Lin
6b288ec44d
make pipelines tests device-agnostic (part2) ( #9400 )
...
* enable on xpu
* add 1 more
* add one more
* enable more
* add 1 more
* add more
* enable 1
* enable more cases
* enable
* enable
* update comment
* one more
* enable 1
* add more cases
* enable xpu
* add one more caswe
* add more cases
* add 1
* add more
* add more cases
* add case
* enable
* add more
* add more
* add more
* enbale more
* add more
* update code
* update test marker
* add skip back
* update comment
* remove single files
* remove
* style
* add
* revert
* reformat
* enable
* enable esingle g
* add 2 more
* update decorator
* update
* update
* update
* Update tests/pipelines/deepfloyd_if/test_if.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* Update src/diffusers/utils/testing_utils.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* Update tests/pipelines/animatediff/test_animatediff_controlnet.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* Update tests/pipelines/animatediff/test_animatediff.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* Update tests/pipelines/animatediff/test_animatediff_controlnet.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* update float16
* no unitest.skipt
* update
* apply style check
* adapt style
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
2024-11-29 11:33:41 +05:30
Aryan
7ac6e286ee
Flux Fill, Canny, Depth, Redux ( #9985 )
...
* update
---------
Co-authored-by: yiyixuxu <yixu310@gmail.com >
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-11-23 01:41:25 -10:00
Fanli Lin
64b3e0f539
make pipelines tests device-agnostic (part1) ( #9399 )
...
* enable on xpu
* add 1 more
* add one more
* enable more
* add 1 more
* add more
* enable 1
* enable more cases
* enable
* enable
* update comment
* one more
* enable 1
* add more cases
* enable xpu
* add one more caswe
* add more cases
* add 1
* add more
* add more cases
* add case
* enable
* add more
* add more
* add more
* enbale more
* add more
* update code
* update test marker
* add skip back
* update comment
* remove single files
* remove
* style
* add
* revert
* reformat
* update decorator
* update
* update
* update
* Update tests/pipelines/deepfloyd_if/test_if.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* Update src/diffusers/utils/testing_utils.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* Update tests/pipelines/animatediff/test_animatediff_controlnet.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* Update tests/pipelines/animatediff/test_animatediff.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* Update tests/pipelines/animatediff/test_animatediff_controlnet.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* update float16
* no unitest.skipt
* update
* apply style check
* reapply format
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
2024-11-22 15:32:54 +05:30
Sayak Paul
2e86a3f023
[Tests] skip nan lora tests on PyTorch 2.5.1 CPU. ( #9975 )
...
* skip nan lora tests on PyTorch 2.5.1 CPU.
* cog
* use xfail
* correct xfail
* add condition
* tests
2024-11-22 12:45:21 +05:30
Aryan
cd6ca9df29
Fix prepare latent image ids and vae sample generators for flux ( #9981 )
...
* fix
* update expected slice
2024-11-21 13:02:31 +05:30
raulmosa
3139d39fa7
Update handle single blocks on _convert_xlabs_flux_lora_to_diffusers ( #9915 )
...
* Update handle single blocks on _convert_xlabs_flux_lora_to_diffusers to fix bug on updating keys and old_state_dict
---------
Co-authored-by: raul_ar <raul.moreno.salinas@autoretouch.com >
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-11-20 12:53:20 -10:00
linjiapro
12358622e5
Improve control net block index for sd3 ( #9758 )
...
* improve control net index
---------
Co-authored-by: YiYi Xu <yixu310@gmail.com >
2024-11-20 12:45:18 -10:00
Sayak Paul
805aa93789
[LoRA] enable LoRA for Mochi-1 ( #9943 )
...
* feat: add lora support to Mochi-1.
2024-11-20 12:07:04 -10:00
Dhruv Nair
f6f7afa1d7
Flux latents fix ( #9929 )
...
* update
* update
* update
* update
* update
* update
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-11-20 17:30:17 +05:30
hlky
637e2302ac
Fix beta and exponential sigmas + add tests ( #9954 )
...
* Fix beta and exponential sigmas + add tests
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-11-20 01:20:34 -10:00
Bagheera
99c0483b67
add skip_layers argument to SD3 transformer model class ( #9880 )
...
* add skip_layers argument to SD3 transformer model class
* add unit test for skip_layers in stable diffusion 3
* sd3: pipeline should support skip layer guidance
* up
---------
Co-authored-by: bghira <bghira@users.github.com >
Co-authored-by: yiyixuxu <yixu310@gmail.com >
2024-11-19 15:22:54 -05:00
Sayak Paul
7d0b9c4d4e
[LoRA] feat: save_lora_adapter() ( #9862 )
...
* feat: save_lora_adapter.
2024-11-18 21:03:38 -10:00
Yuxuan.Zhang
3b2830618d
CogVideoX 1.5 ( #9877 )
...
* CogVideoX1_1PatchEmbed test
* 1360 * 768
* refactor
* make style
* update docs
* add modeling tests for cogvideox 1.5
* update
* make fix-copies
* add ofs embed(for convert)
* add ofs embed(for convert)
* more resolution for cogvideox1.5-5b-i2v
* use even number of latent frames only
* update pipeline implementations
* make style
* set patch_size_t as None by default
* #skip frames 0
* refactor
* make style
* update docs
* fix ofs_embed
* update docs
* invert_scale_latents
* update
* fix
* Update docs/source/en/api/pipelines/cogvideox.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/api/pipelines/cogvideox.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/api/pipelines/cogvideox.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update docs/source/en/api/pipelines/cogvideox.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* Update src/diffusers/models/transformers/cogvideox_transformer_3d.py
* update conversion script
* remove copied from
* fix test
* Update docs/source/en/api/pipelines/cogvideox.md
* Update docs/source/en/api/pipelines/cogvideox.md
* Update docs/source/en/api/pipelines/cogvideox.md
* Update docs/source/en/api/pipelines/cogvideox.md
---------
Co-authored-by: Aryan <aryan@huggingface.co >
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
2024-11-19 00:56:34 +05:30
Dhruv Nair
1b392544c7
Improve downloads of sharded variants ( #9869 )
...
* update
* update
* update
* update
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-11-08 17:49:00 +05:30
Sayak Paul
ded3db164b
[Core] introduce controlnet module ( #8768 )
...
* move vae flax module.
* controlnet module.
* prepare for PR.
* revert a commit
* gracefully deprecate controlnet deps.
* fix
* fix doc path
* fix-copies
* fix path
* style
* style
* conflicts
* fix
* fix-copies
* sparsectrl.
* updates
* fix
* updates
* updates
* updates
* fix
---------
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
2024-11-06 22:08:55 -04:00
SahilCarterr
08ac5cbc7f
[Fix] Test of sd3 lora ( #9843 )
...
* fix test
* fix test asser
* fix format
* Update test_lora_layers_sd3.py
2024-11-05 11:05:20 -10:00
Aryan
3f329a426a
[core] Mochi T2V ( #9769 )
...
* update
* udpate
* update transformer
* make style
* fix
* add conversion script
* update
* fix
* update
* fix
* update
* fixes
* make style
* update
* update
* update
* init
* update
* update
* add
* up
* up
* up
* update
* mochi transformer
* remove original implementation
* make style
* update inits
* update conversion script
* docs
* Update src/diffusers/pipelines/mochi/pipeline_mochi.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* Update src/diffusers/pipelines/mochi/pipeline_mochi.py
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
* fix docs
* pipeline fixes
* make style
* invert sigmas in scheduler; fix pipeline
* fix pipeline num_frames
* flip proj and gate in swiglu
* make style
* fix
* make style
* fix tests
* latent mean and std fix
* update
* cherry-pick 1069d210e1dhruv.nair@gmail.com >
* framewise decoding and conv_cache
* make style
* Apply suggestions from code review
* mochi vae encoder changes
* rebase correctly
* Update scripts/convert_mochi_to_diffusers.py
* fix tests
* fixes
* make style
* update
* make style
* update
* add framewise and tiled encoding
* make style
* make original vae implementation behaviour the default; note: framewise encoding does not work
* remove framewise encoding implementation due to presence of attn layers
* fight test 1
* fight test 2
---------
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
Co-authored-by: yiyixuxu <yixu310@gmail.com >
2024-11-05 20:33:41 +05:30
Sayak Paul
13e8fdecda
[feat] add load_lora_adapter() for compatible models ( #9712 )
...
* add first draft.
* fix
* updates.
* updates.
* updates
* updates
* updates.
* fix-copies
* lora constants.
* add tests
* Apply suggestions from code review
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com >
* docstrings.
---------
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com >
2024-11-02 09:50:39 +05:30
SahilCarterr
f55f1f7ee5
Fixes EMAModel "from_pretrained" method ( #9779 )
...
* fix from_pretrained and added test
* make style
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
2024-11-01 09:20:19 +05:30
YiYi Xu
d2e5cb3c10
Revert "[LoRA] fix: lora loading when using with a device_mapped mode… ( #9823 )
...
Revert "[LoRA] fix: lora loading when using with a device_mapped model. (#9449 )"
This reverts commit 41e4779d98
2024-10-31 08:19:32 -10:00
Sayak Paul
41e4779d98
[LoRA] fix: lora loading when using with a device_mapped model. ( #9449 )
...
* fix: lora loading when using with a device_mapped model.
* better attibutung
* empty
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com >
* Apply suggestions from code review
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com >
* minors
* better error messages.
* fix-copies
* add: tests, docs.
* add hardware note.
* quality
* Update docs/source/en/training/distributed_inference.md
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* fixes
* skip properly.
* fixes
---------
Co-authored-by: Benjamin Bossan <BenjaminBossan@users.noreply.github.com >
Co-authored-by: Marc Sun <57196510+SunMarc@users.noreply.github.com >
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
2024-10-31 21:17:41 +05:30
Sayak Paul
ff182ad669
[CI] add a big GPU marker to run memory-intensive tests separately on CI ( #9691 )
...
* add a marker for big gpu tests
* update
* trigger on PRs temporarily.
* onnx
* fix
* total memory
* fixes
* reduce memory threshold.
* bigger gpu
* empty
* g6e
* Apply suggestions from code review
* address comments.
* fix
* fix
* fix
* fix
* fix
* okay
* further reduce.
* updates
* remove
* updates
* updates
* updates
* updates
* fixes
* fixes
* updates.
* fix
* workflow fixes.
---------
Co-authored-by: Aryan <aryan@huggingface.co >
2024-10-31 18:44:34 +05:30
Sayak Paul
4adf6affbb
[Tests] clean up and refactor gradient checkpointing tests ( #9494 )
...
* check.
* fixes
* fixes
* updates
* fixes
* fixes
2024-10-31 18:24:19 +05:30
Aryan
0d1d267b12
[core] Allegro T2V ( #9736 )
...
* update
* refactor transformer part 1
* refactor part 2
* refactor part 3
* make style
* refactor part 4; modeling tests
* make style
* refactor part 5
* refactor part 6
* gradient checkpointing
* pipeline tests (broken atm)
* update
* add coauthor
Co-Authored-By: Huan Yang <hyang@fastmail.com >
* refactor part 7
* add docs
* make style
* add coauthor
Co-Authored-By: YiYi Xu <yixu310@gmail.com >
* make fix-copies
* undo unrelated change
* revert changes to embeddings, normalization, transformer
* refactor part 8
* make style
* refactor part 9
* make style
* fix
* apply suggestions from review
* Apply suggestions from code review
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
* update example
* remove attention mask for self-attention
* update
* copied from
* update
* update
---------
Co-authored-by: Huan Yang <hyang@fastmail.com >
Co-authored-by: YiYi Xu <yixu310@gmail.com >
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
2024-10-29 13:14:36 +05:30
Sayak Paul
1d1e1a2888
Some minor updates to the nightly and push workflows ( #9759 )
...
* move lora integration tests to nightly./
* remove slow marker in the workflow where not needed.
2024-10-24 23:49:09 +09:00
Dhruv Nair
76c00c7236
is_safetensors_compatible fix ( #9741 )
...
update
2024-10-22 19:35:03 +05:30
Sayak Paul
60ffa84253
[bitsandbbytes] follow-ups ( #9730 )
...
* bnb follow ups.
* add a warning when dtypes mismatch.
* fx-copies
* clear cache.
* check_if_quantized_param
* add a check on shape.
* updates
* docs
* improve readability.
* resources.
* fix
2024-10-22 16:00:05 +05:30
YiYi Xu
e2d037bbf1
minor doc/test update ( #9734 )
...
* update some docs and tests!
---------
Co-authored-by: Aryan <contact.aryanvs@gmail.com >
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
Co-authored-by: Aryan <aryan@huggingface.co >
Co-authored-by: apolinário <joaopaulo.passos@gmail.com >
2024-10-21 13:06:13 -10:00
Sayak Paul
b821f006d0
[Quantization] Add quantization support for bitsandbytes ( #9213 )
...
* quantization config.
* fix-copies
* fix
* modules_to_not_convert
* add bitsandbytes utilities.
* make progress.
* fixes
* quality
* up
* up
rotary embedding refactor 2: update comments, fix dtype for use_real=False (#9312 )
fix notes and dtype
up
up
* minor
* up
* up
* fix
* provide credits where due.
* make configurations work.
* fixes
* fix
* update_missing_keys
* fix
* fix
* make it work.
* fix
* provide credits to transformers.
* empty commit
* handle to() better.
* tests
* change to bnb from bitsandbytes
* fix tests
fix slow quality tests
SD3 remark
fix
complete int4 tests
add a readme to the test files.
add model cpu offload tests
warning test
* better safeguard.
* change merging status
* courtesy to transformers.
* move upper.
* better
* make the unused kwargs warning friendlier.
* harmonize changes with https://github.com/huggingface/transformers/pull/33122
* style
* trainin tests
* feedback part i.
* Add Flux inpainting and Flux Img2Img (#9135 )
---------
Co-authored-by: yiyixuxu <yixu310@gmail.com >
Update `UNet2DConditionModel`'s error messages (#9230 )
* refactor
[CI] Update Single file Nightly Tests (#9357 )
* update
* update
feedback.
improve README for flux dreambooth lora (#9290 )
* improve readme
* improve readme
* improve readme
* improve readme
fix one uncaught deprecation warning for accessing vae_latent_channels in VaeImagePreprocessor (#9372 )
deprecation warning vae_latent_channels
add mixed int8 tests and more tests to nf4.
[core] Freenoise memory improvements (#9262 )
* update
* implement prompt interpolation
* make style
* resnet memory optimizations
* more memory optimizations; todo: refactor
* update
* update animatediff controlnet with latest changes
* refactor chunked inference changes
* remove print statements
* update
* chunk -> split
* remove changes from incorrect conflict resolution
* remove changes from incorrect conflict resolution
* add explanation of SplitInferenceModule
* update docs
* Revert "update docs"
This reverts commit c55a50a271#9135 )"
This reverts commit 5799954dd459462357+stevhliu@users.noreply.github.com >
* contribution guide.
* changes
* empty
* fix tests
* harmonize with https://github.com/huggingface/transformers/pull/33546 .
* numpy_cosine_distance
* config_dict modification.
* remove if config comment.
* note for load_state_dict changes.
* float8 check.
* quantizer.
* raise an error for non-True low_cpu_mem_usage values when using quant.
* low_cpu_mem_usage shenanigans when using fp32 modules.
* don't re-assign _pre_quantization_type.
* make comments clear.
* remove comments.
* handle mixed types better when moving to cpu.
* add tests to check if we're throwing warning rightly.
* better check.
* fix 8bit test_quality.
* handle dtype more robustly.
* better message when keep_in_fp32_modules.
* handle dtype casting.
* fix dtype checks in pipeline.
* fix warning message.
* Update src/diffusers/models/modeling_utils.py
Co-authored-by: YiYi Xu <yixu310@gmail.com >
* mitigate the confusing cpu warning
---------
Co-authored-by: Vishnu V Jaddipal <95531133+Gothos@users.noreply.github.com >
Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com >
Co-authored-by: YiYi Xu <yixu310@gmail.com >
2024-10-21 10:11:57 +05:30
bonlime
5d3e7bdaaa
Fix bug in Textual Inversion Unloading ( #9304 )
...
* Update textual_inversion.py
* add unload test
* add comment
* fix style
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
Co-authored-by: Your Name <you@example.com >
Co-authored-by: YiYi Xu <yixu310@gmail.com >
2024-10-19 02:37:32 -10:00
Aryan
5704376d03
[refactor] DiffusionPipeline.download ( #9557 )
...
* update
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com >
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com >
2024-10-17 12:38:06 -10:00
