* ⚙️chore(train_controlnet) fix typo in logger message
* ⚙️chore(models) refactor modules order; make them the same as calling order
When printing the BasicTransformerBlock to stdout, I think it's crucial that the attributes order are shown in proper order. And also previously the "3. Feed Forward" comment was not making sense. It should have been close to self.ff but it's instead next to self.norm3
* correct many tests
* remove bogus file
* make style
* correct more tests
* finish tests
* fix one more
* make style
* make unclip deterministic
* ⚙️chore(models/attention) reorganize comments in BasicTransformerBlock class
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* add only cross attention to simple attention blocks
* add test for only_cross_attention re: @patrickvonplaten
* mid_block_only_cross_attention better default
allow mid_block_only_cross_attention to default to
`only_cross_attention` when `only_cross_attention` is given
as a single boolean
* Fix invocation of some slow tests.
We use __call__ rather than pmapping the generation function ourselves
because the number of static arguments is different now.
* style
* [Config] Fix config prints and save, load
* Only use potential nn.Modules for dtype and device
* Correct vae image processor
* make sure in_channels is not accessed directly
* make sure in channels is only accessed via config
* Make sure schedulers only access config attributes
* Make sure to access config in SAG
* Fix vae processor and make style
* add tests
* uP
* make style
* Fix more naming issues
* Final fix with vae config
* change more
* add TextToVideoZeroPipeline and CrossFrameAttnProcessor
* add docs for text-to-video zero
* add teaser image for text-to-video zero docs
* Fix review changes. Add Documentation. Add test
* clean up the codes in pipeline_text_to_video.py. Add descriptive comments and docstrings
* make style && make quality
* make fix-copies
* make requested changes to docs. use huggingface server links for resources, delete res folder
* make style && make quality && make fix-copies
* make style && make quality
* Apply suggestions from code review
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* add load textual inversion embeddings draft
* fix quality
* fix typo
* make fix copies
* move to textual inversion mixin
* make it accept from sd-concept library
* accept list of paths to embeddings
* fix styling of stable diffusion pipeline
* add dummy TextualInversionMixin
* add docstring to textualinversionmixin
* add load textual inversion embeddings draft
* fix quality
* fix typo
* make fix copies
* move to textual inversion mixin
* make it accept from sd-concept library
* accept list of paths to embeddings
* fix styling of stable diffusion pipeline
* add dummy TextualInversionMixin
* add docstring to textualinversionmixin
* add case for parsing embedding from auto1111 UI format
Co-authored-by: Evan Jones <evan.a.jones3@gmail.com>
Co-authored-by: Ana Tamais <aninhamoraestamais@gmail.com>
* fix style after rebase
* move textual inversion mixin to loaders
* move mixin inheritance to DiffusionPipeline from StableDiffusionPipeline)
* update dummy class name
* addressed allo comments
* fix old dangling import
* fix style
* proposal
* remove bogus
* Apply suggestions from code review
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Will Berman <wlbberman@gmail.com>
* finish
* make style
* up
* fix code quality
* fix code quality - again
* fix code quality - 3
* fix alt diffusion code quality
* fix model editing pipeline
* Apply suggestions from code review
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Finish
---------
Co-authored-by: Evan Jones <evan.a.jones3@gmail.com>
Co-authored-by: Ana Tamais <aninhamoraestamais@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Will Berman <wlbberman@gmail.com>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Workaround for saving dynamo-wrapped models.
* Accept suggestion from code review
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Apply workaround when overriding pipeline components.
* Ensure the correct config.json is saved to disk.
Instead of the dynamo class.
* Save correct module (not compiled one)
* Add test
* style
* fix docstrings
* Go back to using string comparisons.
PyTorch CPU does not have _dynamo.
* Simple test for save_pretrained of compiled models.
* Helper function to test whether module is compiled.
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
The 'CLIPFeatureExtractor' class name has been renamed to 'CLIPImageProcessor' in order to comply with future deprecation. This commit includes the necessary changes to the affected files.
* [MS Text To Video} Add first text to video
* upload
* make first model example
* match unet3d params
* make sure weights are correcctly converted
* improve
* forward pass works, but diff result
* make forward work
* fix more
* finish
* refactor video output class.
* feat: add support for a video export utility.
* fix: opencv availability check.
* run make fix-copies.
* add: docs for the model components.
* add: standalone pipeline doc.
* edit docstring of the pipeline.
* add: right path to TransformerTempModel
* add: first set of tests.
* complete fast tests for text to video.
* fix bug
* up
* three fast tests failing.
* add: note on slow tests
* make work with all schedulers
* apply styling.
* add slow tests
* change file name
* update
* more correction
* more fixes
* finish
* up
* Apply suggestions from code review
* up
* finish
* make copies
* fix pipeline tests
* fix more tests
* Apply suggestions from code review
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* apply suggestions
* up
* revert
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* updated black format
* update black format
* make style format
* updated line endings
* update code formatting
* Update examples/research_projects/onnxruntime/text_to_image/train_text_to_image.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/models/vae.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/models/vae.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* added vae gradient checkpointing test
* make style
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Will Berman <wlbberman@gmail.com>
* Adding `use_safetensors` argument to give more control to users
about which weights they use.
* Doc style.
* Rebased (not functional).
* Rebased and functional with tests.
* Style.
* Apply suggestions from code review
* Style.
* Addressing comments.
* Update tests/test_pipelines.py
Co-authored-by: Will Berman <wlbberman@gmail.com>
* Black ???
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Will Berman <wlbberman@gmail.com>
* fix AttnProcessor2_0
Fix use of AttnProcessor2_0 for cross attention with mask
* added scale_qk and out_bias flags
* fixed for xformers
* check if it has scale argument
* Update cross_attention.py
* check torch version
* fix sliced attn
* style
* set scale
* fix test
* fixed addedKV processor
* revert back AttnProcessor2_0
* if missing if
* fix inner_dim
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Add support for different model prediction types in DDIMInverseScheduler
Resolve alpha_prod_t_prev index issue for final step of inversion
* Fix old bug introduced when prediction type is "sample"
* Add support for sample clipping for numerical stability and deprecate old kwarg
* Detach sample, alphas, betas
Derive predicted noise from model output before dist. regularization
Style cleanup
* Log loss for debugging
* Revert "Log loss for debugging"
This reverts commit 76ea9c856f.
* Add comments
* Add inversion equivalence test
* Add expected data for Pix2PixZero pipeline tests with SD 2
* Update tests/pipelines/stable_diffusion/test_stable_diffusion_pix2pix_zero.py
* Remove cruft and add more explanatory comments
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* ema test cases.
* debugging maessages.
* debugging maessages.
* add: tests for ema.
* fix: optimization_step arg,
* handle device placement.
* Apply suggestions from code review
Co-authored-by: Will Berman <wlbberman@gmail.com>
* remove del and gc.
* address PR feedback.
* add: tests for serialization.
* fix: typos.
* skip_mps to serialization.
---------
Co-authored-by: Will Berman <wlbberman@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* support for List[ControlNetModel] on init()
* Add to support for multiple ControlNetCondition
* rename conditioning_scale to scale
* scaling bugfix
* Manually merge `MultiControlNet` #2621
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_controlnet.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_controlnet.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_controlnet.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_controlnet.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_controlnet.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_controlnet.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* cleanups
- don't expose ControlNetCondition
- move scaling to ControlNetModel
* make style error correct
* remove ControlNetCondition to reduce code diff
* refactoring image/cond_scale
* add explain for `images`
* Add docstrings
* all fast-test passed
* Add a slow test
* nit
* Apply suggestions from code review
* small precision fix
* nits
MultiControlNet -> MultiControlNetModel - Matches existing naming a bit
closer
MultiControlNetModel inherit from model utils class - Don't have to
re-write fp16 test
Skip tests that save multi controlnet pipeline - Clearer than changing
test body
Don't auto-batch the number of input images to the number of controlnets.
We generally like to require the user to pass the expected number of
inputs. This simplifies the processing code a bit more
Use existing image pre-processing code a bit more. We can rely on the
existing image pre-processing code and keep the inference loop a bit
simpler.
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: William Berman <WLBberman@gmail.com>
