* Add Recent Timestep Scheduling Improvements to DDIM Inverse Scheduler
Roll timesteps by one to reflect origin-destination semantic discrepancy
Restore `set_alpha_to_one` option to handle negative initial timesteps
Remove `set_alpha_to_zero` option not used due to previous truncation
* Bugfix
* Remove unnecessary calls to `detach()`
Use `self.image_processor.preprocess` in DiffEdit pipeline functions
* Preprocess list input for inverted image latents in diffedit pipeline
* Add `timestep_spacing` and `steps_offset` to `DPMSolverMultistepInverseScheduler`
* Update expected test results to account for inverting last forward diffusion step
* Fix inversion progress bar bug
* Add first draft for proper fast tests for DDIMInverseScheduler
* Add deprecated DDIMInverseScheduler kwarg to ConfigMixer registry
* Fix test failure in DPMMultistepInverseScheduler
Invert step specification leads to negative noise variance in SDE-based algs
Add first draft for proper fast tests for DPMMultistepInverseScheduler
* Update expected test results to account for inverting last forward diffusion step
Clean up diffedit fast test
* Quick implementation of t2i-adapter
Load adapter module with from_pretrained
Prototyping generalized adapter framework
Writeup doc string for sideload framework(WIP) + some minor update on implementation
Update adapter models
Remove old adapter optional args in UNet
Add StableDiffusionAdapterPipeline unit test
Handle cpu offload in StableDiffusionAdapterPipeline
Auto correct coding style
Update model repo name to "RzZ/sd-v1-4-adapter-pipeline"
Refactor MultiAdapter to better compatible with config system
Export MultiAdapter
Create pipeline document template from controlnet
Create dummy objects
Supproting new AdapterLight model
Fix StableDiffusionAdapterPipeline common pipeline test
[WIP] Update adapter pipeline document
Handle num_inference_steps in StableDiffusionAdapterPipeline
Update definition of Adapter "channels_in"
Update documents
Apply code style
Fix doc typo and merge error
Update doc string and example
Quality of life improvement
Remove redundant code and file from prototyping
Remove unused pageage
Remove comments
Fix title
Fix typo
Add conditioning scale arg
Bring back old implmentation
Offload sideload
Add supply info on document
Update src/diffusers/models/adapter.py
Co-authored-by: Will Berman <wlbberman@gmail.com>
Update MultiAdapter constructor
Swap out custom checkpoint and update pipeline constructor
Update docment
Apply suggestions from code review
Co-authored-by: Will Berman <wlbberman@gmail.com>
Correcting style
Following single-file policy
Update auto size in image preprocess func
Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_adapter.py
Co-authored-by: Will Berman <wlbberman@gmail.com>
fix copies
Update adapter pipeline behavior
Add adapter_conditioning_scale doc string
Add the missing doc string
Apply suggestions from code review
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Fix few bugs from suggestion
Handle L-mode PIL image as control image
Rename to differentiate adapter resblock
Update src/diffusers/models/adapter.py
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Fix typo
Update adapter parameter name
Update test case and code style
Fix copies
Fix typo
Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_adapter.py
Co-authored-by: Will Berman <wlbberman@gmail.com>
Update Adapter class name
Add checkpoint converting script
Fix style
Fix-copies
Remove dev script
Apply suggestions from code review
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Updates for parameter rename
Fix convert_adapter
remove main
fix diff
more
refactoring
more
more
small fixes
refactor
tests
more slow tests
more tests
Update docs/source/en/api/pipelines/overview.mdx
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
add community contributor to docs
Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Update docs/source/en/api/pipelines/stable_diffusion/adapter.mdx
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
fix
remove from_adapters
license
paper link
docs
more url fixes
more docs
fix
fixes
fix
fix
* fix sample inplace add
* additional_kwargs -> additional_residuals
* move t2i adapter pipeline to own module
* preprocess -> _preprocess_adapter_image
* add TencentArc to license
* fix example code links
* add image converter and fix example doc string
* fix links
* clearer additional residual application
---------
Co-authored-by: HimariO <dsfhe49854@gmail.com>
* add noise_sampler to StableDiffusionKDiffusionPipeline
* fix/docs: Fix the broken doc links (#3897)
* fix/docs: Fix the broken doc links
Signed-off-by: GitHub <noreply@github.com>
* Update docs/source/en/using-diffusers/write_own_pipeline.mdx
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
---------
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Add video img2img (#3900)
* Add image to image video
* Improve
* better naming
* make fix copies
* add docs
* finish tests
* trigger tests
* make style
* correct
* finish
* Fix more
* make style
* finish
* fix/doc-code: Updating to the latest version parameters (#3924)
fix/doc-code: update to use the new parameter
Signed-off-by: GitHub <noreply@github.com>
* fix/doc: no import torch issue (#3923)
Ffix/doc: no import torch issue
Signed-off-by: GitHub <noreply@github.com>
* Correct controlnet out of list error (#3928)
* Correct controlnet out of list error
* Apply suggestions from code review
* correct tests
* correct tests
* fix
* test all
* Apply suggestions from code review
* test all
* test all
* Apply suggestions from code review
* Apply suggestions from code review
* fix more tests
* Fix more
* Apply suggestions from code review
* finish
* Apply suggestions from code review
* Update src/diffusers/schedulers/scheduling_k_dpm_2_ancestral_discrete.py
* finish
* Adding better way to define multiple concepts and also validation capabilities. (#3807)
* - Added validation parameters
- Changed some parameter descriptions to better explain their use.
- Fixed a few typos.
- Added concept_list parameter for better management of multiple subjects
- changed logic for image validation
* - Fixed bad logic for class data root directories
* Defaulting validation_steps to None for an easier logic
* Fixed multiple validation prompts
* Fixed bug on validation negative prompt
* Changed validation logic for tracker.
* Added uuid for validation image labeling
* Fix error when comparing validation prompts and validation negative prompts
* Improved error message when negative prompts for validation are more than the number of prompts
* - Changed image tracking number from epoch to global_step
- Added Typing for functions
* Added some validations more when using concept_list parameter and the regular ones.
* Fixed error message
* Added more validations for validation parameters
* Improved messaging for errors
* Fixed validation error for parameters with default values
* - Added train step to image name for validation
- reformatted code
* - Added train step to image's name for validation
- reformatted code
* Updated README.md file.
* reverted back original script of train_dreambooth.py
* reverted back original script of train_dreambooth.py
* left one blank line at the eof
* reverted back setup.py
* reverted back setup.py
* added same logic for when parameters for prior preservation are used without enabling the flag while using concept_list parameter.
* Ran black formatter.
* fixed a few strings
* fixed import sort with isort and removed fstrings without placeholder
* fixed import order with ruff (since with isort wasn't ok)
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* [ldm3d] Update code to be functional with the new checkpoints (#3875)
* fixed typo
* updated doc to be consistent in naming
* make style/quality
* preprocessing for 4 channels and not 6
* make style
* test for 4c
* make style/quality
* fixed test on cpu
---------
Co-authored-by: Aflalo <estellea@isl-iam1.rr.intel.com>
Co-authored-by: Aflalo <estellea@isl-gpu33.rr.intel.com>
Co-authored-by: Aflalo <estellea@isl-gpu38.rr.intel.com>
* Improve memory text to video (#3930)
* Improve memory text to video
* Apply suggestions from code review
* add test
* Apply suggestions from code review
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* finish test setup
---------
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* revert automatic chunking (#3934)
* revert automatic chunking
* Apply suggestions from code review
* revert automatic chunking
* avoid upcasting by assigning dtype to noise tensor (#3713)
* avoid upcasting by assigning dtype to noise tensor
* make style
* Update train_unconditional.py
* Update train_unconditional.py
* make style
* add unit test for pickle
* revert change
---------
Co-authored-by: root <root@orttrainingdev8.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Prathik Rao <prathikrao@microsoft.com@orttrainingdev8.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net>
* Fix failing np tests (#3942)
* Fix failing np tests
* Apply suggestions from code review
* Update tests/pipelines/test_pipelines_common.py
* Add `timestep_spacing` and `steps_offset` to schedulers (#3947)
* Add timestep_spacing to DDPM, LMSDiscrete, PNDM.
* Remove spurious line.
* More easy schedulers.
* Add `linspace` to DDIM
* Noise sigma for `trailing`.
* Add timestep_spacing to DEISMultistepScheduler.
Not sure the range is the way it was intended.
* Fix: remove line used to debug.
* Support timestep_spacing in DPMSolverMultistep, DPMSolverSDE, UniPC
* Fix: convert to numpy.
* Use sched. defaults when instantiating from_config
For params not present in the original configuration.
This makes it possible to switch pipeline schedulers even if they use
different timestep_spacing (or any other param).
* Apply suggestions from code review
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Missing args in DPMSolverMultistep
* Test: default args not in config
* Style
* Fix scheduler name in test
* Remove duplicated entries
* Add test for solver_type
This test currently fails in main. When switching from DEIS to UniPC,
solver_type is "logrho" (the default value from DEIS), which gets
translated to "bh1" by UniPC. This is different to the default value for
UniPC: "bh2". This is where the translation happens: 36d22d0709/src/diffusers/schedulers/scheduling_unipc_multistep.py (L171)
* UniPC: use same default for solver_type
Fixes a bug when switching from UniPC from another scheduler (i.e.,
DEIS) that uses a different solver type. The solver is now the same as
if we had instantiated the scheduler directly.
* do not save use default values
* fix more
* fix all
* fix schedulers
* fix more
* finish for real
* finish for real
* flaky tests
* Update tests/pipelines/stable_diffusion/test_stable_diffusion_pix2pix_zero.py
* Default steps_offset to 0.
* Add missing docstrings
* Apply suggestions from code review
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Add Consistency Models Pipeline (#3492)
* initial commit
* Improve consistency models sampling implementation.
* Add CMStochasticIterativeScheduler, which implements the multi-step sampler (stochastic_iterative_sampler) in the original code, and make further improvements to sampling.
* Add Unet blocks for consistency models
* Add conversion script for Unet
* Fix bug in new unet blocks
* Fix attention weight loading
* Make design improvements to ConsistencyModelPipeline and CMStochasticIterativeScheduler and add initial version of tests.
* make style
* Make small random test UNet class conditional and set resnet_time_scale_shift to 'scale_shift' to better match consistency model checkpoints.
* Add support for converting a test UNet and non-class-conditional UNets to the consistency models conversion script.
* make style
* Change num_class_embeds to 1000 to better match the original consistency models implementation.
* Add support for distillation in pipeline_consistency_models.py.
* Improve consistency model tests:
- Get small testing checkpoints from hub
- Modify tests to take into account "distillation" parameter of ConsistencyModelPipeline
- Add onestep, multistep tests for distillation and distillation + class conditional
- Add expected image slices for onestep tests
* make style
* Improve ConsistencyModelPipeline:
- Add initial support for class-conditional generation
- Fix initial sigma for onestep generation
- Fix some sigma shape issues
* make style
* Improve ConsistencyModelPipeline:
- add latents __call__ argument and prepare_latents method
- add check_inputs method
- add initial docstrings for ConsistencyModelPipeline.__call__
* make style
* Fix bug when randomly generating class labels for class-conditional generation.
* Switch CMStochasticIterativeScheduler to configuring a sigma schedule and make related changes to the pipeline and tests.
* Remove some unused code and make style.
* Fix small bug in CMStochasticIterativeScheduler.
* Add expected slices for multistep sampling tests and make them pass.
* Work on consistency model fast tests:
- in pipeline, call self.scheduler.scale_model_input before denoising
- get expected slices for Euler and Heun scheduler tests
- make Euler test pass
- mark Heun test as expected fail because it doesn't support prediction_type "sample" yet
- remove DPM and Euler Ancestral tests because they don't support use_karras_sigmas
* make style
* Refactor conversion script to make it easier to add more model architectures to convert in the future.
* Work on ConsistencyModelPipeline tests:
- Fix device bug when handling class labels in ConsistencyModelPipeline.__call__
- Add slow tests for onestep and multistep sampling and make them pass
- Refactor fast tests
- Refactor ConsistencyModelPipeline.__init__
* make style
* Remove the add_noise and add_noise_to_input methods from CMStochasticIterativeScheduler for now.
* Run python utils/check_copies.py --fix_and_overwrite
python utils/check_dummies.py --fix_and_overwrite to make dummy objects for new pipeline and scheduler.
* Make fast tests from PipelineTesterMixin pass.
* make style
* Refactor consistency models pipeline and scheduler:
- Remove support for Karras schedulers (only support CMStochasticIterativeScheduler)
- Move sigma manipulation, input scaling, denoising from pipeline to scheduler
- Make corresponding changes to tests and ensure they pass
* make style
* Add docstrings and further refactor pipeline and scheduler.
* make style
* Add initial version of the consistency models documentation.
* Refactor custom timesteps logic following DDPMScheduler/IFPipeline and temporarily add torch 2.0 SDPA kernel selection logic for debugging.
* make style
* Convert current slow tests to use fp16 and flash attention.
* make style
* Add slow tests for normal attention on cuda device.
* make style
* Fix attention weights loading
* Update consistency model fast tests for new test checkpoints with attention fix.
* make style
* apply suggestions
* Add add_noise method to CMStochasticIterativeScheduler (copied from EulerDiscreteScheduler).
* Conversion script now outputs pipeline instead of UNet and add support for LSUN-256 models and different schedulers.
* When both timesteps and num_inference_steps are supplied, raise warning instead of error (timesteps take precedence).
* make style
* Add remaining diffusers model checkpoints for models in the original consistency model release and update usage example.
* apply suggestions from review
* make style
* fix attention naming
* Add tests for CMStochasticIterativeScheduler.
* make style
* Make CMStochasticIterativeScheduler tests pass.
* make style
* Override test_step_shape in CMStochasticIterativeSchedulerTest instead of modifying it in SchedulerCommonTest.
* make style
* rename some models
* Improve API
* rename some models
* Remove duplicated block
* Add docstring and make torch compile work
* More fixes
* Fixes
* Apply suggestions from code review
* Apply suggestions from code review
* add more docstring
* update consistency conversion script
---------
Co-authored-by: ayushmangal <ayushmangal@microsoft.com>
Co-authored-by: Ayush Mangal <43698245+ayushtues@users.noreply.github.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* add test case for StableDiffusionKDiffusionPipeline noise_sampler
---------
Signed-off-by: GitHub <noreply@github.com>
Co-authored-by: Aisuko <urakiny@gmail.com>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Andrés Mauricio Repetto Ferrero <amd.repetto@gmail.com>
Co-authored-by: estelleafl <estelle.aflalo@intel.com>
Co-authored-by: Aflalo <estellea@isl-iam1.rr.intel.com>
Co-authored-by: Aflalo <estellea@isl-gpu33.rr.intel.com>
Co-authored-by: Aflalo <estellea@isl-gpu38.rr.intel.com>
Co-authored-by: Prathik Rao <prathikr@usc.edu>
Co-authored-by: root <root@orttrainingdev8.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net>
Co-authored-by: Prathik Rao <prathikrao@microsoft.com@orttrainingdev8.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net>
Co-authored-by: dg845 <58458699+dg845@users.noreply.github.com>
Co-authored-by: ayushmangal <ayushmangal@microsoft.com>
Co-authored-by: Ayush Mangal <43698245+ayushtues@users.noreply.github.com>
* Add circular padding option
* Fix style with black
* Fix corner case with small image size
* Add circular padding test cases
* Fix docstring
* Improve docstring for circular padding, remove slow test case
* Update docs for circular padding argument
* Add images comparison for circular padding
* Add timestep_spacing to DDPM, LMSDiscrete, PNDM.
* Remove spurious line.
* More easy schedulers.
* Add `linspace` to DDIM
* Noise sigma for `trailing`.
* Add timestep_spacing to DEISMultistepScheduler.
Not sure the range is the way it was intended.
* Fix: remove line used to debug.
* Support timestep_spacing in DPMSolverMultistep, DPMSolverSDE, UniPC
* Fix: convert to numpy.
* Use sched. defaults when instantiating from_config
For params not present in the original configuration.
This makes it possible to switch pipeline schedulers even if they use
different timestep_spacing (or any other param).
* Apply suggestions from code review
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Missing args in DPMSolverMultistep
* Test: default args not in config
* Style
* Fix scheduler name in test
* Remove duplicated entries
* Add test for solver_type
This test currently fails in main. When switching from DEIS to UniPC,
solver_type is "logrho" (the default value from DEIS), which gets
translated to "bh1" by UniPC. This is different to the default value for
UniPC: "bh2". This is where the translation happens: 36d22d0709/src/diffusers/schedulers/scheduling_unipc_multistep.py (L171)
* UniPC: use same default for solver_type
Fixes a bug when switching from UniPC from another scheduler (i.e.,
DEIS) that uses a different solver type. The solver is now the same as
if we had instantiated the scheduler directly.
* do not save use default values
* fix more
* fix all
* fix schedulers
* fix more
* finish for real
* finish for real
* flaky tests
* Update tests/pipelines/stable_diffusion/test_stable_diffusion_pix2pix_zero.py
* Default steps_offset to 0.
* Add missing docstrings
* Apply suggestions from code review
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Correct controlnet out of list error
* Apply suggestions from code review
* correct tests
* correct tests
* fix
* test all
* Apply suggestions from code review
* test all
* test all
* Apply suggestions from code review
* Apply suggestions from code review
* fix more tests
* Fix more
* Apply suggestions from code review
* finish
* Apply suggestions from code review
* Update src/diffusers/schedulers/scheduling_k_dpm_2_ancestral_discrete.py
* finish
* add paradigms parallel sampling pipeline
* linting
* ran make fix-copies
* add paradigms parallel sampling pipeline
* linting
* ran make fix-copies
* Apply suggestions from code review
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* changes based on review
* add docs for paradigms
* update docs with paradigms abstract
* improve documentation, and add tests for ddim/ddpm batch_step_no_noise
* fix docs and run make fix-copies
* minor changes to docs.
* Apply suggestions from code review
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* move parallel scheduler to new classes for DDPMParallelScheduler and DDIMParallelScheduler
* remove changes for scheduling_ddim, adjust licenses, credits, and commented code
* fix tensor type that is breaking tests
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* added ldm3d pipeline and updated image processor to support depth
* added description
* added paper reference
* added docs
* fixed bug
* added test
* Update tests/pipelines/stable_diffusion/test_stable_diffusion_ldm3d.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update tests/pipelines/stable_diffusion/test_stable_diffusion_ldm3d.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* added reference in indexmdx
* reverted changes tto image processor'
* added LDM3DOutput
* Fixes with make style
* fix failing tests for make fix-copies
* aligned with our version
* Update pipeline_stable_diffusion_ldm3d.py
updated the guidance scale
* Fix for failing check_code_quality test
* Code review feedback
* Fix typo in ldm3d_diffusion.mdx
* updated the doc accordnlgy
* copyrights
* fixed test failure
* make style
* added image processor of LDM3D in the documentation:
* added ldm3d doc to toctree
* run make style && make quality
* run make fix-copies
* Update docs/source/en/api/image_processor.mdx
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* Update docs/source/en/api/pipelines/stable_diffusion/ldm3d_diffusion.mdx
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* Update docs/source/en/api/pipelines/stable_diffusion/ldm3d_diffusion.mdx
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* updated the safety checker to accept tuple
* make style and make quality
* Update src/diffusers/pipelines/stable_diffusion/__init__.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_ldm3d.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* LDM3D output
* up
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Aflalo <estellea@isl-gpu27.rr.intel.com>
Co-authored-by: Anahita Bhiwandiwalla <anahita.bhiwandiwalla@intel.com>
Co-authored-by: Aflalo <estellea@isl-gpu26.rr.intel.com>
Co-authored-by: Aflalo <estellea@isl-iam1.rr.intel.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Aflalo <estellea@isl-gpu42.rr.intel.com>
Co-authored-by: Aflalo <estellea@isl-gpu43.rr.intel.com>
* support views batch for panorama
* add entry for the new argument
* format entry for the new argument
* add view_batch_size test
* fix batch test and a boundary condition
* add more docstrings
* fix a typos
* fix typos
* add: entry to the doc about view_batch_size.
* Revert "add: entry to the doc about view_batch_size."
This reverts commit a36aeaa9ed.
* add a tip on .
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
VaeImageProcessor.preprocess refactor
* refactored VaeImageProcessor
- allow passing optional height and width argument to resize()
- add convert_to_rgb
* refactored prepare_latents method for img2img pipelines so that if we pass latents directly as image input, it will not encode it again
* added a test in test_pipelines_common.py to test latents as image inputs
* refactored img2img pipelines that accept latents as image:
- controlnet img2img, stable diffusion img2img , instruct_pix2pix
---------
Co-authored-by: yiyixuxu <yixu310@gmail,com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* Throw error if strength adjusted num_inference_steps < 1
* Added new fast test to check ValueError raised when num_inference_steps < 1
when strength adjusts the num_inference_steps then the inpainting pipeline should fail
* fix#3487 initial latents are now only scaled by init_noise_sigma when pure noise
updated this commit w.r.t the latest merge here: https://github.com/huggingface/diffusers/pull/3533
* fix
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Add default to inpaint
* Make sure controlnet also works with normal sd for inpaint
* Add tests
* improve
* Correct encode images function
* Correct inpaint controlnet
* Improve text2img inpanit
* make style
* up
* up
* up
* up
* fix more
* Run ControlNet compile test in a separate subprocess
`torch.compile()` spawns several subprocesses and the GPU memory used
was not reclaimed after the test ran. This approach was taken from
`transformers`.
* Style
* Prepare a couple more compile tests to run in subprocess.
* Use require_torch_2 decorator.
* Test inpaint_compile in subprocess.
* Run img2img compile test in subprocess.
* Run stable diffusion compile test in subprocess.
* style
* Temporarily trigger on pr to test.
* Revert "Temporarily trigger on pr to test."
This reverts commit 82d76868dd.
* up
* fix more
* Apply suggestions from code review
* fix more
* fix more
* Check it
* Remove 16:8
* fix more
* fix more
* fix more
* up
* up
* Test only stable diffusion
* Test only two files
* up
* Try out spinning up processes that can be killed
* up
* Apply suggestions from code review
* up
* up
* Added explanation of 'strength' parameter
* Added get_timesteps function which relies on new strength parameter
* Added `strength` parameter which defaults to 1.
* Swapped ordering so `noise_timestep` can be calculated before masking the image
this is required when you aren't applying 100% noise to the masked region, e.g. strength < 1.
* Added strength to check_inputs, throws error if out of range
* Changed `prepare_latents` to initialise latents w.r.t strength
inspired from the stable diffusion img2img pipeline, init latents are initialised by converting the init image into a VAE latent and adding noise (based upon the strength parameter passed in), e.g. random when strength = 1, or the init image at strength = 0.
* WIP: Added a unit test for the new strength parameter in the StableDiffusionInpaintingPipeline
still need to add correct regression values
* Created a is_strength_max to initialise from pure random noise
* Updated unit tests w.r.t new strength parameter + fixed new strength unit test
* renamed parameter to avoid confusion with variable of same name
* Updated regression values for new strength test - now passes
* removed 'copied from' comment as this method is now different and divergent from the cpy
* Update src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Ensure backwards compatibility for prepare_mask_and_masked_image
created a return_image boolean and initialised to false
* Ensure backwards compatibility for prepare_latents
* Fixed copy check typo
* Fixes w.r.t backward compibility changes
* make style
* keep function argument ordering same for backwards compatibility in callees with copied from statements
* make fix-copies
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: William Berman <WLBberman@gmail.com>
* refactor controlnet and add img2img and inpaint
* First draft to get pipelines to work
* make style
* Fix more
* Fix more
* More tests
* Fix more
* Make inpainting work
* make style and more tests
* Apply suggestions from code review
* up
* make style
* Fix imports
* Fix more
* Fix more
* Improve examples
* add test
* Make sure import is correctly deprecated
* Make sure everything works in compile mode
* make sure authorship is correctly attributed
* enable deterministic pytorch and cuda operations.
* disable manual seeding.
* make style && make quality for unet_2d tests.
* enable determinism for the unet2dconditional model.
* add CUBLAS_WORKSPACE_CONFIG for better reproducibility.
* relax tolerance (very weird issue, though).
* revert to torch manual_seed() where needed.
* relax more tolerance.
* better placement of the cuda variable and relax more tolerance.
* enable determinism for 3d condition model.
* relax tolerance.
* add: determinism to alt_diffusion.
* relax tolerance for alt diffusion.
* dance diffusion.
* dance diffusion is flaky.
* test_dict_tuple_outputs_equivalent edit.
* fix two more tests.
* fix more ddim tests.
* fix: argument.
* change to diff in place of difference.
* fix: test_save_load call.
* test_save_load_float16 call.
* fix: expected_max_diff
* fix: paint by example.
* relax tolerance.
* add determinism to 1d unet model.
* torch 2.0 regressions seem to be brutal
* determinism to vae.
* add reason to skipping.
* up tolerance.
* determinism to vq.
* determinism to cuda.
* determinism to the generic test pipeline file.
* refactor general pipelines testing a bit.
* determinism to alt diffusion i2i
* up tolerance for alt diff i2i and audio diff
* up tolerance.
* determinism to audioldm
* increase tolerance for audioldm lms.
* increase tolerance for paint by paint.
* increase tolerance for repaint.
* determinism to cycle diffusion and sd 1.
* relax tol for cycle diffusion 🚲
* relax tol for sd 1.0
* relax tol for controlnet.
* determinism to img var.
* relax tol for img variation.
* tolerance to i2i sd
* make style
* determinism to inpaint.
* relax tolerance for inpaiting.
* determinism for inpainting legacy
* relax tolerance.
* determinism to instruct pix2pix
* determinism to model editing.
* model editing tolerance.
* panorama determinism
* determinism to pix2pix zero.
* determinism to sag.
* sd 2. determinism
* sd. tolerance
* disallow tf32 matmul.
* relax tolerance is all you need.
* make style and determinism to sd 2 depth
* relax tolerance for depth.
* tolerance to diffedit.
* tolerance to sd 2 inpaint.
* up tolerance.
* determinism in upscaling.
* tolerance in upscaler.
* more tolerance relaxation.
* determinism to v pred.
* up tol for v_pred
* unclip determinism
* determinism to unclip img2img
* determinism to text to video.
* determinism to last set of tests
* up tol.
* vq cumsum doesn't have a deterministic kernel
* relax tol
* relax tol
* add inferring_controlnet_cond_batch
* Revert "add inferring_controlnet_cond_batch"
This reverts commit abe8d6311d.
* set guess_mode to True
whenever global_pool_conditions is True
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* nit
* add integration test
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* StableDiffusionInpaintingPipeline now resizes input images and masks w.r.t to passed input height and width. Default is already set to 512. This addresses the common tensor mismatch error. Also moved type check into relevant funciton to keep main pipeline body tidy.
* Fixed StableDiffusionInpaintingPrepareMaskAndMaskedImageTests
Due to previous commit these tests were failing as height and width need to be passed into the prepare_mask_and_masked_image function, I have updated the code and added a height/width variable per unit test as it seemed more appropriate than the current hard coded solution
* Added a resolution test to StableDiffusionInpaintPipelineSlowTests
this unit test simply gets the input and resizes it into some that would fail (e.g. would throw a tensor mismatch error/not a mult of 8). Then passes it through the pipeline and verifies it produces output with correct dims w.r.t the passed height and width
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Fix more torch compile breaks
* add tests
* Fix all
* fix controlnet
* fix more
* Add Horace He as co-author.
>
>
Co-authored-by: Horace He <horacehe2007@yahoo.com>
* Add Horace He as co-author.
Co-authored-by: Horace He <horacehe2007@yahoo.com>
---------
Co-authored-by: Horace He <horacehe2007@yahoo.com>
* fix more
* Fix more
* fix more
* Apply suggestions from code review
* fix
* make style
* make fix-copies
* fix
* make sure torch compile
* Clean
* fix test
* Update Pix2PixZero Auto-correlation Loss
* Add fast inversion tests
* Clarify purpose and mark as deprecated
Fix inversion prompt broadcasting
* Register modules set to `None` in config for `test_save_load_optional_components`
* Update new tests to coordinate with #2953
* add mixin class for pipeline from original sd ckpt
* Improve
* make style
* merge main into
* Improve more
* fix more
* up
* Apply suggestions from code review
* finish docs
* rename
* make style
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* add guess mode (WIP)
* fix uncond/cond order
* support guidance_scale=1.0 and batch != 1
* remove magic coeff
* add docstring
* add intergration test
* add document to controlnet.mdx
* made the comments a bit more explanatory
* fix table
* fix: norm group test for UNet3D.
* chore: speed up the panorama tests (fast).
* set default value of _test_inference_batch_single_identical.
* fix: batch_sizes default value.
* add support for prompt embeds to SD ONNX pipeline
* fix up the pipeline copies
* add prompt embeds param to other ONNX pipelines
* fix up prompt embeds param for SD upscaling ONNX pipeline
* add missing type annotations to ONNX pipes
* ⚙️chore(train_controlnet) fix typo in logger message
* ⚙️chore(models) refactor modules order; make them the same as calling order
When printing the BasicTransformerBlock to stdout, I think it's crucial that the attributes order are shown in proper order. And also previously the "3. Feed Forward" comment was not making sense. It should have been close to self.ff but it's instead next to self.norm3
* correct many tests
* remove bogus file
* make style
* correct more tests
* finish tests
* fix one more
* make style
* make unclip deterministic
* ⚙️chore(models/attention) reorganize comments in BasicTransformerBlock class
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* add load textual inversion embeddings draft
* fix quality
* fix typo
* make fix copies
* move to textual inversion mixin
* make it accept from sd-concept library
* accept list of paths to embeddings
* fix styling of stable diffusion pipeline
* add dummy TextualInversionMixin
* add docstring to textualinversionmixin
* add load textual inversion embeddings draft
* fix quality
* fix typo
* make fix copies
* move to textual inversion mixin
* make it accept from sd-concept library
* accept list of paths to embeddings
* fix styling of stable diffusion pipeline
* add dummy TextualInversionMixin
* add docstring to textualinversionmixin
* add case for parsing embedding from auto1111 UI format
Co-authored-by: Evan Jones <evan.a.jones3@gmail.com>
Co-authored-by: Ana Tamais <aninhamoraestamais@gmail.com>
* fix style after rebase
* move textual inversion mixin to loaders
* move mixin inheritance to DiffusionPipeline from StableDiffusionPipeline)
* update dummy class name
* addressed allo comments
* fix old dangling import
* fix style
* proposal
* remove bogus
* Apply suggestions from code review
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Will Berman <wlbberman@gmail.com>
* finish
* make style
* up
* fix code quality
* fix code quality - again
* fix code quality - 3
* fix alt diffusion code quality
* fix model editing pipeline
* Apply suggestions from code review
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* Finish
---------
Co-authored-by: Evan Jones <evan.a.jones3@gmail.com>
Co-authored-by: Ana Tamais <aninhamoraestamais@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Will Berman <wlbberman@gmail.com>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
