* UniPC Multistep add `rescale_betas_zero_snr`
Same patch as DPM and Euler with the patched final alpha cumprod
BF16 doesn't seem to break down, I think cause UniPC upcasts during some
phases already? We could still force an upcast since it only
loses ≈ 0.005 it/s for me but the difference in output is very small. A
better endeavor might upcasting in step() and removing all the other
upcasts elsewhere?
* UniPC ZSNR UT
* Re-add `rescale_betas_zsnr` doc oops
* UniPC UTs iterate solvers on FP16
It wasn't catching errs on order==3. Might be excessive?
* UniPC Multistep fix tensor dtype/device on order=3
* UniPC UTs Add v_pred to fp16 test iter
For completions sake. Probably overkill?
* Add `final_sigma_zero` to UniPCMultistep
Effectively the same trick as DDIM's `set_alpha_to_one` and
DPM's `final_sigma_type='zero'`.
Currently False by default but maybe this should be True?
* `final_sigma_zero: bool` -> `final_sigmas_type: str`
Should 1:1 match DPM Multistep now.
* Set `final_sigmas_type='sigma_min'` in UniPC UTs
* DPMMultistep rescale_betas_zero_snr
* DPM upcast samples in step()
* DPM rescale_betas_zero_snr UT
* DPMSolverMulti move sample upcast after model convert
Avoids having to re-use the dtype.
* Add a newline for Ruff
* EulerAncestral add `rescale_betas_zero_snr`
Uses same infinite sigma fix from EulerDiscrete. Interestingly the
ancestral version had the opposite problem: too much contrast instead of
too little.
* UT for EulerAncestral `rescale_betas_zero_snr`
* EulerAncestral upcast samples during step()
It helps this scheduler too, particularly when the model is using bf16.
While the noise dtype is still the model's it's automatically upcasted
for the add so all it affects is determinism.
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* Add custom timesteps support to LCMScheduler.
* Add custom timesteps support to StableDiffusionPipeline.
* Add custom timesteps support to StableDiffusionXLPipeline.
* Add custom timesteps support to remaining Stable Diffusion pipelines which support LCMScheduler (img2img, inpaint).
* Add custom timesteps support to remaining Stable Diffusion XL pipelines which support LCMScheduler (img2img, inpaint).
* Add custom timesteps support to StableDiffusionControlNetPipeline.
* Add custom timesteps support to T21 Stable Diffusion (XL) Adapters.
* Clean up Stable Diffusion inpaint tests.
* Manually add support for custom timesteps to AltDiffusion pipelines since make fix-copies doesn't appear to work correctly (it deletes the whole pipeline).
* make style
* Refactor pipeline timestep handling into the retrieve_timesteps function.
* Change LCMScheduler.set_timesteps to pick more evenly spaced inference timesteps.
* Change inference_indices implementation to better match previous behavior.
* Add num_inference_steps=26 test case to test_inference_steps.
* run CI
---------
Co-authored-by: patil-suraj <surajp815@gmail.com>
* Refactor LCMScheduler.step such that prev_sample == denoised at the last timestep in the schedule.
* Make timestep scaling when calculating boundary conditions configurable.
* Reparameterize timestep_scaling to be a multiplicative rather than division scaling.
* make style
* fix dtype conversion
* make style
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* stabilize dpmpp for sdxl by using euler at the final step
* add lu's uniform logsnr time steps
* add test
* fix check_copies
* fix tests
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* initial commit for LatentConsistencyModelPipeline and LCMScheduler based on the community pipeline
* Add callback and freeu support.
* apply suggestions from review
* Clean up LCMScheduler
* Remove timeindex argument to LCMScheduler.step.
* Add support for clipping or thresholding the predicted original sample.
* Remove unused methods and arguments in LCMScheduler.
* Improve comment about (lack of) negative prompt support.
* Change input guidance_scale to match the StableDiffusionPipeline (Imagen) CFG formulation.
* Move lcm_origin_steps from pipeline __call__ to LCMScheduler.__init__/config (as origin_steps).
* Fix typo when clipping/thresholding in LCMScheduler.
* Add some initial LCMScheduler tests.
* add type annotations from review
* Fix type annotation bug.
* Override test_add_noise_device in LCMSchedulerTest since hardcoded timesteps doesn't work under default settings.
* Add generator argument pipeline prepare_latents call.
* Cast LCMScheduler.timesteps to long in set_timesteps.
* Add onestep and multistep full loop scheduler tests.
* Set default height/width to None and don't hardcode guidance scale embedding dim.
* Add initial LatentConsistencyPipeline fast and slow tests.
* Add initial documentation for LatentConsistencyModelPipeline and LCMScheduler.
* Make remaining failing fast tests pass.
* make style
* Make original_inference_steps configurable from pipeline __call__ again.
* make style
* Remove guidance_rescale arg from pipeline __call__ since LCM currently doesn't support CFG.
* Make LCMScheduler defaults match config of LCM_Dreamshaper_v7 checkpoint.
* Fix LatentConsistencyPipeline slow tests and add dummy expected slices.
* Add checks for original_steps in LCMScheduler.set_timesteps.
* make fix-copies
* Improve LatentConsistencyModelPipeline docs.
* Apply suggestions from code review
Co-authored-by: Aryan V S <avs050602@gmail.com>
* Apply suggestions from code review
Co-authored-by: Aryan V S <avs050602@gmail.com>
* Apply suggestions from code review
Co-authored-by: Aryan V S <avs050602@gmail.com>
* Update src/diffusers/schedulers/scheduling_lcm.py
* Apply suggestions from code review
Co-authored-by: Aryan V S <avs050602@gmail.com>
* finish
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Aryan V S <avs050602@gmail.com>
* Update Unipc einsum to support 1D and 3D diffusion.
* Add unittest
* Update unittest & edge case
* Fix unittest
* Fix testing_utils.py
* Fix unittest file
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* fix ddim inverse scheduler
* update test of ddim inverse scheduler
* update test of pix2pix_zero
* update test of diffedit
* fix typo
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Add Recent Timestep Scheduling Improvements to DDIM Inverse Scheduler
Roll timesteps by one to reflect origin-destination semantic discrepancy
Restore `set_alpha_to_one` option to handle negative initial timesteps
Remove `set_alpha_to_zero` option not used due to previous truncation
* Bugfix
* Remove unnecessary calls to `detach()`
Use `self.image_processor.preprocess` in DiffEdit pipeline functions
* Preprocess list input for inverted image latents in diffedit pipeline
* Add `timestep_spacing` and `steps_offset` to `DPMSolverMultistepInverseScheduler`
* Update expected test results to account for inverting last forward diffusion step
* Fix inversion progress bar bug
* Add first draft for proper fast tests for DDIMInverseScheduler
* Add deprecated DDIMInverseScheduler kwarg to ConfigMixer registry
* Fix test failure in DPMMultistepInverseScheduler
Invert step specification leads to negative noise variance in SDE-based algs
Add first draft for proper fast tests for DPMMultistepInverseScheduler
* Update expected test results to account for inverting last forward diffusion step
Clean up diffedit fast test
* refactor prior_transformer
adding conversion script
add pipeline
add step_index from pipeline, + remove permute
add zero pad token
remove copy from statement for betas_for_alpha_bar function
* add
* add
* update conversion script for renderer model
* refactor camera a little bit
* clean up
* style
* fix copies
* Update src/diffusers/schedulers/scheduling_heun_discrete.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/shap_e/pipeline_shap_e.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/shap_e/pipeline_shap_e.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* alpha_transform_type
* remove step_index argument
* remove get_sigmas_karras
* remove _yiyi_sigma_to_t
* move the rescale prompt_embeds from prior_transformer to pipeline
* replace baddbmm with einsum to match origial repo
* Revert "replace baddbmm with einsum to match origial repo"
This reverts commit 3f6b435d65.
* add step_index to scale_model_input
* Revert "move the rescale prompt_embeds from prior_transformer to pipeline"
This reverts commit 5b5a8e6be9.
* move rescale from prior_transformer to pipeline
* correct step_index in scale_model_input
* remove print lines
* refactor prior - reduce arguments
* make style
* add prior_image
* arg embedding_proj_norm -> norm_embedding_proj
* add pre-norm for proj_embedding
* move rescale prompt from pipeline to _encode_prompt
* add img2img pipeline
* style
* copies
* Update src/diffusers/models/prior_transformer.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/models/prior_transformer.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/models/prior_transformer.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/models/prior_transformer.py
add arg: encoder_hid_proj
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/models/prior_transformer.py
add new config: norm_in_type
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/models/prior_transformer.py
add new config: added_emb_type
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/models/prior_transformer.py
rename out_dim -> clip_embed_dim
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/models/prior_transformer.py
rename config: out_dim -> clip_embed_dim
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/models/prior_transformer.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/models/prior_transformer.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* finish refactor prior_tranformer
* make style
* refactor renderer
* fix
* make style
* refactor img2img
* remove params_proj
* add test
* add upcast_softmax to prior_transformer
* enable num_images_per_prompt, add save_gif utility
* add
* add fast test
* make style
* add slow test
* style
* add test for img2img
* refactor
* enable batching
* style
* refactor scheduler
* update test
* style
* attempt to solve batch related tests timeout
* add doc
* Update src/diffusers/pipelines/shap_e/pipeline_shap_e.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/pipelines/shap_e/pipeline_shap_e_img2img.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* hardcode rendering related config
* update betas_for_alpha_bar on ddpm_scheduler
* fix copies
* fix
* export_to_gif
* style
* second attempt to speed up batching tests
* add doc page to index
* Remove intermediate clipping
* 3rd attempt to speed up batching tests
* Remvoe time index
* simplify scheduler
* Fix more
* Fix more
* fix more
* make style
* fix schedulers
* fix some more tests
* finish
* add one more test
* Apply suggestions from code review
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* style
* apply feedbacks
* style
* fix copies
* add one example
* style
* add example for img2img
* fix doc
* fix more doc strings
* size -> frame_size
* style
* update doc
* style
* fix on doc
* update repo name
* improve the usage example in shap-e img2img
* add usage examples in the shap-e docs.
* consolidate examples.
* minor fix.
* update doc
* Apply suggestions from code review
* Apply suggestions from code review
* remove upcast
* Make sure background is white
* Update src/diffusers/pipelines/shap_e/pipeline_shap_e.py
* Apply suggestions from code review
* Finish
* Apply suggestions from code review
* Update src/diffusers/pipelines/shap_e/pipeline_shap_e.py
* Make style
---------
Co-authored-by: yiyixuxu <yixu310@gmail,com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Pedro Cuenca <pedro@huggingface.co>
* initial commit
* Improve consistency models sampling implementation.
* Add CMStochasticIterativeScheduler, which implements the multi-step sampler (stochastic_iterative_sampler) in the original code, and make further improvements to sampling.
* Add Unet blocks for consistency models
* Add conversion script for Unet
* Fix bug in new unet blocks
* Fix attention weight loading
* Make design improvements to ConsistencyModelPipeline and CMStochasticIterativeScheduler and add initial version of tests.
* make style
* Make small random test UNet class conditional and set resnet_time_scale_shift to 'scale_shift' to better match consistency model checkpoints.
* Add support for converting a test UNet and non-class-conditional UNets to the consistency models conversion script.
* make style
* Change num_class_embeds to 1000 to better match the original consistency models implementation.
* Add support for distillation in pipeline_consistency_models.py.
* Improve consistency model tests:
- Get small testing checkpoints from hub
- Modify tests to take into account "distillation" parameter of ConsistencyModelPipeline
- Add onestep, multistep tests for distillation and distillation + class conditional
- Add expected image slices for onestep tests
* make style
* Improve ConsistencyModelPipeline:
- Add initial support for class-conditional generation
- Fix initial sigma for onestep generation
- Fix some sigma shape issues
* make style
* Improve ConsistencyModelPipeline:
- add latents __call__ argument and prepare_latents method
- add check_inputs method
- add initial docstrings for ConsistencyModelPipeline.__call__
* make style
* Fix bug when randomly generating class labels for class-conditional generation.
* Switch CMStochasticIterativeScheduler to configuring a sigma schedule and make related changes to the pipeline and tests.
* Remove some unused code and make style.
* Fix small bug in CMStochasticIterativeScheduler.
* Add expected slices for multistep sampling tests and make them pass.
* Work on consistency model fast tests:
- in pipeline, call self.scheduler.scale_model_input before denoising
- get expected slices for Euler and Heun scheduler tests
- make Euler test pass
- mark Heun test as expected fail because it doesn't support prediction_type "sample" yet
- remove DPM and Euler Ancestral tests because they don't support use_karras_sigmas
* make style
* Refactor conversion script to make it easier to add more model architectures to convert in the future.
* Work on ConsistencyModelPipeline tests:
- Fix device bug when handling class labels in ConsistencyModelPipeline.__call__
- Add slow tests for onestep and multistep sampling and make them pass
- Refactor fast tests
- Refactor ConsistencyModelPipeline.__init__
* make style
* Remove the add_noise and add_noise_to_input methods from CMStochasticIterativeScheduler for now.
* Run python utils/check_copies.py --fix_and_overwrite
python utils/check_dummies.py --fix_and_overwrite to make dummy objects for new pipeline and scheduler.
* Make fast tests from PipelineTesterMixin pass.
* make style
* Refactor consistency models pipeline and scheduler:
- Remove support for Karras schedulers (only support CMStochasticIterativeScheduler)
- Move sigma manipulation, input scaling, denoising from pipeline to scheduler
- Make corresponding changes to tests and ensure they pass
* make style
* Add docstrings and further refactor pipeline and scheduler.
* make style
* Add initial version of the consistency models documentation.
* Refactor custom timesteps logic following DDPMScheduler/IFPipeline and temporarily add torch 2.0 SDPA kernel selection logic for debugging.
* make style
* Convert current slow tests to use fp16 and flash attention.
* make style
* Add slow tests for normal attention on cuda device.
* make style
* Fix attention weights loading
* Update consistency model fast tests for new test checkpoints with attention fix.
* make style
* apply suggestions
* Add add_noise method to CMStochasticIterativeScheduler (copied from EulerDiscreteScheduler).
* Conversion script now outputs pipeline instead of UNet and add support for LSUN-256 models and different schedulers.
* When both timesteps and num_inference_steps are supplied, raise warning instead of error (timesteps take precedence).
* make style
* Add remaining diffusers model checkpoints for models in the original consistency model release and update usage example.
* apply suggestions from review
* make style
* fix attention naming
* Add tests for CMStochasticIterativeScheduler.
* make style
* Make CMStochasticIterativeScheduler tests pass.
* make style
* Override test_step_shape in CMStochasticIterativeSchedulerTest instead of modifying it in SchedulerCommonTest.
* make style
* rename some models
* Improve API
* rename some models
* Remove duplicated block
* Add docstring and make torch compile work
* More fixes
* Fixes
* Apply suggestions from code review
* Apply suggestions from code review
* add more docstring
* update consistency conversion script
---------
Co-authored-by: ayushmangal <ayushmangal@microsoft.com>
Co-authored-by: Ayush Mangal <43698245+ayushtues@users.noreply.github.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Add timestep_spacing to DDPM, LMSDiscrete, PNDM.
* Remove spurious line.
* More easy schedulers.
* Add `linspace` to DDIM
* Noise sigma for `trailing`.
* Add timestep_spacing to DEISMultistepScheduler.
Not sure the range is the way it was intended.
* Fix: remove line used to debug.
* Support timestep_spacing in DPMSolverMultistep, DPMSolverSDE, UniPC
* Fix: convert to numpy.
* Use sched. defaults when instantiating from_config
For params not present in the original configuration.
This makes it possible to switch pipeline schedulers even if they use
different timestep_spacing (or any other param).
* Apply suggestions from code review
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Missing args in DPMSolverMultistep
* Test: default args not in config
* Style
* Fix scheduler name in test
* Remove duplicated entries
* Add test for solver_type
This test currently fails in main. When switching from DEIS to UniPC,
solver_type is "logrho" (the default value from DEIS), which gets
translated to "bh1" by UniPC. This is different to the default value for
UniPC: "bh2". This is where the translation happens: 36d22d0709/src/diffusers/schedulers/scheduling_unipc_multistep.py (L171)
* UniPC: use same default for solver_type
Fixes a bug when switching from UniPC from another scheduler (i.e.,
DEIS) that uses a different solver type. The solver is now the same as
if we had instantiated the scheduler directly.
* do not save use default values
* fix more
* fix all
* fix schedulers
* fix more
* finish for real
* finish for real
* flaky tests
* Update tests/pipelines/stable_diffusion/test_stable_diffusion_pix2pix_zero.py
* Default steps_offset to 0.
* Add missing docstrings
* Apply suggestions from code review
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* add paradigms parallel sampling pipeline
* linting
* ran make fix-copies
* add paradigms parallel sampling pipeline
* linting
* ran make fix-copies
* Apply suggestions from code review
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* changes based on review
* add docs for paradigms
* update docs with paradigms abstract
* improve documentation, and add tests for ddim/ddpm batch_step_no_noise
* fix docs and run make fix-copies
* minor changes to docs.
* Apply suggestions from code review
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* move parallel scheduler to new classes for DDPMParallelScheduler and DDIMParallelScheduler
* remove changes for scheduling_ddim, adjust licenses, credits, and commented code
* fix tensor type that is breaking tests
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Implement option for rescaling betas to zero terminal SNR
* Implement rescale classifier free guidance in pipeline_stable_diffusion.py
* focus on DDIM
* make style
* make style
* make style
* make style
* Apply suggestions from Peter Lin
* Apply suggestions from Peter Lin
* make style
* Apply suggestions from code review
* Apply suggestions from code review
* make style
* make style
---------
Co-authored-by: MaxWe00 <gitlab.9v1lq@slmail.me>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Fix DPM single
* add test
* fix one more bug
* Apply suggestions from code review
Co-authored-by: StAlKeR7779 <stalkek7779@yandex.ru>
---------
Co-authored-by: StAlKeR7779 <stalkek7779@yandex.ru>
* Remove ONNX tests from PR.
They are already a part of push_tests.yml.
* Remove mps tests from PRs.
They are already performed on push.
* Fix workflow name for fast push tests.
* Extract mps tests to a workflow.
For better control/filtering.
* Remove --extra-index-url from mps tests
* Increase tolerance of mps test
This test passes in my Mac (Ventura 13.3) but fails in the CI hardware
(Ventura 13.2). I ran the local tests following the same steps that
exist in the CI workflow.
* Temporarily run mps tests on pr
So we can test.
* Revert "Temporarily run mps tests on pr"
Tests passed, go back to running on push.
* fix multistep dpmsolver for cosine schedule (deepfloy-if)
* fix a typo
* Update src/diffusers/schedulers/scheduling_dpmsolver_multistep.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/schedulers/scheduling_dpmsolver_multistep.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/schedulers/scheduling_dpmsolver_multistep.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/schedulers/scheduling_dpmsolver_multistep.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* Update src/diffusers/schedulers/scheduling_dpmsolver_multistep.py
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* update all dpmsolver (singlestep, multistep, dpm, dpm++) for cosine noise schedule
* add test, fix style
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
