* cosmos predict2.5 base: convert chkpt & pipeline
- New scheduler: scheduling_flow_unipc_multistep.py
- Changes to TransformerCosmos for text embeddings via crossattn_proj
* scheduler cleanup
* simplify inference pipeline
* cleanup scheduler + tests
* Basic tests for flow unipc
* working b2b inference
* Rename everything
* Tests for pipeline present, but not working (predict2 also not working)
* docstring update
* wrapper pipelines + make style
* remove unnecessary files
* UniPCMultistep: support use_karras_sigmas=True and use_flow_sigmas=True
* use UniPCMultistepScheduler + fix tests for pipeline
* Remove FlowUniPCMultistepScheduler
* UniPCMultistepScheduler for use_flow_sigmas=True & use_karras_sigmas=True
* num_inference_steps=36 due to bug in scheduler used by predict2.5
* Address comments
* make style + make fix-copies
* fix tests + remove references to old pipelines
* address comments
* add revision in from_pretrained call
* fix tests
* UniPC Multistep add `rescale_betas_zero_snr`
Same patch as DPM and Euler with the patched final alpha cumprod
BF16 doesn't seem to break down, I think cause UniPC upcasts during some
phases already? We could still force an upcast since it only
loses ≈ 0.005 it/s for me but the difference in output is very small. A
better endeavor might upcasting in step() and removing all the other
upcasts elsewhere?
* UniPC ZSNR UT
* Re-add `rescale_betas_zsnr` doc oops
* UniPC UTs iterate solvers on FP16
It wasn't catching errs on order==3. Might be excessive?
* UniPC Multistep fix tensor dtype/device on order=3
* UniPC UTs Add v_pred to fp16 test iter
For completions sake. Probably overkill?
* Add `final_sigma_zero` to UniPCMultistep
Effectively the same trick as DDIM's `set_alpha_to_one` and
DPM's `final_sigma_type='zero'`.
Currently False by default but maybe this should be True?
* `final_sigma_zero: bool` -> `final_sigmas_type: str`
Should 1:1 match DPM Multistep now.
* Set `final_sigmas_type='sigma_min'` in UniPC UTs
* DPMMultistep rescale_betas_zero_snr
* DPM upcast samples in step()
* DPM rescale_betas_zero_snr UT
* DPMSolverMulti move sample upcast after model convert
Avoids having to re-use the dtype.
* Add a newline for Ruff
* EulerAncestral add `rescale_betas_zero_snr`
Uses same infinite sigma fix from EulerDiscrete. Interestingly the
ancestral version had the opposite problem: too much contrast instead of
too little.
* UT for EulerAncestral `rescale_betas_zero_snr`
* EulerAncestral upcast samples during step()
It helps this scheduler too, particularly when the model is using bf16.
While the noise dtype is still the model's it's automatically upcasted
for the add so all it affects is determinism.
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
* Add custom timesteps support to LCMScheduler.
* Add custom timesteps support to StableDiffusionPipeline.
* Add custom timesteps support to StableDiffusionXLPipeline.
* Add custom timesteps support to remaining Stable Diffusion pipelines which support LCMScheduler (img2img, inpaint).
* Add custom timesteps support to remaining Stable Diffusion XL pipelines which support LCMScheduler (img2img, inpaint).
* Add custom timesteps support to StableDiffusionControlNetPipeline.
* Add custom timesteps support to T21 Stable Diffusion (XL) Adapters.
* Clean up Stable Diffusion inpaint tests.
* Manually add support for custom timesteps to AltDiffusion pipelines since make fix-copies doesn't appear to work correctly (it deletes the whole pipeline).
* make style
* Refactor pipeline timestep handling into the retrieve_timesteps function.
* Change LCMScheduler.set_timesteps to pick more evenly spaced inference timesteps.
* Change inference_indices implementation to better match previous behavior.
* Add num_inference_steps=26 test case to test_inference_steps.
* run CI
---------
Co-authored-by: patil-suraj <surajp815@gmail.com>
* Refactor LCMScheduler.step such that prev_sample == denoised at the last timestep in the schedule.
* Make timestep scaling when calculating boundary conditions configurable.
* Reparameterize timestep_scaling to be a multiplicative rather than division scaling.
* make style
* fix dtype conversion
* make style
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* stabilize dpmpp for sdxl by using euler at the final step
* add lu's uniform logsnr time steps
* add test
* fix check_copies
* fix tests
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* initial commit for LatentConsistencyModelPipeline and LCMScheduler based on the community pipeline
* Add callback and freeu support.
* apply suggestions from review
* Clean up LCMScheduler
* Remove timeindex argument to LCMScheduler.step.
* Add support for clipping or thresholding the predicted original sample.
* Remove unused methods and arguments in LCMScheduler.
* Improve comment about (lack of) negative prompt support.
* Change input guidance_scale to match the StableDiffusionPipeline (Imagen) CFG formulation.
* Move lcm_origin_steps from pipeline __call__ to LCMScheduler.__init__/config (as origin_steps).
* Fix typo when clipping/thresholding in LCMScheduler.
* Add some initial LCMScheduler tests.
* add type annotations from review
* Fix type annotation bug.
* Override test_add_noise_device in LCMSchedulerTest since hardcoded timesteps doesn't work under default settings.
* Add generator argument pipeline prepare_latents call.
* Cast LCMScheduler.timesteps to long in set_timesteps.
* Add onestep and multistep full loop scheduler tests.
* Set default height/width to None and don't hardcode guidance scale embedding dim.
* Add initial LatentConsistencyPipeline fast and slow tests.
* Add initial documentation for LatentConsistencyModelPipeline and LCMScheduler.
* Make remaining failing fast tests pass.
* make style
* Make original_inference_steps configurable from pipeline __call__ again.
* make style
* Remove guidance_rescale arg from pipeline __call__ since LCM currently doesn't support CFG.
* Make LCMScheduler defaults match config of LCM_Dreamshaper_v7 checkpoint.
* Fix LatentConsistencyPipeline slow tests and add dummy expected slices.
* Add checks for original_steps in LCMScheduler.set_timesteps.
* make fix-copies
* Improve LatentConsistencyModelPipeline docs.
* Apply suggestions from code review
Co-authored-by: Aryan V S <avs050602@gmail.com>
* Apply suggestions from code review
Co-authored-by: Aryan V S <avs050602@gmail.com>
* Apply suggestions from code review
Co-authored-by: Aryan V S <avs050602@gmail.com>
* Update src/diffusers/schedulers/scheduling_lcm.py
* Apply suggestions from code review
Co-authored-by: Aryan V S <avs050602@gmail.com>
* finish
---------
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Aryan V S <avs050602@gmail.com>
* Update Unipc einsum to support 1D and 3D diffusion.
* Add unittest
* Update unittest & edge case
* Fix unittest
* Fix testing_utils.py
* Fix unittest file
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
* fix ddim inverse scheduler
* update test of ddim inverse scheduler
* update test of pix2pix_zero
* update test of diffedit
* fix typo
---------
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
