mirror of
https://github.com/huggingface/diffusers.git
synced 2026-01-27 17:22:53 +03:00
5383188c7e
* begin depth pipeline * add depth estimation model * fix prepare_depth_mask * add a comment about autocast * copied from, quality, cleanup * begin tests * handle tensors * norm image tensor * fix batch size * fix tests * fix enable_sequential_cpu_offload * fix save load * fix test_save_load_float16 * fix test_save_load_optional_components * fix test_float16_inference * fix test_cpu_offload_forward_pass * fix test_dict_tuple_outputs_equivalent * up * fix fast tests * fix test_stable_diffusion_img2img_multiple_init_images * fix few more fast tests * don't use device map for DPT * fix test_stable_diffusion_pipeline_with_sequential_cpu_offloading * accept external depth maps * prepare_depth_mask -> prepare_depth_map * fix file name * fix file name * quality * check transformers version * fix test names * use skipif * fix import * add docs * skip tests on mps * correct version * uP * Update docs/source/api/pipelines/stable_diffusion_2.mdx * fix fix-copies * fix fix-copies Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: anton- <anton@huggingface.co>
124 lines
7.2 KiB
Plaintext
124 lines
7.2 KiB
Plaintext
<!--Copyright 2022 The HuggingFace Team. All rights reserved.
|
||
|
||
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
||
the License. You may obtain a copy of the License at
|
||
|
||
http://www.apache.org/licenses/LICENSE-2.0
|
||
|
||
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
||
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
||
specific language governing permissions and limitations under the License.
|
||
-->
|
||
|
||
# Stable diffusion pipelines
|
||
|
||
Stable Diffusion is a text-to-image _latent diffusion_ model created by the researchers and engineers from [CompVis](https://github.com/CompVis), [Stability AI](https://stability.ai/) and [LAION](https://laion.ai/). It's trained on 512x512 images from a subset of the [LAION-5B](https://laion.ai/blog/laion-5b/) dataset. This model uses a frozen CLIP ViT-L/14 text encoder to condition the model on text prompts. With its 860M UNet and 123M text encoder, the model is relatively lightweight and can run on consumer GPUs.
|
||
|
||
Latent diffusion is the research on top of which Stable Diffusion was built. It was proposed in [High-Resolution Image Synthesis with Latent Diffusion Models](https://arxiv.org/abs/2112.10752) by Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Björn Ommer. You can learn more details about it in the [specific pipeline for latent diffusion](pipelines/latent_diffusion) that is part of 🤗 Diffusers.
|
||
|
||
For more details about how Stable Diffusion works and how it differs from the base latent diffusion model, please refer to the official [launch announcement post](https://stability.ai/blog/stable-diffusion-announcement) and [this section of our own blog post](https://huggingface.co/blog/stable_diffusion#how-does-stable-diffusion-work).
|
||
|
||
*Tips*:
|
||
- To tweak your prompts on a specific result you liked, you can generate your own latents, as demonstrated in the following notebook: [](https://colab.research.google.com/github/pcuenca/diffusers-examples/blob/main/notebooks/stable-diffusion-seeds.ipynb)
|
||
|
||
*Overview*:
|
||
|
||
| Pipeline | Tasks | Colab | Demo
|
||
|---|---|:---:|:---:|
|
||
| [pipeline_stable_diffusion.py](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py) | *Text-to-Image Generation* | [](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/stable_diffusion.ipynb) | [🤗 Stable Diffusion](https://huggingface.co/spaces/stabilityai/stable-diffusion)
|
||
| [pipeline_stable_diffusion_img2img.py](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_img2img.py) | *Image-to-Image Text-Guided Generation* | [](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/image_2_image_using_diffusers.ipynb) | [🤗 Diffuse the Rest](https://huggingface.co/spaces/huggingface/diffuse-the-rest)
|
||
| [pipeline_stable_diffusion_inpaint.py](https://github.com/huggingface/diffusers/blob/main/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion_inpaint.py) | **Experimental** – *Text-Guided Image Inpainting* | [](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/in_painting_with_stable_diffusion_using_diffusers.ipynb) | Coming soon
|
||
|
||
## Tips
|
||
|
||
### How to load and use different schedulers.
|
||
|
||
The stable diffusion pipeline uses [`PNDMScheduler`] scheduler by default. But `diffusers` provides many other schedulers that can be used with the stable diffusion pipeline such as [`DDIMScheduler`], [`LMSDiscreteScheduler`], [`EulerDiscreteScheduler`], [`EulerAncestralDiscreteScheduler`] etc.
|
||
To use a different scheduler, you can either change it via the [`ConfigMixin.from_config`] method or pass the `scheduler` argument to the `from_pretrained` method of the pipeline. For example, to use the [`EulerDiscreteScheduler`], you can do the following:
|
||
|
||
```python
|
||
>>> from diffusers import StableDiffusionPipeline, EulerDiscreteScheduler
|
||
|
||
>>> pipeline = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")
|
||
>>> pipeline.scheduler = EulerDiscreteScheduler.from_config(pipeline.scheduler.config)
|
||
|
||
>>> # or
|
||
>>> euler_scheduler = EulerDiscreteScheduler.from_pretrained("CompVis/stable-diffusion-v1-4", subfolder="scheduler")
|
||
>>> pipeline = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4", scheduler=euler_scheduler)
|
||
```
|
||
|
||
|
||
### How to convert all use cases with multiple or single pipeline
|
||
|
||
If you want to use all possible use cases in a single `DiffusionPipeline` you can either:
|
||
- Make use of the [Stable Diffusion Mega Pipeline](https://github.com/huggingface/diffusers/tree/main/examples/community#stable-diffusion-mega) or
|
||
- Make use of the `components` functionality to instantiate all components in the most memory-efficient way:
|
||
|
||
```python
|
||
>>> from diffusers import (
|
||
... StableDiffusionPipeline,
|
||
... StableDiffusionImg2ImgPipeline,
|
||
... StableDiffusionInpaintPipeline,
|
||
... )
|
||
|
||
>>> text2img = StableDiffusionPipeline.from_pretrained("CompVis/stable-diffusion-v1-4")
|
||
>>> img2img = StableDiffusionImg2ImgPipeline(**text2img.components)
|
||
>>> inpaint = StableDiffusionInpaintPipeline(**text2img.components)
|
||
|
||
>>> # now you can use text2img(...), img2img(...), inpaint(...) just like the call methods of each respective pipeline
|
||
```
|
||
|
||
## StableDiffusionPipelineOutput
|
||
[[autodoc]] pipelines.stable_diffusion.StableDiffusionPipelineOutput
|
||
|
||
## StableDiffusionPipeline
|
||
[[autodoc]] StableDiffusionPipeline
|
||
- __call__
|
||
- enable_attention_slicing
|
||
- disable_attention_slicing
|
||
- enable_vae_slicing
|
||
- disable_vae_slicing
|
||
- enable_xformers_memory_efficient_attention
|
||
- disable_xformers_memory_efficient_attention
|
||
|
||
## StableDiffusionImg2ImgPipeline
|
||
[[autodoc]] StableDiffusionImg2ImgPipeline
|
||
- __call__
|
||
- enable_attention_slicing
|
||
- disable_attention_slicing
|
||
- enable_xformers_memory_efficient_attention
|
||
- disable_xformers_memory_efficient_attention
|
||
|
||
## StableDiffusionInpaintPipeline
|
||
[[autodoc]] StableDiffusionInpaintPipeline
|
||
- __call__
|
||
- enable_attention_slicing
|
||
- disable_attention_slicing
|
||
- enable_xformers_memory_efficient_attention
|
||
- disable_xformers_memory_efficient_attention
|
||
|
||
## StableDiffusionDepth2ImgPipeline
|
||
[[autodoc]] StableDiffusionDepth2ImgPipeline
|
||
- __call__
|
||
- enable_attention_slicing
|
||
- disable_attention_slicing
|
||
- enable_xformers_memory_efficient_attention
|
||
- disable_xformers_memory_efficient_attention
|
||
|
||
## StableDiffusionImageVariationPipeline
|
||
[[autodoc]] StableDiffusionImageVariationPipeline
|
||
- __call__
|
||
- enable_attention_slicing
|
||
- disable_attention_slicing
|
||
- enable_xformers_memory_efficient_attention
|
||
- disable_xformers_memory_efficient_attention
|
||
|
||
|
||
## StableDiffusionUpscalePipeline
|
||
[[autodoc]] StableDiffusionUpscalePipeline
|
||
- __call__
|
||
- enable_attention_slicing
|
||
- disable_attention_slicing
|
||
- enable_xformers_memory_efficient_attention
|
||
- disable_xformers_memory_efficient_attention
|