From 21e61eb3a9d16a46245bd284fea3aa19e66772f5 Mon Sep 17 00:00:00 2001 From: YiYi Xu Date: Thu, 15 Dec 2022 13:04:40 -1000 Subject: [PATCH] Added a README page for docs and a "schedulers" page (#1710) Co-authored-by: Patrick von Platen Co-authored-by: Pedro Cuenca --- .gitignore | 3 + docs/README.md | 266 ++++++++++++++++++ docs/source/_toctree.yml | 40 ++- docs/source/api/pipelines/overview.mdx | 50 ++-- docs/source/api/schedulers.mdx | 183 ------------ docs/source/api/schedulers/ddim.mdx | 27 ++ docs/source/api/schedulers/ddpm.mdx | 27 ++ docs/source/api/schedulers/dpm_discrete.mdx | 22 ++ .../api/schedulers/dpm_discrete_ancestral.mdx | 22 ++ docs/source/api/schedulers/euler.mdx | 21 ++ .../source/api/schedulers/euler_ancestral.mdx | 21 ++ docs/source/api/schedulers/heun.mdx | 23 ++ docs/source/api/schedulers/ipndm.mdx | 20 ++ docs/source/api/schedulers/lms_discrete.mdx | 20 ++ .../api/schedulers/multistep_dpm_solver.mdx | 20 ++ docs/source/api/schedulers/overview.mdx | 83 ++++++ docs/source/api/schedulers/pndm.mdx | 20 ++ docs/source/api/schedulers/repaint.mdx | 23 ++ docs/source/api/schedulers/score_sde_ve.mdx | 20 ++ docs/source/api/schedulers/score_sde_vp.mdx | 26 ++ .../api/schedulers/singlestep_dpm_solver.mdx | 20 ++ .../api/schedulers/stochastic_karras_ve.mdx | 20 ++ docs/source/api/schedulers/vq_diffusion.mdx | 20 ++ docs/source/index.mdx | 2 +- docs/source/quicktour.mdx | 2 +- docs/source/using-diffusers/schedulers.mdx | 2 +- src/diffusers/schedulers/README.md | 2 +- 27 files changed, 791 insertions(+), 214 deletions(-) create mode 100644 docs/README.md delete mode 100644 docs/source/api/schedulers.mdx create mode 100644 docs/source/api/schedulers/ddim.mdx create mode 100644 docs/source/api/schedulers/ddpm.mdx create mode 100644 docs/source/api/schedulers/dpm_discrete.mdx create mode 100644 docs/source/api/schedulers/dpm_discrete_ancestral.mdx create mode 100644 docs/source/api/schedulers/euler.mdx create mode 100644 docs/source/api/schedulers/euler_ancestral.mdx create mode 100644 docs/source/api/schedulers/heun.mdx create mode 100644 docs/source/api/schedulers/ipndm.mdx create mode 100644 docs/source/api/schedulers/lms_discrete.mdx create mode 100644 docs/source/api/schedulers/multistep_dpm_solver.mdx create mode 100644 docs/source/api/schedulers/overview.mdx create mode 100644 docs/source/api/schedulers/pndm.mdx create mode 100644 docs/source/api/schedulers/repaint.mdx create mode 100644 docs/source/api/schedulers/score_sde_ve.mdx create mode 100644 docs/source/api/schedulers/score_sde_vp.mdx create mode 100644 docs/source/api/schedulers/singlestep_dpm_solver.mdx create mode 100644 docs/source/api/schedulers/stochastic_karras_ve.mdx create mode 100644 docs/source/api/schedulers/vq_diffusion.mdx diff --git a/.gitignore b/.gitignore index 950e9e5db9..da087d26f4 100644 --- a/.gitignore +++ b/.gitignore @@ -166,3 +166,6 @@ tags .DS_Store # RL pipelines may produce mp4 outputs *.mp4 + +# dependencies +/transformers diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000000..b2d48dfee1 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,266 @@ + + +# Generating the documentation + +To generate the documentation, you first have to build it. Several packages are necessary to build the doc, +you can install them with the following command, at the root of the code repository: + +```bash +pip install -e ".[docs]" +``` + +Then you need to install our open source documentation builder tool: + +```bash +pip install git+https://github.com/huggingface/doc-builder +``` + +--- +**NOTE** + +You only need to generate the documentation to inspect it locally (if you're planning changes and want to +check how they look before committing for instance). You don't have to commit the built documentation. + +--- + +## Previewing the documentation + +To preview the docs, first install the `watchdog` module with: + +```bash +pip install watchdog +``` + +Then run the following command: + +```bash +doc-builder preview {package_name} {path_to_docs} +``` + +For example: + +```bash +doc-builder preview diffusers docs/source/ +``` + +The docs will be viewable at [http://localhost:3000](http://localhost:3000). You can also preview the docs once you have opened a PR. You will see a bot add a comment to a link where the documentation with your changes lives. + +--- +**NOTE** + +The `preview` command only works with existing doc files. When you add a completely new file, you need to update `_toctree.yml` & restart `preview` command (`ctrl-c` to stop it & call `doc-builder preview ...` again). + +--- + +## Adding a new element to the navigation bar + +Accepted files are Markdown (.md or .mdx). + +Create a file with its extension and put it in the source directory. You can then link it to the toc-tree by putting +the filename without the extension in the [`_toctree.yml`](https://github.com/huggingface/diffusers/blob/main/docs/source/_toctree.yml) file. + +## Renaming section headers and moving sections + +It helps to keep the old links working when renaming the section header and/or moving sections from one document to another. This is because the old links are likely to be used in Issues, Forums, and Social media and it'd make for a much more superior user experience if users reading those months later could still easily navigate to the originally intended information. + +Therefore, we simply keep a little map of moved sections at the end of the document where the original section was. The key is to preserve the original anchor. + +So if you renamed a section from: "Section A" to "Section B", then you can add at the end of the file: + +``` +Sections that were moved: + +[ Section A ] +``` +and of course, if you moved it to another file, then: + +``` +Sections that were moved: + +[ Section A ] +``` + +Use the relative style to link to the new file so that the versioned docs continue to work. + +For an example of a rich moved section set please see the very end of [the transformers Trainer doc](https://github.com/huggingface/transformers/blob/main/docs/source/en/main_classes/trainer.mdx). + + +## Writing Documentation - Specification + +The `huggingface/diffusers` documentation follows the +[Google documentation](https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html) style for docstrings, +although we can write them directly in Markdown. + +### Adding a new tutorial + +Adding a new tutorial or section is done in two steps: + +- Add a new file under `docs/source`. This file can either be ReStructuredText (.rst) or Markdown (.md). +- Link that file in `docs/source/_toctree.yml` on the correct toc-tree. + +Make sure to put your new file under the proper section. It's unlikely to go in the first section (*Get Started*), so +depending on the intended targets (beginners, more advanced users, or researchers) it should go in sections two, three, or four. + +### Adding a new pipeline/scheduler + +When adding a new pipeline: + +- create a file `xxx.mdx` under `docs/source/api/pipelines` (don't hesitate to copy an existing file as template). +- Link that file in (*Diffusers Summary*) section in `docs/source/api/pipelines/overview.mdx`, along with the link to the paper, and a colab notebook (if available). +- Write a short overview of the diffusion model: + - Overview with paper & authors + - Paper abstract + - Tips and tricks and how to use it best + - Possible an end-to-end example of how to use it +- Add all the pipeline classes that should be linked in the diffusion model. These classes should be added using our Markdown syntax. Usually as follows: + +``` +## XXXPipeline + +[[autodoc]] XXXPipeline +``` + +This will include every public method of the pipeline that is documented. You can specify which methods should be in the docs: + +``` +## XXXPipeline + +[[autodoc]] XXXPipeline + - __call__ +``` + +You can follow the same process to create a new scheduler under the `docs/source/api/schedulers` folder + +### Writing source documentation + +Values that should be put in `code` should either be surrounded by backticks: \`like so\`. Note that argument names +and objects like True, None, or any strings should usually be put in `code`. + +When mentioning a class, function, or method, it is recommended to use our syntax for internal links so that our tool +adds a link to its documentation with this syntax: \[\`XXXClass\`\] or \[\`function\`\]. This requires the class or +function to be in the main package. + +If you want to create a link to some internal class or function, you need to +provide its path. For instance: \[\`pipeline_utils.ImagePipelineOutput\`\]. This will be converted into a link with +`pipeline_utils.ImagePipelineOutput` in the description. To get rid of the path and only keep the name of the object you are +linking to in the description, add a ~: \[\`~pipeline_utils.ImagePipelineOutput\`\] will generate a link with `ImagePipelineOutput` in the description. + +The same works for methods so you can either use \[\`XXXClass.method\`\] or \[~\`XXXClass.method\`\]. + +#### Defining arguments in a method + +Arguments should be defined with the `Args:` (or `Arguments:` or `Parameters:`) prefix, followed by a line return and +an indentation. The argument should be followed by its type, with its shape if it is a tensor, a colon, and its +description: + +``` + Args: + n_layers (`int`): The number of layers of the model. +``` + +If the description is too long to fit in one line, another indentation is necessary before writing the description +after the argument. + +Here's an example showcasing everything so far: + +``` + Args: + input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): + Indices of input sequence tokens in the vocabulary. + + Indices can be obtained using [`AlbertTokenizer`]. See [`~PreTrainedTokenizer.encode`] and + [`~PreTrainedTokenizer.__call__`] for details. + + [What are input IDs?](../glossary#input-ids) +``` + +For optional arguments or arguments with defaults we follow the following syntax: imagine we have a function with the +following signature: + +``` +def my_function(x: str = None, a: float = 1): +``` + +then its documentation should look like this: + +``` + Args: + x (`str`, *optional*): + This argument controls ... + a (`float`, *optional*, defaults to 1): + This argument is used to ... +``` + +Note that we always omit the "defaults to \`None\`" when None is the default for any argument. Also note that even +if the first line describing your argument type and its default gets long, you can't break it on several lines. You can +however write as many lines as you want in the indented description (see the example above with `input_ids`). + +#### Writing a multi-line code block + +Multi-line code blocks can be useful for displaying examples. They are done between two lines of three backticks as usual in Markdown: + + +```` +``` +# first line of code +# second line +# etc +``` +```` + +#### Writing a return block + +The return block should be introduced with the `Returns:` prefix, followed by a line return and an indentation. +The first line should be the type of the return, followed by a line return. No need to indent further for the elements +building the return. + +Here's an example of a single value return: + +``` + Returns: + `List[int]`: A list of integers in the range [0, 1] --- 1 for a special token, 0 for a sequence token. +``` + +Here's an example of a tuple return, comprising several objects: + +``` + Returns: + `tuple(torch.FloatTensor)` comprising various elements depending on the configuration ([`BertConfig`]) and inputs: + - ** loss** (*optional*, returned when `masked_lm_labels` is provided) `torch.FloatTensor` of shape `(1,)` -- + Total loss is the sum of the masked language modeling loss and the next sequence prediction (classification) loss. + - **prediction_scores** (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`) -- + Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). +``` + +#### Adding an image + +Due to the rapidly growing repository, it is important to make sure that no files that would significantly weigh down the repository are added. This includes images, videos, and other non-text files. We prefer to leverage a hf.co hosted `dataset` like +the ones hosted on [`hf-internal-testing`](https://huggingface.co/hf-internal-testing) in which to place these files and reference +them by URL. We recommend putting them in the following dataset: [huggingface/documentation-images](https://huggingface.co/datasets/huggingface/documentation-images). +If an external contribution, feel free to add the images to your PR and ask a Hugging Face member to migrate your images +to this dataset. + +## Styling the docstring + +We have an automatic script running with the `make style` command that will make sure that: +- the docstrings fully take advantage of the line width +- all code examples are formatted using black, like the code of the Transformers library + +This script may have some weird failures if you made a syntax mistake or if you uncover a bug. Therefore, it's +recommended to commit your changes before running `make style`, so you can revert the changes done by that script +easily. + diff --git a/docs/source/_toctree.yml b/docs/source/_toctree.yml index 2e49b10fd8..b75c658e76 100644 --- a/docs/source/_toctree.yml +++ b/docs/source/_toctree.yml @@ -78,8 +78,6 @@ - sections: - local: api/models title: "Models" - - local: api/schedulers - title: "Schedulers" - local: api/diffusion_pipeline title: "Diffusion Pipeline" - local: api/logging @@ -129,6 +127,44 @@ - local: api/pipelines/audio_diffusion title: "Audio Diffusion" title: "Pipelines" + - sections: + - local: api/schedulers/overview + title: "Overview" + - local: api/schedulers/ddim + title: "DDIM" + - local: api/schedulers/ddpm + title: "DDPM" + - local: api/schedulers/singlestep_dpm_solver + title: "Singlestep DPM-Solver" + - local: api/schedulers/multistep_dpm_solver + title: "Multistep DPM-Solver" + - local: api/schedulers/heun + title: "Heun Scheduler" + - local: api/schedulers/dpm_discrete + title: "DPM Discrete Scheduler" + - local: api/schedulers/dpm_discrete_ancestral + title: "DPM Discrete Scheduler with ancestral sampling" + - local: api/schedulers/stochastic_karras_ve + title: "Stochastic Kerras VE" + - local: api/schedulers/lms_discrete + title: "Linear Multistep" + - local: api/schedulers/pndm + title: "PNDM" + - local: api/schedulers/score_sde_ve + title: "VE-SDE" + - local: api/schedulers/ipndm + title: "IPNDM" + - local: api/schedulers/score_sde_vp + title: "VP-SDE" + - local: api/schedulers/euler + title: "Euler scheduler" + - local: api/schedulers/euler_ancestral + title: "Euler Ancestral Scheduler" + - local: api/schedulers/vq_diffusion + title: "VQDiffusionScheduler" + - local: api/schedulers/repaint + title: "RePaint Scheduler" + title: "Schedulers" - sections: - local: api/experimental/rl title: "RL Planning" diff --git a/docs/source/api/pipelines/overview.mdx b/docs/source/api/pipelines/overview.mdx index 05c8d53adc..1df81495be 100644 --- a/docs/source/api/pipelines/overview.mdx +++ b/docs/source/api/pipelines/overview.mdx @@ -44,31 +44,31 @@ available a colab notebook to directly try them out. | Pipeline | Paper | Tasks | Colab |---|---|:---:|:---:| -| [alt_diffusion](./api/pipelines/alt_diffusion) | [**AltDiffusion**](https://arxiv.org/abs/2211.06679) | Image-to-Image Text-Guided Generation | - -| [audio_diffusion](./api/pipelines/audio_diffusion) | [**Audio Diffusion**](https://github.com/teticio/audio_diffusion.git) | Unconditional Audio Generation | -| [cycle_diffusion](./api/pipelines/cycle_diffusion) | [**Cycle Diffusion**](https://arxiv.org/abs/2210.05559) | Image-to-Image Text-Guided Generation | -| [dance_diffusion](./api/pipelines/dance_diffusion) | [**Dance Diffusion**](https://github.com/williamberman/diffusers.git) | Unconditional Audio Generation | -| [ddpm](./api/pipelines/ddpm) | [**Denoising Diffusion Probabilistic Models**](https://arxiv.org/abs/2006.11239) | Unconditional Image Generation | -| [ddim](./api/pipelines/ddim) | [**Denoising Diffusion Implicit Models**](https://arxiv.org/abs/2010.02502) | Unconditional Image Generation | -| [latent_diffusion](./api/pipelines/latent_diffusion) | [**High-Resolution Image Synthesis with Latent Diffusion Models**](https://arxiv.org/abs/2112.10752)| Text-to-Image Generation | -| [latent_diffusion](./api/pipelines/latent_diffusion) | [**High-Resolution Image Synthesis with Latent Diffusion Models**](https://arxiv.org/abs/2112.10752)| Super Resolution Image-to-Image | -| [latent_diffusion_uncond](./api/pipelines/latent_diffusion_uncond) | [**High-Resolution Image Synthesis with Latent Diffusion Models**](https://arxiv.org/abs/2112.10752) | Unconditional Image Generation | -| [paint_by_example](./api/pipelines/paint_by_example) | [**Paint by Example: Exemplar-based Image Editing with Diffusion Models**](https://arxiv.org/abs/2211.13227) | Image-Guided Image Inpainting | -| [pndm](./api/pipelines/pndm) | [**Pseudo Numerical Methods for Diffusion Models on Manifolds**](https://arxiv.org/abs/2202.09778) | Unconditional Image Generation | -| [score_sde_ve](./api/pipelines/score_sde_ve) | [**Score-Based Generative Modeling through Stochastic Differential Equations**](https://openreview.net/forum?id=PxTIG12RRHS) | Unconditional Image Generation | -| [score_sde_vp](./api/pipelines/score_sde_vp) | [**Score-Based Generative Modeling through Stochastic Differential Equations**](https://openreview.net/forum?id=PxTIG12RRHS) | Unconditional Image Generation | -| [stable_diffusion](./api/pipelines/stable_diffusion) | [**Stable Diffusion**](https://stability.ai/blog/stable-diffusion-public-release) | Text-to-Image Generation | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/training_example.ipynb) -| [stable_diffusion](./api/pipelines/stable_diffusion) | [**Stable Diffusion**](https://stability.ai/blog/stable-diffusion-public-release) | Image-to-Image Text-Guided Generation | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/image_2_image_using_diffusers.ipynb) -| [stable_diffusion](./api/pipelines/stable_diffusion) | [**Stable Diffusion**](https://stability.ai/blog/stable-diffusion-public-release) | Text-Guided Image Inpainting | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/in_painting_with_stable_diffusion_using_diffusers.ipynb) -| [stable_diffusion_2](./api/pipelines/stable_diffusion_2) | [**Stable Diffusion 2**](https://stability.ai/blog/stable-diffusion-v2-release) | Text-to-Image Generation | -| [stable_diffusion_2](./api/pipelines/stable_diffusion_2) | [**Stable Diffusion 2**](https://stability.ai/blog/stable-diffusion-v2-release) | Text-Guided Image Inpainting | -| [stable_diffusion_2](./api/pipelines/stable_diffusion_2) | [**Stable Diffusion 2**](https://stability.ai/blog/stable-diffusion-v2-release) | Text-Guided Super Resolution Image-to-Image | -| [stable_diffusion_safe](./api/pipelines/stable_diffusion_safe) | [**Safe Stable Diffusion**](https://arxiv.org/abs/2211.05105) | Text-Guided Generation | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ml-research/safe-latent-diffusion/blob/main/examples/Safe%20Latent%20Diffusion.ipynb) -| [stochastic_karras_ve](./api/pipelines/stochastic_karras_ve) | [**Elucidating the Design Space of Diffusion-Based Generative Models**](https://arxiv.org/abs/2206.00364) | Unconditional Image Generation | -| [versatile_diffusion](./api/pipelines/versatile_diffusion) | [Versatile Diffusion: Text, Images and Variations All in One Diffusion Model](https://arxiv.org/abs/2211.08332) | Text-to-Image Generation | -| [versatile_diffusion](./api/pipelines/versatile_diffusion) | [Versatile Diffusion: Text, Images and Variations All in One Diffusion Model](https://arxiv.org/abs/2211.08332) | Image Variations Generation | -| [versatile_diffusion](./api/pipelines/versatile_diffusion) | [Versatile Diffusion: Text, Images and Variations All in One Diffusion Model](https://arxiv.org/abs/2211.08332) | Dual Image and Text Guided Generation | -| [vq_diffusion](./api/pipelines/vq_diffusion) | [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://arxiv.org/abs/2111.14822) | Text-to-Image Generation | +| [alt_diffusion](./alt_diffusion) | [**AltDiffusion**](https://arxiv.org/abs/2211.06679) | Image-to-Image Text-Guided Generation | - +| [audio_diffusion](./audio_diffusion) | [**Audio Diffusion**](https://github.com/teticio/audio_diffusion.git) | Unconditional Audio Generation | +| [cycle_diffusion](./cycle_diffusion) | [**Cycle Diffusion**](https://arxiv.org/abs/2210.05559) | Image-to-Image Text-Guided Generation | +| [dance_diffusion](./dance_diffusion) | [**Dance Diffusion**](https://github.com/williamberman/diffusers.git) | Unconditional Audio Generation | +| [ddpm](./ddpm) | [**Denoising Diffusion Probabilistic Models**](https://arxiv.org/abs/2006.11239) | Unconditional Image Generation | +| [ddim](./ddim) | [**Denoising Diffusion Implicit Models**](https://arxiv.org/abs/2010.02502) | Unconditional Image Generation | +| [latent_diffusion](./latent_diffusion) | [**High-Resolution Image Synthesis with Latent Diffusion Models**](https://arxiv.org/abs/2112.10752)| Text-to-Image Generation | +| [latent_diffusion](./latent_diffusion) | [**High-Resolution Image Synthesis with Latent Diffusion Models**](https://arxiv.org/abs/2112.10752)| Super Resolution Image-to-Image | +| [latent_diffusion_uncond](./latent_diffusion_uncond) | [**High-Resolution Image Synthesis with Latent Diffusion Models**](https://arxiv.org/abs/2112.10752) | Unconditional Image Generation | +| [paint_by_example](./paint_by_example) | [**Paint by Example: Exemplar-based Image Editing with Diffusion Models**](https://arxiv.org/abs/2211.13227) | Image-Guided Image Inpainting | +| [pndm](./pndm) | [**Pseudo Numerical Methods for Diffusion Models on Manifolds**](https://arxiv.org/abs/2202.09778) | Unconditional Image Generation | +| [score_sde_ve](./score_sde_ve) | [**Score-Based Generative Modeling through Stochastic Differential Equations**](https://openreview.net/forum?id=PxTIG12RRHS) | Unconditional Image Generation | +| [score_sde_vp](./score_sde_vp) | [**Score-Based Generative Modeling through Stochastic Differential Equations**](https://openreview.net/forum?id=PxTIG12RRHS) | Unconditional Image Generation | +| [stable_diffusion](./stable_diffusion) | [**Stable Diffusion**](https://stability.ai/blog/stable-diffusion-public-release) | Text-to-Image Generation | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/training_example.ipynb) +| [stable_diffusion](./stable_diffusion) | [**Stable Diffusion**](https://stability.ai/blog/stable-diffusion-public-release) | Image-to-Image Text-Guided Generation | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/image_2_image_using_diffusers.ipynb) +| [stable_diffusion](./stable_diffusion) | [**Stable Diffusion**](https://stability.ai/blog/stable-diffusion-public-release) | Text-Guided Image Inpainting | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/huggingface/notebooks/blob/main/diffusers/in_painting_with_stable_diffusion_using_diffusers.ipynb) +| [stable_diffusion_2](./stable_diffusion_2) | [**Stable Diffusion 2**](https://stability.ai/blog/stable-diffusion-v2-release) | Text-to-Image Generation | +| [stable_diffusion_2](./stable_diffusion_2) | [**Stable Diffusion 2**](https://stability.ai/blog/stable-diffusion-v2-release) | Text-Guided Image Inpainting | +| [stable_diffusion_2](./stable_diffusion_2) | [**Stable Diffusion 2**](https://stability.ai/blog/stable-diffusion-v2-release) | Text-Guided Super Resolution Image-to-Image | +| [stable_diffusion_safe](./stable_diffusion_safe) | [**Safe Stable Diffusion**](https://arxiv.org/abs/2211.05105) | Text-Guided Generation | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ml-research/safe-latent-diffusion/blob/main/examples/Safe%20Latent%20Diffusion.ipynb) +| [stochastic_karras_ve](./stochastic_karras_ve) | [**Elucidating the Design Space of Diffusion-Based Generative Models**](https://arxiv.org/abs/2206.00364) | Unconditional Image Generation | +| [versatile_diffusion](./versatile_diffusion) | [Versatile Diffusion: Text, Images and Variations All in One Diffusion Model](https://arxiv.org/abs/2211.08332) | Text-to-Image Generation | +| [versatile_diffusion](./versatile_diffusion) | [Versatile Diffusion: Text, Images and Variations All in One Diffusion Model](https://arxiv.org/abs/2211.08332) | Image Variations Generation | +| [versatile_diffusion](./versatile_diffusion) | [Versatile Diffusion: Text, Images and Variations All in One Diffusion Model](https://arxiv.org/abs/2211.08332) | Dual Image and Text Guided Generation | +| [vq_diffusion](./vq_diffusion) | [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://arxiv.org/abs/2111.14822) | Text-to-Image Generation | **Note**: Pipelines are simple examples of how to play around with the diffusion systems as described in the corresponding papers. diff --git a/docs/source/api/schedulers.mdx b/docs/source/api/schedulers.mdx deleted file mode 100644 index 0e702f2069..0000000000 --- a/docs/source/api/schedulers.mdx +++ /dev/null @@ -1,183 +0,0 @@ - - -# Schedulers - -Diffusers contains multiple pre-built schedule functions for the diffusion process. - -## What is a scheduler? - -The schedule functions, denoted *Schedulers* in the library take in the output of a trained model, a sample which the diffusion process is iterating on, and a timestep to return a denoised sample. That's why schedulers may also be called *Samplers* in other diffusion models implementations. - -- Schedulers define the methodology for iteratively adding noise to an image or for updating a sample based on model outputs. - - adding noise in different manners represent the algorithmic processes to train a diffusion model by adding noise to images. - - for inference, the scheduler defines how to update a sample based on an output from a pretrained model. -- Schedulers are often defined by a *noise schedule* and an *update rule* to solve the differential equation solution. - -### Discrete versus continuous schedulers - -All schedulers take in a timestep to predict the updated version of the sample being diffused. -The timesteps dictate where in the diffusion process the step is, where data is generated by iterating forward in time and inference is executed by propagating backwards through timesteps. -Different algorithms use timesteps that both discrete (accepting `int` inputs), such as the [`DDPMScheduler`] or [`PNDMScheduler`], and continuous (accepting `float` inputs), such as the score-based schedulers [`ScoreSdeVeScheduler`] or [`ScoreSdeVpScheduler`]. - -## Designing Re-usable schedulers - -The core design principle between the schedule functions is to be model, system, and framework independent. -This allows for rapid experimentation and cleaner abstractions in the code, where the model prediction is separated from the sample update. -To this end, the design of schedulers is such that: - -- Schedulers can be used interchangeably between diffusion models in inference to find the preferred trade-off between speed and generation quality. -- Schedulers are currently by default in PyTorch, but are designed to be framework independent (partial Jax support currently exists). - - -## API - -The core API for any new scheduler must follow a limited structure. -- Schedulers should provide one or more `def step(...)` functions that should be called to update the generated sample iteratively. -- Schedulers should provide a `set_timesteps(...)` method that configures the parameters of a schedule function for a specific inference task. -- Schedulers should be framework-specific. - -The base class [`SchedulerMixin`] implements low level utilities used by multiple schedulers. - -### SchedulerMixin -[[autodoc]] SchedulerMixin - -### SchedulerOutput -The class [`SchedulerOutput`] contains the outputs from any schedulers `step(...)` call. - -[[autodoc]] schedulers.scheduling_utils.SchedulerOutput - -### Implemented Schedulers - -#### Denoising diffusion implicit models (DDIM) - -Original paper can be found here. - -[[autodoc]] DDIMScheduler - -#### Denoising diffusion probabilistic models (DDPM) - -Original paper can be found [here](https://arxiv.org/abs/2010.02502). - -[[autodoc]] DDPMScheduler - -#### Singlestep DPM-Solver - -Original paper can be found [here](https://arxiv.org/abs/2206.00927) and the [improved version](https://arxiv.org/abs/2211.01095). The original implementation can be found [here](https://github.com/LuChengTHU/dpm-solver). - -[[autodoc]] DPMSolverSinglestepScheduler - -#### Multistep DPM-Solver - -Original paper can be found [here](https://arxiv.org/abs/2206.00927) and the [improved version](https://arxiv.org/abs/2211.01095). The original implementation can be found [here](https://github.com/LuChengTHU/dpm-solver). - -[[autodoc]] DPMSolverMultistepScheduler - -#### Heun scheduler inspired by Karras et. al paper - -Algorithm 1 of [Karras et. al](https://arxiv.org/abs/2206.00364). -Scheduler ported from @crowsonkb's https://github.com/crowsonkb/k-diffusion library: - -All credit for making this scheduler work goes to [Katherine Crowson](https://github.com/crowsonkb/) - -[[autodoc]] HeunDiscreteScheduler - -#### DPM Discrete Scheduler inspired by Karras et. al paper - -Inspired by [Karras et. al](https://arxiv.org/abs/2206.00364). -Scheduler ported from @crowsonkb's https://github.com/crowsonkb/k-diffusion library: - -All credit for making this scheduler work goes to [Katherine Crowson](https://github.com/crowsonkb/) - -[[autodoc]] KDPM2DiscreteScheduler - -#### DPM Discrete Scheduler with ancestral sampling inspired by Karras et. al paper - -Inspired by [Karras et. al](https://arxiv.org/abs/2206.00364). -Scheduler ported from @crowsonkb's https://github.com/crowsonkb/k-diffusion library: - -All credit for making this scheduler work goes to [Katherine Crowson](https://github.com/crowsonkb/) - -[[autodoc]] KDPM2AncestralDiscreteScheduler - -#### Variance exploding, stochastic sampling from Karras et. al - -Original paper can be found [here](https://arxiv.org/abs/2006.11239). - -[[autodoc]] KarrasVeScheduler - -#### Linear multistep scheduler for discrete beta schedules - -Original implementation can be found [here](https://arxiv.org/abs/2206.00364). - -[[autodoc]] LMSDiscreteScheduler - -#### Pseudo numerical methods for diffusion models (PNDM) - -Original implementation can be found [here](https://github.com/crowsonkb/k-diffusion/blob/481677d114f6ea445aa009cf5bd7a9cdee909e47/k_diffusion/sampling.py#L181). - -[[autodoc]] PNDMScheduler - -#### variance exploding stochastic differential equation (VE-SDE) scheduler - -Original paper can be found [here](https://arxiv.org/abs/2011.13456). - -[[autodoc]] ScoreSdeVeScheduler - -#### improved pseudo numerical methods for diffusion models (iPNDM) - -Original implementation can be found [here](https://github.com/crowsonkb/v-diffusion-pytorch/blob/987f8985e38208345c1959b0ea767a625831cc9b/diffusion/sampling.py#L296). - -[[autodoc]] IPNDMScheduler - -#### variance preserving stochastic differential equation (VP-SDE) scheduler - -Original paper can be found [here](https://arxiv.org/abs/2011.13456). - - - -Score SDE-VP is under construction. - - - -[[autodoc]] schedulers.scheduling_sde_vp.ScoreSdeVpScheduler - -#### Euler scheduler - -Euler scheduler (Algorithm 2) from the paper [Elucidating the Design Space of Diffusion-Based Generative Models](https://arxiv.org/abs/2206.00364) by Karras et al. (2022). Based on the original [k-diffusion](https://github.com/crowsonkb/k-diffusion/blob/481677d114f6ea445aa009cf5bd7a9cdee909e47/k_diffusion/sampling.py#L51) implementation by Katherine Crowson. -Fast scheduler which often times generates good outputs with 20-30 steps. - -[[autodoc]] EulerDiscreteScheduler - - -#### Euler Ancestral scheduler - -Ancestral sampling with Euler method steps. Based on the original (k-diffusion)[https://github.com/crowsonkb/k-diffusion/blob/481677d114f6ea445aa009cf5bd7a9cdee909e47/k_diffusion/sampling.py#L72] implementation by Katherine Crowson. -Fast scheduler which often times generates good outputs with 20-30 steps. - -[[autodoc]] EulerAncestralDiscreteScheduler - - -#### VQDiffusionScheduler - -Original paper can be found [here](https://arxiv.org/abs/2111.14822) - -[[autodoc]] VQDiffusionScheduler - -#### RePaint scheduler - -DDPM-based inpainting scheduler for unsupervised inpainting with extreme masks. -Intended for use with [`RePaintPipeline`]. -Based on the paper [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2201.09865) -and the original implementation by Andreas Lugmayr et al.: https://github.com/andreas128/RePaint - -[[autodoc]] RePaintScheduler diff --git a/docs/source/api/schedulers/ddim.mdx b/docs/source/api/schedulers/ddim.mdx new file mode 100644 index 0000000000..f7de240c60 --- /dev/null +++ b/docs/source/api/schedulers/ddim.mdx @@ -0,0 +1,27 @@ + + +# Denoising diffusion implicit models (DDIM) + +## Overview + +[Denoising Diffusion Implicit Models](https://arxiv.org/abs/2010.02502) (DDIM) by Jiaming Song, Chenlin Meng and Stefano Ermon. + +The abstract of the paper is the following: + +Denoising diffusion probabilistic models (DDPMs) have achieved high quality image generation without adversarial training, yet they require simulating a Markov chain for many steps to produce a sample. To accelerate sampling, we present denoising diffusion implicit models (DDIMs), a more efficient class of iterative implicit probabilistic models with the same training procedure as DDPMs. In DDPMs, the generative process is defined as the reverse of a Markovian diffusion process. We construct a class of non-Markovian diffusion processes that lead to the same training objective, but whose reverse process can be much faster to sample from. We empirically demonstrate that DDIMs can produce high quality samples 10× to 50× faster in terms of wall-clock time compared to DDPMs, allow us to trade off computation for sample quality, and can perform semantically meaningful image interpolation directly in the latent space. + +The original codebase of this paper can be found here: [ermongroup/ddim](https://github.com/ermongroup/ddim). +For questions, feel free to contact the author on [tsong.me](https://tsong.me/). + +## DDIMScheduler +[[autodoc]] DDIMScheduler \ No newline at end of file diff --git a/docs/source/api/schedulers/ddpm.mdx b/docs/source/api/schedulers/ddpm.mdx new file mode 100644 index 0000000000..260e20aced --- /dev/null +++ b/docs/source/api/schedulers/ddpm.mdx @@ -0,0 +1,27 @@ + + +# Denoising diffusion probabilistic models (DDPM) + +## Overview + +[Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2006.11239) + (DDPM) by Jonathan Ho, Ajay Jain and Pieter Abbeel proposes the diffusion based model of the same name, but in the context of the 🤗 Diffusers library, DDPM refers to the discrete denoising scheduler from the paper as well as the pipeline. + +The abstract of the paper is the following: + +We present high quality image synthesis results using diffusion probabilistic models, a class of latent variable models inspired by considerations from nonequilibrium thermodynamics. Our best results are obtained by training on a weighted variational bound designed according to a novel connection between diffusion probabilistic models and denoising score matching with Langevin dynamics, and our models naturally admit a progressive lossy decompression scheme that can be interpreted as a generalization of autoregressive decoding. On the unconditional CIFAR10 dataset, we obtain an Inception score of 9.46 and a state-of-the-art FID score of 3.17. On 256x256 LSUN, we obtain sample quality similar to ProgressiveGAN. + +The original paper can be found [here](https://arxiv.org/abs/2010.02502). + +## DDPMScheduler +[[autodoc]] DDPMScheduler \ No newline at end of file diff --git a/docs/source/api/schedulers/dpm_discrete.mdx b/docs/source/api/schedulers/dpm_discrete.mdx new file mode 100644 index 0000000000..c848195e7e --- /dev/null +++ b/docs/source/api/schedulers/dpm_discrete.mdx @@ -0,0 +1,22 @@ + + +# DPM Discrete Scheduler inspired by Karras et. al paper + +## Overview + +Inspired by [Karras et. al](https://arxiv.org/abs/2206.00364). Scheduler ported from @crowsonkb's https://github.com/crowsonkb/k-diffusion library: + +All credit for making this scheduler work goes to [Katherine Crowson](https://github.com/crowsonkb/) + +## KDPM2DiscreteScheduler +[[autodoc]] KDPM2DiscreteScheduler \ No newline at end of file diff --git a/docs/source/api/schedulers/dpm_discrete_ancestral.mdx b/docs/source/api/schedulers/dpm_discrete_ancestral.mdx new file mode 100644 index 0000000000..5fdc651dc1 --- /dev/null +++ b/docs/source/api/schedulers/dpm_discrete_ancestral.mdx @@ -0,0 +1,22 @@ + + +# DPM Discrete Scheduler with ancestral sampling inspired by Karras et. al paper + +## Overview + +Inspired by [Karras et. al](https://arxiv.org/abs/2206.00364). Scheduler ported from @crowsonkb's https://github.com/crowsonkb/k-diffusion library: + +All credit for making this scheduler work goes to [Katherine Crowson](https://github.com/crowsonkb/) + +## KDPM2AncestralDiscreteScheduler +[[autodoc]] KDPM2AncestralDiscreteScheduler \ No newline at end of file diff --git a/docs/source/api/schedulers/euler.mdx b/docs/source/api/schedulers/euler.mdx new file mode 100644 index 0000000000..718e7031dd --- /dev/null +++ b/docs/source/api/schedulers/euler.mdx @@ -0,0 +1,21 @@ + + +# Euler scheduler + +## Overview + +Euler scheduler (Algorithm 2) from the paper [Elucidating the Design Space of Diffusion-Based Generative Models](https://arxiv.org/abs/2206.00364) by Karras et al. (2022). Based on the original [k-diffusion](https://github.com/crowsonkb/k-diffusion/blob/481677d114f6ea445aa009cf5bd7a9cdee909e47/k_diffusion/sampling.py#L51) implementation by Katherine Crowson. +Fast scheduler which often times generates good outputs with 20-30 steps. + +## EulerDiscreteScheduler +[[autodoc]] EulerDiscreteScheduler \ No newline at end of file diff --git a/docs/source/api/schedulers/euler_ancestral.mdx b/docs/source/api/schedulers/euler_ancestral.mdx new file mode 100644 index 0000000000..3f736fa545 --- /dev/null +++ b/docs/source/api/schedulers/euler_ancestral.mdx @@ -0,0 +1,21 @@ + + +# Euler Ancestral scheduler + +## Overview + +Ancestral sampling with Euler method steps. Based on the original (k-diffusion)[https://github.com/crowsonkb/k-diffusion/blob/481677d114f6ea445aa009cf5bd7a9cdee909e47/k_diffusion/sampling.py#L72] implementation by Katherine Crowson. +Fast scheduler which often times generates good outputs with 20-30 steps. + +## EulerAncestralDiscreteScheduler +[[autodoc]] EulerAncestralDiscreteScheduler \ No newline at end of file diff --git a/docs/source/api/schedulers/heun.mdx b/docs/source/api/schedulers/heun.mdx new file mode 100644 index 0000000000..5539be881d --- /dev/null +++ b/docs/source/api/schedulers/heun.mdx @@ -0,0 +1,23 @@ + + +# Heun scheduler inspired by Karras et. al paper + +## Overview + +Algorithm 1 of [Karras et. al](https://arxiv.org/abs/2206.00364). +Scheduler ported from @crowsonkb's https://github.com/crowsonkb/k-diffusion library: + +All credit for making this scheduler work goes to [Katherine Crowson](https://github.com/crowsonkb/) + +## HeunDiscreteScheduler +[[autodoc]] HeunDiscreteScheduler \ No newline at end of file diff --git a/docs/source/api/schedulers/ipndm.mdx b/docs/source/api/schedulers/ipndm.mdx new file mode 100644 index 0000000000..069928cbbc --- /dev/null +++ b/docs/source/api/schedulers/ipndm.mdx @@ -0,0 +1,20 @@ + + +# improved pseudo numerical methods for diffusion models (iPNDM) + +## Overview + +Original implementation can be found [here](https://github.com/crowsonkb/v-diffusion-pytorch/blob/987f8985e38208345c1959b0ea767a625831cc9b/diffusion/sampling.py#L296). + +## IPNDMScheduler +[[autodoc]] IPNDMScheduler \ No newline at end of file diff --git a/docs/source/api/schedulers/lms_discrete.mdx b/docs/source/api/schedulers/lms_discrete.mdx new file mode 100644 index 0000000000..d7fa878126 --- /dev/null +++ b/docs/source/api/schedulers/lms_discrete.mdx @@ -0,0 +1,20 @@ + + +# Linear multistep scheduler for discrete beta schedules + +## Overview + +Original implementation can be found [here](https://arxiv.org/abs/2206.00364). + +## LMSDiscreteScheduler +[[autodoc]] LMSDiscreteScheduler \ No newline at end of file diff --git a/docs/source/api/schedulers/multistep_dpm_solver.mdx b/docs/source/api/schedulers/multistep_dpm_solver.mdx new file mode 100644 index 0000000000..e3f725fe4a --- /dev/null +++ b/docs/source/api/schedulers/multistep_dpm_solver.mdx @@ -0,0 +1,20 @@ + + +# Multistep DPM-Solver + +## Overview + +Original paper can be found [here](https://arxiv.org/abs/2206.00927) and the [improved version](https://arxiv.org/abs/2211.01095). The original implementation can be found [here](https://github.com/LuChengTHU/dpm-solver). + +## DPMSolverMultistepScheduler +[[autodoc]] DPMSolverMultistepScheduler \ No newline at end of file diff --git a/docs/source/api/schedulers/overview.mdx b/docs/source/api/schedulers/overview.mdx new file mode 100644 index 0000000000..7e139d152b --- /dev/null +++ b/docs/source/api/schedulers/overview.mdx @@ -0,0 +1,83 @@ + + +# Schedulers + +Diffusers contains multiple pre-built schedule functions for the diffusion process. + +## What is a scheduler? + +The schedule functions, denoted *Schedulers* in the library take in the output of a trained model, a sample which the diffusion process is iterating on, and a timestep to return a denoised sample. That's why schedulers may also be called *Samplers* in other diffusion models implementations. + +- Schedulers define the methodology for iteratively adding noise to an image or for updating a sample based on model outputs. + - adding noise in different manners represent the algorithmic processes to train a diffusion model by adding noise to images. + - for inference, the scheduler defines how to update a sample based on an output from a pretrained model. +- Schedulers are often defined by a *noise schedule* and an *update rule* to solve the differential equation solution. + +### Discrete versus continuous schedulers + +All schedulers take in a timestep to predict the updated version of the sample being diffused. +The timesteps dictate where in the diffusion process the step is, where data is generated by iterating forward in time and inference is executed by propagating backwards through timesteps. +Different algorithms use timesteps that can be discrete (accepting `int` inputs), such as the [`DDPMScheduler`] or [`PNDMScheduler`], or continuous (accepting `float` inputs), such as the score-based schedulers [`ScoreSdeVeScheduler`] or [`ScoreSdeVpScheduler`]. + +## Designing Re-usable schedulers + +The core design principle between the schedule functions is to be model, system, and framework independent. +This allows for rapid experimentation and cleaner abstractions in the code, where the model prediction is separated from the sample update. +To this end, the design of schedulers is such that: + +- Schedulers can be used interchangeably between diffusion models in inference to find the preferred trade-off between speed and generation quality. +- Schedulers are currently by default in PyTorch, but are designed to be framework independent (partial Jax support currently exists). + +## Schedulers Summary + +The following table summarizes all officially supported schedulers, their corresponding paper + + +| Scheduler | Paper | +|---|---| +| [ddim](./ddim) | [**Denoising Diffusion Implicit Models**](https://arxiv.org/abs/2010.02502) | +| [ddpm](./ddpm) | [**Denoising Diffusion Probabilistic Models**](https://arxiv.org/abs/2006.11239) | +| [singlestep_dpm_solver](./singlestep_dpm_solver) | [**Singlestep DPM-Solver**](https://arxiv.org/abs/2206.00927) | +| [multistep_dpm_solver](./multistep_dpm_solver) | [**Multistep DPM-Solver**](https://arxiv.org/abs/2206.00927) | +| [heun](./heun) | [**Heun scheduler inspired by Karras et. al paper**](https://arxiv.org/abs/2206.00364) | +| [dpm_discrete](./dpm_discrete) | [**DPM Discrete Scheduler inspired by Karras et. al paper**](https://arxiv.org/abs/2206.00364) | +| [dpm_discrete_ancestral](./dpm_discrete_ancestral) | [**DPM Discrete Scheduler with ancestral sampling inspired by Karras et. al paper**](https://arxiv.org/abs/2206.00364) | +| [stochastic_karras_ve](./stochastic_karras_ve) | [**Variance exploding, stochastic sampling from Karras et. al**](https://arxiv.org/abs/2206.00364) | +| [lms_discrete](./lms_discrete) | [**Linear multistep scheduler for discrete beta schedules**](https://arxiv.org/abs/2206.00364) | +| [pndm](./pndm) | [**Pseudo numerical methods for diffusion models (PNDM)**](https://github.com/crowsonkb/k-diffusion/blob/481677d114f6ea445aa009cf5bd7a9cdee909e47/k_diffusion/sampling.py#L181) | +| [score_sde_ve](./score_sde_ve) | [**variance exploding stochastic differential equation (VE-SDE) scheduler**](https://arxiv.org/abs/2011.13456) | +| [ipndm](./ipndm) | [**improved pseudo numerical methods for diffusion models (iPNDM)**](https://github.com/crowsonkb/v-diffusion-pytorch/blob/987f8985e38208345c1959b0ea767a625831cc9b/diffusion/sampling.py#L296) | +| [score_sde_vp](./score_sde_vp) | [**Variance preserving stochastic differential equation (VP-SDE) scheduler**](https://arxiv.org/abs/2011.13456) | +| [euler](./euler) | [**Euler scheduler**](https://arxiv.org/abs/2206.00364) | +| [euler_ancestral](./euler_ancestral) | [**Euler Ancestral scheduler**](https://github.com/crowsonkb/k-diffusion/blob/481677d114f6ea445aa009cf5bd7a9cdee909e47/k_diffusion/sampling.py#L72) | +| [vq_diffusion](./vq_diffusion) | [**VQDiffusionScheduler**](https://arxiv.org/abs/2111.14822) | +| [repaint](./repaint) | [**RePaint scheduler**](https://arxiv.org/abs/2201.09865) | + +## API + +The core API for any new scheduler must follow a limited structure. +- Schedulers should provide one or more `def step(...)` functions that should be called to update the generated sample iteratively. +- Schedulers should provide a `set_timesteps(...)` method that configures the parameters of a schedule function for a specific inference task. +- Schedulers should be framework-specific. + +The base class [`SchedulerMixin`] implements low level utilities used by multiple schedulers. + +### SchedulerMixin +[[autodoc]] SchedulerMixin + +### SchedulerOutput +The class [`SchedulerOutput`] contains the outputs from any schedulers `step(...)` call. + +[[autodoc]] schedulers.scheduling_utils.SchedulerOutput + + diff --git a/docs/source/api/schedulers/pndm.mdx b/docs/source/api/schedulers/pndm.mdx new file mode 100644 index 0000000000..ed6ae082c0 --- /dev/null +++ b/docs/source/api/schedulers/pndm.mdx @@ -0,0 +1,20 @@ + + +# Pseudo numerical methods for diffusion models (PNDM) + +## Overview + +Original implementation can be found [here](https://github.com/crowsonkb/k-diffusion/blob/481677d114f6ea445aa009cf5bd7a9cdee909e47/k_diffusion/sampling.py#L181). + +## PNDMScheduler +[[autodoc]] PNDMScheduler \ No newline at end of file diff --git a/docs/source/api/schedulers/repaint.mdx b/docs/source/api/schedulers/repaint.mdx new file mode 100644 index 0000000000..717a58372e --- /dev/null +++ b/docs/source/api/schedulers/repaint.mdx @@ -0,0 +1,23 @@ + + +# RePaint scheduler + +## Overview + +DDPM-based inpainting scheduler for unsupervised inpainting with extreme masks. +Intended for use with [`RePaintPipeline`]. +Based on the paper [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://arxiv.org/abs/2201.09865) +and the original implementation by Andreas Lugmayr et al.: https://github.com/andreas128/RePaint + +## RePaintScheduler +[[autodoc]] RePaintScheduler \ No newline at end of file diff --git a/docs/source/api/schedulers/score_sde_ve.mdx b/docs/source/api/schedulers/score_sde_ve.mdx new file mode 100644 index 0000000000..b8fb932576 --- /dev/null +++ b/docs/source/api/schedulers/score_sde_ve.mdx @@ -0,0 +1,20 @@ + + +# variance exploding stochastic differential equation (VE-SDE) scheduler + +## Overview + +Original paper can be found [here](https://arxiv.org/abs/2011.13456). + +## ScoreSdeVeScheduler +[[autodoc]] ScoreSdeVeScheduler \ No newline at end of file diff --git a/docs/source/api/schedulers/score_sde_vp.mdx b/docs/source/api/schedulers/score_sde_vp.mdx new file mode 100644 index 0000000000..1f45439ab7 --- /dev/null +++ b/docs/source/api/schedulers/score_sde_vp.mdx @@ -0,0 +1,26 @@ + + +# Variance preserving stochastic differential equation (VP-SDE) scheduler + +## Overview + +Original paper can be found [here](https://arxiv.org/abs/2011.13456). + + + +Score SDE-VP is under construction. + + + +## ScoreSdeVpScheduler +[[autodoc]] schedulers.scheduling_sde_vp.ScoreSdeVpScheduler \ No newline at end of file diff --git a/docs/source/api/schedulers/singlestep_dpm_solver.mdx b/docs/source/api/schedulers/singlestep_dpm_solver.mdx new file mode 100644 index 0000000000..44231c2d97 --- /dev/null +++ b/docs/source/api/schedulers/singlestep_dpm_solver.mdx @@ -0,0 +1,20 @@ + + +# Singlestep DPM-Solver + +## Overview + +Original paper can be found [here](https://arxiv.org/abs/2206.00927) and the [improved version](https://arxiv.org/abs/2211.01095). The original implementation can be found [here](https://github.com/LuChengTHU/dpm-solver). + +## DPMSolverSinglestepScheduler +[[autodoc]] DPMSolverSinglestepScheduler \ No newline at end of file diff --git a/docs/source/api/schedulers/stochastic_karras_ve.mdx b/docs/source/api/schedulers/stochastic_karras_ve.mdx new file mode 100644 index 0000000000..95437f85da --- /dev/null +++ b/docs/source/api/schedulers/stochastic_karras_ve.mdx @@ -0,0 +1,20 @@ + + +# Variance exploding, stochastic sampling from Karras et. al + +## Overview + +Original paper can be found [here](https://arxiv.org/abs/2206.00364). + +## KarrasVeScheduler +[[autodoc]] KarrasVeScheduler \ No newline at end of file diff --git a/docs/source/api/schedulers/vq_diffusion.mdx b/docs/source/api/schedulers/vq_diffusion.mdx new file mode 100644 index 0000000000..921cc17d43 --- /dev/null +++ b/docs/source/api/schedulers/vq_diffusion.mdx @@ -0,0 +1,20 @@ + + +# VQDiffusionScheduler + +## Overview + +Original paper can be found [here](https://arxiv.org/abs/2111.14822) + +## VQDiffusionScheduler +[[autodoc]] VQDiffusionScheduler \ No newline at end of file diff --git a/docs/source/index.mdx b/docs/source/index.mdx index c293d83724..b4ab609158 100644 --- a/docs/source/index.mdx +++ b/docs/source/index.mdx @@ -23,7 +23,7 @@ specific language governing permissions and limitations under the License. More precisely, 🤗 Diffusers offers: - State-of-the-art diffusion pipelines that can be run in inference with just a couple of lines of code (see [**Using Diffusers**](./using-diffusers/conditional_image_generation)) or have a look at [**Pipelines**](#pipelines) to get an overview of all supported pipelines and their corresponding papers. -- Various noise schedulers that can be used interchangeably for the preferred speed vs. quality trade-off in inference. For more information see [**Schedulers**](./api/schedulers). +- Various noise schedulers that can be used interchangeably for the preferred speed vs. quality trade-off in inference. For more information see [**Schedulers**](./api/schedulers/overview). - Multiple types of models, such as UNet, can be used as building blocks in an end-to-end diffusion system. See [**Models**](./api/models) for more details - Training examples to show how to train the most popular diffusion model tasks. For more information see [**Training**](./training/overview). diff --git a/docs/source/quicktour.mdx b/docs/source/quicktour.mdx index d71f66d0bc..05bf121273 100644 --- a/docs/source/quicktour.mdx +++ b/docs/source/quicktour.mdx @@ -97,7 +97,7 @@ Running the pipeline is then identical to the code above as it's the same model >>> image.save("image_of_squirrel_painting.png") ``` -Diffusion systems can be used with multiple different [schedulers](./api/schedulers) each with their +Diffusion systems can be used with multiple different [schedulers](./api/schedulers/overview) each with their pros and cons. By default, Stable Diffusion runs with [`PNDMScheduler`], but it's very simple to use a different scheduler. *E.g.* if you would instead like to use the [`EulerDiscreteScheduler`] scheduler, you could use it as follows: diff --git a/docs/source/using-diffusers/schedulers.mdx b/docs/source/using-diffusers/schedulers.mdx index 87ff789747..21250b0550 100644 --- a/docs/source/using-diffusers/schedulers.mdx +++ b/docs/source/using-diffusers/schedulers.mdx @@ -13,7 +13,7 @@ specific language governing permissions and limitations under the License. # Schedulers Diffusion pipelines are inherently a collection of diffusion models and schedulers that are partly independent from each other. This means that one is able to switch out parts of the pipeline to better customize -a pipeline to one's use case. The best example of this are the [Schedulers](../api/schedulers.mdx). +a pipeline to one's use case. The best example of this are the [Schedulers](../api/schedulers/overview.mdx). Whereas diffusion models usually simply define the forward pass from noise to a less noisy sample, schedulers define the whole denoising process, *i.e.*: diff --git a/src/diffusers/schedulers/README.md b/src/diffusers/schedulers/README.md index 9494e357fd..31ad27793e 100644 --- a/src/diffusers/schedulers/README.md +++ b/src/diffusers/schedulers/README.md @@ -1,3 +1,3 @@ # Schedulers -For more information on the schedulers, please refer to the [docs](https://huggingface.co/docs/diffusers/api/schedulers). \ No newline at end of file +For more information on the schedulers, please refer to the [docs](https://huggingface.co/docs/diffusers/api/schedulers/overview). \ No newline at end of file