mirror of
https://github.com/huggingface/diffusers.git
synced 2026-01-27 17:22:53 +03:00
Add pipeline_stable_diffusion_xl_attentive_eraser (#10579)
* add pipeline_stable_diffusion_xl_attentive_eraser * add pipeline_stable_diffusion_xl_attentive_eraser_make_style * make style and add example output * update Docs Co-authored-by: Other Contributor <a457435687@126.com> * add Oral Co-authored-by: Other Contributor <a457435687@126.com> * update_review Co-authored-by: Other Contributor <a457435687@126.com> * update_review_ms Co-authored-by: Other Contributor <a457435687@126.com> --------- Co-authored-by: Other Contributor <a457435687@126.com>
This commit is contained in:
Wenhao Sun
GitHub
92
examples/community/README.md
Executable file → Normal file
92
examples/community/README.md
Executable file → Normal file
@@ -77,6 +77,7 @@ Please also check out our [Community Scripts](https://github.com/huggingface/dif
|
||||
PIXART-α Controlnet pipeline | Implementation of the controlnet model for pixart alpha and its diffusers pipeline | [PIXART-α Controlnet pipeline](#pixart-α-controlnet-pipeline) | - | [Raul Ciotescu](https://github.com/raulc0399/) |
|
||||
| HunyuanDiT Differential Diffusion Pipeline | Applies [Differential Diffusion](https://github.com/exx8/differential-diffusion) to [HunyuanDiT](https://github.com/huggingface/diffusers/pull/8240). | [HunyuanDiT with Differential Diffusion](#hunyuandit-with-differential-diffusion) | [](https://colab.research.google.com/drive/1v44a5fpzyr4Ffr4v2XBQ7BajzG874N4P?usp=sharing) | [Monjoy Choudhury](https://github.com/MnCSSJ4x) |
|
||||
| [🪆Matryoshka Diffusion Models](https://huggingface.co/papers/2310.15111) | A diffusion process that denoises inputs at multiple resolutions jointly and uses a NestedUNet architecture where features and parameters for small scale inputs are nested within those of the large scales. See [original codebase](https://github.com/apple/ml-mdm). | [🪆Matryoshka Diffusion Models](#matryoshka-diffusion-models) | [](https://huggingface.co/spaces/pcuenq/mdm) [](https://colab.research.google.com/gist/tolgacangoz/1f54875fc7aeaabcf284ebde64820966/matryoshka_hf.ipynb) | [M. Tolga Cangöz](https://github.com/tolgacangoz) |
|
||||
| Stable Diffusion XL Attentive Eraser Pipeline |[[AAAI2025 Oral] Attentive Eraser](https://github.com/Anonym0u3/AttentiveEraser) is a novel tuning-free method that enhances object removal capabilities in pre-trained diffusion models.|[Stable Diffusion XL Attentive Eraser Pipeline](#stable-diffusion-xl-attentive-eraser-pipeline)|-|[Wenhao Sun](https://github.com/Anonym0u3) and [Benlei Cui](https://github.com/Benny079)|
|
||||
|
||||
To load a custom pipeline you just need to pass the `custom_pipeline` argument to `DiffusionPipeline`, as one of the files in `diffusers/examples/community`. Feel free to send a PR with your own pipelines, we will merge them quickly.
|
||||
|
||||
@@ -4585,8 +4586,8 @@ image = pipe(
|
||||
```
|
||||
|
||||
|  |  |  |
|
||||
| ------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
|
||||
| Gradient | Input | Output |
|
||||
| -------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
|
||||
| Gradient | Input | Output |
|
||||
|
||||
A colab notebook demonstrating all results can be found [here](https://colab.research.google.com/drive/1v44a5fpzyr4Ffr4v2XBQ7BajzG874N4P?usp=sharing). Depth Maps have also been added in the same colab.
|
||||
|
||||
@@ -4634,6 +4635,93 @@ make_image_grid(image, rows=1, cols=len(image))
|
||||
# 50+, 100+, and 250+ num_inference_steps are recommended for nesting levels 0, 1, and 2 respectively.
|
||||
```
|
||||
|
||||
### Stable Diffusion XL Attentive Eraser Pipeline
|
||||
<img src="https://raw.githubusercontent.com/Anonym0u3/Images/refs/heads/main/fenmian.png" width="600" />
|
||||
|
||||
**Stable Diffusion XL Attentive Eraser Pipeline** is an advanced object removal pipeline that leverages SDXL for precise content suppression and seamless region completion. This pipeline uses **self-attention redirection guidance** to modify the model’s self-attention mechanism, allowing for effective removal and inpainting across various levels of mask precision, including semantic segmentation masks, bounding boxes, and hand-drawn masks. If you are interested in more detailed information and have any questions, please refer to the [paper](https://arxiv.org/abs/2412.12974) and [official implementation](https://github.com/Anonym0u3/AttentiveEraser).
|
||||
|
||||
#### Key features
|
||||
|
||||
- **Tuning-Free**: No additional training is required, making it easy to integrate and use.
|
||||
- **Flexible Mask Support**: Works with different types of masks for targeted object removal.
|
||||
- **High-Quality Results**: Utilizes the inherent generative power of diffusion models for realistic content completion.
|
||||
|
||||
#### Usage example
|
||||
To use the Stable Diffusion XL Attentive Eraser Pipeline, you can initialize it as follows:
|
||||
```py
|
||||
import torch
|
||||
from diffusers import DDIMScheduler, DiffusionPipeline
|
||||
from diffusers.utils import load_image
|
||||
import torch.nn.functional as F
|
||||
from torchvision.transforms.functional import to_tensor, gaussian_blur
|
||||
|
||||
dtype = torch.float16
|
||||
device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
|
||||
|
||||
scheduler = DDIMScheduler(beta_start=0.00085, beta_end=0.012, beta_schedule="scaled_linear", clip_sample=False, set_alpha_to_one=False)
|
||||
pipeline = DiffusionPipeline.from_pretrained(
|
||||
"stabilityai/stable-diffusion-xl-base-1.0",
|
||||
custom_pipeline="pipeline_stable_diffusion_xl_attentive_eraser",
|
||||
scheduler=scheduler,
|
||||
variant="fp16",
|
||||
use_safetensors=True,
|
||||
torch_dtype=dtype,
|
||||
).to(device)
|
||||
|
||||
|
||||
def preprocess_image(image_path, device):
|
||||
image = to_tensor((load_image(image_path)))
|
||||
image = image.unsqueeze_(0).float() * 2 - 1 # [0,1] --> [-1,1]
|
||||
if image.shape[1] != 3:
|
||||
image = image.expand(-1, 3, -1, -1)
|
||||
image = F.interpolate(image, (1024, 1024))
|
||||
image = image.to(dtype).to(device)
|
||||
return image
|
||||
|
||||
def preprocess_mask(mask_path, device):
|
||||
mask = to_tensor((load_image(mask_path, convert_method=lambda img: img.convert('L'))))
|
||||
mask = mask.unsqueeze_(0).float() # 0 or 1
|
||||
mask = F.interpolate(mask, (1024, 1024))
|
||||
mask = gaussian_blur(mask, kernel_size=(77, 77))
|
||||
mask[mask < 0.1] = 0
|
||||
mask[mask >= 0.1] = 1
|
||||
mask = mask.to(dtype).to(device)
|
||||
return mask
|
||||
|
||||
prompt = "" # Set prompt to null
|
||||
seed=123
|
||||
generator = torch.Generator(device=device).manual_seed(seed)
|
||||
source_image_path = "https://raw.githubusercontent.com/Anonym0u3/Images/refs/heads/main/an1024.png"
|
||||
mask_path = "https://raw.githubusercontent.com/Anonym0u3/Images/refs/heads/main/an1024_mask.png"
|
||||
source_image = preprocess_image(source_image_path, device)
|
||||
mask = preprocess_mask(mask_path, device)
|
||||
|
||||
image = pipeline(
|
||||
prompt=prompt,
|
||||
image=source_image,
|
||||
mask_image=mask,
|
||||
height=1024,
|
||||
width=1024,
|
||||
AAS=True, # enable AAS
|
||||
strength=0.8, # inpainting strength
|
||||
rm_guidance_scale=9, # removal guidance scale
|
||||
ss_steps = 9, # similarity suppression steps
|
||||
ss_scale = 0.3, # similarity suppression scale
|
||||
AAS_start_step=0, # AAS start step
|
||||
AAS_start_layer=34, # AAS start layer
|
||||
AAS_end_layer=70, # AAS end layer
|
||||
num_inference_steps=50, # number of inference steps # AAS_end_step = int(strength*num_inference_steps)
|
||||
generator=generator,
|
||||
guidance_scale=1,
|
||||
).images[0]
|
||||
image.save('./removed_img.png')
|
||||
print("Object removal completed")
|
||||
```
|
||||
|
||||
| Source Image | Mask | Output |
|
||||
| ---------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |
|
||||
|  |  |  |
|
||||
|
||||
# Perturbed-Attention Guidance
|
||||
|
||||
[Project](https://ku-cvlab.github.io/Perturbed-Attention-Guidance/) / [arXiv](https://arxiv.org/abs/2403.17377) / [GitHub](https://github.com/KU-CVLAB/Perturbed-Attention-Guidance)
|
||||
|
||||
2318
examples/community/pipeline_stable_diffusion_xl_attentive_eraser.py
Normal file
2318
examples/community/pipeline_stable_diffusion_xl_attentive_eraser.py
Normal file
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user