Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion .ai/skills/model-integration/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,22 @@ A pipeline in Diffusers (be it standard or modular) will have multiple component
- [ ] Add a LoRA mixin if applicable
- [ ] Register in the relevant `__init__.py` files (lazy imports)
- [ ] Pipeline-level tests (see **Testing**)
- [ ] **Docs** — see **File structure**
- [ ] **Docs** — see **File structure** and **Docs device examples**
- [ ] **Style** — `make style` and `make quality`

## Docs device examples

When adding or updating examples in `docs/source/en/`, avoid hardcoding CUDA as the default execution device. Prefer device-agnostic snippets so the docs work on CUDA, XPU, MPS, and other supported accelerators:

```python
from diffusers.utils.torch_utils import get_device

device = get_device()
pipe.to(device)
```

Use `device` for `.to(...)`, `device_map`, `torch.Generator(device=...)`, tensors, and component placement where applicable. Only keep explicit CUDA APIs or strings when the guide is truly CUDA/NVIDIA-specific, for example CUDA kernel documentation, NCCL multi-GPU examples, or benchmarks that explicitly target NVIDIA GPUs. In those cases, the surrounding text should make the CUDA-only requirement clear.

## File structure

A new model PR roughly lands these files (the contents of `pipelines/<model>/` and `modular_pipelines/<model>/` live in their guides):
Expand Down
2 changes: 2 additions & 0 deletions .ai/skills/self-review/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ files, scope to your own commits: `git log main..HEAD --oneline`, then
it and review against it; don't rely on a remembered copy. For the areas you
touched, also read `.ai/models.md`, `.ai/pipelines.md`, or `.ai/modular.md`.

When reviewing docs in `docs/source/en/`, check that examples are device agnostic by default. Flag newly added `.to("cuda")`, `device="cuda"`, `device_map="cuda"`, `torch.Generator("cuda")`, or similar hardcoded CUDA usage unless the page is explicitly CUDA/NVIDIA-specific, such as CUDA kernels, NCCL/distributed GPU examples, or NVIDIA GPU benchmarks.

## 3. Report

- **Blocking issues** — numbered. Each: title → explanation → `file.py:line` →
Expand Down
9 changes: 6 additions & 3 deletions docs/source/en/advanced_inference/outpaint.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,9 @@ from diffusers import (
StableDiffusionXLControlNetPipeline,
StableDiffusionXLInpaintPipeline,
)
from diffusers.utils.torch_utils import get_device

device = get_device()

def scale_and_paste(original_image):
aspect_ratio = original_image.width / original_image.height
Expand Down Expand Up @@ -117,10 +120,10 @@ controlnets = [
"diffusers/controlnet-zoe-depth-sdxl-1.0", dtype=torch.float16
),
]
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", dtype=torch.float16).to("cuda")
vae = AutoencoderKL.from_pretrained("madebyollin/sdxl-vae-fp16-fix", dtype=torch.float16).to(device)
pipeline = StableDiffusionXLControlNetPipeline.from_pretrained(
"SG161222/RealVisXL_V4.0", dtype=torch.float16, variant="fp16", controlnet=controlnets, vae=vae
).to("cuda")
).to(device)

def generate_image(prompt, negative_prompt, inpaint_image, zoe_image, seed: int = None):
if seed is None:
Expand Down Expand Up @@ -176,7 +179,7 @@ pipeline = StableDiffusionXLInpaintPipeline.from_pretrained(
dtype=torch.float16,
variant="fp16",
vae=vae,
).to("cuda")
).to(device)
```

Prepare a mask for the final outpainted image. To create a more natural transition between the original image and the outpainted background, blur the mask to help it blend better.
Expand Down
5 changes: 4 additions & 1 deletion docs/source/en/api/models/allegro_transformer3d.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,11 @@ The model can be loaded with the following code snippet.

```python
from diffusers import AllegroTransformer3DModel
from diffusers.utils.torch_utils import get_device

transformer = AllegroTransformer3DModel.from_pretrained("rhymes-ai/Allegro", subfolder="transformer", dtype=torch.bfloat16).to("cuda")

device = get_device()
transformer = AllegroTransformer3DModel.from_pretrained("rhymes-ai/Allegro", subfolder="transformer", dtype=torch.bfloat16).to(device)
```

## AllegroTransformer3DModel
Expand Down
5 changes: 4 additions & 1 deletion docs/source/en/api/models/asymmetricautoencoderkl.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,12 @@ Evaluation results can be found in section 4.1 of the original paper.

```python
from diffusers import AsymmetricAutoencoderKL, StableDiffusionInpaintPipeline
from diffusers.utils.torch_utils import get_device
from diffusers.utils import load_image, make_image_grid



device = get_device()
prompt = "a photo of a person with beard"
img_url = "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/repaint/celeba_hq_256.png"
mask_url = "https://huggingface.co/datasets/hf-internal-testing/diffusers-images/resolve/main/repaint/mask_256.png"
Expand All @@ -41,7 +44,7 @@ mask_image = load_image(mask_url).resize((512, 512))

pipe = StableDiffusionInpaintPipeline.from_pretrained("stable-diffusion-v1-5/stable-diffusion-inpainting")
pipe.vae = AsymmetricAutoencoderKL.from_pretrained("cross-attention/asymmetric-autoencoder-kl-x-1-5")
pipe.to("cuda")
pipe.to(device)

image = pipe(prompt=prompt, image=original_image, mask_image=mask_image).images[0]
make_image_grid([original_image, mask_image, image], rows=1, cols=3)
Expand Down
5 changes: 4 additions & 1 deletion docs/source/en/api/models/autoencoder_dc.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,11 @@ Load a model in Diffusers format with [`~ModelMixin.from_pretrained`].

```python
from diffusers import AutoencoderDC
from diffusers.utils.torch_utils import get_device

ae = AutoencoderDC.from_pretrained("mit-han-lab/dc-ae-f32c32-sana-1.0-diffusers", dtype=torch.float32).to("cuda")

device = get_device()
ae = AutoencoderDC.from_pretrained("mit-han-lab/dc-ae-f32c32-sana-1.0-diffusers", dtype=torch.float32).to(device)
```

## Load a model in Diffusers via `from_single_file`
Expand Down
16 changes: 12 additions & 4 deletions docs/source/en/api/models/autoencoder_rae.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,27 +31,32 @@ The following RAE models are released and supported in Diffusers:

```python
from diffusers import AutoencoderRAE
from diffusers.utils.torch_utils import get_device

device = get_device()
model = AutoencoderRAE.from_pretrained(
"nyu-visionx/RAE-dinov2-wReg-base-ViTXL-n08"
).to("cuda").eval()
).to(device).eval()
```

## Encoding and decoding a real image

```python
import torch
from diffusers.utils.torch_utils import get_device
from diffusers import AutoencoderRAE
from diffusers.utils import load_image
from torchvision.transforms.functional import to_tensor, to_pil_image


device = get_device()
model = AutoencoderRAE.from_pretrained(
"nyu-visionx/RAE-dinov2-wReg-base-ViTXL-n08"
).to("cuda").eval()
).to(device).eval()

image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/cat.png")
image = image.convert("RGB").resize((224, 224))
x = to_tensor(image).unsqueeze(0).to("cuda") # (1, 3, 224, 224), values in [0, 1]
x = to_tensor(image).unsqueeze(0).to(device) # (1, 3, 224, 224), values in [0, 1]

with torch.no_grad():
latents = model.encode(x).latent # (1, 768, 16, 16)
Expand All @@ -66,9 +71,12 @@ recon_image.save("recon.png")
Some pretrained checkpoints include per-channel `latents_mean` and `latents_std` statistics for normalizing the latent space. When present, `encode` and `decode` automatically apply the normalization and denormalization, respectively.

```python
from diffusers.utils.torch_utils import get_device

device = get_device()
model = AutoencoderRAE.from_pretrained(
"nyu-visionx/RAE-dinov2-wReg-base-ViTXL-n08"
).to("cuda").eval()
).to(device).eval()

# Latent normalization is handled automatically inside encode/decode
# when the checkpoint config includes latents_mean/latents_std.
Expand Down
9 changes: 7 additions & 2 deletions docs/source/en/api/models/autoencoder_tiny.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,13 +18,16 @@ To use with Stable Diffusion v-2.1:

```python
import torch
from diffusers.utils.torch_utils import get_device
from diffusers import DiffusionPipeline, AutoencoderTiny


device = get_device()
pipe = DiffusionPipeline.from_pretrained(
"stabilityai/stable-diffusion-2-1-base", dtype=torch.float16
)
pipe.vae = AutoencoderTiny.from_pretrained("madebyollin/taesd", dtype=torch.float16)
pipe = pipe.to("cuda")
pipe = pipe.to(device)

prompt = "slice of delicious New York-style berry cheesecake"
image = pipe(prompt, num_inference_steps=25).images[0]
Expand All @@ -36,12 +39,14 @@ To use with Stable Diffusion XL 1.0
```python
import torch
from diffusers import DiffusionPipeline, AutoencoderTiny
from diffusers.utils.torch_utils import get_device

device = get_device()
pipe = DiffusionPipeline.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0", dtype=torch.float16
)
pipe.vae = AutoencoderTiny.from_pretrained("madebyollin/taesdxl", dtype=torch.float16)
pipe = pipe.to("cuda")
pipe = pipe.to(device)

prompt = "slice of delicious New York-style berry cheesecake"
image = pipe(prompt, num_inference_steps=25).images[0]
Expand Down
5 changes: 4 additions & 1 deletion docs/source/en/api/models/autoencoderkl_allegro.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,11 @@ The model can be loaded with the following code snippet.

```python
from diffusers import AutoencoderKLAllegro
from diffusers.utils.torch_utils import get_device

vae = AutoencoderKLAllegro.from_pretrained("rhymes-ai/Allegro", subfolder="vae", dtype=torch.float32).to("cuda")

device = get_device()
vae = AutoencoderKLAllegro.from_pretrained("rhymes-ai/Allegro", subfolder="vae", dtype=torch.float32).to(device)
```

## AutoencoderKLAllegro
Expand Down
5 changes: 4 additions & 1 deletion docs/source/en/api/models/autoencoderkl_audio_ltx_2.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,11 @@ The model can be loaded with the following code snippet.

```python
from diffusers import AutoencoderKLLTX2Audio
from diffusers.utils.torch_utils import get_device

vae = AutoencoderKLLTX2Audio.from_pretrained("Lightricks/LTX-2", subfolder="vae", dtype=torch.float32).to("cuda")

device = get_device()
vae = AutoencoderKLLTX2Audio.from_pretrained("Lightricks/LTX-2", subfolder="vae", dtype=torch.float32).to(device)
```

## AutoencoderKLLTX2Audio
Expand Down
5 changes: 4 additions & 1 deletion docs/source/en/api/models/autoencoderkl_cogvideox.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,11 @@ The model can be loaded with the following code snippet.

```python
from diffusers import AutoencoderKLCogVideoX
from diffusers.utils.torch_utils import get_device

vae = AutoencoderKLCogVideoX.from_pretrained("THUDM/CogVideoX-2b", subfolder="vae", dtype=torch.float16).to("cuda")

device = get_device()
vae = AutoencoderKLCogVideoX.from_pretrained("THUDM/CogVideoX-2b", subfolder="vae", dtype=torch.float16).to(device)
```

## AutoencoderKLCogVideoX
Expand Down
5 changes: 4 additions & 1 deletion docs/source/en/api/models/autoencoderkl_ltx_2.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,11 @@ The model can be loaded with the following code snippet.

```python
from diffusers import AutoencoderKLLTX2Video
from diffusers.utils.torch_utils import get_device

vae = AutoencoderKLLTX2Video.from_pretrained("Lightricks/LTX-2", subfolder="vae", dtype=torch.float32).to("cuda")

device = get_device()
vae = AutoencoderKLLTX2Video.from_pretrained("Lightricks/LTX-2", subfolder="vae", dtype=torch.float32).to(device)
```

## AutoencoderKLLTX2Video
Expand Down
5 changes: 4 additions & 1 deletion docs/source/en/api/models/autoencoderkl_ltx_video.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,11 @@ The model can be loaded with the following code snippet.

```python
from diffusers import AutoencoderKLLTXVideo
from diffusers.utils.torch_utils import get_device

vae = AutoencoderKLLTXVideo.from_pretrained("Lightricks/LTX-Video", subfolder="vae", dtype=torch.float32).to("cuda")

device = get_device()
vae = AutoencoderKLLTXVideo.from_pretrained("Lightricks/LTX-Video", subfolder="vae", dtype=torch.float32).to(device)
```

## AutoencoderKLLTXVideo
Expand Down
5 changes: 4 additions & 1 deletion docs/source/en/api/models/autoencoderkl_magvit.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,11 @@ The model can be loaded with the following code snippet.

```python
from diffusers import AutoencoderKLMagvit
from diffusers.utils.torch_utils import get_device

vae = AutoencoderKLMagvit.from_pretrained("alibaba-pai/EasyAnimateV5.1-12b-zh", subfolder="vae", dtype=torch.float16).to("cuda")

device = get_device()
vae = AutoencoderKLMagvit.from_pretrained("alibaba-pai/EasyAnimateV5.1-12b-zh", subfolder="vae", dtype=torch.float16).to(device)
```

## AutoencoderKLMagvit
Expand Down
5 changes: 4 additions & 1 deletion docs/source/en/api/models/autoencoderkl_minimax_h3.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,14 @@ The temporal geometry is fixed by `clip_length` (17 pixel frames per encoder chu

```python
import torch
from diffusers.utils.torch_utils import get_device
from diffusers import AutoencoderKLMiniMaxH3


device = get_device()
vae = AutoencoderKLMiniMaxH3.from_pretrained(
"MiniMaxAI/MiniMax-H3", subfolder="vae", dtype=torch.float32
).to("cuda")
).to(device)
```

## AutoencoderKLMiniMaxH3
Expand Down
5 changes: 4 additions & 1 deletion docs/source/en/api/models/autoencoderkl_minimax_h3_audio.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,11 +21,14 @@ The autoencoder is **mono**, and it normalizes latents per channel with `latents

```python
import torch
from diffusers.utils.torch_utils import get_device
from diffusers import AutoencoderKLMiniMaxH3Audio


device = get_device()
audio_vae = AutoencoderKLMiniMaxH3Audio.from_pretrained(
"MiniMaxAI/MiniMax-H3", subfolder="audio_vae", dtype=torch.float32
).to("cuda")
).to(device)
```

## AutoencoderKLMiniMaxH3Audio
Expand Down
5 changes: 4 additions & 1 deletion docs/source/en/api/models/autoencoderkl_mochi.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,11 @@ The model can be loaded with the following code snippet.

```python
from diffusers import AutoencoderKLMochi
from diffusers.utils.torch_utils import get_device

vae = AutoencoderKLMochi.from_pretrained("genmo/mochi-1-preview", subfolder="vae", dtype=torch.float32).to("cuda")

device = get_device()
vae = AutoencoderKLMochi.from_pretrained("genmo/mochi-1-preview", subfolder="vae", dtype=torch.float32).to(device)
```

## AutoencoderKLMochi
Expand Down
5 changes: 4 additions & 1 deletion docs/source/en/api/models/cogvideox_transformer3d.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,11 @@ The model can be loaded with the following code snippet.

```python
from diffusers import CogVideoXTransformer3DModel
from diffusers.utils.torch_utils import get_device

transformer = CogVideoXTransformer3DModel.from_pretrained("THUDM/CogVideoX-2b", subfolder="transformer", dtype=torch.float16).to("cuda")

device = get_device()
transformer = CogVideoXTransformer3DModel.from_pretrained("THUDM/CogVideoX-2b", subfolder="transformer", dtype=torch.float16).to(device)
```

## CogVideoXTransformer3DModel
Expand Down
5 changes: 4 additions & 1 deletion docs/source/en/api/models/cogview3plus_transformer2d.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,11 @@ The model can be loaded with the following code snippet.

```python
from diffusers import CogView3PlusTransformer2DModel
from diffusers.utils.torch_utils import get_device

transformer = CogView3PlusTransformer2DModel.from_pretrained("THUDM/CogView3Plus-3b", subfolder="transformer", dtype=torch.bfloat16).to("cuda")

device = get_device()
transformer = CogView3PlusTransformer2DModel.from_pretrained("THUDM/CogView3Plus-3b", subfolder="transformer", dtype=torch.bfloat16).to(device)
```

## CogView3PlusTransformer2DModel
Expand Down
5 changes: 4 additions & 1 deletion docs/source/en/api/models/cogview4_transformer2d.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,11 @@ The model can be loaded with the following code snippet.

```python
from diffusers import CogView4Transformer2DModel
from diffusers.utils.torch_utils import get_device

transformer = CogView4Transformer2DModel.from_pretrained("THUDM/CogView4-6B", subfolder="transformer", dtype=torch.bfloat16).to("cuda")

device = get_device()
transformer = CogView4Transformer2DModel.from_pretrained("THUDM/CogView4-6B", subfolder="transformer", dtype=torch.bfloat16).to(device)
```

## CogView4Transformer2DModel
Expand Down
5 changes: 4 additions & 1 deletion docs/source/en/api/models/consisid_transformer3d.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,11 @@ The model can be loaded with the following code snippet.

```python
from diffusers import ConsisIDTransformer3DModel
from diffusers.utils.torch_utils import get_device

transformer = ConsisIDTransformer3DModel.from_pretrained("BestWishYsh/ConsisID-preview", subfolder="transformer", dtype=torch.bfloat16).to("cuda")

device = get_device()
transformer = ConsisIDTransformer3DModel.from_pretrained("BestWishYsh/ConsisID-preview", subfolder="transformer", dtype=torch.bfloat16).to(device)
```

## ConsisIDTransformer3DModel
Expand Down
7 changes: 5 additions & 2 deletions docs/source/en/api/models/controlnet.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@ from the original format using [`FromOriginalModelMixin.from_single_file`] as fo

```py
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
from diffusers.utils.torch_utils import get_device


device = get_device()
url = "https://huggingface.co/lllyasviel/ControlNet-v1-1/blob/main/control_v11p_sd15_canny.pth" # can also be a local path
controlnet = ControlNetModel.from_single_file(url)

Expand All @@ -43,8 +46,8 @@ from diffusers import ControlNetModel, UNet2DConditionModel
lora_id = "stabilityai/control-lora"
lora_filename = "control-LoRAs-rank128/control-lora-canny-rank128.safetensors"

unet = UNet2DConditionModel.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", subfolder="unet", dtype=torch.bfloat16).to("cuda")
controlnet = ControlNetModel.from_unet(unet).to(device="cuda", dtype=torch.bfloat16)
unet = UNet2DConditionModel.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", subfolder="unet", dtype=torch.bfloat16).to(device)
controlnet = ControlNetModel.from_unet(unet).to(device=device, dtype=torch.bfloat16)
controlnet.load_lora_adapter(lora_id, weight_name=lora_filename, prefix=None, controlnet_config=controlnet.config)
```

Expand Down
Loading
Loading