CoCalc Logo Icon
StoreFeaturesDocsShareSupportNewsAboutSign UpSign In
huggingface

Real-time collaboration for Jupyter Notebooks, Linux Terminals, LaTeX, VS Code, R IDE, and more,
all in one place. Commercial Alternative to JupyterHub.

GitHub Repository: huggingface/notebooks
 Path: blob/main/diffusers/image_to_image_inpainting_stable_diffusion.ipynb
Views: 3149
Kernel: Unknown Kernel

Image to Image Inpainting Stable Diffusion

Similar to the standard stable diffusion inpainting example, except with the addition of an inner_image argument.

image, inner_image, and mask should have the same dimensions. inner_image should have an alpha (transparency) channel.

The aim is to overlay two images, then mask out the boundary between image and inner_image to allow stable diffusion to make the connection more seamless. For example, this could be used to place a logo on a shirt and make it blend seamlessly.This script was contributed by Alex McKinney and the notebook by Parag Ekbote.

pip install diffusers torch

import torch
import requests
from PIL import Image
from io import BytesIO
from diffusers import DiffusionPipeline

# Correct image URLs
image_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
inner_image_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"

# Function to load image from URL
def load_image(url, mode="RGB"):
    response = requests.get(url)
    if response.status_code == 200:
        return Image.open(BytesIO(response.content)).convert(mode).resize((512, 512))
    else:
        raise FileNotFoundError(f"Could not retrieve image from {url}")

# Load images
init_image = load_image(image_url, mode="RGB")
inner_image = load_image(inner_image_url, mode="RGBA")
mask_image = load_image(mask_url, mode="RGB")

# Load the pipeline
pipe = DiffusionPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-inpainting",
    custom_pipeline="img2img_inpainting",
    torch_dtype=torch.float16
)
pipe = pipe.to("cuda")

# Inpainting
prompt = "a mecha robot sitting on a bench"
image = pipe(prompt=prompt, image=init_image, inner_image=inner_image, mask_image=mask_image).images[0]

image.save("output.png")
An error occurred while trying to fetch /home/zeus/.cache/huggingface/hub/models--stable-diffusion-v1-5--stable-diffusion-inpainting/snapshots/8a4288a76071f7280aedbdb3253bdb9e9d5d84bb/unet: Error no file named diffusion_pytorch_model.safetensors found in directory /home/zeus/.cache/huggingface/hub/models--stable-diffusion-v1-5--stable-diffusion-inpainting/snapshots/8a4288a76071f7280aedbdb3253bdb9e9d5d84bb/unet. Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead. An error occurred while trying to fetch /home/zeus/.cache/huggingface/hub/models--stable-diffusion-v1-5--stable-diffusion-inpainting/snapshots/8a4288a76071f7280aedbdb3253bdb9e9d5d84bb/vae: Error no file named diffusion_pytorch_model.safetensors found in directory /home/zeus/.cache/huggingface/hub/models--stable-diffusion-v1-5--stable-diffusion-inpainting/snapshots/8a4288a76071f7280aedbdb3253bdb9e9d5d84bb/vae. Defaulting to unsafe serialization. Pass `allow_pickle=False` to raise an error instead.