CoCalc Logo Icon
StoreFeaturesDocsShareSupportNewsAboutSign UpSign In
huggingface

Real-time collaboration for Jupyter Notebooks, Linux Terminals, LaTeX, VS Code, R IDE, and more,
all in one place. Commercial Alternative to JupyterHub.

GitHub Repository: huggingface/notebooks
 Path: blob/main/diffusers/prompt_2_prompt_pipeline.ipynb
Views: 3149
Kernel: Unknown Kernel

Prompt2Prompt Pipeline

Prompt2Prompt allows the following edits:

  1. ReplaceEdit (change words in prompt)

  2. ReplaceEdit with local blend (change words in prompt, keep image part unrelated to changes constant)

  3. RefineEdit (add words to prompt)

  4. RefineEdit with local blend (add words to prompt, keep image part unrelated to changes constant)

  5. ReweightEdit (modulate importance of words)

Abbreviated examples for the other edits:

ReplaceEdit with local blend

prompts = ["A turtle playing with a ball",
           "A monkey playing with a ball"]

cross_attention_kwargs = {
    "edit_type": "replace",
    "cross_replace_steps": 0.4,
    "self_replace_steps": 0.4,
    "local_blend_words": ["turtle", "monkey"]
}

RefineEdit

prompts = ["A turtle",
           "A turtle in a forest"]

cross_attention_kwargs = {
    "edit_type": "refine",
    "cross_replace_steps": 0.4,
    "self_replace_steps": 0.4,
}

RefineEdit with local blend

prompts = ["A turtle",
           "A turtle in a forest"]

cross_attention_kwargs = {
    "edit_type": "refine",
    "cross_replace_steps": 0.4,
    "self_replace_steps": 0.4,
    "local_blend_words": ["in", "a" , "forest"]
}

ReweightEdit

prompts = ["A smiling turtle"] * 2

edit_kcross_attention_kwargswargs = {
    "edit_type": "reweight",
    "cross_replace_steps": 0.4,
    "self_replace_steps": 0.4,
    "equalizer_words": ["smiling"],
    "equalizer_strengths": [5]
}

Side note: See this GitHub gist if you want to visualize the attention maps. This script was contributed by Umer Adil and the notebook by Parag Ekbote.

import torch
from diffusers import DiffusionPipeline
import numpy as np
from PIL import Image

# Load the pipeline with custom prompt-to-prompt
pipe = DiffusionPipeline.from_pretrained(
    "CompVis/stable-diffusion-v1-4", 
    custom_pipeline="pipeline_prompt2prompt"
).to("cuda")

# Prompts for image generation
prompts = [
    "A turtle playing with a ball",
    "A monkey playing with a ball"
]

# Custom attention settings
cross_attention_kwargs = {
    "edit_type": "replace",
    "cross_replace_steps": 0.4,
    "self_replace_steps": 0.4
}

# Generate images
outputs = pipe(
    prompt=prompts,
    height=512,
    width=512,
    num_inference_steps=50,
    cross_attention_kwargs=cross_attention_kwargs
)

# Save each image explicitly without a loop
outputs.images[0].save("output_image_0.png")
cannot get type annotation for Parameter vae of <class 'diffusers_modules.git.pipeline_prompt2prompt.Prompt2PromptPipeline'>. cannot get type annotation for Parameter text_encoder of <class 'diffusers_modules.git.pipeline_prompt2prompt.Prompt2PromptPipeline'>. cannot get type annotation for Parameter tokenizer of <class 'diffusers_modules.git.pipeline_prompt2prompt.Prompt2PromptPipeline'>. cannot get type annotation for Parameter unet of <class 'diffusers_modules.git.pipeline_prompt2prompt.Prompt2PromptPipeline'>. cannot get type annotation for Parameter scheduler of <class 'diffusers_modules.git.pipeline_prompt2prompt.Prompt2PromptPipeline'>. cannot get type annotation for Parameter safety_checker of <class 'diffusers_modules.git.pipeline_prompt2prompt.Prompt2PromptPipeline'>. cannot get type annotation for Parameter feature_extractor of <class 'diffusers_modules.git.pipeline_prompt2prompt.Prompt2PromptPipeline'>. cannot get type annotation for Parameter image_encoder of <class 'diffusers_modules.git.pipeline_prompt2prompt.Prompt2PromptPipeline'>. cannot get type annotation for Parameter requires_safety_checker of <class 'diffusers_modules.git.pipeline_prompt2prompt.Prompt2PromptPipeline'>.