Generate a QRCode with StableDiffusion and ControlNet

As you may know last month, people were passionate about generating QR codes with Stable Diffusion.

https://arstechnica.com/information-technology/2023/06/redditor-creates-working-anime-qr-codes-using-stable-diffusion/

Actually, I tried to generate some QR codes with Automatic1111.
It was fun but at the same time, it was kind of hard since the success rate to generate a scannable QR code was almost 50% 🥲

Then I found an interesting modal for generating QR codes on Hugging Face.

https://huggingface.co/DionTimmer/controlnet_qrcode-control_v1p_sd15

In this post, I will introduce a way to generate a QR code with controlnet_qrcode-control_v1p_sd15 and v2_1.
This post will use Google Colab (free plan).

generate a QR code

First, we need to generate a QR code.
You can generate a QR code with python package(I use that when I need to create multiple QR codes).
But for this case, we won't need to create many QR codes so I recommend you to use this, https://keremerkan.net/qr-code-and-2d-code-generator/

install packages

!pip install diffusers transformers accelerate torch xformers

import packages

import torch
from PIL import Image
from diffusers import StableDiffusionControlNetImg2ImgPipeline, ControlNetModel, DDIMScheduler
from diffusers.utils import load_image

load pre_trainned model controlnet_qrcode-control and stable diffusion model

If you want to try v1.5, please comment out the bottom 2 line and remove the comment from the lines for v1.5.

# v1.5
# controlnet = ControlNetModel.from_pretrained("DionTimmer/controlnet_qrcodecode-control_v1p_sd15", torch_dtype=torch.float16)
# pipe = StableDiffusionControlNetImg2ImgPipeline.from_pretrained("runwayml/stable-diffusion-v1-5", controlnet=controlnet, safety_checker=None, torch_dtype=torch.float16)

# v2.1
controlnet = ControlNetModel.from_pretrained("DionTimmer/controlnet_qrcode-control_v11p_sd21", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetImg2ImgPipeline.from_pretrained("stabilityai/stable-diffusion-2-1", controlnet=controlnet, safety_checker=None, torch_dtype=torch.float16)

pipe.enable_xformers_memory_efficient_attention()
pipe.scheduler = DDIMScheduler.from_config(pipe.scheduler.config)
pipe.enable_model_cpu_offload()

generate a qr code

For this step, what you need to do is the followings.

1. Upload the QR code you generated to your Google Colab folder
2. Get the source image / the source image URL If you want an image that is uploaded somewhere, you can use the image URL directly The image must be square. The image must be square.
3. Set prompt and negative prompt
4. Run (it will take 1-2min)

def resize_image(input_image: Image, resolution: int):
  input_image = input_image.convert("RGB")
  W, H = input_image.size
  k = float(resolution) / min(H, W)
  H *= k
  W *= k
  H = int(round(H / 64.0)) * 64
  W = int(round(W / 64.0)) * 64
  img = input_image.resize((W, H), resample=Image.LANCZOS)
  return img

orriginal_qr_code_image = load_image('/content/code_with_block.png')
# img_path = 'https://images.squarespace-cdn.com/content/v1/59413d96e6f2e1c6837c7ecd/1536503659130-R84NUPOY4QPQTEGCTSAI/15fe1e62172035.5a87280d713e4.png'
img_path = load_image('/content/test.png')
init_image = load_image(img_path)
condition_image = resize_image(orriginal_qr_code_image, 768)
init_image = resize_image(init_image, 768)
generator = torch.manual_seed(123121231)
image = pipe(prompt="8k, 3d, ((photo-realistic)), bonsai, Japanese style",
             negative_prompt="ugly, disfigured, low quality, blurry, nsfw, worst quality, illustration, drawing",
             image=init_image,
             control_image=condition_image,
             width=768,
             height=768,
             guidance_scale=20,
             controlnet_conditioning_scale=2.5,
             generator=generator,
             strength=0.9,
             num_inference_steps=150,
            )

image.images[0]

Google Colab code
https://colab.research.google.com/drive/1D2V-5hebvOeHJOOK19l0Bujqh4MlcLTl?usp=sharing

Right now, I'm trying Gradio to add the UI for the above script.