T2I(1). Setting Up a Local Validation Environment for Kaggle's Text-to-Image Generation Challenge

Text-to-Image Generation Challenge Competition

Abstract

• Participated in the past Kaggle competition "Text-to-Image Generation Challenge."
• Built a local validation environment.
• Ran the baseline pipeline end-to-end.

Overview

After graduating from the Kaggle Titanic tutorial, I was thinking about what to do next. My inner voice said, "It has to be computer vision next!" So I decided to take on the "Text-to-Image Generation Challenge" competition.

The goal of this competition is to generate images that match given text prompts(Prompt-to-Image Alignment). It is a cutting-edge challenge and the perfect opportunity to build hands-on skills in generative AI.

In this article, I summarize the steps I took to build a local evaluation environment for image generation and automated evaluation on my PC, along with an overview of how the pipeline works.

Environment Setup

1. Hardware Configuration

The specs of my local PC used for the evaluation environment are as follows:

• OS: Windows 11
• CPU: Intel Core Processor(24 logical cores)
• RAM: 32GB
• GPU: NVIDIA GeForce RTX 4060
• Storage: HDD(500GB free space)

2. Installed Libraries and Selection Rationale

• Python 3.12: Although 3.14 is the latest version, PyTorch and other libraries could not utilize the GPU on 3.14. 😞
• uv: A fast Python package manager.
• diffusers: The de facto standard library provided by Hugging Face for running diffusion models like Stable Diffusion.
• ultralytics(YOLO): Used to automatically detect and evaluate whether objects specified in the prompt are correctly depicted in the generated images.

3. Installation Steps

Run the following commands in PowerShell:

# 1. Install uv
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

# 2. Create a virtual environment specifying Python 3.12
uv venv --python 3.12

# 3. Install PyTorch and Torchvision supporting CUDA 12.4 in the virtual environment
uv pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124

# 4. Install required dependency libraries all together
uv pip install diffusers "transformers<5.0.0" accelerate pandas ultralytics

Now, a fast image generation environment using CUDA on the GPU(NVIDIA GeForce RTX 4060) is ready.

Running the Baseline Pipeline

I verified that the entire pipeline runs end-to-end:

• Load prompts -> Generate images -> Auto-evaluate using the object detection model(YOLO).

uv run src/baseline.py

Processing Flow

1. How Images Are Generated from Text

The core process of generating images from text prompts in Stable Diffusion works as follows.

1.1 Overview of the Generation Process

Stable Diffusion operates on an architecture called the "Latent Diffusion Model(LDM)." Instead of performing computations directly in the high-dimensional pixel space, it compresses the image into a lower-dimensional "Latent Space"¹ and performs noise removal² there. This reduces memory consumption while maintaining generation quality.

Role of Each Component

1. Input

• Text Prompt(String): The input string specifying what objects to depict. Example: "A dog sitting on a chair".
• Random Noise(Latent Space: 64x64): A numeric array generated on the fly at runtime, represented as a 64x64 array.

2. Text Processing

• 2.1. CLIP Tokenizer: Splits the input prompt(string) into tokens(words or subwords) and converts them into "Token IDs(an array of numbers)" based on a predefined dictionary.
  • Example: The string "A dog on a chair" is converted to a numeric array like [320, 2361, 803, 320, 8942].
• 2.2. CLIP Text Encoder: Takes the array of Token IDs and converts it into high-dimensional vectors representing semantic meanings("Text Embeddings"), which capture word relationships and nuances. This serves as the guide map for image generation.

3. Latent Space Reverse Diffusion Process(Denoising Loop)
The reverse diffusion process is like watching ink dispersed in water gather back into a single drop(playing time backward). Starting from complete noise(static), U-Net predicts the noise in the image step-by-step, subtracting it to construct the final image.

• 3.1. U-Net(Noise Prediction in Latent Space): Image generation starts with meaningless random noise(Latent Noise). U-Net takes the noisy image and the text embedding vectors from CLIP as inputs and predicts what noise to remove to bring the image closer to the prompt's meaning. The connection between the text and noise is established using a mechanism called Cross-Attention.
• 3.2. Scheduler(Noise Reduction Control): An algorithm that controls how much of the predicted noise to subtract at each step. By repeating the loop("noise prediction -> subtraction"), the image gradually emerges from the static, completing the latent representation of the final image.
  • Note: The sd-turbo model used in this baseline is a distilled model(Adversarial Diffusion Distortion) that can complete this denoising process in just a single step, making it extremely fast.

4. Reconstruct to Pixel Space

• VAE Decoder: Variational Autoencoder(VAE) decoder takes the completed denoised latent representation(usually a small 64x64 size) and decodes it back into the pixel space(e.g., 512x512 pixels) that humans can perceive.

Implementation

The evaluation pipeline reads inputs from input/DreamLayer-Prompt-Kaggle.txt(provided by Kaggle), generates images, and evaluates them using YOLOv8.

Python Code

import torch
import pandas as pd
import re
from pathlib import Path
from diffusers import StableDiffusionPipeline
from ultralytics import YOLO

# Objects corresponding to common COCO classes
common_objects = {
    'man', 'woman', 'person', 'dog', 'cat', 'car', 'truck', 'train', 'airplane',
    'pizza', 'cake', 'donut', 'chair', 'table', 'bed', 'toilet', 'sink', 'mirror', 'clock', 'umbrella'
    # ... (partially omitted)
}

def extract_expected_objects(text):
    words = re.findall(r'\b\w+\b', text.lower())
    return set(word for word in words if word in common_objects)

def calculate_f1_score(expected, detected):
    if len(expected) == 0 and len(detected) == 0:
        return 1.0
    if len(expected) == 0 or len(detected) == 0:
        return 0.0
    true_positives = len(expected.intersection(detected))
    precision = true_positives / len(detected)
    recall = true_positives / len(expected)
    if precision + recall == 0:
        return 0.0
    return 2 * (precision * recall) / (precision + recall)

In the image generation process, a fixed seed value(seed = 42) is set in torch.Generator to ensure reproducibility. The evaluation script outputs generated images and computes the F1-score to produce submission.csv.

Evaluation

Execution Results and Local Evaluation Score

Running the baseline script on the GPU(CUDA) produced the following console output:

Using device: cuda
Reading prompts from input\DreamLayer-Prompt-Kaggle.txt...
Loaded 49 prompts.
Loading pipeline for stabilityai/sd-turbo...
...
[49/49] Generating: 'A group of people standing on a snow covered hill.' -> 0049.png
Image generation complete.
Loading YOLOv8 model for local evaluation...
Running YOLO detection and F1 score calculation...
Saved results.csv to output\results.csv
Saved submission.csv to output\submission.csv

==================================================
LOCAL EVALUATION COMPLETE
Mean F1 Score: 0.5102
==================================================

The local F1 score using SD-Turbo is 0.5102.
This is a poor score for the competition, but it is a starting point.
Below is the list of generated images. You can see that some images do not match the prompts correctly.

1	2	3	4	5	6	7	8	9	10
11	12	13	14	15	16	17	18	19	20
21	22	23	24	25	26	27	28	29	30
31	32	33	34	35	36	37	38	39	40
41	42	43	44	45	46	47	48	49	50

Output Files(output/)

• output/images/0001.png to 0049.png(Generated images)
• output/results.csv(Detailed log of prompts, detected objects, and F1 scores)
• output/submission.csv(Kaggle submission CSV)
• output/config-dreamlayer.json(Parameter settings for generation)

An F1 score of 0.5102 is definitely a losing score, but it is not a bad start. I will aim for 0.70 from here.

Future Directions

Now that we have a baseline score of 0.5102, we will work on improving it.

1. Switching to Higher-Precision Generation Models:
   While SD-Turbo is fast, its rendering details(especially fine object layouts and shapes) are weak. We plan to switch to more expressive models like SDXL(stabilityai/stable-diffusion-xl-base-1.0) or Flux.

2. Prompt Engineering:
   Modify prompts by adding quality keywords or emphasis tags to ensure target objects are rendered in sizes and layouts that YOLOv8 can easily detect.

3. Self-Feedback Iteration(Maximization Hack):
   Generate multiple images for each prompt using different seed values, and build a system that automatically selects the image with the highest YOLOv8 F1 score to include in the final submission. This allows us to optimize(overfit) the submission set specifically to the evaluator's(YOLOv8) characteristics for a higher score.

Next, I will verify the changes in score by implementing this "Self-Feedback Automatic Selection System" and switching to a higher-quality model.

Conclusion

In this article, I set up a local validation environment for Kaggle's "Text-to-Image Generation Challenge" and verified the end-to-end process, achieving an initial F1 score of 0.5102.

I also gained a deeper understanding of the mechanisms behind how Stable Diffusion generates images from text strings.

From here, I will select different models and perform prompt hacking to improve the score.

I hope this helps.

Japanese Version:

• 16-(1)Kaggle実践2「Text-to-Image Generation Challenge」コンペ用にローカル検証環境を構築してみた

---

1. Q: What is Latent Space?
   A: A digital image(pixel space) is a collection of dots(e.g., 512x512), which contains too much information. Latent space is a compressed digital space that extracts only the key features of the image(outlines, colors, semantic meaning) from the raw pixel data.
    ↩

2. Q: What is noise removal?
   A: This refers to the task handled by the U-Net. The U-Net predicts the "useless noise components" to be subtracted next, working step-by-step. The concept is similar to "sculpting."
    ↩