DEV Community: Oleksiy

I Built an AI Dungeon Master with Gemini to Automate My D&D Campaigns

Oleksiy — Sun, 15 Mar 2026 10:36:05 +0000

This article was written as part of my submission to the Gemini Live Agent Challenge. When sharing on social media, I'll be using the hashtag #GeminiLiveAgentChallenge.

If you've ever been a Game Master (GM) for a tabletop RPG like Dungeons & Dragons, you know the deal. You're a storyteller, an actor, a referee, and... an exhausted bookkeeper. I love crafting epic narratives, but the cognitive load of tracking every NPC, quest status, and inventory item in a messy notebook was burning me out.

I thought: what if an AI could handle the bookkeeping, leaving the creativity to the humans?

I didn't just want a chatbot. I wanted a "World Steward"—an agent that listens to the story and silently updates a structured database of the world in the background. That's why I built LoreForge.

What is LoreForge?

LoreForge is an AI-powered campaign companion that transforms unstructured storytelling into structured data and cinematic visuals.

Automated State Tracking: It listens to gameplay and maintains a live JSON database of the "World State" (NPCs, Factions, Quests) and "Inventory."
Cinematic Visualization: It detects when a scene is being described and uses Imagen to generate atmospheric fantasy illustrations in real-time.
Session Recaps on Autopilot: At the end of a session, it generates a fully styled Reveal.js slide deck with an outline, summaries, and custom background art for an instant "Previously on..." presentation.

The Tech Stack

The project is built on a modern, asynchronous Python backend, leaning heavily on Google's ecosystem:

Backend: Python with FastAPI
Database: Google Cloud Firestore for persisting session data.
AI Models:
- Gemini API: The core LLM for reasoning, state derivation, and content generation.
- Imagen API: For generating all the cinematic visuals.

How It Works: A Look Under the Hood

The magic of LoreForge is in how it orchestrates these services to create something more than a simple chat interface.

1. The "World Steward": State Derivation with Gemini

This is the core of the project. Instead of just having a long chat history, I needed the AI to maintain a canonical, machine-readable "source of truth" for the campaign.

I implemented a "State Derivation" pattern. Periodically, I bundle up the recent gameplay events, the current JSON state, and a complex system prompt, and send it all to Gemini. The model's job isn't to chat, but to return a new, updated JSON object representing the new reality of the game world.

The prompt includes a "schema hint" to guide the model's output. When a player says "I take the 3 healing potions from the chest," the AI doesn't just acknowledge it. It processes the event log and updates the inventory array in the JSON state.

2. Taming the LLM: Forcing Valid JSON

Anyone who has worked with LLMs knows they sometimes get creative, even when you ask for structured data. A stray Markdown fence, a trailing comma, or a truncated response can break your application.

To make LoreForge robust, I wrote a dedicated function, coerce_json_object, to clean up the model's output before parsing. It's a series of defensive heuristics that have proven incredibly effective. This function is a lifesaver; it tries standard parsing, then applies fixes for common LLM mistakes, and as a last resort, even tries parsing the string as a Python literal, which is more forgiving.

3. From Words to Worlds: Cinematic Visuals & Recaps

Visuals with Imagen: When the user's prompt contains the word "scene," LoreForge triggers Imagen to generate a visual. The key here was prompt engineering. I had to explicitly tell the model not to include text, UI elements, or logos to maintain a clean, cinematic feel.
Automated Recaps: This is my favorite feature. The presentation service reads the entire session history and the final world state, then uses a multi-step agentic workflow with Gemini:
1. Generate Outline: Ask Gemini to create a JSON outline for a slide deck, summarizing key events.
2. Generate Image Prompts: For each slide in the outline, ask Gemini to create a new, specific prompt for a background image.
3. Render HTML: Use Jinja2 to render the final outline and image URLs into a Reveal.js HTML file.

The result is a beautiful, shareable slide deck, created with a single button click.

The "It's Alive!" Moment

The first time I described the party finding a treasure chest and then saw the inventory JSON update automatically in my debug view... that was magical. It wasn't just a chatbot anymore; it was an agent that understood the game's state. Clicking the "Generate Presentation" button and seeing a fully-formed slide deck appear moments later felt like pure science fiction.

What's Next?

This project was a deep dive into agentic workflows and state management with LLMs. The next logical step is to integrate Gemini Live to remove the keyboard entirely. I want to be able to simply speak my narration, and have LoreForge listen in as a silent, helpful scribe, updating the world state from my voice alone.

Thanks for reading! Building this has been an incredible learning experience.

Building GeminiLens: An Interactive Educational Explainer with Google Gemini and Cloud Run

Oleksiy — Thu, 12 Mar 2026 07:59:25 +0000

Hi everyone!

I’m excited to share the technical journey behind GeminiLens, my entry for the #GeminiLiveAgentChallenge. GeminiLens is an adaptive AI teacher that doesn't just talk at you—it explains complex concepts using text, dynamically generated diagrams (Imagen), and even generated videos (Veo).

(Disclaimer: I created this piece of content for the purposes of entering this hackathon.)

The Problem: Textbooks Are Static

Learning a new concept often requires more than just reading. You need visualization, summarization, and interactivity. I wanted to build an agent that acts like a human academic mentor—someone who knows when to draw a diagram on the whiteboard or when to switch to a video explanation.

The Solution: A Multi-Modal Agent with Google GenAI

The core of GeminiLens is built using the google-genai SDK. I leveraged Gemini's function calling capabilities (Tools) to give the model "hands" to create content.

1. Defining the Persona

The "brain" of the application is a persistent chat session. In main.py, I defined a strict system instruction to ensure the model behaves like a mentor and uses its visual tools proactively:

system_instruction = (
    "You are the GeminiLens Academic Mentor. "
    "Explain complex concepts clearly, utilizing text, and whenever helpful, generate educational diagrams to illustrate your points. "
    "Use the `generate_educational_diagram` tool to create visuals. "
    "When a user asks to summarize a lesson or create a deck, use the `create_presentation_deck` tool. Map complex concepts to slides. "
    "Use previously generated Imagen diagrams or Veo video URLs in the media_url field to make the slides visual. "
    "CRITICAL: You MUST ALWAYS include a detailed textual explanation in your responses. Never return only an image or diagram without accompanying text. "
    "Return your final explanation in Markdown format..."
)

2. Wiring Up the Tools

GeminiLens isn't limited to text. I registered Python functions as tools that the model can invoke. For example, here is how I integrated Imagen to generate educational diagrams on the fly:

def generate_educational_diagram(prompt: str) -> str:
    """
    Generates an educational diagram or image based on the given prompt using Google's Imagen model.
    Call this tool when you need to visually explain a concept to the user.
    """
    print(f"DEBUG: Generating image for prompt: {prompt}")
    try:
        # Call Imagen model
        result = client.models.generate_images(
            model=settings.IMAGEN_MODEL_ID,
            prompt=prompt,
            config=types.GenerateImagesConfig(
                number_of_images=1, aspect_ratio="16:9", person_generation="DONT_ALLOW"
            ),
        )

        # ... (saving logic) ...

        return f"/static/images/{image_filename}"
    except Exception as e:
        return f"Error generating image: {str(e)}"

I then initialized the chat session with these tools attached, allowing Gemini to decide when to draw and when to speak:

# Global chat session to keep history across requests
global_chat = client.chats.create(
    model=settings.MAIN_MODEL_ID,
    config=types.GenerateContentConfig(
        system_instruction=system_instruction,
        tools=[tool_generate_diagram, tool_create_presentation],
        temperature=0.7,
    ),
)

The Infrastructure: Fast and Scalable with Cloud Run

To host the API, I chose FastAPI running on Google Cloud Run. Cloud Run is perfect for this because it handles the containerization complexity and scales automatically.

The application serves the frontend via Jinja2 templates and exposes endpoints like /api/explain (for the main chat) and /api/generate_video (which triggers the Veo model).

Automated Deployment (Infrastructure-as-Code)

A critical part of modern cloud engineering is automation. Instead of manually clicking through the Google Cloud Console, I wrote a shell script to automate the build and deploy process.

My deploy.sh script handles everything from building the container image with Cloud Build to deploying it to Cloud Run with the necessary environment variables.

Here is the actual script used to deploy GeminiLens:

#!/bin/bash
set -e

# Configuration
PROJECT_ID="gemini-lens-hackathon"
REGION="us-central1"
APP_NAME="gemini-lens"
IMAGE_NAME="gcr.io/$PROJECT_ID/$APP_NAME"

echo "Deploying GeminiLens Interactive Educational Explainer..."

# ... (gcloud checks and config set) ...

echo "Building and submitting Docker image via Cloud Build..."
gcloud builds submit --tag "$IMAGE_NAME"

echo "Deploying to Cloud Run..."
read -p "Enter your GOOGLE_API_KEY to inject into the deployment: " API_KEY

gcloud run deploy "$APP_NAME" \
  --image "$IMAGE_NAME" \
  --region "$REGION" \
  --allow-unauthenticated \
  --set-env-vars="GOOGLE_API_KEY=$API_KEY"

echo "Deployment complete! Visit the URL provided by Cloud Run above."

This script ensures that every deployment is consistent. It creates a container image stored in the Google Container Registry and then spins up a fresh Cloud Run revision.

What I Learned

Building GeminiLens taught me the power of combining models. Using Gemini for reasoning and conversation, while offloading visual tasks to Imagen and Veo, creates a much richer user experience than a standard text chatbot.

Check out the full source code and try deploying it yourself here: Github

Dota2 live matches viewer

Oleksiy — Fri, 09 Aug 2019 09:59:18 +0000

I have built web service to be able to track dota2 pro matches on live and watch recent matches:

http://dotainsider.com

The repo is closed, but I can tell you that it was developed on latest version of aiohttp. This python framework is the best option for me to write small and big projects with async I\O operations. For storing data I use MongoDB. Front part is made with vuejs.

The most challenging part in this project is to collect all useful data is one place, but official web API has some fields in distributed API endpoints.

Now it is in current development, but I think it's stable :)