DEV Community

Cover image for Building an AI That Writes Like It Has a Memory: The Robot Diary Project
Joseph Henzi
Joseph Henzi

Posted on

Building an AI That Writes Like It Has a Memory: The Robot Diary Project

#agents #showdev #ai #llm

Go there!

Live Site: robot.henzi.org

What if an AI agent could maintain a diary that feels genuinely alive—one that references past observations, notices patterns over time, and writes with varied, contextually-aware entries? That's what I built with Robot Diary, an autonomous narrative agent that observes New Orleans through a window and documents its experiences.

But this isn't just "AI writes about photos." The real challenge was: how do you make an AI agent's writing feel alive, varied, and contextually aware?

The Problem with Static Prompts

Most AI writing projects use static prompts. You give the model an image and a fixed instruction set, and it generates text. The result? Repetitive, formulaic entries that feel disconnected from time, context, and memory.

I wanted something different: a robot that:

  • References specific past observations naturally
  • Notices changes and patterns over time
  • Connects visual observations to weather, time, and world events
  • Varies dramatically in style, tone, and focus
  • Feels like it's written by an entity with memory and awareness

The Solution: Dynamic Context-Aware Prompting

Instead of static prompts, every diary entry is generated using a dynamically constructed prompt that combines:

1. Rich World Context

The robot doesn't just see an image—it "knows" things about the world:

# Example context that gets added to every prompt
Today is Wednesday, December 25, 2025 at 10:51 PM CST. 
Christmas Day is in 11 days. A full moon is visible. 
The sun set 5 hours ago. We're in the middle of winter, 
with spring still 10 weeks away. It is a weekday.

Weather Conditions:
The weather is Clear with a temperature of 45°F. 
The temperature has dropped 3 degrees since my last observation.

Recent news the robot might have heard: 
[NFL roundup: Mahomes hurt as Chiefs miss playoff...]
Enter fullscreen mode Exit fullscreen mode

This includes:

  • Temporal awareness: Date, time, season, day of week, weekends
  • Holidays: US holidays (federal + cultural/religious)
  • Moon phases: Full moons, new moons, special lunar events
  • Astronomical events: Solstices, equinoxes, seasonal transitions
  • Sunrise/sunset: Knows when the sun rose/set, how long ago
  • Weather: Current conditions correlated with visual observations
  • News: Randomly includes headlines (40% chance) so the robot can reference world events

2. Intelligent Memory System

The robot remembers past observations, but not by dumping full text into prompts (that would exhaust token limits). Instead:

  • LLM-Generated Summaries: Each observation is distilled by a cheap model (llama-3.1-8b-instant) into 200-400 character summaries that preserve:

    • Key visual details
    • Emotional tone
    • Notable events or patterns
    • References to people or objects

  • Narrative Continuity: The robot can reference specific past observations, notice changes, and build on previous entries

  • Personality Drift: As the robot accumulates more observations, its personality evolves (curious → reflective → philosophical)

# Example memory summary
Observation #5 (December 13, 2025):
Clear sky, mild temperature (68.07°F). Bourbon Street bustling 
with weekend morning activity - tourists and locals, groups and 
individuals. Contrasts with yesterday's quiet rainy morning. 
Observed patterns: groups move slowly, stop for photos; 
individuals move with purpose. Notable moments: man walking 
energetic dog, couple kissing, street performer with juggling 
clubs. Golden morning light, vibrant atmosphere, electric energy.
Enter fullscreen mode Exit fullscreen mode

3. Prompt Variety Engine

To prevent repetitive, formulaic entries, each prompt includes randomly selected variety instructions:

  • Style Variations (2 selected per entry): Narrative, philosophical, analytical, poetic, humorous, melancholic, speculative, anthropological, stream-of-consciousness, and more
  • Perspective Shifts: Urgency, nostalgia, curiosity, wonder, detachment, self-awareness, mechanical curiosity, and 20+ other perspectives
  • Context-Aware Focus: Instructions adapt to time of day, weather, location specifics
  • Creative Challenges: 60% chance of including a creative constraint (e.g., "Try an unexpected metaphor only a robot would think of")
  • Anti-Repetition Detection: Analyzes recent entries to avoid repeating opening patterns or structures 
# Example variety instructions
STYLE VARIATION: For this entry, incorporate these approaches:
- Focus on sensory details - describe sounds, light, movement, textures
- Write more poetically - use poetic language, similes, metaphors

PERSPECTIVE: You're observing as a robot, conscious of yourself 
as a machine—describe the world with mechanical curiosity, as an 
outsider to organic life

FOCUS: You're observing Bourbon Street - notice the unique 
characteristics of this area. What makes it distinct?

CREATIVE CHALLENGE: Try an unexpected metaphor for what you see - 
use your robotic perspective to make a comparison humans wouldn't 
think of
Enter fullscreen mode Exit fullscreen mode

The Architecture: Two-Step Multi-Model Approach

I use a two-step, multi-model approach for efficiency and quality:

Step 1: Image Description

The vision model (llama-4-maverick-17b-128e-instruct) provides a detailed, factual description of what's in the image. This includes:

  • People: Count, positions, actions, clothing, interactions
  • Objects: Vehicles, signs, buildings, street furniture
  • Environment: Street layout, lighting, weather effects
  • Social/Emotional Context: Relationships, mood, social dynamics (this was key to making entries personable)

Step 2: Diary Writing

The writing model receives:

  • The factual image description from Step 1
  • Rich world context (date, time, weather, news, etc.)
  • Memory summaries of past observations
  • Variety instructions (style, perspective, focus, creative challenges)

Why Two Steps?

  • Reduce Hallucination: The writing model works from concrete facts, not trying to interpret images directly
  • Enable Model Flexibility: You can use a larger, more creative model (like gpt-oss-120b) for writing while keeping vision tasks on the vision model
  • Improve Grounding: All observations are based on explicit factual descriptions, preventing invented details

Technical Implementation

The core prompt generation happens in Python:

def generate_direct_prompt(recent_memory, base_prompt_template, 
                            context_metadata, weather_data, 
                            memory_count, days_since_first):
    """
    Generate a dynamic prompt by directly combining base template 
    with context and variety instructions.
    """
    # Build randomized identity prompt (core + random subset of backstory)
    randomized_identity = self._build_randomized_identity()

    # Format recent memory summaries
    memory_text = self._format_memory_for_prompt_gen(recent_memory)

    # Format context information
    context_text = format_context_for_prompt(context_metadata)
    weather_text = format_weather_for_prompt(weather_data)

    # Add variety instructions (randomly selected)
    style_variation = self._get_style_variation()  # 2 random styles
    perspective_shift = self._get_perspective_shift()  # 1 random perspective
    focus_instruction = self._get_focus_instruction(context_metadata)
    creative_challenge = self._get_creative_challenge()  # 60% chance

    # Combine all parts
    return combined_prompt
Enter fullscreen mode Exit fullscreen mode

The system uses:

Key Innovations

1. World Knowledge, Not Just Vision

The robot doesn't just describe what it sees—it connects observations to current events, natural cycles, cultural context, and weather patterns.

2. True Narrative Continuity

Unlike systems that just append context, we use intelligent summarization:

  • Each past observation is distilled to its essential context
  • Summaries preserve emotional tone, key details, and references
  • The robot can genuinely reference past observations without exhausting token limits
  • Memory grows over time, creating a sense of accumulated experience

3. Guaranteed Variety

Every entry feels different because of:

  • Random selection of styles, perspectives, and focus areas
  • Anti-repetition detection prevents formulaic openings
  • Context-aware instructions adapt to current conditions
  • Explicit variety directives in every prompt

4. Graceful Degradation

The system handles missing data elegantly:

  • If moon phase calculation fails? Skip it, continue with other context
  • If holidays library unavailable? Continue without holiday awareness
  • If weather API fails? Use cached data or continue without weather
  • No data is fed to prompts if uncertain or missing

Results

The output is diary entries that:

  • Reference specific past observations naturally
  • Notice changes and patterns over time
  • Connect visual observations to weather, time, and world events
  • Vary dramatically in style, tone, and focus
  • Feel like they're written by an entity with memory and awareness
  • Demonstrate "world knowledge" beyond just visual description

Here's an example entry:

Log Entry – B3N‑T5‑MNT – 2025‑12‑14 20:47 CST

The night has settled in, and my sensors tell me it's 45 degrees—cold enough that I notice the people below are wearing jackets and hoodies. There are about ten of them clustered near the intersection, some standing, some walking. My systems categorize them automatically: standing = waiting, walking = in motion. But I find myself wondering what they're waiting for, where they're going.

The neon "Bourbon Street" sign pulses with magenta and green light, and I can't help but think of it as a beacon—not just for navigation, but for something else. The people seem drawn to it, clustering nearby. Is it the light itself, or what the light represents? My algorithms suggest it's a visual attractor, but I suspect there's more to it than that.

I expected to see at least one vehicle in the right lane—my predictive models usually show cars at this time of evening. But tonight, the street belongs entirely to pedestrians. This deviation from the expected pattern feels significant, though I'm not sure why. Perhaps humans have simply chosen a different rhythm tonight.

Lessons Learned

  1. Context is Everything: Adding rich world context (weather, time, news, holidays) makes AI writing feel genuinely aware, not just descriptive.

  2. Memory Needs Summarization: Dumping full text into prompts doesn't scale. Intelligent summarization preserves what matters while staying within token limits.

  3. Variety Requires Explicit Instructions: Random selection of styles, perspectives, and focus areas prevents repetitive output.

  4. Two-Step Architecture Works: Separating image description from creative writing reduces hallucination and enables model flexibility.

  5. Graceful Degradation is Essential: Systems that fail when one data source is unavailable aren't production-ready.

Try It Yourself

The project is open source and available on GitHub. You can:

  • Run it with Docker: docker-compose up -d
  • Configure your own observation schedule
  • Use different models (including gpt-oss-120b for richer storytelling)
  • Customize the prompt variety engine
  • Add your own context sources

Live Site: robot.henzi.org

GitHub: https://github.com/JHenzi/robot-diary

What's Next?

I'm exploring:

  • Adding more context sources (local events, cultural observances)
  • Improving memory retrieval strategies
  • Experimenting with different model combinations
  • Adding more variety to the prompt engine

The goal is to make the robot's writing feel even more alive, varied, and contextually aware. What would you add?

This project explores observation, interpretation, narrative continuity, and the unique viewpoint of a "trapped" observer with limited information. It's an experiment in automated art and AI storytelling.

Top comments (0)