DEV Community: Joseph Henzi

I built an autonomous Robot Diary with "Boredom Scores" and a sense of time 🤖📖

Joseph Henzi — Sat, 10 Jan 2026 01:17:07 +0000

I’ve always wondered: If a robot were left alone to watch the world go by, what would it actually think about? Would it just catalog data, or would it eventually get... bored?

To find out, I built B3N-T5-MNT (The Maintenance Robot), an autonomous AI agent that lives on Bourbon Street, New Orleans. It wakes up twice a day, looks at the world through a webcam, and writes a diary entry about what it sees.

But this isn't just a basic "Image-to-Text" bot. It has a memory, a mood, and a "Boredom Engine."

🧠 The Tech Stack

Vision: Llama 4 Maverick (via Groq) for high-speed scene analysis.
Brain: A custom "Memory API" (MemVault) that uses hybrid search (Vector + Keyword + Recency) so the robot can remember that "white van from last Tuesday."
Environmental Context: Real-time data fetching for weather, moon phases, and New Orleans holidays (it knows when it’s Mardi Gras!).
The Boredom Engine: A logic layer that tracks visual repetition. If the street stays too quiet for too long, the robot's writing style shifts from factual reporting to existential noir or poetic reflection.

🛠 Recent Changelog Highlights

I’ve been iterating on the robot’s "consciousness" over at robot.henzi.org/changelog:

Memory Integration: Swapped out static JSON for a dynamic RAG (Retrieval-Augmented Generation) layer. Now, the robot can "overhear" news and link it to what it sees on the street.
Astro-Logic: Hard-wired the solar/lunar calendar into the prompt. The robot is now officially "grumpy" during rainy transitions and more reflective during full moons.
Boredom Scoring: Implemented a decay function for "interestingness." If the pixel delta between days is too low, the boredom score spikes, triggering a change in the LLM's system prompt to be more "philosophical."

🎭 Why build this?

This started as a late-night "what if" project sponsored by The Henzi Foundation. It’s an exploration of Environmental Intelligence—the idea that an AI shouldn't just respond to prompts, but should react to the rhythm of the physical world.

Is the robot bored today, or did it find something new?

Check out the live diary here: https://robot.henzi.org

I’d love to hear your thoughts on "AI boredom" or how you're handling long-term memory in your own agents!

AI #Robotics #OpenSource #LLM #CreativeCoding

Building an AI That Writes Like It Has a Memory: The Robot Diary Project

Joseph Henzi — Mon, 15 Dec 2025 04:51:53 +0000

Go there!

Live Site: robot.henzi.org

What if an AI agent could maintain a diary that feels genuinely alive—one that references past observations, notices patterns over time, and writes with varied, contextually-aware entries? That's what I built with Robot Diary, an autonomous narrative agent that observes New Orleans through a window and documents its experiences.

But this isn't just "AI writes about photos." The real challenge was: how do you make an AI agent's writing feel alive, varied, and contextually aware?

The Problem with Static Prompts

Most AI writing projects use static prompts. You give the model an image and a fixed instruction set, and it generates text. The result? Repetitive, formulaic entries that feel disconnected from time, context, and memory.

I wanted something different: a robot that:

References specific past observations naturally
Notices changes and patterns over time
Connects visual observations to weather, time, and world events
Varies dramatically in style, tone, and focus
Feels like it's written by an entity with memory and awareness

The Solution: Dynamic Context-Aware Prompting

Instead of static prompts, every diary entry is generated using a dynamically constructed prompt that combines:

1. Rich World Context

The robot doesn't just see an image—it "knows" things about the world:

# Example context that gets added to every prompt
Today is Wednesday, December 25, 2025 at 10:51 PM CST. 
Christmas Day is in 11 days. A full moon is visible. 
The sun set 5 hours ago. We're in the middle of winter, 
with spring still 10 weeks away. It is a weekday.

Weather Conditions:
The weather is Clear with a temperature of 45°F. 
The temperature has dropped 3 degrees since my last observation.

Recent news the robot might have heard: 
[NFL roundup: Mahomes hurt as Chiefs miss playoff...]

This includes:

Temporal awareness: Date, time, season, day of week, weekends
Holidays: US holidays (federal + cultural/religious)
Moon phases: Full moons, new moons, special lunar events
Astronomical events: Solstices, equinoxes, seasonal transitions
Sunrise/sunset: Knows when the sun rose/set, how long ago
Weather: Current conditions correlated with visual observations
News: Randomly includes headlines (40% chance) so the robot can reference world events

2. Intelligent Memory System

The robot remembers past observations, but not by dumping full text into prompts (that would exhaust token limits). Instead:

LLM-Generated Summaries: Each observation is distilled by a cheap model (llama-3.1-8b-instant) into 200-400 character summaries that preserve:
- Key visual details
- Emotional tone
- Notable events or patterns
- References to people or objects
Narrative Continuity: The robot can reference specific past observations, notice changes, and build on previous entries
Personality Drift: As the robot accumulates more observations, its personality evolves (curious → reflective → philosophical)

# Example memory summary
Observation #5 (December 13, 2025):
Clear sky, mild temperature (68.07°F). Bourbon Street bustling 
with weekend morning activity - tourists and locals, groups and 
individuals. Contrasts with yesterday's quiet rainy morning. 
Observed patterns: groups move slowly, stop for photos; 
individuals move with purpose. Notable moments: man walking 
energetic dog, couple kissing, street performer with juggling 
clubs. Golden morning light, vibrant atmosphere, electric energy.

3. Prompt Variety Engine

To prevent repetitive, formulaic entries, each prompt includes randomly selected variety instructions:

Style Variations (2 selected per entry): Narrative, philosophical, analytical, poetic, humorous, melancholic, speculative, anthropological, stream-of-consciousness, and more
Perspective Shifts: Urgency, nostalgia, curiosity, wonder, detachment, self-awareness, mechanical curiosity, and 20+ other perspectives
Context-Aware Focus: Instructions adapt to time of day, weather, location specifics
Creative Challenges: 60% chance of including a creative constraint (e.g., "Try an unexpected metaphor only a robot would think of")
Anti-Repetition Detection: Analyzes recent entries to avoid repeating opening patterns or structures

# Example variety instructions
STYLE VARIATION: For this entry, incorporate these approaches:
- Focus on sensory details - describe sounds, light, movement, textures
- Write more poetically - use poetic language, similes, metaphors

PERSPECTIVE: You're observing as a robot, conscious of yourself 
as a machine—describe the world with mechanical curiosity, as an 
outsider to organic life

FOCUS: You're observing Bourbon Street - notice the unique 
characteristics of this area. What makes it distinct?

CREATIVE CHALLENGE: Try an unexpected metaphor for what you see - 
use your robotic perspective to make a comparison humans wouldn't 
think of

The Architecture: Two-Step Multi-Model Approach

I use a two-step, multi-model approach for efficiency and quality:

Step 1: Image Description

The vision model (llama-4-maverick-17b-128e-instruct) provides a detailed, factual description of what's in the image. This includes:

People: Count, positions, actions, clothing, interactions
Objects: Vehicles, signs, buildings, street furniture
Environment: Street layout, lighting, weather effects
Social/Emotional Context: Relationships, mood, social dynamics (this was key to making entries personable)

Step 2: Diary Writing

The writing model receives:

The factual image description from Step 1
Rich world context (date, time, weather, news, etc.)
Memory summaries of past observations
Variety instructions (style, perspective, focus, creative challenges)

Why Two Steps?

Reduce Hallucination: The writing model works from concrete facts, not trying to interpret images directly
Enable Model Flexibility: You can use a larger, more creative model (like gpt-oss-120b) for writing while keeping vision tasks on the vision model
Improve Grounding: All observations are based on explicit factual descriptions, preventing invented details

Technical Implementation

The core prompt generation happens in Python:

def generate_direct_prompt(recent_memory, base_prompt_template, 
                            context_metadata, weather_data, 
                            memory_count, days_since_first):
    """
    Generate a dynamic prompt by directly combining base template 
    with context and variety instructions.
    """
    # Build randomized identity prompt (core + random subset of backstory)
    randomized_identity = self._build_randomized_identity()

    # Format recent memory summaries
    memory_text = self._format_memory_for_prompt_gen(recent_memory)

    # Format context information
    context_text = format_context_for_prompt(context_metadata)
    weather_text = format_weather_for_prompt(weather_data)

    # Add variety instructions (randomly selected)
    style_variation = self._get_style_variation()  # 2 random styles
    perspective_shift = self._get_perspective_shift()  # 1 random perspective
    focus_instruction = self._get_focus_instruction(context_metadata)
    creative_challenge = self._get_creative_challenge()  # 60% chance

    # Combine all parts
    return combined_prompt

The system uses:

Groq API for fast, cost-effective LLM inference
Astral for astronomical calculations (sunrise/sunset, moon phases)
Holidays for US holiday detection
Pirate Weather API for weather data
YouTube Live Streams via yt-dlp for video source
Hugo for static site generation

Key Innovations

1. World Knowledge, Not Just Vision

The robot doesn't just describe what it sees—it connects observations to current events, natural cycles, cultural context, and weather patterns.

2. True Narrative Continuity

Unlike systems that just append context, we use intelligent summarization:

Each past observation is distilled to its essential context
Summaries preserve emotional tone, key details, and references
The robot can genuinely reference past observations without exhausting token limits
Memory grows over time, creating a sense of accumulated experience

3. Guaranteed Variety

Every entry feels different because of:

Random selection of styles, perspectives, and focus areas
Anti-repetition detection prevents formulaic openings
Context-aware instructions adapt to current conditions
Explicit variety directives in every prompt

4. Graceful Degradation

The system handles missing data elegantly:

If moon phase calculation fails? Skip it, continue with other context
If holidays library unavailable? Continue without holiday awareness
If weather API fails? Use cached data or continue without weather
No data is fed to prompts if uncertain or missing

Results

The output is diary entries that:

Reference specific past observations naturally
Notice changes and patterns over time
Connect visual observations to weather, time, and world events
Vary dramatically in style, tone, and focus
Feel like they're written by an entity with memory and awareness
Demonstrate "world knowledge" beyond just visual description

Here's an example entry:

Log Entry – B3N‑T5‑MNT – 2025‑12‑14 20:47 CST

The night has settled in, and my sensors tell me it's 45 degrees—cold enough that I notice the people below are wearing jackets and hoodies. There are about ten of them clustered near the intersection, some standing, some walking. My systems categorize them automatically: standing = waiting, walking = in motion. But I find myself wondering what they're waiting for, where they're going.

The neon "Bourbon Street" sign pulses with magenta and green light, and I can't help but think of it as a beacon—not just for navigation, but for something else. The people seem drawn to it, clustering nearby. Is it the light itself, or what the light represents? My algorithms suggest it's a visual attractor, but I suspect there's more to it than that.

I expected to see at least one vehicle in the right lane—my predictive models usually show cars at this time of evening. But tonight, the street belongs entirely to pedestrians. This deviation from the expected pattern feels significant, though I'm not sure why. Perhaps humans have simply chosen a different rhythm tonight.

Lessons Learned

Context is Everything: Adding rich world context (weather, time, news, holidays) makes AI writing feel genuinely aware, not just descriptive.
Memory Needs Summarization: Dumping full text into prompts doesn't scale. Intelligent summarization preserves what matters while staying within token limits.
Variety Requires Explicit Instructions: Random selection of styles, perspectives, and focus areas prevents repetitive output.
Two-Step Architecture Works: Separating image description from creative writing reduces hallucination and enables model flexibility.
Graceful Degradation is Essential: Systems that fail when one data source is unavailable aren't production-ready.

Try It Yourself

The project is open source and available on GitHub. You can:

Run it with Docker: docker-compose up -d
Configure your own observation schedule
Use different models (including gpt-oss-120b for richer storytelling)
Customize the prompt variety engine
Add your own context sources

Live Site: robot.henzi.org

GitHub: https://github.com/JHenzi/robot-diary

What's Next?

I'm exploring:

Adding more context sources (local events, cultural observances)
Improving memory retrieval strategies
Experimenting with different model combinations
Adding more variety to the prompt engine

The goal is to make the robot's writing feel even more alive, varied, and contextually aware. What would you add?

This project explores observation, interpretation, narrative continuity, and the unique viewpoint of a "trapped" observer with limited information. It's an experiment in automated art and AI storytelling.

Henzi NLP — Random AI Story Generator 📖✨

Joseph Henzi — Mon, 08 Dec 2025 22:15:00 +0000

I built Henzi NLP as a fun, experimental app that generates random stories. It’s not a writing assistant or collaborative tool — it just pulls a starter from a GitHub Gist and generates a story from there. Every run is unpredictable, so you never know exactly what you’ll get.

How it works

Starts with a random prompt from a Gist

Uses an NLP model to continue the story

Outputs a full story in one go — completely different each time

You can try it here: Henzi NLP Story Generator - Using BLOOM!
It’s meant as a light, playful exploration of AI-generated stories — ideal if you want to see what random AI storytelling looks like, without worrying about guiding or editing the text.

Under the Hood

Henzi NLP is powered by large language models like BLOOM, which can generate coherent, flowing text based on a short prompt. The app demonstrates how NLP models can produce varied, creative outputs even from minimal input, showing the playful side of AI storytelling in a fully automated way.

Meet Pulsefield — Where News Becomes Living Art 🎨

Joseph Henzi — Mon, 08 Dec 2025 21:50:22 +0000

What if the endless scroll of news could be transformed into something beautiful, immersive, and meaningful? That’s the idea behind Pulsefield — a real-time, AI-powered “living canvas” that turns headlines into a dynamic, breathing art piece. Instead of flat lists or articles, you get floating, glowing orbs that pulse, shift, cluster, and dance — visualizing the world’s news as living, evolving patterns. henzi.org

🌐 What Is Pulsefield?

Pulsefield grabs news from major sources around the world and feeds them into an AI pipeline based on GPT‑OSS 120b. henzi.org
The AI groups related news stories into “topic clusters,” then renders each cluster as a 3D blob. henzi.org
The blobs behave organically — they grow and shrink depending on how many stories cover a topic, pulse when breaking news hits, attract or repel each other based on topic similarity, and change color based on sentiment (e.g. greens/blues for positive, reds for more serious or negative news). henzi.org
Interacting with the blobs reveals more details: you can hover or tap to see the sentiment breakdown, the volume of coverage, and click through to view the underlying headlines. henzi.org

🛠️ The Tech Behind the Magic

Pulsefield isn’t just art — it's a fully engineered real-time system that blends data processing, machine learning, and interactive 3D graphics.

It’s a passion project by Joe Henzi, blending data engineering, AI, and interactive art into one playful, thought-provoking experience. henzi.org

🎯 Why It Matters (And What’s Cool About It)

A new perspective on news — Rather than slog through headlines, you can see what’s trending globally, how stories interconnect, and how mood and sentiment shift over time.
Data meets art — Pulsefield lives at the intersection of journalism, data science, and digital art. It’s a creative exploration of “what’s happening now” through a visual medium.
Engaging & playful — There’s something inherently fun about watching “the world’s pulse” in motion; it invites curiosity, idle exploration, and serendipitous discovery.

A Smarter Way to Reinvest Liquidity Pool Rewards: Using Machine Learning on Solana

Joseph Henzi — Sat, 05 Jul 2025 19:09:44 +0000

I started by tracking Orca LP rewards on Solana and ended up building a contextual bandit that learns when to buy, sell, or hold SOL based on price patterns and past trade outcomes.

This project combines:

A Flask web app
Reinforcement learning (contextual bandits)
Live price prediction
SOL/USDC liquidity pool monitoring

It evolved from a passive analytics tool into a smarter system for simulating trades using real-time data and historical trends.

Liquidity pools like Orca’s SOL/USDC offer passive rewards, but I wanted to go one step further:
What if you reinvested those rewards into SOL using a machine-learning model that understood price context?

That idea led to building a trading simulator that:

Learns from market indicators (Sharpe ratio, momentum, SMA)
Evaluates the impact of each trade
Tracks portfolio performance over time
Python + Flask for the web app
SQLite for local data persistence
LiveCoinWatch API for SOL price data
Helius API for Orca LP activity
Contextual Bandits (via river or Vowpal Wabbit conceptually)
Chart.js + Tailwind for the frontend

A contextual bandit is a type of reinforcement learning algorithm that chooses actions based on the current state (called "context") and learns from the reward.

In this case:

Context = features like price deviation from 24h low/high, rolling mean, Sharpe ratio, momentum, and current portfolio state
Actions = buy, sell, or hold
Reward = based on realized/unrealized profit, market timing, and trade quality

📈 Key Features

Live portfolio tracking and trade logs
Reward calculations that factor in profit, timing, and trend alignment
Automatic SQLite logging for all trades and portfolio snapshots
Model state saved between runs
(Optional) Price predictor using rolling mean and recent volatility

🔍 Challenges & Tradeoffs

The model needs time to learn. Early trades are often naive.
There’s no "real" trading yet—only simulation.
We're still experimenting with the best reward functions.
Future goal: add backtesting, deploy to simulate with live SOL rewards.

Try it!

GitHub: [https://github.com/JHenzi/OrcaRewardDashboard]

To run locally:

Clone the repo
Set up a .env file with your API keys
Run flask run
Visit http://localhost:5030 to explore the dashboard

🛣️ What's Next

Next steps:

Backtest with historical price data
Refactor to make predictions available via API
Possibly deploy as a hosted dashboard for live tracking
Improve reward function based on more advanced trading signals

Let's Talk?

Would love to hear feedback on:

Reward strategy improvements
Trading signal features to add
How you’d use this if it supported real trading

Drop a comment below or reach out on GitHub!