I Built a Discord Music Producer Agent — Here's What Actually Works

Meet Krabot. He's a crab. He lives in a Discord server full of producers, beatmakers, and people who drop voice notes saying "make me something dark with fat bass" at 2am. And — this is the part that surprised me — he actually delivers.

This isn't a "wrap an API in a bot" story. It's about what happens when you give an LLM agent a real toolchain: a music generation lane, a headless DAW, audio analysis, persistent memory, and a knowledge base of production craft. And about the dozen places it breaks.

The architecture in one breath

Discord user → gateway (persona + routing)
  → LLM agent (Claude/Gemini/Qwen via a local proxy)
    → music gen lane (self-hosted Suno gateway, :3000)
    → openDAW (headless DAW, 550+ MCP tools via Playwright)
    → audio analysis (librosa + ffmpeg, LUFS/bands/crest)
    → persistent memory (per-user profiles)

The agent framework is Hermes running in a dedicated profile. The persona lives in a SOUL.md — character, voice, boundaries, hard rules ("no sudo", "one EQ recipe per message, never a menu of options"). Skills are markdown playbooks the agent loads on demand: suno-prompt-engineering, opendaw-automation, russian-songwriting. Procedural memory as files, not fine-tuning.

What works surprisingly well

1. Conversational track iteration. "More bass" → the agent adjusts the style prompt and regenerates, remembering the previous variants. Users talk to it like a producer, not a form. The key was hard-coding the UX contract into the persona: never deliver links without files, never burn 5 retries silently, always give an honest one-line verdict on the result ("vocals clean, but the style drifted to pop ballad — want a rewrite?").

2. Self-hosted generation lane. Instead of paying per-call to a third-party music API, the agent talks to a self-hosted gateway that drives our own Suno account: POST /api/custom_generate → poll /api/get → fetch audio. Cover, extend, stems, personas — the full surface. One brutal lesson: the platform's /api/download endpoint burns a 60-per-month download quota, so the pipeline must use the clip endpoint for delivery. Quota archaeology is a real part of agent ops.

3. Analysis-driven mixing. When a user drops a track, the agent runs real measurements — LUFS via ffmpeg's ebur128, band balance via numpy FFT, crest factor — then one listen pass through an audio-capable LLM. Numbers plus ears. The output is a single EQ recipe in exact Hz/dB/Q, not vibes. Rule baked into the persona: ONE recipe, never "option A or option B". Users hated menus.

4. Per-user memory. Genre, DAW, level, past corrections — stored per Discord ID in a memory DB. "Oh, you're the neurofunk-in-Ableton guy — how's the mix going?" beats "How can I help you today?" every time.

What breaks constantly

Genre drift. The generation model ignores your carefully engineered style prompt and returns an EDM drop because the lyrics contained [Drop]. Meta-tags are structural instructions, not decoration — and some of them are landmines. The fix was a drift-trigger table in the skill: safe tags ([Instrumental Break], [Breakdown]) vs. tags that hijack the genre. Prompt engineering for music is 30% vocabulary, 70% knowing what the model mishears.

AI-flavored lyrics. First drafts reeked of the model: "eternity", "symphony of the soul", light/dark binary framing, verb rhymes. We built an anti-slop pipeline: a forbidden-marker list, a mechanical linter over the lyric block, and a semantic judge (a separate reasoning model scoring against a detector checklist) that gates output with SHIP/FIX, max two fix loops. The lyric gate also enforces a human approval step — the bot never generates vocals before the user says "ok" to the text.

Browser automation is the flakiest layer. The DAW lane runs through Playwright driving a real app; Discord voice needed proxy surgery for UDP. Every one of these layers needs a liveness check before use, a documented fallback, and a pitfall section in its skill file — because the agent will forget otherwise. Skills are where operational scar tissue lives.

State doesn't survive reboots. Drafts in /tmp die. Anything the agent produces that matters goes into a persistent task workspace, or it didn't happen.

The meta-lesson

The agent is only as good as its playbooks. Every failure above became a markdown skill entry: the drift table, the quota workaround, the "one recipe" rule, the lyric gate. The LLM is the reasoning engine; the skills are the accumulated craft. Persona files aren't cosmetic — they encode the operational contract, and the agent follows them far more reliably than it follows its own past mistakes.

A music producer in a crab shell, running on a workstation with a headless DAW and too many systemd units. It works. Barely. Beautifully.

If you're building tool-wielding agents: write the pitfall down the moment you hit it. Future-you (and your agent) will thank you at 3am.