DEV Community: Omkar tripathi

Vent 🙄- I built my friend someone to call when he just needs to vent (and it talks back in my voice)

Omkar tripathi — Sun, 04 Oct 2026 21:34:54 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

What I Built

I have a friend who has bad days, like all of us. The problem is not the bad day. The problem is when he tells someone about it, everyone becomes a life coach.

"You should talk to your manager."

"Have you tried journaling?"

Bro, he just wants to say out loud that some guy in a white SUV cut him off and then flipped him off. That's it. He wants someone to go "wait, what??" and let him finish.

So I built him VENT. It's a friend you can call.

You open it, press the phone button, it rings, and it picks up. Then you just talk. VENT listens and reacts like a friend on a phone call would:

You: my manager blamed me for the delay and it wasn't even my part

VENT: He blamed YOU? For the backend's mess?

It doesn't give advice. If you're in the middle of a sentence, it waits.

Two ways to talk it out

🌈 Breathe is the default. A small cloud called Puff breathes with you, opens its eyes when you talk, and VENT stays calm.
🥊 Punch is devil mode, for the days you're actually angry. A little devil boxer punches a heavy bag every time your voice gets loud. The bag tears a bit more with every hit and bursts on the 20th, a bell rings, and a new bag comes down. VENT gets angry with you here ("Nope. NOT okay.").

When you hang up, VENT asks "Feel a little lighter?" You get two choices:

Let it go. The whole call dissolves and nothing is saved.
Keep it as a page. It goes into your journal.

A journal you don't have to write

The second part is for him also, but on normal days. He never keeps a journal, because writing feels like homework. Talking about your day is easy though.

So there's a Make a journal page call, where VENT is curious instead of quiet ("Ooh, where did you two go?"). When you hang up, Gemma turns the call into a proper page for the day:

how the mood moved through the day
the people, the food, the places
the good moments and the hard ones
things to remember for tomorrow
the day told back in your own words

Later you can ask your own life questions, like "When did I last talk about Greg?" It answers only from your pages and shows which days it used. If something isn't in there, it says so instead of making it up.

Demo

One small confession about the video. My screen recorder only recorded my mic, not VENT's voice. So the captions at the bottom show what I said (transcribed with ElevenLabs Scribe) and what VENT said.

For the Breathe and journal calls these are the real replies. I got them back from Temporal, because every page VENT writes goes through a workflow that stores the full call. The Punch call I let go, so by design nothing was saved. For that one the replies are regenerated by the same model from my words, and the video says so.

Code

triggeredcode / vent

A friend you can call when you just need to talk — open-weight Gemma listens, journals your days, and remembers. Runs locally.

VENT

A friend you can call when you just need to talk. VENT listens without trying to fix you, turns the days you talk through into a visual journal, and lets you ask about your own life later.

Call → talk it out → see your day → ask your life

Every piece of intelligence in VENT is an open-weight model running on your own machine: these are the most private conversations a person has, so they shouldn't have to leave it.

What it does

Vent — a real phone call. VENT reacts like a friend would ("Wait, he blamed you?"), nudges you on, stays quiet when you're mid-thought, and never gives advice. When you hang up, nothing is kept unless you ask.
Journal — tell VENT about your day. When the call ends, Gemma writes a magazine-style page: mood arc, people, food, places, highlights, hard moments, things to…

View on GitHub

Running it takes two commands:

pnpm install
pnpm vent            # everything starts, the browser opens
pnpm voice:record    # optional: make VENT talk in your own voice

pnpm vent starts Ollama and pulls the Gemma models, starts the voice server, starts Temporal and the app, puts a few sample days in the journal and opens the browser. pnpm connect sets up Sentry and ElevenLabs if you want them.

How I Built It

The whole idea dies if it doesn't feel like a call. If you're upset and the thing on the other side sounds like a GPS lady, you hang up.

So most of my weekend actually went into one question: how do I make it sound like a person?

Version 1: the robot

The first version worked on paper: mic, speech-to-text, a small model, and the browser's built-in text-to-speech. It was terrible. The voice was flat, and it said "I'm here" after almost everything I said:

Me: (something really specific about my day)

VENT: I'm here. Take your time.

My friend would last about ten seconds. Two things were wrong, and only one of them was the voice.

Fixing what it says

I was using Gemma 3 1B for replies because it was fast, about 80 ms. But it kept returning only this:

{ "action": "acknowledge" }

There was no text at all, so the app fell back to "I'm here" every time. A fast model saying nothing is still saying nothing.

So I split the job between two Gemma 4 models in Ollama:

Job	Model	Warm time
Hear you (audio → text)	Gemma 4 E2B	~150–400 ms
Reply, with the whole call as context	Gemma 4 E4B	~400–550 ms

I ran E2B and E4B as the listener on the same scripted rants to compare:

You said	Gemma 4 E2B	Gemma 4 E4B
"I literally labelled it" (roommate ate my biryani)	"You labelled it out clearly, right?"	"Labelled it, and still? Unbelievable."
"whatever. I ordered pizza instead"	"Pizza sounds way better right now."	"Pizza fix. Good call."

E4B costs about 200 ms more, and it's completely worth it. The reply now comes back as a small action plus the words:

{ "action": "follow_up", "tone": "fired_up", "text": "Wait, he blamed YOU?" }

The prompt has a few strict rules:

- Never generic: never say "That sounds rough", "I'm here", "I hear you", "Take your time".
- No advice, no tips, no "you should", no therapy talk.
- Choose silence ONLY when their line is obviously cut off mid-sentence.

One funny thing: when my test lines were too close to the examples in the prompt, the model just repeated my examples back to me 😅. I changed the examples to random situations (a landlord raising rent, a friend skipping your birthday), and it started reacting properly.

I also pre-load the listener while the phone is ringing, so the first reply isn't slower than the rest. One full turn, from you stopping to VENT starting to talk, is about a second.

Fixing how it sounds

This is the part I'm happiest about. The voice went through three versions:

Browser text-to-speech. Flat and robotic. Nobody wants to vent to that.
Kokoro-82M, an open-weight TTS running locally. Night and day: about 100 ms per line and it sounds like a real person. But it's a person, some voice from a model card.
My own voice. He's calling a friend, so why not make it sound like one?

I talk to my laptop a lot. I have my own little dictation app, so I had a folder full of recordings of my own voice. I took 16 seconds of me talking normally, cleaned it up with ffmpeg, and cloned it locally with Chatterbox-Turbo (MIT licence), running on my Mac through mlx-audio. It takes around half a second per line.

Now when he calls VENT, the voice that answers is mine.

Honestly, the first time I heard my own voice ask "Wait, he said that to you?" it was a little weird. And then it was kind of perfect.

You can do the same with your own voice, it's one command:

pnpm voice:record

It shows a short script, records you for 45 seconds, cleans up the take (trims silence, removes rumble, evens out the loudness), and restarts the voice server on your cloned voice. The recording stays on your machine and is never committed. If you skip it, VENT just uses Kokoro.

Same voice, but the mood should change with the mode. In Punch, VENT should sound fired up. Chatterbox-Turbo just ignores emotion controls (I checked its source code), so I shape the audio after it's generated:

Mode	Voice style	What changes
Breathe	`calm`	nothing, as it is
Punch	`fired`	~14% faster, small pitch lift, more presence, compression
Journal	`bright`	a bit livelier and warmer, more curious

The words change too. In Punch the model is told to be angry on your side:

VENT (Punch): FLIPPED you off?! ARE YOU SERIOUS?!

With my ElevenLabs creator account, I also cloned my voice there (Instant Voice Clone), so VENT can talk through ElevenLabs Flash v2.5 instead if you don't have a Mac. The punch sounds were generated with the ElevenLabs Sound Effects API: heavy-bag thumps, the uppercut, the bag tearing, the bell, the chain.

Not talking over you

The most annoying thing in early tests was VENT interrupting itself. On laptop speakers its voice went back into the mic, it thought I was talking, and it stopped in the middle of its own sentence. Two fixes:

the mic is deaf while VENT speaks, plus a short echo tail
the server ignores any "turn" that's just VENT's own words coming back

If you want to cut it off, you tap the character. Same as putting your hand up ✋.

The journal page, and why Temporal

After a journal call, Gemma 4 E4B writes the page against a JSON schema (Ollama structured output). Only what the caller said counts as fact, and VENT's lines are just context. Then the page is embedded locally with nomic-embed-text and saved.

On a laptop this part can fail. The model might still be loading, or Ollama might be busy. And I really didn't want my friend to talk for five minutes and lose the page. So it runs as a Temporal workflow:

writeJournalPage:  extractDay  →  embedPage  →  savePage
                   (each step with its own timeout + retries)

I tested it by killing the worker with kill -9 in the middle of writing. When the worker came back, Temporal just picked it up again and the page arrived. And the workflow input is the full call, which is how I got VENT's real lines back for the demo captions.

Remembering, and watching it without reading it

Memories gives Gemma your pages, never the raw transcripts, and it has to tell me which pages it used. If an id doesn't exist I throw it away, so it can't invent a day.
Pages live in a local file. If you want sync, they go to MongoDB Atlas, with Atlas Vector Search over the same local embeddings (tested against Atlas Local).
Every call turn is traced in Sentry: hear → reply → speak → write the page, with timings and token counts.

These are probably the most private conversations someone has, so I switched off everything in Sentry that collects content. The new SDK collects request bodies by default, and that would have quietly sent journal text out. Now it only knows how long things took, never what was said.

Why Does Open Innovation Matter?

Because of what people say to VENT.

Nobody tells a cloud API about their worst day if they think about it for even a second.

With open models, the listening happens on your laptop:

Gemma hears you, replies, writes the page and answers your questions, all through Ollama, all local.
Your voice and journal never leave the machine unless you turn something on.
I could compare two Gemma sizes on my own test calls and pick E2B for hearing and E4B for thinking.
I could clone my own voice with an open model, with no account and no per-minute bill.

A closed API would charge for every "wait, seriously?", and a thing you call at 11 pm on a bad day shouldn't be counting minutes. The hosted parts (ElevenLabs, Sentry, Atlas) are all opt-in.

Prize Categories

Gemma: Gemma 4 E2B hears every turn. Gemma 4 E4B replies on the call (and reads the mood that changes the scene), writes the journal page with a JSON schema, and answers memory questions. All local through Ollama.
ElevenLabs: my voice as an Instant Voice Clone with Flash v2.5, the Punch sound effects from the Sound Effects API, and Scribe for the demo captions.
Sentry: gen_ai traces of every call turn and journal write, with all content collection turned off.
Temporal: every journal page is written by a durable workflow with retries. It survived me killing the worker mid-write.
MongoDB Atlas: optional synced journal with Atlas Vector Search for Memories.

Kizuna 絆 the Gemini-web to Local Environment Bridge

Omkar tripathi — Wed, 04 Mar 2026 11:05:10 +0000

This is a submission for the Built with Google Gemini: Writing Challenge

╔══════════════════════════════════════════════════════════════╗
║ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ║
║                                                              ║
║   ██╗  ██╗ ██╗ ███████╗ ██╗   ██╗ ███╗   ██╗  █████╗         ║
║   ██║ ██╔╝ ██║ ╚══███╔╝ ██║   ██║ ████╗  ██║ ██╔══██╗        ║
║   █████╔╝  ██║   ███╔╝  ██║   ██║ ██╔██╗ ██║ ███████║        ║
║   ██╔═██╗  ██║  ███╔╝   ██║   ██║ ██║╚██╗██║ ██╔══██║        ║
║   ██║  ██╗ ██║ ███████╗ ╚██████╔╝ ██║ ╚████║ ██║  ██║        ║
║   ╚═╝  ╚═╝ ╚═╝ ╚══════╝  ╚═════╝  ╚═╝  ╚═══╝ ╚═╝  ╚═╝        ║
║                                                              ║
║          [ The Gemini to Local Environment Bridge ]          ║
║ ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ ║
╚══════════════════════════════════════════════════════════════╝

Integrating AI into our daily coding workflows is a recurring discussion. The discourse focuses heavily on context windows, reasoning models, and whether AI will replace or augment engineers. But I think centering this debate purely on the models themselves is reductive.

The bigger question to me is the environment. How do we actually connect these floating, cloud-based brains to our physical work?

For a long time, my workflow was agonizing. I was stuck in what I call "Copy-Paste Torture." I would give the AI context, copy a file from my IDE, paste it into Google Gemini, ask for a change, copy the resulting code, paste it back, run it, hit an error, and start over.

Gemini was incredibly capable, but the friction of constant context-switching was killing the momentum. A brain in a browser jar has no hands.

So, I decided to build it a nervous system. I called it Kizuna (絆 - meaning bond or connection).

What I Built with Google Gemini

Kizuna is an end-to-end toolchain that transforms the standard Google Gemini web interface into a localized, agentic IDE companion. I didn't want to just build a wrapper; I wanted to create a system that felt intentional, granting the web-based LLM the ability to read, search, and safely patch a local codebase without compromising my machine.

To achieve this, I broke the system down into three foundational pillars:

1. The Engine (Local Daemon) ⚙️

Code is where software lives, but you can't just give an AI raw shell access—that's a massive security risk. I built a local backend service to act as a secure sandbox. It translates strict JSON intents from the browser into optimized local file reads, writes, and Git operations.

╔══════════════════════════════════════════════════════════════╗
║ ░░░░░░░░░░░░░░ THE PATH JAIL (SANDBOX) ░░░░░░░░░░░░░░░░░░░░░ ║
╠══════════════════════════════════════════════════════════════╣
║                                                              ║
║  ╭── [ ALLOWED WORKSPACE ] ──────────────────────────────╮   ║
║  │                                                       │   ║
║  │   📂 /workspace/my-app/     (Sandbox Root)            │   ║
║  │    ├── 📄 src/main.js             [ ✓ ] OK            │   ║
║  │    └── 📄 package.json            [ ✓ ] OK            │   ║
║  │                                                       │   ║
║  ╰───────────────────────────────────────────────────────╯   ║
║                                                              ║
║  ╭── [ BLOCKED EXTERNALS ] ──────────────────────────────╮   ║
║  │                                                       │   ║
║  │   🚨 /etc/passwd                  [ ⨉ ] BLOCKED: 403  │   ║
║  │                                                       │   ║
║  ╰───────────────────────────────────────────────────────╯   ║
║                                                              ║
║  ▪ Engine drops all path traversal requests (../)            ║
║  ▪ Symlinks are resolved prior to boundary validation        ║
╚══════════════════════════════════════════════════════════════╝

The Engine resolves all symlinks before executing. If an AI hallucinates a path traversal, the system structurally drops the request.

2. The Bridge (Chrome Extension)

A web extension that sits quietly on the right side of the Gemini window. Building this was an architectural challenge. Because Gemini streams text, scraping the DOM naively crashes the parser with incomplete JSON. I had to build a MutationObserver that waits for the absolute "completion" state of the chat UI before parsing. It captures the AI's outputs, relays them to my local engine via a Background Worker (to bypass strict browser CORS restrictions), and injects the results back into the chat.

3. The Protocol (Documentation & Rules)

A deterministic system prompt fed to Gemini at the start of every chat. This acts as the "Operating Manual." LLMs are chaotic; they need boundaries. The protocol forces Gemini to use a strict JSON schema instead of conversational markdown.

Demo

Here is a look at the complete, fully-boxed architectural flow of Kizuna:

╔══════════════════════════════════════════════════════════════╗
║ ▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ BROWSER ENVIRONMENT ▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒ ║
║                                                              ║
║  ┌────────────────┐   [ DOM ]     ┌───────────────────────┐  ║
║  │  🧠 Gemini UI  │ ◀───────────▶ │  🧩 Chrome Extension   │  ║
║  └────────────────┘               └───────────────────────┘  ║
║          ▲                                    │              ║
║          │ (Injects UI Data)                  │ (WebSockets) ║
╠══════════╪════════════════════════════════════╪══════════════╣
║          │                                    ▼              ║
║  ┌───────┴────────────────────────────────────┴───────────┐  ║
║  │  💻 KIZUNA ENGINE (Local Daemon :8080)                 │  ║
║  │                                                        │  ║
║  │  ╭─────────────────╮          ╭─────────────────────╮  │  ║
║  │  │ 🛡️ Path Sandbox │ ───────▶ │ 🛠️ Tool Dispatcher   │  │  ║
║  │  ╰─────────────────╯          ╰───┬──────┬──────┬───╯  │  ║
║  │                                   │      │      │      │  ║
║  │                                [Read] [Write] [Git]    │  ║
║  │                                   │      │      │      │  ║
║  │                                 ╭─┴──────┴──────┴─╮    │  ║
║  │                                 │ 📂 Local Storage│    │  ║
║  │                                 ╰─────────────────╯    │  ║
║  └────────────────────────────────────────────────────────┘  ║
║                                                              ║
║ ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ LOCAL ENVIRONMENT ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ ║
╚══════════════════════════════════════════════════════════════╝

When I ask Gemini to "Update the database connection," it doesn't write me a markdown tutorial. It outputs its intent directly:

{
  "action": "patch_file",
  "path": "src/config.js",
  "search": "localhost",
  "replace": "process.env.DB_HOST"
}

The extension picks this up, the engine verifies the path hasn't escaped the workspace, the file is safely patched, and the success message is injected directly back into the Gemini chat or I pasted it manually sometimes.

What I Learned

Through this work, I learned to question the constraints of LLMs first, not treat them as assumptions.

Abstract Syntax Trees (ASTs) vs. Raw Text

The naive approach to the context window is to just send the AI the whole file. But why send a 1,000-line file when the AI just needs to know what the file does? Sending raw text is a massive waste of tokens and scatters the model's focus.

I built a tool into the engine that parses code into an Abstract Syntax Tree (AST). Instead of returning raw code, it strips the implementation logic and returns a "Skeleton" of class names, imports, and function signatures.

╔══════════════════════════════════════════════════════════════╗
║ ░░░░░░░░░░░░░░░ AST PARSER ROUTING LOGIC ░░░░░░░░░░░░░░░░░░░ ║
╠══════════════════════════════════════════════════════════════╣
║                 [ 📄 Gemini Requests File ]                  ║
║                              │                               ║
║                              ▼                               ║
║                  ╭───────────────────────╮                   ║
║                  │  Is File > 300 Lines? │                   ║
║                  ╰───────────┬───────────╯                   ║
║                              │                               ║
║            ┌───( YES )───────┴───────( NO )────┐             ║
║            │                                   │             ║
║            ▼                                   ▼             ║
║    ╭───────────────╮                   ╭───────────────╮     ║
║    │  🌳 AST Parse │                   │  📝 Raw Read   │    ║
║    ╰───────┬───────╯                   ╰───────┬───────╯     ║
║            │                                   │             ║
║    ╭───────▼───────╮                   ╭───────▼───────╮     ║
║    │ ▪ Classes     │                   │ ▪ Full Logic  │     ║
║    │ ▪ Signatures  │                   │ ▪ Implement.  │     ║
║    │ ▪ Docstrings  │                   │ ▪ Variables   │     ║
║    ╰───────┬───────╯                   ╰───────┬───────╯     ║
║            │                                   │             ║
║    █▓▒░ Token Cost: ~5%                Token Cost: 100% ░▒▓█ ║
╚══════════════════════════════════════════════════════════════╝

To make this work, I learned to rely on highly descriptive function names and docstrings. With good docstrings, the AI could understand the codebase's intent just by pinging the AST, mapping out the architecture flawlessly.

Sandboxing and Heuristics

I actually used Gemini itself to help create a dataset filtering hundreds of developer CLI commands into "safe" and "harmful" categories. This allowed me to build heuristic judgments into the local sandbox, entirely disabling shell: true evaluations in Node.

The Ultimate Safety Net (Auto-Commits)

You can never trust an AI blindly. I didn't want it making untracked changes. I built a feature into the engine that automatically runs a local git commit after every (almost) single file change and kept git files backed up when messing around with git commits and refactoring. If the AI messed up, I had an instant, local undo button. Git became the AI's eyes and my safety net.

Google Gemini Feedback

I think of designing with AI in two stages: when the system hums, and when the material fights back.

✅ The 70% Magic

When it worked, it felt like magic. About 70% of the time, the system operated perfectly. Reading code worked phenomenally well. Gemini 2.5 Pro could absorb the AST skeletons, navigate the directory, and reason about system design with striking clarity.

⚠️ When the material fights back (The 30% Chaos)

The remaining 30% of the time was a battle against the realities of the medium: hallucinations, syntax errors, and context degradation.

1. The Writing Problem: Surgical Diffs vs. End-to-End

While reading was elegant, writing was clumsy. Gemini 2.5 Pro struggled heavily with surgical, line-by-line insertions. Standard diff patching is notoriously flaky with LLMs—they hallucinate line numbers or forget indentation. When I asked for complex changes across a large file, it would misplace the code entirely.

The Workaround: The solution wasn't to push harder; it was to change the abstraction. I stopped asking for line manipulations. Instead, I updated the protocol to force verbatim block patching or end-to-end rewrites of the entire file. The search block had to match the local file exactly, down to the space, or the Engine rejected it.

2. Amnesia and The Autonomous Self-Healing Loop

Sometimes, the AI would just forget its own rules. It would hallucinate bad JSON syntax (trailing commas, unescaped quotes) or start outputting raw markdown. If I fixed it manually in my IDE, the sync between the "brain" and the "hands" broke.

The Workaround: I built an Autonomous Self-Correction Loop. If the Chrome extension failed to JSON.parse() the output, it automatically generated an error payload and injected it directly back into the chat. Gemini would immediately apologize, fix the syntax, and re-emit the tool call without me typing a single word and if this was too often then I inserted Protocol docs again.

╔══════════════════════════════════════════════════════════════╗
║ ░░░░░░░░░░░░░ AUTONOMOUS SELF-HEALING LOOP ░░░░░░░░░░░░░░░░░ ║
╠══════════════════════════════════════════════════════════════╣
║                                                              ║
║   [🧠 Gemini UI]         [🧩 Extension]          [💻 Engine] ║
║         │                      │                      │      ║
║         │ 1. Invalid JSON      │                      │      ║
║         │ ───────────────────▶ │                      │      ║
║         │                      │ ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓   │      ║
║         │ 2. Catch Error       │ ▓ JSON Parse Fails ▓ │      ║
║         │ ◀─────────────────── │ ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓   │      ║
║         │                      │                      │      ║
║         │ 3. Fixes Syntax      │                      │      ║
║         │ ───────────────────▶ │                      │      ║
║         │                      │ 4. Valid Request     │      ║
║         │                      │ ━━━━━━━━━━━━━━━━━━━▶ │      ║
║         │                      │                      │      ║
║         │                      │  ╭────────────────╮  │      ║
║         │                      │  │ ⚙️ Validates   │  │      ║
║         │                      │  │ ⚡ Executes    │  │      ║
║         │                      │  ╰────────────────╯  │      ║
║         │                      │ 5. Returns Data      │      ║
║         │                      │ ◀━━━━━━━━━━━━━━━━━━━ │      ║
║         │ 6. Inject Status     │                      │      ║
║         │ ◀━━━━━━━━━━━━━━━━━━━ │                      │      ║
║                                                              ║
╚══════════════════════════════════════════════════════════════╝

3. Context Degradation (The Final Boss)

This compounds when the conversation gets too long. When dealing with detailed codebases, the LLM eventually loses its understanding of the current situation. The context window fills up, attention mechanisms drift, and it makes bad decisions.

The Master Workaround: To tackle this, I created a specific "Summarization Prompt" within the Chrome extension. When a chat got too long and the AI started losing the plot, I didn't try to salvage it.

I would run the prompt, instructing Gemini to condense its current understanding of the architecture, the problem, and our progress into one cohesive document. I would then open a brand new chat, paste that summary along with my base JSON rules, and resume.

╔══════════════════════════════════════════════════════════════╗
║ ░░░░░░░░░ CONTEXT HANDOVER PROTOCOL (SUMMARIZATION) ░░░░░░░░ ║
╠══════════════════════════════════════════════════════════════╣
║                                                              ║
║  [ ⏳ STAGE 1: Degradation ]                                 ║
║   │                                                          ║
║   ╰─▶ 💬 Long Chat Session ──▶ ⚠️ Hallucinations Begin       ║
║                                                              ║
║  [ 💉 STAGE 2: The Handoff ]                                 ║
║   │                                                          ║
║   ├─▶ 1. Inject "Summarization Prompt" via Extension         ║
║   ╰─▶ 2. Gemini Generates Cohesive 'State Document' 🗂️        ║
║                                                              ║
║  [ ✨ STAGE 3: Resurrection ]                                ║
║   │                                                          ║
║   ├─▶ 1. Close Degraded Session 🗑️                            ║
║   ├─▶ 2. Open Empty Chat Session 🆕                          ║
║   ├─▶ 3. Inject [ Base Rules + State Document ] 📥           ║
║   ╰─▶ 4. Fresh AI Instance with Perfect Context 🚀           ║
║                                                              ║
╚══════════════════════════════════════════════════════════════╝

It was like giving the AI a fresh cup of coffee and a perfect handover document.

Elevating the craft

The Kizuna system did not work flawlessly out of the box. But building it taught me that modern LLMs are not a pipeline; they are a search. They hallucinate, they forget, and they make errors.

But by building the right scaffolding—strict constraints, local safety nets, AST parsers, and clever workarounds like the self-healing loop and summarization prompt—you can harness them to do incredible things. It forced me to be intentional about how software is written, and it laid a profound foundation for the systems I want to build next.

GeminiChallenge #AI #WebDev #Productivity #Agents #Automation #SoftwareDesign

DialogueAI: Interactive Playground for assemblyai with automatic code generation

Omkar tripathi — Sat, 23 Nov 2024 13:47:18 +0000

This is a submission for the AssemblyAI Challenge: : Sophisticated Speech-to-text and No More Monkey Business.

What I Built : DialogueAI ( GITHUB )

I built DialogueAI, an interactive platform that leverages the powerful capabilities of AssemblyAI's sophisticated speech-to-text API and their LeMUR summarization model. The primary goal of this platform is to simplify the process for users who are new to these APIs, helping them overcome the steep learning curve typically associated with diving into new documentation.

Key Features of the Platform:

Interactive Playground: Users can explore and experiment with various API functionalities through an intuitive interface. Input boxes, selection options, model selection, and summary types are all easily adjustable.
Instant Results: With a single click, users can execute API calls and see the results immediately. This feature helps bridge the gap between learning and actual implementation.
Code Generation: For those who prefer to handle API calls manually, the platform generates the necessary code snippets, which can be directly run on their systems. This feature significantly reduces the time and effort required to understand and use the API.
Smart Summary Page: Similar to the main playground, this page offers various configuration options and examples to help users generate summaries of transcripts quickly. Users can also get the generated code to use by themselves.

By providing these features, the platform ensures that users can quickly and efficiently learn how to use AssemblyAI's APIs, reducing the frustration and time typically spent navigating complex documentation. This makes it an invaluable tool for developers and anyone looking to incorporate speech-to-text and summarization capabilities into their projects.

Journey

The inspiration for this platform came from my own experience when I first encountered AssemblyAI's API. I found it a bit confusing to get started with the documentation and the API usage. So, I set out to solve this problem not just for myself but for everyone else who might face the same challenge.

Tech Used

Frontend: React, TypeScript, Tailwind CSS
API: AssemblyAI Speech-to-Text, LeMUR LLM model summary API
Animations: Framer Motion

Working Features

Interactive Speech-to-Text Configurations:
- Users can easily configure and experiment with various settings.
- Single Click Run: Execute the configuration and see results immediately.
- Single Click Code Generation: Generates the code based on the configuration for users to use directly.

Configurations Available:

API Key
Speech Model
Word Boost
Profanity Filter
Audio Range
Audio Intelligence
Summary Model
Summary Type

Interactive Summary Generation with LeMUR:
- Users can generate summaries with various options and configurations.
- Single Click Run: Instantly generate summaries.
- Single Click Code Generation: Provides the code for generating summaries.

Configurations Available:

API Key
Summary Type (Basic, Custom)
Transcript ID
Model
Prompt
Custom Prompt
Max Output Tokens (Example Pre-coded)

In Development

Chat with the Transcript: Using LeMUR API to enable interactions with the generated transcript.
Interactive Quiz Generation: Generate quizzes based on the transcript.

Journey

So far, I've successfully addressed the initial problem statements for the speech-to-text API and LeMUR summary model. This project has been incredibly exciting to work on, pushing the boundaries of what can be done with API interactions and user interface design.

Looking ahead, I plan to expand the platform to include interactive playgrounds and code generation capabilities for real-time APIs and more sophisticated use cases of LeMUR. This will further streamline the learning and implementation process for developers and enhance the overall user experience.