DEV Community

Mininglamp
Mininglamp

Posted on

From meeting speech to task dispatch: wiring on-device ASR into an AI collaboration workflow

From meeting speech to task dispatch: wiring on-device ASR into an AI collaboration workflow

We recently shipped an integration between Octic, our on-device AI recorder, and Octo, the open-source collaboration platform we use for agent-orchestrated work. The goal was straightforward: someone says something actionable in a meeting, and a Loop gets created and assigned without anyone lifting a finger.

This post walks through the pipeline end to end: audio capture, speech recognition, intent extraction, task creation, and agent execution.

The Problem

Our team runs 6+ meetings a day. Decisions get made, action items get called out, and then... they die in someone's notebook. We tracked it for two weeks. Roughly 40% of verbally agreed tasks never made it into any tracking system. People just forgot to write them down, or wrote them down and forgot to transfer them.

We wanted to close that gap automatically.

Architecture Overview

The pipeline has five stages:

Mic → ASR (on-device) → NLU / intent extraction → Octo Loop creation → Agent execution
Enter fullscreen mode Exit fullscreen mode

Each stage hands off a structured artifact to the next. No monolith, no single model doing everything.

Stage 1: On-Device ASR with Octic

Octic is a hardware recorder built by the same team behind the Lingting device, which was designed for noisy-environment speech capture. The key specs that matter for this pipeline:

  • ASR runs locally on the device. Audio never leaves the room.
  • Speaker diarization is built in, so you get per-speaker transcripts, not a single blob.
  • Real-time transcription with 7 built-in skills for in-meeting assistance.
  • Personalized ASR error correction, which actually matters when your codebase has project-specific jargon.

The output is a timestamped, speaker-attributed transcript in JSON. Each segment looks roughly like:

{
  "speaker": "fanrong",
  "start_ms": 124500,
  "end_ms": 131200,
  "text": "Let's have the agent handle the weekly report generation, assign it to Elva's agent by Friday"
}
Enter fullscreen mode Exit fullscreen mode

Privacy note: because ASR is on-device, we can use this in client meetings without worrying about data exfiltration. That was a hard requirement for us.

Stage 2: NLU and Intent Extraction

Raw transcripts are noisy. People repeat themselves, correct themselves mid-sentence, go on tangents. You can't just regex for "assign X to Y."

We run a lightweight NLU pass over the full transcript. The model extracts:

  • Action items: things someone committed to doing or asked someone else to do
  • Decisions: conclusions the group reached
  • Open questions: things raised but not resolved

For action items specifically, we extract:

  • What needs to be done (the task description)
  • Who is responsible (mapped to Workspace members)
  • Deadline if mentioned
  • Any acceptance criteria mentioned verbally

This runs as a post-meeting batch job. We experimented with real-time extraction during the meeting but found that waiting until the end produces much better results, because context from later in the conversation often clarifies earlier ambiguous statements.

Stage 3: Mapping to Octo Loops

This is where Octo comes in. Octo is an IM-native collaboration platform designed for human-agent work. The core abstraction is a Loop: a work unit that goes from conversation to delivery, with an owner, deliverables, and acceptance criteria.

For each extracted action item, we create a Loop via the Octo API:

  • Title: the extracted task description, cleaned up
  • Owner: mapped to the responsible person's Agent (every team member has a digital Agent in Octo that inherits their authorizations and preferences)
  • Acceptance criteria: pulled from the transcript or inferred from context
  • Source context: link back to the meeting transcript segment, so anyone can trace why this Loop exists

The two-way link matters. When someone asks "why is my agent working on this?", the Loop shows the exact meeting moment where it was assigned.

Loops in Octo can be created two ways: manually through the UI, or via natural language. Our pipeline uses the API directly, but the natural language path is interesting for ad-hoc meeting follow-ups: you can literally type "create a loop for the quarterly review deck, assign to my agent" in the Workspace chat.

Stage 4: Agent Execution

Here's where it gets interesting. In Octo, a Loop owner can be an Agent, not just a person. Agents are digital workforce clones: they inherit your authorizations, carry your preferences, and can autonomously pick up and execute assigned work.

When a Loop is created with an Agent as the owner:

  1. The Agent gets notified through the IM channel (Octo uses a three-tier escalation system to make sure notifications land)
  2. The Agent reads the Loop brief, including the acceptance criteria and any attached context
  3. The Agent executes the work. For a weekly report, that might mean pulling data, writing the doc, formatting it, and attaching the output to the Loop.
  4. The human who spawned the agent reviews and accepts or rejects the deliverable.

Every acceptance or rejection gets stored as a Preference: a behavioral rule that the Agent references on future tasks. Over time, the Agent learns things like "this person prefers bullet points over paragraphs" or "always include the raw data table alongside the summary."

Stage 5: The Feedback Loop

Rejected deliverables go back to the agent with specific feedback. The agent revises and resubmits. This creates an iterative cycle that's fully tracked in the Loop's timeline: brief, discussion, output, feedback, revision, acceptance.

A year from now, someone can open any Loop and see the full chain: what was said in the meeting, what task was created, what the agent produced, what got sent back, and what was finally accepted.

What We Learned

Speaker diarization quality is critical. Without reliable speaker attribution, you can't map "I'll handle this" to a specific person. Octic's diarization works well in rooms with 3-6 people, which covers most of our meetings.

Post-meeting batching beats real-time. Real-time intent extraction sounds cool in a demo but produces too many false positives in practice. People say things like "we should probably..." without meaning it as a commitment.

Acceptance criteria extraction is the hardest part. People rarely state explicit acceptance criteria in meetings. We default to a summary of the task context and let the Loop owner refine it before the agent starts work.

Preference accumulation is surprisingly useful. After about two weeks of active use, agents started producing first drafts that needed fewer revisions. The Preference system in Octo compounds over time.

Orchestration Modes

Octo supports six orchestration modes for multi-agent collaboration: Solo, Roundtable, Critic, Pipeline, Split, and Swarm. For meeting-generated tasks, we mostly use Solo (single agent, simple task) and Pipeline (multi-step, ordered handoffs). The orchestration mode is selected based on task complexity at creation time.

Stack

  • Audio capture: Octic hardware recorder
  • ASR: On-device (Octic built-in)
  • NLU: Post-meeting batch extraction
  • Task management: Octo Loops
  • Agent runtime: OpenClaw, connected to Octo via the Agent framework
  • Feedback loop: Octo Preference system

Open Questions

We're still working on a few things:

  • How to handle ambiguous assignments ("someone should look into this") without creating garbage Loops
  • Cross-meeting context: when a task from Meeting A gets updated in Meeting B, should the Loop be updated or should a new one be created?
  • Latency optimization for the NLU stage, since people want their tasks created within minutes of the meeting ending, not hours

If you're building something similar or have experience wiring ASR into structured workflows, I'd be interested in hearing what worked for you.

The Octo repo is at github.com/Mininglamp-OSS. The Loop and Agent systems are the most relevant parts for this use case.

Top comments (0)