DEV Community

Aditya More
Aditya More

Posted on

Devlog - Skillviewer

#hackathon #systemdesign #devlog #ai

1. Foundation

I'll start off with the problem statement. The initial objective of the system was very simple, create an AI interviewer. This was motivated by Wellfound's AI interviewer, as that was the closest AI based interviewer I've had ever interacted with, there were others but they didn't feel that good like theirs'.

After thinking about the idea for a while, I thought let's experiment by letting users communicate with voice based interface by providing it the context in form of SKILL file at load time. My plain idea was that if in coding environment we could create SKILL file and then use that to do the work we wanted instead of writing entire prompt, why we can't do the same when we create an interviewer. This became the foundation for skillviewer, rest everything was just the system design that I worked upon for day & night.

2. Research

The first thing that was necessary after having the idea clear was to figure out the basics I needed to understand so I could really consider investing time in it. Since my idea was to take SKILL file and give it to a model, it was necessary to understand whether this was a sensible thing for the problem statement I was considering.

After researching a bit, I found out that SKILL file was ok for the system I was building, although there are other alternatives I'm considering like RAG ** for instance, but I wanted to build a prototype for the idea first, it's indeed possible to redesign the system with RAG, although I'd first investigate whether that's technically accurate and efficient for the purpose of this system.

So after reading out about SKILL file, I understood that SKILL uses something called as procedural knowledge

Procedural Knowledge:

  • It focuses on skills, actions & processes (for ex., how to tie shoes).
  • Specifically, it refers to habits, skills and techniques people pick up through practical application rather than textbook learning
  • Such skills transfer via conscious experiences rather than explicit instructional methods like classroom lectures / manuals
  • Ex., A new team of engineers won't know how to navigate company's OS until they start using it.

Read more at- https://www.notion.com/blog/procedural-knowledge

Declarative Knowledge

  • It involves the "how" part to perform a task
  • Insteead of learning through hands on experience, declarative learning uses classroom lectures

Why is SKILL file useful for our product

  • It's a structured technical document written specifically for AI Agents
  • Agents are increasingly capable but often they don't have context to work reliably
  • SKILLS solve that by packaging procedural knowledge & user specific context into portable, version controlled folders agents can load as required
  • For our product, interview skills are portable, reusable interview methodologies that’re useful to configure interview agent.

Features of SKILL.md

Domain Expertise in SKILL file- Capture specialized knowledge like data analysis pipelines, presentation formatting as reusable instructions & resouces
Reputable Workflows- Turns multi-step tasks into consistent procedures

How it works

  • A SKILL.md progresses through progressive disclosure in 3 stages
  • Discovery - At startup, only name & description is read so system knows when to load a particular SKILL
  • Activation - When a task matches skill description, agent reads entire SKILL.md in context
  • Execution - Agent follows instructions, execute bundled code OR referenced file if necessary

How to write SKILL.md

  • It's an operational guidance. The goal is to help AI agent execute tasks predictably

Define clean boundaries- Explain not only when it should be used but also when not to use
Add guardrails & common pitfalls- Defines limits. Negative constraint reduce subtle mistakes that'd otherwise require human review

Benefits of using SKILL file

  1. Reusability - Same skill can be reused across interview session
  2. Progressive disclosure - If we later incorporate skill library, skill system can expose metadata first
  3. Resources beyond instructions - An instruction is text only, but with SKILL we can scatter instruction across directory / package
  4. Versioning - Useful for evaluation
  5. Marketplace
    • We can build SKILL based marketplace
    • Anyone can practice for particular role at particular company

3. Existing solutions

After this, I needed to research existing solutions. There indeed was a solution for voice based AI by google - google live ai. But the problem with it was that it didn't fit within the constraints I had for hackathon for which I was making this project - it required gemini model >= 3.5 version.

Although it was really possible to have a workaround for that, but that was just a hack to get around with the constraints, so I designed the system in such a way to stay within constraints and create a rigid system without designing hacks.

4. SYSTEM DESIGN

1. Design #1

  • The initial idea was to use adk bidi streaming
  • The idea was to load SKILL file into agent & then communicate, since ADK + transport layer streaming was already provided out of the box - https://codelabs.developers.google.com/way-back-home-level-3/instructions
  • The only caveat was that it was using gemini live (live api) under the hood, and the latest gemini model for live was gemini 3.1
  • For the hackathon, gemini ≥ version 3.5 was a must.
  • So I needed to think for an alternative

But the important key takeaway from this was that I needed following features from live api in my application

  • Barge in
  • Audio transcriptions

You can check out to read more about live api here - https://ai.google.dev/gemini-api/docs/live-api

2. Design #2

  • For looking at alternative solution, I first needed to understand how gemini live actually does what it does (on a higher level)
  • There was a particular section in live api which was important for moving the project ahead -

https://ai.google.dev/gemini-api/docs/live-api#implementation-approach

  • The key takeaway was that live api could be integrated in one of the 2 ways -
    • server to server
    • client to server
  • The important keyword here was websocket connection, also the guide had the following diagram

!image.png

  • So my next objective was to somehow use something other than Live API to design the solution to this problem.
  • After exploring the solution offered by gemini live + adk, there was an important piece that was powering it up - it was realtime communication platforms - livekit, pipecat (daily), voximplant.
  • After researching a bit about one of the platforms - livekit; I realized I just needed communication layer offered by livekit and then plug it with ADK via some mechanism.

Then on further digging I realized I need a plug & play tool for:

  • VAD
  • audio transport
  • turn detection
  • interruptions / barge-in
  • stop TTS when interrupted
  • conversation state
  • streaming STT
  • streaming TTS
  • latency instrumentation
  • arbitary LLM/agent integration
  • This gave me candidate tools I could use for my solution

I used https://docs.livekit.io/agents/integrations/google/ as guide for developing my system, and used AI assistance in this part (livekit ai + chatgpt) to understand solution offered by them

My architecture became like this

flowchart TB

    %% =========================
    %% CLIENT
    %% =========================
    subgraph CLIENT["User / Client"]
        MIC["🎤 Microphone"]
        SPEAKER["🔊 Speaker"]
    end

    %% =========================
    %% REALTIME LAYER
    %% =========================
    subgraph LIVEKIT["LiveKit Agents — Realtime Voice Layer"]
        RTC["WebRTC / Audio Transport"]
        SESSION["AgentSession"]

        VAD["VAD"]
        TURN["Turn Detection"]
        INTERRUPT["Interruption / Barge-in"]

        STT["STT"]
        TTS["TTS"]

        OBS["Observability / Metrics"]
    end

    %% =========================
    %% AGENT LAYER
    %% =========================
    subgraph ADK["Google ADK — Agent / Orchestration Layer"]
        ADAPTER["Custom LiveKit ↔ ADK LLM Adapter"]

        AGENT["Interview Agent"]

        SKILL["SKILL.md\nInterview Rules"]
        STATE["Agent / Workflow State"]
        TOOLS["Tools"]
    end

    %% =========================
    %% MODEL LAYER
    %% =========================
    subgraph GOOGLE["Google AI"]
        GEMINI["Gemini Model\nGemini API / Vertex AI"]
    end

    %% =========================
    %% AUDIO INPUT
    %% =========================
    MIC -->|"User speech"| RTC
    RTC --> SESSION

    SESSION --> VAD
    VAD --> TURN
    TURN --> STT

    %% =========================
    %% AGENT FLOW
    %% =========================
    STT -->|"Final transcript"| ADAPTER
    ADAPTER --> AGENT

    SKILL --> AGENT
    STATE <--> AGENT
    TOOLS <--> AGENT

    AGENT --> GEMINI
    GEMINI -->|"Streamed text"| AGENT

    AGENT --> ADAPTER
    ADAPTER -->|"ChatChunk / text stream"| TTS

    %% =========================
    %% AUDIO OUTPUT
    %% =========================
    TTS --> RTC
    RTC -->|"Agent speech"| SPEAKER

    %% =========================
    %% INTERRUPTION
    %% =========================
    INTERRUPT -.-> SESSION
    INTERRUPT -.-> TTS
    INTERRUPT -.-> ADAPTER

    %% =========================
    %% OBSERVABILITY
    %% =========================
    SESSION -.-> OBS
    STT -.-> OBS
    ADAPTER -.-> OBS
    TTS -.-> OBS
    AGENT -.-> OBS

3. Design #3 (Pipecat ADK)

  • Livekit + custom adapter + ADK was rejected because it was perfect on paper and by communicating with AI, but it was technically challenging to implement.
  • The main problem was the adapter, I would have to write custom glue code and hope for it to work
  • So I was looking for any other way to build this instead.
  • With LiveKit other solution was not practical, so I moved to pipecat
  • Pipecat provided realtime audio pipeline
  • This made me curious to think whether there’s an existing pipecat + adk integration and indeed there was
  • I’d like to give a shoutout to https://github.com/recruit41/pipecat-adk here, because this project wouldn’t have been possible without their help.
  • They provided features such as
  • Interruption Handling
  • Context Management
  • State Management
  • Function Call Lifecycle
  • Custom Context Injection

I told my coding agent to write a script so I could communicate with it, and it wasn’t much of code to write. They had solved major problems I’d have to write myself with my custom livekit integration

Here’s the high level architecture I had in mind for this

flowchart LR
    USER["Candidate<br/>Browser"]

    VOICE["Realtime Voice Layer<br/>Pipecat + WebRTC<br/><br/>Audio input / output<br/>STT / TTS / VAD"]

    BRIDGE["ADK Bridge<br/>pipecat-adk<br/><br/>Realtime ↔ ADK"]

    AGENT["Interview Agent<br/>Google ADK<br/><br/>Interview logic<br/>Skills<br/>Session state<br/>Tools"]

    MODEL["Gemini 3.5+<br/>Vertex AI"]

    USER <-->|Realtime Audio| VOICE
    VOICE <-->|Streaming Text| BRIDGE
    BRIDGE <-->|ADK Events| AGENT
    AGENT <-->|Model Calls / Responses| MODEL
flowchart LR
    C["Candidate"]

    UI["Voice Interface<br/>Browser + WebRTC"]

    RT["Realtime Voice Engine<br/>Pipecat"]

    ADK["Interview Agent<br/>Google ADK"]

    AI["Gemini 3.5+<br/>Vertex AI"]

    DATA["Application State<br/>Sessions + Skills"]

    C <-->|Voice| UI
    UI <-->|Realtime Audio| RT
    RT <-->|Conversation| ADK
    ADK <-->|Reasoning| AI
    ADK <--> DATA

5. Agentic Authoring

  • From end user’s perspective, it’s tedious to write SKILL files / reference files manually
  • To solve this, my idea was to have agentic builder, where I’d have an AI prompt page & then under the hood this SKILL would be built as users required
  • So the skill part was handled by system and the interface was kept extremely simple for making it really intuitive & easy to use

6. Features

The project was only 50% done, because my first objective was to get a communication + intelligence layer set up, I wanted following features for the system I was designing

For the MVP, I had the following features in mind

  1. Voice based communication - Have bidirectional communication
  2. Skill loading - For demo, it’s loaded into agent’s memory, in prod app it’ll be stored & loaded via cloud storage
  3. UI - A prompt builder page where users can describe what they want to get interviewed on and then system plans it. Users can view this on canvas and then approve/edit/change it.
  4. Guardrails - To avoid platform abuse, I identified several guardrails
    • Interview scoped - Keep conversation relevant to interview
    • Instruction hierarchy - Prevent prompt injection / instruction injection
    • Sensitive content - Prevent Inappropriate / discriminatory Q’s
    • Internal information protection - Prevent SYSTEM prompt, rubric, tools, configurations to be leaked
    • Output safety - Prevent inappropriate / unsafe responses
    • Candidate abuse handling - Don’t reciprocate abusive / inappropirate interaction
    • Tool / action authorization - Prevent LLM from performing unauthorized app actions

7 - Submission & Demo

You can watch my pitch & demo below -
https://devpost.com/software/skillviewer?ref_content=user-portfolio&ref_feature=in_progress

Top comments (0)