The latest articles on DEV Community by Karneeshkar V (@watninja68). https://dev.to/watninja68 https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3283983%2F163455f4-e40d-4eed-a000-40e75cbf35c9.jpeg https://dev.to/watninja68 en Karneeshkar V Sat, 26 Jul 2025 08:03:34 +0000 https://dev.to/watninja68/the-polymath-tool-for-all-your-audio-and-document-needs-1ej https://dev.to/watninja68/the-polymath-tool-for-all-your-audio-and-document-needs-1ej <p><em>This is a submission for the <a href="https://dev.to/challenges/assemblyai-2025-07-16">AssemblyAI Voice Agents Challenge</a></em></p> <h2> What I Built </h2> <p>I built a <strong>Command-Line Interface (CLI) tool</strong> designed to help users manage their <strong>medical and legal conversations</strong> more effectively.</p> <p>This tool can <strong>transcribe audio files or calls</strong> with your doctor or Financial advisor, then organize and retrieve relevant insights to assist in decision-making.</p> <p>The idea stems from a personal pain point — often during important medical or legal discussions, I found it difficult to:</p> <ul> <li>Ask detailed follow-up questions</li> <li>Recall key points accurately</li> <li>Understand complex terminology on the spot</li> </ul> <p>By using <strong>AssemblyAI’s accurate transcription</strong>, especially for <strong>domain-specific (medical/legal) vocabulary</strong>, the project came to life </p> <p>All the CLI commands and flags can be found in the README.md<br> To set it up you will need this in your .env file<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code><span class="nv">ASSEMBLY_AI_API_KEY</span><span class="o">=</span><span class="s2">""</span> <span class="nv">OPENAI_API_KEY</span><span class="o">=</span><span class="s2">""</span> <span class="nv">QDRANT_URL</span><span class="o">=</span><span class="s2">""</span> </code></pre> </div> <p>Make sure to run Qdrant in your local system </p> <h2> Demo </h2> <p><a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ft8kb8xkbp88i31v517oc.png" class="article-body-image-wrapper"><img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Ft8kb8xkbp88i31v517oc.png" alt=" " width="800" height="223"></a></p> <p>Using Assembly AI for transcription and injecting it to rag </p> <p><a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fbaj4rd69kp0idspihnov.png" class="article-body-image-wrapper"><img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fbaj4rd69kp0idspihnov.png" alt=" " width="800" height="290"></a></p> <p><a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fpdvp4cx7f2yzoveq3rl1.png" class="article-body-image-wrapper"><img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2Fpdvp4cx7f2yzoveq3rl1.png" alt=" " width="800" height="320"></a></p> <p>Using memory from past call from doctor</p> <h2> GitHub Repository </h2> <p><a href="https://github.com/KarneeshkarV/-AssemblyAI-Domain-Expert-Voice-Agent" rel="noopener noreferrer">https://github.com/KarneeshkarV/-AssemblyAI-Domain-Expert-Voice-Agent</a></p> <h2> Technical Implementation &amp; AssemblyAI Integration </h2> <ul> <li>Built using the <strong><a href="https://github.com/agno-agi/agno" rel="noopener noreferrer">Agno agent framework</a></strong> </li> <li>Each domain-specific agent (medical or legal) is powered by a <strong>team of sub-agents</strong> <ul> <li>One for RAG (retrieval)</li> <li>One for memory/context management</li> <li>One for web search and knowledge lookups</li> <li>So on ....</li> </ul> </li> <li>I used <strong>OpenAI models</strong> in the primary implementation due to cost-effectiveness, though I found <strong>Claude models</strong> to perform better in tool use during testing</li> <li>Made some audio optimizations to effectively use TTS credits </li> <li>Core transcription powered by <strong>AssemblyAI</strong>, enabling robust handling of domain-specific vocabulary <h2> Future Work </h2> <p>I had plans to:</p> </li> <li>Make the entire injecting of data more easier and user Friendly</li> <li>Integrate <strong>SIP Sorcery</strong> for capturing and analyzing VoIP call streams</li> <li>Add another specialized agent focused on <strong>legal document processing</strong> </li> </ul> <p>However, due to my time constraints — they remain on my Todo list!</p> <p>I am all hears to know how I can improve this project</p> devchallenge assemblyaichallenge ai api