From Idea to Terminal: How I Built My First LLM-Powered CLI Tool (And What I'd Do Differently)

I love the terminal. There's something satisfying about typing a command and watching things happen. But lately, I've been annoyed by how much context-switching I do when I'm working on AI projects. I'd have a question about my code, or need a quick summary of a file, and I'd end up opening a browser, going to some chat interface, pasting content, and waiting.

So I decided to build what I thought would be a simple weekend project: an LLM-powered CLI tool that lives right in my terminal. No browser, no copy-paste, just ask followed by my question.

Spoiler: it wasn't as simple as I expected. But it was one of the most educational things I've built this year.

Here's how I did it, what actually worked, and what I'd do differently if I started today.

Why a CLI Tool for LLMs?

Before diving into code, let me explain why this even made sense.
I spend most of my day in the terminal: running tests, checking logs, managing containers, and messing with configs. When I'm in that flow, opening a browser breaks my concentration. I wanted something that:
Works where I already am: the terminal.

Understands my project context (files, errors, logs).
Gives me quick, focused answers without turning into a distraction machine.
There are already great LLM tools out there, but I wanted something lightweight, customizable, and open source that I could extend as I go. Plus, building it forced me to think seriously about how to use LLMs responsibly in real workflows.

The First Version: "Let's Just Call the API"

My initial approach was embarrassingly simple:
Create a new Python project.
Use argparse to parse command-line arguments.

Call an LLM API with the user's prompt.
Print the response.
Roughly, it looked like this:

``

ask.py (v0.1, very naive)

import argparse
import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def main():
parser = argparse.ArgumentParser()
parser.add_argument("prompt", nargs="+", help="Your question for the LLM")
args = parser.parse_args()

prompt = " ".join(args.prompt)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are a helpful assistant for a software engineer working with AI and open source."},
        {"role": "user", "content": prompt},
    ],
    temperature=0.3,
)

print(response.choices[0].message.content)

if name == "main":
main()
Then I could run:
$ ask "Summarize this error log" < error.log
``

It worked. Sort of.
I could ask questions, get answers, and it felt magical the first few times. But very quickly, I hit problems.
Reality Check: Where Things Went Wrong

1. No Context, Mostly Noise
   The tool had no idea what project I was in, what files mattered, or what I was actually trying to do. Every prompt started from zero. I found myself writing prompts like:
   "I'm in a Python project that uses FastAPI and PostgreSQL. Here's my error log. Explain what's going on and suggest fixes."
   That's a lot of repetitive context to type every time.

2. Token Bloat and Cost Surprises
   When I tried to "fix" the context problem by automatically sending entire files, my token usage exploded. One command costing a few cents doesn't sound like much—until you run 50 of them in a day.
   I had no tracking, no limits, and no idea how much I was spending until I checked my bill.

3. Streaming vs. Blocking UX
   The first version blocked until the full response was ready. For long answers, the terminal just… sat there. No feedback, no progress, nothing. It felt slow and clunky compared to the streaming chat interfaces I was used to.

4. No Memory of Past Conversations
   Every ask call was stateless. If I wanted to follow up—"Okay, but how would this change if I use SQLAlchemy 2.0?"—I had to re-explain everything.
   Version 2: Making It Actually Useful
   After the initial excitement wore off, I rebuilt the tool with a few key improvements.

Adding Project Context
I added a simple config file (ask.config.json) where I could define:
Project type (python, node, etc.)
Key directories to consider
Default system prompts
Then I wrote a small helper to read nearby files and inject them as context:
``

context.py (simplified)

from pathlib import Path

def gather_context(max_files=5):
cwd = Path.cwd()
files = list(cwd.glob("*.py"))[:max_files]
context_parts = []

for file in files:
    content = file.read_text(errors="ignore")
    context_parts.append(f"File: {file.name}

`
{content}
`")

return "

".join(context_parts)
In the main command:
context = gather_context()
user_prompt = " ".join(args.prompt)

full_prompt = f"Here is some context from my project:

{context}
``
User question: {user_prompt}"
Now I could just run:
$ ask "Why is this migration failing?"
and the tool would automatically include relevant Python files from the current directory.
Streaming Responses
To make the UX feel more responsive, I switched to streaming:

response =
``client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{"role": "system", "content": "You are a concise, practical assistant for a software engineer."},
{"role": "user", "content": full_prompt},
],
temperature=0.3,
stream=True,
)

for chunk in response:
content = chunk.choices[0].delta.content
if content:
print(content, end="", flush=True)

print() # final newline
Suddenly, the terminal felt alive. I could see the answer forming in real time, which made long responses way less painful.
Basic Cost Control
I added a rough token estimator and a --max-tokens flag:
parser.add_argument("--max-tokens", type=int, default=500)``
And logged approximate usage per call to a local file. Not perfect, but enough to stop me from blindly sending 10k-token prompts everywhere.

*If I were building this again now, with everything I've
*
learned, here's what I'd change:

1. Start With Open-Source Models First My first version assumed a commercial API. But for a CLI tool, especially one that might handle sensitive code, I'd start with open-source LLMs (via Ollama, LM Studio, or self-hosted options).

Benefits:
Better control over data and privacy.
Easier to reason about costs (or run locally for free).
More aligned with my open-source mindset.

I'd probably structure the tool to support multiple backends from day one:
``

provider abstraction (pseudo-code)

class LLMProvider:
def generate(self, prompt: str, **kwargs) -> str:
...

class OpenAIProvider(LLMProvider):
...

class OllamaProvider(LLMProvider):
``
Then let users choose via config or env vars.

1. Design for "Small, Focused Prompts" Instead of trying to send "everything just in case," I'd design commands around specific tasks:

ask error < error.log
ask summarize README.md
ask explain tests/test_auth.py::test_login_fail
Each command would have a tailored prompt template and a clear scope, keeping token usage low and answers focused.

1. Make Safety and Boundaries Explicit
   I'd build in guardrails from the start:
   A clear system prompt that discourages dangerous suggestions (e.g., "just drop this table").
   Warnings when commands might modify files or run code.
   An explicit "I'm an AI assistant, not a replacement for your judgment" vibe in the UI.
   Given my work in AI research and writing, I care a lot about responsible AI. A CLI tool that runs in your terminal should model that.

2. Add Conversation History (Even If Simple)
   I'd add a minimal session history:
   Store the last N turns in a local SQLite DB or JSON file.
   Allow ask --thread to continue a conversation.
   Show a short context summary before each new prompt.
   This makes follow-up questions natural without building a full-blown chat app.

3. Open Source It Earlier
   I waited until the tool felt "complete" before thinking about sharing it. That was a mistake.
   Releasing an early version on GitHub would have:
   Given me feedback on UX and features.
   Attracted contributors who care about CLI + AI.
   Forced me to write better docs and think more clearly about the design.
   If you're building something similar: ship v0.1 fast, then iterate publicly.

Key Takeaways
Building an LLM-powered CLI tool is a fantastic way to learn about prompts, context, tokens, and UX in a concrete project.
Your first version will be naive; that's fine. The learning comes from using it daily and feeling its pain points.
For developer tools, streaming, context awareness, and cost control are not optional—they're core to the experience.
Starting with open-source models and clear safety boundaries aligns better with how many of us actually work.
Shipping early and iterating in public beats waiting for "perfect."

What About You?
Have you built any LLM-powered tools, CLI or otherwise? What surprised you the most when moving from "cool demo" to "thing I actually use every day"?
If you'd like, I can share the current structure of my repo (without sensitive bits) or brainstorm ideas for your own CLI tool. Drop a comment with what kind of workflow you'd want to automate with AI—I'm curious what patterns others are seeing.