DEV Community: popiol

How I replaced LLM calls with coding agent calls and saved money

popiol — Fri, 10 Jul 2026 21:46:15 +0000

When building an AI agent, you need LLM calls. It can be done via a remote API or a local API, but either way you need to do it. Whether the agent is a simple conversational agent or a ReAct agent with a bunch of tools, whether it's using a complex graph or a simple RAG, it must be based on the concept of sending prompts to the language model.

But, what if we replace the language model with... another agent? Let's say we already have a smart agent with a bunch of tools that can handle complex problems. Why not use it to build a new agent on top of it? This way we can focus on the specific custom functionality we want to achieve, while already having the common functionalities covered by the underlying agent.

You might think this must be expensive. You get a better performance, so you have to pay for it, right? Well, not necessarily. The catch is that the coding assistants are actually surprisingly cheap when compared to API prices. They offer much more than raw LLM calls, they offer amazing agent functionality, but the cost is actually lower, and it's not a small difference. The cost of LLM calls per 1M tokens is usually between $2 and $7. For coding assistant subscriptions, it's a bit more tricky to calculate because you pay for monthly subscription, but it can be still converted to per 1M tokens cost, and from what LLM just told me it is around $0.08 to $2. That's a huge difference! And the complex agents are cheaper than raw LLM's!

according to ChatGPT:

Service	Cost per 1M tokens
Codex / ChatGPT coding plan	~$0.08
Cursor Pro	~$0.08–0.25
GitHub Copilot Pro	~$0.10–0.30
Claude Pro / Claude Code	~$0.74
GPT-5 API	~$2.1
Claude Sonnet API	~$4.2
Claude Opus API	~$7.0

according to Claude:

Service	Cost per 1M tokens
Haiku 4.5 (API)	$1.80
Sonnet 5 (API, intro thru Aug 31 '26)	$3.60
Sonnet 5 (API, standard)	$5.40
Opus 4.8 (API)	$9.00
Fable 5 (API)	$18.00
Claude Code — Pro	~$1.10–$2.15 (est.)
Claude Code — Max 5x	~$2.10–$4.15 (est.)
Claude Code — Max 20x	~$2–$4 (est.)

So, why not use the coding agents to build new agents? The only problem is that they don't have API. But, this is a problem that can be solved by writing a simple wrapper to the agent that will communicate with it via CLI. We just have to install the agent on the machine from which we want to call it and use the wrapper. Simple and brilliant. Here's the code:

"""Claude Code CLI — complete integration module.

Provides plain-text and structured LLM calls via the local `claude` binary,
plus a LangChain-compatible adapter for use with make_llm().

Requires the `claude` binary on PATH.
"""

import json
import subprocess
from typing import Generic, TypeVar

from langchain_core.messages import HumanMessage, SystemMessage
from pydantic import BaseModel

T = TypeVar("T", bound=BaseModel)

_TIMEOUT = 120


# ── Core subprocess layer ─────────────────────────────────────────────────────


def cli_call(system: str, user: str) -> str:
    """Run a plain-text prompt through the Claude CLI and return the response."""
    return _run(system=system, user=user)


def cli_call_structured(system: str, user: str, output_type: type[T]) -> T:
    """Run a structured prompt through the Claude CLI and parse into output_type."""
    schema = json.dumps(output_type.model_json_schema())
    raw = _run(system=system, user=user, json_schema=schema)
    return output_type.model_validate_json(raw)


def _run(system: str, user: str, json_schema: str | None = None) -> str:
    cmd = [
        "claude",
        "--print",
        "--no-session-persistence",
        "--tools", "",
    ]
    if system:
        cmd += ["--system-prompt", system]
    if json_schema:
        cmd += ["--output-format", "json", "--json-schema", json_schema]
    cmd.append(user)

    result = subprocess.run(
        cmd, capture_output=True, text=True, encoding="utf-8", timeout=_TIMEOUT
    )
    if result.returncode != 0:
        stderr = (result.stderr or "").strip()
        raise RuntimeError(f"claude CLI exited {result.returncode}: {stderr}")

    stdout = (result.stdout or "").strip()
    if not stdout:
        raise RuntimeError("claude CLI returned empty output")

    if json_schema:
        envelope = json.loads(stdout)
        payload = envelope.get("result", stdout)
        return payload if isinstance(payload, str) else json.dumps(payload)

    return stdout


# ── LangChain-compatible adapter ──────────────────────────────────────────────


class CliResponse:
    def __init__(self, content: str) -> None:
        self.content = content


class ClaudeCliStructured(Generic[T]):
    def __init__(self, output_type: type[T]) -> None:
        self._type = output_type

    def invoke(self, messages: list) -> T:
        system, user = _extract_messages(messages)
        return cli_call_structured(system=system, user=user, output_type=self._type)


class ClaudeCli:
    """LangChain-compatible adapter that delegates to the claude CLI binary."""

    def invoke(self, messages: list) -> CliResponse:
        system, user = _extract_messages(messages)
        return CliResponse(content=cli_call(system=system, user=user))

    def with_structured_output(self, output_type: type[T]) -> ClaudeCliStructured[T]:
        return ClaudeCliStructured(output_type)


def _extract_messages(messages: list) -> tuple[str, str]:
    system = ""
    user = ""
    for m in messages:
        if isinstance(m, SystemMessage):
            system = m.content if isinstance(m.content, str) else ""
        elif isinstance(m, HumanMessage):
            user = m.content if isinstance(m.content, str) else ""
    return system, user

How I improved BMAD with one simple skill

popiol — Sun, 05 Jul 2026 09:26:52 +0000

When asked how I evaluate BMAD recently I said haven't quite decided yet, but it definitely has strong sides and makes development faster. Of course, it dependss what you are currently work on. If you work in scrum and handle single tasks from Jira, you probably won't fin BMAD useful in any way. It's strength is in handling large projects which have to be taken from brainstorming, through requirement specification to proper design and implementation plan. BMAD does really good job in asking the right questions and pushing you into providing all necessary details. The only thing I found a bit discouraging was the boring implementation phase where you just keep clicking approve.

I was recently pointed to Superpowers framework, which apparently deosn't have that problem and does the implementation part without human interaction. I tested it, but to be honest I wasn't impressed. In my case it didn't work so good. The whole planning phase comes down to brainstorming and implementation plan. This is clearly not meant to be handling large complex projects. In BMAD you have several distinct steps where each of them is handled with care and brings value to the final design. First, the brainstorming in BMAD pushes you into providing all available information, any documents you may already have and then asks a bunch of questions. Superpowers is more like: give me a brief intro and I will make up the rest. In BMAD, brainstorming is just the start, then you go to PRD, UX and architecture. You have to really spend some time to get it right, but that's exactly the real value, you spend the effort on planning, but save long hours of searching for bugs in the implementation or re-implementing the solution.

Having said that, I am going to keep using BMAD, but there is still the issue I mentioned, and it has to be taken care of. The implementation itself is a pain. It's not just the approval thing, this can be fixed easily, by running cluade with --dangerously-skip-permissions, which is not as dangerous as it sounds. If you just have the agent write some code, there's no much risk it will do any harm to your system. In worst case it can write bad code - I can take that risk. But that's not the only problem. The whole imlpementation consists of multiple cyclic steps:

create story
dev story
commit and push

It gets more complicated if you want to manually review each commit, but I figured it is more efficient to review final result rather than each step. However, you could still extend the steps with auto review:

create story
dev story
review and fix (repeat until no issues)
commit and push

So, these are the steps I want to run autonomously, it shouldn't stop to ask me for permissions, it should only stop if a real blocker is faced. The names of commit should refer to the story names. It should be able to continue from current state if interrupted. Also, I don't want all the stories to be implemented in a single session and have the context grow just to spend extra tokens on pointless context comperession. Each story should be implemented independantly with a new context.

The first thought was to write a python script that would run the steps in a loop. Seems to make sense, right? But luckly, I soon realized there is an even easier way to do it - a skill! I can simply describe to the agent what has to be done. There will be the main agent that will spawn sub-agents for each story and it will keep doing it in a loop until all stories are done. And that's the final result:

# BMAD Automation Loop

Orchestrate the full BMAD story implementation loop unattended. For each story, spawn a subagent to do the work in an isolated context, then loop until no new commits are made.

## Orchestrator steps (this session)

### 1. Check for a clean working tree (once, before the loop)

```bash
git status --porcelain
```

If any output is returned, **stop with failure**: "❌ Working tree is not clean. Commit or stash changes before running bmad-loop."

Log: "✅ Working tree is clean. Starting bmad-loop."

---

Repeat the following loop, keeping a counter starting at 1:

### 2. Record the current commit hash

```bash
git log -1 --format="%H" HEAD
```

Store this as `$HASH_BEFORE`.

Log: "🔄 Story #$COUNTER — running bmad-create-story..."

### 3. Spawn a subagent via the Task tool

Pass the following as the subagent prompt:

> You are implementing a single BMAD story. Complete these steps in order and do not stop until all are done:
>
> 1. Run `/bmad-create-story` and wait for it to complete fully.
> 2. Find the uncommitted story file:
>    ```bash
>    git diff --name-only HEAD -- '_bmad-output/implementation-artifacts/*.md'
>    ```
>    If no file is returned, stop — there is nothing to implement.
> 3. Extract the story title from that file:
>    ```bash
>    grep -m1 '^# Story' <file>
>    ```
>    Strip the leading `# ` to get `$STORY_TITLE` (e.g. `Story 5.5: Boss Match Flow & Chapter Unlock`).
>    If no matching line exists, stop with an error.
> 4. Log: "📖 Implementing: $STORY_TITLE"
> 5. Run `/bmad-dev` and wait for it to complete fully.
> 6. Run:
>    ```bash
>    git add .
>    git commit -m "$STORY_TITLE"
>    git push
>    ```
> 7. Log: "✅ Committed and pushed: $STORY_TITLE"

### 4. Check whether anything was committed

```bash
git log -1 --format="%H" HEAD
```

If the hash matches `$HASH_BEFORE`, log "🏁 No new commit detected — bmad-loop complete." and **stop**.
Otherwise increment `$COUNTER`, log "✅ Story #$COUNTER done." and go back to step 2.

## Rules

- The orchestrator never runs `/bmad-create-story` or `/bmad-dev` directly — always delegates to a subagent.
- The commit message must be exactly `$STORY_TITLE` — no prefix, no suffix, no surrounding quotes.
- Never proceed to the next story if the git push failed.
- If `/bmad-create-story` or `/bmad-dev` produces errors or unresolved TODOs, the subagent must stop without committing.
- If the specification is ambiguous or unclear, the subagent must never ask the user for clarification. Instead, stop with failure: "❌ Ambiguous specification: [reason]. Please clarify before re-running bmad-loop."

This works perfectly and makes my life easier. This way I was able to come down with implementation time from several days to several hours. This was the missing part of BMAD for me. I simlpy add it to .claude/skills/bmad-loop/SKILL.md, run claude with --dangerously-skip-permissions and trigger /bmad-loop. It only stops when running out of tokens or facing serious design flaw.

How I'm re-discovering computer science with LLM revolution

popiol — Tue, 30 Jun 2026 19:42:35 +0000

I've been in this business for years and the best (or worst?) thing is that once I gain enough experience to feel comfortable as an expert, something drastically changes and I feel like I have to start over again. It was like that when Python and Node started taking over C++ and Java market. Then it was like that when cloud platforms started taking over traditional hosting. Then machine learning started growing so much I had to keep up. Now, traditional ML seems passe. What's next? Quantum algorithms?

Learning your job again every couple of years is a bit concerning, but as much exciting. It's not a steady improvement, it's a constant fight. The current revolution gives great opportunities, but in the same time if you don't take this opportunity, you may stay behind. How not to? This article is not an answer, but it is an introduction to how I cope with it and stay competitive.

For the past couple of years I have been building ML models, experimenting with them, testing various model types, structures, parameters, validating, following best practices, building pipelines. I have been doing that at work, but I have also been doing that within my personal projects, which is my secret weapon. Personal projects is something I have been doing forever. I love to construct things that don't have to generate profit, but they help you answer the question 'what happens if I do that?'. This is the way to stay in touch with reality. When you join new teams and projects there are always great expectations, but the projects are not always fancy, ambitious, and satisfactory. Personal projects ensure you are always there with the global trends, not just the local needs. They let you invent new solutions, not just implement standards. They let you experiment on a level unreachable in commercial work.

So, how does the LLM revolution change the way I work on my personal projects? The simplest answer would be that they do the job for me, and they do it fast, but that's not quite accurate. When I know very well how to solve a problem, I would argue I can do it even faster than a coding agent because I don't need to look for the solution, I know very well how the code should look like, I can instantly write it and it's already good without reviewing and fixing. When I delegate it to a coding agent, it often misunderstands the task, goes wrong direction, chooses solutions not in line with my instinct and tends to create messy code, by for instance surrounding everything with try except blocks :/

But, the real advantage starts when you want to build a complex solution that goes beyond your expertize. This is where a personal project can stop being just a small demo and can become a fully functional solution to a real problem. For me, the part I usually avoided as a developer is UI. It takes time, it's boring, it's complicated, but not in a way I want it to be complicated. Frontend code is so messy, it doesn't look to me like a code written and maintained by a human. With LLM's it finally starts to make sense - you don't care it's messy, you don't care it's impossible to read it and review it, you don't have to do that at all. You just look at the UI itself, and see directly if the result is what you expected. That's beautiful. Frontend work always seemed to me like something only machines can do, and finally they can.

Another advantage is planning. Working with the coding agent by directly asking it to handle a complex project or even a complex task is naive and leads to a waste of time on manual code review, manual testing, bug fixing and overall frustration. What I like most about LLM's is that they are great conversation partners. They don't have ego's, don't interrupt you, have full commitment, and if your idea is stupid, they will gently suggest it, but not reject it. Sometimes they agree too easily, which is why it's important to remind them occasionally to be more critical. This great conversational skill is perfect for brainstorming and architecture design. And now the best part - having a good design doc, they can do the job practically alone. You don't need to follow every step they make and carefully review it. Most of mistakes the agents make is because of lack of clarity in specification. The most irritating part is when the agent writes almost perfect code, except for a few details that are wrong, and fixing these few issues turns out to be so difficult for the agent, you end up spending more time on prompting than you would on fixing it manually. But, if you invest more time in design, then you can just relax, watch the magic, and don't care that much about details, just review the final result. I am using BMAD framework to achieve that effect, but I had to improve it with additional missing skill - I will write about it on another occasion.

Careful planning, already allowed me building more and better, but that wasn't enough, I had to take it to another level. Planning still requires spending significant time on specifying requirements every time we want to develop a large feature. That makes sense if we have one specific thing to build. But that's not quite the case in ML, where your work is mostly about experimenting, coming up with ideas, testing them, observing, comparing, making conclusions and slowly improving the solution. The standard plan and implement approach is still very much manual. But, why not let the agent come up with new ideas and test them without human intervention? This way we can take advantage of the two great concepts: memory and agentic loop. The agent remembers the long-term plan, creates ideas, scores them, picks the most promising ones, tests them, makes conclusions, updates the plan and starts again. Each idea is materialized as new code written by the agent. This is like a new reinforcement learning method, where instead of updating model weights, we update the code. That's huge.

Another big thing in agents world are tools and skills, but how to use them to take creative process to yet another level? Which tools can change the rules? Typically, we can use it to give our agent access to Github, Jira, some database, some docs, etc. But that's just making work a bit easier. A real game changer is using tools in a more specific use case, such as graphic design. I have no idea about graphic design, and normally I wouldn't even think about it, but with Gen AI, why not? Graphic design is available for everyone now. However, it is much more tricky than with LLM's. Language models reached the level where they understand you most of the time perfectly, sometimes I feel like they understand my ideas better than I do. With image generation we're not there yet tough. Image generation is time consuming and frustrating. I don't know how pro designers work and use the models, I'm sure they do great and can achieve amazing results, but for me it is pain. I don't want to spend time on graphic design, but do want to be able to create a simple 2D game with nice characters and nice locations. So, here's the idea: why not let LLM use an image generation model as a tool? Why not let LLM do the cumbersome job of prompt engineering? If the agent can build backend and frontend, it should be able to handle graphics too. This way we could create a comprehensive framework for building games. Isn't that exciting? This is something I'm currently experimenting with and will surely write about it.

One more thing that boosts my productivity, are internet communicators. They are perfect for keeping in touch with the agent from everywhere, anytime with a mobile phone. It can be used to monitor jobs, handling maintenance tasks, basically anything the agent can normally do, and you don't even have to open your laptop. That's another subject worth exploring and I will post some details on how I'm doing.