Prompting Is Dead. Long Live the Loop.

Rajesh Kumar Kanumetta — Fri, 10 Jul 2026 10:20:13 +0000

A realistic first migration

Before (prompting):

Every PR: manually ask the agent to fix CI, re-prompt after each failure.
After (looping):

/loop 5m check my PR, address review comments, and fix failing CI

Add a skill that says: never mark CI fixed without re-running the full pipeline. Add /goal all checks green, stop after 10 tries for stubborn failures.

Run it for a day. Note where it stalls—too eager merges, wrong files touched, missed edge cases—and encode those failures into skills. That iteration is the product. The prompt was never the product.

The Bottom Line

Prompting isn’t worthless. It’s incomplete.
The best teams still write clear instructions—but they’ve stopped treating a single response as the finish line. They design loops: agents that gather context, act, verify, and repeat until a condition they didn’t have to guess at is satisfied.
Prompting is dead as the default paradigm for serious agentic work. Not because words stopped mattering, but because outcomes beat utterances.
The loop is here. Your job moved upstack: define what done means, how to verify it, when to run, and when to stop.
Start with one loop this week. Hand off one thing you used to re-prompt manually. Watch what breaks. Fix the system, not just the output.

That’s not prompt engineering. That’s engineering.

Inspired by Anthropic’s Claude Code team and their guide: Getting started with loops. Examples use Claude Code primitives (`/goal`, `/loop`, `/schedule`, skills, workflows); the loop mindset applies to any agentic coding tool.

Tags: AI, Agentic AI, Prompt Engineering, Claude Code, Software Development, Developer Productivity, LLM, Automation
Estimated read time: 8 min

Track Every LLM Token in Node.js with all-llm-token-tracker

Rajesh Kumar Kanumetta — Fri, 10 Jul 2026 07:36:09 +0000

Tags (pick up to 4)

node
javascript
openai
llm

(Alternates: typescript, npm, webdev)

"Track input/output LLM tokens in Node.js"

If you build with OpenAI, Anthropic, or any LLM API, you've probably wondered:

How many tokens did that request actually use?

Token usage drives cost, performance, and scaling. But in many Node.js apps, response.usage gets logged once and forgotten.

I built all-llm-token-tracker — a small npm package that tracks input and output tokens for every LLM call, with pluggable storage and built-in provider extractors.

npm: https://www.npmjs.com/package/all-llm-token-tracker

GitHub: https://github.com/rkanumetta/all-llm-token-tracker

The problem

Most integrations look like this:

const response = await openai.chat.completions.create({ ... });
console.log(response.usage); // logged, rarely stored

That works for debugging — not for production. You usually need:

Per-call input and output token counts
Persistent storage you can query later
Summaries by model, provider, or user
Something that works in dev and production

Many existing tools bundle pricing engines, MCP servers, or heavy infra. Sometimes you just want clean token tracking.

The solution

all-llm-token-tracker does one job well:

Feature	Supported
Manual token recording	✅
Auto-wrap LLM calls	✅
OpenAI extractor	✅
Anthropic extractor	✅
Memory storage	✅
File (JSON) storage	✅
MongoDB storage	✅
Query & summarize	✅
Browser dashboard	✅
Zero required deps	✅

Requirements: Node.js 18+

License: MIT

Install

npm install all-llm-token-tracker

MongoDB storage (optional):

npm install all-llm-token-tracker mongodb

Quick start: manual recording

import { createTracker } from 'all-llm-token-tracker';

const tracker = createTracker({ storage: 'memory' });

const record = await tracker.record({
  provider: 'openai',
  model: 'gpt-4o-mini',
  inputTokens: 120,
  outputTokens: 45,
  metadata: { userId: 'user-123' },
});

console.log(record.totalTokens); // 165

Each record gets a UUID, timestamp, and optional metadata.

Auto-track LLM calls (recommended)

Wrap your existing function — usage is recorded automatically:

import {
  createTracker,
  extractOpenAiUsage,
} from 'all-llm-token-tracker';
import OpenAI from 'openai';

const tracker = createTracker({
  storage: 'file',
  file: { filePath: './.llm-usage/usage.json' },
});

const openai = new OpenAI();

const { result, record } = await tracker.track(
  () =>
    openai.chat.completions.create({
      model: 'gpt-4o-mini',
      messages: [{ role: 'user', content: 'Hello!' }],
    }),
  {
    provider: 'openai',
    model: 'gpt-4o-mini',
    extractUsage: extractOpenAiUsage,
    metadata: { feature: 'chat' },
  }
);

console.log(result.choices[0].message.content);
console.log(record.inputTokens, record.outputTokens);

What happens:

Timer starts
Your LLM function runs
Usage is extracted from the response
Record is saved to storage
You get { result, record } back

No changes to your core LLM logic — just a wrapper.

Anthropic support

import { extractAnthropicUsage } from 'all-llm-token-tracker';

await tracker.track(
  () => anthropic.messages.create({
    model: 'claude-3-5-sonnet-20241022',
    max_tokens: 1024,
    messages: [{ role: 'user', content: 'Hello!' }],
  }),
  {
    provider: 'anthropic',
    model: 'claude-3-5-sonnet-20241022',
    extractUsage: extractAnthropicUsage,
  }
);

Storage options

Memory — tests & scripts

const tracker = createTracker({ storage: 'memory' });

File — local JSON

const tracker = createTracker({
  storage: 'file',
  file: { filePath: './.llm-usage/usage.json' },
});

MongoDB — production

const tracker = createTracker({
  storage: 'mongodb',
  mongodb: {
    uri: process.env.MONGODB_URI,
    database: 'my_app',
    collection: 'llm_token_usage',
  },
});

// on shutdown
await tracker.close();

Query & summarize

const calls = await tracker.getCalls({
  provider: 'openai',
  from: '2026-07-01',
  limit: 100,
});

const summary = await tracker.getSummary();
/*
{
  totalCalls: 42,
  totalInputTokens: 12500,
  totalOutputTokens: 3200,
  totalTokens: 15700,
  averageInputTokens: 297.6,
  averageOutputTokens: 76.2,
  averageTotalTokens: 373.8
}
*/

const byModel = await tracker.getSummaryByModel();
const byProvider = await tracker.getSummaryByProvider();

Record shape

Every call is stored like this:

{
  "id": "uuid",
  "provider": "openai",
  "model": "gpt-4o-mini",
  "inputTokens": 120,
  "outputTokens": 45,
  "totalTokens": 165,
  "timestamp": "2026-07-10T06:00:00.000Z",
  "durationMs": 842,
  "metadata": { "userId": "user-123" }
}

Custom providers

import { createUsageExtractor } from 'all-llm-token-tracker';

const extractMyProvider = createUsageExtractor((response) => ({
  inputTokens: response.tokens.in,
  outputTokens: response.tokens.out,
}));

Or plug in your own storage (Postgres, Redis, etc.):

import { createTracker, type StorageAdapter, type LlmCallRecord } from 'all-llm-token-tracker';

class PostgresStorage implements StorageAdapter {
  async save(record: LlmCallRecord) { /* ... */ }
  async find(query) { /* ... */ }
  async count(query) { /* ... */ }
}

const tracker = createTracker({ storage: new PostgresStorage() });

Bonus: browser dashboard

The repo includes a simple dashboard to visualize input vs output tokens:

git clone https://github.com/rkanumetta/all-llm-token-tracker.git
cd all-llm-token-tracker
npm install
npm run example
npm run dashboard

Open http://localhost:4173

You'll see:

Total calls, input tokens, output tokens
Bar chart per call
Breakdown by model
Filterable table with metadata

Real-world use cases

Cost monitoring — track tokens per feature or user
Debugging — find which prompts burn the most tokens
A/B testing — compare models side by side
SaaS billing — feed data into your own pricing logic
Compliance — audit trail with timestamps + metadata

Why I built it this way

Small surface area — one responsibility: track tokens
Storage flexibility — file in dev, MongoDB in prod
Provider-agnostic — extend with custom extractors
TypeScript-first — full types included
Lightweight — MongoDB is optional, zero required runtime deps

Try it

npm install all-llm-token-tracker

Links:

📦 npm: https://www.npmjs.com/package/all-llm-token-tracker
🐙 GitHub: https://github.com/rkanumetta/all-llm-token-tracker
🐛 Issues: https://github.com/rkanumetta/all-llm-token-tracker/issues

If you find it useful, star the repo ⭐ and open an issue if you want support for another provider (Gemini, Mistral, etc.).

Wrapping up

LLM token tracking doesn't need to be complicated. Wrap your API calls, store consistent records, query when you need insights.

Track input. Track output. Store it. Query it. Done.

What storage backend do you use for LLM observability — file, MongoDB, or something else? Drop a comment below.

DEV.to publishing checklist

Go to dev.to/new
Paste the title
Paste the article body (from "If you build with OpenAI..." through the closing question)
Add tags: node, javascript, openai, llm
Add a cover image (dashboard screenshot helps clicks)
Set canonical URL only if you also published on Medium (avoid duplicate SEO issues)
Click Publish

Optional: shorter subtitle for DEV

Paste this as the first line under the title if DEV shows a preview/subtitle field:

A lightweight Node.js npm package to track input/output tokens for OpenAI, Anthropic, and custom LLM providers — with memory, file, and MongoDB storage plus a browser dashboard.

DEV Community: Rajesh Kumar Kanumetta