AI Cost Optimizer CLI: Real-Time AI Cost Tracking, Model Routing, and Shadow AI Detection from the Terminal

Engineering teams running AI at scale eventually hit the same wall: no one knows exactly how much each model call costs, which GPU is idle, whether a cheaper model would have done the job just as well, or whether someone in the organization is quietly using an unauthorized AI service. The answers exist in the data, but nothing surfaces them without instrumentation. By the time costs are visible, the spend has already happened.

AI Cost Optimizer CLI brings that visibility into the terminal before it becomes a problem. It is a command-line toolkit designed for engineering teams who use AI and LLM APIs at scale: real-time cost tracking at the token level, GPU utilization monitoring, intelligent model routing, caching analysis, shadow AI detection, and comprehensive reporting, all from a single CLI with no external API keys and no cloud dependencies.

What This Tool Does

Six capabilities work together from one command entry point.

The cost tracker logs every API call with precise attribution: prompt tokens multiplied by input price plus completion tokens multiplied by output price, per call, per session, per project, stored in a local SQLite database. The GPU monitor queries NVML for live metrics (utilization, memory, temperature, power draw) or generates realistic random-walk simulations when no NVIDIA hardware is present. The model router classifies task complexity using keyword analysis and recommends the cost-optimal model, routing low-complexity tasks to SLMs where the savings range from 60 to 90 percent versus frontier models. The caching advisor analyzes prompt patterns and estimates dollar savings across four caching strategies with daily, monthly, and yearly projections. The shadow AI scanner parses DNS packet captures to detect unauthorized AI API usage across 25+ known domains. The report generator aggregates all of this into a single actionable output: current cost, optimized cost, potential savings, and the routing change that gets there.

A pricing database of 25+ models across 9 providers is built in. No external API keys are needed. No cloud dependencies. GPU monitoring works in simulation mode when no NVIDIA GPU is present.

Architecture

The CLI is built on click, with each subcommand routing to an independent backend module:

┌──────────────────────────────────────────────────────────────────────────┐
│                           neo-cost CLI (click)                            │
│                                                                          │
│   cost     gpu      route     cache     shadow    report    monitor      │
│   ─────   ─────   ───────   ───────   ───────   ───────   ───────      │
└──────────────────────────────────────────────────────────────────────────┘
         │         │         │          │         │          │
         ▼         ▼         ▼          ▼         ▼          ▼
┌─────────────┐ ┌─────────┐ ┌──────────┐ ┌─────────────┐ ┌────────┐
│ CostTracker │ │GPUMonit │ │ModelRoute│ │CachingAdvis │ │ShadowAI│
│             │ │ or      │ │ r        │ │ or          │ │Scanner │
│  ┌───────┐  │ │         │ │          │ │             │ │        │
│  │SQLite │  │ │pynvml + │ │PricingDB │ │PromptAnalys │ │dpkt +  │
│  │  DB   │  │ │ Demo    │ │ +        │ │ is +        │ │ Pcap   │
│  └───────┘  │ │ Mode    │ │Keyword   │ │SavingsCalc  │ │Parser  │
│             │ └─────────┘ │Classifier│ └─────────────┘ └────────┘
│  Per-call   │             └──────────┘
│  logging    │            ┌─────────────┐
│  Aggregated │            │ PRICING_DB  │
│  reports    │            │ (25 models) │
└─────────────┘            └─────────────┘
                                    ▲
                                    │
                              ┌──────────┐
                              │ config   │
                              │ (YAML)   │
                              └──────────┘

The data flow follows each module's responsibility cleanly. The cost tracker logs each API call to a local SQLite database and produces aggregated reports. The GPU monitor queries NVML for live metrics or generates realistic simulations in demo mode. The model router classifies task complexity using keyword analysis and recommends the optimal model from the pricing database. The caching advisor analyzes prompt patterns and estimates dollar savings from different caching strategies. The shadow AI scanner parses DNS packet captures to detect unauthorized AI API usage. The report generator aggregates data across all modules into actionable recommendations.

Features

1. Cost Tracker: Token-Level Attribution
Every API call is tracked with precise cost attribution across five dimensions:

• Per-call costing: Prompt tokens × input price + completion tokens × output price
• Session tracking: Group related calls into sessions for project-level analysis
• Model breakdowns: See exactly what each model costs you
• Project breakdowns: Allocate costs across teams or applications
• Time-series: Daily cost trends over configurable periods

Sample output:

💰 Cost Logged — gpt-4o
╭───────────────────┬──────────────╮
│ Metric            │ Value        │
├───────────────────┼──────────────┤
│ Model             │ gpt-4o       │
│ Provider          │ OpenAI       │
│ Project           │ test-project │
│ Prompt Tokens     │ 500          │
│ Completion Tokens │ 150          │
│ Total Tokens      │ 650          │
│ Input Cost        │ $0.001250    │
│ Output Cost       │ $0.001500    │
│ Total Cost        │ $ 0.002750   │
╰───────────────────┴──────────────╯

2. GPU Monitor: Utilization and Alerts
Live GPU metrics with threshold-based alerts:

• GPU Utilization: NVML or simulated random-walk
• Memory Usage: Used / Total GB with percentage
• Temperature: Live sensor reading in °C
• Power Draw: Current and max power in Watts
• Alert Thresholds: Configurable for temp, util, mem, power

Demo mode generates realistic GPU metrics using a random-walk simulation, for development and testing without dedicated hardware. Continuous monitoring with neo-cost gpu watch shows a live-updating terminal dashboard with ASCII bar charts.

3. Model Router: Intelligent Routing Engine
Task complexity is classified across four dimensions:

• Simple: Summarization, translation, extraction, formatting
• Complex: Analysis, comparison, evaluation, troubleshooting
• Creative: Writing, composition, brainstorming, marketing
• Code: Implementation, debugging, refactoring, architecture

Routing logic maps complexity to model category:

• Low complexity → SLM (saves 60-90% vs frontier)
• Medium complexity → SLM with frontier fallback option
• High complexity → Frontier model for quality

Each model receives a composite score based on task-specific quality multiplier, per-call cost estimate, and token budget prediction.

4. Caching Advisor: Cost Savings Analysis
Four caching strategies evaluated with concrete dollar estimates:

• Exact Match (~15% hit rate): Best for repeated identical prompts
• Semantic (~25% hit rate): Best for similar meaning, different wording
• Prompt Caching (~40% hit rate): Best for shared system prefixes and templates
• TTL-Based (~20% hit rate): Best for time-bounded repeated queries

Each strategy produces daily, monthly, and yearly savings projections based on estimated call volume.

5. Shadow AI Scanner: DNS-Based Detection

Unauthorized AI API usage is detected by analyzing DNS traffic. The detection database covers 25+ known AI API domains across providers including OpenAI, Anthropic, Google, Groq, DeepSeek, Mistral, Cohere, HuggingFace, Replicate, Perplexity, Together AI, and more.

Risk scoring considers the number of queries to each AI domain, the sensitivity of the service (API endpoints score higher risk than web UIs), and the total number of unique AI services detected. Sample PCAP generation creates realistic test data with mixed benign and AI API DNS queries.

6. Report Generator: Comprehensive Insights
All six modules aggregate into a single report:

Current Cost: $152.40                                                        
Optimized Cost: $88.39                                                       
Potential Savings: $64.01                                                    
Savings %: 42.0%                                                             
Route 60% of simple/medium tasks from frontier to SLM models

Installation

Prerequisites

Python 3.8 or higher
pip (Python package manager)

Install from source

# Clone the repository
git clone https://github.com/your-org/neo-cost-optimizer.git
cd neo-cost-optimizer

# Create and activate virtual environment (recommended)
python3 -m venv venv
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

# Install the package
pip install -e .

# Verify installation
neo-cost --help

Quick install via pip

pip install neo-cost

Dependencies

Quick Start

# Track a single API call
neo-cost cost log gpt-4o --prompt-tokens 500 --completion-tokens 150 --project my-app

# View cost summary
neo-cost cost summary

# Check GPU status (simulation mode)
neo-cost gpu status --demo

# Get model routing recommendation
neo-cost route "Write a Python function to sort a list of dictionaries"

# Analyze caching strategy
neo-cost cache "You are a helpful assistant. Summarize the following text:" --volume 5000

# Scan for shadow AI usage (generates sample data automatically)
neo-cost shadow scan

# Generate a comprehensive report
neo-cost report

# List all models with pricing
neo-cost models

# Show current configuration
neo-cost config

Configuration

Configuration is stored in ~/.config/neo-cost/config.yaml (or $XDG_CONFIG_HOME/neo-cost/config.yaml) and is created automatically on first run.

Default configuration

cost_tracker:
  db_path: ~/.neo-cost/cost_tracker.db
  default_project: default

gpu_monitor:
  poll_interval: 2.0
  temp_threshold_c: 85
  util_threshold_pct: 95
  memory_threshold_pct: 90
  power_threshold_w: 400
  demo_mode: false

model_router:
  default_slm: gpt-4o-mini
  default_frontier: gpt-4o
  token_budget_warning_pct: 80

caching_advisor:
  min_similarity: 0.7
  default_ttl_hours: 24
  cache_hit_cost_reduction_pct: 90

shadow_ai:
  sample_pcap_path: ~/.neo-cost/shadow_ai_sample.pcap
  risk_score_threshold: 5

reporting:
  output_dir: ~/.neo-cost/reports
  default_format: terminal

Configuration is edited directly or viewed with neo-cost config.

Command Reference

neo-cost cost

neo-cost gpu

• status: Current GPU metrics (--demo)
• watch: Live monitoring dashboard (--demo, --interval)

neo-cost route

neo-cost cache

• --volume: Daily call volume (default: 1000)
• --ttl: TTL in hours (default: 24)

neo-cost shadow

Other commands

• report: Comprehensive cost optimization report
• models: List all models with pricing
• config: Show current configuration
• monitor: Interactive real-time monitoring dashboard

Example Workflows

Workflow 1: Daily Cost Review

# 1. Log all your API calls (integrate into your app or batch log)
neo-cost cost log gpt-4o --prompt-tokens 15000 --completion-tokens 3000 --project production
neo-cost cost log gpt-4o-mini --prompt-tokens 50000 --completion-tokens 10000 --project production

# 2. Get daily summary
neo-cost cost summary --days 1

# 3. Generate full report
neo-cost report --days 7

Workflow 2: Model Selection Decision

# Compare routing for different task types
neo-cost route "Translate this document to Spanish"
neo-cost route "Debug this production issue with the microservice"
neo-cost route "Write a marketing campaign for our new product"

Workflow 3: Shadow AI Audit

# 1. Capture DNS traffic on your network (using tcpdump)
sudo tcpdump -i eth0 port 53 -w network_traffic.pcap

# 2. Scan with neo-cost
neo-cost shadow scan --pcap network_traffic.pcap

# 3. Review the risk report and take action

Workflow 4: Caching ROI Analysis

# Analyze your most common prompt patterns
neo-cost cache "You are a customer support agent..." --volume 10000
neo-cost cache "Summarize the following article:" --volume 5000

Workflow 5: GPU Capacity Planning

# Monitor GPU over time
neo-cost gpu watch --interval 5

# Check current status
neo-cost gpu status

Cost-Savings Methodology

The cost-optimization engine uses a multi-factor scoring system:

Model Score = QualityMultiplier(TaskType, ModelCategory) / Cost

Quality Multiplier Table:

Savings estimation analyzes the current model mix from cost tracker data, classifies each task by complexity, proposes alternative SLM routing for simple and medium tasks, applies an average savings of ~70% per rerouted call, and assumes a realistic adoption rate of 60%.

Caching savings follow this formula:

Savings = DailyVolume × CacheHitRate × CostPerCall × 0.9

The 0.9 factor represents the 90% cost reduction when serving from cache.

Real-World Use Cases

Engineering Teams: track per-feature AI costs across development sprints, optimize model selection for different product features, monitor GPU utilization in CI/CD pipelines.

FinOps / Cloud Cost Teams: allocate AI costs to business units and projects, identify cost anomalies with daily time-series tracking, audit shadow AI usage before it becomes a security risk.

AI Platform Teams: build an internal model gateway with routing recommendations, implement caching for high-volume inference endpoints, create chargeback reports for internal AI consumers.

Security Teams: detect unauthorized AI API usage in corporate networks, identify data exfiltration risks from shadow AI tools, enforce AI usage policies with DNS-level monitoring.

How I Built This Using NEO

This project was built using NEO. NEO is a fully autonomous AI engineering agent that can write code and build solutions for AI/ML tasks including AI model evals, prompt optimization and end to end AI pipeline development.

The requirement was a terminal-based toolkit that gives engineering teams full visibility into their AI spend: token-level cost attribution, GPU monitoring, intelligent model routing, caching analysis, shadow AI detection, and comprehensive reporting, all from a single CLI with no external dependencies. NEO planned and produced the files in this repository: six backend modules covering cost tracking, GPU monitoring, model routing, caching analysis, shadow AI scanning, and report generation, a built-in pricing database across 25+ models and 9 providers, a click-based CLI layer with 9 commands, a YAML configuration system, and a test suite under test_suite/.

The result is a fully working CLI toolkit that surfaces AI cost decisions in real time: what each call costs, which model would have been cheaper, how much caching would save, and whether unauthorized AI services are active on the network.

How You Can Use This With NEO

Track AI costs per project across an engineering team.
The neo-cost cost log command accepts a model name, token counts, and a project label on every API call. The neo-cost cost summary command then breaks down spending by model, project, and day, turning scattered API usage into an auditable cost record without any external tooling.

Route tasks to cheaper models before making the API call.
The neo-cost route command accepts a prompt and returns a model recommendation based on task complexity. Simple and medium-complexity tasks are routed to SLMs where savings of 60–90% versus frontier models are possible, with the scoring methodology transparently documented in neo_cost/pricing_db.py.

Audit shadow AI usage across a corporate network.
DNS traffic captured with tcpdump can be piped directly into neo-cost shadow scan. The scanner checks against 25+ known AI API domains and produces a risk score, giving security teams a concrete detection path without requiring agent deployment on individual machines.

Calculate caching ROI before committing to an implementation.
The neo-cost cache command takes a prompt pattern and a daily call volume and returns projected savings across four strategies: exact match, semantic, prompt caching, and TTL-based, with daily, monthly, and yearly figures, so the decision to invest in a caching layer is backed by numbers before any code is written.

Final Notes

Most AI cost problems are invisible until they're large. By the time a team notices the spend, the model choices, the redundant calls, and the shadow usage have already accumulated. AI Cost Optimizer CLI moves that visibility to the point where the decisions are still being made: which model, which caching strategy, which teams are spending what, and puts it in the terminal where engineering teams already work.

The code is at https://github.com/dakshjain-1616/AI-Cost-Optimizer-CLI
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code