Infra-Sizer: A CLI That Recommends the Right GPU Infrastructure for Any LLM Deployment

"What hardware do I need to serve this model?" is the first question every team asks before deployment. The answer requires juggling model size, throughput benchmarks, AWS pricing, batching math, and serving framework tradeoffs simultaneously.

infra-sizer answers it in seconds. It is a CLI that takes a model ID, expected request volume, latency SLA, and context length, then outputs a concrete GPU recommendation: instance type, count, serving framework, batching config, and monthly cost estimate. Built autonomously using NEO.

The Problem

Getting the hardware wrong is expensive in both directions.

Overprovision: Wasted $10k+/month on idle H100s.

Underprovision: Latency SLA breaches and on-call pages at 3am.

Wrong framework: vLLM vs TGI vs llama.cpp have 5x throughput differences for certain workloads.

Infra-Sizer gives you the right answer in seconds, not hours of spreadsheet work.

Architecture

The tool moves through five steps: the CLI parses inputs and fetches model metadata from the HuggingFace API, the GPU database looks up throughput benchmarks and AWS on-demand pricing across 6 GPU tiers, the planner sends the constraints and data to Llama-4-Maverick via OpenRouter for LLM-assisted reasoning, and the output is a Rich recommendation table covering GPU tier, instance count, framework, batch size, cost, and confidence score. If no API key is set, a rule-based fallback produces the recommendation without the LLM.

How It Works

1. Parse CLI args: model ID, requests-per-second, latency SLA (seconds), context length
2. Fetch model metadata from HuggingFace API: parameter count, architecture, dtype
3. Query GPU database: look up throughput benchmarks and AWS on-demand pricing for 6 GPU tiers
4. LLM reasoning: send constraints and data to Llama-4-Maverick via OpenRouter, which reasons over tradeoffs and recommends a config
5. Rule-based fallback: if OPENROUTER_API_KEY is unset, a deterministic algorithm produces a recommendation without LLM

GPU Database Coverage

Installation

git clone https://github.com/neo-engine/infra-sizer.git
cd infra-sizer
pip install -r requirements.txt

Requirements: Python 3.11+, openai, requests, rich, pandas

Usage

python main.py --model Qwen/Qwen3-32B --rps 50 --latency-sla 2.0 --context 8192

Arguments

Environment Variables

export OPENROUTER_API_KEY=your_key_here  # Enables LLM-assisted reasoning

Without the key, the tool falls back to rule-based recommendations. No crash, still useful.

Output

╔══════════════════════════════════════════════════════════════╗
║          Infra-Sizer — Infrastructure Recommendation         ║
╠══════════════════════════════════╦═══════════════════════════╣
║ GPU Tier                         ║ A100-80G                  ║
║ Instance Count                   ║ 2                         ║
║ Serving Framework                ║ vLLM                      ║
║ Recommended Batch Size           ║ 32                        ║
║ Max Concurrent Requests          ║ 64                        ║
║ Estimated Monthly Cost (AWS)     ║ $32,058                   ║
║ Confidence Score                 ║ 0.85                      ║
╚══════════════════════════════════╩═══════════════════════════╝

Test Cases

# Small model, low load → CPU or single L4
python main.py --model 3B --rps 5 --latency-sla 10.0 --context 2048

# Large model, high load, tight SLA → multi-GPU H100
python main.py --model 70B --rps 100 --latency-sla 1.0 --context 4096

# Impossible constraint → "No viable configuration" + closest feasible
python main.py --model 1T --rps 1000 --latency-sla 0.1 --context 128000

File Structure

infra-sizer/
├── main.py              # CLI entrypoint
├── hf_metadata.py       # HuggingFace model card fetcher
├── gpu_db.py            # GPU performance database + cost calculator
├── planner.py           # LLM-assisted recommendation engine (OpenRouter)
├── data/
│   ├── gpu_specs.json   # Throughput benchmarks per GPU per model size
│   └── aws_pricing.json # AWS on-demand pricing per instance type
├── requirements.txt
└── README.md

Why This Exists

"Just throw it on an A100" is the most expensive mistake in AI infrastructure. The right hardware choice depends on model size, throughput requirements, latency SLA, and cost constraints, all at once. This tool does that math in seconds so you can make an informed decision before signing a cloud contract.

How I Built This Using NEO

This project was built using NEO. NEO is a fully autonomous AI engineering agent that can write code and build solutions for AI/ML tasks including AI model evals, prompt optimization and end to end AI pipeline development.

The requirement was a CLI that takes an LLM model ID and deployment constraints, then outputs a concrete GPU infrastructure recommendation backed by real throughput benchmarks and AWS pricing. NEO planned and produced the files in this repository: the CLI entrypoint in main.py, the HuggingFace metadata fetcher, the GPU performance database and cost calculator, the LLM-assisted planner using Llama-4-Maverick via OpenRouter with a rule-based fallback, and the two data files covering GPU throughput benchmarks and AWS on-demand pricing across 6 GPU tiers.

The result is a fully working CLI that takes four inputs and returns a concrete infrastructure recommendation with GPU tier, instance count, serving framework, batch size, monthly cost, and confidence score, in seconds.

How You Can Use This With NEO

Run it before signing a cloud contract.
Pass in the model ID, expected RPS, latency SLA, and context length and get a concrete GPU recommendation with a monthly cost estimate before committing to any instance type. Takes seconds instead of hours of spreadsheet work.

Test impossible constraints to find the closest feasible configuration.
Passing an infeasible combination like a 1T-parameter model at 1000 RPS with a 0.1-second SLA returns "No viable configuration" alongside the closest feasible option, so you know exactly where the constraint breaks before you hit it in production.

Use it without an API key for quick rule-based estimates.
Without OPENROUTER_API_KEY set, the tool falls back to a deterministic algorithm and still produces a recommendation. Useful for quick checks in environments where external API calls are not allowed.

Use it as a pre-deployment checklist step.
Before any model goes to production, running Infra-Sizer with the actual expected load validates whether the planned instance type and count will meet the latency SLA, with a confidence score attached to the recommendation.

Final Notes

Every team deploying an LLM in production faces the same sizing problem, and most solve it the same way: by guessing, overprovisioning, and paying for it. Infra-Sizer makes that decision explicit, data-backed, and fast.

The code is at https://github.com/dakshjain-1616/infra-sizer
You can also build with NEO in your IDE using the VS Code extension or Cursor.
You can use NEO MCP with Claude Code: https://heyneo.com/claude-code