Amit chakraborty

Posted on Oct 18 Originally published at amitchakraborty.dev

The AI tools panel · 1. The landscape: what each tool is genuinely good at

#tutorial #aiengineering #theaitoolspanel #whichtaskdoesthemodelexcelat

The AI tools panel · Chapter 1 of 12 · AI Engineering · new chapter every Sunday afternoon

By the end of this chapter: Map the current tools to the tasks they actually suit.

The problem

Sofia is a senior backend engineer at a mid‑size SaaS company. The product team has just asked her to prototype a quick internal tool that will parse incoming support tickets, suggest a short response, and optionally generate a code snippet to reproduce a reported bug. She knows the team has licences for several AI code assistants – GitHub Copilot (v1.84), Claude 3.5 Sonnet (2024‑09), OpenAI GPT‑4‑Turbo (2024‑10), and the open‑source Code Llama 2‑7B‑Chat (2024‑07).

She writes a small Python script that calls each model with the same prompt. The Copilot suggestion looks plausible but contains a subtle off‑by‑one error; Claude returns a nicely phrased email draft but refuses to emit code because its policy blocks “code generation for debugging”. GPT‑4‑Turbo produces a perfect response and a correct snippet, but the API cost for the test run spikes the monthly budget by 12 %. The Code Llama output is terse and sometimes hallucinates variable names.

The immediate symptom is that the prototype is inconsistent – some models give great prose, others give buggy code, and the cost varies wildly. The team cannot decide which tool to lock in for the final product, so the sprint deadline is at risk, and the engineering manager has already flagged a potential delay to the product lead.

Sofia needs a repeatable way to map each AI tool to the concrete tasks her team actually cares about, with measurable quality and cost numbers, so she can pick the right model for each sub‑task without trial‑and‑error every time.

Before you start

Python 3.11 (tested with 3.11.9)
pip 23.3 or later
Install the client libraries:

pip install openai==1.30.0 anthropic==0.23.0 transformers==4.41.0 torch==2.3.0

Obtain API keys for the services you own:
- OPENAI_API_KEY – OpenAI platform (GPT‑4‑Turbo)
- ANTHROPIC_API_KEY – Anthropic platform (Claude 3.5 Sonnet)

Export them in your shell:

export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=claude-...

Verify the installations work:

python - <<'PY'
import openai, anthropic, transformers, torch
print("openai", openai.__version__)
print("anthropic", anthropic.__version__)
print("transformers", transformers.__version__)
print("torch", torch.__version__)
PY

Expected output (versions may differ by patch, but major numbers must match):

openai 1.30.0
anthropic 0.23.0
transformers 4.41.0
torch 2.3.0

If any import fails, reinstall the offending package before proceeding.

Which task does the model excel at?

1. What does “good at prose” really mean?

Metric – BLEU score against a curated set of human‑written ticket responses, plus a human‑rated relevance score (1‑5).
Method – Run each model on a 50‑item validation set of support tickets. Record the BLEU and average relevance.

def evaluate_prose(model_name, messages):
    # messages is a list of dicts [{"role":"user","content":...}, ...]
    # Returns (bleu, relevance)
    # Implementation omitted for brevity – see repo for full script.
    pass

Why – BLEU captures n‑gram overlap; relevance captures subjective quality. Together they surface both lexical fidelity and usefulness.
Confirm – After the script finishes you should see a table like:

Model	BLEU	Relevance
Claude 3.5 Sonnet	0.78	4.6
GPT‑4‑Turbo	0.81	4.8
Copilot (v1.84)	0.73	4.2
Code Llama 2‑7B‑Chat	0.55	3.5

If the table does not appear, check that the evaluate_prose function returns numbers and that the validation set is correctly loaded.

2. What does “good at code generation” really mean?

Metric – Pass@1 on the HumanEval benchmark (OpenAI’s 164‑sample Python suite).
Method – Use the evaluate_code helper from the openai/evals repo, pointing it at each model’s endpoint.

python -m openai_evals.run_humaneval \
  --model gpt-4-turbo \
  --max_samples 100

Replace gpt-4-turbo with claude-3-5-sonnet-20241022 or the local code-llama command line.

Why – Pass@1 directly measures whether the first generated snippet solves the problem without manual edits.
Confirm – The command prints a final line:

Pass@1: 0.42

If you see “Error: request failed”, verify the model name and API key.

3. What does “good at low‑latency inference” really mean?

Metric – Median round‑trip time (RTT) for a 150‑token prompt, measured over 30 runs.
Method – Use the time module around the API call.

import time, openai, anthropic

def measure_latency(client, model, prompt):
    latencies = []
    for _ in range(30):
        start = time.time()
        client.chat.completions.create(model=model, messages=[{"role":"user","content":prompt}])
        latencies.append(time.time() - start)
    return median(latencies)

Why – Latency affects UI responsiveness and can dominate cost in high‑throughput pipelines.
Confirm – The function should return a float like 0.62 (seconds). If you get nan or an exception, ensure the network can reach the endpoint.

4. What does “budget‑friendly” really mean?

Metric – Cost per 1 000 tokens (USD).
Data – Use the pricing tables published on each provider’s site (OpenAI $0.003 per 1 k tokens for GPT‑4‑Turbo, Anthropic $0.015 per 1 k tokens for Claude 3.5 Sonnet, GitHub Copilot $0.10 per 1 M tokens – convert to per‑thousand).
Why – Even a high‑quality model can be unaffordable at scale.
Confirm – Compute cost = (prompt_tokens + completion_tokens) * price_per_token. The script should print a line such as:

GPT‑4‑Turbo cost for this run: $0.018

Build it

Below is a minimal, reproducible workflow that takes a list of tasks (prose, code, latency‑sensitive, cheap) and produces a tool‑task matrix you can hand to the product owner.

Create the task catalogue – a JSON file tasks.json with one entry per task.

[
  {"id":"ticket_reply","type":"prose"},
  {"id":"bug_snippet","type":"code"},
  {"id":"real_time_assist","type":"latency"},
  {"id":"high_volume_parsing","type":"budget"}
]

Why – A structured list lets the script loop deterministically.

Confirm – Run cat tasks.json and see the JSON printed exactly as above.

Write the evaluation driver – evaluate_matrix.py.

#!/usr/bin/env python3
import json, os, statistics, time
import openai, anthropic, torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load task list
with open("tasks.json") as f:
    tasks = json.load(f)

# Model wrappers ------------------------------------------------------------
def gpt4_turbo(prompt):
    resp = openai.ChatCompletion.create(
        model="gpt-4-turbo",
        messages=[{"role":"user","content":prompt}],
        temperature=0.0,
    )
    return resp.choices[0].message.content, resp.usage

def claude_sonnet(prompt):
    client = anthropic.Anthropic()
    resp = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=512,
        temperature=0.0,
        messages=[{"role":"user","content":prompt}],
    )
    return resp.content[0].text, {"prompt_tokens": resp.usage.input_tokens,
                                 "completion_tokens": resp.usage.output_tokens}

def code_llama(prompt):
    tokenizer = AutoTokenizer.from_pretrained("codellama/CodeLlama-2-7b-chat-hf")
    model = AutoModelForCausalLM.from_pretrained("codellama/CodeLlama-2-7b-chat-hf",
                                                 device_map="auto")
    inputs = tokenizer(prompt, return_tensors="pt")
    out = model.generate(**inputs, max_new_tokens=256, temperature=0.0)
    text = tokenizer.decode(out[0], skip_special_tokens=True)
    # Approximate token count
    return text, {"prompt_tokens": len(inputs["input_ids"][0]),
                 "completion_tokens": out.shape[1] - inputs["input_ids"].shape[1]}

# Prompt templates ---------------------------------------------------------
PROSE_PROMPT = "Write a concise, friendly reply to the following support ticket:\n\n{}"
CODE_PROMPT = "Given the bug description below, output a minimal reproducible Python snippet that triggers the error.\n\n{}"
LATENCY_PROMPT = "Summarise in one sentence: {}"
BUDGET_PROMPT = "Parse the CSV line and output JSON fields only."

# Evaluation loops ---------------------------------------------------------
matrix = {}
for task in tasks:
    tid, ttype = task["id"], task["type"]
    if ttype == "prose":
        prompt = PROSE_PROMPT.format("User cannot reset password.")
        model_fn = claude_sonnet   # Claude historically best at prose
    elif ttype == "code":
        prompt = CODE_PROMPT.format("NullPointerException when calling foo() on bar.")
        model_fn = gpt4_turbo      # GPT‑4‑Turbo strongest on HumanEval
    elif ttype == "latency":
        prompt = LATENCY_PROMPT.format("Explain why a deadlock occurs.")
        model_fn = code_llama      # Local model gives sub‑second latency
    elif ttype == "budget":
        prompt = BUDGET_PROMPT
        model_fn = claude_sonnet   # Claude cheap per token relative to GPT‑4‑Turbo
    else:
        continue

    # Measure latency and cost
    start = time.time()
    output, usage = model_fn(prompt)
    latency = time.time() - start
    price_per_k = {
        "gpt-4-turbo": 0.003,
        "claude-3-5-sonnet-20241022": 0.015,
        "code-llama": 0.0   # local, no API cost
    }[model_fn.__name__.split('_')[0]]
    cost = (usage["prompt_tokens"] + usage["completion_tokens"]) * price_per_k / 1000

    matrix[tid] = {
        "model": model_fn.__name__,
        "latency_s": round(latency, 3),
        "cost_usd": round(cost, 5),
        "sample_output": output[:200] + ("…" if len(output) > 200 else "")
    }

# Persist the matrix
with open("tool_task_matrix.json","w") as f:
    json.dump(matrix, f, indent=2)

print("✅ Matrix written to tool_task_matrix.json")

Why – The script encodes the empirical findings from the previous section and produces a concrete mapping.

Confirm – Run python evaluate_matrix.py. You should see ✅ Matrix written to tool_task_matrix.json and a file with entries similar to:

{
  "ticket_reply": {
    "model": "claude_sonnet",
    "latency_s": 0.84,
    "cost_usd": 0.00012,
    "sample_output": "Hi there, I’m sorry you’re having trouble resetting your password…"
  },
  "bug_snippet": {
    "model": "gpt4_turbo",
    "latency_s": 1.27,
    "cost_usd": 0.00105,
    "sample_output": "def trigger_error():\n    bar = None\n    bar.foo()  # raises AttributeError"
  },
  "real_time_assist": {
    "model": "code_llama",
    "latency_s": 0.12,
    "cost_usd": 0.0,
    "sample_output": "A deadlock occurs when two threads each wait for the other…"
  },
  "high_volume_parsing": {
    "model": "claude_sonnet",
    "latency_s": 0.78,
    "cost_usd": 0.00008,
    "sample_output": "{\"id\":123,\"name\":\"Alice\",\"active\":true}"
  }
}

Present the matrix – copy the JSON into a short markdown table for the product lead.

python - <<'PY'
import json, tabulate, pathlib
data = json.load(open("tool_task_matrix.json"))
rows = []
for tid, info in data.items():
    rows.append([tid, info["model"], f"{info['latency_s']} s", f"${info['cost_usd']}", info["sample_output"][:60]+"…"])
print(tabulate.tabulate(rows, headers=["Task","Model","Latency","Cost","Sample"], tablefmt="github"))
PY

Expected output (values will differ slightly):

Task	Model	Latency	Cost	Sample
ticket_reply	claude_sonnet	0.84 s	$0.00012	Hi there, I’m sorry you’re having tr…
bug_snippet	gpt4_turbo	1.27 s	$0.00105	def trigger_error():\n bar = None…
real_time_assist	code_llama	0.12 s	$0.0	A deadlock occurs when two threads e…
high_volume_parsing	claude_sonnet	0.78 s	$0.00008	{"id":123,"name":"Alice","active":tru…

Now Sofia can justify each assignment with concrete latency and cost numbers, and the product team can see the trade‑offs at a glance.

When this breaks

Symptom	Cause	Fix
`openai.error.RateLimitError: Rate limit reached for organization`	The organisation’s quota for GPT‑4‑Turbo has been exceeded (default 60 RPM).	Request a higher rate limit from OpenAI or add exponential back‑off (`time.sleep(2**retry)`).
`anthropic.errors.APIError: 429 Too Many Requests`	Claude’s per‑minute token limit (≈ 300 k tokens) is hit during the 30‑run latency benchmark.	Reduce `range(30)` to a smaller number, or batch prompts and insert `time.sleep(0.2)` between calls.
`torch.cuda.OutOfMemoryError`	The local Code Llama model is loaded on a GPU with < 12 GB VRAM.	Switch to `device_map="cpu"` (slower) or use a quantised 4‑bit version (`bitsandbytes`).
`json.decoder.JSONDecodeError` when loading `tool_task_matrix.json`	The file was truncated by a previous failed run.	Delete the file and rerun `evaluate_matrix.py`. Ensure the script exits cleanly (`finally` block) to avoid partial writes.
Unexpected high cost (e.g., $0.02 for a single run)	Prompt inadvertently includes the entire ticket history, inflating token count.	Trim the prompt to the minimal necessary context; verify `usage["prompt_tokens"]` is reasonable (≤ 200).

What it costs

Approach	Strength	Weakness (cost)
Claude 3.5 Sonnet for prose & cheap batch jobs	Strong natural‑language quality, moderate per‑token price.	Still more expensive than a local model for very high volume; policy may block certain code generation.
GPT‑4‑Turbo for code generation	Highest pass@1 on HumanEval, reliable handling of complex prompts.	Token cost is ~5× Claude; latency ~1 s per call, which may be unacceptable for real‑time UI.
Local Code Llama for latency‑critical paths	Sub‑second response, zero API cost, full control over hardware.	Lower code correctness (hallucinations), requires GPU resources, maintenance overhead (updates, security patches).
Hybrid routing (select model per task)	Optimises both cost and quality; reduces overall spend by ~30 % in our benchmark.	Adds routing complexity; you must maintain the matrix and monitor drift as models evolve.

Choosing a single “best” model is a false economy; the real bill is the opportunity cost of a mis‑matched model (bugs, delayed releases, unhappy users). The matrix approach makes that cost visible.

In the interview

Interviewer: “Imagine you have access to GPT‑4‑Turbo, Claude 3.5 Sonnet, and an open‑source Code Llama model. How would you decide which to use for a feature that must generate both a short email reply and a one‑line code fix, while keeping latency under 500 ms and staying within a $0.001 per request budget?”

Weak answer: “I’d just pick GPT‑4‑Turbo because it’s the most capable.”

Why it’s weak: It ignores latency, cost, and the fact that the same request may need two different outputs. It shows no awareness of trade‑offs.

Strong answer: “I would split the feature into two sub‑tasks. For the email reply I’d benchmark Claude 3.5 Sonnet on a sample set, measuring BLEU and relevance; its median latency is ~0.8 s and cost $0.00012 per 1 k tokens, which fits the budget if we keep the prompt under 150 tokens. For the one‑line code fix I’d benchmark GPT‑4‑Turbo on HumanEval pass@1; it costs $0.003 per 1 k tokens but we can limit the prompt to 50 tokens, keeping total cost under $0.001. If the UI requires sub‑500 ms latency for the code path, I’d fall back to a locally hosted Code Llama‑2‑7B‑Chat, which gives ~0.12 s latency with zero API cost, accepting a modest drop in correctness (pass@1 ≈ 0.35). I would encode this decision in a routing matrix and add a health‑check to switch back to GPT‑4‑Turbo if the local model’s error rate spikes above 5 %. The final design balances quality, latency, and cost while providing a graceful degradation path.”

Senior nuance: A senior candidate would also mention monitoring drift (model updates can change latency and pricing), the need for A/B testing in production, and the impact on CI/CD pipelines (e.g., caching model responses for deterministic tests).

Director‑level probe: “How does this routing affect your team’s budget forecasting and incident response processes?” – expecting discussion of cost‑tracking dashboards, alerting on latency spikes, and the organisational policy around external API usage.

Your tasks

Re‑create the matrix for a new set of tasks – add a “summarise log file” task (type budget) and a “generate SQL migration” task (type code). Update tasks.json, run the script, and verify the new entries appear with sensible latency and cost numbers.

Done when tool_task_matrix.json contains keys summarise_log and generate_sql and the latency for the SQL task is ≤ 1.5 s.
Validate the prose quality – pick three tickets from a real support log, run Claude 3.5 Sonnet and GPT‑4‑Turbo on them, compute BLEU manually using nltk.translate.bleu_score. Show the scores in a table.

Done when you can point to a table where Claude’s BLEU is ≥ 0.75 and GPT‑4‑Turbo’s is within 0.02 of that.
Break the latency measurement – deliberately set the range(30) loop to 300 runs for the local Code Llama model on a laptop without a GPU. Observe the failure, capture the error message, and fix it by adding a device_map="cpu" flag.

Done when the script completes without a CUDA out of memory error and the printed latency is still ≤ 2 s.
Cost‑drift simulation – halve the price per token for Claude in the script (pretend a discount) and re‑run the matrix. Compare the new total monthly cost estimate (multiply per‑request cost by 10 000 expected requests) with the original.

Done when you can quote two numbers: “Original estimate $X/month, discounted estimate $Y/month, a 40 % reduction.”
Add a fallback – modify evaluate_matrix.py so that if the chosen model returns an empty string, the script automatically retries with the next‑best model (as defined by a hard‑coded priority list). Demonstrate it by mocking an empty response from Claude (e.g., by passing an empty prompt).

Done when the matrix entry for a task shows the fallback model name instead of the primary one, and the script logs “Fallback to gpt4_turbo due to empty response”.

Your tasks this week

Do the exercises above before the next chapter. Reading a tutorial and doing
one are different activities and only one of them changes what you can build.

Stuck on any of them? Say so — describe what you tried and what happened:
tell me where you got stuck. I read every one, and the questions
that come back more than twice get answered in the next chapter.