DEV Community: Azeem Siddiqui

If You're Giving Developers AI Tools, You Need Per-User Keys, Audit Logging, and Cost Controls on Day One

Azeem Siddiqui — Mon, 22 Jun 2026 14:36:48 +0000

The rollout looks responsible. A team wants AI in their editors, so someone wires up the tooling against a hosted model, drops a shared key into the config, ships it to the team, and moves on to the next thing. An afternoon's work. Everyone's happy. It feels like the careful version of "just let people use it."

It isn't, and the reason it isn't is that everything wrong with it is invisible on day one. There's no error. Nothing breaks. The tools work, the developers are productive, and the problems you've created don't announce themselves — they wait. That's what makes this the dangerous version, not the careful one.

I've built the thing that sits between a team of developers and a hosted model — the gateway that fronts the API, hands out keys, and logs the traffic. And the single most useful decision I made was treating three controls as non-negotiable from the first commit rather than features to add once we'd "proven it out": per-user keys, audit logging, and cost controls. Not because we were big. Because they're trivial to build in and genuinely painful to bolt on later, and that asymmetry is the whole argument.

Per-user keys

A shared key is one key that everyone's tooling carries. It works right up until you need it not to.

Someone leaves the company. Now you either rotate the key — which breaks every developer's tools at once and turns into a coordinated fire drill — or you don't, and a former employee's laptop still talks to your models. A key leaks into a screenshot, a commit, a pasted config in a support channel. Same fork. Rotate and break everyone, or shrug and hope. Neither is a position you want to be standing in.

And every day in between, you can't answer the most basic question: who did this? Who sent that request? Who's responsible for that spike? With a shared key the answer is "the team," which is the same as no answer.

Per-user keys make all of that boring. One person leaves, you deactivate one key, nobody else notices. One key leaks, you kill one key. Every request carries a person's identity, so "who did this" is a lookup instead of an investigation. The cost to build this in on day one is one table and a five-line CLI to mint a key. The cost to add it later is touching every developer's machine to swap the shared key for an individual one, coordinated across the whole team, while everyone's mid-task. Same outcome, an order of magnitude more pain, and you only do it after something already went wrong.

Audit logging

Here's a question someone will eventually ask you, and the only thing that varies is who and when: what did the AI tool actually see, do, and cost? Could be security after an incident. Could be compliance during a review. Could be you, at 2 a.m., trying to understand why something went sideways.

If you logged from day one, that's a query. If you didn't, it's an investigation with no evidence, because the logs you didn't write don't exist. You cannot audit the past retroactively. That's the trap with logging specifically — every other control you can add late and at least cover yourself going forward, but the gap in your audit trail is permanent. The day you wish you'd been logging is the day it's already too late to have been.

One thing worth getting right, since it's where I see people overcorrect: log metadata by default, not content. Who, when, which model, how many tokens, what it cost. The moment you start logging the actual prompts and completions, you've built a store of potentially sensitive text — source code, internal data, whatever developers paste in — and that store has its own retention, access, and compliance weight hanging off it. Good auditing isn't "record everything." It's recording enough to answer the questions you'll be asked, and making the choice to capture content a deliberate one with a reason behind it, not a default you backed into.

Cost controls

Token spend is invisible until the invoice, and AI usage is spiky in a way that makes that especially dangerous. It's not a smooth line you can extrapolate. It's a flat baseline with occasional cliffs — an agent stuck in a retry loop, someone pasting an entire repository into context, a script that was supposed to run once running ten thousand times. The spend that hurts you is the spend you didn't see coming, and without controls you see none of it coming.

Two cheap things change that. A per-user cap, checked before the request goes through, so one runaway can't run away very far. And a budget alert at the provider level as a backstop, so even if your app-level accounting has a blind spot, the cloud bill itself trips a wire. The difference between having these and not is the difference between a Slack notification on a Tuesday and a meeting with finance about last month.

Notice this one needs the first one. You can't cap or attribute spend per person if you can't tell people apart — per-user keys are what make per-user cost controls possible. The three aren't a checklist of separate features. They're one posture, and they reinforce each other: identity makes attribution possible, attribution makes both auditing and cost control meaningful, and all three come for nearly free if you lay them down together at the start.

"We're moving fast — isn't this premature?"

This is the real objection, and it deserves a real answer rather than a dismissal, because the instinct behind it is correct most of the time. Early on, you shouldn't gold-plate. Most "enterprise readiness" work genuinely is premature, and shipping beats polishing more often than engineers like to admit. If I thought these three were gold-plating, I'd agree with the objection.

They're not, and here's the distinction. Gold-plating is work whose payoff is speculative and whose cost is the same whenever you do it. These three are the opposite: their payoff is near-certain — someone will leave, a key will leak, a bill will surprise you, given enough time — and their cost is wildly different depending on when you do them. A day at the start. A migration under pressure after an incident later. When the cost of doing something is low now and high later, and the need is a matter of when rather than if, "we're moving fast" is the argument for doing it now, not against. Speed is exactly why you don't want to be retrofitting auth and accounting across a live fleet six months from now.

The fast path and the responsible path are the same path here. That's not usually true. It's true this time.

Day one is the cheap day

You don't earn these controls by hitting some headcount or some spend threshold where they suddenly become worth it. The first developer and the first key is the cheapest moment they'll ever be — one table, one CLI, one budget alert, an afternoon. Every day after that, the team grows, the configs proliferate, the unattributed spend piles up, and the same work gets a little more expensive and a little more disruptive to do.

I built all three in on day one. It cost me about a day, and I have never once wished I'd skipped it. The regret in this space only ever runs the other direction — it's always the team that handed out a shared key and figured they'd sort out the rest later, standing in the rubble of a leaked credential or a surprise invoice, doing the expensive version of the work they could have done cheap. Do it cheap. Do it first.

Centralized Logging Across a Mixed-OS Server Fleet with Grafana Alloy and Loki

Azeem Siddiqui — Mon, 22 Jun 2026 14:32:11 +0000

For a long time, finding a log on our fleet meant SSH-ing into the box and running grep. Which box? Depends. If you knew exactly where the thing ran, fine. If you didn't — and during an incident you usually don't — you were hopping between hosts, scrolling Tomcat logs, trying to line up timestamps across three machines by eye. It works right up until the moment you need it to work fast.

So I rolled out centralized logging: every log line from every server, in one place, queryable, with a Slack command so people didn't even have to open Grafana to check something. The stack is Grafana Alloy on each host shipping to Loki. This post is that rollout — including the parts the docs don't warn you about, which is most of what made it take longer than the happy-path tutorial suggests.

A quick word on why Alloy and Loki, since there are other options. I'd run ELK before, and the Elasticsearch side is a lot of machine to feed and babysit for what amounts to "let me search my logs." Loki indexes labels, not the full log text, so it's dramatically cheaper to run and the storage footprint is sane. The paid SaaS log tools are great until you see the bill for a chatty fleet. Alloy is Grafana's agent — it replaced Promtail and the old Grafana Agent — and it does the collecting. The combination gave me searchable logs without standing up a search cluster or signing a contract.

What the pipeline looks like

  each host                          central
┌───────────────┐
│  app logs     │
│  /var/log/... │
│      │        │
│   Grafana     │   push (HTTP)    ┌──────────┐     ┌──────────┐
│   Alloy       │ ───────────────► │   Loki   │ ◄── │ Grafana  │
│  (per box)    │                  └──────────┘     └──────────┘
└───────────────┘                       ▲
                                         │ LogQL over HTTP API
                                   ┌──────────┐
                                   │ Slack bot│
                                   └──────────┘

Alloy reads files on each host, tags each line with a few labels (which host, which environment, which service), and pushes to Loki. Grafana queries Loki for the dashboard view. The Slack bot hits Loki's HTTP API directly so people can pull logs without leaving the channel they're already in. That last piece turned out to be the one everybody actually uses.

Standing up Loki

For trying this out, a single Loki container is plenty. Here's a docker-compose.yml that gets you Loki and Grafana side by side:

services:
  loki:
    image: grafana/loki:3.0.0
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/config.yaml
    volumes:
      - ./loki-config.yaml:/etc/loki/config.yaml
      - loki-data:/loki

  grafana:
    image: grafana/grafana:11.0.0
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin

volumes:
  loki-data:

And a minimal loki-config.yaml that stores everything on the local filesystem:

auth_enabled: false

server:
  http_listen_port: 3100

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  retention_period: 744h   # 31 days

docker compose up -d and you've got Loki listening on 3100 and Grafana on 3000.

Filesystem storage is fine for a single node and for following along. In a real deployment I point object_store at S3 (or whatever object storage you've got) — Loki is built to keep chunks in object storage, and that's where it gets cheap and durable. Don't run a real fleet on local disk. But for getting the pipeline working end to end, this is one less thing to set up.

The Alloy agent

Now the part that runs on every host. Install on Debian/Ubuntu from Grafana's apt repo:

sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor \
  | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" \
  | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y alloy

On RHEL-family boxes (Rocky, Alma, and so on) it's the rpm repo and dnf install alloy instead. Same agent, different package manager — keep that in mind, because "mixed fleet" is the whole theme here and the install step is the first place it shows up.

Alloy's config lives at /etc/alloy/config.alloy and it speaks its own config language. Here's a working config that watches an app's logs and ships them:

// Discover the files to read, and stamp each one with labels.
local.file_match "app" {
  path_targets = [
    {
      __path__ = "/var/log/myapp/*.log",
      host     = constants.hostname,
      env      = "prod",
      service  = "api",
    },
  ]
}

// Read those files.
loki.source.file "app" {
  targets    = local.file_match.app.targets
  forward_to = [loki.process.app.receiver]
}

// Process: stitch multiline entries, pull out the log level.
loki.process "app" {
  stage.multiline {
    firstline     = "^\\d{4}-\\d{2}-\\d{2}[ T]\\d{2}:\\d{2}:\\d{2}"
    max_wait_time = "3s"
  }

  stage.regex {
    expression = "\\s(?P<level>TRACE|DEBUG|INFO|WARN|ERROR|FATAL)\\s"
  }

  stage.labels {
    values = { level = "" }
  }

  forward_to = [loki.write.central.receiver]
}

// Ship to Loki.
loki.write "central" {
  endpoint {
    url = "http://loki-host:3100/loki/api/v1/push"
  }
}

The shape is always the same: find files, read files, process, write. The labels I set in local.file_match — host, env, service — ride along with every line, and they're what you'll select on later. constants.hostname fills in the box's hostname so you don't have to hardcode it per machine, which matters when you're templating this across a fleet.

Reload after editing:

sudo systemctl reload alloy
journalctl -u alloy -f

Watch that journal the first time. It'll tell you if it can't find files, can't reach Loki, or doesn't like your config — and one of those three is almost always your first problem.

Where "mixed-OS fleet" stops being a footnote

The happy-path tutorial assumes every box is the same and every log is readable. A real fleet is neither. Here's what actually slowed me down.

Permissions, every time. Alloy runs as the alloy user. Your application logs are very often owned by the app's user and group, mode 0640, which means a process that isn't root and isn't in that group can't read them. Alloy starts up clean, reports no errors, and ships nothing — because it can't open the file, and "can't read it" looks a lot like "nothing to read." The fix is to put the alloy user in the app's group:

sudo usermod -aG tomcat alloy
sudo systemctl restart alloy

I lost more time to this than I'd like to admit, precisely because it fails silently. If a host is shipping nothing and the journal looks fine, check whether alloy can actually cat the file. That's the first thing now.

The box that's two years past EOL. Every fleet has one. The distro's so old the vendor stopped shipping security patches, nobody's volunteering to migrate it, and it is of course running something important. The agent will usually still install, but the older init and a couple of defaults don't behave like they do on a current release, and you don't want a routine package update yanking the version out from under you. I pin the package version on those boxes and treat them as their own little special case rather than pretending they're like the rest.

Ripping out the old shipper. Most fleets that have been around a while already have some log agent — a leftover vendor thing, an old Promtail, a Filebeat somebody set up in 2019. Leave it running next to Alloy and you've got two processes tailing the same files, double-shipping, and a config nobody remembers the login for. Part of this rollout was finding and removing those:

sudo systemctl stop legacy-log-agent
sudo systemctl disable legacy-log-agent

Decommissioning the old thing is genuinely part of the job, not a cleanup task for later. Two agents disagreeing about what they've shipped is a debugging session you don't want.

Multiline: the parsing that bites everyone

Here's the one that everybody hits. A Java stack trace is one logical event spread across forty physical lines. By default, Alloy treats every line as its own log entry — so your stack trace lands in Loki as forty separate, context-free entries, and when you go looking for the error during an incident, you find the first line and none of the cause.

The stage.multiline block fixes it, and the whole thing turns on one field: firstline. That regex describes what the start of a new entry looks like. Anything that doesn't match gets appended to the entry above it. So if your log lines begin with a timestamp:

stage.multiline {
  firstline     = "^\\d{4}-\\d{2}-\\d{2}[ T]\\d{2}:\\d{2}:\\d{2}"
  max_wait_time = "3s"
}

a new line that starts with 2024-01-15 10:23:45 is a new entry, and the indented at com.example... frames underneath it get glued on where they belong. One readable event instead of forty fragments.

Now the part that actually cost me time. One firstline regex does not cover a mixed fleet, because different applications start their lines differently. One service logs ISO timestamps. Another leads with a bracketed [2024-01-15T10:23:45]. A third uses a syslog-style Jan 15 10:23:45. A regex tuned for the first silently fails on the others — and "silently" is the dangerous word. It doesn't error. It mostly looks fine. The single-line entries parse correctly, so the dashboards look healthy, and you don't find out the multiline is broken until you're staring at a stack trace shredded into thirty rows in the middle of an outage. That is the worst possible moment to learn your firstline regex was written for a different app.

So I stopped trying to write one config to rule them all. Each application family gets its own firstline pattern, and I template the Alloy config per app-type — same skeleton, the firstline and the paths swapped in. The discipline is: when you onboard a new kind of app, go look at three or four raw log lines and confirm what the start-of-entry marker actually is before you trust the multiline. Thirty seconds of looking saves you that incident.

One more knob: max_wait_time is how long Alloy waits for continuation lines before it gives up and flushes what it has. Too short and you'll occasionally guillotine the tail of a slow-written stack trace. A few seconds is a sane default.

Querying it: LogQL

With logs flowing, you query them with LogQL. It reads like PromQL's cousin. You start by selecting a stream with labels, then filter the text:

{env="prod", service="api"} |= "ERROR"

That's "lines from the prod api service containing ERROR." The label selector narrows down which streams Loki even looks at — that's the cheap part — and the |= filters the text. A few more that I reach for constantly:

{env="prod"} |= "ERROR" != "favicon"          # errors fleet-wide, minus the noise
{service="api"} | json | status >= 500         # parse JSON logs, filter on a field
{host="web-03"} | level="ERROR"                 # the label we extracted earlier
sum by (service) (rate({env="prod"} |= "ERROR" [5m]))   # error rate per service

That last one is a metric built straight out of logs, which is where Loki starts earning its place beyond grep-in-a-box: you can alert on it.

Logs in Slack

The dashboard is fine, but the thing that got people to actually adopt this was the Slack bot. Nobody wants to context-switch to a browser to answer "is the api throwing errors right now." So the bot queries Loki's HTTP API and posts results back in the channel.

The core is one function that hits Loki's query_range endpoint:

async function queryLoki({ query, sinceMs, limit = 50 }) {
  const end = `${Date.now()}000000`;            // ns; Loki wants nanoseconds
  const start = `${Date.now() - sinceMs}000000`;

  const url = new URL("http://loki-host:3100/loki/api/v1/query_range");
  url.searchParams.set("query", query);
  url.searchParams.set("start", start);
  url.searchParams.set("end", end);
  url.searchParams.set("limit", String(limit));
  url.searchParams.set("direction", "backward"); // newest first

  const res = await fetch(url);
  const body = await res.json();

  // result: [{ stream: {labels...}, values: [[ts_ns, line], ...] }]
  return body.data.result.flatMap((s) => s.values.map(([ts, line]) => ({ ts, line })));
}

A small gotcha lives in that first line. Loki timestamps are nanoseconds, and Date.now() gives milliseconds, so you'd naturally write Date.now() * 1e6. Don't — that number is bigger than JavaScript's safe integer range and you'll get a subtly wrong, rounded timestamp that makes your queries return the wrong window. Building the string by hand (${Date.now()}000000, six zeros for ms-to-ns) sidesteps the whole floating-point problem. This one is not in the docs; it's in the "why are my time ranges off" stage of debugging.

Now wire it to a slash command. Using Slack's Bolt framework, a /logs command with a tiny argument parser handles the cases people actually ask for:

import pkg from "@slack/bolt";
const { App } = pkg;

const app = new App({
  token: process.env.SLACK_BOT_TOKEN,
  signingSecret: process.env.SLACK_SIGNING_SECRET,
});

// "/logs service=api ERROR"        -> {service="api"} |= "ERROR", last 15m
// "/logs tail service=api"         -> same selector, last few lines
// "/logs service=api ERROR yesterday" -> shift the window back a day
function parse(text) {
  const tokens = text.trim().split(/\s+/).filter(Boolean);
  const selectors = [];
  const filters = [];
  let sinceMs = 15 * 60 * 1000;
  let limit = 50;

  for (const t of tokens) {
    if (t === "tail") { limit = 20; }
    else if (t === "yesterday") { sinceMs = 24 * 60 * 60 * 1000; }
    else if (t.includes("=")) {
      const [k, v] = t.split("=");
      selectors.push(`${k}="${v}"`);
    } else {
      filters.push(`|= "${t}"`);
    }
  }

  const query = `{${selectors.join(", ")}} ${filters.join(" ")}`.trim();
  return { query, sinceMs, limit };
}

app.command("/logs", async ({ command, ack, respond }) => {
  await ack();
  const { query, sinceMs, limit } = parse(command.text);

  let rows;
  try {
    rows = await queryLoki({ query, sinceMs, limit });
  } catch (err) {
    await respond(`Query failed: \`${query}\``);
    return;
  }

  if (rows.length === 0) {
    await respond(`No matches for \`${query}\` in the window.`);
    return;
  }

  const block = rows
    .reverse()                       // oldest -> newest reads better
    .map((r) => r.line)
    .join("\n")
    .slice(0, 3500);                 // stay under Slack's message limit

  await respond("```

\n" + block + "\n

```");
});

await app.start();

So in any channel: /logs service=api ERROR gives you the last fifteen minutes of API errors, tail trims it to a handful of recent lines, and yesterday shifts the window when you're chasing something from the day before. The parser is deliberately dumb — label-ish tokens become selectors, bare words become line filters, two keywords adjust the window. It covers the questions people actually ask, and it's small enough to extend in an afternoon when they ask for more.

(If you want to take this further, the same Loki results make a clean input to an LLM for "summarize what went wrong here" — but that's a different post.)

Before you trust it in production

A few things I'd make sure are handled before leaning on this:

Object storage, not local disk. Point Loki at S3 or equivalent. Local filesystem is for the demo; it doesn't survive the node, and it doesn't scale.
Retention, set on purpose. Decide how long you keep logs and configure retention_period to match — for compliance reasons, for cost reasons, or both. Don't let it default into "forever" by accident.
Label cardinality is the trap. This is the Loki mistake. Labels are an index; every unique combination of label values is a separate stream. Put something high-cardinality in a label — a request ID, a user ID, a raw timestamp — and you'll blow up the index and bring Loki to its knees. Keep labels low-cardinality (host, env, service, level) and filter on everything else inside the query with |= and | json. When in doubt, it's a filter, not a label.
Alert on patterns. Once error rate is a LogQL metric, wire an alert to it — a Grafana alert rule into a Slack channel, say — so a spike pages you instead of waiting to be noticed. The logs being centralized is step one; the logs telling you something's wrong before a human looks is the actual payoff.

Wrapping up

The rollout itself is the easy part: install Alloy, point it at your files, label the streams, ship to Loki. The work that doesn't show up in the quickstart is the fleet being a fleet — the permission that fails silently, the EOL box that needs special handling, the old agent you have to evict, and the multiline regex that's wrong for half your apps and won't admit it until the worst possible moment. None of those are hard once you know to look. They're just the difference between "it worked on my one test box" and "it works across everything I actually run."

What you get at the end is worth the fuss: every line from every server in one place, a query language that turns logs into alertable metrics, and a Slack command that means people check the logs because it's easy instead of avoiding it because it isn't. The day an incident hits and the answer is one /logs away instead of a tour of five machines, the whole thing pays for itself.

Putting an OpenAI-Compatible Gateway in Front of AWS Bedrock

Azeem Siddiqui — Mon, 22 Jun 2026 14:24:46 +0000

The request that kicked this off was simple enough: the developers wanted an AI coding assistant in their editors. The constraints were where it got interesting. No AWS credentials on laptops. No third-party SaaS that meant a data-handling review. And whoever set it up had to be able to answer "who spent what" when the bill showed up.

AWS Bedrock solved the backend question. The models are good, inference stays inside our account, and there's nothing to sign off on legally. The trouble is everything sitting in front of Bedrock.

Here's the mismatch. Bedrock has its own API and authenticates with AWS SigV4. The tools developers actually reach for — Continue.dev, Cursor, aider, a big chunk of the LangChain ecosystem — all speak the OpenAI API instead: POST /v1/chat/completions, an Authorization: Bearer header, done. So if you want your team using their preferred tools against Bedrock, something has to bridge those two worlds.

The naive bridge is to mint an IAM user per developer so their editor can sign requests. Don't do this. You'll have AWS credentials scattered across a dozen machines, no per-person attribution, and no way to throttle or cut off one person without going and editing IAM policies. I went a different way: a small gateway that speaks OpenAI on the front, Bedrock on the back, and owns auth, rate limiting, audit logging, and cost tracking in between.

I've been running a version of this in production for a team of about a dozen developers. What follows is a clean, self-contained build of it — you can have it answering requests in an afternoon.

What we're building

Developer tools (Continue.dev, Cursor, aider, curl)
        │   OpenAI format + Bearer key (sk-...)
        ▼
┌──────────────────────────────────────┐
│              Gateway                  │
│   auth → rate limit → translate →     │
│   call Bedrock → translate back →     │
│   audit + cost log                    │
└──────────────────────────────────────┘
        │   SigV4, one IAM role
        ▼
        AWS Bedrock (Converse API)

One IAM role lives on the gateway and nowhere else. Developers hold a sk-... key that I issue and can kill from one place. Every request passes through a single service, which is exactly where you want auth, throttling, and accounting to sit. That last point is the whole argument for doing this, so I'll keep coming back to it.

Before any code: should you even build this? LiteLLM has a proxy that covers a lot of the same ground, and AWS publishes a bedrock-access-gateway sample. Look at both. I went custom because I wanted auth tied to our own notion of identity, audit logs in a shape our existing tooling already understood, and per-developer cost numbers rather than a per-key blob — and because the translation layer turned out to be about 80 lines. If your needs are generic, use the off-the-shelf option and save yourself the maintenance. If you want to own the four things in the title, it's a weekend, and you'll know every line.

Prerequisites

Node.js 18+ and an AWS account with Bedrock model access turned on. (You enable it per-model in the Bedrock console. For most models it's effectively instant.)
An IAM role or user the gateway can assume, with bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream on the model ARNs you plan to serve.
Comfort with Express. Nothing fancy.

I'm using SQLite here so the whole thing runs with zero external services. In production I back it with a managed database instead, but the schema doesn't change — you swap the driver and move on.

Step 1: Project setup

mkdir bedrock-gateway && cd bedrock-gateway
npm init -y
npm install express @aws-sdk/client-bedrock-runtime express-rate-limit better-sqlite3

Add "type": "module" to package.json so the ES module imports work.

Give the gateway its AWS credentials however you normally would — an instance role if it's on EC2 or ECS, environment variables if you're running it locally to test. The one rule: these credentials live on the gateway and only on the gateway.

export AWS_REGION=us-east-1

Step 2: The data layer (keys + usage)

Two tables. One maps an issued key to a developer. One records every call so you can audit it and bill it back.

db.js:

import Database from "better-sqlite3";

export const db = new Database("gateway.db");

db.exec(`
  CREATE TABLE IF NOT EXISTS api_keys (
    id          TEXT PRIMARY KEY,
    name        TEXT NOT NULL,
    key_hash    TEXT NOT NULL UNIQUE,
    active      INTEGER NOT NULL DEFAULT 1,
    created_at  TEXT NOT NULL DEFAULT (datetime('now'))
  );

  CREATE TABLE IF NOT EXISTS usage (
    id            INTEGER PRIMARY KEY AUTOINCREMENT,
    developer_id  TEXT NOT NULL,
    ts            TEXT NOT NULL DEFAULT (datetime('now')),
    model         TEXT NOT NULL,
    input_tokens  INTEGER NOT NULL,
    output_tokens INTEGER NOT NULL,
    cost_usd      REAL NOT NULL,
    latency_ms    INTEGER NOT NULL,
    streamed      INTEGER NOT NULL,
    status        INTEGER NOT NULL
  );
`);

The keys table stores a hash of the key, never the key itself. You see the raw value once, at creation, and it's unrecoverable after that — same deal as a password. If gateway.db ever leaks, no usable key leaks with it.

Step 3: Issuing keys

A throwaway CLI so you can hand a developer a key without building a UI first:

create-key.js:

import crypto from "node:crypto";
import { db } from "./db.js";

const name = process.argv[2];
if (!name) {
  console.error("usage: node create-key.js \"developer name\"");
  process.exit(1);
}

const raw = "sk-" + crypto.randomBytes(24).toString("hex");
const id = crypto.randomUUID();
const keyHash = crypto.createHash("sha256").update(raw).digest("hex");

db.prepare(
  "INSERT INTO api_keys (id, name, key_hash) VALUES (?, ?, ?)"
).run(id, name, keyHash);

console.log(`Created key for ${name}`);
console.log(`Key (shown once, store it now): ${raw}`);

node create-key.js "Jordan"
# Key (shown once, store it now): sk-1f3c...e9a2

The sk- prefix is cosmetic, but it earns its keep. Some OpenAI-compatible clients sanity-check the shape of the key, and it tells the developer this drops into the same field where an OpenAI key would go. Small thing, fewer support pings.

Step 4: Auth middleware

Pull the bearer token, hash it, look it up. Attach the developer to the request so everything downstream — the rate limiter, the audit log — knows who's calling.

auth.js:

import crypto from "node:crypto";
import { db } from "./db.js";

const lookup = db.prepare(
  "SELECT id, name FROM api_keys WHERE key_hash = ? AND active = 1"
);

export function authenticate(req, res, next) {
  const header = req.get("authorization") || "";
  const match = header.match(/^Bearer\s+(.+)$/i);
  if (!match) {
    return res.status(401).json({
      error: { message: "Missing bearer token", type: "invalid_request_error" },
    });
  }

  const keyHash = crypto.createHash("sha256").update(match[1]).digest("hex");
  const dev = lookup.get(keyHash);
  if (!dev) {
    return res.status(401).json({
      error: { message: "Invalid API key", type: "invalid_request_error" },
    });
  }

  req.developer = dev;
  next();
}

I'm matching OpenAI's error envelope — { error: { message, type } } — on purpose. Client tools parse that shape. Return a bare string and some of them surface a vague generic failure instead of your message, and then you're debugging "the gateway is down" tickets that were really just expired keys. Ask me how I know.

Step 5: The translation layer

This is the part that does the actual work. OpenAI's chat format and Bedrock's Converse format are close, with three differences that'll trip you up if you don't watch for them:

Converse roles are only user and assistant. The system prompt goes in its own top-level system field, not in the messages array.
Converse message content is an array of blocks — [{ text: "..." }] — not a plain string.
Inference params live under inferenceConfig, and the keys are renamed (maxTokens, not max_tokens).

bedrock.js:

import {
  BedrockRuntimeClient,
  ConverseCommand,
  ConverseStreamCommand,
} from "@aws-sdk/client-bedrock-runtime";

const client = new BedrockRuntimeClient({
  region: process.env.AWS_REGION || "us-east-1",
});

// Friendly names the clients send -> real Bedrock model IDs.
// Check current IDs with: aws bedrock list-foundation-models
// IDs and regional availability move around. If you need a model in a
// region that doesn't host it directly, look at cross-region inference
// profiles (the "us." / "eu." prefixed IDs).
export const MODELS = {
  "claude-3-5-sonnet": "anthropic.claude-3-5-sonnet-20240620-v1:0",
  "claude-3-5-haiku": "anthropic.claude-3-5-haiku-20241022-v1:0",
};

// OpenAI content can be a string or an array of parts. Normalize to text.
function extractText(content) {
  if (typeof content === "string") return content;
  if (Array.isArray(content)) {
    return content
      .filter((p) => p.type === "text")
      .map((p) => p.text)
      .join("");
  }
  return "";
}

// OpenAI request body -> Converse arguments
export function toBedrock(body) {
  const modelId = MODELS[body.model];
  if (!modelId) throw new ModelNotFound(body.model);

  const system = [];
  const messages = [];

  for (const m of body.messages) {
    if (m.role === "system") {
      system.push({ text: extractText(m.content) });
    } else {
      messages.push({
        role: m.role, // "user" | "assistant"
        content: [{ text: extractText(m.content) }],
      });
    }
  }

  const inferenceConfig = {};
  if (body.max_tokens != null) inferenceConfig.maxTokens = body.max_tokens;
  if (body.temperature != null) inferenceConfig.temperature = body.temperature;
  if (body.top_p != null) inferenceConfig.topP = body.top_p;
  if (body.stop != null) {
    inferenceConfig.stopSequences = Array.isArray(body.stop) ? body.stop : [body.stop];
  }

  return {
    modelId,
    ...(system.length ? { system } : {}),
    messages,
    ...(Object.keys(inferenceConfig).length ? { inferenceConfig } : {}),
  };
}

export class ModelNotFound extends Error {}

const FINISH = {
  end_turn: "stop",
  stop_sequence: "stop",
  max_tokens: "length",
  tool_use: "tool_calls",
  content_filtered: "content_filter",
};

export function finishReason(stopReason) {
  return FINISH[stopReason] || "stop";
}

export async function converse(args) {
  return client.send(new ConverseCommand(args));
}

export async function converseStream(args) {
  return client.send(new ConverseStreamCommand(args));
}

Step 6: Cost calculation

Bedrock bills per token, at a rate that depends on the model. Keep a price table and compute cost from the usage Bedrock hands back. Prices change, so I keep this in one file and treat it as config, not as numbers buried in the request path.

pricing.js:

// USD per 1M tokens. Confirm against current Bedrock pricing before trusting it.
const PRICES = {
  "claude-3-5-sonnet": { input: 3.0, output: 15.0 },
  "claude-3-5-haiku":  { input: 0.8, output: 4.0 },
};

export function costUsd(model, inputTokens, outputTokens) {
  const p = PRICES[model];
  if (!p) return 0;
  return (inputTokens / 1e6) * p.input + (outputTokens / 1e6) * p.output;
}

Step 7: Wiring it together

Now the server pulls it all in: auth, a per-developer rate limiter, the two routes, and the audit-and-cost write that fires on every request.

server.js:

import express from "express";
import crypto from "node:crypto";
import rateLimit from "express-rate-limit";
import { db } from "./db.js";
import { authenticate } from "./auth.js";
import {
  toBedrock,
  converse,
  converseStream,
  finishReason,
  MODELS,
  ModelNotFound,
} from "./bedrock.js";
import { costUsd } from "./pricing.js";

const app = express();
app.use(express.json({ limit: "2mb" }));

const recordUsage = db.prepare(`
  INSERT INTO usage
    (developer_id, model, input_tokens, output_tokens, cost_usd, latency_ms, streamed, status)
  VALUES (?, ?, ?, ?, ?, ?, ?, ?)
`);

// One audit + cost write per request. Lives in a finally block so it runs
// even when a streaming client hangs up mid-response.
function audit(req, { model, usage, latencyMs, streamed, status }) {
  const input = usage?.inputTokens ?? 0;
  const output = usage?.outputTokens ?? 0;
  const cost = costUsd(model, input, output);

  recordUsage.run(
    req.developer.id, model, input, output, cost, latencyMs, streamed ? 1 : 0, status
  );

  // Structured line for log shipping. Metadata only — no prompt or
  // completion text. Logging content is a privacy/compliance call you
  // make on purpose, not a default you back into.
  console.log(JSON.stringify({
    evt: "llm_request",
    developer: req.developer.name,
    developer_id: req.developer.id,
    model, input_tokens: input, output_tokens: output,
    cost_usd: Number(cost.toFixed(6)), latency_ms: latencyMs,
    streamed, status,
  }));
}

app.use("/v1", authenticate);

// Per-developer rate limit. The keyGenerator is the whole trick — it makes
// the limit per-person instead of per-IP. Run more than one instance and
// you'll want a shared store (rate-limit-redis), or each instance counts
// its own window and the real limit is N times what you set.
app.use("/v1", rateLimit({
  windowMs: 60_000,
  limit: 60,
  keyGenerator: (req) => req.developer?.id || req.ip,
  standardHeaders: true,
  legacyHeaders: false,
  handler: (req, res) => res.status(429).json({
    error: { message: "Rate limit exceeded", type: "rate_limit_error" },
  }),
}));

app.get("/v1/models", (req, res) => {
  res.json({
    object: "list",
    data: Object.keys(MODELS).map((id) => ({
      id, object: "model", owned_by: "bedrock-gateway",
    })),
  });
});

app.post("/v1/chat/completions", async (req, res) => {
  const started = Date.now();
  const model = req.body.model;
  const streamed = req.body.stream === true;
  let usage = null;
  let status = 200;

  try {
    const args = toBedrock(req.body);

    if (!streamed) {
      const out = await converse(args);
      usage = out.usage;
      res.json(toCompletion(model, out));
    } else {
      usage = await streamCompletion(res, model, args);
    }
  } catch (err) {
    status = err instanceof ModelNotFound ? 400 : 502;
    const payload = {
      error: {
        message: err instanceof ModelNotFound
          ? `Unknown model: ${model}`
          : "Upstream model error",
        type: err instanceof ModelNotFound ? "invalid_request_error" : "api_error",
      },
    };
    if (res.headersSent) res.end();
    else res.status(status).json(payload);
  } finally {
    audit(req, { model, usage, latencyMs: Date.now() - started, streamed, status });
  }
});

function toCompletion(model, out) {
  return {
    id: "chatcmpl-" + crypto.randomBytes(12).toString("hex"),
    object: "chat.completion",
    created: Math.floor(Date.now() / 1000),
    model,
    choices: [{
      index: 0,
      message: {
        role: "assistant",
        content: out.output.message.content.map((b) => b.text || "").join(""),
      },
      finish_reason: finishReason(out.stopReason),
    }],
    usage: {
      prompt_tokens: out.usage.inputTokens,
      completion_tokens: out.usage.outputTokens,
      total_tokens: out.usage.totalTokens,
    },
  };
}

app.listen(3000, () => console.log("Gateway listening on :3000"));

Step 8: Streaming

Streaming is where gateways tend to get sloppy, and it's also where two of the four concerns get tested at once. Bedrock's ConverseStream gives you an async iterable of events; you translate each text delta into an OpenAI SSE chunk and flush it right away. The catch that got me the first time: token counts only show up in the final metadata event. If you care about cost tracking — and the entire point here is that you do — you have to let the stream finish, or at least catch that event, even when the client has already walked away.

Add this to server.js:

async function streamCompletion(res, model, args) {
  res.writeHead(200, {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache",
    Connection: "keep-alive",
  });

  const id = "chatcmpl-" + crypto.randomBytes(12).toString("hex");
  const created = Math.floor(Date.now() / 1000);
  const send = (delta, finish = null) => {
    const chunk = {
      id, object: "chat.completion.chunk", created, model,
      choices: [{ index: 0, delta, finish_reason: finish }],
    };
    res.write(`data: ${JSON.stringify(chunk)}\n\n`);
  };

  let usage = null;
  const out = await converseStream(args);

  send({ role: "assistant" }); // first chunk carries the role, no content

  for await (const event of out.stream) {
    if (event.contentBlockDelta?.delta?.text) {
      send({ content: event.contentBlockDelta.delta.text });
    }
    if (event.messageStop) {
      send({}, finishReason(event.messageStop.stopReason));
    }
    if (event.metadata?.usage) {
      usage = event.metadata.usage; // this is the part you can't skip
    }
  }

  res.write("data: [DONE]\n\n");
  res.end();
  return usage;
}

Two details that matter. The first chunk announces { role: "assistant" } with no content — clients expect the role up front, once. And the terminating data: [DONE] line is part of the OpenAI contract; tools wait for it to know the stream closed cleanly, and they'll hang if it never comes.

The reason the usage capture lives where it does: that function returns usage, and the route's finally block writes it. So a developer who cancels a long generation halfway through still gets billed for the tokens Bedrock actually produced. That's the difference between cost numbers that tie out at the end of the month and cost numbers that quietly run low and make you look like you don't know what your platform costs.

Testing it

Plain curl, no streaming:

curl http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer sk-YOUR-KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-3-5-sonnet",
    "messages": [{"role": "user", "content": "Say hello in five words."}]
  }'

Streaming — note the -N so curl doesn't buffer:

curl -N http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer sk-YOUR-KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-3-5-haiku","stream":true,
       "messages":[{"role":"user","content":"Count to five."}]}'

And the test that actually matters — pointing a real editor at it. In Continue.dev, the gateway is just an "openai" provider with a custom base URL:

{
  "models": [
    {
      "title": "Bedrock Gateway",
      "provider": "openai",
      "model": "claude-3-5-sonnet",
      "apiBase": "http://your-gateway-host:3000/v1",
      "apiKey": "sk-YOUR-KEY"
    }
  ]
}

The developer never has to know Bedrock is back there. They get chat and autocomplete in their editor. You get a key you can revoke, a limit you can tune, and a usage row for every single call.

Checking the books

Cost tracking is only worth anything if you can read it back. The reason every usage row carries a developer id is so questions like "who's driving the bill this month" are one query and not an afternoon:

SELECT
  k.name,
  COUNT(*)                  AS requests,
  SUM(u.input_tokens)       AS input_tokens,
  SUM(u.output_tokens)      AS output_tokens,
  ROUND(SUM(u.cost_usd), 2) AS cost_usd
FROM usage u
JOIN api_keys k ON k.id = u.developer_id
WHERE u.ts >= datetime('now', '-30 days')
GROUP BY u.developer_id
ORDER BY cost_usd DESC;

I wire this into a dashboard and a monthly summary, but even the raw query does the job — it catches a runaway script the day it starts instead of the day the invoice lands.

What I'd add before calling it done

The build above is honest about what it is: a clean core, not a hardened production deployment. A few things I'd want in place before I trusted it with a real team:

Shared rate-limit state. The in-memory limiter resets per process. Two instances behind a load balancer means a developer's real limit is double what you configured. Move the window into Redis with rate-limit-redis.
A real database. SQLite is great for the tutorial and fine for one small instance. For anything multi-instance, point db.js at managed Postgres or MySQL. The schema is identical; it's a driver swap.
Content logging is a decision. This gateway logs metadata only. The moment you start logging prompts and completions, you've created a store of possibly-sensitive text with its own retention, access, and compliance questions hanging off it. Choose that deliberately, write down why, and tell your developers either way.
Budget guardrails. Usage rows let you see spend. A hard stop is a separate thing — add a per-developer monthly cap checked in the auth path, and set an account-level AWS budget alert as a backstop. When the app's number and AWS's number disagree, something in your accounting is wrong, and you want to find out from a graph, not a finance meeting.
Key lifecycle. You've already got active = 0 for revocation. Add rotation and an expiry column so contractor keys age out on their own instead of living forever.
Health and timeouts. A /healthz route for your load balancer, and a request timeout so one stuck upstream call doesn't pin a connection open indefinitely.

Wrapping up

The whole thing is around 200 lines, and it converts "every developer holds AWS credentials and spend is a mystery" into "every developer holds a key I issued, every request is attributed to a person, and I can throttle or cut off anyone from one place." The translation layer — OpenAI in, Converse out — is small and it stays put. The four behaviors in the title aren't bolt-ons; they're the reason the service exists. Auth is one middleware. Rate limiting is one more. Audit and cost are a single write per request. You get all of it precisely because every call funnels through one service you control.

If your requirements are generic, save yourself the upkeep and try LiteLLM or the AWS sample first. But if you want to own these four things end to end, it's a weekend of work — and the first time something goes sideways in production, knowing exactly why each piece is there pays for itself.