tl;dr: We released Milliseconds.ai, a multimodal API for AI decisions, classification and extraction from text and images. It grew out of models we'd built for another product. Here's how that happened.
Watching TypeSafe AI launch Jev this week, we saw a totally new use case for models we already had.
Jev returns typed decisions: labels, probabilities and source spans that software can use directly. A question like "is this refund request legitimate?" gets a boolean and a score rather than a written explanation.
CloudRaker's Paperwork API has been using small models for similar work for months: classifying pages, checking statements against contracts, and extracting fields from forms. Those calls take tens of milliseconds. We'd treated the models as part of our document processing service and hadn't thought much about offering them separately.
The response to TypeSafe's launch made us reconsider that stragy: developers seemed interested in an API for these small decisions, even without the rest of a document workflow attached.
Paperwork already handled API keys, login, per-organization rate limits and billing. Our first attempt was to expose the decision routes through its existing gateway.
Where the time went
The Paperwork gateway handles two-hundred-page PDFs, signature envelopes that stay open for weeks, and redaction jobs spread across three workers and different GPUs. Each request goes through authentication, fine-grained authorization, tenant context, logging, tracing and metering. That overhead barely matters for a forty-second job. For a model call that takes a few milliseconds, it can dominate the response time.
We benchmarked the decision routes and checked the traces in Dash0. In the worst cases, the authorization calls, context propagation, logging and network hops took twelve times as long as inference.
We gave the decision API its own service so we could shorten that path. Building Milliseconds.ai, the API for decision-machine-1, took two days. The models and supporting services already existed; the work was in changing how a request reached them.
The request path
Each request needs authentication, an organization-level usage check, and an available inference slot. Afterwards, we need to record the usage for billing. We put a single Cloudflare Worker in front of the models, with Durable Objects for organization state and inference scheduling.
The Worker runs Hono. Bindings connect it to D1 for hashed keys, Analytics Engine for request metrics, and our metering service.
Two Durable Objects
The scheduler
A GPU runner has a limited number of concurrent inference slots. We track them in a scheduler Durable Object, one per GPU region, using a location hint to place it near the runners.
The scheduler keeps the slots in memory. It prefers GPU slots and falls back to CPU slots when the GPUs are full. A request takes a lease, POSTs through the tunnel and releases the lease when the answer comes back. Requests wait in a queue when all slots are busy; if none becomes available in time, the client gets a 529 and a retry hint.
If a call fails, we skip its slot for thirty seconds and send the retry to a different GPU host.
The scheduler also manages the spot fleet. It requests more GPUs as slots fill up and releases them when demand drops. That keeps us from paying for a fixed fleet sized for peak traffic.
Its state is temporary: an in-flight lease keeps the object alive, and a cold start rebuilds the pool. We don't persist the scheduler's state.
The namespace
A Milliseconds.ai key looks like sk-ms-{namespace}-{entropy}, where the namespace is the organization id. Including it in the key lets the Worker find the right Durable Object without a database lookup.
Each organization's namespace object mirrors its keys from the database into local storage and memory. It maintains two token buckets, for requests per minute and input tokens per minute, plus a usage ledger in fifteen-minute buckets.
For an admission check, the Worker sends the token count calculated from the request body. The object returns an admission, rate-limit or billing-block verdict, along with OpenAI-style x-ratelimit-* headers. The Worker caches that verdict, which I'll get to below.
Usage goes to our metering Worker over a service binding. That Worker deducts credits in Schematic and returns the billing verdict. The namespace object stores it for subsequent admission checks.
We reused Paperwork's Schematic setup for plans, credits, entitlements and the customer-facing usage page. Adding Milliseconds.ai meant configuring a credit type and burn rate, then calling it from the metering Worker. Billing took an afternoon, and four hours of that afternoon went into arguing about what a token should cost.
Caching admission checks
A Durable Object call adds a round trip. To avoid one admission check per request, the Worker caches the verdict per key for sixty seconds in the Cache API, at each location that sees the key. The cache holds the key id, any block reason, and the rate-limit headers.
We record usage after sending the response. Combined with the cached verdict, this means a burst can exceed the limit before a block takes effect. We accepted that delay to keep admission checks off most requests' critical path.
The tunnel
The GPUs aren't on Cloudflare. They're spot instances spread across managed instance groups in several regions.
Each VM runs two containers: the inference runner and cloudflared. We configure the tunnel on Cloudflare's side, and each VM joins it with a token.
The scheduler reaches the runners with fetch() through a Workers VPC binding. We don't have to expose a public endpoint for each runner. When a VM gets preempted, its connector drops and the tunnel continues serving through the others.
Images
Paperwork already used these models to classify scanned pages and extract fields from forms. Milliseconds.ai reuses that image support: calls such as yes-no, classify, rate and extract accept images alongside text and return typed results.
Images go in the request body as base64. We don't accept image URLs because fetching from another server would add latency outside our control. Three detail tiers resize the longest edge to 512, 768 or 1024 pixels. Each tier has a fixed token cost, so the namespace object can check usage before inference. Image decisions take roughly 45 to 120 ms depending on the tier.
Images pass through the Worker and tunnel to the runner, where they're processed in memory. We don't write customer images to disk or include them in logs. That also avoided adding an image store to manage and include in our compliance scope.
What we reused
Milliseconds.ai uses our existing Worker template, configuration conventions, three environments and CI. It publishes an API spec that generates a typed client, gets secrets from our shared vault, and deploys through our release pipeline.
Traces go to Dash0, where we could compare a Milliseconds.ai request with a Paperwork request from the first day. We also reused our access policies for admin surfaces.
Those choices mattered for our SOC 2 Type II controls as well as for development time. Milliseconds.ai uses the deployment accounts, secret store, access policies and logging infrastructure we already operate at CloudRaker. We could build within that setup instead of establishing a separate one for the new product.
What took two days
The title leaves out months of work on Paperwork. We spent two days assembling a product from models and infrastructure we'd already built, with most of the new work going into scheduling, admission checks and the path to the GPU.
I had wondered whether the service templates, generated configuration, secret sync and infrastructure as code were too much ceremony for a small, bootstrapped team. They certainly took longer than setting up any one service by hand. This time, they let us concentrate on the part that needed to change: getting a short inference request through the system without spending most of its time in our gateway.
Milliseconds.ai is live, with a free plan if you'd like to try it. The API docs cover the calls and include examples.
Top comments (0)