I've been meaning to do this for a while. I have a small job that runs a language model over my own blog posts, and I'd been running models like that locally with Ollama. This morning I moved it to DigitalOcean's serverless Inference and kept notes as I went. It reads like a video log, minus the video.
The rule I set myself: same model on both sides, same prompts, same code. Only the place the model runs changes.
Early morning, the account is blocked
My first call of the day to DigitalOcean Inference:
402 {"id": "Payment Required", "message": "This account has been blocked"}
Yesterday I drained my prepaid inference credit in the middle of another experiment, and I hadn't topped it up. That's how prepaid works here. When the balance hits zero, calls stop. There's no surprise bill at the end of the month. I topped up at 10:27, and at 10:31 the same kind of call came back 200.
Before that, picking the model
I wanted a model that exists in both places, so the comparison is fair. DigitalOcean's model list has openai-gpt-oss-20b, and Ollama has gpt-oss:20b. Both are OpenAI's open-weight 20B model. It also fits on my laptop, an M4 MacBook with 24 GB of memory.
brew install ollama
ollama pull gpt-oss:20b
The download is 13 GB. I started it while I was still sorting out the credit.
10:34, the job
Something real and small. I have 57 published posts on dev.to, and each has four tags I chose myself. The job asks the model to suggest four tags for each post from its title, description and the first 1,500 characters of the text. Then I count how many match the tags I actually picked.
That gives me a score I didn't make up: my own tagging. It's a harsh score. If the model says nodejs and I tagged node, that counts as a miss. But it's the same harsh score for every run.
The script talks to any OpenAI-compatible endpoint, and both Ollama and DigitalOcean's Inference are OpenAI-compatible. The base URL, model name and key come from the environment:
BASE_URL = os.environ["BASE_URL"]
MODEL = os.environ["MODEL"]
API_KEY = os.environ.get("API_KEY", "ollama")
10:34, first local call
I sent a two-word prompt to warm the model up. It took 21.9 seconds, most of it loading 13 GB into memory. The second call took 1.9 seconds.
10:35, the local run starts
All 57 posts, one after another. It finished at 10:52, 17.1 minutes later.
- median time per post: 16.6 seconds
- slowest 10%: 29.1 seconds or more
- memory used by the model server: 14 GB, more than half of the laptop
- tags matching mine: 91 of 228, which is 40%
- posts with at least one matching tag: 51 of 57
gpt-oss is a reasoning model, so most of each call goes into thinking before it answers. On a laptop that thinking is slow. The answers were fine.
I didn't wait for it to finish. Ten minutes in, about halfway through, I got impatient and started the migration alongside it.
10:45, the migration
This is the whole migration. Three environment variables:
BASE_URL=https://inference.do-ai.run/v1
MODEL=openai-gpt-oss-20b
API_KEY=<a DigitalOcean model access key>
No code changes. Not a new SDK, and not a different response format to parse.
10:45 to 10:47, the same run on DigitalOcean
The same 57 posts took 2.1 minutes. It finished while the local run was still going.
- median time per post: 2.1 seconds
- slowest 10%: 3.1 seconds or more
- tags matching mine: 93 of 228, which is 41%
- posts with at least one matching tag: 51 of 57
- cost: $0.0068 for all 57 posts, at $0.05 per million input tokens and $0.45 per million output
Same accuracy, about eight times faster, and none of the work happened on my laptop.
One detail I liked: both runs counted exactly 31,451 input tokens across the 57 prompts. Same model, same tokenizer, the same prompts. That's a decent sign the two really are the same model.
10:47, a false alarm
I grepped the results file for "error" and got one hit. My heart sank for a second. It was a tag. For a post about QR codes, the model suggested errorcorrection.
10:47 to 10:51, the model I can't run at home
The real reason to move is that the bigger models become a one-word change. openai-gpt-oss-120b wouldn't fit in my laptop's memory. On DigitalOcean it's just another model name:
MODEL=openai-gpt-oss-120b
- tags matching mine: 103 of 228, which is 45%
- posts with at least one matching tag: 54 of 57
- median time per post: 3.9 seconds
- cost: $0.010 for all 57 posts, at $0.10 per million input tokens and $0.70 per million output
It was a bit slower than the 20B model and still four times faster than my laptop. It also wrote less reasoning: 9,722 output tokens against 11,685 for the 20B.
What the misses looked like
Many of the "wrong" tags were reasonable. nodejs where I used node, postgresql where I used postgres, python on posts where Python was only the language of the test script. All three runs also suggested digitalocean on posts where I hadn't tagged it. In fairness to the models, those posts were about DigitalOcean.
What I got wrong
I tried to download all 57 posts from the dev.to API at once and got 429 Too Many Requests after a few. I had to slow down to about one request a second.
I wrapped the warm-up call in timeout, which isn't installed on macOS by default, so the first attempt didn't even run.
The grep that scared me at 10:47 was my own sloppiness. I should have checked for an error key, not the word anywhere in the line.
Yesterday's blocked account was my mistake too. I knew the credit was small and didn't estimate the experiment's cost before starting it. This time I did: 57 short prompts on these models cost about a cent each run.
Caveats
This is one laptop, one job, and one run of each. A faster Mac, a smaller quantized model or a GPU box would close some of the speed gap. Ollama ran with its default settings. The timings from DigitalOcean include the network round trip from Bulgaria. The two DigitalOcean runs overlapped with the end of the local run. They only use the network, so I don't think they slowed the laptop down, but I didn't measure that. My scoring counts exact tag matches, so it undersells all three models. The cost figures are my own token counts multiplied by DigitalOcean's published prices. The month-to-date usage on my account moved by 7 cents over the morning, which also covers my test calls.
Where I ended up
For this job, DigitalOcean was the better place to run the model. Same model, same answers, about eight times faster, under a cent for the whole run, and a bigger model one variable away. My laptop got its 13 GB and its memory back.
When I was done I uninstalled Ollama and deleted the model. If I ever want it local again, the three environment variables go back the other way.
Top comments (0)