You want to add AI to your Chrome extension.
The obvious path: spin up a Node.js server, hold a master API key, charge users monthly, eat the AI c...
For further actions, you may consider blocking this person and/or reporting abuse
These days, just BYOM, everyone uses AI, they have their favorites, so let them use their favorites, but allow them to experiment with the rest. A single Google OAuth is good enough to sign them in to just about all of them. Users are also scared of 'companies stealing their data', they dont much care that Google, OpenAI, Anthropic, SpaceXAI all do the same... But if you're going to leak your own data, atleast let it hide in a crowd of billions?
But it's their choice. The best you can do as a provider, is provide a choice. It lowers your running costs, it lowers your liability and it creates customer trust that the app is there as a tool for the user, not for data harvesting.
Really good point on BYOM — and honestly the direction I'm leaning for the next iteration. Supporting OpenRouter already gets you most of the way there (30+ models, one key, user picks their favorite). The OAuth angle is interesting though: the friction of "paste a key" vs "sign in with Google" is real, and for less technical users OAuth wins every time.
The privacy paradox you're describing is accurate and a bit uncomfortable to sit with. Users are worried about my server seeing their code, while their chosen AI provider is training on it anyway (unless they're on an API tier with opt-out). The honest answer is that BYOK gives them control and transparency, not actual privacy — they can audit exactly what leaves their browser and to whom. That's different from trusting a black box, even if the end result is similar.
The liability point is the one I don't see discussed enough. As a solo developer, not holding a master API key means I'm not a target. If someone's key gets misused, it's between them and their provider. That asymmetry matters a lot at small scale.
What's your stack? Curious if you've shipped BYOM in production and what the OAuth + multi-provider flow looks like in practice.
For the API end, Essentially the model garden models are the third parties, those require different api calls, so you create a template, for each, then tie it to the provider's URL in the model declaration, so anthropic means claude, etc. That way you have set API structure for each model type and it means as the time goes on, if they release new models, they can upgrade the second it releases, not wait a week for a patch.
Really appreciate you walking through the Doccit setup in that level of detail — that's a genuinely different shape of BYOM than what I expected.
The GCP service account handoff is clever — it sidesteps the "paste a raw key" friction entirely while still keeping you out of the liability chain, since the user is provisioning their own service account against their own billing. That's BYOK and OAuth-style delegation at the same time, which I hadn't considered as a combination.
The per-document-profile model selection is the part that stands out most. Storing "this model works best for this document type" as a persistent decision tied to the profile, rather than a single global setting, is more practical than what most BYOK implementations do. Did that come from observing real usage — users manually overriding the default often enough that you made it sticky per profile — or was it designed in from the start?
The admin-assigns-allowed-models / user-picks-within-that-set pattern solves the enterprise procurement problem cleanly. "So someone doesn't select Opus" made me laugh, but it's a real cost-control need that most BYOK writeups (mine included) don't address — I've only been thinking about individual users, not teams with a budget owner.
On the per-provider templates: that's effectively the same abstraction I'm using (one config object per provider, swap endpoint/model), just one layer deeper since you're proxying through Vertex's Model Garden instead of calling providers directly. Makes sense that new model releases become a config change rather than a code change either way.
Curious — with Gemini API being "pretty bad" as you mentioned, did that push usage toward the Model Garden third-party models (Claude, etc.) more than you expected? Interesting signal if Google's own routing layer ends up making the case for competitors' models within their own product.
Design from the start, 6 years ago when I started with the original Doccit, I had tried out DocTR which had different datasets you could preload. Results were Master worked for most docs, but failed on others, whereas FUNSD worked well on some outliers, but was much faster, list goes on. That was the original switch, which dataset to use in general. Fast forward, I had to rebuild Doccit from scatch because the source code died with my ssd (Yes, I know, use Git, I do now... I had it locally hosted and a power surge took out my NAS and SSD on dev pc).
The move from Gemini to Vertex AI, was more a matter of use-case. Gemini api is great for basic chatbot support, but it's not viable for any heavy workloads (concurrency and quota), whereas Vertex AI opened that door up, while providing better solutions for regional regulations (that happened since the original), namely, data privacy laws that state financial docs need to stay in the same geographical location, unless you completely sanitize all potential identifiers (which is impossible to guarantee), so for liability's sake, it had to move to Vertex AI, for rate's sake, it had to move to Vertex AI, configurability was just a bonus really. Especially when you compare gemini 3.1 flash lite vs Claude Haiku... Lite is cheaper, faster, higher rate limit, availability and success rate than Haiku, so for 99% of cases, you'd just use the default Lite, but the option is what people care about. If a doc scans poorly and they have a million of them, they dont want to struggle with corrections, they'd sooner pay the premium by choice to upgrade that specific doc type to a higher model. When dealing with financial docs, mistaking a 0 for an o costs more in liability for the customer than it does to just pay an extra 50% to ensure it passes. For the company using it, it's the difference between spending 30 seconds reviewing and 30 minutes reviewing each doc for correctness. Either case, it's worth the premium, but it's never a forced premium, it's a choice.
That being said, if you want to cost optimize, Cloudflare worker ai is generally cheaper I mean you can run Kimi K2.7 for the price of 3.1 flash... a 1t param model...
The SSD/NAS story is the kind of thing that should be required reading before anyone skips backups — glad Doccit survived the rebuild, even if the rebuild itself sounds brutal.
The regulatory angle is the part I hadn't considered at all. I've been thinking about BYOK purely as a privacy/trust lever for the user, but you're describing something stricter: data residency as a hard compliance requirement, where the choice of provider isn't about quality or cost, it's about which jurisdiction the data is legally allowed to touch. That's a different category of constraint than anything I deal with — PR diffs don't have geographic residency laws attached to them. Financial documents clearly do.
The "0 vs o" example is a great way to make the stakes concrete. It reframes the whole model-selection problem: you're not optimizing for average accuracy, you're optimizing for the cost of being wrong on the tail end. A 30-second review vs a 30-minute review is the real unit of value there, not tokens or latency.
The Cloudflare Workers AI / Kimi K2.7 pointer is new to me — I hadn't clocked that a trillion-parameter model was reachable at that price point. Have you actually moved any production traffic onto it, or is it still in the "watching the price/performance curve" phase? I ask because for my use case, the calculus is different (Groq's free tier means $0 is already on the table for most users), but I'm curious whether Workers AI is reliable enough yet for anyone to trust it as a primary path rather than a cost-optimization experiment.
On the streaming question you floated at the end, the cleaner path most people land on now is an offscreen document. The service worker still gets killed around 30 seconds in mid-stream, but an offscreen doc gives you a context that actually stays alive to hold the fetch open, and it relays tokens back over messages the same way you're already doing. The service worker's only job becomes spawning that doc and tearing it down when the stream ends. One caveat before you commit, though. You can only have one offscreen document at a time, so concurrent streams need a small queue. Solid writeup on the BYOK trade-offs otherwise, the cost table makes the case better than any pitch could.
Nazar — this is exactly the kind of detail that makes the difference between "works in dev" and "works in production." Thanks for sharing it.
The offscreen document approach is something I've been meaning to test, but the 30-second service worker termination window is real. The pattern you're describing (SW spawns the doc → doc holds the fetch → relays tokens back) is cleaner than what I have now, which is a hack that keeps the SW alive longer than it should.
One caveat I'd add to your caveat: the "one offscreen document at a time" limit means queueing concurrent streams, but in practice for PR Focus, I've rarely seen more than one stream running simultaneously. Most users are reviewing one PR at a time.
A question back to you: have you tested how the offscreen document behaves with Chrome's memory pressure policies? My concern is that if the browser is under memory pressure, the offscreen document might get terminated just like the SW, albeit with a longer grace period. Do you know if it's treated differently by the browser's lifecycle policies?
Also worth noting: the offscreen document API is still relatively new (Chrome 109+). For developers supporting older Chrome versions, the SW-only approach might still be the safer compatibility bet — even with the 30-second termination risk.
I'll be testing the offscreen document approach this week. If it holds up, Build Log #008 is coming.
Thanks again for the sharp observation — this is exactly the kind of feedback that makes writing these articles worthwhile.
Nice approach, BYOK makes sense for cutting backend cost and improving privacy, but onboarding is still the main friction point, especially for non-dev users.
Overall solid dev-first architecture, especially with MV3 streaming workaround.
Exactly — onboarding friction is the number I watch most closely. Install-to-first-AI-call is the metric that matters, not install count.
The mitigation that's moved the needle most: making every non-AI feature work without a key, so the first session has value before setup. Users who get value in session one come back for session two and actually configure the key. Users who hit a setup wall in session one don't.
For non-dev users specifically, I think BYOK is the wrong architecture — you'd want OAuth or a managed key. But for a tool aimed at developers who review code on GitHub, "get a free Groq key in 2 minutes" is a friction level they'll tolerate.
The question is always: does your user understand what an API key is and do they care enough about privacy to go get one?
What's your use case? Building something or evaluating architectures?
This is a nice breakdown of the BYOK tradeoff. Removing the backend simplifies cost and privacy, especially for developer tools that may touch private code or PR diffs, but it also shifts responsibility to onboarding, key storage, quota errors, and provider-specific UX. I like that you included where the approach breaks down instead of presenting it as universally better. For production extensions, I’d also think about traceability: when an AI review looks wrong, users need enough local context to understand what diff, prompt, and provider response produced it.
Raju — you've hit on something I deliberately left out of the main article because it deserves its own treatment, but you're absolutely right: traceability is the missing piece in most BYOK writeups.
When an AI review produces a bad summary or a risk score that doesn't match the reviewer's intuition, the user needs to answer three questions:
Currently, PR Focus logs all three locally in IndexedDB with the PR ID as the key. If a user flags a bad summary, I can ask them to export the log and see exactly what happened — but that's manual.
What I'd like to build next: a "debug view" in the extension that shows the prompt, the diff sent, and the raw provider response for any PR. That turns "this AI review is wrong" from a support ticket into a self-serve investigation.
The constraint: logging the full prompt + diff + response can get large quickly. I'm thinking of a rolling log — keep the last 20 AI calls, purge older ones. For most users, that's enough to debug a bad review.
One more thought: the traceability requirement changes depending on who's using the tool. For individual developers, "I trust it or I don't" is enough. For teams, you need to be able to audit why a review was flagged as high-risk. That's a different scale of requirement.
I documented the initial tracing approach in Build Logs, but I'd like to improve it. Do you have a specific tracing setup in your projects that you've found works well? Always curious to see what others are doing in this space.
the streaming implementation has a subtle chunk boundary issue worth knowing before it bites you.
decoder.decode(value)gives you whatever bytes arrived in this read, which does not always align with SSE event boundaries. if a chunk ends in the middle of a line, yourchunk.splitcall silently drops the tail and the next chunk starts without that continuation. in practice this shows up as occasional token drops that look like the model cutting itself short, because the logic still runs without throwing. the fix is a line buffer across reads: accumulate the incomplete fragment in a variable and prepend it before splitting the next chunk. the malformed chunk catch block handles parse failures but not partial line carryover, which is a quieter failure mode. have you seen unexplained short responses in production streams that might trace back to this?Mudassir — you're right, and I should have caught this earlier. The chunk boundary issue is real and exactly as quiet as you describe. I had seen occasional short responses that I attributed to model behavior, but your diagnosis makes more sense: partial line carryover that the catch block doesn't handle because it never throws.
The line buffer fix is the correct approach. Here's what I'm moving to:
The
{ stream: true }flag ondecodealso matters — it tells the decoder not to flush the internal state between calls, which handles multi-byte UTF-8 sequences that might split across chunk boundaries as well.I've included this fix with credit to you in the next article I'm publishing — it's specifically about decisions I'd make differently building PR Focus, and the streaming bug fits exactly there. Thanks for taking the time to write it up properly instead of just flagging that something was off.
Super interesting approach! I'm curious how you
Thanks! Appreciate the interest — and I'm genuinely curious about what you were going to ask.
The comment cut off, but I'd love to hear what part you're curious about. Some possible angles:
If there's a specific piece you'd like me to go deeper on, just ask. I've been running this in production for ~6 weeks and have hit most of the edge cases.
Or if you're working on something similar and want a second pair of eyes on the implementation, happy to take a look — the Summer Review Swap is open right now if you want to exchange a PR review.
Either way, thanks for reading and for the engagement 🙌