DEV Community: Ashish Mishra

Building a Disposable Notion Agent on Cheap Models

Ashish Mishra — Thu, 20 Aug 2026 03:47:10 +0000

TL;DR: We built a one-shot HTTP worker that talks to Notion through MCP. Version one worked. Version two got cheaper and more readable, then failed in a new way. The harness was fine. The tool surface, the model, and the prompt were not the same problem, and we kept treating them as one.

We keep seeing the same pitch: put an agent in the cloud, give it tools, let it live in Slack, let it remember you. That is a product. It is not the product we needed.

We needed something dumber and more useful. Another service should be able to say "read this Notion page, write a summary somewhere, stop." No chat history. No personality that accretes over weeks. No always-on process. If nobody is calling it, it should cost nothing.

We started calling that shape a one-shot agent. One HTTP request. Tools for that request. A JSON result. Then the instance can go away.

This is the path we actually walked: first working version, what it got wrong, the Markdown fork, and the cheaper tricks that mattered more than swapping frameworks.

The job was never "build a chatbot"

The first real task was almost boring. Once a week, pull a skill write-up from Notion, extract what mattered, and append it to a digest page. Callers would name pages in English. They would not paste Notion ids. If a name was ambiguous, the agent should refuse to write rather than guess.

If that loop is wrong, people stop trusting write-back. If it is expensive, nobody schedules it. If it needs a human to babysit a terminal, it is not a system.

So the constraints were social as much as technical:

An external caller owns the schedule. The agent does not.
The agent must be allowed to use tools, not just talk about them.
Secrets stay in the environment, never in the request body.
Idle time should be free.

Question you will probably ask: why not a cron script that hits the Notion API directly?

Because the task changes every call. This week it is a weekly digest. Next week it is "list in-progress rows and do not write." We did not want a new Python file per job. We wanted one worker that takes a prompt plus a list of tool servers.

Common pitfall: starting from a coding agent (shell, files, memory, a learning loop) and trying to shrink it into a request/response worker. You spend months deleting features you never wanted.

The shape we committed to

The loop is short on purpose.

caller
  |  POST /run  { system prompt, user prompt, which tools }
  v
one-shot agent  (starts if needed, dies when idle)
  |  model + tool loop
  v
MCP server for Notion
  |  integration token
  v
Notion API
  |
  +--> JSON back: result, tools used, tokens, cost

A few choices fell out of that picture.

HTTP, synchronous. The caller waits. A weekly digest can wait two minutes. We did not want a job queue for v1.

The agent is not the Notion client. Notion (and later Slack, GitHub, whatever) lives behind MCP: a small server that exposes tools. The agent image stays thin. The Notion token never enters the agent process as "the caller's header."

Search first. Callers say "Weekly Skill," not a 32-character id. The model searches, picks a title match, and stops if it cannot.

A cheap default model behind an OpenRouter-style gateway. We did not want a Claude bill on every internal trigger. The first default was a very cheap DeepSeek flash model, on the order of a few cents per million tokens.

Decision: keep the harness thin (validate request, attach tools, run the loop, return JSON). Put intelligence in the model, the tool server, and the prompt. If quality is bad, we change those three before we change the loop.

That last sentence sounds obvious. We still almost violated it.

What we looked at and walked past

A hosted Notion MCP

Notion's hosted connector is built for Claude Desktop and similar clients. It wants OAuth. We wanted an internal integration token in a secret store and a server we could run next to the agent. Self-hosting the official Notion MCP was the boring path that matched that.

Identity for every caller

The "correct" cloud version is: every caller has a service account, mints a short-lived token, the platform checks it. That is great when all callers live in your cloud. It is miserable when the next caller is a script on a laptop or a third-party job.

We shipped a shared API key on the agent instead. Cloud ingress is open. The app checks the header and fails closed. Onboarding is "here is a key." Revoke is "rotate the key."

Tradeoff: you lose per-caller identity until you grow a key set. We accepted that for v1.

A heavier harness

When quality wobbled, the tempting move was "use a real agent": something with memory, skills, a terminal, a personality that improves. We looked hard at that family (the persistent, self-improving runtimes people mean when they say they want Hermes-class agents).

They are good at being companions. They are the wrong shape for "POST, work, return, stop." They want state on disk. They want to stay up. They would make our Cloud Run bill and our threat model worse for a problem we did not have.

Decision: do not swap the loop because the model cannot name a tool. A thin loop plus OpenRouter already is the harness. Quality lives one layer up.

Baking the weekly-skill job into the system prompt

This one is ideological, and it showed up in a real request.

We had a generic worker. Someone sent a specific Notion page and a specific question ("summarize in-progress tasks"). The easy fix is to stuff that page name, that Status property, that persona into the system prompt.

That prompt then becomes unusable for the next job. The agent is no longer one-shot and generic. It is a weekly-skill bot wearing a generic coat.

The pattern we now follow: two layers.

How to drive the tools (stable). Search by name. Read before writing. Do not invent ids. If markdown shows a database stub, query the database. Do not write unless asked.
What to do this run (the caller). The actual task lives in the user prompt.

Claude Desktop users barely write layer 1, because a strong model infers it from the tool list. A cheap model will not. Layer 1 has to be explicit, and it still must not contain this week's page title.

Version one: named tools and a pile of JSON

The first MVP was honest and a little ugly.

Official Notion MCP, with a flat list of named tools (search, retrieve page, append blocks, and friends).
The model could see those names and call them.
Reads came back as Notion's raw JSON block trees.
Default model: the cheapest flash-class ID we could point at.
Guardrails: 12 model steps, 20 tool calls, two minutes on the clock.
Infra: two small containers that scale to zero. Notion's API is free for internal integrations. The bill that matters is tokens.

It worked. Search found pages by title. A digest got appended. A successful run we logged was on the order of 40,000 input tokens, a few hundred output tokens, and about a quarter of a cent.

For a weekly job, that is fine. For something you want to call all day, you start staring at the 40k.

What version one actually got wrong

The failure was not "it does not work." The failure was "it works in a way that does not travel."

JSON is a terrible reading format for a summarizer. Claude's own Notion connector feels good because it reads Markdown. We were stuffing block trees into a cheap model and asking it to sound like a colleague. Input tokens were the tax. Quality was the interest.

The cheap model was good enough for named tools. That hid a landmine. When every tool has a clear name, flash-class models can finish in a handful of steps. We thought we had "the model problem" solved. We had only solved it for a flat tool list.

Deploy was a human ritual. Build three images on a laptop, remember linux/amd64 on Apple Silicon, push, hope. One arm64 image and Cloud Run will not start. That is not an agent problem. It still blocks the agent.

We almost taught callers the wrong lesson. Version one accepted a vague system prompt. Official tools are discoverable. Callers copied that prompt to the next server. Then nothing worked, and it looked like the new server was broken.

Question you will probably ask: why not raise the step cap when a run dies?

Because a confused loop is a money printer. Twelve steps is a fuse. If the model is guessing tool names, more steps buys more guesses. Fix the prompt and the model first. Raise the cap last, and only for jobs you have already seen succeed in fewer rounds.

The Markdown experiment (this is version two's raw material)

We wanted reads that look like documents, not like API dumps. We source-reviewed two community MCP servers that return Markdown. Both were MIT, both spoke Streamable HTTP, both kept the Notion token in the environment.

	Awkoy-style (two meta-tools)	Flat Markdown server (~40 named tools)
How the model calls Notion	`execute(operation, payload)` plus a schema helper	Direct tool names
Schema size in the prompt	Tiny (hundreds of tokens)	Large
Notion 429 handling	Built-in pacing and retry	You own it
Risk	The model must name the operation	Tool-list overload

Awkoy's trick is the interesting one. Instead of 40 tools, it exposes two. Every real action is an operation name inside a payload. The tool schema that lands in the prompt shrinks from something like 17,000 tokens to something like 400.

That is a real cost win if the model can dispatch. There is almost no public evidence of people driving that two-tool surface from a flash-class model. Desktop Claude does it because Claude can. We were about to find out what DeepSeek flash does instead.

Decision: do not replace version one. Run both. Callers pick a catalog name. Official named tools stay the simple path. Markdown stays the cheap-token path. Another scale-to-zero container is roughly free when idle.

Version two: same worker, different tools, a more expensive cheap model

We pointed the same /run body at the Markdown server: same generic prompt, same flash model, use: markdown_notion instead of use: official_notion.

Two things happened, in order.

1. The run hit the step fuse

The official server exposes search as a tool the model can see. The Markdown server does not. It exposes two verbs. The model has to say search_pages inside execute. Our prompt never said that. Flash guessed, asked for schemas, retried, and burned 12 steps.

A clean Markdown run is 3 to 5 steps. A confused one is 12 and an error.

Before: named tools + vague prompt + flash = success.

After: two meta-tools + the same prompt + flash = step_limit_exceeded.

The worker did not get worse. The instructions no longer matched the tools.

2. When it did not fuse, it lied politely

get_page_markdown renders the page body. An inline database shows up as a stub: a title and an id, not the rows. The model would say "there is a database" and stop. That looks like a product bug. It is a missing rule.

Rows only come from query_database. If the user asked about tasks, status, or items, we have to query. We also cannot assume the property is called Status. Real databases are named State, Stage, a checkbox, a board.

Common pitfall: writing a "working" prompt for one database (Status = In progress) and putting it in the system prompt. The next caller has different properties. The agent filters into empty and looks broken.

Version two's actual upgrade was not a new framework. It was:

A tool primer that names the two verbs, the search-then-read loop, and the database-stub rule.
A stronger cheap model for that path (a V3-class DeepSeek, roughly three times the input price of flash, still far under frontier). Flash stays the default for named tools.
The task still lives in the user prompt. No page names in the primer.

Quality jumped because the model could finish the loop, not because we grew a memory subsystem.

We also stopped shipping images from laptops as the happy path. Merge to main builds and deploys. Terraform still owns env and IAM. The pipeline owns tags. Those two fighting each other is its own footgun (ignore image changes in Terraform, never apply Terraform from the image job).

What we would copy if we started tomorrow

These are the optimizations that paid rent. Not a toolkit dump. A short list of bets.

1. Pay for tokens where they leak.

Official JSON reads were the 40k-input problem. Markdown plus a tiny tool schema attacks that directly. Extra round-trips from a weak dispatcher can eat the savings. Measure both.

2. Price the model to the tool surface, not to the brand.

Flash is fine when tools have names. Meta-dispatch wants a model that can emit query_database on the first try. We would rather spend ~$0.21 per million input tokens and finish in four steps than spend ~$0.07 and fail at twelve. Failed runs are not cheap. They are delayed work plus a retry.

3. Keep idle at zero.

The worker and each MCP server scale to nothing. Notion's API did not add a monthly line item. The LLM bill is the bill.

4. Put a fuse on the loop.

Steps, tool calls, wall clock. Return a structured error, not a 500 with a stack trace. Log a trace id. The caller should be able to paste that id, not a screenshot of Cloud logs.

5. Split "how to use tools" from "what to do."

This is the one we would tattoo on the repo. It is also how you keep one worker generic while still running a weekly digest.

6. Search by name. Refuse to guess ids.

Callers should not hold Notion's internal ids. If two pages match, stop. Write-back to the wrong page is worse than a failed run.

7. Do not confuse "simpler implementation" with "fewer moving parts in the prompt."

A two-tool MCP is simpler for the server. It is harder for a cheap model. Simplicity moved. We had to put it back in English.

Rough numbers we actually use as a gut check (prices move; re-check before you budget):

Piece	Order of magnitude
Official MCP success, flash	~40k input, ~$0.002 to $0.01
Markdown tool schema vs official tool list	~400 tokens vs ~17k
Flash vs V3-class DeepSeek (input / 1M)	~$0.07 vs ~$0.21
Idle Cloud Run	~$0
Notion internal integration	$0 API

What we are trying next

None of this is a promise. It is where the pain is.

More tools on the same worker. Slack, GitHub, meeting transcripts. The catalog should grow without a new harness. If we cannot add a server without rewriting /run, the one-shot idea failed.

A harness that is actually seamless. Today the "harness" is still a pile of conventions: which catalog name, which model, which primer. We want a structure where a caller says the job and the worker picks a sane tool server and a sane model. Disposable should feel like calling a function, not like configuring an IDE.

A way to learn without becoming a companion. Persistent agents learn by writing skills to disk and living forever. We still want one-shot. The interesting version is: after a good run, store a primer delta or a eval fixture somewhere external. The next cold start can load it. The instance still dies.

Use it on real internal work. Weekly digest was the wedge. The point is other systems posting jobs: standups, research write-backs, "what changed in this database." If we only ever curl it ourselves, we built a demo.

Compare answers without a human reading every one. Judging the final response is the slow part now. We need a fixture shape: same prompt, same Notion snapshot (or a recorded tool trace), two models or two primers, a score that is not "the author liked it." Until that exists, we will keep arguing from anecdotes and burning evenings on single traces.

If you are in the same spot

Start with a named-tool MCP and a cheap model. Prove search, read, write, and "do not guess." Then, and only then, switch the read path to Markdown and spend a little more on the model that has to dispatch.

Keep the worker boring. Make the prompt boring in the right way: how tools work, not what this week's page is called.

And if a run starts saying "there is a database," believe the stub. Query the rows. The page was never the table.

Automated First-Pass PR Reviews - Build, Buy, or Use Your Coding Agent’s Cloud?

Ashish Mishra — Fri, 24 Jul 2026 02:52:25 +0000

TL;DR: I wanted every new PR to get an automatic first-turn brief — summary, risks, security/performance smells — before I deep-dive. After comparing SaaS review bots, building a custom agent, and using Cursor’s cloud agents from a GitHub Action, I chose the Action + cloud-agent path because I already pay for Cursor Pro+, I wanted the pipeline in my repo, and I cared more about a trustworthy first pass than a fully productized review platform.

The problem I was actually solving

Not “replace human review.” Not “ship a full AppSec program.”

I wanted a first turn: when a PR opens, something reads the change and leaves a structured brief so I’m not starting cold. Ideal shape:

What is this PR trying to do?
Bugs / logic risks worth noticing early
Security smells (secrets, auth, injection-shaped code)
Performance smells (hot paths, N+1s, wasteful work)
A clear signal: looks fine / needs changes / needs discussion

Later I also want staging previews — but that’s usually a separate pipeline. This piece is about the analysis brief.

I already had: GitHub repos + a Cursor Pro+ subscription. I didn’t have a settled opinion on build vs buy vs “use the agent product I already pay for.”

What “good enough” first-pass analysis can (and can’t) do

Almost every AI review path — bot, DIY agent, or cloud coding agent — is strong at reasoning over the diff and nearby code:

Usually catches	Usually needs extra tools
Logic bugs, edge cases, off-by-ones	Live CVE / dependency vulns (`npm audit`, Snyk, etc.)
Code smells, complexity, inconsistency	Binary / artifact scanning
Security patterns (secrets in code, missing checks)	Formal coverage gates (unless tests are run and parsed)
Performance anti-patterns in visible code	Org-wide policy engines and compliance attestations

So the industry split is less “AI vs not AI” and more:

Deterministic CI — lint, tests, build, secret scan, dependency audit
AI first-pass — narrative brief + judgment calls
Human — merge decision
Preview/staging — clickable environment (often orthogonal)

Treat the AI layer as a sharp intern’s notes, not the only gate.

Industry pattern underneath every option

Whatever logo is on the bot, the system is always:

Listen  →  something wants a review (PR open, /review, Slack)
Think   →  model + tools gather context and reason
Speak   →  comment on the PR (or Slack, or both)

If you build it yourself, you own all three.

If you buy a product, they own Think (and often Speak).

If you use a coding agent’s cloud runtime, you often own Listen/Speak and rent Think.

The main roads people take

1. Buy a PR review product (e.g. CodeRabbit and peers)

What you get: Install a GitHub App, get inline/threaded comments, summaries, sometimes learning from your repo style, dashboards, team defaults.

Strengths

Fastest path to “PRs get reviewed comments”
Productized UX (inline nits, severity, ignore rules)
Less prompt-engineering on day one

Tradeoffs

Another vendor in the trust/security review
Opinionated comment style; tuning is product settings, not “your agent”
Overlap with tools you may already pay for (IDE agents, CI scanners)
Cost scales with seats/repos/PR volume in their pricing, not your existing AI plan

Best when: You want a dedicated review product and don’t care that Think lives outside your coding agent.

2. Build your own Listen → Think → Speak system

Classic DIY: GitHub webhook or Actions → call a model API with tools (fetch PR, get file, maybe gh) → post a comment. Optional Slack: paste a PR link, get a thread reply.

Strengths

Full control of prompt, severity bar, and where results go
Can mix models, scanners, and internal APIs
Clear architecture; easy to explain to security teams

Tradeoffs

You maintain tool loops, auth, retries, prompt drift
Large diffs need careful context management
Slack needs an immediate ACK (3s) and async work
You’re on the hook for reliability (“why didn’t the bot run?”)

Best when: Review is a core workflow you want to own, or you must stay inside strict data boundaries.

3. Use your coding agent’s cloud features (e.g. Cursor cloud agents / Automations)

Newer path: the same class of agent you use in the IDE can run in the cloud against a PR — either via native Automations (PR opened → comment) or via a thin GitHub Action that starts a cloud agent through an API and posts the result.

Strengths

Think quality close to the agent you already trust day to day
Can reuse one subscription’s cloud usage (for me: Pro+)
Automations minimize glue; Actions keep the trigger in .github/workflows

Tradeoffs

Usage is token/spend based — deep reviews can run 10–15+ minutes and burn budget faster than IDE chat
API-started agents may need the Action to Speak (post the comment) even if Think succeeded
Org repos need GitHub App + Actions permissions sorted
Not a full substitute for dependency CVE scanners

Best when: You already live in that agent daily and want the first-pass voice to match.

4. Hybrid (what most mature teams quietly do)

Secret scan + npm audit / SAST in CI (Speak: check failures)
AI brief on PR open (Speak: one summary comment)
Preview deploy URL from hosting/CI (Speak: environment link)
Humans for merge

Best when: You want seatbelts and a narrative first turn.

Comparison at a glance

Path	Time to first useful comment	Ownership	Cost shape	Feels like “my agent”?	First-pass summary quality
SaaS review bot	Hours	Low	Per product pricing	Low–medium	High UX, fixed personality
DIY agent system	Days–weeks	High	Model API + your time	High	As good as your prompt/tools
Cloud coding agent + Automation	Hours	Medium	Included in agent plan usage	Medium–high	High if prompt is good
Cloud coding agent + GitHub Action	Half day–days	High	Plan usage + CI minutes	High	High if prompt is good
Hybrid	Ongoing	Medium	CI + AI + host	Medium	Best overall risk coverage

What I chose — and why

I went with GitHub Action → Cursor cloud agent → PR comment.

Not because SaaS bots are bad. Not because DIY is “more pure.” Because of my constraints:

I already pay for Cursor Pro+. I wanted the first-pass brain to be the same one I trust in the editor, not a second personality I have to learn to ignore.
I wanted a brief, not a product. One structured comment before I dive in — summary, bugs, security, performance, verdict — was enough for v1.
I wanted the trigger in my repo. A workflow file I can read, diff, and tweak beats a dashboard full of vendor toggles (for me). Automations were a strong alternative; I still preferred Option-B-style glue I own.
I accepted the honest limits. Pattern-level security/perf in the brief; real CVE scanning stays in CI later. Staging deploys stay a separate preview pipeline.
Build-from-scratch DIY was attractive in research, but I didn’t need to own the tool loop on day one when a cloud agent already knows how to explore a repo.

What the decision felt like in practice

Listen: pull_request opened / ready for review (skip drafts)
Think: cloud agent with prUrl — no analysis on the Actions runner itself
Speak: Action posts with gh pr comment --repo … (because the runner never checked out git)

The surprising lesson: people ask “how does it analyze if Actions didn’t checkout?” — it doesn’t analyze there. The cloud agent clones elsewhere. The Action is a remote control.

The painful lesson: deep first-pass reviews are slower and hungrier than IDE chat. Budget them. Read your usage dashboard after the first few runs.

The quality lesson: the prompt is the product. A generic “review this” gets generic notes. A forced structure (summary → risks → security → performance → verdict) is what makes the comment useful before you dive in.

How I’d advise someone still deciding

Ask yourself, in order:

Do I want a productized review UX (inline threads, team policies)? → Start with a SaaS bot; add CI scanners beside it.
Do I already pay for a coding agent with cloud/Automations? → Try native Automation or a thin Action before building a custom tool loop.
Do I need Slack-first ChatOps or weird internal tools? → DIY Listen/Speak; rent any Think backend.
Do I need CVE/policy gates? → Those are CI jobs either way; don’t make the LLM the only security control.
Do I need staging? → Preview deploys are usually CI/CD; let the AI brief link to the URL later.

My answer to (2) was yes, so I didn’t buy another review logo — I aimed the agent I already have at GitHub and made the first turn automatic.

Takeaways

A first-pass PR brief is a different job from full human review or full AppSec.
Every approach is Listen → Think → Speak; you’re only choosing who owns each box.
Buy for speed and review UX; build for control; reuse your coding agent’s cloud when the subscription and trust are already there.
Hybrid with deterministic scanners + AI summary + human merge is still the grown-up default.
If you automate cloud agents from Actions, own the comment step, expect multi-minute runs, and treat prompt design as the real feature.

I didn’t want more noise in the PR timeline. I wanted a calm first paragraph from something that already thinks like I do when I’m in the editor — waiting for me before I dive in.

From "You Have a Bug" to "Here's the Root Cause" - Adding AI Code Analysis to My App Review Pipeline

Ashish Mishra — Tue, 21 Jul 2026 01:33:50 +0000

TL;DR: I built an app review pipeline that classifies bugs and crashes. It was useful but stopped at "what's wrong." I extended it to answer "where in the code" and "what to change" — using a PydanticAI agent with codebase exploration tools, then made the analysis engine pluggable so it can use Grok Build, Claude Code, or OpenAI Codex as backends. Here's how I got there, what broke along the way, and how the architecture ended up.

Where I Started

I have a CLI tool called AppPulse that monitors my app's reviews and crash data. Every morning it:

Pulls new reviews from Google Play / App Store
Classifies each one with an LLM — bug, crash, feature request, performance, praise
Correlates reviews with crash data from Sentry/Firebase
Sends me a digest

apppulse run
# → "12 new reviews: 3 bugs (1 critical), 2 feature requests, 7 praise"

apppulse reviews --category bug
#  ID │ Rating │ Summary                              │ Severity
#  42 │  ★☆☆☆☆ │ Photo upload crashes for >10MB images │ critical

This was genuinely useful. I knew what users were complaining about and which crashes were affecting the most people. But the pipeline stopped at classification. When I saw "Photo upload crashes for >10MB images," I still had to:

Open the project in my IDE
Search for upload-related code
Cross-reference the stacktrace from Sentry
Mentally map the review to the actual code
Figure out what to change

For one bug, that's fine. For a batch of 5 critical bugs on a Monday morning, it's a lot of manual work before I've even started fixing anything.

I wanted the pipeline to go further — to take the classified review, point to the specific files and lines causing the issue, and hand me a brief I could act on immediately.

The First Version: A PydanticAI Agent with Codebase Tools

The core idea was simple: give an LLM agent access to my codebase through read-only tools and let it investigate the bug, just like I would.

Why PydanticAI

I considered LangGraph, LangChain, and rolling my own agent loop. PydanticAI won for three reasons:

Structured output by default. I needed the analysis to come back as a validated Pydantic model — not free-form text I'd have to parse. PydanticAI auto-validates the LLM output against the schema and retries with correction instructions if it's wrong.
Minimal dependency footprint. AppPulse is a lightweight CLI. I didn't want to drag in LangChain's dependency tree for a single-agent linear workflow.
It already supports the LLM providers I use — OpenAI, Anthropic, Ollama. No new API keys.

The Output Model

Every analysis produces a validated AnalysisBrief:

class AnalysisBrief(BaseModel):
    issue_type: str              # bug | crash | feature_request
    summary: str                 # one-paragraph assessment
    root_cause: str | None       # for bugs/crashes
    feature_rationale: str | None # for feature requests
    affected_files: list[AffectedFile]
    proposed_changes: list[ProposedChange]
    complexity: str              # small | medium | large
    risks: list[str]             # potential side effects
    testing_notes: str           # what tests to write/update

This is the contract between the analysis engine and the rest of the pipeline. Whether the analysis runs in-process or via an external coding agent, it always produces the same structure.

Four Built-in Tools

The agent gets four tools — all read-only, all operating on the local codebase:

@analysis_agent.tool
async def tool_search_code(ctx: RunContext[AnalysisDeps], keyword: str, file_extensions: str = "") -> str:
    """grep/ripgrep search — finds where relevant code lives."""
    return search_code(ctx.deps.code_path, keyword, file_extensions)

@analysis_agent.tool
async def tool_read_file(ctx: RunContext[AnalysisDeps], file_path: str, start_line: int = 0, end_line: int = 0) -> str:
    """Read source with line range, capped at 500 lines, path-validated."""
    return read_file(ctx.deps.code_path, file_path, start_line, end_line)

@analysis_agent.tool
async def tool_list_files(ctx: RunContext[AnalysisDeps], directory: str = ".", pattern: str = "") -> str:
    """Directory listing with sizes."""
    return list_files(ctx.deps.code_path, directory, pattern)

@analysis_agent.tool
async def tool_find_symbols(ctx: RunContext[AnalysisDeps], name: str) -> str:
    """Regex-based definition search — classes, functions, methods."""
    return find_symbols(ctx.deps.code_path, name)

Every tool validates paths against code_path to prevent directory traversal — no ../../etc/passwd games. The agent can explore but never escape the project root.

The Problem with V1: Too Many API Calls

The first version worked, but the agent was expensive. It would make 40-50 tool calls — listing directories, searching for broad terms, reading entire files — before settling on a hypothesis. At gpt-4o-mini rates that's still cheap, but it hit the API request limit I'd set and sometimes never produced output.

Optimization 1: The Repo Map

The biggest waste was the agent discovering project structure. It would call list_files recursively, trying to understand where things lived before it could even start investigating.

The fix: generate a repo map upfront and include it in the system prompt. The repo map is a compact text overview of every source file with its top-level symbols (classes, functions) extracted via regex:

src/services/
  ImageUploadService.java — ImageUploadService, compress, validateSize, uploadToS3
  UserService.java — UserService, getProfile, updatePreferences
src/utils/
  ImageUtils.java — ImageUtils, downscale, getExifOrientation

This gives the agent a "GPS" before it makes any tool calls. Instead of blindly listing directories, it can read the repo map and go straight to ImageUploadService.java when investigating an upload crash.

Caching the Repo Map

Generating the repo map means walking the entire source tree and reading every file for definitions. For a medium project (~500 files), that's 100-500ms. Fine for a one-off, but I didn't want to pay that cost on every analysis.

The cache strategy uses a git-based fingerprint:

def _compute_fingerprint(root: Path) -> str:
    # Git repos: HEAD commit + dirty-file count (~5ms)
    head = subprocess.run(["git", "rev-parse", "HEAD"], ...)
    status = subprocess.run(["git", "status", "--porcelain"], ...)
    dirty_count = len(status.stdout.strip().splitlines())
    return hashlib.sha256(f"{commit}:{dirty_count}".encode()).hexdigest()[:16]

Cache hit: ~0ms. Cache miss (new commit or uncommitted changes): regenerate. The cache self-invalidates on every git operation that changes the working tree.

Optimization 2: Progressive Tool Gating

Even with the repo map, the agent would sometimes keep exploring forever — finding tangential files, searching for related patterns, going down rabbit holes. Sound familiar? It's what I do when debugging at 2am.

The fix is a funnel that progressively restricts the agent's tools:

async def _prepare_tools(ctx: RunContext[AnalysisDeps], tool_defs: list[ToolDefinition]):
    tool_call_count = count_tool_calls(ctx.messages)

    if tool_call_count >= 18:
        return []           # No tools — force the agent to produce output

    if tool_call_count >= 12:
        # Only read_file — no more searching, refine what you already found
        return [td for td in tool_defs if td.name not in EXPLORATION_TOOLS]

    return tool_defs        # Full access during exploration phase

Phase	Tool Calls	Available Tools
Exploration	0-12	All 4 tools
Deep-dive	12-18	`read_file` only
Forced output	18+	None

This dropped the average analysis from 40+ tool calls to 8-12, with no noticeable quality difference. The agent just gets to the point faster.

The Bigger Step: Pluggable Analysis Backends

The in-process PydanticAI agent worked well for straightforward bugs. But for complex, multi-file issues — or when I wanted deeper analysis — I kept wishing I could point a full coding agent like Grok Build or Claude Code at the codebase and just get back a structured brief.

All of these tools support headless mode:

# Grok Build
grok -p "analyze this bug..." --cwd /path/to/code --tools "read_file,grep,list_dir" --yolo

# Claude Code
claude -p "analyze this bug..." --cwd /path/to/code --dangerously-skip-permissions

# OpenAI Codex
codex --quiet --approval-mode full-auto "analyze this bug..."

The challenge: each agent returns its output differently (JSON, plain text, structured conversations), and I still needed a validated AnalysisBrief at the end.

The Two-Stage Pipeline

The solution is a two-stage pipeline for external backends:

Stage 1: Coding Agent → Raw Markdown
Stage 2: Cheap LLM (structurer) → Validated AnalysisBrief

Stage 1 runs the coding agent with a prompt that asks it to produce a markdown analysis with specific sections (Summary, Root Cause, Affected Files, Proposed Changes, etc.). The agent uses its own tools to explore the codebase — it's much better at this than my four built-in tools because it has full access to grep, file reading, and code understanding.

Stage 2 is a single, cheap gpt-4o-mini call that extracts the markdown into the AnalysisBrief Pydantic model. No tools, no exploration — just structured extraction. Costs fractions of a cent.

The raw markdown from Stage 1 is saved as an inspectable artifact at ~/.apppulse/analyses/raw/. If the structured output looks wrong, I can check exactly what the coding agent found.

The Backend Architecture

I wanted to be able to add new backends without touching the core analysis flow. The architecture uses a lazy-loaded registry with a strategy pattern:

# backends/__init__.py
BACKEND_REGISTRY: dict[str, str] = {
    "builtin":     "apppulse.code_analysis.backends.builtin:BuiltinBackend",
    "grok":        "apppulse.code_analysis.backends.grok:GrokBackend",
    "claude_code": "apppulse.code_analysis.backends.claude_code:ClaudeCodeBackend",
    "codex":       "apppulse.code_analysis.backends.codex:CodexBackend",
}

def get_backend(name: str) -> BaseAnalysisBackend:
    module_path, class_name = BACKEND_REGISTRY[name].rsplit(":", 1)
    module = importlib.import_module(module_path)
    return getattr(module, class_name)()

Backends are imported lazily — if you never use Grok, grok.py is never loaded. Adding a new backend is three steps:

Create a module that subclasses ExternalAgentBackend
Implement _build_command() — return the argv list
Add one line to BACKEND_REGISTRY

Here's the entire Grok backend:

class GrokBackend(ExternalAgentBackend):
    name = "grok"
    display_name = "Grok Build"
    cli_command = "grok"

    def _build_command(self, prompt, cwd, max_turns):
        return [
            "grok", "-p", prompt,
            "--cwd", cwd,
            "--tools", "read_file,grep,list_dir",   # read-only
            "--yolo",
            "--output-format", "json",
            "--max-turns", str(max_turns),
        ]

Everything else — prompt assembly, subprocess management, output parsing, raw artifact saving, and the Stage 2 structurer call — is handled by the ExternalAgentBackend base class. The concrete backend only knows how to build its CLI command.

Switching Backends

In config.yaml:

code_analysis:
  enabled: true
  backend: "grok"        # swap to "builtin", "claude_code", or "codex"
  max_turns: 15           # how many exploration turns for external agents

The setup wizard auto-detects which CLIs are installed on your PATH and lets you pick:

Available analysis backends:
  1. builtin     — Built-in (PydanticAI) — lightweight, no extra tools needed
  2. grok        — Grok Build — headless mode, read-only tool access
  3. claude_code — Claude Code — deep exploration, 200K context
Choose a backend [1]:

What the Output Looks Like

Whether using the builtin or an external backend, the output is the same validated structure:

$ apppulse analyze 42

Analyzing review #42 against MyApp Android codebase...
  Code path: /Users/me/projects/myapp
  Issue type: bug
  Backend:   grok

  > Launching Grok Build agent...
  ... Agent exploring codebase (this may take 30-60s)...
  OK Raw analysis saved to ~/.apppulse/analyses/raw/MyApp_bug_20250710_091523.md
  > Structuring analysis into brief...
  OK Analysis complete

┌──────────────────────────────────────────────────────────────┐
│  CODE ANALYSIS — Review #42                                   │
│  Photo upload crashes for >10MB images                        │
│  Type: bug │ Complexity: medium                               │
├──────────────────────────────────────────────────────────────┤
│  ROOT CAUSE                                                   │
│  ImageUploadService.compress() loads the full bitmap into     │
│  memory before compression. Images >10MB exceed the heap      │
│  limit on devices with ≤4GB RAM → OutOfMemoryError.           │
│                                                               │
│  AFFECTED FILES                                               │
│  ImageUploadService.java (lines 45-89) — compress() method    │
│  ImageUtils.java (lines 12-35) — no input size validation     │
│                                                               │
│  PROPOSED CHANGES                                             │
│  1. Add BitmapFactory.Options.inSampleSize for downsampling   │
│  2. Add file size validation before compress()                │
│  3. Use streaming compression instead of in-memory byte array │
│                                                               │
│  RISKS                                                        │
│  • Downsampling reduces image quality for server-side use     │
│  • Need to test across API levels 26-34                       │
└──────────────────────────────────────────────────────────────┘

With --output file, it saves the analysis as markdown with a ready-to-paste prompt section at the bottom — hand it directly to a coding agent to implement the fix.

Watch Out For

Config nesting. The code_analysis block goes at the top level of your config YAML, not nested under an app entry. I made this mistake myself — the parser reads code_analysis from the root, so nesting it under apps[] means it silently gets ignored and falls back to defaults.
prepare_tools is called every turn. It receives the full message history, so counting tool calls is O(n) over all messages. For 25 turns this is negligible, but if you remove the request limit, it could get slow on very long conversations.
External agent stdout formats vary. Claude Code with --output-format json wraps its response in a JSON object; Codex prints plain text. The ExternalAgentBackend._extract_text() method handles both, but if you add a new backend, check what it actually writes to stdout.
Model name prefix matters. PydanticAI wants openai-chat:gpt-4o-mini (not just gpt-4o-mini). Without the prefix, you get a deprecation warning and potentially the wrong provider.

What I'm Thinking About Next

This pipeline keeps evolving. Some things on the radar:

Per-app backend config. Right now code_analysis is global. If you have multiple apps, you might want Grok for one and builtin for another. The config structure needs to support this.
Batch analysis. apppulse analyze --category bug --severity critical to analyze all critical bugs in one run instead of one-by-one.
Analysis history. Tracking which analyses led to successful fixes — creating a feedback loop that improves future analysis quality.
Auto-analyze in pipeline. Option to run analysis on all critical bugs during apppulse run, so the morning digest includes code-level briefs, not just classification.

Takeaways

Start with classification, then add analysis. Getting reviews categorized (bug vs. feature request vs. praise) is the foundation. Code analysis is the layer on top — it's valuable but depends on good classification first.
Give the agent a map before tools. The repo map cut tool calls by 60%. An LLM exploring a codebase blind will waste most of its budget on orientation. Front-load structure.
Funnel, don't limit. Progressive tool gating works better than a hard request limit. The agent naturally transitions from exploration to deep-dive to output, rather than being cut off mid-investigation.
Make the backend a strategy, not a hard dependency. The two-stage pipeline (explore → structure) decouples the exploration engine from the output format. Today it's Grok and Claude Code; tomorrow it could be any agent that can read files and write markdown.
Save raw artifacts. The raw markdown from external agents is invaluable for debugging bad analyses. If the structured output looks wrong, the raw artifact tells you whether the problem was Stage 1 (bad exploration) or Stage 2 (bad extraction).

AppPulse is a personal project I keep iterating on. The code analysis pipeline is the part I'm most excited about — it's the difference between "you have 3 critical bugs" and "here's exactly where they are and what to change." If you're building something similar, the pluggable backend pattern and two-stage pipeline might save you some architecture headaches.

I built a very cheap App Review analysis pipeline that also finds Bugs in my app code

Ashish Mishra — Mon, 20 Jul 2026 19:18:20 +0000

I got tired of reading app reviews manually every morning, so I built a command-line tool that pulls reviews from Google Play and the App Store, classifies them (bug, feature request, crash, praise), and sends me a daily summary. It costs about $0.50/month. In version 2, I added the ability to point it at my source code and get an analysis: which files are causing the bug, what to change, and how complex the fix is.

If you need to checkout code for your own setup : https://github.com/Mr-Ashish/AppPulse

The Problem

I have a few apps on Google Play. Every morning, I'd open the Play Console, scroll through new reviews, and try to figure out:

Is this a bug or a complaint?
Is this the same issue three other people mentioned yesterday?
Does this match any of the crashes I'm seeing in Firebase?

About 15 minutes of work. But it's unfocused work. Scanning text, making mental classifications, switching between tabs. On days with 20+ reviews, I'd either rush through them or skip the whole exercise.

I wanted a morning briefing:

"You got 12 new reviews. 3 are bugs (1 critical, photo upload crashes on large images). 2 are feature requests. 7 are praise. The critical bug matches a Firebase crash that affected 312 users this week."

So I built one.

Version 1: Classify and Digest

The Flow

Google Play ──┐
               → Pull new reviews→Classify review → Save to database
App Store ───┘         │             (using AI)              
                         │                                           
Firebase/Sentry ──────
---------------------------------------------------------------    
     ↓

Daily Digest                                                                             ┌──────────────────┐
│ 3 bugs (1 critical)
│ 2 feature requests
│ 7 praise
│ Top crash: OOM in
│  ImageUploadService
└──────────────────┘
      ↓         ↓
     Terminal   Slack

Classification

Each review goes through an LLM (a language model; I use OpenAI's gpt-4o-mini, their cheapest option). The LLM reads the review text and categorizes it:

Category	Meaning	Example review
bug	Something broken	"Upload button does nothing after I pick a photo"
crash	App closes or freezes	"App crashes every time I open settings"
feature_request	User wants something new	"Please add dark mode"
performance	Slow, laggy, battery drain	"App drains 20% battery in an hour"
praise	Positive feedback	"Best app I've used for this!"
complaint	General dissatisfaction	"Used to be good, now it's terrible"

Along with the category, the LLM extracts:

Severity (critical, major, or minor)
Keywords (the specific feature or area mentioned, e.g., "photo upload", "settings screen")
Functional area (which part of the app is affected)

Reviews go to the LLM in batches of 10, so one API call classifies 10 reviews at once. That's what keeps the cost low.

Cost

For an app with about 500 reviews per week:

Component	Monthly Cost
Google Play API	Free
App Store Connect API	Free
Firebase/Sentry API	Free (free tier)
OpenAI classification (~2,000 reviews/month)	~$0.50
Storage (SQLite, local file)	Free
Total	~$0.50/month

If you use Ollama (runs AI models on your own computer), the cost drops to $0.00/month.

The Tech Stack

Python , the whole tool is a Python command-line app
Click , a library for building command-line interfaces (handles apppulse run, apppulse reviews, etc.)
SQLAlchemy + SQLite , stores everything in a single local database file at ~/.apppulse/data.db
Rich , makes the terminal output clean with tables, colors, and panels
OpenAI / Anthropic / Ollama , LLM providers for classification (you pick one during setup)

No servers. No cloud infrastructure. No Docker. Run pip install and a 5-minute setup wizard.

A Typical Morning

$ apppulse run
# Pulling reviews for MyApp Android... 12 new reviews
# Classifying... done (0.8s)
# Pulling crash data from Firebase... 3 active crashes
# Correlating reviews with crashes... 1 match found

# ── Daily Digest ───────────────────────────────────────
#  12 new reviews: 3 bugs (1 critical), 2 feature requests, 7 praise
#
#  🔴 Critical: Photo upload crashes for >10MB images
#     → Matches Firebase crash: OutOfMemoryError (312 users affected)
#     → 3 reviews mention this issue
#
#  🟡 Major: Search doesn't update after applying filters
#     → 2 reviews
#
#  💡 Feature requests: Dark mode (2 requests)

Five seconds. I know what matters. No tab-switching, no scrolling through the Play Console.

Version 2: From Classification to Code Analysis

Version 1 told me what users were experiencing. It didn't tell me where in my code the problem lived or what to change.

For the critical bug above, "Photo upload crashes for >10MB images," I'd still need to:

Open the project
Search for upload-related code
Look at the Firebase stacktrace
Connect the dots
Figure out a fix

So I built a second stage: code analysis.

The Mechanism

                    apppulse analyze 42
                         │
                         ▼
        ┌────────────────────────────────┐
        │    Load the review from the    │
        │    database + crash data       │
        └───────────────┬────────────────┘
                        │
                        ▼
        ┌────────────────────────────────┐
        │    AI agent explores your      │
        │    source code with read-only  │
        │    tools (search, read files,  │
        │    find function definitions)  │
        └───────────────┬────────────────┘
                        │
                        ▼
        ┌────────────────────────────────┐
        │    Produces an Analysis Brief  │
        │    • Root cause                │
        │    • Affected files + lines    │
        │    • Proposed changes          │
        │    • Complexity estimate       │
        │    • Testing notes             │
        └────────────────────────────────┘

You point AppPulse at your app's source code folder during setup. Running apppulse analyze 42 (where 42 is a review ID) triggers the AI agent to:

Read the review and any matched crash data
Get a structural overview of your project (a map of every file and its key classes/functions)
Search the code for relevant keywords, error messages, class names, the feature area mentioned in the review
Read the specific files that look promising
Produce a structured analysis with file paths, line numbers, and proposed changes

The Output

┌──────────────────────────────────────────────────────────────┐
│  CODE ANALYSIS — Review #42                                   │
│  Photo upload crashes for >10MB images                        │
│  Type: bug │ Complexity: medium                               │
├──────────────────────────────────────────────────────────────┤
│                                                               │
│  ROOT CAUSE                                                   │
│  ImageUploadService.compress() loads the full image into      │
│  memory. Images >10MB exceed the memory limit on phones       │
│  with ≤4GB RAM, causing the app to crash.                     │
│                                                               │
│  AFFECTED FILES                                               │
│  ImageUploadService.java (lines 45-89) — compress() method    │
│  ImageUtils.java (lines 12-35) — no file size check           │
│                                                               │
│  PROPOSED CHANGES                                             │
│  1. Resize large images before loading them into memory       │
│  2. Add a file size check before compression starts           │
│  3. Use streaming compression instead of loading everything   │
│     into memory at once                                       │
│                                                               │
│  TESTING NOTES                                                │
│  Test with 1MB, 10MB, and 50MB images on a low-memory device  │
└──────────────────────────────────────────────────────────────┘

Instead of a category label, I get the file, the line, what to change, and how to test it.

Pick Your Analysis Engine

The analysis engine is pluggable. I built it with four options:

Backend	Description	Good for
Built-in	A lightweight AI agent that runs inside AppPulse, using PydanticAI (a Python framework for building AI agents with structured output)	Quick analyses, small codebases
Grok Build	xAI's coding assistant in headless mode	Fast, targeted exploration
Claude Code	Anthropic's coding assistant in headless mode	Deep multi-file investigations
OpenAI Codex	OpenAI's coding assistant	Sandboxed, autonomous mode

The external backends (Grok, Claude Code, Codex) use a two-stage approach:

The coding assistant explores the codebase and writes its findings as a markdown document
A cheap AI call ($0.001) extracts that document into the structured analysis format

Consistent output regardless of which AI did the exploration. Adding a new backend takes about 20 lines of code.

Switching is one line in the config file:

code_analysis:
  enabled: true
  backend: "grok"    # or "builtin", "claude_code", "codex"

Lessons

1. Classify before you analyze

Categorization (bug vs. feature request vs. praise) determines everything downstream. Code analysis needs good classification as input, or it wastes time investigating a feature request as if it were a bug.

2. Cheap AI models handle most of this

The classification pipeline runs on gpt-4o-mini, OpenAI's cheapest model. For categorizing app reviews, which are 1-3 sentences, it's accurate enough. Expensive models are overkill.

3. Give the AI a map of your codebase

The biggest analysis quality improvement came from generating a project map upfront, a compact overview of every file and its key components. Without the map, the AI spent most of its budget figuring out where things lived. With it, the AI goes to the relevant files on its first move. A GPS for the codebase.

4. Save raw output

Every analysis saves the AI's raw output as a markdown file. If the structured summary looks off, I can check what the AI found and pinpoint whether the problem was in the exploration or the extraction.

5. No infrastructure required

The whole tool runs as a single command-line app. The database is a SQLite file on disk. The cache is a text file. No server, no Docker container, no cloud deployment. I run apppulse run by hand or on a daily schedule.

Up Next

Batch analysis , analyze all critical bugs at once instead of one at a time
Per-app configuration , different analysis settings for different apps
Auto-analysis , run code analysis on critical bugs during the daily pipeline, so the morning digest includes root causes alongside classifications
Tracking outcomes , connect analyses to fixes, so the tool can learn which investigations led to successful resolutions

The classification half took a weekend. Code analysis took another. The pipeline costs less per month than a cup of coffee, and it replaced 15 minutes of unfocused tab-switching with a 5-second terminal readout.

Your Team Can Build Dashboards — But Can You Share Them Safely?

Ashish Mishra — Mon, 20 Jul 2026 19:17:14 +0000

TL;DR: Non-tech teammates can already build useful internal dashboards — often as a single HTML file with AI help. The real problems are sharing, secrets, and control. We solved that with a small publish app, two repositories, and a clear rule: code goes through review; passwords never go into Git.

The Problem

More teams can create internal tools than ever before. Someone in support, ops, or finance asks an AI assistant for “a dashboard that shows X,” downloads one HTML file, and it works on their laptop.

Then reality hits:

How do we share it? Emailing files and “open this on your machine” does not scale.
Where do API keys go? Those keys unlock company systems. Pasting them into a file that gets copied around is a security incident waiting to happen.
Who checked it? One edit can break the dashboard — or worse, leak data. Someone should review changes before the whole company uses them.
Who can open the link? It should be company people only, not anyone on the internet who finds the URL.

So the issue is not “can we build dashboards?” It’s can we publish them like a product: safely, repeatedly, without forcing every creator to learn Git.

What We Were Aiming For

Think of it as internal websites for ops dashboards, not a full developer platform.

Goal	In plain terms
Easy publish	Drag a file into a simple UI — no terminal
Safe secrets	Keys live in a locked vault (database), not in the file people share
Review before live	Changes go through a pull request (a proposed change someone can approve)
Company-only access	Sign in with work Google account
Room to grow	Same design can later support bots, coding agents, and stricter permissions

The Overall Idea

  Creator + AI                Publish app                 Two stores
 ─────────────               ────────────                ──────────
  One HTML file  ──upload──►  Company login               ┌─ Git: redacted HTML (via PR)
                              Save settings               └─ Database: config + secrets
                              Strip secrets
                                      │
                                      ▼
                              After review + merge
                                      │
                                      ▼
                              Shareable company link
                                      │
                                      ▼
                              Server calls APIs with secrets
                              (keys never shown in the page)

In one sentence: people upload a dashboard; the app saves safe settings in a database, sends a cleaned-up copy of the HTML for review in Git, and only after approval serves a company login–protected link — while secrets stay on the server.

Why Two Repositories? (The Original Thinking)

This was the biggest structural choice. Early on it was tempting to keep everything in one place: the publish website and all the dashboard HTML together.

Here’s why we didn’t.

What each repo is for

Repository	What’s inside	Who cares
Platform repo	The publish website, login, “serve this dashboard” logic, secret proxy	Engineers who ship the product
Dashboards repo	Only the HTML (and a tiny list of setting names, not values)	Reviewers, automated checks, and later AI agents that edit dashboards

Why split them

1. Different jobs, different risk

Platform code is “how publishing works.” Dashboard HTML is “what the business sees.” Mixing them means a dashboard tweak and a security fix compete in the same history. Splitting them makes reviews clearer: Is this a product change or a dashboard change?

2. Secrets and HTML must not travel together

If HTML and the app live in one pile, it’s too easy for keys to end up next to the files people download and copy. A dedicated dashboards repo has a hard rule: only cleaned-up HTML. Real keys never belong there.

3. Review that non-engineers can still understand

A pull request on a dashboards-only repo looks like: “here’s the new support board page.” Reviewers aren’t wading through unrelated website code. That matches how the team actually works — ops owns the dashboard content; engineering owns the platform.

4. Safer automation later

We knew we might want Slack bots or coding agents to open dashboard updates. Giving an automation access to only the dashboards repo is much safer than giving it keys to the whole platform.

5. Cleaner automated checks

Checks on dashboards can be simple: “is there one HTML file?”, “does this look like a leftover API key?”, “is the file too big?” Those checks don’t need to understand the full website.

Tradeoff we accepted

Two repos	One repo
Slightly more setup	Simpler at first
Clear ownership and safer automation	Faster to start, messier later
Platform releases don’t churn when dashboards change	Every dashboard edit looks like a product change

Original thinking in short: treat dashboards like content and the publish app like product. Content gets a content repo. Product gets a product repo. Secrets get neither — they get the database.

How Publishing Works (For Humans)

Sign in with your work Google account.
Upload one HTML file (title, optional team label).
The app reads any preview settings used for local testing, saves them for that dashboard, and strips secrets out of the copy that goes to Git.
It opens a pull request — a proposed change — on the dashboards repo.
Someone reviews and merges.
The owner marks it live.
Teammates open a share link, sign in, and use the dashboard.

Updates follow the same path: new upload → update the open pull request (or open a new one) → merge → live.

Security — What Can Go Wrong, and How This Helps

Risk	What it looks like in real life	How the design responds
Keys in the file	HTML emailed around with an API key inside	Keys saved encrypted in the database; Git only gets placeholders
Keys in the browser	Anyone can “View Source” and steal a key	Page only gets non-secret config; the server calls external APIs on your behalf
Open link on the internet	Company data on a URL with no login	Company Google sign-in before viewing
Silent go-live	Broken or unsafe change hits everyone at once	Pull request + merge before live
One mega-repo blast radius	A bot or leak touches the platform and every dashboard	Two repos; automation can be limited to dashboards only

The simple rule we tell creators:

Config (folder IDs, public URLs) can appear on the page. Secrets (API keys, tokens) never do — the server holds them.

The same HTML can still work on a laptop for preview: it uses a local preview block when it’s not on the platform, and platform-provided config when it is. Creators aren’t maintaining two different apps.

Architecture at a Glance

                    ┌─────────────────────────────────────┐
                    │           Platform app              │
   Creator ───────► │  Upload · Settings · Google login   │
                    │  Serve link · Secret proxy          │
                    └───────────┬─────────────┬───────────┘
                                │             │
                    save config │             │ redacted HTML
                    + secrets   │             │ as a pull request
                                ▼             ▼
                         ┌──────────┐   ┌──────────────┐
                         │ Database │   │ Dashboards   │
                         │ (locked) │   │ Git repo     │
                         └────┬─────┘   └──────┬───────┘
                              │                │
                              │   after merge  │
                              └───────┬────────┘
                                      ▼
                               Teammate opens
                            share link (+ login)

Three ideas to remember:

Git holds the dashboard pages (after review).
The database holds settings and secrets (encrypted).
The platform app is the only door: login, upload, serve, and proxy.

Tradeoffs (Honest Ones)

Choice	Upside	Downside
Pull request before live	Safer, reviewable history	Not “instant publish”
Secrets in the database + server proxy	Keys stay off pages and out of Git	Slightly more setup for API connections
Two repositories	Clear ownership, safer bots later	Two places to understand
Company sign-in only	Fits internal tools	Not for public / customer-facing sites
Single HTML dashboards first	Matches how people actually create with AI today	Multi-file apps come later

We deliberately deferred finer team permissions, Slack bots, and heavy hosting complexity. The skeleton — two repos, secrets out of Git, company login, review before live — is what has to be right first.

Infrastructure (Small Picture)

You don’t need a huge cloud setup for this.

Need	Typical choice
Run the publish website	A small hosted web service (for example on Google Cloud or AWS)
Database for settings and secrets	Managed Postgres
Keep the platform’s own production secrets safe	The cloud provider’s secret store

Traffic is light for an internal pilot — tens of dashboards, a handful of people using any one at a time. Cost is usually dominated by the database staying on, not by the website. Google Cloud and AWS both work; pick wherever your company already lives.

Takeaways

The bottleneck isn’t building dashboards — it’s sharing them without leaking keys or skipping review.
Treat dashboard HTML as content and the publish website as product: that’s why two repositories beat one pile of everything.
Security for humans: keys in the vault, pages for viewers, review before live, company login on the link.
Start simple on infrastructure; invest first in the rules of the road — where HTML lives, where secrets live, who can open the URL.

If your non-tech teams are already creating internal dashboards, you don’t need to stop them — you need a front door that makes the safe path the easy path.