Out of curiosity, how many of you have had concerns regarding AI agents having access to your codebase and credentials in your codebase (eg. appsettings, API keys, etc.). How many of you realized it's a problem after it's saved to cloud and I imagine on a subscription, not API where your data is kept private...
Bump if you think we need a lightweight (eg. 100m param) draft model, that is trained explicitly to filter any sensitive data and scramble it, as a middleware service, to prevent data leaks to cloud. Either that, or a tiny model to run on a codebase to replace all sensitive data with references and an interaction layer for the agentic harness to call the reference, instead of directly calling the credentials from file?
What that would mean, is essentially, your SQL connection string stays safe, your API keys stay safe, your logins stay safe, if you expand, it can keep your banking details safe, while allowing you to use your AI to it's fullest, without slowing it down?
Idk, thought crossed my mind again and I thought I'd actually ask who else has thought of it?
Top comments (21)
I have ALWAYS thought about that since the first time I used a chatbot, let alone an agent. Every time I pasted a .env file, I thought, "It would be nice if a small model replaced those values with appropriate placeholders automatically when I pasted it in." I like that your posts are shorter, by the way.
Thanks! Yeah mostly the personal posts are short, mostly cuz it's spur of the moment questionnaires meant to judge whether I expand on a thought or not, like Drone and Installer, if people are genuinely interested, then I polish them up and release them.
But yeah, draft models, literally something tiny, that can run on a modern NPU, or CPU while still being faster than a cloud LLM, so it doesnt slow anything down. Just a little 100m param model, designed strictly for text stripping, indexing and recompiling. Pascal made a post earlier about an awesome MCP project he made, where it's essentially stack overflow for LLMs, giving systems that actually work, so the same thought doesnt happen twice, it got me thinking again on the drafter idea, if it can take the cloud LLM's query, then find relevant info on the repo and return it, that'd already be a cool feature, but if it strips credentials at send time and recompiles them at receive time, so it's seamless... Then it starts becoming a pretty decent safety net. Wont be 100% accurate, or never make mistakes, but obvious things like URLs, secrets, etc. it can essentially strip them faster than the cloud model can read them... I might actually give it a go sometime, though it'd probably need to be custom trained.
About making the local model attach the relevant context, YES I have also thought about that! Specifically when I made ReptClip. That was a small CLI tool that copies the required context to your clipboard so that you can paste it into the chatbot. But the problem was that we had to manually specify which files to include using glob patterns, either when calling it or using a config file. I thought it would be nice if the tool fetched the required context automatically from the prompt. And yes, if we also make it strip the credentials, it'd be a genuinely useful product at least for me. That's something that I'll definitely use if it existed. Also, could you say which Pascal you are talking about? I searched on Google and found Blaise Pascal 😅
No, Pascal on here, sorry, @pascal_cescato_692b7a8a20 You should read his posts, they're always insightful and he's always up for answering questions!
I remember your ReptClip, it actually gave me a 3rd use-case, namely model switching. How often do you switch models during a session? Did you know your context cache disappears and your context gets polluted? What if a mini-model kept a 'compressed context' live while your agents run, so it can essentially pre-seed a model with the context, without dumping 200k+ tokens worth of context bloat it didnt work through (essentially like pasting a book into a single prompt and expect it to have read it all in the 15 seconds it takes to respond), but here's the catch, that compressed context maps to a postgres where the full context for each section is... That way if the switched model needs more info on a sector, it queries the tiny model to fetch it from the postgres, so you seed with max 20k tokens, but preserve over 400k tokens worth of knowledge in the postgres for when it needs it. Alot more efficient than having to grep through files and reading chunks of lines at a time to isolate a method or trace it's usages... And if it runs locally (even a potato can run a 100m parameter model), it's effectively free? So saves you tokens, while also improving signal to noise, especially when switching between models.
Oooh... I hadn't thought about the fact that we lose context caching when we switch models. I can't believe I missed that. However, I have thought about giving the AI the ability to dynamically remove things from its context. For example, when we solve a math problem, we often write intermediate steps in a scratchpad and discard them once we find the answer, so we can stay focused on the core problem, right? Wouldn't it be great if the AI could do that too? And what if it could also use a Postgres tool, just in case it needs any specific details like you mentioned? That would save some tokens, I guess.
Download Qoder and try out their Generate Wikis feature. What it does, is it turns your codebase into a set of human readable documents (like a git wiki), while also making itself flash cards (knowledge cards and memories), which it uses to prevent drift. Though it uses Sqlite, it still works shockingly well.
The context stripping idea I've experimented with a bit. I had a theory, if we reformat how we treat KV-Cache, then we can keep a sitemap too, that sitemap is the full-context codebase, while KV-Cache gets reserved for the small subset the agent needs immediately. So you have live context, but at a fraction of the size, without losing any depth when it needs it and it stays live due to the merkle root backing that keeps it chronological, no matter how many agents edited it. A cheap diff check that fires before it affects it's declared files just notifies it if any dependencies of that section has been changed, so it knows to re-check before it suggests an edit, so it's write-time conflict resolution, instead of merge time.
I already use Qoder because you told me to. It asked me to generate wikis, but I was busy with something else, so I skipped it. However, unlike what you said, the free tier actually ends after one month or when you run out of free credits. So I can't generate wikis right now. I can't even use the Lite model, which they say consumes "0x credits" :(
I haven't uninstalled it yet because the free AI code completion is still super useful! But since I don't have much free time right now due to college, the GitHub Copilot free tier (which actually refreshes every month) seems to be enough. So, I switch to VS Code for agentic tasks and use Qoder (and ReptClip) when I'm typing code myself. Also, regarding your theory, isn't that how many agentic tools already work, or am I missing something?
Oh yeah! Yeah I think that's their anti-abuse system, so people dont spawn a ton of fake accounts to mass-use. But if you switch to a paid plan, I can actually testify that Lite doesnt use any credits and it doesnt run out either (efficient now too). Which is really worth it imo, but if that's not an option, you could always sign up for alibaba cloud's free trial and use it through qoder? Though Generate wikis is only for auto, which costs credits...
They kinda do and they kinda dont. Qoder spawns dedicated background agents to keep it up to date at all times, vs others rely purely on the main model's discretion, eg. Claude, which just does it when explicitly told to.
Oh that theory, not quite. they store chat history, then dump that as 'pre-load' context to a different model, which needs to be processed. That's why if you run API, it immediately blows up any cost saving you get from switching to a lighter model. Hence long-running tasks are usually best kept to 1 model
Oh, so you were talking about the context of the whole codebase? I was talking about the context of what the AI did previously. I understand that they dump the chat history into the new model when we switch, but how does the codebase sitemap method fix those issues? Aren't they meant for solving a different problem?
Codebase sitemap is to keep track of live-changes and to act as a lookup table, instead of raw code-dumps. That way it acts as a minimal interface to the knowledge the model needs, so they can scope their requirements and fetch only what's relevant and once they move on, it's cleared from their state memory. That way it's always fresh and unpolluted, ontop of being a fraction of the size and always live, so stale context doesnt break implementations. A drafter would essentially take the prompt, research the sitemap for relevant methods across it, so instead of grepping hundreds of files, it directly pulls the hundreds of relevant methods, without any overshoot. Instead of copy-pasting in a single prompt 800k tokens worth of context, it moves in with 0 context, then live-fetches it via the drafter as a compiled synopsis that's precise. Right now all LLM providers just raw dump it across models, expecting that it's a '1 time cost' to put it all in cache, even though that discards changes (why a model remembers the previous state for a few passes), instead of being live.
The best way to explain it, is currently 2 models reading and modifying the same file. Eg. a single file app in python. Model A reads it, Model B reads it, Model A edits it, Model B doesnt know model A edited it and tries to apply it's edit. That immediately breaks both implementations.
Whereas with a live sitemap and a drafter, suddenly before Model B can write, it's made aware of the updated state and can adjust it's approach to match the current state of the file. That scales exponentially the more agents you add, which is why live-data is always important, if 2 hit the same spot and want to argue, they can, but atleast when they resolve it, both can continue and the rest of the agents are made aware of the change if it affects them. If each model had to keep full context, that'd immediately fall apart on cost, but with the sitemap as the ground truth and continuous cache pruning, it can keep all operations lightweight enough to do the extra work and still clear a profit over traditional systems.
That's especially true if you're switching models. You cant just copy paste the kv-cache from sonnet to opus and back to haiku, each are architecturally different, so you cant use them as drafters (in the traditional sense) each switch involves a full transplant of the entire context window as plaintext and hope it translates over well. It's usually fine stepping up, but it hardly ever works well switching down. Idk if you've tried antigravity (btw, if you're on freebie route, make 10 accounts and just hotswap them), or in copilot, when you switch models, your usage immediately skyrockets, that's the cache pre-filling with the old context fed as input tokens and even switching back to the original model doesnt preserve the old cache, it needs to re-create it from an input prompt again. Each step dilutes the context and makes it more and more hallucinated instead of genuine continuation. Kinda why model swapping always feels like a 'fresh perspective', it's not difference in the model, just the re-ingesting of all the information.
Thank You! You didn't have to explain it in such detail 🥹 This is going to take me a while to decode...
I've read through this several times, but I still have some questions.
I follow you up to this point:
What I don't understand is your point about the main model starting with zero context and fetching it dynamically from the drafter. How does that differ from current tools if the drafter is still ultimately passing context into the main model's prompt? How is it dynamic and live if that's the case?
Additionally, how is Model B made aware of Model A's changes? Is Model B sent a diff/summary of the changes, or is its entire context replaced? Also, how does this approach solve the model-switching issue?
Currently, it's mass-dumped context, or you start clean and the model reads files to understand and pulls chat history to understand intent.
Changes are registered at the tool call, because of the merkle root, which keeps track of all changes. When model B tries to execute on something related to what model A changed, eg. something that inherits from a shared enum, the tool interrupts and says 'changes here'. With a drafter, the drafter would run through the changes made and summarize how it changes it, with a snippet, which is then sent to Model B, so it can understand what changed, so it can judge it's own write before committing to it.
Model switching is traditionally just a chatlog dump, or at most, a summary, it has no index, no information that helps the model search for answers effectively. Take Claude and watch what happens when it compresses context (shrinks chatlog into a summary), it loses all reference points and it needs to re-grep large sections of files just to understand where it's standing. Keeping all of that separate. The user instruction in bucket A, the codebase context in bucket B, they dont dilute when switching, the user's instructions are preserved for reference, the codebase context is indexed and easily referenceable, with a drafter, when it queries the indexes, it fetches exact blocks, no overflow, with reference to the merkle root sitemap's context history, so if it sees the enum was changed, it can also see why it was changed. That way it doesnt make the same mistake twice and when it tries to, it has to justify it.
So essentially,
Is this atleast somewhat correct?
A summary of it yes, though how it does it does make a big difference. Idk if you're familiar with file-locks? Think about line-lock. If you affect a property used by another model, you and the other model need to agree it's fine. That way both know it wont break their work. If it's a simple agree, it's just a thumbs up, if it's breaking change, then discuss a better solution together.
It looks like you know a lot about this 🤔 Are you working on something related?
Yip, though without the drafter, already part of my IDE.
You mean the thing you've built?
Yip
What?!