I Let a Free Model Prune My Feature Flags for 48 Hours. It Found More Than I Did.

Feature flags are the duct tape of modern deployments. They let you ship dark, roll back instantly, and run gradual rollouts. But every flag you forget to remove becomes a layer of dead conditional logic that future-you has to untangle. We all know the feeling: a codebase full of if (flags.isEnabled("old_checkout")) branches that nobody dares delete because the flag might still be toggled in production.

I had a pile of 60 flags in a small service. Manually auditing them would take a full afternoon, and I still wouldn't trust my own grep results. So I asked the obvious question: could a free model make the pruning decision for me, while a free server watched the clock for two days?

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access and its free server option for this experiment. I'm describing what I actually ran, not a benchmark or a guarantee.

The Experiment Design

The goal was simple: every six hours, for 48 hours, a Python service would pull my flag definitions from a JSON file, run code searches for each flag name, feed the results to a free model, and archive the model's verdict. The service would run on MonkeyCode's free server so I could see whether a free deployment could handle periodic cron-like workloads without falling asleep.

The verdict had to be one of three labels:

• PRUNE – zero real usages found, safe to delete
• KEEP – clear references found, either logic or config
• REVIEW – ambiguous; a human should look before touching

Here's the core function that built the prompt for each flag:

import json

def build_prompt(flag_name, search_hits):
    hits_summary = "\n".join(
        f"{h['file']}:{h['line']}: {h['code'].strip()}"
        for h in search_hits[:10]
    ) or "(no direct references in code)"

    return f"""
You are a code maintainer reviewing feature flags.

Flag: {flag_name}

Relevant code references:
{hits_summary}

Also consider that flags can be referenced dynamically (e.g., built from env vars or
toggled via config files not included in this search).

Reply with exactly one line:
PRUNE, KEEP, or REVIEW
Then a short reason.
"""

def classify_flag(model_fn, flag_name, search_hits):
    prompt = build_prompt(flag_name, search_hits)
    output = model_fn(prompt)  # your free-model call goes here
    output = output.strip()
    for label in ("PRUNE", "KEEP", "REVIEW"):
        if output.startswith(label):
            return label, output
    return "REVIEW", f"Unparseable output: {output}"

I created a ground-truth set of 10 flags that I manually audited beforehand. That set became the sanity check: if the model disagreed with my manual call too often, I'd stop trusting the whole pipeline.

What Broke First

The Model Could Not See Dynamic References

The first run flagged payment_v2 as PRUNE because grep found zero literal occurrences in the repo. But I knew payment_v2 was toggled via a remote config service. The model had no way to discover that from my narrow search results. This taught me something important: the artifact I feed the model is the real bottleneck. Garbage context in, confident false positives out.

Search Results Were Incomplete

My quick grep -r missed references inside JSON config files and environment templates. The model then reasoned over incomplete evidence and produced confident verdicts. A human would have said "I didn't look hard enough." The model just said PRUNE.

It Hallucinated a Flag

On run four, the model returned REVIEW for checkout_flow_v3 and justified it with a file and line number that didn't exist in the repository. I verified: the file was services/checkout/handlers.py:214, but that file ended at line 198. The model had fabricated a source to support its uncertainty. That was the boundary condition that made me add a "never auto-delete" rule.

What Actually Worked

To be fair, the model was consistently correct on the high-signal cases. For flags with zero references across every file type I included, and no toggles in external config, it said PRUNE and stayed PRUNE across all eight runs. That consistency was useful. It gave me a shortlist of 14 flags that all three verdicts agreed on, and a human could review those in ten minutes.

The REVIEW bucket also proved valuable. The model sent 23 of the 60 flags to human review, which forced me to actually look at them. It turned a tedious audit into a triage session where I only focused on the ambiguous five percent.

What I'd Repeat

• Always keep a small human-labeled gold set. I don't trust any model's verdict on code without a baseline of 10–20 cases I've checked by hand.
• Feed the model more than a grep. Next time I'll include config files, docs, and maybe a dependency graph. The model is only as good as its context.
• Treat PRUNE as a recommendation, not a command. The free model made a false positive in four hours. A CI pipeline that auto-deletes based on its output would be dangerous.
• Use a free server for this kind of scheduled job. The server didn't sleep between runs, and the periodic execution was enough for a low-frequency audit. Your mileage may vary, but for a once-every-six-hours script, it held up.

Limitations and Who Should Not Use This

This approach is not a replacement for a proper feature-flag lifecycle tool. If your team uses a hosted feature-management service with built-in flag analytics, you already have better data than any model reading grep output. Also, if your flags are heavily toggled from runtime state or external systems, this entire exercise is likely to produce false confidence.

Don't run this on a codebase where a wrong deletion can break a compliance requirement or an irreversible data path. Use it only as a first-pass sorter, not as an autonomous remover.

The Verdict After 48 Hours

The free model was good at pointing out the obvious and terrible at knowing what it couldn't see. The free server ran the script reliably for two days without a hiccup, which made the experiment possible. What I'll keep is the triage loop: a free model that proposes, a human who disposes, and a validation set that catches the hallucinations before they reach main.

If you're curious about running a similar experiment, MonkeyCode's free tier is an easy place to start. But treat every verdict like a pull request from a very confident intern — review it carefully, then merge the boring parts.