🏴‍☠️ Distilling a Closed Cloud Compiler with an Autonomous AI Agent

#ai #agents #automation #python

Differential testing, a self-maintained ledger, and a prompt that keeps working while you sleep. You can see a transpilation example here.

The setup: Pine Script, PyneCore and PyneComp

In algorithmic trading, TradingView is the de facto monopoly. It created Pine Script, published an open specification — but never shipped a local implementation. So you can only write Pine Script in their cloud editor. No local debugger, no test runner, and no way to plug the language into AI coding agents.

A company called PyneSys took the classic open-core route. Their open-source library, PyneCore, lets you write trading strategies in Python using the same execution model as TradingView. The missing piece — compiling actual Pine Script into Python — is PyneComp, a closed-source cloud compiler behind a subscription.

That architecture creates an interesting target. The compiler's interface is fully observable: Pine Script in, Python out. Which means its behavior can be learned.

Why bother: agents need a strategy corpus

The end goal isn't the compiler itself. It's what the compiler unlocks.

An AI agent can sweep a portfolio in seconds — BTC +261%, SOL +285%, LINK +201% — and tell you the drawdown comes from ATOM −64%, LTC −61%, DOT −58%.

But here's the catch: to get there, you need to feed the agent building blocks harvested from the most popular public Pine Script strategies. No single strategy performs on its own — the value is in recombination. Give an agent an empty context and ask it to "make money," and it politely declines. Give it a dataset of ~2,000 battle-tested strategy snippets, and it becomes a constructor. Porting two thousand scripts by hand is not an option.

So the real task is: build an open transpiler (pine2pyne) whose output matches both PyneComp and, more importantly, Pine Script semantics on TradingView itself.

The method: the closed compiler as an oracle

This is differential testing with a twist. The agent treats PyneComp as a black-box oracle:

Pick a real-world Pine script from the corpus.
Compile it through PyneComp (within the subscription's rate limits: 120 compilations/hour, 2,100/week).
Compile the same script through the open pine2pyne.
Diff the outputs. Where they differ, design minimal probe scripts that isolate the rule the cloud compiler applies.
Fix pine2pyne, add the probe as a regression test, raise the match-rate floor, commit.

Two details make this distillation rather than blind imitation:

Pine Script is the ground truth, not PyneComp. When the probes reveal that PyneComp itself computes something differently from TradingView, the agent does not copy the bug. It follows Pine semantics, records the discrepancy as a "known difference" in the test suite, and logs it in a findings document.
Every finding ships with evidence. A TradingView measurement, a probe script, or a test — logged in the same commit as the fix.

The byproduct is docs/pynecore_vs_pine.md: a growing, evidence-backed registry of places where the PyneCore runtime or the PyneComp compiler diverge from real Pine Script behavior. Those entries convert directly into upstream bug reports for the PyneCore maintainers — the kind with a minimal repro attached, the kind maintainers actually like receiving.

The prompt

The agent runs unattended, overnight, in iterations. Everything that makes that safe and productive lives in one prompt. Here it is, translated in full:

# Autonomous iteration (the user is asleep, the machine is on): do NOT stop
and wait for input; the only thing that throttles work is the PyneComp quota
(120 compilations/hour, 2,100/week — python tools/pynecomp_probe.py --usage).
If a task is already in progress — continue it and ignore this message.

**HARD RULES from the user:** do NOT edit README.md. In docstrings and
comments inside the pine2pyne/ package, do NOT reference probes, cards (#NN),
asset file ids, tmp/ or anything opaque to an outside reader (tests, assets,
tools and probes never ship to PyPI) — describe the PyneComp rule itself.
References to probes/files are allowed only in tests/ and tools/.

**Goal:** the PyneComp segments that could not be derived from the project's
own tests. Material — assets/test/strategy, assets/test/indicator,
assets/test/lib (hardest first). Guide to the folder and to downloading from
TradingView: assets/test/README.md. If MCP playwright (.mcp.json) is
unavailable — python scripts with HTTPS GET (pubscripts-suggest-json,
pine-facade), a browser User-Agent, a 0.4 s pause between requests.

**STANDING DUTY:** maintain docs/pynecore_vs_pine.md as you go. Every newly
found divergence from Pine Script goes into that file immediately, together
with its evidence (a TradingView measurement, a probe, a test, an
assets/test script). Section 1 — the PyneCore runtime computes differently
from Pine. Section 2 — PyneComp emits code that computes differently from
Pine: a new known difference in tests/test_pynecomp_probes.py or a
"known difference" note in the ledger. Section 3 — not yet measured on
TradingView; once measured, the item moves to section 1 or 2. The file edit
goes in the same commit as the finding.

# Each iteration:

1. python tools/assets_test_ledger.py --probe-names — scripts with no
   PyneComp answer yet; the ledger is assets/test/processed.tsv
   (status: processed | skipped_duplicate | pending;
   pynecomp: identical | final-newline | differs | error | not_compiled;
   notes). Mark a duplicate of already-covered logic skipped_duplicate
   without compiling it.
2. Copy the picked scripts to tmp/probes/*.pine and compile:
   python tools/pynecomp_probe.py <names>; then
   python tools/assets_test_ledger.py.
3. For rows marked differs: derive the PyneComp rule (small probes in
   tmp/probes), fix pine2pyne so the match rate grows AND the output
   computes like Pine (where PyneComp is wrong — follow Pine, add a known
   difference to tests/test_pynecomp_probes.py and a line to
   docs/pynecore_vs_pine.md). Proof probes go to assets/probes,
   tests go to tests/.
4. Before committing: the full python -m pytest (no new failures) and
   python -m pine2pyne.compat (matches not below MATCHING_FLOOR in
   tests/test_compat_harness.py — raise the floor as the number grows).
   Update the ledger (python tools/assets_test_ledger.py) and
   docs/pynecore_vs_pine.md. Commit and push to the working branch
   (git pull --no-rebase origin main before pushing if main moved ahead;
   never commit tmp/ or the key from tmp/config/api.toml;
   git diff --cached | grep -c eyJhbGci must equal 0).
5. If the hourly quota is exhausted — commit what is done and end the
   iteration; the next one will pick it up.

Why this prompt works

A few design decisions carry most of the weight:

A ledger instead of memory. processed.tsv is the agent's externalized state. Each script has a status (processed / skipped_duplicate / pending) and a compile verdict (identical / final-newline / differs / error / not_compiled). A fresh iteration doesn't need to remember anything — it reads the ledger and continues. This is what makes "the user is asleep" viable: context windows end, the TSV file doesn't.

A ratchet, not a target. MATCHING_FLOOR in the compat harness only moves up. The agent is allowed to raise the floor when the match rate grows and forbidden to commit below it. Progress becomes monotonic by construction — the classic ratchet pattern, enforced by a test rather than by hope.

Quota as the only brake. The prompt explicitly names the single legitimate reason to stop (the compiler's hourly limit) and tells the agent what to do when it hits it: commit, end the iteration, let the next one continue. Without this, autonomous agents tend to invent reasons to pause and ask questions nobody is awake to answer.

Evidence coupled to findings in the same commit. "The file edit goes in the same commit as the finding" is a small sentence with a big effect: the divergence registry can never drift out of sync with the code, because the agent is not allowed to separate them.

Hygiene rules that survive publication. The package ships to PyPI; the probes, the ledger and the tooling don't. Hence the rule: comments inside the package must describe the rule ("PyneComp normalizes X to Y"), never the internal artifact that proved it ("see probe #47"). An outside reader of the published code should never hit a dangling reference.

Releasing the kraken

Run the prompt, go to bed. In the morning: a longer ledger, a higher MATCHING_FLOOR, and new entries in pynecore_vs_pine.md.

The entries in that file — actual, measured bugs in the original compiler and runtime, each with a minimal repro — get filed as issues to the PyneCore maintainers. The closed compiler gets distilled; the open project gets bug reports; the agent gets its strategy corpus.

Thanks for reading.