DEV Community: Orkas

Your Moat Is Someone Else's API

Orkas — Wed, 12 Aug 2026 10:17:29 +0000

You built an AI product. People use it. The numbers go up.

Then someone asks: how is this different from just using the model directly?

And you don't have a good answer.

Not because you didn't build anything. You built prompts, workflows, context management, eval harnesses, a UI people actually like. But none of that lives in the intelligence layer.

And the intelligence layer is rented.

What the business actually is

Strip the narrative away and most consumer AI startups today are doing the same thing: buying intelligence by the token and reselling it by the month.

You buy capability from a model vendor at metered rates, wrap it in a product, and charge a subscription. The spread is the business.

That's not an insult, it's a description. Plenty of good businesses are built on resale. But it has one structural property worth being honest about: the thing your product is actually made of doesn't belong to you.

What that costs you

Four consequences follow, and they compound.

Prices move, and your margin moves with them. Your unit economics are set by someone else's pricing page.

Behavior changes, and your product changes with it. A model update ships, your outputs shift, and you usually find out from users rather than from a changelog.

Your scenario can get absorbed. If what you do is valuable and general enough, there's a real chance the vendor ships it natively. At that point you're not competing on product, you're competing on still being necessary.

Your layer thins as the base model improves. This one is slower and easier to miss. A meaningful share of your prompt engineering, retry logic, format coercion, and orchestration scaffolding exists because the model can't do it reliably yet. Each generation absorbs some of it.

The "it's just a wrapper" criticism stings not because it's rude, but because it names something true:

How much value you hold is a function of what the model hasn't done yet.

"But we have user data"

The standard answer to the moat question is the data flywheel: more users → more data → better product → more users.

Under the constraint that you don't own a model, that flywheel mostly doesn't spin. It jams in three places.

1. You can't use the data

User data carries privacy obligations. Whether it can be used for training, how it gets de-identified, what your consent language actually covers — each is a real gate. Plenty of teams have the logs and can't get past legal.

2. Even if you can use it, there's nowhere to put it

You don't own a model. The data can go into prompts, into retrieval, into a thin adapter. It cannot go into the intelligence itself.

3. The quality isn't there

What actually improves a model is labeled, outcome-verified, feedback-carrying samples. What most applications accumulate is logs. Logs are excellent for analytics. They are not training data.

So for most application companies, "data flywheel" has been a story rather than a mechanism.

So why not train your own?

Two years ago this was a bad idea, for two solid reasons.

Open models were meaningfully behind. Building on one meant starting from a lower ceiling. You'd invest months and still ship something worse than a direct API call.

The base models were improving fast. Any domain advantage you spent six months building could be erased by the next general release. The ROI didn't close.

So "the application layer shouldn't touch models" became close to consensus. And at the time, that consensus was correct.

What changed

Two things, roughly at once.

Open models crossed into trillion-parameter territory, with substantially stronger coding and agentic capability. They handle genuinely complex long-horizon tasks now. On general capability the gap to the closed frontier has narrowed considerably.

Domain-targeted post-training became a real lever. In a specific domain, with the right data, there is now a chance of beating the closed SOTA model in that domain.

Read that carefully: a chance, not a certainty. I'm not claiming you will. I'm claiming it moved from "essentially impossible" to "possible" — and that shift is large enough to change a decision that was previously automatic.

"Owning" is a spectrum, not a binary

Worth being precise here, because "own your model" gets thrown around loosely and the levels have very different economics.

Level	What you own	Flywheel works?
Prompt + retrieval	Nothing	No — this is renting with extra steps
Adapter / LoRA on a hosted model	A small delta you can't deploy independently	Barely — still bounded by the host
Post-training an open base you deploy yourself	The weights you ship	Yes — this is where the argument starts to hold
Pre-training from scratch	Everything	Yes, but the economics here have not changed

When I say "owning intelligence" in this post, I mean the third row. The first two are worth doing and often sufficient; they just don't buy you the thing this post is about.

What owning intelligence actually buys you

	Renting	Owning
Performance	General capability; latency set by the vendor	Trim capability you don't need; optimize inference for your shape
Cost	Linear in tokens	Marginal cost drops sharply past scale
Control	Price changes, deprecations, rate limits, absorption	Yours
Data flywheel	Doesn't spin — nowhere to put the data	Data goes into post-training; gains feed usage

The first three rows are cost and risk management. The fourth is compounding, and it's the one that decides long-run separation. Owning a model is what finally gives your data somewhere to go.

Should you do this?

Six questions. Be honest on each.

[ ] Is general-purpose capability clearly insufficient in your domain?
[ ] Do you have data nobody else can get?
[ ] Can that data legally be used for training?
[ ] Is inference cost a material line item yet?
[ ] Is latency central to the product experience?
[ ] Can you staff and fund a post-training team?

4+ yes → worth seriously evaluating.
2–3 yes → try it on one narrow slice. Don't restructure the company around it.
0–1 yes → keep renting. That is the rational call, not the cowardly one.

This is a framing device, not a scorecard. Where you set the threshold depends on your runway and your window.

Three things that have to be said

1. This is not free.

Owning intelligence means a team that understands post-training, compute, a data pipeline, and eval infrastructure you trust. It also means accepting real risk that your iteration speed gets worse before it gets better. Those costs are not small, and they're mostly fixed.

2. Most companies should not do this.

If general models are good enough for your domain, if your volume is still small, if you have no proprietary data — renting is correct. Don't do this because it makes a better story for the next round.

3. This judgment will change again.

Once intelligence is cheap and ubiquitous, the application layer will grow new business models, and "owns a model" may stop being an interesting axis at all. I'd expect that to happen.

So I'm not going to tell you that you should.

The accurate statement is narrower:

For the first time, owning your intelligence is a real option rather than a wrong answer. It used to be a mistake. Now it's a calculation worth actually running.

What this post is and isn't

This is a judgment call, not a benchmark report.

I've deliberately included no scores. How far open models have come in your domain, and what post-training actually buys you on your data, are things you have to measure yourself. The general trend is real; the size of the effect is entirely domain-dependent. Don't take anyone's word for the magnitude — including mine.

If you disagree with the framing, I'd genuinely like to hear where. The weakest link in the argument is probably the claim that domain post-training can beat a closed frontier model in-domain — that's the part most sensitive to what "domain" means and how much data you actually have.

If you're building on rented intelligence right now: which row of that table hurts most?

Cost — volume goes up and the economics stop working
Control — you're one roadmap decision away from being redundant
Flywheel — the data is piling up and there's nowhere for it to go

Curious whether the ordering matches what I'd expect. I'd guess control ranks higher than cost for most people, and that flywheel only ranks first for teams who've already tried and hit the wall.

Your Agent Isn't Repeating Itself. That's Why You Can't Catch It Spinning

Orkas — Tue, 28 Jul 2026 03:51:54 +0000

I've been reading BEACON from Zhejiang University's ZJU-REAL lab (arXiv:2605.06078), and the more I read it the more directly applicable it felt to the long-horizon agent work we're doing. So I'm turning that into a series.

The framing

To build a long-horizon agent that actually works, I think two things have to be right:

1. The agent keeps making progress toward the goal

Context compaction — what is safe to forget
Milestone design — how do you know a step is really done
Spin prevention — when it gets stuck, something has to notice

2. The agent can reflect and improve

This post covers spin prevention, from the first group. It goes first because it's the one that hurt us most.

The symptom

We had user reports, and saw it internally too: some very long tasks just stop moving forward partway through.

Look at the logs and the agent is busy — reading files, searching, running commands, never idle. But half an hour goes by and nothing has advanced.

My first explanation was context loss: compaction dropped the record that "this path was already tried," so the agent went and tried it again.

Digging through the code and the logs, that turned out to be only half the story.

We already had guards. Three tiers of them.

Tier	Criterion	Thresholds
Exact repeat	Tool name + canonicalized args, byte-identical	`LOOP_WARN=3` warn / `LOOP_HARD=5` force-stop
Near-duplicate	Identical except volatile id / timestamp fields	`NEAR_DUP_LOOP_WARN=6` / `NEAR_DUP_LOOP_HARD=12`
Spin convergence	≥2 compactions and ≥75% of the tool-loop budget burned	`SPIN_CONVERGENCE_MIN_COMPACTIONS=2` `SPIN_CONVERGENCE_TOOL_LOOP_RATIO=0.75`

So the problem wasn't that nobody built anything. It's that what we built couldn't catch it.

The blind spot: all three are input-side detectors

toolCallSignature = tool name + canonicalized args

That signature answers exactly one question: are you doing the same thing twice?

But a capable model that's stuck doesn't repeat calls. It does this:

read file A → grep X → re-read A with a different line range
  → run a slightly different command → read file B → grep again …

Every single call has a distinct signature. All three tiers stay silent.

And during that entire stretch, the number of verifiable state changes is zero. No file actually rewritten. No command producing a new result. Nothing irreversible happening at all.

One line:

Loop detection ≠ stall detection. The first looks at inputs. The second has to look at outputs.

The paper gives exactly that output-side definition

BEACON's in-segment reward:

r_t = R_ms · γ^(t_k − t)    if the segment ends in a milestone
    = 0                     otherwise

A segment that doesn't terminate in a milestone earns exactly zero — regardless of how many actions it contained. Action count doesn't enter the formula at all.

That's the cleanest formalization of busy ≠ progressing I've seen.

The baseline layer is harsher still. The baseline is the group-average per-step return, so if your segment took 8 steps while the group averaged 5, the entire segment's advantage skews negative, and the penalty scales with how far over you went.

Spinning isn't merely unrewarded. It's actively penalized, proportionally.

Figure 8 in the paper has a failed trajectory where the last two actions receive an identical −2.20. That tail — the stretch after the last milestone that never reaches another one — is the mathematical shape of spinning.

How common is it? Their data: trajectories that complete at least one subgoal but fail the task hold steady at 39–47% of samples.

One ablation that kills the "split by step count" idea

This is the number I found most valuable:

Partitioning	Score	vs. baseline (72.8)
Random 5-way split	74.2	+1.4
Real milestones	91.4	+17.2

Translated into product terms: splitting by arbitrary step counts is worth almost nothing. Splitting by real structure is worth a lot.

Now look back at our third-tier criterion — "75% of the tool-loop budget consumed." That's an arbitrary-step-count trigger. It asks how much you've burned, not what you've achieved.

The right shape is:

❌  if steps > N                          → intervene
✅  if steps > N AND zero verified milestones → intervene

The second one won't misfire on a legitimately long task that's progressing (it hits milestones along the way). The first one will.

There's also a positive feedback loop

Put a few constants side by side:

context at 82%                        → trigger compaction
2 compactions + 75% of loop budget    → only then flag spinning

Spin → context fills up → compaction fires → durable state gets summarized away → re-derive what was lost → spin more.

The spin detector infers spinning by observing that compaction happened repeatedly — but compaction is the very step that causes the amnesia. It's detecting a downstream symptom, and it has to wait for the loop to go around twice.

Worse, its intervention is to nudge the model to re-anchor on its durable state. If the durable state is precisely what got compacted away, there's nothing left to re-anchor on.

This is a prompt-layer patch for a state-layer problem.

What we plan to change

Add a fourth tier: output-side stall detection. Two counters, both derived mechanically from tool observations. No model judgment required.

1. Steps since the last verified milestone

2. Deduplicated new state changes ← the more useful one

File read with an identical hash → not new information (re-reading the same file at a different line range leaves the hash unchanged)
Command + exit code + stdout hash identical → not new information
Write where afterHash didn't change → nothing was actually written

This counter targets exactly the case signature matching misses: every action different, information gain zero. The criterion is mechanical — no semantic understanding needed.

Then make the pressure continuous instead of a one-shot nudge:

surface the counter in context so the model can see it
  → force a plan revision (admit this path is dead)
    → ask the user
      → abort, but preserve the milestones already achieved

That last rung matters: if you're going to stop, stop holding what you earned — which echoes the 39–47% number above.

One honest discount

BEACON is a training method. It shapes gradients so the trained policy is less prone to wandering, but it has no runtime detection or intervention mechanism of its own.

It gives you a definition of progress, not a controller. Thresholds, escalation ladders, when to abort — all of that you design yourself. The paper doesn't help there.

Also, its per-step baseline needs a group-average segment length as reference. In production a given user task usually runs exactly once, so there's no group. You can only approximate with historical similar tasks, which is considerably noisier.

But the first thing to do isn't to fix it — it's to measure it

Instrument only. Change no decision logic. Answer a question we currently can't:

Of the spins happening in production, how many are amnesia-type vs. no-gradient-type?

Mostly accompanied by information gain at zero while call signatures all differ → no-gradient-type. The existing three tiers structurally cannot catch it; we need output-side detection.
Mostly accompanied by re-reading content that was already compacted away → amnesia-type. What needs fixing is what compaction preserves.

Those two directions call for completely different investments. Measuring first is cheaper than designing first.

Next post: why AI self-reflection always seems to conclude with "be more careful."

If you're building long-running agents — what does yours look like when it gets stuck? Repeating the same action, or looking busy while nothing changes? Curious what others are seeing.

Why RL Training Collapses on Long-Horizon Agents

Orkas — Mon, 27 Jul 2026 04:12:06 +0000

Been heads-down lately, but I finally had a couple of days to catch up on some recent papers. Sharing one of them here.

Milestone-Guided Policy Learning for Long-Horizon Language Agents
Zhejiang University (ZJU-REAL) · arXiv:2605.06078 · github.com/ZJU-REAL/BEACON

It tackles a very concrete and fairly painful problem: when you train an agent on long tasks with RL, it doesn't just get a bit worse — it falls off a cliff as the horizon grows.

TL;DR

Trajectory-level RL (GRPO and friends) fails on long horizons for two measurable reasons: contradictory gradients and wasted partial successes.
BEACON splits trajectories at milestones (verifiable state transitions), shapes rewards within each segment, and estimates advantages at two scales.
ALFWorld long tasks: 53.5% → 92.9%. Effective sample utilization: 23.7% → 82.0%.
The gains grow with horizon length — which is a much stronger claim than a higher average score.
There's a metric in the paper (CCR) that appears to contradict the method's own design. It doesn't — but working out why is the most interesting part of the paper.

1. First, look at how cleanly they diagnose the problem

What I appreciate most is that they don't open with the method. They open with an empirical autopsy: Qwen2.5-1.5B + GRPO on ALFWorld, breaking "long-horizon training collapses" into two quantifiable failure modes.

Failure mode 1: credit misattribution

GRPO treats a trajectory as a flat action sequence. Every action shares one terminal score.

The consequence: the same correct action gets a positive gradient in a successful trajectory and a negative gradient in a failed one. Whether it was "right" depends on what happened afterwards.

They define CAR (Contradictory Action Ratio) — the fraction of actions that receive opposite-sign advantages across trajectories despite being executed at identical states.

It peaks above 40%. Nearly half the gradient updates for repeated state-action pairs point in conflicting directions. After cancellation, effective learning signal drops below 20%.

Failure mode 2: sample inefficiency

Bucket trajectories three ways: full success, partial success (completed at least one subgoal but failed the task), and total failure.

Partial successes hold steady at 39–47% of samples throughout training
Full successes stay below 27%

Under GRPO, a partial success and a total failure both get a reward of 0. That means over 73% of samples produce no learning signal at all.

Both problems compound as horizons extend: longer tasks have lower success rates (more partial successes) and more opportunity for downstream stochasticity to corrupt credit.

The number that makes it concrete: ALFWorld short tasks 76.7%, long tasks 53.5%.

2. The core insight

Long-horizon tasks have structure. Flat trajectory optimization just throws it away.

They naturally decompose into phases bounded by milestones — verifiable state transitions marking subgoal completion.

The authors formalize this as the Milestone Markov Property: once you reach a milestone state, the distribution over the remaining trajectory depends mostly on which subgoals are left, not on the full history of how you got there.

In plain terms: once you have the key, what happens next depends on what you do with it — not on how you found it.

That approximate Markov property is what makes credit decoupleable across segments.

3. The method: three steps

Step 1 — partition at milestones

A detector Φ flags milestone timesteps and splits the trajectory into segments.

Here's the design decision I think is underrated: Φ requires no learned model and no human annotation. It reads observable state changes straight from environment feedback.

ALFWorld: object state transitions (successful pickup, heating complete)
WebShop: page transitions toward the target product
ScienceWorld: the environment emits explicit subgoal signals

Zero extra models, zero extra rollouts. That's the whole cost advantage over process reward models and Monte Carlo value estimation.

Step 2 — temporal reward shaping inside segments

r_t = R_ms * γ^(t_k − t)   if segment k ends in a completed milestone
    = 0                    otherwise

Only segments that terminate in a milestone get positive reward. Within such a segment, actions closer to the milestone get more.

Two effects: every action in a completed segment now carries positive signal (partial successes stop being wasted), and there's an implicit push toward efficient execution.

Step 3 — dual-scale advantage (the crux)

Trajectory level — standard GRPO normalization over terminal rewards. Captures global task performance.

Segment level — the trick is in how the comparison group is defined:

G_k = { i : K_i ≥ k }          # only trajectories that ALSO reached milestone k

A_seg(i,t) = r_t − (1/|G_k|) · Σ_{j∈G_k}  R_k(j) / |Seg_k(j)|
                               └── group-average PER-STEP return ──┘

Actions in segment k are compared only against trajectories that also reached milestone k. From this the authors prove a variance isolation property: whether later segments succeed or fail cannot mathematically contaminate credit for the current segment.

Final advantage is A_traj + λ · A_seg, optimized with a standard PPO clipped surrogate.

Hyperparameters γ=0.95, λ=1.0 are fixed across all benchmarks — no per-task tuning. Worth noting.

4. Results

ALFWorld, Qwen2.5-1.5B:

Method	Short	Medium	Long	Avg
GRPO	76.7	73.9	53.5	72.8
GiGPO	90.7	84.3	79.5	86.1
BEACON	96.8	87.0	92.9	91.4

Success rate on the other two:

Method	ScienceWorld	WebShop
GRPO	21.1	56.8
GiGPO	25.8	65.0
BEACON	45.3	75.6

A few things worth calling out:

The 1.5B model beats GPT-4o. ALFWorld 91.4 vs 48.0, WebShop 75.6 vs 23.7. (Fair caveat: the closed models are prompted with ReAct, not trained. And on ScienceWorld it's a tie — 45.3 vs 45.4. Still, a 1.5B model reaching parity is not nothing.)

Sample utilization goes from 23.7% to 82.0% — a 3.5× increase in trajectories that produce useful gradient. Zero-advantage sample ratio drops from ~55% under GRPO to ~10%.

Convergence is faster too: 60% success rate by iteration 50, where GRPO needs iteration 120.

But the most convincing result is that the gains scale with horizon. On 7B, relative improvement over GRPO goes from +13% on short tasks to +39% on long tasks. GiGPO only goes from +11% to +22%.

The reason is interesting. GiGPO builds step-level comparison groups by finding repeated states across trajectories. But as the policy improves and trajectories diversify, state recurrence gets sparser — its own signal source erodes. Milestone anchors don't decay as the policy gets stronger.

A method whose benefit grows as the problem gets harder is a much stronger claim than a higher average score.

5. What's actually novel here

Four things look like real contributions rather than assembly:

5.1 Redefining the comparison group

Process reward models need expensive annotation and carry reward-hacking risk. Monte Carlo value estimation needs multiple rollouts per decision point. GiGPO leans on state recurrence — incidental structure.

BEACON swaps in semantic milestones, which are intrinsic to the task, essentially free to detect, and become more reliable as horizons grow rather than less. That's a methodological switch, not an increment.

5.2 Variance isolation has a formal guarantee

Most reward shaping work stops at "it's denser, therefore better." This one actually proves why cutting here works.

5.3 Dual-scale isn't redundancy — the ablation proves both are load-bearing

Variant	ALFWorld	WebShop
BEACON	91.4	75.6
w/o segment-level advantage	72.8	56.8
w/o trajectory-level advantage	23.4	67.9

Drop the segment level and you get GRPO back exactly. Drop the trajectory level and ALFWorld craters to 23.4%.

That second row deserves thought: with only segment-level signal, the policy reinforces behaviors that hit intermediate milestones but are doomed to fail overall. Every subtask executed beautifully, the whole thing drifting off target.

The divergence between environments is also telling. WebShop still manages 67.9% without the trajectory level, because its milestones align more directly with final success. ALFWorld collapses. So how much terminal verification you need is a function of task structure.

5.4 A counterintuitive finding — CCR

This one deserves its own section.

6. CCR: the number I assumed was a typo

The authors define CCR (Credit Concentration Ratio) = average advantage magnitude for milestone actions ÷ that for non-milestone actions. CCR > 1 means credit concentrates on milestones.

Measured:

Method	CCR
GiGPO	2.36
GRPO	1.37
BEACON	0.84

So the best-performing method is the one that concentrates credit on key steps the least.

My first reaction: this directly contradicts "actions closer to the milestone get more reward." I had to work through the math to see that the two statements measure different quantities, with two transformations in between.

Transformation 1 — shaped reward

Within a segment, reward is monotonically increasing toward the milestone. ✅ This part is exactly as advertised.

With γ=0.95 and segment length 5, the five actions get 0.8145, 0.857, 0.9025, 0.95, 1.0.

But this is reward, not the credit that reaches the gradient.

Transformation 2 — subtract the group baseline

The baseline is the group-average per-step return (segment return ÷ segment length). For a segment of length L, per-step return is (1−γ^L) / (L(1−γ)):

Segment length	3	5	8	10
Per-step return	0.951	0.905	0.842	0.803

So if your segment took 8 steps while the group averaged 5, your per-step return sits below the baseline and the entire segment's advantage skews negative.

Note what this means: the penalty for wandering doesn't come from the decay itself — it comes from the decay acting through the per-step baseline. I initially attributed it to the decay alone, which is wrong. Decay alone only orders actions within a segment.

At this point, CCR inside a completed segment is still greater than 1.

Transformation 3 — add the trajectory-level advantage

This is where CCR falls below 1, via two asymmetric effects:

(a) The tail segment of a failed trajectory enters no comparison group at all. Since G_k = {i : K_i ≥ k} and the incomplete tail has index K_i + 1 > K_i, trajectory i is excluded. That tail gets no segment-level advantage — only the trajectory-level term, at full magnitude. And every one of those is a non-milestone action, inflating the denominator.

(b) In failed trajectories, the negative trajectory-level term cancels against the positive segment-level credit on milestone actions, squeezing their magnitude toward zero.

Figure 8 confirms it numerically

On a failed trajectory that completed milestones S3 and S4:

	go to toilet	put soapbar (S3✓)	go to counter (S4✓)	go to counter	go to holder
GRPO	−2.50	−2.50	−2.50	−2.50	−2.50
BEACON	−0.92	+0.51	+0.32	−2.20	−2.20

The last two values are identical — the fingerprint of "tail segment gets only the trajectory-level term," which lets you read off A_traj ≈ −2.20. The two milestone actions are positive, but their magnitude is only ~0.5 because the negative trajectory term ate most of the segment-level credit.

CCR for this trajectory: 0.23. For a successful trajectory: 2.95. The global 0.84 is the mixture.

So what does CCR < 1 actually mean

CCR measures the concentration of gradient magnitude, not "who got rewarded." The positive credit milestones are supposed to receive within a segment is fully intact.

The more precise framing: low CCR isn't a design goal, it's a byproduct of dense in-segment allocation plus dual-scale stacking.

But the authors' conclusion still holds, and it's a good one: don't dump all your gradient energy on the key steps. GiGPO's 2.36 means the preparatory actions in between get almost no signal — and those are exactly what makes reaching the milestone possible.

7. Something the paper doesn't spell out

There's an odd cell in the ablation table: γ=1 (uniform in-segment credit) scores 71.8, worse than no shaping at all (γ=0, 81.2), and even below GRPO's 72.8.

The paper's explanation: "assigning equal credit to all actions obscures the distinction between critical and preparatory actions, producing misleading gradients."

Work through the math and the real answer is sharper.

With γ=1, every action in a completed segment gets the same reward R_ms. Then every completed segment, regardless of length, has a per-step return of exactly R_ms — so the baseline is R_ms too, and:

A_seg(i,t) ≡ R_ms − R_ms = 0    for every action

It's not that the gradients are misleading. The segment-level channel vanishes identically, and the method reduces exactly to GRPO.

Check it against the table: 71.8 vs GRPO's 72.8. One point apart — run-to-run noise. It lines up perfectly.

Which means temporal decay isn't just "differentiating actions within a segment." It's a necessary condition for the segment-level signal to exist at all. Without decay, the per-step baseline cancels it out on the spot.

The paper doesn't say this.

8. Three experiments that shut down the obvious objections

Credit where due — they proactively close three holes:

"Isn't this just behavior cloning in disguise?"
SFT on oracle trajectories gets 43%. BEACON gets 91.4%. The policy discovers execution strategies better than the oracle, so it isn't imitating.

"Are the gains just from chunking the trajectory?"
Random partitioning (5 arbitrary split points) scores 74.2% — only 1.4 points above GRPO's 72.8. Real milestones score 91.4%, a 17.2-point gap. The benefit genuinely comes from task-intrinsic structure. Clean control.

"What if the detector is unreliable?"
Randomly dropping 50% of milestones still yields 82.8%, 10 points above GRPO. Degradation is graceful, not a cliff.

9. Limitations, stated honestly

The biggest bottleneck is whether Φ is even obtainable.

All three benchmarks get milestones from rules: pattern matching on environment responses, page transitions, or explicit subgoal signals from the env.

Real open-ended settings — browser automation, codebase refactoring, deep research — have no such ready-made verifiable state transitions. The authors themselves list automated milestone discovery as an open problem in the appendix.

So this reads as a paradigm validated in structured environments, not an engineering recipe you can lift as-is.

Also: milestone granularity is sensitive (too sparse degenerates to GRPO, too dense makes segment advantages noisy); the Markov property is only approximate, and variance isolation rests on it; experiments stop at 7B with discrete text action spaces — continuous control and multi-agent settings are untested.

Closing thought

The ideas in this paper matter for anyone designing agents that take on long-horizon tasks — even if you're not training a policy. The transferable part isn't the advantage formula; it's the claim that long tasks have exploitable compositional structure, and that structure should be a first-class runtime object rather than something you hope the model tracks in context.

We've been working on long-horizon task support in Orkas recently, and I'll keep sharing our design thinking on that as it develops.

If you're building agents for long tasks — how are you handling intermediate progress signals? Genuinely curious what's working in production.

Rebuilding Orkas for Long-Running Agents

Orkas — Mon, 20 Jul 2026 09:24:06 +0000

AI agents don’t only break because the model is weak. They often break because the product treats agent work like a single chatbot response.

In the Orkas 1.0 foundation refactor, we rebuilt the lower layers of the system: the agent runtime, provider rotation, orchestration, external hosting, memory, and context management.

This post summarizes the architecture changes and the lessons behind them.

Why the refactor happened

Orkas is a local-first desktop workspace for AI agents. Agents can work with local files, shell commands, project folders, connectors, skills, and knowledge bases.

That creates a different set of problems from a normal chat UI:

long-running tasks can span many steps
context grows quickly
tool calls need ordering and safety rules
providers can fail mid-run
users may interrupt or correct the agent
multi-agent workflows need coordination, not just a static plan

The refactor was about moving these concerns into Orkas’s own foundation instead of relying on chat-style assumptions.

1. An in-process agent runtime

The core change was a standalone, dynamically loadable, in-process agent runtime.

It is split into two layers:

Engine layer: owns the generic agent loop, tool calling, streaming, context compaction, retry behavior, provider abstraction, memory, and self-evolution.
Adapter layer: connects that engine to Orkas-specific storage, permissions, skills, connectors, provider rotation, and event formats.

This boundary adds some complexity, but it keeps the reusable agent machinery separate from product-specific wiring.

2. Desktop agents need desktop-grade tool behavior

A local desktop agent can touch real files, run local commands, and work across project directories. That means the tool loop needs stronger guarantees.

Some details became important:

read before write
stale edit protection
parallel read-only operations
ordered writes
loop detection for repeated tool calls
interruption handling at safe boundaries
enough output room for long reports or large edits

These are not flashy features, but they are what make agent work feel predictable instead of fragile.

3. Context is a cost and reliability problem

Long tasks burn tokens in small repeated steps: reading, searching, summarizing, retrying, compacting, and recovering from errors.

The refactor made context handling more model-aware. Instead of using one conservative context limit, the runtime reads the model’s actual context window and compacts around a usage threshold.

It also avoids compaction when summarizing would not free useful space.

The lesson: context management is not just a UX feature. It affects cost, latency, and whether the agent can finish the task.

4. Provider rotation belongs below the agent runner

Model calls fail. Keys hit limits. Networks break.

Orkas moved provider rotation below the agent runner so a user turn can survive safe failures without duplicating session state.

The rotation rule is conservative:

if failure happens before meaningful output, another provider can be tried
once text or tool calls start, rotation stops
transient failures can retry
request or policy errors should not be hidden by blind retries

The point is not to make providers invisible. It is to put failover where it can be handled safely.

5. From static plans to group-chat orchestration

Earlier orchestration used a more static plan/DAG model. That looked clean, but real agent work changes as new information appears.

Orkas moved toward dynamic group-chat orchestration:

a Commander coordinates the room
worker agents receive focused slices of context
dispatch happens through structured tool calls
the Commander can fan out, synthesize, or hand off work

This fits long-running tasks better than compiling one fixed plan upfront.

6. Opening the system without removing boundaries

The refactor also made Orkas more open to external capabilities:

external packages
custom skills
local CLIs
user-configured MCP servers
external agents launched from Orkas

The important constraint is that risky actions still go through controlled boundaries: explicit user confirmation, local process isolation where possible, encrypted credentials, and permission checks for external side effects.

Open hosting is useful only if the trust boundary remains clear.

7. Memory and self-evolution need limits

Orkas also rebuilt memory and self-evolution around a simple principle:

bounded, observable, and off by default where appropriate.

Memory is local and separated into categories such as user preferences and agent notes. Retrieval combines semantic and keyword search.

Self-evolution is agent-private. It can update private skills or competence notes based on corrections, error recovery, and repeated task patterns, but it is bounded by cost controls and explicit session behavior.

“Agents that improve with use” is only useful if users can understand and control what is changing.

What we learned

Agent infrastructure has a different shape from chat infrastructure.

A production agent runtime needs to handle:

tool ordering
side effects
provider failures
context pressure
token cost
user interruption
orchestration state
local security boundaries
memory limits

The main challenge is deciding where complexity should live.

For Orkas, the answer became:

generic agent behavior in the runtime engine
product-specific wiring in the adapter
orchestration in the group-chat message bus
risky external actions behind explicit consent
memory and self-evolution behind local controls

Closing

The Orkas 1.0 foundation refactor was not about adding more surface features. It was about rebuilding the operating layer underneath long-running local agents.

If you are building agent systems, especially local-first or desktop agents, the takeaway is simple:

You eventually stop building around the model API and start building the runtime around the agent.

The Real Cost Problem in AI Agents

Orkas — Fri, 03 Jul 2026 10:16:23 +0000

AI agents have a cost problem.

A single "task" often means many model calls: reading context, calling tools, summarizing results, deciding the next step, retrying, validating output. If every step hits a frontier LLM, the unit economics get ugly fast.

One big model for everything is probably the wrong shape

The better question isn't "which model is smartest?" — it's "which part of the task actually needs the smartest model?"

LLMs should handle the hard parts: planning, backtracking, judgment, ambiguous decisions.

Small language models can handle the boring but frequent parts: extraction, routing, JSON formatting, tool parameters, log summaries, simple validation.

Most agent workflows contain a lot of that second category.

Why desktop agents are interesting

Cloud agents pay for tokens at almost every step — every retry, every summary, every tool-call decision, every formatting pass usually goes through a remote model.

Desktop agents have another option: local compute. They can run small local models or deterministic code for cheap, repetitive work, and only call cloud LLMs when the task actually needs deeper reasoning.

That changes the cost structure. Instead of:

every step → cloud LLM token cost

you get something closer to:

routine work → local compute · hard decisions → cloud LLMs

The long-term loop

start with LLMs → log agent traces → find repeated task patterns → distill them into SLMs / LoRAs → run them locally or cheaply → keep LLMs as fallback

In other words, agents should get cheaper as they're used more. The more traces you collect, the clearer it gets which tasks are repeated, narrow, and safe to move off frontier models.

My takeaway

The next wave of agents won't just be about stronger models — it'll be about better compute allocation: LLMs for judgment, SLMs for narrow repeated work, code for deterministic checks, local compute wherever possible.

That may be what makes agent economics work.

Paper: Small Language Models are the Future of Agentic AI — https://arxiv.org/abs/2506.02153