DEV Community: Divy Yadav

[Boost]

Divy Yadav — Sun, 30 Aug 2026 20:31:02 +0000

Divy Yadav

Aug 30

Your Sandbox Shouldn't Keep Its Install-Time Network Access

#ai #programming #devops #security

12 min read

Your Sandbox Shouldn't Keep Its Install-Time Network Access

Divy Yadav — Sun, 30 Aug 2026 20:29:59 +0000

Give setup the access it needs, then lock the sandbox down before untrusted code runs, without restarting it.

Your AI agent needed internet access to install a dependency. Why should it still have that access when it starts running code you never wrote?

Say your setup step needs pypi.org, files.pythonhosted.org, and a Git host. The code your model writes next needs one internal API and nothing else. Both phases run in the same sandbox under the same firewall rules, because the network policy was decided before either phase existed.

If the policy stays unchanged after installation, the untrusted phase inherits the installer's network reach and keeps it until the sandbox dies. The broader requirement wins simply because the narrower one would have broken setup.

That's the problem with treating network policy as part of the environment spec, alongside CPU, memory, and disk. For workloads where trust level changes during execution, permissions should follow the phase, not the sandbox.

The permission window is wider than the work

![Photo from AI]https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/i6dgpjktl7llp7fsi6z8.png)

Setting network policy at environment creation feels natural, it's how we usually think about containers and VMs: give the environment a set of resources and permissions, then leave them alone until it goes away.

But agent workloads don't stay in one trust state. During setup, you run dependencies and tools you explicitly chose. A few minutes later, the same sandbox might execute code generated by a model or submitted by a user. Those phases can need very different access, but they sit behind the same firewall.

Keep one policy for both, and the broader permission set carries into the less-trusted phase, not because the sandbox requires it, but because the policy was tied to the sandbox instead of to the work happening inside it.

There are a few obvious workarounds, none particularly attractive:

Separate environments. Works, but you move state between them and pay the setup cost twice.
Enforce the policy from inside the sandbox. Puts the control mechanism in the same environment you're trying to restrict.
Recreate the sandbox between phases. Keeps the security boundary, but throws away the state you just built.

What I actually want is simpler: keep the sandbox, change its egress policy when the phase changes, and see exactly what policy is active at each point. An egress policy defines where a sandbox can make outbound connections, and those rules don't have to stay the same for the sandbox's entire lifetime. That requires enforcement to live outside the workload, with the transition controlled by the system orchestrating the run.

This is where Tensorlake's Sandboxes are useful: the egress policy can be replaced on a running sandbox with a single update call. I'm using Tensorlake here because the behavior is explicit enough in the documentation and SDK source to verify what actually happens, rather than filling gaps with assumptions.

What a live policy change has to guarantee to be worth using

Before trusting a mechanism like this in a phased design, I want to know what happens when something goes wrong. Four things matter:

Atomicity — no moment where old rules have disappeared but new ones aren't active yet. Failure containment — a bad policy leaves the previous one in place. An external control path — the transition is driven by the orchestrator, not the workload. Clear replacement semantics — updates replace rather than merge, so permissions can't quietly accumulate across phases.

Tensorlake documents the first three directly: the swap is atomic with no enforcement gap, sending a policy object replaces the entire policy, and a policy naming an unresolvable hostname is rejected while the previous policy stays enforced.

The fourth is more about how you design the control path than a guarantee Tensorlake gives you. The update runs through an authenticated API, so keeping that path outside the sandbox is up to how you handle credentials, code inside the sandbox could still call the API if it holds a key with enough permissions. The test is mine. The behavior it checks is not.

How the policy swap behaves on a running sandbox

You can change a sandbox's egress policy without recreating or suspending it. The new policy applies to the running firewall in a single atomic swap, no gap where old rules are gone and new ones haven't taken effect.

The important detail is where enforcement happens. In the Python SDK, NetworkConfig is enforced host-side, per sandbox, not inside the guest's own network stack. Changing routes or firewall rules from inside the sandbox doesn't change the policy; doing that would require calling the same authenticated API the orchestrator uses, which makes credential placement a separate security concern.

The firewall is also stateful: established and related connections remain permitted, so changing the policy doesn't cut off traffic already in progress.

The live update surface is deliberately limited to name, exposed ports, unauthenticated access for those ports, and egress policy. CPU, memory, and disk are fixed at creation and require a new sandbox to change. Egress policy is one of the few resource properties you can change while running, and a replacement naming an unresolvable hostname is rejected outright, leaving the sandbox on its prior policy in full. A rejected update leaves you where you already were, never somewhere weaker.

The network argument is tri-state:

You pass	Result
Nothing (omit `network`)	Current policy unchanged. Update `name` or ports freely.
A policy object	The entire policy is replaced by what you sent.
An explicit clear	The sandbox returns to unrestricted egress.

Updates replace rather than merge, so every update has to express the complete intended policy for that phase. Clearing removes restrictions rather than imposing them, so a phase with no network at all is allow_internet_access=false with an empty allow_out, not a clear.

from tensorlake.sandbox import CLEAR_NETWORK_POLICY, NetworkConfig, Sandbox

sandbox = Sandbox.connect("<sandbox-id>")

# Replace the whole policy: reach api.example.com and nothing else.
sandbox.update(
    network=NetworkConfig(
        allow_internet_access=True,
        allow_out=["api.example.com"],
    )
)

# Later: return the sandbox to unrestricted egress.
sandbox.update(network=CLEAR_NETWORK_POLICY)

A few SDK details worth knowing if you're writing a wrapper: Python's None means keep, not clear (the explicit CLEAR_NETWORK_POLICY sentinel is required to clear), while TypeScript expresses the clear as network: null. Over HTTP it's a PATCH to /sandboxes/{sandbox_id}. The field is named network in the SDK but network_policy in the raw HTTP response schema, worth pinning to whichever surface you're parsing.

allow_internet_access defaults to true, which makes a policy object that only lists allow_out a DNS-permitted allowlist rather than a lockdown. I write all three fields on every update, deny_out=[] included when I mean none, to remove the guesswork.

The CLI follows the same replace-or-clear rule:

# Replace the policy: api.example.com plus DNS to the sandbox's resolvers.
tl sbx update <sandbox-id-or-name> -A api.example.com

# Replace the policy with an absolute block on outbound traffic.
tl sbx update <sandbox-id-or-name> --no-internet

# Clear the policy and restore unrestricted egress.
tl sbx update <sandbox-id-or-name> --clear-network

--no-internet can't combine with a rule flag, and --clear-network can't combine with any replacement-policy flag, since each describes a whole policy on its own.

On error handling: a missing sandbox raises SandboxNotFoundError; anything else the API rejects arrives as RemoteAPIError with a status code and message; a failure to reach the API at all raises SandboxConnectionError. That split matters for retries, since a call that never landed and a policy the server refused are different problems. The one state-specific case is 409 ("Sandbox is terminated and cannot be updated"), recoverable via restart for 48 hours, which restores from the most recent snapshot if one exists or cold-boots otherwise, re-entering Pending either way, so re-establish the phase policy explicitly rather than assume it survived. The exact status code for an unresolvable-hostname rejection isn't clearly pinned in the schema, so branch on the message too, not just the status code, before treating a policy as permanently bad.

Tensorlake's documentation reports two public-cloud checks of the enforcement itself: DNS resolution failing under allow_internet_access=False, and a deny_out entry blocking one destination while another stayed reachable. Those are existence proofs for specific destinations, not throughput or general enforcement measurements.

What `allow_internet_access` actually controls once `allow_out` is non-empty

Whether this flag opens general egress or only DNS depends on allow_out. A non-empty allowlist is itself a default-deny rule, which changes what the boolean is doing:

`allow_internet_access`	`allow_out`	Outbound behavior
`true`	empty	Everything reachable, minus whatever `deny_out` matches.
`true`	non-empty	Default-deny: nothing but the listed destinations, DNS to the sandbox's resolvers permitted.
`false`	non-empty	Default-deny again, and DNS fails unless the resolver's own IP appears in `allow_out`.
`false`	empty	Nothing leaves the sandbox, DNS included.

Reading allow_internet_access=True, allow_out=[...] as "internet on, with an allowlist on top" gets the model backwards: you have an allowlist, with DNS permitted. The false plus non-empty row is where an afternoon disappears, since hostnames can't resolve unless the resolver's IP is also listed, or unless you list only IPv4 addresses and CIDR ranges.

allow_out accepts domains, leading-wildcard domains like *.example.com, IPv4 addresses, and CIDR ranges; deny_out accepts the same except wildcards. deny_out takes precedence on any overlap, including against the sandbox's own resolver, which is the failure mode where a correct-looking allowlist suddenly stops resolving anything. A :port suffix is accepted but doesn't make a rule port-specific. A wildcard covers subdomains but not the apex (*.example.com reaches api.example.com but not example.com itself), and hostname rules follow DNS changes, so an allowlisted CDN-backed domain keeps working as its IPs rotate.

A policy state machine for a phased agent run

Treat the run as a state machine whose states are policies: install → fetch → execute → deliver → sealed, one update call per arrow, a complete policy statement in every state. The phase model is mine, not a documented Tensorlake pattern, though every mechanic it relies on is documented.

Phase	`allow_internet_access`	`allow_out`	Purpose
`install`	`true`	Registries and source hosts	Dependency installation under default-deny
`fetch`	`true`	Input data hosts only	Retrieve inputs, install-time reach already retired
`execute`	`true` (allowlist) or `false` if no network needed	One endpoint, or empty	Smallest surface for least-trusted code
`deliver`	`true`	Upload/result endpoint only	Output path separate from execute
`sealed`	`false`	empty	No egress at all, DNS included

The swap is atomic, so no window exists where egress goes unfiltered, and a failed transition (unresolvable hostname) leaves the previous state enforced rather than stranding the sandbox somewhere wider. What the swap doesn't do is close what's already open: established connections survive it, so a connection opened during install is still usable in execute. Narrowing allow_out shrinks what's reachable going forward; it doesn't retroactively make survivors safe.

# Setup finished. Retire the install-time surface before running generated code.
sandbox.update(
    network=NetworkConfig(
        allow_internet_access=True,
        allow_out=["api.internal.example.com"],
    )
)

Past two phases, I'd stop writing update calls by hand and keep the policy next to the phase definition instead, so the two can't drift apart:

from tensorlake.sandbox import NetworkConfig, Sandbox

PHASE_ORDER = ("install", "fetch", "execute", "deliver", "sealed")

PHASE_POLICY = {
    "install": NetworkConfig(
        allow_internet_access=True,
        allow_out=["pypi.org", "files.pythonhosted.org", "github.com"],
        deny_out=[],
    ),
    "fetch": NetworkConfig(
        allow_internet_access=True,
        allow_out=["inputs.internal.example.com"],
        deny_out=[],
    ),
    "execute": NetworkConfig(
        allow_internet_access=True,
        allow_out=["api.internal.example.com"],
        deny_out=[],
    ),
    "deliver": NetworkConfig(
        allow_internet_access=True,
        allow_out=["results.internal.example.com"],
        deny_out=[],
    ),
    "sealed": NetworkConfig(
        allow_internet_access=False,
        allow_out=[],
        deny_out=[],
    ),
}


def enter_phase(sandbox: Sandbox, phase: str, current: str | None = None) -> str:
    if current is not None and PHASE_ORDER.index(phase) <= PHASE_ORDER.index(current):
        raise RuntimeError(f"refusing to move from {current} back to {phase}")
    intended = PHASE_POLICY[phase]
    info = sandbox.update(network=intended)
    effective = info.network
    if (
        effective is None
        or effective.allow_internet_access != intended.allow_internet_access
        or set(effective.allow_out) != set(intended.allow_out)
        or set(effective.deny_out) != set(intended.deny_out)
    ):
        raise RuntimeError(f"{phase}: policy did not take effect, got {effective}")
    return phase

Every entry spells out all three fields, since leaning on the allow_internet_access default is how an intended lockdown becomes an allowlist. The check compares all three too, so a policy whose boolean or deny list drifted from the phase definition still fails loudly rather than passing on an allow_out match alone. I want that assertion because if the phase table and the sandbox ever disagree, the run should stop rather than continue under a policy nobody chose. The ordering guard enforces the declared sequence, it doesn't prove each policy is a strict subset of the one before it, but it's the cheap version of the property I actually want.

The install entry is where the real work hides. Enumerating what your build actually touches is harder than it looks, a dependency resolver can reach a CDN you never named, so the honest approach is to start strict and widen from the failures rather than guess wide and never revisit it.

Each transition returns a record you can keep

Every phase change goes through an authenticated control-plane call, and on success, the response gives you the sandbox record as it stands after the update, including the current policy. I'd keep those responses: they let you answer "what was this sandbox allowed to reach at 14:32?" later, without instrumenting the workload itself.

The security benefit is straightforward, but the operational benefit is what I find more useful. After an incident, knowing a policy existed isn't enough, you need to know which policy was actually in place at that point. A create-time configuration can't give you that history for a sandbox that's been running for six hours. It also gives you a boundary you can test in CI: run against a real sandbox and verify the setup-time hosts are unreachable before the untrusted phase starts.

What the swap does not do for you

Connection survival is the detail I'd pay the most attention to. Tightening the policy isn't the same as cutting off everything already connected. If your threat model requires setup-time connections to disappear before the next phase, close them yourself.

Egress and ingress are separate. Changing one doesn't change the other. Inbound access runs through exposed_ports and allow_unauthenticated_access, while the management port 9501 always requires authentication. They clear differently too: an empty exposed_ports array still leaves the management port available, while clearing the egress policy requires an explicit null.

The API key is part of this boundary. Updating the policy is an authenticated operation whether through SDK, CLI, or PATCH. Tensorlake API keys are project-scoped with project-member permissions, so a key that can update the sandbox can widen its network policy, which is why I'd keep the credential in the orchestrator rather than inside the environment it controls.

Idle timeout is a separate failure mode. timeout_secs is an idle threshold, not a wall-clock lifetime, measured against traffic through the sandbox proxy and defaulting to 600 seconds. A named sandbox suspends and can resume under the same ID; an ephemeral one terminates. Naming a sandbox is worth deciding before a phased run, since it makes suspend/resume available, and the same update call can set name alongside the policy. I wouldn't assume a policy update resets the idle clock: the update request has no timeout_secs field, and whether a control-plane PATCH counts as the "traffic through the proxy" the timeout is defined against isn't obvious, so I'd measure that directly rather than assume it. I'd also resume a suspended sandbox explicitly before updating it rather than assume the update handles that state transition; Sandbox.connect() doesn't auto-resume, though a request through an exposed port can auto-resume a suspended named sandbox.

When this is the wrong tool

If the sandbox stays at one trust level start to finish, or every phase genuinely needs the same broad access, set the policy once and leave it. Switching policies at that point just adds something to manage without reducing the sandbox's actual reach.

If untrusted code needs no network at all, skip the transitions entirely: create the sandbox with allow_internet_access=False and an empty allow_out from the start.

And egress policy doesn't solve a filesystem isolation problem. If untrusted code must never share an environment with credentials or artifacts left over from setup, tightening network access isn't enough, the policy controls where the sandbox connects, not what's already sitting inside it. Use separate environments for that instead.

The takeaway

Permission lifetime is a design choice. Tie it to the sandbox's lifetime, and you carry the combined network access every phase needs for as long as the sandbox exists.

Once you can change the policy while running, the egress policy can match the phase instead. List what setup actually needs rather than opening access broadly, replace that policy before untrusted code runs, and check the returned policy at each transition.

The version I'd actually use is simple: keep policies in a phase dictionary, one update per boundary, refuse to move back to a broader phase, and fail loudly if the policy that comes back isn't the one you expected.

It isn't much code, but it changes the security model. The sandbox no longer carries the installer's network reach for its entire lifetime. It only has it while the installer needs it.

References

Where Sandbox Ingress Speed Actually Comes From

Divy Yadav — Sat, 22 Aug 2026 20:17:34 +0000

A proxy chain into an isolated sandbox usually has more than one hop, and each one deserves its own answer to the same question:

Does this specific layer need to understand the application protocol, or is it mainly moving bytes between two points that already trust each other?

Many production proxy chains combine L7 and L4 functions at different hops. The interesting engineering work is in separating the cost of parsing and buffering at L7 from the cost of moving bytes at L4.

Get that separation wrong, and a technique like kernel Transport Layer Security (kTLS) can end up credited with a win that actually came from removing a parser.

A recent ingress rebuild provides a concrete test of that separation. The team moved one dataplane hop from a full L7 reverse proxy to an L4 forwarder using kTLS and splice(2).

They measured each change on its own rather than only comparing the old system to the new one. Most of the CPU saving in that staged comparison came from removing the parsing layer; kTLS and splice(2) added a smaller, separate throughput gain on top.

Tensorlake’s rebuild is the worked example; the underlying decision applies to proxy chains more broadly.

The questions underneath it, when an L7 hop earns its cost, when L4 becomes attractive, what disappears when a hop stops parsing, what has to be rebuilt at L4, and how to test whether removing L7 actually matters, apply to any proxy chain, not just this one.

The real question: which hop needs to understand the protocol

The path into a Tensorlake sandbox runs through an edge gateway and a separate dataplane-side hop. The edge gateway, built on Cloudflare's Pingora, terminates the client's Transport Layer Security (TLS) connection, accepts HTTP/S, WebSocket, and gRPC traffic, and extracts sandbox-routing information from the request for those protocols. SSH uses a different extraction path from the connection.

That hop is still L7 today, and stayed that way through the entire redesign. What changed sits behind it: a dataplane-side hop that used to run a full L7 reverse proxy speaking HTTP/2 over mutual TLS (mTLS), and now runs an L4 forwarder.

That's the shape of the actual decision, and it's made per hop, not once for the whole path: which layer needs to read the application protocol, and which layers only need to move authenticated bytes?

Keep L7 where a hop needs request-aware behavior, such as protocol-aware routing, application-aware retry policy, request deadlines, or synthesized responses. L4 becomes a candidate when an earlier hop has already made those decisions and the next hop only needs to move bytes over an authenticated channel.

The team also considered lower-level alternatives for that dataplane hop, including VXLAN, eBPF-steered routing, a CNI plugin, and host-level encapsulation. The engineering post says the team chose a non-invasive, application-layer mechanism instead, because the platform runs across AWS, GCP, and planned GPU neoclouds.

The data path shouldn't depend on any one provider's networking primitives. L4 forwarding won out over L7 for this one hop; the choice against those lower-level options was a separate portability decision.

The change comes down to one hop:

OLD - two L7 hops between the client and the app

+--------+      +--------------+      +----------------+      +-------------+
| Client | -->  | Edge gateway | -->  |    L7 proxy    | -->  | Sandbox app |
|        |      |  TLS + auth  |      |  mTLS+HTTP/2   |      |             |
+--------+      +--------------+      +----------------+      +-------------+


NEW - one L4 hop between the client and the app

+--------+      +--------------+      +----------------+      +-------------+
| Client | -->  | Edge gateway | -->  |  L4 forwarder  | -->  | Sandbox app |
|        |      |  TLS + auth  |      | kTLS+splice(2) |      |  plaintext  |
+--------+      +--------------+      +----------------+      +-------------+

Only the third box changes. The edge gateway stays L7 in both paths.

The public-facing side of that edge layer is documented separately, in the networking docs: the proxy preserves the request path and query string, supports WebSocket upgrades, and forwards gRPC over HTTP/2. Those are request- and protocol-aware behaviors, the kind of work that generally belongs at an L7 boundary.

When L7 is worth keeping

An L7 hop earns its cost by reading the request. Because it can see what's actually in flight, it can support:

routing on a path, header, or other application-level field
an application-aware retry policy
a per-request timeout or deadline
transforming a request or response in flight
a synthesized, application-level error when something downstream misbehaves

The edge gateway here is a clean example of a hop where that trade still holds. For HTTP/S, WebSocket, and gRPC, it extracts routing information from the request itself and authenticates the user; SSH uses a different extraction path from the connection.

An L4 hop cannot perform those application-protocol-aware functions without parsing the traffic, although it can authenticate its own transport channel using mechanisms such as mTLS.

Nothing about the dataplane redesign touched that layer, because the argument for dropping L7 never applied to it.

When L4 starts looking attractive

A hop becomes a real candidate for L4 once its cost is dominated by parsing and buffering bytes it never needs to act on, and that cost scales with data volume rather than request count. That's a different profile from a hop handling many small, distinct calls, where the request-level work is the entire reason the hop exists.

The dataplane hop here fit that profile closely. Traffic included thousands of small calls to start, stop, and observe sandboxes, alongside file uploads and downloads that moved substantially more data.

The engineering post says the old dataplane proxy parsed and buffered that bulk traffic even though the hop never needed to interpret it, since routing had already happened at the edge.

There was also a separate, structural problem layered on top. That proxy shared a binary with the sandbox orchestrator, so a routine orchestrator deploy could interrupt live connections that had nothing to do with the update.

That's a separate issue from the performance question: a team facing only the deploy-interruption problem could address it by decoupling the proxy's lifecycle from the orchestrator's, without changing the network layer at all.

What you give up when a hop drops to L4

Nothing about moving a hop to L4 is free. An L4 proxy that does not parse the application protocol can forward bytes but cannot make request-semantic decisions, so the removed L7 hop no longer provides:

a different routing decision for each request on the same connection
intelligent, application-aware retries
a real, synthesized error response when something downstream fails
watching individual requests to know a connection is still active

Any capability that depended on reading traffic has to move somewhere else or get rebuilt on a signal the L4 layer can actually observe. This forwarder takes on two of those: routing and liveness.

Routing information, which used to be obtained from each request, now arrives before tenant data in a short, length-bounded preamble naming the sandbox ID and the sandbox’s private target address (ip:port). The gateway learned that target from the scheduler.

That happens over a connection that's mutually authenticated in both directions: the forwarder presents a certificate that the gateway pins, and the gateway presents a client certificate whose identity the forwarder checks against an allowlist.

Liveness, which used to come from watching requests, is rebuilt in this implementation by metering bytes moving through each connection and reporting those byte counts to the layer that owns the sandbox’s idle timer.

In Tensorlake's implementation, that idle timer runs on a threshold rather than a fixed lifetime: the forwarder meters bytes and reports them to the dataplane, which resets the sandbox's idle timer as long as traffic keeps flowing, and only lets it time out once nothing has moved for the configured period.

Both routing and liveness took real engineering work. An L4 hop doesn't produce either one on its own.

Why kernel TLS enables the zero-copy path, but isn't where most of the performance gain comes from

The forwarder still terminates mutual TLS on the gateway connection, which is what makes the routing preamble trustworthy in the first place. A plain L4 forwarder that does this the ordinary way, decrypting into a userspace buffer and writing the plaintext back out, already gets most of the benefit of dropping the L7 hop.

This setup measured 2.07 GB/s and 0.50 CPU-seconds per GB at exactly that stage, before kTLS entered the picture at all. Removing the request-parsing layer is what did that.

The team's own expectation going in was that kTLS would be responsible for most of the savings, that eliminating two userspace copies was where the CPU went. The isolated measurement said otherwise.

What kTLS adds on top is narrower than it first sounds, and it's worth being precise about what kind of narrower. In a conventional userspace TLS termination path, encrypted bytes cross into userspace, are decrypted there, and the resulting plaintext is then written onward, creating the userspace staging and boundary-crossing work that this design aims to avoid.

Kernel TLS moves symmetric TLS record processing into the kernel after the userspace handshake and key installation, so application data is decrypted on receive and encrypted on transmit by the kernel. In this socket-to-pipe-to-socket implementation, that enables splice(2) to avoid the ordinary userspace staging path for tenant payloads; it should not be read as a guarantee that every possible internal copy is eliminated in every kTLS configuration.

The plain userspace-copy stage above still had to copy every decrypted byte through the application; it just did that without also parsing a request first. That alone is what produced most of the measured gain in this staged test, before kTLS entered the picture.

What kTLS specifically supplies is a way for a hop to keep terminating TLS, which the routing preamble depends on, while still avoiding the ordinary userspace staging path in this implementation, instead of requiring a userspace copy for every decrypted byte because the hop needs to inspect the TLS record layer.

Tensorlake reports requiring Linux with CONFIG_TLS and kTLS receive support and using Linux 6.x in production; it reports Linux 5.1 or newer for its TLS 1.3 path. Actual availability depends on the kernel build and configuration, distribution, TLS-library integration, cipher support, and the socket path.

This implementation uses software kTLS, with cryptography on the CPU rather than required NIC offload.

It's also fail-closed by design: the forwarder daemon holds no sandbox state of its own and refuses to start at all if the kernel can't attach the TLS ULP, rather than silently falling back to a slower, userspace-copying path.

On the CPU that's left, the encryption itself is a minority cost.

AES-256-GCM ran at roughly 7.9 GB/s per core. At 0.49 CPU-s/GB, that implies approximately 0.127 CPU-s/GB for the cryptographic work, or about 26% of the measured forwarder CPU. The remainder was attributed to TCP handling and syscalls in that measurement; this should not be treated as a portable AES performance constant.

How to know whether removing L7 actually matters for you

Comparing only the old system to the new one won't tell you what caused the gain.

The useful version of this test has three stages: the existing L7 path, an L4 path that still does a plain userspace copy, and the L4 path with kTLS and splice(2) added on top. The middle stage helps isolate the effect of removing the parser and buffering layer from the incremental effect of the kTLS-plus-splice(2) path, provided the workload and measurement conditions remain controlled.

The staged measurements:

STAGED MEASUREMENT: isolate each change before crediting it

+--------------------+      +--------------------+      +--------------------+
|    L7, two hops    |      | L4, userspace copy |      | L4, kTLS+splice(2) |
|     1.12 GB/s      | -->  |     2.07 GB/s      | -->  |     2.50 GB/s      |
|   0.90 CPU-s/GB    |      |   0.50 CPU-s/GB    |      |   0.49 CPU-s/GB    |
+--------------------+      +--------------------+      +--------------------+

removing L7: +0.95 GB/s, -0.40 CPU-s/GB
adding kTLS + splice(2): +0.43 GB/s, -0.01 CPU-s/GB

Measuring those as separate steps is what makes it possible to say that removing the L7 hop accounted for most of the throughput gain and nearly all of the CPU saving, with kTLS contributing the smaller remainder.

That conclusion describes this one test. It isn't a claim that every L7-to-L4 migration will split the same way.

The test also didn't stop at a single connection, and it was direct about the limits of what it measured. Under concurrency, eight tunnels on that same host aggregated to 8.67 GB/s, at which point the bottleneck was serialization in the splice loop rather than CPU.

The headline numbers come from one connection, one direction, over loopback, on a single host, stated plainly as a bound on what this specific test can tell you rather than a production capacity claim.

A benchmark that changes routing, authentication, and workload shape all at once, and only reports one before/after number, won't tell you which change actually caused the result.

When this doesn't matter at all

This throughput optimization is unlikely to materially improve a hop dominated by short, frequent calls.

Start, stop, and observe calls were never bandwidth-bound, and kTLS adds little to them, since the handshake stays in userspace and dominates a short connection.

No end-to-end latency numbers have been published for the new path, so this piece doesn't infer any.

Those calls can still see a different failure model, unrelated to throughput.

A forwarder that never parses a response cannot synthesize an application-level status code. In Tensorlake’s implementation, a half-close is forwarded as a half-close and an upstream reset reaches the client as a reset, rather than being converted into a proxy-manufactured response or timeout. TCP defines the underlying half-close and reset signals, but transparent propagation is a property of this forwarder’s implementation.

A bandwidth gain and a latency gain are two different claims. A workload bound by connection setup rather than data volume can see close to nothing from this change, even while bulk transfers improve substantially.

The takeaway

Use this sequence on your own ingress path, independent of what any one platform did:

Find the hop that's actually costing you something, and check whether the cost is per-byte, per-connection, or per-request. Bulk transfers expose parsing, buffering, and copy costs. Short calls stay dominated by connection setup and the handshake.
Name what that hop currently does that depends on reading traffic: routing, retries, deadlines, liveness, honest errors. All of it has to move elsewhere or get rebuilt on a signal the lower layer can actually see.
If the hop still needs to authenticate its traffic, work out separately whether you can keep that authentication without paying for a userspace copy on every byte. That's a distinct problem from removing the parsing layer, and conflating the two is how a technique like kTLS ends up credited with a win it didn't cause.
Benchmark the removal in three stages under concurrency, before you believe the number.

If the hop in question is mostly handling small, frequent calls, this is often a lower-priority throughput optimization unless measurements show a meaningful per-byte or proxy-CPU bottleneck. The redesign may still be worthwhile for lifecycle isolation or a cleaner failure model.

References

[Boost]

Divy Yadav — Fri, 21 Aug 2026 17:29:04 +0000

Divy Yadav

Aug 21

Best AI Document Processing Tools in 2026: Top 5 Compared for Developers and Engineering Teams

#ai #programming #machinelearning #datascience

15 min read

Best AI Document Processing Tools in 2026: Top 5 Compared for Developers and Engineering Teams

Divy Yadav — Fri, 21 Aug 2026 17:09:56 +0000

Every vendor on this list claims "99% accuracy." None of them are lying, and none of them are telling you the whole story.

I pulled up an invoice while writing this. Nothing exotic.

A print shop billing a camera store for three line items, a 20% discount, tax, a total due.

The kind of document any tool on this list would claim to handle without blinking.

Except it has four payment-mode checkboxes with only one ticked.

A "past dues inclusive" checkbox sitting next to the issue date. A discount buried inside a line label instead of its own field.

And a stray line of placeholder text, "Describe your item," left behind by whatever invoice template made the file. Sitting exactly where a lazy parser would scoop it up as if it were real content.

None of that is unusual. It's a Tuesday. It's what your accounts-payable inbox actually looks like once you get past the one vendor you tested against during the demo.

*That's the whole test for any AI document processing software. Not whether a tool can read a clean sample PDF, but whether it survives the messy, half-filled, inconsistent documents your real vendors send you every day.
*
I spent the last several days pulling apart the architecture, the pricing, and the actual verified capabilities of five tools built to handle exactly that. Here's what I found, and where each one quietly falls short.

What Actually Separates These Tools

"Accuracy" is the number every vendor leads with. It's also the least useful number for comparing tools, because each vendor tests it on their own benchmark, their own documents, under their own definition of "correct."

If you're evaluating ai for document processing, what actually decides whether it survives production is less flashy than a benchmark score:

Layout preservation. Does the parsing layer keep table structure, multi-column text, and checkbox state intact, or does it flatten everything into a stream of text that destroys the spatial relationships a human reader relies on?
Schema and prompt automation. Do you manually define every field and write every extraction rule, or does the platform infer structure from sample documents?
Validation and confidence. Does a wrong extraction look identical to a right one, or does the system flag uncertainty before bad data reaches your database?
Deployment flexibility. Can this run in your VPC, on-prem, or only in the vendor's cloud? That answer alone rules tools in or out for regulated industries.
Pricing you can actually model. Per-page, per-credit, or "talk to sales"? The difference changes whether a finance team can approve this without a quarter-long procurement cycle.

I evaluated the shortlist against these five criteria, not just benchmark scores. Here's how they stack up.

1. Unstract: Best Overall for Production-Grade Document Automation

Overview

Unstract is an open-source, LLM-native document processing platform built by Zipstack. It ships in three forms: a free, self-hosted open-source edition under AGPL-3.0, a managed Cloud edition, and an On-Premise Enterprise edition for regulated environments.

The GitHub repository sits at 6.6k stars and 629 forks, with 1,604 commits and 547 releases behind it. That's a decent sign of a project still being actively built, not one abandoned after a launch blog post.

It's built for teams who want document extraction to work like software they control, not a black box they rent. You define what to extract using prompts instead of training custom models per document type, and it deploys as an API endpoint or an ETL pipeline into your existing data stack.

Key Capabilities

Unstract's core differentiator is that it treats extraction as a pipeline with distinct, inspectable stages rather than a single opaque call.

Layout-aware parsing comes from LLMWhisperer, Unstract's own OCR layer. Where standard OCR reads a page left to right and flattens a multi-column table into a jumbled text stream, LLMWhisperer preserves the spatial structure so the extraction LLM is reasoning over an intact table instead of noise. On that sample invoice, this is the difference between correctly separating the "Discount(20%)" label from the subtotal line, versus merging them into unusable text.

Schema and prompt automation happens through the Agentic Prompt Studio, a pipeline of six AI agents split into two stages.

Three agents build the schema. A Summarizer reads each sample document on its own. A Uniformer merges duplicate fields across documents. A Finalizer turns the result into a clean, standard-compliant JSON schema.

Three more agents build and test the extraction prompts. A Pattern Miner finds the labels and formatting clues behind each field. A Prompt Architect writes the actual extraction instructions. A Critic Dry-Runner stress-tests the prompt against the schema before it ever touches a real document.

In practice, you feed it sample invoices, bank statements, or claims forms. It builds and checks the extraction logic. You don't write it by hand.

Confidence validation comes from LLMChallenge, which runs two LLMs in parallel on the same extraction, an extractor and a challenger, and only returns a value if both agree. Mismatches return NULL instead of a silently wrong guess, and the full comparison log is available via API for debugging. This is available on Cloud and Enterprise plans, not the open-source edition.

Human-in-the-loop review routes low-confidence extractions to a human reviewer with document highlighting, so a person corrects the field with the source document visible, not a blind form. This is included on Cloud plans starting at Starter.

Deployment and integration is where Unstract's flexibility shows. It connects to nine LLM providers, including OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Google Gemini, and Mistral, plus Ollama for local, offline inference. It also connects to five vector databases.

On the data side, it pulls from or pushes to a long list of destinations: Snowflake, Amazon Redshift, Google BigQuery, PostgreSQL, MySQL, S3, Dropbox, Google Drive, and more. It ships an MCP server and an n8n node too, so it slots into agent workflows and existing automation tools without custom glue code.

Strengths & Limitations

Strengths:

Automated schema generation, dual-LLM validation, and native human review, in one integrated pipeline. None of the other four tools on this list offer all three together.
LLM-agnostic. Bring your own OpenAI, Anthropic, or Bedrock key, so you're never locked into one model vendor's pricing or quality ceiling.
The open-source edition is a real, usable product, not a crippled trial meant to push you toward a paid plan.

Limitations:

LLM-agnostic pricing means your total cost includes LLM API usage on top of the platform fee. That takes more modeling than an all-in-one flat rate.
LLMChallenge and the token-saving SinglePass and Summarized Extraction modes are Cloud and Enterprise features, not part of the free self-hosted edition.
More flexibility means more surface area to configure. Multiple LLMs, multiple vector databases, multiple deployment modes, it's not a narrow, single-purpose tool you can set up in five minutes.

Ideal Use Case

Engineering teams and automation leads who need document variety handled without a template-per-vendor maintenance burden, especially in finance, insurance, healthcare, and KYC workflows where both accuracy and auditability matter. It's also the strongest fit if you specifically need self-hosted or air-gapped deployment for compliance reasons, since that's a first-class option, not an enterprise afterthought.

Pricing Model

Open Source: Free, self-hosted via Docker Compose, AGPL-3.0 licensed.
Cloud Starter: $499/month (or $416/month billed annually) for 5,000 pages/month, $0.10/page overage.
Cloud Growth: $2,249/month (or $1,874/month billed annually) for 25,000 pages/month, $0.09/page overage.
Enterprise (Cloud or On-Prem): custom pricing, adds SOC 2, ISO 27001, HIPAA, GDPR compliance, SSO/SAML, and a dedicated support SLA.
Free tier: 100 pages/day, no credit card required, plus a 14-day full trial with a pre-configured LLM stack.
LLMWhisperer is also sold standalone, from $1 per 1,000 pages for native-text PDFs up to $15 per 1,000 pages for forms and tables with checkbox detection.

2. Reducto: Best for Developer-First RAG and LLM Pipelines

Overview

Where Unstract is built for teams who want a full pipeline, Reducto is built for one job done extremely well: turning messy documents into clean structured data for engineers feeding RAG systems, AI agents, and LLM applications. It's raised $108M in total funding, including a $75M Series B led by a16z, and reports having processed billions of pages for customers that include Harvey, Scale AI, and Vanta.

Reducto treats documents as visual objects rather than raw text, which is the architectural choice behind its accuracy claims on messy, real-world files.

Key Capabilities

Reducto's core engine combines computer vision with vision-language models in a multi-pass pipeline it calls Agentic OCR. A VLM agent reviews the baseline OCR output and corrects errors, similar to a human proofreading a first draft.

Every extracted value comes back with a bounding box and a confidence score, so you can trace any output back to its exact spot on the source page.

The platform ships five APIs: Parse, Extract, Classify, Split, and Edit, across 30+ file types including PDFs, spreadsheets, and slide decks. It reports a 0.90 score on the public RD-TableBench benchmark and claims roughly 20 percentage points of accuracy advantage over major cloud document APIs, on its own internal benchmarks. Worth checking against your own documents before you take that number at face value, as with any vendor-run benchmark.

Strengths & Limitations

Strengths:

Strong performance on messy, non-templated documents, multi-column layouts, and complex financial tables. This is Reducto's core reputation.
Bounding-box citations make it well suited for compliance-sensitive review work.
Deployment options extend to VPC, on-prem, and air-gapped for enterprise customers.

Limitations:

It's a parsing and extraction API, not a full workflow platform. You define your own schema by hand; there's no equivalent to Unstract's agent-driven schema generation.
Native human review and correction workflows aren't a headline feature, so teams generally build that layer themselves.
A signed BAA and zero data retention require the Growth tier, not the entry-level Standard plan.

Ideal Use Case

Engineering teams building RAG pipelines or AI agents who need top-tier parsing on real, messy documents, and who have the engineering bandwidth to build their own schema definitions, review workflows, and downstream integrations around the API.

Pricing Model

Standard: first 15,000 credits free, then $0.015/credit; a standard parse costs 1 credit per page.
Growth: custom pricing, required for Studio evaluations, zero data retention, a signed BAA, premium rate limits, and data residency endpoints.
Enterprise: custom pricing for SSO/SAML, dedicated SLAs, and air-gapped deployment.
No published free-forever tier beyond the initial credit grant.

3. Rossum: Best for High-Volume Transactional Invoice Processing

Overview

Rossum picks a narrower fight than the other tools on this list. It's built specifically around transactional business documents, invoices, purchase orders, delivery notes, sales orders, and explicitly stays out of contracts or legal documents. That narrower scope is a deliberate trade, and it's sharpened what Rossum is good at. Over 450 organizations use it, including Bosch, Siemens, Panasonic, and Flexport.

Worth knowing before you evaluate it: Coupa, the spend-management platform, acquired Rossum in May 2026. Rossum's product and pricing are unchanged as of this writing, but it's no longer an independent company. If long-term product direction matters to your decision, factor in that Rossum's roadmap now sits inside Coupa's.

Key Capabilities

Rossum's extraction engine, called Aurora, is a proprietary AI model trained entirely in-house on millions of transactional documents. That's different from Unstract or Reducto, which call out to third-party LLM APIs. Rossum says this cuts the risk of your data leaking to an outside model provider. The trade-off: you can't swap in a different model if Rossum's own model underperforms on your specific document type.

The platform supports 276 languages, including handwriting, and checks extracted data against your ERP master data automatically. It also includes what Rossum calls Loop Validation, a built-in human review step that feeds corrections back into the model over time.

Rossum reports reaching 90%+ accuracy within 10 to 20 processed documents for a given layout, with a named case study (the Port of Rotterdam) cited at that level.

Strengths & Limitations

Strengths:

Strong accuracy reputation for its specific niche, accounts payable and other transactional documents, well documented across independent reviews.
ERP integrations (SAP, Coupa, Workday, Oracle, NetSuite, QuickBooks Online) are deep, not superficial.
The built-in human review loop means less engineering work to stand up a review process, compared to a bare API.

Limitations:

Pricing is the clearest constraint. There's no public per-page rate, no free tier, and independent sources put the Starter plan floor around $18,000 a year.
That Starter plan excludes custom business logic, master data matching, and most ERP integrations. Those need Business or Enterprise tiers, both quote-only.
The proprietary, closed model means no flexibility to choose or swap the underlying AI. Real user reviews on G2 and Capterra consistently flag weaker accuracy on unusual layouts and non-English documents.

Ideal Use Case

Mid-market to enterprise accounts payable teams processing high volumes of standardized transactional documents, where the budget supports a five-figure annual commitment and the document types stay within Rossum's transactional focus rather than expanding into contracts, claims, or general document variety.

Pricing Model

Starter: approximately $18,000/year minimum (published floor, per multiple independent sources), includes unlimited seats, email/API ingestion, and 12 months of document archive.
Business / Enterprise: custom, quote-based, required for SSO, custom business logic, and most ERP connectors beyond the basics.
SAP Marketplace reference tiers: Silver at $40,000/year for 100,000 pages, Gold at $70,000/year for 250,000 pages.
No free tier; trials and pilots are available through the sales process.

4. Nanonets: Best for No-Code Workflow Automation on a Budget

Overview

Nanonets is an AI-powered IDP platform built around a no-code, block-based workflow builder rather than a developer-first API. It's designed so operations teams, not just engineers, can assemble a document automation workflow by chaining together blocks: extraction, classification, formatting, routing to an integration.

Key Capabilities

Nanonets' extraction runs on its own OCR model, Nanonets-OCR-s, built on the Qwen2.5-VL vision-language model and trained on 250,000+ documents. It converts documents to structured markdown rather than raw text.

The AI Guidelines feature lets you describe extraction logic in plain language instead of touching a rules engine, things like applying jurisdiction-specific VAT rules or pulling only certain pages from a mixed document.

Pricing runs on a credit-per-block model. A document typically runs through four to six blocks (extraction, classification, formatting, integration) at $0.02 to $0.30 per block depending on complexity. Nanonets' own documentation estimates this lands under $2 per invoice end to end.

It connects to ERP systems including SAP, Oracle, and Salesforce, and supports SOC 2, HIPAA, GDPR, and ISO 27001 compliance with private cloud and on-prem deployment options.

Strengths & Limitations

Strengths:

The free tier is genuinely usable: $200 in credits that never expire, enough to process thousands of documents before paying anything.
The no-code block builder lowers the barrier for non-developers to own a workflow.
Broad ERP connector support and multi-region data residency (US/EU/APAC) make it viable for teams outside pure engineering.

Limitations:

Per-block, per-run pricing is metered at every step. A document that touches five blocks bills five times, and a rework loop bills again, which makes total cost harder to predict at real scale than a flat per-page rate.
Independent reviews note that advanced custom extraction can still need real setup time, and sometimes model training, despite the "template-free" positioning.
No option to bring your own foundation LLM. You're using Nanonets' own model, unlike Unstract.

Ideal Use Case

Operations-heavy teams, particularly finance and accounts payable groups, who want a workflow automation tool that non-engineers can configure and maintain, and whose document volume is moderate enough that per-block credit costs stay predictable.

Pricing Model

Starter: free to start with $200 in credits, credits never expire.
Usage-based blocks: $0.02/run for simple operations, $0.10/run for standard AI blocks, $0.30/run for complex AI extraction.
Growth / Enterprise: custom, quote-based, with volume discounts reported up to 40%.
No flat monthly platform fee on the entry tier; you pay only for blocks run.

5. Google Document AI: Best for Teams Already Built on Google Cloud

Overview

If your infrastructure already lives on Google Cloud, Google Document AI is probably already on your shortlist by default. It's Google Cloud's API-based document processing service, built on Vertex AI and organized around "processors": pre-trained or custom models for specific tasks, from general OCR to prebuilt invoice and tax-form parsers to generative custom extractors you can bootstrap with as few as 10 to 50 labeled examples.

It's consumed entirely through REST/gRPC APIs and client libraries, with Document AI Workbench as the console for labeling, training, and evaluating processors, not a turnkey business-user interface.

Key Capabilities

Document AI's processors split into three families: digitize (Enterprise Document OCR), extract (Form Parser, Layout Parser, Custom Extractor, and prebuilt parsers for invoices, receipts, W-2s, and other tax forms), and classify (custom splitters and classifiers for routing mixed document batches).

Recent updates have introduced Gemini 3 Flash and Gemini 3 Pro-powered versions of the Layout Parser and Custom Extractor, currently in preview. That's a signal Google is folding its newer models into the processor stack, not leaving it as a static OCR product.

Synchronous requests are capped at 10 pages per call. Anything larger needs the asynchronous batch API, which accepts up to 200 pages per file. Output includes bounding-box coordinates for every extracted value, so you can trace any field back to its exact source location.

Strengths & Limitations

Strengths:

Integrates natively with BigQuery, Cloud Storage, and the rest of the GCP stack, no separate vendor relationship needed if you're already there.
Per-page pricing is genuinely transparent and published, unlike two of the other tools on this list.
The generative Custom Extractor can bootstrap from a small labeled set, a real strength for niche document types.

Limitations:

Costs scale in a way that's easy to underestimate. Enterprise OCR is cheap at $1.50 per 1,000 pages, but Custom Extractor and Form Parser run $30 per 1,000 pages, and real G2 reviews consistently flag pricing as a pain point once volume grows.
No built-in workflow layer, no native human review interface, and no automated schema generation from sample documents. You're assembling a pipeline out of API calls, not adopting a packaged platform.
GCP-native by design, so teams on AWS or Azure take on real integration overhead to use it.

Ideal Use Case

Teams already standardized on Google Cloud who need specific pre-built processors (invoices, tax forms, identity documents) and are comfortable building the orchestration, review, and workflow layer themselves in exchange for tight GCP integration and transparent per-page pricing.

Pricing Model

Enterprise Document OCR: $1.50 per 1,000 pages (1 to 5 million pages/month), dropping to $0.60 per 1,000 pages above that.
Custom Extractor / Form Parser: $30 per 1,000 pages (up to 1 million pages/month), dropping to $20 per 1,000 pages above that.
Layout Parser: $10 per 1,000 pages, flat.
Custom splitter / classifier: $5 per 1,000 pages, dropping to $3 per 1,000 above 1 million pages/month.
No dedicated free tier beyond standard Google Cloud trial credits; custom processor hosting adds $0.05 per deployed version-hour.

Side-by-Side Comparison

Criteria	Unstract	Reducto	Rossum	Nanonets	Google Document AI
Architecture	LLM-agnostic, agentic pipeline	CV + VLM, Agentic OCR	Proprietary in-house model	Own VLM-based OCR model	Pretrained + generative processors
Schema automation	Yes, six-agent auto schema + prompt generation	No, manual schema definition	No, pretrained for transactional docs	Partial, via plain-language AI Guidelines	No, manual per processor
Native HITL review	Yes, on Cloud plans	No, build your own	Yes, Loop Validation	Via workflow blocks	No
Confidence validation	Yes, dual-LLM LLMChallenge (Cloud/Ent.)	Per-field confidence scores	Per-field confidence scores	Per-block confidence	Per-field confidence scores
Deployment options	OSS, Cloud, On-Prem/VPC/air-gapped	Cloud, VPC, on-prem, air-gapped (Ent.)	Cloud only	Cloud, private cloud, on-prem	Google Cloud only
Entry pricing	Free (OSS) / $499/mo	Free 15K credits / $0.015/credit	~$18,000/year, no free tier	Free ($200 credits)	Pay-per-use, no platform fee
Best fit	General-purpose production IDP	RAG/LLM pipeline parsing	High-volume transactional AP	No-code ops workflows	GCP-native teams

Decision Matrix: Which Tool Fits Your Situation

When Not to Buy Any of These

Every tool on this list earns its cost when document variety is high, volume is real, and layouts change often enough that a hand-built parser becomes a maintenance job in itself. That's the honest threshold for when AI document processing automation pays for itself instead of adding a new vendor bill for no real gain.

If you're processing invoices from one vendor with a layout that hasn't changed in three years, none of this infrastructure pays for itself. A basic PDF text extraction script and a handful of regex patterns will outperform any of these tools on cost, and you can build it in an afternoon.

The moment that calculus flips is the moment a second vendor's layout doesn't match the first, or a third document type shows up that your rules engine can't handle without another round of hand-tuning. That's the point where automated schema generation, layout-aware parsing, and confidence-based routing stop being nice-to-haves and start being the thing standing between you and a quietly broken production pipeline.

The Real Test

Before picking a tool off this list, run your worst 20% of documents, not your cleanest ones, through whichever platform you're evaluating. The invoice with checkboxes nobody filled in consistently. The scan with a coffee stain. The vendor who changed their template last quarter and didn't tell you.

Vendor benchmarks measure best-case documents on someone else's dataset. Your production pipeline will run on your worst-case documents, every day, indefinitely. The tool that survives that test is the one worth paying for.

[Boost]

Divy Yadav — Sat, 15 Aug 2026 12:07:07 +0000

Divy Yadav

Aug 14

3 Statement Descriptor Mistakes That Quietly Turn Real Purchases Into Chargebacks

#news #fintech #ai #productivity

4 min read

[Boost]

Divy Yadav — Fri, 14 Aug 2026 10:35:27 +0000

Divy Yadav

Aug 14

3 Statement Descriptor Mistakes That Quietly Turn Real Purchases Into Chargebacks

#news #fintech #ai #productivity

4 min read

3 Statement Descriptor Mistakes That Quietly Turn Real Purchases Into Chargebacks

Divy Yadav — Fri, 14 Aug 2026 10:34:28 +0000

I spent a week chasing what looked like a fraud spike. It wasn't fraud. It was 22 characters of bad UX that I'd shipped myself.

A support ticket queue that's usually quiet started filling up with the same complaint: "I didn't make this charge." Different customers, different amounts, same pattern.

My first instinct was fraud

Rotate keys, check for a leaked webhook secret, the usual on-call checklist. None of it panned out. Every single "disputed" charge turned out to be a real purchase, made by the actual customer, on the actual date.

The problem wasn't the payment

It was what the payment looked like on a bank statement.

The 22-character problem nobody warns you about

Card networks give you roughly 22 characters to explain a charge to your customer. Not 22 words. Twenty-two characters, including spaces.

If you're on Stripe, this isn't a soft guideline, it's enforced. Statement descriptors are capped at 22 characters, and you can't use <, >, ', ", or *, and the descriptor can't be only numbers.

That limit is tight enough that your actual product name often doesn't fit. "RunClub Premium Monthly" doesn't fit in 22 characters. Neither does most of what a real business is called.

So it gets cut. And a cut-off name looks nothing like the thing your customer remembers buying.

Where dynamic descriptors make it worse

If you're setting a dynamic suffix so each transaction shows different context, the math gets tighter, not looser. Here's a basic example using Stripe's Payment Intents API:

curl https://api.stripe.com/v1/payment_intents \
  -u "sk_test_yourkey:" \
  -d amount=1099 \
  -d currency=usd \
  -d "statement_descriptor_suffix=RUN CLUB APR"

Your account-level prefix, an asterisk, a space, and your suffix all share that same 22-character budget. Stripe allots roughly 10 characters for the dynamic part by default. Go over, and Stripe truncates it, sometimes into something that reads as gibberish or, worse, as a duplicate of your own business name stacked twice.

I've seen this exact failure mode described in payment platform support docs: a merchant's static prefix and their dynamic suffix both containing a version of the brand name, so the customer sees the company name twice in a row and assumes something's broken. Nothing was broken. It was just two systems both trying to be helpful in the same 22 characters.

Testing this before it reaches a real customer

Here's the part most teams skip: nobody on the engineering side actually checks what the final descriptor renders as on a real bank statement, because you can't easily see that in a sandbox environment.

What I started doing instead: after setting a new descriptor pattern, I paste the exact string I'm about to ship into UnknownCharges' descriptor decoder, the same tool a confused customer would use if they searched their statement. If a decoder built for confused consumers can't make sense of what I just shipped, neither will they.

It's a five-second sanity check, and it catches the "wait, this reads as nonsense" problem before a support ticket does.

Understanding processor prefixes as the person who set them

If your payment stack routes through a processor like Square, Stripe Connect, or Toast instead of settling directly, your customers see the processor's name before they see yours. That's not a bug in your integration. It's how card networks handle sub-merchants.

SQ *YOURBUSINESS is a completely normal, expected pattern for Square-routed charges. Customers who don't know that read it as a stranger's name attached to their statement. UnknownCharges keeps a running reference page for the SQ* prefix specifically because this exact confusion generates enough support volume across enough businesses to be worth documenting on its own.

If you're building on top of a processor, it's worth reading how your specific prefix looks to an end user who has zero context on your backend architecture, because that's the only context your customer actually has.

What happens after the customer still disputes it

Even with a clean descriptor, some percentage of customers will still file a dispute instead of contacting you first. That's a habit, not a fixable bug, and it's worth understanding what happens on their end so your support responses actually match reality.

Credit card disputes in the US run under the Fair Credit Billing Act, with a 60-day window from the statement date. Debit card disputes run under Regulation E instead, with a much tighter two-business-day window before liability caps start rising. If your support team is telling customers "just call your bank" without knowing which rule applies to which card type, you're giving advice that's sometimes wrong.

I've pointed support-facing teammates at UnknownCharges' dispute playbook when writing our own canned responses, mainly because it lays out the credit-versus-debit distinction more clearly than the two paragraphs buried in most payment processor docs.

A quick checklist before you ship a descriptor change

Situation	What to check
Setting a static prefix	Does it fit in 22 characters, no truncation guesswork?
Adding a dynamic suffix	Does prefix + `*` + space + suffix stay under 22 total?
Routing through Square, Stripe Connect, or similar	Does the resulting `PROCESSOR *NAME` pattern make sense out of context?
Multi-line-item or subscription billing	Is the descriptor different enough per product to avoid support confusion?
Before shipping any descriptor change	Paste the exact rendered string into a decoder and read it like a confused customer would

None of this is complicated engineering. It's just the one part of the payment flow that has zero automated tests, because the failure mode isn't a 500 error. It's a confused human, three weeks later, staring at their banking app.

Statement descriptors are the last UI your product shows a customer, and it's the one team most engineers never design on purpose.

PDF Structural Extraction: Why Your AI Has Never Read a Single Page

Divy Yadav — Sat, 01 Aug 2026 18:09:55 +0000

Ask an AI tool to read a scanned contract, and something strange happens: it tells you it understood the document.

It didn't.

I've watched this go wrong enough times that it stopped surprising me. Most PDF tools don't read a PDF at all.

They copy whatever characters happen to sit on the page, in whatever order the page happens to store them, and call that "extraction." A table becomes a wall of numbers with no rows. A form becomes text with no fields.

A scanned page becomes nothing, because there was never any text there to copy, just a picture of text pretending to be text.

If you've ever asked an AI assistant a question about a PDF and gotten a confidently wrong answer, this is very likely why. Here's what's really happening, and what it takes to fix it, including the exact engineering breakdown this piece is pulling from.

What you see is not what's stored

Open a PDF and you see a clean page: a heading, a paragraph, a tidy little table. None of that is actually in the file. Not really.

WHAT YOU SEE ON THE PAGE            WHAT THE PDF ACTUALLY STORES

  Q3 Report                           "Q" at (72, 40)
  ─────────                           "3" at (84, 40)
  Revenue    Costs                    " " at (96, 40)
  142,500    88,200                   "R" at (72, 58)
                                       "e" at (81, 58)
  (a clean heading                    "v" at (90, 58)
   and a tidy table)                  ... character by character,
                                       position by position

                                     no row. no column. no table.
                                     just ink, and where it goes.

I didn't quite believe this the first time someone explained it to me either. A PDF, underneath the page you see, is closer to a set of typesetting instructions: put this exact letter here, at this exact spot, in this font, at this size. It never says "this is a table." It never says "this is a heading." It just says where the ink goes.

That's fine for printing a page. It falls apart the moment you want a computer to pull meaning out of that page instead of just displaying it. Foxit's technical breakdown walks through this exact storage model in far more depth than a Medium post has room for.

Copying text isn't the same as understanding it

I used to assume "PDF extraction" meant a computer read the document roughly the way I would. It doesn't. Not by default.

Most extraction tools do exactly one thing well: they read those typesetting instructions and hand you back the characters, roughly in the order they were drawn. Engineers call this content serialization. You don't need the term. You just need to see where it breaks.

A table with mismatched row heights, extremely common in financial statements, gets its rows silently merged into the wrong place.
A signature, a stamp, or a logo isn't text at all, so it never gets copied, even when it matters for a real decision downstream.
A scanned page has no underlying characters whatsoever. It's a photograph. Ask a basic tool to "extract" it and you get back nothing.
A form field can hold a value stored completely separately from what's visually printed on the page. Copy the visible text, and you can lose the actual data entirely.

None of this is a bug in any one tool. It's the ceiling of what "copy the characters" was ever going to be able to do.

What actually understanding a document looks like

Real structural extraction asks a different question entirely. Not "what characters are here," but "what is this, and how does it relate to everything around it."

In practice, that's a few concrete things happening at once:

Two blocks of text sitting side by side get recognized as two separate columns, not one garbled sentence.
A cluster of numbers gets tied to a specific row of a specific table, not left floating as a page full of digits.
A scanned page runs through actual character recognition first, so a photo of text becomes real, usable text.
A form field's stored value gets read directly, instead of guessed at from whatever happens to be visually printed nearby.

A properly built system doesn't stop at "text" versus "not text" either. It typically recognizes something like a dozen distinct kinds of content on a page: titles, headings, ordinary paragraphs, tables, images, footnotes, hyperlinks, form fields, even stamped annotations, each one tagged for what it actually is.

Get all of that right, and a PDF stops being a picture a computer merely displays. It becomes structured data a computer can use: feed to an AI, drop into a spreadsheet, route into a CRM.

A quick before-and-after

Here's the same table, seen two ways. I find this comparison does more work than any explanation I could write around it.

BASIC EXTRACTION                      STRUCTURAL EXTRACTION

"Q3 Revenue Total   142,500           { "type": "table",
Q3 Costs    88,200   Net    54,300"     "rows": [
                                          ["Q3 Revenue", "142,500"],
one wall of characters,                  ["Q3 Costs", "88,200"],
no rows, no columns,                     ["Net", "54,300"]
no idea what belongs                   ]}
where

                                       every value knows exactly
                                       which row and column
                                       it belongs to

Left side, a machine can display it. Right side, a machine can actually use it. This is the exact transformation a properly built structural extraction engine is responsible for.

How this works, behind the scenes

You don't need to write any code to understand the shape of it. A structural extraction system like this usually runs in four plain steps.

UPLOAD          ANALYZE            CHECK STATUS         DOWNLOAD
(get back a  ──▶ (runs in the  ──▶  (poll every    ──▶  (get the finished,
 ticket for       background,        couple of            structured file)
 the file)        not instantly)     seconds)

You hand over the document, and the system stores it and gives you back a ticket number for that file. You ask it to analyze the document, and because genuinely reading a messy page takes real processing time, it starts the job in the background instead of answering instantly. You check on that job every couple of seconds until it reports back done. Then you download the finished result: a structured file listing every table, form, heading, paragraph, and image the system found, each one tagged with exactly what it is and exactly where it sits on the page.

The first time I watched one of these jobs run, I kept refreshing before I realized the wait was the whole point, not a glitch. There's no single instant "read this PDF" button, because genuinely reading a messy real-world document was never going to be an instant operation. It's a careful pass through the whole page, once, done properly.

The proof isn't just anecdotal

This isn't just an opinion floating around engineering teams. NVIDIA ran its own comparison in 2025, testing a purpose-built extraction pipeline against a general-purpose AI vision model on real financial filings and reports. The dedicated extraction approach came out roughly 7% more accurate at pulling back the right information, and processed pages about 32 times faster.

I'll admit the size of that speed gap surprised me. The general-purpose model wasn't bad at describing a page in broad strokes. It was worse specifically at the boring, structural part: keeping every number tied to the correct row, every field tied to the correct label, consistently, every single time.

That consistency is exactly what a real business process needs. A rough guess can't reliably provide it, no matter how confident it sounds. It's exactly this gap that a dedicated structural extraction engine is built to close.

The part that quietly wrecks AI assistants specifically

If you've built or used any tool that lets an AI answer questions from your own documents, this is worth sitting with for a second.

That kind of tool works by chopping documents into chunks, then handing the AI the most relevant chunks for a given question.

If the reading order feeding those chunks is scrambled, say two columns merged into one garbled paragraph, or a table's numbers floating with no row labels, then no amount of clever search logic layered on top can recover the original meaning. You can't retrieve your way out of a bad starting chunk.

Correct structure has to happen before retrieval, not after.

It's not just AI assistants that get burned

RAG pipelines get most of the attention here, but they're not the only place this breaks down. I only started noticing this pattern once I stopped thinking of it as an "AI problem" at all.

A finance team pulling numbers into a dashboard needs every value tied to the correct row label, or the dashboard is just confidently wrong.
A sales team auto-filling a CRM from scanned contracts needs the actual form field values, not a best guess at whatever text happened to sit near a checkbox.
A compliance team building an audit trail needs to know which stamps, signatures, and annotations exist on a document, not just the paragraph text.

Different teams, different tools, the same root cause every time: something upstream copied characters instead of understanding structure.

Signs your own pipeline already has this problem

A few blunt questions, if you're dealing with PDFs at any real volume. I've asked variations of these in enough conversations to know the honest answer usually comes with a pause first.

Do your table exports ever have the right numbers sitting in the wrong row?
Does anything break specifically on scanned or faxed documents, but work fine on ones typed directly into a PDF?
Do form values ever come back blank even though the form clearly has data filled in?
Does your AI assistant's answer quality quietly drop on longer, multi-column, or older documents specifically?

A yes to any of these usually traces back to the same root cause: something in the pipeline is copying characters, not reading structure.

Where this gets solved

Companies building document-heavy products don't solve this by hoping the AI figures it out. They solve it at the extraction layer, before the AI ever sees the document, using tools purpose-built to recognize tables, forms, scanned pages, and headers as the distinct things they actually are, not just characters on a grid.

Foxit's engineering team recently published a detailed technical walkthrough of exactly how this works under the hood, including the specific failure points in popular extraction libraries and how a properly built structural extraction engine avoids them. If you're building anything that touches PDFs at scale, invoices, contracts, scanned forms, it's worth the read: Inside Foxit's PDF Structural Extraction Engine.

The takeaway

Next time an AI hands you a confidently wrong answer about a document, look upstream before blaming the model. Somewhere before the AI ever got involved, a table probably lost its rows, or a scanned page never got read at all.

I think about this every time I see a demo where an AI "reads" a document flawlessly on stage. Stage demos use clean PDFs. Production doesn't. The gap between the two is exactly the gap this piece has been describing: copying characters versus genuinely understanding structure. Now you know which question to ask before you trust either one.

If you're building anything document-heavy

Foxit's developer team publishes breakdowns like this one regularly: PDF APIs, document automation, agentic document workflows. If "here's what's actually happening under the hood" is useful to you, Foxit Developer Solutions on LinkedIn is where they post it first, and the original technical write-up this piece is based on is worth bookmarking directly: Inside Foxit's PDF Structural Extraction Engine.

You can also follow Foxit's main LinkedIn page for the wider product news, outside of the developer-specific posts.

References and further reading

Foxit, "Inside Foxit's PDF Structural Extraction Engine"
LinkedIn: "https://www.linkedin.com/company/foxit-corporation"

OpenCode Is Powerful. That's Exactly the Problem.

Divy Yadav — Tue, 21 Jul 2026 12:31:40 +0000

OpenCode is the free, open source alternative to Claude Code, with full shell access to match. Here's how I ran it safely using a sandbox instead of my laptop.

OpenCode is an open-source coding agent with a workflow similar to Claude Code.

You type what you want, it reads your codebase, edits files, runs shell commands on its own. Same basic idea as Claude Code. Free though, open source, and you're not locked into one model.

Which is great, until you actually sit with what "runs shell commands on its own" means.

The first time you hand a terminal agent that kind of access, there's one question you can't quite shake: what happens the moment it runs something you didn't expect?

Most people pick a folder they don't care about and hope for the best. Or they don't think about it at all until something breaks.

I didn't love either option, so before I let OpenCode near anything that mattered, I went looking for a third one.

That's how I ended up moving the agent's shell into a disposable sandbox from Tensorlake, that runs these as a cloud service, while leaving everything else right where it was on my laptop.

Your Laptop
      │
      ▼
   OpenCode
      │
      ▼
Tensorlake Plugin
      │
      ▼
 Disposable Linux Sandbox

Here's what that looked like, and what actually surprised me once I had it running.

What I Kept Picturing Before I Typed Anything

A coding agent's real risk has nothing to do with intelligence. Most of the time it gets things right. What it doesn't do is pause. It doesn't stop to double-check whether the command it's about to fire off is the one you actually meant to approve.

One scenario kept nagging at me before I typed anything. I ask it to clean up build artifacts, and it runs rm -rf ./build. Except ./build turns out to be a symlink into somewhere I still needed, the kind of thing an agent skims right past, and honestly, the kind of thing I don't always catch either when I'm moving fast.

Or I ask it to install a dependency. npm install fires a postinstall script. That script rewrites a config file I never agreed to touch. I've seen postinstall scripts do weirder things than that with a human sitting right there watching.

Neither one is a bug to be honest.

That's just what shell access does when nothing stands between a command and your machine.

Then I thought bigger than my own laptop. Same agent, but now it's refactoring a production monorepo that ten other engineers are actively pushing commits to. Same misread symlink. Same rogue postinstall script. Except now it's not just my afternoon on the line.

bash, write, edit. These aren't agent-specific features. They're shell and filesystem access, handed to a process making its own calls about what to run next.

The instinct is to fix this with more caution. Review every diff. Approve every command. That works right up until it doesn't, because the entire point of an autonomous agent is that you eventually stop reviewing every single step.

The real fix turned out to be a different blast radius.

Put the agent's commands somewhere disposable instead of on my machine, and now a bad command won't harm my system.

Running locally isn't wrong, to be clear. For a throwaway project, or a workflow you already trust, it's exactly what you want. What changes the math is pointing autonomous shell access at something a mistake would actually cost you.

Brain Local, Hands in a Sandbox

One table covers the whole shift. The wiring behind it takes a little longer to explain.

	Local OpenCode	OpenCode + Tensorlake
Commands run	On your laptop	Remotely, in a sandbox
Filesystem	Your actual filesystem	Isolated, disposable
Dependencies	Whatever's on your machine	Disposable environment
A bad command affects	Your machine	The sandbox, not you

That's the outcome. The wiring is simpler than it sounds.

Tensorlake ships a plugin called tensorlake-opencode. The plugin doesn't replace OpenCode.It intercepts specific tool calls and reroutes them. Once I actually understood that distinction, the idea turned out simpler than I expected.

OpenCode on your laptop, tools in a sandbox, that's the actual shape of it. The OpenCode harness itself, the interface, your session, all of that keeps running on your machine exactly like before.

The model call still goes out to whichever provider you've configured, Anthropic, OpenAI, whoever, the same as it always did; that part was never local to begin with, and this plugin doesn't change it.

What changed was where each individual tool call landed:

                Your Laptop
                     │
                     ▼
          OpenCode Harness
                     │
      ┌──────────────┴──────────────┐
      ▼                             ▼
Model Provider              Tensorlake Sandbox
(Claude/OpenAI/etc.)      Shell + Filesystem

The model call still goes wherever you configured your provider. Only the tool calls, the hands, move into the sandbox.

webfetch and websearch are the two exceptions that stay local, since neither touches a filesystem. bash, write, edit, read, ls, glob, and grep are the ones that get rerouted.

Every intercepted command now makes a network round trip instead of running instantly on my machine. Tensorlake's documentation says a sandbox starts up in a few seconds, with the underlying VM image itself booting in hundreds of milliseconds. Their GitHub page and product site separately claim resume from a suspended state also lands under a second. My own first sandbox, the one the plugin spun up automatically on that uname -a call, took 2.3 seconds end to end, per the timestamp the plugin logged, which fits comfortably inside what the docs describe. That's likely the plugin's own provisioning and connection overhead stacked on top of the raw VM boot, not just the VM starting up. Either way, it's not something you sit around waiting on.

Setting It Up

The whole setup turned out to be one config entry and one environment variable, typed in this order.

First, the plugin, added to ~/.config/opencode/opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "plugin": ["tensorlake-opencode"]
}

OpenCode installs it automatically, no separate npm install needed. Then the API key, exported in the same shell I was about to launch OpenCode from:

export TENSORLAKE_API_KEY=your_api_key_here
opencode

Nothing happened. No sandbox spun up on launch, which threw me for a second. First instinct was that I'd messed up the config somehow, and I almost went back to check the JSON for a typo before actually rereading what I'd just set up: lazy creation, not eager. Tailing ~/.local/share/opencode/log/tensorlake.log just confirmed the plugin had loaded, nothing more, until I actually asked the agent to do something that needed a sandbox.

The First Real Test

Sandbox creation in this plugin is lazy. It waits for the first tool call that actually needs the filesystem or a shell, not the moment you launch the session.

I didn't want to point it at anything that mattered yet, so the first command I actually gave it was about as low-stakes as they come:

Run: uname -a

That single bash call was what triggered everything. A "Sandbox created" toast showed up in the terminal, and I had the log tailing in a second window the whole time. It read almost exactly what the docs describe: a line saying a new sandbox was being created for the session, then a second line confirming it was live, at 2.3 seconds.

The output said Linux. My laptop runs macOS. Not going to lie, that one word convinced me faster than any architecture diagram would have.

Next I asked it to prove the filesystem side too:

Write "Hello Tensorlake" to /tmp/workspace/test.txt, then read it back.

The write and read both happened entirely inside the sandbox, confirming that filesystem operations were isolated from my machine.

/tmp/workspace was the agent's working directory inside the sandbox, not a folder anywhere on my disk. Nothing in either test would have cost me anything if it had gone wrong.

With the sandbox working, I wanted to move beyond smoke tests and verify the full developer workflow on a real repository.

Clone https://github.com/benjaminp/six.git, find the ensure_str function, 
improve its TypeError message so it names the function it came from, 
then run the test suite and tell me if anything broke.

This was intentionally a tiny change—the goal wasn't to contribute a feature, but to verify that OpenCode could complete the same edit–test loop I'd expect during normal development.

six is a small, well-known Python compatibility library: one source file, a straightforward test suite, and just enough structure to exercise a realistic edit–test workflow.

OpenCode cloned the repository, located ensure_str, modified the function, and ran the test suite entirely inside the sandbox.

The original implementation raised TypeError("not expecting type '%s'" % type(s)).

The edit made it TypeError("ensure_str: not expecting type '%s'" % type(s))—a deliberately small edit that was easy to verify with the existing test suite. Because the tests only verify that a TypeError is raised rather than checking the exact message text, this made for a safe, minimal edit, and the run confirmed it: 184 passed, 16 skipped, 0 failed.

Same as before the change.

That's the part that mattered to me. Not the specific repo or the specific one-line fix, but that a clone, a real edit, and a full test run all happened inside the sandbox, on an actual project, without me once worrying about what would happen to my own machine if something in that chain went sideways.

Three Things That Clicked Once It Was Running

I expected the isolation. The other two took me by surprise.

Isolation. A bad command genuinely has nowhere real to land. A runaway install, an rm aimed at the wrong path, a dependency that half-installs and leaves things broken. All of it happens in a sandbox I can throw away, not my actual working tree.

Reproducibility mattered more once I pictured someone else on the team running this same setup. Whatever's actually installed on my laptop stops being relevant, because the agent isn't running on my laptop anymore. Register one image with the right toolchain baked in, point every future session at it, and the "works on my machine" conversation just stops happening.

Then there's persistence, the one I hadn't planned for. A named sandbox doesn't vanish when OpenCode restarts. It just sits there, parked. Come back later and the same working directory, the same installed packages, the same warm caches are all still exactly where I left them. Nothing rebuilt, nothing reinstalled.

A disposable CI container vanishes the second a run finishes. This one was still there the next day, same as I'd left it.

Took me an hour of lost installed state before that actually registered.

Configuring the Sandbox for Real Work

Once I trusted it with something more than a test file, I went back and actually sized it properly. The plugin reads a small set of environment variables once, at the moment the first sandbox gets created, so they need to be set before you launch OpenCode, not after:

export TENSORLAKE_CPUS=4
export TENSORLAKE_MEMORY_MB=8192
export TENSORLAKE_DISK_MB=20480
export TENSORLAKE_IMAGE=my-custom-image

Variable	Default	Controls
`TENSORLAKE_CPUS`	2	vCPUs
`TENSORLAKE_MEMORY_MB`	4096	RAM in MB
`TENSORLAKE_DISK_MB`	10240	Disk in MB
`TENSORLAKE_IMAGE`	platform default	The image the sandbox boots from

If you're using a Personal Access Token instead of a project-scoped key, you'll also need TENSORLAKE_ORGANIZATION_ID and TENSORLAKE_PROJECT_ID.

TENSORLAKE_IMAGE is the one actually worth using. Bake your language runtime, system packages, and project dependencies into a registered image once:

tl sbx image create Dockerfile --registered-name my-custom-image
export TENSORLAKE_IMAGE=my-custom-image

Every session after that starts already warm. I stopped watching the agent reinstall my stack from scratch every time I opened a new session.

Note: export only lasts for the current shell, so I added these to my shell profile once I knew I'd be using this setup regularly.

The Operational Questions I Actually Had

Before trusting this with anything real, I wanted answers to a few things that don't come up in a quick demo. Worth being upfront about one thing here: most of what follows is documented at the Tensorlake SDK and platform level. I haven't confirmed that the OpenCode plugin specifically exposes or manages each of these the same way, only that the underlying sandboxes it creates support them. Where that distinction matters, I've called it out below.

What if I just walk away mid-session? Named sandboxes auto-suspend after their idle timeout rather than terminating, at the platform level. Tensorlake's product site says the meter stops the moment it suspends, and their GitHub page puts resume at under a second, with filesystem, memory, and running processes exactly where you left them, not rebuilt. I haven't independently timed a resume myself, and I haven't confirmed whether the OpenCode plugin surfaces any control over this behavior or just inherits the platform default, so take the specific number as Tensorlake's claim about the platform, not a tested claim about this plugin.

What if the agent kicks off something long-running, like a build or a test suite? The underlying Tensorlake SDK supports starting background processes inside a sandbox that outlive the single command that launched them. I haven't tested whether the OpenCode plugin's bash interception uses this pattern specifically, or whether a long-running command inside OpenCode just holds the tool call open for the duration. Either way, the sandbox itself isn't the bottleneck.

What if my connection drops mid-command? Tensorlake has written publicly about this exact failure mode at the platform level: when the transport hiccups mid-run, they retry and reap orphaned sandboxes so a flaky connection doesn't cost you the whole task. Again, this is a platform-level guarantee. I haven't tested how the plugin itself behaves if your local connection drops mid-tool-call.

What if the sandbox itself crashes outright, not just an idle timeout? Here's where I'll be straight with you: I didn't find documentation covering a hard crash mid-command specifically, at either the platform or plugin level, and I didn't manage to force one during testing either. If you're planning to run this against something you really can't afford to lose, that's worth confirming directly with Tensorlake rather than taking my word for it.

Where I'd Actually Use This

By the end I had a rough rule for myself. Picture a coding agent refactoring a production monorepo while another engineer is pushing commits at the same time. The question was never whether the model was smart enough. It was whether I wanted its shell commands landing on my own workstation.

This isn't the right setup for every OpenCode session, and I don't pretend otherwise:

My situation	What I'd do
Personal throwaway project	Run locally
Client repository	Use Tensorlake
Production codebase	Use Tensorlake
Letting the agent experiment freely with shell commands	Use Tensorlake
Shared engineering environment	Use Tensorlake

The sandbox earns its place anywhere the blast radius of a wrong command actually matters. If I'm the only person who'll ever touch the repo and I'd shrug off losing it, I'm solving a problem I don't have yet.

Worth a quick word on why this isn't just Docker or a Codespace with extra steps. A Docker container shares your host's kernel, which is fine for packaging an app but a thinner isolation boundary than a full VM if you're worried about what an autonomous agent might run.

A local VM gives you the stronger boundary but is heavy and slow to spin up and tear down for something you might want to throw away every few minutes. GitHub Codespaces and Dev Containers solve a different problem: a consistent, persistent dev environment tied to a specific repo, not a fast, disposable environment built to be created and discarded on every session.

The MicroVM approach here is closer to a VM's isolation with something closer to a container's boot speed, and it's built specifically to be thrown away.

If You Want to Try This Yourself

Here's the order I'd do it in, knowing what I know now.

Step 1: Install OpenCode, add tensorlake-opencode to opencode.json. This just gets the plugin loaded, nothing else happens yet.

Step 2: Export TENSORLAKE_API_KEY, launch OpenCode, confirm the plugin loaded via the log file. This is the "is it actually on" check.

Step 3: Ask it to run something trivial, like uname -a. This is what triggers sandbox creation and gives you visible proof the command ran remotely.

Step 4: Set TENSORLAKE_CPUS, TENSORLAKE_MEMORY_MB, and TENSORLAKE_DISK_MB to match your real workload, before your next session starts, not mid-session.

Later: Build a custom image with your actual toolchain baked in, once the default image starts feeling like it's missing things you keep reinstalling.

Closing

I went looking for a third option: an agent with full autonomy, running somewhere that wasn't my laptop. Got that part working fast. What actually surprised me was how little I had to think about afterward.

I stopped reading every command before approving it once this was running. Wasn't carelessness. Just nothing left on my machine for a bad command to reach.

Turns out the part that mattered was never whether the agent could run commands. It was where they landed. Not stopping mistakes. Just making sure they happen somewhere disposable instead of somewhere that costs me.

If you've started letting an autonomous coding agent anywhere near your shell, moving that shell execution into a disposable sandbox is one of the highest-leverage changes you can make, and Tensorlake's own numbers put the setup at a few minutes, not an afternoon. The config entry and the environment variable up above are the whole thing.

References

Tensorlake OpenCode Integration: Full setup, configuration, and lazy sandbox creation model
OpenCode: The open source, terminal-first coding agent, MIT-licensed and model-agnostic
tensorlake-opencode on npm: The plugin package referenced in this article
Plugin source on GitHub: Tool interceptors, session manager, lifecycle handling
Tensorlake Sandbox Lifecycle: The suspend, resume, and snapshot model underneath this integration
Tensorlake Sandbox Images: Building and registering a custom image

Master These 6 AI Concepts to Become an AI Engineer ( A Visual Explanation)

Divy Yadav — Wed, 15 Jul 2026 09:27:36 +0000

Job postings asking for AI skills are up 143% in a single year. The people filling those roles didn't spend years on advanced math. They learned six ideas, in the right order, and started building.

Most people think you need a math degree, or years of computer science training, to become a serious AI developer.

You don't.

The gap between someone who can barely get a chatbot to behave and someone building real, production AI systems isn't intelligence. It isn't years of study. It's six specific concepts, learned in an order that actually makes sense, instead of scattered across a hundred confusing tutorials.

Here they are, explained assuming you know nothing yet.

A quick honest note before we start

This roadmap covers how to build AI-powered applications, chatbots, assistants, and tools that use existing AI models well. That's different from becoming a machine learning researcher who builds new models from scratch, which genuinely does need heavy math and years of study.

Most people who say "I want to become an AI developer" mean the first path. That's also where almost all the current job demand is. AI-related job postings are up 143% year over year, and the engineers filling those roles are, overwhelmingly, application builders, not research scientists.

This is that path.

Level 1: Beginner

1. The Context Window

Here's the concept that trips up almost every beginner, because it isn't obvious until something breaks.

An AI model can only "see" a limited amount of text at once. That limit is called the context window. Picture a whiteboard in a meeting room. You can write a lot on it, but once it's full, anything written past the edge simply isn't there anymore. It's not that the AI forgot. It never saw it.

This matters the moment you try to have a long conversation, or feed the AI a huge document. Past a certain point, older parts of the conversation quietly fall off the edge of the whiteboard, and the AI starts responding as if they never happened.

Understanding this one limit explains most of the "why did the AI suddenly get confused" moments beginners run into.

Level 2: Intermediate

2. RAG (Retrieval-Augmented Generation)

An AI model only knows what it learned during training. Ask it about your company's internal policies, and it has nothing, because it never read them. RAG fixes this.

Think of it as the difference between a closed-book exam and an open-book one. Without RAG, the AI answers purely from memory. With RAG, it's allowed to flip open a specific book—your documents, your database, your knowledge base—and check before answering.

In practice, this happens in three steps:

Chunking: Your documents get broken into smaller pieces and stored in a searchable format.
Retrieval: The AI searches that storage for the pieces most relevant to the question.
Augmentation and Generation: It answers using both what it already knew and what it just found.

Virtually every AI chatbot that answers questions about a specific company's internal information is running RAG under the hood. It is arguably the single most valuable skill on this entire list, because it is the difference between a generic AI and one that actually knows your business.

3. Fine-Tuning

Prompting hands the AI a fresh set of instructions every single time. Fine-tuning is different: it actually reshapes how the model behaves, permanently, by training it further on a narrow, specific set of examples.

Think of the difference between handing someone a manual before every task versus sending them through months of specialized training. The manual works for most jobs. Specialized training changes how someone thinks about the job itself.

Pro Tip: Most developers should reach for RAG first. Fine-tuning costs more, takes longer, and is usually only worth it when you need a very specific style, format, or behavior that no amount of careful instruction alone can reliably produce.

4. AI Agents

Everything so far has been about the AI answering questions. An agent is about the AI actually doing things.

Picture the difference between a consultant who gives you advice and an assistant who actually goes and books the flight. A consultant-style AI just answers. An agent actually takes real action instead of just describing what action you should take: searching the web, running code, sending an email, or updating a database.

The AI does this by deciding, on its own, which step to take next based on what it just learned. Ask it to fix a bug, and it can read the error, try a fix, check if that fix worked, and try something else if it didn't—all without you typing a new instruction after every single step.

This is where AI development starts feeling genuinely powerful, and also where it starts requiring real care, which is exactly why the next two concepts exist.

Level 3: Advanced

5. MCP (Model Context Protocol)

Imagine every country having a different shape of electrical socket, and every single appliance needing its own custom adapter just to work when you travel. That was the old way AI agents connected to outside tools: every connection was custom-built, one at a time, for every tool and every AI model.

MCP works like a universal socket standard.

Once a tool speaks MCP, any AI agent that understands MCP can plug into it directly. No custom adapter needed. Anthropic introduced this standard, and it has quickly become the common way agents connect to databases, files, and other services, regardless of which AI model is actually doing the work. If you're building an agent that needs to reach outside tools, MCP is very likely how that connection gets made.

6. Harness Engineering

This is one of the most critical operational concepts for production engineering. Picture a stunt performer on a film set. They're genuinely skilled. But nobody lets them attempt a dangerous stunt without a safety harness, a spotter, a hard limit on how many takes they get, and a director watching every single shot.

The performer's skill isn't what keeps them safe on set. The equipment and process wrapped around them is.

An AI agent works the same way. The model is the skilled performer. The harness is everything wrapped around it:

A hard limit on how much it's allowed to spend before it has to stop.
Checkpoints that save progress so a crash doesn't waste hours of work.
Guardrails on which actions it's actually allowed to execute.
Logs so a person can audit exactly what happened if something goes wrong.

This concept exists because of one striking number: 88% of AI agent projects never make it into real production use. Not because the models were bad, but because nobody built the harness around them.

How these six AI concepts fit together

None of these exist alone in a real system. A production AI assistant, the kind companies actually pay for, typically uses several of these at once: RAG to pull in company-specific knowledge, an agent to actually take action, MCP so that agent can reach real tools, and a harness watching the whole thing to make sure nothing goes wrong.

Learning them one at a time makes sense. Using only one at a time, in a real product, almost never does.

Key Takeaways

The context window is the AI's limited working memory. Understanding its limit explains most confusing AI mistakes.
RAG lets an AI answer from your own documents instead of just what it learned in training. It is the most valuable skill on this list.
Fine-tuning permanently reshapes a model's behavior. Reach for it only when instructions genuinely can't get you there.
An AI Agent takes real action instead of just answering, deciding its own next step based on what it just learned.
MCP is the universal standard that lets agents plug into outside tools without a custom connection built for every single one.
Harness engineering is what turns a smart model into a reliable system. The missing harness is usually why agent projects fail to reach production.

The part worth remembering

Nobody starts as an advanced AI developer. Every person building serious production AI systems today started exactly where you might be starting now—watching a conversation quietly lose track of what was said three messages ago and wondering why.

What separates a beginner from someone advanced isn't talent. It's this list, learned in order, applied to something real. The math and the deep model architecture that everyone assumes is required? Most working AI developers never touch it. They learned six ideas, built things with them, and kept going.

You can start with the context window today. Nothing on this list requires anything you don't already have.

References

[Boost]

Divy Yadav — Fri, 10 Jul 2026 12:39:27 +0000

Divy Yadav

Jul 10

Why Your AI Experiments Keep Starting From Scratch (And How Tensorlake Fixes It)

#ai #webdev #programming #productivity

12 min read

DEV Community: Divy Yadav

[Boost]

Your Sandbox Shouldn't Keep Its Install-Time Network Access

Your Sandbox Shouldn't Keep Its Install-Time Network Access

The permission window is wider than the work

What a live policy change has to guarantee to be worth using

How the policy swap behaves on a running sandbox

What allow_internet_access actually controls once allow_out is non-empty

A policy state machine for a phased agent run

Each transition returns a record you can keep

What the swap does not do for you

When this is the wrong tool

The takeaway

References

Where Sandbox Ingress Speed Actually Comes From

The real question: which hop needs to understand the protocol

When L7 is worth keeping

When L4 starts looking attractive

What you give up when a hop drops to L4

Why kernel TLS enables the zero-copy path, but isn't where most of the performance gain comes from

How to know whether removing L7 actually matters for you

When this doesn't matter at all

The takeaway

References

[Boost]

Best AI Document Processing Tools in 2026: Top 5 Compared for Developers and Engineering Teams

Best AI Document Processing Tools in 2026: Top 5 Compared for Developers and Engineering Teams

What Actually Separates These Tools

1. Unstract: Best Overall for Production-Grade Document Automation

Overview

Key Capabilities

Strengths & Limitations

Ideal Use Case

Pricing Model

2. Reducto: Best for Developer-First RAG and LLM Pipelines

Overview

Key Capabilities

Strengths & Limitations

Ideal Use Case

Pricing Model

3. Rossum: Best for High-Volume Transactional Invoice Processing

Overview

Key Capabilities

Strengths & Limitations

Ideal Use Case

Pricing Model

4. Nanonets: Best for No-Code Workflow Automation on a Budget

Overview

Key Capabilities

Strengths & Limitations

Ideal Use Case

Pricing Model

5. Google Document AI: Best for Teams Already Built on Google Cloud

Overview

Key Capabilities

Strengths & Limitations

Ideal Use Case

Pricing Model

Side-by-Side Comparison

Decision Matrix: Which Tool Fits Your Situation

When Not to Buy Any of These

The Real Test

[Boost]

3 Statement Descriptor Mistakes That Quietly Turn Real Purchases Into Chargebacks

[Boost]

3 Statement Descriptor Mistakes That Quietly Turn Real Purchases Into Chargebacks

3 Statement Descriptor Mistakes That Quietly Turn Real Purchases Into Chargebacks

The 22-character problem nobody warns you about

Where dynamic descriptors make it worse

Testing this before it reaches a real customer

Understanding processor prefixes as the person who set them

What happens after the customer still disputes it

A quick checklist before you ship a descriptor change

PDF Structural Extraction: Why Your AI Has Never Read a Single Page

What you see is not what's stored

Copying text isn't the same as understanding it

What actually understanding a document looks like

A quick before-and-after

How this works, behind the scenes

The proof isn't just anecdotal

What `allow_internet_access` actually controls once `allow_out` is non-empty