Dan Stephenson, ThoughtPivot
When Apify retired rental pricing for Actors on October 1, 2026, a lot of developers lost the pricing model that actually fit their heaviest customers. Pay-per-event works well for long-tail and agent-style usage. It is a worse fit for teams that run the same Actor on a schedule every day and want a predictable monthly line item.
Bytewells is our answer: a hosted, Apify-compatible Actor marketplace where developers can list the same codebase they already maintain, and renters pay by the month, with compute billed separately. This post is the build story. It covers what we pulled from the open-source community, what we built ourselves on ThoughtPivot infrastructure, and why owning the seams mattered as much as speaking the protocol.
The short version of our approach: take the best open foundations (Crawlee, the Apify SDK shape, and the Crawlee Cloud lineage), then build the product layer on ThoughtPivot's own open tools, Flight and Scribe, so we can operate, price, and evolve Bytewells without waiting on someone else's roadmap.
Two different things named Crawlee
People mix these up, so I will separate them up front.
Crawlee is the open-source scraping library. Cheerio, Playwright, Puppeteer, request queues, storage helpers: the toolkit Actor developers already use. Bytewells does not replace Crawlee. If your Actor uses Crawlee today, that is the code we want to run.
Crawlee Cloud is something else: an MIT-licensed, self-hosted platform for running Crawlee and Apify Actors on your own infrastructure. Its core idea is drop-in compatibility with the Apify SDK. Existing Actor code keeps working when you point APIFY_API_BASE_URL at your own instance instead of Apify's API.
That second project matters to our lineage. Bytewells did not start by inventing a new Actor runtime from a blank page. We started from the Crawlee Cloud approach: an Apify-compatible API surface, containerized Actor runs, and the storage primitives production scrapers already expect. Then we built a hosted marketplace and commercial layer on top of it.
Credit where it is due: Amine Mbarki and the Crawlee Cloud work made "keep the SDK, change the host" a concrete, open-source path instead of a wishlist item.
We are not Apify. We are not Crawlee Cloud either. Bytewells is a ThoughtPivot product: hosted runners, a Store, Stripe rentals and payouts, import tooling, and the operational work that turns a self-hostable runtime into something customers and developers can rely on without standing up the stack themselves.
What compatibility has to mean in code
Compatibility is easy to claim and hard to keep. Our rule was simple: an Actor written against the Apify SDK and Crawlee should run without a rewrite, and apify-client consumers should keep working after a base-URL change.
In practice, that means:
- Point the SDK at
https://bytewells.com/api/v2with a Bytewells token. - Inside a run, inject the environment the Apify SDK expects: Actor and run IDs, default dataset, key-value store and request queue IDs, memory and timeout, APIFY_IS_AT_HOME=1, storage paths, and the usual surface.
- Treat Actor.init(), Actor.getInput(), Actor.pushData(), Actor.setValue(), request queues, webhooks, and schedules as first-class paths, not nice-to-haves.
- Keep a living compatibility ledger of known gaps versus Apify's wire format: what is done, what is accepted for later, and what we deliberately diverge on. Compatible stays a checklist instead of a vibe.
One deliberate divergence is worth calling out. On Bytewells, Actor code often belongs to someone other than the person running it. So the runner does not inject the renter's API key into the container.
APIFY_TOKEN inside a run is a run-scoped token. It authenticates as the run owner while the run is active, only on the endpoints an Actor needs: datasets, key-value stores, request queues, its own run and log, and a narrow set of related calls. Everything else returns 403. That is not Apify's model. It is the model you want when third-party Actor code is executing on shared infrastructure.
Import follows the same philosophy. bw import apify, or Connect Apify in the console, pulls source, README, input schema, and listing defaults from an Actor you already own on Apify. It builds the Actor on Bytewells and creates a draft Store listing.
It only reads from Apify. Your Apify listing stays yours. There is no exclusivity and no forced migration.
npm install -g @thoughtpivot/bytewells-cli
bw signup --username you
bw import apify you/your-actor
That is the whole fork-avoidance story in three lines. One codebase. Two storefronts when you want both.
What we built on top of the open-source runtime
An Actor runtime is not a marketplace. The product work that sits on top of the compatible runtime is where most of Bytewells lives, and where ThoughtPivot's own stack starts to matter.
We did not rent every seam of the product. After the Crawlee Cloud-shaped runtime gave us a path to run existing Actors, we still had to ship a Store people can browse, a console operators can trust, billing that pays developers, import tooling that does not force a rewrite, and a data model that does not drift every time a listing field changes. Those layers are Bytewells. They sit on Flight and Scribe.
Store and console. Next.js surfaces for browsing, listing, pricing, and operating Actors. Flight backs the app and API shapes behind those surfaces so features stay colocated with the backends that serve them.
API. Services that speak Apify-shaped /v2 routes plus the marketplace endpoints rentals, credits, listings, and payouts need. Compatibility lives at the edge; product logic lives in code we own.
Billing. Stripe for rentals and prepaid compute credits, and Stripe Connect for developer payouts. Developers keep 90% of each rental charge before Stripe's card and billing fees, passed through at cost. We take 0% commission on customers they refer.
CLI. The @thoughtpivot/bytewells-cli package, bw, handles signup, push, import, publish, and day-to-day operator work.
Execution fleet. Builders and runners turn source into a built image, then run that image for entitled renters in isolated containers.
Product data via Scribe. Scribe holds schema-driven product and marketplace data: structured components with validation and version history instead of a pile of ad-hoc tables that drift from the API.
We have spent real engineering time on fleet hygiene: disk admission so a full runner cannot drain the READY queue by failing every pull, registry-scoped image eviction so tagged Actor images do not silently fill disks, and infrastructure-level retries for host problems that are not Actor bugs.
Our data-plane choices are boring on purpose: Postgres, Redis, and S3-compatible object storage. Boring is a feature when the interesting question is whether yesterday's Actor still runs tomorrow.
Why we build on our own tools: Flight and Scribe in Bytewells
Open source got us a compatible Actor runtime. It did not give us a product we could operate, price, and evolve on our terms. For that, Bytewells runs on ThoughtPivot's own open tools: Flight and Scribe.
Flight is our Apache 2.0 app server for component-shaped full-stack apps. Backends live next to features as .backend.ts files. We use it for Bytewells app and API surfaces where we need horizontal scale, Redis-backed sessions and cache layers, and a production path we can deploy on ordinary hosting without locking the product to someone else's edge.
Scribe is our Apache 2.0 schema-driven data modeling service. Components are JSON Schema files. We use it for Bytewells product and marketplace data so listings, account-facing records, and marketplace structures get validation, version history, and a clear schema instead of reinventing that stack inside the marketplace every time the product moves.
Both are open source under Apache 2.0. Building Bytewells on our own open tools is not a closed-stack story. Anyone can read, fork, and use the same foundations we run in production. That is the same habit that made Crawlee and Crawlee Cloud useful to us: pull the best from open source, contribute and maintain the layers we rely on, and keep the hosted marketplace as the product we operate.
Early Bun research and benchmarks for Flight and Scribe by Vasilen Lyubomirov (github.com/VasilenL), whose work helped start this.
Why own these layers instead of renting every seam?
- Control over the seams. Protocol compatibility at the edge is not the same as owning Store, console, and billing flows. When a marketplace endpoint has to evolve, we change our code instead of waiting on a vendor roadmap.
- Ownership of the product, runtime, and data model. Crawlee and the Apify SDK define how Actors talk. Flight and Scribe define how Bytewells itself is built and how product data is stored and versioned.
- Performance and cost tuning we can actually do. We can profile, trim, and re-host our own services when the bottleneck is our product path.
- Predictable operations. Fewer external dependencies on the critical path means fewer mystery outages in the layers that render the Store, accept imports, or serve account data.
- Root-cause fixes without a ticket queue. If a schema migration or an API surface is wrong, we ship the fix.
- Portability and openness. Flight and Scribe are Apache 2.0 and run where we point them. Useful components stay public; the hosted marketplace stays ours. Developers and operators outside ThoughtPivot can build on the same tools.
A concrete example: when the marketplace needs a new listing field or a stricter input contract, Scribe's schema-first model lets us version the change in one place instead of chasing mismatched validators across routes. When the console needs a new operator flow, Flight keeps the backend next to the feature so the API and UI do not drift. Neither win requires abandoning open source. They require owning the layers where product differentiation and day-two operations live, and keeping those layers readable in public repos.
We are not anti-open-source. Crawlee and the Apify SDK are the right foundation for Actor code. Crawlee Cloud (MIT, by Amine Mbarki) showed a workable shape for an Apify-compatible host. Flight and Scribe are Apache 2.0 ThoughtPivot projects we keep building in the open, and we use them inside Bytewells so we can keep the product sharp without forking the ecosystem developers already trust.
The pattern: stay compatible at the edge, own the product layer on open tools you control, and put your own stack where you need to fix root causes and ship.
Source, ownership, and what renters actually get
This needs to be said plainly, because the wrong version of this sentence travels fast.
When a developer connects Apify or imports an Actor, Bytewells receives the source. We need it to build and run the listing. Renters get the built image, not a checkout of your repo. Developers keep ownership.
Under our Publisher Terms, the license we take is to host, build, store the build, run the image for entitled customers, and display the listing. We may inspect source for legal, compliance, or security reasons: malware, credential theft, or a listing that does not match the Actor. There are no buyouts. There is no exclusivity.
That is the honest trade for a second store that can actually execute your Actor.
Architectural lessons you can steal
Whether or not you ever list on Bytewells, these are the decisions I would make again.
Treat the protocol as the product boundary. If Actors already speak Apify SDK and Crawlee, competing on a new SDK is a tax on the people you want to serve. Compete on runtime, pricing, and distribution.
Keep a written compatibility ledger. Works with apify-client is not a feeling. It is a table of endpoints, statuses, and intentional divergences.
Scope secrets to the job. Run-scoped tokens beat handing the user's full API key to third-party code every time on a multi-tenant runner.
Separate build from rent. Import builds a draft. Publish is a deliberate step. That gap is where pricing, Stripe Connect, and human review belong.
Fleet failures are product bugs. Disk-full runners, bad image pulls, and missing containers are not Actor-quality problems. Retry and admission policy belong in the platform.
Open source the ecosystem seams; own your product stack on open tools. Stand on Crawlee, the Apify SDK shape, and the Crawlee Cloud lineage. Build the marketplace, billing, and day-two operations on tools you control, keep those tools open under a clear license, and let the community read the same foundations you run.
Where this is going next
Developer signups open to everyone on Monday, November 9. If you already publish Actors on Apify and want a second channel that keeps your codebase intact, join the waitlist and import when you are ready.
Join the Bytewells developer waitlist
We are still closing compatibility gaps, hardening the runner fleet, and learning from the first developers who import real Actors. If you have a production Actor whose renters never fit pay-per-event cleanly, I want the sharp edges. Comments welcome.
Top comments (0)