Somebody owns the Kafka cluster you need, and it is not you. You have a topology in mind: read orders, keep the big ones, write them somewhere. You have a predicate in mind too, and you are fairly sure it is right. "Fairly sure" is the problem. To find out, you need to run it, and the only place to run it is the shared cluster: a consumer group in everyone's list, internal topics with your name on them, and an output topic that real consumers may already be reading. Kafka does not take records back. A wrong predicate there means wrong records that somebody downstream has already acted on.
So you do not run it. The idea is shelved, and you never learn that it was the right one.
This post is about the other order: find out first, in a place where being wrong costs nothing, and only then touch the cluster. It needs two things. A stack of your own that you are free to break, and a way to carry what you learned across the border without carrying the stack with it.
The expensive part is not the topology
Writing a Kafka Streams topology is not where the time goes. The time goes into the question the code cannot answer for itself: does the predicate do what you meant, on the records that actually arrive? We made that argument at length in The compiler was never what you wanted: the cost is in the loop between having an idea and finding out, and on a shared cluster that loop is long, so you run it rarely.
The loop is short when three things are true. The records you evaluate against are real. The evaluation you see before deploying is the one that will run after. And being wrong leaves nothing behind. The earlier post was about the first two. This one is about the third, and about what happens after you are right.
A stack of your own, free to break
One docker compose up -d gives you a single-node Kafka, a Schema Registry, and a console that runs Kafka Streams topologies, bound to your machine, with no volumes. It comes seeded with topics, real Avro records against a registered subject, and a topology already running, so the first thing you see is a canvas with numbers on it. docker compose down returns the machine to the state it was in. That is the whole safety story. Nothing here can hurt anyone, because nothing here is shared.
Seeded orders are fine for the first ten minutes. After that you want your own topic. Say it is shipments, with JSON Schema values. Two folders beside the compose file take it in, and because they are read at every up, what you put there survives the down that erases everything else:
playground/
docker-compose.yml
schemas/
shipments-value.json the subject, saved out of your registry
records/
shipments.json-schema.jsonl a sample of the topic, one record per line
The file name is the contract. shipments-value.json is registered as the subject shipments-value; .avsc and .proto do the same for Avro and Protobuf. shipments.json-schema.jsonl is produced into the topicshipments, each value framed against that subject, and a line is either a keyed record or a bare one:
s-1|{"shipmentId":"s-1","orderId":"ord-1001","carrier":"DHL","weightKg":2.4,"destination":"Zagreb"}
s-2|{"shipmentId":"s-2","orderId":"ord-1003","carrier":"UPS","weightKg":11.0,"destination":"Milano"}
Keep the topic and subject names identical to the real ones, because the design you are about to make carries them. A small sample is enough, if it is the right one: the ordinary record, and then the ones that break things. A null where a field is optional, an empty list, a value sitting on the boundary of your predicate, a key you did not expect. What you are proving is logic against the shape of your data, and shape is a matter of variety, not volume.
Production may not be able to give you that sample. The ordinary record is there a million times over; the one that breaks your predicate may not have happened yet, and a shared topic is no place to produce it to find out, with everyone else's consumers downstream. Here it is one more line in the file: written by hand, framed against the same subject as the sampled ones, seen by nobody's consumer but yours.
Anything personal in the sampled records is yours to redact before it lands in the folder. The playground's repository has these two files as templates to copy.
Now the loop. Drop a source node on shipments, bound to shipments-value. Add a filter, and start typing the predicate. As you type, it is evaluated against the newest record on the topic, and the result sits next to the editor: the input record, the output, the verdict. Put a mapValues in front of it and the filter sees the shape themapValues produced, not a guess. Misspell a field and the editor tells you on the keystroke. On an Avro or Protobuf record the access fails, with the same message the deployed operator would throw, because it is the same evaluation. On a JSON Schema record, where a missing key is a quietnull at runtime, the editor warns on the name and offers the closest one the subject declares. This is the short loop: the few seconds between the typo and finding it.
Fix the name, finish the predicate, and the same panel judges it against a record you brought:
Submit, and a gate runs before anything deploys: a source topic that does not exist, a source whose subject the registry does not have, join inputs that are not co-partitioned, a windowed key on its way to a sink that cannot write it. Each refusal names the node and the fix. Pass the gate, and the stream runs here, on this broker. Produce a few records, written by hand or generated at random from the subject's schema, one click each. Watch the sink fill, open the State tab and see the threads, the stores and the lag. Delete it, keep the state or remove it. Deploy it again under the same id, and the console tells you what the earlier run left on the cluster, a changelog and a committed offset, and asks whether to continue on it or start clean.
None of this is a simulation. It is the Kafka Streams client, running your topology, on a broker you can throw away. The console in the loop is Alginte, in the mode it callsthe playground, and the one claim that makes the loop worth trusting is the one we test hardest: an expression that evaluates one way in the editor evaluates the same way deployed, on Avro, JSON Schema and Protobuf alike. That promise is held by a corpus of rows that run each expression on both surfaces and refuse to ship a disagreement.
Carrying it across the border
A design that works here has to reach the cluster that matters, and it must not bring the sandbox with it.
The design travels as a file. Download writes the whole thing: the configuration, every node with its expressions and serdes, the edges, the layout. The file is named after the application.id, and it belongs in git, beside the code it feeds. It is plain, indented JSON, so a changed expression is a one-line diff, a new node is a block a reviewer can read in a pull request, and the history of the stream is the history of the file:
"data": {
"name": "light DHL parcels",
"expression": {
- "expression": "value.get('carrier') == 'DHL' && value.get('weightKg') < 5",
+ "expression": "value.get('carrier') == 'DHL' && value.get('weightKg') < 10",
"language": "SpEL"
}
That is what makes it the durable artefact: the sandbox is thrown away, the file is not.
What the file does not carry is as important: not the records, not the topic, not the schema, not the stream's state. Those are the cluster's, on both sides.
On the console that runs against the real cluster, Upload the file. The same wizard opens with the same validation. Three things are different at the border, and each is handled by a rule rather than by remembering.
A design carries its logic, not its cluster. If the file hasbootstrap.servers, a registry URL or security settings in it, the upload leaves them out and says which, so the connection that applies is this console's, never the sandbox's.
Names are checked against this cluster. The gate refuses a source topic that does not exist here, and a source whose subject this registry lacks, naming the node. Register the subject first, with the same text you put inschemas/, and the refusal goes away.
The id is checked against this cluster's memory. If a stream under thisapplication.id has run here before, the console says so before deploying: these internal topics, this committed offset. Continue on them, or start fresh. It never guesses.
Then Submit. The deploy is the same gate and the same client. The only thing that changed is the cluster it joined.
That is one of two routes out of the sandbox, and neither is the lesser one. The other: your team ships its streams as Java services, with a compiler and a test suite, and what you carry across is the shape, the expressions and the evidence that they are right. Either way you arrive having already answered the question the compiler cannot.
What the sandbox does not prove
Scale. A design proven on a small, well-chosen sample is proven for what it does to a record, not for its throughput, its partition layout under load, or how long its stores take to restore after a restart. Those are questions for the real cluster, and there they have answers. In the sandbox, the State tab showed a handful of records moving. On the real cluster, the same tab shows what matters at scale: a restore as it replays, the threads and the tasks they hold, the lag of every store, and on the canvas the throughput of every node. It is the same console on both sides of the border: the workbench here, the instrument there. The sandbox tells you the logic is right. The first hour on the real cluster tells you the rest, and it is an hour you can watch rather than infer.
The order of operations
Start where being wrong is free. Bring the shape of your data, and a sample that covers its corners. Write the expression against a real record and let the editor tell you on the keystroke. Deploy on a broker you can throw away, and throw it away. Then, if you choose to run it as it is, carry the design across to the production cluster as a file, and let the gate on the other side check the names, the id and the connection. Deploy there once, on purpose.
The production cluster is still somebody else's. The difference is that your stream now arrives there with its logic already proven, and "fairly sure" left behind in the sandbox.
Top comments (0)