SpEL in Kafka Streams: from evaluator to editor

A Kafka Streams application is two things: a graph, and the logic inside it.
The DSL writes both in one chain — operators like mapValues, filter and
join become the nodes, the chaining becomes the edges, and the lambda inside
each operator is where the work happens:

record Enriched(String customer, String product, double total) { }

builder.stream("orders", Consumed.with(Serdes.Void(), orderSerde))
       .mapValues(value -> new Enriched(
               value.getCustomerId(),
               value.getItem(),
               value.getQuantity() * value.getPriceEur()))
       .filter((key, value) -> value.total() > 50)
       .to("orders-enriched", Produced.with(Serdes.Void(), enrichedSerde));

A graph is a picture, so we draw it: operators dragged onto a canvas and wired
together.

The same four operators as the Java above, drawn rather than chained — one box
each for stream, mapValues, filter and to. The two that need an
expression each carry a red badge, and the step above counts them; the source and
sink already have their topics. The graph is finished and the contents are not.
Watching it built, and what happens after, is the subject of an earlier post.

The body of an operator is not a picture. It is a small computation over one
record — and it is the only part of the whole thing doing anything specific to
your data. That part, and only that part, is what an expression language is
for.

Written as an expression, that body needs no class and no build around it:

{'customer': value.get('customerId'),
 'product': value.get('item'),
 'total': value.get('quantity') * value.get('priceEur')}

The filter's body is one line: value.get('total') > 50.

Writing those bodies in Java is not the hard part. Finding out whether they are
right is the hard part: compile, package, point the app at a broker, produce
a few records, read what came out. Every step is routine, the whole loop is
slow, and it takes exactly as long whether the expression was correct or not.

Evaluated as an expression against a real record while you type, that loop
collapses. It widens the set of people who can experiment with stream
processing, and it makes learning it cheap — the barrier was never the DSL, it
was everything you had to stand up before the DSL would run.

Why SpEL

It is built for this.
SpEL has driven routers and transformers in Spring Integration for years: a language built to take expressions at runtime.

It is already there. In a Spring application it arrives on the classpath with the framework.

It can be contained. A Kafka Streams lambda is compiled by whoever owns the deployment — your classpath, your risk. An expression typed into a browser arrives at runtime and runs in a process the author may not own. So containment comes first: a restricted evaluation context, with no type references, no constructors and no bean resolution. Spring provides that deliberately, which is what makes the whole approach possible — though the honest word is containment rather than safety.

There is a tokenizer to reuse. That expression needs a real editor around
it, and what that editor shows has to agree with what will actually run. The
surest way is to tokenize with the same code the evaluator uses. SpEL has one.
Getting hold of it is where this gets interesting.

Where the knowledge comes from

In the Java above, the knowledge came from types — a class on the classpath, there at build time, whether generated from a schema or written by hand.

An expression editor in a browser has none of it. No compile step, no class to consult — the only things that know anything about the data are the Schema Registry and the records sitting on the topic. Everything the editor offers has to come from those two, while someone is typing.

Drawing the graph is the easy half, and on its own it buys very little: a
canvas with a bare text box for the logic trades away the help the IDE was giving and offers nothing in its place. Separating the two only pays off if the expression half gets something of its own.

It will not be an IDE — but it has what an IDE cannot: the data. An IDE knows priceEur is a number; it does not know this record's is 17. One expression, evaluated against a real record as you type, is closer to a REPL than to autocompletion.

The editor is Monaco, the
component behind VS Code, so it already knows how to highlight tokens, show a completion list and put a marker on a line. What it knows nothing about is SpEL, or your data.

Completions from types, and from content

Completions come from two different places, and the difference is worth keeping visible.

At a source node — reading a topic whose values have a registered schema — the completion list is the schema's own fields with their declared types. That is a contract: something else asserted it, and it holds for every record on the topic.

One operator downstream, after a mapValues, there is no schema any more. The value is whatever the expression returned. The fields offered there are derived by evaluating that expression against a real record and looking at what came out — and the UI marks them (inferred), because that is what they are: an observation of the sample in hand, not a promise. Draw another record and they may differ; if the data is heterogeneous, no single sample will say so.

Inference is not the last word. A node can declare its output in the same
vocabulary a source uses — a serde per side, optionally a registered subject — and a node that declares one becomes a fresh start: its completions are a contract again. What you cannot do is type a list of field names, and that is deliberate. Fields belong to schemas, and a shape worth describing is usually a shape worth registering.

The same dropdown at two different nodes, two kinds of knowledge, labelled differently. It would be easy to present both as "fields", and it would be wrong.

An empty topic is not a dead end. With no record to draw, the sample is
randomly generated from the registered schema instead, and the preview says which one it is holding. That keeps the editor working before any data exists — while being the weaker evidence of the two, since it demonstrates the shape rather than your data.

One operator after the mapValues, the filter offers customer, product and total — none of which exist in any schema. They are there because the expression on the node behind was evaluated against a real record, and the type hint says so. In the source node the same list would have come from the registry instead. The earlier post has this as a
clip.

What an expression receives, and returns

All of that depends on what the deserializer actually hands over — and that is rarely the Java type you would guess from the schema. Avro gives you a GenericRecord, and a field the schema calls a string does not arrive as a String: it arrives as org.apache.avro.util.Utf8, a CharSequence wrapping the raw bytes. Equality still works — value.get('item') == 'Kettle' matches on the orders stream — because SpEL compares CharSequence content. .contains(...) does not, because it is a String method that CharSequence never declared, so it needs a .toString() first. That is the kind of detail that lives in serde Javadoc rather than anywhere you would think to look, and
it costs an afternoon the first time.

Typing value.get('item').contains('Kettle') into a map expression; the editor reports one issue; opening the status shows EL1004E, method contains(java.lang.String) cannot be found on type org.apache.avro.util.Utf8; adding .toString() clears it to no issues.

The same afternoon, spent in under thirty seconds. .contains(...) on an Avro string, the message naming org.apache.avro.util.Utf8 rather than "invalid expression", and .toString() clearing it.

Types are one half of what an editor can offer; the record is the other. Each node's sample preview puts that record in and the evaluated output out, side by side, re-running as the expression changes — so what an expression actually returns is on screen while it is still being written, against the data it will run on.

The Sample preview panel on a map node, showing a real record drawn from the orders topic on the in line. Typing value.get('item') makes the out line read Kettle; replacing the expression with value.get('quantity') * value.get('priceEur') makes the out line read 51. The editor reports no issues throughout and nothing is deployed. in is a real record off orders; out is whatever the expression returns, and it follows the expression as that changes — value.get('item') gives "Kettle", then the intro's own quantity * priceEur gives 51, the value the
> 50 filter downstream exists to judge. Nothing has been deployed, and the editor never claims either answer is correct. It only makes them impossible to miss.

Avro's type names leak into error messages too, which the next section has to deal with.

Errors a user can act on

SpelCompilerMode.IMMEDIATE exists to make expressions faster by compiling them to bytecode. We use it in the validator, as a type checker. Compiling forces type resolution to happen while someone is still typing, which is exactly when you want to hear that a method does not exist on the type you actually have.

Catching it early, at the right line and column, is half of a diagnostic. The other half is what it says.

SpEL's messages assume a developer who knows the JVM types involved. Ours are read by someone who has never heard of GenericData$Record and should not have to. So the validator rewrites them.

An indexing failure — someone reasonably tries value['item'] — becomes:

indexing with ['...'] is not supported on schema-backed records — use
get('fieldName') instead (the completions suggest it). A deployed topology
fails the same way on Avro / Protobuf values.

The fix, a pointer at where it is offered, and a promise about the
runtime. That last clause matters most: it tells the reader this is not an editor quirk to be worked around, because the deployed topology fails
identically.

Everything else gets a blunter rule: any message mentioning an internal sample type has that type replaced with "the sample record". Someone who writes a bad expression should learn what they did, not what class the deserializer happened to return. A few cases are worth naming precisely and get hand-written messages, like "Invalid map access syntax. Use value['key'] without a dot."

The rewriting covers Avro's generic types, Jackson's nodes and — since 0.10.0
— Protobuf's message types, which became reachable the moment get('field')
started evaluating on them.

The tokenizer you cannot reach

An editor needs three things an evaluator does not provide: syntax
highlighting, completions, and errors at the right line and column. All three start from tokens.

Spring has a tokenizer — the one its own parser uses. It is package-private.
Against Spring Framework 7.0.8:

class org.springframework.expression.spel.standard.Tokenizer {
  public org.springframework.expression.spel.standard.Tokenizer(java.lang.String);
  public java.util.List<org.springframework.expression.spel.standard.Token> process();
  static {};
}

The constructor and process() are already public. The class is not, so
from outside the package there is no way to call them.

The public surface of that package is SpelExpressionParser, SpelExpression
and SpelCompiler. Parsing, evaluation, compilation — no tokenization.

SpelExpression does expose a parse tree — getAST() returns SpelNodes with
start and end positions, which looks like exactly the supported route we missed.
But a parser needs a valid expression, and an editor spends most of its life
holding an invalid one. value. is precisely when completions have to fire, and
it does not parse; nor does this post's own map literal before its closing
brace. Both tokenize.

So we copied Tokenizer, Token and TokenKind from
Spring's spel.standard package,
under Apache-2.0, into a package of the same name in our own tree — which is
what puts those package-private types within reach.

All of it exists to enable one line:

List<Token> tokens = new Tokenizer(inputData).process();

What we add on top is thin, and most of it is a coordinate transform. Spring
reports absolute character offsets; editors want line and column. So each
Token becomes a SpELToken(start, end, line, charPositionInLine, type,
modifiers) — the shape Monaco and LSP want — and goes over a WebSocket to the
browser. There they are served through a DocumentSemanticTokensProvider,
which is the highlighting: every colour in the expression comes from the same
tokenizer the evaluator parses with, so what the editor shows and what runs
cannot drift apart.

That is the whole of it. Spring's authors had already written the hard part;
nothing was missing but a way to call it. What we carry is not
complexity — it is their file.

What it costs

SpEL is not Java. Broadening who can write a transformation costs the type
system, the IDE, refactoring, unit tests, and the entire library ecosystem, in
exchange for a string in a text box. That trade is defensible for "what does
this do to my records" and indefensible for a thousand-line pipeline.

The editor cannot catch a wrong answer. It catches expressions that are
invalid — a method that does not exist, a syntax it cannot parse. One that is
type-correct and simply always false passes every check. The preview above will
faithfully show you the wrong answer; nothing will tell you that is what it
is.

Package-private internals carry no compatibility promise. Our copy of the
tokenizer can break between Spring versions with no deprecation cycle, and that
is our problem rather than Spring's. We took the risk knowingly; it is still a
risk.

Protobuf was opaque when this was first published — a DynamicMessage
offers no field access by name — and unlike the rest of this, it was ours to
fix. It shipped in 0.10.0: one SpEL MethodResolver gives
get('field') the same meaning on Protobuf as on Avro and JSON Schema,
registered at all three places an evaluation context is built, so the editor
and the runtime cannot disagree about it.

Is there a better way to do this?

That is our workaround, and we are not claiming it is the right answer.
Package-private is a deliberate choice — internals stay private precisely so
they can change — and we went around it.

So the question is genuine. If there is a supported path to a SpEL token stream
that we missed, we would much rather be told than go on carrying a copy of
somebody else's tokenizer. And if there is not, we are probably not the last
people who will want one.

Either way the pieces are all there: the tokenizer works, the errors carry
positions, the compiler doubles as a type checker. We would just like a
supported way to hold one of them.

Updates

• 2026-09-07 — when first published (2026-08-31), this post said Protobuf values were opaque to the editor and that the fix was on the list. It shipped in 0.10.0 on 2026-09-03: get('field') evaluates on Protobuf in the editor and deployed, and Protobuf's type names joined the message rewriting. The two passages above were corrected to say so.

---

This came out of building Alginte, a visual Kafka
Streams builder — self-hosted, free to run.