There is a certain embarrassment in discovering that a team of physicists, trying to catch ripples in spacetime, quietly solved a software architecture problem that data engineering teams have been wrestling with for years.
The problem is this: building a real-time streaming pipeline that does not require you to first adopt a distributed system the size of a small country.
The Tax of Modern Streaming
If you want to process a continuous stream of data today, the conventional wisdom points you toward Kafka, Flink, Spark Streaming, or some managed cloud equivalent. These are legitimate tools for legitimate scale problems. But they carry a tax. That tax is operational complexity, infrastructure overhead, and a learning curve that often has nothing to do with the actual problem you are trying to solve.
A lot of real-world streaming problems are not "process ten million events per second across a global fleet." They are "take this incoming sensor feed, run it through three transformations and a filter, and tell me something useful with minimal latency." For that class of problem, standing up a Kafka cluster is roughly equivalent to hiring a construction crew to hang a picture frame.
What you actually want is a clean abstraction for wiring computation together, running it against a live data source, and getting out of your own way.
A DAG in an Event Loop
That is essentially what SGN (Stream Graph Navigator) is. It is a zero-dependency Python library that lets you define computational components and connect them into a directed acyclic graph. That graph then runs inside a Python event loop, processing data as it arrives.
This is not a new idea. DAGs are everywhere in data tooling. But the execution of this one is unusually clean. There is no broker to configure, no cluster to spin up, no serialization format to negotiate. You define nodes, you wire them together, and you run.
A rough sketch of how that looks in practice:
from sgn import Node, Pipeline
class Bandpass(Node):
def process(self, sample):
# apply filter, return result
return filtered_sample
class Threshold(Node):
def process(self, sample):
if sample > self.limit:
self.emit(sample)
pipeline = Pipeline()
pipeline.connect(Bandpass(), Threshold(limit=0.5))
pipeline.run(source=my_data_stream)
The shape of this will be immediately familiar to anyone who has spent time with frameworks like Luigi or Prefect, but the execution model is different. This is not batch orchestration. This is a live graph processing samples as they arrive.
Why the Origin Matters
SGN was not built as a framework-first project. It was built because a team doing real-time gravitational-wave detection at LIGO needed it. That context is worth sitting with for a moment.
Gravitational-wave detection is about as demanding a latency-sensitive workload as exists outside of high-frequency trading. You are looking for extraordinarily faint signals buried in noise, in real time, with enough speed that astronomers can point telescopes at an event while it is still happening. The error budget for "slow and clunky" is essentially zero.
The fact that SGN was built under those constraints, and then matured into SGN-TS for time-series signal processing and SGNL for full matched-filtering search pipelines, means this is not a weekend project that happens to have a README. It is production infrastructure for one of the most demanding physics experiments on the planet, now being adopted across multiple low-latency gravitational-wave analysis projects.
That provenance matters because it tells you something about what the design was optimized for. This is not a framework that prioritized developer experience and hoped performance would follow. It prioritized correctness and latency under real conditions, and ended up being clean as a side effect.
What Data Engineers Can Steal From This
The lesson here is not "abandon Kafka and use a physics library instead." The lesson is narrower and more useful.
When you are designing a streaming system, the first question worth asking is whether the problem is actually a distributed systems problem or a computation graph problem. A lot of pipelines that end up deployed as distributed infrastructure would be better served by a clean local graph runner with well-defined node interfaces.
The overhead of distributed streaming infrastructure is not just operational. It shapes how you think about your pipeline. When everything runs through a broker, you start designing around message passing and consumer groups and offset management, even when none of that complexity is serving your actual requirements.
SGN represents an alternative starting point. Define your computation as a graph. Run it in-process. Add distribution only when the workload actually demands it.
The physicists figured this out because they had to. They could not afford to let infrastructure complexity eat into their latency budget. Most engineering teams can afford that waste, and so they absorb it without noticing.
The Concrete Takeaway
Before your next streaming project defaults to the full distributed stack, spend an afternoon with a graph-based in-process runner. You may find that the problem you thought required a cluster requires nothing more than a well-wired DAG and a tight event loop. If it was good enough to catch gravitational waves, it is probably good enough for your telemetry pipeline.
Top comments (0)