DEV Community: Bart van Deenen

Love for Go and Rust rant!

Bart van Deenen — Sat, 13 Dec 2025 17:38:37 +0000

Warning: rant!

I'm so tired of this shit. Every few weeks, some Rustacean/Rustafarian/Rustbucket slithers in here with their smug "but have you tried fearless concurrency?" spiel, acting like Rust is the Second Coming of programming languages that finally fixes everything wrong with the world. Newsflash: it's not. It's a bloated, over-engineered mess masquerading as a panacea, and its concurrency story is one of the biggest hyped-up lies in modern PL discourse.

Go (and yeah, Erlang/BEAM before it) actually got concurrency right. Goroutines are dirt-cheap, channels are built-in and idiomatic, the scheduler is battle-tested, and you can spin up thousands—hell, millions—of concurrent tasks without your brain melting or the compiler throwing a tantrum. It's simple, predictable, and scales to real-world production systems without needing a PhD in lifetime annotations.

Rust? Oh boy. They brag about "fearless concurrency" because the borrow checker catches data races at compile time. Cool story, bro. But guess what? That's only half the battle—and a tiny half at that. Real concurrency bugs aren't just shared mutable state; they're deadlocks, livelocks, starvation, priority inversion, and worst of all: blocking the damn executor.

In Rust's dominant async ecosystem (Tokio, async-std, whatever flavor of the week), everything's cooperative multitasking on a thread pool. One idiot calls a blocking function—.await a future that secretly does std:🧵:sleep, or reads a file synchronously, or hits some crate that blocks under the hood—and BAM, you've starved the entire runtime. Your "massively concurrent" app grinds to a halt on one thread while the rest sit idle. Sound familiar? It's the exact same footgun as Python's GIL or Java's thread-blocking nonsense. Rust doesn't prevent it; it just makes you chase it down manually, wrap it in spawn_blocking, or pray your dependencies are perfectly async-all-the-way-down (spoiler: they're not).

And don't give me that "but Send/Sync traits!" crap. Traits don't magically make third-party libraries non-blocking. One lazy dev (or one outdated crate) and your fearless concurrency is as vulnerable as any GC'd language. Meanwhile in Go, you just go func() and communicate over channels—it's hard to accidentally block the world because the model is designed for real engineers building real systems, not theoretical purity.

Rustaceans act like their language is the cure-all because the compiler yells at you for borrowing wrong. Big whoop—that's like bragging your car has the best seatbelts while ignoring it has no engine. The borrow checker is great for memory safety (congrats, you reinvented GC without the GC), but it comes at the cost of endless fighting the compiler, lifetimes in every signature, and async code that's "colored" and infectious, turning your whole codebase into a pin-projecting nightmare.

Go gets shit done. Fast compile times, single binary deploys, boring readability that onboard new devs in days, and concurrency that's actually enjoyable. Rust? Steep learning cliff, borrow errors that make you question your life choices, and a community of evangelists who think verbosity = virtue.

Rust is fine for what it is: low-level systems crap where you need zero-cost abstractions and no GC pauses. But for servers, backends, networked services—the stuff we do in r/golang? It's overkill wrapped in hype. Stop shilling your cult language here. We have goroutines. We have channels. We have simplicity that scales.

Go is king. Rust is just the loud kid in the back yelling "but my ownership model!"

Downvote me, Rustbucks. I'll be over here shipping code while you're still wrestling the compiler.

Streammachine export to Pandas

Bart van Deenen — Thu, 12 Aug 2021 11:42:53 +0000

This post is a second in a series of posts on how to use the Stream Machine platform. The first post is about getting started.

This time, I'll look at how to get Stream Machine events into a Google Cloud Storage Bucket and from there into a Pandas dataframe. Let's get started.

I assume you have a Google Cloud Bucket somewhere, and you have created a service account.

IAM -> Service Accounts -> Create Service Account

The post assumes you've stored the Service Account credentials in a file credentials.json in the working directory. The file contents are something like this:

{
  "type": "service_account",
  "project_id": "stream-machine-development",
  "private_key_id": "bae39......d6efda",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG......3kS+0=\n-----END PRIVATE KEY-----\n",
  ...
}

Creating a sink and an exporter

In order to store event data in a cloud bucket, you need to

configure the bucket (name, credentials and path) in a so-called sink entity.
create a batch-exporter from a stream to that sink.

strm create sink demo strm-demo --credentials-file=credentials.json
{
  "ref": { "billingId": "strmbart5986941267", "name": "demo" },
  "sinkType": "GCLOUD",
  "bucket": { "bucketName": "strm-demo", "credentials": "..." }
}

# batch exporter for the encrypted stream at path demo-in on the bucket
strm create batch-exporter demo --interval 30 --path-prefix demo-in --sink demo

# batch exporter for the decrypted level 2 stream at path demo-2 on the bucket
strm create batch-exporter demo-2 --interval 30 --sink demo --path-prefix demo-2
{
  "ref": {
    "billingId": "strmbart5986941267",
    "name": "demo-demo-2"
  },
  "streamRef": {
    "billingId": "strmbart5986941267",
    "name": "demo-2"
  },
  "interval": "30s",
  "sinkName": "demo",
  "pathPrefix": "demo-2"
}

Start sending some data

The cli has a simulator on board. Use version 1.4.0 of the cli, otherwise you'll get different simulated events.

strm sim run-random demo --interval 100
Starting to simulate random streammachine/demo/1.0.2 events to stream demo.
Sending one event every 100 ms.
Sent 50 events
Sent 100 events
Sent 150 events
Sent 200 events
...

We should see some data in our bucket:

gsutil ls gs://strm-demo/demo-in | head -5
gs://strm-demo/demo-in/2021-08-12T09:38:00-stream-69c9bbcd-98c2-40e6-b041-fe1af8498752---0-1-2-3-4.jsonl
gs://strm-demo/demo-in/2021-08-12T09:38:30-stream-69c9bbcd-98c2-40e6-b041-fe1af8498752---0-1-2-3-4.jsonl
gs://strm-demo/demo-in/2021-08-12T11:09:00-stream-69c9bbcd-98c2-40e6-b041-fe1af8498752---0-1-2-3-4.jsonl
gs://strm-demo/demo-in/2021-08-12T11:09:30-stream-69c9bbcd-98c2-40e6-b041-fe1af8498752---0-1-2-3-4.jsonl
gs://strm-demo/demo-in/2021-08-12T11:10:00-stream-69c9bbcd-98c2-40e6-b041-fe1af8498752---0-1-2-3-4.jsonl

gsutil ls gs://strm-demo/demo-2 | head -5
gs://strm-demo/demo-2/2021-08-12T09:38:00-stream-4932a907-1972-4982-8018-45a5cd648003---0-1-2-3-4.jsonl
gs://strm-demo/demo-2/2021-08-12T09:38:30-stream-4932a907-1972-4982-8018-45a5cd648003---0-1-2-3-4.jsonl
gs://strm-demo/demo-2/2021-08-12T09:39:00-stream-4932a907-1972-4982-8018-45a5cd648003---0-1-2-3-4.jsonl
gs://strm-demo/demo-2/2021-08-12T11:09:00-stream-4932a907-1972-4982-8018-45a5cd648003---0-1-2-3-4.jsonl
gs://strm-demo/demo-2/2021-08-12T11:09:30-stream-4932a907-1972-4982-8018-45a5cd648003---0-1-2-3-4.jsonl

and in one of the files we can see JSON events, newline based (for readability I've formatted an example JSON event below).

gsutil cat gs://strm-demo/demo-in/.....jsonl | head -1

{
  "strmMeta": {
    "eventContractRef": "streammachine/example/1.3.0",
    "nonce": -1557054268,
    "timestamp": 1628761079144,
    "keyLink": "1d960e7a-4169-4bcf-bece-e8fcc3243c06",
    "billingId": "strmbart5986941267",
    "consentLevels": [ 0, 1 ]
  },
  "uniqueIdentifier": "AQq8Ihq3DBOahkZNXpBfdky8m04pb6c02RIUNOHo",
  "consistentValue": "AQq8IhqopLqWCpIDd1+xZUw/KCtXObL7irK5NbgE1I4=",
  "someSensitiveValue": "AQq8Ihq62mxCv1fqEZ+0bcijMEFZ/VFnpA4EEs8XRp0P",
  "notSensitiveValue": "not-sensitive-30"
}

The demo-2 directory will contain decrypted data.

gsutil cat gs://strm-demo/demo-2/.....jsonl | head -1
{
  "strmMeta": {
    "eventContractRef": "streammachine/example/1.3.0",
    "nonce": -62169113,
    "timestamp": 1628761077936,
    "keyLink": "1f118159-a27a-4468-a540-23a4938bce14",
    "billingId": "strmbart5986941267",
    "consentLevels": [ 0, 1, 2 ]
  },
  "uniqueIdentifier": "unique-25",
  "consistentValue": "session-488",
  "someSensitiveValue": "AXHCR9j0KLyYy7Bivvrk+xfU0D4pRJkIlHAE/PtvtsPx",
  "notSensitiveValue": "not-sensitive-19"
}

To Jupyter and Pandas

The next step is to get these into a Pandas Dataframe in a Jupyter notebook. You need credentials.json that defines the Google Cloud Service Account credentials.

The essence of the Jupyter notebook is:

pip install jupyter gcsfs pandas

import pandas as pd
from pandas import json_normalize
import gcsfs
import json

# set these to your own project and bucket
bucket = "strm-demo"
project = "stream-machine-development"

Load the data

fs = gcsfs.GCSFileSystem(project=project, token="credentials.json")

def one_object_to_df(path):
    with fs.open(path, "r") as _f:
        df = json_normalize([json.loads(l) for l in _f.readlines()])
        df['strmMeta.timestamp'] = pd.to_datetime(df['strmMeta.timestamp'], unit='ms')
        df = df.set_index('strmMeta.timestamp').sort_index()
    return df

def make_df(folder):
    print(folder)
    df = None
    for f in fs.find(f"{bucket}/{folder}"):
        if df is None:
            df = one_object_to_df(f)
        else:
            df = df.append(one_object_to_df(f))
    return df

demo = make_df("demo-in")
demo_2 = make_df("demo-2")

Show some data

demo.consistentValue.value_counts()

AScCSOgnaoVZw2nqRtTSBlwV8pWe5R7SXXJcL1tXC5M=    55
AV3MqegMMQ3Bikr1klqdNN+X+6rykyNtftZizyRKA6U=    52
AQMdVpGbhAScpImKh1I5Lx1FLhb19N97/1reQhZd0ig=    50
AV4eJeE0kNrUw5svSpJTUryc+C4ZZ/zCdZL++VEgY6g=    48
AX2OWCN2zha+odokuX9rTDwOkM47lbNBGnMDPbJZieU=    48
                                                ..
AV8dQPY3mopfROkxlMDeLqcYAxA3qqbYG89J0SSEDio=    19
AVn0ykPCKcX5vtT07DjoV+tcTgUeLmknOytDWzcPAQ==    19
ATKFsJ5+fXeCnsLOc1k+IA+VSYtWR+wT7Iq/4IfwPjA=    19
AXRHjzm9RTN60ocpLNVgMvdW8mSEKKg0/fezhT/We78=    19
ARtzjnb15acL/xVvsS6dslv3M7A8WHUkCmy94LmK5g==    19
Name: consistentValue, Length: 1000, dtype: int64

First steps with Streammachine.io

Bart van Deenen — Wed, 28 Oct 2020 13:56:28 +0000

This post aims to show some first steps with the Stream Machine platform. The target audience is developers, data engineers and possibly data scientists. This post uses Python, but it can similarly be done with Stream Machine's Other language drivers.

About me: I'm Bart van Deenen, lead engineer of Stream Machine, so I should sort of know what I'm talking about :-). This post is meant to be unbiased, and from the perspective of a third-party developer.

Stream Machine

Stream Machine promises to provide Lightning fast, privacy secured, customer data - you can actually use.

So what does this actually mean?

Stream Machine accepts events with a strictly defined serialization schema (currently Apache Avro and Json-Schema are supported). Any valid schema can be used, but needs to be registered with Stream Machine.
Stream Machine handles events that are subject to an event-contract that defines which fields in the event schema (the serialization schema) contain Personally Identifiable Information ( referred to as PII or PII Data).
Stream Machine event contracts may contain customizable validation rules that define field value validity.
Stream Machine processes events with a highly available fault tolerant stream processing system that encrypts all PII field values. The encryption keys are rotated every 24 hours, and this leads to a GDPR compliant stream of event data that can be used by everyone in your company. During the 24 hours, the encrypted values remain static.
Stream Machine events contains consent-level information, and only those events that allow decryption of PII data for certain purposes can be decrypted into decrypted stream(s) that can only be used by those inside your company that are allowed.

This post uses a debugging output of the stream data that uses a websocket. Production level output streams require hooking up our internal Apache Kafka streams. AWS S3 and Google Cloud Storage buckets can be used for batch processing. This will be explored in a next blog post.

The plan

I'm going to build a Python application that mimics users clicking around on a dummy web-shop, that will send a click stream to Stream Machine. I want to retrieve the anonymized data from a Google Cloud bucket, and show them in a Jupyter notebook. I also want to see that only for those simulated users that have given full personalized marketing permissions I retrieve their click stream events. This first post just gets the basics working, i.e. sending events to Stream Machine, and retrieving them.

The steps

An account

I went to streammachine.io to register an account, and after confirming my email, I was shown this page:

Get the CLI

The cli is the most mature way to interact with Stream Machine. Install the most recent version and don't forget to install the shell auto-completion.

Let's create a new stream

strm auth login your-email@wherever
strm create stream demo --save

The --save options causes the credentials to be saved in
~/.config/stream-machine/Stream/demo.json.

Let's send an event!

I'm following along with the Python example in the documentation. I'm going to use the syncsender because I want to play with it in ipython.

The demo example uses the streammachine/demo/1.0.2 schema. You can see its full definition via strm get schema streammachine/demo/1.0.2. The corresponding event-contract that defines validation rules and such is streammachine/example/1.3.0.

For convenience I've copied the saved credentials file (see above) into the working directory.

Keep an eye on the stream

strm egress demo

This starts a websocket connection where you'll see a json serialization of the messages on a stream.

python3 -m venv .venv
. .venv/bin/activate
make
# observe the installed streammachine stuff.
pip freeze | grep streammachine
streammachine-driver==1.0.0
streammachine-schemas-common==1.0.0
streammachine-schemas-demo-avro==1.0.2

Ok, let's start ipython

python3 -m pip install ipython
$> ipython

# import a class that matches the structure of the demo schema
# note that this package name is so long because it's derived from the Stream
# Machine schema ref streammachine/demo and also the `namespace` inside the Avro
# schema (which is io.streammachine.schemas.demo.v1)

from streammachine_io_streammachine_schemas_demo_v1.io.streammachine.schemas.demo.v1 import DemoEvent

from streammachine.driver import StreamMachineEvent, current_time_millis
from streammachine.driver.client.syncsender import SyncSender

event = DemoEvent()

import json
creds = json.load(open("demo.json"))
sender = SyncSender(creds['ref']['billingId'], creds['credentials'][0]['clientId'],
    creds['credentials'][0]['clientSecret'])
sender.start()
sender.wait_ready()
sender.send_event(event)

After the send_event(event) I get the following error:

Error while sending event to Stream Machine (https://in.strm.services/event),
    response status = 400, response: Invalid event contract: . Not supported.

We need to set strmMeta.eventContractRef to a valid contract reference (strm list event-contracts). While a schema defines the shape/structure of an event, an event-contract defines the rules that apply to an event, i.e. what validations, what fields contain personally identifiable data, which field ties the events into a sequence etc... We'll use streammachine/example/1.3.0 (inspect with strm get event-contract ..)

event.strmMeta.eventContractRef="streammachine/example/1.3.0"
sender.send_event(event)
Error while sending event to Stream Machine (https://in.strm.services/event),
    response status = 400, response: Field: 'consistentValue' in event with schema: 'streammachine/demo/1.0.2'
    with value: '' doesn't match regex: '^.+$'

So the field consistentValue needs at least one character?

event.consistentValue="hi there"
sender.send_event(event)
# nothing ....

A whole lot of nothing is returned. Stream Machine is meant for very high throughput and logging anything at thousands of events per second will quickly break the bank if you're using StackDriver. So None and http status 204 are the indicators that everything went fine.

But if we have the strm egress demo running, we would see something like

strm egress demo
{
  "strmMeta": {
    "eventContractRef": "streammachine/example/1.3.0",
    "nonce": -1515353731,
    "timestamp": 1628688372401,
    "keyLink": "cd181172-4ec7-4f0d-86bb-662fc0ee854b",
    "billingId": "strmbart5986941267",
    "consentLevels": []
  },
  "uniqueIdentifier": null,
  "consistentValue": "AWpvnLU8hBPWRfYrAjWPs0wWt6vBMMXXEnSqGTw=",
  "someSensitiveValue": null,
  "notSensitiveValue": null
}

We can observe that the consistentValue field has some data in it (it's actually base64 encoded encrypted "hi there"). Explanation of the various fields in strmMeta are explained in the documentation

Let's fill in some more fields:

import random, uuid

def create_avro_event():
    event = DemoEvent()

    event.strmMeta.eventContractRef = "streammachine/example/1.3.0"
    event.strmMeta.consentLevels = [random.randint(0, 3)]

    event.uniqueIdentifier = str(uuid.uuid4())
    event.someSensitiveValue = "A value that should be encrypted"
    event.consistentValue = "a-user-session"
    event.notSensitiveValue = "Anyone is free to see this text."
    return event

event = create_avro_event()

r = sender.send_event(event)
print(r)
None

In the strm egress demo pane we see:

{
  "strmMeta": {
    "eventContractRef": "streammachine/example/1.3.0",
    "nonce": 1820364498,
    "timestamp": 1628689078226,
    "keyLink": "5074c4de-c51b-4321-a70b-c87db4c79bde",
    "billingId": "strmbart5986941267",
    "consentLevels": [ 0 ]
  },
  "uniqueIdentifier": "AUvu95+NUDFf9krvvVUSU+pJsRBl9XahrMVCTpjqHDa9lTHBTzbRdjazyyMVi3xDy2Ps7HDxJHWA",
  "consistentValue": "AUvu958J4Lf8JWlxwEfdMXXSZpjxdkBSL4hl8Tk5MVHp3L4=",
  "someSensitiveValue": "AUvu95+bq6bw4Z1l9pTYLNQwd/ecdtntrH5mcBJNWv8n6n9jzYxKwEuSDUjig5lPNYqpZpU=",
  "notSensitiveValue": "Anyone is free to see this text."
}

Just looking at it, I can see that all the PII field in the event-contract have been encrypted

Decrypting

We can create a decrypted stream that does contain personal data.

strm create stream --derived-from demo --levels 2 --save

Start sending data in a loop

import time
while True:
    time.sleep(0.1)
    sender.send_event(create_avro_event())

And observe in the egress:

strm egress demo-2
{
  "strmMeta": {
    "eventContractRef": "streammachine/example/1.3.0",
    "nonce": 1165221388,
    "timestamp": 1628689364936,
    "keyLink": "5074c4de-c51b-4321-a70b-c87db4c79bde",
    "billingId": "strmbart5986941267",
    "consentLevels": [
      3
    ]
  },
  "uniqueIdentifier": "279e9dcb-a9a5-497b-8d46-b213106f7fab",
  "consistentValue": "a-user-session",
  "someSensitiveValue": "AUvu95+bq6bw4Z1l9pTYLNQwd/ecdtntrH5mcBJNWv8n6n9jzYxKwEuSDUjig5lPNYqpZpU=",
  "notSensitiveValue": "Anyone is free to see this text."
}

Note that someSensitiveValue is still encrypted (observe the base64 encoding).
This is because

we're asking for a stream with consent-levels up to 2.
the data owner has given consent level 3, but we don't need it, didn't ask for it and so we don't get it. If we'd asked for level 3 this would also have been decrypted.

If we observe the consent levels in the decrypted stream:

strm egress demo-2 | jq -c .strmMeta.consentLevels
[3]
[3]
[2]
[2]

we only see [3] and [2]. Events with consent levels lower than these
don't get exported to this decrypted stream.

Conclusions

We can send data into Stream Machine, and receive them privacy safe and of guaranteed quality.

The next post in this series will show how to get all these data into a Google Cloud bucket, and how to use those data in a Jupyter notebook.