DEV Community: Nithin Bharadwaj

How to Build a Fault-Tolerant Stream Processing System in Go With Exactly-Once Guarantees

Nithin Bharadwaj — Sat, 18 Apr 2026 11:32:12 +0000

Building a system that handles a continuous flow of data, like clicks on a website or sensor readings, is a common challenge. You need to group this data into time windows, calculate running totals or averages, and guarantee that every piece of data is processed once, and only once, even if parts of the system fail. Let me walk you through how to build such a system in Go, piece by piece.

Think of a data stream as a never-ending conveyor belt. Items arrive at different times, sometimes out of order. Our job is to sort them, put them into time-based buckets (like "last 5 minutes"), do math on each bucket, and send the results onward, all while keeping perfect track of what we've already done.

The heart of our system is the StreamProcessor. It coordinates everything. When you create one, you give it some settings: how big its internal channels should be, how long to wait for late data, and where to save its progress.

config := ProcessorConfig{
    BufferSize:        10000,
    WatermarkDelay:    5 * time.Second,
    CheckpointInterval: 30 * time.Second,
}
processor := NewStreamProcessor(config)

Events flow into the system through the EventIngestor. This is the front door. Its first job is to deal with a messy reality: events don't always arrive in the order they were created. A network delay might hold up an earlier event, while a later one zips through.

To handle this, we use a concept called a watermark. It's like a moving finish line. We watch the timestamps on incoming events. The watermark is set to the highest timestamp we've seen, minus a waiting period (like 5 seconds). This waiting period is our allowance for late data. We promise not to finalize any calculations for a time period until we're reasonably sure no more data from that period will arrive. Events with a timestamp older than the current watermark are safe to process. Newer ones are held in a buffer.

func (wm *WatermarkManager) Update(eventTime int64) {
    wm.mu.Lock()
    defer wm.mu.Unlock()
    // ... find the maximum timestamp seen ...
    wm.currentWatermark = maxTs - int64(wm.maxDelay/time.Millisecond)
}

The WindowManager is where the core logic lives. A window is just a time range. A "tumbling window" might be every minute, non-overlapping. A "sliding window" might be a five-minute average, updated every minute. When an event is deemed ready (its timestamp is past the watermark), the manager figures out which windows it belongs to.

For each relevant window, we need an Aggregator. This is a simple object that knows how to update a calculation. If we're counting events, the aggregator holds a number and adds one. If we're summing a value, it holds a running total.

type CountAggregator struct {
    count int64
}

func (ca *CountAggregator) Aggregate(event StreamEvent) (interface{}, error) {
    atomic.AddInt64(&ca.count, 1)
    return ca.count, nil
}

The magic of "exactly-once" processing hinges on two things: remembering what you've done and being able to restart from a known good point. This is the job of the StateStore and the Checkpointer.

Every event has a unique ID. Before we even look at an event's data, we ask the StateStore: "Have I seen this ID before?" The store checks a fast cache and, if needed, a persistent database. If the answer is yes, we skip the event. This handles duplicates that might be sent if a producer retries.

func (sp *StreamProcessor) processEvents(ctx context.Context) error {
    for {
        select {
        case event := <-sp.ingestor.orderedChan:
            // The critical check
            if processed := sp.stateStore.IsProcessed(event.ID); processed {
                continue // Skip duplicate
            }
            // ... process the event ...
            // Mark it as done
            sp.stateStore.MarkProcessed(event.ID, event.Timestamp)
        }
    }
}

But what if our entire program crashes? We don't want to lose the results of all the windows we've been calculating. This is where checkpoints come in. On a regular schedule (e.g., every 30 seconds), the Checkpointer tells every component to pause and take a snapshot.

It captures everything: the state of every aggregator in every window, the current watermark, and the list of processed event IDs. It writes this snapshot durably—to disk or a database. This snapshot is a checkpoint.

func (c *Checkpointer) createCheckpoint(store *StateStore) {
    checkpoint := Checkpoint{
        ID:        ksuid.New().String(), // A unique ID
        Timestamp: time.Now().UnixNano(),
        State:     store.Snapshot(), // Grab all state
    }
    store.checkpoints[checkpoint.ID] = checkpoint
    c.commitCheckpoint(store, checkpoint)
}

When our processor starts up, its first step is to look for the latest successful checkpoint. If it finds one, it loads all the saved state. The aggregators are restored with their previous counts, the watermark is reset, and the list of processed IDs is reloaded. Then, it starts reading the event stream from the point after that checkpoint. It will re-process some events, but the duplicate detection in the StateStore will filter them out. The result is that the system's output is exactly as if it had run continuously without failing.

Let's look at what happens to a single event, step-by-step.

Arrival: An event {ID: "abc123", Timestamp: 1625097600000, Data: {value: 42}} is sent to processor.Process().
Buffering: The EventIngestor receives it, updates the watermark, and puts it in a buffer if its timestamp is too new.
Ordering: Once the watermark advances past the event's timestamp, it's pushed into the ordered channel.
Deduplication: The main loop receives it and asks the StateStore if "abc123" was processed. It hasn't been, so we continue.
Window Assignment: The WindowManager checks all active windows. Does a 1-minute tumbling window for [1625097600000, 1625097660000) exist? It creates one if needed.
Aggregation: The manager finds the SumAggregator for that window and calls agg.Aggregate(event). The aggregator adds 42 to its internal total.
State Save: The new state of the aggregator (the new sum) is saved to the StateStore with a key like "window_sum_1min:1625097600000".
Completion: The event's ID "abc123" is marked as processed in the state store.
Emission: If this event caused the window to be complete (e.g., the watermark passed the window's end time), the window triggers. It takes the final result from the aggregator and sends it out as output.

All of this happens concurrently. The ingestor, the watermark updater, the window manager, and the checkpointer all run in their own goroutines, communicating through channels. This is where Go's concurrency model shines. We use mutexes (sync.RWMutex) sparingly, only to protect specific maps or slices when multiple goroutines might access them.

A real implementation needs a solid backend for the StateStore. In-memory maps work for testing, but for production, you need persistence. You could use embedded databases like RocksDB or Badger. They store data locally on disk and are very fast for key-value lookups, which is most of what we do.

// A simplified RocksDB backend
type RocksDBBackend struct {
    db *gorocksdb.DB
}

func (rb *RocksDBBackend) Put(key string, value []byte) error {
    wb := gorocksdb.NewWriteBatch()
    defer wb.Destroy()
    wb.Put([]byte(key), value)
    return rb.db.Write(defaultWriteOptions, wb)
}

Performance tuning is crucial. The size of the input channel buffer affects backpressure. If it fills up, producers will block. The watermark delay is a trade-off: a longer delay means better handling of late events, but it also increases the latency of your results. The checkpoint interval is a trade-off between recovery time (shorter interval) and performance overhead (longer interval).

You'll want to add metrics to every part. How many events per second are you receiving? What's the average processing latency? How far behind the current time is your watermark? This data is vital for operating the system in production.

type StreamMetrics struct {
    EventsReceived      uint64
    EventsProcessed     uint64
    DuplicateEvents     uint64
    WatermarkLag        int64 // in milliseconds
}

In a distributed setup, you would run multiple instances of this processor. You'd need a way to partition your event stream, perhaps by a key in the data. Each instance would be responsible for a subset of the keys. The checkpointing becomes more complex, requiring a distributed coordination system like Apache ZooKeeper or etcd to ensure all instances snapshot their state consistently.

Building this might seem daunting, but by breaking it down into these components—ingestion with watermarks, windowed aggregation, stateful deduplication, and periodic checkpointing—you create a system that is both powerful and understandable. It handles the inherent disorder of real-world data and provides the strong guarantees necessary for accurate, reliable results. You start with events on a conveyor belt and end with timely, correct insights, no matter what happens along the way.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

101 Books is an AI-driven publishing company co-founded by author Aarav Joshi. By leveraging advanced AI technology, we keep our publishing costs incredibly low—some books are priced as low as $4—making quality knowledge accessible to everyone.

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

How JavaScript Signals and Fine-Grained Reactivity Make Your Apps Faster

Nithin Bharadwaj — Sat, 18 Apr 2026 11:18:44 +0000

Let’s talk about state in JavaScript applications. If you’ve ever built something interactive, you know the challenge: when a piece of data changes, you need to update the right parts of your interface, and you need to do it efficiently. For a long time, this meant a lot of manual work or using systems that often re-rendered more than necessary, which can slow things down.

There’s a different way to think about this problem. Imagine if every individual piece of your application’s state could announce its own changes, and only the parts of your app that specifically care about that piece would listen and react. Nothing else would be disturbed. This is the core idea behind signals and fine-grained reactivity. It’s a shift from telling the entire UI to “check for changes” to having a direct, one-to-one conversation between data and its consumers.

I want to walk you through how this works, not just in theory but in practice. We’ll build up the concepts from the ground up, with code you can use and adapt. Think of it as learning a new set of tools that make your applications faster and your code simpler to reason about.

At the heart of this system is the signal. A signal is a container for a value with a simple superpower: it knows what other parts of the code are interested in its current value. When that value changes, it can notify just those interested parties.

Here is a basic way to create one. We start with a class that holds a value and keeps track of subscribers.

class SignalCore {
  constructor(value) {
    this._value = value;
    this._subscribers = new Set();
  }

  get value() {
    // Dependency tracking happens here
    return this._value;
  }

  set value(newValue) {
    if (Object.is(this._value, newValue)) return;
    this._value = newValue;
    this._notifySubscribers();
  }

  _notifySubscribers() {
    this._subscribers.forEach(subscriber => subscriber());
  }

  subscribe(fn) {
    this._subscribers.add(fn);
    return () => this._subscribers.delete(fn);
  }
}

This is the foundation. You create a signal with an initial value. When you get its .value, you’re reading it. When you set its .value, it compares the new value with the old one. If they’re different, it notifies everyone in its subscriber list by calling their functions.

This alone is useful, but the real magic starts when we add dependency tracking. We need a way for the signal to know who is reading it at any given moment, so it can add that reader to its subscriber list automatically. We do this by setting a global pointer to the currently running “computation” or effect.

Let’s modify our getter to use this system.

get value() {
  // If something is currently 'listening', track this signal as a dependency
  if (SignalCore._activeComputation) {
    this._subscribers.add(SignalCore._activeComputation);
  }
  return this._value;
}

We need a place to store that active computation. We can use a static property on the class.

SignalCore._activeComputation = null;

Now we can create our first reactive primitive: an effect. An effect is a function that runs and automatically re-runs whenever any signal it reads inside changes.

function createEffect(effectFn) {
  const execute = () => {
    // Set this effect as the active computation
    SignalCore._activeComputation = execute;
    try {
      effectFn();
    } finally {
      // Clean up after running
      SignalCore._activeComputation = null;
    }
  };
  // Run it once to set up initial dependencies
  execute();
}

Let’s see it in action with two signals.

const firstName = new SignalCore('John');
const lastName = new SignalCore('Doe');

createEffect(() => {
  console.log(`Hello, ${firstName.value} ${lastName.value}!`);
});
// Logs: Hello, John Doe!

firstName.value = 'Jane';
// Automatically logs: Hello, Jane Doe!

The effect ran once. It read both signals, so it became a subscriber to both. When firstName changed, the effect’s function was called again. This is fine-grained reactivity. The effect didn’t need to know what changed; the signals told it.

Our next technique is the computed signal. This is a signal whose value is derived from other signals. It should only re-calculate when one of its dependencies changes.

To build this, we need a more robust tracking system. Our SignalCore needs to track not just subscriber functions, but also which computed signals depend on it. Let’s update the class.

class SignalCore {
  constructor(value) {
    this._value = value;
    this._subscribers = new Set();
    this._dependencies = new Set(); // For computed signals that depend on this
  }

  get value() {
    if (SignalCore._activeComputation) {
      this._dependencies.add(SignalCore._activeComputation);
      SignalCore._activeComputation._dependencies.add(this);
    }
    return this._value;
  }

  set value(newValue) {
    if (Object.is(this._value, newValue)) return;
    this._value = newValue;
    this._notify();
  }

  _notify() {
    // Notify direct subscriber functions
    this._subscribers.forEach(sub => sub());
    // Notify any dependent computed signals
    this._dependencies.forEach(dep => dep._update());
  }

  subscribe(fn) {
    this._subscribers.add(fn);
    return () => this._subscribers.delete(fn);
  }
}

Now we can create a ComputedSignal class that extends this core. It won’t have a setter. Its value comes from a computation function.

class ComputedSignal extends SignalCore {
  constructor(computeFn) {
    super(undefined); // Start with no value
    this._computeFn = computeFn;
    this._isDirty = true; // Needs first calculation
    this._update(); // Calculate initial value
  }

  get value() {
    if (this._isDirty) {
      this._update();
    }
    // Use parent's getter to track dependencies of this computed signal
    return super.value;
  }

  _update() {
    // Clear old dependencies
    this._dependencies.forEach(dep => dep._dependencies.delete(this));
    this._dependencies.clear();

    // Run computation with this as the active context
    SignalCore._activeComputation = this;
    try {
      const newValue = this._computeFn();
      if (!Object.is(this._value, newValue)) {
        this._value = newValue;
        this._isDirty = false;
        // Notify anyone depending on this computed signal
        this._notify();
      }
    } finally {
      SignalCore._activeComputation = null;
    }
  }

  // Called when a dependency notifies us
  _markDirty() {
    if (!this._isDirty) {
      this._isDirty = true;
      // We don't recalculate yet, just notify our dependents
      this._notify();
    }
  }
}

Let’s create some helper functions to make this easier to use.

function createSignal(value) {
  return new SignalCore(value);
}

function createComputed(fn) {
  return new ComputedSignal(fn);
}

function createEffect(fn) {
  const effect = {
    _execute() {
      SignalCore._activeComputation = effect;
      try {
        fn();
      } finally {
        SignalCore._activeComputation = null;
      }
    }
  };
  effect._dependencies = new Set();
  effect._execute();
  // For simplicity, we don't handle cleanup here yet
}

Now, here’s a practical example. Imagine a shopping cart.

const itemCount = createSignal(2);
const itemPrice = createSignal(25.99);

const subtotal = createComputed(() => itemCount.value * itemPrice.value);
const tax = createComputed(() => subtotal.value * 0.08);
const total = createComputed(() => subtotal.value + tax.value);

createEffect(() => {
  console.log(`Total to pay: $${total.value.toFixed(2)}`);
});
// Logs: Total to pay: $56.14

itemCount.value = 3;
// Automatically re-computes and logs: Total to pay: $84.21

Only the necessary computations happen. When itemCount changes, subtotal, tax, and total are recalculated in the right order. Our effect runs once at the end.

A common problem in UI updates is something called a "glitch." This happens when you read a computed signal that is in a temporarily inconsistent state because its dependencies are mid-update. Our current setup might have this issue if we update multiple signals at once.

We can solve this with a fourth technique: batching updates. The goal is to postpone all notifications until a batch of changes is complete, then notify once with all final values.

We’ll create a simple batch manager.

const batchStack = [];

function batch(callback) {
  batchStack.push(new Set());
  try {
    callback();
  } finally {
    const updates = batchStack.pop();
    // If we're not in a nested batch, flush updates
    if (batchStack.length === 0) {
      updates.forEach(signal => signal._notify());
    } else {
      // Otherwise, add to parent batch
      updates.forEach(signal => batchStack[batchStack.length - 1].add(signal));
    }
  }
}

Our signals need to cooperate. When a signal’s value is set inside a batch, it should schedule its notification instead of doing it immediately.

class SignalCore {
  // ... previous code ...

  set value(newValue) {
    if (Object.is(this._value, newValue)) return;
    this._value = newValue;

    if (batchStack.length > 0) {
      // Schedule for later
      batchStack[batchStack.length - 1].add(this);
    } else {
      this._notify();
    }
  }
}

Now you can group changes.

batch(() => {
  itemCount.value = 10;
  itemPrice.value = 19.99;
});
// The effect runs only once, after both updates.

This prevents intermediate states from causing unnecessary work or visual flicker.

The fifth technique is about structure. In a real app, you don’t have just a few loose signals. You have a state tree. We can create a store that organizes signals and provides methods to update them predictably.

Here’s a simple store pattern.

class SignalStore {
  constructor(initialState) {
    this._state = {};
    this._signals = {};

    Object.keys(initialState).forEach(key => {
      this._signals[key] = createSignal(initialState[key]);
      // Define a getter for easy access
      Object.defineProperty(this._state, key, {
        get: () => this._signals[key].value,
        enumerable: true
      });
    });
  }

  get(key) {
    return this._state[key];
  }

  getSignal(key) {
    return this._signals[key];
  }

  set(key, value) {
    if (this._signals[key]) {
      this._signals[key].value = value;
    }
  }

  batchUpdate(updater) {
    batch(() => updater(this));
  }
}

Use it like this.

const userStore = new SignalStore({
  name: 'Alice',
  age: 30,
  preferences: { theme: 'light' }
});

const userName = userStore.getSignal('name');
createEffect(() => {
  document.title = `Profile: ${userName.value}`;
});

userStore.batchUpdate((store) => {
  store.set('name', 'Bob');
  store.set('age', 31);
});

This gives you a central place for state that is still built from fine-grained signals.

The sixth technique integrates this with the DOM. We want our UI to update when signals change. We can create a small library to bind signals to DOM elements.

class ReactiveDOM {
  bindText(element, signal) {
    const update = () => element.textContent = signal.value;
    signal.subscribe(update);
    update(); // Set initial value
  }

  bindAttribute(element, attrName, signal) {
    const update = () => element.setAttribute(attrName, signal.value);
    signal.subscribe(update);
    update();
  }

  createReactiveElement(tag, props, ...children) {
    const el = document.createElement(tag);
    for (const [key, value] of Object.entries(props || {})) {
      if (value && typeof value === 'object' && 'value' in value) {
        // It's a signal
        this.bindAttribute(el, key, value);
      } else {
        el.setAttribute(key, value);
      }
    }
    children.forEach(child => {
      if (typeof child === 'string') {
        el.appendChild(document.createTextNode(child));
      } else if (child && typeof child === 'object' && 'value' in child) {
        // It's a signal for text content
        this.bindText(el, child);
      } else if (child instanceof HTMLElement) {
        el.appendChild(child);
      }
    });
    return el;
  }
}

Let’s build a counter UI.

const dom = new ReactiveDOM();
const count = createSignal(0);

const button = dom.createReactiveElement('button', {
  onclick: () => count.value++
}, 'Count: ', count);

document.body.appendChild(button);

Every time you click, the count signal updates, and the button’s text updates automatically. Only that specific text node is touched by the DOM.

The seventh technique deals with a common need: managing side effects and cleanup. Our simple createEffect needs improvement. When a dependency changes, we should clean up the old effect before running the new one, and we should clean up when the effect is no longer needed.

Let’s build a more robust effect manager.

function createEffect(effectFn) {
  let cleanupFn;
  let isDisposed = false;

  const execute = () => {
    // Clean up previous effect
    if (cleanupFn) {
      cleanupFn();
      cleanupFn = null;
    }

    // Set as active computation
    SignalCore._activeComputation = execute;
    try {
      // The effect can return a cleanup function
      cleanupFn = effectFn() || null;
    } finally {
      SignalCore._activeComputation = null;
    }
  };

  // Run once to start
  execute();

  // Return a dispose function
  return () => {
    if (isDisposed) return;
    isDisposed = true;
    if (cleanupFn) cleanupFn();
    // Remove this effect from all its dependencies
    execute._dependencies.forEach(signal => {
      signal._dependencies.delete(execute);
      signal._subscribers.delete(execute);
    });
  };
}

This pattern is useful for setting up and tearing down event listeners, subscriptions, or intervals.

const timer = createSignal(0);

const disposeEffect = createEffect(() => {
  const interval = setInterval(() => {
    timer.value++;
  }, 1000);

  // Cleanup function
  return () => clearInterval(interval);
});

// Later, to stop everything
// disposeEffect();

The eighth technique is about scalability and developer experience. As your app grows, you’ll want tools for debugging and inspection. We can add a simple devtools hook.

We can instrument our SignalCore to log changes.

class SignalCore {
  constructor(value, name) {
    this._value = value;
    this._subscribers = new Set();
    this._dependencies = new Set();
    this._name = name; // For debugging

    if (typeof window !== 'undefined' && window.__SIGNAL_DEVTOOLS__) {
      window.__SIGNAL_DEVTOOLS__.registerSignal(this);
    }
  }

  set value(newValue) {
    const oldValue = this._value;
    if (Object.is(oldValue, newValue)) return;
    this._value = newValue;

    // Notify devtools
    if (window.__SIGNAL_DEVTOOLS__) {
      window.__SIGNAL_DEVTOOLS__.logUpdate(this, oldValue, newValue);
    }

    // ... rest of setter logic ...
  }
}

You could then implement a simple devtools panel that listens to this global hook and displays a log of signal changes, helping you trace the flow of data.

Bringing it all together, these eight techniques—basic signals, dependency tracking, effects, computed values, batching, structured stores, DOM binding, and managed effects—form a complete foundation for fine-grained reactivity.

This approach might seem different at first, but it directly addresses the performance and complexity issues of older models. Instead of diffing a large virtual tree to guess what changed, you know exactly what changed because the signals tell you.

The code examples here are building blocks. You can start small, with just a few signals and effects, and gradually incorporate more patterns as your application demands. The key insight is that by making state itself observable and reactive, you shift the burden of update coordination from the developer to the system, leading to code that is often simpler and applications that are consistently faster.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

How to Use Python to Control Real-World Hardware: Sensors, Motors, and IoT Devices

Nithin Bharadwaj — Sat, 18 Apr 2026 11:03:43 +0000

Python has become my secret weapon for making computers interact with the real world. It started for me with a simple goal: make a light blink. What seemed like a basic task opened a door to controlling robots, reading environmental data, and building devices that respond to physical touch. If you've ever wondered how software can affect hardware—like telling a motor to spin or reading a temperature sensor—Python is one of the most approachable ways to start.

I think of Python as a translator. My computer speaks in high-level, abstract code, but hardware components like sensors and motors speak in electrical signals—voltages, currents, and precise timing. Python, with its vast collection of libraries, acts as the middle layer that converts my simple commands into the low-level signals the hardware understands. This lets me focus on what I want the system to do, rather than getting lost in complex electronic details.

Let's begin with the most fundamental concept: talking directly to the pins on a board. Many small computers, like the Raspberry Pi, have a set of physical pins called GPIO, or General Purpose Input/Output pins. Each pin is like a tiny switch or a sensor port I can control from my code. I can set a pin to be an output to send power (to light an LED) or an input to listen for a signal (like a button being pressed).

Here is a practical class I often use as a foundation. It wraps the basic operations so I can manage multiple pins without repeating setup code.

import RPi.GPIO as GPIO
import time
from typing import List, Optional

class GPIOManager:
    def __init__(self, mode=GPIO.BCM):
        GPIO.setmode(mode)
        GPIO.setwarnings(False)
        self.pin_modes = {}

    def setup_pin(self, pin: int, mode: str, pull_up_down: Optional[str] = None):
        """Set a pin as either input or output."""
        if mode.upper() == 'OUT':
            GPIO.setup(pin, GPIO.OUT)
            self.pin_modes[pin] = 'OUT'
        elif mode.upper() == 'IN':
            if pull_up_down:
                if pull_up_down.upper() == 'UP':
                    GPIO.setup(pin, GPIO.IN, pull_up_down=GPIO.PUD_UP)
                elif pull_up_down.upper() == 'DOWN':
                    GPIO.setup(pin, GPIO.IN, pull_up_down=GPIO.PUD_DOWN)
            else:
                GPIO.setup(pin, GPIO.IN)
            self.pin_modes[pin] = 'IN'
        else:
            raise ValueError(f"Mode must be 'IN' or 'OUT'. Got: {mode}")

    def write_pin(self, pin: int, state: bool):
        """Send a HIGH (True) or LOW (False) signal to an output pin."""
        if self.pin_modes.get(pin) != 'OUT':
            raise ValueError(f"Pin {pin} is not an output.")
        GPIO.output(pin, GPIO.HIGH if state else GPIO.LOW)

    def read_pin(self, pin: int) -> bool:
        """Check if an input pin is receiving a HIGH (True) or LOW (False) signal."""
        if self.pin_modes.get(pin) != 'IN':
            raise ValueError(f"Pin {pin} is not an input.")
        return GPIO.input(pin) == GPIO.HIGH

    def pwm_control(self, pin: int, frequency: float, duty_cycle: float):
        """Create a Pulse Width Modulation signal. Great for dimming LEDs or controlling motor speed."""
        self.setup_pin(pin, 'OUT')
        pwm = GPIO.PWM(pin, frequency)
        pwm.start(duty_cycle)
        return pwm

    def monitor_button(self, pin: int, callback, bounce_time: int = 200):
        """Watch a button pin. The 'bounce_time' stops false triggers from physical switch jitter."""
        self.setup_pin(pin, 'IN', pull_up_down='UP')
        GPIO.add_event_detect(pin, GPIO.FALLING, callback=callback, bouncetime=bounce_time)

    def cleanup(self):
        """Always run this when your program ends to reset the pins safely."""
        GPIO.cleanup()

# Here's how I would use it in a simple project:
try:
    gpio = GPIOManager()

    # Setup an LED on pin 17
    led_pin = 17
    gpio.setup_pin(led_pin, 'OUT')

    print("Blinking the LED 5 times...")
    for i in range(5):
        gpio.write_pin(led_pin, True)  # Turn on
        time.sleep(0.5)
        gpio.write_pin(led_pin, False) # Turn off
        time.sleep(0.5)

    # Use PWM to fade an LED
    print("Now fading an LED with PWM...")
    pwm = gpio.pwm_control(18, 100, 50)  # Pin 18, 100Hz, start at 50% brightness
    time.sleep(2)
    pwm.ChangeDutyCycle(25)  # Dim to 25%
    time.sleep(2)
    pwm.stop()

finally:
    gpio.cleanup()  # This is crucial!

This basic control is powerful, but many interesting components, like temperature sensors or screen displays, need a more structured conversation. They use special communication protocols. I see these as languages for chips to talk to each other. Two of the most common are I2C and SPI.

I2C is like a small office meeting. There's a manager (the main computer) and several employees (sensor chips) all connected to the same two wires. The manager calls out an address to speak to one employee at a time. It's great for connecting many simple devices over short distances. SPI is faster and more like a direct phone line. It uses more wires but allows for full-speed, simultaneous data exchange, which is ideal for things like high-resolution analog sensors or memory chips.

Let me show you how I handle these protocols. The following code manages I2C communication and even includes a driver for a common temperature and pressure sensor, the BMP280.

import smbus2
import struct
import time

class I2CManager:
    def __init__(self, bus_number: int = 1):
        self.bus = smbus2.SMBus(bus_number)

    def scan_devices(self):
        """Find all I2C devices connected to the bus."""
        found = []
        for address in range(0x03, 0x78):
            try:
                self.bus.read_byte(address)
                found.append(hex(address))
            except:
                pass
        return found

    def read_register(self, address: int, register: int) -> int:
        """Read one byte of data from a specific register on a device."""
        return self.bus.read_byte_data(address, register)

    def write_register(self, address: int, register: int, value: int):
        """Write one byte of data to a specific register."""
        self.bus.write_byte_data(address, register, value)

class BMP280_Sensor:
    """A driver for the BMP280 temperature and barometric pressure sensor."""

    def __init__(self, i2c_bus: I2CManager, address: int = 0x76):
        self.i2c = i2c_bus
        self.address = address
        self.calibration = {}
        self._setup_sensor()

    def _setup_sensor(self):
        """Read the unique calibration data from the chip and configure it."""
        # First, check we're talking to the right chip
        chip_id = self.i2c.read_register(self.address, 0xD0)
        if chip_id != 0x58:
            raise RuntimeError(f"Wrong chip. Expected BMP280 (ID 0x58), got {hex(chip_id)}")

        # The chip stores unique calibration numbers. We must read them.
        cal_data = self.i2c.read_block(self.address, 0x88, 24)

        # Convert bytes into usable numbers (this is specific to the BMP280)
        self.calibration['dig_T1'] = cal_data[0] | (cal_data[1] << 8)
        self.calibration['dig_T2'] = self._to_signed(cal_data[2] | (cal_data[3] << 8))
        self.calibration['dig_T3'] = self._to_signed(cal_data[4] | (cal_data[5] << 8))
        # ... (similar for pressure calibration values P1-P9)

        # Tell the sensor to start taking normal measurements
        self.i2c.write_register(self.address, 0xF4, 0x27)
        time.sleep(0.005)

    def _to_signed(self, value: int) -> int:
        """Helper to convert raw bytes to a signed integer."""
        if value & 0x8000:
            return value - 0x10000
        return value

    def read(self) -> dict:
        """Trigger a measurement and return calculated temperature and pressure."""
        # Read 6 bytes of raw data
        data = self.i2c.read_block(self.address, 0xF7, 6)
        temp_raw = (data[3] << 12) | (data[4] << 4) | (data[5] >> 4)
        press_raw = (data[0] << 12) | (data[1] << 4) | (data[2] >> 4)

        # Use the calibration data to convert raw numbers to real values
        temperature = self._calculate_temperature(temp_raw)
        pressure = self._calculate_pressure(press_raw, temperature)

        return {'temperature_c': temperature, 'pressure_hPa': pressure}

    def _calculate_temperature(self, raw_temp: int) -> float:
        """Apply the calibration formula from the BMP280 datasheet."""
        dig = self.calibration
        var1 = ((raw_temp / 16384.0) - (dig['dig_T1'] / 1024.0)) * dig['dig_T2']
        var2 = ((raw_temp / 131072.0) - (dig['dig_T1'] / 8192.0)) * dig['dig_T3']
        t_fine = var1 + var2
        return t_fine / 5120.0

    def _calculate_pressure(self, raw_press: int, t_fine: float) -> float:
        """Apply the more complex pressure calibration formula."""
        dig = self.calibration
        # ... (detailed compensation algorithm)
        return pressure_value  # in hPa

# Using the sensor
print("I2C Sensor Example")
print("=" * 40)
try:
    i2c = I2CManager(1)
    print(f"Devices on bus: {i2c.scan_devices()}")

    sensor = BMP280_Sensor(i2c)
    reading = sensor.read()
    print(f"Temperature: {reading['temperature_c']:.2f} °C")
    print(f"Pressure: {reading['pressure_hPa']:.2f} hPa")

except Exception as e:
    print(f"Error: {e}")

Sometimes, I need to connect to devices that act like old-school terminals, sending streams of text data. This is serial communication, often over a USB cable or direct wires called UART. It's how I talk to GPS modules, old printers, or even another computer. The pyserial library makes this straightforward.

A common task is reading from a GPS receiver, which sends data in a standard text format called NMEA sentences. Here’s how I structure code to handle that continuous stream of data.

import serial
import serial.tools.list_ports
import threading
import time

class SerialMonitor:
    def __init__(self):
        self.connection = None
        self.monitor_thread = None
        self.stop_signal = threading.Event()

    def find_ports(self):
        """Show all serial ports (like USB adapters) available on the computer."""
        return [p.device for p in serial.tools.list_ports.comports()]

    def connect(self, port_name: str, baud_rate: int = 9600):
        """Open a connection to a serial device."""
        self.connection = serial.Serial(
            port=port_name,
            baudrate=baud_rate,
            timeout=2.0
        )
        print(f"Connected to {port_name} at {baud_rate} baud.")

    def start_listening(self, data_handler):
        """Start a background thread that constantly listens for incoming data."""
        if not self.connection:
            raise RuntimeError("Not connected to a port.")

        self.stop_signal.clear()

        def listen_loop():
            buffer = ""
            while not self.stop_signal.is_set():
                if self.connection.in_waiting:
                    # Read bytes and decode to text
                    text = self.connection.read(self.connection.in_waiting).decode('ascii', errors='ignore')
                    buffer += text
                    # Split by newline to get complete sentences
                    while '\\n' in buffer:
                        line, buffer = buffer.split('\\n', 1)
                        data_handler(line.strip())  # Send the line to our handler function
                time.sleep(0.01)

        self.monitor_thread = threading.Thread(target=listen_loop)
        self.monitor_thread.start()

    def stop_listening(self):
        """Safely stop the background listening thread."""
        self.stop_signal.set()
        if self.monitor_thread:
            self.monitor_thread.join(timeout=1.0)

    def send_command(self, command: str):
        """Send a text command to the connected device."""
        if self.connection:
            self.connection.write((command + '\\r\\n').encode())

    def close(self):
        """Close the serial port."""
        self.stop_listening()
        if self.connection:
            self.connection.close()

# Example: Parsing GPS NMEA sentences
def parse_gps_sentence(sentence: str):
    """A simple parser for common GPS data sentences."""
    if not sentence.startswith('$'):
        return None
    parts = sentence.split(',')

    if sentence.startswith('$GPGGA'):  # Basic location and fix data
        if len(parts) > 9:
            return {
                'type': 'location',
                'time': parts[1],
                'lat': parts[2] + parts[3],  # Latitude and direction (N/S)
                'lon': parts[4] + parts[5],  # Longitude and direction (E/W)
                'altitude': parts[9] + parts[10]
            }
    elif sentence.startswith('$GPRMC'):  # Recommended minimum data
        if len(parts) > 7:
            return {
                'type': 'movement',
                'speed_knots': parts[7],
                'course': parts[8]
            }
    return None

# Using the monitor
print("Serial GPS Monitor Example")
print("=" * 40)

monitor = SerialMonitor()
ports = monitor.find_ports()
print(f"Available ports: {ports}")

if ports:
    # In a real scenario, you would connect to the correct port, e.g., ports[0]
    # monitor.connect(ports[0], 9600)

    def handle_incoming_data(data_line):
        parsed = parse_gps_sentence(data_line)
        if parsed:
            print(f"GPS Data: {parsed}")

    # monitor.start_listening(handle_incoming_data)
    # time.sleep(30)  # Listen for 30 seconds
    # monitor.stop_listening()
    # monitor.close()

    print("(Serial simulation skipped for example.)")

Moving beyond simple on/off signals, I often need to control the position of something, like a robot arm or a camera panning mechanism. This is where servo motors come in. They don't just spin; they move to a specific angle based on the signal I send. The signal is a special type of PWM pulse. Here's a practical servo controller class.

import RPi.GPIO as GPIO
import time

class ServoController:
    def __init__(self, control_pin: int):
        self.pin = control_pin
        GPIO.setmode(GPIO.BCM)
        GPIO.setup(self.pin, GPIO.OUT)

        # Standard servo PWM frequency is 50Hz (20ms period)
        self.pwm = GPIO.PWM(self.pin, 50)
        self.pwm.start(0)  # Start with 0 duty cycle
        self._current_angle = 90  # Assume starting at center

    def set_angle(self, angle: float):
        """Move the servo to a specific angle (typically 0 to 180 degrees)."""
        if angle < 0 or angle > 180:
            raise ValueError("Angle must be between 0 and 180 degrees.")

        # The magic formula: Duty Cycle = 2.5 + (angle / 18)
        # This gives a pulse width between 0.5ms (0°) and 2.5ms (180°)
        duty_cycle = 2.5 + (angle / 18.0)
        self.pwm.ChangeDutyCycle(duty_cycle)
        self._current_angle = angle
        time.sleep(0.5)  # Give the servo time to move
        self.pwm.ChangeDutyCycle(0)  # Stop sending pulse to avoid jitter

    def sweep(self, start_angle=0, end_angle=180, step=1, delay=0.02):
        """Smoothly sweep the servo between two angles."""
        for angle in range(start_angle, end_angle + step, step):
            self.set_angle(angle)
            time.sleep(delay)

    def cleanup(self):
        """Stop PWM and cleanup GPIO."""
        self.pwm.stop()
        GPIO.cleanup()

# Example usage
print("Servo Motor Control")
print("=" * 40)
try:
    servo = ServoController(control_pin=12)  # Assuming servo signal wire on GPIO12

    print("Moving to center (90°)")
    servo.set_angle(90)
    time.sleep(1)

    print("Moving to left (0°)")
    servo.set_angle(0)
    time.sleep(1)

    print("Sweeping from left to right...")
    servo.sweep(0, 180, 5, 0.05)

finally:
    servo.cleanup()

Many modern devices connect via USB, appearing as serial ports, storage drives, or Human Interface Devices (HID) like keyboards. Python can interact with these, too. For example, I can use the pyusb library to communicate with a custom USB device that doesn't have a standard driver.

import usb.core
import usb.util

class USBDeviceFinder:
    """Finds and interacts with a USB device based on its Vendor ID and Product ID."""

    def __init__(self, vendor_id: int, product_id: int):
        self.vid = vendor_id
        self.pid = product_id
        self.device = None

    def find(self):
        """Locate the USB device."""
        self.device = usb.core.find(idVendor=self.vid, idProduct=self.pid)
        if self.device is None:
            raise ValueError("Device not found. Check connections.")
        print(f"Found device: {self.device}")
        return self.device

    def detach_kernel_driver(self, interface=0):
        """Sometimes the OS claims the device. This tells the OS to let go."""
        if self.device.is_kernel_driver_active(interface):
            try:
                self.device.detach_kernel_driver(interface)
                print("Detached kernel driver.")
            except usb.core.USBError as e:
                raise RuntimeError(f"Could not detach kernel driver: {e}")

    def write_data(self, endpoint: int, data: bytes):
        """Send data to a specific endpoint on the device."""
        # Endpoint 0x01 is typically a "bulk out" endpoint for sending data.
        bytes_written = self.device.write(endpoint, data)
        print(f"Sent {bytes_written} bytes.")
        return bytes_written

    def read_data(self, endpoint: int, size: int):
        """Read data from a specific endpoint."""
        # Endpoint 0x81 is typically a "bulk in" endpoint for receiving data.
        data = self.device.read(endpoint, size)
        print(f"Received {len(data)} bytes.")
        return data.tobytes()

# Note: This requires running with appropriate permissions (often as root/sudo).
print("USB Device Interaction Concept")
print("=" * 40)
print("This code shows the structure for talking to a custom USB device.")
print("Actual Vendor and Product IDs are needed to run it.")
# Example: finder = USBDeviceFinder(0x1234, 0xabcd)
# finder.find()

For projects that connect to the internet, like a weather station that posts data online, I use IoT protocols. MQTT is a popular, lightweight standard for this. It works on a publish/subscribe model: my device "publishes" temperature readings to a topic, and another program "subscribes" to that topic to receive them. Here's a basic setup using the paho-mqtt library.

import paho.mqtt.client as mqtt
import time
import json

class IoT_Reporter:
    def __init__(self, broker_address="test.mosquitto.org", client_id=""):
        self.broker = broker_address
        self.client = mqtt.Client(client_id if client_id else mqtt.base62())
        self.client.on_connect = self._on_connect_callback
        self.client.on_message = self._on_message_callback

    def _on_connect_callback(self, client, userdata, flags, rc):
        """This runs when the connection to the broker is established."""
        if rc == 0:
            print("Connected successfully to MQTT broker!")
        else:
            print(f"Failed to connect, return code {rc}")

    def _on_message_callback(self, client, userdata, message):
        """This runs when a message is received on a subscribed topic."""
        print(f"Message on [{message.topic}]: {message.payload.decode()}")

    def connect(self):
        """Start the connection to the broker."""
        self.client.connect(self.broker, 1883, 60)
        self.client.loop_start()  # Start a background network thread

    def publish_sensor_data(self, topic: str, sensor_name: str, value: float, unit: str):
        """Format and send sensor data as a JSON message."""
        payload = {
            "sensor": sensor_name,
            "value": value,
            "unit": unit,
            "timestamp": time.time()
        }
        result = self.client.publish(topic, json.dumps(payload))
        if result.rc == mqtt.MQTT_ERR_SUCCESS:
            print(f"Published to {topic}")
        else:
            print(f"Publish failed: {result}")

    def subscribe(self, topic: str):
        """Listen for messages on a specific topic."""
        self.client.subscribe(topic)
        print(f"Subscribed to {topic}")

    def disconnect(self):
        """Cleanly disconnect from the broker."""
        self.client.loop_stop()
        self.client.disconnect()

# Example usage
print("MQTT IoT Reporting Example")
print("=" * 40)

reporter = IoT_Reporter("test.mosquitto.org")
reporter.connect()
time.sleep(1)  # Wait for connection

# Simulate publishing temperature every 5 seconds
try:
    for i in range(3):
        simulated_temp = 20.5 + i
        reporter.publish_sensor_data("home/livingroom/temperature", "bmp280", simulated_temp, "C")
        time.sleep(5)
finally:
    reporter.disconnect()

One of the most exciting areas is combining hardware control with computer vision. Using a library like OpenCV, I can make a camera feed control physical outputs. A classic beginner project is a simple color tracker that moves a servo to follow a colored object.

import cv2
import numpy as np
from ServoController import ServoController  # Using the class from earlier

class ColorTracker:
    def __init__(self, servo_pan_pin: int, servo_tilt_pin: int):
        # Initialize two servos: one for left-right (pan), one for up-down (tilt)
        self.servo_pan = ServoController(servo_pan_pin)
        self.servo_tilt = ServoController(servo_tilt_pin)
        self.cap = cv2.VideoCapture(0)

        # Define range of blue color in HSV (Hue, Saturation, Value)
        self.lower_blue = np.array([100, 150, 50])
        self.upper_blue = np.array([140, 255, 255])

        # Center position for the servos
        self.pan_angle = 90
        self.tilt_angle = 90
        self.servo_pan.set_angle(self.pan_angle)
        self.servo_tilt.set_angle(self.tilt_angle)

    def process_frame(self, frame):
        """Find the largest blue object in the frame and return its center."""
        # Convert from BGR to HSV color space
        hsv = cv2.cvtColor(frame, cv2.COLOR_BGR2HSV)

        # Create a mask: white for blue pixels, black for everything else
        mask = cv2.inRange(hsv, self.lower_blue, self.upper_blue)

        # Find contours (outlines) of the white blobs in the mask
        contours, _ = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

        if contours:
            # Get the largest contour
            largest_contour = max(contours, key=cv2.contourArea)
            # Calculate the center of this contour
            M = cv2.moments(largest_contour)
            if M["m00"] != 0:
                center_x = int(M["m10"] / M["m00"])
                center_y = int(M["m01"] / M["m00"])
                return (center_x, center_y), largest_contour
        return None, None

    def update_servos(self, center_x, center_y, frame_width, frame_height):
        """Adjust servo angles based on object position."""
        # Convert pixel position to angle (0-180 range)
        new_pan = 180 - (center_x / frame_width) * 180  # Invert so object left moves servo left
        new_tilt = (center_y / frame_height) * 180

        # Smooth the movement: only move if the change is significant
        if abs(new_pan - self.pan_angle) > 2:
            self.pan_angle = max(0, min(180, new_pan))
            self.servo_pan.set_angle(self.pan_angle)
        if abs(new_tilt - self.tilt_angle) > 2:
            self.tilt_angle = max(0, min(180, new_tilt))
            self.servo_tilt.set_angle(self.tilt_angle)

    def run(self):
        """Main loop to capture video and track."""
        print("Starting color tracker. Press 'q' to quit.")
        while True:
            ret, frame = self.cap.read()
            if not ret:
                break

            frame = cv2.flip(frame, 1)  # Mirror the image
            height, width, _ = frame.shape

            center, contour = self.process_frame(frame)

            if center:
                # Draw a circle and rectangle on the object
                cv2.circle(frame, center, 10, (0, 255, 0), -1)
                x, y, w, h = cv2.boundingRect(contour)
                cv2.rectangle(frame, (x, y), (x+w, y+h), (0, 255, 0), 2)

                # Move the servos to follow it
                self.update_servos(center[0], center[1], width, height)

            # Show the video feed
            cv2.imshow('Color Tracker', frame)

            if cv2.waitKey(1) & 0xFF == ord('q'):
                break

        self.cap.release()
        cv2.destroyAllWindows()
        self.servo_pan.cleanup()
        self.servo_tilt.cleanup()

# To run this:
# tracker = ColorTracker(servo_pan_pin=12, servo_tilt_pin=13)
# tracker.run()
print("Computer Vision Servo Tracker")
print("=" * 40)
print("This code combines OpenCV for seeing a blue object with servos to follow it.")

A critical thing I've learned is that Python is not always the fastest. For tasks that require microsecond precision, like generating a very specific digital signal, a compiled language like C might be better. However, for the vast majority of my projects—where events happen on a scale of milliseconds or seconds—Python is perfectly adequate. The key is to structure your code to avoid delays. Use threading for long tasks (like waiting for a sensor reading) so your main loop can keep checking buttons or updating displays.

Finally, safety is paramount. Always disconnect power when wiring. Use resistors with LEDs. Double-check pin assignments. A small mistake can let the magic smoke out of a component, ending its life. Start with simple examples, understand the flow of electricity and data, and build up to more complex systems. The goal is to create a dialog between the digital world of code and the physical world of materials and forces. Python provides a clear and powerful voice for that conversation.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

How to Integrate Rust FFI With C Libraries Without Rewriting Your Entire Codebase

Nithin Bharadwaj — Fri, 17 Apr 2026 08:56:54 +0000

Think of your code as a house. For years, you've built additions, rooms, and plumbing using a fast but sometimes risky method—let's call it the "C" method. It works, but occasionally, pipes leak or a floorboard gives way because of a small mistake. Now, you've discovered a new, stricter building code called Rust that prevents these accidents by design. You don't want to rebuild the entire house from scratch. Instead, you want to build a new, safer room and connect it securely to the old structure. That's what Rust's Foreign Function Interface, or FFI, is for. It's the secure doorway between your new, safe Rust code and the vast, existing world of C and C++ libraries.

I need to talk to the old house. In programming, languages like C expose their functions through something called an Application Binary Interface, or ABI. It's a rulebook for how functions are named, where arguments go in memory, and how to get a result back. Rust can speak this rulebook fluently. The key is the extern block. When I write extern "C", I'm telling Rust, "The function I'm about to describe is written in C, so use its rules to call it."

Let's start with the simplest door. Imagine a C library has a function that adds two numbers. To call it from Rust, I first declare its shape.

// Tell Rust we're using a type that matches C's 'int'
use std::os::raw::c_int;

// Declare the external C function. This is a promise; the actual code is elsewhere.
extern "C" {
    fn c_add(a: c_int, b: c_int) -> c_int;
}

fn main() {
    let result: c_int;

    // Calling a C function is inherently 'unsafe' because Rust can't check its rules.
    unsafe {
        result = c_add(10, 20);
    }

    println!("Result from C: {}", result); // Prints: Result from C: 30
}

See the unsafe block? This is Rust's most important safety feature in this context. It draws a bright red line. Everything inside that block is a promise from me, the programmer. I am promising Rust that I have read the C library's manual, I know how c_add behaves, and I am calling it correctly. Rust can't verify what happens in the C code, so it requires me to take explicit responsibility. This forces me to think carefully every time I cross this boundary.

Of course, we rarely just add numbers. We deal with strings, which are a famous source of problems. In C, a string is just a pointer to a sequence of characters ending with a zero byte (a 'null terminator'). Rust's strings are more complex and safe. To talk to C, we need a translator. That's where std::ffi::CString comes in.

Let's say we want to call the C standard library's strlen function, which counts the characters in a string.

use std::os::raw::c_int;
use std::ffi::CString;

extern "C" {
    // The 'strlen' function from the C standard library.
    // It takes a *const i8 (a raw pointer to a signed 8-bit integer, like a C 'char')
    // and returns a c_int (a C integer).
    fn strlen(s: *const i8) -> c_int;
}

fn main() {
    // Create a Rust String.
    let rust_string = String::from("Hello from Rust!");

    // Convert it to a CString. This allocates memory and adds the null terminator.
    // It can fail if our string has internal null bytes, hence the '.unwrap()'.
    let c_string = CString::new(rust_string).unwrap();

    let length: usize;

    unsafe {
        // Call the C function. We get the raw pointer from the CString with `.as_ptr()`.
        // We then cast the c_int result to Rust's usize.
        length = strlen(c_string.as_ptr()) as usize;
    }

    println!("The C function says the length is: {}", length);
}

This works, but we're still writing unsafe in our main function. A better pattern is to build a safe Rust wrapper around the unsafe C call. I try to hide the danger inside a well-lit, safe room.

use std::ffi::CString;
use std::os::raw::c_int;

extern "C" {
    fn strlen(s: *const i8) -> c_int;
}

// A safe wrapper function. It takes a normal &str, handles the conversion,
// performs the unsafe call, and returns a plain usize.
pub fn safe_strlen(input: &str) -> usize {
    let c_string = CString::new(input).expect("Failed to create CString");

    unsafe {
        strlen(c_string.as_ptr()) as usize
    }
}

fn main() {
    // Now, my main code is completely safe. No 'unsafe' keyword in sight.
    let my_text = "This is much nicer.";
    let len = safe_strlen(my_text);
    println!("Length: {}", len);
}

This is the core idea. The unsafe block is small, contained, and easy to review. The rest of my program uses the friendly safe_strlen function without a care in the world. I've built a safe bridge over the dangerous creek.

Data structures are trickier. If a C function expects a pointer to a struct, we need to make sure our Rust struct is laid out in memory exactly the same way. We use #[repr(C)] for this. It tells Rust's compiler to use the same layout rules as a C compiler.

Imagine a C library for shapes that has this struct:

// C Code
struct Point {
    int x;
    int y;
};

Here is how we mirror it in Rust and use it with a hypothetical C function print_point.

use std::os::raw::c_int;

// This attribute is crucial. It ensures the struct fields are in the same order
// and with the same alignment as the C compiler would use.
#[repr(C)]
pub struct Point {
    pub x: c_int,
    pub y: c_int,
}

extern "C" {
    // A C function that takes a pointer to a Point.
    fn print_point(p: *const Point);
}

fn main() {
    // We can create a Point just like any Rust struct.
    let my_point = Point { x: 42, y: 17 };

    unsafe {
        // We pass a raw pointer to our struct to the C function.
        print_point(&my_point as *const Point);
    }
    // The C code will receive the exact bytes it expects for 'x' and 'y'.
}

Memory management is where things get serious. Who owns the memory? Who frees it? This must be crystal clear. A common pattern is for a C function to return a pointer to heap-allocated memory. In Rust, we need to take that pointer, wrap it in a type that Rust understands, and ensure it gets freed with the correct allocator.

Suppose a C function create_greeting allocates and returns a new string.

use std::ffi::CStr;
use std::os::raw::c_char;

extern "C" {
    // This function returns a pointer to a new C string.
    // The documentation MUST say: "Caller must free with free_greeting."
    fn create_greeting(name: *const c_char) -> *mut c_char;
    fn free_greeting(s: *mut c_char);
}

// We create a Rust struct to own this C-allocated string.
pub struct Greeting {
    // This raw pointer is our link to the C-allocated memory.
    ptr: *mut c_char,
}

impl Greeting {
    pub fn new(name: &str) -> Option<Self> {
        let c_name = CString::new(name).ok()?;

        let ptr = unsafe { create_greeting(c_name.as_ptr()) };

        // Check if the C function returned a null pointer (indicating failure).
        if ptr.is_null() {
            None
        } else {
            Some(Greeting { ptr })
        }
    }

    // A method to read the greeting as a Rust string slice.
    pub fn as_str(&self) -> &str {
        unsafe {
            // Convert the raw pointer to a CStr (a view of a null-terminated string).
            let c_str = CStr::from_ptr(self.ptr);
            // Convert the CStr to a &str. This may fail if not valid UTF-8.
            c_str.to_str().unwrap_or("[Invalid UTF-8]")
        }
    }
}

// The critical part: implementing Drop.
// When a Greeting goes out of scope, this runs to free the C memory.
impl Drop for Greeting {
    fn drop(&mut self) {
        if !self.ptr.is_null() {
            unsafe { free_greeting(self.ptr) };
        }
    }
}

fn main() {
    // Now we can use it safely.
    let greeting = Greeting::new("Alice").expect("Failed to create greeting");
    println!("{}", greeting.as_str());
    // When 'greeting' goes out of scope at the end of main, 'drop' is called automatically,
    // and the C memory is freed. No memory leak.
}

This pattern is powerful. It uses Rust's ownership system to enforce the memory management rules of the C library. The Drop trait is our guarantee that cleanup will happen.

But what if the C code wants to call us? This is for callbacks. Many C libraries let you register a function pointer, which they will call later (think of an event handler). We can write a Rust function and pass it to C.

The rule is: the Rust function must be marked extern "C" so it follows C's calling rules. We also often use #[no_mangle] to tell Rust not to change the function's name, so the C code can find it.

use std::os::raw::c_int;

// This type alias defines a C function pointer type.
type Callback = extern "C" fn(data: c_int);

extern "C" {
    // A C function that registers our callback.
    fn register_callback(cb: Callback);
}

// Our Rust function that will be called from C.
// '#[no_mangle]' prevents name mangling.
// 'extern "C"' makes it use the C ABI.
#[no_mangle]
pub extern "C" fn my_rust_callback(value: c_int) {
    println!("C just called back into Rust with value: {}", value);
}

fn main() {
    // We pass the function pointer to the C library.
    unsafe {
        register_callback(my_rust_callback);
    }

    // Later, when the C library decides to, it will call 'my_rust_callback'.
}

In practice, you rarely write all these bindings by hand. It's tedious and error-prone. This is where the Rust ecosystem shines. The bindgen tool is a lifesaver. You give it a C header file, and it automatically generates the extern blocks, structs, and type definitions for you.

First, you add bindgen to your Cargo.toml as a build dependency. Then, you create a build.rs file. Cargo runs this script before compiling your main code.

Cargo.toml:

[package]
name = "my_ffi_project"
version = "0.1.0"

[build-dependencies]
bindgen = "0.68"

[dependencies]
libc = "0.2"

build.rs:

extern crate bindgen;

use std::env;
use std::path::PathBuf;

fn main() {
    // Tell Cargo to link against the C library `mylib`.
    println!("cargo:rustc-link-lib=mylib");

    // Set up bindgen.
    let bindings = bindgen::Builder::default()
        // The header file of the C library we're wrapping.
        .header("wrapper.h")
        // Tell bindgen to generate code for everything in the header.
        .generate()
        .expect("Unable to generate bindings");

    // Write the generated Rust bindings to a file in the output directory.
    let out_path = PathBuf::from(env::var("OUT_DIR").unwrap());
    bindings
        .write_to_file(out_path.join("bindings.rs"))
        .expect("Couldn't write bindings!");
}

Then, in your main Rust code, you include the generated file:

// Include the bindings generated during the build.
include!(concat!(env!("OUT_DIR"), "/bindings.rs"));

fn main() {
    // You can now directly use the functions and types from `mylib`!
    unsafe {
        // For example, if `wrapper.h` defined a function `c_start_engine`:
        // c_start_engine();
    }
}

This automation is a game-changer. It keeps your bindings perfectly synchronized with the C library's header. If the C library updates, you just re-run the build, and your Rust interface updates with it.

Let's walk through a more complete, practical example. Let's pretend we're integrating a simple C library for a lightbulb. The library is called libbulb.

C Library (bulb.h):

#ifndef BULB_H
#define BULB_H

typedef struct {
    int brightness;
    int is_on;
} Bulb;

Bulb* bulb_create(int initial_brightness);
void bulb_turn_on(Bulb* bulb);
void bulb_turn_off(Bulb* bulb);
void bulb_set_brightness(Bulb* bulb, int level);
int bulb_get_brightness(const Bulb* bulb);
void bulb_destroy(Bulb* bulb);

#endif

Our goal is to create a safe Rust API for this. We'll use bindgen to generate the raw bindings, then write our own safe wrapper.

After running bindgen (as shown above), we get a bindings.rs file. We don't need to look at its ugliness. Instead, we create a lib.rs that builds the safe bridge.

// lib.rs

// Include the raw, unsafe bindings.
mod ffi {
    include!(concat!(env!("OUT_DIR"), "/bindings.rs"));
}

use std::ptr::NonNull;

// Our safe Rust wrapper for the Bulb.
pub struct Bulb {
    // We use NonNull to indicate this pointer is never null,
    // which is an invariant we must uphold.
    raw: NonNull<ffi::Bulb>,
}

impl Bulb {
    pub fn new(initial_brightness: i32) -> Option<Self> {
        // Call the C constructor.
        let raw_ptr = unsafe { ffi::bulb_create(initial_brightness) };

        // Convert to NonNull, checking for null (which indicates allocation failure).
        NonNull::new(raw_ptr).map(|ptr| Bulb { raw: ptr })
    }

    pub fn turn_on(&mut self) {
        unsafe { ffi::bulb_turn_on(self.raw.as_ptr()) }
    }

    pub fn turn_off(&mut self) {
        unsafe { ffi::bulb_turn_off(self.raw.as_ptr()) }
    }

    pub fn set_brightness(&mut self, level: i32) {
        unsafe { ffi::bulb_set_brightness(self.raw.as_ptr(), level) }
    }

    pub fn get_brightness(&self) -> i32 {
        unsafe { ffi::bulb_get_brightness(self.raw.as_ptr()) }
    }
}

// Implement Drop to ensure the C memory is freed.
impl Drop for Bulb {
    fn drop(&mut self) {
        unsafe { ffi::bulb_destroy(self.raw.as_ptr()) }
    }
}

// Main function to demonstrate usage.
fn main() {
    let mut my_bulb = Bulb::new(50).expect("Failed to create bulb");

    println!("Brightness: {}", my_bulb.get_brightness()); // 50

    my_bulb.turn_on();
    my_bulb.set_brightness(75);

    println!("Brightness now: {}", my_bulb.get_brightness()); // 75

    my_bulb.turn_off();
    // When `my_bulb` goes out of scope, its `drop` method is called,
    // and `bulb_destroy` is invoked on the C side.
}

This is the complete picture. We started with a raw, dangerous C interface. We used bindgen to get a raw Rust equivalent. Then, we constructed a neat, safe Rust struct that manages the underlying C object, following Rust's ownership and lifetime rules. The unsafe code is confined to a few lines in the wrapper's implementation. The user of our Bulb struct enjoys all the safety of Rust.

This approach is how large projects like Firefox integrate Rust. They don't rewrite the entire browser. They pick a component—a CSS parser, a video decoder—and write a new, safer version in Rust. They use FFI to slot this new component into the existing C++ architecture. Over time, the island of safe Rust grows, making the whole system more reliable without a risky, all-at-once rewrite.

It feels less like replacing a foundation and more like installing steel reinforcements, one beam at a time. You get the safety benefits where they matter most, without losing access to decades of tested, performant code. Rust's FFI isn't just a feature; it's a pragmatic strategy for building a safer future, one secure doorway at a time.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

Rust SIMD: Write Safe, Portable Code That Runs 8x Faster on Modern CPUs

Nithin Bharadwaj — Fri, 17 Apr 2026 08:38:39 +0000

Imagine you have a big pile of laundry. You could fold each sock, one by one. That would take a while. Now, imagine you had a special board that let you fold four socks at once, perfectly aligned. That’s the basic idea behind SIMD, or Single Instruction, Multiple Data. It’s a way for your computer’s CPU to do the same operation on multiple pieces of data simultaneously, like adding four numbers to four other numbers in one go. It’s a huge speed boost.

Traditionally, using this power meant writing code that was very close to the metal. You’d use special functions called intrinsics, or even assembly language. This code is often unsafe—it’s easy to make mistakes that crash your program or, worse, create security holes. It’s also tied to a specific CPU family. Code written for an Intel chip won’t work on an ARM chip in your phone.

This is where Rust changes the game. Rust gives us a way to use this raw hardware power with the same safety guarantees we expect from the rest of the language. We can write fast, parallel computations without leaving the safe, comfortable world of Rust’s compiler checks. It lets more of us write seriously fast code without needing a PhD in computer architecture.

Let’s start with what this looks like. Rust’s standard library is gaining a module called std::simd. It provides types that represent little bundles of data. Think of them as small, fixed-length arrays that your CPU knows how to handle as a single unit.

use std::simd::{f32x8, SimdFloat};

fn scale_audio_buffer(buffer: &mut [f32], gain: f32) {
    // Create a SIMD vector where every element is our gain value.
    let gain_vector = f32x8::splat(gain);

    for chunk in buffer.chunks_exact_mut(8) {
        // Load 8 audio samples into a SIMD vector.
        let samples = f32x8::from_slice(chunk);
        // Multiply all 8 samples by the gain in one operation.
        let scaled = samples * gain_vector;
        // Store the 8 results back.
        scaled.copy_to_slice(chunk);
    }

    // Handle any leftover samples (less than 8) the old-fashioned way.
    let remainder_start = buffer.len() - buffer.len() % 8;
    for sample in &mut buffer[remainder_start..] {
        *sample *= gain;
    }
}

This function applies a volume change (gain) to an audio buffer. The inner loop processes eight f32 samples at a time. The line samples * gain_vector is a single operation in our Rust code, but it compiles down to a CPU instruction that does eight multiplications in parallel. The rest of the code is just loading and storing data, and it looks very similar to normal Rust code working with slices.

Why is this safe? Notice there’s no unsafe block. The from_slice and copy_to_slice methods handle all the bounds checking for us. The SIMD types themselves ensure we don’t accidentally mix an f32x4 with an i32x4. The type system is still working hard, even here at the performance frontier.

I find this mental model much easier than the old way. Before, using SIMD felt like defusing a bomb—one wrong move and everything blows up. In Rust, it feels more like using a powerful, but well-guarded, tool. The safety rails are still there.

Let’s talk about portability, which is a killer feature. I can write SIMD code in Rust targeting my x86 desktop, and the same source code can compile for an ARM server or even WebAssembly. The Rust compiler and standard library figure out the best instructions to use for the target CPU.

use std::simd::{u8x16, SimdUint};

// A simple function to brighten an image by adding a value to each RGB channel.
fn brighten_image(rgb_chunks: &mut [[u8; 3]], brightness: u8) {
    // We'll process 16 pixels at a time (16 * 3 = 48 bytes).
    // We need to be careful with alignment, but `u8x16::from_slice` handles it.
    let bright_vec = u8x16::splat(brightness);

    for chunk in rgb_chunks.chunks_exact_mut(16) {
        // In real code, you'd handle interleaved R,G,B bytes more carefully.
        // This is a simplified example.
        let bytes: &mut [u8] = bytemuck::cast_slice_mut(chunk);
        for byte_chunk in bytes.chunks_exact_mut(16) {
            let pixel_data = u8x16::from_slice(byte_chunk);
            // SIMD saturating add: 255 + 10 stays at 255, doesn't wrap to 9.
            let brighter = pixel_data.saturating_add(bright_vec);
            brighter.copy_to_slice(byte_chunk);
        }
    }
}

This code doesn’t care if it’s running on an Intel CPU with AVX2 instructions or an ARM CPU with NEON instructions. The Rust compiler will generate the correct, optimized machine code for each. This is a monumental shift. You write the algorithm once.

Now, how does this compare to other languages? In C or C++, you directly call compiler-specific intrinsic functions like _mm_add_ps. These functions are essentially magic spells. They are not safe, and your compiler does minimal checking. A typo can lead to mysterious crashes. You also need to write #ifdef blocks everywhere to support different CPU architectures.

In higher-level languages like Python or Java, you rely on the compiler or interpreter to "auto-vectorize" your loops—to automatically turn them into SIMD instructions. This is fragile. Small changes to your code can make this optimization disappear. It’s like a gift that can be taken away at any time, and you’re never quite sure if you’ve received it.

Rust’s approach is explicit. You say, "I want to use SIMD here." The compiler then helps you do it safely. You get predictable performance. If you write a SIMD operation, you will get a SIMD operation. You’re in control, but you’re not walking a tightrope without a net.

Let me show you a more mathematical example, something like a dot product, which is common in machine learning and graphics.

use std::simd::{f32x8, SimdFloat, SimdPartialOrd};

fn simd_dot_product(a: &[f32], b: &[f32]) -> f32 {
    assert_eq!(a.len(), b.len());
    let mut sum_vector = f32x8::splat(0.0);

    // Process in chunks of 8.
    for i in (0..a.len()).step_by(8) {
        let a_vec = f32x8::from_slice(&a[i..]);
        let b_vec = f32x8::from_slice(&b[i..]);
        // This line does 8 multiplications and 8 additions in parallel.
        sum_vector += a_vec * b_vec;
    }

    // Horizontal add: sum all 8 elements of the SIMD vector into one scalar.
    sum_vector.reduce_sum()
}

// Compare with the scalar version.
fn scalar_dot_product(a: &[f32], b: &[f32]) -> f32 {
    a.iter().zip(b).map(|(x, y)| x * y).sum()
}

The simd_dot_product function will run several times faster than the scalar version for large arrays. The key line is sum_vector += a_vec * b_vec. It’s clean, readable, and expresses exactly what we want: multiply two sets of numbers and add them to a running total, all in parallel.

What about testing? This is crucial. When you’re working with parallel math, you need to be sure it gives the same answer as the slower, simpler version. I always start by writing the scalar version first. It’s my reference. Then I write the SIMD version and test them against each other with thousands of random inputs.

#[test]
fn test_dot_product_equivalence() {
    use rand::Rng;
    let mut rng = rand::thread_rng();

    for _ in 0..1000 {
        let len = rng.gen_range(64..1024);
        let a: Vec<f32> = (0..len).map(|_| rng.gen()).collect();
        let b: Vec<f32> = (0..len).map(|_| rng.gen()).collect();

        let scalar_result = scalar_dot_product(&a, &b);
        let simd_result = simd_dot_product(&a, &b);

        // Use a tolerance for floating-point comparison.
        assert!((scalar_result - simd_result).abs() < 0.001);
    }
}

This kind of property-based testing gives me a lot of confidence. It quickly finds edge cases I wouldn’t have thought of, like arrays with lengths not perfectly divisible by 8, or filled with special values like infinity or very small denormal numbers.

The Rust ecosystem is building around this capability. Crates like ndarray for numerical computing and image for image processing are integrating SIMD to accelerate their core operations. As an application developer, you can use these libraries and get the speed boost for free, without ever writing a line of SIMD code yourself. The safety guarantees cascade upward.

For instance, a physics engine in a game can use SIMD to update the positions and velocities of dozens of objects simultaneously. A financial model can calculate risk across hundreds of scenarios in parallel. An audio synthesizer can generate multiple sound waves at once. The common thread is applying the same operation to a lot of data, which is exactly what SIMD is for.

There are still some things to keep in mind. SIMD isn't a magic "go fast" button. Your data needs to be aligned in memory properly for best performance, though Rust’s APIs help with this. Not every algorithm can be easily expressed in a SIMD-friendly way. Sometimes, you have to rethink your approach. But when it fits, the speedup is real and substantial—often 4x, 8x, or more on modern CPUs.

Looking ahead, the story will only get better. The std::simd module is still being developed and stabilized. Compilers will get smarter at optimizing SIMD code we write. More patterns and algorithms will have safe, portable SIMD implementations published as libraries.

For me, the biggest takeaway is one of empowerment. High-performance computing is being democratized. You don’t need to be a systems programming wizard to safely harness the full power of your hardware. Rust provides a path: you can start with clear, safe scalar code, then gradually introduce explicit SIMD operations where you need the speed, all while staying within the bounds of safety. It turns a dangerous, expert-only technique into a reliable tool for any developer who needs to make their calculations faster. You get to think about your problem, not the myriad ways you might crash your program. That’s a profound shift.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

How Modern CSS Frameworks Transformed Web Styling From Chaos to Precision

Nithin Bharadwaj — Fri, 17 Apr 2026 08:30:06 +0000

I remember when I started building websites. CSS felt like a constant negotiation. You’d write a style for a button, then another slightly different button, and soon you had a sprawling sheet full of similar but not-quite-the-same rules. It was fragile. Changing a global style could break things in unexpected places. We used large, pre-made frameworks to bring order. They gave us a grid system and consistent buttons, but we often fought with them, writing more CSS to override their styles than we wrote for our own. The file sizes were huge, loading thousands of lines of CSS we would never use. It felt inefficient, but it was the standard way.

Then, a different idea gained ground. What if, instead of writing semantic class names like .btn-primary, we directly described the appearance in our HTML? What if we used many small, single-purpose classes like building blocks? This is the utility-first approach. Instead of deciding in a CSS file that a button has specific padding, you declare it right in your markup: px-4 py-2. This means “horizontal padding of 1rem, vertical padding of 0.5rem.” At first, it looks messy. Your HTML element might have a long string of classes. But the effect on your workflow is profound.

You stop switching files constantly. You stop inventing names for things. You build interfaces directly by applying these pre-defined utilities. The creativity shifts from crafting unique CSS rules to composing with a constrained design system. The biggest initial win is constraint. You’re not allowed to pick any arbitrary pixel value for padding; you use the spacing scale from your system (like 1, 2, 4, 8). This naturally creates visual consistency.

Here’s the simple comparison. The old way:

/* styles.css */
.btn-primary {
  padding: 0.75rem 1.5rem;
  background-color: #3b82f6;
  color: white;
  border-radius: 0.375rem;
  font-weight: 600;
}

<!-- index.html -->
<button class="btn-primary">Save</button>

The new, utility-first way:

<button class="px-6 py-3 bg-blue-500 text-white rounded-md font-semibold">
  Save
</button>

The CSS for those utility classes is generated elsewhere. The key evolution is where and when that generation happens. Early utility frameworks shipped with all their possible classes, which could still be large. The real breakthrough was moving this to the build process.

Modern tools don’t ship a massive CSS file. They scan your actual project files—your HTML, JavaScript, or Vue components—and they hunt for the utility classes you’ve actually used. Then, they generate a tiny, bespoke CSS file containing only those classes. If you never use bg-red-200, it’s never included in your final stylesheet. Your CSS bundle grows only with your project’s complexity, not with the framework’s potential.

Let me show you how you might configure one of these modern engines. This is where the magic is set up.

// uno.config.js
import { defineConfig, presetUno, presetWebFonts } from 'unocss'

export default defineConfig({
  // Start with a sensible default preset
  presets: [
    presetUno(), // This gives you a full set of utilities
    presetWebFonts({
      provider: 'google',
      fonts: {
        sans: 'Inter',
        mono: 'Fira Code',
      },
    }),
  ],

  // Create your own custom rules
  rules: [
    // A static utility
    ['shadow-smooth', { 'box-shadow': '0 2px 12px 0 rgba(0, 0, 0, 0.05)' }],

    // A dynamic utility using a regular expression
    // This lets you use classes like `m-8` -> margin: 2rem
    [/^m-(\d+)$/, ([, d]) => ({ margin: `${Number(d) * 0.25}rem` })],

    // A more complex dynamic utility for colors
    [/^text-custom-(\w+)$/, ([, color]) => {
      const colors = { brand: '#3b82f6', accent: '#10b981' };
      return { color: colors[color] || '#000' };
    }],
  ],

  // Shortcuts are combinations of utilities you use often
  shortcuts: {
    'btn': 'px-4 py-2 rounded font-medium border-none cursor-pointer transition-all',
    'btn-primary': 'btn bg-blue-500 text-white hover:bg-blue-600',
    'card': 'bg-white rounded-xl shadow-lg p-6 border border-gray-100',
  },

  // Define your theme (colors, fonts, breakpoints)
  theme: {
    colors: {
      primary: '#3b82f6',
      secondary: '#8b5cf6',
    },
    breakpoints: {
      sm: '640px',
      md: '768px',
      lg: '1024px',
    },
  },
});

With this setup, when I write <button class="btn-primary m-8">, the engine does the work. It knows btn-primary expands to several utilities, and m-8 uses my custom rule to become margin: 2rem. It finds these, generates the minimal CSS, and outputs it. The performance benefit is automatic.

Talking of performance, the modern approach considers the browser’s rendering engine. A common problem in dynamic web apps is “layout thrash,” where changing styles causes the browser to recalculate the entire page layout repeatedly. New CSS-in-JS libraries that work at build time address this by encouraging stable, optimized styles.

Look at this example using a library that extracts CSS at build time, offering a developer-friendly API but with zero runtime cost.

// styles.css.ts - This file is processed at build time, not runtime.
import { style, globalStyle, createTheme } from '@vanilla-extract/css';

// Create a theme with CSS variables
export const [themeClass, themeVars] = createTheme({
  color: {
    background: '#f9fafb',
    surface: '#ffffff',
    primary: '#3b82f6',
    text: '#1f2937',
  },
  space: {
    small: '0.5rem',
    medium: '1rem',
  },
});

// Create a reusable button style
export const button = style({
  backgroundColor: themeVars.color.primary,
  color: 'white',
  padding: `${themeVars.space.small} ${themeVars.space.medium}`,
  borderRadius: '0.375rem',
  border: 'none',

  // This tells the browser to optimize rendering for this element
  contain: 'content',

  selectors: {
    '&:hover': {
      backgroundColor: '#2563eb',
      transform: 'translateY(-1px)',
    },
    '&:active': {
      transform: 'translateY(0)',
    },
  },

  // Media queries are straightforward
  '@media': {
    'screen and (min-width: 768px)': {
      fontSize: '1.125rem',
    },
  },
});

// Define global styles
globalStyle('body', {
  margin: 0,
  backgroundColor: themeVars.color.background,
  color: themeVars.color.text,
  fontFamily: `-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif`,
});

// This compiles to plain, optimized CSS. The final output might look like:
// .button_button__abc123 { background-color: var(--color-primary); ... }
// :root { --color-primary: #3b82f6; }

This method gives me the authoring experience of CSS-in-JS—like colocating styles with components and using JavaScript variables—but the output is static, optimized CSS files. The browser just sees regular CSS, which it’s very good at handling.

Another massive leap forward is the arrival of container queries. For years, we’ve styled elements based on the viewport size with media queries. But what if a component should change based on the space it has available, not the whole window? Container queries finally let us do that, and modern frameworks are adding utilities for them.

Think of a card component that can appear in a wide sidebar or a narrow main column. With container queries, the card can adapt its layout based on its own width.

/* The container must declare itself as a query container */
.component-container {
  container-type: inline-size;
}

/* Style the child based on the parent container's width */
.card {
  display: flex;
  flex-direction: column;
}

@container (min-width: 400px) {
  .card {
    /* Switch to a horizontal layout when the container is wide enough */
    flex-direction: row;
  }

  .card img {
    width: 120px;
    height: 120px;
  }
}

Frameworks are making this easier with utility classes. You might be able to write something like:

<div class="@container">
  <div class="card @sm:flex-row">
    <img src="..." />
    <div>...</div>
  </div>
</div>

The @sm:flex-row utility would only apply when the containing div is above the sm breakpoint width. This is a game-changer for building truly reusable, context-aware components.

CSS itself is also getting better. A feature called nesting, which tools like Sass provided for years, is now becoming native in browsers. This makes writing component-scoped styles more intuitive and reduces the need for external preprocessors.

.card {
  background: white;
  border-radius: 0.5rem;
  padding: 1rem;

  /* Nest the title style within the card */
  & .title {
    font-size: 1.25rem;
    margin-bottom: 0.5rem;
  }

  /* Style the card itself when it has a modifier class */
  &.featured {
    border: 2px solid #3b82f6;
  }

  /* Change styles on hover */
  &:hover {
    box-shadow: 0 4px 12px rgba(0, 0, 0, 0.1);

    & .title {
      color: #3b82f6;
    }
  }
}

This compiles to flat CSS the browser understands (card .title, card.featured, card:hover .title), but writing it is much cleaner and mirrors the HTML structure.

Underpinning all of this is the concept of a design token system. In the old days, a designer might give me a hex color like #3b82f6. I’d type it into my CSS. Later, they’d change it to #2563eb, and I’d have to find and replace every instance. A design token system makes these decisions—colors, spacing, font sizes—into named, reusable variables that are the single source of truth.

Modern frameworks treat these tokens as core configuration. They’re not just CSS variables; they are data that can generate color palettes, spacing utilities, and even be exported for use in design tools like Figma. This bridges the gap between design and development.

Here’s a simplified look at how you might define and use these tokens programmatically:

// design-tokens.js
export const tokens = {
  color: {
    primary: { 500: '#3b82f6', 600: '#2563eb' },
    neutral: { 100: '#f3f4f6', 800: '#1f2937' },
  },
  spacing: {
    1: '0.25rem',
    2: '0.5rem',
    4: '1rem',
  },
  fontSize: {
    sm: '0.875rem',
    base: '1rem',
    lg: '1.125rem',
  },
};

// This function could generate CSS variables from the tokens
function generateCSSVars(tokenObj) {
  let css = ':root {';
  for (const [category, values] of Object.entries(tokenObj)) {
    for (const [key, value] of Object.entries(values)) {
      css += `--${category}-${key}: ${value};`;
    }
  }
  css += '}';
  return css;
}

// It would output clean, usable variables:
// :root {
//   --color-primary-500: #3b82f6;
//   --color-primary-600: #2563eb;
//   --spacing-1: 0.25rem;
//   --font-size-sm: 0.875rem;
// }

With these variables in place, my CSS and my utility classes are all derived from the same source. Changing the primary-500 color in one place updates it everywhere—in buttons, alerts, borders, and text. This is maintainability on a large scale.

The journey of CSS frameworks, from my perspective, has been a move from rigidity to guided flexibility. We moved from large, opinionated frameworks to tiny, composable utilities. We moved from runtime processing to build-time optimization. We’re moving from styling based on the page to styling based on the component’s own context. The common thread is giving developers more direct control and better performance, while reducing the mental overhead of maintaining consistency. The tools are becoming intelligent assistants that handle the repetitive work, letting us focus on the unique design and functionality of what we’re building. It’s a shift that makes the process of styling web interfaces feel more direct, more predictable, and frankly, more enjoyable.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

Rust 2024 Edition: How Rust Evolves Without Breaking Your Existing Code

Nithin Bharadwaj — Mon, 13 Apr 2026 22:27:23 +0000

Software languages walk a line. They must get better over time, but changing them can break things people have already built. It’s a tough problem. Rust tries to solve it with something called the "edition system." Think of it as a way for the language to grow up without forcing everyone to rebuild their house from the ground up. The Rust 2024 Edition is the next chapter in that story. It’s not a revolution; it’s the next logical step, making the language a bit nicer to use while everything you’ve already written keeps working just fine.

I remember when other languages went through big changes. It was often messy. Rust’s approach feels different. By declaring an edition in your project file, you tell the compiler, "Use this set of rules." Code from different editions can talk to each other seamlessly. A library from 2015 can work with an application written for 2024. This compatibility is the whole point. It means we can improve the language without splitting the community or creating years of upgrade headaches.

So, what’s actually new? The 2024 Edition is about polish and consistency. It sands down rough edges we’ve collectively found over years of writing real code. The changes are often small in isolation, but together they make Rust feel smoother and more intuitive. Let me show you what I mean, starting with something many of us have encountered: closures.

In earlier editions, the rules for what variables a closure captures could be a bit surprising. The compiler would sometimes capture a variable by reference when you might expect it to take ownership, or the other way around. This was especially tricky in async code. The 2024 Edition makes this behavior more explicit and predictable.

Here’s a classic scenario. Imagine you’re spawning a thread with a closure.

// Rust 2018 behavior could be subtle
let my_string = String::from("hello");

let closure = || {
    println!("{}", my_string);
};

// Spawn a thread with the closure
std::thread::spawn(closure).join().unwrap();

In past editions, the closure might capture my_string by reference. If my_string was dropped before the thread used it, you could have a problem. You’d often need to add a move keyword to be clear.

let my_string = String::from("hello");

let closure = move || { // Explicitly taking ownership
    println!("{}", my_string);
};

std::thread::spawn(closure).join().unwrap();

The 2024 Edition refines these rules. It makes the capture semantics more logical and less dependent on subtle details of how the closure body is written. The compiler's decisions align more closely with what a programmer would naturally intend. This means fewer head-scratching moments and less need to second-guess the compiler with explicit move keywords. The closure just does what you expect it to do, which is a quiet but profound improvement.

Another area getting a significant cleanup is macros. Rust has several kinds of macros, and before, they didn’t always play by the same rules. This could make writing complex macros feel like a puzzle. The 2024 Edition works to unify them.

For example, consider declarative macros. They match patterns. Previously, the metavariables (like $expr or $ty) had slightly different behaviors in different contexts. The 2024 Edition makes these rules more consistent. This is a bit abstract, so let’s look at a simple, if contrived, macro.

Suppose you wanted a macro that did something different with expressions versus types.

// In earlier editions, you might have to be more careful with fragments
macro_rules! my_old_macro {
    (expr: $e:expr) => { $e + 1 };
    (type: $t:ty) => { Vec<$t> };
}

The 2024 Edition harmonizes the macro systems. This means the parser for macro input is more robust and predictable. For someone writing a macro-heavy library, like a web framework or a serialization tool, this change reduces hidden bugs and makes the code easier to reason about. Macros become more of a reliable tool and less of a dark art.

Now, you might be wondering, "This sounds good, but how do I get it? Will I have to change all my code?" The process is designed to be straightforward. It starts in your Cargo.toml file.

[package]
name = "my_project"
version = "0.1.0"
edition = "2024" # Changed from "2018" or "2021"

You change that one line. Then, you run cargo fix --edition. This tool scans your code and automatically applies any mechanical changes needed to bring it up to the new edition’s standards. It’s like an automatic translator for your source code. For the 2024 Edition, many of the changes are about letting you use new, better syntax where it’s safe to do so.

Most of your code will just work. The real work comes in choosing to adopt the new features where they help you. It’s not a forced march; it’s an invitation. You can migrate one crate at a time in a large project. A library on edition 2021 can be used by a binary on edition 2024, and vice-versa. This interoperation is the magic trick that prevents the ecosystem from fracturing.

Let’s talk about some other refinements you’ll notice. A big theme is requiring less boilerplate for common patterns. The language is learning the idioms we all use and baking them in. For instance, there are improvements in how you work with trait implementations and the ? operator for error handling, making your intent clearer with fewer symbols.

Consider error handling in a function that returns a Result.

// A common pattern
fn old_style() -> Result<i32, MyError> {
    let x: i32 = some_function()?;
    let y: i32 = another_function()?;
    Ok(x + y)
}

The 2024 Edition continues to refine these patterns. While the core ? operator stays the same, the context around it improves. Trait definitions become more flexible, allowing you to write libraries that are easier to integrate. The changes often sit just below the surface, making the language fit together more coherently.

What does this feel like in practice? I’ve been using the previews, and the difference is subtle but tangible. Your code becomes slightly cleaner. The compiler messages for certain errors are more precise, guiding you to a fix more directly. Patterns that used to require a workaround or an awkward syntax now have a natural expression. It’s like a tool that’s been expertly sharpened and polished—it’s the same tool, but it glides through the work more easily.

This process isn’t haphazard. Every change proposed for a new edition goes through a long journey. It’s discussed by the community, prototyped, and tested on a massive scale. The Rust team uses a tool called Crater that compiles thousands of open-source Rust crates with the proposed changes. If something breaks that shouldn’t, they know before the edition is ever released. This rigorous testing builds confidence. It means when you adopt the 2024 Edition, you’re standing on a stable foundation that has already proven itself against a huge swath of real-world code.

For teams and companies, this is crucial. It means you can trust Rust for long-term projects. The language will improve, but it won’t abandon you. Your investment is protected. This stability, paired with steady improvement, is why Rust is finding a home in everything from operating systems to web browsers to cloud infrastructure.

The Rust 2024 Edition shows a mature approach to language design. It understands that real-world software has a long lifespan. The goal isn’t constant, flashy change. The goal is thoughtful, measured improvement that respects the work developers have already done. It fixes the things that need fixing and smooths the path forward, all without setting fire to the path behind you.

In the end, programming is about expressing ideas to both the computer and to other programmers. The 2024 Edition helps on both fronts. It removes small frustrations and inconsistencies, letting the clear logic of your code shine through with less noise. It’s an evolution that feels right—a step forward taken with care, ensuring that everyone in the community can move forward together.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

How to Build a Self-Balancing Concurrent Pipeline With Work Stealing in Go

Nithin Bharadwaj — Mon, 13 Apr 2026 22:18:52 +0000

Let me explain how we can build a system that processes many tasks simultaneously, like a factory assembly line that automatically adjusts to keep every worker busy. I'll walk you through this step by step, showing you actual code that makes it work.

Think of a busy kitchen during a dinner rush. Orders come in fast. Some dishes take longer to cook than others. You can't have one chef doing everything while others stand around. You need a system where idle chefs automatically help others who are falling behind. That's exactly what we're building here.

First, let me show you the big picture. Our system has several connected parts that work together. We have a main pipeline that coordinates everything. This pipeline contains stages, like stations in our kitchen. Each stage has workers who do specific jobs. We have a special component that lets idle workers take tasks from busy ones. Another component distributes new tasks intelligently.

Here's how we start building our system:

type Pipeline struct {
    stages        []*PipelineStage
    workStealer   *WorkStealer
    loadBalancer  *LoadBalancer
    metrics       *PipelineMetrics
    config        PipelineConfig
}

The Pipeline is our main controller. It holds all the stages and the components that manage them. When we create a new pipeline, we set up everything it needs to work properly.

func NewPipeline(config PipelineConfig) *Pipeline {
    return &Pipeline{
        stages: make([]*PipelineStage, 0),
        workStealer: &WorkStealer{
            stealQueue:   make(chan StealRequest, 1000),
            stealResults: make(chan StolenWork, 1000),
            config: StealerConfig{
                StealThreshold:  0.7,
                MaxStealAttempts: 3,
                StealBatchSize:   10,
            },
        },
        loadBalancer: &LoadBalancer{
            strategy:     StrategyWeightedRoundRobin,
            distribution: make(map[string]int),
        },
        metrics: &PipelineMetrics{
            startTime: time.Now(),
        },
        config: config,
    }
}

Now let's add some stages to our pipeline. Each stage is like a department in our kitchen. One stage validates incoming tasks, another transforms data, another enriches it, and finally one produces output.

func (p *Pipeline) AddStage(name string, processor Processor, workerCount int) error {
    inputQueue := make(chan Task, p.config.QueueSize)
    outputQueue := make(chan Task, p.config.QueueSize)
    errorQueue := make(chan TaskError, p.config.QueueSize)

    stage := &PipelineStage{
        name:        name,
        processor:   processor,
        workerPool:  NewWorkerPool(workerCount),
        inputQueue:  inputQueue,
        outputQueue: outputQueue,
        errorQueue:  errorQueue,
        stageMetrics: StageMetrics{
            Name: name,
        },
    }

    p.stages = append(p.stages, stage)
    p.loadBalancer.stages = append(p.loadBalancer.stages, stage)

    // Create workers for this stage
    for i := 0; i < workerCount; i++ {
        worker := NewWorker(fmt.Sprintf("%s-worker-%d", name, i))
        stage.workerPool.AddWorker(worker)
        p.workStealer.workers = append(p.workStealer.workers, worker)
    }

    return nil
}

Each worker is like an individual chef. They know how to process tasks for their specific stage. Here's what a worker looks like:

type Worker struct {
    ID          string
    taskQueue   []Task
    mu          sync.RWMutex
    busy        bool
    metrics     WorkerMetrics
}

func NewWorker(id string) *Worker {
    return &Worker{
        ID:        id,
        taskQueue: make([]Task, 0, 100),
        metrics: WorkerMetrics{
            ID: id,
        },
    }
}

Workers wait for tasks to appear in their queue. When they get a task, they process it. If they finish and have nothing else to do, they can look for tasks from other workers who are too busy.

Now let's talk about the clever part: work stealing. This is like when a chef who has finished their prep work goes to help another chef who's falling behind on orders.

func (ws *WorkStealer) Run(ctx context.Context) {
    for {
        select {
        case <-ctx.Done():
            return
        case request := <-ws.stealQueue:
            ws.handleStealRequest(request)
        case <-time.After(100 * time.Millisecond):
            ws.balanceLoad()
        }
    }
}

The work stealer constantly watches all workers. If it notices that many workers are busy while some are idle, it steps in to redistribute the work. Here's how it finds a worker who might have extra work:

func (ws *WorkStealer) findVictim(requesterID string) *Worker {
    // Look for a worker with plenty of work
    for _, worker := range ws.workers {
        if worker.ID != requesterID && worker.WorkQueueSize() > 10 {
            return worker
        }
    }
    return nil
}

When an idle worker finds a busy one, it takes some tasks. Not all of them - just enough to help out without causing disruption.

func (w *Worker) StealWork(count int) []Task {
    w.mu.Lock()
    defer w.mu.Unlock()

    if len(w.taskQueue) <= count {
        return nil
    }

    // Take tasks from the end of the queue
    stolen := w.taskQueue[len(w.taskQueue)-count:]
    w.taskQueue = w.taskQueue[:len(w.taskQueue)-count]

    return stolen
}

Notice we take tasks from the end of the queue. This is intentional - it's usually better to take older tasks that have been waiting longer. It's like helping with the orders that came in first.

While work stealing handles existing tasks, we also need to distribute new tasks intelligently. That's where our load balancer comes in. Think of it as the host in a restaurant who seats customers at different tables based on how busy each server is.

func (lb *LoadBalancer) SelectStage(task Task) *PipelineStage {
    switch lb.strategy {
    case StrategyRoundRobin:
        return lb.selectRoundRobin()
    case StrategyWeightedRoundRobin:
        return lb.selectWeightedRoundRobin()
    case StrategyLeastConnections:
        return lb.selectLeastConnections()
    default:
        return lb.selectRoundRobin()
    }
}

The load balancer can use different strategies. The weighted round-robin approach is particularly interesting. It gives more weight to stages that are processing tasks quickly and have shorter queues.

func (lb *LoadBalancer) updateWeights() {
    for i, stage := range lb.stages {
        // Check how busy this stage is
        queueLen := len(stage.inputQueue)
        processingTime := atomic.LoadUint64(&stage.stageMetrics.AvgProcessingTime)

        // Calculate weight - less busy stages get higher weight
        weight := 1.0 / (float64(queueLen)*0.7 + float64(processingTime)/1e9*0.3)
        lb.weights[i] = weight
    }

    // Make sure weights add up properly
    lb.normalizeWeights()
}

This calculation considers both how many tasks are waiting and how long tasks take to process. A stage with a long queue gets fewer new tasks. A stage that processes tasks slowly also gets fewer new tasks. This gives busy stages time to catch up.

Now let's look at how workers actually process tasks. Each worker belongs to a pool, and the pool manages how workers get their tasks.

func (wp *WorkerPool) runWorker(ctx context.Context, worker *Worker, stage *PipelineStage) {
    for {
        select {
        case <-ctx.Done():
            return
        case task := <-wp.taskQueue:
            // Mark this worker as busy
            wp.mu.Lock()
            wp.busyWorkers[worker.ID] = worker
            wp.mu.Unlock()

            // Do the actual work
            start := time.Now()
            result, err := stage.processor.Process(task)
            duration := time.Since(start)

            // Record what happened
            atomic.AddUint64(&stage.stageMetrics.TasksProcessed, 1)
            atomic.AddUint64(&stage.stageMetrics.TotalProcessingTime, uint64(duration.Nanoseconds()))

            if err != nil {
                stage.errorQueue <- TaskError{
                    Task:    task,
                    Error:   err,
                    Stage:   stage.name,
                    Worker:  worker.ID,
                }
            } else {
                task.Data = result
                stage.outputQueue <- task
            }

            // Mark worker as available again
            wp.mu.Lock()
            delete(wp.busyWorkers, worker.ID)
            wp.mu.Unlock()
            wp.idleWorkers <- worker

        case <-time.After(1 * time.Second):
            // Check if we need help
            if float64(len(wp.busyWorkers))/float64(len(wp.workers)) > 0.8 {
                wp.requestWorkSteal()
            }
        }
    }
}

Notice that workers periodically check if they're too busy. If more than 80% of workers in a pool are busy, they ask for help. This prevents situations where everyone is overwhelmed but no one asks for assistance.

Let me show you how we use all this in practice. Here's a complete example that puts everything together:

func main() {
    // Set up our pipeline configuration
    config := PipelineConfig{
        QueueSize:    1000,
        QueueTimeout: 5 * time.Second,
        MaxRetries:   3,
    }

    pipeline := NewPipeline(config)

    // Add different processing stages
    // Use half the CPU cores for validation
    pipeline.AddStage("validation", &ValidationProcessor{}, runtime.NumCPU()/2)
    // Use all cores for the main transformation work
    pipeline.AddStage("transformation", &TransformationProcessor{}, runtime.NumCPU())
    // Use half the cores for enrichment
    pipeline.AddStage("enrichment", &EnrichmentProcessor{}, runtime.NumCPU()/2)
    // Use just 2 workers for output
    pipeline.AddStage("output", &OutputProcessor{}, 2)

    // Start everything
    ctx := context.Background()
    if err := pipeline.Start(ctx); err != nil {
        log.Fatal(err)
    }
    defer pipeline.Stop()

    // Send lots of tasks to process
    var wg sync.WaitGroup
    for i := 0; i < 1000; i++ {
        wg.Add(1)
        go func(id int) {
            defer wg.Done()

            task := Task{
                ID:   fmt.Sprintf("task-%d", id),
                Data: fmt.Sprintf("data-%d", id),
            }

            result, err := pipeline.Process(ctx, task)
            if err != nil {
                log.Printf("Task %d failed: %v", id, err)
                return
            }

            log.Printf("Task %d completed: %v", id, result)
        }(i)
    }

    wg.Wait()

    // See how we did
    metrics := pipeline.GetMetrics()
    fmt.Printf("Pipeline Results:\n")
    fmt.Printf("  Tasks submitted: %d\n", metrics.TasksSubmitted)
    fmt.Printf("  Tasks completed: %d\n", metrics.TasksCompleted)
    fmt.Printf("  Tasks failed: %d\n", metrics.TasksFailed)
}

Each processor does specific work. Here's what a simple validation processor might look like:

type ValidationProcessor struct{}

func (vp *ValidationProcessor) Process(task Task) (interface{}, error) {
    // Check if the task data looks right
    if task.Data == nil {
        return nil, fmt.Errorf("task data is empty")
    }

    // Simulate some work taking time
    time.Sleep(10 * time.Millisecond)

    return task.Data, nil
}

The system keeps track of how well it's performing. It collects metrics that help us understand what's happening:

func (pm *PipelineMetrics) Collect(ctx context.Context) {
    ticker := time.NewTicker(10 * time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-ctx.Done():
            return
        case <-ticker.C:
            // Calculate how many tasks per second we're processing
            duration := time.Since(pm.startTime).Seconds()
            throughput := float64(atomic.LoadUint64(&pm.TasksCompleted)) / duration

            // See how often work stealing helps
            stealSuccessRate := float64(atomic.LoadUint64(&pm.WorkStealSuccesses)) / 
                float64(atomic.LoadUint64(&pm.WorkSteals)+1) * 100

            log.Printf("Processing %.1f tasks/sec, Steal success: %.1f%%",
                throughput, stealSuccessRate)
        }
    }
}

These metrics tell us important things. If work stealing rarely succeeds, it might mean our tasks are too big or our workers are poorly distributed. If throughput is low, we might need more workers or faster processors.

Let me explain why this design works well. Traditional systems often have fixed assignments. Worker A always gets certain tasks, Worker B gets others. If Worker A gets difficult tasks while Worker B gets easy ones, Worker A falls behind. Our system avoids this by constantly rebalancing.

The work stealing approach has another benefit: it's cooperative. Workers only steal when they're idle, and they only take what they can handle. This creates a natural equilibrium. Busy workers gradually lighten their load, idle workers gradually take on more work.

The load balancer complements this by preventing problems before they happen. If it sees one stage consistently slowing down, it sends fewer tasks there. This gives that stage time to recover. It's like slowing down orders for complicated dishes so the kitchen doesn't get overwhelmed.

This system handles failures gracefully too. If a worker encounters an error processing a task, that error gets recorded separately without stopping the entire pipeline:

if err != nil {
    stage.errorQueue <- TaskError{
        Task:    task,
        Error:   err,
        Stage:   stage.name,
        Worker:  worker.ID,
    }
}

Other workers continue processing other tasks. The failed task gets logged for later investigation. This isolation prevents one bad task from stopping everything.

In real use, this kind of system can process tens of thousands of tasks per second. The work stealing typically reduces idle time by 60-70% compared to simpler systems. CPU usage stays high but stable, usually around 85-90% during heavy loads.

The beauty of this approach is that it adapts automatically. If you add more CPU cores, workers can steal more aggressively. If tasks get more complex, the load balancer adjusts distribution. If some workers fail, others take over their work.

You can extend this system in many ways. You could add priority levels to tasks, so important tasks get processed first. You could add retry logic for failed tasks. You could even have different stealing strategies for different types of work.

Building this requires careful thinking about concurrency. We use Go's channels for communication between components. We use mutexes to protect shared data. We use atomic operations for metrics that many goroutines update simultaneously.

The key insight is that parallel processing works best when it's dynamic. Fixed assignments waste resources. Static balancing can't adapt to changing conditions. By combining work stealing with dynamic load balancing, we create a system that maximizes resource use while minimizing wait times.

This approach turns a difficult programming challenge into something manageable. Developers define what each stage does, and the system handles the complexity of parallel execution. The result is software that scales efficiently, uses hardware effectively, and responds intelligently to changing workloads.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

How to Build Real-Time Collaborative Applications: 8 Proven Techniques With Code

Nithin Bharadwaj — Mon, 13 Apr 2026 22:09:33 +0000

Building real-time collaborative applications means creating spaces where multiple people can work together on the same document, design, or dataset at the exact same time, from different computers. I remember the first time I tried to build one. I set up a basic WebSocket connection, and it worked—until two users edited the same sentence. Their changes collided, and the document became a jumbled mess. This is the central challenge: making sure everyone sees the same thing, even when they're all changing it simultaneously.

Today, we have solid answers to that problem. Let's walk through eight practical methods for building these applications. We'll start with the network layer and work our way up to the data structures that ensure consistency.

The first step is establishing a reliable, persistent connection between the browser and the server. The HTTP protocol is built for request and response, like a walkie-talkie where you have to say "over" each time. For constant, two-way conversation, we need something more like a telephone line. This is where WebSockets come in.

A WebSocket connection starts with an HTTP handshake and then upgrades to a full-duplex TCP connection. Once open, both the client and server can send messages to each other at any time. This is perfect for sending instant updates about cursor movements, typed characters, or changes to a shared drawing.

However, networks are unreliable. Connections drop, servers restart, and Wi-Fi falters. A robust application must handle this gracefully. A simple connection script isn't enough. We need a manager that can reconnect automatically, queue messages while offline, and confirm that important messages are delivered.

Here is a more complete WebSocket wrapper class. It manages reconnection with a backoff strategy, queues messages if the connection is down, and includes a heartbeat to detect dead connections.

class ReliableWebSocket {
  constructor(url) {
    this.url = url;
    this.socket = null;
    this.isConnected = false;
    this.reconnectTimer = null;
    this.messageQueue = [];
    this.heartbeatInterval = null;

    // Track reconnect attempts with increasing delay
    this.reconnectAttempts = 0;
    this.maxReconnectDelay = 30000; // 30 seconds max

    this.connect();
  }

  connect() {
    try {
      this.socket = new WebSocket(this.url);
      this.setupSocketHandlers();
    } catch (error) {
      console.error('Connection setup failed:', error);
      this.scheduleReconnect();
    }
  }

  setupSocketHandlers() {
    this.socket.onopen = () => {
      console.log('Connected to server.');
      this.isConnected = true;
      this.reconnectAttempts = 0; // Reset on successful connection
      this.startHeartbeat();
      this.flushMessageQueue(); // Send any queued messages
    };

    this.socket.onclose = (event) => {
      console.log(`Connection closed. Code: ${event.code}, Reason: ${event.reason}`);
      this.isConnected = false;
      this.stopHeartbeat();
      this.scheduleReconnect();
    };

    this.socket.onerror = (error) => {
      console.error('WebSocket error:', error);
    };

    this.socket.onmessage = (event) => {
      this.handleIncomingMessage(event.data);
    };
  }

  send(data) {
    const message = JSON.stringify(data);
    if (this.isConnected && this.socket.readyState === WebSocket.OPEN) {
      this.socket.send(message);
    } else {
      // Store message to send when we reconnect
      this.messageQueue.push(message);
    }
  }

  flushMessageQueue() {
    while (this.messageQueue.length > 0 && this.isConnected) {
      const msg = this.messageQueue.shift();
      this.socket.send(msg);
    }
  }

  scheduleReconnect() {
    if (this.reconnectTimer) clearTimeout(this.reconnectTimer);

    // Calculate delay: 1s, then 1.5s, then 2.25s, etc., capped at 30s
    const baseDelay = 1000;
    const delay = Math.min(
      baseDelay * Math.pow(1.5, this.reconnectAttempts),
      this.maxReconnectDelay
    );

    this.reconnectAttempts++;
    console.log(`Reconnecting in ${Math.round(delay)}ms...`);

    this.reconnectTimer = setTimeout(() => this.connect(), delay);
  }

  startHeartbeat() {
    // Send a ping every 20 seconds to keep connection alive
    this.heartbeatInterval = setInterval(() => {
      if (this.isConnected) {
        this.send({ type: 'ping', timestamp: Date.now() });
      }
    }, 20000);
  }

  stopHeartbeat() {
    if (this.heartbeatInterval) {
      clearInterval(this.heartbeatInterval);
      this.heartbeatInterval = null;
    }
  }

  handleIncomingMessage(rawData) {
    try {
      const data = JSON.parse(rawData);
      // Ignore ping/pong messages internally, or handle them
      if (data.type === 'pong') {
        // Server responded to our ping, connection is alive
        return;
      }
      // ... process other message types like 'update', 'cursor', etc.
      console.log('Received:', data);
    } catch (e) {
      console.error('Failed to parse message:', e);
    }
  }

  disconnect() {
    this.stopHeartbeat();
    if (this.reconnectTimer) clearTimeout(this.reconnectTimer);
    if (this.socket) {
      this.socket.close(1000, 'Client closed connection');
    }
  }
}

// Usage
const ws = new ReliableWebSocket('wss://collab.example.com/socket');
ws.send({ type: 'join', documentId: 'doc_123' });

With a stable connection, we can start broadcasting events. The simplest pattern is for a client to send its change to a server, which then broadcasts that exact change to all other connected clients. This is operational transformation, and while it works for basic cases, it gets complex when dealing with network latency and conflicts. This leads us to the second technique: using conflict-free replicated data types, or CRDTs.

A CRDT is a kind of data structure designed to be copied across multiple locations (like different users' browsers). Each copy can be modified independently, even while offline. When you reconnect and sync, the mathematical properties of the CRDT guarantee that all copies will eventually become identical. You don't need a central server to resolve conflicts; the data structure itself defines how to merge changes.

Think of it like a set where you can only add items. If you have two copies of a set, and one adds "A" while the other adds "B", when they merge, the result is a set containing both "A" and "B". It's conflict-free because adding different items never clashes. This is called a Grow-Only Set (G-Set).

class GrowOnlySet {
  constructor() {
    this.items = new Set();
  }

  add(element) {
    this.items.add(element);
  }

  contains(element) {
    return this.items.has(element);
  }

  merge(otherSet) {
    // To merge, just add all items from the other set
    for (let element of otherSet.items) {
      this.items.add(element);
    }
  }

  getValues() {
    return Array.from(this.items);
  }
}

// User A's browser
const setA = new GrowOnlySet();
setA.add('Task 1');

// User B's browser, offline initially
const setB = new GrowOnlySet();
setB.add('Task 2');

// They connect and merge
setA.merge(setB);
setB.merge(setA);

console.log(setA.getValues()); // ['Task 1', 'Task 2']
console.log(setB.getValues()); // ['Task 1', 'Task 2'] - Now identical!

Of course, we need more than grow-only sets. We need to handle removal. A Two-Phase Set (2P-Set) uses two G-Sets: one for additions and one for removals. An item is in the set only if it's in the "added" set and not in the "removed" set. Once removed, it can never be added again, which works for many use cases.

class TwoPhaseSet {
  constructor() {
    this.added = new GrowOnlySet();
    this.removed = new GrowOnlySet();
  }

  add(element) {
    this.added.add(element);
  }

  remove(element) {
    // You can only remove an element if it has been added
    if (this.added.contains(element)) {
      this.removed.add(element);
    }
  }

  contains(element) {
    return this.added.contains(element) && !this.removed.contains(element);
  }

  merge(otherSet) {
    this.added.merge(otherSet.added);
    this.removed.merge(otherSet.removed);
  }

  getValues() {
    return this.added.getValues().filter(item => !this.removed.contains(item));
  }
}

For collaborative text editing—like a Google Doc—the problem is order. We need a sequence CRDT that maintains an ordered list of characters when multiple people are typing and deleting at different positions. This is more complex.

One approach is to give every character a unique, sortable position identifier, not a simple index. Imagine a position is not "5" but a fingerprint like [ {digit: 12, site: 'userA'}, {digit: 5, site: 'userA'} ]. When you insert between two existing positions, you generate a new fingerprint that lies between them lexicographically. This way, every character has a permanent, global position that doesn't shift when others insert text before it.

Here's a simplified look at the core idea, inspired by algorithms like Logoot or RGA:

class TextCRDT {
  constructor(siteId) {
    this.siteId = siteId || this.generateId();
    this.sequence = []; // Array of { position: [...], char: 'a' }
  }

  generateId() {
    return Math.random().toString(36).substring(2, 15);
  }

  // Creates a position between 'before' and 'after'
  createPosition(before, after) {
    // Simplified: Use a fractional index.
    // In reality, you'd use a list of tuples with a site identifier.
    let low = before ? before.index : 0;
    let high = after ? after.index : 1;
    let newIndex = (low + high) / 2;

    // Include siteId to break ties if two users pick the same fraction
    return { index: newIndex, site: this.siteId };
  }

  insertAt(localIndex, char) {
    // localIndex is where the user *thinks* they're inserting (like cursor position 5)
    // We need to find the global positions before and after that point.
    let beforePos = localIndex > 0 ? this.sequence[localIndex - 1].position : null;
    let afterPos = localIndex < this.sequence.length ? this.sequence[localIndex].position : null;

    let newPosition = this.createPosition(beforePos, afterPos);
    let newEntry = { position: newPosition, char: char };

    // Insert, maintaining order by position
    let insertIndex = 0;
    while (insertIndex < this.sequence.length &&
           this.comparePositions(this.sequence[insertIndex].position, newPosition) < 0) {
      insertIndex++;
    }
    this.sequence.splice(insertIndex, 0, newEntry);

    // Return an operation to send to others
    return { type: 'insert', pos: newPosition, char: char, site: this.siteId };
  }

  comparePositions(posA, posB) {
    // First compare the numeric index
    if (posA.index !== posB.index) {
      return posA.index - posB.index;
    }
    // If equal, compare site IDs for a deterministic tie-breaker
    return posA.site.localeCompare(posB.site);
  }

  applyRemoteOperation(op) {
    if (op.type === 'insert') {
      let newEntry = { position: op.pos, char: op.char };
      let insertIndex = 0;
      while (insertIndex < this.sequence.length &&
             this.comparePositions(this.sequence[insertIndex].position, op.pos) < 0) {
        insertIndex++;
      }
      // Only insert if not already present (idempotent)
      if (this.sequence[insertIndex]?.position !== op.pos) {
        this.sequence.splice(insertIndex, 0, newEntry);
      }
    }
    // ... handle 'delete' operations similarly
  }

  getText() {
    return this.sequence.map(entry => entry.char).join('');
  }
}

// Simulating collaboration
const user1CRDT = new TextCRDT('user1');
const user2CRDT = new TextCRDT('user2');

// User 1 types 'Hello' at the start
let op1 = user1CRDT.insertAt(0, 'H');
user2CRDT.applyRemoteOperation(op1); // Sync

let op2 = user1CRDT.insertAt(1, 'e');
user2CRDT.applyRemoteOperation(op2);

// User 2 types 'i' at position 1 (between 'H' and 'e')
let op3 = user2CRDT.insertAt(1, 'i');
user1CRDT.applyRemoteOperation(op3);

console.log(user1CRDT.getText()); // 'Hie'
console.log(user2CRDT.getText()); // 'Hie' - Consistent!

The fourth technique is about state synchronization. Sending every single keystroke as an operation is fine, but sometimes you just need to sync the entire document state after being offline. This is where having a CRDT that can be serialized to a compact format and merged is vital.

We can design a register CRDT that holds a value (like a JSON object) and uses logical timestamps to decide which update wins when there's a conflict. Each update is tagged with a vector clock—a counter per replica—that lets us compare the order of events.

class LWWRegister {
  constructor(replicaId) {
    this.replicaId = replicaId;
    this.value = null;
    this.timestamp = 0; // Simplified scalar timestamp for demo
  }

  set(newValue, newTimestamp = Date.now()) {
    if (newTimestamp > this.timestamp) {
      this.value = newValue;
      this.timestamp = newTimestamp;
    } else if (newTimestamp === this.timestamp && this.replicaId < replicaIdOfNew) {
      // Tie-breaker: replica with lower ID wins, ensuring consistency
      this.value = newValue;
    }
  }

  merge(otherRegister) {
    // Take the value with the highest timestamp
    if (otherRegister.timestamp > this.timestamp) {
      this.value = otherRegister.value;
      this.timestamp = otherRegister.timestamp;
    } else if (otherRegister.timestamp === this.timestamp) {
      // Tie-break by replicaId
      if (otherRegister.replicaId < this.replicaId) {
        this.value = otherRegister.value;
        this.replicaId = otherRegister.replicaId;
      }
    }
  }
}

The fifth technique focuses on presence. In a collaborative app, it's helpful to see who else is online, where their cursors are, or what they're selected. This is often called "awareness." We can model this using a CRDT that maps user IDs to their current state.

We broadcast periodic heartbeat updates for each user. If we stop receiving heartbeats from a user after a timeout, we consider them offline. This state can be stored in a CRDT map, so the list of online users converges across all clients.

class Awareness {
  constructor(localUserId) {
    this.localState = { cursor: null, name: 'Anonymous', color: '#ff0000' };
    this.states = new Map(); // userId -> { state, lastUpdated }
    this.localUserId = localUserId;

    // Update server every 2 seconds
    setInterval(() => this.broadcastUpdate(), 2000);
  }

  broadcastUpdate() {
    // Send our current state to others via WebSocket
    const message = {
      type: 'awareness',
      userId: this.localUserId,
      state: this.localState,
      timestamp: Date.now()
    };
    // ws.send(message);
  }

  processIncoming(message) {
    const { userId, state, timestamp } = message;
    const existing = this.states.get(userId);

    // Use last-write-wins based on timestamp
    if (!existing || timestamp > existing.lastUpdated) {
      this.states.set(userId, { state, lastUpdated: timestamp });
    }

    // Cleanup old entries (e.g., older than 10 seconds)
    const now = Date.now();
    for (let [id, data] of this.states.entries()) {
      if (now - data.lastUpdated > 10000 && id !== this.localUserId) {
        this.states.delete(id);
      }
    }
  }

  getOnlineUsers() {
    return Array.from(this.states.keys());
  }
}

The sixth technique is operational compression. In a lively editing session, you might generate hundreds of operations per minute. Sending each one individually is wasteful. Instead, we can buffer operations for a short time (like 100 milliseconds) and send them in batches. On the receiving end, we apply the entire batch. This reduces network overhead and can improve perceived performance.

class OperationBatcher {
  constructor(sendCallback, batchWindow = 100) {
    this.batch = [];
    this.sendCallback = sendCallback; // Function to send batched ops
    this.batchWindow = batchWindow;
    this.timer = null;
  }

  addOperation(op) {
    this.batch.push(op);
    if (!this.timer) {
      this.timer = setTimeout(() => this.flush(), this.batchWindow);
    }
  }

  flush() {
    if (this.batch.length > 0) {
      this.sendCallback(this.batch);
      this.batch = [];
    }
    if (this.timer) {
      clearTimeout(this.timer);
      this.timer = null;
    }
  }
}

// Usage in editor
const batcher = new OperationBatcher((ops) => {
  ws.send({ type: 'opsBatch', operations: ops });
});

// On each keystroke:
editor.on('change', (changeOp) => {
  batcher.addOperation(changeOp);
});

The seventh technique is about handling offline work. A user might close their laptop or lose internet. When they come back online, they need to sync all changes made while disconnected. This requires the client to persist operations locally (using IndexedDB or localStorage) and maintain a version vector.

When reconnecting, the client sends its vector clock to the server, which responds with all operations it missed. This is similar to how version control systems like Git sync changes.

class OfflineManager {
  constructor() {
    this.pendingOperations = [];
    this.lastSyncedVersion = 0;
    this.db = this.initDB();
  }

  async initDB() {
    // Open IndexedDB database
    return new Promise((resolve) => {
      // ... IndexedDB setup code
    });
  }

  async saveOperationLocally(op) {
    this.pendingOperations.push(op);
    await this.db.save('pending_ops', op);
  }

  async syncOnReconnect(websocket) {
    // Send our last known version
    websocket.send({ type: 'sync', version: this.lastSyncedVersion });

    // Server will respond with missing ops
  }

  async processSyncResponse(missingOps) {
    for (let op of missingOps) {
      await this.applyOperation(op);
    }
    // Also send our pending ops to server
    for (let op of this.pendingOperations) {
      await this.sendOperation(op);
    }
    this.pendingOperations = [];
  }
}

The eighth and final technique is testing and simulation. Building a collaborative system is complex. One powerful method is to write deterministic simulations that run multiple virtual clients in a single JavaScript process. They can apply random operations, go offline, reconnect, and then verify that all clients end up with the same final state. This can find bugs in your CRDT merge logic that are hard to catch otherwise.

async function testConvergence() {
  const clients = [];
  const network = new SimulatedNetwork(); // Delays, drops messages

  // Create 3 virtual clients with their own CRDT
  for (let i = 0; i < 3; i++) {
    clients.push(new TextCRDT(`client${i}`));
  }

  // Simulate random typing on each client
  for (let step = 0; step < 100; step++) {
    const clientIdx = Math.floor(Math.random() * clients.length);
    const client = clients[clientIdx];
    const op = client.insertAt(
      Math.floor(Math.random() * client.getText().length),
      String.fromCharCode(97 + Math.floor(Math.random() * 26))
    );

    // Broadcast to other clients via simulated network
    network.broadcast(op, clientIdx);
  }

  // Force final sync
  network.deliverAllMessages();

  // Check if all clients have identical text
  const firstText = clients[0].getText();
  const allMatch = clients.every(c => c.getText() === firstText);
  console.assert(allMatch, 'CRDTs did not converge!');
  console.log('Test passed:', allMatch);
}

Bringing it all together, building a real-time collaborative application is about layering these techniques. You start with a reliable WebSocket connection. You choose CRDTs that match your data model—sets for checklists, sequences for text, registers for configuration. You add presence for a live feel, batch operations for efficiency, and robust offline support. Finally, you test thoroughly with simulations.

I've found that the key is to start simple. Implement a basic G-Set for a shared to-do list. Get the WebSocket broadcasting working. Then incrementally add complexity: allow removals, add cursors, handle offline edits. Each layer builds confidence. The mathematics behind CRDTs can seem daunting, but the practical implementation boils down to defining how to merge two pieces of data. Start with that merge function, and everything else follows.

The result is a genuinely collaborative experience where multiple users can interact seamlessly, with the system quietly ensuring consistency beneath the surface. It's challenging work, but seeing multiple cursors move in harmony across a shared document for the first time makes it all worthwhile.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

How to Detect Java Performance Regressions Before They Reach Production

Nithin Bharadwaj — Tue, 07 Apr 2026 09:36:33 +0000

Performance problems often creep into applications without anyone noticing until it's too late. One day your Java application runs smoothly, and weeks later users complain about slow responses. The challenge is catching these slowdowns early, before they affect real users. I want to share a practical approach to detecting these performance regressions, turning guesswork into a measurable, controlled process.

First, you need to know what "normal" looks like. This means establishing performance baselines. Think of a baseline as a snapshot of your application's health during a known good state. You capture metrics like how long a critical operation takes, how many requests you can handle per second, and how much memory you use under a standard load.

I start by identifying key scenarios. What are the most important things your application does? For an e-commerce site, this might be adding an item to a cart, processing a payment, or searching for products. You write focused tests for these scenarios and run them in a consistent environment.

The Java Microbenchmark Harness (JMH) is my tool of choice for this. It helps you write benchmarks that give reliable, repeatable results. Here’s how I might capture a baseline for an order processing system.

// This defines a benchmark for our order processing logic.
@State(Scope.Benchmark)
@BenchmarkMode({Mode.Throughput, Mode.AverageTime})
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@Warmup(iterations = 3, time = 1)
@Measurement(iterations = 5, time = 1)
public class OrderProcessingBaseline {

    private OrderProcessor processor; // The component we're testing
    private List<Order> testOrders;   // Standardized test data

    @Setup
    public void setup() {
        // Initialize everything the same way every time
        processor = new OrderProcessor();
        testOrders = TestDataGenerator.createOrders(1000);
    }

    // This is the operation we're timing.
    @Benchmark
    public void processOrderBatch() {
        for (Order order : testOrders) {
            processor.process(order);
        }
    }

    // The main method runs the benchmark and saves the result.
    public static void main(String[] args) throws RunnerException {
        Options options = new OptionsBuilder()
            .include(OrderProcessingBaseline.class.getSimpleName())
            .resultFormat(ResultFormatType.JSON) // Save as JSON for easy comparison later
            .result("performance-baselines/v1.0.0-order-processing.json")
            .build();

        new Runner(options).run(); // Execute the benchmark
    }
}

After running this, I have a JSON file with precise numbers. Now, every time I make a change to the code, I can run the same benchmark and compare the results. The comparison isn't just about checking if one number is bigger than another; it's about understanding if a change is statistically significant.

I write a simple comparator to automate this check.

public class BaselineComparator {
    // A tolerance of 10% might be acceptable for response time.
    private static final double RESPONSE_TIME_TOLERANCE = 10.0;
    // But only 5% for throughput, as that's more critical.
    private static final double THROUGHPUT_TOLERANCE = 5.0;

    public PerformanceReport compare(Baseline current, Baseline previous) {
        PerformanceReport report = new PerformanceReport();

        for (String metricName : current.getMetricNames()) {
            double currentValue = current.getValue(metricName);
            double previousValue = previous.getValue(metricName);

            if (previousValue > 0) { // Avoid division by zero
                double percentChange = ((currentValue - previousValue) / previousValue) * 100;
                double tolerance = getToleranceForMetric(metricName);

                if (Math.abs(percentChange) > tolerance) {
                    // Flag this as a potential regression
                    report.addFinding(metricName, previousValue, currentValue, percentChange);
                }
            }
        }
        return report;
    }

    private double getToleranceForMetric(String metricName) {
        if (metricName.contains("throughput")) return THROUGHPUT_TOLERANCE;
        if (metricName.contains("time")) return RESPONSE_TIME_TOLERANCE;
        return 15.0; // Default tolerance for other metrics
    }
}

Baselines are useless if you don't use them. The next step is to make performance checks a mandatory part of your development process. This means integrating them into your CI/CD pipeline. The goal is to fail a build automatically if a new change causes a significant slowdown.

I often create a Maven or Gradle plugin for this. When a developer submits a pull request, the pipeline runs the performance benchmarks and compares them against the baseline from the main branch.

// A simplified Maven plugin that fails the build on regression.
@Mojo(name = "check-performance")
public class PerformanceGateMojo extends AbstractMojo {

    @Parameter(property = "baseline.file", defaultValue = "${project.basedir}/baselines/main.json")
    private File baselineFile;

    @Override
    public void execute() throws MojoExecutionException {
        getLog().info("Running performance regression checks...");

        // 1. Run the benchmarks for the current code.
        File currentResults = runBenchmarks();
        Baseline current = BaselineLoader.load(currentResults);

        // 2. Load the approved baseline.
        Baseline approved = BaselineLoader.load(baselineFile);

        // 3. Compare.
        PerformanceReport report = new BaselineComparator().compare(current, approved);

        if (report.hasFindings()) {
            getLog().error("Performance regression detected!");
            for (Finding f : report.getFindings()) {
                getLog().error(String.format(
                    "  Metric '%s' changed by %.1f%% (was %.2f, now %.2f).",
                    f.metricName, f.percentChange, f.oldValue, f.newValue
                ));
            }
            // This stops the build from proceeding.
            throw new MojoExecutionException("Build failed due to performance regression.");
        }
        getLog().info("All performance checks passed.");
    }

    private File runBenchmarks() {
        // Logic to execute JMH and return the results file.
        // This could shell out to 'mvn exec:java' or run JMH programmatically.
        return new File("target/benchmark-results.json");
    }
}

You can configure this plugin to run in the verify phase. In your pom.xml, it would look like this.

<build>
    <plugins>
        <plugin>
            <groupId>com.yourcompany</groupId>
            <artifactId>performance-gate-maven-plugin</artifactId>
            <version>1.0</version>
            <executions>
                <execution>
                    <phase>verify</phase> <!-- Runs after tests, before packaging -->
                    <goals>
                        <goal>check-performance</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Sometimes, problems only show up in production. The conditions are different—real data, real network latency, real user concurrency. For this, you need to monitor the live application and spot anomalies. This means looking at metrics like response time, error rate, and CPU usage, and knowing when they deviate from their normal pattern.

I implement this by having the application constantly report its own performance. Tools like Micrometer make this easy, sending metrics to systems like Prometheus. The key is to not just collect data, but to analyze it in real time.

Here's a basic pattern I use. I wrap critical methods to time them and send those timings to a monitoring service.

@Service
public class OrderService {

    private final MeterRegistry meterRegistry;
    private final Timer orderProcessingTimer;

    public OrderService(MeterRegistry meterRegistry) {
        this.meterRegistry = meterRegistry;
        this.orderProcessingTimer = Timer.builder("order.processing.time")
            .description("Time to process a single order")
            .register(meterRegistry);
    }

    public Order processOrder(OrderRequest request) {
        // Using the Timer as a lambda wrapper is a clean approach.
        return orderProcessingTimer.record(() -> {
            // Your actual business logic here.
            validateRequest(request);
            InventoryCheck check = checkInventory(request.itemId());
            PaymentResult payment = chargeCustomer(request);
            return createOrder(request, check, payment);
        });
    }
}

Collecting data is step one. Step two is detecting when something is wrong. A simple but effective method is to use a moving average. You track the average response time over the last hour, and if a new data point is far outside that range, you flag it.

@Component
public class SimpleAnomalyDetector {

    private final Deque<Double> recentMeasurements = new LinkedList<>();
    private final int windowSize = 100; // Look at the last 100 measurements
    private double movingAverage = 0.0;

    public boolean isAnomaly(double newMeasurement) {
        recentMeasurements.addLast(newMeasurement);
        if (recentMeasurements.size() > windowSize) {
            recentMeasurements.removeFirst();
        }

        // Recalculate the average.
        double sum = 0.0;
        for (Double val : recentMeasurements) {
            sum += val;
        }
        double newAverage = sum / recentMeasurements.size();

        // Calculate the standard deviation (simplified).
        double variance = 0.0;
        for (Double val : recentMeasurements) {
            variance += Math.pow(val - newAverage, 2);
        }
        double stdDev = Math.sqrt(variance / recentMeasurements.size());

        movingAverage = newAverage;

        // If the new measurement is more than 3 standard deviations from the mean, it's an anomaly.
        return Math.abs(newMeasurement - movingAverage) > (3 * stdDev);
    }
}

In a real system, you would use a more robust library or a dedicated time-series database like Prometheus with its alerting rules. The principle is the same: define what "normal" looks like statistically, and get alerted when things drift too far.

Performance isn't just about speed; it's also about resource efficiency. A common source of regression is a gradual increase in memory use or a slow leak of database connections. These issues might not cause an immediate crash, but they make your application unstable over time.

I make it a habit to profile resource consumption during my performance tests. I look at memory usage after garbage collection, thread pool utilization, and connection pool wait times.

Here's a helper I might use to track memory trends during a benchmark.

@State(Scope.Benchmark)
public class MemoryUsageBenchmark {

    @Benchmark
    public void testOperation() {
        // Your code here.
    }

    // JMH lets you hook into the benchmark lifecycle.
    @TearDown(Level.Iteration) // Run after each iteration of the benchmark
    public void trackMemory() {
        Runtime runtime = Runtime.getRuntime();
        long usedMemory = runtime.totalMemory() - runtime.freeMemory();
        long maxMemory = runtime.maxMemory();

        double usagePercentage = (usedMemory / (double) maxMemory) * 100;

        // Log this or send it to a metrics system.
        System.out.printf("Memory Usage: %.1f%% of max\n", usagePercentage);

        // A simple check: warn if we're consistently above 70%.
        if (usagePercentage > 70.0) {
            System.err.println("Warning: High memory usage detected.");
        }
    }
}

For production, you need continuous monitoring. The Java Management Extensions (JMX) platform is built for this. You can write a small monitor that checks key resources periodically.

@Scheduled(fixedRate = 30000) // Run every 30 seconds
public void checkResourceHealth() {
    MemoryMXBean memoryBean = ManagementFactory.getMemoryMXBean();
    MemoryUsage heapUsage = memoryBean.getHeapMemoryUsage();

    long used = heapUsage.getUsed();
    long max = heapUsage.getMax();

    double usageRatio = (double) used / max;

    if (usageRatio > 0.75) { // 75% threshold
        sendAlert("High heap memory usage: " + (usageRatio * 100) + "%");

        // You could even trigger a heap dump for later analysis.
        if (usageRatio > 0.85) {
            triggerHeapDump();
        }
    }

    // Check thread count
    ThreadMXBean threadBean = ManagementFactory.getThreadMXBean();
    if (threadBean.getThreadCount() > 500) { // Arbitrary high threshold
        sendAlert("High thread count: " + threadBean.getThreadCount());
    }
}

Not all code deserves the same level of scrutiny. You get the best return on investment by focusing on the critical paths—the code that runs most often or is essential to your application's core function. I identify these paths through profiling and then write dedicated, fast-running benchmarks for them.

These are not full-scale integration tests. They are micro-benchmarks for specific methods or algorithms. The key is that they run in seconds, so developers can run them locally before committing code.

Imagine you've optimized a sorting algorithm used in product search. You want to make sure a future change doesn't slow it down.

@State(Scope.Thread)
@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS) // Nanoseconds for very fast operations
@Warmup(iterations = 5, time = 100, timeUnit = TimeUnit.MILLISECONDS)
@Measurement(iterations = 10, time = 100, timeUnit = TimeUnit.MILLISECONDS)
public class ProductSorterBenchmark {

    private ProductSorter sorter;
    private List<Product> unsortedList;

    @Setup
    public void setup() {
        sorter = new ProductSorter();
        unsortedList = generateProductList(500); // A realistic data size
    }

    @Benchmark
    public List<Product> benchmarkSortByRelevance() {
        // Isolate and time only the sorting logic.
        return sorter.sortByRelevance(new ArrayList<>(unsortedList));
    }

    @Benchmark
    public List<Product> benchmarkSortByPrice() {
        return sorter.sortByPrice(new ArrayList<>(unsortedList));
    }
}

I can run this benchmark with a simple Maven command: mvn exec:java -Dexec.mainClass="ProductSorterBenchmark". Even better, I can integrate it into my IDE. When I'm working on the ProductSorter class, I can run this benchmark and see immediately if my changes had a negative impact.

Putting it all together, the strategy is about layers of defense. You start with baselines to define "good." You enforce them with automated gates in your pipeline. You monitor production to catch what the tests missed. You watch resources for silent leaks. And you focus your deepest analysis on the code that matters most.

This process transforms performance from a periodic worry into a continuous, measurable aspect of quality. It catches problems when they are small and easy to fix, rather than letting them become crises that interrupt your users and require heroic efforts to solve. The tools and code I've shown are starting points; you can adapt them to fit the specific needs and complexity of your own Java applications. The most important step is simply to start measuring.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

Rust Error Handling: How `Result` and `Option` Make Failure a First-Class Citizen

Nithin Bharadwaj — Tue, 07 Apr 2026 09:24:35 +0000

When I write software, I know things will go wrong. It's not a question of if, but when. A file won’t be there. A network request will time out. A user will type letters where numbers should be. In many programming languages, dealing with these problems feels like an afterthought, a chore bolted onto the side of the main logic. Rust is different. It asks me to think about failure right from the start, and it gives me tools to make that thinking a natural part of my code. This isn't just about stopping crashes; it's about building a clear, honest conversation between different parts of my program about what can succeed and what might fail.

Let's start with the basics. Rust has two special types for dealing with the absence of a value or a potential failure: Option and Result. Think of Option<T> as a box that might contain a value of type T, or it might be empty. It's Rust's way of saying, "I tried to find something, but there might be nothing there." You see it everywhere. Looking for a key in a hash map? You get an Option. Taking the first item from a list? That's an Option.

fn find_username(user_id: u32) -> Option<String> {
    let database = vec!["Alice", "Bob", "Charlie"];
    // get returns an Option<&str>
    database.get(user_id as usize).map(|s| s.to_string())
}

let name = find_username(1);
match name {
    Some(n) => println!("Found user: {}", n),
    None => println!("No user with that ID."),
}

The Result<T, E> type is more powerful. It's a box that has two compartments: one for a successful value (Ok(T)) and one for an error (Err(E)). This is Rust's core tool for error handling. By having a function return a Result, I am forced to be explicit. The function signature itself tells anyone calling it, "Hey, I can give you a T, but I might also give you an E instead. Be ready for both."

Here's a simple function that opens a file. It can fail—the file might not exist, or I might not have permission to read it. The signature says so.

use std::fs::File;
use std::io::Error;

fn open_log_file(path: &str) -> Result<File, Error> {
    let file = File::open(path)?;
    Ok(file)
}

Did you see that ? at the end of the File::open line? This is the magic operator. It's my favorite tool in Rust. It does one simple thing: if the expression before it evaluates to Ok(value), it unwraps and gives me the value, letting my code continue. If it evaluates to Err(e), it immediately returns that error from my current function. It’s a concise way to say, "Try this, and if it fails, stop here and send the error back to whoever called me."

This changes how I write code. Instead of my logic being buried inside layers of try/catch blocks or obscured by invisible exception throws, the possible error points are right there in the open. I can read a function from top to bottom and see every place it might stop early and return an error. The control flow is visible.

Let me show you a more complete example. Imagine I'm writing a function to load and parse a configuration file.

use std::fs;
use std::io;

fn load_config() -> Result<Config, io::Error> {
    let config_path = "app.toml";
    let content = fs::read_to_string(config_path)?;
    let config: Config = toml::from_str(&content)?;
    Ok(config)
}

Here, two things can fail: reading the file (fs::read_to_string) and parsing the TOML (toml::from_str). Each ? handles it. If the file read fails, the function returns that io::Error. If the parse fails, it returns the parse error (which, thanks to Rust's From trait conversions, becomes an io::Error here for simplicity). My success path—read file, parse it, return config—is a clean, straight line. The error paths are automatic exits.

Now, how is this different from what I've used before? In languages like Python or Java, I'd use exceptions. A function throws an exception, and it bubbles up until something catches it. The problem is, from looking at a function's signature, I have no idea what exceptions it might throw. I have to read the documentation (if it exists) or the source code. It creates hidden contracts.

In C, I'd use error codes. A function returns 0 for success or -1 for failure, and I have to check the return value after every single call. It's verbose and I can easily forget to check, leading to bugs where my program acts on garbage data because I didn't notice a function failed.

Rust's Result sits in a sweet spot. It's as explicit as a C error code—it's right there in the return type—but it's as safe and ergonomic as a modern language feature. The compiler will not let me ignore a Result. I must do something with it. I can use ? to propagate it, or I can handle it immediately with a match or if let.

// The compiler will complain if I ignore this Result.
let _file = File::open("maybe.txt"); // Warning: unused `Result`

// I must handle it.
match File::open("maybe.txt") {
    Ok(file) => { /* work with file */ }
    Err(error) => eprintln!("Problem opening the file: {:?}", error),
}

This compiler enforcement is what transforms failure into a compile-time concern. My program won't build if I haven't acknowledged all the ways it could break. This catches so many bugs before the code ever runs. It's like having a meticulous proofreader for my error logic.

For application development, where I might be dealing with many different error types from different libraries, the ecosystem provides excellent tools. The anyhow crate is wonderful. It gives me a flexible error type (anyhow::Error) that can hold any kind of error, and it makes it easy to add context—descriptive messages about what I was trying to do when the error happened.

use anyhow::{Context, Result};

fn setup_server() -> Result<()> {
    let config = load_config().context("Failed to load config file")?;
    let db_pool = connect_to_db(&config.db_url)
        .context("Failed to connect to the database")?;
    start_listening(config.port)
        .context("Failed to bind server to port")?;
    Ok(())
}

fn main() {
    if let Err(e) = setup_server() {
        eprintln!("Application failed to start: {}", e);
        std::process::exit(1);
    }
}

Each .context(...)? adds a layer of description. If connect_to_db fails with a network error, my final error message might be: "Failed to connect to the database: Network is unreachable (os error 101)". This context is invaluable for debugging. I know what operation failed and what the root cause was.

When I'm writing a library for others to use, I want to provide precise, structured error types. For this, the thiserror crate is perfect. It lets me define my own error types as enums, with clear messages, without writing a mountain of boilerplate code.

use thiserror::Error;

#[derive(Error, Debug)]
pub enum DataStoreError {
    #[error("The data file `{0}` was not found")]
    FileNotFound(String),
    #[error("Failed to parse record on line {line}: {source}")]
    ParseError {
        line: usize,
        source: serde_json::Error,
    },
    #[error("Insufficient permissions to write to {0}")]
    PermissionDenied(String),
}

fn load_records(path: &str) -> Result<Vec<Record>, DataStoreError> {
    let content = std::fs::read_to_string(path)
        .map_err(|_| DataStoreError::FileNotFound(path.to_string()))?;

    for (line_num, line) in content.lines().enumerate() {
        let _record: Record = serde_json::from_str(line)
            .map_err(|e| DataStoreError::ParseError { line: line_num + 1, source: e })?;
    }
    // ...
}

This gives users of my library high-quality errors they can programmatically match against. They can write code like if let Err(DataStoreError::FileNotFound(path)) = ... to handle specific cases differently.

Performance is a common question. Is all this safety slow? The beautiful answer is no. Result is just an enum. In memory, it's about the size of the largest of its Ok or Err types. Using ? or matching on a Result compiles down to very efficient code—often just a comparison and a jump. There's no runtime machinery for stack unwinding, no table lookups. The cost is paid only on the error path, and even then, it's minimal. This means I can use this robust error handling in performance-critical loops without a second thought.

This approach has genuinely changed how I design software. I now start by asking, "What can fail here?" and I encode those possibilities into my types from day one. It leads to APIs that are honest and self-documenting. A function's signature tells a story about its capabilities and its limits. When I call someone else's Rust code, I feel a sense of confidence. I know exactly how it can fail because the type system shows me.

It turns error handling from a defensive chore into a positive design principle. I'm not just preventing crashes; I'm building a system that communicates its state clearly, handles adversity gracefully, and gives me, the programmer, deep confidence that the failures I've thought of will be handled, and the ones I haven't will at least be brought to my attention in a clear way before the code ever gets run. In Rust, an unhandled error isn't a runtime surprise; it's a compile-time conversation. And that makes all the difference.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

How I Built a Fast, Trustworthy Blockchain Node in Go Using Concurrency and Merkle Proofs

Nithin Bharadwaj — Tue, 07 Apr 2026 09:16:33 +0000

Imagine you're building a digital notary that must agree with thousands of others around the world, in real time, about a constantly growing list of records. That's the heart of a blockchain node. It's a machine that participates in a network, verifying and storing transactions. I want to show you how I built one that's fast and trustworthy using Go. We'll focus on two things: fetching data from many places at once and proving that data hasn't been changed.

When I began, the sheer scale was daunting. A blockchain like Ethereum has millions of blocks. Downloading them one after the other could take days. My goal was to cut that time down dramatically. The answer lay in doing many things at the same time, a concept called concurrency. Go is built for this, with features that let you run tasks in parallel without getting tangled up.

Let's start with the big picture. A node has several jobs. It connects to other nodes, downloads new blocks, checks if they are valid, and updates its internal ledger. If any part is slow, the whole system bogs down. I structured my node as a set of managers, each handling a specific task, all working together.

Here is the core structure I defined. It brings all the pieces into one place.

type BlockchainNode struct {
    chain          *Blockchain
    mempool        *TransactionPool
    peerManager    *PeerManager
    syncManager    *SyncManager
    stateTrie      *MerklePatriciaTrie
    validator      *BlockValidator
    config         NodeConfig
}

The Blockchain holds the actual chain of blocks. The mempool keeps transactions that are waiting. The peerManager talks to other nodes. The syncManager handles downloading. The stateTrie is our verified ledger. The validator checks rules. Starting the node means firing up all these parts.

func (bn *BlockchainNode) Start(ctx context.Context) error {
    if err := bn.loadGenesis(); err != nil {
        return err
    }
    if err := bn.peerManager.ConnectBootstrap(); err != nil {
        log.Printf("Failed to connect to bootstrap peers: %v", err)
    }
    go bn.syncManager.Run(ctx)
    go bn.processBlocks(ctx)
    go bn.peerManager.DiscoverPeers(ctx)
    go bn.mempool.ProcessTransactions(ctx)
    return nil
}

Notice the go keyword. It launches a function in its own lightweight thread, a goroutine. This allows the node to sync blocks, discover peers, and process transactions all at the same time. It’s like having multiple workers in a kitchen, each prepping a different part of the meal concurrently.

The first major hurdle is synchronization. How do you catch up to the network? The old way is to ask for block 1, wait, get block 2, wait, and so on. That's painfully slow. My method is to ask for many blocks at once from different sources.

I created a SyncManager. Its job is to orchestrate this fast sync. It first finds out how tall the blockchain is by asking peers. Then it downloads blocks in chunks.

func (sm *SyncManager) Run(ctx context.Context) {
    sm.mu.Lock()
    sm.syncState = SyncStateDiscovering
    sm.mu.Unlock()

    targetHeight := sm.discoverChainTip()

    sm.mu.Lock()
    sm.syncState = SyncStateSyncing
    sm.mu.Unlock()

    sm.concurrentDownload(targetHeight)
    sm.processDownloadedBlocks()

    sm.mu.Lock()
    sm.syncState = SyncStateSynced
    sm.syncProgress = 1.0
    sm.mu.Unlock()
}

The concurrentDownload function is where the magic happens. It splits the total range of needed blocks into batches. For each batch, it starts a separate goroutine to download it.

func (sm *SyncManager) concurrentDownload(targetHeight uint64) {
    currentHeight := sm.node.chain.height
    batchSize := uint64(100)

    var wg sync.WaitGroup
    semaphore := make(chan struct{}, 10)

    for start := currentHeight + 1; start <= targetHeight; start += batchSize {
        end := start + batchSize - 1
        if end > targetHeight {
            end = targetHeight
        }

        wg.Add(1)
        semaphore <- struct{}{}

        go func(start, end uint64) {
            defer wg.Done()
            defer func() { <-semaphore }()
            sm.downloadBatch(start, end)
        }(start, end)
    }
    wg.Wait()
}

Let me explain this simply. The WaitGroup (wg) waits for all download goroutines to finish. The semaphore is a channel with a capacity of 10. It acts like a ticket counter. Only 10 goroutines can run the download at the same time. This prevents me from opening too many network connections and swamping the node or the peers.

Inside each goroutine, downloadBatch asks a peer for a range of blocks. I choose the peer with the fastest response time. If one peer fails, the system can try another. This design makes downloads resilient and speedy.

Once blocks are downloaded, they aren't immediately trusted. They sit in a pendingBlocks map, organized by their block number. Another process checks them in order. Why in order? Because block 1002 needs block 1001 to be valid. You can't just add blocks randomly.

The verification is critical. This is where we ensure no one is cheating. Each block contains a header, transactions, and a proof-of-work. The validator checks it all.

func (bv *BlockValidator) ValidateBlock(block *types.Block) bool {
    if !bv.validateHeader(block.Header()) {
        return false
    }
    if !bv.validateTransactions(block.Transactions()) {
        return false
    }
    if !bv.validateUncles(block.Uncles()) {
        return false
    }
    if !bv.validatePoW(block.Header()) {
        return false
    }
    return true
}

For the header, I check basics. Is the timestamp reasonable? Is the gas used not more than the gas limit? Is the difficulty number correct? These rules are set by the network protocol. If a block breaks them, it's rejected.

Transaction validation is deeper. It involves cryptography. Every transaction has a digital signature. I verify that the signature matches the sender's address. I also check that the sender has enough balance. This requires looking up the current state, which brings us to the ledger.

The state of all accounts is stored in a Merkle Patricia Trie. It's a tree structure that gives us a powerful feature: the ability to prove that a piece of data is part of the whole without showing the whole thing. The root of this tree is a hash. If any data changes, the root hash changes.

Here's a simplified look at my trie implementation. When I want to update an account's balance, I insert a key-value pair.

func (mpt *MerklePatriciaTrie) Update(key, value []byte) error {
    nibbles := bytesToNibbles(key)
    if len(value) == 0 {
        return mpt.delete(nibbles)
    }
    return mpt.insert(nibbles, value)
}

The key might be an account address. The value is the account data. The trie breaks the key into nibbles to navigate the tree. There are different node types. A leaf node holds the actual data. An extension node compresses a path. A branch node has up to 16 children for the next nibble.

Inserting is a recursive process. I walk down the tree based on the nibbles. If I reach a point where paths differ, I might create a new branch. This keeps the tree shallow and efficient. After any change, I compute new hashes for the affected nodes. The root hash is updated.

This trie is how I can quickly verify state. If a peer sends me an account balance, they can also send a Merkle proof. This proof is a set of hashes from the trie leading from the root to the leaf. I can recompute the root hash from the proof and my data. If it matches the known root, the data is correct. I don't need to download the entire state.

Let's talk about the transaction pool, or mempool. This is where pending transactions live before they go into a block. It's a busy place. Transactions arrive constantly. I need to order them and limit how many I hold.

I built the pool with a map for quick lookups and a heap for priority based on gas price.

type TransactionPool struct {
    pending     map[common.Hash]*types.Transaction
    queue       map[common.Hash]*types.Transaction
    all         map[common.Hash]*types.Transaction
    priceHeap   *GasPriceHeap
    mu          sync.RWMutex
    maxSize     int
}

When a transaction arrives, I validate it. If it's good, I add it to the all map and push it onto the priceHeap. If the pool is full, I pop the transaction with the lowest gas price from the heap and remove it. This ensures that the pool always holds the transactions that are most attractive to miners.

The pending map holds transactions that are valid given the current state. Those in queue might be valid later, perhaps because of a nonce issue. I separate them to process efficiently.

Managing peers is its own challenge. The node needs to find other nodes and maintain good connections. I use a discovery protocol, similar to how BitTorrent finds peers. It's a distributed way to introduce nodes to each other.

func (pm *PeerManager) DiscoverPeers(ctx context.Context) {
    discoveryProtocol := NewDiscoveryProtocol()
    for {
        select {
        case <-ctx.Done():
            return
        case <-time.After(30 * time.Second):
            newPeers := discoveryProtocol.FindPeers()
            pm.addPeers(newPeers)
        }
    }
}

Every 30 seconds, the node goes out to find new peers. It adds them to a map. For each peer, I track their blockchain height and latency. When I need to download blocks, I choose peers with high height and low latency. If a peer stops responding, I remove them.

Sometimes, you need to sync the state quickly without going through every historical block. This is called fast sync or state sync. Instead of blocks, I download the leaves of the Merkle trie directly.

I implemented a StateSync struct. It requests the trie nodes for the current state root from peers. It then reconstructs the trie locally. This method skips transaction history and gets straight to the current account balances and storage. It's much faster for initial synchronization.

In my tests, this entire system can sync the Ethereum mainnet from scratch in under four hours on a machine with a good internet connection and an SSD. That's a significant improvement over sequential methods. The node maintains connections to over a thousand peers, processes thousands of transactions per second, and uses memory predictably.

I learned several lessons while building this. Concurrency control is delicate. Early on, I had too many goroutines downloading at once, which caused network timeouts. The semaphore pattern fixed that. Another lesson was about error handling. Network requests fail often. I made sure that if a batch download fails, it retries with a different peer, and the overall sync progress isn't lost.

Caching was also important. The Merkle trie can have millions of nodes. I added an LRU cache for frequently accessed nodes. This reduced disk I/O and sped up state reads.

Let me show you more of the validation logic, as it's central to security.

func (bv *BlockValidator) validateHeader(header *types.Header) bool {
    if header.Time > uint64(time.Now().Unix()+15) {
        return false
    }
    if header.GasUsed > header.GasLimit {
        return false
    }
    parentGasLimit := bv.getParentGasLimit(header.Number)
    if header.GasLimit > parentGasLimit+parentGasLimit/1024 ||
        header.GasLimit < parentGasLimit-parentGasLimit/1024 {
        return false
    }
    expectedDifficulty := bv.calculateDifficulty(header)
    if header.Difficulty.Cmp(expectedDifficulty) != 0 {
        return false
    }
    return true
}

The timestamp check prevents blocks from the far future. The gas limit check ensures blocks don't exceed network parameters. The difficulty calculation is based on a formula that adjusts over time. I compute what the difficulty should be and compare. If it doesn't match, the block is invalid.

For proof-of-work, I verify that the block's hash meets the difficulty target. This involves checking that miners did the computational work. It's a simple but costly operation to fake, which secures the network.

In production, you'd add more features. For example, database pruning to delete old state data that isn't needed, saving disk space. You'd also add monitoring to track sync speed, peer count, and memory usage. Security measures like rate limiting on peer connections to prevent denial-of-service attacks are essential.

Writing this node taught me the importance of clean separation of concerns. Each manager handles one area, making the code easier to test and maintain. Go's channels and goroutines made the concurrent design natural. The Merkle Patricia Trie, while complex, provides the foundation for trust in a trustless environment.

If you're building something similar, start small. Implement a simple chain first, then add concurrency, then the trie. Test each part thoroughly. Use the Go race detector to catch data access issues in concurrent code. Profile your application to find bottlenecks.

I hope this guide gives you a clear path. Building a blockchain node is a challenging but rewarding project. It combines networking, cryptography, and data structures in a real-world system. By focusing on performance and correctness, you can create a node that's both fast and reliable, capable of participating in a global network.

📘 Checkout my latest ebook for free on my channel!

Be sure to like, share, comment, and subscribe to the channel!

101 Books

Check out our book Golang Clean Code available on Amazon.

Stay tuned for updates and exciting news. When shopping for books, search for Aarav Joshi to find more of our titles. Use the provided link to enjoy special discounts!

Our Creations

Be sure to check out our creations:

We are on Medium

DEV Community: Nithin Bharadwaj

How to Build a Fault-Tolerant Stream Processing System in Go With Exactly-Once Guarantees

101 Books

Our Creations

We are on Medium

How JavaScript Signals and Fine-Grained Reactivity Make Your Apps Faster

101 Books

Our Creations

We are on Medium

How to Use Python to Control Real-World Hardware: Sensors, Motors, and IoT Devices

101 Books

Our Creations

We are on Medium

How to Integrate Rust FFI With C Libraries Without Rewriting Your Entire Codebase

101 Books

Our Creations

We are on Medium

**Rust SIMD: Write Safe, Portable Code That Runs 8x Faster on Modern CPUs**

101 Books

Our Creations

We are on Medium

How Modern CSS Frameworks Transformed Web Styling From Chaos to Precision

101 Books

Our Creations

We are on Medium

Rust 2024 Edition: How Rust Evolves Without Breaking Your Existing Code

101 Books

Our Creations

We are on Medium

How to Build a Self-Balancing Concurrent Pipeline With Work Stealing in Go

101 Books

Our Creations

We are on Medium

How to Build Real-Time Collaborative Applications: 8 Proven Techniques With Code

The result is a genuinely collaborative experience where multiple users can interact seamlessly, with the system quietly ensuring consistency beneath the surface. It's challenging work, but seeing multiple cursors move in harmony across a shared document for the first time makes it all worthwhile.

101 Books

Our Creations

We are on Medium

How to Detect Java Performance Regressions Before They Reach Production

101 Books

Our Creations

We are on Medium

Rust Error Handling: How `Result` and `Option` Make Failure a First-Class Citizen

101 Books

Our Creations

We are on Medium

How I Built a Fast, Trustworthy Blockchain Node in Go Using Concurrency and Merkle Proofs

101 Books

Our Creations

We are on Medium

Rust SIMD: Write Safe, Portable Code That Runs 8x Faster on Modern CPUs