How I Refactored a Distributed Hardware Control System into a Reusable Framework

Building software for industrial machines taught me an unexpected lesson: scaling isn't about adding more hardware, it's about reducing architectural complexity.

A few years ago, I led the development of a distributed hardware control system that orchestrated multiple industrial machines. The master controller coordinated production, while a React dashboard gave operators real-time visibility through WebSockets. Under the hood, we were running a fleet of Raspberry Pis, each communicating with an Arduino Due running the g2core motion control firmware.

At first, everything seemed straightforward. But as we added more robotic cells, our software began to buckle under the weight of its own abstractions.

This is the story of how a simple collection of control scripts morphed into an overly generic, bloated framework, and how I eventually had to tear it all down to build a clean, domain-driven architecture.

---

1. The Day Scripts Stopped Scaling

When you are controlling physical motors, early success is intoxicating. You send a JSON payload over TCP, a stepper motor whirs to life, and a robotic arm moves exactly 150 millimeters to the right.

But distributed systems have a funny way of punishing organic growth. As we expanded the production line by adding six more robotic cells, bringing the system to around sixteen distributed nodes, the software architecture began showing its limits. We experienced dropped TCP connections, skipped heartbeats, and UI state that desynced from the physical hardware.

The real challenge wasn't controlling motors anymore. It was controlling complexity.

To understand why the system started fighting us, you first have to understand the hardware stack we were building on.

2. Understanding the Hardware Stack

Industrial environments require strict separation of concerns between high-level orchestration and low-level real-time execution. Our stack looked like this:

• HMI (Human-Machine Interface): A React frontend communicating via WebSockets for real-time monitoring and HTTP for configuration.
• Master Controller: An asynchronous Python application orchestrating the entire factory floor.
• Raspberry Pi (Slave): A local edge computer for each machine, running a Python TCP server.
• Arduino Due: The real-time microcontroller running g2core, translating high-level JSON/G-code into precise electrical pulses for the motor drivers.

This hardware separation is standard. The architectural problem was how we mapped our software onto it.

3. The First Design: Simple, But Naive

For the first version of the system, we kept the slave software intentionally dumb. The Raspberry Pi exposed a TCP server, accepted JSON commands, forwarded them to g2core over a serial port, and returned the results.

It looked something like this:

# Early design: Networking, request routing, and hardware control
# all live inside a single class.
class DeviceServer:
    def __init__(self, serial_port):
        # Open a direct serial connection to the Arduino/g2core firmware.
        self.ser = serial.Serial(serial_port, 115200)

        # Simple command router: map incoming JSON commands
        # to Python handler functions.
        #
        # This worked well initially, but as more commands and
        # workflows were added, this dictionary became the center
        # of the entire application.
        self.COMMAND_HANDLER = {
            "raw": self._handle_raw,
            "G1": self._handle_move,
            "get_status": self._handle_status,
        }

    async def handle_client(self, reader, writer):
        # Receive a JSON request from a TCP client.
        data = await reader.read(1024)
        request = json.loads(data.decode())

        # Find the corresponding command handler.
        command = request.get("cmd")
        handler = self.COMMAND_HANDLER.get(command)

        if handler:
            # Execute business logic, which often communicates
            # with the Arduino over the serial port.
            response = handler(request["payload"])

            # Send the result back to the client.
            writer.write(json.dumps(response).encode())
            await writer.drain()

    def _send_serial(self, data):
        # Send a command to the Arduino/g2core controller.
        self.ser.write((data + "\n").encode())

        # ⚠️ Problem:
        # This blocks until the Arduino responds.
        #
        # Although the server uses asyncio, every command that
        # waits here prevents this request from making progress.
        # As more machines and longer-running operations were added,
        # serial communication became a scalability bottleneck.
        return self.ser.readline().decode()

The Original Request Flow

Simple. Easy to understand. Easy to debug. For one machine, it was exactly the right architecture.

4. The Architectural Smells

As the system grew, this simple script became a nightmare. We encountered problems that every distributed system eventually faces:

1. Blocking I/O Killed Concurrency: Look at _send_serial in the snippet above. It calls ser.readline(). Because serial communication is slow, this blocked the entire asynchronous event loop. While the Pi waited for the Arduino to confirm a motor move, it couldn't respond to network heartbeats. The Master Controller assumed the Pi was dead and initiated a fault recovery.
2. State Lived Everywhere: If an operator pressed "Pause" on the HMI, where did that state live? The Controller had a is_paused flag. The Pi TCP server had a state = PAUSED variable. The Arduino also had its own internal motion planner state. Keeping these three in sync across unreliable TCP connections resulted in massive race conditions.
3. Workflow Leaked into Infrastructure: We started adding retry logic, pause/resume handling, and fault recovery directly inside the socket handlers. Our networking code was suddenly making business decisions.

5. The Wrong Refactor: Abstract Everything

Like many engineers facing a messy codebase, my instinct was to extract every recurring pattern into its own module. I decided to build a "Hardware Control Framework."

The directory structure gradually looked like this:

framework/
├── cycles/        # Lifecycle managers
├── events/        # Global event bus (Pub/Sub)
├── hardware/      # Base classes for hardware
├── mixins/        # StateMachineMixin, RetryMixin, etc.
├── network/       # Abstracted TCP/UDP logic
├── serial/        # Serial communication wrappers
└── system/        # Global singletons

At first glance, this looked like progress. Everything was abstract. Everything was reusable.

But I had fallen into a classic trap: The framework had started solving problems that didn't actually exist.

I had heavily utilized mixins and deep inheritance. A single RoboticArm class inherited from HardwareBase, StateMachineMixin, and NetworkNode. When a TCP packet failed to send, the stack trace hopped through 15 different abstract files.

Some abstractions represented real domain concepts. Others existed simply because I could create them. They made the framework larger without making the architecture clearer.

6. Designing the Architecture I'd Build Today

Looking back, I realized the project contained three completely different kinds of software:

1. Infrastructure: TCP sockets, connection pooling, heartbeats.
2. Device Runtime: Serial communication, g2core protocols, hardware limits.
3. Production Logic: Recipes, machine coordination, calibration steps.

If I were starting over, I would abandon the "general-purpose framework" approach and strictly organize the software around domain responsibilities.

The Refactored Architecture

Here is why each component exists and the specific problems it solves:

Why the Controller Exists

The controller orchestrates the system. It starts production scripts, manages the lifecycle, and enforces that only one production workflow runs at a time.

• The Problem it Solves: Without it, workflow ownership is unclear, and pause/resume logic spreads across the codebase.
• The Constraint: It doesn't know anything about serial ports, motor drivers, or sockets. It coordinates. Nothing more.

Why the Machine Agent Exists

Each physical machine becomes an object with ownership over its own state on the master server.

• The Problem it Solves: Instead of scattering networking code throughout the application, the MachineAgent owns the TCP client, the status cache, and high-level operations.
• The Result: The caller never thinks about sockets or retries. Production scripts just call machine.rotate(10) or machine.home(). The agent encapsulates the network complexity.

Why the Job Queue Exists (The Game Changer)

Industrial machines require deterministic execution. Motion commands cannot execute concurrently just because they're asynchronous in software. If you send a "Move X" and "Move Y" command at the same time, the hardware might attempt to interpolate them, crash, or ignore one entirely.

Each edge device now owns a strict, sequential Job Queue.

This simple rule removed an entire class of race conditions. Concurrency is handled by the transport layer (so heartbeats can still be processed), but hardware execution is strictly sequential.

Why the Transport Layer is Isolated

Communication deserves its own module. Nothing more. Nothing less. It handles request/response tracking, the JSON protocol, connection pooling, and fault recovery. It is not a service discovery framework, and it does not know what a "motor" is.

Why Recipes are Separated from Workflows

In the old system, scripts contained hardcoded values. In the new architecture, Recipes are configuration, and Scripts are execution. A recipe stores parameters (e.g., target_temperature = 200, feed_rate = 50). Production scripts decide how to use them. Keeping data and logic separate makes both trivially easy to test.

7. Why This Design Scales Better

This domain-driven approach fundamentally changed the stability of our system.

1. Fault Isolation: When a serial cable was unplugged, the Device Runtime threw an error. The Job Queue paused. The Transport layer cleanly reported the error back to the Machine Agent. The Controller saw the error and halted the workflow. No hanging sockets, no deadlocks.
2. No More God Objects: By favoring composition over inheritance, we deleted all our Mixins. A Machine doesn't inherit from a NetworkNode; a Machine has a NetworkNode.
3. True Asynchronous I/O: By separating the network transport from the hardware execution queue, the Pi could instantly respond to a "status check" or "heartbeat" over TCP, even if the motors were in the middle of a 30-second physical movement.

8. Lessons Learned: Design Principles I Follow Today

When engineers hear the word "framework," it's tempting to think about flexibility and abstraction. But experience has taught me that frameworks should emerge from domain concepts, not from implementation details.

If I were starting this system over today, I would follow these seven rules:

1. The Master orchestrates; the Slave executes. Never put business logic on the edge device.
2. Scripts describe workflows; recipes hold data. Don't hardcode production logic.
3. Machines own their own state and queues. Protect your hardware with strict, sequential execution queues.
4. Prefer composition over deep inheritance or mixins. Avoid the trap of the "clever" object-oriented hierarchy.
5. Create new modules only when a second real use case appears. Don't preemptively abstract.
6. Wire dependencies explicitly at startup instead of relying on global singletons.
7. Keep infrastructure invisible to business logic whenever possible.

These principles aren't specific to robotics or mechatronics. You could replace the Arduino with a cloud database, the stepper motors with background worker tasks, and the Raspberry Pi with a microservice. The architectural boundaries remain exactly the same.

The most valuable part of this project wasn't writing code that controlled motors. It was learning to separate orchestration, communication, and device control. Ironically, the final architecture contained fewer modules than my original "framework", yet it explained the system much more clearly.

Sometimes the best refactoring isn't adding another abstraction. It's deleting one.

---

I'd love to hear how you've approached similar problems. Have you ever built a framework that later became too generic? Or simplified an architecture by removing abstractions instead of adding them? Let's discuss it in the comments below.