High-Throughput IoT Log Aggregator

#go #performance #testing #productivity

Imagine an industrial monitoring system receiving telemetry packets from thousands of sensors every second. The system must:

Ingest a batch of raw data packets.
Filter inactive sensors.
Aggregate temperature readings by Device ID.
Generate a textual summary log for the dashboard.

The Challenge: Since this runs continuously, any inefficiency (like unnecessary memory allocation) will cause "Garbage Collection" pauses, causing data loss. We must use memory-efficient patterns.

We used Go language for this performance test

System Flow Diagram

    A[Raw Data Ingestion] -->|Slice Pre-allocation| B(Batch Processing)
    B -->|Value Semantics| C{Filter Inactive}
    C -->|Map Pre-allocation| D[Aggregation]
    D -->|strings.Builder| E[Log Generation]
    E --> F[Final Report]

Code Segment	Optimization Technique	Why it matters in this scenario?
`type SensorPacket struct`	Struct Alignment	Millions of packets are kept in RAM. Saving 8 bytes per packet saves ~8MB of RAM per million records.
`make([]SensorPacket, 0, n)`	Slice Pre-allocation	The simulation loads 100,000 items. Without this, Go would resize the array ~18 times, copying memory each time.
`make(map[int32]float64, 100)`	Map Size Hint	We aggregate by device. Allocating buckets upfront prevents expensive "rehashing" when the map fills up.
`var sb strings.Builder`	String Builder	Generating the report log involves many string additions. Builder prevents creating hundreds of temporary trash strings.
`func processBatch(...)`	Value vs Pointer	`config` is passed by Value (fast stack access). The `Report` is returned by Pointer (avoids copying the big map).

Optimized (Current Code):

[ Timestamp (8) ] [ Value (8) ] [ DeviceID (4) | Active (1) | Pad (3) ]
Total: 24 Bytes / Block

Unoptimized (If mixed):

[ Active (1) | Pad (7) ] [ Timestamp (8) ] [ DeviceID (4) | Pad (4) ] [ Value (8) ]
Total: 32 Bytes / Block (33% Wasted Memory!)

Example Results

--- Processing Complete in 6.5627ms --- --- BATCH REPORT --- Batch ID: 1766689634 Device 79: Avg Temp 44.52 Device 46: Avg Temp 46.42 Device 57: Avg Temp 45.37 Device 11: Avg Temp 44.54 Device 15: Avg Temp 46.43 ... (truncated)

📊 Benchmark Results

The following benchmarks compare the performance of "naive" implementations versus "optimized" patterns using Go's best practices. The tests were run on an Intel Core i5-10300H CPU @ 2.50GHz.

Operation Type	Implementation	Time (ns/op)	Memory (B/op)	Allocations (op)	Performance Gain
Slice Append	Inefficient	66,035 ns	357,626 B	19	-
	Efficient (Pre-alloc)	15,873 ns	81,920 B	1	~4.1x Faster
String Build	Inefficient (+)	8,727 ns	21,080 B	99	-
	Efficient (Builder)	244.7 ns	416 B	1	~35.6x Faster
Map Insert	Inefficient	571,279 ns	591,485 B	79	-
	Efficient (Size hint)	206,910 ns	295,554 B	33	~2.7x Faster
Struct Pass	By Value (Copy)	0.26 ns	0 B	0	-
	By Pointer (Ref)	0.25 ns	0 B	0	Similar

Note on Structs: In micro-benchmarks with tight loops, the Go compiler heavily optimizes (inlines) function calls, making the difference between Value and Pointer negligible. However, in real-world applications with complex call stacks, passing large structs by Pointer significantly reduces CPU usage by avoiding memory copy operations.

💡 Key Takeaways
String Concatenation: Never use + in loops. The strings.Builder approach is over 35 times faster and uses 98% less memory because it avoids creating intermediate garbage strings.

Memory Pre-allocation: Telling Go how much memory you need upfront (for Slices and Maps) eliminates the overhead of resizing and copying data (Rehashing/Reallocation).

Slice: Reduced allocations from 19 to 1.

Map: Reduced allocations from 79 to 33.

Allocations matter: Every allocation (allocs/op) forces the Garbage Collector to work harder. Keeping this number low makes the entire application more stable and responsive.