The 3% Drop That Only Showed Up on a Clean Server: A Debugging Retrospective

My forwarder lost 3% of its messages. My laptop reported zero. CI reported zero. Staging lost thousands every hour.

I blamed the network. I blamed the VM. I blamed everything except my read loop.

This is the retrospective: one wrong assumption, three counters, a twelve-line fix.

The Symptom

The service was a small TCP forwarder. Accept a connection, read text lines, forward them. Nothing exotic.

Staging showed the loss: 97,004 lines in, 94,183 lines out. No errors. No exceptions. Just missing messages.

Replays on my laptop always passed. 100,000 lines in, 100,000 lines out.

That mismatch is the first clue. I was not testing what production actually did.

When a bug only lives in one environment, do you fix the environment first? Or the assumption your tests never stressed?

Step 1: Reproduce Outside Your Laptop

Before touching code, I changed only the machine. Same binary, same load script, same counters.

I started a fresh server from the free server option of MonkeyCode, an open source AI coding project.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The loss reappeared. 100,000 lines in, 97,128 lines out. The environment changed the probability, not the existence of the bug.

A second machine is a microscope for timing bugs. A fresh one has no history and no bias.

Step 2: Count, Don't Log

Logs tell stories. Counters add up. I placed a counter at every layer.

1. Client counts lines sent.
2. Server counts newlines parsed.
3. Server counts replies written.
4. Client counts replies received.

Run once, compare four numbers. The gap between them names the losing layer.

Counter gap	Suspect
parsed < sent	read loop or buffer size
replies < parsed	reply logic (this bug)
received < replies	client read logic

The clean-server run was loud: sent 100,000, parsed 100,000, replies 99,987, received 99,987.

The server parsed everything. It just did not reply enough. That gap is the fingerprint.

Step 3: The Reproducer

Smallest code that fails. One socket, one buffer, no frameworks.

load.py - burst mode, no waiting between sends:

# load.py - burst mode. No waiting between sends.
import socket

N = 100_000
s = socket.create_connection(("127.0.0.1", 9000))
s.settimeout(5)
acks = 0
try:
    for _ in range(N):
        s.sendall(b"PING\n")
    while acks < N:
        data = s.recv(65536)
        if not data:
            break
        acks += data.count(b"\n")
except TimeoutError:
    pass
print(f"sent={N} acks={acks} lost={N - acks}")

bad_server.cpp - one reply per read, not per line:

// bad_server.cpp - one reply per read, not per line.
#include <arpa/inet.h>
#include <netinet/in.h>
#include <sys/socket.h>
#include <unistd.h>

#include <cstdio>

int main() {
  int listen_fd = socket(AF_INET, SOCK_STREAM, 0);
  int one = 1;
  setsockopt(listen_fd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof one);

  sockaddr_in addr{};
  addr.sin_family = AF_INET;
  addr.sin_port = htons(9000);
  addr.sin_addr.s_addr = INADDR_ANY;
  bind(listen_fd, (sockaddr*)&addr, sizeof addr);
  listen(listen_fd, 64);

  char buf[256];
  long parsed = 0;
  long replies = 0;
  for (;;) {
    int c = accept(listen_fd, nullptr, nullptr);
    if (c < 0) continue;
    for (;;) {
      ssize_t r = read(c, buf, sizeof buf);
      if (r <= 0) break;
      for (ssize_t i = 0; i < r; ++i)
        if (buf[i] == '\n') parsed++;
      write(c, "ok\n", 3);  // one reply per read, not per line
      replies++;
    }
    close(c);
    fprintf(stderr, "parsed=%ld replies=%ld\n", parsed, replies);
  }
}

Look at the reply line. One read can return many lines.

TCP is a byte stream, not a message queue. read() has no idea what a PING is. The kernel may merge ten messages into one segment, and one read consumes all ten.

My polite local test sent one PING, waited for the ack, sent the next. One message per segment. The bug stayed asleep.

The staging load was a burst. Thousands of messages per second, batching inside the kernel. One read swallowed a hundred lines, the server answered once.

Test shape decides which bugs survive. Polite tests hide pipelining bugs. Burst tests expose them.

Step 4: Fix the Boundary, Not the Byte

The small fix replies per newline instead of per read.

for (ssize_t i = 0; i < r; ++i) {
  if (buf[i] == '\n') {
    parsed++;
    write(c, "ok\n", 3);
  }
}

The robust fix also survives a line split across two reads. That is the other half of the same bug family.

std::string acc;
ssize_t r;
while ((r = read(c, buf, sizeof buf)) > 0) {
  acc.append(buf, static_cast<size_t>(r));
  size_t pos;
  while ((pos = acc.find('\n')) != std::string::npos) {
    acc.erase(0, pos + 1);
    parsed++;
    write(c, "ok\n", 3);
  }
}

After the fix, both environments produced identical numbers. sent == parsed == replied == received.

What I Would Do Differently

• Change one variable at a time. In the first hour I changed the binary, the server, and the load. Wasteful.
• Write the burst test first. The polite test validated my hope, not the system.
• Audit syscall contracts with a second pair of eyes. I asked MonkeyCode's free model access to review the read loop. It flagged the one-reply-per-read pattern and pointed me to man 2 read. The man page says: "read() attempts to read up to count bytes." It never promises one message. Verify that claim at man7.org.

Limitations

This workflow targets timing and batching bugs. Data bugs need property-based tests, not a second machine.

Free servers are run-to-failure tools, not persistent hosts. Do not store state there, do not run compliance workloads, and do not benchmark the server itself. Use one for a clean slate and one reproduction.

The fix changed reply counts, not throughput. Measure both after touching a read loop.

The Takeaway

Your local test is not your load test. Your laptop is not your server. One read is not one message.

Start your next local-only bug with a burst client and a second machine. Then count at every layer.

The counters know the truth. Listen to them.