The Mutex Never Unlocked Because Retry Slept Inside It

Retry does not belong inside a locked section. That single rule would have saved me an afternoon. An AI-generated C++ helper compiled, passed tests, and then froze a worker thread. This post is a lab reconstruction, not a production war story.

The conclusion first

A retry loop that catches everything hides the real fault. A mutex held across sleep looks like a deadlock. You will debug the hang for hours. You should have debugged the lock scope first.

What I asked the model to build

I wanted a tiny in-memory ticket gate. Each key maps to a remaining count. Callers take one ticket and continue. Transient failures should retry a few times. Was that request as harmless as it sounded?

I used free model access to draft the first class. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The generated code looked tidy and complete. Tidy C++ is often the first smell.

The reduced lab code

I stripped comments and kept the shape. This is the hanging version.

// lab reconstruction — not production code
#include <chrono>
#include <mutex>
#include <stdexcept>
#include <string>
#include <thread>
#include <unordered_map>

class TicketGate {
  std::mutex mu_;
  std::unordered_map<std::string, int> tickets_;

 public:
  void seed(std::string id, int n) {
    std::lock_guard<std::mutex> lock(mu_);
    tickets_[std::move(id)] = n;
  }

  template <class F>
  static auto retry(F&& fn, int attempts) {
    using namespace std::chrono_literals;
    for (int i = 0; i < attempts; ++i) {
      try {
        return fn();
      } catch (...) {
        if (i + 1 == attempts) throw;
        std::this_thread::sleep_for(50ms);
      }
    }
    throw std::logic_error("unreachable");
  }

  int take(const std::string& id) {
    std::lock_guard<std::mutex> lock(mu_);
    return retry([&] {
      auto it = tickets_.find(id);
      if (it == tickets_.end()) {
        throw std::runtime_error("missing ticket");
      }
      if (it->second <= 0) {
        throw std::runtime_error("empty ticket");
      }
      return it->second--;
    }, 5);
  }
};

Do you already see the trap in take? The lock wraps the whole retry. The catch path sleeps. No other thread can enter that map.

Symptom: a quiet hang

I wrote a happy-path test first. It passed on the first clean run. Missing keys never appeared in that first fixture. Those green tests made me trust the wrapper. Should those green tests have earned that trust?

Then I added a second thread. One thread takes a bad key, and one thread takes a good key. The good caller never returned a ticket at all. CPU stayed near zero the whole time. Logs stopped after the first retry line. How do you debug a process that says nothing?

Root cause, in six moves

1. take locked mu_ for the entire retry.
2. A missing key threw std::runtime_error immediately.
3. catch (...) treated that error as retryable work.
4. sleep_for ran while the same mutex stayed owned.
5. The second thread blocked on mu_ and waited.
6. Nothing could insert the missing key during sleep.

The model assumed retry was always a safe wrapper. It assumed every exception was a transient blip. It assumed holding a lock across backoff was cheap. Those three bad assumptions produced a single hang. Which assumption would you have caught in review?

Reusable debug steps

Do not start with extra logging statements here. Start with lock ownership and the exception class.

1. Name the lock scope on paper

Draw one box for the mutex lifetime. Then put every nested call inside that box. If sleep_for sits inside the box, stop now. You already found the bug without a debugger.

2. Build a two-thread hang repro

One thread takes a bad key, and one thread takes a good key. Give the whole process a short hard timeout. A hang that cannot fail will return tomorrow.

# lab commands — adjust paths on your machine
c++ -std=c++17 -pthread -g -O0 ticket_gate_lab.cpp -o ticket_gate_lab
timeout 3s ./ticket_gate_lab
echo $?

A timed-out exit is the first honest failure. Record that status before you change code.

3. Turn on thread and undefined sanitizers

Rebuild the same source file with sanitizers enabled. ThreadSanitizer may stay quiet on this hang. That quiet result still teaches you a lesson. This hang is lock convoy, not a data race.

c++ -std=c++17 -pthread -fsanitize=thread -g -O1 ticket_gate_lab.cpp -o gate_tsan
c++ -std=c++17 -pthread -fsanitize=undefined,address -g -O1 ticket_gate_lab.cpp -o gate_asan
timeout 3s ./gate_tsan; echo tsan:$?
timeout 3s ./gate_asan; echo asan:$?

Did the sanitizers print a useful stack trace? If not, your next tool is scope, not more flags.

4. Classify each exception before retry

Ask one hard question at every throw site. Can waiting overnight change this failing condition? If the map cannot gain a key during sleep, retry is theater.

Exception	Retry?	Why
missing ticket	no	waiting cannot insert the key
empty ticket	no	the count will not refill itself
short read / EINTR	maybe	the cause can vanish
std::bad_alloc	no	sleep does not create memory
catch (...) unknown	no	you just deleted the only clue

Would you still wrap take after reading that table? After that table I would not wrap it.

5. Move retry outside the lock

The fix is structural, not a clever trick. Lock only around the actual map update. Do not sleep, and do not catch every type.

int take(const std::string& id) {
  std::lock_guard<std::mutex> lock(mu_);
  auto it = tickets_.find(id);
  if (it == tickets_.end()) {
    throw std::runtime_error("missing ticket");
  }
  if (it->second <= 0) {
    throw std::runtime_error("empty ticket");
  }
  return it->second--;
}

If a caller truly needs backoff, it retries take itself. It must retry without holding mu_ at all. Keep retry policy outside the gate class.

6. Prove the hang cannot return

Add a focused test with a real deadline. If the good thread cannot join, fail CI.

// lab test — two threads, one bad id
TEST(TicketGate, BadIdDoesNotStallGoodId) {
  TicketGate gate;
  gate.seed("ok", 1);

  std::atomic<int> got{-1};
  std::thread bad([&] {
    try {
      gate.take("nope");
    } catch (...) {
    }
  });
  std::thread good([&] {
    got.store(gate.take("ok"));
  });

  good.join();
  bad.join();
  ASSERT_EQ(got.load(), 1);
}

Set a hard test timeout in the runner. A join without a timeout is another silent hang. Did your CI already hide this class of bug?

What the free compile box was for

I needed a second compiler invocation with sanitizer flags. My local run already showed the hard timeout. A clean remote rebuild still had real value. MonkeyCode's free server option rebuilt the same lab files. Free model access drafted the first retry wrapper. I rejected most of that wrapper after the hang.

The model does not own your lock scopes. You still own every lock scope in this class. Do not treat a remote compile as proof. Treat it as a second set of flags.

If the hang reproduces there, the bug is in the source. If it does not, check timing and thread startup.

Limitations

This lab uses C++17 and a plain std::mutex. It does not use timed locks or network jitter. It does not claim any speedup number. It does not name models, quotas, or machines.

Those extra claims go stale very fast. The lock-scope rule does not go stale. Sanitizers can change thread scheduling under load.

A hang can shrink or stretch under sanitizers. A shell timeout is not a liveness proof. Treat this article as a debugging drill only.

Who should skip this approach

Skip it if your code has no shared mutex. Skip it if retries talk to a real network.

Skip it if you need formal deadlock detection. Skip it if you cannot add a failing test. A blog repro is not a lock graph.

Also skip AI-generated retry helpers for semantic failures. Missing map keys are not transient brownouts. Empty counters are not lost network packets.

Retrying them only burns wall clock time. Why wrap a function that cannot succeed later?

Sticky note I now keep

1. Never sleep while a mutex is owned.
2. Never catch (...) inside a library helper.
3. Never retry a condition waiting cannot change.
4. Always add a second thread before trusting a gate.
5. Always give the test runner a timeout.

Would I ask a model for boilerplate again? Yes, but only for the first draft.

Would I paste that first answer into a mutex class? I would not paste it after this hang. The useful part was the hanging failure. The reusable part was the debug checklist.

If you reconstruct this lab, compile the two-thread test first. Then delete the retry helper from the class. Then decide whether any retry belongs in that class.

Need a second compile box for sanitizer builds? The free server option is enough for this drill.