I am a long-time Symfony developer. I love the framework. I use it in countless projects—whether they are large corporate systems, smaller work tasks, or my own private pet projects. When the Symfony Messenger component was introduced years ago, I instantly fell in love with it and deployed it pretty much everywhere.
Everything worked like a Swiss watch. At least, that’s what I thought... until recently.
This article is a confession about how I spent three months solving two seemingly completely different, critical issues that ultimately shared one silent killer: untamed asynchrony. And also about how I tore my hair out and lost half my beard while debugging, all while my colleagues were cursing me.
Case One: The Mystery of the Evening Attachment Disappearances
We have a robust ticketing system. It works simply: a user creates a ticket and adds attachments. To avoid making the user wait for the upload, the attachments are processed asynchronously via Messenger, which moves them to Azure folders in the background.
Three months ago, the first report came in: "Sometimes, an attachment gets lost after creating a ticket."
When you looked at the data, you'd find an interesting anomaly. It mostly happened in the evening hours, around 9 PM. The cat-and-mouse game began. We spent a long time examining the code, going through the frontend JavaScript, but we found nothing. So we started adding logging. When that didn't help, we added more logs. And even more.
Finally, after weeks of frustration, we found the crime scene. In our Message Handler, we had this innocent-looking piece of code:
$forum = $this->emRds->getRepository(Forum::class)->find($message->getForumId());
if (!$forum) {
$logger->log("Forum not found", context: [
"id" => $message->getForumTicketFileId(),
"forumId" => $message->getForumId(),
]);
return null; // Here is the silent killer
}
Why was it there? A protective mechanism. If someone deleted the forum/ticket in the meantime, we didn't want the handler trying to assign files to a non-existent record and throwing exceptions (crashing). Logical, right?
But what was actually happening at 9 PM?
At 9 PM, almost nobody is working in the system. The server is completely idle and has plenty of free resources. When someone created a ticket at that time, Messenger processed the Message instantly. And by instantly, I mean so brutally fast that the handler spun up and finished before the original web request even had time to complete and commit the database transaction with the new forum!
The handler was looking for the forum in the database, but it physically wasn't there yet (the transaction wasn't closed). So the code said to itself: "Ah, the forum doesn't exist, someone probably deleted it. OK, I'm discarding the attachment and exiting." By the time I figured this out, my colleagues—who had spent three months listening to client complaints about lost files—wanted to defenestrate me.
Case Two: The Phantom of the Inventory Documents
As if that weren't enough, I was running into another problem in parallel, in a completely different part of the system.
We process inventory documents. Our CRON connects to a good old Windows SFTP server, downloads an XML file, and sends it (guess via what)—yes, via Messenger—for asynchronous processing.
Once again, a bug was reported: "We are losing inventory documents!"
I look into our monitoring system... no errors, no failed messages. Everything was flashing green.
Round two of the cat-and-mouse game began. I looked for bugs in the code. I gradually added logs. Logging these kinds of asynchronous errors is pure hell. You always forget to put a log exactly where it's needed. So you add it two days later, deploy, and wait for the error to reproduce. In the meantime, you forget about the task because you have a million other things to do, and two weeks later you realize: "Oh, the code can also branch over here, I need to add a log here too."
What was the resolution?
We were connecting to a free version of a SolarWinds SFTP server. This version had one funny limitation: it allowed only one single active connection at any given time. There was no log about this on the server.
My PHP script tried to connect, but if another process was already holding the connection, the SFTP silently rejected it. And what did my code do in the Handler?
To be "defensive," before processing, I double-checked via SFTP just to be sure: "Does this XML file still exist on the server?" Since the connection failed due to the limit, the function returned that the file wasn't there.
And my logic? "Ah, the file isn't there. Probably some other system or an external person deleted it. That's a legitimate use-case. I'm discarding the message, return null."
BOOM. Another critical bug, another three months of stress, lost documents, and my beard half as thick. Basically, it was the exact same mental lapse as the first problem: I ignored a missing resource because I assumed a conscious user action.
The Common Denominator and the Solution
Both problems were of the exact same nature. In both cases, I created a graceful fallback for a situation I thought was a user-caused edge-case (deleting a record). I never dreamed that one problem was caused by excessive server speed and the other by the technical limits of a 1-connection SFTP.
How did I solve it? Very simply.
In both cases, I just turned off asynchrony and let them process synchronously.
For the first problem with the attachments, it could have been solved using a DelayStamp (so the message waits a bit for the DB transaction to complete), but we needed the attachment to be available in the system immediately after the page refresh. Synchronous processing solved that perfectly.
The second case with the SFTP couldn't be solved any other way. Since the server allowed only one connection, parallelization and asynchrony were a bad idea from the very beginning.
Final Takeaways
Symfony Messenger is a truly great servant. It can incredibly speed up application response times and tuck heavy logic away into the background. But it is a damn fast and powerful tool.
If you hand it code that silently "swallows" Not Found states without accounting for Race conditions or the limits of external infrastructure, get ready to spend long nights staring at empty logs.
And if you ever meet me and notice I'm missing a chunk of my beard... at least now you know why. Blame it on asynchrony.
Do you have similar horror stories with asynchronous processing? Share them in the comments so I know I'm not alone in this!
Top comments (0)