The last few days I had been receiving errors when a new user would sign up to my site. It was a deserialization error that wasn't actually causing any problems. Someone would sign up and after_create they would receive a welcome message. I decided to take a look at Google to see if I could find a fix for it.
What was happening was sidekiq was trying to complete the welcome message job so fast that it was running into errors as all of the information wasn't available in the database just yet. The fix I saw basically had the same explanation that made sense but I didn't look at the context of the original poster. They weren't sending welcome messages. Their fix was to change after_create to after_commit. As you may know, after_create happens a total of once per user, after_commit has no limits, it fires every time that user record is committed into the database. DOH!
The issue was found quickly and my users were apologized to and notified that this didn't affect any of their customers. It was definitely a reminder to slow down and take an extra few minutes to think about what you're about to deploy. Especially when you're the only check and balance and don't have a team to give you a quick "dummy check" as we called it in the Air Force.
It's never fun messing something up, even if it was something that resulted in a customer needing to delete a few emails. Regardless it was a learning experience I thought I would share.
Have an awesome week.
Top comments (8)
Glad it got worked out!
Definitely! Glad it was only a few extra welcome emails to my customers and not a bunch of emails to my customers customers.
btw we ran into the same Sidekiq/
after_createrealization a little while back, luckily we switched directly toafter_create_commit.I know what I’m doing after I’m done eating drinking this glass of orange pineapple juice! Don’t think I’ve seen after_create_commit. Thanks!
I think you should use after_create_commit, as after_commit will get called for every update operation. If you end up updating the signup record in any situation, you might send a duplicate notification if you don't have any conditions to check whether is it a new record or not while sending notification. or use like this
after_commit :do_foo_bar, on: [:create]Thanks. Yeah after_commit was a mistake, hence the title of my article. I swapped to after_commit_create after Ben pointed it out.
It is an irony that this might not be reproducible always in the production server.
If the sidekiq is slow enough, this might not happen always as the request might have already finished and the transaction is committed to DB.
We have faced a similar issue related to sidekiq. We queued a job to sidekiq with the model primary key. inside sidekiq we will fetch from DB using PK and we will do the heavy operation (We don't want to push the entire object to sidekiq as it was costly in our case). Sometimes we delete the record even before it is getting processed by sidekiq. during this operation, it started throwing null pointer exception.
adding one more thing to your issue,
If the database is set to 'Read Uncommitted', you might not face the same issue. But most of the system won't have that transaction isolation setting as it is the lowest of all isolation level and has a lot of problems in the real-world application.
Yeah sidekiq is surprisingly fast. I was regularly running into the issue although it wasn’t actually an issue. Just delayed an email an extra 30 seconds or so. On my service I send all sorts of things directly to sidekiq. Most of which are webhooks so they’re not huge. I’m going to start adding more analytics so I’ll either just continue to listen to a bunch of webhooks or ping the API every hour or so to update the metrics for my customers. Both of which I’ll have sidekiq handle. As with the database and reading uncommitted things the only issue I was having was with new users being added. We’ll see as the load increases whether that sort of issue appears again.