DEV Community: Adam Tal

From Production Outage to the Front Page of Hacker News

Adam Tal — Sat, 20 May 2023 20:48:45 +0000

Hello, everyone! I'm Adam, currently the Director of Payments Engineering at Vimeo. It's graduation season, meaning many are about to embark on their own professional journeys. It's an exciting time, and in the spirit of this, I'd like to share a story from my career, which spans over a decade in engineering.

We often celebrate our wins and milestones, as we should, but we tend to gloss over the challenging times, the missteps, and the lessons they bring. I've experienced my fair share of these lessons; I've brought down production at every company I've worked for at least once.

While that might sound alarming, it's in those moments of crisis that I've learned the most, grown exponentially as an engineer, and come away with valuable insights.

Today, I'd like to share one such story.

To set the stage, I was leading the Platform Engineering team at a startup credit card company. Buried in refactoring work, I discovered an empty Database (DB) table named "cards." It was a mystery – not referenced anywhere in the code, devoid of data, and mentioned in dusty old documents earmarked for deletion.

After discussing it with colleagues and doing some due diligence, I decided to axe it. A DB migration to drop the table was prepared and launched. A simple enough task. But as we're aware in tech, simple rarely equates to trouble-free.

Within 15 minutes post-deployment, our production environment buckled and crashed. A seemingly innocuous migration had spiraled into an engineering catastrophe. As the orchestrator of this chaos, I was now on a mission for answers.

Diving into DB logs and audit trails, the puzzle began to unravel. The empty table had a Foreign Key (FK) to another table. To drop it, Postgres needed to lock the referenced table. That table? ... "users." A relatively essential table, hard to get a lock on.

A database deadlock ensued. The migration query was unable to secure the necessary locks due to these hostage-taking threads. This query also held a lock on the main application table, lining up all other queries in a deadlock-laden conga line that would never move.

The application, overwhelmed by the surge of waiting queries, went down. The immediate resolution? Terminate the migration query.

From this tumultuous experience, the idea for my tool was born.


pg_explain_locks "DROP TABLE cards"

+-------------+---------------+---------------------+
| Relation ID | Relation Name | Lock Type           |
+-------------+---------------+---------------------+
| 16415       | cards         | AccessExclusiveLock |
| 16416       | users         | AccessExclusiveLock |
+-------------+---------------+---------------------+

This ordeal became an intensive crash course in database fundamentals, a challenging yet enriching learning experience that left an indelible mark on my professional journey.

In the aftermath, I developed a tool to better understand DB locks' interactions and their impacts on an application. Four years later, this humble tool found itself on the front page of Hacker News.

Continue to build, stumble, and learn. The journey is long, and each step, misstep, or fall is a stepping stone toward growth. You never know what your next mistake might teach you.

Created a Fully Automated Daily Trivia Podcast with ChatGPT - It's a Blast!

Adam Tal — Sat, 20 May 2023 20:35:36 +0000

Hey, fellow developers!

I'm excited to share my latest project with you all: I used ChatGPT to create a fully automated daily trivia podcast, and it's turning out to be an absolute blast! We're three episodes in so far, and I'm loving the results. I'll make sure to post an update later this week with full stats on the podcast's performance.

One of the coolest aspects of this project is that it really showcases the vast and generalized trivia knowledge embedded within the ChatGPT LLM. I think you'll be pleasantly surprised by the trivia questions and answers it generates.

You can find this podcast on all the usual platforms, so feel free to give it a listen. To make things easier, I used Buzzsprout for podcast hosting. If you're ever thinking about starting your own podcast, I highly recommend them – they made it incredibly simple to get started and host on all platforms for free. Here's the link to my podcast: https://www.buzzsprout.com/2185630/share

For the first 20 episodes, I used Wondercraft AI, but I'm currently developing my own code for the next set of episodes. I'm using Eleven Labs and PyDub for this, and I'll definitely share the results when everything is ready.

So, check out the podcast, and let me know what you think! I'd love to hear your feedback and any suggestions you might have. Happy listening!

Convert a Subreddit into a Podcast with AI

Adam Tal — Sat, 20 May 2023 20:33:26 +0000

Hey all,

Wanted to share this code I co-wrote with ChatGPT.

https://github.com/AdmTal/crowdcast

It’s a script that converts a subreddit into a podcast. Pretty neat!

I made it specifically for my new sub /r/crowdcast

I thought it would be neat to make a crowd sourced podcast using AI - so there it is!

Here’s how it turns out:
Listen to Xrowdcast podcast

There are two episodes so far.

I hope some of you leave some comments and are part of next week’s cast!

Thanks for reading, please reach out with any questions or feedback you might have