I'm Not on the DEV Team. I Still Debugged Their Missing Badges From Public Code. (And My First Test Was Wrong.)

I’m not on the DEV team. I don’t have an admin panel, production database access, or internal Slack. Everything I know about how DEV works comes from reading the public forem/forem repository. 🕵️‍♂️

Today, that was enough to diagnose why nobody on DEV was receiving multi-commit contributor badges, ship the service fix upstream, and get a "Thank you for this" and merge from @benhalpern Sir himself. 🚀

The 4x DEV Contributor Club badge: "Awarded for 4+ merged commits to Forem — Thank you so much for your contributions!"

It also taught me a humbling lesson about tests that pass for the wrong reason. 🧪

---

1. The Itch: 4 Merged PRs, 1 Badge 🤔

Over the past few days, I went on a contribution sprint in forem/forem and merged four pull requests:

• #24003 — Docs: fix broken Kamal TOC anchor and index missing guides
• #24056 — A11y: remove duplicate aria-describedby attributes in admin forms
• #24094 — A11y: remove dangling aria-describedby references in creator settings
• #24095 — Views/Liquids: quote src and add missing alt in user_subscription tag

My 1x (dev-contributor) badge showed up right away after PR #1. ✅

When PR #4 merged, I knew from config/schedule.yml that BadgeAchievements::BadgeAwardWorker runs every hour at :20 past the hour (cron: "20 * * * *"). I checked GitHub’s Contributors API (/repos/forem/forem/contributors) and confirmed my account was sitting on page 2 with contributions: 4. Because Forem sets Octokit.auto_paginate = true, the worker was definitely reaching page 2.

One hourly cron passed. Then two. Then three. Still no 4x-commit-club badge. 💀

Me at :20 past the hour, every hour, refreshing my profile like it owes me money.

Instead of refreshing my profile for another 24 hours, I cracked open app/services/badges/award_contributor_from_github.rb. 🔍

---

2. Reading the Code Like a Detective 🕵️‍♂️

Here is how Badges::AwardContributorFromGithub looked on main:

BADGE_SLUGS = {
  "dev-contributor": 1,
  "4x-commit-club": 4,
  "8x-commit-club": 8,
  "16x-commit-club": 16,
  "32x-commit-club": 32
}.freeze

When I compared award_single_commit_contributors (which awarded my 1x badge) against award_multi_commit_contributors (which handles 4x, 8x, 16x, and 32x), three bugs jumped right off the screen:

🐛 Problem 1: Silent Slug Drops

Each key in BADGE_SLUGS is resolved via Badge.id_for_slug(slug). If a slug in the production badges table doesn't match the hardcoded constant, id_for_slug returns nil. The service called .compact on the resulting hash and moved on. No error. No log line. Just a badge tier that quietly vanishes into the void. 👻

Because Forem only exposes resources :badges, only: [:index] without public /badges/:slug show routes, you can't query individual badge slugs directly. However, inspecting the public dev.to/badges catalog gave me a massive clue:

Badge #423 on /badges:

• Title: 4x DEV Contributor Club
• Description: 4x-commit-club

Wait—why was 4x-commit-club sitting in the description field while the title was 4x DEV Contributor Club? Because Badge#generate_slug parameterizes the title when no explicit slug is set (4x-dev-contributor-club), the stored slug column in production didn't match the hardcoded "4x-commit-club" constant at all! 💡

Connecting Badge#generate_slug to DOM element IDs on /badges at 9 AM with zero database access.

💌 Problem 2: Missing Context Messages

Single-commit badges were awarded with:

BadgeAchievement
  .where(user_id: i.user_id, badge_id: dev_contributor_badge_id)
  .first_or_create(rewarding_context_message_markdown: msg)

Multi-commit badges, however, were awarded with a bare .create:

BadgeAchievement.create(user_id: user_id, badge_id: slug_id)

Even when a multi-commit badge was created, it omitted rewarding_context_message_markdown: msg ("Thank you so much for your contributions!"). Four times the commits, zero times the thank-you note. 😭

🔁 Problem 3: Missing Idempotency Guard

create_badges_for_user called BadgeAchievement.create(...) on every hourly run instead of .where(...).first_or_create(...). Every hour, it re-attempted already-earned badges for every contributor on the list, relying on ActiveRecord's uniqueness validation to fail silently. Worse, if a badge ever had allow_multiple_awards: true, bare .create would insert a duplicate achievement every hour—firing after_create_commit :notify_recipient and send_email_notification (flooding your inbox from [e-mail]) each time. 📬💥

---

3. Filing the Issue & Shipping PR #24113 🛠️

I opened Issue #24114 laying out the slug mismatch on badge IDs 423–426, and paired it with PR #24113 to fix the service itself:

1. ✅ Attached rewarding_context_message_markdown: msg to multi-commit badge awards.
2. 🔒 Replaced bare BadgeAchievement.create with .where(...).first_or_create(...).
3. 📊 Deferred slug lookup behind the GitHub auth provider check and added logging whenever configured slugs are missing from the database.

---

4. The Plot Twist in Code Review 🎬

The initial deep-dive review on @benhalpern Sir’s account was posted by Forem's automated review agent (OpenHands), and it did something awesome—it ran a read-only lookup against the badges table and confirmed my exact diagnosis from Issue #24114:

"I ran a read-only lookup on the badges table for the five BADGE_SLUGS keys. Only one of the five slugs resolves. The four Nx-commit-club slugs have no matching row, so create_badges_for_user skips them and nobody can earn them today. The issue reporter's guess is correct."

Bingo. 🎯

Even better, the review caught two subtle flaws in my PR that taught me more than the bug itself:

📢 Lesson A: My Rails.logger.warn Was Log Spam

I originally logged a Rails.logger.warn inside the loop for each missing slug. Why was that bad?

1. Until the production slugs were aligned, DEV would log 4 warn lines every hour.
2. Forem powers many independent communities. Most self-hosted Forems enable GitHub auth without seeding DEV-specific contributor badges. Five warn lines an hour for expected configuration turns warn into permanent background noise.

The Fix (5e36999): Collect all missing slugs and emit a single Rails.logger.info line per run. 🧹

🤡 Lesson B: My Regression Test Passed on main (For the Wrong Reason!)

To test idempotency, I wrote a spec that ran described_class.call twice and asserted that the user still had only 1 badge achievement for 4x-commit-club.

It passed! Green dots everywhere. 🟢

Except the review pointed out something brutal: that test also passed on origin/main without my fix.

When your shiny new regression test passes on main without your bugfix.

Why? The badge factory defaults to allow_multiple_awards: false. On main, the second BadgeAchievement.create hit validates :badge_id, uniqueness: { scope: :user_id, if: :single_award_badge? }, failed validation silently, and left the count at 1. My test wasn't testing first_or_create at all—it was testing ActiveRecord's uniqueness validator! 🤦‍♂️

The Fix (5e36999): I updated the spec to create the badge with allow_multiple_awards: true (where bare .create genuinely inserts duplicates on main) and asserted expect { described_class.call }.not_to change(BadgeAchievement, :count) on the second run.

After I pushed 5e36999 and all 34 checks went green, @benhalpern Sir stepped in personally, dropped a comment:

That "Thank you for this" from @benhalpern Sir genuinely made my entire day! 🙌 And right after that, he merged #24113 into forem:main as d1b962e (my 5th merged PR!). 🎉

Shortly after, the badge email from [e-mail] hit my inbox and the 4x DEV Contributor Club badge landed on my profile—complete with the "Thank you so much for your contributions!" context message my PR wired up. The code is cleaner, the PR is merged, and the green 4x circuit board is finally home. 💚

---

💡 3 Takeaways I’d Tell Past Me

1. 🔴 A regression test isn't real until you watch it fail on main. Before you celebrate a green spec, git stash your change and run the spec against main. If it still passes, you haven't written a regression test—you've written a test that agrees with you.
2. 🪵 Log levels are an operator contract. warn means someone should act. If a platform supports optional data seeds, missing optional rows are info, not warn.
3. 🔓 You don't need production access to debug production. You just need the public source code, a falsifiable hypothesis, and the curiosity to follow the trail.

Your turn: What’s the sneakiest time one of your tests passed for the completely wrong reason? Drop it in the comments 👇

---

Comments

[Dhruv Jani]

🐛 Plot twist: the bug wasn't the only thing I had to debug. My own test was wrong, too.

This started with four merged PRs and a missing badge. It turned into a dive through Forem's public code, a fix that got merged, and a code review that taught me why a passing test isn't always proof that your fix works.

Huge thanks to everyone who reviewed the PR and helped me catch what I'd missed!

Your turn: Have you ever had a test pass green while completely missing the bug it was supposed to catch? I'd love to hear that story. 👀

[Dhruv Jani]

And Sir @francistrdev, I'd love a comment in here because I know you've contributed to forem before and skilled in ruby on rails.

Don't want to bother you, only when you're free!😅

[Ben Halpern]

Thank you for this!

This bug has been around so long it had completely faded from everyone's consciounseness.

[Dhruv Jani]

Thank you so much, Sir! That really means a lot. It was such an awesome rabbit hole to trace from the outside.
Also, your reply really made my day!
Always happy to support the DEV Community!

Have A Great Day!

[Emmanuel Paul]

Nicely done 👏

[Dhruv Jani]

Thanks!😄
Have A Great Day!

[Yug Vasava]

Bro, this is great! keep up the good work.

[Dhruv Jani]

Thanks Bro!
I'm still learning ruby on rails. That's why my initial PRs were docs/front-end related and I made it to 4 merged PRs but didn't get the badge so today was the BUG SMASH DAY!😂

[Kaj]

Same class, from the other side. I published a fix for a 422 "Invalid authenticity token" on a scripted comment POST, then tested it by posting a comment. Green. Posted a second one to be sure. Green.

Then I ran the same recipe cold, fresh cookie jar, and it 422'd. The token I'd been grabbing off the article page belonged to a different form. It only ever worked because of state left over in a warm session.

The test wasn't wrong about the fix. It was wrong about the world. Same shape as your factory default: the thing that made it green was a default I didn't know I'd chosen.

Borrowed from your takeaway: for anything auth-shaped, "it passed" usually means "it passed with the session I happened to be holding." The regression run has to start from nobody. That's the git stash step, just for cookies.

(Token-side writeup, if the details are useful: dev.to/kaj_ilands/your-scripted-po...)

[Stefan]

There's something pretty funny about earning your fifth merged PR by fixing the badge for your fourth 😄 The factory default was a sneaky one: the count was right, but the service was still attempting the duplicate. Nice example of why checking the final state alone can miss what went wrong.