Jess Lee

for The DEV Team

Posted on Sep 30

Behind the Scenes: How We Judge DEV Challenge Submissions

#devchallenge

One of the questions we hear most from our community is: "How exactly do you decide who wins these challenges?" It's a great question, and we believe in being transparent about our process while continuing to refine it based on what we learn from each challenge.

Here's a look at how your submissions make their way from the #challengesubmission tag to the winner's circle.

Our judging system combines automated checks, community input, professional review, and sponsor involvement. This approach helps ensure fairness while managing the hundreds of submissions we often receive. Here are the multiple steps we take:

Step 1: AI Validation

We take an initial automated step to confirm that posts tagged with #challengesubmission are legitimate submissions. Meaning, they're not empty posts, aren't quick "boost" posts, and actually relate to the challenge topic (not users simply trying to utilize a popular tag to get visibility for a non-submission post). Think of this as making sure all the entries are actually entries before the real judging begins.

Steps 2 & 3: Volunteer and Paid Judge Review (Parallel Process)

Once submissions pass AI validation, we have two types of judges reviewing entries simultaneously:

Volunteer Judges: Our amazing community volunteers are often the first humans to review submissions. These dedicated community members help us identify standout entries and share their winner recommendations with our team. Their passion for the community and diverse perspectives add tremendous value to the process. Important note: volunteer judges who participate in a challenge are not allowed to vote for their own submissions—we've had to remove judges in the past for attempting to vioalte this rule, as it ensures fairness for all participants.

Paid Judges: We work with contracted professional judges who are not community members. These external judges bring additional expertise and bandwidth while helping to reduce any potential bias that might exist in community-based judging. Like our volunteers, they evaluate entries and provide winner recommendations to help us identify the most deserving submissions.

Both groups work in parallel, which allows us to process submissions more efficiently while benefiting from different perspectives and areas of expertise.

Step 4: DEV Team Review

Our internal team reviews all recommendations from both volunteer and paid judges. We also spot-check additional submissions that may not have been surfaced in earlier rounds to ensure we're not missing any gems. From this step, we compile a shortlist of top submissions for our sponsors.

Step 5: Sponsor Input

Depending on the specific challenge and sponsor preferences, sponsors may participate directly in the final judging process. Many choose to be actively involved in selecting winners from our shortlist. Sponsors also have access to the full list of submissions and can advocate for specific entries they find compelling, either during the review process or at the final selection stage.

Our Commitment to Fairness

This process is something we're always working to improve. We regularly evaluate what's working well and what could be better, adjusting our approach based on feedback from participants, judges, and sponsors. Our goal is to create a system that's both thorough and fair, one that gives every quality submission a genuine chance to be recognized.

We know that behind every submission is a developer who put time, creativity, and effort into their work. That's why we've built multiple layers of review and involve different perspectives in the judging process. No single person or automated system determines the winners, it's a collaborative effort designed to surface the best submissions.

Want to Help Shape Our Community Challenges?

We're always looking for community members to join our volunteer judging team! As a volunteer judge, you'll play a direct role in recognizing and celebrating the amazing work our community creates. Your perspective helps ensure that high-quality submissions get the recognition they deserve.

Being a volunteer judge means you're not just consuming content—you're actively helping to foster the culture of learning, sharing, and innovation that makes DEV such a special place. Plus, you'll get a special volunteer judge badge on your DEV Profile for every challenge you review.

Interested? Here' more information on the application process:

Seeking Volunteer Judges for DEV Challenges!

dev.to staff for The DEV Team ・ Nov 1 '24

#devchallenge

Final Note

We're happy to address concerns or answer questions about how challenges are judged. While we can't share specific details about individual submissions or judge deliberations (to maintain the integrity of the process), we're committed to being as transparent as possible about our overall approach.

Thanks for being part of our community <3

Top comments (6)

Fayaz • Oct 1

A very timely post!

Personally, I highly value this community and respect your judgement, but I've noticed some complains and concerns regarding the evaluation process!

I think this post addressed those concerns very well. 🫡

A suggestion:

Perhaps it'll be better if you could make the submissions anonymous until the winner is announced.

This will for sure need some time and effort to implement, but once implemented, I believe the judgement will be even more unbiased and well accepted across the community!

Just a thought! ❤️

Anna Villarreal • Oct 1

i like your anonymous idea!

Jess Lee The DEV Team • Oct 1

This will for sure need some time and effort to implement, but once implemented, I believe the judgement will be even more unbiased and well accepted across the community!

Completely agree and appreciate the suggestion!