Being criticized by random strangers on social media sucks, but it feels even worse when your it's your own community that's calling you out.
Yet that's exactly what happened to me a couple days ago, in the wake of opening this year's State of JS survey.
The List
The source of the controversy was a question asking respondents which streamers or YouTubers they follow:
Respondents soon noticed a big issue with the list of options provided: it didn't include any women.
This seems like a straightforward enough matter: the list is not representative, therefore it's a bad list, and the person who drafted it (that would be me) is to blame.
But wait! Things may not be as simple as they seem… Let's dig deeper.
The Data
That list was not the result of an arbitrary process. It was taken straight from the results to last year's survey:
And it so happens that that year's survey did not include a list of options to pick from at all. Instead, it featured a freeform textfield where respondents could just type in whoever they wanted, precisely to “reset” any previous bias.
So there you go: the lack of diversity in the controversial list did not result from any bias. Instead, it simply reflected a lack of diversity in who the top programming-related video creators actually are.
But wait! There's something else we didn't take into account…
The Audience
This is the gender breakdown for the 2023 State of JS survey, the very same survey whose data was used to draft the 2024 list:
If you're surprised by the extreme imbalance in gender, I suggest checking out a previous article I wrote about this topic. Such extreme ratios are sadly not uncommon in developer surveys, since they reflect the makeup of online communities that traditionally haven't been very welcoming to women to say the least.
But leaving aside the why, that imbalance certainly hurts out list's credibility: it now seems quite probable that the reason why our list consists entirely of men is that the survey itself is filled out mostly by men.
So the list was problematic after all. Glad we could settle this.
But wait! There is an unspoken assumption in our reasoning: that men are less likely to follow women creators. Is that even true?
The Assumption
The State of JS survey results let you define your own custom filters, and thanks to this handy feature we can compare the overall 2023 top video creator list with that same list, but filtered to only show responses from women respondents:
As you can see, neither men or women respondents wrote in enough women for their name to show up in the top 10. In other words, a respondent's gender does not seem to be correlated with the gender of the creators they follow.
Note: for the sake of this analysis I'm using my own judgement to determine who is a woman or not. I realize this is problematic in its own way, and I apologize if I happen to mis-gender anyone in the process.
So to recap: the reason the list did not contain women was because it reflected last year's results – and even when based on an all-women subset of the data, the list remains virtually identical.
So no bias after all. That's just the way things are. Except…
The Long Tail
The "women respondents" side of the chart above only includes the top 9 answers because we cut off any answer that gets fewer than 10 mentions. After all, at such low numbers the statistical significance of a datapoint becomes harder to ensure.
But I went straight to the source and queries our back-end directly to get data on the top 100 responses, including items that only got 1 or 2 mentions. And I also did the same for the related People question ("Which individuals do you read, follow, or just want to highlight?").
This gave me the following data:
Video Creators
Number of women in top 100:
- All respondents: 1
- Women respondents: 6
People
Number of women in top 100:
- All respondents: 3
- Women respondents: 14
So a respondent's gender does impact the gender of their picks after all! But that correlation only appears in the long tail of answers, which do not show up in charts.
So the list was problematic after all, because it was based on a disproportionately male audience, and men tend to mention women at a lower rate, except, if we restrict it to the top 10 then that correlation disappears; however…
Wait. We've dug so deep that we somehow ended up right back where we started.
The List (Again)
Let's take another look at that list:
It took me a while, but I finally realized I had missed the point all along.
At the end of the day in really doesn't matter how well I can justify the process that led to this list, to myself or others. The list itself sends respondents the message that women creators don't exist, which alienates women taking the survey. And anything that turns away women will then deepen the survey's gender skew.
In other words, I was so focused on looking at things from a deeper perspective that I didn't realize that the very fact people had an issue was in itself an issue.
Lessons Learned
- Diversity is a complex topic with few easy answers. And while supporting diversity and inclusivity on a moral and ethical level may be a straightforward choice, actually translating that in real-world action can be a lot trickier.
- Social media tends to strip context and nuance of everything – but don't discount feedback just because of that. A superficial "hot take" may seem uninformed to a domain expert, but it still reveals a lot about people's first impression – which if you're not careful, will also be their last.
- The lack of survey participation by women is the cause of most of the issues discussed here, and something I want to take on this year.
So if you're able to, please share the survey with any women developers who you think might benefit from taking it and making their opinions known:
Oh, and you might be wondering what I did about that list in the end. Well, almost immediately after that initial feedback, I replaced it by a freeform text field, just like it was last year:
That was a 5-minute fix. But addressing the real issues at the root of all this will take a lot longer than that…
P.S. if you want to hear more about what concrete steps are being taken, you can learn more here.
Top comments (0)