Hello, wonderful DEV community!
My name is Amy, and I’m a product manager here at Forem. I started my journey here this past August so if we haven’t interacted on DEV before, it’s great to meet you!
Today, I wanted to share an update about a topic that impacts anyone who regularly visits and/or writes on DEV – our home feed and the algorithm behind it.
Addressing some important concerns
We have heard our community's frustrations about our feed. Some examples include your feed not reflecting tag preferences, too many listicles, and more recent posts rather than relevant posts. We have come across posts by community members exploring how to improve their experience of the feed with a handful of unofficial workarounds.
While the ingenuity of our community doesn't surprise us (and in fact, delights us), we don't think you should have to find "hacks" in order to make this space work as you expect.
Our commitment on continuous improvement
We strive to make DEV an enjoyable place for our members. However, this is not always easy. To solve these problems effectively, we first need to know what might contribute to these problems. As some of you probably already know, our feed algorithm is complex. It is often difficult for us, as core team members at Forem, to digest what is going on in it. This challenge makes it difficult for the community to chime in, as it hasn’t been clear what factors are diminishing our feed’s quality. As technology evolves over time naturally, we need to untangle and refactor different layers of decision making in our software so as to pinpoint the underlying root cause(s).
We are committed to making a great feed experience while being transparent in how we work. The first step is for us to shed light on how our algorithm works to ensure that the code base is accessible and readable for our community. This post, the changes to our feed algorithm, plus all the great refactoring work going into it by Jeremy are our first baby steps towards this commitment. We also renamed the name of the default sorting on your feed from “Feed” to “Relevant” for clarity.
A peek at how the DEV feed works
Without further ado, here is a high-level summary of how our feed algorithm works.
When deciding which posts should be in your home feed, four things come into play:
- The freshness of a post (how new is a post?)
- The relevance of a post (relevance as in, do you follow any of the tags on this post? Do you follow the author?)
- The popularity of a post (how many comments have been left? How many reactions have been given?)
- Moderator feedback for a post (has this post been reported?)
After a series of calculations based on the above, we will then assign a “score” to each post. Then, the posts with the highest scores are delivered to your feed.
Possible causes of DEV’s feed algorithm quality issues
After some investigation, we realized we are unintentionally giving reactions too much weight when calculating the aforementioned score of a post. It manifests itself in the following ways:
- It’s really easy to game the algorithm by farming reactions
- Sometimes too many irrelevant posts show up in feed because they have gotten lots of reactions
- As a consequence, there are too many irrelevant posts to a point that you don’t feel like your feed is relevant to you anymore
What we have changed so far
We are launching an A/B test in our algorithm, with the treatment of suppressing the effect of reactions in calculation. We will start with 20% of our users, and observe the performance to decide whether it is ready to be rolled out to more users – or if it requires some tweaks.
We also have identified several other opportunities to improve the algorithm, and we see this as an iterative effort. In other words, you’ll be hearing from me again with more updates and information about our feed as we work on improving it!
What do you think?
We would love to hear your thoughts on anything mentioned above. Do you think our fixes will address the issues you have encountered on your home feed? Are there other problems you are experiencing with the feed? Let me know in the comments below!
Finally, on behalf of the entire Forem team, I want to thank you not only for being a member of DEV, but also for being active and vocal with your frustrations and the changes you’d like to see here. We appreciate your patience and commitment to being stewards of this community.
Oldest comments (30)
Yes! I was waiting for this.
Those "bad" posts gathering instant reactions and being visible for a long period to gather even more reactions to become even more visible .. are annoying. I have even seen people who intentionally create alt accounts to have that first boost and make sure their posts are visible on the top and if you check them, it's only promotional content.
I would also love if we can give more weight to the "rate quality" feature given to trusted users. I know we might have some abuse here as well but it would be insignificant compared to gaming the reactions especially that trusted users are manually selected by the DEV team.
I believe this is already the case - if I'm remembering correctly the Thumbs Up/Down by Trusted users are about 10x as valuable as a standard user's reaction. (This is based on a discussion in the connect mod chat so I can't go back to confirm this 😅)
Considering that a user can perform 3 reactions and that you can add reactions to your own post, 10x is still low. With 2 extra accounts you can get 9 reactions immediately so you are immune to the first vote.
Based on my experience, If I am late to downvote a bad post before 10/15 reactions then it's too late unless more trusted users do it before that post gather more reactions.
Once you have your first boost, you are on the top and more reactions will come easily so it become more difficult to fight. Bad posts can easily reach 100+ reactions in a few hours.
@afif This type of fine-tuning is exactly what @jeremyf and I have in mind as potential candidates for future experimentations. We believe this is just a start for many iterations to come. I am logging it here in an issue to make sure we don't miss it when we are ready for the next round of experimentation.
Thanks for your feedback!
These are awesome news that the DEV team is working to fix the feed 👍
I've posted regularly for 2 years and really love the platform and the community 💖
It will be interesting to see whether this works, I hope it does.
Being the cynic that I am I think this is the wrong move, however.
While reaction farming is a big problem every site faces, the reduced relevance of votes and comments will have a side effect,
Posts that are high quality, but from lesser known authors will not get the boost they deserve if people find their article useful and upvote it and comment to show their support.
The problem (as I see it) is the tag system and always has been as far as I can tell.
If you write an article just add the #javascript tag and you will double your views for example. Albeit a mod may come and remove the tag but I have gamed the system by adding a JS example that doesn't add much to the article just so I can include the tag and it worked.
That is where I would reduce the relevance.
As for tackling farming reactions, having a hidden system that decides whether or not to count a vote for the algorithm, based on account age or some other factor, would be great.
So you have a public likes counter and then you have a separate one for the algorithm that only counts votes and comments that are from accounts xxxx days older or with a total of yyyy votes.
Don't get me wrong, I have seen some big improvements in the feed since moderators seemed to have been more involved and some of the experiments you have run have been great.
The only other thing that would be interesting to run an experiment on would be "DEV picks".
Similar to the "Top 7 posts" weekly update but actually incorporated into the feed (increased visibility and longevity perhaps?) so there is a place we can see posts that have been hand picked by the dev team and the mods. These are likely to be very high quality and hopefully quite diverse in nature.
Anyway those are my 2 cents and I am always impressed by the work the team at FOREM and DEV do to try and always improve. ❤🦄
Just to make sure I understand you correctly (you know, I am still rather new here 😉), when tagging a post with
JavaScript, you will double the views because there are a lot of people following this tag. Correct? I do see how people can abuse the tags and hence reduce the relevance here. This is going to be an interesting problem to solve.I have also gathered your suggestions on feed experimentation here for review at a later time.
Thanks so much for the write-up! I love your candid feedback.
Yes that is correct.
But the problem isn’t abusing the tags as such it is more that you have to abuse the tags if you want to get an article to show up enough to get traction (if that makes sense?)
I Love the steps the team are taking, I don’t envy you all as trying to solve biases and abuse in a feed is not an easy task! ❤️
Ah. I see. This is one of the discovery problems we are facing for less known tags (i.e. with fewer followers), similar to new authors having a hard time getting their posts surfaced, albeit of high quality.
Thanks for the extra context. It helps me see what problems you are facing with the feed as an author. 🥰
All I have to say now is welcome to DEV and as your first 2 official posts / features these are great (despite the criticism! 🤣) and super positive steps!
I look forward to seeing what you tackle next / the outcome of this feed experiment! ❤
It is worth noting, of course, that the extra weight (previously) put on reactions is not totally unfounded. After all, it is one of the main metrics for user engagement.
The "too-many-listicles" phenomenon only occurs simply because listicles are inherently designed to be bookmarked (i.e. one of the three reaction types that DEV considers). Hence, we end up in a situation where in addition to the hearts and unicorns a listicle receives, the extra bookmarks greatly amplify what would have otherwise been a regular article.
However, the same argument can be said about regular articles, tutorials, and visualizations. That is, they can also "game the Algorithm" by being designed to be bookmarked—intentionally or otherwise. But, what separates listicles from regular articles is that they are more easily automated than full articles, guides, etc. And that is exactly why the feed becomes bloated. To make a long story short, automation and bookmarks make a perfect storm.
Unfortunately, I have no recommendations for addressing this at the moment. It's really tricky because—as mentioned earlier—certain patterns are just inherently gameable.
Anyway, with that said, I just wanted to bring some additional perspective to the table. Looking forward to hearing about the proposed solutions! 🚀
This is a great insight! 🤔 We should test treating bookmarks differently from other reactions. I am logging it here for review.
Thanks for the great suggestion!
It's great that you are trying to optimise this feature.
Since anyone could make alt accounts to give an initial boost to the post, these are my suggestions:
Also, IMO, algorithm doesn't need to be publicly known, so it would be less abused. It's important that it works good and feedback is collected from users.
These are some great ideas, and one goal I have is to allow Forem creators to configure the feed "levers" to better match their community use case. And these "properties" that you're referencing may well be worth adding to the "levers" we make available.
The "algorithm" of the alternate feed is to look at the "properties" of each article and give those properties weight.
Here's the relevant weighting, mostly with comments: github.com/forem/forem/blob/main/a...
I hope I didn't come off as aggressive. My intention was to highlight the implementation and how I hope it reflects people's choices over the more opaque current algorithm (which encodes and obfuscates people's choices).
As the architect of the feed update, there are two things I'm super curious about:
1) System performance of the new feed algorithm
2) Higher utility for the DEV community
In the case of system performance, if this new approach is more performant than the current approach, I’ll be super happy. Why? Because of how I architected the new feed algorithm. More on that in a bit, but it informs the higher utility for DEV.
The second point I’m curious about is the higher utility for DEV. In writing the weights for each “property” of a post, the weights were intuitive guesses. We needed something to propose, and those current weights are what we’re going with.
However, the weights are very configurable. To the point where we could envision allowing Forem creators to choose which “properties” to consider for their feed and the weights to apply to those properties.
So, as I see it, if the performance is better with the new feed algorithm, we’re going to then run experiments into tweaking the properties and weights of that algorithm.
In other words, if the new feed algorithm is viable from a performance stand-point, we will now have a more transparent and extensible approach that is adaptable for all Forem instances.
Hey Jeremy, quick heads up I think I am one of the ones (currently, not sure if you randomise at a certain time frame) using the new feed.
I think there is a tweak needed somewhere as my feed is pretty much in date order now like "latest" but with selected articles.
I hope that is useful feedback (if you weren't already aware) as it is burying things of value.
Also to save you having to look it up I have no preferred tags selected, just in case it only affects people without any feed preferences.
Thank you for sharing this feedback. The "Relevant" feed in the experiment first finds the relevant posts, limits the results to the highest relevancy, then sorts by date.
No problem, I know how tough something like this can be to balance when there are so many variations of tag preferences etc.
Just a personal preference but relevancy first felt better (if that is the difference) and you might have slightly overcooked the pulling back of reactions (I could be entirely wrong, a sample size of one is not very useful I know!) as that seems to have 0 bearing now?
But I would imagine the end goal is to see which feed produces more balanced engagement and if this works then that is great!
Hope the experiments continue! Can't wait for the write up on the results! ❤
At present, the levers we are using are found here:
github.com/forem/forem/blob/main/a...
And the reactions are very limited in impact:
github.com/forem/forem/blob/main/a...
These "levers" are our first configuration. But we're planning on testing the configuration of those levers.
In addition, we're looking at the computational performance of the two feed options.
Super, thanks for those links, I knew I have seen that somewhere in the codebase but couldn't find it again!
The bit I didn't get is does that config mean that reactions have a slightly negative impact of 0.9988 if there are 0,1,2 or 3 reactions and then fall back to 1 for more than 3 reactions?
So at this stage essentially reactions are completely pointless (as I can give myself 3 reactions) to let you fine tune the other parts of your experiment? (which makes sense, A / B test one thing at a time so you know which lever pulls the feed which way!)
It is super interesting as where my gut leads me is that reactions are one of the more important metrics but the way they are gathered / assessed is the bit that needs addressing (I mentioned only counting votes from accounts that meet a certain criteria such as certain age or total number of likes for example), but often you can think one thing and find it has a completely unintended side effect!
The way you have the levers set is smart, wish I was the one playing with them as I would cause havoc going to the extremes 🤣!
Reactions are such a challenging component to consider. They are very low effort, so a question is how much do we "weight" do we give for low effort tasks? A second variable class, as you hint at, is related to the person giving the reaction.
And yes, in this experiment, we've greatly limited the impact of reactions.
One goal I have is to find "lever" configurations for communities to adopt. I can envision a lever configuration that says "We want to see more posts from new members to hear what they have to say, but we want to ensure that old members reactions are more heavily considered." (That's an odd example but one that I hope the levers options can help make feasible).
Oh I love that last concept! The predefined preferences would be great for people running different instances of Forem!
The low effort part is a fair point! (I would love to see what would happen if you could only upvote 3/5 articles a day for example, that would be super interesting to see how people behaved when their votes were "valuable"!)
One idea could be time on page / scroll percentage to assess article quality / engagement? I know that is getting complex though (but one positive is it can nearly all be done client side with little performance impact)!
The only other idea I have to throw in the pot (and I have no idea if it is a good one) is to weight comments with a certain length / word count (so the "thank you" comments get ignored) more heavily.
Then the number of comments on an article that meet this criteria are then used as a weighting factor.
Yet again this is just based on my usage of the site where I look for posts with a lot of comments as they tend to either be really good or really controversial (and most of the DEV team know I am the one who loves to be involved in controversial stuff on here 🤣).
In all seriousness, looking at number of comments tend to lead me to some cracking content (or at least to engaged authors who respond), so it may be another lever to add after all of your planned experiments?
Anyway I have taken up plenty of your time I am just enjoying the complexity of the task you have undertaken as it is so nuanced!
A super quick update, we found a bug in the new strategy (see github.com/forem/forem/pull/15528 for details).
For the time being we've toggled off the new feed, until we can deploy the bug fix.
What did the bug look like? After interacting with DEV for a bit, you might have noticed your Relevant Feed went blank or became very sparse. That had to do with faulty logic (on my part) of limiting publication date ranges.
Love it. The idea of making the feed more relevant. I'm far less interested in popular than I am in relevant and interesting to me. Twitter, Facebook, Instagram are all for the popularity junkies.
I see some rather poor quality Dev articles on my feed at times but mostly just let them slide on by. Seeing less of those would please me.
Above all I don't need quantity either or volume in preference to quality. I don't mind a feed that delivers more slowly, the right stuff. I do read som top quality stuff here too. And I bookmark it for future reference. A lot in fact!
Good job to those that worked on this. It's always great to improve algorithms.
That said, I don't use the algorithm-driven feed; I use the latest feed that shows all the articles in the order they were posted, so instead of actually dealing with the low quality articles this seems to me like a way to sweep them under the proverbial rug where most users simply won't see them. So I have to ask, is anything being done to actually deal with low quality articles and those that post them?
Quick update, I've re-activated the alternate feed experiment.