Recovering interrupter with occasional relapses, lover of spreadsheets, blogger, programmer, adept debugger, conjurer of analogies, and probably other things.
As the architect of the feed update, there are two things I'm super curious about:
1) System performance of the new feed algorithm
2) Higher utility for the DEV community
In the case of system performance, if this new approach is more performant than the current approach, I’ll be super happy. Why? Because of how I architected the new feed algorithm. More on that in a bit, but it informs the higher utility for DEV.
The second point I’m curious about is the higher utility for DEV. In writing the weights for each “property” of a post, the weights were intuitive guesses. We needed something to propose, and those current weights are what we’re going with.
However, the weights are very configurable. To the point where we could envision allowing Forem creators to choose which “properties” to consider for their feed and the weights to apply to those properties.
So, as I see it, if the performance is better with the new feed algorithm, we’re going to then run experiments into tweaking the properties and weights of that algorithm.
In other words, if the new feed algorithm is viable from a performance stand-point, we will now have a more transparent and extensible approach that is adaptable for all Forem instances.
Accessibility First DevRel. I focus on ensuring content created, events held and company assets are as accessible as possible, for as many people as possible.
Recovering interrupter with occasional relapses, lover of spreadsheets, blogger, programmer, adept debugger, conjurer of analogies, and probably other things.
Thank you for sharing this feedback. The "Relevant" feed in the experiment first finds the relevant posts, limits the results to the highest relevancy, then sorts by date.
Accessibility First DevRel. I focus on ensuring content created, events held and company assets are as accessible as possible, for as many people as possible.
No problem, I know how tough something like this can be to balance when there are so many variations of tag preferences etc.
Just a personal preference but relevancy first felt better (if that is the difference) and you might have slightly overcooked the pulling back of reactions (I could be entirely wrong, a sample size of one is not very useful I know!) as that seems to have 0 bearing now?
But I would imagine the end goal is to see which feed produces more balanced engagement and if this works then that is great!
Hope the experiments continue! Can't wait for the write up on the results! ❤
Recovering interrupter with occasional relapses, lover of spreadsheets, blogger, programmer, adept debugger, conjurer of analogies, and probably other things.
Accessibility First DevRel. I focus on ensuring content created, events held and company assets are as accessible as possible, for as many people as possible.
Super, thanks for those links, I knew I have seen that somewhere in the codebase but couldn't find it again!
The bit I didn't get is does that config mean that reactions have a slightly negative impact of 0.9988 if there are 0,1,2 or 3 reactions and then fall back to 1 for more than 3 reactions?
So at this stage essentially reactions are completely pointless (as I can give myself 3 reactions) to let you fine tune the other parts of your experiment? (which makes sense, A / B test one thing at a time so you know which lever pulls the feed which way!)
It is super interesting as where my gut leads me is that reactions are one of the more important metrics but the way they are gathered / assessed is the bit that needs addressing (I mentioned only counting votes from accounts that meet a certain criteria such as certain age or total number of likes for example), but often you can think one thing and find it has a completely unintended side effect!
The way you have the levers set is smart, wish I was the one playing with them as I would cause havoc going to the extremes 🤣!
Recovering interrupter with occasional relapses, lover of spreadsheets, blogger, programmer, adept debugger, conjurer of analogies, and probably other things.
Reactions are such a challenging component to consider. They are very low effort, so a question is how much do we "weight" do we give for low effort tasks? A second variable class, as you hint at, is related to the person giving the reaction.
And yes, in this experiment, we've greatly limited the impact of reactions.
One goal I have is to find "lever" configurations for communities to adopt. I can envision a lever configuration that says "We want to see more posts from new members to hear what they have to say, but we want to ensure that old members reactions are more heavily considered." (That's an odd example but one that I hope the levers options can help make feasible).
Accessibility First DevRel. I focus on ensuring content created, events held and company assets are as accessible as possible, for as many people as possible.
Oh I love that last concept! The predefined preferences would be great for people running different instances of Forem!
The low effort part is a fair point! (I would love to see what would happen if you could only upvote 3/5 articles a day for example, that would be super interesting to see how people behaved when their votes were "valuable"!)
One idea could be time on page / scroll percentage to assess article quality / engagement? I know that is getting complex though (but one positive is it can nearly all be done client side with little performance impact)!
The only other idea I have to throw in the pot (and I have no idea if it is a good one) is to weight comments with a certain length / word count (so the "thank you" comments get ignored) more heavily.
Then the number of comments on an article that meet this criteria are then used as a weighting factor.
Yet again this is just based on my usage of the site where I look for posts with a lot of comments as they tend to either be really good or really controversial (and most of the DEV team know I am the one who loves to be involved in controversial stuff on here 🤣).
In all seriousness, looking at number of comments tend to lead me to some cracking content (or at least to engaged authors who respond), so it may be another lever to add after all of your planned experiments?
Anyway I have taken up plenty of your time I am just enjoying the complexity of the task you have undertaken as it is so nuanced!
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
As the architect of the feed update, there are two things I'm super curious about:
1) System performance of the new feed algorithm
2) Higher utility for the DEV community
In the case of system performance, if this new approach is more performant than the current approach, I’ll be super happy. Why? Because of how I architected the new feed algorithm. More on that in a bit, but it informs the higher utility for DEV.
The second point I’m curious about is the higher utility for DEV. In writing the weights for each “property” of a post, the weights were intuitive guesses. We needed something to propose, and those current weights are what we’re going with.
However, the weights are very configurable. To the point where we could envision allowing Forem creators to choose which “properties” to consider for their feed and the weights to apply to those properties.
So, as I see it, if the performance is better with the new feed algorithm, we’re going to then run experiments into tweaking the properties and weights of that algorithm.
In other words, if the new feed algorithm is viable from a performance stand-point, we will now have a more transparent and extensible approach that is adaptable for all Forem instances.
Hey Jeremy, quick heads up I think I am one of the ones (currently, not sure if you randomise at a certain time frame) using the new feed.
I think there is a tweak needed somewhere as my feed is pretty much in date order now like "latest" but with selected articles.
I hope that is useful feedback (if you weren't already aware) as it is burying things of value.
Also to save you having to look it up I have no preferred tags selected, just in case it only affects people without any feed preferences.
Thank you for sharing this feedback. The "Relevant" feed in the experiment first finds the relevant posts, limits the results to the highest relevancy, then sorts by date.
No problem, I know how tough something like this can be to balance when there are so many variations of tag preferences etc.
Just a personal preference but relevancy first felt better (if that is the difference) and you might have slightly overcooked the pulling back of reactions (I could be entirely wrong, a sample size of one is not very useful I know!) as that seems to have 0 bearing now?
But I would imagine the end goal is to see which feed produces more balanced engagement and if this works then that is great!
Hope the experiments continue! Can't wait for the write up on the results! ❤
At present, the levers we are using are found here:
github.com/forem/forem/blob/main/a...
And the reactions are very limited in impact:
github.com/forem/forem/blob/main/a...
These "levers" are our first configuration. But we're planning on testing the configuration of those levers.
In addition, we're looking at the computational performance of the two feed options.
Super, thanks for those links, I knew I have seen that somewhere in the codebase but couldn't find it again!
The bit I didn't get is does that config mean that reactions have a slightly negative impact of 0.9988 if there are 0,1,2 or 3 reactions and then fall back to 1 for more than 3 reactions?
So at this stage essentially reactions are completely pointless (as I can give myself 3 reactions) to let you fine tune the other parts of your experiment? (which makes sense, A / B test one thing at a time so you know which lever pulls the feed which way!)
It is super interesting as where my gut leads me is that reactions are one of the more important metrics but the way they are gathered / assessed is the bit that needs addressing (I mentioned only counting votes from accounts that meet a certain criteria such as certain age or total number of likes for example), but often you can think one thing and find it has a completely unintended side effect!
The way you have the levers set is smart, wish I was the one playing with them as I would cause havoc going to the extremes 🤣!
Reactions are such a challenging component to consider. They are very low effort, so a question is how much do we "weight" do we give for low effort tasks? A second variable class, as you hint at, is related to the person giving the reaction.
And yes, in this experiment, we've greatly limited the impact of reactions.
One goal I have is to find "lever" configurations for communities to adopt. I can envision a lever configuration that says "We want to see more posts from new members to hear what they have to say, but we want to ensure that old members reactions are more heavily considered." (That's an odd example but one that I hope the levers options can help make feasible).
Oh I love that last concept! The predefined preferences would be great for people running different instances of Forem!
The low effort part is a fair point! (I would love to see what would happen if you could only upvote 3/5 articles a day for example, that would be super interesting to see how people behaved when their votes were "valuable"!)
One idea could be time on page / scroll percentage to assess article quality / engagement? I know that is getting complex though (but one positive is it can nearly all be done client side with little performance impact)!
The only other idea I have to throw in the pot (and I have no idea if it is a good one) is to weight comments with a certain length / word count (so the "thank you" comments get ignored) more heavily.
Then the number of comments on an article that meet this criteria are then used as a weighting factor.
Yet again this is just based on my usage of the site where I look for posts with a lot of comments as they tend to either be really good or really controversial (and most of the DEV team know I am the one who loves to be involved in controversial stuff on here 🤣).
In all seriousness, looking at number of comments tend to lead me to some cracking content (or at least to engaged authors who respond), so it may be another lever to add after all of your planned experiments?
Anyway I have taken up plenty of your time I am just enjoying the complexity of the task you have undertaken as it is so nuanced!