Big improvements incoming π
Finding the right balance for a feed algorithm is historically really hard. If you optimize purely for clicks and comm...
For further actions, you may consider blocking this person and/or reporting abuse
Very much interested to see how much it improves the personalized feed!
Although, I think significant improvement will take time.
I've used embeddings in an n8n RAG challenge last year.
While sometimes the similarity match works pretty well, at times it becomes very difficult to get the desired results.
In this case, for example, a person may engage in posts s/he disagrees with and/or is familiar with, a lot more than in posts s/he likes and wants to learn from.
Disagreement and familiarity are almost instant and hence result in more engagement, whereas unfamiliar learnable posts may take a long time to engage in.
Hence, these sort of algorithm may increase engagement while decreasing learning!
Hence, I'd be cautiously optimistic β€οΈ
We do have many other factors also being weighted completely separate, such as the "clickbait score" which punishes typical clickbait articles etc.
It is always veeeery much a balance of finding the right outcomes for learning. We'll try and come up with success criteria that really do reward learning to weigh as feedback. Will report back :)
Just a quick feedback: I usually bookmark a post if I think I have something to learn from that post later, but not yet sure what to say in comments, or whether or not I'd like it! Sometimes I engage with it weeks or months later.
Can this metric be used in any useful way? Just thinking out loud! π€
The interest_embedding injected directly into the ranking SQL is the right call. Static tag weights miss the person β someone who reacts to advanced Rust posts and beginner Python posts looks like "Rust + Python developer" when they're really two different people depending on the week.
One thing I'm curious about: how are you handling embedding drift? A user's interest_embedding from six months ago may actively work against what they care about now. Rolling window, decay weighting, or full recompute on new interactions?
I built a similar pipeline β hybrid BM25 + vector search on 100k personal documents. The thing that surprised me most was how much reranking changed the result set after retrieval. The cosine score alone didn't match what actually mattered. Curious whether the weighted SQL blend is your final signal or if there's a reranking layer on top.
Reranking is expensive, what did you use for reranking?
I've used Pinecone before. Any better suggestion?
This is the logic we're using, you can see
blend_factor(which is a different weight depending on the trigger).It's designed to reward recency and cycle through your interests over time, which is different than indicators such as what tags you follow which are more permanent in nature.
github.com/forem/forem/tree/main/a...
Since we're open source we more than welcome PRs suggesting more effective approaches here :)
The EMA blend is the right mechanism β it handles recency without requiring a separate decay job, and the variable blend_factor per trigger is smart. A reaction carries different signal than a page view; same vector update logic, different weight.
The drift question answers itself in the code: low blend_factor means old preferences dominate for a long time even after behavior shifts. Worth watching whether 0.2 is sticky enough at the edges
β a user who spent six months in one niche and then pivots hard may feel like the feed "doesn't know them yet" for longer than it should.
On the PR invitation . I'll take a look. Curious whether you've experimented with asymmetric blend rates: higher factor when the new article is far from the current vector (big shift), lower when
it's close (reinforcement).
Pinecone's reranking is managed but you're paying for the latency and the lock-in. What moved the needle for me was a cross-encoder running locally β slower than cosine similarity alone but the result quality difference on ambiguous queries was significant enough to justify it.
The retrieval gets you candidates; the reranker picks the one that actually matches intent. I built this out on Cloudflare Workers if you want to see it in production:github.com/dannwaneri/vectorize-mc...
This is actually a really smart direction for recommendation systems.
I like that you're not replacing community signals with AI, but blending semantic relevance with human-driven interactions. That balance is what most platforms miss.
The audit layer around AI calls is also underrated β logging latency, token usage, wrappers, versions, etc. becomes essential once AI touches production ranking systems.
How are you considering wild factors? Like occasionally its nice to see a post outside of your interest scope, for freshness. Is there a loose allowance after preferences are consideted, im guessing? I hope that makes sense.
Honestly really excited about this. I love how thoughtful the implementation is while also leveraging some bleeding edge tech from our partners :) and hopefully continuing to zig zag us towards the ultimate goal of always surfacing the right content to the right people when they're spending time on DEV
the part I'd worry about is model pinning - gemini-embedding-exp-03-07 today, different dimensions next cycle, full recalibration project. what's your migration path when the embedding model version changes?
Big improvements!! Thanks Ben for updating us and keeping it transparent! Good work :D
The AiAudit-by-default pattern is the part more teams should steal β most bolt AI in and add logging only after the first incident. Two things it quietly buys you beyond debugging: (1) it's your governance boundary β when someone asks "what exactly did we send Gemini, and when," you have an answer instead of a guess; (2) it's where you'd catch payload drift (PII or secrets sneaking into an embedding call) before it's a compliance problem. Curious whether you redact payloads before they hit the AiAudit log or store them raw for fidelity β that trade-off (forensic completeness vs. not warehousing sensitive text) is the one I keep seeing teams get wrong. Really like that the embedding layer is auditable rather than magic.
Balancing a feed algorithm between clickbait popularity and chronological noise is incredibly difficult. Using semantic embeddings to cluster topics rather than relying purely on tag-matching is a huge step forward for content discovery. Iβm really looking forward to seeing how this impacts the visibility of deeper, technical deep-dives over listicles!
This is really interesting to me because I tried applying a tiny version of this idea in my college blog website.
Iβd love to see how this goes at DEVβs scale, especially around how different actions affect the interest_embedding. A like, bookmark, comment, or long read can all mean different things. Sometimes bookmarking means βI want to learn this later,β while commenting can even mean disagreement or familiarity.
So Iβm looking forward to learning from how this works in practice, and seeing which parts I could have done better in my own small implementation.
Also love that youβre blending embeddings with community signals instead of making vector similarity the whole feed. That feels like the right balance.
Really great write-up. Iβve been following DEVβs infra posts for a while, and this is one of the most actionable ones.
What stands out to me is the pragmatic choice of sticking with Pgvector + Gemini embeddings instead of jumping to a dedicated vector database. For most content platforms (and honestly, many API-first systems), the simplicity of keeping vectors inside PostgreSQL is massively underrated. The operational overhead of syncing between two systems often kills the ROI of a "more scalable" solution.
The shift from tag-based to semantic similarity also solves a problem Iβve felt as a reader: tags are great for filtering, but terrible for discovery. Two posts about βMCP serversβ and βAI agent infrastructureβ might share zero tags but are conceptually adjacent. This is exactly where embeddings shine.
Iβm building something slightly different (agentshare.dev β an API for procurement agents), but we ran into a similar challenge on the machine side: how to help AI agents discover the right tool or endpoint without hardcoding. We ended up using a mix of JSON-LD and semantic hints in API responses. Seeing how DEV approaches the same "discovery vs retrieval" tension from a human-feed perspective is really valuable.
One question: did you experiment with any hybrid approach (e.g., tags + embeddings) to handle edge cases where semantic similarity is too "loose"? Iβve seen some teams add a small weight to explicit tags to keep the feed from drifting too far.
Thanks for sharing the latency numbers too β sub-50ms for embedding generation is impressive for a community platform at DEVβs scale.
Looking forward to the follow-up on how you handle real-time updates (new posts, edits) in the vector index.
Yess ! And will the search be available as an API or a customized rss feed ? That could be quite interesting...
Interesting approach using Gemini embeddings for community feed ranking. Curious about the cold start problem -- how do you handle new users who haven't built up enough signal for the collaborative filtering side? Do you fall back to pure content-based similarity, or is there some hybrid approach?
Also for the embedding storage layer, what's your latency budget? I've been exploring similar approaches for robot navigation data, and the embedding lookup latency becomes critical when you're doing real-time context injection under memory pressure.
The hardest part of an embedding-based feed isn't the model, it's deciding what "relevant" means for a community that contains both deep technical posts and beginner journeys side by side. Curious whether you're treating those as distinct corpora or letting the embedding space sort it out β the choice quietly shapes whose work surfaces.
Methodology note: embeddings without a labeled eval set degrade silently. We ran a similar embeddings-driven feed and watched cluster quality drop 15% over 6 weeks as content distribution shifted, with zero observable signal in any production metric. The DDD-style pattern that helped: treat the feedback signal as its own bounded context with its own ubiquitous-language (impression, engagement, dwell, intentional-skip), version those signals like a schema, and run a labeled calibration check weekly. FastAPI plus pgvector for the index is fine, but the eval layer is where the work actually is.
Curious if you ran into cost surprises here β embedding APIs are deceptive because the per-call cost looks tiny but volume Γ frequency compounds fast. Did you batch the embedding writes or push them through real-time on every post creation? Also wondering if Gemini's longer context window (vs OpenAI's 8K) actually moved the needle on recommendation quality, or if you ended up truncating to a similar length anyway in practice.
This is a really smart application of embeddings for content discovery. The idea of using semantic similarity rather than just recency or popularity makes a lot of sense for a community-driven platform.
I'm curious about the cold start problem β how do you handle new articles that haven't accumulated enough interaction data yet? And have you experimented with multilingual embeddings for the global DEV community?
Followed to follow your work on this! Interesting stuff π
Love the usage of vectoring here! Excited to see how it plays out.
Insightful breakdown - love how youβre using embeddings to boost relevance and surface better community content. Curious to see how this evolves as the model and the platform grow.
Deep stuff - cool to see how you're able to use AI in a 'controlled' and auditable way ...