How to Replace a Rules-Based Feed Ranker Without Rebuilding Your Feed

Rules-based feed ranker: a ranking stage that orders feed items with hand-written logic, such as "boost followed authors, then sort by recency," instead of a scoring model trained on engagement data.

• Candidate generation: the stage that retrieves the eligible items for one user. It decides what can appear.
• Ranking layer: the stage that orders those candidates. It decides what appears first.
• Shadow mode: running a new ranker on live requests while users still see the old ranker's output. The new ranker only produces logs.
• Interleaving: an online test that merges two rankers' results into one list and credits each click to the ranker that contributed the item.
• Overlap@k: the share of items that two rankers both place in their top k positions.

TL;DR

• The ranker is one stage of a feed pipeline. Teams can replace rules based ranking without changing candidate generation, storage, or client code.
• The migration has five steps: isolate the ranking call, convert rules into features or constraints, run shadow mode, test on live users, and cut over with a kill switch.
• Shadow mode compares both rankers on real traffic with zero user exposure.
• Policy rules, such as "never show blocked users," stay as hard filters outside the new ranker.

Why can a feed ranker be replaced without rebuilding the feed?

A feed ranker can be replaced in isolation because ranking is a stage boundary. The ranker receives a user, a surface, and a candidate list, and it returns the same candidates in a new order. If the new ranker accepts the same input and returns the same output shape, nothing upstream or downstream has to change.

Most feed pipelines have four stages: candidate generation, filtering, ranking, and rendering. The migration below changes one function call and its configuration.

One exception applies. If the current ranker is also the retrieval step, for example a SQL query with ORDER BY and LIMIT 20, the candidate set must be widened first. The section on edge cases covers this.

Step 1: How do you isolate the ranking call?

Isolation means every ranking decision passes through one function with a fixed signature: rank(user, surface, candidates) -> ordered candidates. Nothing else in the codebase reorders items after this call.

Find every place that sorts candidates after retrieval: SQL ORDER BY clauses, sort calls in API handlers, and client-side re-sorting. Move that logic into the single function. Then add one configuration flag with three values: rules, shadow, and new.

Step 1 is complete when the feed output is byte-identical to production and all existing tests pass.

Step 2: What happens to the existing rules?

Each existing rule becomes either a feature that the new ranker can weigh or a hard constraint applied outside the ranker. Rule #7 in Google's Rules of Machine Learning by Martin Zinkevich gives the same instruction: turn heuristics into features, or handle them externally.

Existing rule	What it becomes
Boost posts from followed accounts	Feature: is_followed (0 or 1)
Show newer posts first	Feature: item age in hours
Never show blocked or muted users	Hard filter before ranking
Max 2 posts per author in the top 10	Diversity constraint after ranking
Pin announcements to slot 1	Fixed placement after ranking

A preference becomes a feature, because a model may learn to weigh it differently. A policy becomes a constraint, because no ranker should be able to violate legal, safety, or contractual rules.

Step 3: How do you run a new ranker in shadow mode?

Shadow mode ranking calls the new ranker asynchronously on each request, logs its output next to the served output, and never shows it to users. The served response keeps the rules-based order, so a failure in the new ranker cannot affect the feed.

The harness below runs with the Python standard library. Replace new_ranker with the call to your model or vendor.

import json, random, time
from concurrent.futures import ThreadPoolExecutor

def rules_ranker(user, candidates):
    # Current production logic: followed authors first, then newest.
    return sorted(candidates,
                  key=lambda c: (c["author"] in user["follows"], c["created_at"]),
                  reverse=True)

def new_ranker(user, candidates):
    # Stand-in. Replace with your model or HTTP client.
    return sorted(candidates, key=lambda c: c["score"], reverse=True)

def overlap_at_k(a, b, k):
    return len({c["id"] for c in a[:k]} & {c["id"] for c in b[:k]}) / k

def mean_rank_shift(a, b):
    pos = {c["id"]: i for i, c in enumerate(b)}
    return sum(abs(i - pos[c["id"]]) for i, c in enumerate(a)) / len(a)

pool = ThreadPoolExecutor(max_workers=4)

def shadow_compare(user, candidates, served, k):
    start = time.perf_counter()
    try:
        shadow = new_ranker(user, candidates)
    except Exception as exc:
        print(json.dumps({"event": "shadow_error", "error": str(exc)}))
        return
    print(json.dumps({
        "event": "shadow_compare",
        "user": user["id"],
        "latency_ms": round((time.perf_counter() - start) * 1000, 2),
        "overlap_at_k": overlap_at_k(served, shadow, k),
        "mean_rank_shift": round(mean_rank_shift(served, shadow), 2),
    }))

def rank_feed(user, candidates, k=10):
    served = rules_ranker(user, candidates)
    pool.submit(shadow_compare, user, candidates, served, k)
    return served  # users always see the rules-based order

if __name__ == "__main__":
    user = {"id": "u1", "follows": {"a1", "a3"}}
    candidates = [{"id": f"p{i}", "author": f"a{i % 5}",
                   "created_at": i, "score": random.random()} for i in range(50)]
    rank_feed(user, candidates)
    pool.shutdown(wait=True)

Log four signals per request: overlap@10, mean rank shift, latency, and error rate. An overlap@10 near 1.0 means the new ranker barely changes the feed. An overlap@10 near 0 means a large change, so sample those requests and review the lists by hand. Run shadow mode for at least one full week. Ron Kohavi, Diane Tang, and Ya Xu recommend full-week durations in Trustworthy Online Controlled Experiments (Cambridge University Press, 2020) because behavior varies by day of week.

Step 4: How do you test the new ranker on live users?

A live test exposes a small share of users to the new ranker and measures engagement against the rules-based baseline. Two methods work: an A/B test, which splits users between rankers, and interleaving, which mixes both rankers into one list for each user.

Interleaving needs less traffic. Chapelle, Joachims, Radlinski, and Yue found interleaved comparisons more sensitive than standard A/B metrics in a large-scale search study (ACM Transactions on Information Systems, 2012). A/B tests are still needed for session length and retention.

Define guardrail metrics before the test starts. For feeds, useful guardrails are the hide or report rate, the share of creators receiving impressions, and p99 latency.

Step 5: How do you cut over safely?

Cut over by ramping traffic in stages, for example 1%, 10%, 50%, then 100%, while keeping the rules ranker as an automatic fallback. If the new ranker errors or exceeds the latency budget, the service serves the rules-based order for that request.

Keep the fallback path in production for at least one release cycle after reaching 100%.

When is this approach the wrong choice?

Stage-by-stage replacement fails in four situations:

• Ranking happens inside retrieval. A query that returns 20 sorted rows gives a new ranker nothing to reorder. Widen candidate generation to a few hundred items first.
• No interaction logging exists. Every learned or calibrated ranker needs impression and click data. Add logging before Step 3.
• Traffic is very low. Online tests may not reach significance. Rely on shadow mode and manual list review instead.
• The rules encode policy. Those rules stay as constraints permanently, as Step 2 describes.

Where does a ranking API fit in this migration?

A hosted ranking API replaces only the new_ranker function. Steps 1, 2, 4, and 5 stay the same, which is why the shadow harness doubles as a vendor evaluation tool: each candidate vendor can run on real traffic before any contract.

Gortex is one option for that slot, currently in invite-only beta. One request with a recipient, a surface, and a candidate set returns the ranked order and fills sponsored slots in the same response, at sub-200ms p99. Calibrated linear ranking and deterministic slot-filling ship today. A learned ranker is planned for mid-2027 and a real-time auction for Q1 2027. Details are on the Gortex feed ranking API page.

Get a free feed teardown

Send your current ranking rules and feed surface, and the Gortex team will map which rules become features, which stay as constraints, and what a shadow test should measure. Request access at gortex.ai, or book a demo with the founder at [e-mail].

FAQ

How long does it take to replace a rules-based ranker?
Isolation and rule mapping usually fit in one sprint. Shadow mode runs at least one full week, and the live test runs until it reaches significance.

Does replacing a rules-based ranker require a data science team?
No. Shadow mode, guardrails, and staged rollout are backend engineering tasks. A team needs modeling skills only if it trains its own ranker.

Can some rules stay after the migration?
Yes. Policy rules stay as filters or placement constraints. Only preference rules move into the ranker as features.