Predicting Content Decay: How I Built a Leak-Free ML Pipeline with 79M Rows of Search Data

Syed Muhammad Ibrahim Ali — Tue, 08 Sep 2026 18:02:27 +0000

If you manage high-volume publishing, content decay is the silent killer of organic traffic. By the time you notice a drop in your dashboard, the damage is already done.

For my FlyRank Machine Learning Internship capstone, I set out to solve this by building an offline analytical pipeline that predicts content decay before traffic plummets, using historical search telemetry.

Here is a look under the hood at what I built, the architectural decisions that mattered, and the limitations of predicting the future.

The Results

Instead of letting editors blindly guess what to update, the pipeline translates 30,000 anonymized search performance rows into a prioritized action queue (e.g., refresh, monitor, expand_and_refresh).

My final model, an optimized Random Forest, achieved a 0.750 ROC-AUC and a 0.74 Precision@50, massively outperforming the naive recency baseline (which sat at 0.24 Precision@50).

The Core Design Decision: Beating Data Leakage

When I started, I thought the hardest part would be tuning hyperparameters. I was wrong. The hardest part was structural hygiene.

My biggest design decision was abandoning random k-fold cross-validation in favor of a strict chronological client_holdout split.
When dealing with time-series search data and rolling-window velocity features (like log_impressions_90d), random splits inevitably leak future information into the training set. By enforcing a temporal embargo gap and grouping data strictly by page entities, I ensured the model’s evaluation metrics were completely leak-free and production-realistic.

The Reality Check: A Core Limitation

Honesty reads as credibility in data science, so here is the major limitation of this build: It calculates statistical probability, not causal certainty.
The model identifies that a page is exhibiting the mathematical signals of decay based on historical telemetry. However, it cannot predict unobserved external shocks—like a sudden core algorithm update from a search engine or a macroeconomic shift in user intent. It is a powerful reviewer aid, but it requires a human editor to make the final publishing decision.

AI as an Architectural Partner

As part of the AI Fluency framework, I used Claude during this build not as a code-generator shortcut, but as a rigorous sounding board. I used it to stress-test my leakage assumptions, audit my validation split logic, and scaffold my documentation, while manually verifying every metric against my local telemetry execution.

See It Live

You can inspect the full pipeline, evaluation metrics, and my generated action queues here:

📄 Live Research Paper: My Netlify Portfolio
💻 Code & Architecture: GitHub Repository