03:14 AM. My PagerDuty app started screaming with that specific high-frequency tone that suggests my weekend is effectively over. The alert: Critical: Feature Store Latency Breach > 500ms.
I rolled over, checked the dashboard on my phone, and saw the throughput on our Kafka topic for user credit-risk features had flatlined. Simultaneously, the ML monitoring platform—our separate, shiny, "model-specific" tool—started firing alerts for Feature Attribution Drift.
In most shops, this is where the finger-pointing begins. The data engineers blame the upstream ingestion pipeline, and the data scientists blame the "flaky" model. We were about to waste four hours arguing over who broke what, until I realized they were both looking at different symptoms of the exact same systemic rot.
What we saw
The initial dashboard showed a massive spike in null values for credit_score_bucket. Naturally, the platform team assumed a schema change in the upstream ingestion service—our standard culprit. We checked the protobuf definitions in the user-profile-service. Nothing.
We checked the airflow logs for the etl-spark-job. It finished successfully, albeit three minutes slower than the P99 baseline. That was our first false lead. We spent forty-five minutes digging into Spark executor memory settings, thinking we had a data skew issue causing a timeout.
Meanwhile, the data science team was losing their minds because the model was outputting 0.98 probability of default for every single applicant. They were convinced the weights had corrupted or that someone pushed a bad model version to v2.4.1. They were busy rolling back to v2.3.9 while I was busy trying to figure out why the data pipeline was "succeeding" while producing zero output.
Photo by Savannah Bolton on Unsplash
Root cause
It wasn't a schema change. It wasn't a Spark memory leak. It was a configuration drift in our feature store, specifically in the feature-config.yaml of our Redis cache layer.
We had recently upgraded our client library to redis-py 4.5.4 to support connection pooling optimizations. The new library introduced a subtle change in how it handled None values during serialization. When an upstream service failed to fetch a credit score, it passed a null to the feature store. The new client library, instead of passing that null through or raising an exception, was silently defaulting to an empty string.
The downstream pipeline didn't crash because the code was "resilient." It just quietly ingested empty strings. The Spark job finished because it successfully processed the data. The model "drifted" because it was suddenly receiving empty strings for a critical feature it expected to be an integer.
The ML monitoring tool caught the drift, and the pipeline monitoring caught the latency, but because they were siloed, nobody saw the connection. The failure wasn't in the data; it was in the translation layer between the platform and the model.
The fix
We reverted the redis-py version back to 4.3.5 immediately to stop the bleeding. But the real fix was in how we handled the FeatureStore writer. I refactored the ingestion wrapper to include a strict validation step using Pydantic.
I added a FeatureSchema class that explicitly fails if a required feature like credit_score_bucket is missing or the wrong type. We stopped relying on implicit "it worked, so it's fine" pipeline logs.
class FeatureSchema(BaseModel):
user_id: int
credit_score_bucket: int
last_login: datetime
@validator('credit_score_bucket')
def must_be_int(cls, v):
if not isinstance(v, int):
raise ValueError('Feature integrity violation')
return v
By enforcing this at the entry point of the feature store, we turned a "silent model performance degradation" into a "loud pipeline failure." I would rather have a pipeline fail and alert me at 3 AM than have a silent model failure that costs us money for eight hours before someone notices the drift.
What we changed so it never happens again
We stopped running two separate monitoring stacks.
If you have a separate "Data Observability" tool that only looks at row counts and freshness, and an "ML Monitoring" tool that only looks at distribution drift, you are failing. We collapsed both into a single incident workflow centered around our observability platform.
We now use custom metrics exported from our ML models directly into our primary observability stack (in our case, Prometheus/Grafana). We don't use the black-box "drift alerts" provided by ML-specific platforms anymore. We define our own drift thresholds as part of the pipeline metadata.
When a pipeline job starts, it pushes its expected schema version and feature distribution baseline to a shared state store. If the model starts seeing data that deviates from that baseline, the same observability stack that monitors the Kafka throughput fires the alert.
We consolidated our alerts into one Slack channel: #incident-data-core. No more "Data Science" channel versus "Data Engineering" channel. If a model drifts, the engineers see it. If a pipeline lags, the data scientists see it.
The lesson here is simple: stop treating data as a "plumbing" problem and model performance as a "math" problem. They are the same problem. If your monitoring isn't telling you the story of how your data became a prediction, you’re just looking at noise.
Infrastructure and intelligence are not separate concerns. The moment you treat them as such, you’re just waiting for the next 3 AM page to prove you wrong.
Tags: #data #observability #mlops #engineering
Cover photo by Tom Martin on Unsplash.
Top comments (0)