How I Built an Anomaly Detection System for Critical Infrastructure

Critical infrastructure does not usually announce failure.

It whispers first.

A sensor drifts a little. A vibration pattern changes slightly. A machine still “looks fine,” but not quite. That is the problem with these systems. By the time the failure becomes obvious, the cost has already gone up.

That is why I like anomaly detection. It sits in that awkward but important space between raw data and real operational decisions. It is not just about spotting weird numbers. It is about helping people catch early warning signs before they turn into downtime, safety issues, or expensive surprises.

In this article, I want to show how I would build a practical anomaly detection system for critical infrastructure data, not as a toy example, but as something that could actually be used in the real world.

Start with the problem, not the model

A lot of ML projects fail before the first line of code because the team starts with the model. They ask, “Should we use LSTM, XGBoost, or autoencoders?” too early.

The better question is, “What does abnormal actually mean in this system?”

That answer changes everything.

In one environment, an anomaly may be a sudden spike. In another, it may be a slow drift over several hours. In another, it may be a combination of values that are individually normal but suspicious when viewed together.

If you are monitoring a pump, a transformer, a building system, or an industrial line, you are rarely dealing with a neat classification problem. You are dealing with behaviour.

That is why the first step is always to define the behaviour you expect.

Building the data pipeline

The model itself is only one piece. The real work starts with the pipeline.

Before anything else, I want clean timestamps, consistent units, and features that make sense over time. In a real system, the data might come from sensors, logs, SCADA feeds, telemetry, maintenance records, or operational dashboards.

Here is a simple way to think about the flow:

raw_data -> cleaning -> feature engineering -> anomaly scoring -> alerting -> feedback loop

That sounds simple, but the details matter.

A missing value can be harmless in one signal and disastrous in another. A duplicate timestamp can break a sequence model. A shifted time zone can make an entire dataset misleading.

So before training anything, I would usually do something like this:

import pandas as pd

df = pd.read_csv("sensor_data.csv")
df["timestamp"] = pd.to_datetime(df["timestamp"])
df = df.sort_values("timestamp")

# basic cleaning
df = df.drop_duplicates(subset=["timestamp"])
df = df.fillna(method="ffill").fillna(method="bfill")

That is not glamorous, but it is the kind of unglamorous work that makes the rest of the pipeline usable.

Feature engineering is where the signal appears

Raw sensor values are rarely enough.

A reading might be normal on its own, but suspicious when compared to the last few minutes or the long-term baseline. That is why rolling features are so useful in anomaly detection.

I usually start with simple temporal features like rolling mean, rolling standard deviation, lag values, and rate of change.

window = 5

df["rolling_mean"] = df["sensor_value"].rolling(window).mean()
df["rolling_std"] = df["sensor_value"].rolling(window).std()
df["lag_1"] = df["sensor_value"].shift(1)
df["lag_2"] = df["sensor_value"].shift(2)
df["delta"] = df["sensor_value"] - df["lag_1"]

Once you do this, the data starts telling a better story.

A single spike becomes easier to spot. A slow drift becomes more visible. A pattern of instability starts to stand out.

If you are working with infrastructure data, those little temporal shifts often matter more than the raw reading itself.

A strong baseline is always worth it

I like starting with a baseline model before moving to something more complex.

In many cases, Isolation Forest is a strong place to begin because it is quick, works well on tabular features, and gives you a usable anomaly score without needing labelled examples.

from sklearn.ensemble import IsolationForest
from sklearn.preprocessing import StandardScaler

features = ["sensor_value", "rolling_mean", "rolling_std", "lag_1", "delta"]

X = df[features].dropna()

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

model = IsolationForest(
    n_estimators=100,
    contamination=0.02,
    random_state=42
)

predictions = model.fit_predict(X_scaled)

df.loc[X.index, "anomaly_flag"] = predictions == -1

That gives you a first pass.

It will not be perfect, and it is not meant to be. What it does give you is a reference point. If this baseline is already useful, you may not need something much heavier. If it is weak, then you know where to improve.

That saves a lot of time.

Why sequence models become useful

Some anomalies are not obvious in a single row.

They only make sense when you look at the sequence.

That is where models like LSTM, BiLSTM, and GRU become useful. They are especially helpful when the order of events matters and when the system’s behaviour depends on what happened before.

For example, imagine a machine that slowly vibrates out of range, returns to normal, then drifts again. One reading may look harmless. The sequence tells the real story.

A simple GRU-style setup might look something like this:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import GRU, Dense

model = Sequential([
    GRU(64, input_shape=(30, 1), return_sequences=False),
    Dense(32, activation="relu"),
    Dense(1, activation="sigmoid")
])

model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])

Of course, in a real anomaly detection setup, you may not even have labels in the traditional sense. In that case, the model may be trained to reconstruct normal sequences, and then reconstruction error becomes the anomaly signal.

That is often more useful than forcing a classification label onto a problem that is really about behaviour.

Turning scores into action

A model is not useful if it just says “something is strange.”

Operators need priority.

They need to know what to look at first.

That is why anomaly scoring matters so much. Instead of only returning a yes or no result, I prefer systems that assign a severity score.

df["anomaly_score"] = model.decision_function(X_scaled)

Once you have a score, you can rank events by urgency, trigger alerts above a threshold, and send only the most important cases for human review.

That makes the system much more usable.

A score also gives you flexibility. You can tune sensitivity based on the context. In a low-risk situation, you may want fewer alerts. In a high-risk operational setting, you may want to catch more potential issues, even if that means reviewing more false positives.

Validation is where trust is won

This is the part people often rush.

They train a model, see a decent metric, and assume the system is ready.

It is usually not.

For critical infrastructure, validation has to respect time. You cannot randomly shuffle data and call it a day. That creates leakage and gives you an unrealistic view of performance.

Instead, I would validate using time-aware splits and measure things like recall, precision, false alarm rate, and detection delay.

The question is not just “Did the model find anomalies?”

The better question is “Did the model find them early enough, often enough, and without creating too much noise?”

That is a very different standard.

If your model misses rare but important anomalies, it can still have a great accuracy score and still be completely wrong for the business.

Deployment is part of the model

One thing I have learned is that a model does not become useful when training finishes. It becomes useful when it starts helping someone make decisions.

That means deployment matters.

A practical production setup might look like this:

def score_new_batch(batch_df, model, scaler, features):
    X_new = batch_df[features].fillna(0)
    X_new_scaled = scaler.transform(X_new)
    scores = model.decision_function(X_new_scaled)
    batch_df["anomaly_score"] = scores
    return batch_df

From there, the output can feed a dashboard, a notification system, or an operations workflow.

But deployment is not the end either. You need monitoring. Data changes. Behaviour changes. Business operations change. A model that worked well last month may drift quietly over time.

So I would also monitor:

- feature drift
- alert frequency
- false positives
- missed incidents
- score distribution changes

That feedback loop is what keeps the system alive.

The human layer matters

This part is easy to ignore, but it is one of the most important.

In critical infrastructure, a good anomaly detection system should not replace human judgment. It should support it.

The best systems I have seen do not just produce alerts. They help teams investigate, confirm, reject, and learn from them.

That means building room for feedback.

If an operator marks an alert as useful, that becomes valuable signal. If they say it was noise, that matters too. Over time, that feedback helps improve thresholds, retraining decisions, and the overall quality of the system.

This is where a model becomes part of an operational workflow instead of just a research experiment.

What the final system should do

At the end of the day, the goal is not to have the fanciest model.

The goal is to catch meaningful changes early, reduce blind spots, and give teams a better chance to act before a small issue becomes a big one.

A good anomaly detection system for critical infrastructure should be able to:

- learn normal behaviour
- detect unusual patterns
- rank alerts by severity
- adapt to changing conditions
- support human review

If it does those things well, it is already doing something valuable.

Final thoughts

Anomaly detection is one of those areas where good engineering matters just as much as good modelling.

You need clean data, thoughtful features, a sensible baseline, a way to score risk, and a feedback loop that keeps the system honest.

That is the real work.

And honestly, that is also what makes it interesting.

Because when done well, anomaly detection is not just about spotting unusual numbers. It is about building systems that help people respond sooner, work smarter, and protect things that matter.