The annual flow of the Nile at Aswan from 1871 to 1970 is one of the classic series in statistics: a hundred numbers that every time-series textbook seems to use. It is famous for one feature, a drop in the level around the turn of the century. I wanted to find that break properly, put an uncertainty on it, and then ask a practical question: how much does a forecaster actually gain from knowing about it?
Three methods, one answer
The data is the series published by Durbin and Koopman (2001), available as datasets::Nile in R: 100 annual values in units of 10⁸ m³.
| Method | Break after | Mean before → after |
|---|---|---|
Binary segmentation, one break (ruptures) |
1900 | 1,078 → 851 |
| PELT, penalty picks the number of breaks (it found one) | 1900 | 1,078 → 851 |
| Two-mean scan with a 2,000-draw bootstrap | 1898 (95% CI 1896–1902) | 1,098 → 850 |
Three methods built on different ideas land within two years of each other, and the bootstrap interval covers all of them. The mean flow falls by a little over a fifth and never recovers within the record.
I like running more than one detector on the same data. Agreement between methods with different assumptions is much stronger evidence than a single method's p-value, and when they disagree, the disagreement tells you something about the data.
Is anything left after the step?
Once the step is removed, the residuals pass a Ljung–Box test (p = 0.23). There is no year-to-year memory left: a good year does not predict another good year. So a forecast for this river is a level plus noise, and the only decision that really matters is which level.
What does knowing the break buy you?
I held out the last 20 years (1951–1970) and compared three forecasts:
| Forecast of 1951–1970 | RMSE | Mean forecast | Actual mean |
|---|---|---|---|
| AR(1) on the whole history (ignores the break) | 131 | 928 | 877 |
| AR(1) on the post-break years only | 128 | 840 | 877 |
| The post-break mean | 128 | 840 | 877 |
The model that ignores the break aims too high, the post-break models aim a little low, and the improvement is three RMSE points against a year-to-year standard deviation of about 120. Knowing about the break removes a bias. It cannot remove the noise, and it would be easy to dress up this result as a bigger win than it is.
That is the part I find most useful outside hydrology. Structural breaks show up in every business series: a pricing change, a new data source, a migration. Detecting them is worth doing, but the honest question is always how much the forecast improves, measured on data the model has not seen.
Why around 1898?
The data cannot say. Two explanations are usually given: the first Aswan dam, built in the same years and opened in 1902, and a drier period over the Ethiopian highlands, where most of the flood water comes from. A statistical change point needs a cause from outside the data before it becomes a story, and with a single gauge the two cannot be separated.
Reproduce
The notebook is generated from source and runs top to bottom:
pip install -r requirements.txt
python make_notebook.py
jupyter nbconvert --to notebook --execute --inplace nile_river_flow.ipynb
- Code and notebook: github.com/engomaressam/nile-river-flow
- Kaggle notebook: Nile at Aswan: finding the year the river changed
Omar Essam El Sherbiny. Head of Data at WalaPlus. Data platforms on Google Cloud, data governance, applied ML. More at omaressam.web.app.
Top comments (0)