Real-Time Forecasting Without Retraining

The Real Bottleneck Is Not Accuracy

For more than a decade, time series modelling has improved significantly: better handling of non-stationarity, increased robustness to noise, support for multiple horizons, and strong performance on controlled benchmarks. These are real advances. But in production systems — non-stationary, distributed, resource-constrained, continuously evolving — a different constraint dominates.

The challenge is no longer how well models can predict, but how fast they can adapt without stopping. Most approaches still rely on a cycle of training on historical data, deploying the model, and retraining as performance degrades. At small scale, the delay between system change and model update is manageable. At large scale, it becomes structural.

The Hidden Assumption: Train → Freeze → Predict

Most time series systems — statistical, machine learning, or deep learning — follow the same cycle: train on historical data, freeze the model, use it for prediction, and retrain when performance degrades. This is not a limitation of specific algorithms. It is an underlying assumption about how learning should occur: only when necessary.

While intuitive, this approach introduces a critical limitation in continuously evolving systems: the model cannot learn while it is acting. Each adaptation requires retraining, creating a learning gap where the system has already changed but the model still operates on outdated assumptions.

The scalability challenge

As the number of time series grows, the cost of retraining scales rapidly: more signals require more models, more drift demands more frequent updates, and more updates increase compute, latency, and operational complexity. This leads to a structural trade-off — either simplify models to keep them deployable, or centralise processing and absorb the latency cost. In both cases, adaptability is compromised.

The Latency of Learning

In most systems, learning is not continuous. It happens in discrete steps, triggered by retraining cycles. Between those steps, the model operates on assumptions that are already becoming outdated. This is the latency of learning: the delay between a change in the system and the model's ability to incorporate it.

In static environments, this delay is acceptable. In continuously evolving systems, it becomes the defining limitation. When learning is delayed, everything downstream is affected:

• Anomalies are detected later than they occur
• Predictions reflect past conditions rather than present ones
• Decisions are made with partial awareness of what is actually happening

A Different Approach: Continuous Learning

Most time series approaches treat the problem as function approximation: learn a global relationship from past data and use it to predict the future. But in continuously evolving systems, signals do not behave like stable functions. They behave like sequences of transitions between states.

Instead of asking "what is the function that fits this data?", a different question emerges: "what are the patterns, and how do they evolve over time?"

In this view, learning is no longer a periodic event — it becomes a continuous process. Each new observation updates the internal representation, refines existing patterns, or creates new ones as the system evolves. There is no retraining cycle. There is no freeze phase. Learning and inference are no longer separate steps; they become the same operation, happening continuously as data flows.

For the full architectural details — online behavioural clustering, Temporal Transition Graphs, the three forecasting paths, and how patterns are stored as structural memory — see the DriftMind architecture deep-dive.

The Benchmark: A Direct Comparison Under Continuous Load

To evaluate this approach under realistic conditions, DriftMind was benchmarked against two widely used baselines: Adaptive ARIMA using Kalman Filters and Triggered Prophet. The setup was intentionally simple — all models exposed to the same input data, simulating a continuous streaming scenario, with no HTTP overhead and no artificial batching. Each model runs in-process, point by point, predicting the next value before observing the actual one.

To avoid the classic trap of single-dataset benchmarks, the same protocol was run on four datasets from the Numenta Anomaly Benchmark (NAB), chosen to span different domains and drift profiles:

How "Adaptive" and "Triggered" Actually Work

Both baselines need a retraining strategy, because neither ARIMA nor Prophet learns online. The question is when to retrain, and that decision matters more than people usually admit — too aggressive and you spend all your compute rebuilding models; too lazy and you drift away from the signal. The trigger had to be defensible for both.

Each baseline uses the same two-condition trigger, applied independently after every prediction:

1. MASE breach (with debounce)

Track the mean absolute scaled error over a rolling window of the last 10 predictions. If that windowed MASE exceeds 3.0 — meaning the model is, on average, 3× worse than a naïve persistence forecast — for 20 consecutive steps, the model retrains on the last 200 observations. The 20-step debounce is important: it prevents single-spike anomalies from triggering a retrain. The model has to be sustainably wrong, not momentarily surprised.

2. Correlation collapse

Over a rolling window of the last 100 predictions, compute the Pearson correlation between predicted and actual values. If that correlation drops below 0.20, the model retrains immediately, with no debounce. This catches the case where the model isn't necessarily large-error but has lost the shape of the signal — predicting the right magnitude in the wrong direction.

The retrain itself fits the model on a sliding window of the most recent 200 points. Both baselines start from the same 200-point warm-up before they make their first prediction. DriftMind, by contrast, predicts from observation #1.

What the retrain counts in the results table are telling you, then, is not "how often did I force these models to retrain" but "how often did these models themselves admit they were misaligned with the data."

Results Across All Four Datasets

Green values indicate the winning model on each metric. ARIMA(5,1,0) and Prophet with changepoint_prior_scale=0.3, both wrapped in identical MASE+correlation retrain triggers. All models run in-process on the same CPU.

What the Numbers Actually Show

Two things become clear when you stop looking at a single dataset.

On drift-heavy series, DriftMind wins both accuracy and speed

Machine Temperature and CPU Utilization both contain real-world failure events and gradual concept drift. On these, DriftMind achieves the lowest MAE and the highest throughput, by margins of 344× and 1,225× respectively. ARIMA's 5,837 retraining cycles on the CPU dataset, and Prophet's 17,486, are not anomalies — they are what triggered retraining looks like when the data is actually changing. Each retrain is the model conceding that it is misaligned with the present.

On stable series, ARIMA catches up on accuracy

On NYC Taxi Demand and Ambient Temperature, ARIMA achieves slightly lower MAE than DriftMind — and notably, with zero retraining cycles on both. This is exactly what theory predicts. When the underlying signal is well-behaved and the initial fit is good, a parametric model that converges once and stays there is hard to beat on pure accuracy.

But even here, DriftMind is 362× faster on Taxi and 248× faster on Ambient, with no warm-up window. ARIMA needs 200 points of history before it can make its first prediction. DriftMind starts forecasting from the very first observation. In a production system processing thousands of new time series per day — new sensors, new tenants, new endpoints — that warm-up requirement alone changes the operational economics.

This is not a marginal improvement. It is a fundamentally different scaling behaviour — and crucially, a different failure mode. ARIMA and Prophet can be made accurate at the cost of compute. DriftMind cannot be made faster than it already is, but its accuracy degrades gracefully on data it has never seen, without ever stopping to retrain.

A note on Prophet

Prophet is included as a reference, not because it's well-suited to high-frequency industrial data — it isn't. Prophet was designed for daily and weekly business time series with strong calendar seasonality. On 5-minute machine sensor signals, it's being asked to run a marathon in dress shoes. But it remains the most widely deployed forecasting library in industry, and many teams still reach for it by default. This benchmark shows what happens when you do.

Anomaly Detection Comes for Free

Because DriftMind tracks transitions between observed states, anomaly detection isn't a separate model — it falls out of the architecture. Every forecast is returned alongside an anomaly score derived from how well the current observation matches the learned transition graph. There is no additional model to train, deploy, or version. There is no second inference pipeline.

For the telecom and IoT use cases DriftMind is deployed in production for, this is the more commercially interesting capability. Forecasting tells you what the system will look like. Anomaly scoring tells you when it stops looking like itself. Both come from the same single-pass operation.

Why This Changes the Economics of AI

Removing retraining is not just a performance optimisation. It changes the cost structure of operating a forecasting system at scale:

In environments such as telecom or IoT, this distinction is critical. The question is no longer whether models can handle drift in theory, but whether they can do so under real-world constraints of scale, cost, and latency.

Reproduce It Yourself

The full benchmark — all four datasets, all three models — runs from a single Docker command:

docker run -p 8080:8080 -p 8888:8888 thngbk/driftmind-edge-lab

Then open http://localhost:8888 and run multi_benchmark.py. The script downloads each NAB dataset directly from the official repository, runs all three models in-process, and writes the results to benchmark_results.json. No cloud credentials, no API keys, no hidden configuration.

You can verify any number in the table above on your own laptop in under an hour.