AI vs Reality
Do the new pretrained AI forecasting models predict the real world better than classic methods? We replayed nearly 2,000 forecasts of German train delays and electricity prices, giving every model exactly the same information. New here? Start with the 45-second film below.
The question
New pretrained AI forecasting models (the same idea as ChatGPT, but trained on millions of number series instead of text) promise to predict almost anything without extra training. Do they really beat older, simpler methods on messy real-world data?
How we tested it, fairly
We picked two things Germany runs on: train delays at five big stations and wholesale electricity prices. Then we went back in time and made 18,628 forecasts. Each time, every model saw exactly the same past data and made its prediction; only then did we compare it with what really happened. No model could peek at the future.
What we found
1. Trains: a tie, and nobody sees a disruption coming. All serious models were about 33% more accurate than the simple rule "same as yesterday at this hour". But the best AI model and the best classic machine-learning model were only 0.07 minutes apart. When Frankfurt (Main) Hbf suddenly hit 61 minutes of average delay (the hour after Thu 05 Feb 2026, 02:00), every model had expected between 1.5 and 7.1.
2. Electricity prices: AI wins clearly. Pretrained AI models were about 33% more accurate than repeating yesterday's prices; classic machine learning only 19%. AI was also far better at spotting hours when prices go below zero, which happens when there is so much wind and sun that producers pay to get rid of power.
3. It is worth real money. A business that must use 4 MWh of electricity a day but can choose the hours, and follows the AI forecast to pick the cheapest ones, saves 40 € a day compared with always running at midday. That is 74% of what perfect knowledge of the future would save.
4. The honest catch. AI is not magic. On trains it was no better than models trained in seconds on a laptop, and no model predicts sudden disruptions from history alone.
5. New in the arena. German weather stations (30 cities, hourly temperature) and wind + solar generation are now benchmarked with the same protocol - early results are in the Leaderboard and Replay tabs.
Try it yourself in Replay a forecast: pick a moment, bet on a model, reveal reality.
- Forecast: a prediction of a future value, made at one specific moment (the yellow line).
- Pretrained AI (foundation model): Timestral 28M and 209M (Mistral), Chronos-2 (Amazon), TimesFM 3.0 (Google). Trained once on huge collections of time series and used here as-is, without retraining on this data ("zero-shot").
- Classic machine learning: LightGBM and XGBoost. Retrained on this data before every forecast; fast, runs on a laptop.
- Simple rules (baselines): "same as yesterday at this hour" (seasonal naive) and "the typical value for this hour of the week" (climatology). A useful model must beat them.
- Average miss (MAE): how far a prediction is from reality, on average. 1.8 minutes means typically off by about 1.8 minutes.
- Better than 'same as yesterday' (%): how much smaller the average miss is than the simple rule. +33% means the miss is a third smaller.
- 80% range: the band a model says reality will fall into 8 times out of 10. A good model's band really does contain reality about 80% of the time.
- Negative price: hours with so much wind and solar power that sellers pay buyers to take electricity.
- €/MWh: euros per megawatt-hour, the wholesale unit. One MWh is roughly a third of what a typical German household uses in a year.
Pick a moment, choose your contenders, call the winner. Then reveal what really happened.
Runtime is wall-clock seconds for every forecast of the run on one CPU; compare orders of magnitude, not decimals.
A flexible consumer moves 4 MWh a day into the hours a model predicts are cheapest, then pays the real auction price. Wholesale cost only, not a household bill.
How the arena works
- Rolling origins. Forecasts are replayed chronologically; every model sees exactly the same history, truncated at the moment the forecast is made. No shuffling, no peeking.
- Trains (ADR-0001): hourly average arrival delay at 5 hub stations, forecast 1 to 24 hours ahead from completed hours.
- Power (ADR-0002): forecasts made at 10:00 the day before, ahead of the day-ahead auction, for all 24 delivery hours.
- Foundation models run zero-shot (Timestral 28M/209M, Chronos-2, TimesFM 3.0); gradient boosting (LightGBM, XGBoost) is retrained at every origin on lag and calendar features.
- Thresholds frozen on training data before any test forecast was scored.
Data: piebro/deutsche-bahn-data (CC-BY-4.0), SMARD Bundesnetzagentur (dl-de/by-2-0). Everything on this page is read from committed result files; nothing is inferred live.
- Forecast: a prediction of a future value, made at one specific moment (the yellow line).
- Pretrained AI (foundation model): Timestral 28M and 209M (Mistral), Chronos-2 (Amazon), TimesFM 3.0 (Google). Trained once on huge collections of time series and used here as-is, without retraining on this data ("zero-shot").
- Classic machine learning: LightGBM and XGBoost. Retrained on this data before every forecast; fast, runs on a laptop.
- Simple rules (baselines): "same as yesterday at this hour" (seasonal naive) and "the typical value for this hour of the week" (climatology). A useful model must beat them.
- Average miss (MAE): how far a prediction is from reality, on average. 1.8 minutes means typically off by about 1.8 minutes.
- Better than 'same as yesterday' (%): how much smaller the average miss is than the simple rule. +33% means the miss is a third smaller.
- 80% range: the band a model says reality will fall into 8 times out of 10. A good model's band really does contain reality about 80% of the time.
- Negative price: hours with so much wind and solar power that sellers pay buyers to take electricity.
- €/MWh: euros per megawatt-hour, the wholesale unit. One MWh is roughly a third of what a typical German household uses in a year.