Zum Inhalt springen
Noether Analytics

How we judge a forecast

A vendor tells you their forecast is 95% accurate. Before that number means anything, you need three more pieces of information, and vendors rarely volunteer them.

Which error, in which unit

"Accuracy" has no single definition. The most common one in practice is 100% minus the mean absolute percentage error, or MAPE. It has a known flaw: it punishes errors on small numbers much more than errors on large ones. If you sell 2 units of a part and the forecast said 4, that is a 100% error. If you sell 2,000 units of another part and the forecast said 2,400, that is a 20% error, although it is the second mistake that fills your warehouse.

We prefer metrics in the unit you plan in. The mean absolute error in units per week tells a planner how far off the forecast is on a typical week. Where products differ a lot in volume, we scale the error by what a naive forecast would have achieved on the same product, which makes the result comparable across the range.

For forecasts that come with a range, there is a second question: is the range honest? If we say demand will fall inside the range in 8 weeks out of 10, we count how often it did. A model whose 80% range contains the outcome 60% of the time is overconfident, and stock levels built on it will be too low.

Compared with what

A forecast error of 12% is good or bad depending on what was achievable. So every result needs a baseline, and we use two.

The first is your current method, whatever it is: the planner's judgement, a spreadsheet, the forecast module of your ERP system. That comparison answers the business question of whether a new model is worth having.

The second is a simple benchmark, such as "the same as the same week last year". It costs nothing to produce, and a surprising number of elaborate models fail to beat it. If ours does not, the honest conclusion is that the data does not support a better forecast, and we tell you so.

Tested on which period, and how

It is easy to build a model that explains the past perfectly and predicts poorly. The safeguard is a backtest that imitates real use. We go back to a date in the past, give the model only the data that existed on that date, let it forecast the following weeks, and compare with what happened. Then we move the date forward and repeat. Over two years of weekly data, that produces about a hundred forecasts made under realistic conditions.

Two details decide whether a backtest can be trusted. The test period must be long enough to include the difficult stretches: a seasonal peak, a price shock, a product launch. And nothing from the future may leak into the model, including through data that was corrected or completed later.

Some questions cannot be answered on history at all. If the forecast changes what you do, and what you do changes the outcome, only a live test shows the effect.

What we agree before a pilot

Before a pilot starts, we write down the metric, the two baselines and the test period together with the client. When the results come in, there is then no room to choose the comparison that flatters the model.

Alle Beiträge