The previous post showed that AR, MA, and ARMA break down on the airline series because they assume stationarity. The common fix is to difference the series manually and then fit ARMA on the result. ARIMA formalises exactly this. The “I” stands for integrated, and it refers to the differencing step being built directly into the model. No manual pre-processing needed.
What differencing does
When a series has a trend, consecutive values are highly correlated, yesterday predicts today almost perfectly, simply because the series keeps going up. This makes it hard for a model to learn anything beyond “more of the same.” Taking the first difference, y_t − y_{t−1}, removes that trend and leaves behind only the change from one period to the next — a much more stationary quantity.
Figure 1. ACF of the raw series (left) and after first-differencing (right).
On the left, the raw series has autocorrelations that barely decay. lag 1 is near 0.9, and significant correlations persist through lag 24. After one round of differencing (right), most lags fall within the confidence bands. The trend is gone. One difference — d=1 — is sufficient.
The model
ARIMA(p, d, q) has three parameters:
p — AR order: how many lagged observed values to included
d — degree of differencing: how many times to difference the series
q — MA order: how many lagged error terms to include
With d=1, the model internally computes y_t − y_{t−1} and fits ARMA(p, q) on that differenced series. Forecasts are then inverted back to the original scale. Unlike ARMA on raw data, ARIMA with d=1 can track a trend — in that sense it’s analogous to DES in the smoothing family: both add a mechanism for a trending series that ARMA or SES alone can’t capture.
Fitting ARIMA(1,1,0)
Starting with the simplest non-trivial model — one AR lag, one round of differencing, no MA:
Figure 2. ARIMA(1,1,0) forecast on the airline series.
The forecast follows the upward trend — already a genuine improvement over ARMA on raw data. However, seasonal oscillations are entirely absent. ARIMA has no component for seasonality, so the forecast runs through the middle of the test values without ever oscillating.
Choosing p, d, q: AIC and BIC
With d fixed at 1, the question is how many AR lags to include. Two widely used criteria help here.
AIC — Akaike Information Criterion
AIC = n · log(SSR/n) + 2k
where SSR is the sum of squared residuals on the training set and k is the number of model parameters. AIC rewards a good fit but penalises complexity. The intuition is that adding more parameters always improves training fit — AIC corrects for this by charging a cost of 2 per additional parameter. Lower AIC is better.
BIC — Bayesian Information Criterion
BIC = n · log(SSR/n) + k · log(n)
BIC applies a stronger penalty than AIC — log(n) per parameter instead of 2. As a result, BIC tends to favour simpler models, especially on larger datasets. When AIC and BIC agree, the choice is clear. When they disagree, AIC leans toward the more complex model and BIC toward the simpler one.
Both are computed on training data only — no test set needed. This makes them useful when you want to select a model without touching held-out data, or when no held-out set exists.
Figure 3. AIC and BIC for ARIMA(p,1,0), p = 1 to 5. Green bar = best by AIC.
Both criteria decrease as p grows from 1 to 5, with ARIMA(5,1,0) coming out best. The improvement from p=4 to p=5 is small — a sign that adding more lags is buying less and less.
Optimised forecast
Figure 4. ARIMA(5,1,0) forecast — selected by AIC.
The trend is well captured and the MAE drops to 53.3, an improvement over ARIMA(1,1,0). Even so, the forecast remains a smooth line that ignores the annual seasonal cycle. The gap to Holt-Winters (13.1) is almost entirely due to seasonality.
What comes next
ARIMA handles trend via differencing but has no mechanism for seasonality. The natural extension is SARIMA — Seasonal ARIMA — which adds a second set of parameters (P, D, Q, s) working at the seasonal lag. With s=12 for monthly data, seasonal differencing removes the annual pattern the same way regular differencing removes the trend. That’s the next post.
Figures generated using the airline passengers dataset (monthly, January 1949–December 1960).
Leave a Reply