AR, MA, and ARMA

The smoothing methods from the previous posts; SES, DES, Holt-Winters work by taking weighted averages of past observations. This post introduces a different family: AR, MA, and ARMA models. Instead of smoothing, these treat forecasting as a regression problem. The predictors are either past observed values, past forecast errors, or a combination of both.

The models

AR(p) — Autoregression

An AR model predicts y_t as a linear combination of its own past values:

AR(1):  y_t = a_1 · y_{t−1} + ε_t

AR(2):  y_t = a_1 · y_{t−1} + a_2 · y_{t−2} + ε_t

AR(p):  y_t = a_1 · y_{t−1} + … + a_p · y_{t−p} + ε_t

The coefficients a_1, …, a_p are estimated by ordinary least squares — exactly like standard regression, except the predictors are lagged versions of the same variable. Conceptually, this is similar to SES: both use past observations to forecast the next value. The difference is that SES uses a single α to control how all lags decay together, while AR assigns a separate coefficient to each lag.

MA(q) — Moving Average

Rather than past observed values, MA models use past forecast errors:

MA(1):  y_t = m_1 · ε_{t−1} + ε_t

MA(q):  y_t = m_1 · ε_{t−1} + … + m_q · ε_{t−q} + ε_t

If the model has been consistently underestimating, those positive past errors carry forward and help correct future predictions. Note: this has nothing to do with the moving averages used in decomposition — the name refers to a weighted average of error terms, not observed values.

ARMA(p, q) — Both combined

ARMA(1,1):  y_t = a_1 · y_{t−1} + m_1 · ε_{t−1} + ε_t

ARMA(p, q) = AR(p) + MA(q)

The AR part learns from past values; the MA part learns from past errors. Together they’re more expressive than either alone — in theory.

Applying them to the airline series

All three methods come with an important constraint: they assume the series is stationary — no trend, no changing variance, no systematic seasonality. The airline series has all three. Rather than transforming the data first, let’s apply these models directly and see what happens.

AR results

AR(1) and AR(2) produce nearly flat forecasts — the model anchors to the recent level and has no mechanism to project growth or seasonality. AR(12) does better by incorporating observations from 12 months ago, which accidentally picks up some seasonal structure. Even so, the MAE of 56 is far worse than the Holt-Winters result of 13.

MA results

The MA models tell essentially the same story. Small q values produce flat lines that can’t follow the trend. MA(12) again picks up some seasonal pattern through the lagged error structure, but the fundamental limitation remains: none of these models know how to project a trend.

ARMA comparison

Even ARMA(12,2) — which uses 12 AR lags and 2 MA terms — only achieves a MAE of 56. The forecast follows the broad seasonal shape but consistently underestimates because the upward trend is not being modelled. Combining AR and MA doesn’t help when the core problem is non-stationarity.

One pattern worth noting: AR and MA models with the same order produce nearly identical MAE values. This isn’t a coincidence — on a non-stationary series, both methods end up anchoring to the same level and failing in the same way. The dominant effect is the trend, which neither can model, so the difference between “use past values” and “use past errors” becomes irrelevant.

Why they fail: the ACF

The autocorrelation function (ACF) makes the problem visible. For a stationary series, the ACF drops off quickly — lags beyond a few steps become uncorrelated. For a non-stationary series, correlations decay slowly and stay high for many lags.

Lag 1 has autocorrelation above 0.9, and significant correlations persist all the way to lag 24. This is the signature of a non-stationary series — the trend is carrying information forward indefinitely. AR, MA, and ARMA cannot handle this directly.

What’s missing

The problem isn’t the regression framework — it’s that the series needs to be made stationary before these models can work properly. One approach is to difference the series manually and then apply ARMA to the result. A cleaner solution is to build the differencing step directly into the model.

That’s what ARIMA does. The “I” in ARIMA stands for “integrated” — meaning the model includes a differencing step as part of its specification. Instead of requiring stationary input, ARIMA handles the transformation internally, applies ARMA to the result, and then inverts the differencing to produce forecasts on the original scale. That’s the next post.

Figures generated using the airline passengers dataset (monthly, January 1949–December 1960).