Time Series with Machine Learning (Part 2)

Part 2: Teaching the Model About the Past

Lag Features

Calendar features: month, day of week, and is_weekend tell the model what time it is. That is useful for capturing seasonality and weekly rhythms, but it leaves out something equally important: what actually happened in the recent past. A store that sold 90 units last month is probably in a different situation than one that sold 20. The model has no way of knowing that from the calendar alone.

Lag features solve this by packaging historical sales values directly as input variables. Instead of asking the model to infer momentum from indirect signals, you hand it the actual numbers. This post covers how it works, why the lag distances are chosen the way they are, and what the random noise term does in the mix.

What Is a Lag Feature?

A lag feature is simply the value of a variable at some earlier point in time. If today is day t and yesterday’s sales were y(t−1), then lag1 is y(t−1). Lag2 is y(t−2), and so on. Each row in the dataset pulls the value from k days earlier and carries it forward as a new column.

Concretely, the shift operation moves the series down by k rows. The first k rows have no history to look back on, so they receive NaN. Everything after that gets the value from k days prior.

• sales at t=0  →  lag1 = NaN  (no prior observation)
• sales at t=1  →  lag1 = sales(t=0)
• sales at t=2  →  lag1 = sales(t=1),  lag2 = sales(t=0)

One subtlety matters here: the data must be sorted correctly before shifting. With 500 distinct store-item series concatenated into one dataframe, a naive global shift would bleed the last sale of one series into the first row of the next. Sorting by store, item, and date first and then computing lag within each group using groupby-transform keeps each series isolated.

Why These Specific Lag Distances?

The lag distances are not arbitrary. The forecast horizon determines the minimum usable lag, and domain knowledge about seasonality determines the rest.

The competition asks for 3-month-ahead forecasts, roughly 91 days. That creates a hard lower bound. When the model is asked to predict sales on day t, the most recent known sales figure is from day t−91 (everything after that belongs to the forecast period and is unavailable). Using any lag shorter than 91 would mean looking at data that does not exist at prediction time, which is data leakage.

Within the safe window, the lag choices reflect the structure of the data. The first cluster, 91, 98, 105, 112, 119, 126, steps through a six-week span in weekly increments starting three months back. This approach captures the seasonal pattern around the same period last quarter while also giving the model some texture: rather than a single snapshot, it sees how demand was moving week by week in that window. Longer lags at 182, 364, 546, and 728 days anchor the model to the same period six months ago, one year ago, and two years ago, each a useful reference point for annual seasonality and multi-year trends.

Lag (days)	What it looks back to	Purpose
91, 98, 105, 112, 119, 126	Weekly steps from 3 months back	Core seasonal window
182	6 months back	Mid-year seasonality
364	1 year back	Same period, last year
546	1.5 years back	Extended memory
728	2 years back	Longest memory

Table 1. The ten lag features used and their seasonal rationale.

What Happens to the NaN Values?

Every lag feature starts with missing values. Lag91 has NaN for the first 91 rows of each store-item series; lag728 has NaN for the first 728 rows, nearly two years’ worth. With 500 series and lags up to 728 days, a substantial portion of the training data carries missing lag values.

In this case, that is not a problem. The modeling approach uses gradient boosted trees (LightGBM), which handle NaN natively: at each split, observations with missing feature values are routed to whichever branch reduces loss more. The model learns the best direction for missing values from the data itself, so no imputation is needed.

There is also a structural reason to expect the NaN count to shrink naturally. The training data starts in January 2013. A lag91 value for any observation on or after April 2013, three months into the series, will be available. By the time the model sees the years with the most data, the vast majority of lag features are populated.

With date features and lag features in place, the dataset now carries two complementary kinds of signal. Calendar features encode when in the year each observation falls. Lag features encode what the sales trend looked like at comparable past moments. Together, they give a tree-based model the raw material to learn both seasonal patterns and recent momentum, without ever explicitly telling it what a trend or a season is.

The Role of Random Noise

Before getting to the noise, it helps to understand what overfitting actually looks like in practice.

Imagine you are training a model and one of your input features is lag91, the sales value from 91 days ago. For a particular row in your training data, lag91 happens to be exactly 41.0, and the actual sales that day were 43. The model sees this pair (41.0 → 43) and learns from it.

Now imagine that same exact value, 41.0, shows up hundreds of times across your 500 store-item series, because many stores sold similar amounts three months ago. Each time, the actual sales are slightly different: 40, 45, 39, 47. But the input is always the same number. A powerful tree model, given enough depth, can learn to treat “lag91 = exactly 41.0” as its own special case and memorize the average output for that exact value. It stops learning a general relationship between past and future sales. It starts pattern-matching on specific numbers.

This works beautifully on training data. It fails on anything new, because the next time lag91 is 41.0, the sales will not be 43 again. It will be whatever it happens to be that day. The model has no generalization to fall back on.

The fix is to make sure no lag value is ever exactly the same twice. You do this by adding a small, random number to each lag observation before it is added to the training data.

That random number is drawn from a normal distribution — also called a bell curve. Here is why that choice matters. A normal distribution centered at zero means the noise is equally likely to push the lag value up or down, and most draws will be small, with large deviations being rare. If you used a uniform distribution instead — say, a random number between −5 and +5 — every size of perturbation would be equally likely, including ones large enough to meaningfully change the value. The normal distribution concentrates the noise near zero, which means most lag values shift by only a tiny amount. The signal stays intact; only the exact value changes.

Concretely, each lag value gets perturbed like this:

$sales_{lag_{91}}  =  y(t − 91)  +  ε,    ε ~ N(0, 1.6²)$

ε (epsilon) is the noise term — a single random draw from a normal distribution with mean 0 and standard deviation 1.6. The mean being 0 means the noise does not systematically push values up or down; it just introduces variation. The standard deviation of 1.6 controls the typical size of the perturbation.

To feel how small 1.6 actually is: the median daily sales in this dataset are 47 units, and the natural day-to-day standard deviation is about 29 units. Adding noise at σ = 1.6 is a perturbation of roughly 3–4%. It is like measuring a 100 cm table and writing down 100.3 or 99.7, close enough to be useful but different enough that no two measurements are ever identical.

The result is that the model can no longer find two training rows with exactly the same lag value. It cannot memorize. It has to learn the general shape of the relationship—”higher past sales tend to mean higher future sales”—rather than the exact coordinates of individual data points. Meanwhile, the overall signal is preserved almost completely. The correlation between the noisy lag and actual sales sits at r ≈ 0.24, virtually identical to the clean version. The trend is still there. The noise just sanded off the sharp edges that invited memorization.

This is a lightweight form of regularization, conceptually related to dropout in neural networks or label smoothing in classification. The key requirement is that the noise be small relative to the natural variance of the target. Sales values in this dataset have a standard deviation of roughly 29 units, so injecting noise at σ = 1.6 represents about a 5% perturbation. Large enough to break exact repetition; small enough to leave the seasonal pattern intact.