Time Series with Machine Learning (Part 9)

Part 9: The Final Model and Submission In this part, we will cover machine learning model testing as part of preparing your final submission.

Everything up to this point has been preparation. The feature engineering produced 142 inputs. The validation run found the optimal number of boosting iterations and confirmed a validation SMAPE around 13.6%. Feature importance analysis identified which of those 142 features actually contributed. Now all of that feeds into one final step: train the model on every available data point, generate predictions for the test period, and produce a submission file.

Splitting Train and Test

At this stage, the dataframe still contains both training rows (with known sales) and test rows (where sales is NaN—the values to predict). The split is straightforward: rows with known sales become the training set, while rows with NaN values become the test set.

train  =  df.loc[~df.sales.isna()]   # 2013-01-01 to 2017-12-31

test   =  df.loc[ df.sales.isna()]   # 2018-01-01 to 2018-03-31

This differs from the earlier validation split. During validation, we held out a slice of 2017 data as a proxy test set to tune the hyperparameters. Now that tuning is complete, the model trains on all rows with known sales—all five years of data. With more data available, it should produce slightly better predictions than the validation run suggested.

Why Refit on the Full Dataset?

The validation model’s purpose was to find the right number of boosting iterations—when to stop training before the model starts overfitting. That number (model.best_iteration) is now known, so there’s no reason to hold data back for evaluation. The final model can train on the full training dataset.

The final model keeps the same hyperparameters as the validation run, with one critical change: it removes early_stopping_rounds and sets num_boost_round to model.best_iteration. Without a validation set to monitor, the model simply trains for the optimal number of iterations determined during validation and then stops.

lgb_params[‘num_boost_round’] = model.best_iteration

final_model = lgb.train(lgb_params, lgbtrain_all, num_boost_round=model.best_iteration)

One optional step before training: drop the zero-importance features identified in the previous post. The feature list cols is filtered into imp_feats, excluding any column with a gain score of zero. This reduces the feature matrix slightly, speeds up training, and removes noise that was never contributing signal.

Generating and Converting Predictions

Once the final model is trained, a single call generates predictions for the test period. The model takes X_test — the feature matrix for the 45,000 test rows — and returns one predicted value per row.

test_preds = final_model.predict(X_test, num_iteration=model.best_iteration)

These predictions are on the log scale because we transformed the target variable with log1p before training. The raw output looks like this:

array([2.553, 2.742, 2.773, …, 4.375, 4.412, 4.471])

These are log(1 + sales) values, not unit sales. Before submission, we convert the predictions back to actual sales using expm1, the exact inverse of log1p. For any value x, expm1(x) = eˣ − 1.

After conversion, the predicted sales values are in the same units as the original dataset — daily items sold per store-item pair. The distribution looks reasonable: the median prediction sits around 46 units per day, which is close to the 47-unit median in the training data. The distribution remains right-skewed, matching the pattern observed in the raw data at the beginning of this series.

Building the Submission File

The competition expects a CSV file with two columns: id and sales. The id column comes directly from the test set — Kaggle assigned a unique integer to each row when it released the test data. The sales column receives the expm1-converted predictions.

• submission_df = test.loc[:, [‘id’, ‘sales’]]
• submission_df[‘sales’] = np.expm1(test_preds)
• submission_df[‘id’]    = submission_df.id.astype(int)
• submission_df.to_csv(‘submission_demand.csv’, index=False)

The id column is cast to integer before saving because missing values elsewhere in the dataframe promote it to a float, while Kaggle expects integer IDs. index=False prevents pandas from writing the row index as an extra column, preserving the required submission format.

The result is a CSV with 45,000 rows: 90 days of predictions for each of 500 store-item combinations. Each row contains one id and one predicted sales value. That file is what gets uploaded to Kaggle for scoring.

What This Series Covered

This post closes the modelling section of the series. Starting from a four-column CSV (date, store, item, sales), the pipeline engineered 142 features across five families: calendar, lag, rolling mean, exponentially weighted mean, and one-hot encoded identity features. We trained a LightGBM model with a custom SMAPE objective, tuned it using time-based validation, and evaluated it on a held-out quarter of 2017.

Feature importance showed that long-horizon rolling averages and year-ago lag features carried most of the predictive signal, confirming annual seasonality as the dominant pattern. The final model, trained on all five years of data, produced 45,000 predictions for the test period.

A validation SMAPE of approximately 13.6% suggests the model captures the overall level and seasonal shape of demand for each store-item pair. Whether that is competitive on Kaggle depends on the benchmark, but the pipeline is complete, reproducible, and ready for further iteration.

Series complete — Machine Learning for Time Series