Deep Learning from Scratch, Part 31: From Search Space to Final Model

We have chosen a search method.

We have considered initialization, architecture, batch size, learning rate, and regularization.

Now comes the part that looks deceptively simple:

Run the tuner and keep the winner.

Unfortunately, a good hyperparameter-tuning workflow contains several boundaries. If we cross them carelessly, we can produce a model with an excellent score and very little evidence that it will work outside our notebook.

The best trial is not the end of the process. It is the middle.

Give Every Dataset One Job

Before tuning begins, the data should be separated into three parts:

• Training set: learns the model parameters
• Validation set: compares hyperparameter configurations
• Test set: evaluates the selected procedure at the end

During the search, every trial trains on the training set and reports its performance on the validation set.

The test set remains untouched.

Not “mostly untouched.”

Not “checked once or twice to make sure things look reasonable.”

Untouched.

[Insert Figure 1 here — Three Datasets, Three Jobs]

The validation set can be used repeatedly because model selection is its job. But this also means that, over many trials, we may gradually adapt our decisions to that validation set.

The test set provides a final check using data that did not participate in those decisions.

Turn the Search Space into a Model Builder

A tuner needs a function that can build different models.

In Keras Tuner, this is commonly written as build_model(hp).

Instead of using fixed values, the function asks the tuner for choices:

• How many hidden layers?
• How many units in each active layer?
• Which hidden activation?
• How much dropout?
• Which learning rate?
• Which regularization strength?

Each call creates one complete model configuration.

Conditional choices matter here. If a trial selects two hidden layers, there should be no active units_3 setting pretending that a third layer exists.

The model builder should also preserve valid relationships:

• ReLU hidden layers can use He initialization.
• Binary classification requires an appropriate output and loss.
• Optimizer-specific settings should appear only when that optimizer is selected.

The tuner explores possibilities. The builder prevents impossible combinations.

Choose the Objective Before the Search

The tuner needs one quantity that determines which trial is better.

For a regression problem, we might minimize validation mean absolute error.

For a balanced classifier, validation loss may be reasonable. For an imbalanced classifier, accuracy may hide the mistakes we care about, so AUC, recall, precision, or another business-relevant metric may be more useful.

Direction matters:

• Validation loss → minimize
• Mean absolute error → minimize
• AUC → maximize
• Recall → maximize

A tuner cannot understand our real goal by reading the room. It will optimize exactly what we give it, even if that quantity is a poor description of success.

Set a Budget

A search space may contain thousands of possible configurations.

max_trials tells the tuner how many it is allowed to evaluate.

This is not merely a software setting. It is our experimental budget.

The budget may be limited by:

• Available time
• Hardware
• Cost
• Number of hyperparameters
• Variability between training runs

With a small budget, the search space should remain focused. Allowing every imaginable layer size, optimizer, learning rate, and regularization method gives the tuner freedom—but not enough time to use it intelligently.

A ten-trial search through a universe-sized space is less exploration and more sightseeing.

What Happens Inside One Trial?

For each trial, the tuner:

1. Selects a hyperparameter configuration.
2. Builds a new model.
3. Trains it on the training data.
4. Monitors validation performance.
5. Records the objective value.
6. Moves to the next configuration.

[Insert Figure 2 here — Anatomy of a Tuning Trial]

Every trial must begin with a newly initialized model. Reusing the weights from a previous configuration would make the comparison unfair because some models would inherit learning that others did not receive.

Training conditions should otherwise remain consistent: the same data split, maximum epoch budget, callbacks, preprocessing, and evaluation objective.

Stop Trials That Have Stopped Improving

Giving every trial a large fixed number of epochs can waste enormous amounts of computation.

Early stopping ends training when the monitored validation quantity has not improved for a selected number of epochs.

Its main settings include:

• Monitor: the validation quantity to follow
• Mode: whether lower or higher is better
• Patience: how long to wait without improvement
• Minimum change: how large an improvement must be to count
• Restore best weights: whether to return to the best observed epoch

Patience should not be too short. Validation curves can wobble, especially with small batches or noisy data. Stopping after one disappointing epoch may eliminate a model that was still learning.

Restoring the best weights is also important. Otherwise, the trial may finish with the weights from its final epoch rather than the epoch that produced its best validation result.

Early stopping does not make every trial equally cheap, but it prevents clearly unproductive ones from occupying the entire budget.

One Configuration Can Produce Several Scores

Neural-network training is stochastic.

The same hyperparameters can produce slightly different results because of random initialization, batch order, dropout masks, and other sources of randomness.

If each configuration is trained only once, a lucky run may win the search.

One solution is to run the same configuration several times with different seeds and compare its average performance.

[Insert Formula Panel 1 here — Averaging Multiple Executions]

Keras Tuner supports this idea through executions_per_trial.

Multiple executions improve the reliability of the comparison, but they also multiply the cost. With 30 trials and three executions per trial, we may train as many as 90 models.

For an early exploratory search, one execution may be practical. For the most promising configurations, repeated runs provide stronger evidence.

Retrieve the Configuration, Not Just the Trophy

After the search, the tuner can return the best stored model.

That is useful for inspection, but a clean final workflow should also retrieve the best hyperparameters.

Those hyperparameters describe the selected training recipe:

• Architecture
• Initialization
• Activation functions
• Dropout
• Regularization
• Optimizer settings
• Learning rate

We can then rebuild a fresh model from that configuration.

This separates two ideas:

• The best trial helped us choose the recipe.
• The final training run produces the model we intend to evaluate or deploy.

The distinction may feel ceremonial until a trial checkpoint is missing, partially saved, or linked to an experimental preprocessing pipeline. Then it becomes less ceremonial.

Retrain Without Leaking the Test Set

Once the hyperparameters have been selected, we can use more of the available development data for final training.

One option is:

1. Use the best epoch information from tuning to choose a fixed training duration.
2. Combine the training and validation sets.
3. Build a fresh model with the selected hyperparameters.
4. Train it once on the combined development data.
5. Evaluate it once on the untouched test set.

[Insert Formula Panel 2 here — Final Retraining and Test Evaluation]

If we still need early stopping during final training, we must retain a small internal validation split. Early stopping cannot monitor data that is simultaneously being used for weight updates.

After the final test result has been reported, we may retrain a deployment model using all available labelled data, including the old test set.

But that exact deployment model no longer has an untouched test set. The previously reported test score belongs to the selection-and-training procedure evaluated before this final all-data retraining.

Save More Than the Neural Network

A model rarely works alone.

The final package may also need:

• Feature names and order
• Missing-value rules
• Encoders
• Scalers
• Classification threshold
• Selected hyperparameters
• Training seed
• Software versions

If the model was trained on scaled features but receives raw values in production, its beautifully tuned learning rate will not save it.

Preprocessing is part of the model system.

A Search Is an Experiment

A reliable tuning workflow can be reduced to a simple sequence:

Define the data boundaries. Build a valid search space. Choose an objective that reflects the real task. Give every trial comparable conditions. Record the results. Confirm promising configurations. Retrain cleanly. Touch the test set once.

[Insert Figure 3 here — The Complete Tuning Workflow]

Hyperparameter optimization is not about asking software to make every decision.

It is about automating repeated experiments without losing control of what those experiments mean.

We now have a configuration and a clean evaluation boundary. The next question moves inside the optimizer itself: should each update use the entire dataset, a single example, or something in between?