In Part 25, we built a search space.
Now we need to decide how to explore it.
We could move through every possible combination in perfect order. Or we could sample configurations randomly and accept a little uncertainty.
The first approach sounds scientific.
The second sounds like something we might do after losing patience.
Surprisingly, random search is often the smarter use of a limited training budget.
Grid Search: Check Every Combination
Suppose our search space contains:
- Four learning rates
- Three batch sizes
- Three dropout rates
- Two network depths
Grid search trains a model for every possible combination.
It starts with the first learning rate, combines it with every batch size, dropout rate, and network depth, and continues until the entire grid has been visited.
[Insert Formula Panel 1 here — How Fast a Grid Grows]
For this small experiment, we already need 72 training runs.
Add four optimizer choices, and the number grows to:
[
72\times4=288
]
If each trial takes ten minutes, the complete search requires 48 hours of continuous training.
At that point, the laptop is no longer a computer. It is a small radiator with ambitions.
Thorough—Inside the Grid
Grid search is often described as exhaustive.
That is true, but only within the values we selected.
If our learning-rate candidates are:
[
0.001,\quad 0.01,\quad 0.1
]
grid search will test all three carefully. It will never try (0.003), even if that value would have worked better.
So grid search does not explore every possible model. It exhaustively explores a small set of predefined points.
This can be useful when:
- The search space is small
- Hyperparameters have a few meaningful discrete values
- We want every candidate to be compared systematically
- Training is inexpensive
The difficulty appears when the number of hyperparameters grows.
Random Search: Spend the Budget Differently
Random search begins with the same search space but does not visit every combination.
Instead, it samples a fixed number of configurations.
For one trial, it might choose:
- Learning rate: (0.0027)
- Batch size: (64)
- Dropout rate: (0.2)
- Hidden layers: (3)
The next trial samples another combination.
We decide the number of trials in advance. If our budget allows 30 experiments, random search runs 30 experiments—even if the full space contains hundreds or thousands of possibilities.
This does not guarantee that we will find the best configuration.
But neither does grid search, unless the actual best configuration happens to be one of the points on the grid.
Why Random Search Can See More
Imagine that we are tuning two hyperparameters:
- Learning rate
- Dropout rate
We run 25 trials.
A (5\times5) grid tests five learning rates and five dropout rates. Each learning rate is repeated five times.
That seems balanced.
But suppose learning rate strongly affects performance while dropout has only a small effect for this particular dataset.
The grid has spent 25 experiments testing only five distinct learning rates.
Random search can use the same 25 trials to test many more learning-rate values. If learning rate is the setting that truly matters, random search explores the important direction in much greater detail.
[Insert Figure 1 here — Same Budget, Different Coverage]
This is the key insight.
In many machine-learning problems, not every hyperparameter matters equally. A few settings may dominate performance while others make only minor differences.
The problem is that we rarely know which ones matter before running the experiments.
Grid search divides its attention evenly. Random search spreads its trials across more distinct values, giving it a better chance of exploring an important dimension.
Random does not mean careless. The sampling is random; the experiment should still be disciplined.
A Useful Region, Not a Perfect Point
Hyperparameter optimization is often presented as a hunt for one magical configuration.
In practice, we are usually looking for a useful region: a part of the search space where several configurations perform well.
Suppose useful configurations occupy about 5% of our search space.
One random trial has only a 5% chance of landing there. But repeated independent trials gradually improve our chances.
[Insert Formula Panel 2 here — The Chance of Reaching a Useful Region]
With 40 trials, the probability of reaching that region at least once is approximately 87%.
This is not a promise that random search will succeed. The useful region may be much smaller than expected, and our search space may have been poorly designed.
It simply shows why several modest random attempts can become surprisingly effective.
A Fair Comparison Needs the Same Rules
To compare grid and random search fairly, both should use:
- The same training and validation data
- The same evaluation metric
- The same early-stopping rule
- The same search-space boundaries
- A comparable number of trials or computation time
Otherwise, we may believe that one search strategy is better when it was simply given more opportunities.
Random search also introduces variation between runs. A different random seed may produce different configurations and a different winner.
That is not a flaw we should hide.
We should record the seed, save every configuration and its validation result, and repeat important searches when the budget allows.
Reproducibility does not require removing randomness. It requires keeping track of it.
Which One Should We Use?
Grid search is a reasonable choice when the space is small and discrete. If we are comparing two optimizers and three batch sizes, testing all six combinations is perfectly sensible.
Random search becomes more attractive when:
- Several hyperparameters are being tuned together
- Values come from continuous ranges
- Some ranges span several orders of magnitude
- Training each model is expensive
- We have a fixed experiment budget
Neither method learns from previous trials.
Grid search follows a map drawn in advance. Random search throws darts at the map. If an early trial discovers a promising region, both methods continue as if nothing happened.
The next step is to let the search remember what it has already learned—and use that information to choose where to look next.
Leave a Reply