Grid search follows a fixed map.
Random search throws darts at that map.
Both can find good hyperparameters, but neither pays much attention to what happened during the previous trials.
If a learning rate of (0.001) performs terribly and (0.003) performs surprisingly well, the next random trial does not care. It may jump to a completely unrelated part of the search space.
Bayesian optimization takes a different approach:
Use previous experiments to decide which experiment should come next.
A Search That Remembers
Imagine searching for buried treasure.
Grid search digs holes at evenly spaced locations. Random search chooses locations by chance. Bayesian optimization begins with a few holes, studies what they reveal, and gradually builds a rough map of where the treasure may be.
The map is not perfect. But after every new hole, it becomes a little more informed.
In hyperparameter optimization, each “hole” is expensive. We must build a model, train it, and measure its validation performance.
Bayesian optimization tries to learn as much as possible from each of those expensive trials.
[Insert Figure 1 here — A Search That Learns from Every Trial]
The Expensive Model and the Cheap Guide
Suppose a hyperparameter configuration contains:
- Learning rate
- Batch size
- Number of hidden layers
- Dropout rate
We can give that configuration to our real training process and obtain a validation loss.
But doing this may take minutes, hours, or even days.
Bayesian optimization therefore builds a second, much cheaper model called a surrogate model.
The surrogate does not replace the neural network. It tries to predict how the neural network is likely to perform under different hyperparameter configurations.
Its training data consists of our previous trials:
- Configuration A produced a validation loss of 0.46
- Configuration B produced 0.39
- Configuration C produced 0.52
- Configuration D produced 0.41
From these observations, the surrogate estimates two things for unexplored configurations:
- The performance we might expect
- How uncertain that prediction is
[Insert Formula Panel 1 here — What the Surrogate Learns]
Common surrogate choices include Gaussian processes and tree-based probabilistic models. The exact method can change, but the job remains the same: create a cheap approximation of an expensive experiment.
Good Predictions Are Not Enough
Suppose the surrogate predicts that one configuration will achieve a validation loss of 0.38.
Another configuration has an expected loss of 0.40, but the surrogate is much less certain about it. It might perform poorly—or it might turn out to be the best configuration we have seen.
Which one should we try?
This is the central tension in Bayesian optimization.
Exploitation
Test configurations near regions that already look promising.
This is the safe option. We use what we have learned and search for small improvements around the current best result.
Exploration
Test configurations in uncertain regions.
This is riskier. The result may be disappointing, but it may reveal a better part of the search space.
Too much exploitation can trap the search around a decent but unremarkable configuration. Too much exploration can waste the budget jumping between unknown regions.
It is a little like choosing a restaurant. We can return to the place that served a good meal last week, or try the suspiciously empty restaurant around the corner.
One choice is safer. The other may produce either a wonderful discovery or a very long evening.
[Insert Figure 2 here — Exploration and Exploitation]
The Acquisition Function
Bayesian optimization manages this balance using an acquisition function.
The acquisition function examines the surrogate’s prediction and uncertainty, then assigns a value to each possible next configuration.
Unlike training a neural network, evaluating this function is usually cheap. We can examine many candidate configurations without actually training the real model.
The configuration with the most attractive acquisition value becomes the next experiment.
[Insert Formula Panel 2 here — Choosing the Next Trial]
One popular acquisition strategy is Expected Improvement.
Instead of asking only:
Which configuration is predicted to perform best?
it asks:
Considering both prediction and uncertainty, where do we expect the greatest improvement over our current best result?
After the selected configuration is trained, its real validation result is added to the history. The surrogate is updated, uncertainty changes, and the process repeats.
Bayesian optimization is therefore a loop:
- Observe previous trials
- Update the surrogate
- Choose a promising next configuration
- Train the real neural network
- Record the validation result
- Repeat
The search becomes more informed as the experiment continues.
Neural Network Results Are Noisy
There is one complication.
Training the same neural network configuration twice may not produce exactly the same validation score.
Random weight initialization, shuffled mini-batches, dropout, and other stochastic operations introduce variation. A configuration may appear excellent partly because it received a lucky run.
If Bayesian optimization treats every score as perfect truth, one unusually good result can pull the search toward the wrong region.
We can reduce this problem by:
- Recording random seeds
- Repeating promising configurations
- Comparing results across multiple seeds when possible
- Allowing the surrogate to represent observation noise
- Avoiding conclusions based on tiny score differences
A validation loss of 0.401 is not automatically meaningfully better than 0.403. Sometimes it is just Tuesday.
When Bayesian Optimization Helps
Bayesian optimization is especially useful when:
- Training each model is expensive
- The search budget is limited
- Hyperparameters include continuous values
- We want later trials to benefit from earlier results
- Evaluating the surrogate is much cheaper than training the real model
It may be less attractive when the search space is tiny. If only six configurations exist, simply training all six may be easier.
It can also be harder to run many trials simultaneously. Later Bayesian trials benefit from the results of earlier ones, while independent random trials can be launched in parallel without waiting for each other.
Modern implementations offer parallel versions, but the basic trade-off remains: Bayesian optimization gains efficiency by learning sequentially.
An Informed Guess, Not a Fortune Teller
Bayesian optimization does not know the true validation landscape.
Its surrogate is built from limited observations. Early in the search, its map may be rough and occasionally wrong.
Poorly chosen ranges still create a poor search. No acquisition function can discover a learning rate that lies outside the allowed space. No surrogate can repair a validation metric that does not reflect what we actually care about.
What Bayesian optimization offers is not certainty.
It offers organized curiosity.
Each trial does more than test one configuration. It improves the guide used to choose the next one.
But even the cleverest search method depends on where the neural network begins. Before the first gradient update, the initial weights have already influenced whether signals will flow, fade, or grow out of control.
Leave a Reply