Imagine that our neural network is performing poorly.
We lower the learning rate and the validation loss improves. Good. We keep the new learning rate fixed and try several batch sizes. Then we choose a dropout rate. After that, perhaps we add another hidden layer.
One decision at a time.
This approach feels careful and scientific. It is certainly better than changing everything at random and hoping the model appreciates our enthusiasm.
But it has a hidden weakness:
Hyperparameters can change how other hyperparameters behave.
The best value for one setting may depend on the values of several others.
The Coffee Problem
Consider making coffee.
Grind size affects extraction. Brewing time also affects extraction. But the “best” grind size cannot always be chosen independently of brewing time.
A very fine grind might work well with a short extraction but become unpleasantly bitter if the brewing time is also long. A coarser grind may need more time to produce the same strength.
Testing grind size while keeping a poor brewing time fixed could lead us to the wrong conclusion.
Neural networks contain similar relationships.
A learning rate that works well with one batch size may become unstable with another. A wide network may benefit from stronger regularization, while the same regularization may unnecessarily restrict a smaller model.
The settings are not separate passengers. They are travelling in the same car—and occasionally arguing over the radio.
The Problem with One-at-a-Time Tuning
A common manual strategy looks like this:
- Choose a baseline model.
- Tune the learning rate.
- Keep the winning learning rate fixed.
- Tune the batch size.
- Keep that fixed.
- Continue with the next hyperparameter.
This is known as one-at-a-time tuning.
It is easy to understand and relatively inexpensive. It can also be useful for quick debugging. If a model is clearly unstable, changing one setting at a time helps us see what caused the change.
But as an optimization strategy, it has an important limitation.
Every choice is judged against the current baseline.
If that baseline is poor, a promising value may appear ineffective simply because it was tested with the wrong companions.
[Insert Figure 1 here — When Hyperparameters Interact]
Suppose a small learning rate works best with a large batch, while a larger learning rate works best with a small batch.
If we test learning rates using only the large batch, we choose the small learning rate. We then freeze it and test batch sizes.
The combination of a larger learning rate and a smaller batch is never evaluated.
Nothing went wrong during training. The search procedure simply never looked in the right place.
A Search Space Is a Set of Possibilities
Instead of treating hyperparameters as isolated decisions, we can define a search space.
A search space describes which settings may change and which values they are allowed to take.
For example, we might search over:
- Four learning rates
- Three batch sizes
- Two network depths
- Three dropout rates
A single configuration contains one value from each group.
[Insert Formula Panel 1 here — Defining the Search Space]
The search space is more than a list of numbers. It defines the question we are asking.
If the space contains only tiny learning rates, the experiment cannot discover that a larger one works better. If every candidate network is shallow, the search cannot tell us whether extra depth would help.
An optimization method cannot find a configuration that we never allowed it to test.
Too Narrow, Too Wide
A narrow search space is efficient but risky.
We may test learning rates between (0.001) and (0.003), obtain a clear winner, and feel very precise. But if the useful region begins near (0.01), we have precisely searched the wrong neighbourhood.
A very wide search space creates the opposite problem.
If we allow almost every imaginable architecture, learning rate, batch size, and regularization strength, most experiments may be wasted on obviously poor configurations.
The goal is not to create the largest possible search space.
It is to create a plausible one.
[Insert Figure 2 here — Building a Useful Search Space]
Prior knowledge helps. ReLU networks suggest certain initialization methods. Memory limits restrict possible batch sizes. Earlier experiments reveal whether the model is underfitting or overfitting. The dataset size may make extremely deep networks unnecessary.
Search is not a replacement for understanding. Understanding helps us decide where to search.
Not Every Hyperparameter Uses the Same Scale
Some hyperparameters live naturally on an ordinary linear scale.
For example, we might test:
The difference between two and three layers is comparable to the difference between four and five.
Learning rates behave differently. Their useful values often span several orders of magnitude:
Here, each value is ten times larger than the previous one.
Testing:
may examine one tiny region very carefully while ignoring most of the meaningful range.
For hyperparameters such as learning rate and regularization strength, a logarithmic scale is often more informative than evenly spaced decimal steps.
[Insert Formula Panel 2 here — Linear and Logarithmic Search]
Some Choices Are Conditional
Search spaces can also contain rules.
Momentum matters when we use optimizers that include momentum. The number of neurons in a third hidden layer matters only if the network actually has a third hidden layer.
Testing every value in every situation would create meaningless combinations.
A sensible search space can therefore say:
- If the optimizer is SGD, test several momentum values.
- If the model has three layers, choose a width for the third layer.
- If dropout is enabled, search for its rate.
This keeps the experiment focused on configurations that can actually exist.
A Better Kind of Search
One-at-a-time tuning is not useless. It remains valuable when we want to understand a single effect, debug a problem, or perform a quick local improvement.
The mistake is assuming that it can reliably discover the best combination.
When hyperparameters interact, we need to explore configurations together. That does not mean trying every imaginable possibility. It means giving the search enough room to notice useful partnerships.
A good search space is therefore a balance:
- Wide enough to contain meaningfully different options
- Narrow enough to avoid obviously wasteful experiments
- Structured enough to respect conditional choices
- Grounded in what we already know about the model and data
Once that space has been defined, the next problem is deciding how to travel through it.
We could visit every possible point in an orderly grid. It sounds thorough. It can also turn a small experiment into several hundred training runs before lunch.
Leave a Reply