Imagine walking downhill in thick fog.
You cannot see the bottom, so you estimate the slope beneath your feet and take a step.
Three decisions matter:
- How much ground do you inspect before choosing a direction?
- How large a step do you take?
- How strongly do you avoid complicated or risky paths?
In neural-network training, these decisions resemble batch size, learning rate, and regularization.
They are often introduced separately. In practice, they pull on the same training process.
Batch Size: How Much Evidence Before an Update?
A batch is the group of training examples used to estimate the gradient before the model updates its weights.
With a batch size of 32, the model examines 32 examples, calculates their average gradient, and performs one update.
A smaller batch produces more frequent updates. Because each update is based on fewer examples, the gradient estimate is usually noisier.
A larger batch averages information from more examples. Its gradient is generally smoother, but each update requires more memory and computation.
[Insert Formula Panel 1 here — Batch Size and the Gradient Update]
For a dataset containing 10,000 examples:
- Batch size 32 gives roughly 313 updates per epoch.
- Batch size 256 gives roughly 40 updates per epoch.
- Full-batch training gives one update per epoch.
The model sees the same dataset, but its journey through the loss landscape is different.
Small batches take many somewhat uncertain steps. Large batches take fewer, more informed steps.
Neither is automatically better.
A little gradient noise can help the optimizer move away from narrow or unhelpful regions. Too much noise can make training unstable. Very large batches may use hardware efficiently, but they also reduce the number of updates made during each epoch.
Learning Rate: How Large Is the Step?
The learning rate determines how far the weights move in the direction suggested by the gradient.
If the learning rate is too small, training may progress painfully slowly. The direction may be correct, but the model moves through the loss landscape like someone searching an entire house while taking steps the length of a paperclip.
If it is too large, the optimizer can repeatedly jump across a good region instead of settling into it.
[Insert Figure 1 here — Batch Size Sets the Noise, Learning Rate Sets the Step]
A useful learning rate needs to be large enough to make progress and small enough to remain stable.
This is why values such as (0.1), (0.01), (0.001), and (0.0001) are more informative search candidates than tiny linear changes such as (0.001), (0.002), and (0.003).
Learning rates are usually searched on a logarithmic scale because meaningful values can span several orders of magnitude.
Batch Size and Learning Rate Interact
Changing the batch size changes the gradient estimate. That can change which learning rates behave well.
A learning rate that is stable with a large, smooth batch may behave differently with a small, noisy batch. A very cautious learning rate may work, but train unnecessarily slowly when the gradient estimate is already stable.
This does not give us a universal rule such as “double the batch size, always double the learning rate.”
Scaling rules can be useful in particular training regimes, but they are not laws that apply to every model and dataset.
The safer lesson is simpler:
Batch size and learning rate should be evaluated as a pair.
If we tune one while permanently freezing the other, we may miss useful combinations.
The Learning Rate Does Not Have to Stay Fixed
The best learning rate at the beginning of training may not be the best one near the end.
Early in training, the model is usually far from a good solution. Larger steps can help it move quickly.
Later, smaller steps may allow more precise improvements without repeatedly overshooting a useful region.
A learning-rate schedule changes the learning rate as training progresses.
One option is exponential decay, where the learning rate is gradually multiplied by a decay factor.
[Insert Formula Panel 2 here — Exponential Learning-Rate Decay]
For example, a decay rate of (0.96) reduces the learning rate by 4% after each decay period.
A schedule does not rescue a terrible initial learning rate. Beginning with an unstable value and promising to reduce it later is a little like driving into a wall while announcing that we plan to brake eventually.
The initial rate, decay amount, and decay timing all matter.
Regularization: A Preference for Simpler Solutions
A flexible neural network can learn useful patterns, but it can also assign large weights to accidental details in the training data.
Regularization adds a cost for certain kinds of complexity.
Two common choices are L1 and L2 regularization.
L1 Regularization
L1 adds a penalty based on the absolute values of the weights.
It can encourage some weights to become exactly or nearly zero, producing a sparser model.
L2 Regularization
L2 adds a penalty based on the squared weights.
It discourages very large weights and usually shrinks them more smoothly rather than forcing many of them to zero.
[Insert Formula Panel 3 here — L1 and L2 Regularization]
The regularization strength controls how strongly this preference affects training.
If it is too weak, the model may still overfit.
If it is too strong, the model may become unable to learn useful relationships. The simplest possible model is not automatically the best one. A horizontal line is wonderfully simple, but it makes a poor map of a mountain road.
[Insert Figure 2 here — Too Little, Balanced, Too Much Regularization]
Regularization Also Interacts with Architecture
A large network has more capacity and may benefit from stronger regularization.
A small network may already be constrained. Adding strong L1, L2, and dropout at the same time can leave it with too little freedom to learn.
This creates another interaction:
- Network size controls available capacity.
- Dropout temporarily removes part of that capacity.
- L1 and L2 discourage the use of large or numerous weights.
- Early stopping limits how long the network can adapt.
These methods can support one another, but they can also overcorrect.
Regularization is not a competition in which we win by collecting the largest number of penalties.
Be Careful About the Objective
When regularization is included, the reported model loss may contain two parts:
- Error on the data
- Regularization penalty
That total loss is appropriate for training because it is the quantity the optimizer should minimize.
During hyperparameter selection, however, we should know exactly what our objective represents. If one trial uses strong regularization and another uses none, comparing their total validation losses may mix predictive error with different penalty sizes.
Depending on the goal, we may prefer to compare a separate validation metric or an unpenalized validation loss while still using the regularized loss for training.
The important point is consistency: the tuner should optimize the quantity that reflects what we actually want from the final model.
Search the Combination, Not the Knobs
A practical search space might include:
- Batch size: 32, 64, or 128
- Initial learning rate: sampled on a logarithmic scale
- Learning-rate decay: none or a small set of schedules
- L1 or L2 strength: sampled on a logarithmic scale
- Dropout rate: a small plausible range
Not every experiment needs all of them.
If the baseline model underfits, adding stronger regularization is unlikely to be the first useful move. If training is unstable, learning rate deserves attention before we add more layers.
Hyperparameter optimization works best when the search space reflects what the learning curves are already telling us.
Batch size controls how the gradient is estimated. Learning rate controls how strongly we respond to it. Regularization controls which solutions we prefer.
Once these choices are placed into one search space, we can finally assemble the complete tuning workflow: build models, stop weak trials, record results, select a configuration, and retrain without quietly leaking information from the test set.
Leave a Reply