A neural network with one small hidden layer may fail to capture the patterns in our data.
The obvious response is to make it bigger.
Add more neurons. Add another layer. Perhaps add several more, just to make sure the network understands that we are serious.
Sometimes this works.
Sometimes we simply create a slower model that memorizes the training set with greater confidence.
Network architecture is a balance between having enough capacity to learn and enough restraint to generalize.
Capacity Is Not Intelligence
The number of layers and neurons determines part of a network’s capacity: its ability to represent complex relationships.
A model with too little capacity may underfit. Training and validation performance both remain poor because the network cannot capture the underlying structure.
A model with too much capacity can overfit. It performs extremely well on the training data but struggles with examples it has not seen before.
A larger network is not automatically more intelligent.
Giving a confused model more neurons can be like inviting more people to a meeting with no agenda. The room becomes crowded, but the problem remains.
[Insert Figure 1 here — Too Small, Balanced, Too Large]
Width: More Neurons in a Layer
The number of neurons in a hidden layer is called its width.
A wider layer can learn more features at the same stage of processing. One neuron may respond to one pattern, while others respond to different combinations of inputs.
But every added neuron creates new weights connecting it to the previous layer.
A layer receiving 100 inputs and producing 64 outputs does not contain only 64 trainable values. Each neuron needs a weight for every input, plus a bias.
[Insert Formula Panel 1 here — How Width Increases the Parameter Count]
Widening consecutive layers can therefore increase the parameter count quickly.
More parameters may improve the model’s flexibility, but they also require:
- More memory
- More computation
- More training data
- Stronger protection against overfitting
Width should be treated as a hyperparameter, not a volume knob that we keep turning until the neighbours complain.
Depth: More Stages of Representation
Depth refers to the number of layers through which information passes.
A deeper network can build representations in stages. Earlier layers may learn simple relationships, and later layers can combine them into more complex ones.
For image data, simple edges may contribute to shapes, and shapes may contribute to objects. In tabular data, the hierarchy is usually less visible, but deeper layers can still represent interactions between features.
Depth, however, introduces its own costs.
A deeper network takes longer to train and may be harder to optimize. Signals and gradients must travel through more transformations. Extra depth can also be unnecessary when the dataset is small or the relationship is relatively simple.
Adding another layer is useful when the model needs another level of representation—not merely because a dropdown menu allows it.
Activation Functions Change What Layers Can Learn
Without activation functions, stacking dense layers would not give us the nonlinear flexibility we expect from a neural network.
No matter how many purely linear layers we stack, their combined transformation is still linear.
Hidden activations allow the network to model more complex relationships.
Common candidates include:
- ReLU: simple, efficient, and widely used in hidden layers
- Leaky ReLU: keeps a small response for negative inputs
- Tanh: produces values between (-1) and (1)
- Sigmoid: produces values between (0) and (1), but can saturate in deep hidden layers
The hidden-layer activation can be part of our search space, but it should not be considered independently from initialization. As we saw in Part 28, ReLU is commonly paired with He initialization, while tanh often works with Xavier initialization.
The output activation is different.
For binary classification, a sigmoid output is commonly paired with binary cross-entropy. Multiclass classification often uses softmax, while regression commonly uses a linear output.
This is not a setting we should randomize without thought. The output activation must match what the model is being asked to predict and how the loss is defined.
Dropout: Making Training Slightly More Difficult
A large network may become too dependent on particular neurons.
Dropout interrupts that dependence by randomly setting a fraction of activations to zero during training.
With a dropout rate of (0.2), each affected activation has a 20% chance of being dropped during a training step.
The model cannot rely on the same complete pathway every time. It must learn representations that remain useful even when some contributions temporarily disappear.
[Insert Figure 2 here — Dropout During Training and Inference]
[Insert Formula Panel 2 here — Inverted Dropout]
Dropout is active during training but not during ordinary inference. In implementations such as Keras, the surviving activations are scaled during training so that their expected total scale remains consistent.
A small amount of dropout may reduce overfitting.
Too much dropout can make learning unnecessarily difficult. If the network is already small, repeatedly removing much of its capacity may cause underfitting instead.
Dropout is not a repair button for every struggling model.
These Choices Interact
Layers, units, activations, and dropout do not work independently.
A wide network may tolerate or benefit from more dropout. A small network may need little or none. A deeper ReLU network may depend more strongly on appropriate initialization than a shallow model.
This makes architecture tuning a conditional search problem.
If a trial chooses one hidden layer, settings such as units_2 and dropout_2 should not affect the model. They belong only to configurations in which a second layer exists.
[Insert Figure 3 here — A Conditional Architecture Search Space]
The search procedure should first choose the number of layers and then choose the units, activation, and dropout rate for each layer that actually exists.
Otherwise, the tuner may report impressive-looking hyperparameter dictionaries containing settings the final model never used.
Comparing Architectures Fairly
Larger models may learn at different speeds from smaller ones.
If every architecture receives only a few epochs, we may reject a promising larger network before it has had time to learn. If every model trains for a very long time, we waste computation and give overfitting more room to develop.
A consistent early-stopping rule helps compare architectures under a shared maximum training budget.
We should also record more than validation performance:
- Number of trainable parameters
- Training time
- Best epoch
- Gap between training and validation performance
- Variation across random seeds
A tiny improvement may not justify a model that is ten times larger and slower.
The goal is not to discover the biggest network our hardware can tolerate. It is to find the smallest architecture that captures the useful structure in the data without learning every accidental detail.
Once that structure is chosen, we still need to decide how the training process should move through it: how many examples to use at once, how large each update should be, and how strongly the weights should be restrained.
Leave a Reply