A neural network with one small hidden layer may fail to capture the patterns in our data. The obvious response is to make it bigger. Add more neurons. Add another layer. Perhaps add several more, just to make sure the network understands that we are serious. Sometimes this works. Sometimes we simply create a slower... Continue Reading →

Before training begins, a neural network knows nothing about the problem. But it does not begin as an empty object. Its connections already contain numbers: the initial weights. These values are temporary. Backpropagation will update them thousands of times. Still, their starting scale can determine whether useful signals travel through the network—or disappear before learning... Continue Reading →

Grid search follows a fixed map. Random search throws darts at that map. Both can find good hyperparameters, but neither pays much attention to what happened during the previous trials. If a learning rate of (0.001) performs terribly and (0.003) performs surprisingly well, the next random trial does not care. It may jump to a... Continue Reading →

In Part 25, we built a search space. Now we need to decide how to explore it. We could move through every possible combination in perfect order. Or we could sample configurations randomly and accept a little uncertainty. The first approach sounds scientific. The second sounds like something we might do after losing patience. Surprisingly,... Continue Reading →

Imagine that our neural network is performing poorly. We lower the learning rate and the validation loss improves. Good. We keep the new learning rate fixed and try several batch sizes. Then we choose a dropout rate. After that, perhaps we add another hidden layer. One decision at a time. This approach feels careful and... Continue Reading →

Training a neural network is often described as a learning process. We give the model data. It makes predictions, measures its mistakes, and adjusts itself. That description is correct, but incomplete. A neural network can learn millions of weights on its own, yet it cannot decide how many layers it should have, which optimizer to... Continue Reading →

Hyperparameters are choices that control how a neural network is built and trained, but are not learned directly through backpropagation. The network learns its weights and biases from gradients. It does not ordinarily learn how many layers it should contain, how wide those layers should be, what batch size to use, or how much dropout... Continue Reading →

A trained model is useful only if we can recover the correct version and reproduce the complete path from a new raw observation to its prediction. Saving the final weights is not always enough. The final epoch may not be the best epoch, the scaler may be missing, the feature columns may arrive in a... Continue Reading →

Before a neural network can learn, it needs somewhere to begin. Weight initialization determines the scale and diversity of the network’s starting connections. Batch Normalization then helps control the scale of intermediate activations as those connections change during training. These mechanisms are related to optimization, but they act at different times: Initialization defines the network... Continue Reading →

In Part 19, we trained a Keras neural network by passing NumPy arrays directly to model.fit(). That approach is perfectly valid for a small dataset that fits comfortably in memory. But as data preparation becomes more complex, the model needs more than a collection of arrays. It needs an organized route through which observations are... Continue Reading →