Deep Learning from Scratch, Part 21: Weight Initialization and Batch Normalization

Before a neural network can learn, it needs somewhere to begin.

Weight initialization determines the scale and diversity of the network’s starting connections. Batch Normalization then helps control the scale of intermediate activations as those connections change during training.

These mechanisms are related to optimization, but they act at different times:

• Initialization defines the network at step zero.
• Batch Normalization operates throughout training and later uses accumulated statistics during inference.

Neither method replaces data preprocessing, regularization, validation, or careful model design. Their purpose is more specific: they help signals and gradients travel through the network in numerically useful ranges.

In this article, we will derive Glorot and He initialization, examine how poor initialization produces vanishing or exploding signals, and add Batch Normalization to the structured-data model from Part 20.

Why Initial Weights Cannot Be Arbitrary

Consider a hidden layer receiving activations from the previous layer. Each new pre-activation is a weighted sum of those incoming values.

If the weights begin extremely small, the outputs may shrink as they pass through successive layers. Eventually, later layers receive values so close to zero that useful distinctions become difficult to preserve.

If the weights begin extremely large, the outputs may grow from one layer to the next. Activations can explode, sigmoid-like units can saturate, and gradients may become unstable.

A useful initialization therefore tries to keep the scale of signals reasonably stable across depth.

Under simplifying assumptions—independent inputs and weights, approximately zero means, and similar input variances—the variance of a layer’s pre-activation depends on three quantities:

• The number of incoming connections
• The variance of the weights
• The variance of the incoming activations

This is why a fixed random standard deviation is not suitable for every layer. A layer with eight inputs and a layer with 512 inputs should not necessarily draw weights from the same distribution.

The initializer must account for the layer’s fan-in, fan-out, or both.

Why Zero Initialization Fails for Kernels

Setting every weight to zero may appear safe because it avoids exploding values. But it creates a symmetry problem.

If two neurons in the same layer begin with identical incoming weights and identical biases, they calculate identical outputs. During backpropagation, they also receive identical gradients and remain identical after every update.

A layer containing 32 identical neurons does not learn 32 different features. It behaves more like one neuron copied 32 times.

With ReLU, an all-zero kernel is even more problematic because all pre-activations begin at zero, where the usual implemented derivative does not provide the diverse learning signals needed to separate the neurons.

Weight initialization therefore needs randomness—not because randomness itself is intelligent, but because it breaks symmetry.

Bias vectors, by contrast, can usually begin at zero. Random kernels already make the neurons different.

layer = keras.layers.Dense(
    units=32,
    kernel_initializer="he_normal",
    bias_initializer="zeros",
)

Random kernels break symmetry. Zero biases do not restore it.

Variance Is More Important Than the Distribution’s Name

We can draw initial values from a normal distribution or a uniform distribution. In many cases, the crucial property is not whether the distribution is bell-shaped or flat but whether its scale is appropriate for the layer.

Two initializers can use different distributions while targeting approximately the same variance.

Keras therefore provides both normal and uniform versions of several initialization strategies:

• GlorotNormal
• GlorotUniform
• HeNormal
• HeUniform

The choice between Glorot and He is connected more strongly to the activation function than to the words normal and uniform.

Glorot or Xavier Initialization

Glorot initialization, also known as Xavier initialization, balances the scale using both the number of inputs and the number of outputs.

The original motivation was to help signals and gradients retain useful scale as they travel forward and backward through deep networks. The method is associated especially with approximately symmetric activation functions such as tanh, although its use is broader in practice. Glorot and Bengio, 2010

Keras offers both forms:

glorot_normal = keras.initializers.GlorotNormal()
glorot_uniform = keras.initializers.GlorotUniform()

A Keras Dense layer uses glorot_uniform by default. Keras Dense documentation

This means the ReLU layers we created in Part 19 and Part 20 were already initialized with a reasonable variance-aware strategy, even though we did not specify one explicitly.

layers.Dense(
    32,
    activation="relu",
)

is effectively using:

layers.Dense(
    32,
    activation="relu",
    kernel_initializer="glorot_uniform",
    bias_initializer="zeros",
)

Glorot initialization is not wrong for ReLU networks. Many such models train successfully. But ReLU discards negative pre-activations, so an initializer designed specifically around rectifying nonlinearities can preserve the signal scale more directly.

He or Kaiming Initialization

He initialization, also called Kaiming initialization, accounts for the fact that ReLU sets negative values to zero.

For a symmetric pre-activation distribution centered around zero, approximately half the values are negative. ReLU removes those values, reducing the activation’s second moment.

He initialization compensates by using a larger starting variance based on fan-in.

The method was derived for rectifier networks, including ReLU and its variants. He et al., 2015

Keras provides:

he_normal = keras.initializers.HeNormal()
he_uniform = keras.initializers.HeUniform()

For a hidden Dense layer followed by ReLU, we can write:

layers.Dense(
    32,
    activation="relu",
    kernel_initializer="he_normal",
)

The practical pairing is usually:

• Glorot initialization for tanh, sigmoid-like, or roughly symmetric activations
• He initialization for ReLU and related rectifying activations
• Glorot initialization for our final sigmoid classification layer

These are principled defaults, not inviolable rules. Architecture, residual connections, normalization layers, and specialized activations may lead to other initialization strategies.

[Insert Figure 1 here]

Watching Activation Scale Across Depth

Instead of treating initialization as an abstract formula, we can observe what happens to activations before training begins.

The following experiment builds a deeper ReLU network and records the standard deviation of each hidden layer’s output.

import numpy as np
import pandas as pd
import keras
from keras import layers
def make_initializer(initializer_name):
    if initializer_name == "large_random":
        return keras.initializers.RandomNormal(
            mean=0.0,
            stddev=1.0,
        )
    return initializer_name
def build_activation_probe(
    number_of_features,
    initializer_name,
):
    inputs = keras.Input(
        shape=(number_of_features,)
    )
    x = inputs
    hidden_outputs = []
    for layer_index in range(6):
        x = layers.Dense(
            64,
            use_bias=True,
            kernel_initializer=make_initializer(
                initializer_name
            ),
            bias_initializer="zeros",
            name=f"dense_{layer_index + 1}",
        )(x)
        x = layers.ReLU(
            name=f"relu_{layer_index + 1}"
        )(x)
        hidden_outputs.append(x)
    return keras.Model(
        inputs=inputs,
        outputs=hidden_outputs,
    )

We will compare four starting strategies:

initializer_names = [
    "zeros",
    "large_random",
    "glorot_uniform",
    "he_normal",
]
activation_records = []
probe_input = np.asarray(
    X_train_scaled,
    dtype="float32",
)
for initializer_name in initializer_names:
    keras.utils.set_random_seed(42)
    probe = build_activation_probe(
        number_of_features=probe_input.shape[1],
        initializer_name=initializer_name,
    )
    layer_outputs = probe(
        probe_input,
        training=False,
    )
    for layer_number, output in enumerate(
        layer_outputs,
        start=1,
    ):
        activation_records.append(
            {
                "initializer": initializer_name,
                "layer": layer_number,
                "mean": float(
                    keras.ops.mean(output)
                ),
                "standard_deviation": float(
                    keras.ops.std(output)
                ),
                "zero_fraction": float(
                    keras.ops.mean(
                        keras.ops.cast(
                            output == 0,
                            "float32",
                        )
                    )
                ),
            }
        )
activation_statistics = pd.DataFrame(
    activation_records
)
print(activation_statistics)

We can visualize the activation scale:

import matplotlib.pyplot as plt
for initializer_name in initializer_names:
    selected_rows = activation_statistics[
        activation_statistics["initializer"]
        == initializer_name
    ]
    plt.plot(
        selected_rows["layer"],
        selected_rows["standard_deviation"],
        marker="o",
        label=initializer_name,
    )
plt.xlabel("Hidden layer")
plt.ylabel("Activation standard deviation")
plt.title(
    "Activation Scale Before Training"
)
plt.legend()
plt.show()

[Insert Figure 2 here]

The purpose of this experiment is not to prove that one initializer always produces the highest test accuracy.

It asks a narrower question:

Does the initial signal remain in a numerically useful range as it travels through the network?

The all-zero model collapses immediately. A large fixed random scale may produce rapidly growing activations. Glorot and He initialization adapt their scale to layer dimensions and are therefore more likely to preserve useful signals.

Exact curves will depend on the dataset, layer widths, activation function, random seed, and number of layers. We should generate the figure from the actual output rather than inventing a universal curve.

Initialization Also Affects Backpropagation

Forward activations are only half the story.

During backpropagation, gradients are repeatedly multiplied by weight matrices and activation derivatives. Poor scale choices can therefore produce:

• Vanishing gradients, where early layers receive almost no update
• Exploding gradients, where updates become numerically unstable
• Saturated activations, where derivatives are extremely small
• Dead ReLU units that remain inactive for much of the data

Initialization does not guarantee that gradients remain perfectly stable throughout training. The parameters change after every update. But it gives the network a more workable starting geometry.

A suitable initializer makes optimization possible. It does not determine the final solution on its own.

What Batch Normalization Does

Even with a carefully chosen initializer, activation distributions change as the weights are updated.

Batch Normalization inserts a learnable normalization operation inside the network. During training, it calculates the mean and variance of the current mini-batch, normalizes the values, and then applies a learned scale and offset.

The final scale and offset are important.

If Batch Normalization permanently forced every hidden unit to have exactly zero mean and unit variance, it could restrict what the network represents. The learned parameters gamma and beta allow the model to recover or modify an appropriate scale and location.

In Keras, the layer is:

layers.BatchNormalization()

Its default learned scale begins at one, and its learned offset begins at zero. Keras BatchNormalization documentation

[Insert Figure 3 here]

The Explanation Has Evolved

The original Batch Normalization paper described the method as reducing internal covariate shift: changes in the distributions received by layers as earlier parameters are updated. It also demonstrated faster training and support for larger learning rates in its experiments. Ioffe and Szegedy, 2015

That historical explanation should not be presented as a fully settled account of why Batch Normalization works.

Later research found that reducing internal distribution shift was not sufficient to explain its effectiveness and proposed that Batch Normalization instead improves optimization by making gradients more stable and the optimization landscape smoother. Santurkar et al., 2018

The safest conclusion is narrower:

Batch Normalization changes the optimization geometry and often makes deep networks easier to train, but its benefit cannot be reduced to one universally accepted slogan.

Training and Inference Use Different Statistics

Batch Normalization behaves differently during training and prediction.

During training, it uses statistics calculated from the current mini-batch.

During validation, testing, and prediction, it uses moving averages accumulated during training.

Keras maintains:

• A trainable scale called gamma
• A trainable offset called beta
• A non-trainable moving mean
• A non-trainable moving variance

The moving statistics are updated during training but are not optimized through ordinary gradient descent.

This distinction has a deep consequence: during training, one observation’s normalized representation depends partly on the other observations in its mini-batch. During inference, that same observation is normalized using stored population estimates instead.

[Insert Figure 4 here]

Keras automatically selects the correct behavior:

model(
    batch_features,
    training=True,
)

uses current-batch statistics, while:

model(
    batch_features,
    training=False,
)

this one uses moving statistics.

model.fit() uses training behavior for training batches and inference behavior for validation. model.evaluate() and model.predict() use inference behavior.

Momentum in Batch Normalization

Batch Normalization uses a momentum parameter when updating its moving mean and variance.

This momentum is not the momentum used by an optimizer.

Keras’s current default Batch Normalization momentum is 0.99. A high value gives substantial weight to the existing moving statistic and a smaller weight to the current batch statistic.

The moving statistics therefore change gradually rather than becoming equal to the most recent batch.

This can stabilize inference estimates, but it also means they need enough representative training batches to become useful.

For a very small dataset or very few training steps, the moving statistics may be poorly estimated. Changing momentum can affect that behavior, but it should be treated as an experimental choice rather than an automatic fix.

Small Batches Make Batch Statistics Noisier

A mean and variance calculated from four observations are generally less stable than statistics calculated from 64 observations.

This creates a tension:

• Small batches can provide useful gradient noise.
• Batch Normalization benefits from sufficiently representative batch statistics.

When batches become extremely small, training and inference behavior may differ substantially because the batch statistics are noisy while inference relies on moving estimates.

Possible responses include:

• Testing a larger batch size
• Adjusting Batch Normalization momentum carefully
• Omitting Batch Normalization if it provides no validation benefit
• Considering methods that do not depend on batch statistics

The correct response is not automatically to increase model complexity.

For a small structured dataset, Batch Normalization must earn its place through validation evidence.

Batch Normalization Does Not Replace Input Scaling

In Part 19, we standardized the input columns using statistics fitted only on the training set.

Batch Normalization does not make that preprocessing principle obsolete.

Input scaling and hidden-layer Batch Normalization act on different objects:

• Input scaling standardizes named input features using training-set statistics.
• Batch Normalization normalizes intermediate activations generated by the evolving network.
• Input preprocessing must later be reproduced for every new observation.
• Batch Normalization’s moving statistics are stored inside the model.

Leaving one feature measured in thousands and another in fractions still makes the model’s first layer harder to reason about. Hidden-layer normalization does not justify careless input preparation.

Where Should Batch Normalization Be Placed?

A common pattern for a Dense ReLU block is:

Dense → Batch Normalization → ReLU

In Keras:

x = layers.Dense(
    32,
    use_bias=False,
    kernel_initializer="he_normal",
)(x)
x = layers.BatchNormalization()(x)
x = layers.Activation("relu")(x)

This places normalization on the linear pre-activations before ReLU.

Some architectures place Batch Normalization after the activation. Both patterns exist, and the best choice can depend on the architecture and experiment. We will use the pre-activation pattern because it makes the mathematical sequence explicit.

Why Set use_bias=False?

A Dense bias adds a constant offset before Batch Normalization. Batch Normalization then subtracts a batch mean and introduces its own learned offset, beta.

The Dense bias is therefore usually redundant when a Batch Normalization layer follows immediately. Keras’s Dense documentation also recommends use_bias=False in this configuration. Keras Dense documentation

Rebuilding Our Structured-Data Model

We can now combine:

• He initialization for the ReLU hidden blocks
• Batch Normalization before each ReLU
• L2 regularization on Dense kernels
• Dropout after the first hidden activation
• Glorot initialization for the sigmoid output

import keras
from keras import layers, regularizers
keras.utils.set_random_seed(42)
model = keras.Sequential(
    [
        keras.Input(
            shape=(X_train_scaled.shape[1],),
            name="features",
        ),
        layers.Dense(
            32,
            use_bias=False,
            kernel_initializer="he_normal",
            kernel_regularizer=regularizers.L2(
                1e-4
            ),
            name="hidden_1_linear",
        ),
        layers.BatchNormalization(
            name="hidden_1_normalization"
        ),
        layers.Activation(
            "relu",
            name="hidden_1_relu",
        ),
        layers.Dropout(
            0.20,
            name="dropout",
        ),
        layers.Dense(
            16,
            use_bias=False,
            kernel_initializer="he_normal",
            kernel_regularizer=regularizers.L2(
                1e-4
            ),
            name="hidden_2_linear",
        ),
        layers.BatchNormalization(
            name="hidden_2_normalization"
        ),
        layers.Activation(
            "relu",
            name="hidden_2_relu",
        ),
        layers.Dense(
            1,
            activation="sigmoid",
            kernel_initializer="glorot_uniform",
            name="probability",
        ),
    ],
    name="normalized_structured_classifier",
)

The hidden layers use He initialization because they feed ReLU activations. The final layer uses Glorot initialization because its output is passed through a sigmoid.

Counting Batch Normalization Parameters

For every normalized hidden unit, Batch Normalization maintains four values:

• gamma
• beta
• Moving mean
• Moving variance

The first two are trainable. The moving statistics are non-trainable state.

For our complete model:

• First Dense kernel: 256 parameters
• First Batch Normalization: 128 parameters
• Second Dense kernel: 512 parameters
• Second Batch Normalization: 64 parameters
• Output kernel and bias: 17 parameters
• Total: 977 parameters

Of these:

• 881 are trainable
• 96 are non-trainable moving statistics

model.summary()

The phrase non-trainable does not mean the moving statistics never change. They are updated during training through moving-average rules rather than ordinary optimizer gradients.

[Insert Figure 5 here]

Compile and Train with the tf.data Pipelines

The model can use the datasets created in Part 20:

model.compile(
    optimizer=keras.optimizers.Adam(
        learning_rate=1e-3,
    ),
    loss=keras.losses.BinaryCrossentropy(),
    metrics=[
        keras.metrics.BinaryAccuracy(
            name="accuracy"
        ),
        keras.metrics.AUC(name="auc"),
    ],
)
early_stopping = keras.callbacks.EarlyStopping(
    monitor="val_loss",
    mode="min",
    patience=25,
    min_delta=1e-4,
    restore_best_weights=True,
)
history = model.fit(
    train_dataset,
    validation_data=validation_dataset,
    epochs=1000,
    callbacks=[early_stopping],
    verbose=0,
)

Batch Normalization sometimes permits a larger learning rate, but that is an empirical possibility—not permission to increase the learning rate blindly.

We should first verify stable training at the existing learning rate. Then learning rate can be tested as a separate hyperparameter.

Batch Normalization Is Not Primarily an Overfitting Method

Batch Normalization introduces noise because training statistics depend on the current mini-batch. That noise can sometimes produce a regularizing side effect.

But Batch Normalization should not be presented as a direct substitute for:

• L1 or L2 regularization
• Dropout
• Early stopping
• More representative data
• Controlled model capacity

Its principal role is connected to optimization and activation scaling.

A model can use Batch Normalization and still overfit severely. A model can also generalize well without Batch Normalization.

If validation performance does not improve—or if training becomes unstable because batches are too small—the layer may not be useful for that experiment.

How to Compare Models Fairly

To determine whether Batch Normalization helps, we need more than two unrelated training runs.

The comparison should hold constant:

• Training, validation, and test partitions
• Preprocessing
• Hidden-layer widths
• Activation functions
• Initializer strategy
• Optimizer
• Learning rate
• Batch size
• Early-stopping rules
• Maximum epochs

The training dataset should also be reconstructed using the same shuffle seed for each run.

A reusable model function helps:

def hidden_block(
    x,
    units,
    block_name,
    use_batch_normalization,
):
    x = layers.Dense(
        units,
        use_bias=not use_batch_normalization,
        kernel_initializer="he_normal",
        kernel_regularizer=regularizers.L2(
            1e-4
        ),
        name=f"{block_name}_linear",
    )(x)
    if use_batch_normalization:
        x = layers.BatchNormalization(
            name=f"{block_name}_normalization"
        )(x)
    x = layers.Activation(
        "relu",
        name=f"{block_name}_relu",
    )(x)
    return x

def build_candidate(
    number_of_features,
    use_batch_normalization,
):
    inputs = keras.Input(
        shape=(number_of_features,),
        name="features",
    )
    x = hidden_block(
        inputs,
        units=32,
        block_name="hidden_1",
        use_batch_normalization=(
            use_batch_normalization
        ),
    )
    x = layers.Dropout(
        0.20,
        name="dropout",
    )(x)
    x = hidden_block(
        x,
        units=16,
        block_name="hidden_2",
        use_batch_normalization=(
            use_batch_normalization
        ),
    )
    outputs = layers.Dense(
        1,
        activation="sigmoid",
        kernel_initializer="glorot_uniform",
        name="probability",
    )(x)
    candidate = keras.Model(
        inputs=inputs,
        outputs=outputs,
    )
    candidate.compile(
        optimizer=keras.optimizers.Adam(
            learning_rate=1e-3
        ),
        loss=keras.losses.BinaryCrossentropy(),
        metrics=[
            keras.metrics.BinaryAccuracy(
                name="accuracy"
            ),
            keras.metrics.AUC(name="auc"),
        ],
    )
    return candidate

We should run each configuration more than once with controlled seeds. A single result can reflect favorable or unfavorable random initialization, batch order, or dropout masks.

The test set must remain outside this comparison. Select the configuration using validation evidence, then evaluate the chosen procedure on the test set once.

Common Mistakes

Initializing Every Weight to Zero

This prevents neurons within the same layer from learning distinct features.

Using a Fixed Large Standard Deviation

The appropriate scale depends on fan-in, fan-out, and activation behavior.

Assuming the Keras Default Is He Initialization

The current Dense default is glorot_uniform. Specify he_normal or he_uniform explicitly when that is the intended strategy.

Believing He Initialization Is Always Better

He initialization is designed around rectifying activations. It is not automatically superior for every activation or architecture.

Treating Batch Normalization as Input Standardization

Batch Normalization acts on internal activations. It does not replace leakage-safe feature preprocessing.

Forgetting Training–Inference Differences

Training uses current-batch statistics. Evaluation and prediction use moving statistics.

Using Extremely Small Batches Without Checking Stability

Noisy batch statistics can create a mismatch between training and inference behavior.

Keeping an Unnecessary Dense Bias

A Dense bias is usually redundant when Batch Normalization follows immediately.

Calling Batch Normalization an Overfitting Cure

It may have a regularizing side effect, but its primary role is optimization.

Increasing the Learning Rate Automatically

Batch Normalization may tolerate larger learning rates, but the appropriate value must still be validated.

Frequently Asked Questions

What does weight initialization do?

It chooses the network’s starting parameter values. A good initializer breaks symmetry and keeps initial signals and gradients in a workable numerical range.

What is the difference between Xavier and Glorot initialization?

There is no difference in this context. Xavier initialization is another name for Glorot initialization.

When should I use He initialization?

He initialization is a principled default for layers followed by ReLU or related rectifying activations.

Does Batch Normalization normalize the dataset?

No. It normalizes internal layer inputs using mini-batch statistics during training and moving statistics during inference.

Should Batch Normalization come before or after ReLU?

Both patterns exist. This article uses Dense → Batch Normalization → ReLU, which normalizes the linear pre-activations before applying ReLU.

Can Batch Normalization replace dropout?

Not reliably. They operate through different mechanisms. Whether both, one, or neither is useful should be determined through validation experiments.

Why does Batch Normalization have non-trainable parameters?

Its moving mean and moving variance are updated during training but are not optimized through gradient descent. They are stored for inference.

Is Batch Normalization useful for small structured datasets?

Sometimes, but not automatically. Small batches and limited observations may produce unstable estimates. Its value must be tested against a comparable model without it.

Final Perspective

A neural network is not only a set of equations. It is a chain through which numerical signals and gradients must travel.

Poor initialization can damage that chain before learning begins. Values may vanish, explode, saturate, or remain trapped in symmetrical behavior.

Glorot initialization scales weights using both fan-in and fan-out. He initialization adjusts the scale for the signal-reducing effect of rectifying activations. These methods give the network a statistically informed starting point.

Batch Normalization acts after that starting point. It uses batch statistics during training, learns an affine transformation, and accumulates moving statistics for inference.

But these tools should not be turned into slogans.

He initialization does not guarantee convergence. Batch Normalization does not guarantee generalization. Neither method repairs data leakage, poor labels, an inappropriate metric, or a biased dataset.

Their value becomes clear when we understand the exact problem they address.

In Part 22, we will make the training process recoverable and reproducible. We will use callbacks and checkpoints, save the best model rather than merely the final epoch, preserve preprocessing state, and reconstruct the complete path from a raw observation to a production-ready prediction.