Deep Learning from Scratch (Part 7) – Logistic Regression 2

How Logistic Regression Learns

In the previous part, we followed a logistic regression model from input features to a predicted probability.

The model calculated a weighted score:$$z = w_1x_1 + w_2x_2 + dots + b$$

Then it passed that score through the sigmoid function:$$hat{y} = sigma(z)$$

This gave us a probability between 0 and 1.

But one question was left open: how does the model find useful values for the weights and bias?

At the beginning, the weights may be zero, random, or simply poor. The model might assign a probability of 0.9 to the wrong class or remain close to 0.5 even when the answer should be clear.

Training is the process of adjusting those parameters so that the predictions become better.

To do that, the model needs two things:

1. A way to measure how poor its predictions are.
2. A way to change the weights and bias in the right direction.

For logistic regression, these are usually handled by log loss and gradient descent.

A correct class label does not tell the whole story

Suppose the true class is 1.

Two models produce the following probabilities:

With a threshold of 0.5, both predictions are counted as correct.

Accuracy treats them in exactly the same way.

But the predictions are not equally strong.

Model A is barely above the threshold. Model B gives much more probability to the correct class.

Now consider two wrong predictions:

Again, accuracy treats both as wrong.

Yet Model C was uncertain and only narrowly missed the threshold. Model D was extremely confident in the wrong answer.

A useful training objective should notice these differences.

This is why logistic regression is not usually trained by directly minimizing classification errors. The threshold creates a hard decision, while training needs a smoother signal that tells the model how far its probabilities are from the correct values.

That signal comes from log loss.

Log loss measures probability quality

For a single observation, binary log loss is:$$L(y,hat{y}) = -left[ ylog(hat{y}) + (1-y)log(1-hat{y}) right]$$

Here:

• $y$ is the actual class, either 0 or 1.
• $hat{y}$ is the predicted probability of class 1.
• $L$ is the loss for that observation.

The formula may look crowded, but only one side is active at a time.

When the true class is 1

If y=1, then:$$L = -log(hat{y})$$

The second part disappears because $1-y=0$.

If the model gives a high probability to class 1, the loss is small.

The closer the prediction gets to 1, the smaller the loss becomes.

A confident wrong prediction receives a much larger penalty.

When the true class is 0

If $y=0$, then:$$L = -log(1-hat{y})$$

This time, the first part disappears.

For a class 0 observation, a prediction close to 0 is good. A prediction close to 1 is heavily penalized.

This is the main behavior we want:

• correct and confident predictions produce low loss,
• uncertain predictions produce moderate loss,
• wrong and confident predictions produce high loss.

Why the logarithm?

The logarithm makes the penalty grow sharply when the model becomes confident in the wrong class.

Suppose the true class is 1.

A prediction of 0.6 is not ideal, but it still gives more probability to the correct class than the wrong one.

A prediction of 0.01 is very different. The model is almost certain that the observation belongs to class 0.

Log loss treats the second mistake as much more serious.

That matters during training. Without this stronger penalty, the model would have less reason to correct badly calibrated, overconfident predictions.

Loss and cost

The words loss and cost are often used loosely, but there is a useful distinction.

Loss usually refers to the error for one observation.

$$L_i = -left[ y_ilog(hat{y}_i) + (1-y_i)log(1-hat{y}_i) right]$$

Cost usually refers to the average loss across the dataset.

$$J(w,b) = -frac{1}{m} sum_{i=1}^{m} left[ y_ilog(hat{y}_i) + (1-y_i)log(1-hat{y}_i) right]$$

Here, $m$m is the number of observations.

In practice, you will often see the terms used interchangeably. The important part is that the model is trained by reducing the average error over the training data.

A lower log loss means that the predicted probabilities are, on average, closer to the actual classes.

It is not the same as accuracy.

Accuracy asks:

How many final class predictions were correct?

Log loss asks:

How much probability did the model assign to the correct classes?

Two models can have the same accuracy and very different log loss values.

The goal of training

At this point, logistic regression has:

• input features,
• weights,
• a bias,
• predicted probabilities,
• and a cost function.

The cost depends on the model parameters:$$J(w,b)$$

Different values of $w$w and $b$b produce different predictions. Those predictions produce different cost values.

Training means finding values for the weights and bias that make the cost as small as possible.

$$text{Find } w,b text{ that minimize } J(w,b)$$

We are not changing the input data or the true labels. We are changing the parameters inside the model.

Derivatives as directions

To reduce the cost, we need to know what will happen if a parameter changes.

Suppose there is only one weight.

We can ask:

If the weight increases slightly, does the cost increase or decrease?

The derivative gives us this local change information.

A simple example is:$$y = 5x + 10$$

The derivative with respect to $x$ is:$$frac{dy}{dx}=5$$

This means that when $x$ increases by a small amount, $y$ changes at a rate of about 5 units per unit of $x$.

For a curved function, the derivative may change depending on where we are.

$$y = 2x^2 + 50x$$

Its derivative is:$$frac{dy}{dx}=4x+50$$

The rate of change is no longer constant. It depends on the current value of $x$.

In model training, we use the same idea, but instead of asking how $y$y changes with $x$, we ask how the cost changes with each parameter.

Partial derivatives

Logistic regression usually has more than one parameter.

If there are 30 input features, the model has 30 weights plus a bias.

The cost can be written as:$$J(w_1,w_2,dots,w_p,b)$$

A partial derivative measures how the cost changes with respect to one parameter while treating the others as fixed.

For example:$$frac{partial J}{partial w_1}$$

describes how the cost changes when $w_1$ changes.

Similarly:$$frac{partial J}{partial b}$$

describes how the cost changes when the bias changes.

Each partial derivative answers a local question:

Should this parameter increase or decrease, and how strongly?

The gradient

When we collect all partial derivatives together, we get the gradient.

$$nabla J = left[ frac{partial J}{partial w_1}, frac{partial J}{partial w_2}, dots, frac{partial J}{partial w_p}, frac{partial J}{partial b} right]$$

The gradient is a vector.

It contains:

• a direction,
• and a magnitude.

The direction points toward the steepest increase in the cost.

If we want to reduce the cost, we move in the opposite direction.

This is the central idea behind gradient descent.

Gradient descent

Gradient descent updates the model parameters step by step.

For a weight:$$w := w – alphafrac{partial J}{partial w}$$

For the bias:$$b := b – alphafrac{partial J}{partial b}$$

The symbol $alpha$ represents the learning rate.

The minus sign is important.

The gradient points toward increasing cost. Subtracting it moves the parameters toward decreasing cost.

Each update follows the same pattern:

1. Make predictions.
2. Calculate the cost.
3. Calculate the gradients.
4. Update the weights and bias.
5. Repeat.

The logistic regression gradients

For binary logistic regression, the weight gradient can be written as:

$$frac{partial J}{partial w} = frac{1}{m} sum_{i=1}^{m} (hat{y}_i-y_i)x_i$$

The bias gradient is:$$frac{partial J}{partial b} = frac{1}{m} sum_{i=1}^{m} (hat{y}_i-y_i)$$

The term:$$hat{y}_i-y_i$$

is the prediction error for one observation.

If the model predicts 0.8 and the true class is 1:$$0.8-1=-0.2$$

If the model predicts 0.8 and the true class is 0:$$0.8-0=0.8$$

The sign and size of the error affect how the parameters are updated.

Why is the weight gradient multiplied by the input?

The weight appears inside the model as:$$z = wx+b$$

Changing the weight affects the score through the input value $x$.

That is why the weight gradient contains:$$(hat{y}-y)x$$

The bias appears as a separate added term:$$z = wx+b$$

Its derivative with respect to itself is 1, so there is no input multiplier in the bias gradient.

This gives us:$$frac{partial J}{partial b} = frac{1}{m} sum(hat{y}-y)$$

The input feature affects how strongly a particular observation contributes to the weight update. The bias update only depends on the prediction errors.

A single parameter update

Suppose we have one feature and one observation:

x = 2
y = 1

The current model parameters are:

w = 0.3
b = 0.1

The learning rate is:

α = 0.1

First, calculate the score: $z=wx+b$

$z=(0.3×2)+0.1=0.7$

Then calculate the probability:$$hat{y}=sigma(0.7)approx0.668$$

The prediction error is:$$hat{y}-y=0.668-1=-0.332$$

The weight gradient is:$$dw=(hat{y}-y)x$$

The bias gradient is:$$db=hat{y}-y=-0.332$$

Now update the parameters:$$w_{text{new}}=w-alpha dw$$$$b_{text{new}} = 0.1-(0.1times-0.332) = 0.1332$$

Both parameters increase.

That makes sense. The true class is 1, but the predicted probability was only 0.668. The update should push the next prediction upward.

The same update in Python

import numpy as np
def sigmoid(z: float) -> float:
    return 1 / (1 + np.exp(-z))
x = 2.0
y = 1.0
weight = 0.3
bias = 0.1
learning_rate = 0.1
# Forward pass
z = weight * x + bias
prediction = sigmoid(z)
# Gradients
dw = (prediction - y) * x
db = prediction - y
# Parameter updates
weight -= learning_rate * dw
bias -= learning_rate * db
print(f"Prediction: {prediction:.3f}")
print(f"Updated weight: {weight:.4f}")
print(f"Updated bias: {bias:.4f}")

Output:

Prediction: 0.668
Updated weight: 0.3664
Updated bias: 0.1332

This is only one update.

A real model repeats the process many times across many observations.

Learning rate

The learning rate controls the size of each update.$$theta := theta-alphanabla J$$

Here, $theta$θ is a general symbol for the model parameters.

A small learning rate produces small steps.

A large learning rate produces larger steps.

Neither extreme is automatically better.

If the learning rate is too small

• Training may be very slow.
• The cost may decrease only slightly after many iterations.
• The model may require far more epochs.

If the learning rate is too large

• The updates may jump past the minimum.
• The cost may oscillate.
• Training may become unstable.
• The cost may even increase or return NaN.

For standard logistic regression, the cost function is convex. This means there is a single global minimum rather than many competing local minima.

So the main concern is not getting trapped in a poor local minimum. The more common problem is taking steps that are too large to converge smoothly.

Epochs and iterations

An epoch usually means one complete pass through the training dataset.

If we use the entire dataset to calculate one gradient and then update the parameters, one epoch contains one update.

In larger models, data is often divided into smaller batches. In that case, one epoch contains several updates.

The training loop may look like this:

for epoch in range(number_of_epochs):
    make_predictions()
    calculate_loss()
    calculate_gradients()
    update_parameters()

During training, we often save or print the loss every few epochs.

For example:

Epoch 1   Loss: 0.693
Epoch 10  Loss: 0.258
Epoch 20  Loss: 0.193
Epoch 30  Loss: 0.165
Epoch 40  Loss: 0.146

A falling loss suggests that the model is learning on the training data.

But training loss alone is not enough. We also need to check performance on validation data to see whether the model generalizes beyond the examples it was trained on.

When should training stop?

There is no need to continue forever.

Training can stop when:

• the loss stops improving meaningfully,
• the validation performance begins to worsen,
• a fixed number of epochs is reached,
• or the gradient becomes very small.

If training loss keeps falling while validation loss starts rising, the model may be overfitting.

For logistic regression, this is often less dramatic than in large neural networks, but it can still happen, especially with many features and limited data.

Accuracy and log loss answer different questions

After training, we may evaluate the model using both accuracy and log loss.

Suppose two models make predictions for 100 observations.

Both classify 90 of them correctly.

Their accuracy is the same:$$text{Accuracy}=0.90$$

But one model may produce probabilities close to the true classes, while the other may be frequently overconfident.

Their log loss values can therefore be very different.

Model A has better probability predictions, even though the final number of correct labels is the same.

This becomes especially important when probabilities are used directly, such as in:

• medical risk estimates,
• fraud scores,
• customer churn probabilities,
• credit risk models,
• and ranking systems.

A model can be useful not only because it chooses the correct class, but because its probabilities carry meaningful information.

The full training cycle

Logistic regression training can now be summarized as a repeated sequence:

1. Calculate the weighted score

$$z=Xw+b$$

2. Convert scores into probabilities

$$hat{y}=sigma(z)$$

3. Measure the error

$$J(w,b) = -frac{1}{m} sum left[ ylog(hat{y}) + (1-y)log(1-hat{y}) right]$$

4. Calculate the gradients

$$dw= frac{1}{m}X^T(hat{y}-y)$$$$db= frac{1}{m}sum(hat{y}-y)$$

5. Update the parameters

$$w:=w-alpha dw$$$$b:=b-alpha db$$

6. Repeat

Each cycle should move the model toward parameters that produce better probability estimates.

From one model to neural networks

The same training logic appears again in neural networks.

A neural network also:

• combines inputs using weights,
• adds biases,
• applies activation functions,
• calculates a loss,
• computes gradients,
• and updates parameters using an optimizer.

The main difference is scale.

Logistic regression has one direct path from the input features to the output probability.

A neural network may have many layers and millions of parameters. The chain of derivatives becomes longer, but the basic learning cycle remains the same.

That is why logistic regression is a useful model to build from scratch. It gives us a small, visible version of the same process used in deeper models.

In the next part, we will put these pieces together in NumPy and train a logistic regression model on a real classification dataset.