Deep Learning from Scratch (Part 18): A Validation Set Is Not a Test Set

In the previous post, early stopping watched validation loss and restored the parameters from the best epoch.

That raises an important question:

Why do we need a separate validation set? Why not monitor the test set and stop when its loss is lowest?

Because the moment a result influences one of our modelling decisions, it becomes part of the development process.

A trustworthy experiment gives each dataset one clear responsibility:

The training set teaches the model, the validation set helps us choose the model, and the test set evaluates the finished process.

Confusing these roles is one of the easiest ways to make an overfit model appear trustworthy.

Three Datasets, Three Different Jobs

Suppose we divide the available observations into three non-overlapping sets:

• training;
• validation;
• test.

The separation is not merely organizational.

Each set is allowed to influence different decisions.

The training set participates directly in optimization.

The validation set does not calculate ordinary parameter gradients, but its results still influence which model we keep.

The test set should not influence either process.

The Training Set Teaches the Parameters

During training, the model receives observations from the training set.

The forward pass produces predictions.

The loss function compares those predictions with the training targets.

Backpropagation calculates gradients.

The optimizer updates the weights and biases.

If we write the model as ($f(x;\theta,h)$), then:

• ($\theta$) represents learned parameters such as weights and biases;
• ($h$) represents choices made outside ordinary backpropagation.

Examples of ($h$) include:

• number of hidden layers;
• neurons per layer;
• learning rate;
• L1 or L2 strength;
• dropout rate;
• batch size;
• patience;
• and decision threshold.

Training learns ($\theta$).

It does not ordinarily decide ($h$).

The Validation Set Helps Us Choose

Suppose we train six networks using six different L2 values.

All six networks learn their weights from the training set.

We then compare their validation losses.

The validation result influences which lambda value we select.

That means the validation set has influenced the final model, even though it never appeared inside the weight-gradient calculation.

The same applies when we use validation data to choose:

• the number of layers;
• the number of neurons;
• an optimizer;
• a learning rate;
• a dropout rate;
• a stopping epoch;
• a probability threshold;
• or a preprocessing strategy.

Formula Panel 2 separates parameter learning, model selection and final evaluation.

• $\theta$: weights and biases
• $h$: architecture and hyperparameters
• $E_{\mathrm{final}}$: final test estimate

The Test Set Judges the Finished Procedure

The test set is used after the model-development decisions have been made.

It answers:

How well does the selected modelling procedure perform on data that did not influence its construction?

This is why repeatedly checking test performance is a problem.

Imagine that we try ten architectures and inspect test accuracy after each one.

We then choose the architecture with the highest test score.

The test set has now influenced the architecture.

It has become another validation set.

Its score may be optimistic because we selected the model partly for its performance on those specific observations.

The individual network may never have calculated gradients from the test set, but the complete development process has adapted to it.

The Human Is Part of the Learning Process

Machine learning is not limited to what happens inside model.fit().

The developer also observes results and makes decisions.

Suppose we see a disappointing validation curve and respond by:

• adding a layer;
• changing lambda;
• increasing patience;
• removing a feature;
• changing the threshold;
• or trying a new split.

Each decision transfers information from the validation results into the modelling process.

This is the correct purpose of a validation set.

But repeated experimentation can eventually overfit the validation set too.

The model may become increasingly specialized to the dataset used for model selection.

A separate test set allows us to evaluate the entire process, including the decisions made by the developer.

Creating a Train–Validation–Test Split

Suppose we want:

• (60%) training data;
• (20%) validation data;
• (20%) test data.

A clear way to create this split is to reserve the test set first:

from sklearn.model_selection import train_test_split
X_development, X_test, y_development, y_test = (
    train_test_split(
        X,
        y,
        test_size=0.20,
        random_state=42,
        stratify=y
    )
)

We now have:

• (80%) development data;
• (20%) test data.

Next, we take (25%) of the development data for validation:

X_train, X_val, y_train, y_val = (
    train_test_split(
        X_development,
        y_development,
        test_size=0.25,
        random_state=42,
        stratify=y_development
    )
)

Why (25%)?

Because (25%) of the remaining (80%) is (20%) of the original dataset.

For 1,000 observations, this produces approximately:

• 600 training observations;
• 200 validation observations;
• 200 test observations.

We can verify the sizes:

print("Training:", len(X_train))
print("Validation:", len(X_val))
print("Test:", len(X_test))

What Does stratify Do?

In a classification problem, one class may be less common than another.

A purely random split can accidentally produce noticeably different class proportions across the subsets, especially when the dataset is small.

Passing:

stratify=y

asks the splitting procedure to preserve the target-class proportions approximately.

If the original data contains (65%) class 0 and (35%) class 1, stratification tries to maintain similar proportions in each subset.

We can inspect them:

import numpy as np
def positive_class_rate(y):
    return np.mean(np.asarray(y) == 1)
print(
    "Training positive rate:",
    positive_class_rate(y_train)
)
print(
    "Validation positive rate:",
    positive_class_rate(y_val)
)
print(
    "Test positive rate:",
    positive_class_rate(y_test)
)

Stratification controls class proportions.

It does not guarantee that every other feature distribution will be identical.

It also does not solve problems involving repeated people, time order or grouped observations.

Split Before Preprocessing

Our earlier structured-data notes contained a dangerous ordering:

1. scale the complete dataset;
2. split it afterward.

That allows information from validation and test observations to influence the scaler.

A standard scaler learns:

• a mean for each feature;
• and a standard deviation for each feature.

If those values were calculated using the complete dataset, the training transformation contains information about data that was supposed to remain unseen.

The leakage may look small.

The principle is not.

The correct order is:

1. split the raw data;
2. fit preprocessing on the training set;
3. apply the learned transformation to validation and test data.

In code:

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(
    X_train
)
X_val_scaled = scaler.transform(
    X_val
)
X_test_scaled = scaler.transform(
    X_test
)

Only the training set uses fit_transform().

Validation and test sets use transform().

Why Validation and Test Data Still Need Transformation

Preventing leakage does not mean leaving validation and test data unprocessed.

The network was trained on scaled inputs.

It expects new observations to use the same coordinate system.

If one feature had training mean (100) and training standard deviation (20), a new value of (130) should be transformed using those training statistics.

We do not calculate a new mean and standard deviation for each dataset.

That would place training, validation and test data in different coordinate systems.

The same saved scaler must later be used for real predictions.

Preprocessing Is Part of the Model

The learned weights alone are not enough to reproduce a prediction.

Suppose we save the neural network but lose:

• the scaler;
• the input-column order;
• missing-value rules;
• categorical encodings;
• and the probability threshold.

We have not preserved the complete prediction procedure.

A production prediction is closer to:

raw observation → preprocessing → network → probability → decision rule

Every learned preprocessing object belongs to the fitted system.

This is why preprocessing must follow the same data-separation rules as the neural network.

Leakage Is Larger Than Scaling

Standardization is only one possible source of leakage.

Any operation that learns something from data must be fitted without access to validation or test information.

Examples include:

• imputing missing values with a learned mean or median;
• selecting features;
• applying PCA;
• learning categorical vocabularies;
• estimating outlier thresholds;
• balancing classes through synthetic examples;
• choosing transformations;
• and tuning a probability threshold.

Even seemingly simple actions can leak information.

Suppose we remove columns because they correlate poorly with the target after examining the complete dataset.

The test targets influenced feature selection.

Suppose we fill missing test values using the mean of the complete dataset.

The test features influenced preprocessing.

Suppose we choose a classification threshold that maximizes test F1.

The test labels influenced the decision rule.

In each case, the final score is no longer a clean estimate of unseen performance.

Target Leakage

Some leakage is more direct.

A feature may contain information that would not actually exist when a real prediction is made.

Imagine predicting whether a patient will be diagnosed with diabetes, but including a variable recorded only after the diagnosis.

The model may achieve excellent results.

It has not learned to predict the diagnosis from information available beforehand.

It has learned to read evidence from the future.

Target leakage is not repaired by a random train–test split because the leaked feature can appear in every subset.

The feature design itself must respect prediction time.

A useful question is:

Would this value be available at the exact moment when the prediction is supposed to be made?

If the answer is no, it should not be used as an input.

Duplicate and Group Leakage

Random splitting assumes that observations can be separated independently.

That assumption is not always valid.

Suppose the dataset contains several visits from the same patient.

A random split may place one visit in training and another visit from the same person in test.

The model can exploit patient-specific information.

The test set is technically separate at the row level but not at the subject level.

Similar problems occur with:

• several images of the same object;
• multiple transactions from one customer;
• repeated measurements from one machine;
• documents from the same author;
• or samples from the same location.

In these situations, all observations from one group should usually remain in the same subset.

A group-aware split is more appropriate than ordinary random splitting.

Time Leakage

Time-ordered data creates another exception.

Suppose we are predicting next month’s demand.

A random split may train the model on future observations and evaluate it on earlier ones.

That reverses the real prediction direction.

For forecasting and other time-dependent tasks, training should generally occur on the past and evaluation on the future.

Preprocessing must follow the same direction.

A scaler used for an earlier period should not learn statistics from later observations.

Random shuffling is not automatically appropriate merely because it produces balanced-looking subsets.

When One Validation Split Is Not Enough

A fixed validation set is simple and useful.

But its result depends on which observations happened to be selected.

This is especially important when the dataset is small.

A model may look strong on one split and weaker on another.

Cross-validation provides a broader view.

In (K)-fold cross-validation, the development data is divided into (K) folds.

The procedure is repeated (K) times:

1. use (K-1) folds for training;
2. use the remaining fold for validation;
3. rotate the validation fold;
4. collect the score from each run.

Every observation is used for validation once and for training in the other folds.

Cross-Validation Produces Multiple Models

Five-fold cross-validation does not train one model that somehow moves through five datasets.

It trains five separate models.

Each model has:

• its own training fold combination;
• its own fitted preprocessing objects;
• its own initialized parameters;
• and its own validation score.

The result is summarized using the fold scores.

The mean estimates average validation performance.

The variation tells us how sensitive the result is to the split.

Two model configurations can have the same mean score but very different stability.

For example:

• Model A: mean AUC (0.82), standard deviation (0.01);
• Model B: mean AUC (0.82), standard deviation (0.09).

The average is identical.

The evidence is not.

Preprocessing Must Be Refit Inside Every Fold

Cross-validation does not prevent leakage automatically if preprocessing is performed before the folds are created.

This is wrong:

X_scaled = scaler.fit_transform(X)
scores = cross_validate(
    model,
    X_scaled,
    y,
    cv=5
)

The scaler saw every observation before cross-validation began.

The correct logic is:

for train_indices, val_indices in cv.split(X, y):
    X_fold_train = X[train_indices]
    X_fold_val = X[val_indices]
    y_fold_train = y[train_indices]
    y_fold_val = y[val_indices]
    scaler = StandardScaler()
    X_fold_train = scaler.fit_transform(
        X_fold_train
    )
    X_fold_val = scaler.transform(
        X_fold_val
    )
    parameters = initialize_parameters(
        n_inputs=X_fold_train.shape[1],
        n_hidden=8
    )
    # Train a fresh network on this fold.
    # Evaluate it on X_fold_val.

Each fold needs a fresh scaler and a fresh model.

Otherwise, information can flow between folds.

Stratified, Grouped and Time-Based Cross-Validation

Ordinary (K)-fold cross-validation is not suitable for every dataset.

Stratified K-fold

Used for classification when class proportions should remain approximately consistent.

Group K-fold

Used when observations belong to groups that must not be divided across training and validation.

Time-series split

Used when temporal order must be preserved.

The correct splitter depends on the structure of the data, not merely its number of rows.

Early Stopping Inside Cross-Validation

Early stopping makes cross-validation more subtle.

The held-out fold is supposed to estimate performance for that fold.

If we repeatedly monitor it to choose the stopping epoch, it is also being used for model selection.

For ordinary development experiments, this may be an accepted practical compromise.

For a stricter performance estimate, each outer training fold needs an additional internal validation split for early stopping.

The outer held-out fold should then be evaluated only after training decisions are complete.

This leads to nested cross-validation.

Nested Cross-Validation

Nested cross-validation contains two loops.

The inner loop selects hyperparameters.

The outer loop estimates the performance of the complete selection procedure.

For each outer fold:

1. reserve one outer fold for evaluation;
2. use only the remaining outer-training data for hyperparameter selection;
3. perform inner cross-validation on that outer-training data;
4. choose the best configuration;
5. refit using the selected configuration;
6. evaluate once on the outer held-out fold.

The outer evaluation data does not choose lambda, dropout rate, architecture or stopping configuration.

Nested cross-validation can be computationally expensive, especially for neural networks.

But it reveals an important principle:

The data used to choose a model should not also be treated as independent evidence of that model’s final performance.

Cross-Validation Does Not Eliminate the Need for a Final Test Set

A common workflow is:

1. reserve a final test set;
2. use cross-validation on the remaining development data;
3. select the hyperparameters;
4. train the selected model;
5. evaluate it once on the test set.

Nested cross-validation provides another option when the dataset is too small for a permanent test holdout or when we want to estimate the entire selection procedure across several outer splits.

The word “test” can become confusing because software libraries sometimes call the held-out fold in cross-validation a test fold.

Conceptually, that fold is often acting as validation data unless it belongs to the outer evaluation loop of a properly nested procedure.

The role of the data matters more than the variable name.

What Happens After Model Selection?

Suppose validation experiments identify:

• two hidden layers;
• L2 strength of (10^{-4});
• dropout rate of (0.20);
• learning rate of (0.01);
• and a suitable training schedule.

We now have two reasonable options.

Keep the selected trained model

Use the model from the train–validation process and evaluate it once on the test set.

This is the simplest clean workflow.

Refit using more development data

Combine training and validation data, rebuild the model using the selected hyperparameters and train again.

This uses more data, but it creates a new question: how should we determine the stopping epoch without the original validation set?

Possible solutions include:

• fixing the number of epochs using the earlier experiments;
• creating a new internal validation split;
• or using cross-validation to estimate the training duration.

Refitting is not automatically better if it changes the procedure in an uncontrolled way.

A Practical Checklist

Before accepting a model score, ask:

1. Did we split the data before fitting preprocessing?
2. Did validation or test targets influence feature selection?
3. Did we tune any hyperparameter using the test set?
4. Are repeated observations from the same subject separated?
5. Does the split respect time order?
6. Was preprocessing refitted inside every cross-validation fold?
7. Was the probability threshold selected on validation data?
8. Did early stopping monitor data reserved for final evaluation?
9. Have we inspected variability across splits or random seeds?
10. Can the complete preprocessing and prediction procedure be reproduced?

A high score is useful only when we understand how it was produced.

The Main Idea

A validation set and a test set may look identical.

Both contain inputs and targets that are not used for ordinary weight updates.

Their difference is not their file format.

Their difference is their role.

The validation set helps us make decisions.

The test set evaluates the process after those decisions are finished.

Once we inspect a result and change the model because of it, that result has entered the development loop.

This is why splitting data is not enough by itself.

We must also control how information moves after the split.

In the next post, we will return to the network itself and rebuild the learning process with Keras.

The code will become shorter, but the separation rules established here must remain intact.