Deep Learning from Scratch, Part 19: Building a Keras Neural Network for Structured Data

In the previous parts of this series, we built neural networks almost entirely by hand. We wrote the forward pass, calculated the loss, propagated gradients backward, and updated every parameter ourselves.

That work was not merely an exercise before reaching a “real” framework. It gave us the mental model needed to understand what a framework is doing on our behalf.

Keras is a high-level neural-network API that lets us describe the layers and training rules while it handles gradient calculation, batching, and parameter updates. The mathematics has not disappeared; we have simply handed its repetitive execution to a tested computational system.

In this article, we will build a Keras neural network for structured data and connect each Keras instruction to the NumPy operations we already understand.

We will also preserve the most important lesson from Part 18: a sophisticated model is still unreliable if information leaks across the training, validation, and test boundaries.

What Changes When We Move from NumPy to Keras?

Our NumPy implementation required us to manage several operations explicitly:

• Initialize weight matrices and bias vectors
• Calculate linear transformations
• Apply activation functions
• Store intermediate values
• Derive and calculate gradients
• Update parameters
• Divide the data into mini-batches
• Record losses and metrics
• Stop training at an appropriate point

Keras performs most of these operations for us.

But Keras does not decide whether our experiment is scientifically valid. It cannot know that we fitted a scaler on the complete dataset, repeatedly inspected the test set, selected an inappropriate metric, or encoded missing values incorrectly.

Automation reduces implementation work. It does not replace reasoning.

Keras and TensorFlow Are Related, but They Are Not Identical

Keras defines the model-building and training interface we will use. TensorFlow is the computational backend responsible for tensor operations, automatic differentiation, and hardware execution in this example.

Keras 3 can work with TensorFlow, JAX, or PyTorch as its backend. We will use TensorFlow while keeping our imports consistently within the standalone Keras API. The backend must be selected before importing Keras. Keras getting started guide

import os
os.environ["KERAS_BACKEND"] = "tensorflow"
import keras
from keras import layers, regularizers

Avoid mixing imports such as keras.layers and tensorflow.keras.layers throughout the same project. Even when such code appears to run, a consistent interface makes versioning, serialization, and debugging easier to reason about.

Step 1: Load the Structured Dataset

We will continue with a binary classification dataset stored in diabetes.csv. Each row represents one observation, the input columns contain numerical features, and Outcome is the binary target.

import numpy as np
import pandas as pd
data = pd.read_csv("diabetes.csv")
X = data.drop(columns="Outcome")
y = data["Outcome"].astype("float32")
print("Input shape:", X.shape)
print("Target shape:", y.shape)
print("Class distribution:")
print(y.value_counts(normalize=True))

Keeping X as a pandas DataFrame at this stage preserves the column names. That matters because structured-data pipelines depend not only on values but also on feature meaning and feature order.

A Dataset-Specific Warning About Missing Values

In some versions of the Pima Indians Diabetes dataset, zero may be used as a missing-value marker in columns where a literal zero is physiologically implausible, such as glucose concentration or body mass index.

We should not automatically convert every zero into a missing value. Pregnancy count, for example, may legitimately be zero. The correct treatment depends on the dataset documentation.

If missing-value imputation is necessary, its parameters must be learned from the training set only—exactly like feature scaling. Otherwise, the imputer becomes another path through which validation or test information can enter training.

Step 2: Create Honest Train, Validation, and Test Sets

We will reserve:

• 60% of the observations for training
• 20% for validation
• 20% for final testing

The first split separates the test set. The second split divides the remaining development data into training and validation sets.

from sklearn.model_selection import train_test_split
X_development, X_test, y_development, y_test = train_test_split(
    X,
    y,
    test_size=0.20,
    random_state=42,
    stratify=y,
)
X_train, X_validation, y_train, y_validation = train_test_split(
    X_development,
    y_development,
    test_size=0.25,
    random_state=42,
    stratify=y_development,
)
print("Training observations:", len(X_train))
print("Validation observations:", len(X_validation))
print("Test observations:", len(X_test))

Taking 25% of the remaining 80% produces a validation set containing 20% of the original observations.

stratify approximately preserves the class ratio across the three subsets. This is particularly useful when one class is less common than the other.

The test set should now be placed conceptually behind a closed door. Architecture decisions, regularization strength, early stopping, and threshold selection must be based on the training and validation data—not on test performance.

Step 3: Scale Without Leaking Information

Neural networks usually train more effectively when numerical features operate on comparable scales. However, a scaler learns statistics from data, so it must be treated as part of the model-fitting process.

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_validation_scaled = scaler.transform(X_validation)
X_test_scaled = scaler.transform(X_test)

Notice the difference:

• fit_transform() is used only on the training set.
• transform() is used on the validation and test sets.

The validation and test distributions are not allowed to influence the training mean or standard deviation.

Keras commonly performs its numerical operations using 32-bit floating-point values, so we can convert the arrays explicitly:

X_train_scaled = X_train_scaled.astype("float32")
X_validation_scaled = X_validation_scaled.astype("float32")
X_test_scaled = X_test_scaled.astype("float32")
y_train = y_train.to_numpy(dtype="float32")
y_validation = y_validation.to_numpy(dtype="float32")
y_test = y_test.to_numpy(dtype="float32")

Because this dataset fits comfortably in memory, NumPy arrays are sufficient. Keras officially supports NumPy arrays in its built-in training methods; we will introduce tf.data pipelines in Part 20. Keras training and evaluation guide

Step 4: Define the Neural Network

We will use the following architecture:

• Eight input features
• A hidden layer with 32 ReLU units
• A dropout layer
• A hidden layer with 16 ReLU units
• One sigmoid output unit

keras.utils.set_random_seed(42)
number_of_features = X_train_scaled.shape[1]
model = keras.Sequential(
    [
        keras.Input(
            shape=(number_of_features,),
            name="features",
        ),
        layers.Dense(
            32,
            activation="relu",
            kernel_regularizer=regularizers.L2(1e-4),
            name="hidden_1",
        ),
        layers.Dropout(
            0.20,
            name="dropout",
        ),
        layers.Dense(
            16,
            activation="relu",
            kernel_regularizer=regularizers.L2(1e-4),
            name="hidden_2",
        ),
        layers.Dense(
            1,
            activation="sigmoid",
            name="probability",
        ),
    ],
    name="structured_data_classifier",
)

A Sequential model is appropriate here because the network is a simple stack: every layer has one input and one output, and information moves through the layers in a single direction. More complex models with multiple branches, multiple inputs, or residual connections require Keras’s Functional API. Keras Sequential model guide

Why ReLU in the Hidden Layers?

Without nonlinear activation functions, stacking Dense layers would still collapse into a single linear transformation. ReLU allows the network to learn piecewise-linear decision surfaces and interactions that a single linear classifier cannot represent.

Why Sigmoid in the Output Layer?

The target has two possible values, zero and one. A single sigmoid unit maps the final logit into a value between zero and one, which we interpret as the model’s estimated probability for the positive class.

Why Include L2 and Dropout?

L2 regularization discourages excessively large weights, while dropout temporarily removes a random fraction of hidden activations during training. They reduce overfitting through different mechanisms, as discussed in Part 16 and Part 17.

Their inclusion does not guarantee better generalization. The values 1e-4 and 0.20 are reasonable starting points, not universal constants. They must ultimately be compared through controlled validation experiments.

Step 5: Inspect Shapes and Parameter Counts

Because we supplied an explicit input shape, Keras can create the model’s weights immediately and display the architecture:

model.summary()

The expected trainable parameter counts are:

• First Dense layer: 288
• Second Dense layer: 528
• Output layer: 17
• Total: 833

The dropout layer has no trainable parameters. It changes how activations are used during training but does not create a new weight matrix.

L2 regularization also adds no weights. It adds a penalty derived from the existing weights to the objective being minimized.

Counting parameters manually is still useful when working with Keras. It verifies that the model received the expected input shape and helps us detect architectures that are unexpectedly large relative to the dataset.

Step 6: Compile the Model

Defining the architecture tells Keras how information moves forward through the network. Compilation tells Keras how the network should learn and how its performance should be monitored.

model.compile(
    optimizer=keras.optimizers.Adam(
        learning_rate=1e-3,
    ),
    loss=keras.losses.BinaryCrossentropy(),
    metrics=[
        keras.metrics.BinaryAccuracy(name="accuracy"),
        keras.metrics.AUC(name="auc"),
    ],
)

The three components have different responsibilities.

The Optimizer

Adam determines how the parameters are updated after gradients have been calculated. The learning rate controls the approximate scale of those updates.

Keras’s automatic differentiation calculates the gradients, while the optimizer uses them to update the parameters.

Adam modifies the effective update using running estimates of the gradient’s first and second moments; the expression above shows the underlying gradient-based idea.

The Loss Function

Binary cross-entropy measures the disagreement between the observed binary labels and the predicted probabilities.

Because the Dense layers use L2 regularization, the loss reported by Keras is not always pure binary cross-entropy. The regularization penalties are included in the total optimization objective.

This distinction matters when interpreting the loss curves. Accuracy or AUC may improve even when small differences in the regularized loss are difficult to interpret directly.

The Metrics

Accuracy reports the fraction of classifications that are correct at its configured threshold. AUC evaluates how well the model ranks positive observations above negative observations across many possible thresholds.

Metrics describe performance, but the optimizer does not normally minimize them directly. The loss function remains the mathematical training objective.

Step 7: Train with Explicit Validation Data

We will use early stopping to prevent the model from continuing long after validation performance has stopped improving.

early_stopping = keras.callbacks.EarlyStopping(
    monitor="val_loss",
    mode="min",
    patience=25,
    min_delta=1e-4,
    restore_best_weights=True,
)
history = model.fit(
    X_train_scaled,
    y_train,
    validation_data=(
        X_validation_scaled,
        y_validation,
    ),
    epochs=1000,
    batch_size=32,
    callbacks=[early_stopping],
    verbose=0,
)

model.fit() now performs the repeating training cycle that we previously wrote by hand:

1. Select a mini-batch.
2. Run the forward pass.
3. Calculate the loss.
4. Calculate gradients through backpropagation.
5. Update the parameters.
6. Repeat until the epoch is complete.
7. Evaluate the current model on the validation data.

The validation set is evaluated but not used for gradient updates.

The maximum of 1,000 epochs does not mean the model must train for 1,000 epochs. It gives early stopping enough room to locate a useful stopping point. Training ends when validation loss fails to improve by at least min_delta for 25 consecutive epochs.

restore_best_weights=True is crucial. Without it, training would stop after the patience period while retaining the weights from the final epoch—not necessarily the weights that produced the lowest validation loss.

trained_epochs = len(history.history["loss"])
best_epoch = np.argmin(history.history["val_loss"]) + 1
print("Epochs completed:", trained_epochs)
print("Best validation-loss epoch:", best_epoch)

Step 8: Read the Training Curves

The object returned by fit() contains the recorded training and validation values.

import matplotlib.pyplot as plt
history_values = history.history
epochs = range(1, len(history_values["loss"]) + 1)
figure, axes = plt.subplots(
    1,
    3,
    figsize=(16, 4),
)
axes[0].plot(
    epochs,
    history_values["loss"],
    label="Training loss",
)
axes[0].plot(
    epochs,
    history_values["val_loss"],
    label="Validation loss",
)
axes[0].set_title("Loss")
axes[0].set_xlabel("Epoch")
axes[0].legend()
axes[1].plot(
    epochs,
    history_values["accuracy"],
    label="Training accuracy",
)
axes[1].plot(
    epochs,
    history_values["val_accuracy"],
    label="Validation accuracy",
)
axes[1].set_title("Accuracy")
axes[1].set_xlabel("Epoch")
axes[1].legend()
axes[2].plot(
    epochs,
    history_values["auc"],
    label="Training AUC",
)
axes[2].plot(
    epochs,
    history_values["val_auc"],
    label="Validation AUC",
)
axes[2].set_title("AUC")
axes[2].set_xlabel("Epoch")
axes[2].legend()
plt.tight_layout()
plt.show()

We should not judge the model from a single curve in isolation.

A widening gap between training and validation loss may indicate overfitting. Persistently high losses on both sets may suggest underfitting, poor optimization, insufficiently informative features, or data-quality problems.

It is also possible for validation accuracy to temporarily exceed training accuracy. Dropout is active while the training batches are processed but inactive during validation. The training task is therefore deliberately noisier.

That does not automatically mean the validation set is easier or that leakage has occurred. The entire pattern—and the data pipeline—must be considered.

Step 9: Evaluate the Test Set Once

After the architecture and training process have been finalized using validation results, we can evaluate the untouched test set:

test_results = model.evaluate(
    X_test_scaled,
    y_test,
    return_dict=True,
    verbose=0,
)
for metric_name, metric_value in test_results.items():
    print(f"{metric_name}: {metric_value:.4f}")

Keras’s built-in workflow distinguishes between training with fit(), final evaluation with evaluate(), and inference with predict(). Keras training and evaluation guide

We have deliberately not inserted invented test results into this article. The actual values depend on the exact dataset version, split, preprocessing rules, software environment, and random initialization.

If the test performance is disappointing, we should not repeatedly redesign the model around that result. Doing so gradually converts the test set into another validation set.

Step 10: Convert Probabilities into Class Predictions

The sigmoid output produces probabilities rather than final class labels:

test_probabilities = model.predict(
    X_test_scaled,
    verbose=0,
).ravel()
test_predictions = (
    test_probabilities >= 0.50
).astype("int32")

The threshold of 0.50 is a conventional starting point, not a law of binary classification.

In a medical screening problem, a false negative may have a very different cost from a false positive. A lower threshold could improve sensitivity while increasing false alarms. A higher threshold could do the reverse.

The threshold must be selected using validation data according to the real objective. Choosing it after examining test outcomes would leak test information into the decision process.

We can inspect several classification measures after the threshold has been fixed:

from sklearn.metrics import (
    classification_report,
    confusion_matrix,
    roc_auc_score,
)
print("Confusion matrix:")
print(confusion_matrix(y_test, test_predictions))
print("\nClassification report:")
print(
    classification_report(
        y_test,
        test_predictions,
        digits=3,
    )
)
print(
    "ROC AUC:",
    roc_auc_score(y_test, test_probabilities),
)

Accuracy alone can hide asymmetric errors or weak minority-class performance. The confusion matrix, precision, recall, and AUC answer different questions and should be interpreted in relation to the application.

What Keras Automated—and What It Did Not

Keras automated:

• Weight and bias creation
• Forward computation
• Automatic differentiation
• Backpropagation
• Mini-batch iteration
• Adam parameter updates
• Metric tracking
• Validation evaluation
• Early-stopping logic

Keras did not automate:

• Understanding the feature meanings
• Recognizing invalid missing-value codes
• Preventing preprocessing leakage
• Choosing a scientifically valid split
• Selecting the correct target and loss
• Deciding which metric reflects the real cost
• Determining an appropriate classification threshold
• Guaranteeing generalization to future data

This distinction is the central lesson of moving from NumPy to Keras.

A framework can make neural-network training concise. It cannot make an experiment honest on our behalf.

Common Mistakes When Building Keras Models for Structured Data

Scaling Before Splitting

Fitting a scaler on the full dataset exposes training to validation and test statistics. Split first, then fit preprocessing operations exclusively on the training set.

Using the Test Set as Validation Data

Passing the test set into validation_data makes it part of model selection. The test set should remain untouched until the experiment is finalized.

Treating Accuracy as a Complete Evaluation

Accuracy may be misleading when classes or error costs are imbalanced. Include ranking and class-specific measures where appropriate.

Believing More Layers Automatically Mean a Better Model

Structured datasets are often much smaller than image or language datasets. An excessively large network can memorize the training data without learning a stable relationship.

Assuming Dropout Is Active During Prediction

Keras automatically disables dropout during validation and inference. Predictions are made using the complete network, with dropout’s scaling convention already accounted for.

Comparing Hyperparameters with Different Data Splits

Changing the split while changing the architecture makes the comparison ambiguous. Keep the data partitions fixed during a controlled experiment.

Mixing Keras APIs

Choose a consistent import style. This article uses import keras and objects from keras, with TensorFlow configured as the backend.

Frequently Asked Questions

Is Keras the same as TensorFlow?

No. Keras provides the high-level model-building and training API. In this article, TensorFlow supplies the computational backend. Keras 3 can also use JAX or PyTorch.

Why use Dense layers for structured data?

Each row is represented as a fixed-length feature vector. Dense layers can combine every feature with learned weights and construct nonlinear interactions across successive layers.

Dense neural networks are not automatically superior to tree-based methods on tabular data. Their value depends on the dataset size, feature representation, optimization, and evaluation protocol.

Why use sigmoid with binary cross-entropy?

A sigmoid output produces a value between zero and one for a binary target. Binary cross-entropy evaluates how well those predicted probabilities agree with the observed labels and provides useful gradients during training.

How many hidden layers and neurons should I use?

There is no universal answer. Model capacity should be treated as a hyperparameter and compared on fixed validation data. The network should be large enough to learn meaningful structure but not so large that it merely memorizes a limited dataset.

Why can validation accuracy be higher than training accuracy?

Dropout is active during training and inactive during validation. Training metrics can therefore be measured under a deliberately more difficult, noisier version of the model. Sampling variation and regularization can also contribute.

Does Keras prevent overfitting?

No. Keras provides regularizers, dropout, callbacks, and monitoring tools, but the practitioner must configure and evaluate them correctly. Overfitting is a modeling and experimental problem, not something a framework removes automatically.

Final Perspective

Our NumPy network and our Keras network are not fundamentally different kinds of models. Both perform linear transformations, apply nonlinear activation functions, calculate a loss, propagate gradients backward, and update parameters.

Keras compresses those operations into a more reliable and scalable interface.

The benefit is not that we can forget the mathematics. The benefit is that understanding the mathematics now allows us to inspect the abstraction instead of treating it as magic.

We can read a model summary and verify the parameter counts. Then, we can examine the loss and understand what is being optimized. Afterwards, we can interpret early stopping without confusing it with test evaluation. We can also recognize that preprocessing still belongs inside the experimental boundary.

That is the real transition from implementing neural networks to engineering them.

In Part 20, we will move beyond in-memory NumPy arrays and examine tensors, batches, shuffling, caching, and tf.data pipelines. The model will remain familiar, but the way data reaches it will become faster, more flexible, and more suitable for larger experiments.