Suppose a bank wants to predict which customers are likely to subscribe to a term deposit after a marketing campaign.
For this example, we will use the publicly available UCI Bank Marketing dataset.
The complete bank-full.csv file contains 45,211 customer records:
- 39,922 did not subscribe
- 5,289 subscribed
Only about 11.7% belong to the positive class.
This creates an easy way to build a model with 88.3% accuracy:
Predict no for everyone.
No neural network required. No optimizer. Barely even a keyboard.
The accuracy looks respectable, but the model fails to identify a single potential subscriber.
Accuracy Answers Only One Question
Accuracy measures the fraction of all predictions that were correct.
In a balanced dataset, this can be informative. In an imbalanced dataset, the majority class may dominate the result.
Our all-negative model correctly classifies 39,922 customers and misses every positive customer.
Its accuracy is high because “no” is common, not because the model understands the problem.
[Insert Figure 1 here — The 88% Accuracy Trap]
Accuracy is not mathematically lying.
It is answering a question that may not match what we actually need to know.
Four Possible Outcomes
A binary classifier can make four kinds of predictions.
For the bank-marketing example:
- True positive: We predict subscription, and the customer subscribes.
- False positive: We predict subscription, but the customer does not subscribe.
- False negative: We predict no subscription, but the customer would have subscribed.
- True negative: We predict no subscription, and the customer does not subscribe.
These outcomes form the confusion matrix.
[Insert Figure 2 here — The Confusion Matrix in Real Life]
The two kinds of error have different consequences.
A false positive may lead to an unnecessary call, consuming employee time and possibly annoying a customer.
A false negative means missing someone who might have accepted the offer.
Which error matters more depends on the goal and the cost of each action.
The model cannot decide that for us.
Precision: When We Say Yes, How Often Are We Right?
Precision focuses on positive predictions.
It asks:
Among the customers predicted to subscribe, how many actually subscribed?
High precision means the positive predictions are reliable.
This is useful when acting on a positive prediction is expensive. If each marketing call requires significant time or cost, we may not want a model that produces thousands of weak leads.
A model can improve precision by becoming more selective.
But becoming too selective may cause it to miss many real positives.
Recall: How Many Positives Did We Find?
Recall looks at the problem from the other direction.
It asks:
Among all customers who actually subscribed, how many did the model identify?
High recall means fewer positive cases are missed.
This matters when false negatives are costly. If the main goal is to discover as many potential subscribers as possible, recall may deserve more attention than precision.
The model that predicts “no” for everyone has:
- Accuracy: 88.3%
- Recall for subscribers: 0%
Its impressive accuracy suddenly looks less impressive.
F1: A Compromise Between Precision and Recall
Precision can be increased by making very few positive predictions.
Recall can be increased by predicting positive more often. In the extreme, predicting “yes” for everyone gives perfect recall—but produces many false positives.
F1 combines precision and recall using their harmonic mean.
[Insert Formula Panel 1 here — Accuracy, Precision, Recall, and F1]
F1 becomes high only when both precision and recall are reasonably strong.
It is useful when we want one summary score and neither false positives nor false negatives can be ignored.
But F1 also assumes that precision and recall deserve equal importance. Real-world costs do not always agree.
The Threshold Is a Decision
A binary neural network commonly produces a score between zero and one.
We often classify a customer as positive when that score is at least (0.5).
But (0.5) is only a default threshold—not a law of nature.
Lowering the threshold usually creates more positive predictions:
- Recall tends to increase.
- False negatives tend to decrease.
- False positives tend to increase.
- Precision may decrease.
Raising the threshold usually does the opposite.
[Insert Figure 3 here — Moving the Decision Threshold]
The best threshold depends on how the predictions will be used.
If marketing calls are inexpensive and missed subscribers are costly, a lower threshold may be sensible.
If each call is expensive and the team can contact only a small number of customers, higher precision may matter more.
Model training produces scores. The threshold turns those scores into actions.
AUC Looks Beyond One Threshold
Precision, recall, and F1 depend on a selected threshold.
ROC-AUC evaluates how well the model ranks positive examples above negative ones across many thresholds.
A model with a high ROC-AUC generally gives higher scores to positive cases than to negative ones.
This makes AUC useful during tuning because we can compare ranking performance without committing to one operational threshold too early.
But AUC does not tell us:
- Which threshold to use
- How many false positives the business can tolerate
- Whether predicted probabilities are well calibrated
- What precision and recall will be at the chosen threshold
With a rare positive class, a precision-recall curve can also be informative because it focuses directly on performance for the positive class.
No single metric provides the entire story.
Make Positive Mistakes More Expensive
Standard binary cross-entropy treats each example according to the same loss rule.
With strong class imbalance, the many negative examples can dominate the total training objective.
One response is weighted binary cross-entropy.
It assigns a larger cost to mistakes involving the minority class.
[Insert Formula Panel 2 here — Weighted Binary Cross-Entropy]
If positive examples receive more weight, missing a subscriber contributes more strongly to the loss. The optimizer is encouraged to pay greater attention to the minority class.
In Keras, this can also be introduced using class_weight during training.
Class weighting does not magically improve every metric. It often increases recall, but precision may fall because the model predicts more positive cases.
It changes what the model is punished for. The final trade-off still needs to be measured.
Monitor the Metric That Matters
Early stopping and hyperparameter tuning need an objective.
If we monitor validation accuracy on this dataset, a model can look strong while ignoring many subscribers.
Better candidates may include:
- Validation AUC
- Validation recall
- Validation precision
- Validation F1 or average precision
- A task-specific measure of cost
Monitoring recall alone can also be misleading. A model that predicts everyone as positive achieves perfect recall.
The metric should reward the behaviour we want without making a trivial solution look brilliant.
For this case, AUC or average precision can guide the search, while precision and recall at a chosen threshold describe the final operational trade-off.
One Feature Arrives Too Late
The dataset contains a feature called duration: the length of the most recent marketing call.
It may be highly predictive. Long conversations may be associated with successful subscriptions.
But imagine that our goal is to decide whom to call before the call happens.
At that moment, the call duration does not exist yet.
Using it would give the model information from the future relative to the decision we want to make. Validation performance may look excellent, but the feature cannot be used at prediction time.
In that scenario, duration should be excluded.
If the task is instead to predict the outcome after the call has finished, the feature may be available—but we are now solving a different problem.
This is a reminder that leakage is not always hidden inside complicated code. Sometimes it is sitting in the dataset with a perfectly ordinary column name.
A Metric Is a Value Judgment
Accuracy counts every correct prediction equally.
Business decisions rarely do.
Missing a disease, approving a fraudulent payment, overlooking a customer likely to leave, and sending an unnecessary marketing message do not have the same cost.
Precision, recall, F1, AUC, thresholds, and class weights let us examine different parts of that reality.
They do not choose what matters.
That decision must come before model selection, because the model that wins depends on the scoreboard we create.
Classification errors fit neatly into four boxes. Regression errors are less cooperative. A house-price model can be wrong by €1,000 or €100,000, and those two mistakes should not look identical.
Leave a Reply