In classification, every prediction eventually enters one of four boxes: true positive, false positive, false negative, or true negative.
Regression is less tidy.
A house-price prediction can be wrong by $1,000, $10,000, or $100,000. All three are errors, but they do not tell the same story.
For this final post, we will use the publicly available Ames House Prices dataset on OpenML. It contains 79 explanatory variables describing homes in Ames, Iowa, with SalePrice as the target.
A single metric can tell us how wrong the model is on average.
Error analysis tells us where, how, and for whom it is wrong.
Return Predictions to the Real World
Neural networks often train more easily when numerical targets are scaled.
Instead of predicting a sale price such as $180,000 directly, the model may learn a transformed value near zero or inside a small range.
That is useful during training, but a scaled error such as 0.032 means very little to someone trying to price a house.
Before interpreting the results, both predictions and target values should be returned to the original price scale.
[Insert Formula Panel 1 here — Returning Predictions to the Original Scale]
The target transformer must be fitted using the training targets only. Test-set prices should not influence its mean, standard deviation, minimum, or maximum.
The same fitted transformer is then used to convert predictions back.
It is also safer to keep separate transformers for:
- Input features (X)
- Target values (y)
A feature scaler fitted to square footage, year built, and garage size is not secretly qualified to restore house prices. Enthusiasm is not a substitute for compatible dimensions.
One Score Is Not Enough
Three common regression metrics are MAE, MSE, and RMSE.
Mean Absolute Error
MAE calculates the average absolute distance between the real and predicted values.
If MAE is $20,000, the model is wrong by about $20,000 per house on average.
It is easy to interpret because it remains in the original target unit.
Mean Squared Error
MSE squares every error before averaging.
This gives large mistakes much more influence.
An error of $10,000 contributes 100 times more squared error than an error of $1,000—not ten times more.
MSE is useful when large mistakes deserve a particularly strong penalty, but its unit is squared dollars, which is not especially friendly to human intuition.
Root Mean Squared Error
RMSE takes the square root of MSE.
It returns the result to the original price unit while preserving the stronger sensitivity to large errors.
[Insert Formula Panel 2 here — MAE, MSE, and RMSE]
If RMSE is much larger than MAE, a small number of very large mistakes may be pulling the squared metric upward.
That difference is not proof, but it is an invitation to investigate.
Look at Predictions, Not Only Their Average
An actual-versus-predicted scatter plot places the real sale price on one axis and the predicted price on the other.
A perfect model would place every house on the diagonal line where:
Predicted price = Actual price
[Insert Figure 1 here — Actual vs. Predicted Prices]
Points above the line are overpredictions. Points below it are underpredictions.
The shape of the cloud can reveal patterns hidden by a single metric:
- Predictions may work well for ordinary homes but fail for expensive ones.
- Low-priced homes may be consistently overvalued.
- Predictions may be compressed toward the average.
- A few unusual properties may sit far away from the rest.
A model can have a respectable MAE while systematically underpricing the most expensive houses.
The average error does not complain about who received it.
Residuals: Give Every Mistake a Direction
A residual is the difference between the real value and the prediction.
[Insert Formula Panel 3 here — Residuals]
With this definition:
- A positive residual means the model predicted too low.
- A negative residual means the model predicted too high.
- A residual near zero means the prediction was close.
Residual direction matters.
Absolute error tells us how large a mistake was. Residuals also tell us which way the model was wrong.
The average residual should ideally be near zero, but that alone is not enough. Positive and negative mistakes can cancel even when they are individually enormous.
A person who is $100,000 too high on one house and $100,000 too low on another has an average residual of zero. Few homeowners would describe this as flawless balance.
What a Residual Plot Can Reveal
A residual plot commonly places predicted values on the horizontal axis and residuals on the vertical axis.
For a well-behaved model, we hope to see points scattered around zero without a strong visible structure.
[Insert Figure 2 here — Four Stories Told by Residuals]
Different patterns suggest different problems.
A Curved Pattern
If residuals form a curve, the model may be missing a nonlinear relationship.
The architecture, features, or transformations may not represent the underlying pattern adequately.
A Funnel Shape
If residuals spread out as predicted prices increase, the error variance is changing across the price range.
Expensive houses may naturally have larger absolute errors than inexpensive ones. This is often called heteroscedasticity.
A log transformation of the target or a relative-error metric may be worth considering.
Residuals Mostly Above or Below Zero
If residuals remain mostly positive in one region, the model systematically underpredicts there.
If they remain negative, it overpredicts.
This is local bias even when the overall mean residual looks harmless.
A Few Isolated Points
Large isolated residuals may be outliers, data errors, rare property types, or houses whose prices depend on information missing from the dataset.
They deserve inspection, not automatic deletion.
Find the Largest Mistakes
A useful error-analysis table contains:
- Actual price
- Predicted price
- Residual
- Absolute residual
- Percentage error
- Important property features
Sort this table by absolute residual and inspect the worst predictions.
Perhaps the model struggles with:
- Very expensive homes
- Unusually large living areas
- Rare neighbourhoods
- Uncommon sale conditions
- Missing garage or basement information
- Renovated homes whose features are poorly represented
This is where error analysis becomes more than evaluation.
Repeated mistakes can suggest:
- New features
- Better missing-value handling
- A different target transformation
- Separate treatment of rare categories
- More representative training data
- A more suitable model family
The largest errors are not merely embarrassing rows to hide at the bottom of a notebook. They are often the most informative part of the result.
Outlier Does Not Mean Useless
Suppose one house has an unusually large residual.
There are several possibilities:
- The recorded price is incorrect.
- One or more feature values are incorrect.
- The house is real but extremely unusual.
- Important information is missing.
- The model does not understand that region of the data.
Only the first two clearly suggest a data-quality correction.
A rare but genuine luxury home is not automatically a bad observation. Removing it may make the metric look better while making the model less useful for rare homes.
Outlier rules should be developed using training data and documented clearly. Removing test examples after seeing that the model predicts them poorly would corrupt the evaluation.
Cleaning the scoreboard is not the same as improving the player.
Log-Transforming House Prices
House-price distributions are often right-skewed: most homes occupy a common range, while a smaller number are extremely expensive.
A logarithmic target transformation compresses the upper tail.
This can make the learning problem easier and reduce the domination of very expensive homes.
But it also changes what the model prioritizes.
An error from $100,000 to $120,000 and an error from $500,000 to $600,000 are both 20% errors. On a log scale, they are treated more similarly than they would be under ordinary MSE in dollars.
That may be desirable if relative error matters.
It may be undesirable if a $100,000 absolute mistake must receive a much larger penalty.
The transformation should match the meaning of the task, not merely improve a leaderboard number. Predictions must also be transformed back before being presented as prices.
Compare Distributions
A histogram or density plot of actual and predicted prices provides another useful view.
If the real prices have a wide distribution but predictions cluster tightly around the mean, the model is playing it safe.
It may perform reasonably for typical homes while failing to represent the extremes.
Similar overall distributions do not guarantee correct individual predictions, but very different distributions reveal a problem immediately.
This is why several views belong together:
- MAE, MSE, and RMSE summarize error magnitude.
- Actual-versus-predicted plots show agreement across the price range.
- Residual plots reveal structure and bias.
- Error tables identify the individual cases behind the averages.
- Distribution plots show whether the model reproduces the range of the target.
Understanding the Mistakes
Hyperparameter optimization can reduce a validation metric.
It cannot tell us whether the remaining error comes from luxury homes, missing variables, unusual neighbourhoods, noisy labels, or a transformation that emphasizes the wrong type of mistake.
For that, we must look beyond the score.
A model is not trustworthy merely because its average error is small. Trust begins when we understand where it performs well, where it struggles, and whether those failures are acceptable for the decision it will support.
The series began with choosing how a neural network learns.
It ends with something just as important: learning from the neural network’s mistakes.
Leave a Reply