Discover more from Daily Dose of Data Science
The Most Overlooked Problem With Imputing Missing Values Using Zero (or Mean)
...and here's what you should try instead.
Replacing (imputing) missing values with mean or zero or any other fixed value:
alters summary statistics
changes the distribution
inflates the presence of a specific value
This can lead to:
incorrect conclusions, and more.
Instead, always try to impute missing values with more precision.
kNN imputer is often a great choice in such cases.
It imputes missing values using the k-Nearest Neighbors algorithm.
Missing features are imputed by running a kNN on non-missing feature values.
The following depicts how it works:
Step 1: Select a row (
r) with a missing value.
Step 2: Find its k nearest neighbors using the non-missing feature values.
Step 3: Impute the missing feature of the row (
r) using the corresponding non-missing values of k nearest neighbor rows.
Step 4: Repeat for all rows with missing values.
Its effectiveness over Mean/Zero imputation is evident from the image below.
Mean/Zero alters the summary statistics and distribution.
kNN imputer preserves them.
Get started with kNN imputer: Sklearn Docs.
Thanks for reading Daily Dose of Data Science! Subscribe for free to learn something new and insightful about Python and Data Science every day. Also, get a Free Data Science PDF (350+ pages) with 250+ tips.
👉 Over to you: What are some other better ways to impute missing values?
👉 If you liked this post, don’t forget to leave a like ❤️. It helps more people discover this newsletter on Substack and tells me that you appreciate reading these daily insights. The button is located towards the bottom of this email.
Thanks for reading!
Whenever you’re ready, here are a couple of more ways I can help you:
Get the full experience of the Daily Dose of Data Science. Every week, receive two extensive deep dives, which:
Explain fundamental concepts of data science and statistics.
Help you approach data science problems with intuition.
Share ideas that you might be overlooking in data science.
Promote yourself (or your brand) to 27,000 subscribers by sponsoring this newsletter.
👉 Tell the world what makes this newsletter special for you by leaving a review here :)
👉 If you love reading this newsletter, feel free to share it with friends!