Pandas Missing Data (Nulls) Handling: The Ultimate Guide

Data cleaning ka sabse mushkil hissa ab banega sabse aasan. Sikhiye missing values ko handle karne ke saare 16 professional methods details ke saath.

📑 Is Blog Post Mein Aap Kya Sikhenge:

Missing data kya hota hai, yeh analytics ko kaise kharab karta hai, aur isse tackle karne ke saare technical tools:

• Null Values Detect Karna: isnull(), isna(), notnull()
• Null Values Summary Metrics: isnull().sum(), percentage metrics
• Data Dropping Strategies: dropna() aur uske advanced variations (axis, subset, thresh)
• Statistical & Sequential Imputation: fillna() with Mean/Median/Mode, ffill(), bfill(), interpolate()
• Placeholder Cleaning: replace() mapping to true NaN

🔍 Kya Hai: isnull() aur isna() Pandas ke identical twin functions hain jo pure dataset ke ek-ek cell ko scan karte hain aur batate hain ki kahan data missing (NaN) hai aur kahan valid data hai.

🎯 Kyu Use Hota Hai: Machine Learning models NaN values ko process nahi kar sakte. Isliye model building se pehle dataset ki baseline quality aur errors ko locate karne ke liye iska use hota hai.

💡 Kab Use Hota Hai: Jab aapko dataset load karte hi yeh check karna ho ki pooray data grid mein missing spots kahan hain, ya kisi specific row ko target karke null check lagana ho.

💻 Real-World Code Examples:

Example 1: Pure employees dataframe par missing data ka boolean structure check karna.

import pandas as pd
employees = pd.read_csv("employees.csv")
# Pure dataset par null check mask generate karna
null_mask = employees.isnull()

Example 2: Sirf un employees ki list filter karna jinki "Salary" missing (null) hai.

# Salary column par boolean indexing lagana
missing_salary_employees = employees[employees["Salary"].isnull()]

📊 Expected Output:

# Example 2 ka filter output:
   EmpID      Name   Salary   Department
2    103  Rajesh S      NaN           IT
5    106   Neha Sharma   NaN        Sales

✅ Best Practices:

• isnull() aur isna() bilkul identical hain. Apne pure project mein kisi ek hi method ko use karein consistency barkarar rakhne ke liye.
• Kabhi bhi baday datasets par direct df.isnull() print na karein, yeh screen ko heavy boolean flags se bhar dega. Hamesha filtering ke sath use karein.
• Agar object data columns mein space string (e.g. " ") hai, toh isnull() use detect nahi karega. Pehle string spaces ko true NaN mein replace karein.

💬 Crack the Interview:

🔍 Kya Hai: notnull() aur notna() functions isnull() ke opposite hote hain. Yeh har cell par True flag karte hain jahan valid non-null numerical ya string data available hota hai.

🎯 Kyu Use Hota Hai: Jab hume missing data se koi lena-dena nahi hota aur hum sirf wahi clean records chahte hain jo analytics ke liye fully formatted hain.

💡 Kab Use Hota Hai: Jab primary analytics models chalane hon ya reports banani hon aur hum raw database se non-missing parameters ko filter out karna chahte hon.

💻 Real-World Code Examples:

Example 1: Sirf un profiles ko analyze karna jinka standard "Department" populated hai.

assigned_staff = employees[employees["Department"].notnull()]

Example 2: Bank records se non-null target scores trace karke segments create karna.

valid_accounts = bank[bank["CreditScore"].notna()]

📊 Expected Output:

# Example 1 ka output (All rows where Department is filled):
EmpID      Name  Salary Department
0    101   Rahul S 65000.0         IT
1    102  Preeti K 72000.0      Sales

✅ Best Practices:

• Agar baday operations chains hain, toh filter mask ko safe copy ke roop mein store karein: df.copy() taaki structural issues na aayein.
• Is function ko negate karne ke liye python ka ~ logical operator bhi use kiya ja sakta hai (e.g. ~df.notnull()).
• Data reporting aur auditing reports generate karte waqt clean files ko filter out karne ke liye yeh standard technique hai.

💬 Crack the Interview:

🔍 Kya Hai: isnull().sum() Pandas ke sub-system functions ko combine karke banaya gaya hai. Pehle isnull() booleans (True/False or 1/0) banata hai aur sum() har column ke true values (1s) ko jodta hai.

🎯 Kyu Use Hota Hai: Ek bade dataset mein har ek individual null value ko visual scan karna impossible hai. Yeh pure columns ka missing count report ek glance mein dikhata hai.

💡 Kab Use Hota Hai: Exploratory Data Analysis (EDA) ke sabse pehle phase mein, jab hum quality checks metrics validation tests design karte hain.

💻 Real-World Code Examples:

Example 1: Pure employees column wise total missing counters check karna.

missing_counts = employees.isnull().sum()

Example 2: Specific segments targets parameters fields indicators lists totals check calculations.

specific_gaps = employees[["Salary", "Age"]].isnull().sum()

📊 Expected Output:

# Example 1 ka summary stats:
EmpID          0
Name           2
Salary        14
Department     5
dtype: int64

✅ Best Practices:

• Agar baday enterprise records databases hain, toh direct print karne ke bajaye checks filter targets indicators laga sakte hain: df.isnull().sum().filter(like='target_col').
• Is output report format ko values sorted format mein display karein: df.isnull().sum().sort_values(ascending=False) takki highly damaged columns top par aayein.
• Data Quality Check pipelines mein iska threshold trigger standard rules integration systems banayein.

💬 Crack the Interview:

🔍 Kya Hai: Yeh formula missing value counts ko column ki total length se divide karke 100 se multiply karta hai. Isse direct null ratio percentage metrics details print hoti hain.

🎯 Kyu Use Hota Hai: Kuch columns mein 10 missing values ho sakti hain. Agar dataset 100 rows ka hai, toh yeh 10% hua. Agar dataset 10 lakh rows ka hai, toh yeh negligible (0.001%) hai. Isliye percentage ratio dekhna critical hai.

💡 Kab Use Hota Hai: Jab hume dropping thresholds limits set karne hon (jaise 40% se zyada missing data hone par poore column ko hi delete karna hai).

💻 Real-World Code Examples:

Example 1: Pure employees metadata checks percentage analysis calculations.

missing_percentage = employees.isnull().mean() * 100

Example 2: Filter and display only columns that have more than 30% missing values.

highly_damaged_cols = missing_percentage[missing_percentage > 30]

📊 Expected Output:

# Example 1 output ratios percentage:
EmpID          0.00%
Name           0.40%
Salary        14.20%
Department     5.10%
dtype: float64

✅ Best Practices:

• Is output value report decimals points ratios limit setups targets adjust: (df.isnull().mean() * 100).round(2) takki presentation tidy lage.
• General guidelines: Agar kisi column mein 50% se zyada value missing hain, toh feature engineering se pehle domain expert se consult karein.
• Data cleaning templates standard function modules design system triggers create templates loops profiles formats.

💬 Crack the Interview:

🔍 Kya Hai: dropna() simple and ultimate system tool function clean structure parameters, jo database se un sabhi individual row frames records ko delete kar deta hai jahan kisi ek bhi cell coordinate mein NaN/null markers filled system tha.

🎯 Kyu Use Hota Hai: Jab missing records percentage bohot low ho (jaise < 2%) aur rows drop karne se overall analytics baseline predictions matrices limits metrics target structures standard records lists variance templates drop checks parameters scale deviations limits indicators.

💡 Kab Use Hota Hai: Quality filters operations systems limits checks setups, jahan pure indicators entries validation coordinates models.

💻 Real-World Code Examples:

Example 1: Pure dataset se standard basic drops, filtering structural records templates.

clean_rows_df = employees.dropna()

Example 2: Dropping only when all columns cells are fully empty (NaN).

clean_empty_records = employees.dropna(how="all")

📊 Expected Output:

# Original Employees shape: (500, 8)
# After Example 1 shape check:
(432, 8)  # 68 rows dropped containing at least one NaN cell

✅ Best Practices:

• Pehle checks run karein, direct inplace=True use karne se bachein kyuki isse deleted index levels records easily reverse nahi ho pate.
• Bade databases targets maps setups run indicators verify lengths changes after drop setups.
• Ensure drops limits boundaries scales, data distribution should not distort after drop systems.

💬 Crack the Interview:

🔍 Kya Hai: Jab hum dropna() system call standard parameter configuration axis=1 pass karte hain, tab execution system row levels horizontal arrays ko target nahi karta. Yeh vertical columns targets structures ko search/delete karta hai.

🎯 Kyu Use Hota Hai: Kisi system fields datasets (e.g. users secondary descriptions registers) mein full column attributes arrays cells standard blanks properties registers profiles mappings limits models structure completely redundant properties variables columns dropping setup elements patterns.

💡 Kab Use Hota Hai: Column patterns filtering, jahan standard threshold rules profiles systems parameters checks verify patterns logs parameters coordinates levels.

💻 Real-World Code Examples:

Example 1: Deleting whole columns if they contain even a single missing value cell.

clean_columns_df = employees.dropna(axis=1)

Example 2: Deleting columns when they are 100% empty containing only NaN values.

absolute_clean_cols_df = employees.dropna(axis=1, how="all")

📊 Expected Output:

# Original Employees columns: ['EmpID', 'Name', 'Salary', 'Department']
# After Example 1 columns remain:
['EmpID']  # Columns 'Name', 'Salary', 'Department' were dropped due to null cells

✅ Best Practices:

• Bina baseline review system details metrics target lists checked structures parameters, direct vertical drops execute karne se bachein (useful data delete ho sakta hai).
• Set thresholds check configurations always inside standard parameters maps profiles registers metrics.
• Alternative approach: Pehle target columns indexes details values verify metrics lists values logs systems.

💬 Crack the Interview:

🔍 Kya Hai: Yeh standard dropna() function ka ek highly precise controller parameter hai. Isme hum specific column labels pass karte hain (subset array), aur drop conditions sirf unhi listed columns par restrict ho jati hain.

🎯 Kyu Use Hota Hai: Dataset mein kuch columns secondary hote hain (jaise Age ya City) jinke blank hone se farq nahi padta, lekin primary fields (jaise Salary ya Employee_ID) missing nahi honi chahiye. Yeh targeted rows clear karne ke kaam aata hai.

💡 Kab Use Hota Hai: Jab key features ya machine learning labels columns mein gaps ho aur aap baki data records ko intact rakhte hue sirf critical missing entries ko flush out karna chahte hon.

💻 Real-World Code Examples:

Example 1: Sirf tabhi row drop karna jab key columns (Name ya Salary) missing ho.

clean_salary_df = employees.dropna(subset=["Name", "Salary"])

Example 2: Bank transaction logs se accounts profiles and balances level non-null filter records structures.

active_trans_df = bank.dropna(subset=["AccountHolder", "Balance"])

📊 Expected Output:

# If Department column had NaN but Salary was valid, that row stays.
# Output dataset maintains maximum density keeping secondary gaps but ensuring core validity.

✅ Best Practices:

• Subset mein hamesha target keys array pass karein: subset=['EmpID'] aur multiple indices labels checks adjust variables platforms.
• Data distributions balance parameters, verifying missing percentages before subset drops.
• Keep copies outputs: clean_df = df.dropna(subset=['col']).copy().

💬 Crack the Interview:

🔍 Kya Hai: dropna(thresh=n) row values deletion ko control karta hai ek threshold counter ke zariye. Isme 'n' ka matlab hai "Minimum Non-Null (Valid) Values". Kisi row mein kam se kam n cells valid honi chahiye, warna row delete ho jayegi.

🎯 Kyu Use Hota Hai: Excel sheet or custom database templates inputs form profiles, jahan customers fields blanks parameters skip details cards lists, but overall valid records profile standard indicators target maps. This avoids aggressive dropping of semi-filled records.

💡 Kab Use Hota Hai: Jab surveys or detailed questionnaires analytics platforms handle karte hain, jahan multiple Optional features coordinates values variables fields blanks structures.

💻 Real-World Code Examples:

Example 1: Row target details threshold level parameters limits (must have at least 5 valid columns).

clean_thresh_rows = employees.dropna(thresh=5)

Example 2: Dynamically setting threshold based on columns matrix size percentage ratios.

min_required_valid = int(len(employees.columns) * 0.7)
clean_dynamic_df = employees.dropna(thresh=min_required_valid)

📊 Expected Output:

# If row has EmpID, Name, Salary, and Department (4 valid cells) but thresh=5,
# That row gets dropped because it did not meet the minimum 5 valid criteria.

✅ Best Practices:

• Dynamic percentages targets levels constraints are much more reliable than hardcoded values.
• Understand that 'n' counts valid (non-null) records cells, not missing spots. Setting thresh=2 means 2 cells must have data.
• Check how shapes curves variations limits before final deploy standard metrics designs patterns.

💬 Crack the Interview:

🔍 Kya Hai: fillna(value) missing cells parameters spaces mapping ka primary baseline function clean structure setups hai, jo standard parameters direct constants strings labels standard placeholder values inputs update sets indicators profiles maps.

🎯 Kyu Use Hota Hai: Gaps blank status ko replace karke standard labels configurations systems properties maintain triggers arrays targets patterns (e.g. replacing missing Customer City with 'Unknown' string parameters).

💡 Kab Use Hota Hai: Categorical attributes fields processing levels, jahan absolute clean constant labels representations standard requirements.

💻 Real-World Code Examples:

Example 1: Replacing missing departments entries with static label "Bench" profiles.

employees["Department"] = employees["Department"].fillna("Bench")

Example 2: Replacing missing bonus numeric columns structures with flat zero value.

employees["Bonus"] = employees["Bonus"].fillna(0)

📊 Expected Output:

# All null slots in 'Bonus' column are now securely registered as 0.0.
# Output maintains type integrity preventing mathematical calculation errors.

✅ Best Practices:

• Hamesha target columns matching values labels formats compatibility ensure structures (e.g. don't write string labels into purely numeric float columns).
• Maintain structural metadata details records profiles updates checks indicators.
• Avoid generic placeholder filling on target predictions target parameters.

💬 Crack the Interview:

🔍 Kya Hai: Mean Imputation ek statistical data filling technique hai. Yeh numerical column ke non-null values ka arithmetic average nikal kar column ki sabhi missing values ko us average se replace kar deti hai.

🎯 Kyu Use Hota Hai: Kisi group attributes (jaise Employee Age or Height) ke normal distributions curves templates maintain metrics triggers check values models formats, preserving overall mean baseline.

💡 Kab Use Hota Hai: Symmetrical, normally distributed numerical data metrics tracks models, where data does not contain extreme outliers values.

💻 Real-World Code Examples:

Example 1: Standard Mean Imputation on employees "Age" column.

mean_age = employees["Age"].mean()
employees["Age"] = employees["Age"].fillna(mean_age)

Example 2: Category-specific mean imputation using groupby loops.

employees["Salary"] = employees.groupby("Department")["Salary"].transform(lambda x: x.fillna(x.mean()))

📊 Expected Output:

# If mean salary of IT is 60000.0, any IT employee with a NaN Salary
# is now filled with 60000.0 (Example 2 output logic).

✅ Best Practices:

• Avoid flat mean imputation on columns containing massive outliers (e.g. Salary column with high skewness due to executive pays).
• Category-wise group mean imputation (Example 2) is highly realistic compared to flat global imputation.
• Verify skewness before and after filling process profiles models.

💬 Crack the Interview:

🔍 Kya Hai: Median Imputation statistical mapping ka ek robust method hai, jo numerical column ke middlemost value (Median) ko target karke saari missing values ko replace karta hai.

🎯 Kyu Use Hota Hai: Real-world financial datasets (jaise salaries, balances, item prices) mein massive outliers hote hain. Outliers, global average (Mean) ko pull kar lete hain, lekin Median par iska asar nahi padta.

💡 Kab Use Hota Hai: Outliers heavy metric fields, highly skewed data series distributions mappings tracks levels templates profiles models.

💻 Real-World Code Examples:

Example 1: Safe Median Imputation on employees "Salary" column.

salary_median = employees["Salary"].median()
employees["Salary"] = employees["Salary"].fillna(salary_median)

Example 2: Imputing missing bank transaction balances using median scores.

bank["Balance"] = bank["Balance"].fillna(bank["Balance"].median())

📊 Expected Output:

# Symmetrical placement of middle
values in blank spots.
# Protects analytical graphs against extreme artificial spikes.

✅ Best Practices:

• Hamesha skewed columns metrics check verify systems logs.
• Median scores should be calculated only on train slices during ML pipelines to prevent data leakage.
• Verify variances trends checks profiles setups.

💬 Crack the Interview:

🔍 Kya Hai: Mode Imputation ek statistical imputation technique hai, jo dataset mein sabse zyada baar aayi values index (Mode) ko select karke categorical/string column ke missing gaps ko fill karti hai.

🎯 Kyu Use Hota Hai: Categorical attributes (jaise City, Department, Gender) ka mean ya median nahi nikal sakte. In ke empty gaps ko fill karne ke liye is segment ki sabse popular category (Mode) ko choose kiya jata hai.

💡 Kab Use Hota Hai: Categorical attributes fields imputation processing pipelines targets checks models configurations frameworks.

💻 Real-World Code Examples:

Example 1: Safely filling missing "City" entries using Mode Indexing.

city_mode = employees["City"].mode()[0]
employees["City"] = employees["City"].fillna(city_mode)

Example 2: Replacing missing ecommerce product categories using mode indicators.

product_mode = ecommerce["Category"].mode()[0]
ecommerce["Category"] = ecommerce["Category"].fillna(product_mode)

📊 Expected Output:

# Mode returns a series. We must slice [0] to extract the actual string
values.
# Output fills NaN with popular items keeping text profiles clean.

✅ Best Practices:

• Remember that mode() returns a Series because multiple items can have the same frequency. Always slice with [0] index to safely extract a single label.
• Imputing with Mode might artificially amplify popular class shares. Be careful with highly biased category sets.
• Document category distributions shifts patterns.

💬 Crack the Interview:

🔍 Kya Hai: ffill() or pad() sequential arrays values updates, jo pichli non-null cell entry registers values ko down direction standard coordinates levels forward fill updates sets properties maps.

🎯 Kyu Use Hota Hai: Time-series analyses logs, stock trends rates, sensor parameters tracks readings, dynamic metrics change setups coordinates (jahan current cell metrics previous state boundaries continuous tracks represent setups).

💡 Kab Use Hota Hai: Time-series databases, dynamic progressive stocks tracks charts logs tracking indicators.

💻 Real-World Code Examples:

Example 1: Imputing progressive stock prices trends series via ffill.

ecommerce["Price"] = ecommerce["Price"].ffill()

Example 2: Time series logs, limiting consecutive forward fills using limit parameters.

ecommerce["Price"] = ecommerce["Price"].ffill(limit=2)

📊 Expected Output:

# If
values are: [120.0, NaN, NaN, 140.0]
# Output after Example 2: [120.0, 120.0, 120.0, 140.0]

✅ Best Practices:

• Ensure datasets are correctly sorted chronological order before executing ffill.
• Use limit parameter (Example 2) to avoid copy pasting values across a long blank gap which distorts data validity.
• Verify boundaries start parameters indices patterns.

💬 Crack the Interview:

🔍 Kya Hai: bfill() function ffill() ka exact opposite hai. Yeh sequence series list mein horizontal lines up-to-down checks parameters upward indices targets valid copy updates, copying next valid cell backwards.

🎯 Kyu Use Hota Hai: Processing parameters sequences logs formats setups levels templates (jaise forecast predictions timelines systems, or filling values looking ahead to next target event properties records maps).

💡 Kab Use Hota Hai: Time-series forecasting preparation loops, data alignment targets indices tracking levels profiles models.

💻 Real-World Code Examples:

Example 1: Backward filling missing transaction dates records series profiles bank logs.

bank["TransactionDate"] = bank["TransactionDate"].bfill()

Example 2: Multidimensional category columns bfill with explicit limit boundary checks.

bank["Branch"] = bank["Branch"].bfill(limit=1)

📊 Expected Output:

# If series was: [NaN, NaN, 'Mumbai']
# After Example 1: ['Mumbai', 'Mumbai', 'Mumbai']

✅ Best Practices:

• Be extremely careful during train-test splits; using bfill() blindly on time series can cause "Data Leakage" where future values are leaked backwards.
• Set safe logical limit values boundaries (Example 2).
• Verify indexes positions chronologies maps profiles variables parameters loops.

💬 Crack the Interview:

🔍 Kya Hai: interpolate() missing numerical fields variables calculations ka advanced tool function hai, jo missing cells values coordinates ko extreme edge points boundaries values ke beech coordinate math calculation trends draw karke filled maps metrics parameters setups configurations.

🎯 Kyu Use Hota Hai: Flat fill configurations like mean ya constant data trend levels curves patterns ko distortion check metrics configurations systems, whereas math interpolation maintains original smooth trend parameters curves outlines.

💡 Kab Use Hota Hai: Time series, continuous physics sensors tracking values logs, progressive temperature monitoring platforms registers.

💻 Real-World Code Examples:

Example 1: Standard linear interpolation on progressive employees age/salary metrics.

employees["Salary"] = employees["Salary"].interpolate()

Example 2: Advanced polynomial trend interpolation to match custom curves trajectories.

employees["Salary"] = employees["Salary"].interpolate(method="polynomial", order=2)

📊 Expected Output:

# If
values were: [10000.0, NaN, 30000.0]
# After linear interpolation: [10000.0, 20000.0, 30000.0]

✅ Best Practices:

• Linear method is default and highly reliable for standard interval calculations.
• For advanced curves (Example 2), verify order scaling factors (order=2) safely without overfitting.
• Avoid applying interpolation on purely categorical attributes fields indices loops.

💬 Crack the Interview:

🔍 Kya Hai: replace() method database placeholders values patterns (jaise "?", "N/A", "missing") mapping ko absolute clean, standardized system np.nan values mein substitute karta hai.

🎯 Kyu Use Hota Hai: Multiple legacy systems data registers inputs me blank fields ke badle special characters generate kar dete hain. Jab tak ye true nulls (NaN) nahi banenge, Pandas ke null handling functions kaam nahi karenge.

💡 Kab Use Hota Hai: Dataset import karne ke turant baad, jab initial profiling me generic placeholder values dikhein.

💻 Real-World Code Examples:

Example 1: Converting legacy "?", "missing" strings to NumPy NaN values.

import numpy as np
employees = employees.replace(["?", "missing", "N/A"], np.nan)

Example 2: Target replacement dictionary mapping across specific target columns.

clean_map = {"Salary": {"-999": np.nan}, "Department": {"N/A": np.nan}}
employees = employees.replace(clean_map)

📊 Expected Output:

# If row was: [102, 'Preeti', -999.0, 'N/A']
# After Example 2: [102, 'Preeti', NaN, NaN]  # Standardized for pandas algorithms

✅ Best Practices:

• Hamesha custom codes values mappings check limits (like Example 2 where -999 was successfully replaced as True Null).
• Maintain structural dtypes; converting objects placeholders strings to NaN might trigger automatic float casts which is standard in pandas.
• Verify metrics count before and after.

💬 Crack the Interview:

Apne analytics scenarios ke basis par right missing value strategy chunye:

Next Post Preview: Masterclass Part 2

Next tutorial mein hum cover karenge: Duplicate Records Management ko pooray detailed frameworks and questions ke sath.

Happy Coding & Stay Standard! 🚀