Loops, Apply & Lambda Functions in Pandas: Complete Guide

Data transformation mein custom logic apply karna padta hai — standard functions se sab kuch nahi hota. Sikhiye 6 powerful iteration aur transformation tools jo Python loops se lekar vectorized operations tak cover karte hain aur aapke code ko 100x faster banate hain.

📑 Is Masterclass Guide Mein Aap Kya Sikhenge:

Custom data transformation aur iteration ke 6 professional tools (Slowest → Fastest):

• Basic Iteration: iterrows() / itertuples() — Row-by-row DataFrame traversal
• Pythonic Loop: List Comprehension — Fast inline loop transformation
• Inline Function: lambda — Anonymous one-line functions
• Series Transform: apply() on Series — Column-level custom function
• Row-Level Logic: apply(axis=1) on DataFrame — Multi-column row operations
• Ultimate Speed: Vectorization — NumPy-powered zero-loop operations

🔍 Kya Hai: iterrows() DataFrame ki har row ko (index, Series) pair ke roop mein iterate karta hai. itertuples() har row ko namedtuple ke roop mein iterate karta hai jo iterrows() se 10-100x faster hai. Dono row-by-row traversal ke methods hain.

🎯 Kyu Use Hota Hai: Jab complex row-level operations karni hon jo vectorized functions se possible nahi hain — external API calls per row, database lookups, file I/O per record, ya complex conditional logic jo multiple columns depend karta hai aur np.select() se handle nahi ho raha.

💡 Kab Use Hota Hai: LAST RESORT — jab koi vectorized alternative na ho tab hi. External API calls per row, complex stateful operations (previous row dependent calculations), ya debugging mein specific rows inspect karne ke liye.

💻 Real-World Code Examples:

Example 1: iterrows() se har employee ka customized message generate karna (slow approach).

import pandas as pd
employees = pd.read_csv("employees.csv")
# iterrows() — SLOW (returns index, Series)
messages = []
for index, row in employees.head(5).iterrows():
    msg = f"Dear {row['Name']}, Your salary ₹{row['Salary']:,.0f} in {row['Department']}"
    messages.append(msg)
    print(msg)

Example 2: itertuples() se same task 10x faster — always prefer over iterrows().

# itertuples() — 10-100x FASTER than iterrows()
messages = []
for row in employees.head(5).itertuples():
    msg = f"Dear {row.Name}, Your salary ₹{row.Salary:,.0f} in {row.Department}"
    messages.append(msg)
    print(msg)
# Speed comparison
import time
start = time.time()
for _ in employees.iterrows(): pass
print(f"iterrows: {time.time()-start:.3f}s")
start = time.time()
for _ in employees.itertuples(): pass
print(f"itertuples: {time.time()-start:.3f}s")

📊 Expected Output:

# Dear Rahul, Your salary ₹85,000 in IT
# Dear Priya, Your salary ₹65,000 in HR
# Dear Amit, Your salary ₹92,000 in Finance

# Speed Comparison (10,000 rows):
# iterrows:   1.250s   🐌 Slow!
# itertuples: 0.035s   ⚡ 35x Faster!

✅ Best Practices:

• iterrows() se HAMESHA bachein — yeh sabse slow Pandas operation hai. Har row ko Series mein convert karta hai jo massive overhead create karta hai. itertuples() ko prefer karein.
• itertuples() mein column access dot notation se hota hai (row.Salary), bracket notation nahi (row['Salary']). Column names mein spaces hain toh getattr(row, 'Full Name') use karein.
• Agar loop ke andar DataFrame modify kar rahe hain toh STOP — yeh anti-pattern hai. Pehle list banayein, phir ek saath assign karein ya vectorized methods use karein.

💬 Crack the Interview:

🔍 Kya Hai: List Comprehension Python ka compact loop syntax hai jo ek line mein list create karta hai. Pandas mein yeh for loops se 2-5x faster hai aur readable bhi. Syntax: [expression for item in iterable if condition]. Custom transformations ke liye iterrows() ka best replacement hai.

🎯 Kyu Use Hota Hai: For loops verbose aur slow hote hain. List comprehension same logic ko ek concise line mein likhta hai jo Pythonic bhi hai aur performant bhi. Conditional transformations, string formatting, calculated columns — sab mein for loop ki jagah list comprehension preferred hai.

💡 Kab Use Hota Hai: Simple to moderate transformations, conditional value assignment, string manipulations, calculated columns, aur jab vectorized Pandas methods available nahi hain lekin logic simple hai tab list comprehension sabse practical choice hai.

💻 Real-World Code Examples:

Example 1: Conditional column create karna — salary slab assign karna list comprehension se.

# Simple conditional — for loop ka 1-line replacement
employees["Slab"] = [
    "Senior" if sal >= 75000 else "Mid" if sal >= 45000 else "Junior"
    for sal in employees["Salary"]
]
print(employees[["Name", "Salary", "Slab"]].head())

Example 2: Multiple columns combine karke custom formatted column banana.

# Combining multiple columns with zip()
employees["Profile"] = [
    f"{name} ({dept}) - {age}yr"
    for name, dept, age in zip(
        employees["Name"], employees["Department"], employees["Age"]
    )
]
print(employees["Profile"].head())
# With filtering — only senior employees
senior_names = [name for name, sal in zip(employees["Name"], employees["Salary"]) if sal > 80000]
print(f"Senior Employees: {len(senior_names)}")

📊 Expected Output:

# Slab Assignment:
# Name    Salary   Slab
# Rahul   85000    Senior
# Priya   45000    Mid
# Amit    92000    Senior
# Sneha   38000    Junior

# Profile Column:
# Rahul (IT) - 28yr
# Priya (HR) - 34yr

# Senior Employees: 3420

✅ Best Practices:

• Multiple columns access karne ke liye zip() use karein — zip(df['A'], df['B'], df['C']) se har iteration mein corresponding values milti hain bina iterrows() ke.
• Nested ternary (if-else inside if-else) 2 levels se zyada mat karein — readability bahut kharab hoti hai. 3+ conditions ke liye np.select() ya named function use karein.
• List comprehension apply() se faster hota hai simple operations ke liye kyunki Python-level loop overhead kam hota hai.

💬 Crack the Interview:

🔍 Kya Hai: lambda Python ka anonymous (nameless) function hai jo ek line mein define hota hai. Syntax: lambda arguments: expression. Yeh primarily apply(), map(), filter() ke saath use hota hai jab chhoti si transformation ke liye separate named function define karna overkill ho.

🎯 Kyu Use Hota Hai: Har chhoti transformation ke liye def function_name() likhna verbose hai. Lambda inline throwaway function deta hai jo ek baar use karke bhool jaate hain. apply(), map(), agg(), transform() — sabhi Pandas methods ke saath lambda sabse zyada use hota hai.

💡 Kab Use Hota Hai: Simple 1-line transformations apply() ke saath, sorting key functions, groupby() ke andar custom aggregation, aur jab bhi chhota sa function chahiye jo reusable nahi hai — lambda perfect hai.

💻 Real-World Code Examples:

Example 1: Lambda ke different use cases — apply, map, sort, filter sabke saath.

# 1. apply() ke saath — salary ko lakhs mein convert
employees["Salary_Lakhs"] = employees["Salary"].apply(lambda x: round(x / 100000, 2))
# 2. map() ke saath — name capitalize karna
employees["Name_Upper"] = employees["Name"].map(lambda x: x.upper() if isinstance(x, str) else x)
# 3. sort_values() ke saath — custom sort key
sorted_df = employees.sort_values("Name", key=lambda col: col.str.len())
print(employees[["Name", "Salary", "Salary_Lakhs"]].head())

Example 2: Lambda ke saath groupby() aur agg() — custom aggregation functions.

# Custom aggregation with lambda
dept_stats = employees.groupby("Department")["Salary"].agg([
    ("Avg", "mean"),
    ("IQR", lambda x: x.quantile(0.75) - x.quantile(0.25)),
    ("Range", lambda x: x.max() - x.min()),
    ("CV%", lambda x: (x.std() / x.mean() * 100))
]).round(2)
print(dept_stats)
# groupby + transform with lambda — group percentage
employees["Dept_Salary_%"] = employees.groupby("Department")["Salary"].transform(
    lambda x: (x / x.sum() * 100).round(2)
)

📊 Expected Output:

# Salary in Lakhs:
# Name    Salary   Salary_Lakhs
# Rahul   85000    0.85
# Priya   65000    0.65
# Amit    92000    0.92

# Custom Aggregations:
# Department   Avg       IQR       Range      CV%
# IT           82500.00  32000.00  218000.00  26.80
# Finance      75200.50  28000.00  150000.00  24.60
# HR           55800.75  18000.00  92000.00   22.20

✅ Best Practices:

• Lambda sirf 1-line logic ke liye use karein. Multi-line logic, try-except, ya loops chahiye toh proper def function banayein — PEP 8 guidelines bhi yehi kehti hain.
• Lambda ko variable mein assign mat karein: f = lambda x: x*2 anti-pattern hai. Iske bajaye def f(x): return x*2 likhein — debugging mein function name dikhega.
• Lambda ke andar heavy computation avoid karein — yeh har row/element par call hota hai, vectorized alternative hamesha prefer karein.

💬 Crack the Interview:

🔍 Kya Hai: Series.apply(func) ek column (Series) ke har element par custom function apply karta hai. Yeh map() jaisa hai lekin zyada flexible — function mein complex logic, error handling, external lookups sab possible hai. Result same-length Series return hota hai.

🎯 Kyu Use Hota Hai: Built-in Pandas methods (str.upper(), astype()) har transformation cover nahi karte. Custom string parsing (phone number formatting), mathematical formulas (tax calculation), data validation (email format check), aur any domain-specific logic apply karne ke liye Series.apply() sabse common method hai.

💡 Kab Use Hota Hai: Single column par custom transformation — string parsing, format conversion, mathematical formulas, data validation, category mapping with complex rules, aur jab vectorized alternative available na ho.

💻 Real-World Code Examples:

Example 1: Tax calculation with slab-based rules — complex multi-slab logic.

def calculate_tax(salary):
    if pd.isna(salary):
        return 0
    elif salary 250000:
        return 0
    elif salary 500000:
        return (salary - 250000) * 0.05
    elif salary 1000000:
        return 12500 + (salary - 500000) * 0.20
    else:
        return 112500 + (salary - 1000000) * 0.30

employees["Annual_Tax"] = employees["Salary"].apply(calculate_tax).round(2)
print(employees[["Name", "Salary", "Annual_Tax"]].head())

Example 2: Email validation — complex regex pattern check per cell.

import re
def validate_email(email):
    if pd.isna(email):
        return "Missing"
    pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
    return "Valid" if re.match(pattern, str(email)) else "Invalid"

employees["Email_Status"] = employees["Email"].apply(validate_email)
print(employees["Email_Status"].value_counts())

📊 Expected Output:

# Tax Calculation:
# Name    Salary    Annual_Tax
# Rahul   85000     0.00       # Below 2.5L threshold
# Priya   650000    42500.00   # 12500 + (150000 × 0.20)
# Amit    1200000   172500.00  # 112500 + (200000 × 0.30)

# Email Validation:
# Valid      8920
# Invalid     850
# Missing     230

✅ Best Practices:

• apply() ke andar NaN handling hamesha karein — pd.isna() check lagayein function ke start mein warna TypeError/ValueError aa sakte hain unexpected NaN inputs par.
• Complex logic ke liye proper named function banayein (def), lambda mat use karein — testing, debugging, aur reusability sab better hoti hai.
• apply() slow hai — pehle check karein ki vectorized alternative hai ya nahi. df['col'].str.upper() apply(lambda x: x.upper()) se 10x faster hai.

💬 Crack the Interview:

🔍 Kya Hai: DataFrame.apply(func, axis=1) har row ko ek Series ke roop mein function ko pass karta hai. Function ke andar row['col1'], row['col2'] se multiple columns access kar sakte hain. Yeh multi-column dependent complex business logic implement karne ka standard method hai.

🎯 Kyu Use Hota Hai: Real business rules aksar multiple columns par depend karte hain — "Agar department IT hai AND experience 5+ years AND rating A hai toh bonus 20%". Aise multi-column conditional logic ke liye axis=1 zaroori hai jab np.select() se handle nahi ho raha.

💡 Kab Use Hota Hai: Multi-column business rules, row-level score calculation (weighted formula across columns), complex conditional transformations, data quality row-level validation, aur jab np.where()/np.select() sufficient nahi ho.

💻 Real-World Code Examples:

Example 1: Employee performance score — multiple columns ka weighted combination.

def performance_score(row):
    # Weighted score from multiple columns
    rating_score = row["Rating"] * 0.4 if pd.notna(row["Rating"]) else 0
    exp_score = min(row["Experience"] / 20, 1) * 5 * 0.3 if pd.notna(row["Experience"]) else 0
    project_score = min(row["Projects"] / 10, 1) * 5 * 0.3 if pd.notna(row["Projects"]) else 0
    return round(rating_score + exp_score + project_score, 2)

employees["Perf_Score"] = employees.apply(performance_score, axis=1)
print(employees[["Name", "Rating", "Experience", "Projects", "Perf_Score"]].head())

Example 2: E-commerce order priority classification — multi-column dependent rules.

def classify_priority(row):
    if row["Revenue"] > 50000 and row["Customer_Type"] == "Premium":
        return "🔴 Critical"
    elif row["Revenue"] > 20000 or row["Customer_Type"] == "Premium":
        return "🟡 High"
    elif row["Revenue"] > 5000:
        return "🟢 Medium"
    else:
        return "⚪ Low"

ecommerce["Priority"] = ecommerce.apply(classify_priority, axis=1)
print(ecommerce["Priority"].value_counts())

📊 Expected Output:

# Performance Score:
# Name    Rating  Experience  Projects  Perf_Score
# Rahul   4.5     7           8         3.87
# Priya   3.0     4           5         2.70
# Amit    5.0     12          10        4.50

# Order Priority:
# 🟢 Medium     5420
# ⚪ Low        4200
# 🟡 High       3120
# 🔴 Critical   2260

✅ Best Practices:

• axis=1 BHOOLNA sabse common bug hai — default axis=0 (column-wise) hai. Row-level operations ke liye axis=1 explicitly likhna MANDATORY hai warna unexpected results aayenge.
• apply(axis=1) SLOW hai — 1M+ rows par minutes lag sakte hain. Pehle np.select() ya np.where() try karein. apply(axis=1) sirf tab use karein jab logic genuinely complex ho.
• Function ke andar har column access par pd.notna() check lagayein — ek bhi NaN se puri calculation galat ho sakti hai ya error aa sakta hai.

💬 Crack the Interview:

🔍 Kya Hai: Vectorization ka matlab hai loops completely eliminate karke operations ko poore array/column par ek saath apply karna. NumPy aur Pandas internally C-compiled code use karte hain jo millions of operations parallel execute karta hai. Yeh sabse fastest approach hai — apply() se 100-1000x faster.

🎯 Kyu Use Hota Hai: Professional data engineering mein speed critical hai — lakho rows process karne mein loops minutes lete hain jabki vectorized operations seconds mein ho jaate hain. Production pipelines, real-time processing, aur large-scale ETL mein vectorization mandatory hai.

💡 Kab Use Hota Hai: HAMESHA pehle try karein! Mathematical operations, string operations (str accessor), conditional operations (np.where/np.select), datetime operations (dt accessor) — sab vectorized hain. Sirf jab vectorized option na ho tab apply()/loop consider karein.

💻 Real-World Code Examples:

Example 1: Loop vs Apply vs Vectorization — speed comparison same task par.

import numpy as np
import time
# Task: Calculate bonus = salary * 0.15 if rating > 3, else salary * 0.05

# Method 1: For Loop (SLOWEST) 🐌
start = time.time()
bonuses = []
for _, row in employees.iterrows():
    if row["Rating"] > 3:
        bonuses.append(row["Salary"] * 0.15)
    else:
        bonuses.append(row["Salary"] * 0.05)
employees["Bonus_Loop"] = bonuses
print(f"Loop: {time.time()-start:.4f}s")

# Method 2: Apply (MEDIUM) 🚶
start = time.time()
employees["Bonus_Apply"] = employees.apply(
    lambda row: row["Salary"] * 0.15 if row["Rating"] > 3 else row["Salary"] * 0.05, axis=1)
print(f"Apply: {time.time()-start:.4f}s")

# Method 3: Vectorized np.where (FASTEST) 🚀
start = time.time()
employees["Bonus_Vec"] = np.where(
    employees["Rating"] > 3, employees["Salary"] * 0.15, employees["Salary"] * 0.05)
print(f"Vectorized: {time.time()-start:.4f}s")

Example 2: Complex multi-column vectorized operations — complete cheat sheet.

# ✅ Vectorized Math — direct column arithmetic
employees["Monthly_Salary"] = employees["Salary"] / 12
employees["Experience_Months"] = employees["Experience"] * 12

# ✅ Vectorized String — str accessor
employees["Name_Upper"] = employees["Name"].str.upper()
employees["Name_Length"] = employees["Name"].str.len()

# ✅ Vectorized Conditional — np.where / np.select
employees["Senior"] = np.where(employees["Experience"] >= 5, "Yes", "No")

# ✅ Vectorized Between — between()
employees["Mid_Age"] = employees["Age"].between(30, 45)

# ✅ Vectorized Clip — boundary capping
employees["Salary_Capped"] = employees["Salary"].clip(lower=20000, upper=200000)

print("All vectorized — zero loops! 🚀")

📊 Expected Output:

# Speed Comparison (10,000 rows):
# Loop:       1.2500s    🐌 Baseline
# Apply:      0.3200s    🚶 ~4x faster
# Vectorized: 0.0008s    🚀 ~1500x faster!

# Vectorized Operations — All instant:
# Monthly_Salary, Experience_Months — Direct math ✅
# Name_Upper, Name_Length — str accessor ✅
# Senior — np.where conditional ✅
# Mid_Age — between() range check ✅
# Salary_Capped — clip() boundaries ✅

✅ Best Practices:

• Speed hierarchy yaad rakhein: Vectorized (np/pd) >> List Comprehension >> apply() >> itertuples() >> iterrows(). HAMESHA left se start karein aur right ki taraf tab jayein jab left possible na ho.
• String operations ke liye .str accessor use karein — .str.contains(), .str.replace(), .str.split() sab vectorized hain aur apply(lambda) se 10x faster hain.
• Boolean operations bhi vectorized hain: (df['A'] > 5) & (df['B'] < 10) — ek bhi loop nahi lagta aur millions of rows instant process hote hain.

💬 Crack the Interview:

HAMESHA fastest option se start karein aur neeche jayein sirf jab zaroorat ho:

Next Post Preview: Masterclass Part 12

Next masterclass mein hum cover karenge: Merge, Join & Concat Functions — pd.merge(), join(), pd.concat(), aur multiple DataFrames ko professionally combine karne ke complete tools.

Happy Coding & Stay Analytically Pure! 🚀