My study notes on Descriptive Statistics: Understanding the "Center" and the "Spread" 📊
In professional Data Science, Exploratory Data Analysis (EDA) is where the real work happens. While we all want to jump into complex modeling, skipping a rigorous descriptive analysis is a recipe for biased results.
Here is a high-level framework for how I evaluate a new dataset:
1. Central Tendency & Data Integrity
- The Mean vs. Median Delta: This is my first "sanity check." If the mean is significantly higher than the median, I’m immediately looking for positive skewness or extreme outliers that could compromise linear models.
- Mode for Categorical Imputation: When handling missing values in categorical features, the mode is a critical baseline for imputation strategies.
2. Variance & Distribution Shape
- Standard Deviation ($\sigma$): Essential for understanding volatility, but it’s the Interquartile Range (IQR) that I rely on for robust outlier detection.
- Skewness & Kurtosis: These aren't just "shapes"—they are indicators of whether the data needs transformation (Log, Box-Cox) before it hits a model. High Kurtosis (Leptokurtic) is a major red flag for "fat-tail" risks that standard algorithms often underestimate.
3. The Strategy
Descriptive stats aren't just numbers; they dictate the cleaning strategy. If my IQR shows heavy outliers in a pricing column (like used car datasets), I’m immediately deciding between capping values or using a more robust model like a Tree-based regressor that handles outliers better than OLS.
Question for the seniors here: Do you automate your descriptive profiling (using tools like ydata-profiling) or do you prefer a manual deep-dive into the coefficients to "feel" the data first?
#DataScience #MachineLearning #Statistics #EDA #DataEngineering