Feature Scaling, Side by Side

1Who is my nearest neighbor?

Click any district. Its 5 nearest neighbors light up, measured by the ordinary (Euclidean) distance on total_rooms and median_income, as kNN would measure it. The shaded shape is the neighborhood: every district inside it is at least as close as the 5th neighbor.

The plot always shows the original units, so you can compare the three modes. Districts with more than 10,000 rooms are drawn at the right edge. Statistics for scaling (min, max, mean, standard deviation) come from these 2,000 districts, as fit() would compute them.

2One bad value, two scalers

All 16,512 training districts have a median_income between 0.5 and 15 (tens of thousands of dollars). Add one more district whose income was typed in wrong, and move it up.

Min-max scaling

Standardization

3Scaling vs. transforming

Pick a feature and a treatment. Watch the numbers on the x-axis, and watch the shape. Square root, x^0.25 and log are steps on one ladder: each shrinks large values harder than the one before.

Bonus: a new feature, similarity to age 35

The Gaussian RBF exp(-γ (age − 35)²) is 1 at age 35 and falls toward 0 away from it. Its values lie between 0 and 1, while the histogram counts run into the hundreds: on one y-axis, the curve disappears.

The same in scikit-learn

# fit on the training set only, then transform any set
from sklearn.preprocessing import MinMaxScaler, StandardScaler
min_max_scaler = MinMaxScaler(feature_range=(0, 1))
housing_num_min_max_scaled = min_max_scaler.fit_transform(housing_num)
std_scaler = StandardScaler()
housing_num_std_scaled = std_scaler.fit_transform(housing_num)

# a transformation changes the shape
np.log(housing_num["population"])

# similarity to age 35 as a new feature
from sklearn.metrics.pairwise import rbf_kernel
age_simil_35 = rbf_kernel(housing[["housing_median_age"]], [[35]], gamma=0.1)

Reading the panels

• Distances add up the features in their own units. Unscaled, a difference of 50 rooms outweighs the whole income range, so the "nearest" districts can be rich or poor. Scaling puts both features on equal footing.
• Min-max scaling lets the extremes decide. One wrong value becomes the new max and squeezes everybody else into a sliver of [0, 1].
• Standardization looks at the whole crowd. The mean and standard deviation of 16,513 values hardly notice one outlier, so everybody else keeps their z-score.
• Scaling changes the units, not the shape. A heavy tail stays heavy after standardizing; the skewness does not move. Square root, powers and the log change the shape - and which one is right depends on the feature.
• Mean 0 and std 1 is not a bell curve. The standard normal distribution has mean 0 and std 1, but so does every standardized feature, whatever its shape. The z-scores of income in panel 2 are still skewed to the right.
• Plain linear regression does not care about scaling (its coefficients absorb the units), but kNN, gradient descent, Ridge and Lasso do. It does care about the shape: that is what the log helps with.