Random vs. Stratified Test Sets

The income mix: full dataset vs. the last test set

Shares in percent of each set; errors relative to the full dataset, as in Géron's comparison table. Click a row (or a group of bars) to follow that category below.

Every draw so far, for one category

"Average miss" is how far a test set's share of this category lies from the full dataset's share, on average, as a fraction of that share: 10% means a true share of 4.0% typically comes out as 3.6% or 4.4%. The formula is the one from class: Var = p(1−p)/n · (N−n)/(N−1).

The two splits in code

# random: the income mix of the test set is left to chance
train_set, test_set = train_test_split(
    housing_full, test_size=0.2, random_state=42)

# stratified: same share of every income category in both sets
strat_train_set, strat_test_set = train_test_split(
    housing_full, test_size=0.2,
    stratify=housing_full["income_cat"], random_state=42)

The page draws the same way: the random split picks rows uniformly from the whole dataset; the stratified split first fixes how many rows each income category gets (its share of the test set, rounded to whole rows, as scikit-learn does), then picks those rows at random within the category.

Reading the spread

• Random sampling is right on average, wrong in any single draw. The random histogram is centered on the true share, but each test set lands somewhere in that cloud. You only ever get one.
• random_state=42 does not fix this. It picks one draw from the cloud and makes it repeatable. A repeatable unlucky test set is still unlucky.
• Small datasets and rare categories suffer most. Switch to 200 districts, or follow category 1 instead of 3: the cloud gets much wider relative to the true share.
• Stratifying removes the wobble in the mix, not the randomness. Which districts end up in the test set is still random; how many come from each income category is fixed. Any leftover difference is rounding to whole rows.
• Why income? Median income is the feature most strongly correlated with house value. A test set with the wrong income mix gives a misleading error estimate.