A4.2 · Book pp. 215–222

Data preprocessing

HL only This whole section is for Higher Level students.

Most of the work in machine learning happens before training starts. A simple model on clean data will usually beat a clever model on messy data.

What you need to know

A4.2.1 Describe the significance of data cleaning.
A4.2.2 Describe the role of feature selection.
A4.2.3 Describe the importance of dimensionality reduction.

For the exam, know the effect of each step, including its risks. Filling in missing data can add bias, for example, and deleting it can lose useful information.

Next: data cleaning →

A4.2.1 Data cleaning

Outliers	Detect with Z-scores or the interquartile range, then cap, transform or remove them
Duplicates	Remove them so that over-represented values do not bias the model. Hashes can compare long records
Incorrect data	Use validation rules on ranges, and check anomalies by hand
Irrelevant data	Drop inputs with no link to the outcome
Bad formatting	Use one date style, one precision, and one image size and orientation
Missing data	Fill with the mean or mode, k-nearest neighbours or regression, or delete the records
Scaling	Normalize or standardize, so that large-valued features do not dominate

import numpy as np

data = np.array([12, 14, 15, 15, 16, 18, 19, 21, 95])

z = (data - np.mean(data)) / np.std(data)
print(data[np.abs(z) > 2])

q1, q3 = np.percentile(data, [25, 75])
cut = 1.5 * (q3 - q1)
print(data[(data < q1 - cut) | (data > q3 + cut)])

Normalization and standardization

normalized = (x − min) ÷ (max − min) · standardized = (x − mean) ÷ std

import numpy as np

data = np.array([10, 20, 30, 40, 50])
print((data - data.min()) / (data.max() - data.min()))   # 0 to 1
print((data - np.mean(data)) / np.std(data))             # mean 0, std 1

✏️ Book corrections

The book links standardization to a “Gaussian distribution”. Standardizing sets the mean to 0 and the standard deviation to 1, but it does not change the shape of the data. Skewed data stays skewed.
It says spurious correlations are correlations that “are, in fact, not”. They really are correlations. What is missing is any causal link, which is the lesson here: correlation does not imply causation.

Next: feature selection →

A4.2.2 Feature selection

A feature is an input variable. Keeping only the relevant ones gives a more general model, which handles unseen data better. Good features often matter more than the choice of model.

Method	How it works	Strength	Weakness
Filter	A statistic, such as correlation, ranks each feature	Fast, and works with any model	Misses interaction between features
Wrapper	Trains on different subsets of features and compares	Considers interaction, tailored to the model	Slow, with a risk of overfitting
Embedded	Selection happens during model training	Efficient, with no separate step	Depends on the model

The common filter is Pearson’s r, from −1 to +1. Near 0 means there is little linear relationship with the outcome.

✏️ Book correction

The book says “records” with certain r values can be flagged for deletion. Correlation scores features (columns), so it is features that get removed, not records.

Discussion: are features neutral?

In a credit model, including a postcode can bring in bias linked to where people live. Choosing features reflects the values of the people who build the model.

Next: dimensionality reduction →

A4.2.3 Dimensionality reduction

Each feature adds a dimension. Too many dimensions for the amount of data leads to the curse of dimensionality. The data becomes sparse, so patterns are hard to find.

Why too many dimensions cause harm

Overfitting: the model memorizes details, including noise.
Distances lose meaning: all points become similarly far apart.
Visualization: humans struggle beyond three dimensions.
Cost: more samples need more time and memory, which excludes limited hardware.

Reducing dimensions

You can remove features by hand, or use statistical tools. The maths is beyond the course, but you may use these in your IA.

PCA (principal component analysis) ignores labels. It is good for compression, visualization and speed.
LDA (linear discriminant analysis) uses labels to separate classes, so it suits classification.

from sklearn.datasets import make_classification
from sklearn.decomposition import PCA
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA

x, y = make_classification(n_samples=1000, n_features=20, n_informative=10,
                           n_clusters_per_class=1, n_classes=3)
print(PCA(n_components=2).fit_transform(x).shape)        # (1000, 2)
print(LDA(n_components=2).fit_transform(x, y).shape)     # (1000, 2)

⚠️ Common mistake

Reducing dimensions does not always improve a model. It can throw away important information, so test the result.

Next: review →

Quick quiz

1. Why does poor data give a poor model?

2. Which rule flags a value more than 1.5 × IQR beyond the quartiles?

3. Normalization usually…

4. Standardization gives data…

5. Which feature selection method is usually cheapest?

6. Which considers feature interaction by training on subsets?

7. With a fixed number of records, adding features makes data…

8. Overfitting means a model…

9. Which technique uses class labels to reduce dimensions?

Exam-style questions

These are written for this site in the style of Paper 1. Answer on paper first, then open the mark points.

Describe one common issue in customer survey data that requires cleaning. [2]
Mark points
- Missing answers where respondents skipped questions [1].
- These could bias the model, so they must be filled in or removed [1].
Describe how feature selection affects a model’s performance. [3]
Mark points
- It removes irrelevant or redundant features [1].
- This lowers the risk of overfitting, so the model generalizes better [1].
- Training is also faster, with less data to process [1].
Outline the role of dimensionality reduction for high-dimensional data. [3]
Mark points
- It reduces the number of input variables [1].
- This reduces sparsity, making patterns easier to find [1].
- It also cuts memory and processing, but risks losing some information [1].
Explain why features should be scaled before training. [2]
Mark points
- Features can have very different ranges [1].
- Without scaling, large-valued features can dominate distance-based or gradient-based methods [1].
A school district plans to feed raw marks for every test question into a model. Discuss the implications. [4]
Mark points
- Thousands of features create high dimensionality [1].
- Too few students for that many features leads to overfitting [1].
- Training needs more time and memory [1].
- It would be better to select or combine features, such as using topic totals [1].
Compare filter and wrapper methods. [3]
Mark points
- Filter methods use statistics without training a model [1].
- Wrapper methods train on feature subsets, so they see interaction between features [1].
- Filter is cheap, while wrapper is expensive and may overfit [1].

Mean and standard deviation
Outliers by Z-score
IQR fences
Outliers by IQR

Grid cells, with range 0 to 4 per feature
Data points per cell
Points needed to stay at 2.5 per cell