13 Data preparation and pipelines

A model expects inputs with a consistent meaning. If training cadence is standardized but new cadence remains in steps per minute, the model receives a different numerical representation. If missing readings are filled using information from the final test set, the evaluation has already influenced training. Good preparation therefore has two jobs: create useful inputs and preserve the boundary between learning and assessment.

13.1 Learning objectives and prerequisites

After this chapter, you should be able to distinguish fixed transformations from learned preprocessing, impute numerical and categorical values, standardize numerical features using training statistics, encode nominal categories, and assemble these operations into a pipeline fitted inside every cross-validation fold. You should also recognize what a pipeline cannot fix: unavailable features, invalid group boundaries, incorrect labels, or a change in the population.

Prerequisites are array shapes, pandas tables, and the training, validation, and leakage concepts in Chapter 12. We use a regularized regression model as a final pipeline step. You do not need its full derivation to follow the data preparation; the important contract is that fit learns from training examples and predict applies the learned procedure to new inputs.

13.2 Preparation is part of the fitted model

Let \(\mathbf{X}_{\mathrm{train}}\in\mathbb{R}^{n\times p}\) contain \(n\) training observations and \(p\) numerical features. A learned transformation has parameters \(\boldsymbol{\theta}\) determined using training inputs, such as one mean per column. Write its transformed matrix as

\[ \mathbf{Z}_{\mathrm{train}}=T_{\boldsymbol{\theta}}(\mathbf{X}_{\mathrm{train}}). \]

The predictor is fitted using \(\mathbf{Z}_{\mathrm{train}}\). For new observations \(\mathbf{X}_{\mathrm{new}}\), use

\[ \hat{\mathbf{y}}_{\mathrm{new}}= \hat{f}\!\left(T_{\boldsymbol{\theta}}(\mathbf{X}_{\mathrm{new}})\right), \]

with the same learned \(\boldsymbol{\theta}\). Refitting a scaler separately on a new batch changes the coordinate system. Even if the resulting numbers look similarly distributed, the fitted predictor’s coefficients refer to the original training transformation.

This distinguishes two operations:

Operation	What it does	Where it belongs
`fit`	Learns preparation or model parameters	Training portion of the current fit
`transform`	Applies an already learned representation	Training, validation, test, or new data
`fit_transform`	Learns, then applies, a transformation	Training portion only
`predict`	Applies the fitted preparation and predictor	New inputs after fitting

Some transformations have no learned numerical parameters. Converting milliseconds to seconds with a fixed factor does not estimate a statistic from the sample. A predetermined log1p transformation also has no learned centering parameters. Nevertheless, choosing whether to use a transformation after comparing validation scores is a development decision. If fitted objects or selected rules depend on a dataset, their proper place is inside the development procedure.

13.3 Missing measurements: preserve their meaning

A missing value means that a measurement is unavailable under the collection or processing procedure. It does not mean zero. Zero cadence is a meaningful value; replacing a missing cadence by zero can falsely describe a stationary person. Before selecting an imputer, investigate why values are missing and whether the same pattern can occur when predicting.

Imputation supplies a replacement for a missing value. Simple choices include a training-column mean, median, or a documented constant. A categorical feature can use a training most-frequent category or a distinct missing category. These are assumptions about how to represent incomplete records, not recovery of the unknown measurement.

A numerical median is resistant to a few extreme values, but a single repeated replacement can distort a distribution or a relationship. If missingness itself is informative, a binary missingness indicator can accompany imputation. Its availability and meaning must also be valid at prediction time. An indicator may capture a temporary recording fault that changes after deployment.

Dropping all incomplete rows is sometimes defensible, but it changes which observations receive predictions and can discard a systematic subset. A system expected to serve incomplete inputs needs an explicit strategy for them. Missing targets present another problem: do not casually fill outcomes and present the artificial values as observed labels for supervised evaluation.

For feature \(j\), median imputation learns \(a_j\) from the observed training values and applies

\[ x_{ij}^{\mathrm{filled}}= \begin{cases} a_j, & x_{ij}\text{ is missing},\\ x_{ij}, & x_{ij}\text{ is observed}. \end{cases} \]

Here \(i\) indexes rows and \(j\) columns; \(a_j\) has the same units as feature \(j\). If a whole training column is missing, its median is undefined. Decide whether to exclude that feature, supply a justified fixed value, or redesign the collection. The example below explicitly rejects that case rather than silently constructing a value.

The library SimpleImputer API provides strategies such as median and most frequent, and options concerning missingness indicators and completely empty features. Specify the choices that matter to the lesson instead of relying on unexamined defaults.

13.4 Scaling: change coordinates, preserve the training reference

Standardization of a filled training column computes

\[ \mu_j=\frac{1}{n}\sum_{i=1}^{n}x_{ij}^{\mathrm{filled}}, \qquad \sigma_j=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(x_{ij}^{\mathrm{filled}}-\mu_j)^2}, \qquad z_{ij}=\frac{x_{ij}^{\mathrm{filled}}-\mu_j}{\sigma_j}. \]

\(\mu_j\) is a training-column mean and \(\sigma_j\) is its population-style standard deviation, using denominator \(n\) for this transformation. This denominator is a scaling convention, distinct from the sample standard deviation with denominator \(n-1\) used for descriptive estimation. If the feature is in steps/minute, its mean and scale share those units and the standardized value is dimensionless.

For a constant column, the numerical standard deviation is zero. Divide by one as an explicit convention: training values then center to zero without division by zero. Standardization does not make a distribution Gaussian, remove outliers, or eliminate dependence among columns. Training values need not lie between zero and one, and a new value can be far outside the range seen in training. The StandardScaler documentation specifies its per-feature centering and scaling behavior and sensitivity to outliers.

Scaling is especially consequential for distance calculations, gradient-based fitting, and penalties on coefficient magnitudes. A numerical feature measured in millimeters can dominate a raw distance beside the same kind of feature measured in meters. Some methods, such as ordinary threshold-based trees, are much less sensitive to positive rescaling. Choose the operation for the model and task rather than standardizing every column mechanically.

Alternative transformations include min-max scaling using training minima and maxima, or centering and scaling by a training median and interquartile range. Min-max scaling does not guarantee that future values fall within the training range. Robust statistics reduce some sensitivity to extremes but do not establish that extreme cases are invalid. Record which rule was fitted and apply it consistently.

13.5 From scratch: make the training statistics visible

The following two-feature example teaches the numeric preparation logic independently of a predictive model. The first column is cadence in steps/minute; the second is a dimensionless movement summary. Arrays have shape (rows, features). Separate functions learn statistics and apply them. These educational functions accept only numerical arrays with ordinary finite values or np.nan, where np.nan represents a missing value.

Compute training medians from available readings.
Fill missing training readings with those medians.
Compute training means and scales after filling.
Apply the same medians, means, and scales to a new array.
Compare the result with library transformers on the identical arrays.

13.5.1 Step 1: Learn only from training rows

# Step 1: Implement numeric imputation and scaling with explicit column loops.
import numpy as np


def learn_numeric_preparation(training_features):
    """Learn medians, means, and scales from a numeric (n, p) training array.

    NaN marks missing readings. Every feature must have an observed value.
    Return three length-p arrays; constant filled columns receive scale one.
    Means and scales are computed after training-median imputation.
    """
    # Copy values so filling does not overwrite the caller's original array.
    filled_training_features = np.array(training_features, dtype=float, copy=True)
    # Store the number of separate feature columns to process.
    number_of_features = filled_training_features.shape[1]
    # Allocate one learned median for every column.
    training_medians = np.empty(number_of_features)
    # Allocate the post-imputation mean for every column.
    training_means = np.empty(number_of_features)
    # Allocate the post-imputation standardization scale for every column.
    training_scales = np.empty(number_of_features)

    for feature_number in range(number_of_features):
        # Select this feature's readings across all training rows.
        feature_values = filled_training_features[:, feature_number]
        # Mark available readings rather than treating missing values as zero.
        observed_rows = ~np.isnan(feature_values)
        if not observed_rows.any():
            raise ValueError("Every training feature needs an observed reading.")
        # Estimate the replacement using only this training column.
        training_medians[feature_number] = np.median(feature_values[observed_rows])
        # Fill this column before learning its mean and scale.
        feature_values[~observed_rows] = training_medians[feature_number]
        training_means[feature_number] = np.mean(feature_values)
        training_scales[feature_number] = np.std(feature_values, ddof=0)
        if training_scales[feature_number] == 0.0:
            training_scales[feature_number] = 1.0

    return training_medians, training_means, training_scales


def apply_numeric_preparation(features, medians, means, scales):
    """Apply learned length-p statistics to a numeric (m, p) feature array.

    Return a new (m, p) standardized array, leaving the input unchanged.
    Statistics must come from the associated training fit, not these rows.
    Column order must match training; NaN represents a missing reading.
    """
    # Preserve the supplied array while creating its prepared representation.
    prepared_features = np.array(features, dtype=float, copy=True)
    for feature_number in range(prepared_features.shape[1]):
        # Select readings from one column without mixing measurement units.
        feature_values = prepared_features[:, feature_number]
        # Locate missing readings in the array currently being transformed.
        missing_rows = np.isnan(feature_values)
        # Reuse the training replacement, even if this is a new batch.
        feature_values[missing_rows] = medians[feature_number]
        # Center and scale using the original training coordinate system.
        prepared_features[:, feature_number] = (
            feature_values - means[feature_number]
        ) / scales[feature_number]
    return prepared_features


# Define small input matrices with different measurement units by column.
# Four training rows provide the statistics that may be learned.
small_training_features = np.array([
    [100.0, 0.5], [110.0, np.nan], [np.nan, 0.9], [130.0, 0.7]
])
# Two new rows must use the stored training statistics without refitting.
small_new_features = np.array([[115.0, 0.8], [np.nan, 0.6]])
# Learn the three arrays exclusively from the four training rows.
learned_medians, learned_means, learned_scales = learn_numeric_preparation(
    small_training_features
)
# Apply those arrays to the same training rows for a centered representation.
prepared_training_features = apply_numeric_preparation(
    small_training_features, learned_medians, learned_means, learned_scales
)
# Apply precisely the same transformation to the new observations.
prepared_new_features = apply_numeric_preparation(
    small_new_features, learned_medians, learned_means, learned_scales
)
print("Training medians:", learned_medians)
print("Filled-training means:", learned_means)
print("Training scales:", learned_scales.round(4))
print("Prepared new rows:")
print(prepared_new_features.round(4))
assert np.allclose(learned_medians, [110.0, 0.7])
assert np.allclose(prepared_training_features.mean(axis=0), 0.0)
assert np.isfinite(prepared_new_features).all()

Training medians: [110.    0.7]
Filled-training means: [112.5   0.7]
Training scales: [10.8972  0.1414]
Prepared new rows:
[[ 0.2294  0.7071]
 [-0.2294 -0.7071]]

The first filled training column is \([100,110,110,130]\), whose mean is 112.5. Therefore a new cadence of 115 is centered by subtracting 112.5, not by subtracting the new batch’s mean. New rows generally do not have mean zero after this transformation; that is expected and preserves their relationship to training.

The column slices in the learning function refer to the copied array, so filling a slice updates that copy. The original input retains its missing values. The functions deliberately omit a production transformer’s broader validation, feature-name tracking, sparse-array support, and fitted-state management. Their purpose is to expose the equations, not replace a tested library implementation.

13.5.2 Step 2: Check the same numerical operations with the library

# Step 2: Check explicit calculations against established numeric transformers.
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

# Apply median filling first, then scale the filled training columns.
numeric_comparison_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median", keep_empty_features=True)),
    ("scaler", StandardScaler(with_mean=True, with_std=True)),
])
numeric_comparison_pipeline.fit(small_training_features)
# Transform new rows without relearning either preparation step.
library_prepared_new_features = numeric_comparison_pipeline.transform(small_new_features)
assert np.allclose(prepared_new_features, library_prepared_new_features)
print("Manual and library transformations agree:", True)

Manual and library transformations agree: True

13.6 Categorical features need a representation

A nominal category is a name without a numerical ordering: sensor placement, material type, or device family. Encoding waist=0, ankle=1, and wrist=2 as one scalar can imply an artificial order and distance. A linear model would interpret the change from 0 to 1 through the same coefficient as the change from 1 to 2. That is an assumption about the feature, not a neutral formatting choice.

One-hot encoding creates an indicator column for each learned category. With training categories ankle and waist, an ankle record can become \([1,0]\) and a waist record \([0,1]\), in the encoder’s documented column order. There is no implied ordering, but there are more columns. With \(c\) categories, retaining all indicators creates \(c\) columns; large-cardinality identifiers can create unwieldy representations and opportunities to memorize records.

With an intercept and unregularized linear coefficients, all category indicators are linearly dependent with the intercept. Dropping a reference category is one way to get an identifiable coefficient representation. Predictions can still be well defined under suitable numerical solvers with redundant columns; individual coefficient interpretation then requires care. The regularized example below retains all indicators and uses a positive ridge penalty. It is intended for predictions, not causal interpretation of category coefficients.

An unseen category needs an explicit policy. handle_unknown="ignore" produces zeros in that feature’s learned category columns for an unrecognized value. That preserves input shape but does not demonstrate accurate predictions for the new category. Alternatively, a system may reject unknown values or use a predefined other category, depending on its task. The OneHotEncoder API specifies handle_unknown, category handling, and the sparse_output option. Dense output is chosen explicitly below because the dataset is tiny and easy to inspect.

An ordinal category really does have an order, such as a documented severity scale. Coding it numerically still imposes assumptions about spacing if the model uses arithmetic distances. Keep the measurement interpretation explicit. Do not give a nominal field an order merely because a library accepts integer arrays.

13.7 Feature transformations: decide what relation to represent

A fixed logarithm can make multiplicative relationships easier to represent, but its domain matters: \(\log(x)\) requires positive \(x\). log1p(x) computes \(\log(1+x)\) accurately near zero and is defined for \(x>-1\); it is often convenient for nonnegative counts. It does not automatically repair a bad distribution or make an evaluation valid. If the target is transformed, predictions must be interpreted back on the original scale and evaluated there when original-scale errors matter. Exponentiating a predicted log mean does not generally recover an original-scale conditional mean.

Polynomial expansion creates features such as \(x\), \(x^2\), and \(x^3\). It changes model flexibility without requiring the final predictor to be nonlinear in its coefficients. Powers can amplify large values and increase numerical sensitivity. For multiple inputs, interactions such as \(x_1x_2\) can also expand feature count rapidly. Tune the degree or interaction choices within development data as in Chapter 12.

Transformations that estimate parameters, including power transforms or dimensionality reduction, require training-only fitting. Even a fixed mathematical transformation can leak future information if the input feature was calculated from a future interval. Check where a measurement came from before considering how to rescale it.

13.8 A mixed-column pipeline fitted within training folds

Real tables often mix numeric and categorical fields. ColumnTransformer sends specified columns through separate preparation branches, then concatenates their output. Pipeline runs these preparation steps before the final estimator. When a complete pipeline is passed to cross-validation, every fold fits its own preprocessing using that fold’s training rows. The composition guide describes these APIs and how they support consistent transformation and model selection.

The complete example below simulates 180 independent people with one trial each. Cadence, a dimensionless acceleration summary, and nominal sensor placement are inputs. A target speed is generated from a stated artificial equation before some features are hidden. The numerical coefficients are chosen for teaching and provide no biomechanical evidence. The same source code runs without downloading data.

The experiment will:

Generate independent synthetic records and split off the final test.
Define numeric and categorical preparation branches.
Combine them with a ridge-regression predictor.
Select penalty strength using development folds only.
Inspect the fitted representation and evaluate the fixed pipeline.
Demonstrate the mechanical handling of an unseen category.

13.8.1 Step 3: Generate a table with mixed types and missing inputs

# Step 3: Generate independent synthetic rows before creating missing readings.
import pandas as pd
from sklearn.model_selection import train_test_split, KFold, GridSearchCV
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
from sklearn.linear_model import Ridge
from sklearn.metrics import mean_absolute_error, root_mean_squared_error

# Fix randomness so the same artificial table and missingness can be reproduced.
mixed_generator = np.random.default_rng(19)
# Use one trial per simulated person to keep this example's rows independent.
number_of_trials = 180
# Generate complete cadence values in steps/minute before hiding any readings.
complete_cadence = mixed_generator.normal(105.0, 10.0, number_of_trials)
# Generate a dimensionless movement summary available from the same trial.
complete_movement_summary = mixed_generator.uniform(0.2, 1.0, number_of_trials)
# Assign nominal sensor locations with no assumed numerical ordering.
complete_placement = mixed_generator.choice(["waist", "ankle"], number_of_trials)
# Assign a small artificial placement contribution for the target generator.
placement_contribution = np.zeros(number_of_trials)
placement_contribution[complete_placement == "ankle"] = 0.05
# Add independent target noise in m/s.
mixed_target_noise = mixed_generator.normal(0.0, 0.04, number_of_trials)
# Generate artificial reference speed in m/s from the complete input quantities.
mixed_targets = (
    0.3 + 0.006 * complete_cadence
    + 0.25 * complete_movement_summary + placement_contribution
    + mixed_target_noise
)
# Store named input columns so each branch receives the intended measurements.
mixed_features = pd.DataFrame({
    "cadence_steps_per_minute": complete_cadence,
    "movement_summary": complete_movement_summary,
    "sensor_placement": complete_placement,
})
# Predetermine separate missing rows for each input feature.
mixed_features.loc[np.arange(0, number_of_trials, 13), "cadence_steps_per_minute"] = np.nan
mixed_features.loc[np.arange(3, number_of_trials, 17), "movement_summary"] = np.nan
mixed_features.loc[np.arange(5, number_of_trials, 23), "sensor_placement"] = np.nan

# Reserve a final holdout before fitting any data-dependent preparation.
mixed_development_features, mixed_test_features, mixed_development_targets, mixed_test_targets = (
    train_test_split(mixed_features, mixed_targets, test_size=0.25, random_state=42)
)
print("Development shape:", mixed_development_features.shape)
print("Final-test shape:", mixed_test_features.shape)
print("Missing development inputs:")
print(mixed_development_features.isna().sum())

Development shape: (135, 3)
Final-test shape: (45, 3)
Missing development inputs:
cadence_steps_per_minute    10
movement_summary            10
sensor_placement             4
dtype: int64

If multiple trials came from each real person, replace this random row split and the shuffled folds below with participant-aware equivalents. Having a pipeline does not change the independence requirements established by the task.

13.8.2 Step 4: Build two branches and a final predictor

The numeric branch fills missing values, then standardizes. The categorical branch fills missing values with its training most-frequent category, then learns indicator columns. remainder="drop" explicitly excludes all unspecified columns; this is useful for preventing accidental inclusion of later-added IDs or target-derived metadata. It does not replace auditing the listed input columns.

# Step 4: Route different measurement types through appropriate preparation.
# List the two numerical inputs in the same order used for inspection later.
numeric_columns = ["cadence_steps_per_minute", "movement_summary"]
# Keep the nominal input separate from numerical scaling.
categorical_columns = ["sensor_placement"]

# Learn medians and scaling statistics on the rows used by each fit.
numeric_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="median", keep_empty_features=True)),
    ("scaler", StandardScaler(with_mean=True, with_std=True)),
])
# Learn category filling and one-hot columns on the same training rows.
categorical_pipeline = Pipeline([
    ("imputer", SimpleImputer(strategy="most_frequent", keep_empty_features=True)),
    ("encoder", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
])
# Concatenate the two prepared branches into one feature matrix.
column_preparation = ColumnTransformer(
    transformers=[
        ("numeric", numeric_pipeline, numeric_columns),
        ("categorical", categorical_pipeline, categorical_columns),
    ],
    remainder="drop",
)
# Keep all learned preparation and the predictor inside one fitted object.
# A positive ridge penalty stabilizes coefficients; its strength is selected next.
mixed_pipeline = Pipeline([
    ("preparation", column_preparation),
    ("regression", Ridge(alpha=1.0, fit_intercept=True)),
])

Calling mixed_pipeline.fit first fits each preparation branch, transforms its training inputs, concatenates the prepared columns, and fits the regressor. Calling predict transforms new rows using those stored branch objects and passes the resulting matrix to the fitted regressor. No manual concatenation or separate test scaler is necessary.

13.8.3 Step 5: Tune the complete pipeline within development folds

The double underscore in regression__alpha means “the alpha setting of the step named regression.” The same notation can target settings deeper in the preparation branches. Ridge penalizes squared coefficient magnitudes; alpha is its library penalty setting, with its own objective convention. It should not be equated numerically with a penalty under a differently normalized loss. Only its role as a tunable strength matters here.

# Step 5: Select penalty strength with all preparation repeated within each fold.
# Use four shuffled folds because these simulated observations are independent.
mixed_development_folds = KFold(n_splits=4, shuffle=True, random_state=42)
# Predetermine a small set of positive penalty strengths.
penalty_candidates = {"regression__alpha": [0.01, 1.0, 100.0]}
# Search the entire pipeline, not a matrix prepared before cross-validation.
mixed_search = GridSearchCV(
    mixed_pipeline, penalty_candidates, cv=mixed_development_folds,
    scoring="neg_root_mean_squared_error", refit=True,
    error_score="raise", n_jobs=1
)
mixed_search.fit(mixed_development_features, mixed_development_targets)
# Store the final pipeline refitted on all development rows after selection.
fitted_mixed_pipeline = mixed_search.best_estimator_
print("Selected penalty:", mixed_search.best_params_)
print("Selected mean CV RMSE (m/s):", round(-mixed_search.best_score_, 3))

Selected penalty: {'regression__alpha': 0.01}
Selected mean CV RMSE (m/s): 0.043

Within any one fold, medians, means, scales, most-frequent categories, one-hot categories, and coefficients all come from the training portion of that fold. Fitting column_preparation once on all development rows and cross-validating only the prepared matrix would allow validation-feature information into preparation. Although no target was used by the imputer or scaler, those validation inputs still influenced the representation. The complete pipeline prevents that particular leak.

13.8.4 Step 6: Inspect the fitted representation and evaluate once

named_steps retrieves a fitted pipeline component by its step name. A ColumnTransformer similarly exposes fitted branch objects through named_transformers_. These are inspection tools; do not refit these components on the test set. get_feature_names_out() helps connect output columns to the original measurement branches.

# Step 6: Inspect learned training statistics without modifying the fitted object.
# Retrieve the final development-fitted preparation component.
fitted_preparation = fitted_mixed_pipeline.named_steps["preparation"]
# Retrieve the numerical branch for checking its training medians.
fitted_numeric_branch = fitted_preparation.named_transformers_["numeric"]
# Read medians already learned during the final development refit.
fitted_numeric_medians = fitted_numeric_branch.named_steps["imputer"].statistics_
# Record the prepared column names to understand the expanded representation.
prepared_column_names = fitted_preparation.get_feature_names_out()
# Transform a small development sample only to inspect dimensions and values.
prepared_preview = fitted_preparation.transform(mixed_development_features.iloc[:3])
print("Numeric training medians:", fitted_numeric_medians.round(3))
print("Prepared column names:", prepared_column_names)
print("Prepared preview shape:", prepared_preview.shape)
assert prepared_preview.shape[1] == len(prepared_column_names)
assert np.isfinite(prepared_preview).all()

# Produce the final predictions after all model-selection choices are fixed.
mixed_test_predictions = fitted_mixed_pipeline.predict(mixed_test_features)
# Keep evaluation in m/s, even though the numeric inputs were standardized.
mixed_test_rmse = root_mean_squared_error(mixed_test_targets, mixed_test_predictions)
# Report a complementary absolute-error summary in the same target units.
mixed_test_mae = mean_absolute_error(mixed_test_targets, mixed_test_predictions)
print("Final-test RMSE (m/s):", round(mixed_test_rmse, 3))
print("Final-test MAE (m/s):", round(mixed_test_mae, 3))
assert mixed_test_predictions.shape == mixed_test_targets.shape
assert np.isfinite(mixed_test_predictions).all()

Numeric training medians: [106.131   0.586]
Prepared column names: ['numeric__cadence_steps_per_minute' 'numeric__movement_summary'
 'categorical__sensor_placement_ankle'
 'categorical__sensor_placement_waist']
Prepared preview shape: (3, 4)
Final-test RMSE (m/s): 0.052
Final-test MAE (m/s): 0.043

Standardizing inputs does not standardize the target. The model still predicts speed in m/s because those were the labels supplied during fitting. The final errors describe this synthetic holdout only. Missing inputs and target noise make perfect reconstruction unnecessary and generally unavailable; a small error here does not establish validity for another sensor placement or a real population.

13.8.5 Step 7: Understand an unknown category without claiming accuracy

This last call demonstrates an inference policy after evaluation. It has no measured reference target and is not used to choose a setting. The unknown category is mechanically accepted, but its prediction has no established accuracy. A real application may instead require such records to be flagged or rejected.

# Step 7: Check how the declared unknown-category policy behaves at inference.
# Supply a new input row with one missing numeric value and an unseen placement.
unseen_input = pd.DataFrame({
    "cadence_steps_per_minute": [110.0],
    "movement_summary": [np.nan],
    "sensor_placement": ["wrist"],
})
# Transform with the development-fitted objects, without fitting to this row.
unseen_prepared_input = fitted_preparation.transform(unseen_input)
# Generate a mechanical prediction; no accuracy is asserted for this new category.
unseen_prediction = fitted_mixed_pipeline.predict(unseen_input)
print("Prepared row for an unknown placement:", unseen_prepared_input.round(3))
print("Unvalidated prediction for that row (m/s):", unseen_prediction.round(3))
assert unseen_prepared_input.shape[1] == len(prepared_column_names)
assert np.isfinite(unseen_prediction).all()

Prepared row for an unknown placement: [[ 0.433 -0.038  0.     0.   ]]
Unvalidated prediction for that row (m/s): [1.125]

In the checked run, development CV selects alpha 0.01, with mean CV RMSE of 0.043 m/s. The final-test RMSE is 0.052 m/s and MAE is 0.043 m/s. The prepared matrix has four columns: two numeric columns and two placement indicators. These are observed outputs of this specified synthetic experiment, not promised values for different missingness patterns or real sensor data.

13.9 Practical limitations and costs

A pipeline enforces the sequence it was given; it cannot determine whether the sequence is scientifically appropriate. It does not remove a future-derived feature, discover an invalid clinical label, or stop related participants from crossing folds. It also cannot make unknown categories familiar or prevent a sensor calibration shift. Audit the task and collection process alongside the code.

Column names, numerical units, and category definitions are part of the model interface. A column named cadence can still contain steps/second when the fitted model expects steps/minute. Preserve schemas and unit checks when reusing the pipeline. Avoid silently coercing a malformed numeric string into a plausible-looking value without a recorded parsing policy.

For \(n\) rows and \(p\) numeric columns, applying imputation and scaling requires work proportional to \(np\) and stores roughly \(p\) statistics per stage. Fitting medians can require more work depending on the implementation used to select or sort values. One-hot output can have many columns: if nominal features have category counts \(c_1,\ldots,c_q\), retaining all categories adds \(\sum_{j=1}^{q}c_j\) indicator columns. Dense storage grows with rows times that expanded width; sparse storage can be preferable for large category sets. Our tiny dense matrix is a deliberate teaching choice.

Cross-validation multiplies preparation and model fitting cost by the number of folds and candidates. That repetition is necessary for honest validation of learned preparation. Caching can help expensive repeated steps in larger projects, but it does not justify fitting transforms outside training folds. Keep the first experiment small enough to inspect fully.

13.10 Exercises and worked answers

Calculate a new standardized value. Training values are 10, 12, and 14. Using the denominator-\(n\) scale, transform a new value of 15.

Answer: The mean is 12. The squared deviations sum to 8, so the scale is \(\sqrt{8/3}\approx1.633\). The new value becomes \((15-12)/1.633\approx1.837\). Do not center it using the new batch’s mean.
Spot preprocessing leakage. A scaler is fitted on all development inputs before five-fold CV, with no test data involved. Is CV evaluating an entirely training-only preparation procedure?

Answer: No. Each validation fold influenced the fitted scaler. Pass the complete preparation-and-model pipeline into CV so each fold learns its own statistics. The final test has remained separate, but the development CV calculation is still different from the intended training-only procedure.
Explain categories. Why can integer codes 0, 1, and 2 be inappropriate for nominal sensor placements?

Answer: Arithmetic models may impose an order and equal spacing between codes. One-hot indicators represent membership without that numerical ordering. If the feature is truly ordinal, document its order and consider whether equal spacing is justified.
Interpret unknown handling. An unseen wrist category yields all zeros in the placement indicators. Does that mean it behaves like an average known placement?

Answer: That is not established. The encoder’s output is a policy for maintaining the feature schema. The predictor’s numerical result follows from that representation, but its accuracy for unseen placements requires relevant evidence.
Distinguish fixed and learned operations. Which needs fitted statistics: dividing milliseconds by 1,000, centering by a dataset mean, or choosing a polynomial degree after validation?

Answer: Unit conversion uses a fixed rule. Centering learns a mean and must be fitted within training folds. Degree selection is a development decision evaluated using validation, even though generating polynomial powers has no estimated mean or median.
Handle an empty column. Every training cadence is missing, but test cadence is complete. Should test values supply its training median?

Answer: No. There is no observed training cadence from which to learn that statistic. Use a justified predetermined policy, exclude or redesign the feature, or collect suitable training measurements. Borrowing test values breaks the intended boundary.
Practical extension. Before a new run, increase the missing-cadence frequency from one row in 13 to one row in four. Keep the candidate grid and evaluation procedure fixed. What should be inspected?

Answer: Inspect development missingness, selected CV error, and the final prespecified evaluation, then compare the two constructed missingness scenarios. Higher missingness can remove useful information, but the measured effect depends on the generator, feature redundancy, and imputation. Do not assume an improvement or a fixed amount of degradation without running it.

13.11 Revision summary

Preparation belongs to the fitted prediction procedure. Learn imputations, scales, categories, and selectors from training rows; apply their stored rules to validation, test, and new data. Missing is not zero. Standardization uses training means and scales and does not force new batches to have mean zero. One-hot encoding avoids imposing an artificial order on nominal categories; unknown handling preserves shape without establishing accuracy. Use ColumnTransformer for different column types and Pipeline for sequential preparation and fitting. Pass the whole pipeline into CV, refit the selected procedure on development data, and assess the reserved test in the original target units. Keep units, timing, groups, and label validity under review even when the code runs successfully.