15 Milestone revision and troubleshooting
This release reaches one complete supervised regression workflow. The long-term contents cover general machine learning and deep learning, including regularized models, classification, ensembles, unsupervised learning, and applications. Those later algorithms are not implemented in the current release. Use this chapter to choose a review path and diagnose common errors in the examples you have studied.
15.1 Reconstruct the complete workflow
- Define the task. State the observation unit, target, available inputs, units, and intended deployment population. Revisit Chapter 1 and Chapter 11.
- Represent data correctly. Keep rows aligned, inspect missing values and joins, and distinguish a vector from a matrix. Revisit Chapter 2, Chapter 3, and Chapter 4.
- Specify valid evaluation. Reserve the test set, choose a baseline and metrics, and respect groups/time. Revisit Chapter 12.
- Fit training-only transformations. Apply the resulting transformation unchanged to validation and test data. Use pipelines inside cross-validation; revisit Chapter 13.
- Choose a model and objective. For continuous predictions, derive what squared loss asks a model to estimate. Revisit Chapter 9, Chapter 10, and Chapter 14.
- Verify numerical behavior. Check gradient signs, shapes, loss normalization, convergence, and agreement with a numerical reference.
- Interpret evidence. Compare held-out error with a baseline, distinguish sample results from population claims, and state uncertainty and limitations. Revisit Chapter 7 and Chapter 8.
15.2 Symptoms and useful checks
| Symptom | What to check first | Why it matters |
|---|---|---|
| Subtraction produces a square error matrix | Are targets shaped \((n,1)\) while predictions are \((n,)\)? | Broadcasting can create \((n,n)\) rather than rowwise errors. |
| A mean changes after joining tables | Are join keys duplicated and were row counts checked? | A many-to-many join can multiply observations. |
| Training loss grows or becomes nonfinite | Learning rate, feature scales, gradient sign, and arithmetic overflow. | A numerical failure is different from model underfitting. |
| Training error is small but validation error is large | Model flexibility, sampling differences, leakage, and data quality. | Memorizing development details need not generalize. |
| Scratch and library coefficients differ | Intercept convention, feature order, scaling, objective, convergence, and rank. | Different parameterizations can also represent the same predictions. |
| Coefficients are large and unstable | Near-dependent columns and unit changes. | Individual effects may be poorly identifiable. |
| Excellent random-row scores on repeated participants | Do participants or overlapping windows cross partitions? | The evaluation may measure familiarity with people rather than unseen-person performance. |
| Negative test \(R^2\) | Compare squared error with the evaluation-mean denominator; inspect shift and sample size. | Negative values are possible and do not alone diagnose optimizer failure. |
| Validation improves after many searches | How many choices have used this set? | A validation set can itself be overfit. |
15.3 Equations to rebuild from understanding
For \(n\) rows and \(p\) features, \(X\in\mathbb R^{n\times p}\). With design matrix \(A=[\mathbf1\ X]\) and \(\boldsymbol\theta=(b,\mathbf w^T)^T\),
\[ \hat{\mathbf y}=A\boldsymbol\theta,\qquad J(\boldsymbol\theta)=\frac1{2n}\|A\boldsymbol\theta-\mathbf y\|_2^2,\qquad \nabla J=\frac1n A^T(A\boldsymbol\theta-\mathbf y). \]
Explain each shape before multiplying. Derive the gradient by the chain rule. Setting it to zero gives the normal equations; a unique coefficient solution requires independent design columns. A least-squares solver is preferable to implementing an explicit inverse. A gradient method additionally needs a suitable learning rate and convergence check.
For evaluation targets \(y_i\) and predictions \(\hat y_i\), RMSE is \(\sqrt{\frac1n\sum_i(y_i-\hat y_i)^2}\) in target units. MSE has squared target units. These are not the half-MSE objective \(J\) without the corresponding factor. See Chapter 14 for the hand example, derivation, and measured experiments.
15.4 Questions for a final self-check
- Can you explain why a feature available only after the outcome is leakage?
- Can you translate a matrix product into individual row calculations?
- Can you distinguish a probability, a sample proportion, an estimator, and one estimate?
- Can you derive a loss gradient rather than recall its shape?
- Can you fit preprocessing inside each training fold and explain what validation scores select?
- Can you say why a polynomial model remains linear in its coefficients?
- Can you explain why implementation agreement is a correctness check rather than independent evidence of generalization?
Use the worked answers in the relevant chapter when an explanation is incomplete. The next planned reading sequence is Ridge Regression → Lasso and Elastic Net → Logistic Regression, followed by other supervised families, reliable model development, and unsupervised learning. These chapters will reuse the workflow established here while introducing different assumptions and tradeoffs. The contents page also provides a complete classical-ML route and a continuation into deep learning.