Lesson 11 of 21 · Part III, Validation and theory

Validation (part 3) — typical mistakes and FAQ

The most frequent mistakes: fixed epochs, early stopping inside CV, random initializations, sequential selection, the test set used to redesign, overfitted CV; which CV to choose, with examples.

3 figures to manipulate.

In this lesson

• Stopping the training
• Random initialization and model selection
• Sequential selection of the hyperparameters
• What to do after selection and assessment?
• Cross-validation can overfit
• Which cross-validation?
• Examples of model selection
• Practical conclusions

Other lessons

Validation (part 2) — formal schemes · Statistical Learning Theory and VC-dimension · All lessons

Notes by Fabio Piscitelli — Machine Learning (654AA), Prof. Alessio Micheli, University of Pisa, a.y. 2026/27.