Machine Learning — Interactive notes

Machine Learning notes

The 21 lessons of the Machine Learning course of the University of Pisa as pages to explore: formulas explained symbol by symbol, 188 figures to manipulate, a glossary and exam questions with answer outlines.

All lessons

  1. Introduction to Machine Learning — What ML is, why it is a necessity, how the course works and the maths refresher that will be needed throughout.
  2. ML in the Master’s curricula — Where the course sits in the Master’s degree in Computer Science and why to pair it with Computational Mathematics.
  3. Fundamental concepts of ML — Data, task, model, algorithm, validation: the ingredients of an ML system, the inductive bias and loss functions.
  4. Generalization and validation — Underfitting and overfitting on polynomial fitting, the VC-bound, hold-out and K-fold, confusion matrix and ROC curve.
  5. Linear models and K-nearest neighbors — Linear regression and classification with LMS, normal equations and gradient descent, regularization; then K-NN, the Bayes classifier and the curse of dimensionality.
  6. Neural networks (part 1) — from the neuron to the MLP — From the biological neuron to the Perceptron and its convergence theorem, sigmoids, the Multi-Layer Perceptron as an adaptive basis expansion and universal approximator.
  7. Notes on backpropagation — The full derivation of backpropagation: the delta of each unit, the backward pass from the output to the hidden layers, the training loop and its linear cost.
  8. Neural networks (part 2) — training in practice — How an MLP is actually trained: initialization, on-line/batch/mini-batch, learning rate and momentum, early stopping and weight decay, Cascade Correlation, inputs and outputs, the MONK benchmark.
  9. Validation (part 1) — model selection and assessment — Why estimates go wrong: model selection versus model assessment, the random-target counterexample, grid and random search, K-fold CV, sampling and error measures.
  10. Validation (part 2) — formal schemes — Risk and empirical risk, then the rigorous schemes: model selection and model assessment with hold-out and K-fold CV, and how to combine them (TR–VL–TS, CV + test, double CV).
  11. Validation (part 3) — typical mistakes and FAQ — The most frequent mistakes: fixed epochs, early stopping inside CV, random initializations, sequential selection, the test set used to redesign, overfitted CV; which CV to choose, with examples.
  12. Statistical Learning Theory and VC-dimension — Shattering and VC-dimension (lines in the plane have 3), VC-dim and number of parameters, the VC-bound on the risk and Structural Risk Minimization on nested structures.
  13. Support Vector Machines — Maximum margin and support vectors, the quadratic problem (primal and dual), soft margin and slack variables, kernels and the kernel trick, SVMs for regression with the ε-insensitive loss.
  14. SVMs and kernels — practical aspects and a critical view — Strengths and weaknesses of SVMs, the results on the two-class problem, the critical role of C and of the kernel parameters, the myths to debunk and kernel methods as similarity measures.
  15. Bias-variance and ensembles — The expected error over training sets decomposed into variance, bias² and noise, the role of λ in the trade-off, then ensembles: committees, bagging and boosting, and a note on feature selection.
  16. Convolutional neural networks (CNN) — Local connections, shared weights, pooling and many layers: 1D and 2D convolution, the receptive field that grows with depth, LeCun’s networks, MNIST, AlexNet and ImageNet.
  17. Deep learning — Why many layers: hierarchical abstraction, no-flattening and compositionality; representation learning, autoencoders and transfer learning; distributed representations; the techniques: ReLU, clipping, batch normalization, dropout.
  18. Randomized neural networks — Randomness as a resource: Random Forests, hidden layers with random weights that are never trained, a linear readout in one step, Cover’s theorem; pros and cons of random features.
  19. Unsupervised learning — K-means and SOM — Clustering as vector quantization: Voronoi cells, quantization error, on-line and batch K-means; then Self-Organizing Maps, which preserve topology and make it possible to visualize the data.
  20. Recurrent neural networks (RNN) — Sequences and transductions, the finite memory of IDNNs and the state of recurrent units, the Simple RNN, unfolding and backpropagation through time, then transformers, Echo State Networks and recursive networks.
  21. Learning on structured data and graphs — From vectors to graphs: transductions on graphs, message passing and Deep Graph Networks (GCN, NN4G, GNN, GraphESN), the problems of depth (over-smoothing, over-squashing, heterophily) and kernels for structures.

Prerequisites · Glossary · Italiano

Notes by Fabio Piscitelli — Machine Learning (654AA), Prof. Alessio Micheli, University of Pisa, a.y. 2026/27.