A comprehensive discussion of generalization and regularization

Model generalization ability is an extremely important dimension when designing and evaluating a machine learning or deep learning method, so I would like to comprehensively discuss generalization/regularization in machine learning and deep learning through a series of articles. (regularization), on the one hand, understand the generalization problem of the model from multiple perspectives, on the other hand, explain many methods in machine learning and deep learning from the perspective of generalization (norm penalty, weight decay, dropout, parameter sharing, etc.).
Much of the content here is based on Chapter 7 “Regularization for Deep Learning” of Ian Goodfellow’s book Deep Learning (wall crack recommendation!), combined with some other articles and my own experience.
The main contents of this series are:
Introduction
1. Definition: Regularization is a general term for all methods used to reduce the generalization error of an algorithm.
2. There are various means of regularization, reducing variance at the expense of increasing bias.
3. The best deep learning model, in reality, is often [complex model (large and deep)] + [effective regularization].
Table of Contents: Regularizer
1. Norm penalty: L1/L2 regularization is commonly used. The theoretical mechanism is different. According to the importance of the feature direction, L1 “cuts” the parameter component with a threshold, and L2 “shrinks” the parameter component; the implementation methods in deep learning include [weight decay] and There are two kinds of [hard…