As the tuning parameter λ increases in ridge regression, what happens to the squared bias?

In ridge regression, the tuning parameter λ plays a crucial role in controlling the extent of regularization applied to the model. As λ increases, the regularization effect strengthens, which influences the estimated coefficients. Specifically, as λ rises, the coefficients are shrunk towards zero. This shrinking leads to an increase in the squared bias of the model.

The relationship can be understood through the bias-variance tradeoff in statistical learning. When λ is small, the model can fit the training data more closely, resulting in lower bias but potentially higher variance. However, as we increase λ, the model becomes more biased because it is effectively simplifying the model by penalizing the size of the coefficients more aggressively. This simplification may lead to a poorer fit of the training data but can improve the model's generalization performance on unseen data.

Therefore, with a larger λ, the penalties on the coefficients lead to greater bias in the predictions, thus increasing the squared bias. This insight into ridge regression highlights the delicate balance between bias and variance that practitioners must manage when tuning hyperparameters.