Does overfitting depend only on validation loss or both training and validation loss?

Question

There are several scenarios that can occur while training and validating:

1. Both training loss and validation loss are decreasing, with the training loss lower than the validation loss.
2. Both training loss and validation loss are decreasing, with the training loss higher than the validation loss.
3. Training loss decreasing, but validation loss is increasing.

I am aware that overfitting occurs in scenario 3, but does overfitting occur in scenario 1? If so, does this mean that overfitting only occurs when either scenario 1 or scenario 3 occur? Otherwise, if overfitting only occurs in scenario 3, does this mean that overfitting only occurs when validation loss is increasing?

Answer 1

In my opinion, only case 3 should be considered overfitting. As @stans has mentioned, there is not a very rigorous definition of overfitting so other people might think differently.

I wouldn't say the point where the validation loss stops decreasing is where bias and variance are minimized since there is a trade-off between bias and variance:

• A constant model will have very low variance, but very high bias.
• An overfitting model will have very low bias, but very high variance.

The point where the validation loss starts increasing can be considered optimal in terms of the sum of squared bias and variance, that is, an optimum of the generalization error.