3

I am running a regression analysis on a 7000 row dataset with a train/test split of 70%/30%. I am using one variable X to predict a variable Y.

  • X ranges between 300 and 810 (mean 712).
  • Y is an integer (number of occurrences) ranging between 0 and 20 (mean 0.2).

Without standardisation or normalising X, I receive:

Train score:  0.082
Test score:  0.077

However upon normalising (X = (X-X.min())/(X.max()-X.min())), I receive:

Train score:  0.0000
Test score:  -0.0001

Is there something incorrect about normalising for a Lasso regression? The same applies to standardising the data. Would anyone be able to advise me on the best course of action?

Romain Reboulleau
  • 1,387
  • 9
  • 26
atom
  • 31
  • 1

1 Answers1

4

Standardizing/normalizing is generally the right thing to do, but it will make little/no difference with just one independent variable if you also adjust the regularization strength.

With more than one independent variable, standardizing assures that the lasso's penalty applies more equally to all the variables. Without standardizing, a large-scale variable will be penalized less than small-scale ones (because the coefficients will already tend to be small).

In your case, the original model needs a fairly small coefficient on X because its scale is so large, and so you don't receive much penalty from the lasso. After you standardize, now the model wants a larger coefficient on X, but that means the lasso penalty makes more of a contribution. That you get an $R^2$ of zero suggests that the lasso penalty is large enough now to push the coefficient to zero, so that the model is just a horizontal line. If you reduce the regularization strength a bit, you should be able to recover the old model (exactly maybe, with the right adjustment? If I get some spare time I'll look into this).

Example Colab notebook

Ben Reiniger
  • 12,855
  • 3
  • 20
  • 63