2

it is my first time doing something with financial data. I have a dataset with account numbers and some other information about each client (some clients span more than one row since we have info for each month in a different row). I managed to clean and create some models, here are the confusion matrices, the classification reports and AUC:

Logistic regression

[[185847  62897]
 [     1   1061]]

         precision    recall  f1-score   support

 not buy       1.00      0.75      0.86    248744
     buy       0.02      1.00      0.03      1062

accuracy                           0.75    249806



macro avg       0.51      0.87      0.44    249806
weighted avg       1.00      0.75      0.85    249806


AUC train = 0.9168592981611143
AUC test = 0.9150300677458543

Random Forest Classifier:

[[245503   3241]
 [   960    102]]

          precision    recall  f1-score   support

 not buy       1.00      0.99      0.99    248744
     buy       0.03      0.10      0.05      1062

accuracy                           0.98    249806



macro avg       0.51      0.54      0.52    249806
weighted avg       0.99      0.98      0.99    249806


AUC train = 0.9996866568080237
AUC test = 0.9139101966925902

Gradient Boosting Classifier:

[[184940  63804]
 [     3   1059]]

          precision    recall  f1-score   support

 not buy       1.00      0.74      0.85    248744
     buy       0.02      1.00      0.03      1062

accuracy                           0.74    249806



macro avg       0.51      0.87      0.44    249806
weighted avg       1.00      0.74      0.85    249806


AUC train = 0.8800353734759541
AUC test = 0.8657829269466372

Voting Classifier (from all the three above):

[[211316  37428]
 [   213    849]]

          precision    recall  f1-score   support

 not buy       1.00      0.85      0.92    248744
     buy       0.02      0.80      0.04      1062

accuracy                           0.85    249806



macro avg       0.51      0.82      0.48    249806
weighted avg       0.99      0.85      0.91    249806


AUC train = 0.9987531510931085
AUC test = 0.9160262741936392

Since I do not have any experience I am not sure which model is producing better results. Can you help me understand which one and why? Thank you!

oaksandbrooms
  • 53
  • 6

2 Answers2

2

Well, evaluating your model just by comparing matrixes can be pretty hard. In your case, your matrixes are not done with the same number of row classified 1 or 0, so comparing them is really hard. Let's take an example : your logistic regression classifies around 64000 as 1, while your RandomForest only classifies around 4500, so comparing these data by this matrix is pretty hard.

I'd suggest you to use the ROC AUC metric, that is really useful to compare models. You might find many info on this subject in the Internet. The closer your AUC is than 1, the better your model is. If it's 0.5 or less, your model is less efficient than a random classifier.

Adept
  • 904
  • 6
  • 17
2

There are some different methods you can use to measure your model's accuracy. As @BeamsAdept mentioned in his own answer you can use the ROC AUC metric. Alternatively you mind find the Odds Ratio useful and if your dataset is big enough I would highly encourage you look into something like K-fold cross validation in order to get more representative results. I would also be cautious when the training accuracy is significantly higher than your testing accuracy which is the result of overfitting. Perhaps finding the P-values of your features would help you filter out potentially useless information and as a result improve your accuracy in new observations. Hope this helps!

Tony
  • 131
  • 3