How to improve accuracy on a single class out of 3 classes in model

Question

I am training a classification model with 3 classes using a deep neural network.

The classes have been resampled and balanced.

I have around 600000 samples... equally distributed.

The dataset is also divided equitably in the train/test/validation dataset.

After training, the overall accuracy is ~65% but individual classes have a disparity.

Class 0 and 1 have high precision and recall, but the class 2 has very low precision and recall... How can I fix this...

Model defined:


model = tf.keras.models.Sequential([
    tf.keras.layers.InputLayer(input_shape=(20, 4)),
    tf.keras.layers.BatchNormalization(),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(1024, activation='relu'),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(1024, activation='relu'),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(512, activation='relu'),
    tf.keras.layers.Dropout(0.3),
    tf.keras.layers.Dense(3, activation='softmax'),
])
hist = evaluate(
    model=model,
    train_data=(X_train, Y_train),
    val_data=(X_test, Y_test),
    epochs=100,
    batch_size=256,
    verbose=True,
    loss='categorical_crossentropy',
    metrics=['accuracy'],
)

LR scheduler, Early Stopping has been implemented.

Classification report:

              precision    recall  f1-score   support
       0       0.66      0.99      0.79    196323
       1       0.68      0.99      0.80    196323
       2       0.57      0.03      0.06    196323

accuracy                           0.67    588969

macro avg       0.64      0.67      0.55    588969
weighted avg       0.64      0.67      0.55    588969

Confusion Matrix

[[193909    177   2237]
 [   298 193918   2107]
 [ 98184  92287   5852]]

accuracy_score: 0.6684205790117986
roc_auc_score: 0.7927754415211714

UPDATE

I tried the OVA approach and below are the results

So the high accuracy score and the high ROC value is because i oversampled the data and that artificially boosted the accuracy score.

in practice, the real accuracy for class 1 (which was 1 in 11 occurrence), was as below:

              precision    recall  f1-score   support
     0.0       1.00      0.73      0.84    125570
     1.0       0.25      0.98      0.40     11977

accuracy                           0.75    137547

macro avg       0.63      0.85      0.62    137547
weighted avg       0.93      0.75      0.80    137547

I think this extreme imbalance was the reason for the original problem as well

How do i fix this??

score 0 · Accepted Answer · answered Sep 21 '23 at 06:46

Try creating one model for one class. Basically you can call it as class detector, it works as following:

One model will predict class 0. One model will predict 1 and another one for class 2.
Then when you want to classify, you can use decision score from each model for that data and you select the class whose classifier outputs the highest score. This is called the one-versus-all (OvA) strategy (also called one-versus-the-rest).
Another strategy is to train a binary classifier for every pair of digits: one to distinguish class0 and class 1, another to distinguish class 0 and class 2, another for class 1 and class 2, and so on. This is called the one-versus-one (OvO) strategy. If there are N classes, you need to train N × (N – 1) / 2 classifiers. For your problem, this means training 4 binary classifiers! When you want to classify the data, you have to run the data through all 4 classifiers and see which class wins the most duels. The main advantage of OvO is that each classifier only needs to be trained on the part of the training set for the two classes that it must distinguish.

This will improve the overall model for your metrics.

score 0 · Answer 2 · answered Sep 21 '23 at 11:35

Looking at the numbers:

Your classifier is quite good (or even excellent) on distinguishing class 0 and class 1.
Contrary to that, class 2 is either recognized as class 0 or class 1.

I would expect such a result if class 2 would be a mixture of class 0 and class 1. Imagine for example to take about 100 000 cases from class 0 and class 1 and label them class 2. Of course, the effect will probably more subtle than that.

Before tweeking with the classifier, I strongly suggest to look at the data and understand what is going there. There might be some problem with the data collection itself.

Also, a good start might be a projection to the 2-dimensional plane to have a look at the distributions. Train t-SNE or umap on all your training data and use a scatter-plot (with colors for the classes) to have a look at the data. I would expect that class 2 would mix in with class 0 and 1, while both of these classes have little intersection.

How to improve accuracy on a single class out of 3 classes in model

UPDATE

2 Answers2