I have an imbalanced dataset (True labels are ~10x than False labels) and thus use the f_beta score as a metric for model performance, as such:
def fbeta(beta):
def f1(y_true, y_pred):
def recall(y_true, y_pred):
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
possible_positives = K.sum(K.round(K.clip(y_true, 0, 1)))
recall = true_positives / (possible_positives + K.epsilon())
return recall
def precision(y_true, y_pred):
true_positives = K.sum(K.round(K.clip(y_true * y_pred, 0, 1)))
predicted_positives = K.sum(K.round(K.clip(y_pred, 0, 1)))
precision = true_positives / (predicted_positives + K.epsilon())
return precision
beta_squared = beta ** 2
precision = precision(y_true, y_pred)
recall = recall(y_true, y_pred)
fbeta_score = (beta_squared + 1) * (precision * recall) / (beta_squared * precision + recall + K.epsilon())
return fbeta_score
return f1
Running my model I get the following output.
Epoch 00001: val_f1 improved from -inf to 0.90196, saving model to output\LSTM\weights\floods_weights.h5 Epoch 2/3
- 0s - loss: 0.4114 - f1: 0.8522 - val_loss: 0.4193 - val_f1: 0.9020 Epoch 00002: val_f1 did not improve from 0.90196
Epoch 3/3
- 0s - loss: 0.3386 - f1: 0.8867 - val_loss: 0.3589 - val_f1: 0.9020 Epoch 00003: val_f1 did not improve from 0.90196
Evaluating :
51/51 [==============================] - 0s 372us/step
Final Accuracy : 0.9019607305526733
However, when running the model again with swapped y-labels (True->False, False->True) I get the exact same values and do not understand why as the f-measure should be highly unlikely to give the exact same results.
When assessing the final model using the following code for precision and recall, I do get different results: 0.6 vs 0.95788:
prediction = model.predict(data['X_test'], batch_size=64)
tp = 0
fp = 0
fn = 0
for y, p in zip(data['y_test'], np.argmax(prediction, axis=1)):
y = y[0]
if y == 1 and p == 1:
tp += 1
elif p == 1 and y == 0:
fp += 1
elif p == 0 and y == 1:
fn += 1
precision = tp / (tp + fp)
recall = tp / (tp + fn)
print(2 * (precision * recall) / (precision + recall))
What's happening?
Edit:
I am compiling the model as such:
beta = 1
model.compile(
optimizer=Adam(lr=1e-3),
loss='categorical_crossentropy',
metrics=[fbeta(beta)]
)