I am analyzing the results of various machine learning models for a regression task, using four metrics: RMSE, MAE, MAPE, and $R^2$. My approach involves two types of analyses:
- Individual Metric Analysis: Where each metric is considered separately.
- Composite Performance Metric Analysis: A combined metric to determine the best model among the candidates.
The Composite Performance Metric (CPM) incorporates the standardized mean of the metrics derived from the individual results obtained through cross-validation: $$ \begin{equation} \text{CPM} = M_{norm}(\text{RMSE}) + M_{norm}(\text{MAE}) + M_{norm}(\text{MAPE}) - M_{norm}(R^2) \end{equation} $$
where
$$ \begin{equation} M_{norm}(\text{metric}) = \frac{\text{metric} - \mu_{\text{metric}}}{\sigma_{\text{metric}}} \end{equation} $$ and $\mu_\text{metric}$ is the overall mean of the metric across all models, and $\sigma_\text{metric}$ is the overall standard deviation of the metric across all models.
The CPM prioritizes models with lower errors (RMSE, MAE, MAPE) and higher $R^2$ values. By standardizing each metric, the CPM ensures that all metrics contribute equally, allowing for a fair comparison across different models. It essentially functions as a weighted sum where each metric is equally weighted.
The Issue with Sign Changes
However, I've encountered a potential issue: during the standardization process, the signs of some metrics may change. This could lead to misleading results when calculating the CPM. For instance, if the sign of $R^2$ flips due to standardization, it might inadvertently increase the CPM, penalizing models that should be rewarded for their higher $R^2$ values.
Proposed Solution
To address this issue, I am considering the following approach:
- Error Metrics (RMSE, MAE, MAPE): Since these metrics are non-negative, I propose taking the absolute value of the standardized metrics to ensure they always contribute positively to the CPM.
- $R^2$ Metric: I suggest preserving the original sign of $R^2$ before standardization and reapplying it afterward to maintain its intended influence on the CPM.
Request for Feedback
Does this approach seem valid, or am I overlooking something? Are there alternative methods or considerations that might better address the issue of sign changes during standardization in the context of the CPM?
