2

During prediction phase, fully trained supervised models may have to deal with data representing new classes, that weren't part of the training and test sets. A real world example for this issue is Supervised Condition Monitoring, where a classifier can be trained on data representing several conditions of a mechanical system. Since it is impossible to know and/or generate data for all possible conditions, such a classifier may encounter data representing new and unknown classes during runtime.

What I am looking for, is the general term for this issue and approaches that address it. Semi-Supervised-Learning doesn't seem to be a good fit here, since in my example, all training and test data is labeled, and new classes only occur during runtime. For similar reasons, Data Drift doesn't apply, since the drift only refers to data of known classes. One-Class-Classification would not help either, since the example clearly represents a multiclass scenario.

A good approach to address the issue would be classifiers, that for each test sample and each known class output the probability of the test sample belonging to this class. For a test sample belonging to an unknown class, all these probabilities should be low.

Does anyone know the scientific term for this general issue and can point me into the direction of relevant research and possible solutions?

Tripartio
  • 153
  • 5
user1934212
  • 141
  • 3

2 Answers2

2

One option would be to take a Bayesian approach by estimating the confidence in the class prediction. If all of the confidences are below a threshold, predict "new class".

Another option is zero-shot learning (ZSL) which goes one step further and attempts to predict a specific class for the unobserved classes. ZSL often requires auxiliary information, which includes distinguishing properties of the items.

Brian Spiering
  • 23,131
  • 2
  • 29
  • 113
1

It might not be exactly what you're asking, but your example describes a clear use case for one-class classification: there is a representative sample only for the 'good' class, anything else is is a fault condition. It's different from regular classification because it does not rely on discriminating between the classes: in regular classification (usually with one-against-all), the model predicts the most likely class with respect to the other classes. As a consequence, the model is not meant to deal with cases which are completely different from the training set in the regular classification setting. By contrast, in the one-class classification setting the model attempts to represent the class on its own, so in theory is best prepared to deal with any completely different case which doesn't belong to the class.

Erwan
  • 26,519
  • 3
  • 16
  • 39