3

I am looking for as close as possible for a exhaustive taxonomy of each train-test split approach.

For example, the 3 main splits that come to mind are:

  • A non-time based problem - would lead you to a random, maybe stratified train-test split.
  • A single time-series problem, would require a purely time based train-test split.
  • A multiple item, time-series (panel) problem, would require a split across time? And maybe additional splits between observations?

I think a separate question is around k-fold validation, which would typically be computed within the train section of this split.

Interested to hear what problems I have missed above, links to great papers/resources that break this down.

GooJ
  • 445
  • 2
  • 11

1 Answers1

1

I don't know if there are many train/test split solutions, but it mainly depends on 3 things:

  • The data quantity
  • The data complexity/balance
  • The data type
  • Multi-class or single class

To answer your question completely, we should make a table considering every scenario, but most solutions would have been the classic 80% / 20% distribution.

Nevertheless, we can focus on the specific cases, i.e:

  • Complete data with high complexity: Create groups of data from which you take representative train/test splits (ex: if there is a very small group of 10 values, you take a random split of 7 and 3 for train/test, instead of mixing this small group with a general random train/test split).
  • Data scarcity with low complexity: Use sampling with replacement to have many train/test bags of data and select the one that has the best results.
  • Data scarcity with high complexity: In this case, we can apply a cost-sensitive split. We weigh scarce data to make our model learn from specific cases. This is a lot of data preparation.

https://fraud-detection-handbook.github.io/fraud-detection-handbook/Chapter_6_ImbalancedLearning/CostSensitive.html

As you've mentioned, time series have different behavior, but they could have specific cases. Data with cyclic behavior can have their cycles randomized because cycles can be compared independently to each other (as long as the correlation between cycles is not very important).

Nicolas Martin
  • 5,014
  • 1
  • 8
  • 15