How to interpret the output of XGBoost importance?

Question

I ran a xgboost model. I don't exactly know how to interpret the output of xgb.importance.

What is the meaning of Gain, Cover, and Frequency and how do we interpret them?

Also, what does Split, RealCover, and RealCover% mean? I have some extra parameters here

Are there any other parameters that can tell me more about feature importances?

From the R documentation, I have some understanding that Gain is something similar to Information gain and Frequency is number of times a feature is used across all the trees. I have no idea what Cover is.

I ran the example code given in the link (and also tried doing the same on the problem that I am working on), but the split definition given there did not match with the numbers that I calculated.

importance_matrix

Output:

           Feature         Gain        Cover    Frequence
  1:            xxx 2.276101e-01 0.0618490331 1.913283e-02
  2:           xxxx 2.047495e-01 0.1337406946 1.373710e-01
  3:           xxxx 1.239551e-01 0.1032614896 1.319798e-01
  4:           xxxx 6.269780e-02 0.0431682707 1.098646e-01
  5:          xxxxx 6.004842e-02 0.0305611830 1.709108e-02

214:     xxxxxxxxxx 4.599139e-06 0.0001551098 1.147052e-05
215:     xxxxxxxxxx 4.500927e-06 0.0001665320 1.147052e-05
216:   xxxxxxxxxxxx 3.899363e-06 0.0001536857 1.147052e-05
217: xxxxxxxxxxxxxx 3.619348e-06 0.0001808504 1.147052e-05
218:  xxxxxxxxxxxxx 3.429679e-06 0.0001792233 1.147052e-05

Answer 1

From your question, I'm assuming that you're using xgboost to fit boosted trees for binary classification. The importance matrix is actually a data.table object with the first column listing the names of all the features actually used in the boosted trees.

The meaning of the importance data table is as follows:

1. The Gain implies the relative contribution of the corresponding feature to the model calculated by taking each feature's contribution for each tree in the model. A higher value of this metric when compared to another feature implies it is more important for generating a prediction.
2. The Cover metric means the relative number of observations related to this feature. For example, if you have 100 observations, 4 features and 3 trees, and suppose feature1 is used to decide the leaf node for 10, 5, and 2 observations in tree1, tree2 and tree3 respectively; then the metric will count cover for this feature as 10+5+2 = 17 observations. This will be calculated for all the 4 features and the cover will be 17 expressed as a percentage for all features' cover metrics.
3. The Frequency (/'Frequence') is the percentage representing the relative number of times a particular feature occurs in the trees of the model. In the above example, if feature1 occurred in 2 splits, 1 split and 3 splits in each of tree1, tree2 and tree3; then the weightage for feature1 will be 2+1+3 = 6. The frequency for feature1 is calculated as its percentage weight over weights of all features.

The Gain is the most relevant attribute to interpret the relative importance of each feature.

The measures are all relative and hence all sum up to one, an example from a fitted xgboost model in R is:

> sum(importance$Frequence)
[1] 1
> sum(importance$Cover)
[1] 1
> sum(importance$Gain)
[1] 1

Answer 2

Thanks Sandeep for your detailed answer. I would like to correct that cover is calculated across all splits and not only the leaf nodes.

Let's go through a simple example with the data provided by the xgboost library.

library('xgboost')

data(agaricus.train, package='xgboost')
data(agaricus.test, package='xgboost')

train <- agaricus.train
test <- agaricus.test

X = train$data

y = train$label

bst <- xgboost(data = X, label = y, max.depth = 2,
           eta = 1, nthread = 2, nround = 2,objective = "binary:logistic",
           reg_lambda = 0, reg_alpha = 0)

xgb.model.dt.tree(agaricus.train$data@Dimnames[[2]], model = bst)

xgb.importance(agaricus.train$data@Dimnames[[2]], bst)

Output -

Let's try to calculate the cover of odor=none in the importance matrix (0.495768965) from the tree dump.

Cover of each split where odor=none is used is 1628.2500 at Node ID 0-0 and 765.9390 at Node ID 1-1.

Total cover of all splits (summing across cover column in the tree dump) = 1628.2500*2 + 786.3720*2

Cover of odor=none in the importance matrix = (1628.2500+765.9390)/(1628.2500*2+786.3720*2)

Hence we are sure that cover is calculated across all splits!