7

I want to obtain the prediction intervals of my xgboost model which I am using to solve a regression problem. I am using the python code shared on this blog, and not really understanding how the quantile parameters affect the model (I am using the suggested parameter values on the blog). When I apply this code to my data, I obtain nonsense results, such as negative predictions for my target values while my target values are always over 10K. I don't understand how should this code vary according to my data and would really appreciate any help.

Differences in my data to the data that is used on the blog are:

  • My distribution is Poisson like.
  • I have over 100 features.

Note: I tried tuning the delta, threshold and var parameters, but they don't seem to have a controllable effect on the results and predictions remains nonsense.

mari
  • 141
  • 1
  • 2
  • 8

3 Answers3

3

To produce confidence intervals for xgboost model you should train several models (you can use bagging for this). Each model will produce a response for test sample - all responses will form a distribution from which you can easily compute confidence intervals using basic statistics. You should produce response distribution for each test sample.

this answer is provided here: https://stackoverflow.com/questions/37418938/how-to-obtain-a-confidence-interval-or-a-measure-of-prediction-dispersion-when-u

Nidhi Agarwal
  • 225
  • 1
  • 8
0

As an alternative to Nidhi's answer, you could also train two models to directly predict the bounds of the confidence interval. One model would have the upper bound as its training target, the other model the lower bound.

One downside of this approach is that you cannot simply change the confidence interval, because that requires retraining of the models.

joba2ca
  • 146
  • 4
0

Try the following code. It must work fine. It might take a lot of time (more than 100 features).

Change max_depth to 6 if you want more accuracy. (because of 100 features.)

We can change learning_rate between 0 and 1, to improve the efficiency.

import xgboost as xgb
model_xgb = xgb.XGBRegressor(colsample_bytree=0.4603, gamma=0.0468, 
                             learning_rate=0.05, max_depth=3, 
                             min_child_weight=1.7817, n_estimators=4200,
                             reg_alpha=0.4640, reg_lambda=0.8571,
                             subsample=0.5213, silent=1,
                             nthread=-1)

X_train, X_test, Y_train, Y_test= train_test_split(X, Y, random_state= 0)
def model_score_error(model):
    prepared_model=model.fit(X_train, Y_train)
    x=prepared_model.score(X_test,Y_test)
    print('Score: ',x)
    Target_predicted=prepared_model.predict(X_test) 
    MSE=mean_squared_error(Y_test,Target_predicted) 
    print('mean square error', MSE)

model_score_error(model_xgb)
Nidhi Agarwal
  • 225
  • 1
  • 8