Bagging vs pasting in ensemble learning

Question

This is a citation from "Hands-on machine learning with Scikit-Learn, Keras and TensorFlow" by Aurelien Geron:

"Bootstrapping introduces a bit more diversity in the subsets that each predictor is trained on, so bagging ends up with a slightly higher bias than pasting, but this also means that predictors end up being less correlated so the ensemble’s variance is reduced."

I can't understand why bagging, as compared to pasting, results in higher bias and lower variance. Can anyone provide an intuitive explanation of this?

Answer 1

Let's say we have a set of 40 numbers from 1 to 40. We have to pick 4 subsets of 10 numbers.

Case 1 - Bagging -
We will pick the first number, put it back, and then pick the next. This makes all the draw independent and consequently have very little correlation.
So, if you make a Tree on the first 10 samples and another Tree on the next, both the trees will have little correlation and high variance(among them) (more independent splits).
At the same time, because of selection with replacement the data points will be repeated [~63% unique] [Ref], which will increase the bias of individual Trees.

In case of bagging, the sample size is equal to the size of the dataset but we just created this scenario to compare it with Pasting.

Same logic goes for splitting with Random Features subset i.e. RandomForest.
It might be possible that a Split on a particular Feature may result in the correlated next split(always). So if we randomly pick a subset of features before each split, then this will further reduce the Correlation. But again, we will have increased Bias.

Case 2 - Pasting -
Here, because of selection without replacement, the data points in each sample will be unique which will result in lesser bias of individual Trees.

Answer 2

In the context of ensembling, the aim of both bagging and pasting is to get a diverse set of estimators despite each estimator using the same algorithm.

The diversity comes from how you set up the data for individual estimators. Both bagging and pasting randomly select samples for each estimator, meaning each estimator's data will be different.

When selecting samples for an estimator in the ensemble, bagging and pasting both start off with the entire dataset. Bagging selects a sample, and it is allowed to re-select the sample if it randomly comes up again, meaning that you end up with some duplicated samples. Pasting stipulates a constraint: once you've selected a sample for this estimator, the sample cannot be selected a second time.

As an example, if the entire dataset is [10, 20, 30, 40, 50] and you want each estimator trained on 80% of the dataset (4 samples), then a bagged subset could be [10, 10, 10, 20] (duplication is permitted), whereas a pasted subset could be [10, 20, 30, 40] (no sample duplication).

Since bagging duplicates samples, there's more potential for subsets to be different across estimators. Pasting, on the other hand, constrains the samples you can choose from, so pasted subsets end up looking more similar between estimators.

The diversity introduced by bagging means that it is less representative of the original data, and therefore will have higher bias. But the diversity also means that the individual estimators are less similar, which results in better generalisation performance to new data (lower variance).

Similarly, since pasted subsets don't duplicate samples, the individual estimators have a better idea of the original dataset, and therefore they score more highly on the training data (lower bias). By following the original dataset more closely, the estimators in a pasted ensemble are more correlated. Ensembles hinge on and exploit estimator diversity, which puts pasting at a disadvantage and is why it might not generalise as well (higher variance).

Bagging generally results in better models, but you'd need to run CV to see whether this holds for your particular scenario. It's probably not worth tuning this aspect until you've assessed more pertinent model parameters.