Chapter 3 Super (Ensemble Machine) Learning

Ivana Malenica and Rachael Phillips

Based on the sl3 R package by Jeremy Coyle, Nima Hejazi, Ivana Malenica, and Oleg Sofrygin.

Updated: 2019-11-13

3.1 Introduction

Once the statistical estimation problem is defined, as described in the The Targeted Learning Roadmap, we are ready to construct the TMLE: an asymptotically efficient substitution estimator of this target quantity.

The first step in the estimation procedure is an initial estimate of the data-generating distribution, or the relevant part of this distribution that is needed to evaluate the target parameter. For this initial estimation, we use the super learner (van der Laan, Polley, and Hubbard 2007), an important step for creating a robust estimator.

3.1.1 Super Learning

  • A common task in statistical data analysis is estimator selection (e.g., for prediction).
  • There is no universally optimal machine learning algorithm for density estimation or prediction.
  • For some data, one needs learners that can model a complex function.
  • For others, possibly as a result of noise or insufficient sample size, a simple, parametric model might fit best.
  • Super Learner, an ensemble learner, solves this issue, by allowing a combination of learners from the simplest (intercept-only) to most complex (neural nets, random forests, SVM, etc).
  • It works by using cross-validation in a manner which guarantees that the resulting fit will be as good as possible, given the learners provided.
  • Note: even a combination of poor learners can sometimes result in good fit. It is very important to have good candidates in our library, possibly incorporating known knowledge about the system in question.

3.1.1.1 General Overview of the Algorithm

What is cross-validation and how does it work?

  • There are many different cross-validation schemes, designed to accommodate different study designs and data structures.
  • The figure below shows an example of 10-fold cross-validation.

General step-by-step overview of the Super Learner algorithm:

  • Break up the sample evenly into V-folds (say V=10).
  • For each of these 10 folds, remove that portion of the sample (kept out as validation sample) and the remaining will be used to fit learners (training sample).
  • Fit each learner on the training sample (note, some learners will have their own internal cross-validation procedure or other methods to select tuning parameters).
  • For each observation in the corresponding validation sample, predict the outcome using each of the learners, so if there are \(p\) learners, then there would be \(p\) predictions.
  • Take out another validation sample and repeat until each of the V-sets of data are removed.
  • Compare the cross-validated fit of the learners across all observations based on specified loss function (e.g., squared error, negative log-likelihood, …) by calculating the corresponding average loss (risk).

  • Either:

    • choose the learner with smallest risk and apply that learner to entire data set (resulting SL fit),

    • do a weighted average of the learners to minimize the cross-validated risk (construct an ensemble of learners), by

      • re-fitting the learners on the original data set, and
      • use the weights above to get the SL fit.

Note, this entire procedure can be itself cross-validated to get a consistent estimate of the future performance of the SL fit.

How to pick a Super Learner library?

  • A library is simply a collection of algorithms.
  • The algorithms in the library should come from contextual knowledge and a large set of “default” algorithms.
  • The algorithms may range from a simple linear regression model to multi-step algorithms involving screening covariates, penalizations, optimizing tuning parameters, etc.

3.1.1.2 Example: Super Learner In Prediction

  • We observe a learning data set \(X_i=(Y_i,W_i)\), for \(i=1, ..., n\).
  • Here, \(Y_i\) is the outcome of interest, and \(W_i\) is a p-dimensional set of covariates.
  • Our objective is to estimate the function \(\psi_0(W) = E(Y|W)\).
  • This function can be expressed as the minimizer of the expected loss: \(\psi_0(W) = \text{argmin}_{\psi} E[L(X,\psi(W))]\).
  • Here, the loss function is represented as \(L\) (e.g., squared error loss, \(L: (Y-\psi(W))^2)\)).

3.1.1.3 Why use the Super Learner?

  • For prediction, one can use the cross-validated risk to empirically determine the relative performance of SL and competing methods.
  • When we have tested different algorithms on actual data and looked at the performance (e.g., MSE of prediction), never does one algorithm always win (see below).
  • Below shows the results of such a study, comparing the fits of several different learners, including the SL algorithms.
  • Super Learner performs asymptotically as well as best possible weighted combination.
  • By including all competitors in the library of candidate estimators (glm, neural nets, SVMs, random forest, etc.), the Super Learner will asymptotically outperform any of its competitors- even if the set of competitors is allowed to grow polynomial in sample size.
  • Motivates the name “Super Learner”: it provides a system of combining many estimators into an improved estimator.

Review of the Super Learner

  • Loss-function-based tool that uses V-fold cross-validation to obtain the best prediction of the relevant part of the likelihood that’s needed to evaluate target parameter.

  • Requires expressing the estimand as the minimizer of an expected loss, and proposing a library of algorithms (“learners” in sl3 nomenclature) that we think might be consistent with the true data-generating distribution.

  • The discrete super learner, or cross-validated selector, is the algorithm in the library that minimizes the V-fold cross-validated empirical risk.

  • The super learner is a weighted average of the library of algorithms, where the weights are chosen to minimize the V-fold cross-validated empirical risk of the library. Restricting the weights (“metalearner” in sl3 nomenclature) to be positive and sum to one (convex combination) has been shown to improve upon the discrete super learner (Polley and van der Laan 2010; van der Laan, Polley, and Hubbard 2007).

  • Proven to be asymptotically as accurate as the best possible prediction algorithm that is tested (van der Laan and Dudoit 2003; van der Vaart, Dudoit, and van der Laan 2006).

  • This background material is described in greater detail in the accompanying tlverse handbook sl3 chapter.

3.2 Learning Objectives

By the end of this lesson you will be able to:

  1. Assemble an ensemble of learners based on the properties that identify what features they support.
  2. Customize learner hyperparameters to incorporate a diversity of different settings.
  3. Select a subset of available covariates and pass only those variables to the modeling algorithm.
  4. Fit an ensemble with nested cross-validation to obtain an estimate of the performance of the ensemble itself.
  5. Calculate sl3 variable importance metrics.
  6. Interpret the discrete and continuous super learner fits.
  7. Rationalize the need to remove bias from the super learner to make an optimal bias-variance tradeoff for the parameter of interest.

3.3 sl3 “Microwave Dinner” Implementation

We begin by illustrating the core functionality of the super learner algorithm as implemented in sl3. For those who are interested in the internals of sl3, see this sl3 introductory tutorial.

The sl3 implementation consists of the following steps:

  1. Load the necessary libraries and data
  2. Define the machine learning task
  3. Make a super learner by creating library of base learners and a metalearner
  4. Train the super learner on the machine learning task
  5. Obtain predicted values

WASH Benefits Study Example

Using the WASH data, we are interested in predicting weight-for-height z-score whz using the available covariate data.

0. Load the necessary libraries and data

whz tr fracode month aged sex momage momedu momheight hfiacat Nlt18 Ncomp watmin elec floor walls roof asset_wardrobe asset_table asset_chair asset_khat asset_chouki asset_tv asset_refrig asset_bike asset_moto asset_sewmach asset_mobile
0.00 Control N05265 9 268 male 30 Primary (1-5y) 146.40 Food Secure 3 11 0 1 0 1 1 0 1 1 1 0 1 0 0 0 0 1
-1.16 Control N05265 9 286 male 25 Primary (1-5y) 148.75 Moderately Food Insecure 2 4 0 1 0 1 1 0 1 0 1 1 0 0 0 0 0 1
-1.05 Control N08002 9 264 male 25 Primary (1-5y) 152.15 Food Secure 1 10 0 0 0 1 1 0 0 1 0 1 0 0 0 0 0 1
-1.26 Control N08002 9 252 female 28 Primary (1-5y) 140.25 Food Secure 3 5 0 1 0 1 1 1 1 1 1 0 0 0 1 0 0 1
-0.59 Control N06531 9 336 female 19 Secondary (>5y) 150.95 Food Secure 2 7 0 1 0 1 1 1 1 1 1 1 0 0 0 0 0 1
-0.51 Control N06531 9 304 male 20 Secondary (>5y) 154.20 Severely Food Insecure 0 3 1 1 0 1 1 0 0 0 0 1 0 0 0 0 0 1

1. Define the machine learning task

To define the machine learning “task” (predict weight-for-height z-score whz using the available covariate data), we need to create an sl3_Task object.

The sl3_Task keeps track of the roles the variables play in the machine learning problem, the data, and any metadata (e.g., observational-level weights, id, offset).

Warning in .subset2(public_bind_env, "initialize")(...): Missing Covariate Data
Found. Imputing covariates using sl3_process_missing.
A sl3 Task with 4695 obs and these nodes:
$covariates
 [1] "tr"              "fracode"         "month"           "aged"           
 [5] "sex"             "momage"          "momedu"          "momheight"      
 [9] "hfiacat"         "Nlt18"           "Ncomp"           "watmin"         
[13] "elec"            "floor"           "walls"           "roof"           
[17] "asset_wardrobe"  "asset_table"     "asset_chair"     "asset_khat"     
[21] "asset_chouki"    "asset_tv"        "asset_refrig"    "asset_bike"     
[25] "asset_moto"      "asset_sewmach"   "asset_mobile"    "delta_momage"   
[29] "delta_momheight"

$outcome
[1] "whz"

$id
NULL

$weights
NULL

$offset
NULL

2. Make a super learner

Now that we have defined our machine learning problem with the task, we are ready to “make” the super learner. This requires specification of

  • Base learning algorithms, to establish a library of learners that we think might be consistent with the true data-generating distribution.
  • Metalearner, to ensemble the base learners.

We might also incorporate

  • Feature selection, to pass only a subset of the predictors to the algorithm.
  • Hyperparameter specification, to tune base learners.

Learners have properties that indicate what features they support. We may use sl3_list_properties() to get a list of all properties supported by at least one learner.

 [1] "binomial"             "categorical"          "continuous"          
 [4] "cv"                   "density"              "ids"                 
 [7] "multivariate_outcome" "offset"               "preprocessing"       
[10] "timeseries"           "weights"              "wrapper"

Since we have a continuous outcome, we may identify the learners that support this outcome type with sl3_list_learners().

 [1] "Lrnr_arima"                     "Lrnr_bartMachine"              
 [3] "Lrnr_bilstm"                    "Lrnr_caret"                    
 [5] "Lrnr_condensier"                "Lrnr_dbarts"                   
 [7] "Lrnr_earth"                     "Lrnr_expSmooth"                
 [9] "Lrnr_gam"                       "Lrnr_gbm"                      
[11] "Lrnr_glm"                       "Lrnr_glm_fast"                 
[13] "Lrnr_glmnet"                    "Lrnr_grf"                      
[15] "Lrnr_h2o_glm"                   "Lrnr_h2o_grid"                 
[17] "Lrnr_hal9001"                   "Lrnr_HarmonicReg"              
[19] "Lrnr_lstm"                      "Lrnr_mean"                     
[21] "Lrnr_nnls"                      "Lrnr_optim"                    
[23] "Lrnr_pkg_SuperLearner"          "Lrnr_pkg_SuperLearner_method"  
[25] "Lrnr_pkg_SuperLearner_screener" "Lrnr_polspline"                
[27] "Lrnr_randomForest"              "Lrnr_ranger"                   
[29] "Lrnr_rpart"                     "Lrnr_rugarch"                  
[31] "Lrnr_solnp"                     "Lrnr_stratified"               
[33] "Lrnr_svm"                       "Lrnr_tsDyn"                    
[35] "Lrnr_xgboost"

Now that we have an idea of some learners, we can construct them using the make_learner function.

We can customize learner hyperparameters to incorporate a diversity of different settings.

Documentation for the learners and their hyperparameters can be found in the sl3 Learners Reference.

We can also include learners from the SuperLearner R package.

In order to assemble the library of learners, we need to “stack” them together.

A Stack is a special learner and it has the same interface as all other learners. What makes a stack special is that it combines multiple learners by training them simultaneously, so that their predictions can be either combined or compared.

We will fit a non-negative least squares metalearner using Lrnr_nnls. Note that any learner can be used as a metalearner.

We can optionally select a subset of available covariates and pass only those variables to the modeling algorithm.

Let’s consider screening covariates based on their correlation with our outcome of interest (cor.test p-value \(\leq 0.1\)).

Warning: `lang_tail()` is deprecated as of rlang 0.2.0.
This warning is displayed once per session.
Warning: `mut_node_cdr()` is deprecated as of rlang 0.2.0.
This warning is displayed once per session.
[1] "Lrnr_pkg_SuperLearner_screener_screen.corP"
$selected
 [1] "tr"             "fracode"        "aged"           "momage"        
 [5] "momedu"         "momheight"      "hfiacat"        "Nlt18"         
 [9] "elec"           "floor"          "walls"          "asset_wardrobe"
[13] "asset_table"    "asset_chair"    "asset_khat"     "asset_chouki"  
[17] "asset_tv"       "asset_refrig"   "asset_moto"     "asset_sewmach" 
[21] "asset_mobile"

To “pipe” only the selected covariates to the modeling algorithm, we need to make a Pipeline, which is a just set of learners to be fit sequentially, where the fit from one learner is used to define the task for the next learner.

Now our learners will be preceded by a screening step.

We also consider the original stack, just to compare how the feature selection methods perform in comparison to the methods without feature selection.

Analogous to what we have seen before, we have to stack the pipeline and original stack together, so we may use them as base learners in our super learner.

We have made a library/stack of base learners and a metalearner, so we are ready to make the super learner. The super learner algorithm fits a metalearner on the validation-set predictions.

We can also use Lrnr_cv to build a super learner, cross-validate a stack of learners to compare performance of the learners in the stack, or cross-validate any single learner (see “Cross-validation” section of this sl3 introductory tutorial).

Furthermore, we can Define New sl3 Learners which can be used in all the places you could otherwise use any other sl3 learners, including Pipelines, Stacks, and the Super Learner.

3. Train the super learner on the machine learning task

Now we are ready to “train” our super learner on our sl3_task object, washb_task.

4. Obtain predicted values

Now that we have fit the super learner, we are ready to obtain our predicted values, and we can also obtain a summary of the results.

[1] -0.5285481 -0.9118507 -0.7555671 -0.7374887 -0.6580733 -0.7360829
[1] "SuperLearner:"
List of 2
 $ : chr "Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)"
 $ : chr "Stack"
[1] "Lrnr_nnls"
                                                                                                  lrnrs
 1:                           Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_glm_TRUE
 2:                               Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_mean
 3:                  Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_ranger_100_TRUE_1
 4: Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_glmnet_NULL_deviance_10_1_100_TRUE
 5:            Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_pkg_SuperLearner_SL.gam
 6:       Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_pkg_SuperLearner_SL.bayesglm
 7:                                                                                 Stack_Lrnr_glm_TRUE
 8:                                                                                     Stack_Lrnr_mean
 9:                                                                        Stack_Lrnr_ranger_100_TRUE_1
10:                                                       Stack_Lrnr_glmnet_NULL_deviance_10_1_100_TRUE
11:                                                                  Stack_Lrnr_pkg_SuperLearner_SL.gam
12:                                                             Stack_Lrnr_pkg_SuperLearner_SL.bayesglm
       weights
 1: 0.00000000
 2: 0.00000000
 3: 0.34430490
 4: 0.00000000
 5: 0.14888084
 6: 0.00000000
 7: 0.06413515
 8: 0.00000000
 9: 0.05745541
10: 0.39039616
11: 0.00000000
12: 0.00000000
[1] "Cross-validated risk (MSE, squared error loss):"
                                                                                                learner
 1:                           Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_glm_TRUE
 2:                               Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_mean
 3:                  Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_ranger_100_TRUE_1
 4: Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_glmnet_NULL_deviance_10_1_100_TRUE
 5:            Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_pkg_SuperLearner_SL.gam
 6:       Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_pkg_SuperLearner_SL.bayesglm
 7:                                                                                 Stack_Lrnr_glm_TRUE
 8:                                                                                     Stack_Lrnr_mean
 9:                                                                        Stack_Lrnr_ranger_100_TRUE_1
10:                                                       Stack_Lrnr_glmnet_NULL_deviance_10_1_100_TRUE
11:                                                                  Stack_Lrnr_pkg_SuperLearner_SL.gam
12:                                                             Stack_Lrnr_pkg_SuperLearner_SL.bayesglm
13:                                                                                        SuperLearner
    coefficients mean_risk    SE_risk    fold_SD fold_min_risk fold_max_risk
 1:           NA  1.015128 0.02363317 0.07629401     0.8927540      1.131594
 2:           NA  1.065282 0.02502664 0.09191791     0.9264292      1.196647
 3:           NA  1.019054 0.02357973 0.08215682     0.8750611      1.157305
 4:           NA  1.013120 0.02360115 0.07902932     0.8822292      1.130862
 5:           NA  1.015128 0.02363317 0.07629401     0.8927540      1.131594
 6:           NA  1.015119 0.02363328 0.07631510     0.8926608      1.131570
 7:           NA  1.018612 0.02380402 0.07799191     0.8956048      1.134940
 8:           NA  1.065282 0.02502664 0.09191791     0.9264292      1.196647
 9:           NA  1.023817 0.02370594 0.08537651     0.8824159      1.168307
10:           NA  1.012201 0.02358754 0.07954388     0.8826979      1.131795
11:           NA  1.018612 0.02380402 0.07799191     0.8956048      1.134940
12:           NA  1.018596 0.02380414 0.07801948     0.8954820      1.134909
13:           NA  1.006803 0.02344266 0.08015240     0.8723953      1.133771

3.4 Extensions

3.4.1 Cross-validated Super Learner

We can cross-validate the super learner to see how well the super learner performs on unseen data, and obtain an estimate of the cross-validated risk of the super learner.

This estimation procedure requires an “external” layer of cross-validation, also called nested cross-validation, which involves setting aside a separate holdout sample that we don’t use to fit the super learner. This external cross-validation procedure may also incorporate 10 folds, which is the default in sl3. However, we will incorporate 2 outer/external folds of cross-validation for computational efficiency.

We also need to specify a loss function to evaluate super learner. Documentation for the available loss functions can be found in the sl3 Loss Function Reference.

Warning in .subset2(public_bind_env, "initialize")(...): Missing Covariate Data
Found. Imputing covariates using sl3_process_missing.
learner coefficients mean_risk SE_risk fold_SD fold_min_risk fold_max_risk
Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_glm_TRUE NA 1.0365 0.0241 0.0370 1.0103 1.0627
Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_mean NA 1.0673 0.0251 0.0464 1.0345 1.1001
Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_ranger_100_TRUE_1 NA 1.0467 0.0244 0.0331 1.0233 1.0701
Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_glmnet_NULL_deviance_10_1_100_TRUE NA 1.0216 0.0239 0.0414 0.9924 1.0509
Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_pkg_SuperLearner_SL.gam NA 1.0365 0.0241 0.0370 1.0103 1.0627
Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Stack)_Lrnr_pkg_SuperLearner_SL.bayesglm NA 1.0364 0.0241 0.0371 1.0102 1.0626
Stack_Lrnr_glm_TRUE NA 1.0458 0.0252 0.0342 1.0216 1.0700
Stack_Lrnr_mean NA 1.0673 0.0251 0.0464 1.0345 1.1001
Stack_Lrnr_ranger_100_TRUE_1 NA 1.0400 0.0243 0.0426 1.0098 1.0701
Stack_Lrnr_glmnet_NULL_deviance_10_1_100_TRUE NA 1.0213 0.0239 0.0436 0.9905 1.0521
Stack_Lrnr_pkg_SuperLearner_SL.gam NA 1.0458 0.0252 0.0342 1.0216 1.0700
Stack_Lrnr_pkg_SuperLearner_SL.bayesglm NA 1.0458 0.0252 0.0342 1.0216 1.0699
SuperLearner NA 1.0237 0.0240 0.0415 0.9944 1.0531

3.4.2 Variable Importance Measures with sl3

The sl3 varimp function returns a table with variables listed in decreasing order of importance, in which the measure of importance is based on a risk difference between the learner fit with a permuted covariate and the learner fit with the true covariate, across all covariates.

In this manner, the larger the risk difference, the more important the variable is in the prediction.

X risk_diff
aged 0.0379
momedu 0.0092
tr 0.0057
Nlt18 0.0042
momheight 0.0034
floor 0.0032
asset_refrig 0.0029
asset_chair 0.0029
elec 0.0024
month 0.0021
asset_chouki 0.0017
hfiacat 0.0009
asset_tv 0.0009
asset_moto 0.0008
walls 0.0008
asset_wardrobe 0.0005
asset_khat 0.0005
sex 0.0000
delta_momage 0.0000
roof -0.0001
momage -0.0001
fracode -0.0001
watmin -0.0001
asset_bike -0.0002
delta_momheight -0.0002
asset_table -0.0005
Ncomp -0.0005
asset_mobile -0.0006
asset_sewmach -0.0006

3.5 Exercise

3.5.1 Predicting Myocardial Infarction with sl3

Follow the steps below to predict myocardial infarction (mi) using the available covariate data. We thank Prof. David Benkeser at Emory University for making the this Cardiovascular Health Study (CHS) data accessible.

  1. Which learner was the discrete super learner? What was the cross validated mean risk of the discrete super learner?
  2. What was the cross-validated risk of the continuous super learner?
  3. Did your group face any challenges?
  4. Any additional comments/questions about this sl3 section of the workshop?
waist alcoh hdl beta smoke ace ldl bmi aspirin gend age estrgn glu ins cysgfr dm fetuina whr hsed race logcystat logtrig logcrp logcre health logkcal sysbp mi
110.1642 0.0000 66.4974 0 0 1 114.2162 27.9975 0 0 73.5179 0 159.9314 70.3343 75.0078 1 0.1752 1.1690 1 1 -0.3420 5.4063 2.0126 -0.6739 0 4.3926 177.1345 0
89.9763 0.0000 50.0652 0 0 0 103.7766 20.8931 0 0 61.7723 0 153.3888 33.9695 82.7433 1 0.5717 0.9011 0 0 -0.0847 4.8592 3.2933 -0.5551 1 6.2071 136.3742 0
106.1941 8.4174 40.5059 0 0 0 165.7158 28.4554 1 1 72.9312 0 121.7145 -17.3017 74.6989 0 0.3517 1.1797 0 1 -0.4451 4.5088 0.3013 -0.0115 0 6.7320 135.1993 0
90.0566 0.0000 36.1750 0 0 0 45.2035 23.9608 0 0 79.1191 0 53.9691 11.7315 95.7823 0 0.5439 1.1360 0 0 -0.4807 5.1832 3.0243 -0.5751 1 7.3972 139.0182 0
78.6143 2.9790 71.0642 0 1 0 131.3121 10.9656 0 1 69.0179 0 94.3153 9.7112 72.7109 0 0.4916 1.1028 1 0 0.3121 4.2190 -0.7057 0.0053 1 8.2779 88.0470 0
91.6593 0.0000 59.4963 0 0 0 171.1872 29.1317 0 1 81.8346 0 212.9066 -28.2269 69.2184 1 0.4621 0.9529 1 0 -0.2872 5.1773 0.9705 0.2127 1 5.9942 69.5943 0
  1. Create an sl3 task, setting myocardial infarction mi as the outcome and using all available covariate data.
  2. Make a library of seven relatively fast base learning algorithms (i.e., do not consider BART or HAL). Customize hyperparameters for one of your learners. Feel free to use learners from sl3 or SuperLearner. You may use the same base learning library that is presented above.
  3. Incorporate feature selection with the SuperLearner screener screen.corP.
  4. Fit the metalearning step with non-negative least squares, Lrnr_nnls.
  5. With the metalearner and base learners, make the super learner and train it on the task.
  6. Print your super learner fit by calling print() with $.
  7. Cross-validate your super learner fit to see how well it performs on unseen data. Specify loss_squared_error as the loss function to evaluate the super learner. Like above, create a new task with 2 folds of external cross-validation for computational efficiency.

3.6 Summary

  • The general ensemble learning approach of super learner can be applied to a diversity of estimation and prediction problems that can be defined by a loss function.

  • Plug-in estimators of the estimand are desirable because a plug-in estimator respects both the local and global constraints of the statistical model.

  • Asymptotically linear estimators are also advantageous, since they converge to the estimand at \(\frac{1}{\sqrt{n}}\) rate, and thereby permit formal statistical inference.

  • If we plug in the estimator returned by super learner into the target parameter mapping, then we would end up with an estimator that has the same bias as what we plugged in. This estimator would not be asymptotically linear.

  • Targeted maximum likelihood estimation (TMLE) is a general strategy that succeeds in constructing asymptotically linear plug-in estimators.

  • In the chapters that follow, we focus on the targeted maximum likelihood estimator and the targeted minimum loss-based estimator, both referred to as TMLE.

References

Polley, Eric C, and Mark J van der Laan. 2010. “Super Learner in Prediction.” bepress.

van der Laan, Mark J, and Sandrine Dudoit. 2003. “Unified Cross-Validation Methodology for Selection Among Estimators and a General Cross-Validated Adaptive Epsilon-Net Estimator: Finite Sample Oracle Inequalities and Examples.” bepress.

van der Laan, Mark J, Eric C Polley, and Alan E Hubbard. 2007. “Super Learner.” Statistical Applications in Genetics and Molecular Biology 6 (1).

van der Vaart, Aad W, Sandrine Dudoit, and Mark J van der Laan. 2006. “Oracle Inequalities for Multi-Fold Cross Validation.” Statistics & Decisions 24 (3). Oldenbourg Wissenschaftsverlag: 351–71.