sl3 package provides a modern framework for machine learning. This includes the Super Learner algorithm (van der Laan, Polley, and Hubbard 2007), a method for performing stacked regressions (Breiman 1996), combined with covariate screening and cross-validation.
sl3 uses an Object Oriented Programming (OOP) approach and leverages
R6 classes to define both Tasks (machine learning problems) and Learners (machine learning algorithms that attempt to solve those problems) in a way that is both flexible and extensible. The design of
sl3 owes a lot to the
mlr packages, which also provide unified frameworks for Super Learning and machine learning, respectively.
Throughout this vignette, we use data from the Collaborative Perinatal Project (CPP) to illustrate the features of
sl3 as well as its proper usage. For convenience, we’ve included an imputed version of this dataset in the
sl3 package. Below, we load some useful packages, load the
cpp_imputed dataset, and define the variables (columns) from the data set we’re interested in:
set.seed(49753) # packages we'll be using library(data.table) library(SuperLearner) library(origami) library(sl3) # load example data set data(cpp_imputed) # here are the covariates we are interested in and, of course, the outcome covars <- c("apgar1", "apgar5", "parity", "gagebrth", "mage", "meducyrs", "sexn") outcome <- "haz"
As mentioned above,
sl3 is designed using basic OOP principles and the
R6 OOP framework. While we’ve tried to make it easy to use
sl3 without worrying much about OOP, it is helpful to have some intuition about how
sl3 is structured. In this section, we briefly outline some key concepts from OOP. Readers familiar with OOP basics are invited to skip this section. The key concept of OOP is that of an object, a collection of data and functions that corresponds to some conceptual unit. Objects have two main types of elements, fields, which can be thought of as nouns, are information about an object, and methods, which can be thought of as verbs, are actions an object can perform. Objects are members of classes, which define what those specific fields and methods are. Classes can inherit elements from other classes (sometimes called base classes) – accordingly, classes that are similar, but not exactly the same, can share some parts of their definitions.
Many different implementations of OOP exist, with variations in how these concepts are implemented and used. R has several different implementations, including S3, S4, reference classes, and R6.
sl3 uses the
R6 implementation. In R6, methods and fields of a class object are accessed using the
$ operator. The next section explains how these concepts are used in
sl3 to model machine learning problems and algorithms.
sl3_Task class defines machine learning problems. An
sl3_Task object keeps track of the task data, as well as what variables play what roles in the machine learning problem. We can see an example of that here, using the
cpp dataset described above:
We use the
make_sl3_Task method to create a new
task. Here, we specified the underlying data,
cpp_imputed, and vectors indicating which varaibles to use as covariates and outcomes.
Let’s take a look at this object:
## A sl3 Task with 1441 obs and these nodes: ## $covariates ##  "apgar1" "apgar5" "parity" "gagebrth" "mage" "meducyrs" "sexn" ## ## $outcome ##  "haz" ## ## $id ## NULL ## ## $weights ## NULL ## ## $offset ## NULL
In addition to the simple usage demonstrated above,
make_sl3_Task supports a range of options in order to facilitate the proper articulation of more advanced specifics potentially informative of the machine learning problem of interest. For example, we can specify the
offset nodes listed above. These additional features are documented in the help for
Lrnr_base is the base class for defining machine learning algorithms, as well as fits for those algorithms to particular
sl3_Tasks. Different machine learning algorithms are defined in classes that inherit from
Lrnr_base. For instance, the
Lrnr_glm class inherits from
Lrnr_base, and defines a learner that fits generalized linear models. We will use the term learners to refer to the family of classes that inherit from
Lrnr_base. Learner objects can be constructed from their class definitions using the
Because all learners inherit from
Lrnr_base, they have many features in common, and can be used interchangeably. All learners define three main methods:
chain. The first,
train, takes a
sl3_task object, and returns a learner_fit, which has the same class as the learner that was trained:
##  TRUE
Here, we fit the learner to the CPP task we defined above. Both
lrnr_glm_fit are objects of class
Lrnr_glm, although the former defines a learner and the latter defines a fit of that learner. We can distiguish between the learners and learner fits using the
is_trained field, which is true for fits but not for learners.
Now that we’ve fit a learner, we can generate predictions using the
##  0.36298498 0.36298498 0.25993072 0.25993072 0.25993072 0.05680264
Here, we specified
task as the task for which we wanted to generate predictions. If we had omitted this, we would have gotten the same predictions because
predict defaults to using the task provided to
train (called the training task). Alternatively, we could have provided a different task for which we want to generate predictions.
The final important learner method,
chain, will be discussed below, in the section on learner composition. As with
sl3_Task, learners have a variety of fields and methods we haven’t discussed here. More information on these is available in the help for
Learners have properties that indicate what features they support. You can use
sl3_list_properties to get a list of all properties supported by at least one learner. You can then use
sl3_list_learners to find learners supporting any set of properties. For example:
##  "binomial" "categorical" "continuous" ##  "cv" "density" "ids" ##  "multivariate_outcome" "offset" "preprocessing" ##  "sampling" "timeseries" "weights" ##  "wrapper"
##  "Lrnr_glm" "Lrnr_glm_fast" "Lrnr_h2o_glm" "Lrnr_h2o_grid" ##  "Lrnr_mean" "Lrnr_optim" "Lrnr_solnp" "Lrnr_xgboost"
The list of supported learners is currently somewhat limited. Despite current limitations, some learners not yet supported natively in
sl3 can be used via their corresponding wrappers in the
SuperLearner wrappers, screeners, and methods can all be used as
sl3 learners via
Lrnr_pkg_SuperLearner_method respectively. To learn more about
SuperLearner wrappers, screeners, and methods, consult the documentation provided with that R package. Here’s an example of defining a
sl3 learner that uses the
SL.glmnet wrapper from
In most cases, using these wrappers will not be as efficient as their native
sl3 counterparts. If your favorite learner is missing from
sl3, please consider adding it by following the “Defining New Learners” vignette.
In general, learners can be instantiated without providing any additional parameters. We’ve tried to provide sensible defaults for each learner; however, if you would like to modify the learners’ behavior, you may do so by instantiating learners with different parameters.
sl3 Learners support some common parameters that work with all learners for which they are applicable:
covariates: subsets covariates before fitting. This allows different learners to be fit to the same task with different covariate subsets.
outcome_type: overrides the
task$outcome_type. This allows different learners to be fit to the same task with different outcome_types.
...: abitrary parameters typically passed directly to the internal learner method. The documentation for each learner will direct to the appropriate function documentation for the learner method.
sl3 defines two special learners,
Stack, that allow learners to be composed in a flexible manner.
A pipeline is a set of learners to be fit sequentially, where the fit from one learner is used to define the task for the next learner. There are many ways in which a learner can define the task for the downstream learner. The
chain method defined by learners defines how this will work. Let’s look at the example of pre-screening variables. For now, we’ll rely on a screener from the
SuperLearner package, although native
sl3 screening algorithms will be implemented soon.
Below, we generate a screener object based on the
screen.corP and fit it to our task. Inspecting the fit, we see that it selected a subset of covariates:
##  "Lrnr_pkg_SuperLearner_screener_screen.corP" ## $selected ##  "parity" "gagebrth"
chain may be called on this learner fit to define a downstream task:
## A sl3 Task with 1441 obs and these nodes: ## $covariates ##  "parity" "gagebrth" ## ## $outcome ##  "haz" ## ## $id ## NULL ## ## $weights ## NULL ## ## $offset ## NULL
predict, we can omit a task from the call to
chain, in which case the call defaults to using the same task that was used for training. We can see that the chained task reduces the covariates to the subset selected by the screener. We can fit this new task using the
lrnr_glm we defined above:
##  0.38084472 0.38084472 0.29887623 0.29887623 0.29887623 -0.00987784
Pipeline class automates this process. It takes an arbitrary number of learners and fits them sequentially, training and chaining each one in turn. Since
Pipeline is a learner like any other, it shares the same interface. We can define a pipeline using
make_learner, and use
predict just as we did before:
##  0.38084472 0.38084472 0.29887623 0.29887623 0.29887623 -0.00987784
We see that the pipeline returns the same predictions as manually training
glm on the chained task from the screening learner.
We can visualize the pipeline we defined above:
Stacks combine multiple learners.
Stacks train learners simultaneously, so that their predictions can be either combined or compared. Again,
Stack is just a special learner and so has the same interface as all other learners:
## Lrnr_glm_TRUE ## 1: 0.36298498 ## 2: 0.36298498 ## 3: 0.25993072 ## 4: 0.25993072 ## 5: 0.25993072 ## 6: 0.05680264 ## Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Lrnr_glm_TRUE) ## 1: 0.38084472 ## 2: 0.38084472 ## 3: 0.29887623 ## 4: 0.29887623 ## 5: 0.29887623 ## 6: -0.00987784
Above, we’ve defined and fit a
stack comprised of a simple
glm learner as well as a pipeline that combines a screening algorithm with that same learner. We could have included any abitrary set of learners and pipelines, the latter of which are themselves just learners. We can see that the
predict method now returns a matrix, with a column for each learner included in the stack.
We can visualize the stack:
We see one “branch” for each learner in the stack.
Having defined a stack, we might want to compare the performance of learners in the stack, which we may do using cross-validation. The
Lrnr_cv learner wraps another learner and performs training and prediction in a cross-validated fashion, using separate training and validation splits as defined by
Below, we define a new
Lrnr_cv object based on the previously defined
stack and train it and generate predictions on the validation set:
We can also use the special
cv_risk to estimate cross-validated risk values:
## learner ## 1: Lrnr_glm_TRUE ## 2: Pipeline(Lrnr_pkg_SuperLearner_screener_screen.corP->Lrnr_glm_TRUE) ## coefficients mean_risk SE_risk fold_SD fold_min_risk fold_max_risk ## 1: NA 1.600411 0.1042245 0.3521137 1.062662 2.139263 ## 2: NA 1.601875 0.1032100 0.3419678 1.066897 2.108442
In this example, we don’t see much difference between the two learners, suggesting the addition of the screening step in the pipeline learner didn’t improve performance much.
We can combine all of the above elements,
Stacks, and cross-validation using
Lrnr_cv, to easily define a Super Learner. The Super Learner algorithm works by fitting a “meta-learner”, which combines predictions from multiple stacked learners. It does this while avoiding overfitting by training the meta-learner on validation-set predictions in a manner that is cross-validated. Using some of the objects we defined in the above examples, this becomes a very simple operation:
Here, we used a special learner,
Lrnr_nnls, for the meta-learning step. This fits a non-negative least squares meta-learner. It is important to note that any learner can be used as a meta-learner.
The Super Learner finally produced is defined as a pipeline with the learner stack trained on the full data and the meta-learner trained on the validation-set predictions. Below, we use a special behavior of pipelines: if all objects passed to a pipeline are learner fits (i.e.,
TRUE), the result will also be a fit:
##  0.34125504 0.34125504 0.25355166 0.25355166 0.25355166 0.02901442
A Super Learner may be fit in a more streamlined manner using the
Lrnr_sl learner. For simplicity, we will use the same set of learners and meta-learning algorithm as we did before:
##  0.34125504 0.34125504 0.25355166 0.25355166 0.25355166 0.02901442
We can see that this generates the same predictions as the more hands-on definition above.
Fitting a Super Learner is composed of many different training and prediction steps, as the procedure requires that the learners in the stack and the meta-learner be fit on cross-validation folds and on the full data. For large datasets, this can be extremely time-consuming. To alleviate this complication, we’ve developed a specialized parallelization framework
delayed that parallelizes across these tasks in a way that takes into account their inter-dependent nature. Consider a Super Learner with three learners:
We can plot the network of tasks required to train this Super Learner:
delayed then allows us to parallelize the procedure across these tasks using the
future package. For more information on specifying
plans for parallelization, see the documentation of the
future package. Performance comparisons can be found in the “SuperLearner Benchmarks” vignette that accompanies this package. This feature is currently experimental and hasn’t yet been throughly tested on a range of parallel backends.
## R version 3.6.3 (2020-02-29) ## Platform: x86_64-pc-linux-gnu (64-bit) ## Running under: Ubuntu 18.04.4 LTS ## ## Matrix products: default ## BLAS: /usr/lib/x86_64-linux-gnu/openblas/libblas.so.3 ## LAPACK: /usr/lib/x86_64-linux-gnu/libopenblasp-r0.2.20.so ## ## locale: ##  LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C ##  LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8 ##  LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8 ##  LC_PAPER=en_US.UTF-8 LC_NAME=C ##  LC_ADDRESS=C LC_TELEPHONE=C ##  LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C ## ## attached base packages: ##  stats graphics grDevices utils datasets methods base ## ## other attached packages: ##  origami_1.0.3 SuperLearner_2.0-26 nnls_1.4 ##  data.table_1.12.8 sl3_1.3.7 ## ## loaded via a namespace (and not attached): ##  progress_1.2.2 shape_1.4.4 tidyselect_1.0.0 ##  xfun_0.12 purrr_0.3.3 listenv_0.8.0 ##  lattice_0.20-40 colorspace_1.4-1 vctrs_0.2.3 ##  htmltools_0.4.0 yaml_2.2.1 rlang_0.4.5 ##  pkgdown_1.4.1 pillar_1.4.3 glue_1.3.1 ##  uuid_0.1-4 foreach_1.4.8 lifecycle_0.1.0 ##  stringr_1.4.0 munsell_0.5.0 gtable_0.3.0 ##  visNetwork_2.0.9 future_1.16.0 htmlwidgets_1.5.1 ##  codetools_0.2-16 memoise_1.1.0 evaluate_0.14 ##  knitr_1.28 parallel_3.6.3 rstackdeque_1.1.1 ##  Rcpp_1.0.3 delayed_0.3.0 backports_1.1.5 ##  scales_1.1.0 checkmate_2.0.0 desc_1.2.0 ##  imputeMissings_0.0.3 jsonlite_1.6.1 abind_1.4-5 ##  fs_1.3.1 ggplot2_3.2.1 hms_0.5.3 ##  digest_0.6.25 stringi_1.4.6 BBmisc_1.11 ##  dplyr_0.8.4 rprojroot_1.3-2 grid_3.6.3 ##  tools_3.6.3 magrittr_1.5 glmnet_3.0-2 ##  lazyeval_0.2.2 tibble_2.1.3 randomForest_4.6-14 ##  future.apply_1.4.0 crayon_1.3.4 pkgconfig_2.0.3 ##  Matrix_1.2-18 MASS_7.3-51.5 prettyunits_1.1.1 ##  iterators_1.0.12 assertthat_0.2.1 rmarkdown_2.1 ##  R6_2.4.1 globals_0.12.5 igraph_126.96.36.199 ##  compiler_3.6.3