(1) Data as a random variable with a probability distribution, \(O \sim P_0\)

The data set we’re confronted with is the result of an experiment and we can view the data as a random variable, \(O\), because if we repeat the experiment we would have a different realization of this experiment. In particular, if we repeat the experiment many times we could learn the probability distribution, \(P_0\), of our data. So, the observed data \(O\) with probability distribution \(P_0\) are \(n\) independent identically distributed (i.i.d.) observations of the random variable \(O; O_1, \ldots, O_n\). Note that while not all data are i.i.d., there are ways to handle non-i.i.d. data, such as establishing conditional independence, stratifying data to create sets of identically distributed data, etc. It is crucial that researchers be absolutely clear about what they actually know about the data-generating distribution for a given problem of interest. Unfortunately, communication between statisticians and researchers is often fraught with misinterpretation. The roadmap provides a mechanism by which to ensure clear communication between research and statistician – it truly helps with this communication!

(2) The statistical model \(\mathcal{M}\) such that \(P_0 \in \mathcal{M}\)

The statistical model \(\mathcal{M}\) is defined by the question we asked at the end of . It is defined as the set of possible probability distributions for our observed data. Often \(\mathcal{M}\) is very large (possibly infinite-dimensional), to reflect the fact that statistical knowledge is limited. In the case that \(\mathcal{M}\) is infinite-dimensional, we deem this a nonparametric statistical model.

Alternatively, if the probability distribution of the data at hand is described by a finite number of parameters, then the statistical model is parametric. In this case, we prescribe to the belief that the random variable \(O\) being observed has, e.g., a normal distribution with mean \(\mu\) and variance \(\sigma^2\). More formally, a parametric model may be defined \[\begin{equation*} \mathcal{M} = \{P_{\theta} : \theta \in \mathcal{R}^d \} \end{equation*}\]

Sadly, the assumption that the data-generating distribution has a specific, parametric forms is all-too-common, even when such is a leap of faith. This practice of oversimplification in the current culture of data analysis typically derails any attempt at trying to answer the scientific question at hand; alas, such statements as the ever-popular quip of Box that “All models are wrong but some are useful,” encourage the data analyst to make arbitrary choices even when that often force significant differences in answers to the same estimation problem. The Targeted Learning paradigm does not suffer from this bias since it defines the statistical model through a representation of the true data-generating distribution corresponding to the observed data.

Now, on to Step 3: ``What are we trying to learn from the data?"

(3) The statistical target parameter \(\Psi\) and estimand \(\Psi(P_0)\)

The statistical target parameter, \(\Psi\), is defined as a mapping from the statistical model, \(\mathcal{M}\), to the parameter space (i.e., a real number) \(\mathcal{R}\). That is, \(\Psi: \mathcal{M}\rightarrow\mathbb{R}\). The target parameter may be seen as a representation of the quantity that we wish to learn from the data, the answer to a well-specified (often causal) question of interest. In contrast to purely statistical target parameters, causal target parameters require identification from the observed data, based on causal models that include several untestable assumptions, described in more detail in the section on causal target parameters.

For a simple example, consider a data set which contains observations of a survival time on every subject, for which our question of interest is “What’s the probability that someone lives longer than five years?” We have, \[\begin{equation*} \Psi(P_0) = \mathbb{P}(O > 5) \end{equation*}\]

This answer to this question is the estimand, \(\Psi(P_0)\), which is the quantity we’re trying to learn from the data. Once we have defined \(O\), \(\mathcal{M}\) and \(\Psi(P_0)\) we have formally defined the statistical estimation problem.

(4) The estimator \(\hat{\Psi}\) and estimate \(\hat{\Psi}(P_n)\)

To obtain a good approximation of the estimand, we need an estimator, an a priori-specified algorithm defined as a mapping from the set of possible empirical distributions, \(P_n\), which live in a non-parametric statistical model, \(\mathcal{M}_{NP}\) (\(P_n \in \mathcal{M}_{NP}\)), to the parameter space of the parameter of interest. That is, \(\hat{\Psi} : \mathcal{M}_{NP} \rightarrow \mathbb{R}^d\). The estimator is a function that takes as input the observed data, a realization of \(P_n\), and gives as output a value in the parameter space, which is the estimate, \(\hat{\Psi}(P_n)\).

Where the estimator may be seen as an operator that maps the observed data and corresponding empirical distribution to a value in the parameter space, the numerical output that produced such a function is the estimate. Thus, it is an element of the parameter space based on the empirical probability distribution of the observed data. If we plug in a realization of \(P_n\) (based on a sample size \(n\) of the random variable \(O\)), we get back an estimate \(\hat{\Psi}(P_n)\) of the true parameter value \(\Psi(P_0)\).

In order to quantify the uncertainty in our estimate of the target parameter (i.e., to construct statistical inference), an understanding of the sampling distribution of our estimator will be necessary. This brings us to Step 5.

(5) A measure of uncertainty for the estimate \(\hat{\Psi}(P_n)\)

Since the estimator \(\hat{\Psi}\) is a function of the empirical distribution \(P_n\), the estimator itself is a random variable with a sampling distribution. So, if we repeat the experiment of drawing \(n\) observations we would every time end up with a different realization of our estimate and our estimator has a sampling distribution. The sampling distribution of an estimator can be theoretically validated to be approximately normally distributed by a Central Limit Theorem (CLT).

A class of Central Limit Theorems (CLTs) are statements regarding the convergence of the sampling distribution of an estimator to a normal distribution. In general, we will construct estimators whose limit sampling distributions may be shown to be approximately normal distributed as sample size increases. For large enough \(n\) we have, \[\begin{equation*} \hat{\Psi}(P_n) \sim N \left(\Psi(P_0), \frac{\sigma^2}{n}\right), \end{equation*}\] permitting statistical inference. Now, we can proceed to quantify the uncertainty of our chosen estimator by construction of hypothesis tests and confidence intervals. For example, we may construct a confidence interval at level \((1 - \alpha)\) for our estimand, \(\Psi(P_0)\): \[\begin{equation*} \hat{\Psi}(P_n) \pm z_{1 - \frac{\alpha}{2}} \left(\frac{\sigma}{\sqrt{n}}\right), \end{equation*}\] where \(z_{1 - \frac{\alpha}{2}}\) is the \((1 - \frac{\alpha}{2})^\text{th}\) quantile of the standard normal distribution. Often, we will be interested in constructing 95% confidence intervals, corresponding to mass \(\alpha = 0.05\) in either tail of the limit distribution; thus, we will typically take \(z_{1 - \frac{\alpha}{2}} \approx 1.96\).

Note: we will typically have to estimate the standard error, \(\frac{\sigma}{\sqrt{n}}\).

A 95% confidence interval means that if we were to take 100 different samples of size \(n\) and compute a 95% confidence interval for each sample then approximately 95 of the 100 confidence intervals would contain the estimand, \(\Psi(P_0)\). More practically, this means that there is a 95% probability (or 95% confidence) that the confidence interval procedure will contain the true estimand. However, any single estimated confidence interval either will contain the true estimand or will not.