Study guide: Statistical Learning Basic Concepts

Course Outline

  1. Learning from Examples
  2. Data Spaces and Distributions
  3. Data Models and Noise
  4. Loss and Expected Risk
  5. Learning Algorithms and Generalization
  6. Consistency and Sample Complexity
  7. Fitting Stability and Regularization

1. Learning from Examples

Key Concepts & Definitions

  • Machine learning: Systems trained from data rather than explicitly programmed.
  • Supervised learning: Supervised learning seeks an underlying input-output relation that predicts an output y for a new input x.

Essential Points

  • A training set consists of n input-output examples, written S={(x1,y1),…,(xn,yn)}S = \{(x_1,y_1),\ldots,(x_n,y_n)\}.

2. Data Spaces and Distributions

Key Concepts & Definitions

  • Data space: The product space XƗYX \times Y of input and output spaces.

ā˜… Must-know

šŸ“Œ The data are assumed to be independently and identically sampled from a fixed but unknown distribution p(x,y).

  • Inputs belong to an input space XāŠ†RDX \subseteq \mathbb{R}^D, while outputs belong to an output space Y, typically a subset of the real numbers.

  • Regression uses an output space YāŠ†RY \subseteq \mathbb{R}, binary classification uses Y={āˆ’1,1}Y=\{-1{,}1\}, and multiclass classification uses Y={1,2,…,T}Y=\{1{,}2,\ldots,T\}.

Further detail

šŸ“ Formula - The joint distribution factorizes as p(x,y)=pX(x)p(y∣x)p(x,y)=p_X(x)p(y\mid x).

3. Data Models and Noise

ā˜… Must-know

  • The conditional distribution p(y|x) models uncertainty or noise in the output for a given input, while the marginal p_X(x) models uncertainty in input sampling.

šŸ“ Formula - In regression, the data model is y=fāˆ—(x)+εy=f^*(x)+\varepsilon, where f* is a fixed unknown regression function and ε is random noise.

Further detail

  • The regression noise may follow a Gaussian distribution N(0,σI)\mathcal{N}(0,\sigma I), with σ∈[0,āˆž)\sigma\in[0,\infty).

  • Noiseless classification has p(1∣x)∈{0,1}p(1\mid x)\in\{0{,}1\} for every x in X.

šŸ“ Formula - In binary classification, p(1∣x)=1āˆ’p(āˆ’1∣x)p(1\mid x)=1-p(-1\mid x) for every x.

4. Loss and Expected Risk

Key Concepts & Definitions

  • Loss function: A loss function maps an actual and predicted output to a nonnegative point-wise error, ā„“:YƗY→[0,āˆž)\ell:Y\times Y\to[0,\infty).
  • Target function: Given a loss and a data distribution, the function fāˆ—:X→Yf^*:X\to Y that minimizes expected risk.

ā˜… Must-know

šŸ“ Formula - The expected risk of a predictor is E(f)=E[ā„“(y,f(x))]=∫p(x,y)ā„“(y,f(x)) dx dyE(f)=\mathbb{E}[\ell(y,f(x))]=\int p(x,y)\ell(y,f(x))\,dx\,dy.

Further detail

  • Expected risk measures prediction error on past as well as future data.

5. Learning Algorithms and Generalization

Key Concepts & Definitions

  • Learning algorithm: A procedure that maps a training set S to an estimator f_S.
  • Generalization: A property of an estimator that mimics the target function, corresponding to a small excess expected risk.

ā˜… Must-know

  • The target function cannot generally be computed because the distribution p is unknown, so learning seeks to estimate it from data.

Further detail

  • The excess expected risk is E(fS)āˆ’E(fāˆ—)E(f_S)-E(f^*).

6. Consistency and Sample Complexity

Key Concepts & Definitions

  • Consistency: A learning algorithm is consistent if lim⁔nā†’āˆžES[E(fS)āˆ’E(fāˆ—)]=0\lim_{n\to\infty}\mathbb{E}_S[E(f_S)-E(f^*)]=0.
  • Learning rate: A learning rate gives finite-sample information by bounding expected excess risk for a specified accuracy ε when the sample size n is sufficiently large.
  • Sample complexity: The sample complexity n(ε) is a sample-size threshold such that, for n≄n(ε), the expected excess risk is at most ε.

Essential Points

  • The expected excess risk over training sets is ES[E(fS)āˆ’E(fāˆ—)]\mathbb{E}_S[E(f_S)-E(f^*)].

7. Fitting Stability and Regularization

Key Concepts & Definitions

  • Overfitting: An algorithm whose estimator fits the data while being unstable.
  • Oversmoothing: An algorithm whose estimator is stable while disregarding the data.

Essential Points

  • Good learning requires both fitting the data well and remaining stable when the data change slightly.

  • Regularization algorithms use one or more parameters to control the trade-off between data fitting and stability.

Memory Hook

Overfitting fits but is unstable; oversmoothing is stable but disregards the data.

Test yourself

Take a 18-question practice quiz on Statistical Learning Basic Concepts, with every answer explained.

1. A software system improves its performance by learning patterns from examples rather than receiving explicit instructions for each task. Which field studies this approach?

2. A researcher has input-output examples and wants a model that predicts an output for a previously unseen input. What is the goal of this supervised-learning task?

Take the quiz →

Study with flashcards

Memorize the key concepts of Statistical Learning Basic Concepts with 35 flashcards.

What does machine learning study?

Systems trained from data rather than explicitly programmed.

What relation does supervised learning seek to predict outputs for new inputs?

An underlying input-output relation.

In what space do inputs belong, and how is it bounded?

XāŠ†RDX \subseteq \mathbb{R}^D.

See the flashcards →

Related classes

Make your own study guides

Upload your class notes and Revizly writes a study guide, a quiz and flashcards in about a minute.

Study guide maker