Study guide: Convolutional Neural Networks

Course Outline

  1. Machine Learning Tasks
  2. Deep Learning and Feature Learning
  3. Neural Networks and MLPs
  4. Training and Backpropagation
  5. CNNs and Convolution Filtering
  6. Gradient Orientation and HOG
  7. CNN Topology and Convolution Layers
  8. Pooling and Overfitting Prevention
  9. Hierarchical Features and Transfer Learning
  10. CNN Dimensions and Applications
  11. ImageNet and CNN Architectures

1. Machine Learning Tasks

Key Concepts & Definitions

  • Machine learning: Designs algorithms that learn to solve a problem from data and improve their performance on a task through experience.
  • Regression: A supervised learning task that models and predicts continuous numeric variables using labeled data with a numeric target value for each observation.
  • Classification: A supervised learning task that models and predicts categorical variables, such as classes or class probabilities.
  • Clustering: An unsupervised learning task that finds natural groupings of observations based on the dataset’s inherent structure.

Essential Points

  • Examples of regression include:

    • predicting real-estate prices
    • predicting stock price movements
    • predicting student test scores
  • Examples of classification include:

    • predicting email spam
    • predicting financial fraud
    • predicting student letter grades
  • Examples of clustering include:

    • customer segmentation
    • grouping similar e-commerce items
    • social network analysis

2. Deep Learning and Feature Learning

Key Concepts & Definitions

  • Deep learning: Composes nonlinear transformations of data to learn useful representations, or features, directly from the data.

β˜… Must-know

πŸ“Œ Traditional machine learning uses fixed feature extractors, whereas deep learning uses trainable feature extractors learned from data.

  • An end-to-end recognition system maps raw pixels directly to labels by composing simple nonlinear functions with learnable parameters.

Further detail

  • Traditional feature extraction can require:

    • expert knowledge
    • time-consuming hand-tuning
    • problem-specific choices
  • Deep learning methods can be supervised or unsupervised.

Memory Hook

Fixed traditional features versus trainable features learned from data

3. Neural Networks and MLPs

Key Concepts & Definitions

  • Neural network: A set of connected neurons with randomly initialized weights and nonlinear activation functions, optimized on training data to minimize prediction error.
  • Multilayer perceptron: A non-convolutional neural-network architecture built by composing layers of simple models.

β˜… Must-know

πŸ“ Formula - A single-layer neuron computes z=βˆ‘iWixi+bz = \sum_i W_i x_i + b and outputs a=f(z)a = f(z), where the weights are learned and f is a nonlinear function.

Further detail

  • Neural networks can address:

    • supervised learning
    • semi-supervised learning
    • unsupervised learning
  • A neural network maps inputs x to outputs y = fΞΈ(x), and the output type determines whether the task is classification or regression.

  • Sigmoid and tanh gradients saturate and become close to zero as their outputs approach extreme values.

  • Leaky ReLU helps address the dead-neuron problem of ReLU when its input is negative.

4. Training and Backpropagation

Key Concepts & Definitions

  • Backpropagation: Propagates prediction error backward through a multilayer feedforward neural network to calculate each weight’s contribution to the error and support weight adjustment.

β˜… Must-know

  • πŸ”„ Neural-network training proceeds by: initializing weights, computing outputs from left to right for each training example, measuring prediction error with a loss function, propagating error backward through the network, adjusting weights using gradient descent

πŸ“ Formula - Gradient descent updates a weight according to winew=wioldβˆ’Ξ·βˆ‚Eβˆ‚wiw_i^{\mathrm{new}} = w_i^{\mathrm{old}} - \eta \frac{\partial E}{\partial w_i}, where Ξ· is the learning rate and E is the error.

Further detail

  • Training repeats weight adjustments across randomly selected training instances thousands or possibly millions of times.

  • Backpropagation can theoretically perform any input-output mapping and can learn linearly inseparable problems.

Memory Hook

Forward pass β†’ measure error β†’ backward pass β†’ adjust weights

5. CNNs and Convolution Filtering

Key Concepts & Definitions

  • Convolutional neural network: A feed-forward network that extracts topological properties from images and recognizes visual patterns directly from pixel images with minimal preprocessing.
  • Convolution: Processes a pixel using information from its neighboring pixels.

β˜… Must-know

  • Yann LeCun and collaborators introduced convolutional neural networks trained by backpropagation in 1989. - Yann LeCun and Yoshua Bengio, 1995

  • Image filtering applies a convolution kernel by sliding a window across the image and approximating the result from a neighborhood of pixels.

  • The identity filter with kernel [000010000]\begin{bmatrix}0&0&0\\0&1&0\\0&0&0\end{bmatrix} leaves the image unchanged.

  • The filter with kernel [000100000]\begin{bmatrix}0&0&0\\1&0&0\\0&0&0\end{bmatrix} shifts the image left by one pixel.

  • The Sobel kernel [βˆ’101βˆ’202βˆ’101]\begin{bmatrix}-1&0&1\\-2&0&2\\-1&0&1\end{bmatrix} detects vertical edges by their absolute response.

  • The Sobel kernel [βˆ’1βˆ’2βˆ’1000121]\begin{bmatrix}-1&-2&-1\\0&0&0\\1&2&1\end{bmatrix} detects horizontal edges by their absolute response.

Further detail

  • Convolutional neural networks are a special kind of multilayer neural network trained with a version of backpropagation.

  • In the Fourier domain, multiplication is equivalent to convolution in the spatial domain.

  • Convolution kernels are typically 3Γ—3, 5Γ—5, or 7Γ—7.

  • Applying the one-dimensional kernel [2, 3, 1] to the sequence [1, 2, 3, 4, 5, 6, 7] produces the response [11, 17, 23, 29, 35].

  • The 2D convolution material presents a convolution operation applied across two dimensions of an image.

Memory Hook

Slide a small kernel window across an image to build a response

6. Gradient Orientation and HOG

β˜… Must-know

  • Horizontal and vertical Sobel gradient kernels can be convolved with an image to detect orientations.

  • In a histogram of oriented gradients (HOG), gradient norm and gradient direction are calculated, and a threshold on the norm detects points of interest. - Dalal, 2005

Further detail

  • A histogram of directions is a possible descriptor for detected orientations.

7. CNN Topology and Convolution Layers

Key Concepts & Definitions

  • Feature map: the output produced by convolving one feature

β˜… Must-know

  • CNNs reuse the same kernel across image locations because features such as edges can occur anywhere in the image.

  • πŸ”„ A CNN may build image features in this order:

    1. Learn edges from raw pixels
    2. Combine edges into simple shapes
    3. Use shapes to detect higher-level features such as facial shapes
  • A CNN is a classic deep neural network with local connections and shared weights, allowing the same feature to be detected at different image positions and drastically reducing the number of free parameters.

πŸ“Œ Convolution cannot be performed at border pixels without padding; padding enlarges the original image, for example by adding zeros, and the feature-map size depends on the padding and filter size.

Further detail

  • CNN topology reduces connections to local regions; for a 1000Γ—1000 image with 1 million hidden units, a 10Γ—10 filter would require 10 million parameters in the stated example.

  • For a single-channel 1000Γ—1000 image, 100 shared convolutional filters of size 10Γ—10 require 10,000 weight parameters, excluding biases.

  • For an input with three RGB channels, a separate kΓ—kk\times k filter is applied to each channel.

  • A CNN architecture combines feature-extraction layers, convolution layers, pooling or subsampling layers, and fully connected layers.

  • The weights of a convolutional layer correspond to its learned filters, and its activations form a two-dimensional feature map representing the convolution response.

πŸ“Œ Stride is the pixel step between convolution operations; stride is usually identical in width and height, may differ, and values greater than 1 reduce feature-map size.

  • A 3Γ—3 kernel applied to a 5Γ—5 input matrix with a 2Γ—2 stride produces a 2Γ—2 output.

  • Convolution-layer settings include:

    • Filter size
    • Number of filters
    • Stride
    • Padding
    • Non-linearity
    • Bias
    • Regularization

πŸ“ Formula - With stride 1 and half padding, the output has FF feature maps of size WΓ—HW\times H; the stated weight count for four convolutional layers with 64 feature maps and 3Γ—3 kernels each is (3Γ—3)Γ—64Γ—4=2,304(3\times3)\times64\times4=2{,}304.

Memory Hook

Imagine a shared filter sliding across an image, detecting the same feature wherever it appears.

8. Pooling and Overfitting Prevention

Key Concepts & Definitions

  • Overfitting: occurs when a learned hypothesis fits training data very well, including noise or outliers, but fails to generalize to new test examples

β˜… Must-know

  • Pooling downsamples convolutional outputs by reducing the spatial resolution of each feature map, yielding some shift and distortion invariance.
  • Dropout randomly removes units and their connections during training to encourage robust features, reduce complex co-adaptations, and alleviate overfitting, especially in dense layers.

Further detail

  • Max pooling and average pooling are common pooling types, and the illustrated setup uses 2Γ—2 filters with stride 2.

πŸ“Œ Dropout is applied during training, not during testing.

  • Data augmentation methods include:
    • Small rotations
    • Translation
    • Scaling
    • Flipping
    • Brightness changes
    • Noise

Memory Hook

Pooling shrinks feature maps; dropout removes units during training.

9. Hierarchical Features and Transfer Learning

β˜… Must-know

  • A pre-trained network is a saved network previously trained on a large dataset, whose learned spatial feature hierarchy can serve as a generic model for different computer-vision problems.

πŸ“Œ Feature extraction reuses representations from a previous network to produce features for new samples and trains a new classifier from scratch, whereas fine-tuning unfreezes selected top layers and trains them jointly with the newly added model part.

  • Earlier convolutional layers extract local, generic features such as edges, colors, and textures, while higher layers extract abstract concepts such as a cat ear or dog eye.

Further detail

  • A fully connected layer has a one-dimensional structure in which each node connects to every input node and computes a weighted sum of all input nodes.

  • A fully connected layer uses high-level features from previous layers to classify an input pattern.

  • Deeper CNNs stack convolution and pooling layers, and the course states that experiments showed better results with deeper replication of these layers.

πŸ“Œ When a new dataset differs greatly from the original training dataset, using only the first few layers for feature extraction may be preferable to using the entire convolutional base.

  • A CNN pre-trained on ImageNet can be used as fixed features or as an initialization, then further trained on new data to fine-tune its weights.

Memory Hook

Pixels β†’ edges β†’ parts β†’ object-level features.

10. CNN Dimensions and Applications

β˜… Must-know

  • Conv2D is generally used on image data and its kernel slides along two dimensions.
  • Conv3D kernels slide in three dimensions and are used mainly with 3D images such as MRI and CT data and with video sequences.

Further detail

  • For a Conv2D image input, input_shape (128, 128, 3) represents height, width, and depth, while kernel_size (3, 3) gives kernel height and width and the kernel depth matches the image depth.

  • Conv3D activation maps preserve temporal or volumetric context, supporting analysis of frame sequences for action recognition and medical imaging such as MRI and CT.

  • For Conv3D, input_shape (128, 128, 128, 3) has four dimensions, and kernel_size (3, 3, 3) specifies kernel height, width, and depth while the fourth dimension matches the color channel.

Memory Hook

Conv2D slides in two dimensions; Conv3D slides in three.

11. ImageNet and CNN Architectures

β˜… Must-know

  • ImageNet is described as containing 1000 classes and 1.2 million labeled photos, with human top-5 error of approximately 5%.

Further detail

  • LeNet5 achieved 99.2% classification accuracy on MNIST for handwritten-character recognition; the source attributes it to Yann LeCun et al. in 1998, in β€œGradient-Based Learning Applied to Document Recognition”. - Yann LeCun, et al.
  • AlexNet, attributed to Alex Krizhevsky et al., was the largest neural network on ImageNet subsets at publication and was trained for six days on two Nvidia Geforce GTX 580 GPUs. - Alex Krizhevsky, et al., ImageNet Classification with Deep Convolutional Neural Networks

  • VGG-16 was presented by Karen Simonyan and Andrew Zisserman in 2014, with the approach of improving CNN efficiency by stacking more layers. - Very Deep Convolutional Networks for Large-Scale Image Recognition

  • Inception-v1 (GoogLeNet) is attributed to Christian Szegedy et al. in β€œGoing Deeper with Convolutions” (2014). - Christian Szegedy, et al.

  • Inception-v3 is attributed to Christian Szegedy et al. in β€œRethinking the Inception Architecture for Computer Vision” (2015). - Christian Szegedy, et al.

  • ResNet-50 is attributed to Kaiming et al. in β€œDeep Residual Learning for Image Recognition” (2015). - Kaiming, et al.

Test yourself

Take a 24-question practice quiz on Convolutional Neural Networks, with every answer explained.

1. Which statements correctly describe machine learning and regression?

2. A project predicts whether each email is spam and estimates house prices; which statements classify these tasks correctly?

Take the quiz β†’

Study with flashcards

Memorize the key concepts of Convolutional Neural Networks with 82 flashcards.

What does machine learning algorithms improve through experience?

Their performance on a task.

What kind of data does regression use to predict continuous numeric variables?

Labeled data with a numeric target for each observation.

What kind of variables does classification predict?

Categorical variables, such as classes or class probabilities.

See the flashcards β†’

Related classes

Make your own study guides

Upload your class notes and Revizly writes a study guide, a quiz and flashcards in about a minute.

Study guide maker