Examples of regression include:
Examples of classification include:
Examples of clustering include:
β Must-know
π Traditional machine learning uses fixed feature extractors, whereas deep learning uses trainable feature extractors learned from data.
Further detail
Traditional feature extraction can require:
Deep learning methods can be supervised or unsupervised.
Fixed traditional features versus trainable features learned from data
β Must-know
π Formula - A single-layer neuron computes and outputs , where the weights are learned and f is a nonlinear function.
Further detail
Neural networks can address:
A neural network maps inputs x to outputs y = fΞΈ(x), and the output type determines whether the task is classification or regression.
Sigmoid and tanh gradients saturate and become close to zero as their outputs approach extreme values.
Leaky ReLU helps address the dead-neuron problem of ReLU when its input is negative.
β Must-know
π Formula - Gradient descent updates a weight according to , where Ξ· is the learning rate and E is the error.
Further detail
Training repeats weight adjustments across randomly selected training instances thousands or possibly millions of times.
Backpropagation can theoretically perform any input-output mapping and can learn linearly inseparable problems.
Forward pass β measure error β backward pass β adjust weights
β Must-know
Yann LeCun and collaborators introduced convolutional neural networks trained by backpropagation in 1989. - Yann LeCun and Yoshua Bengio, 1995
Image filtering applies a convolution kernel by sliding a window across the image and approximating the result from a neighborhood of pixels.
The identity filter with kernel leaves the image unchanged.
The filter with kernel shifts the image left by one pixel.
The Sobel kernel detects vertical edges by their absolute response.
The Sobel kernel detects horizontal edges by their absolute response.
Further detail
Convolutional neural networks are a special kind of multilayer neural network trained with a version of backpropagation.
In the Fourier domain, multiplication is equivalent to convolution in the spatial domain.
Convolution kernels are typically 3Γ3, 5Γ5, or 7Γ7.
Applying the one-dimensional kernel [2, 3, 1] to the sequence [1, 2, 3, 4, 5, 6, 7] produces the response [11, 17, 23, 29, 35].
The 2D convolution material presents a convolution operation applied across two dimensions of an image.
Slide a small kernel window across an image to build a response
β Must-know
Horizontal and vertical Sobel gradient kernels can be convolved with an image to detect orientations.
In a histogram of oriented gradients (HOG), gradient norm and gradient direction are calculated, and a threshold on the norm detects points of interest. - Dalal, 2005
Further detail
β Must-know
CNNs reuse the same kernel across image locations because features such as edges can occur anywhere in the image.
π A CNN may build image features in this order:
π Convolution cannot be performed at border pixels without padding; padding enlarges the original image, for example by adding zeros, and the feature-map size depends on the padding and filter size.
Further detail
CNN topology reduces connections to local regions; for a 1000Γ1000 image with 1 million hidden units, a 10Γ10 filter would require 10 million parameters in the stated example.
For a single-channel 1000Γ1000 image, 100 shared convolutional filters of size 10Γ10 require 10,000 weight parameters, excluding biases.
For an input with three RGB channels, a separate filter is applied to each channel.
A CNN architecture combines feature-extraction layers, convolution layers, pooling or subsampling layers, and fully connected layers.
The weights of a convolutional layer correspond to its learned filters, and its activations form a two-dimensional feature map representing the convolution response.
π Stride is the pixel step between convolution operations; stride is usually identical in width and height, may differ, and values greater than 1 reduce feature-map size.
A 3Γ3 kernel applied to a 5Γ5 input matrix with a 2Γ2 stride produces a 2Γ2 output.
Convolution-layer settings include:
π Formula - With stride 1 and half padding, the output has feature maps of size ; the stated weight count for four convolutional layers with 64 feature maps and 3Γ3 kernels each is .
Imagine a shared filter sliding across an image, detecting the same feature wherever it appears.
β Must-know
Further detail
π Dropout is applied during training, not during testing.
Pooling shrinks feature maps; dropout removes units during training.
β Must-know
π Feature extraction reuses representations from a previous network to produce features for new samples and trains a new classifier from scratch, whereas fine-tuning unfreezes selected top layers and trains them jointly with the newly added model part.
Further detail
A fully connected layer has a one-dimensional structure in which each node connects to every input node and computes a weighted sum of all input nodes.
A fully connected layer uses high-level features from previous layers to classify an input pattern.
Deeper CNNs stack convolution and pooling layers, and the course states that experiments showed better results with deeper replication of these layers.
π When a new dataset differs greatly from the original training dataset, using only the first few layers for feature extraction may be preferable to using the entire convolutional base.
Pixels β edges β parts β object-level features.
β Must-know
Further detail
For a Conv2D image input, input_shape (128, 128, 3) represents height, width, and depth, while kernel_size (3, 3) gives kernel height and width and the kernel depth matches the image depth.
Conv3D activation maps preserve temporal or volumetric context, supporting analysis of frame sequences for action recognition and medical imaging such as MRI and CT.
For Conv3D, input_shape (128, 128, 128, 3) has four dimensions, and kernel_size (3, 3, 3) specifies kernel height, width, and depth while the fourth dimension matches the color channel.
Conv2D slides in two dimensions; Conv3D slides in three.
β Must-know
Further detail
AlexNet, attributed to Alex Krizhevsky et al., was the largest neural network on ImageNet subsets at publication and was trained for six days on two Nvidia Geforce GTX 580 GPUs. - Alex Krizhevsky, et al., ImageNet Classification with Deep Convolutional Neural Networks
VGG-16 was presented by Karen Simonyan and Andrew Zisserman in 2014, with the approach of improving CNN efficiency by stacking more layers. - Very Deep Convolutional Networks for Large-Scale Image Recognition
Inception-v1 (GoogLeNet) is attributed to Christian Szegedy et al. in βGoing Deeper with Convolutionsβ (2014). - Christian Szegedy, et al.
Inception-v3 is attributed to Christian Szegedy et al. in βRethinking the Inception Architecture for Computer Visionβ (2015). - Christian Szegedy, et al.
Take a 24-question practice quiz on Convolutional Neural Networks, with every answer explained.
1. Which statements correctly describe machine learning and regression?
2. A project predicts whether each email is spam and estimates house prices; which statements classify these tasks correctly?
Memorize the key concepts of Convolutional Neural Networks with 82 flashcards.
What does machine learning algorithms improve through experience?
Their performance on a task.
What kind of data does regression use to predict continuous numeric variables?
Labeled data with a numeric target for each observation.
What kind of variables does classification predict?
Categorical variables, such as classes or class probabilities.
Upload your class notes and Revizly writes a study guide, a quiz and flashcards in about a minute.
Study guide maker