Introduction to Deep Learning

Teacher Introduction

Together, you'll get a solid theoretical foundation followed by hands-on, applied experience.

Deep Neural Networks – Course Overview

This course introduces deep neural networks and the powerful methods associated with them.

Learning Outcomes

Upon completion of the course, you will be able to:

1. Explain the fundamental concepts and mechanics of deep neural networks
2. Train deep neural networks to solve practical problems
3. Utilize the PyTorch library for model implementation
4. Differentiate between and apply major types of neural networks:
   • Convolutional Neural Networks (CNNs)
   • Recurrent Neural Networks (RNNs)
   • Generative Models
5. Design and implement solutions for a variety of tasks using deep neural networks

Course schedule (Weeks 1-6)

*Subject to change

Course Schedule (Remaining Weeks)

The detailed schedule for these weeks will be announced by Hlynur when he takes over.

Course Logistics

Assessment & Grading

Prerequisites

Collaboration Policy

Contact Information

Before the Revolution

The State of AI

• Computer vision relied on hand-crafted features (SIFTScale-Invariant Feature Transform: A computer vision algorithm developed in 1999 that detects and describes local features in images. SIFT features are invariant to scale, rotation, and partially invariant to illumination changes. Before deep learning, these hand-crafted features were the state-of-the-art for tasks like object recognition and image matching., HOGHistogram of Oriented Gradients: A feature descriptor technique that counts occurrences of gradient orientation in localized portions of an image. Developed in 2005, HOG was particularly successful for pedestrian detection. It works by dividing the image into cells, computing gradients in each cell, and creating histograms of gradient directions.)
• Training limited to CPUs - weeks for modest networks
• Lack of large labeled datasets
• Deep networks suffered from vanishing gradient problemA difficulty in training deep neural networks where gradients become exponentially small as they propagate back through many layers. This makes it nearly impossible to train networks with many layers using traditional activation functions like sigmoid or tanh, as the weights in early layers barely update.
• Best ImageNet accuracy: ~74% (2011 results)

The AlexNet Moment

September 30, 2012

AlexNet's Key Innovations

• 8-layer deep CNNConvolutional Neural Network: A specialized type of neural network designed for processing grid-like data such as images. CNNs use convolutional layers that apply filters across the image to detect features like edges, shapes, and textures. The key insight is that the same feature detector (filter) can be useful across different parts of an image, leading to parameter sharing and translation invariance. trained on 2 GPUs
• ReLURectified Linear Unit: An activation function defined as f(x) = max(0, x). ReLU was revolutionary because it's much faster to compute than sigmoid or tanh, and it doesn't suffer from vanishing gradients for positive inputs. This simple change allowed training much deeper networks. Today, ReLU and its variants are the most common activation functions in deep learning. activation (instead of sigmoid/tanh)
• DropoutA regularization technique where random neurons are "dropped out" (set to zero) during training with a certain probability (typically 0.5). This prevents neurons from co-adapting too much and forces the network to learn more robust features. It's like training an ensemble of networks that share weights. During inference, all neurons are active but their outputs are scaled accordingly. for regularization
• Data augmentation techniques
• Local Response NormalizationA normalization technique inspired by lateral inhibition in neuroscience, where the activity of a neuron is normalized by the activities of its neighboring neurons. While innovative at the time, LRN has largely been replaced by Batch Normalization in modern architectures as it's more effective and easier to implement.

Immediate Impact (2013-2016)

Every major tech company established AI research labs

Today's Landscape

What Can Deep Learning Do?

Deep learning is transforming industries across the spectrum: healthcare diagnostics, climate science, drug discovery, and creative applications in art and media.

About This Course

Our Mission

Make deep learning approachable by teaching:

Interactive Demo

Interactive Demo: Overfitting

Fit a polynomial to 50 noisy data points — what happens as you raise the degree?

Learning by Doing

Traditional textbooks: exhaustive detail on each topic

Our approach: Just-in-time learning

• Learn concepts when you need them
• Taste success early - train your first model quickly
• One working example per notebook
• Real datasets, practical applications

Course Structure

Getting Started

• Chapter 1: Introduction
  • The deep learning revolution
  • Key concepts and terminology
  • Course overview and goals
• Chapter 2: Preliminaries
  • Data manipulation and preprocessing
  • Linear algebraThe mathematics of vectors, matrices, and linear transformations. Essential for understanding how neural networks transform data through layers. Key concepts include matrix multiplication (how layers compute), eigenvalues (for understanding network dynamics), and vector spaces (for understanding representations). essentials
  • CalculusThe mathematics of change and optimization. In deep learning, we use derivatives to understand how changing weights affects the output, and gradients to find the direction of steepest improvement. The chain rule is fundamental to backpropagation - the algorithm that trains neural networks. and automatic differentiationA technique to compute derivatives of complex functions automatically. Unlike symbolic or numerical differentiation, autodiff computes exact derivatives efficiently by breaking down computations into elementary operations and applying the chain rule. This is what makes training deep networks feasible - frameworks like PyTorch handle this automatically.
  • Probability and statistics

Course Structure

Linear Models

• Chapter 3: Linear Neural Networks for Regression
  • Linear regressionThe simplest form of supervised learning where we fit a line (or hyperplane) to data. Despite its simplicity, linear regression introduces key concepts: loss functions (how we measure error), gradient descent (how we improve), and the bias-variance tradeoff. It's the foundation upon which neural networks are built. from scratch
  • Gradient descentAn optimization algorithm that finds the minimum of a loss function by repeatedly moving in the direction of steepest descent. Think of it like finding the lowest point in a valley by always walking downhill. The 'gradient' is the slope, and we use it to update our model's weights to reduce errors. and optimization
  • Vectorized implementationWriting code that operates on entire arrays/matrices at once instead of using loops. This leverages optimized linear algebra libraries (like NumPy) and GPU parallelization, making code run 10-100x faster. For example, computing dot products of many vectors simultaneously rather than one at a time.
• Chapter 4: Linear Neural Networks for Classification
  • Softmax regressionGeneralizes logistic regression to multiple classes. The softmax function converts a vector of scores into a probability distribution over classes. This is how most neural networks handle multi-class classification - the final layer typically uses softmax with cross-entropy loss.
  • Cross-entropy lossA loss function that measures the difference between predicted probabilities and true labels in classification. It heavily penalizes confident wrong predictions. For example, if the model says '90% cat' but it's actually a dog, the loss is very high. It's the standard loss for classification tasks.
  • Fashion-MNISTA dataset of 70,000 grayscale images (28x28 pixels) of clothing items in 10 categories (T-shirt, trouser, dress, etc.). Created as a drop-in replacement for the handwritten digits MNIST dataset, it's more challenging and representative of real computer vision tasks while still being small enough for quick experiments. dataset

Course Structure

First Deep Networks

• Chapter 5: Multilayer PerceptronsThe first true neural networks with hidden layers between input and output. MLPs can theoretically approximate any continuous function (universal approximation theorem), but in practice, deeper and more specialized architectures work better. MLPs remain important for tabular data and as building blocks in larger architectures.
  • Hidden layers and activation functionsNon-linear functions applied after each layer's linear transformation. Without them, stacking layers would be pointless - multiple linear transformations collapse to a single one. Common functions include ReLU (max(0,x)), sigmoid (S-shaped curve), and tanh. They allow networks to learn complex, non-linear patterns in data.
  • BackpropagationThe algorithm that makes training deep networks possible. It efficiently computes gradients by applying the chain rule backwards through the network. Developed in the 1980s, backprop is still the foundation of how we train neural networks today. The \"backward pass\" in modern frameworks implements this algorithm. algorithm
  • Weight initializationThe strategy for setting initial values of network weights before training. Poor initialization can cause vanishing/exploding gradients or slow convergence. Common methods include Xavier/Glorot (good for sigmoid/tanh) and He initialization (good for ReLU). The goal is to keep activations and gradients at reasonable scales throughout the network. and dropoutA regularization technique where random neurons are "turned off" during training with probability p (typically 0.5). This prevents neurons from co-adapting and forces the network to be robust. It's like training an ensemble of networks. During inference, all neurons are active but their outputs are scaled by (1-p).
• Chapter 6: Builders' Guide
  • Layers, blocks, and modelsThe hierarchical organization of neural networks. A layer is a single transformation (e.g., linear, convolution). A block is a reusable group of layers (e.g., ResNet block). A model is the complete network made of blocks. This modular design makes it easy to build and experiment with complex architectures.
  • Parameter managementTechniques for accessing, modifying, and sharing network weights (parameters). This includes initialization strategies, accessing specific layer weights, sharing parameters between layers, and saving/loading model weights. Modern frameworks handle this automatically but understanding it is crucial for debugging and custom architectures.
  • Custom layersCreating your own layer types beyond what the framework provides. This involves defining the forward pass (computation) and backward pass (gradient calculation). Useful for implementing new research ideas or domain-specific operations. Modern frameworks make this surprisingly easy with automatic differentiation. and saving/loading

Course Structure

Convolutional Neural Networks

• Chapter 7: Convolutional Neural Networks
  • ConvolutionA mathematical operation that slides a small matrix (filter/kernel) across an image, computing dot products at each position. This creates a feature map showing where certain patterns appear. Convolution is translation-equivariant: if an object moves in the input, its detection moves correspondingly in the output. and cross-correlation
  • Padding and strideTechniques to control CNN output dimensions. Padding adds zeros around the input to preserve spatial dimensions after convolution. Stride is how many pixels the filter moves each step - stride 2 halves the output size. These control the trade-off between preserving spatial information and computational efficiency.
  • Pooling layersLayers that reduce spatial dimensions by taking the maximum (max pooling) or average (average pooling) of small regions. This makes the network more robust to small translations and reduces computation. For example, 2x2 max pooling takes the maximum value in each 2x2 region, reducing dimensions by half.
  • LeNetOne of the first successful CNNs (1998) by Yann LeCun. LeNet-5 had 7 layers and was used for digit recognition. Though tiny by today's standards (60K parameters), it established the pattern of alternating convolution and pooling layers that influenced all future CNN architectures. - the first ConvNet
• Chapter 8: Modern Convolutional Neural Networks
  • AlexNet - deep learning breakthrough
  • VGG - using blocks
  • ResNet - skip connectionsAlso called residual connections or shortcuts. These are direct connections that bypass one or more layers, allowing the gradient to flow directly through the shortcut during backpropagation. The key insight: instead of learning a transformation H(x), the network learns the residual F(x) = H(x) - x. This makes it easier to learn identity mappings and enables training of very deep networks.
  • DenseNet and EfficientNetA family of models that achieves state-of-the-art accuracy with far fewer parameters than previous architectures. EfficientNet uses neural architecture search to find optimal depth, width, and resolution scaling. The key insight: it's better to scale all dimensions together rather than just making networks deeper.

Course Structure

Recurrent Neural Networks

• Chapter 9: Recurrent Neural Networks
  • Sequence modelingProcessing data where order matters - text, speech, time series, video. Unlike images where we can process all pixels at once, sequences require handling variable lengths and maintaining context over time. This is fundamentally different from standard neural networks which assume fixed-size, independent inputs.
  • Text preprocessingConverting raw text into numerical format for neural networks. Steps include tokenization (splitting into words/subwords), building vocabulary, converting to indices, and handling unknown words. Modern approaches use subword tokenization (like BPE) to handle rare words and multiple languages effectively.
  • Language modelsModels that predict the probability of the next word given previous words. They learn grammar, facts, and reasoning from text. Can be used for generation (writing), completion, or as features for other tasks. Modern LMs like GPT are trained on massive text corpora and show emergent abilities like few-shot learning.
  • Backpropagation through timeHow we train RNNs by "unrolling" them over time steps and applying regular backpropagation. The challenge is that gradients must flow through many time steps, leading to vanishing (gradients → 0) or exploding (gradients → ∞) problems. This limits how far back RNNs can remember, motivating LSTM/GRU designs.
• Chapter 10: Modern Recurrent Neural Networks
  • LSTMLong Short-Term Memory: A sophisticated RNN architecture with gates that control information flow. LSTMs have a cell state (long-term memory) and three gates: forget gate (what to discard), input gate (what to store), and output gate (what to output). This design allows them to capture dependencies over hundreds of time steps. - solving vanishing gradients
  • GRUGated Recurrent Unit: A simplified version of LSTM with only two gates (reset and update) instead of three. GRUs achieve similar performance to LSTMs with fewer parameters, making them faster to train. They combine the forget and input gates into a single update gate, making the architecture more streamlined. - simplified gating
  • Bidirectional RNNsRNNs that process sequences in both forward and backward directions, then combine the representations. This gives each position information about both past and future context. Extremely effective for tasks like named entity recognition where knowing what comes next helps. Can't be used for generation since future isn't available.
  • Encoder-decoder architectureA two-part architecture for sequence-to-sequence tasks (translation, summarization). The encoder processes the input sequence into a fixed representation, then the decoder generates the output sequence from this representation. The bottleneck forces the model to capture essential information. Attention mechanisms later removed this bottleneck limitation.

Course Structure

Attention and Transformers

• Chapter 11: Attention MechanismsA mechanism where each position in a sequence attends to all positions to compute a representation. For each position, we compute query, key, and value vectors. The attention scores (softmax of query-key dot products) determine how much each position contributes to the output. This allows capturing relationships regardless of distance in the sequence. & Transformers
  • Attention poolingInstead of simple averaging or max pooling, use learned weights to combine features. The weights are computed based on the features themselves - important parts get higher weights. This is the foundation of attention mechanisms: letting the model decide what to focus on rather than treating everything equally.
  • Bahdanau attentionThe first successful attention mechanism (2014) for neural machine translation. It allows the decoder to "attend" to different parts of the input sequence at each step, rather than relying on a single fixed representation. This solved the information bottleneck problem and dramatically improved translation quality for long sentences.
  • Self-attentionA special case of attention where queries, keys, and values all come from the same sequence. This allows each position to attend to all other positions in the same sequence, capturing dependencies regardless of distance. Self-attention is the core mechanism that makes Transformers so powerful. and positional encodingSince Transformers process all positions in parallel (no inherent order), we must inject position information. This is done by adding special vectors to the input embeddings that encode each position. The original paper used sine/cosine functions of different frequencies, but learned embeddings also work well.
  • The Transformer architectureThe revolutionary architecture from "Attention is All You Need" (2017). Built entirely on self-attention and feed-forward layers - no recurrence or convolution. Processes all positions in parallel, making it much faster to train than RNNs. The foundation of all modern language models (GPT, BERT, etc.) and increasingly used in vision too.
  • Multi-head attentionInstead of one attention function, Transformers use multiple attention \"heads\" in parallel, each learning different relationships. The outputs are concatenated and linearly transformed. This allows the model to jointly attend to information from different representation subspaces at different positions.

Course Structure

Optimization and Performance

• Chapter 12: Optimization Algorithms
  • SGDStochastic Gradient Descent: The fundamental optimization algorithm that updates weights in the direction opposite to the gradient. \"Stochastic\" means we use a random subset (batch) of data for each update rather than the full dataset. Despite its simplicity, SGD with momentum often outperforms more complex optimizers. and momentum
  • AdaGradAdaptive Gradient: An optimizer that adapts the learning rate for each parameter based on historical gradients. Parameters with large gradients get smaller learning rates. Great for sparse data but can stop learning too early as rates decay to zero. and RMSpropRoot Mean Square Propagation: Fixes AdaGrad's aggressive learning rate decay by using an exponential moving average of squared gradients instead of accumulating all history. This keeps the learning rate adaptive but prevents it from vanishing. Often works better than AdaGrad in practice.
  • AdamAdaptive Moment Estimation: Combines ideas from momentum and RMSprop. Adam maintains running averages of both gradients (first moment) and squared gradients (second moment), using these to adapt the learning rate for each parameter. It's the default optimizer for most deep learning applications due to its robustness. optimizer
  • Learning rate schedulingChanging the learning rate during training according to a schedule. Common strategies: step decay (drop by factor every N epochs), exponential decay, cosine annealing, or warm-up (start small, increase, then decay). This helps convergence - large rates explore, small rates refine. Critical for training large models.
• Chapter 13: Computational Performance
  • Hardware: CPUs vs GPUs vs TPUsTensor Processing Units: Google's custom chips designed specifically for neural networks. Optimized for matrix multiplications with reduced precision (bfloat16). Much faster and more power-efficient than GPUs for large models. Available on Google Cloud. TPUv4 pods can train models with trillions of parameters.
  • Multiple GPUs and parallelizationTechniques for training on multiple GPUs/machines. Data parallelism splits the batch across devices. Model parallelism splits the model itself. Pipeline parallelism splits layers across devices. Modern training often combines all three. Frameworks like PyTorch DDP and Horovod make this easier but it's still complex at scale.
  • Training on the cloudUsing cloud services (AWS, Google Cloud, Azure) for training instead of local hardware. Benefits: access to latest GPUs/TPUs, scale up/down as needed, pre-configured environments. Considerations: data transfer costs, security, pricing (spot vs on-demand instances). Services like SageMaker and Vertex AI simplify deployment.

Course Structure

Computer Vision

• Chapter 14: Computer Vision
  • Image augmentationCreating variations of training images to improve generalization. Common augmentations: rotation, flipping, cropping, color jittering, mixup. This artificially increases dataset size and makes models robust to variations. Modern approaches like AutoAugment learn optimal augmentation policies. Critical for good performance with limited data. techniques
  • Fine-tuningTaking a model pre-trained on a large dataset (like ImageNet) and adapting it to your specific task. You typically replace the final layer(s) and train on your data with a small learning rate. This transfers learned features and works amazingly well even with little data. The standard approach for most applications. pretrained models
  • Object detection: R-CNNRegion-based CNN: A family of object detection methods that first propose regions likely to contain objects, then classify each region. Faster R-CNN improves speed with a Region Proposal Network. Though slower than YOLO, R-CNN methods often achieve higher accuracy, especially for small objects. family
  • Single shot detection: YOLOYou Only Look Once: A family of real-time object detection models that treat detection as a regression problem. Unlike earlier methods that use region proposals, YOLO divides the image into a grid and predicts bounding boxes and class probabilities directly. This \"single shot\" approach enables real-time performance.
  • Semantic segmentationClassifying every pixel in an image into categories (road, car, person, sky, etc.). Unlike object detection which gives bounding boxes, segmentation provides pixel-perfect boundaries. Used in medical imaging, autonomous driving, photo editing. Architectures like U-Net and DeepLab are designed specifically for this task.

Course Structure

Natural Language Processing

• Chapter 15: NLP Pretraining
  • Word embeddings: Word2VecA technique that learns vector representations of words where similar words have similar vectors. It uses either CBOW (predict word from context) or Skip-gram (predict context from word) objectives. Word2Vec showed that word vectors can capture semantic relationships: vector(\"king\") - vector(\"man\") + vector(\"woman\") ≈ vector(\"queen\"). and GloVeGlobal Vectors: Combines the benefits of matrix factorization (like LSA) and local context window methods (like Word2Vec). GloVe constructs a co-occurrence matrix of words and factorizes it to get word vectors. It often produces better vectors for word analogy tasks than Word2Vec.
  • Subword embeddingBreaking words into smaller pieces (subwords) for better coverage. Methods like Byte-Pair Encoding (BPE) or WordPiece handle rare words by decomposing them ("unhappiness" → "un" + "happiness"). This gives finite vocabulary while handling any word, crucial for multilingual models and avoiding out-of-vocabulary issues.
  • BERTBidirectional Encoder Representations from Transformers: Google's breakthrough model that pre-trains on masked language modeling (predicting missing words) and next sentence prediction. Unlike GPT's left-to-right processing, BERT looks at context from both directions. This bidirectional approach made it superior for tasks like question answering and named entity recognition. for understanding
  • GPTGenerative Pre-trained Transformer: OpenAI's series of large language models trained to predict the next word in a sequence. GPT models are \"autoregressive\" - they generate text one token at a time. GPT-3 (175B parameters) showed that large models could perform many tasks without task-specific training, just by conditioning on prompts. GPT-4 further improved capabilities with multimodal inputs. for generation
• Chapter 16: NLP Applications
  • Sentiment analysisDetermining the emotional tone or opinion in text - positive, negative, or neutral. Can be fine-grained (1-5 stars) or aspect-based ("great food, terrible service"). Used in social media monitoring, product reviews, customer feedback. Modern approaches use pre-trained language models fine-tuned on labeled sentiment data.
  • Natural language inferenceDetermining the logical relationship between two sentences: entailment (A implies B), contradiction (A contradicts B), or neutral. For example: "A man is running" entails "A person is moving". This tests deep language understanding and is used as a benchmark for language models.
  • Question answeringSystems that answer questions based on given context or knowledge. Can be extractive (finding the answer span in text) or generative (creating new answer text). Modern QA systems use Transformers and can handle complex reasoning. Applications include search engines, virtual assistants, and educational tools.

Course Structure

Advanced Topics I

• Chapter 17: Reinforcement Learning
  • Markov Decision ProcessesMathematical framework for modeling sequential decision-making under uncertainty. Consists of states, actions, transition probabilities, and rewards. The "Markov" property means the future depends only on the current state, not history. MDPs are the foundation of reinforcement learning - we're trying to find the optimal policy in an MDP.
  • Q-LearningA reinforcement learning algorithm that learns the value (Q) of taking each action in each state. Q(s,a) represents expected future reward. The algorithm updates Q-values based on experiences, eventually converging to optimal values. Deep Q-Networks (DQN) use neural networks to approximate Q-values for large state spaces.
  • Deep Q-Networks (DQN)Breakthrough that combined Q-learning with deep neural networks to play Atari games from pixels (DeepMind, 2013). Key innovations: experience replay (storing and resampling past experiences) and target networks (separate network for stable targets). This showed deep RL could learn complex behaviors from raw sensory input.
  • Policy gradient methodsInstead of learning value functions, directly optimize the policy (action selection) using gradients. The REINFORCE algorithm estimates gradients using Monte Carlo sampling. More stable for continuous actions and stochastic policies. Advanced versions like PPO and TRPO are used in robotics and game AI.
• Chapter 18: Gaussian Processes
  • Bayesian approachTreating model parameters as probability distributions rather than fixed values. Instead of finding one "best" set of weights, we maintain uncertainty about them. This gives us principled uncertainty estimates - the model knows what it doesn't know. More computationally expensive but crucial for safety-critical applications. to ML
  • Kernel methodsTechniques that implicitly map data to high/infinite dimensional spaces using kernel functions (similarity measures). The "kernel trick" computes dot products in this space without explicitly doing the mapping. Examples: RBF (Gaussian) kernel, polynomial kernel. Gaussian Processes use kernels to define distributions over functions.
  • Uncertainty quantificationMeasuring how confident a model is in its predictions. Distinguishes epistemic uncertainty (model uncertainty from limited data) from aleatoric uncertainty (inherent noise). Critical for decision-making: a self-driving car should know when it's unsure. Methods include Bayesian neural networks, ensembles, and dropout at test time.

Course Structure

Advanced Topics II

• Chapter 19: Hyperparameter Optimization
  • Grid and random searchSimple hyperparameter optimization methods. Grid search tries all combinations of specified values (exhaustive but expensive). Random search samples random combinations - surprisingly effective because usually only a few hyperparameters matter. Random search often finds good solutions faster than grid search for the same budget.
  • Bayesian optimizationSmart hyperparameter search using probabilistic models. Builds a model (often Gaussian Process) of the objective function, then uses it to decide where to sample next. Balances exploration (uncertain regions) and exploitation (promising regions). Much more efficient than random search for expensive evaluations.
  • Automated ML (AutoML)Automating the entire machine learning pipeline: data preprocessing, feature engineering, model selection, hyperparameter tuning. Tools like AutoGluon, H2O AutoML, and Google AutoML make ML accessible to non-experts. Also includes Neural Architecture Search (NAS) for automatically designing network architectures.
• Chapter 20: Generative Adversarial Networks
  • The GANGenerative Adversarial Networks: Two networks competing against each other - a generator creating fake samples and a discriminator trying to distinguish real from fake. Through this adversarial training, GANs can generate remarkably realistic images, though they're notoriously difficult to train and can suffer from mode collapse. framework
  • Deep Convolutional GANsDCGAN - the first stable architecture for training GANs on images. Key insights: use strided convolutions (no pooling), batch normalization, ReLU in generator, LeakyReLU in discriminator. These architectural guidelines made GAN training much more reliable and became the foundation for most image GANs.
  • StyleGANNVIDIA's GAN architecture that generates incredibly realistic images by controlling styles at different scales. Introduces style mixing, adaptive instance normalization, and progressive growing. Can generate and edit faces, art, and more with unprecedented quality. StyleGAN2/3 further improve quality and remove artifacts. and applications

Course Structure

Real-World Applications

• Chapter 21: Recommender Systems
  • Collaborative filteringRecommendation technique based on user-item interactions. User-based CF finds similar users and recommends what they liked. Item-based CF recommends similar items to what you've liked. The key insight: you don't need to understand content, just patterns in user behavior. Amazon's "Customers who bought X also bought Y" is item-based CF.
  • Matrix factorizationDecomposing the user-item interaction matrix into low-rank factors (user and item embeddings). This compresses sparse data and enables prediction of missing values. Netflix Prize winner used sophisticated matrix factorization. Modern systems combine this with deep learning for better representations.
  • Deep learning for recommendationsUsing neural networks to learn complex user-item interactions. Can incorporate multiple data types (text, images, sequences) and model non-linear patterns. Examples: YouTube's video recommendations, Spotify's Discover Weekly. Often combines collaborative filtering with content understanding for best results.
  • Real-world considerationsPractical challenges in recommendation systems: cold start (new users/items), scalability (millions of users/items), real-time serving, diversity vs accuracy trade-off, filter bubbles, fairness, and manipulation/attacks. Production systems must balance many objectives beyond just accuracy.

Prerequisites & Tools

📚 All materials freely available at d2l.ai

Let's Get Started! 🚀

Mathematical Notation Reference

Throughout this course, we'll use standard mathematical notation

This is a quick reference - we'll cover details as needed

Reference: d2l.ai notation guide

Numbers and Arrays I

Numbers and Arrays II

Set Theory I - Common Sets

Set Theory II - Operations

Functions I

Functions II

Operators & Norms

Calculus

Probability I

Probability II

Statistics & Information Theory

Welcome to Deep Learning!

"The goal is to make deep learning concepts intuitive and engaging."

Questions?

Next lecture: Introduction to the Basics of Deep Learning

⚠️ This repository may change throughout the course. If you have cloned it, git pull regularly to get the newest updates.