Machine Learning

• Supervised Learning
• Unsupervised Learning
  • Supervised Learning — Reference Detail
  • Unsupervised Learning — Reference Detail
• Python for Data Analysis
• The Discipline of Machine Learning
• Weka Commands
• Reading List
• Bayesian Inference
  • Intro
  • Joint Distribution
  • Adding Variables
  • Conditional Independence
  • Belief Networks (Bayes Nets)
  • Sampling From a Bayes Net
  • Recovering the Joint Distribution
  • Sampling and Approximate Inference
  • Inference Rules
  • Inference by Hand: Boxes and Balls
  • Naive Bayes
  • Why Naive Bayes Works
  • Summary
• Bayesian Learning
  • Introduction
  • Bayes’ Rule
  • Bayes’ Rule Quiz: Rare Disease
  • MAP and Maximum Likelihood
  • Bayesian Learning: Noise-Free Version Space
  • Noisy Data Quiz
  • Gaussian Noise → Sum of Squared Errors
  • Best Hypothesis Quiz
  • Minimum Description Length (MDL)
  • Bayesian Classification vs MAP Hypothesis
  • Summary
• Beautiful Probability
• Comp Learning Theory
• Decision Trees
  • Supervised Learning — Classification vs Regression
  • Classification vs Regression
  • Key Terminology
  • Classification Learning — Worked Example
  • Contingency Tables
  • Data Mining & Information Theory
  • Learning Decision Trees
  • Dating Example — Problem Setup
  • Decision Tree Representation
  • Representation Quiz — Tracing Instances
  • Best Attribute
  • Decision Tree Expressiveness
    • How Many Decision Trees?
  • ID3 Algorithm
    • Entropy
    • Entropy Example — Coin Flips
  • ID3 Inductive Bias
  • Continuous Attributes
  • Other Considerations
  • Regression Trees
  • Regression vs Classification Trees
  • Basic Decision Tree — Quick Reference
  • Summary
  • References
• Ensemble Learning: Boosting and Bagging
  • Ensemble Learning Overview
    • Core Idea
    • Ensemble Learning Algorithm (General Form)
  • Bagging (Bootstrap Aggregation)
    • Subset Selection
    • Combination
    • Housing Data Example
  • Boosting
    • Motivation vs Bagging
    • Distribution-Weighted Error
    • Weak Learner
    • Boosting Algorithm (Pseudocode)
    • Distribution Update
    • Final Hypothesis
    • Worked Example: Three Boxes
    • Why Boosting Works (Intuition)
  • Summary
• Instance Based Learning
  • Eager vs Lazy Learning
  • 1-Nearest Neighbor
  • k-Nearest Neighbor (k-NN)
  • Distance Metrics
  • Complexity (1-D, sorted data, n points)
  • Domain Knowledge Example
  • k-NN Preference Biases
  • Curse of Dimensionality
  • Locally Weighted Regression
  • Summary
• Kernel Methods and SVM
  • Support Vector Machines
    • Motivation: Maximum Margin
    • Hyperplane Notation
    • Margin Derivation
    • Primal Optimization Problem
    • Dual Form (Lagrange Multipliers)
    • Support Vectors
    • Non-Linearly Separable Data
  • Feature Mapping and the Kernel Trick
    • Feature Map φ — Worked Example
  • Kernels
    • Definition
    • Common Kernels
    • Kernel Properties
  • SVM Summary
  • Boosting and Margins
    • Why Boosting Resists Overfitting
    • When Boosting Overfits
• Neural Networks
  • Overview
  • Perceptron Unit
  • Perceptron = Half-Plane
  • Boolean Functions via Perceptrons
  • XOR Requires a Network
  • Perceptron Training
    • The Bias Trick
    • Perceptron Rule
    • Linear Separability
  • Gradient Descent (Delta Rule)
    • Error Metric
    • Partial Derivative
    • Weight Update
  • Comparison of Learning Rules
    • Why Not Gradient Descent on \(\hat{y}\)?
  • Sigmoid Activation
  • Neural Network Architecture
    • Back-Propagation Details
    • Local Optima
  • Optimizing Weights & Overfitting
    • Overfitting in Neural Nets
    • Training vs. Validation Error
  • Restriction Bias
  • Preference Bias
    • Weight Initialization
    • Preferences
    • Occam’s Razor
  • Neural Nets vs Other Supervised Learners
  • Perceptron vs Sigmoid — Comparison
  • Summary
• Regression
  • Regression and Function Approximation
  • Linear Regression
  • Finding the Best Fit
  • Order of the Polynomial
  • Polynomial Regression (matrix form)
  • Errors
  • Cross Validation
  • Housing Example Revisited
  • Other Input Spaces
• VC Dimensions
  • Which Hypothesis Spaces are Infinite
  • Maybe It Is Not So Bad
  • Power of Hypothesis Space
  • What Does VC Stand For?
  • Quiz: Intervals on the Real Line
  • Quiz: Linear Separators in \(\mathbb{R}^2\)
  • Quiz: Convex Polygons in \(\mathbb{R}^2\)
  • Sample Complexity
  • VC Dimension of Finite Hypothesis Classes
  • Intervals: Proof Checklist
  • Linear Separators: Additional Arguments
  • Convex Polygons: VC = ∞
  • Connecting to Original Sample Bound
  • Summary