• Semantic gap: The gap between low-level machine readable data and high-level human readable representation
• Can we hard-code rules to do recognition by if-else? Usually very hard.
• Let us try to describe the appearance of cats

• we use our knowledge of geometry to derive the camera model, which is a projective transformation
• we use our knowledge of analysis to derive the optical flow equation, which is a linear constraint

1. Collect a dataset of images and labels
2. Use Machine Learning algorithms to train a classifier
3. Evaluate the classifier on new images

def train(images, labels):
    # machine learning!
    return model
                            

def predict(model, test_images):
    # use model to predict labels
    return test_labels
                            

• A data sample is a vector $x\in\R{n}$
  • e.g., we can represent an image as a $32\times 32\times 3=3072$ dimensional vector
• The data sample $x$ belongs to one of $C$ predefined categories
  • e.g., $C=3$ for boat/cat/plane
• We use an integer $y\in \{1,\ldots, C\}$ to represent the groundtruth label of $x$. Each number corresponds to a category.
  • e.g., $y=1$ for boat, $y=2$ for cat, $y=3$ for plane

• Train set: a set of $n_{train}$ data samples with labels \[ \mathcal{D}_{train}=\{(x_i, y_i)\}_{i=1}^{n_{train}} \]
• When we estimate the parameters $\theta$ of $f$, we can only use the train set
• Test set: a set of $n_{test}$ data samples with labels \[ \mathcal{D}_{test}=\{(x_i, y_i)\}_{i=1}^{n_{test}} \]
• When we evaluate the performance of a classifier, we can only use the test set

• On a dataset $\mathcal{D}$, the prediction accuracy is: \[ Acc_{\mathcal{D}}=\frac{\text{number of samples with the same groundtruth and predicted labels}}{\text{total number of samples in }\mathcal{D}} \]
• For the training procedure, we estimate $\theta$ on $\mathcal{D}_{train}$ so that $Acc_{\mathcal{D}_{train}}$ is maximized
• For the testing procedure, we run the classifier $f_{\theta}$ on $\mathcal{D}_{test}$ to compute $Acc_{\mathcal{D}_{test}}$

1. First design a loss function of $(W,b)$ which represents the badness of prediction
2. Then design an efficient algorithm to minimize the loss function (optimization)

• $f(x;\theta)$ to denote the classifier (e.g., linear classifier) parameterized by $\theta$.
• $f_i(x;\theta)$ to denote the score for the $i$-th class
• Then the loss function is \[ L(\theta)=-\frac{1}{n}\sum_{i=1}^n \log \left(\frac{e^{f_{y_i}(x_i;\theta)}}{\sum_j e^{f_j(x_i;\theta)}}\right) \]

• One issue is that we need to compute the exponential of $f_j(x;\theta)$, which may cause numerical issues (overflow) if some $f_j(x;\theta)$ is big.
• Trick:
  • Let $t=\max_j f_j(x;\theta)$, then \[ L(\theta)=-\frac{1}{n}\sum_{i=1}^n \log \left(\frac{e^{f_{y_i}(x_i;\theta)-t}}{\sum_j e^{f_j(x_i;\theta)-t}}\right) \]
  • Note that $f_j(x_i;\theta)-t\le 0\ \forall j$, so $e^{f_j(x_i;\theta)-t}$ will not overflow

# Vanilla Gradient Descent

weights = random_init() # e.g., sample a vector from Gaussian distribution
while True:
    weights_grad = evaluate_gradient(loss_fun, data, weights)
    weights += -step_size * weights_grad # perform parameter update
                    

• In GD, gradient computation needs to average the gradient of every data sample.
• If the dataset is large, gradient computation will be very slow!

• A very simple modification to GD
• At every step, randomly sample a minibatch of data points (e.g., 256 data points)
• Compute the average gradient on the minibatch to update $\theta$

# Vanilla Stochastic Gradient Descent

weights = random_init() # e.g., sample a vector from Gaussian distribution
while True:
    data_batch = sample_training_data(data, 256) # sample 256 examples
    weights_grad = evaluate_gradient(loss_fun, data_batch, weights)
    weights += -step_size * weights_grad # perform parameter update
                    

• Recall $f_{W,b}(x)=Wx+b$, where $W=\begin{bmatrix}w_1^T\\w_2^T\\w_3^T\end{bmatrix}$.
• Therefore, the score for each category are \( \begin{cases} f_1=w_1^Tx+b_1\\ f_2=w_2^Tx+b_2\\ f_3=w_3^Tx+b_3\\ \end{cases} \)
• After learning, we obtain the $w_i$'s to minimize the negative log-likelihood loss

• Binary classification: When there are only two categories
• Decision boundary: A simple straight line