How does pytorch SGD implementation performs sample selection for optimization?

nile649 · October 9, 2022, 11:49pm

Hi, I am not able to find or understand how does pytorch implementation of SGD is able to randomly select samples for optimization?

github.com

pytorch/pytorch/blob/master/torch/optim/sgd.py

import torch
from torch import Tensor
from .optimizer import Optimizer, required, _use_grad_for_differentiable
from typing import List, Optional

__all__ = ['SGD', 'sgd']

class SGD(Optimizer):
    r"""Implements stochastic gradient descent (optionally with momentum).

    .. math::
       \begin{aligned}
            &\rule{110mm}{0.4pt}                                                                 \\
            &\textbf{input}      : \gamma \text{ (lr)}, \: \theta_0 \text{ (params)}, \: f(\theta)
                \text{ (objective)}, \: \lambda \text{ (weight decay)},                          \\
            &\hspace{13mm} \:\mu \text{ (momentum)}, \:\tau \text{ (dampening)},
            \:\textit{ nesterov,}\:\textit{ maximize}                                     \\[-1.ex]
            &\rule{110mm}{0.4pt}                                                                 \\
            &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do}                         \\
            &\hspace{5mm}g_t           \leftarrow   \nabla_{\theta} f_t (\theta_{t-1})           \\

This file has been truncated. show original

I would really appreciate if anyone is able to point me to the line or function which does the random selection or correct me.
Thanks

nile649 · October 9, 2022, 11:50pm

@ptrblck sorry for tagging you. It is just you and few are only familiar with autograd.

AlphaBetaGamma96 · October 10, 2022, 12:38am

Hi @nile649,

The SGD optimizer doesn’t pick any samples at all, the ‘stochastic’ part of the algorthim comes from the data you use in the forward pass.

If you were to use mini-batches, that would create the stochasticity in gradient descent, likewise if you were to use all of your batch, it’d be full-batch gradient descent. It’s just a naming convention for it to be called “SGD”.

@ptrblck wil know more precise details, so correct me if I’m wrong!

ptrblck · October 10, 2022, 4:28am

Yes, @AlphaBetaGamma96 is right and the optimizer does not create the samples but is responsible for the actual parameter updates. The naming (especially the S in SGD) might be a bit confusing.