# Performing mini-batch gradient descent or stochastic gradient descent on a mini-batch

**URL:** <https://discuss.pytorch.org/t/performing-mini-batch-gradient-descent-or-stochastic-gradient-descent-on-a-mini-batch/21235>\
**Category:** Uncategorized\
**Created:** [July 16, 2018, 7:01pm UTC](https://discuss.pytorch.org/t/performing-mini-batch-gradient-descent-or-stochastic-gradient-descent-on-a-mini-batch/21235 "2018-07-16T19:01:46Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Joseph\_Santarcangelo](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joseph_santarcangelo/32/12526_2.png) [@Joseph\_Santarcangelo](https://discuss.pytorch.org/u/Joseph_Santarcangelo)\
**Post date:** [July 16, 2018, 7:01pm UTC](https://discuss.pytorch.org/t/performing-mini-batch-gradient-descent-or-stochastic-gradient-descent-on-a-mini-batch/21235/1 "2018-07-16T19:01:46Z")

</div>

Hello, I have created a data-loader object, I set the parameter batch size equal to five and I run the following code. I would like some clarification, is the following code performing mini-batch gradient descent or stochastic gradient descent on a mini-batch.

```auto
from torch import nn
import torch
import numpy as np
import matplotlib.pyplot as plt
from torch import nn,optim
from torch.utils.data import Dataset, DataLoader
class Data(Dataset):
    def __init__ (self):
        self.x=torch.arange(-3,3,0.1).view(-1, 1)
        self.y=-3*self.x+1+0.1*torch.randn(self.x.size())
        self.len=self.x.shape[0]
    def __getitem__ (self,index):    
            
        return self.x[index],self.y[index]
    def __len__ (self):
        return self.len
class linear_regression(nn.Module):
    def __init__ (self,input_size,output_size):
        super(linear_regression,self). __init__ ()
        self.linear=nn.Linear(input_size,output_size)
    def forward(self,x):
        yhat=self.linear(x)
        return yhat

class linear_regression(nn.Module):
    def __init__ (self,input_size,output_size):
        super(linear_regression,self). __init__ ()
        self.linear=nn.Linear(input_size,output_size)
    def forward(self,x):
        yhat=self.linear(x)
        return yhat
model=linear_regression(1,1)
optimizer = optim.SGD(model.parameters(), lr = 0.01)
criterion = nn.MSELoss()
dataset=Data()
trainloader=DataLoader(dataset=dataset,batch_size=5)
LOSS=[]

n=1;
for epoch in range(5):
    for x,y in trainloader:
        yhat=model(x)
        loss=criterion(yhat,y)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        LOSS.append(loss)

```

i

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [July 16, 2018, 10:17pm UTC](https://discuss.pytorch.org/t/performing-mini-batch-gradient-descent-or-stochastic-gradient-descent-on-a-mini-batch/21235/2 "2018-07-16T22:17:00Z")

</div>

> [@Joseph\_Santarcangelo](#):
>
> mini-batch gradient descent or stochastic gradient descent on a mini-batch

I’m not sure what stochastic gradient descent on a mini-batch is, since as far as my understanding is, stochastic gradient descent uses only one sample by definition.  
Because you use a batch size of 5, your code applies mini-batch gradient descent.

---

<div class="post-metadata">

**Author:** ![Joseph\_Santarcangelo](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joseph_santarcangelo/32/12526_2.png) [@Joseph\_Santarcangelo](https://discuss.pytorch.org/u/Joseph_Santarcangelo)\
**Post date:** [July 16, 2018, 11:35pm UTC](https://discuss.pytorch.org/t/performing-mini-batch-gradient-descent-or-stochastic-gradient-descent-on-a-mini-batch/21235/3 "2018-07-16T23:35:31Z")

</div>

Thanks for the response, my confusion comes from the fact that the code included below calculates SGD by taking a tensor that is the size of my training set and performing at update one sample at the time. i.e for each iteration of the loop an update of SGD is performed for every sample in the tensor  
In the initial code in the second nested loop, the data loader provides a tensor the size of my mini-batch, then how come it does not follow the same procedure as above, i.e for each iteration performing an update on each sample in the tensor.

```auto
from torch import nn
import torch
import numpy as np
import matplotlib.pyplot as plt
from torch import nn,optim
from torch.utils.data import Dataset, DataLoader
class Data(Dataset):
   def __init__ (self):
       self.x=torch.arange(-3,3,0.1).view(-1, 1)
       self.y=-3*self.x+1+0.1*torch.randn(self.x.size())
       self.len=self.x.shape[0]
   def __getitem__ (self,index):    
           
       return self.x[index],self.y[index]
   def __len__ (self):
       return self.len
class linear_regression(nn.Module):
   def __init__ (self,input_size,output_size):
       super(linear_regression,self). __init__ ()
       self.linear=nn.Linear(input_size,output_size)
   def forward(self,x):
       yhat=self.linear(x)
       return yhat

class linear_regression(nn.Module):
   def __init__ (self,input_size,output_size):
       super(linear_regression,self). __init__ ()
       self.linear=nn.Linear(input_size,output_size)
   def forward(self,x):
       yhat=self.linear(x)
       return yhat
model=linear_regression(1,1)
optimizer = optim.SGD(model.parameters(), lr = 0.01)
criterion = nn.MSELoss()
dataset=Data()

x,y=dataset[:]
LOSS=[]

n=1;
for epoch in range(5):
       yhat=model(x)
       loss=criterion(yhat,y)
       optimizer.zero_grad()
       loss.backward()
       optimizer.step()
       LOSS.append(loss)

```

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [July 17, 2018, 6:43am UTC](https://discuss.pytorch.org/t/performing-mini-batch-gradient-descent-or-stochastic-gradient-descent-on-a-mini-batch/21235/4 "2018-07-17T06:43:53Z")

</div>

In your current code snippet you are assigning `x` to your complete dataset, i.e. you are performing batch gradient descent.  
In the former code your `DataLoader` provided batches of size 5, so you used mini-batch gradient descent.  
If you use a dataloader with `batch_size=1` or slice each sample one by one, you would be applying stochastic gradient descent.

The averaged or summed loss will be computed based on your batch size. E.g. if your batch size is 5, and you are using your criterion with its default setting `size_average=True`, the average or the losses for each sample in the batch will be calculated and used to compute the gradients.

---

<div class="post-metadata">

**Author:** ![Joseph\_Santarcangelo](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joseph_santarcangelo/32/12526_2.png) [@Joseph\_Santarcangelo](https://discuss.pytorch.org/u/Joseph_Santarcangelo)\
**Post date:** [July 17, 2018, 5:23pm UTC](https://discuss.pytorch.org/t/performing-mini-batch-gradient-descent-or-stochastic-gradient-descent-on-a-mini-batch/21235/5 "2018-07-17T17:23:52Z")

</div>

Thanks for clearing that up! I was confused because I could not see the explicit relationship between the data loader object and the optimizer.

---

<div class="post-metadata">

**Author:** ![jon](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jon/32/24563_2.png) [@jon](https://discuss.pytorch.org/u/jon)\
**Post date:** [August 24, 2018, 3:33am UTC](https://discuss.pytorch.org/t/performing-mini-batch-gradient-descent-or-stochastic-gradient-descent-on-a-mini-batch/21235/6 "2018-08-24T03:33:28Z")

</div>

Sorry to necro this post, but it bothers me: why is it named a stochastic gradient descent optimizer even if we are doing full batch gradient descent? Is there any sort of difference between what vanilla gradient descent would do vs. the sgd optimizer would do when we run it with the full batch?

---

<div class="post-metadata">

**Author:** ![jon](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jon/32/24563_2.png) [@jon](https://discuss.pytorch.org/u/jon)\
**Post date:** [August 24, 2018, 3:39am UTC](https://discuss.pytorch.org/t/performing-mini-batch-gradient-descent-or-stochastic-gradient-descent-on-a-mini-batch/21235/7 "2018-08-24T03:39:33Z")

</div>

Ahh, actually sorry, it’s just a mismatch in terminology. The SGD optimizer _is_ vanilla gradient descent (i.e. literally all it does is subtract the gradient \* the learning rate from the weight, as expected). See here: [How SGD works in pytorch](https://discuss.pytorch.org/t/how-sgd-works-in-pytorch/8060)

---

<div class="post-metadata">

**Author:** ![vinaykumar2491](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vinaykumar2491/32/44652_2.png) [@vinaykumar2491](https://discuss.pytorch.org/u/vinaykumar2491)\
**Post date:** [October 22, 2018, 5:32am UTC](https://discuss.pytorch.org/t/performing-mini-batch-gradient-descent-or-stochastic-gradient-descent-on-a-mini-batch/21235/8 "2018-10-22T05:32:30Z")

</div>

> [@Joseph\_Santarcangelo](#):
>
> LOSS.append(loss)

If you want to store the whole computational graph then its okay to use `LOSS.append(loss)`; but if you are just looking to store the loss value then use `LOSS.append(loss.item())`.

> [@Joseph\_Santarcangelo](#):
>
> ```python
> LOSS=[]
> 
> n=1;
> for epoch in range(5):
> for x,y in trainloader:
> yhat=model(x)
> loss=criterion(yhat,y)
> optimizer.zero_grad()
> loss.backward()
> optimizer.step()
> LOSS.append(loss)
> 
> ```

Also, shouldn’t you be accumulating the loss (for each batch) within each epoch and then appending it to loss array like this (below):

```python
LOSS=[]
...
for epoch in range(5):
    running_loss = 0 #accumulates loss of each batch
    for x,y in trainloader:
        # do something......
        loss.backward()
        optimizer.step()
        running_loss += loss.item() # accumulating loss
    
    LOSS.append(running_loss) # saving final loss for each epoch

```

---

<div class="post-metadata">

**Author:** ![Joseph\_Santarcangelo](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joseph_santarcangelo/32/12526_2.png) [@Joseph\_Santarcangelo](https://discuss.pytorch.org/u/Joseph_Santarcangelo)\
**Post date:** [October 22, 2018, 8:58pm UTC](https://discuss.pytorch.org/t/performing-mini-batch-gradient-descent-or-stochastic-gradient-descent-on-a-mini-batch/21235/9 "2018-10-22T20:58:12Z")

</div>

Thanks [Vinay Kumar](https://discuss.pytorch.org/u/vinaykumar2491), I found out about Item it’s super useful.  
For the loss, I was just calculating the average loss for each batch.

---

<div class="post-metadata">

**Author:** ![Jaddi\_abd\_elaziz](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jaddi_abd_elaziz/32/25083_2.png) [@Jaddi\_abd\_elaziz](https://discuss.pytorch.org/u/Jaddi_abd_elaziz)\
**Post date:** [June 2, 2020, 8:28pm UTC](https://discuss.pytorch.org/t/performing-mini-batch-gradient-descent-or-stochastic-gradient-descent-on-a-mini-batch/21235/10 "2020-06-02T20:28:29Z")

</div>

just a question, if we write running\_loss += loss.item() after an update of the parameters, I think like that we calculate these:  
for epoch in range(5):  
running\_loss = 0 #accumulates loss of each batch  
for x,y in trainloader:  
running\_loss += loss.item() \<=\> F\* = F(x,y, w\_{0})+ F(x,y, w\_{1})+F(x,y, w\_{2})+…

in an epoch with F isobjective function.  
however our objective function defined by :  
F(data,w) = F(x,y, w)+ F(x,y,w)+F(x,y,w)+… for (x,y) in data