# What pytorch means by buffers?

**URL:** <https://discuss.pytorch.org/t/what-pytorch-means-by-buffers/120266>\
**Category:** Uncategorized\
**Created:** [May 5, 2021, 2:12am UTC](https://discuss.pytorch.org/t/what-pytorch-means-by-buffers/120266 "2021-05-05T02:12:25Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![seyeeet](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/seyeeet/32/21931_2.png) [@seyeeet](https://discuss.pytorch.org/u/seyeeet)\
**Post date:** [May 5, 2021, 2:12am UTC](https://discuss.pytorch.org/t/what-pytorch-means-by-buffers/120266/1 "2021-05-05T02:12:25Z")

</div>

Can someone explain to me what do we mean by buffers in pytorch.  
what is its characteristics and when we will use that and when we should not use that?  
If a buffer is basically the same as tensor, Why would I even need a buffer when I can simply create my tensor inside the module?  
an example would be appreciated!!  
thanks

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [May 5, 2021, 6:53am UTC](https://discuss.pytorch.org/t/what-pytorch-means-by-buffers/120266/2 "2021-05-05T06:53:29Z")

</div>

Buffers are tensors, which are registered in the module and will thus be inside the `state_dict`.  
These tensors do not require gradients and are thus not registered as parameters.  
This is useful e.g. to track the mean and std in batchnorm layers etc. which should be stored and loaded using the `state_dict` of the module.

---

<div class="post-metadata">

**Author:** ![seyeeet](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/seyeeet/32/21931_2.png) [@seyeeet](https://discuss.pytorch.org/u/seyeeet)\
**Post date:** [June 8, 2021, 4:55pm UTC](https://discuss.pytorch.org/t/what-pytorch-means-by-buffers/120266/3 "2021-06-08T16:55:24Z")

</div>

I understand, thanks.

- is there a way to know all the buffers in a model?
- in case of bn that you mentioned how I can get the mean and std values? I reach to this point, but not sure how to get mean and std

```auto
import torchvision
import torch.nn as nn
import torch.nn.functional as F
resnet = torchvision.models.resnet50()
resnet.layer1[0].bn1.state_dict

```

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [June 8, 2021, 5:12pm UTC](https://discuss.pytorch.org/t/what-pytorch-means-by-buffers/120266/4 "2021-06-08T17:12:01Z")

</div>

You can get all buffers via `model.buffers()` or `model.named_buffers()` (same as with `.parameters()` and `.named_parameters()`).  
To access the buffers in a specific layer, you can access them directly:

```python
model.bn.running_mean
model.bn.running_var

```

---

<div class="post-metadata">

**Author:** ![Konstantin\_Burlachen](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/konstantin_burlachen/32/36090_2.png) [@Konstantin\_Burlachen](https://discuss.pytorch.org/u/Konstantin_Burlachen)\
**Post date:** [November 9, 2021, 4:37pm UTC](https://discuss.pytorch.org/t/what-pytorch-means-by-buffers/120266/5 "2021-11-09T16:37:49Z")

</div>

> [@ptrblck](#):
>
> named\_parameters

Q1: Do I understand correctly that named\_buffers() and buffers() is the same parameters?

Q2: Is Pytotrch automatically determinates the name of the parameter?  
Because ctor does not obtain this as argument ([Parameter — PyTorch 2.1 documentation](https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html))

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [November 10, 2021, 6:22am UTC](https://discuss.pytorch.org/t/what-pytorch-means-by-buffers/120266/6 "2021-11-10T06:22:23Z")

</div>

1. `named_buffers()` and `buffers()` returns the same buffers where the first operation returns the corresponding name for each buffer. I’m explicitly using “buffer” to avoid conflicting it with parameters, which are different. Both are registered to the `nn.Module` where parameters are trainable while buffers are not.

2. Yes, the name of the buffer or parameter is determined by its variable name (e.g. `self.my_param = nn.Parameter(...)` would return “my\_param” as the name) or explicitly via `self.register_buffer(name, buffer)` or `self.register_parameter(name, param)`.

---

<div class="post-metadata">

**Author:** ![Konstantin\_Burlachen](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/konstantin_burlachen/32/36090_2.png) [@Konstantin\_Burlachen](https://discuss.pytorch.org/u/Konstantin_Burlachen)\
**Post date:** [May 1, 2022, 12:33pm UTC](https://discuss.pytorch.org/t/what-pytorch-means-by-buffers/120266/7 "2022-05-01T12:33:41Z")

</div>

It’s an interesting example of using Python reflection like behaviour in such way that there is no need to register named\_pararamters with specific names:

```auto
import torch
import torch.nn as nn

class NN_Network(nn.Module):
    def __init__ (self,in_dim,hid,out_dim):
        super(NN_Network, self). __init__ ()
        self.linear1 = nn.Linear(in_dim,hid)
        self.linear2 = nn.Linear(hid,out_dim)

    def forward(self, input_array):
        h = self.linear1(input_array)
        y_pred = self.linear2(h)
        return y_pred

in_d = 5
hidn = 2
out_d = 3
model = NN_Network(in_d, hidn, out_d)

for name, param in model.named_parameters():
     print(name, "[", type(name), "]", type(param), param.size())

```

---

<div class="post-metadata">

**Author:** ![Konstantin\_Burlachen](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/konstantin_burlachen/32/36090_2.png) [@Konstantin\_Burlachen](https://discuss.pytorch.org/u/Konstantin_Burlachen)\
**Post date:** [June 6, 2022, 6:45pm UTC](https://discuss.pytorch.org/t/what-pytorch-means-by-buffers/120266/8 "2022-06-06T18:45:37Z")

</div>

I have updated a code snippet to play with parameters and buffers from public Python API side:

```auto
#!/usr/bin/env python3

import torch
import torch.nn as nn

class NN_Network(nn.Module):
    def __init__ (self,in_dim,hid,out_dim):
        super(NN_Network, self). __init__ ()

        # https://pytorch.org/docs/stable/generated/torch.nn.Linear.html?highlight=torch%20nn%20linear#torch.nn.Linear
        self.linear1 = nn.Linear(in_dim,hid)
        self.linear2 = nn.Linear(hid,out_dim)

        # https://pytorch.org/docs/stable/generated/torch.nn.parameter.Parameter.html?highlight=torch%20nn%20parameter#torch.nn.parameter.Parameter
        self.my_param_a = nn.Parameter(torch.zeros(3,3))

        # https://pytorch.org/docs/stable/generated/torch.nn.Module.html?highlight=register_buffer#torch.nn.Module.register_buffer
        self.my_buffer_1 = self.register_buffer("A", torch.zeros(3,3))
        self.my_buffer_2 = self.register_buffer("B", torch.zeros(4,4))

    def forward(self, input_array):
        h = self.linear1(input_array)
        y_pred = self.linear2(h)
        return y_pred

in_d = 5
hidn = 2
out_d = 3
model = NN_Network(in_d, hidn, out_d)

print("\nNAMED PARAMS")
for name, param in model.named_parameters():
     print(" ", name, "[", type(name), "]", type(param), param.size())

print("\nNAMED BUFFERS")
for name, param in model.named_buffers():
     print(" ", name, "[", type(name), "]", type(param), param.size())

print("\nSTATE DICT (KEYS ONLY)")
for k, v in model.state_dict().items():
   print(" ", k)

print("\nSTATE DICT (KEYS VALUE PAIRS)")
for k, v in model.state_dict().items():
   print(" ", "(", k, "=>", v, ")")

```