# How do modules like \`nn.Conv2d\` allocate memory?

**URL:** <https://discuss.pytorch.org/t/how-do-modules-like-nn-conv2d-allocate-memory/189953>\
**Category:** Uncategorized\
**Created:** [October 15, 2023, 3:21am UTC](https://discuss.pytorch.org/t/how-do-modules-like-nn-conv2d-allocate-memory/189953 "2023-10-15T03:21:40Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![omega9123](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/o/a9adbd/32.png) [@omega9123](https://discuss.pytorch.org/u/omega9123)\
**Post date:** [October 15, 2023, 3:21am UTC](https://discuss.pytorch.org/t/how-do-modules-like-nn-conv2d-allocate-memory/189953/1 "2023-10-15T03:21:40Z")

</div>

I’m working throught the tutorials and I wanted to understand how `nn.Conv2d` is created. It looks like we can pass in a `device` param during the init function and I’ve checked that this param create a tensor in the GPU (rather than CPU by default). What I’m wondering is if I wanted to create an empty tensor instead of random values for the layer, how do I do that? It seems like by default random values are setup for that tensor.

A follow up question, If I use the `load_state_dict` method to load a model to a GPU allocated tensor, I’m assuming the method would just overwrite the values that were randomly setup?

---

<div class="post-metadata">

**Author:** ![J\_Johnson](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/j_johnson/32/55494_2.png) [@J\_Johnson](https://discuss.pytorch.org/u/J_Johnson)\
**Post date:** [October 15, 2023, 4:28am UTC](https://discuss.pytorch.org/t/how-do-modules-like-nn-conv2d-allocate-memory/189953/2 "2023-10-15T04:28:32Z")

</div>

PyTorch layers are first set to empty, and then call `self.reset_parameters()` to initialize them with a kaiming uniform distribution. See the **init** for ConvNd here: [torch.nn.modules.conv — PyTorch 2.1 documentation](https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html)

As to your second question, that is correct. The previous model values are overwritten by the saved state.

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [October 15, 2023, 3:07pm UTC](https://discuss.pytorch.org/t/how-do-modules-like-nn-conv2d-allocate-memory/189953/3 "2023-10-15T15:07:27Z")

</div>

To skip the initialization of parameters avoiding wasting compute as you will overwrite these, you could use [`torch.nn.utils.skip_init`](https://pytorch.org/docs/stable/generated/torch.nn.utils.skip_init.html#torch.nn.utils.skip_init).