# Freezing just a few weights (instead of the whole layer)

**URL:** <https://discuss.pytorch.org/t/freezing-just-a-few-weights-instead-of-the-whole-layer/64772>\
**Category:** autograd\
**Created:** [December 22, 2019, 8:07pm UTC](https://discuss.pytorch.org/t/freezing-just-a-few-weights-instead-of-the-whole-layer/64772 "2019-12-22T20:07:54Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![numbpy](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/n/e9bcb4/32.png) [@numbpy](https://discuss.pytorch.org/u/numbpy)\
**Post date:** [December 22, 2019, 8:07pm UTC](https://discuss.pytorch.org/t/freezing-just-a-few-weights-instead-of-the-whole-layer/64772/1 "2019-12-22T20:07:54Z")

</div>

Heya, I was trying to replicate the pruning experiments mentioned in the paper, [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://arxiv.org/abs/1803.03635) which is written in Tensorflow. I have managed to create the mask and apply it but unable to figure out how to freeze the pruned weights. As much as I checked, it is only possible to freeze one layer or a parameter at once, not the individual weights.

Tried looking at [Rethinking network pruning (code)](https://github.com/Eric-mingjie/rethinking-network-pruning/tree/master/cifar/lottery-ticket/weight-level) as it has a implementation but couldn’t find how they froze the pruned weights. In another code [Lottery ticket in Pytorch](https://github.com/rahulvigneswaran/Lottery-Ticket-Hypothesis-in-Pytorch), they explicitly zeroed the gradients for pruned weights which seems cumbersome and extra computation for auto\_grad as the gradients are still being computed. Is there a workaround?

Also, if there is no other way can backward hooks make the process a little bit easier as I am trying to use this with fastai package. Any help is really appreciated.

---

<div class="post-metadata">

**Author:** ![Michela](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/michela/32/18830_2.png) [@Michela](https://discuss.pytorch.org/u/Michela)\
**Post date:** [July 14, 2020, 5:20pm UTC](https://discuss.pytorch.org/t/freezing-just-a-few-weights-instead-of-the-whole-layer/64772/2 "2020-07-14T17:20:10Z")

</div>

Check out the pruning functionalities added to PyTorch v1.4 under `torch.nn.utils.prune`. They were built with lottery ticket hypothesis research in mind. The masks are added as forward pre-hooks, and are therefore automatically included as an operation in the backward pass too, thus effectively freezing pruned parameters.

You might also want to check out the PyTorch pruning tutorial: [https://pytorch.org/tutorials/intermediate/pruning\_tutorial.html](https://pytorch.org/tutorials/intermediate/pruning_tutorial.html)