# How to replace all ReLU activations in a pretrained network?

**URL:** <https://discuss.pytorch.org/t/how-to-replace-all-relu-activations-in-a-pretrained-network/31591>\
**Category:** vision\
**Created:** [December 7, 2018, 11:53pm UTC](https://discuss.pytorch.org/t/how-to-replace-all-relu-activations-in-a-pretrained-network/31591 "2018-12-07T23:53:05Z")\
**Posts on this page:** 1\
**Showing post:** 12

<div class="post-metadata">

**Author:** ![Brando\_Miranda](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/brando_miranda/32/14355_2.png) [@Brando\_Miranda](https://discuss.pytorch.org/u/Brando_Miranda)\
**Post date:** [October 1, 2020, 6:56pm UTC](https://discuss.pytorch.org/t/how-to-replace-all-relu-activations-in-a-pretrained-network/31591/12 "2020-10-01T18:56:54Z")

</div>

Got it working! I thought of your idea but didn’t see how to do it well. Perhaps you can share me your code to see how it should had been done. For now this is what works for me:

```auto
def replace_bn(module, name):
    '''
    Recursively put desired batch norm in nn.module module.

    set module = net to start code.
    '''
    # go through all attributes of module nn.module (e.g. network or layer) and put batch norms if present
    for attr_str in dir(module):
        target_attr = getattr(m, attr_str)
        if type(target_attr) == torch.nn.BatchNorm2d:
            print('replaced: ', name, attr_str)
            new_bn = torch.nn.BatchNorm2d(target_attr.num_features, target_attr.eps, target_attr.momentum, target_attr.affine,
                                          track_running_stats=False)
            setattr(module, attr_str, new_bn)

    # iterate through immediate child modules. Note, the recursion is done by our code no need to use named_modules()
    for name, immediate_child_module in module.named_children():
        replace_bn(immediate_child_module, name)

replace_bn(model, 'model')

```

source post: [How to modify a pretrained model](https://discuss.pytorch.org/t/how-to-modify-a-pretrained-model/60509/10)

Thanks AlbanD! 🙂

---

_[View the full topic](https://discuss.pytorch.org/t/how-to-replace-all-relu-activations-in-a-pretrained-network/31591)._