Dilation convolution takes a lot of memory

I try to use the dilation convolution layer described in Multi-Scale Context Aggregation by Dilated Convolutions to make a new Vgg net,but it takes too much memory than I thought,I can’t run it on my 980Ti with [1,3,224,224] tensor
here is my net:

(0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
(1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
(2): ReLU (inplace)
(3): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(2, 2), dilation=(2, 2))
(4): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
(5): ReLU (inplace)
(6): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4))
(7): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
(8): ReLU (inplace)
(9): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(4, 4), dilation=(4, 4))
(10): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
(11): ReLU (inplace)
(12): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(8, 8), dilation=(8, 8))
(13): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
(14): ReLU (inplace)
(15): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(8, 8), dilation=(8, 8))
(16): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
(17): ReLU (inplace)
(18): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(16, 16), dilation=(16, 16))
(19): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
(20): ReLU (inplace)
(21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(16, 16), dilation=(16, 16))
(22): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
(23): ReLU (inplace)

def make_layers(cfg, batch_norm=False):
layers = []
in_channels = 3
i=1
for v in cfg:
if v == ‘M’:
i=i*2;
else:
conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1,dilation=(i,i))

        if batch_norm:
            layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU(inplace=True)]
        else:
            layers += [conv2d, nn.ReLU(inplace=True)]
        in_channels = v
return nn.Sequential(*layers)

-Do you have cuDNN6 or above installed properly?

-Can you run the model with all dilations set to 1 (and paddings appropriately adjusted)?