How to find deep size of each layer of GAN?

Hi , I am really new to Pytorch and facing some difficulties while determining deep size of generator and discriminator models in GAN.

I want to find memory footprint of each and every layer of GAN. Here is my sample code for finding so.
However , this code returns only the shallow size.

import torch
import torch.nn as nn
from torch.autograd import Variable
import numpy as np

class SizeEstimator(object):

    def __init__(self, model, input_size=(1,100,32,32), bits=32):
        Estimates the size of PyTorch models in memory
        for a given input size
        self.model = model
        self.input_size = input_size
        self.bits = 32

    def get_parameter_sizes(self):
        '''Get sizes of all parameters in `model`'''
        mods = list(self.model.modules())
        for i in range(1,len(mods)):
            m = mods[i]
            p = list(m.parameters())
            sizes = []
            for j in range(len(p)):

        self.param_sizes = sizes

    def get_output_sizes(self):
        '''Run sample input through each layer to get output sizes'''
        input_ = Variable(torch.cuda.FloatTensor(*self.input_size), volatile=True)
        mods = list(self.model.modules())
        out_sizes = []
        for i in range(1, len(mods)):
            m = mods[i]
            out = m(input_)
            input_ = out

        self.out_sizes = out_sizes

    def calc_param_bits(self):
        '''Calculate total number of bits to store `model` parameters'''
        total_bits = 0
        for i in range(len(self.param_sizes)):
            s = self.param_sizes[i]
            bits =*self.bits
            total_bits += bits
        self.param_bits = total_bits

    def calc_forward_backward_bits(self):
        '''Calculate bits to store forward and backward pass'''
        total_bits = 0
        for i in range(len(self.out_sizes)):
            s = self.out_sizes[i]
            bits =*self.bits
            total_bits += bits
        # multiply by 2 for both forward AND backward
        self.forward_backward_bits = (total_bits*2)

    def calc_input_bits(self):
        '''Calculate bits to store input'''
        self.input_bits =*self.bits

    def estimate_size(self):
        '''Estimate model size in memory in megabytes and bits'''
        total = self.param_bits + self.forward_backward_bits + self.input_bits

        total_megabytes = (total/8)/(1024**2)
        return total_megabytes, total

So , I tried another code which gives deep size of any python object.

import sys
import inspect

def get_size(obj, seen=None):
    """Recursively finds size of objects in bytes"""
    size = sys.getsizeof(obj)
    if seen is None:
        seen = set()
    obj_id = id(obj)
    if obj_id in seen:
        return 0
    # Important mark as seen *before* entering recursion to gracefully handle
    # self-referential objects
    if hasattr(obj, '__dict__'):
        for cls in obj.__class__.__mro__:
            if '__dict__' in cls.__dict__:
                d = cls.__dict__['__dict__']
                if inspect.isgetsetdescriptor(d) or inspect.ismemberdescriptor(d):
                    size += get_size(obj.__dict__, seen)
    if isinstance(obj, dict):
        size += sum((get_size(v, seen) for v in obj.values()))
        size += sum((get_size(k, seen) for k in obj.keys()))
    elif hasattr(obj, '__iter__') and not isinstance(obj, (str, bytes, bytearray)):
        size += sum((get_size(i, seen) for i in obj))
    if hasattr(obj, '__slots__'): # can have __slots__ with __dict__
        size += sum(get_size(getattr(obj, s), seen) for s in obj.__slots__ if hasattr(obj, s))
    return size

But , I am facing issues with second code when I am passing a layer of generator.

It is calculating object size recursively.

On this line

elif hasattr(obj, '__iter__') and not isinstance(obj, (str, bytes, bytearray)):
        size += sum((get_size(i, seen) for i in obj))

size += sum((get_size(i, seen) for i in obj))
– on this line I am getting Runtime error : 0-d tensor.

Could you please check this once ?

Have a look at @jacobkimmel’s pytorch modelsize estimator.
I’m not sure, if it’s still executable, but it still might give you some insights on the calculations.

hi @ptrblck , thanks a lot for such a prompt reply. :blush:

However , this is the same piece of code which I have used. But , this is giving me only shallow sizes of a layer / parameters.

Could you please provide me with any other example which gives deep size ?

Thanks in anticipation.

I’m not sure what shallow vs. deep size means.
Could you explain it a bit?

Hi @ptrblck,

When we measure a size of an object, we really want the size of all of it’s attributes, and their attributes, etc. But sys.getsizeof only gives us the size of the object and their attributes, however it does not recursively add the size of sub-attributes.

So , we can say this as Shallow size.

It has been clearly mentioned in this link :

Please let me know , if I interpreted the aforementioned content correctly.