Reproducibility of weighted cross entropy loss

vision
1

Hi,

When I need to keep my calculations reproducible, shouldn’t I define a decimal point for the cross-entropy error weights? (Such as [1.0, 100.0, 50.0, 30.0, 20.0])

I am working on a semantic segmentation. The training set has 5 different classes. I applied torch.nn.CrossEntropyLoss and set weights option:

def calc_weights(cat0_num, cat1_num, cat2_num, cat3_num, cat4_num):
    catnum = \
        np.array([cat0_num, cat1_num, cat2_num, cat3_num, cat4_num], 
                      dtype=np.float32)
    cat_sum = catnum.sum()
    catnum = catnum / cat_sum
    catnum = 1.0 / catnum
    catnum = np.round(catnum, decimals=5) # !!!decimals=5!!!
    catnum = torch.FloatTensor(catnum)

    return catnum

But with this setting, the loss value is different every time the calculation is performed.
On the other hand, the following settings always give the same result.

def calc_weights(cat0_num, cat1_num, cat2_num, cat3_num, cat4_num):
    catnum = \
        np.array([cat0_num, cat1_num, cat2_num, cat3_num, cat4_num], dtype=np.float32)
    cat_sum = catnum.sum()
    catnum = catnum / cat_sum
    catnum = 1.0 / catnum
    catnum = np.round(catnum, decimals=0) # !!!decimals=0!!!
    catnum = torch.FloatTensor(catnum)

    return catnum

Shouldn’t I define a decimal point for the cross-entropy error weights?

I already manually make the model deterministic by setting these properties:

def init_seed(seed=1234):
    np.random.seed(seed)
    random.seed(seed)
    torch.manual_seed(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False

def worker_init_fn(worker_id):
    new_seed = np.random.get_state()[1][0] + worker_id
    np.random.seed(new_seed)
    random.seed(new_seed)
    return

Pardon my broken English.
Thanks,

Ryo.

2

For decimal=5 case, try printing value of catnum for two different runs and see if they are different. If they are same, then print the previous catnum value.

3

Thank you very much for your reply.

I tried printing value of “catnum” and here is my code.

def calc_weights(cat0_num, cat1_num, cat2_num, cat3_num, cat4_num):
    catnum = \
        np.array([cat0_num, cat1_num, cat2_num, cat3_num, cat4_num], 
                      dtype=np.float32)
    print(catnum) # -> Standard output A
    cat_sum = catnum.sum()
    catnum = catnum / cat_sum
    catnum = 1.0 / catnum
    catnum = np.round(catnum, decimals=5) # !!!decimals=5!!!
    catnum = torch.FloatTensor(catnum)
    print(catnum) # -> Standard output B
    return catnum

I run it two times and got same standard outputs.
This is the standard outputs of this code.

# Standard output A
[5.1066736e+07 1.0025117e+08 1.5670813e+07 8.5966000e+05 1.2038700e+05]

# Standard output B
tensor([   3.2892,    1.6755,   10.7186,  195.3898, 1395.2401])

If there is anything that you noticed about that please point them out for me.

Best regards,
Ryo．

4

I don’t think the issue is due to this function. I got the same output when running this function. You can make the function shorter.

def calc_weights(cat0_num, cat1_num, cat2_num, cat3_num, cat4_num):
    catnum = np.array([cat0_num, cat1_num, cat2_num, cat3_num, cat4_num], dtype=np.float32)
    catnum = (catnum.sum()/catnum).round(5)
    catnum = torch.from_numpy(catnum)
    return catnum