 # Reproducibility of weighted cross entropy loss

Hi,

When I need to keep my calculations reproducible, shouldn’t I define a decimal point for the cross-entropy error weights? (Such as [1.0, 100.0, 50.0, 30.0, 20.0])

I am working on a semantic segmentation. The training set has 5 different classes. I applied torch.nn.CrossEntropyLoss and set weights option:

``````def calc_weights(cat0_num, cat1_num, cat2_num, cat3_num, cat4_num):
catnum = \
np.array([cat0_num, cat1_num, cat2_num, cat3_num, cat4_num],
dtype=np.float32)
cat_sum = catnum.sum()
catnum = catnum / cat_sum
catnum = 1.0 / catnum
catnum = np.round(catnum, decimals=5) # !!!decimals=5!!!
catnum = torch.FloatTensor(catnum)

return catnum
``````

But with this setting, the loss value is different every time the calculation is performed.
On the other hand, the following settings always give the same result.

``````def calc_weights(cat0_num, cat1_num, cat2_num, cat3_num, cat4_num):
catnum = \
np.array([cat0_num, cat1_num, cat2_num, cat3_num, cat4_num], dtype=np.float32)
cat_sum = catnum.sum()
catnum = catnum / cat_sum
catnum = 1.0 / catnum
catnum = np.round(catnum, decimals=0) # !!!decimals=0!!!
catnum = torch.FloatTensor(catnum)

return catnum
``````

Shouldn’t I define a decimal point for the cross-entropy error weights?

I already manually make the model deterministic by setting these properties:

``````def init_seed(seed=1234):
np.random.seed(seed)
random.seed(seed)
torch.manual_seed(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False

def worker_init_fn(worker_id):
new_seed = np.random.get_state() + worker_id
np.random.seed(new_seed)
random.seed(new_seed)
return
``````

Pardon my broken English.
Thanks,

Ryo.

For decimal=5 case, try printing value of catnum for two different runs and see if they are different. If they are same, then print the previous catnum value.

I tried printing value of “catnum” and here is my code.

``````def calc_weights(cat0_num, cat1_num, cat2_num, cat3_num, cat4_num):
catnum = \
np.array([cat0_num, cat1_num, cat2_num, cat3_num, cat4_num],
dtype=np.float32)
print(catnum) # -> Standard output A
cat_sum = catnum.sum()
catnum = catnum / cat_sum
catnum = 1.0 / catnum
catnum = np.round(catnum, decimals=5) # !!!decimals=5!!!
catnum = torch.FloatTensor(catnum)
print(catnum) # -> Standard output B
return catnum
``````

I run it two times and got same standard outputs.
This is the standard outputs of this code.

``````# Standard output A
[5.1066736e+07 1.0025117e+08 1.5670813e+07 8.5966000e+05 1.2038700e+05]

# Standard output B
tensor([   3.2892,    1.6755,   10.7186,  195.3898, 1395.2401])
``````

If there is anything that you noticed about that please point them out for me.

Best regards,
Ryo．

I don’t think the issue is due to this function. I got the same output when running this function. You can make the function shorter.

``````def calc_weights(cat0_num, cat1_num, cat2_num, cat3_num, cat4_num):
catnum = np.array([cat0_num, cat1_num, cat2_num, cat3_num, cat4_num], dtype=np.float32)
catnum = (catnum.sum()/catnum).round(5)
catnum = torch.from_numpy(catnum)
return catnum
``````