Apex and Random Seed

1

Hi, I am wondering have you ever encountered such situation when using Apex Fp16, random seed seems to be out of work.
This is how I set random seed

def setup_seed(seed):
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    np.random.seed(seed)
    random.seed(seed)
    torch.backends.cudnn.deterministic = True

This is where things get weird. I ran two process simultaneously on GPU0 and GPU1 with same conda environment.

image
image3002×778 336 KB

image
image2980×558 149 KB

2

apex.amp is deprecated so please use the native mixed-precision implementation via torch.cuda.amp as described here.