I think you might pass the wrong weights to WeightedRandomSampler
.
The sequence of weights should correspond to your samples in the dataset.
Here is a small example:
weights = 1. / torch.tensor(class_sample_counts, dtype=torch.float)
samples_weights = weights[train_targets]
sampler = WeightedRandomSampler(
weights=samples_weights,
num_samples=len(samples_weights),
replacement=True)