The samping of RDP

Opacus
1

1、the uniform samping with Replacement ，uniform samping with Replacement and the Poisson samping . What is the relationship between the three of them?
2、the paper《RDP of the sampled Gussian Mechanism》 use Poisson samping?
3、Opacus provide uniform samping with Replacement and Poisson samping methods, do they both correspond to the same privacy calculation formula？

2

1、the uniform samping with Replacement ，uniform samping without Replacement and the Poisson samping . What is the relationship between the three of them?

3

Hi,
Re 1:
We say Uniform sampling to mean Poisson sampling: Each sample is selected with a probability equal to sample_rate
The with replacement is due to the fact that during a batch construction, we consider the whole dataset (all the samples), and sample each data point with the probability sample_rate

Re 2: Yes

Re 3: Yes, because the two are one and the same.

5

Thank you for your answer, I have two more questions as follows.
1, the uniform sampling with replacement , the uniform sampling without replacement and the Poisson sampling are differential. In uniform sampling, sample rate is the number of samples sampled In uniform sampling, the sample rate is the ratio of the number of samples sampled to the total number of samples, so the number of samples sampled is fixed and the same for each sample. In poisson samping, the sample rate is the probability of each sample being sampled, which means that the number of samples sampled is not necessarily the same each time. Am I understanding this correctly?
2. In GDP privacy calculation, uniform sampling and poisson sampling have different privacy budgets for the same parameters, why are the privacy losses calculated the same for both in RDP?

6

3、what is the differential of UniformWithReplacementSampler and DistributedUniformWithReplacementSampler?

7

there are somebody can answer my question? thanks

8

Hey JeffffFu,

Thanks for your interest! To answer your questions:

  1. That’s correct, with Poisson sampling your batch size may vary (hence we only set the expected batch size).
  2. Poisson sampling is necessary to obtain privacy budget computations under the paper you mentioned.
  3. The UniformWithReplacementSampler is to be used when you train on a single process (generally a single GPU), the DistributedUniformWithReplacementSampler is when you train in a distributed way, generally on multiple GPUs.

Hope this helps,
Pierre

9

thank you very much!