Hyperparameters for CIFAR-10

Hi all,

I am wondering if anyone has figured out any “standard” hyperparameter configurations for obtaining the best accuracy at a given epsilon privacy level on CIFAR-10?

If so, it might be a good idea to include them in the tutorial scripts?

Thanks in advance!

I don’t think these experiments are published yet, but am sure that any contribution is welcome.
Would you be interested in working on such a tutorial? :slight_smile:

1 Like

I have done some hp sweeps and am planning to do some more over the holidays. I don’t expect them to be entirely exhaustive but they’d be a good starting place. I’ll share in early 2021. Thanks!

Hi @tginart! Sorry for the delay in getting back to you, we are still getting used to the forums ourselves and figuring out how to setup notifications :smiley:

We are still in the process of figuring it out ourselves. Getting >70 top-1 accuracy for reasonable privacy levels (let’s say under 10 epsilon at delta 1e-5) is quite hard and an area of active development.

One shortcut could be to pretrain without dp: in this case, you can get there quite easily. See this colab: bit.ly/opacus-dev-day.

In another experiment, we were able to get to 68 without any pretraining using a smaller model, but we are still in the process of cleaning up the results before we publish them :slight_smile:

Looking forward to seeing more experiments and results! We would indeed welcome PRs, or tutorials. Even if you want to write it externally, we’d still be happy to link to it in our resources page :slight_smile:

Hello, @Darktex!

Sorry for the late reply. Somehow this got buried in my notifications before the new year.

With regards to using smaller models — since the privacy budget can depend on the parameter count, it does stand to reason that a smaller model might give you more “bang for buck.” I actually tested this out a little bit by comparing MobileNetV2 to ResNet18 on a few runs but I wasn’t able to get a major improvement (granted, I could’ve tried much harder).

I’m looking forward to seeing your results on this! I’ll keep you posted on anything from my end as well. I think it would be useful/cool to maintain a sort of “state-of-the-art” leaderboard for provably DP models at a given (eps,delta). I could help kick this off with some of my experiments, although 68% is already significantly better than anything I’ve obtained. My best runs don’t even make it to 60%.

1 Like