Hi @tginart! Sorry for the delay in getting back to you, we are still getting used to the forums ourselves and figuring out how to setup notifications
We are still in the process of figuring it out ourselves. Getting >70 top-1 accuracy for reasonable privacy levels (let’s say under 10 epsilon at delta 1e-5) is quite hard and an area of active development.
One shortcut could be to pretrain without dp: in this case, you can get there quite easily. See this colab: bit.ly/opacus-dev-day.
In another experiment, we were able to get to 68 without any pretraining using a smaller model, but we are still in the process of cleaning up the results before we publish them
Looking forward to seeing more experiments and results! We would indeed welcome PRs, or tutorials. Even if you want to write it externally, we’d still be happy to link to it in our resources page