What kind of data augmentation methods are you currently using?
If you are using PIL, you might want to install PIL-SIMD, which is a drop-in replacement and will use SIMD operations to speed up the transformations.
Are you using multiple workers in your DataLoader? This should usually speeding up the data loading and processing.
If you are bottlenecked by the CPU and have enough GPU resources, you might also try to use NVIDIA/DALI. @JanuszL might give you some more information about it. Have a look at his post here for some additional information.