ImageFolder uses the pil_loader by default, if accimage is not the used backend and if no other loader was specified.
This loader converts the images to RGB as seen here.
To use the RGBA channels, you could write a custom loader (basically just copy-paste the pil_loader and remove the convert call or convert to RGBA) and pass it as the loader argument to ImageFolder.