Performance drops when setting persistent_workers=True

I encounter a performance drop when setting persistent_workers equals True in the datamodule. Should it just handle the workers? Why does it affect in some way also the optimization of the model?

So, what is the proper way to use persistent workers?

A working Colab to reproduce the behavior.

Are you seeing the same issue without wrapping your model into Lightning and if so could you share a minimal and executable code snippet, please?