Can __torch_function__ only intercept specific API?

I defined a custom DeviceContext (like the one in torch/utils/ based on torch_function. I find that the context will intercept calling of all torch apis, which decrease the performance of training especially in model with multiple small ops like InceptionV3.

Is there a way to limit the interception?