Expert Parallelism and Expert Parallelism + Tensor Parallelism need

distributed
1

Hi, I have no idea how to implement ep and ep+tp with current torch api. Is there any examples or tutorials? Is there anyone has tried it?

2

If you mean MoE, yes it is possible to implement MoE with TP via DTensor. We are also planning to explore this parallelism combination but it is not ready yet.

1 Like
3

Hi! If Iā€™m not mistaken expert parallelism is implemented in GitHub - databricks/megablocks

1 Like
4

Thanks. I will check it.

5

Yes, it is what i mean. Pytorch has great api to support TP. But i cannot find any tutorials helpful to implement EP or EP+TP. So i am confused.