Difference between DDP vs FSDP.NO_SHARD

distributed
1

Could anyone know the difference between DistributedDataParallel

fsdp.ShardingStrategy = NO_SHARD

They looks pretty similar. regarding memory usage and training speed are they the same ?

2

I think these 2 are the same. cc @agu @weifengpy

3

They are the same high-level algorithm: vanilla data parallelism. However, their implementation differs.

DistributedDataParallel (DDP) uses a C++ reducer to bucket gradients for all-reduce. FullyShardedDataParallel with NO_SHARD follows the module wrapping to bucket gradients for all-reduce.

DDP is more mature and generally the preferred solution over NO_SHARD.

2 Likes
4

Since they are the same, is there a plan to replace it with the FSDP one?