Here is the tutorial to build data-parallel and model parallel by using PyTorch https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html

The communication of outputs can be very costly and restricts the level of parallelism that can be achieved by forcing some sort of synchronization.