GPipe: Easy Scaling with Micro-Batch Pipeline…

GPipe divides a neural network into sequential stages and sends micro-batches through them as a pipeline. Read it to see how the schedule trades idle time for throughput, why accumulated gradients preserve synchronous updates, and how rematerialization makes much larger models fit in accelerator memory.

Reading focus: How layer partitions and micro-batches turn model parallelism into a diagonal pipeline schedule. Why increasing the number of micro-batches shrinks the fill-and-drain bubble without changing the mini-batch update. How activation rematerialization saves memory and why stage balance still limits throughput.

NeurIPS 2019. Yanping Huang et al.. 32 min read, easy difficulty.