Update description2.md
Browse files- description2.md +10 -2
description2.md
CHANGED
|
@@ -1,5 +1,7 @@
|
|
| 1 |
## Building a pipeline
|
| 2 |
|
|
|
|
|
|
|
| 3 |
### Building Block
|
| 4 |
It starts by laying out the passes for a single microbatch, which we call a building
|
| 5 |
block. For example, the building block of 1F1B schedule is made of a sequence of forward passes
|
|
@@ -7,12 +9,14 @@ followed by backward passes in the reverse order.
|
|
| 7 |
|
| 8 |
### Repeating
|
| 9 |
More microbatches are then introduced. The building blocks are repeated and woven
|
| 10 |
-
together to form a pipeline.
|
|
|
|
| 11 |
passes from two building blocks should not overlap with each other.
|
| 12 |
|
| 13 |
### Squeezing
|
| 14 |
Depending on the building block, there may be redundant bubbles in the pipeline, which
|
| 15 |
-
can be simply removed by squeezing without changing the order of the passes.
|
|
|
|
| 16 |
|
| 17 |
### Reordering (optional)
|
| 18 |
We can reorder the passes in the warm-up and cool-down phase to further
|
|
@@ -20,6 +24,10 @@ improve the computation throughput. Intuitively, the peak of memory happens in t
|
|
| 20 |
the pipeline, while in the warm-up and cool-down phases the RAM is under utilized, leaving some
|
| 21 |
space for improving the computation throughput without changing peak memory.
|
| 22 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 23 |
## Alternative schedules
|
| 24 |
|
| 25 |
By utilizing the building block, we can search for different types of schedules depending on the need. We illustrate few of them here below:
|
|
|
|
| 1 |
## Building a pipeline
|
| 2 |
|
| 3 |
+
Most of existing pipeline schedules can be explained under the following 4 step framework. In the example, we illustrate the construction of 1F1B and Eager 1F1B.
|
| 4 |
+
|
| 5 |
### Building Block
|
| 6 |
It starts by laying out the passes for a single microbatch, which we call a building
|
| 7 |
block. For example, the building block of 1F1B schedule is made of a sequence of forward passes
|
|
|
|
| 9 |
|
| 10 |
### Repeating
|
| 11 |
More microbatches are then introduced. The building blocks are repeated and woven
|
| 12 |
+
together to form a pipeline. In the figure below, the repeating building blocks are shown in different shades of grey.
|
| 13 |
+
Notably, legit building blocks are required to repeat without a collision, namely, the
|
| 14 |
passes from two building blocks should not overlap with each other.
|
| 15 |
|
| 16 |
### Squeezing
|
| 17 |
Depending on the building block, there may be redundant bubbles in the pipeline, which
|
| 18 |
+
can be simply removed by squeezing without changing the order of the passes. For example,
|
| 19 |
+
Eager 1F1B shows a case where squeezing produces more efficient pipeline.
|
| 20 |
|
| 21 |
### Reordering (optional)
|
| 22 |
We can reorder the passes in the warm-up and cool-down phase to further
|
|
|
|
| 24 |
the pipeline, while in the warm-up and cool-down phases the RAM is under utilized, leaving some
|
| 25 |
space for improving the computation throughput without changing peak memory.
|
| 26 |
|
| 27 |
+
| 1F1B | Eager 1F1B |
|
| 28 |
+
|-|-|
|
| 29 |
+
| <img src="https://cdn-uploads.huggingface.co/production/uploads/646968e05d7015663950e95b/olufjIyulR25CvAY6V2Hi.jpeg"/> | <img src="https://cdn-uploads.huggingface.co/production/uploads/646968e05d7015663950e95b/MbxeLnxyCGXNa6HVTt3sx.jpeg"/> |
|
| 30 |
+
|
| 31 |
## Alternative schedules
|
| 32 |
|
| 33 |
By utilizing the building block, we can search for different types of schedules depending on the need. We illustrate few of them here below:
|