Chapters:
00:00 Introduction
00:51 Theory: Tensor Parallel
02:38 Theory: Column Parallel
03:43 Theory: Row Parallel
05:14 Theory: best combination of Column/Row ?
11:00 Practice: Step 4 new changes
11:55 Practice: apply_tensor_parallel()
14:15 Practice: Column Parallel (init)
16:20 Practice: Row Parallel (init)
16:39 Practice: Column/Row Parallel (forward)
18:50 Practice: Row/Row Parallel (backward)
19:50 Practice: final_proj layer
21:30 Theory: Vocab Embedding Parallel
29:18 Practice: Vocab Embedding Parallel

Part written by Haojun Zhao: [ Ссылка ]
Picotron tutorial: [ Ссылка ]
Picotron codebase: [ Ссылка ]

свернуть