A prompt of one million tokens is roughly two full-length novels. Feed that to a model and the first thing you hit is not a clever-algorithm problem — it is a wall. Two walls, actually. The memory wall: holding the intermediate state for a million tokens, across every layer, will not fit on one GPU. And the throughput wall: if you spread the model across many GPUs the naive way, most of them sit idle most of the time, and the user waits the better part of a minute before the first token appears. The LMSYS post this reading is built on takes those two walls head-on, and the answer it arrives at is, on its face, surprising: pipeline parallelism — the parallelism strategy that everyone is taught to avoid for inference — turns out to be the best way to serve ultra-long context across machines.
一个一百万 token 的 prompt， 大概是两本长篇小说。 把它喂给模型， 你第一个撞上的不是什么"巧妙算法"问题 —— 是一堵墙。 准确说是两堵。 内存墙： 要为一百万 token、 在每一层都保存中间状态， 单张 GPU 装不下。 还有吞吐墙： 如果你用最朴素的办法把模型摊到很多张卡上， 大多数卡在大多数时间里都在发呆， 用户得等将近一分钟才看到第一个 token。 本篇精读所基于的那篇 LMSYS 博客正面硬刚这两堵墙， 而它给出的答案乍看让人意外： 流水线并行 —— 那个推理场景里人人都被教导要躲开的并行策略 —— 恰恰是跨机器服务超长上下文的最优解。
Why surprising, and why it still works, is the whole story. This is not a summary of the post — it is a rebuild of it from first principles, so that by the end you can feel why each piece is there. We will earn the three communication-volume formulas instead of quoting them, derive the pipeline bubble from counting idle squares on a clock, watch one long prompt shatter into a stream of chunks, and finish at the subtlest idea in the paper: sizing each chunk so that every one takes the same amount of wall-clock time, even though attention makes later chunks intrinsically more expensive.
为什么意外、 又为什么照样成立 —— 这就是全部故事。 本文不是对那篇博客的复述， 而是从第一性原理把它重建一遍， 让你读到最后能真切感到每一块为什么在那儿。 我们会把三条通信量公式一步步挣出来， 而不是照抄； 会通过数时钟上的空格子把流水线气泡推导出来； 会看着一条长 prompt 碎成一串 chunk； 最后落到全篇最微妙的那个想法： 给每个 chunk 定一个尺寸， 让它们花的墙钟时间都一样长 —— 哪怕注意力机制让后面的 chunk 天生更贵。
If you have twenty minutes, read § 1 (the three cuts) and § 3 (chunked pipelining). Those carry the mental model. § 5 (dynamic chunking) is the deepest part — it rewards a second pass. The benchmark reading in § 7 is the one section you can hand to someone who only cares whether it works.
如果你只有二十分钟， 读 § 1（三种切法）和 § 3（分块流水线）。 这两节给出心智模型。 § 5（动态分块）是最深的一段 —— 值得读第二遍。 § 7 那段"读懂数字"， 是你可以直接丢给"只关心它到底行不行"的人的一节。
A note on where I sit. I work on AMD GPU kernels — MI300X, MI355X, ROCm, CK, Triton-for-ROCm — so I read this post through a roofline lens, and I flag throughout where the AMD cross-node story is the same physics as NVIDIA and where it is sharper. Pipeline parallelism is, at heart, a way of choosing which slow link your bottleneck lives on; on a cluster without NVLink between nodes, that choice matters more, not less.
说一句我的位置。 我做的是 AMD GPU kernel —— MI300X、 MI355X、 ROCm、 CK、 ROCm 上的 Triton —— 所以我是带着 roofline 的视角读这篇博客的， 全文我都会标出： AMD 的跨节点故事在哪里和 NVIDIA 是同一套物理、 又在哪里更尖锐。 流水线并行本质上是在选"你的瓶颈落在哪条慢链路上"； 在一个节点之间没有 NVLink 的集群上， 这个选择只会更重要， 不会更次要。
§ 1 · Where do you make the cut?Three ways to split a model
§ 1 · 刀往哪儿切？切分模型的三种方式
One fact forces everything that follows: a model too big — or a context too long — to run on a single GPU has to be cut into pieces, and the pieces have to talk across the network. Communication is the tax you pay for cutting. Every named parallelism strategy is just a different answer to where do I make the cut?, and the cut you choose decides what data crosses the wire, how often, and over which link. That is the whole game.
有一个事实决定了之后的一切： 一个大到（或者一段上下文长到）单张 GPU 跑不动的模型， 必须被切成块， 而这些块得跨网络通信。 通信， 就是你为"切"付的税。 每一个有名字的并行策略， 不过是对"刀往哪儿切"给出的不同答案； 而你切的位置， 决定了哪些数据要过线、 多频繁地过、 走哪条链路。 这就是全部的博弈。
Picture the transformer forward pass as a loaf of bread with three axes. H is the hidden dimension — the width of every vector (say 8192). S is the sequence — the number of tokens (up to a million). L is the layer stack — the depth (say 60 stacked blocks). There are exactly three clean ways to slice the loaf, and each named strategy slices along one axis.
把 transformer 的前向过程想象成一条有三个轴的面包。 H 是 hidden 维度 —— 每个向量的宽度（比如 8192）。 S 是序列 —— token 的数量（最多一百万）。 L 是层栈 —— 深度（比如 60 个堆叠的块）。 把这条面包切干净， 正好只有三种切法， 而每个有名字的策略都沿着其中一个轴下刀。
Tensor parallelism slices H
张量并行切 H
Inside every layer, the big weight matrices — the QKV projection, the MLP up and down projections — get cut into vertical columns. GPU 0 holds the first H/N columns, GPU 1 the next, and so on. Each GPU computes a partial result for the full sequence but only its slice of the hidden dimension. Here is the catch, and it is not optional: to get the true output of attention or the MLP, you must sum the partial results back together across all GPUs. That summation is an AllReduce, and it fires twice per layer — once after attention, once after the MLP — because the math literally does not produce a correct activation until the slices are recombined. So TP couples its GPUs tightly: they synchronize 2L times in a single forward pass.
在每一层内部， 那些大权重矩阵 —— QKV 投影、 MLP 的上投影和下投影 —— 被切成竖直的列。 GPU 0 拿前 H/N 列， GPU 1 拿下一段， 以此类推。 每张卡都为完整的序列算出一个部分结果， 但只算它那一片 hidden 维度。 关键的坑在这里， 而且不是可选项： 要拿到注意力或 MLP 的真正输出， 你必须把所有卡上的部分结果加回到一起。 这个求和就是一次 AllReduce， 每层触发两次 —— 注意力之后一次， MLP 之后一次 —— 因为在切片重新合并之前， 数学上根本就没产出一个正确的激活。 所以 TP 把它的 GPU 绑得很紧： 一次前向里它们要同步 2L 次。
Context parallelism slices S
上下文并行切 S
Each GPU owns a slab of the tokens — GPU 0 holds tokens 0–250k, GPU 1 holds 250k–500k — and keeps the full hidden dimension and the full layer stack. This is the natural answer to "my context is too long," which is exactly the million-token regime. The catch is attention: token i must attend to every earlier token, including ones living on other GPUs. So at every layer the shards exchange their K and V tensors around a ring until each query block has seen every key block. The post calls this an AllGather of KV, which is right for the byte count; the production implementation is ring attention — point-to-point KV passing that never materializes the whole KV at once. Either way the cost lands once per layer (×L), and crucially it moves only KV — not the full hidden state. With GQA or MLA shrinking H_KV far below H, that makes CP cheaper than TP.
每张 GPU 拥有一块 token —— GPU 0 拿 token 0–250k， GPU 1 拿 250k–500k —— 同时保留完整的 hidden 维度和完整的层栈。 这是对"我的上下文太长了"最自然的回答， 也正是百万 token 这个场景。 坑在注意力： token i 必须 attend 到每一个更早的 token， 包括住在别的卡上的那些。 于是在每一层， 各分片把自己的 K 和 V 沿着一个环互相传递， 直到每个 query 块都见过每个 key 块。 博客把这叫做对 KV 的 AllGather， 就字节数而言这没错； 但生产实现是 ring attention —— 点对点地传 KV， 从不把整个 KV 一次性物化出来。 无论哪种， 代价都是每层一次（×L）， 而且关键是： 它只搬 KV， 不搬完整的 hidden state。 当 GQA 或 MLA 把 H_KV 压到远小于 H 时， 这就让 CP 比 TP 便宜。
Pipeline parallelism slices L
流水线并行切 L
GPU 0 holds layers 0–14, GPU 1 holds 15–29, and so on. A token's hidden state flows GPU 0 → GPU 1 → … like an assembly line. The only thing that crosses a boundary is the hidden state itself, handed off once at each of the P−1 stage seams with a plain point-to-point send — no collective, no summation. That is the cheapest pattern of the three, and it hides the asymmetry that the whole post turns on: TP and CP pay their tax at every layer (L times); PP pays only at stage seams (P−1 times). Since real clusters have far fewer stages than layers (P ≪ L), PP's total communication is roughly an order of magnitude less. The price it pays instead is the bubble — GPU 1 sits idle until GPU 0 finishes the first piece of work — and the next section is entirely about that bubble.
GPU 0 拿第 0–14 层， GPU 1 拿第 15–29 层， 以此类推。 一个 token 的 hidden state 像流水线一样流过 GPU 0 → GPU 1 → …。 跨越边界的只有 hidden state 本身， 在 P−1 个 stage 接缝处各交接一次， 用一个朴素的点对点 send —— 没有集合通信， 没有求和。 这是三者里最便宜的模式， 它也藏着整篇博客的命门： TP 和 CP 在每一层都交税（共 L 次）； PP 只在 stage 接缝处交（共 P−1 次）。 由于真实集群的 stage 数远少于层数（P ≪ L）， PP 的总通信量大约小一个数量级。 它转而付出的代价是气泡 —— GPU 1 要一直发呆， 直到 GPU 0 做完第一份活儿 —— 下一节就完全在讲这个气泡。
Earning the three formulas
把三条公式挣出来
The post states three communication-volume formulas. Rather than quote them, let us build them from one seed. For a ring collective over N GPUs on a tensor of total size K bytes, the bandwidth-optimal cost per GPU is 2(N−1)/N·K for an AllReduce and (N−1)/N·K for an AllGather. As N grows these factors saturate at 2 and 1 — note saturate, they do not keep climbing with N. That single fact is the seed of all three constants.
博客给了三条通信量公式。 与其照抄， 不如从一颗种子把它们长出来。 对于一个跨 N 张 GPU、 张量总大小为 K 字节的环形集合通信， 带宽最优的每卡代价是： AllReduce 为 2(N−1)/N·K， AllGather 为 (N−1)/N·K。 当 N 增大， 这两个系数分别饱和到 2 和 1 —— 注意是饱和， 它们不会随 N 一直往上爬。 这一个事实， 就是全部三个常数的种子。
Now divide. Vol(TP) / Vol(PP) ≈ 4L/(P−1). For a 60-layer model at P=4 that is about 80× — nearly an order of magnitude. TP and CP synchronize once or twice per layer, so their frequency scales with L; PP synchronizes only at stage seams, so its frequency scales with P. Because P ≪ L on any real cluster, PP slashes the bytes that must cross the slow links. (One unit caveat the post folds away: B·S·H counts tensor elements; multiply by 2 for BF16 to get bytes. It cancels in the ratio, so the comparison is unaffected.)
现在做除法。 Vol(TP) / Vol(PP) ≈ 4L/(P−1)。 对一个 60 层的模型、 P=4 时， 大约是 80× —— 接近一个数量级。 TP 和 CP 每层同步一两次， 所以频率随 L 缩放； PP 只在 stage 接缝处同步， 所以频率随 P 缩放。 由于任何真实集群上都有 P ≪ L， PP 大幅削减了必须穿过慢链路的字节数。 （博客省掉的一个单位提醒： B·S·H 数的是张量元素个数； 乘以 2 才是 BF16 的字节数。 它在比值里会约掉， 所以不影响比较。）
4L/(P−1). The red-dense TP and CP tracks vs the near-empty PP track is the order-of-magnitude saving, drawn to scale.4L/(P−1) 背后的画面。 密密麻麻的 TP、 CP 轨道， 对比几乎空白的 PP 轨道， 就是那个"按比例画出来"的数量级节省。The hierarchy that falls out
由此自然得出的层级
Put the four properties side by side and the deployment recipe writes itself. TP and CP are bubble-free — every device computes at once — but they are communication-heavy, firing a collective on every layer. PP is the inverse: almost no communication, but it pays a pipeline bubble. So you do not pick one; you nest them. TP or CP inside a node, where NVLink-class bandwidth can absorb the frequent collectives, and PP across nodes, where the slow inter-node link can only afford PP's rare, small point-to-point sends. The post's headline configs — PP4 TP8 for DeepSeek, PP8 TP4 for Qwen — are exactly this: tensor-parallel within each 8- or 4-GPU node, pipeline-parallel across the nodes.
把四条性质并排放在一起， 部署配方就自己写出来了。 TP 和 CP 是无气泡的 —— 所有设备同时计算 —— 但它们通信很重， 每一层都要发一次集合通信。 PP 正相反： 几乎不通信， 但要付一个流水线气泡。 所以你不是择其一， 而是把它们嵌套： 节点内用 TP 或 CP， 让 NVLink 级别的带宽去吸收那些频繁的集合通信； 节点间用 PP， 让慢速的跨节点链路只需承担 PP 那些稀疏、 小巧的点对点 send。 博客里的头条配置 —— DeepSeek 的 PP4 TP8、 Qwen 的 PP8 TP4 —— 正是这个意思： 每个 8 卡或 4 卡节点内部做张量并行， 节点之间做流水线并行。
| Property | TP | CP | PP | 性质 | TP | CP | PP |
|---|---|---|---|---|---|---|---|
| Splits | 切分 | hidden H | sequence S | layers L | |||
| Pattern | 通信模式 | AllReduce ×L | KV ring ×L | P2P × (P−1) | |||
| Volume | 通信量 | high · 4BSHL | medium · 2BS·H_KV·L | low · BSH(P−1) | |||
| Bubble | 气泡 | 0 | 0 | (P−1)/(P−1+M) | |||
| Best link | 最适链路 | NVLink (intra-node) | NVLink (intra-node) | Ethernet/IB (cross-node) |
It is tempting to think PP communicates less because each send is smaller. It is not the packet size — it is the frequency. PP sends at P−1 seams; TP and CP synchronize at every one of L layers. And do not expect per-GPU TP cost to keep climbing as you add ranks: the ring factor 2(N−1)/N saturates near 2. What actually worsens with TP degree is latency (more ring steps) and the moment a TP group is forced to span nodes — then its frequent collectives hit the slow link, which is exactly the situation PP exists to avoid.
人们很容易以为 PP 通信更少是因为每次 send 更小。 不是包的大小 —— 是频率。 PP 在 P−1 个接缝处 send； TP 和 CP 在 L 层的每一层都同步。 另外别指望每卡 TP 代价会随 rank 增加一直上涨： 环形系数 2(N−1)/N 在 2 附近就饱和了。 真正随 TP degree 变糟的是延迟（更多环步）， 以及一个 TP 组被迫跨节点的那一刻 —— 那时它频繁的集合通信会打到慢链路上， 而这恰恰是 PP 存在的目的所在。
§ 2 · The cost of slicing by layerThe pipeline bubble
§ 2 · 按层切的代价流水线气泡
PP buys cheap communication, but it borrows against a different account: data dependency. Stage 2 cannot start until stage 1 has produced something for it to consume. Feed the pipeline one monolithic batch and watch the clock. At t=1, stage 1 computes the whole batch while stages 2…P sit idle. At t=2, stage 2 computes while stage 1 — already finished, nothing new to do — goes idle. The single batch walks the pipe one stage at a time, and only one of your P GPUs is ever busy. You bought P GPUs and got the throughput of one. That dead time is the bubble.
PP 换来了便宜的通信， 但它是从另一个账户上借的： 数据依赖。 Stage 2 没法开工， 除非 stage 1 先产出点东西给它消费。 给流水线喂一个整块的 batch， 盯着时钟看。 t=1 时， stage 1 在算整个 batch， 而 stage 2…P 都在发呆。 t=2 时， stage 2 在算， 而 stage 1 —— 已经算完、 没有新活儿 —— 闲下来了。 这一个 batch 一次走一个 stage， 任何时刻都只有 P 张卡里的一张在忙。 你买了 P 张卡， 拿到的是一张卡的吞吐。 这段死时间就是气泡。
The fix is micro-batching, introduced by GPipe (Huang et al., NeurIPS 2019). Chop the batch into M smaller micro-batches and stream them. Now stage 1 finishes micro-batch 1 and immediately starts micro-batch 2, while stage 2 picks up micro-batch 1. Once the pipeline fills, every stage works on a different micro-batch at once — full utilization. PipeDream (Narayanan et al., SOSP 2019) pushed this further with the 1F1B schedule that interleaves forward and backward; for inference prefill we only care about the forward half, but the bubble intuition is identical.
解法是微批（micro-batching）， 由 GPipe（Huang 等， NeurIPS 2019）提出。 把 batch 切成 M 个更小的微批， 流式地喂进去。 现在 stage 1 做完微批 1 就立刻开始微批 2， 而 stage 2 接手微批 1。 一旦流水线填满， 每个 stage 同时在处理不同的微批 —— 满负荷。 PipeDream（Narayanan 等， SOSP 2019）用 1F1B 调度把这一步推得更远， 交错前向和反向； 推理的 prefill 阶段我们只关心前向那一半， 但气泡的直觉完全一样。
P, around a steady-state band that grows with M. The whole efficiency game is making M dwarf P.P 决定的、 固定面积的三角形， 夹着一条随 M 增长的稳态带。 整个效率博弈就是让 M 远远盖过 P。Deriving the bubble ratio
推导气泡比
Use one micro-batch's per-stage time as the unit. The first micro-batch enters stage 1 at slot 1, reaches stage 2 at slot 2, … reaches stage P at slot P — so P−1 slots elapse before the last stage does any useful work. That is the fill. Once the pipe is full, pushing all M micro-batches through takes M slots of useful, zero-waste work — the steady state. After stage 1 consumes the last micro-batch it idles while that piece drains down the remaining stages — another P−1 slots, the drain, symmetric to fill. Following Megatron-LM's accounting, the useful time is M and the bubble overhead is P−1, so:
用一个微批在单个 stage 上的时间作为单位。 第一个微批在第 1 个时隙进入 stage 1， 第 2 个时隙到 stage 2， …… 第 P 个时隙到 stage P —— 所以在最后一个 stage 干任何有用活儿之前， 已经过去了 P−1 个时隙。 这就是填充。 一旦流水线填满， 把全部 M 个微批推完需要 M 个时隙的有用、 零浪费的工作 —— 这是稳态。 stage 1 消费完最后一个微批后就闲下来， 等那一块流过剩下的 stage —— 又是 P−1 个时隙， 即排空， 与填充对称。 按 Megatron-LM 的记账方式， 有用时间是 M， 气泡开销是 P−1， 于是：
As M grows with P fixed, the ratio collapses toward (P−1)/M → 0. The bubble is a fixed cost set by pipeline depth; useful work is amortizable by feeding more micro-batches. GPipe's rule of thumb — keep M ≳ 4P — is the practical reading. But read the inverse too, because it is the warning the post is built around: for a fixed workload M, raising P raises the bubble. Deeper pipelines are strictly worse unless you also grow M. That is why "a pure high-degree PP configuration is generally not recommended."
当 M 增大、 P 固定时， 这个比值朝 (P−1)/M → 0 坍缩。 气泡是由流水线深度决定的固定成本； 有用的工作可以靠多喂微批来摊薄。 GPipe 的经验法则 —— 保持 M ≳ 4P —— 就是它的实用读法。 但也要读它的反面， 因为这正是整篇博客所围绕的警告： 对固定的工作量 M， 增大 P 会增大气泡。 更深的流水线严格地更糟， 除非你同时把 M 也做大。 这就是为什么"纯粹的高 degree PP 配置一般不被推荐"。
Here is the move. In long-context prefill, one long prompt is chopped into chunks, and each chunk is a micro-batch. A million-token prompt at a 4K chunk size is ~244 micro-batches — so M ≫ P for free, and the bubble that normally kills PP vanishes on its own. The workload supplies the huge M that PP needs. That is why PP, the wrong tool for decode (one token at a time, tiny M), is the right tool for million-token prefill. Same hardware, opposite verdict — decided entirely by where M comes from.
关键的一步在这里。 在长上下文的 prefill 阶段， 一条长 prompt 被切成 chunk， 而每个 chunk 就是一个微批。 一条百万 token 的 prompt、 4K 的 chunk 大小， 大约是 244 个微批 —— 于是 M ≫ P 白送给你， 那个平时会要了 PP 命的气泡自己就消失了。 工作量自带了 PP 所需要的那个巨大的 M。 这就是为什么 PP 这个 decode 的错误工具（一次一个 token、 M 极小）， 偏偏是百万 token prefill 的正确工具。 同样的硬件， 相反的判决 —— 完全由 M 从哪儿来决定。
§ 3 · The heart of itChunked pipeline parallelism
§ 3 · 核心所在分块流水线并行
With P=8 stages, stage 0 holds layers 0–9, stage 1 holds 10–19, and a token's hidden state must flow through all eight before you get a logit. The naive way to run a long prompt is to push the whole prompt through stage 0, send the result to stage 1, push it all through stage 1, and so on. Correct, but it makes your time-to-first-token catastrophic: you wait for stage 0 to finish all million tokens before stage 1 even starts. Startup latency is proportional to total sequence length, and seven of eight GPUs idle at any instant.
在 P=8 个 stage 下， stage 0 拿第 0–9 层， stage 1 拿第 10–19 层， 一个 token 的 hidden state 要流过全部八个 stage 才能得到一个 logit。 跑一条长 prompt 的朴素办法， 是把整条 prompt 推过 stage 0， 把结果送给 stage 1， 再整条推过 stage 1， 如此往复。 这是对的， 但它让你的 time-to-first-token 灾难性地差： 你得等 stage 0 把全部一百万 token 都算完， stage 1 才开始。 启动延迟正比于总序列长度， 而任何时刻八张卡里有七张在闲着。
Chunked pipeline parallelism (CPP) stops treating the prompt as a monolith. Cut it into chunks of a few thousand tokens and treat each chunk like a GPipe micro-batch. Stage 0 processes Chunk 1, fires it downstream, and immediately starts Chunk 2 while stage 1 works on Chunk 1. Now all stages can be busy at once — but the deeper win is the startup. The pipeline produces its first downstream work after just the first chunk, so startup latency becomes proportional to first-chunk size, not total length. For a 1M-token prompt at 4K chunks, that is a ~244× smaller fill cost before the pipe is full. This is also why CPP is a prefill story specifically: a long prompt is naturally chunkable along the sequence dimension, and SGLang already had chunked prefill for memory reasons, so PP rides on top of machinery that was already there.
分块流水线并行（CPP）不再把 prompt 当成铁板一块。 把它切成几千 token 一块的 chunk， 把每个 chunk 当成一个 GPipe 微批。 stage 0 处理 Chunk 1， 把它发往下游， 然后立刻开始 Chunk 2， 而 stage 1 正在处理 Chunk 1。 现在所有 stage 都能同时忙起来 —— 但更深的胜利在启动上。 流水线只需第一个 chunk 就能产出第一份下游工作， 所以启动延迟变成正比于首块大小， 而不是总长度。 对一条 1M token、 4K 分块的 prompt， 这是在流水线填满之前小了约 244× 的填充成本。 这也是为什么 CPP 专门是个 prefill 故事： 一条长 prompt 天然可以沿序列维度分块， 而 SGLang 出于内存原因本就有 chunked prefill， 所以 PP 是搭在一套已经存在的机制之上的。
Make the speedup precise. Let t(n) be one stage's time to forward n tokens, and split the prompt of length L into k chunks of size s = L/k. Naive monolithic PP costs P·t(L) — every stage chews the entire prompt before passing it on. Chunked PP costs P·t(s) to fill the pipe with the first chunk, then (k−1)·t(s) in steady state: (P + k − 1)·t(s). If t is roughly linear in tokens, t(L) = k·t(s), and the ratio is P·k / (P + k − 1). At P=8, k=244, that is ~7.6× — and the fill portion before any output shrank by the full factor of k. (That 7.6× is the speedup over a fully-serialized monolithic-PP strawman; the measured wins later — 81% TTFT cut, 3.31× throughput — are against TP, a much stronger baseline.)
把加速比写精确。 设 t(n) 是一个 stage 前向 n 个 token 的时间， 把长度为 L 的 prompt 切成 k 个大小 s = L/k 的 chunk。 朴素整块 PP 的代价是 P·t(L) —— 每个 stage 都要嚼完整条 prompt 才往下传。 分块 PP 的代价是 P·t(s) 把第一个 chunk 填满流水线， 再加稳态的 (k−1)·t(s)： 合计 (P + k − 1)·t(s)。 若 t 大致随 token 线性， 则 t(L) = k·t(s)， 比值为 P·k / (P + k − 1)。 在 P=8、 k=244 时大约是 7.6× —— 而产出第一份输出前的填充部分整整缩小了 k 倍。 （这个 7.6× 是相对"完全串行的整块 PP"这个稻草人的加速比； 后面那些实测胜绩 —— TTFT 降 81%、 吞吐 3.31× —— 是对 TP 这个强得多的基线而言的。）
§ 4 · Keeping the pipe fullAsync P2P & three streams
§ 4 · 让流水线保持满载异步 P2P 与三条流
Chunking shrinks the fill, but it exposes a new hazard: the boundary between stages is a network send. If stage 0 calls a blocking send after each chunk, it stalls — it cannot start Chunk 2 until Chunk 1's bytes are fully handed off. The pipeline would re-serialize on communication instead of compute, and you would have traded one bubble for another. SGLang's fix is asynchronous P2P: the send returns a handle immediately and the stage keeps working while data is in flight; synchronization is deferred to the last possible moment, right before the buffer must be safe to reuse.
分块缩小了填充， 但它暴露出一个新风险： stage 之间的边界是一次网络 send。 如果 stage 0 在每个 chunk 之后调用一个阻塞的 send， 它就卡住了 —— 在 Chunk 1 的字节完全交接出去之前， 它没法开始 Chunk 2。 这样流水线会从"按算力串行"变成"按通信串行"， 等于用一个气泡换了另一个气泡。 SGLang 的解法是异步 P2P： send 立刻返回一个句柄， stage 在数据还在飞的时候继续干活； 同步被推迟到最后一刻 —— 就在缓冲区必须可以安全复用之前。
The mechanism lives in scheduler_pp_mixin.py, and the symbols are worth seeing because they make the design concrete. The send path is _pp_send_pyobj_to_next_stage(self, data, async_send=False); when async_send=True it issues a non-blocking torch.distributed.isend and returns a list of P2PWork handles instead of blocking. Each handle is a tiny dataclass — P2PWork(work, payload) — bundling the torch.distributed.Work object you will wait() on with the tensor payload. Bundling the payload is not incidental: keeping a reference alive stops the buffer from being freed while isend is still reading it, the classic async-comm correctness trap. Synchronization is deferred to _pp_commit_comm_work(self, work), which loops for p2p_work in work: p2p_work.work.wait() and clears the list — the "sync later" half of the design.
这套机制就在 scheduler_pp_mixin.py 里， 它的符号值得一看， 因为它们把设计落到了实处。 send 路径是 _pp_send_pyobj_to_next_stage(self, data, async_send=False)； 当 async_send=True 时， 它发起一个非阻塞的 torch.distributed.isend， 返回一组 P2PWork 句柄而不是阻塞。 每个句柄是一个小 dataclass —— P2PWork(work, payload) —— 把你将要 wait() 的那个 torch.distributed.Work 对象， 连同张量 payload 一起打包。 打包 payload 不是顺手为之： 保住一份引用， 才能防止缓冲区在 isend 还在读它的时候被释放 —— 这是异步通信里经典的正确性陷阱。 同步被推迟到 _pp_commit_comm_work(self, work)， 它循环 for p2p_work in work: p2p_work.work.wait() 然后清空列表 —— 这是设计里"晚点再同步"的那一半。
The third idea is overlap via separate CUDA streams. A stream is an ordered queue of GPU work; separate streams run concurrently. SGLang runs a forward_stream for the current micro-batch's matmuls, a copy_stream for device-to-host transfers of a previous micro-batch's results, and the default stream plus a launch_event to enforce the few real cross-stream dependencies. So while the GPU grinds on micro-batch N's forward pass, the CPU is already building micro-batch N+1 (self.mbs[mb_id] = self.get_next_batch_to_run()) and the copy stream is moving micro-batch N−1's output off the device. Compute, communication, and CPU bookkeeping all happen at once instead of in a sad serial line — the difference between a pipeline that idles between chunks and one that stays saturated.
第三个想法是用不同的 CUDA stream 做重叠。 一条 stream 是一个有序的 GPU 工作队列； 不同的 stream 可以并发运行。 SGLang 跑一条 forward_stream 处理当前微批的矩阵乘， 一条 copy_stream 做上一个微批结果的 device-to-host 传输， 再加默认 stream 和一个 launch_event 来强制那少数几个真正的跨流依赖。 于是当 GPU 在啃微批 N 的前向时， CPU 已经在构建微批 N+1（self.mbs[mb_id] = self.get_next_batch_to_run()）， 而 copy stream 正在把微批 N−1 的输出搬下卡。 计算、 通信、 CPU 记账同时发生， 而不是排成一条可怜的串行队 —— 这就是"chunk 之间会发呆的流水线"和"始终满载的流水线"之间的差别。
Pipeline parallelism landed in SGLang via PR #5724 ("[PP] Add pipeline parallelism," by Ying1123) and was made compatible with prefill-decode disaggregation in PR #8846; the roadmap is tracked in issue #11857. It needs SGLang ≥ v0.5.7. "Async P2P does not mean no synchronization" — it is deferred, not eliminated; forget to commit before reusing a buffer and you get silent corruption, which is exactly why P2PWork keeps the payload alive.
流水线并行通过 PR #5724（"[PP] Add pipeline parallelism"， 作者 Ying1123）进入 SGLang， 又在 PR #8846 中做到了与 prefill-decode 分离兼容； 路线图在 issue #11857 跟踪。 它需要 SGLang ≥ v0.5.7。 "异步 P2P 不等于不同步" —— 是推迟， 不是取消； 在复用缓冲区之前忘了 commit， 你就会得到静默的数据损坏 —— 这正是 P2PWork 要保住 payload 的原因。
§ 5 · The subtlest idea in the paperDynamic chunking
§ 5 · 全篇最微妙的想法动态分块
The assembly line only runs smoothly if every stage takes the same wall-clock time per chunk. If stage k finishes a chunk in 10 ms but the next chunk arriving from stage k−1 took 14 ms, stage k idles 4 ms — and that idle does not stay local; it ripples downstream and compounds at the higher PP ranks. A balanced pipeline needs per-chunk time to be constant across chunks. Here is the trap with a fixed chunk size, and it is a good one: you would think "same token count = same work." False. In a transformer each new chunk's tokens must attend to all previously cached tokens. Chunk 1 attends to ~0 prior tokens; chunk 10, sitting at prefix length L, attends to ~L. Same token count, far more attention work. So per-chunk time rises as the prefix grows — fixed chunks guarantee a slowly-worsening misalignment, worst at the highest PP ranks.
这条流水线只有在"每个 stage 处理每个 chunk 的墙钟时间都一样"时才跑得顺。 如果 stage k 用 10 ms 处理完一个 chunk， 但从 stage k−1 传来的下一个 chunk 花了 14 ms， stage k 就会空转 4 ms —— 而这段空转不会留在原地； 它向下游扩散， 并在更高的 PP rank 上叠加。 一条平衡的流水线需要每 chunk 的时间在各 chunk 之间恒定。 固定 chunk 大小的陷阱在这里， 而且是个漂亮的陷阱： 你会以为"token 数一样 = 工作量一样"。 错。 在 transformer 里， 每个新 chunk 的 token 都必须 attend 到所有已缓存的 token。 Chunk 1 attend 到约 0 个先前 token； chunk 10 处在前缀长度 L， 要 attend 到约 L 个。 token 数一样， 注意力工作量却多得多。 所以每 chunk 时间随前缀增长而上升 —— 固定分块注定会带来一种缓慢恶化的错位， 在最高的 PP rank 上最糟。
The fix is conceptually clean: don't hold chunk size constant, hold chunk time constant. The first chunk is cheap (no prefix yet); its runtime becomes the pipeline's target tempo. Every later chunk should be sized so it takes exactly that same wall-clock time — and since each later chunk is more expensive per token, you give it fewer tokens. Big chunks early, shrinking chunks late. Deep in a million-token prompt a chunk might be a quarter the size of the first but still take the same time, because each of its tokens does four times the attention work.
解法在概念上很干净： 别让 chunk 的大小恒定， 让 chunk 的时间恒定。 第一个 chunk 便宜（还没有前缀）； 它的运行时间成为流水线的目标节拍。 之后每个 chunk 都应被定成花同样的墙钟时间 —— 而既然越往后每 token 越贵， 你就给它更少的 token。 前面 chunk 大， 后面 chunk 缩。 深入到百万 token prompt 的后段， 一个 chunk 可能只有第一个的四分之一大， 却仍花同样的时间， 因为它的每个 token 要做四倍的注意力工作。
Why Runtime(L) is quadratic
为什么 Runtime(L) 是二次的
To do this at runtime you need a model of how runtime grows with prefix length, and the shape is not a guess — it falls out of attention. Take a chunk of ΔL new query tokens appended after a prefix of L cached tokens. Two cost terms. The attention term: each of the ΔL queries scores against ~L cached keys, so attention ≈ \(\Delta L \cdot L\) — this is the term that depends on the prefix. The linear term: each new token does a fixed amount of projection and MLP matmul, independent of prefix, so ≈ ΔL. The per-token marginal cost at position L is therefore \(\alpha L + b\). Integrate that from 0 to S and you get the cumulative runtime — a clean quadratic:
要在运行时做到这点， 你需要一个"运行时间如何随前缀长度增长"的模型， 而它的形状不是猜的 —— 它从注意力里掉出来。 取一个 ΔL 个新 query token 的 chunk， 接在 L 个已缓存 token 的前缀之后。 两个代价项。 注意力项： ΔL 个 query 中每一个都要对约 L 个已缓存 key 算分， 所以注意力 ≈ \(\Delta L \cdot L\) —— 这是依赖前缀的那一项。 线性项： 每个新 token 做固定量的投影和 MLP 矩阵乘， 与前缀无关， 所以 ≈ ΔL。 因此位置 L 处的每 token 边际代价是 \(\alpha L + b\)。 把它从 0 积到 S， 你得到累计运行时间 —— 一个干净的二次式：
This is exactly the quadratic SGLang fits — by profiling requests at a range of input token lengths (ITLs) it recovers a and b. One symbol-hygiene warning the verifier flagged: keep the per-query attention coefficient α distinct from the cumulative coefficient \(a = \alpha/2\); conflate them and your ΔL comes out a factor of two wrong.
这正是 SGLang 拟合的那个二次式 —— 通过在一系列输入 token 长度（ITL）上 profile 请求， 它把 a 和 b 找回来。 一个被审校者点出的符号卫生警告： 把每 query 的注意力系数 α 和累计系数 \(a = \alpha/2\) 区分开； 混为一谈， 你解出的 ΔL 就会差两倍。
Solving for the next chunk size
解出下一个 chunk 的大小
Now the payoff. We want each chunk to take the same time as the cheap first chunk, of size C₀: target \(T = \mathrm{Runtime}(C_0) = a\,C_0^2 + b\,C_0\). For the next chunk of unknown size ΔL appended at current prefix L, its time is the increment \(\mathrm{Runtime}(L{+}\Delta L) - \mathrm{Runtime}(L)\). Expand and set equal to T — that is the post's stated condition \(\mathrm{Runtime}(L{+}\Delta L) - \mathrm{Runtime}(L) = \mathrm{Runtime}(\text{InitialChunkSize})\) — and you get a quadratic in the unknown ΔL:
现在是回报。 我们要每个 chunk 都花和便宜的第一个 chunk（大小 C₀）一样的时间： 目标 \(T = \mathrm{Runtime}(C_0) = a\,C_0^2 + b\,C_0\)。 对接在当前前缀 L 之后、 大小未知 ΔL 的下一个 chunk， 它的时间是增量 \(\mathrm{Runtime}(L{+}\Delta L) - \mathrm{Runtime}(L)\)。 展开并令其等于 T —— 这就是博客给出的条件 \(\mathrm{Runtime}(L{+}\Delta L) - \mathrm{Runtime}(L) = \mathrm{Runtime}(\text{InitialChunkSize})\) —— 你得到一个关于未知量 ΔL 的二次方程：
One quadratic solve per chunk, at runtime. Then three engineering clamps the post specifies, in order: scale by SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR (default 0.75) to damp overshoot from an imperfect fit; align the result downward to a multiple of max(--page-size, 64) so chunk boundaries respect KV-cache paging and land on efficient kernel-tile shapes; and floor it so later chunks stay at least 1/4 of the initial size, keeping kernels out of inefficient tiny-shape territory.
每个 chunk 在运行时解一次二次方程。 然后是博客规定的三道工程夹紧， 按顺序： 乘上 SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR（默认 0.75）来抑制拟合不完美带来的过冲； 把结果向下对齐到 max(--page-size, 64) 的倍数， 让 chunk 边界尊重 KV 缓存的分页、 并落在高效的 kernel tile 形状上； 再加一个下限， 让后面的 chunk 至少保持初始大小的 1/4， 把 kernel 挡在低效的小形状区之外。
L, so a constant time-rise T covers an ever-shorter span of tokens; that span is the next chunk. Fixed chunks (left) drift slower and leak bubbles; dynamic chunks (right) hold the beat.L 变陡， 所以一个恒定的时间升幅 T 覆盖的 token 跨度越来越短； 那段跨度就是下一个 chunk。 固定 chunk（左）越拖越慢、 漏出气泡； 动态 chunk（右）踩住了节拍。Dynamic chunking equalizes TIME, not SIZE — sizes deliberately shrink so times stay flat. SMOOTH_FACTOR=1.0 is not "maximum smoothing" — it means follow the raw model strictly (no damping); lower values (0.6–0.85) are more conservative; 0 disables dynamic chunking entirely. And ITL here is input token length (the profiling x-axis), not inter-token latency. One more, easy to miss: when layers don't divide evenly, put the larger slice on the highest PP rank (SGLANG_PP_LAYER_PARTITION=15,15,15,16 beats 16,15,15,15) — the last stage idles most, so give it more work.
动态分块让时间相等， 不是大小相等 —— 大小故意收缩， 是为了让时间保持平。 SMOOTH_FACTOR=1.0 不是"最大平滑" —— 它表示严格follow原始模型（不阻尼）； 更小的值（0.6–0.85）更保守； 0 则彻底关掉动态分块。 还有， 这里的 ITL 是输入 token 长度（profile 的横轴）， 不是 inter-token latency。 再补一个容易漏的： 当层数除不尽时， 把更大的那片放到最高的 PP rank 上（SGLANG_PP_LAYER_PARTITION=15,15,15,16 优于 16,15,15,15）—— 最后一个 stage 闲得最多， 就该给它更多活儿。
§ 6 · Two phases, two machinesPD disaggregation & HiCache
§ 6 · 两个阶段， 两套机器PD 分离与 HiCache
One observation explains the production architecture: a transformer serving a request does two completely different jobs that stress the hardware in opposite ways. Prefill ingests the prompt — push all N tokens through at once, every matmul is large, attention is N-attends-to-N, lots of arithmetic per byte of weights loaded. Prefill is compute-bound. Decode generates output one token at a time — to make a single token you load the per-token weight set from HBM and do arithmetic for just that one token, so the arithmetic-per-byte ratio collapses. Decode is memory-bandwidth-bound. (For the MoE models here — DeepSeek-V3.1, Qwen3-235B-A22B — decode loads the active experts plus attention and shared weights, not the full parameter count; the bandwidth-bound conclusion is unchanged.)
有一个观察解释了整个生产架构： 一个 transformer 在服务一个请求时， 做的是两件完全不同的工作， 它们以相反的方式压榨硬件。 Prefill 吞入 prompt —— 一次性把全部 N 个 token 推进去， 每个矩阵乘都很大， 注意力是 N attend N， 每加载一字节权重就做大量算术。 Prefill 是计算受限的。 Decode 一次生成一个 token —— 为产出单个 token， 你要从 HBM 加载每 token 的权重集， 却只为这一个 token 做算术， 于是算术 / 字节比塌缩。 Decode 是访存带宽受限的。 （对这里的 MoE 模型 —— DeepSeek-V3.1、 Qwen3-235B-A22B —— decode 加载的是激活的专家加上注意力和共享权重， 不是全部参数； 带宽受限的结论不变。）
This is the roofline picture, applied to a whole serving phase: prefill sits to the right (high arithmetic intensity, hits the FLOPS ceiling), decode sits to the left (low intensity, hits the bandwidth ceiling). Same silicon, two ceilings. Two phases with opposite bottlenecks should not share one node pool, because tuning that helps one hurts the other — that is the whole motivation for Prefill-Decode disaggregation. Run prefill on one pool (here: high-degree PP over the long context) and decode on a separate pool tuned purely for token-generation throughput. The pools talk by shipping the KV cache, and the clever part is they don't wait: as each prompt chunk finishes prefill, its KV is streamed to the decode node (the mooncake backend does chunk-by-chunk transfer), so the network cost hides behind ongoing compute instead of landing as a serial tax.
这就是 roofline 的画面， 应用到整个服务阶段： prefill 在右边（高算术强度， 撞 FLOPS 天花板）， decode 在左边（低强度， 撞带宽天花板）。 同一块硅， 两条天花板。 两个瓶颈相反的阶段不该共用一个节点池， 因为对一个有利的调优会害了另一个 —— 这就是 Prefill-Decode 分离的全部动机。 prefill 跑在一个池上（这里是： 在长上下文上做高 degree 的 PP）， decode 跑在另一个纯粹为 token 生成吞吐调优的池上。 两个池靠搬运 KV 缓存通信， 而精妙之处在于它们不等： 每个 prompt chunk 一 prefill 完， 它的 KV 就被流式送往 decode 节点（mooncake 后端做逐块传输）， 于是网络成本藏在持续的计算之后， 而不是作为一笔串行的税砸下来。
Two latency numbers to keep straight. TTFT (time to first token) is the wait before the first output appears — dominated by prefill, since the model must read the whole prompt before it can emit anything. For a 1M-token prompt, TTFT is tens of seconds, and shrinking it is what all of this is for. ITL / TPOT (inter-token latency) is the gap between successive output tokens once generation is rolling — a decode-phase number, governed by memory bandwidth. Disaggregation lets you attack each on its own pool.
两个延迟数字要分清。 TTFT（time to first token）是第一个输出出现前的等待 —— 由 prefill 主导， 因为模型必须读完整条 prompt 才能吐出任何东西。 对一条 1M token 的 prompt， TTFT 是几十秒， 而把它缩小正是这一切的目的。 ITL / TPOT（inter-token latency）是生成滚动起来后， 相邻输出 token 之间的间隔 —— 一个 decode 阶段的数字， 由访存带宽决定。 分离让你能在各自的池上分别攻击它们。
HiCache attacks TTFT from a different angle: don't recompute prefill you have already done. It is a radix tree (prefix tree) over KV-cache chunks. Two requests that share a prefix — the same system prompt, or the same conversation history in a multi-turn / agentic loop — share a path down the tree. If a prefix chunk was already prefilled and its KV is still resident across the memory hierarchy HiCache manages (GPU → CPU → disk), the new request reuses it and skips that chunk's prefill entirely. In agentic workloads where every turn re-sends a growing history, this is the difference between re-prefilling 100K tokens each turn and prefilling only the new tokens — a direct TTFT cut. The one caveat: it only helps when requests actually share prefixes; for all-unique prompts the tree has nothing to reuse.
HiCache 从另一个角度攻击 TTFT： 别重算你已经算过的 prefill。 它是一棵架在 KV 缓存 chunk 上的 radix tree（前缀树）。 两个共享前缀的请求 —— 同一个 system prompt， 或多轮 / agentic 循环里同一段对话历史 —— 共享树上一条向下的路径。 如果某个前缀 chunk 已经 prefill 过、 它的 KV 还驻留在 HiCache 管理的内存层级里（GPU → CPU → 磁盘）， 新请求就复用它， 完全跳过那个 chunk 的 prefill。 在每一轮都重发一段不断增长的历史的 agentic 工作里， 这是"每轮重 prefill 10 万 token"和"只 prefill 新 token"之间的差别 —— 直接砍 TTFT。 唯一的提醒： 它只在请求确实共享前缀时有用； 对全都互不相同的 prompt， 树上没有东西可复用。
§ 7 · The one section for skepticsReading the numbers
§ 7 · 给怀疑者的一节读懂这些数字
The benchmarks run on 6 H20 nodes (8 × 96 GB each), on DeepSeek-V3.1 and Qwen3-235B-A22B-FP8, averaged over 16 consecutive requests. Before quoting the headline number, learn the one mental move that keeps you honest: always divide the speedup by the GPU-count multiple. "3.31× throughput vs TP8" used PP4-TP8 = 32 GPUs against a TP8 = 8-GPU baseline — that is 4× the hardware. So the real figure is 3.31 / 4 = 82.8% of perfect linear scaling. The speedup and the "82.8% strong-scaling efficiency" the post reports are the same fact stated twice.
基准测试跑在 6 个 H20 节点上（每节点 8 × 96 GB）， 模型是 DeepSeek-V3.1 和 Qwen3-235B-A22B-FP8， 取 16 个连续请求的平均。 在引用那个头条数字之前， 先学会一个让你保持诚实的心算动作： 永远把加速比除以 GPU 数的倍数。 "对 TP8 的 3.31× 吞吐"用的是 PP4-TP8 = 32 张 GPU， 对的是 TP8 = 8 张的基线 —— 也就是 4 倍的硬件。 所以真实数字是 3.31 / 4 = 82.8% 的完美线性缩放。 加速比和博客报告的"82.8% 强缩放效率"， 是同一个事实说了两遍。
| Config | vs baseline | GPU multiple | Efficiency | 配置 | 相对基线 | GPU 倍数 | 效率 |
|---|---|---|---|---|---|---|---|
| DeepSeek PP4-TP8 · DCK 12K | 3.31× | 4× (vs TP8) | 82.8% | ||||
| DeepSeek PP4-TP8 vs TP32 | +30.5% | 1× (equal 32 GPU) | PP > deep TP | ||||
| DeepSeek worst fixed-12K vs TP32 | +18.4% | 1× (equal 32 GPU) | axis > tuning | ||||
| Qwen PP8-TP4 · DCK 18K | 6.14× | 8× (vs PP1) | 76.9% | ||||
| Qwen PP8-TP4 · static 6K | — | 8× (vs PP1) | 69.6% |
Three readings fall out of that table. First, the fair comparison is equal-GPU: PP4-TP8 vs TP32, both 32 GPUs — and PP wins by 30.5%. Why? TP32 splits every matmul 32 ways and fires an AllReduce across all 32 GPUs every layer; across nodes, without NVLink, that frequent collective on the slow link crushes you. PP4-TP8 only passes activations at 4 stage seams. Second, even the worst fixed-chunk PP config still beats TP32 by 18.4% — meaning the parallelism-axis choice (PP over deep TP for long context) dominates the chunk-tuning choice. Pick the right axis first, then tune. Third, dynamic chunking buys ~7 efficiency points (76.9% vs 69.6% for Qwen at PP8) purely by flattening pipeline bubbles — same hardware, better schedule.
这张表里掉出三条读法。 其一， 公平的比较是等 GPU 的： PP4-TP8 对 TP32， 都是 32 张卡 —— PP 赢 30.5%。 为什么？ TP32 把每个矩阵乘切成 32 份， 每层在全部 32 张卡上发一次 AllReduce； 跨节点、 没有 NVLink 时， 这种频繁的集合通信打在慢链路上会把你压垮。 PP4-TP8 只在 4 个 stage 接缝处传激活。 其二， 即便是最差的固定分块 PP 配置， 仍比 TP32 快 18.4% —— 这意味着并行轴的选择（长上下文用 PP 而非深 TP）压倒了分块调优的选择。 先选对轴， 再调参。 其三， 动态分块纯靠拉平流水线气泡就买来约 7 个效率点（Qwen 在 PP8 时 76.9% 对 69.6%）—— 同样的硬件， 更好的调度。
Efficiency decaying as PP grows is not a regression — it is physics. The bubble fraction (P−1)/(P−1+M) grows with stage count, and per-boundary cross-node cost does not shrink as you add stages, so E(PP) slopes downward. DeepSeek's 82.8% at PP4 being higher than Qwen's 76.9% at PP8 is exactly what you should expect. A learner who reads flat-or-rising efficiency into more stages is misreading the bubble.
效率随 PP 增大而衰减， 不是退步 —— 是物理。 气泡占比 (P−1)/(P−1+M) 随 stage 数增长， 而每接缝的跨节点代价不会因为加了 stage 就变小， 所以 E(PP) 向下倾。 DeepSeek 在 PP4 的 82.8% 高于 Qwen 在 PP8 的 76.9%， 正是你该预期的。 一个把"stage 越多效率越平甚至越高"读进去的人， 是读错了气泡。
And the TTFT cuts that all of this is for: on a fixed 1M-token prompt, DeepSeek drops from 48.5s (PP1-TP8) to 15.5s (PP4-TP8), a 67.9% cut; Qwen drops from 55.5s (PP1-TP4) to 10.5s (PP8-TP4), 81.1%. Those are the user-facing payoff of every formula above — the minute-long wait turned into ten seconds.
还有这一切所为的 TTFT 降幅： 在固定的 1M token prompt 上， DeepSeek 从 48.5s（PP1-TP8）降到 15.5s（PP4-TP8）， 降 67.9%； Qwen 从 55.5s（PP1-TP4）降到 10.5s（PP8-TP4）， 降 81.1%。 这些是上面每一条公式面向用户的回报 —— 把将近一分钟的等待变成了十秒。
Read through the roofline I use for MI300X / MI355X kernels: PP does not change the per-GPU work — each stage's attention and GEMMs are still bound by the same HBM-bandwidth and peak-FLOPS ceilings, whether the kernels are CK, AITER, or Triton-for-ROCm. What PP does is move the binding constraint to the cross-node link. On an AMD cluster without NVLink between nodes, that link is even more decisive than on an NVLink-DGX fabric — so async P2P and chunk scheduling matter more here, not less. The AMD-vs-NVIDIA gap on this workload is link maturity and software, not compute physics, which is precisely the kind of gap systematic search and agent-driven tuning can close.
用我做 MI300X / MI355X kernel 的 roofline 视角来读： PP 不改变每卡的工作 —— 每个 stage 的注意力和 GEMM 仍受同样的 HBM 带宽和峰值 FLOPS 天花板约束， 无论 kernel 是 CK、 AITER 还是 ROCm 上的 Triton。 PP 做的是把那条起约束作用的瓶颈挪到跨节点链路上。 在一个节点间没有 NVLink 的 AMD 集群上， 那条链路比 NVLink-DGX 架构上更具决定性 —— 所以异步 P2P 和分块调度在这里更重要， 而非更次要。 这个工作负载上的 AMD 对 NVIDIA 差距， 是链路成熟度和软件， 不是计算物理 —— 而这恰恰是系统化搜索和 agent 驱动调优能够弥合的那类差距。
§ 8 · Where learners run agroundReefs
§ 8 · 学习者容易触礁的地方易混淆礁石
The confusions that cost the most, collected in one place.
最费时间的那些混淆， 收在一处。
| The trap | The correction | 陷阱 | 纠正 |
|---|---|---|---|
| "PP communicates less per send" | It's the frequency, not packet size. PP sends at P−1 seams; TP/CP at every one of L layers. |
"PP 每次 send 更少" | 是频率， 不是包大小。 PP 在 P−1 个接缝 send； TP / CP 在 L 层的每层都发。 |
| "More PP stages = faster" | For fixed M, raising P raises the bubble. Deeper is worse unless M grows too. |
"PP stage 越多越快" | 对固定的 M， 增大 P 会增大气泡。 除非 M 也增大， 否则越深越糟。 |
| "3.31× means great scaling" | It used 4× the GPUs. Divide first: 3.31/4 = 82.8% efficiency. | "3.31× 说明缩放很好" | 它用了 4 倍的 GPU。 先做除法： 3.31/4 = 82.8% 效率。 |
| "Dynamic chunking equalizes size" | It equalizes time. Sizes deliberately shrink so each chunk takes the same wall-clock. | "动态分块让大小相等" | 它让时间相等。 大小故意收缩， 好让每个 chunk 花同样的墙钟时间。 |
SMOOTH_FACTOR=1.0 = "max smoothing" |
Opposite: 1.0 follows the model strictly. Lower (0.6–0.85) is more conservative; 0 disables it. | SMOOTH_FACTOR=1.0 = "最大平滑" |
恰恰相反： 1.0 严格 follow 模型。 更低（0.6–0.85）更保守； 0 关闭。 |
| "Async P2P = no sync" | Deferred, not removed. _pp_commit_comm_work still calls wait(); skip it and you corrupt buffers. |
"异步 P2P = 不同步" | 是推迟， 不是取消。 _pp_commit_comm_work 仍要 wait()； 跳过它就会损坏缓冲区。 |
| "TTFT scales linearly with length" | Super-linear: each doubling ×3+. Attention is O(N²); model prefill as quadratic. |
"TTFT 随长度线性" | 超线性： 每翻倍 ×3 以上。 注意力是 O(N²)； 把 prefill 当二次的来建模。 |
| "CP and DP are the same" | DP runs independent sequences; CP splits one sequence, so attention forces a KV ring every layer. | "CP 和 DP 一样" | DP 跑互相独立的序列； CP 切一条序列， 所以注意力每层都强制一次 KV 环传递。 |
§ 9 · From reading to runningRun it yourself
§ 9 · 从读到跑自己跑一遍
The fastest way to make any of this stick is to launch a server and watch the TTFT move. You need SGLang ≥ v0.5.7. Start with fixed chunking to get a baseline, then turn on dynamic chunking and compare. The two knobs that matter most are --chunked-prefill-size (the initial chunk = the pipeline's target tempo) and SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR (how hard it damps the shrink).
让这一切真正记住的最快办法， 是启一个 server， 盯着 TTFT 动。 你需要 SGLang ≥ v0.5.7。 先用固定分块拿个基线， 再打开动态分块对比。 最要紧的两个旋钮是 --chunked-prefill-size（初始 chunk = 流水线的目标节拍）和 SGLANG_DYNAMIC_CHUNKING_SMOOTH_FACTOR（它把收缩阻尼得多狠）。
A tuning path that respects the bubble math: first find the best fixed --chunked-prefill-size for your target PP, then set the dynamic initial size to 2–3× that (4× for extremely long inputs), then sweep the smooth factor in 0.6–0.85. If layers don't divide evenly across ranks, set SGLANG_PP_LAYER_PARTITION with the larger slice on the highest rank. Watch TTFT and the per-stage profile; the win shows up as the higher PP ranks filling in instead of waiting.
一条尊重气泡数学的调优路径： 先为你的目标 PP 找到最好的固定 --chunked-prefill-size， 再把动态初始大小设成它的 2–3 倍（极长输入用 4 倍）， 然后在 0.6–0.85 之间扫 smooth factor。 如果层数在各 rank 上除不尽， 用 SGLANG_PP_LAYER_PARTITION 把更大的片放到最高 rank 上。 盯着 TTFT 和每 stage 的 profile； 胜利的样子是更高的 PP rank 被填满， 而不是在等。
Pipeline parallelism is the parallelism strategy that communicates the least but bubbles the most — and long-context prefill is the one workload that supplies, for free, the enormous micro-batch count that makes the bubble vanish. Chunk the prompt to fill the pipe, send asynchronously so the network never blocks compute, and size each chunk by time rather than tokens so every stage keeps the beat. That is the whole post, and it is why the strategy everyone avoids for inference is the right one at a million tokens.
流水线并行是通信最少、 气泡最多的那种并行策略 —— 而长上下文 prefill 恰好是那个会免费提供巨量微批、 让气泡自行消失的工作负载。 把 prompt 分块来填满流水线， 异步地 send 好让网络永不阻塞计算， 再按时间而非 token 数给每个 chunk 定大小， 让每个 stage 都踩住节拍。 这就是整篇博客 —— 也是为什么那个推理场景里人人躲开的策略， 在一百万 token 上反而是对的。
Source: Shangming Cai, "Pipeline Parallelism in SGLang: Scaling to Million-Token Contexts and Beyond" (LMSYS, 2026-01-15). Code paths verified against sglang/srt/managers/scheduler_pp_mixin.py and distributed/parallel_state.py on main; provenance #5724, #8846, #11857. Companion readings in this Library: SGLang and mini-SGLang (the engine), the AMD kernel-design and rocprof entries (the per-GPU roofline this post sits on top of).
原文： 蔡尚明， 《Pipeline Parallelism in SGLang: Scaling to Million-Token Contexts and Beyond》（LMSYS， 2026-01-15）。 代码路径已对照 main 分支的 sglang/srt/managers/scheduler_pp_mixin.py 和 distributed/parallel_state.py 核验； 来历见 #5724、 #8846、 #11857。 本文库里的同伴： SGLang 与 mini-SGLang（引擎本体）、 AMD kernel 设计与 rocprof 两篇（这篇博客所立足的每卡 roofline）。