Per-step cost versus accepted tokens每步成本， 与每步接受的 token

The one trade-off唯一的权衡

Speculative decoding replaces one expensive step per token with one expensive step per batch of guessed tokens. Everything that follows is a consequence of that sentence.投机解码把「每个 token 一次昂贵前向」换成「每一批猜测 token 一次昂贵前向」。 后面所有内容都是这句话的推论。

A draft model proposes k tokens; the target model verifies all of them in a single forward pass and keeps the longest correct prefix. Throughput is therefore not a single number you tune — it is a ratio:草稿模型提议 k 个 token， 目标模型在一次前向里验证全部， 然后保留最长的正确前缀。 所以吞吐不是一个你去调的单一数字， 它是一个比值：

throughput  =  accepted tokens per step
               ─────────────────────────
                    step latency

Two ways to make that bigger, and they fight each other. Widen the verify window and the draft proposes more tokens per step, so the numerator grows — but the target now attends over a longer speculative suffix, so the denominator grows too. Narrow the window and the reverse happens. There is no setting that improves both, which is exactly why a search is necessary rather than a rule of thumb.有两种办法把它变大， 而这两种办法互相打架。 把验证窗口加宽， 草稿每步提议更多 token， 分子变大——但目标模型要在更长的投机后缀上做 attention， 分母也跟着变大。 把窗口收窄则相反。 没有哪个设置能同时改善两者， 这正是为什么需要搜索而不是经验法则。

What makes this worth writing down is that the same ratio explains findings that look unrelated. The single largest throughput win in the search — a 1.68x jump from one flag — is a case of trading numerator for a much cheaper denominator. The single largest throughput loss in the sweeps — a 6.35x collapse at long input — is both terms degrading at once. Once you read every number as a fraction, the whole result set stops being a table and becomes one mechanism.值得把这件事写下来的原因是： 同一个比值能解释一批看起来毫不相干的发现。 搜索里最大的单项吞吐收益——某个 flag 带来的 1.68 倍跳升——是拿分子去换一个便宜得多的分母。 扫描里最大的单项吞吐损失——长输入下 6.35 倍的崩塌——是两项同时恶化。 一旦你把每个数字都读成分数， 整份结果就不再是一张表， 而是一套机制。

Shape of the search搜索的形状

Six knobs, each with several plausible values, times a concurrency axis, is thousands of combinations. One configuration costs about 194 seconds to boot — 1.5 TB of weights — plus minutes to benchmark. A full grid is not slow, it is impossible: at three minutes of boot alone, a thousand configurations is fifty hours of doing nothing but loading weights.六个旋钮、每个几个可选值， 再乘一条并发轴， 就是几千种组合。 单个配置光启动就要约 194 秒——1.5 TB 权重——再加上几分钟跑 benchmark。 全网格搜索不是慢， 是不可能： 仅按三分钟启动算， 一千个配置就是五十小时纯粹在装权重。

So the search is staged, and the staging order is not arbitrary. The memory fraction goes first because it is a hard constraint that changes the meaning of every other measurement: a knob evaluated at a different memory budget is a knob evaluated at a different batch capacity, and the comparison is worthless. Fix the ceiling, then sweep knobs at one matched concurrency, then take the winner up the concurrency curve, then repeat the winner to separate signal from noise, then gate on accuracy.所以搜索是分阶段的， 而且阶段顺序不是随意的。 显存占比放在最前面， 因为它是一个硬约束， 会改变其他所有测量的含义： 在不同显存预算下评估的旋钮， 就是在不同 batch 容量下评估的旋钮， 这种对比毫无价值。 先钉住上限， 再在单一匹配并发下扫旋钮， 然后把胜者带上并发曲线， 接着复跑胜者以区分信号与噪声， 最后过精度闸门。

The memory ceiling, and why prediction overshoots显存上限， 以及为什么预测会过头

The bisection starts with an anchor probe at --mem-fraction-static 0.85, reads how much device memory is left after the pools are carved, and extrapolates the fraction that would consume it. For the non-speculative lane the anchor left 35.13 GB free, which predicts 0.95. Both 0.95 and 0.94 died with torch.OutOfMemoryError — at 0.95 the allocator had 1.01 GB free on GPU 1 and wanted 1.75 GB. The winner was 0.93, leaving 12.21 GB.二分从一次 --mem-fraction-static 0.85 的锚点探测开始， 读出各内存池切分完之后设备上还剩多少， 再外推出能把这些吃掉的占比。 非投机通道的锚点剩 35.13 GB， 外推得到 0.95。 结果 0.95 和 0.94 都以 torch.OutOfMemoryError 崩掉——0.95 时分配器在 GPU 1 上只剩 1.01 GB 空闲， 却要申请 1.75 GB。 胜出的是 0.93， 剩 12.21 GB。

Why did a measurement-based prediction overshoot by two whole steps? Because free memory after boot and free memory during work are different quantities. Booting allocates the static pools: weights, KV cache, SSM state. Serving additionally allocates transient activations, and their peak scales with how much work is in flight — the prefill chunk size times the batch. The anchor probe sees the floor of memory use, then the search extrapolates as if that floor were the whole story.为什么一个基于实测的预测会高出整整两档？ 因为「启动后的空闲显存」和「工作中的空闲显存」是两个不同的量。 启动分配的是静态池： 权重、KV cache、SSM 状态。 服务时还要额外分配瞬时激活， 而它的峰值随在途工作量增长——prefill 分块大小乘以 batch。 锚点探测看到的是显存占用的下界， 搜索却把这个下界当成了全貌来外推。

The consequence is a rule worth carrying to any inference deployment: a successful boot does not prove a safe memory setting. The harness therefore gates each candidate with a deliberately hostile probe — input length 16384 to fill an entire prefill chunk, fanned out at the concurrency the lane will actually run at, one wave only. One wave because this is a gate, not a measurement; it must be cheap enough to run on every candidate. Without it the search would happily crown a configuration that boots, benchmarks a small workload, and then dies in production on the first long prompt.由此得到一条值得带到任何推理部署里的规则：启动成功并不能证明显存设置是安全的。 所以 harness 给每个候选都加了一道故意刁难的探测——输入长度 16384 以填满一整个 prefill 分块， 并按该通道实际运行的并发扇出， 只跑一个波次。 只跑一个波次， 因为这是闸门而不是测量， 它必须便宜到每个候选都跑得起。 没有它， 搜索会心满意足地给一个「能启动、在小负载上跑出好成绩、然后在生产环境第一个长 prompt 上暴死」的配置戴上王冠。

The two lanes hit different walls两条通道撞的是两道不同的墙

Kimi-K3 is a hybrid: most layers carry a recurrent SSM state, a minority do full attention. That single architectural fact splits the tuning problem in two, because the two serving lanes exhaust two different pools.Kimi-K3 是混合架构： 大多数层带一个循环 SSM 状态， 少数层做全 attention。 就这一个架构事实把调参问题劈成了两半， 因为两条服务通道耗尽的是两个不同的池。

The speculative lane is capped at 48 concurrent requests, and the cap is not a policy choice — it is what the state pool affords. Speculation needs somewhere to keep the recurrent state for every token it has guessed but not yet verified, so a speculative request reserves several state slots where a plain request reserves one. At 48 running requests the state pool is 98% full while the KV cache sits at 29%.投机通道被限制在 48 个并发请求， 而这个上限不是策略选择——它是状态池能供养的数量。 投机需要地方存放每一个已猜测但尚未验证的 token 的循环状态， 所以一个投机请求要预留数个状态槽位， 而普通请求只占一个。 在 48 个运行请求时， 状态池已经 98% 满， 而 KV cache 才 29%。

The obvious move is to raise the cap, and the search tried it four times. It does not work, and the way it fails is instructive: --max-running-requests and max_total_num_tokens trade against each other, because reserving more request slots means reserving more state per slot from the same fixed pool. Raising the cap therefore shrinks the token budget, so requests that do get admitted have less room, and the ones that do not simply queue.显而易见的动作是把上限提高， 搜索试了四次。 它不管用， 而失败的方式很有教育意义：--max-running-requests 和 max_total_num_tokens 是互相换的， 因为预留更多请求槽位就意味着要从同一个固定池里为每个槽位预留更多状态。 于是提高上限反而缩小了 token 预算， 被放进来的请求空间更少， 没被放进来的就只能排队。

Lowering the cap raises the token budget and the accept length, and destroys latency: at 16 slots the median first token arrives after 96 seconds because 34 requests are waiting. The default cap with a tuned verify window beats all four.降低上限会抬高 token 预算和接受长度， 同时毁掉延迟： 16 个槽位时首 token 中位要等 96 秒， 因为有 34 个请求在排队。 默认上限配上一个调好的验证窗口， 胜过全部四者。

Which knobs moved anything哪些旋钮真的动了指标

Most of them did not, and the null results are worth more than they look — each one retires a hypothesis and narrows what is left to explain.大多数没有， 而这些空结果比看上去更有价值——每一个都退役掉一个假设， 缩小了还需要解释的范围。

Chunked prefill at 32768 and 65536 produced byte-identical capacity and throughput within noise of the default 16384. Setting it to 8192 was actively worse: 815.79 out tok/s against the default's 875.08, a 6.8% loss. The asymmetry is the tell. Above the default, the knob does nothing because prefill is already compute-saturated at 16K tokens per chunk — more tokens per chunk buys no extra parallelism, only a larger activation peak. Below it, chunks become small enough that per-chunk fixed overhead starts to matter. The knob has a floor, not a slope.chunked prefill 设成 32768 和 65536 时， 容量与吞吐和默认的 16384 逐字节相同， 差异在噪声内。 设成 8192 反而更差： 输出 815.79 tok/s， 对比默认的 875.08， 损失 6.8%。 这种不对称正是线索所在。 在默认值以上， 这个旋钮什么都不做， 因为 prefill 在每块 16K token 时已经算力饱和——每块更多 token 买不到额外并行度， 只会抬高激活峰值。 在默认值以下， 分块小到每块的固定开销开始占比。 这个旋钮有一个地板， 而不是一条斜坡。

CUDA-graph batch size 384 changed throughput by 0.03% while costing 2.13 GB of headroom; 512 ran out of memory outright. Schedule conservativeness at 0.6 moved nothing, which answered the question it was raised to answer: the scheduler had queued 54 requests without retracting any, so the suspicion was that it was being too cautious about admission. It was not — the queue was the state pool being full, not a conservative admission policy. Right diagnosis to test, wrong hypothesis, and one benchmark settled it.CUDA-graph batch size 384 让吞吐变化了 0.03%， 代价是 2.13 GB 余量；512 直接 OOM。 schedule conservativeness 设 0.6 什么也没动， 而这恰好回答了它被提出来要回答的问题： 调度器排了 54 个请求却没有回撤任何一个， 所以怀疑它在准入上过于保守。 并非如此——排队是状态池满了， 不是准入策略保守。 值得一测的诊断方向， 错误的假设， 一次 benchmark 就结了案。

The radix cache costs more than it saves — unless prefixes actually repeatradix cache 的代价大于收益——除非前缀真的重复

Enabling the prefix cache on random-prefix traffic dropped throughput from 875.08 to 748.61 out tok/s and cut the runnable batch from 570 slots to 114. The mechanism: with the radix cache on, the hybrid model's state allocation switches to a strategy that reserves several state buffers per request instead of one, because a cached prefix must be able to fork. Pay for forking you never use and you get 20% of the batch capacity for nothing.在前缀随机的流量上开启前缀缓存， 吞吐从输出 875.08 tok/s 掉到 748.61， 可运行 batch 从 570 个槽位砍到 114。 机制是： radix cache 打开后， 混合模型的状态分配会切到一种每请求预留数个状态缓冲而非一个的策略， 因为被缓存的前缀必须能够分叉。 为你从不使用的分叉付费， 换来的是 20% 的 batch 容量， 且一无所得。

On a workload where prefixes do repeat, the same flag is the single biggest win in the entire search. A shared-prefix workload — 32 groups of 8 prompts behind a 4K system prompt — hit 68.2% prompt-token cache hit rate and 12,560.81 total tok/s, against 8,275.57 with the cache disabled. That is a 1.52x speedup from the flag that cost 14% on random traffic.而在前缀确实重复的负载上， 同一个 flag 是整场搜索里最大的单项收益。 一个共享前缀负载——32 组、每组 8 条 prompt、共用一个 4K 的系统 prompt——命中率 68.2%， 总吞吐 12,560.81 tok/s， 对比关闭缓存时的 8,275.57。 那个在随机流量上损失 14% 的 flag， 在这里带来 1.52 倍加速。

The one big knob, and why it works backwards最大的那个旋钮， 以及它为何反着起作用

One flag produced a 1.68x throughput jump — more than every other knob in the search combined. It is --speculative-dspark-block-size, the width of the verify window, and the direction of the win is the interesting part: the value that won proposes fewer tokens per step than the default.有一个 flag 带来了 1.68 倍的吞吐跳升——比搜索里其他所有旋钮加起来还多。 它是 --speculative-dspark-block-size， 也就是验证窗口的宽度， 而这次收益的方向才是有意思的地方： 胜出的取值每步提议的 token 数比默认更少。

Concurrency 48, ISL 8192 / OSL 1024. Step latency is accept_len x TPOT — the wall time of one verify step, which the benchmark does not report directly. Wider windows accept more per step and cost more per step; the product is what ships.并发 48， ISL 8192 / OSL 1024。 每 step 耗时是 accept_len x TPOT——即一次验证 step 的墙钟时间， benchmark 不直接报告这个量。 窗口越宽， 每步接受得越多， 每步也越贵；最终交付的是两者的乘积。

Read the table as the fraction from section 01 and it resolves cleanly. Going from the default window to 3 gives up 15% of the numerator — accept length falls from 2.999 to 2.552 — and buys a denominator that is 2.01x cheaper, 512.7 ms per step down to 255.3 ms. Net: 1.68x. Widening to 5 does the opposite trade at a bad price, recovering accept length to 2.893 while pushing the step back to 505.5 ms, which lands it within 1.2% of the untuned default.把这张表当成第 01 节那个分数来读， 一切就清楚了。 从默认窗口调到 3， 放弃了 15% 的分子——接受长度从 2.999 降到 2.552——换来一个便宜 2.01 倍的分母， 每 step 从 512.7 ms 降到 255.3 ms。 净效果 1.68 倍。 加宽到 5 是以糟糕的价格做相反的交换， 把接受长度拉回 2.893， 却把每 step 推回 505.5 ms， 结果落在未调优默认值的 1.2% 以内。

Why is the denominator so sensitive? Because verification is not free parallelism. The target must attend over the speculative suffix, and for a hybrid model it must also advance the recurrent state through every speculated position — work that scales with window width and is discarded for every token the draft got wrong. A wider window multiplies the wasted work by the rejection rate. Accept length grows sub-linearly with the window while cost grows at least linearly, so the optimum is narrow, and it is narrow for reasons that hold for any speculative implementation, not just this one.为什么分母这么敏感？ 因为验证并不是免费的并行。 目标模型必须在投机后缀上做 attention， 而对混合模型来说， 它还必须把循环状态推进过每一个被投机的位置——这份工作量随窗口宽度增长， 并且对草稿猜错的每个 token 都是白做的。 窗口越宽， 白做的工作量就被拒绝率放大得越多。 接受长度随窗口次线性增长， 而成本至少线性增长， 所以最优点偏窄——而且它偏窄的原因适用于任何投机实现， 不只是这一个。

The accuracy gate, and the two knobs it exists for精度闸门， 以及它为之存在的两个旋钮

Six of the knobs in this search cannot change what the model outputs. Memory fraction, chunked prefill size, CUDA-graph batch size, max running requests, schedule conservativeness, radix strategy — every one of them only decides where tensors live and in what order requests run. The token distribution is untouched by construction, so measuring accuracy after changing them is a formality.这次搜索里有六个旋钮不可能改变模型的输出。 显存占比、chunked prefill 大小、CUDA-graph batch size、最大运行请求数、schedule conservativeness、radix 策略——它们每一个都只决定张量放在哪里、请求以什么顺序运行。 token 分布在构造上就不受影响， 所以改完它们再测精度只是形式。

Two knobs are different. --mamba-ssm-dtype bfloat16 lowers the precision of the recurrent state, which is arithmetic that feeds every subsequent token. --speculative-dspark-block-size changes the verify window, and while the theory says speculation is lossless — the target verifies every token, so accepted tokens are exactly the tokens the target would have produced — a theory is not a measurement. The gate exists for these two.有两个旋钮不同。 --mamba-ssm-dtype bfloat16 降低了循环状态的精度， 而那是要喂给后续每一个 token 的算术。 --speculative-dspark-block-size 改变验证窗口， 虽然理论上投机是无损的——目标模型验证每一个 token， 所以被接受的 token 恰好就是目标模型本来会产出的 token——但理论不是测量。 闸门就是为这两个而存在的。

Both re-run through exactly the protocol that produced the published day-0 numbers: GSM8K greedy at temperature 0, AIME25 with 8 repeats at temperature 1.0 / top-p 0.95 with thinking enabled.两者都按产出已发布 day-0 数字的同一套协议重跑： GSM8K 贪心解码、温度 0；AIME25 重复 8 次、温度 1.0 / top-p 0.95、开启思维链。

The speculative window came out clean, and pleasingly so — AIME25 landed 0.84 points above the day-0 speculative baseline with a perfect stop rate. That is what lossless is supposed to look like, and it is the evidence that let the knob ship.投机窗口的结果是干净的， 而且干净得令人满意——AIME25 落在 day-0 投机基线之上 0.84 个百分点， 停止率完美。 这就是「无损」该有的样子， 也正是让这个旋钮得以发布的证据。

The state-precision knob passed too, and was dropped anyway. It is the only configuration in the gate that lost samples to truncation — 0.83% truncated, 0.83% with no answer, stop rate 99.17% against the baseline's 100% — and it carries a 1.66-point AIME25 deficit. Pooled, that deficit is 0.88 sigma, so it is not a detectable regression; it is also not nothing, and it bought only 1.24% more throughput. A knob that changes numerics, cannot be shown to be safe, and is worth barely more than measurement noise is not a knob worth shipping. It is documented as opt-in for anyone willing to re-run their own gate.状态精度那个旋钮也过了， 但还是被剔除了。 它是闸门里唯一有样本因截断而丢失的配置——0.83% 被截断、0.83% 没有答案、停止率 99.17%， 对比基线的 100%——而且它背着 1.66 个百分点的 AIME25 缺口。 合并标准误后这个缺口是 0.88 sigma， 所以它不是可检出的性能退化；但它也不是没有， 而它换来的吞吐只多 1.24%。 一个改变数值、无法被证明安全、收益又勉强高于测量噪声的旋钮， 不值得发布。 它被记录为可选项， 留给愿意自己重跑一遍闸门的人。

Speculation degrades with context, not with batch投机随上下文衰减， 不随批量衰减

With a recipe chosen and gated, the second campaign asks a different question: not which setting is best, but how the chosen setting behaves as the workload moves. Two sweeps at concurrency 1 — input length, then output length — and one up the concurrency curve. The most useful result comes from putting two of them side by side.配方选定并过闸之后， 第二轮问的是另一个问题： 不是哪个设置最好， 而是选定的设置在负载变化时表现如何。 两条并发 1 的扫描——先输入长度， 再输出长度——加一条并发曲线。 最有用的结果来自把其中两条并排放。

Across concurrency 1 to 48 accept length only jitters between 2.473 and 2.732 — a spread of 10.5% with no monotone trend, which is the size of the run-to-run noise established in section 12. Across input length 128 to 64K it falls monotonically, a spread of 74.6%. One of these is noise and the other is a mechanism.在并发 1 到 48 之间， 接受长度只在 2.473 到 2.732 之间抖动——幅度 10.5%， 且无单调趋势， 这正是第 12 节确定的 run-to-run 噪声量级。 而在输入长度 128 到 64K 之间它单调下降， 幅度 74.6%。 一个是噪声， 另一个是机制。

Why should batch size leave acceptance alone? Because acceptance is decided per sequence. The draft proposes for one request, the target verifies that request's tokens, and the accept/reject decision reads only that sequence's own logits. Batching other requests alongside changes how much silicon the step uses and therefore how long it takes, but it does not change what the draft would have guessed. Batch affects the denominator only.为什么 batch 大小不影响接受率？ 因为接受是按序列决定的。 草稿为某一个请求提议， 目标模型验证该请求的 token， 接受/拒绝的判定只读这条序列自己的 logits。 把其他请求一起打包会改变这一步用掉多少硅、因而改变它耗时多久， 但不会改变草稿本来会猜什么。 batch 只影响分母。

Context is different: it is an input to the draft model's own prediction. A longer prefix is a harder prediction problem, and a small draft model degrades on it faster than the target does. Context affects the numerator and, through attention cost, the denominator. That is why one sweep is flat and the other collapses.上下文不同： 它是草稿模型自身预测的输入。 更长的前缀是更难的预测问题， 而小草稿模型在这上面的退化比目标模型更快。 上下文既影响分子， 又通过 attention 成本影响分母。 这就是为什么一条扫描是平的， 另一条却在崩塌。

Total tok/s counts input tokens, so it rises with ISL and saturates near 1,040 — the prefill ceiling at concurrency 1. Prefill rate itself peaks at 12,874 tok/s around ISL 8K–16K and decays to 7,953 at 64K, which is the quadratic term in attention becoming visible.总 tok/s 计入输入 token， 所以它随 ISL 上升并在 1,040 附近饱和——那是并发 1 时 prefill 的天花板。 prefill 速率自身在 ISL 8K–16K 附近达到峰值 12,874 tok/s， 到 64K 退化到 7,953， 这是 attention 的二次项开始显现。

Splitting the 6.35x loss拆开这 6.35 倍损失

Output throughput at concurrency 1 falls 6.35x between input length 1K and 64K. A single ratio like that is not actionable — it does not tell you whether to work on the draft model or the attention kernel. Splitting it does.并发 1 时， 输出吞吐在输入长度 1K 到 64K 之间下降 6.35x。 单看这样一个比值没法指导行动——它不告诉你该去改草稿模型还是 attention kernel。 拆开它就可以。

Because throughput is accepted-tokens-per-step over step-latency, and both factors are measurable, the loss factorises exactly:因为吞吐等于「每步接受 token 数」除以「每步耗时」， 而两个因子都可测， 所以这个损失可以精确因式分解：

step latency   22.8 ms → 79.6 ms      = 3.50x   (attention cost)
accept length  2.514  → 1.440        = 1.75x   (draft degradation)
                                       ─────────
predicted                              6.11x
measured                               6.35x

The 4% residual is the model's error bar, and it is small enough that these two terms account for essentially the whole collapse. Neither dominates: roughly 57% of the loss is the step getting more expensive, 43% is the draft getting worse. Median TPOT over the same range goes from 9.05 ms to 55.30 ms.4% 的残差就是这个模型的误差棒， 小到足以说明这两项解释了几乎全部崩塌。 两者都不占绝对主导： 约 57% 的损失来自每步变贵， 43% 来自草稿变差。 同一区间内 TPOT 中位从 9.05 ms 变到 55.30 ms。

That split is the reason to bother computing it. Halving the attention cost at 64K would recover about 1.9x, not 6x — a better kernel alone cannot fix long-context speculation. Getting the draft model to hold its accept length would recover about 1.75x. Neither intervention on its own is close to sufficient, which is worth knowing before spending a quarter on either.这个分解正是值得费力算它的理由。 把 64K 处的 attention 成本减半只能挽回约 1.9 倍， 而不是 6 倍——单靠更好的 kernel 修不好长上下文投机。 让草稿模型守住它的接受长度能挽回约 1.75 倍。 任何单一介入都远不够， 而这件事值得在为其中任何一个投入一个季度之前先知道。

Output length barely matters输出长度几乎不重要

A null result, and a useful one: it isolates the previous section's cause. If decode degraded with generation length on its own, the output sweep would fall as steeply as the input sweep. It does not.一个空结果， 而且是有用的： 它把上一节的原因隔离了出来。 如果 decode 本身会随生成长度退化， 那输出扫描应该和输入扫描一样陡地下坠。 它没有。

The y-axis is zoomed deliberately — on a zero baseline this plate would be a flat line and the reader would learn nothing about the one real feature, the fall at 4096. That fall is not a new effect. Generating 4096 tokens from a 1024-token prompt means the effective context reaches roughly 5,120 tokens by the end, which is the input sweep's mechanism arriving by a different road. Median TTFT stays pinned between 176 and 180 ms across the whole sweep, exactly as expected: output length cannot affect the time to produce the first token.纵轴是故意放大的——若以零为基线， 这张图版会是一条平线， 读者就学不到那个唯一的真实特征， 即 4096 处的下降。 那个下降不是新效应。 从 1024 token 的 prompt 生成 4096 个 token， 意味着到末尾时有效上下文已达约 5,120 token， 这是输入扫描的机制换了条路到达。 TTFT 中位在整条扫描上都锁在 176 到 180 ms 之间， 完全符合预期： 输出长度不可能影响产出首个 token 的时间。

Scaling, and where the knee is扩展， 以及拐点在哪

Concurrency 1 to 48 buys 9.66x aggregate throughput, peaking at 1,061 output tok/s. Forty-eight times the concurrency realising under ten times the throughput is 20% scaling efficiency — which sounds like a failure and is actually the expected shape, because the thing being shared is fixed silicon. What each request loses is direct: per-request throughput falls from 109.8 to 24.9 tok/s.并发 1 到 48 换来 9.66x 聚合吞吐， 峰值 1,061 输出 tok/s。 四十八倍并发只兑现不到十倍吞吐， 即 20% 的线性度——听起来像失败， 其实是应有的形状， 因为被共享的是固定的硅。 每个请求付出的代价是直接的： 单请求吞吐从 109.8 掉到 24.9 tok/s。

The knee is at 32. Sixteen to 32 is the best-value segment on the curve: 51% more throughput for a median TTFT that moves from 849 ms to 888 ms. Going on to 48 adds 11.7% throughput while TTFT jumps to 2,271 ms and TPOT rises from 29.67 ms to 38.06 ms. Stop at 32 if latency is a product requirement; go to 48 only when aggregate throughput is the only thing being sold.拐点在 32。 16 到 32 是曲线上性价比最好的一段： 吞吐多 51%， 而 TTFT 中位只从 849 ms 移到 888 ms。 继续到 48， 吞吐只多 11.7%， TTFT 却跳到 2,271 ms， TPOT 也从 29.67 ms 升到 38.06 ms。 如果延迟是产品要求就停在 32；只有当卖的只是聚合吞吐时才上 48。

Note also what does not break down: the per-step arithmetic holds across the whole range. At concurrency 48 the step costs 101.9 ms and delivers 114.2 tokens, which predicts 1,121 tok/s against 1,061 measured — 5% high, the gap being ramp-up and drain at the edges of the run. The batching win is entirely in the numerator: 42x more tokens per step for 4.25x the step cost.同时注意什么没有崩： 每步的算术在整个区间都成立。 并发 48 时每步耗时 101.9 ms、产出 114.2 个 token， 预测 1,121 tok/s， 实测 1,061——高了 5%， 差距来自运行两端的爬坡与收尾。 批处理的收益全部落在分子上： 每步多 42 倍的 token， 只付 4.25 倍的每步成本。

How DSpark works, and why it runs hereDSpark 如何工作， 以及它为何能在这里运行

Section 06 treated the verify window as a knob and found the narrow setting won. This section is the mechanism behind that, because on this particular architecture speculation is more expensive than the usual account of it suggests — and the extra cost is exactly what caps concurrency at 48.第 06 节把验证窗口当成一个旋钮， 测出窄的那一档胜出。 这一节讲它背后的机制——因为在这个架构上， 投机解码比通常的说法要贵， 而多出来的这部分代价， 恰好就是并发被锁在 48 的原因。

Why a guess is worth anything为什么猜测能值钱

Start with the cost of one ordinary token, because that number is what makes everything after it possible. To produce a single token, the machine has to move every weight the model needs from memory into the compute units. On a 1.5 TB model that transfer is the whole cost of the step; the arithmetic units spend most of it waiting for data to arrive.先从产出一个普通 token 的成本说起， 因为正是这个数字让后面的一切成为可能。 要产出一个 token， 机器必须把模型需要的每一个权重从显存搬到计算单元里。 在一个 1.5 TB 的模型上， 这次搬运就是这一步的全部成本；计算单元的大部分时间都在等数据到达。

That leads somewhere counterintuitive. If the weights get streamed once either way, then a forward pass that evaluates four positions costs almost the same as one that evaluates a single position. The expensive resource is the memory traffic, and the memory traffic does not care how many positions you asked about.这引出一个反直觉的结论。 既然无论如何权重都只被流过一次， 那么一次评估四个位置的前向， 和一次只评估一个位置的前向， 成本几乎相同。 昂贵的那样资源是显存流量， 而显存流量并不在乎你问了多少个位置。

So checking is cheap in a way that generating is not, and that gap is the entire opportunity. Suppose you had four candidate tokens from somewhere, and one target forward confirmed the first three. You paid for one step and collected three tokens. Nothing about the model changed — you simply used capacity that was already sitting idle. Without this, the three tokens the plate below commits would have cost three full target forwards; with it, they cost one.所以检查的便宜是生成所不具备的， 而这个落差就是全部的机会所在。 假设你从某处拿到了四个候选 token， 一次目标前向确认了前三个。 你付了一步的钱， 收到了三个 token。 模型没有任何改变——你只是用掉了本来就闲着的容量。 没有这一点， 下面图版里提交的那三个 token 需要三次完整的目标前向；有了它， 只需要一次。

Which moves the problem: the candidates have to come from somewhere cheap enough that they do not eat the saving. A small model could write them, but writing four tokens the ordinary way means four of its forward passes, and that just relocates the serial cost instead of removing it. DSpark takes a different route — it runs its draft once, over four blank positions at the same time.于是问题被搬走了： 候选必须来自某个足够便宜的地方， 便宜到不会把省下来的吃掉。 一个小模型可以写出它们， 但用常规方式写四个 token 意味着它要做四次前向， 那只是把串行成本搬了个地方而不是消除它。 DSpark 走了另一条路——它让草稿模型只跑一次， 同时覆盖四个空白位置。

That buys the speed and creates a problem worth stating plainly: each blank position is being predicted without knowing what the position before it turned out to be. Four predictions made in mutual ignorance rarely form a sentence. It is like asking four people to each write the next word without letting any of them see what the previous one wrote.这买到了速度， 也造出一个值得明说的问题： 每个空白位置在被预测时， 并不知道它前面那个位置最终是什么。 四个在彼此无知的情况下作出的预测， 很少能组成一句话。 这就像让四个人各写下一个词， 却不让任何人看见前一个人写了什么。

The repair is deliberately the smallest thing that could work, and it is the piece worth understanding. After the single draft forward, a tiny head walks the four positions in order, and at each one adds a correction derived from the token that was just chosen: an embedding into a small rank-r space and a projection back out to the vocabulary. No attention. No second transformer pass. Just a nudge that carries "here is what the previous position settled on." That is why, in the plate below, the draft lane fills one cell at a time even though only one draft forward ever happened — and why the cells can be filled serially without the step becoming slow.这个修补被刻意做成了「能起作用的最小的东西」， 而它正是值得弄懂的那一块。 在那一次草稿前向之后， 一个极小的 head 按顺序走过这四个位置， 并在每个位置上加一个由刚刚选定的那个 token 推导出的修正： 一次到低维 rank-r 空间的 embedding， 再投影回词表。 没有 attention。 没有第二次 transformer 前向。 只是一个携带着「前一个位置定下来的是什么」的轻推。 这就是为什么在下面的图版里， draft 行是一格一格填的， 尽管全程只发生过一次草稿前向——也是为什么这些格子可以串行地填， 却不会让这一步变慢。

One cycle, watched看一轮完整的循环

Now the three facts have a picture. Watch the two lanes at different speeds: the draft lane fills a cell at a time because of the nudge, and the verify lane flashes all at once because it is one forward. Where those two rates meet is what the block-size knob moves.现在那三个事实有了一张图。 注意两行的节奏不同： draft 行因为那个轻推而一格一格地填， verify 行因为只是一次前向而整体闪一下。 这两种节奏交汇的位置， 正是 block-size 这个旋钮在移动的东西。

Read the lanes against each other and the trade is visible without any numbers. The draft lane is serial — the Markov head cannot compute position i before it has drafted position i−1, because the correction it applies is a function of the token it just produced. But each of those steps is an embedding lookup and a rank-r projection, so serial does not mean slow. The verify lane is the opposite: every position is computed in the same forward pass, so it is not serial at all, but that one pass is a full target-model forward and it is where the step's cost lives.把两行对着读， 权衡不需要任何数字就能看见。 draft 行是串行的——Markov head 在草拟出位置 i−1 之前算不了位置 i， 因为它施加的修正是刚产出的那个 token 的函数。 但这些步骤每一步只是一次 embedding 查表加一个 rank-r 投影， 所以串行不等于慢。 verify 行恰好相反： 每个位置都在同一次前向里算完， 所以它完全不串行， 但那一次前向是一整个目标模型的前向， 这一步的成本就住在那里。

That asymmetry is the whole reason a speculative step can be worth taking, and also the reason the block cannot be made arbitrarily wide. Widening γ adds positions to the expensive parallel pass at close to linear cost, while the marginal draft token it buys is the least likely one to survive — the Markov correction at position i is conditioned on a token that was itself drafted, so an error at any position poisons every position after it. Cost grows about linearly in γ; expected accepted length grows sublinearly and eventually flattens. An optimum in the middle is not a tuning accident, it is the shape of the two curves.这种不对称正是一次投机步骤值得做的全部理由， 同时也是块不能被任意加宽的理由。 加宽 γ 会以接近线性的成本给那次昂贵的并行前向增加位置， 而它换来的边际草稿 token 恰好是最不可能活下来的那一个——位置 i 的 Markov 修正所依赖的那个 token 本身也是草拟出来的， 所以任何一个位置出错都会污染它之后的每个位置。 成本大致随 γ 线性增长， 期望接受长度则次线性增长并最终走平。 中间存在一个最优点不是调参的偶然， 它就是这两条曲线的形状。

What SGLang actually runsSGLang 实际执行的东西

Three details from the source, for anyone who wants to find this in the tree. The "blank" positions are a mask token, and column 0 of the block is not blank — it holds last step's bonus token, so the draft has an anchor to condition on. The base logits come from the target's own lm_head rather than an output layer of the draft's, which is why the draft needs none. And the nudge has a name — a Markov head — with three available shapes: a plain rank-r embedding correction, a gated variant, and a small GRU-like recurrence.三个来自源码的细节， 供想在代码树里找到它们的人。 那些「空白」位置是 mask token， 而块的第 0 列并不空白——它放着上一步的 bonus token， 好让草稿有个锚可以条件化。 基础 logits 来自目标模型自己的 lm_head 而不是草稿自带的输出层， 这也是草稿为什么不需要自己的输出层。 那个轻推有名字——Markov head——有三种可选形状： 朴素的 rank-r embedding 修正、一个带门控的变体， 以及一个小的类 GRU 递推。

Acceptance has two forms and both are worth naming, because the sampling one is what made this port possible. For an all-greedy batch it is the longest matching prefix: compare each draft against the target's own argmax at that position, take a cumulative product of the matches, and the sum of that product is the accepted length. For any batch containing a sampled request it is chain rejection sampling — the textbook rule, accept when a uniform draw times the draft probability falls under the target probability, and resample from the positive residual otherwise. Either way the step commits accepted drafts plus one bonus token, which is the target's own prediction at the position where the chain broke. That bonus is why even a fully rejected block still makes progress.接受判定有两种形式， 两个都值得点名， 因为其中采样那一种正是让这次移植成为可能的东西。 全贪心的批次走最长匹配前缀： 把每个草稿与目标模型在该位置自己的 argmax 比较， 对匹配取累积乘积， 该乘积之和就是接受长度。 只要批次里有一个采样请求， 就走链式拒绝采样——教科书规则， 均匀采样乘草稿概率落在目标概率之下就接受， 否则从正残差重新采样。 两种方式下这一步都提交「接受的草稿 + 一个 bonus token」， 后者是目标模型在链断裂那个位置上自己的预测。 这个 bonus 正是为什么即使一整块全被拒绝， 这一步依然有进展。

Why it retargets to ROCm without being ported为什么它不用移植就能重定向到 ROCm

This is the part worth understanding, because the answer is not "AMD did the work." Nobody ported DSPARK. It runs because of where its verify math is written down and what its dispatch keys on.这部分值得弄懂， 因为答案不是「AMD 做了适配」。 没有人移植过 DSPARK。 它能跑， 靠的是它的验证数学写在什么地方， 以及它的分派看的是什么。

Every operator on the DSPARK hot path — greedy accept, chain rejection sampling, the draft's per-step sampler, the verify-window scatter, the length scheduler, the KDA verify recurrence, the causal conv, the post-verify state scatter — is written in Triton. Triton is a compiler input, not a binary: the same kernel text retargets to AMDGCN. And the dispatch that decides whether to take those kernels asks one question about the data, tensor.is_cuda, not one question about the toolchain.DSPARK 热路径上的每一个算子——贪心接受、链式拒绝采样、draft 的逐步采样器、verify 窗口的 scatter、长度调度器、KDA 的验证递推、因果卷积、验证后的状态 scatter——都是用 Triton 写的。 Triton 是编译器的输入而不是二进制： 同一份 kernel 文本可以重定向到 AMDGCN。 而决定是否走这些 kernel 的分派， 问的是关于数据的一个问题——tensor.is_cuda——而不是关于工具链的问题。

Those two facts land differently on ROCm than one might expect, because PyTorch-HIP deliberately keeps the CUDA device namespace. Measured on this node: torch.version.cuda is None and torch.version.hip is 7.2.26015, yet torch.cuda.is_available() is True, a device tensor reports its device as cuda:0, and tensor.is_cuda is True. So the codebase carries two different notions of "is this CUDA", and they disagree here — which turns out to be exactly what is wanted:这两个事实在 ROCm 上的落点和人们的直觉不同， 因为 PyTorch-HIP 刻意保留了 CUDA 的设备命名空间。 在这台机器上实测：torch.version.cuda 是 None、torch.version.hip 是 7.2.26015， 但 torch.cuda.is_available() 是 True， 设备张量报告自己的 device 是 cuda:0，tensor.is_cuda 是 True。 所以代码库里存在两个不同的「是否 CUDA」概念， 而它们在这里给出不同答案——结果这恰好就是我们想要的：

is_cuda() additionally requires torch.version.cuda is not None, which is what makes it the strict test — the one that correctly excludes the single NVIDIA-only kernel on this path.is_cuda() 额外要求 torch.version.cuda is not None， 这正是它成为严格判据的原因——它正确地把这条路径上唯一一个 NVIDIA 专属 kernel 排除掉了。

The one NVIDIA-only kernel is a fused KDA verify written in CuTeDSL, and it is reachable only when the strict test passes, CUTLASS imports, and the device is SM100. On ROCm the strict test fails first, so the KDA verify falls back to the Triton recurrence — a slower path, but a present one. Contrast the sibling algorithm: DFLASH's non-greedy verify calls tree_speculative_sampling_target_only, a compiled C++ symbol from sgl_kernel that is only bound when the strict test passes and is None otherwise. There is no fallback to fall back to. DSPARK proposes a strict chain rather than a tree, so it never needed the general tree kernel, and it ships its own Triton chain sampler instead.那个唯一的 NVIDIA 专属 kernel 是一个用 CuTeDSL 写的融合 KDA 验证， 只有在严格判据通过、CUTLASS 可导入、且设备是 SM100 时才可达。 在 ROCm 上严格判据先失败， 于是 KDA 验证回退到 Triton 递推——一条更慢但存在的路。 对比它的兄弟算法： DFLASH 的非贪心 verify 会调用 tree_speculative_sampling_target_only， 那是 sgl_kernel 里一个编译好的 C++ 符号， 只在严格判据通过时才绑定， 否则是 None。 没有回退可以回退。 DSPARK 提议的是严格的链而不是树， 所以它从来不需要那个通用的树 kernel， 而是自带了一个 Triton 的链式采样器。

Verification on a recurrent layer is not a cache write在循环层上做验证不是一次缓存写入

On a pure-attention model, verifying k drafted tokens is one forward pass over k positions. The KV entries for those positions get written, the target checks them, and the rejected tail is simply abandoned — the cache is indexed by position, so abandoning it costs nothing but the bandwidth already spent.在纯 attention 模型上， 验证 k 个草稿 token 就是在 k 个位置上做一次前向。 这些位置的 KV 条目被写入， 目标模型检查它们， 被拒绝的尾部直接丢弃——缓存是按位置索引的， 丢弃它除了已经花掉的带宽之外没有额外代价。

Most of K3's layers do not have a cache. They carry a recurrent state, and a recurrent state is a value, not a table: advancing it through k speculated positions is a serial chain, and there is no index to forget: wherever the chain breaks, the state that survives has to be chosen, which means every candidate has to still exist by the time acceptance runs. So a speculative request keeps γ+1 recurrent states alive where a non-speculative request keeps one.K3 的大多数层没有缓存。 它们携带一个循环状态， 而循环状态是一个值而不是一张表： 把它推进过 k 个投机位置是一条串行链， 而且没有索引可以遗忘： 无论链在哪里断裂， 存活下来的那个状态必须被选出， 这意味着接受判定运行时每个候选都还得存在。 所以一个投机请求要保有 γ+1 份循环状态， 而非投机请求只保有一份。

That is the mechanism behind the wall in section 04. At 48 running requests the state pool sits at 98% while the KV cache is at 29%: the speculative lane is not short of cache, it is short of state slots. And it is why the cap is not a policy the operator can raise — the server says so at boot, Max running requests is reset to 48 for speculative decoding, and section 04's four attempts to raise it all traded token budget for slots and lost.这就是第 04 节那道墙的机制。 48 个运行请求时状态池占到 98%， 而 KV cache 只有 29%： 投机通道缺的不是缓存， 缺的是状态槽位。 这也解释了为什么这个上限不是运维方能调高的策略——服务器在启动时就说了，Max running requests is reset to 48 for speculative decoding， 而第 04 节里四次抬高它的尝试， 全都是拿 token 预算换槽位， 全都输了。

It also sharpens why the window optimum is narrow. The work discarded on a rejection is not just attention over a few extra positions; it is state advancement, which is serial in the sequence dimension and therefore cannot be recovered by throwing more parallelism at it. A wider window multiplies serial work by the rejection rate, which is why block-size 5 landed within 1.2% of the untuned default while block-size 3 won by 1.68x.它也让「为什么最优窗口偏窄」这件事更清楚。 拒绝时丢掉的工作不只是多几个位置的 attention， 还有状态推进——那是在序列维度上串行的， 因此没法靠加并行度捞回来。 窗口越宽， 串行的白做工作就被拒绝率放大得越多， 这就是为什么 block-size 5 落在未调优默认值的 1.2% 以内， 而 block-size 3 赢下 1.68 倍。

The context finding, seen from somewhere else另一个来源看到的同一件事

Section 08's claim — that accept length decays with context and not with batch — was measured here on one node, which is exactly the kind of result that deserves a second source. There is one, arrived at independently: an upstream issue filed against 8x MI350X reports DSPARK getting slower at roughly 26k of context, with accept length degrading to about 1.6–2.0. Our sweep put accept length at 1.892 at input length 16K and 1.741 at 32K, so 26k lands between those two and the reported range brackets ours.第 08 节那个结论——接受长度随上下文衰减、而不随批量衰减——是在单个节点上测出来的， 这正是需要第二个来源印证的那类结果。 有一个， 而且是独立得到的： 上游一个针对 8x MI350X 的 issue 报告 DSPARK 在约 26k 上下文时反而更慢， 接受长度退化到约 1.6–2.0。 我们的扫描在输入长度 16K 处测到 1.892、32K 处 1.741， 26k 落在两者之间， 而报告的区间正好把我们的数值夹在中间。

Two caveats on that comparison, since it is someone else's number: the hardware is MI350X rather than MI355X, and the figure is a reporter's observation rather than a protocol like the one behind this page's tables. What it supports is the direction and the rough magnitude, not the third decimal place. It is worth recording anyway, because the decay is not tracked anywhere upstream — the K3 roadmap issue lists no AMD items at all.对这个对比要加两条限定， 毕竟那是别人的数字： 硬件是 MI350X 而不是 MI355X， 而且那个数值是报告者的观察， 不是本页表格背后那种协议化测量。 它支持的是方向和量级， 不是小数点后第三位。 但仍然值得记下来， 因为这个衰减在上游没有任何地方被跟踪——K3 的 roadmap issue 里一条 AMD 相关的条目都没有。

Reefs暗礁

Five hazards this campaign ran into: one that took the server down, and four that nearly reported something false. All five were caught; the reason to write them down is that none of them announce themselves.这轮测量差点报出错误结论的四种方式。 四个都抓住了；把它们写下来的理由是， 它们没有一个会自我声明。

A verify path that only dies on half your test suite只在半套测试上暴死的验证路径

DSPARK reaches its verify-time probability renormalisation from its own triton accept kernel on every device. But the two kernels that renormalisation calls, top_k_renorm_prob and top_p_renorm_prob, are imported from sgl_kernel only under is_cuda() or is_musa(), and are left as None everywhere else. On ROCm, therefore, any request with top_p < 1 or top_k > 1 killed the scheduler outright with TypeError: 'NoneType' object is not callable.DSPARK 在所有设备上都会从它自己的 triton accept kernel 走到验证期的概率重归一化。 但重归一化调用的那两个 kernel——top_k_renorm_prob 和 top_p_renorm_prob——只在 is_cuda() 或 is_musa() 下才从 sgl_kernel 导入， 其他情况一律是 None。 所以在 ROCm 上， 任何带 top_p < 1 或 top_k > 1 的请求都会直接以 TypeError: 'NoneType' object is not callable 打死 scheduler。

What makes this a reef rather than an ordinary bug is which traffic finds it. Greedy decoding never touches the renormalisation, so the GSM8K half of the accuracy gate — temperature 0, 1319 questions — passes cleanly on a configuration that cannot serve a single sampled request. The AIME half, at temperature 1.0 and top_p 0.95, takes the server down on its first decode batch. A test suite can be half green and wholly misleading, and the half that stays green is the half most people run first.让它成为暗礁而不是普通 bug 的， 是哪种流量会撞上它。 贪心解码永远碰不到重归一化， 所以精度闸门里 GSM8K 那一半——温度 0、1319 道题——在一个连一个采样请求都服务不了的配置上干净通过。 而 AIME 那一半， 温度 1.0、top_p 0.95， 会在第一个 decode batch 就把服务打死。 一套测试可以一半是绿的却整体误导， 而保持绿色的那一半恰好是大多数人先跑的那一半。

The neighbouring speculative implementation, DFLASH, survives the same code path by gating its entire non-greedy verify on an availability check and silently degrading to greedy argmax verification. That is the wrong repair here, and worth saying why: verifying greedily against a target that was sampled changes the output distribution. It converts a crash into a silent accuracy change — the exact failure the gate in section 07 exists to catch, now moved upstream of the gate where the gate cannot see it. Routing the renormalisation to a real implementation keeps the sampling semantics intact; degrading does not.相邻的那个投机实现 DFLASH 在同一条代码路径上活了下来， 办法是把整个非贪心 verify 挂在一个可用性检查后面， 静默降级成贪心 argmax 验证。 这在这里是错的修法， 而且值得说清为什么： 拿贪心去验证一个被采样出来的目标， 会改变输出分布。 它把一次崩溃换成了一次静默的精度变化——正是第 07 节那道闸门要抓的失败， 却被搬到了闸门上游、搬到了闸门看不见的地方。 把重归一化接到一个真实实现上能保住采样语义；降级不能。

A regex that read the command line as a result一个把命令行当成结果读的正则

The gate's first run reported FAIL on five of six checks. Nothing was wrong with the model. The GSM8K scraper scanned forward from the word Score and picked up the 64 in --max-tokens 64000; the AIME25 scraper read the 8 out of the label pass@1[avg-of-8]. Both now anchor on the = sign or on the exact metric key. A parser that searches forward for "the next number" will eventually find a number from the invocation rather than the output, and it will do so silently.闸门第一次运行报告六项检查中五项 FAIL。 模型没有任何问题。 GSM8K 的抓取器从 Score 这个词向前扫描， 抓到了 --max-tokens 64000 里的 64；AIME25 的抓取器从标签 pass@1[avg-of-8] 里读出了那个 8。 现在两者都锚定在 = 号或精确的指标键上。 一个「向前找下一个数字」的解析器， 终究会找到来自调用命令而非输出的数字， 而且它会静默地这么做。

A degeneration probe that flagged good prose一个把好文章标为退化的探测器

The same run failed one sample at 29.5% 8-gram repetition. Inspecting it showed fluent, correct English — a reasoning trace summarising repetitive source text — with a longest consecutive token run of 1. Repetition alone does not distinguish degenerate looping from legitimately repetitive content. The threshold now requires repetition and consecutive runs together, or one signal far out. No eval was re-run; only the scoring changed, which is the honest way to fix a scorer.同一次运行里有一个样本因 8-gram 重复率 29.5% 而未通过。 检查后发现那是流畅正确的英文——一段在总结重复性原文的推理轨迹——最长连续 token 串长度为 1。 单看重复率无法区分退化式循环与本就重复的内容。 现在阈值要求重复率与连续串长度同时超标， 或某一个信号远超。 没有重跑任何评测， 只改了打分方式——这才是修一个打分器的诚实做法。

Random tokens make accept length noisy随机 token 让接受长度变噪

The same ISL/OSL 1024 point measures 101.92 tok/s with four requests and 109.84 with eight — 7.8% apart, and the throughput ratio is almost exactly the accept-length ratio. --dataset-name random emits random token IDs, and the draft model's acceptance is content-sensitive, so low sample counts give a noisy numerator. Every bsz=1 figure here should be read as carrying 8–10% run-to-run variance, and as a conservative lower bound relative to real text. For reference, the search's own p3-lat-win row on this recipe recorded 111.58 tok/s at accept length 2.757, against sweep C's 109.84 and 2.693 — 1.6% and 2.3% apart.同一个 ISL/OSL 1024 点， 四条请求时测出 101.92 tok/s， 八条时测出 109.84——相差 7.8%， 而吞吐比几乎正好等于接受长度比。 --dataset-name random 产出的是随机 token ID， 草稿模型的接受率对内容敏感， 所以低样本数会给出一个噪声很大的分子。 这里每个 bsz=1 数字都该被理解为带有 8–10% 的 run-to-run 波动， 并且相对真实文本是保守下界。 作为参照， 搜索自己在该配方上的 p3-lat-win 行记录为 111.58 tok/s、接受长度 2.757， 对比扫描 C 的 109.84 与 2.693——分别差 1.6% 与 2.3%。

A scaling segment that looked too good一段好得反常的扩展区间

The 4 → 8 segment shows 90.3% scaling efficiency, higher than its neighbours on both sides. That is an artifact, not a sweet spot: at concurrency 4 with 8 requests there are only two waves, so ramp-up and drain dominate and achieved concurrency reaches just 3.39 — 85% of what was requested. Understating the denominator of the previous point inflates the ratio into the next one. From concurrency 8 up, achieved concurrency exceeds 92% and the segments are comparable. A curve measured with a fixed small request count will always bend upward at its low end.4 → 8 那一段显示 90.3% 的线性度， 比它两侧的邻居都高。 那是假象， 不是甜点： 并发 4 配 8 条请求时只有两个波次， 爬坡与收尾占了主导， 实测并发只到 3.39——是请求量的 85%。 低估了前一个点的分母， 就会把比值虚增到下一个点上。 从并发 8 往上， 实测并发超过 92%， 各段之间才可比。 一条用固定的小请求数测出来的曲线， 在低端永远会向上弯。

Epilogue结语

The shipped recipe is four lines long, and none of them is surprising on its own:最终发布的配方只有四行， 单看每一行都不令人惊讶：

MEM_FRAC=0.92
RADIX=0
CUDA_GRAPH_MAX_BS=256
EXTRA_ARGS=--speculative-dspark-block-size 3

What the 450 minutes bought was not those four lines. It was knowing that the memory fraction cannot go higher because activation peaks are invisible at boot, that the request cap cannot go higher because it trades against the token budget, that the prefix cache is off for a reason that a real deployment should reverse, and that the one flag doing all the work does it by making the verify step cheaper rather than more productive.那 450 分钟买到的不是这四行。 买到的是知道： 显存占比不能再高， 因为激活峰值在启动时不可见；请求上限不能再高， 因为它要和 token 预算互换；前缀缓存是有理由关掉的， 而真实部署应该把这个理由反过来；以及那个包办了全部收益的 flag， 靠的是让验证步骤更便宜， 而不是让它更多产。

The through-line is worth restating in a form that outlives this model and this hardware. Speculative decoding turns generation into a bet, and the return on that bet is tokens accepted divided by the cost of the step that verified them. Almost every tuning lever moves one of those two terms, usually at the other's expense. When a lever seems to win on the metric you can read most easily — accept length, or TPOT — check which half of the fraction you are looking at. In this search, tuning on accept length alone would have picked a verify window that was 41% slower, and tuning on TPOT alone would have attributed a draft-model problem to the attention kernel.这条主线值得用一种能活得比这个模型和这批硬件更久的形式重述一遍。 投机解码把生成变成一场赌注， 而赌注的回报是「被接受的 token 数」除以「验证它们那一步的成本」。 几乎每一个调优杠杆都在动这两项之一， 而且通常以另一项为代价。 当某个杠杆在你最容易读到的指标上看起来赢了——接受长度， 或者 TPOT——先确认你看的是这个分数的哪一半。 在这次搜索里， 只按接受长度调参会选出一个慢 41% 的验证窗口；只按 TPOT 调参则会把草稿模型的问题归因到 attention kernel 上。

Everything here is reproducible from two scripts and two CSVs, archived alongside the recipe and the gate that justified it. The numbers on this page are spliced in from those CSVs by a generator, so the page cannot drift from the measurements even if the prose around it is rewritten.这里的一切都可以从两个脚本和两份 CSV 复现， 它们与配方、以及为配方提供依据的闸门一起归档。 本页所有数字都由生成器从那些 CSV 回写进来， 所以即使周围的文字被重写， 页面也不会与测量结果发生漂移。