Hourly update #18 — 32K master table online (PP8)

Built the one-command 32K master table; PP8 baseline c1 20.6 / c4 53.4 / c8 78.9 Gen tok/s, 512K@c16 98.9 tok/s; util 0.96 OOMs 512K so lowered to 0.94

When (PT)2026-09-12 12:33 PT

ServingTP1xPP8 (5,5,5,5,5,5,5,5), max-num-seqs 32, util 0.94

Enginevllm-backport-v41:sm80-v13 (v0.13.0 + Schaka DSV41 kit + patches 0007/0008)

KVfp8_ds_mla, 7,859,683-token pool (7.50x concurrency @1M), block 128

TRITON_MLA_SPARSE_DSV41, DSpark k=5, VLLM_SPARSE_DECODE_HEAD_BLOCK_SIZE=32

Benchmarks

Metric	Value	Δ vs previous	Unit	Context
`accept_pct_c1` (DSpark acceptance @ c1 32K)	23.4 %	first point	%	server /metrics, real text
`agg_512k_c16_tok_s` (512K aggregate decode @ c16)	98.9 tok/s	first point	tok/s	shared 487K prefix, pure decode, output 256
`c1_gen_tok_s` (c1 Gen tok/s @ 32K)	20.6 tok/s	first point	tok/s	32K in / 256 out, 16 real-text prompts, ignore_eos, PP8 util 0.94
`c4_gen_tok_s` (c4 Gen tok/s @ 32K)	53.4 tok/s	first point	tok/s	32K in / 256 out, 16 prompts
`c8_c1_scaling` (c8/c1 scaling @ 32K)	3.82 x	first point	x	derived from the 32K table
`c8_gen_tok_s` (c8 Gen tok/s @ 32K)	78.9 tok/s	first point	tok/s	32K in / 256 out, 16 prompts
`kv_pool_tokens` (KV token capacity (PP8))	7859683 tokens	-15.9%	tokens	util 0.94

What was found

vllm bench `custom` dataset needs pandas (vllm[bench]) and the model has no chat template -> must pass --skip-chat-template
util 0.96 with max-num-seqs 32 OOMs the 512K prefill (torch.OutOfMemoryError in a GEMM, GPU 63.01/63.39 GiB)

What was fixed

one-command 32K master-table harness (scripts/master_table.py): grid -> per-cell vllm bench serve -> JSON DB -> markdown table
folded in VLLM_SPARSE_DECODE_HEAD_BLOCK_SIZE=32 (patches/0008)
lowered PP8 util 0.96 -> 0.94 so the 512K cell is reliable (pool 7.86M tokens, 7.50x @1M)

What was tried

random vs sonnet vs custom dataset for the master table (random gives ~1% acceptance -> unrepresentative; custom real-text gives ~23%)
util 0.96 vs 0.94 for the 512K prefill

What didn't work

util 0.96 + SEQS 32: 512K prefill OOM (regression-gate failure, not a small regression)
sonnet dataset: requires a Jinja chat template this tokenizer does not ship
custom dataset without pandas: ImportError

What's next

attack the c4/c8 ITL p99 tail (~0.6-2.3 s vs ~100 ms median) via chunked-prefill/scheduling (first loop hypothesis)
bake pandas into the serving image so the master table never depends on an ad-hoc install
re-run the 32K table at util 0.94 as the frozen baseline; then one hypothesis per loop

Notes

First cycle of the new 32K-master-table loop. PP8 is the serving layout; PP6 remains an A/B reference. 500 tok/s @512k still far off (98.9 at c16 shared-prefix).