vllm-loop model-improvement progress tracker junk-tokens
models / / #44
▼ MADE NEGATIVE PROGRESS

Hourly update #19 — chunk-size loop: both hypotheses rejected

Tested prefill-chunk size as the ITL-tail fix: MAXBATCH=8192 collapses KV (7.86M->3.23M) and fails c1; MAXBATCH=1024 is rejected by the multimodal min. No free ITL fix; baseline restored

When (PT)2026-09-12 13:02 PT
ServingTP1xPP8 (5,5,5,5,5,5,5,5), max-num-seqs 32, util 0.94
Enginevllm-backport-v41:sm80-v13 (v0.13.0 + Schaka DSV41 kit + patches 0007/0008)
KVfp8_ds_mla, 7,859,683-token pool (7.50x @1M), block 128

TRITON_MLA_SPARSE_DSV41, DSpark k=5, VLLM_SPARSE_DECODE_HEAD_BLOCK_SIZE=32, default max-num-batched-tokens

Benchmarks

MetricValueΔ vs previousUnitContextNote
agg_512k_c16_tok_s (512K aggregate decode @ c16) 98.9 tok/s 0.0% tok/s shared 487K prefix, pure decode, output 256
c1_gen_tok_s (c1 Gen tok/s @ 32K) 20.6 tok/s 0.0% tok/s frozen PP8 util-0.94 baseline, 32K in / 256 out, 16 real-text prompts, ignore_eos
c4_gen_tok_s (c4 Gen tok/s @ 32K) 53.4 tok/s 0.0% tok/s 32K in / 256 out, 16 prompts
c8_c1_scaling (c8/c1 scaling @ 32K) 3.82 x 0.0% x derived
c8_gen_tok_s (c8 Gen tok/s @ 32K) 78.9 tok/s 0.0% tok/s 32K in / 256 out, 16 prompts
kv_pool_tokens (KV token capacity (PP8 util 0.94)) 7859683 tokens 0.0% tokens default max-num-batched-tokens

What was found

  • max-num-batched-tokens trades directly against KV capacity: MAXBATCH=8192 dropped the PP8 pool from 7.86M to 3.23M tokens and failed c1
  • max-num-batched-tokens is floored by the vision item (max_tokens_per_mm_item=1025); MAXBATCH=1024 is rejected at startup
  • the ITL p99 tail also appears in shared-prefix decode without prefill interleaving -> it is a depth decode-step tail, not purely chunked-prefill interleaving
  • master-table c8 is noisy: 101.5 (util 0.96) vs 78.9 (util 0.94) for a change that should not matter -> raise prompts/reps for a stable verdict

What was tried

  • MAXBATCH=8192 (larger prefill chunks -> fewer decode interruptions)
  • MAXBATCH=1024 (smaller chunks -> finer interleave)

What didn't work

  • MAXBATCH=8192: KV 7,859,683 -> 3,234,468 tokens and c1 fail=16 (dead)
  • MAXBATCH=1024: ValueError Chunked MM input disabled but max_tokens_per_mm_item (1025) > max_num_batched_tokens (1024)

What's next

  • harden the master table (32+ prompts, 2 reps, report variance) so the verdict is not noise-limited
  • attack the c4/c8 ITL p99 tail as a decode-side problem (sparse-MLA decode step variability / DSpark verify), since it persists without prefill interleaving
  • re-run the 512K@c16 milestone on the restored baseline and one live opencode c1 run (realness gate)

Notes

Chunk size is constrained to roughly [1030, small] and larger values cost KV, so it is not a free ITL fix. Baseline config restored and healthy.

raw JSON: /api/reports/44