vllm-loop model-improvement progress tracker junk-tokens
models / / #40
▼ MADE NEGATIVE PROGRESS

PP6 depth/concurrency campaign conclusion

Scripted depth/concurrency campaign complete with NO PP6 production promotion: K7 passes the correctness + token-accounting matrix, but c3 depth is the limiting production-shaped behavior (c3/c1 retained rate 0.87 @32k -> 0.10 @192k); K3 failed its eager admission gate (branch stops before…

When (PT)2026-09-08 05:00 PT
ServingPP6 AutoRound W4A16 scope; GPU6 untouched; PP8 and P2P out of scope
Engine
KV

occurred_at is date-precision (report file date)

Benchmarks

MetricValueΔ vs previousUnitContextNote
c1_decode_tok_s (c1_decode_tok_s) 81.6 tok/s first point tok/s 8k depth input
c1_decode_tok_s (c1_decode_tok_s) 82.52 tok/s first point tok/s 32k depth (TTFT 5.95 s)
c1_decode_tok_s (c1_decode_tok_s) 69.85 tok/s first point tok/s 64k depth (TTFT 10.04 s)
c1_decode_tok_s (c1_decode_tok_s) 63.15 tok/s first point tok/s 128k depth (TTFT 18.62 s)
c1_decode_tok_s (c1_decode_tok_s) 78.79 tok/s first point tok/s 192k depth (TTFT 27.67 s)
c1_decode_tok_s (c1_decode_tok_s) 61.32 tok/s first point tok/s 261,888 depth (TTFT 37.09 s)
c3_decode_tok_s (c3_decode_tok_s) 72.12 tok/s first point tok/s 32k depth (c3/c1 0.87)
c3_decode_tok_s (c3_decode_tok_s) 32.97 tok/s first point tok/s 64k depth (c3/c1 0.47)
c3_decode_tok_s (c3_decode_tok_s) 11.43 tok/s first point tok/s 128k depth (c3/c1 0.18)
c3_decode_tok_s (c3_decode_tok_s) 8.22 tok/s first point tok/s 192k depth (c3/c1 0.10)

What didn't work

  • c3 depth collapses with context (concurrency-3 per-user rate falls to 10% of c1 at 192k)
  • K3 failed its eager admission gate - no repartitioning, P2P, or winner qualification attempted

raw JSON: /api/reports/40