diff --git a/AGENTS.md b/AGENTS.md index 6632749..6a8da83 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -427,8 +427,12 @@ digest-pinned) serving **`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8`** **fp8 KV**, **no speculative decode**, **expert prefetch OFF**, and since 2026-10-02 the **shape-D** geometry: `--max-model-len 200704` × 3 slots with `--kv-cache-memory-bytes 3,150,000,000`/rank, so the pool admits **~2 -full-length conversations (measured: 398,658 tokens = 1.99×)** while the argv keeps the only slot count -that compiles (below). +full-length conversations** (measured live: `398,658 tokens, Maximum +concurrency for 200,704 tokens per request: 1.99×`) while the argv keeps the +only slot count that compiles (below). `models.yml` carries +`contextWindow: 200704` for this endpoint — the served window, not the model's +native 262144, so the harness compacts before vLLM answers a longer prompt +with a 400. **`--max-num-seqs 2` is impossible on this model.** Measured 2026-10-02 by single-variable boot probe against the then-deployed shape: SEQS 2 fails at diff --git a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md index 46714e9..693fd66 100644 --- a/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md +++ b/serverAppConfigs-docs/LLM/frank-small-llama-mimo-q5km.md @@ -105,13 +105,13 @@ The per-slot window is the whole price. Two consequences the fleet depends on: - **A prompt over 180,224 tokens is a hard 400**, so `~/.omp/agent/models.yml` sets `contextWindow: 180224` for this endpoint — the harness compacts before it reaches the ceiling instead of discovering it. -- **frank-primary's compaction prompt no longer fits with room to answer.** - Its trigger is 183,501 tokens — already above this slot — and its summary - budget is 32,768 output tokens. On the two-slot shape (262,144) that left - 32.8k of output room, which is why that number was chosen; on this shape a - single compaction request must be truncated or split. Nothing in the fleet - routes compaction here automatically, but any runbook that claims frank-small - "fits a compaction request in ONE request" is describing the 262,144 shape. +- **frank-primary's compaction prompt fits inside one slot again.** At + frank-primary's shape-D window (200,704) its compaction trigger is 140,492 + tokens (70 %), so one compaction request lands here with 39,732 tokens of + room. At the previous 262,144 window the trigger was 183,501 — *above* this + slot — and the request had to be truncated or split. Nothing in the fleet + routes compaction here automatically; the point is that the 2026-10-02 window + trim restored the fit. ## Client contract diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index a2a51a5..74620ce 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -4,18 +4,19 @@ Documentation for the LLM inference fleet on the two hosts (2026-10-02): **Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s -on :7770, TP=2 with automatic tiered-expert offload, **4 seqs x 262144 ctx -(the model's full native window)**, fp8 KV (5 boot but leave no runtime -headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98, -reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on, -measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus -the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw +on :7770, TP=2 with automatic tiered-expert offload, **shape D since +2026-10-02: 3 slots x 200704 ctx** with the KV pool sized for **~2 full-length +conversations** — measured 398,658 tokens = 1.99x — fp8 KV, no speculative +decode, MNBT 1024, MoE gather 32x64, util 0.98, reasoning/tool parsers wired +(qwen3 / qwen3_xml) with preserve_thinking on. `--max-num-seqs` must stay 3: +SEQS 2 cannot compile this model. Rollback frank-primary-fp8-mtp-20261001), +and Dankclaw runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770: kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF Q8_0_ROCMFPX_AGENT, a ftype-115 file only that fork can read, with the repo's MTP head drafting 4 tokens, ONE session at 262,144, f16 KV, vision via the bundled projector)** and **llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q5_K_M -GGUF, 2 x 262,144, asymmetric KV f16 K + q8_0 V, on the RTX 3090)**. +GGUF, 3 x 180,224, asymmetric KV f16 K + q8_0 V, on the RTX 3090)**. > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -31,9 +32,9 @@ graph LR gfx1201, 32 GiB each] B2[tcclaviger RDNA4 vLLM fork, digest-pinned podman run, frank-primary, port 7770] - C[Qwen3.8-Flash-Next FP8 bf16 no-spec -TP=2, expert offload automatic, 4 x 262144 ctx, -fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers] + C[Qwen3.8-Flash-Next FP8, no spec decode +TP=2, expert offload automatic, 3 x 200704 ctx, +fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers] end subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] @@ -46,7 +47,7 @@ frank-large, port 7770] G[llama.cpp master CUDA sm_86 frank-small, port 7771] H[MiMo-V2.6 Q5_K_M GGUF -2 x 262144 ctx, f16 K + q8_0 V, flash-attn] +3 x 180224 ctx, f16 K + q8_0 V, flash-attn, mmproj vision] E2 --> E --> F E2 --> G --> H end @@ -79,10 +80,17 @@ frank-small, port 7771] THREE slots of 180,224 tokens (pool 540,672) with K f16 + V q8_0 and the mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free. The KV arithmetic, the flash-attn dependency, rollback to Q6_K +- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since + 2026-10-02 (shape D)** (root systemd unit, port 7770): the tcclaviger RDNA4 + vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both + R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3 + slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x), + MNBT 1024, gather 32x64, no spec decode, prefetch off. Why `--max-num-seqs` + must stay 3 (SEQS 2 cannot compile this model), why the arena lever is + `max_model_len` rather than the KV pool, and the measured envelope - [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the -- failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro): - (2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork - (2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork + failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro): + the tcclaviger RDNA4 vLLM fork (digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`, no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric int4 group 128, ~170 GiB, 34 shards + 2 extras, 113,074 tensors — on BOTH diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 012f75d..95addee 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -971,9 +971,10 @@ the 3090", then **"frank-small should have vision"**. - **Three contexts cost window, and that is the tradeoff Frank was told:** 180,224 tokens per conversation, not the native 262,144. A longer prompt is a hard `400 exceed_context_size_error`, so `models.yml` sets - `contextWindow: 180224`. Consequence: frank-primary's 183,501-token - compaction request no longer fits here with summary room; the two-slot - rollback is the way back. + `contextWindow: 180224`. Consequence at the time: frank-primary's then + 183,501-token compaction request did not fit here with summary room — + resolved later the same day when frank-primary's window moved to 200,704 + and its trigger fell to 140,492 (see the shape-D entry below). - Measured (2026-10-02, this shape): decode **106.8 t/s**, prefill **3,814 t/s** (~5 % faster decode than 2 x 262,144 — shorter slot, shorter attention scan), three concurrent requests answered in parallel in 1.7 s with a fourth @@ -993,3 +994,79 @@ the 3090", then **"frank-small should have vision"**. payloads). Kept in place for rollback: `MiMo-Q6K/`, `MiMo-Q5KM/`, `Qwen3.8-27B-UD-Q4_K_XL/` and `Qwen3.8-27B-UD-Q8_K_XL/` (the latter holds the gufo DFlash2 drafter). Runbook [[frank-small-llama-mimo-q5km]]. + +## 2026-10-02 — frank-primary shape D: 3 x 200704, fp8, pool sized for 2 contexts + +Frank's order: shrink frank-primary to a 200704-token window with about two +resident full-length contexts, cut the KV pool, and drop MNBT to 1024. The +first attempt (`30904ba`, 2 x 200704 + SEQS=2) **crash-looped**; the revert +went live as `e460bbb`, and the shape was then re-applied one law at a time on +top of the proven-good base, root cause first. + +- **ROOT CAUSE: `--max-num-seqs 2` cannot compile this model. It was never OOM, + never MNBT, never the KV dtype.** Four single-variable boot probes against + the then-deployed shape (production stopped for each, candidate booted AS the + service): + + | MAXLEN | SEQS | MNBT | result | + |---|---|---|---| + | 262144 | 2 | 1024 | **Engine core initialization failed** | + | 262144 | 2 | 2048 | **Engine core initialization failed** | + | 262144 | 3 | 1024 | healthy - 792,069 tokens, 3.02x, arena 6,248 | + | 262144 | 3 | 2048 | healthy - the then-deployed incumbent | + + `torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints + violated (L['query_start_loc'].size()[0]) ... specialized it to be a constant + (3)`, raised from AOT `build_guards(strict_error=True)` during + `determine_available_memory -> profile_run -> _dummy_run`, ~7 min in. + `model_runner.py` sizes that input buffer `max_num_reqs + 1` and `_dummy_run` + sets `num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so its width is + **SEQS + 1** for ANY `MNBT >= SEQS` - 3 at SEQS 2, 4 at SEQS 3, exactly the + constant in the message. MNBT only moves the compile RANGE ((1,1024) vs + (1,2048)), which is why the identical error appears at both rungs. + `--enforce-eager` bypasses the path but drops CUDA graphs. The 2026-10-01 + bf16 2 x 262144 failure was this same error class; its message named "KV + dtype and SEQS" together and nobody bisected, so the dtype was blamed - it + was a red herring. +- **THE ARENA LEVER IS `max_model_len`, NOT `--kv-cache-memory-bytes`.** The + planner sizes expert cache as `budget - non-expert - KV reserve - graphs - + overhead`, and the KV reserve is registry-priced at **9190 B/token x + max_model_len**, independent of the pool override: + + | MAXLEN | KV reserve | arena | + |---|---|---| + | 262144 | 2.48 GiB | 14.30 GiB / **6,248 slots** | + | 200704 | 1.91 GiB | 14.87 GiB / **6,496 slots** | + + d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9 + pool cut (2.75 GiB/rank) bought **zero** arena slots. The previous session's + premise - "the pool cut becomes arena, ~7,500 slots" - was wrong; the pool + cut is a runtime-HEADROOM lever (the shape it replaced died with `0 bytes + free` on a real 995-token request). +- **Final shape (option D):** `--max-model-len 200704` x 3 slots, fp8 KV, + `--kv-cache-memory-bytes 3,150,000,000`/rank, MNBT 1024, gather 32x64, no + speculative decode, prefetch off, no `--expert-precise-memory`. The argv + keeps the only slot count that compiles; the POOL caps resident + conversations. Deployed `c769ff2`, healthy in 558 s, e2e 33/33. +- **Measured live, and one prediction corrected.** `expert cache space 14.87 + GiB / 6,496 slots` - the arena prediction landed exactly. But `GPU KV cache + size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request: + 1.99x` (3,146,129,408 B reserved), not the 2.04x the arithmetic predicted: + **the per-token KV price is shape-dependent** - 7,701 B/token at MAXLEN + 262144 (6,100,000,000 B -> 792,069 tokens) vs **7,892** at 200704, ~2.5 % + dearer. Never extrapolate a concurrency figure across geometries. +- **Smoke-verified on the live endpoint:** three concurrent 67-token requests + answered in parallel (~4.1 s each) - the 3 slots are real; native `tool_calls` + (`get_weather {"city":"Seattle"}`); thinking in `reasoning` with clean + `content`; `reasoning_effort` accepts only {xhigh, medium, low} and answers + 400 for anything else (re-measured; the fork's message names the set). +- **Harness contract followed the window:** `models.yml` `contextWindow` + 262144 -> **200704** (compaction trigger 140,492 + 36,864 = 177,356 <= + 200,704). Side effect: frank-primary's compaction request now fits inside a + frank-small slot again, which it had stopped doing when frank-small went to + 3 x 180,224. +- **Comment-only follow-up** `295d578` recorded the measured numbers and was + deployed so the per-service revision marker sits at HEAD; e2e 33/33 again. +- Rollback for the pre-shape-D fp8+MTP4 era stays git tag + `frank-primary-fp8-mtp-20261001`. Runbook + [[dankserver-qwen38-fp8-expert-offload-tuning]].