docs: LLM index + work log for frank-primary shape D, harness window
index.md intro and mermaid nodes still described 4 x 262144 fp8/bf16 and frank-small at 2 x 262144; the active-deployment list had no frank-primary entry at all. Fixed both, plus a malformed GLM list item. log.md gains the shape-D entry (the SEQS=2 constraint, the arena-lever measurement, the 2x2 probe table, the measured 1.99x), and the frank-small entry's stale consequence bullet now points at the resolution. frank-small runbook: the compaction-fit note is corrected - the 200704 window moved frank-primary's trigger to 140,492, which fits a 180,224 slot again.
This commit is contained in:
@@ -105,13 +105,13 @@ The per-slot window is the whole price. Two consequences the fleet depends on:
|
||||
- **A prompt over 180,224 tokens is a hard 400**, so `~/.omp/agent/models.yml`
|
||||
sets `contextWindow: 180224` for this endpoint — the harness compacts before
|
||||
it reaches the ceiling instead of discovering it.
|
||||
- **frank-primary's compaction prompt no longer fits with room to answer.**
|
||||
Its trigger is 183,501 tokens — already above this slot — and its summary
|
||||
budget is 32,768 output tokens. On the two-slot shape (262,144) that left
|
||||
32.8k of output room, which is why that number was chosen; on this shape a
|
||||
single compaction request must be truncated or split. Nothing in the fleet
|
||||
routes compaction here automatically, but any runbook that claims frank-small
|
||||
"fits a compaction request in ONE request" is describing the 262,144 shape.
|
||||
- **frank-primary's compaction prompt fits inside one slot again.** At
|
||||
frank-primary's shape-D window (200,704) its compaction trigger is 140,492
|
||||
tokens (70 %), so one compaction request lands here with 39,732 tokens of
|
||||
room. At the previous 262,144 window the trigger was 183,501 — *above* this
|
||||
slot — and the request had to be truncated or split. Nothing in the fleet
|
||||
routes compaction here automatically; the point is that the 2026-10-02 window
|
||||
trim restored the fit.
|
||||
|
||||
## Client contract
|
||||
|
||||
|
||||
@@ -4,18 +4,19 @@ Documentation for the LLM inference fleet on the two hosts (2026-10-02):
|
||||
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
|
||||
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
|
||||
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
|
||||
on :7770, TP=2 with automatic tiered-expert offload, **4 seqs x 262144 ctx
|
||||
(the model's full native window)**, fp8 KV (5 boot but leave no runtime
|
||||
headroom; 6 do not fit), no speculative decode, MNBT 1024, MoE gather 32x64, util 0.98,
|
||||
reasoning/tool parsers wired (qwen3 / qwen3_xml) with preserve_thinking on,
|
||||
measured 36.4-39.4 tok/s decode — the 2026-10-01 bf16 no-spec cutover plus
|
||||
the night ladder; rollback frank-primary-fp8-mtp-20261001), and Dankclaw
|
||||
on :7770, TP=2 with automatic tiered-expert offload, **shape D since
|
||||
2026-10-02: 3 slots x 200704 ctx** with the KV pool sized for **~2 full-length
|
||||
conversations** — measured 398,658 tokens = 1.99x — fp8 KV, no speculative
|
||||
decode, MNBT 1024, MoE gather 32x64, util 0.98, reasoning/tool parsers wired
|
||||
(qwen3 / qwen3_xml) with preserve_thinking on. `--max-num-seqs` must stay 3:
|
||||
SEQS 2 cannot compile this model. Rollback frank-primary-fp8-mtp-20261001),
|
||||
and Dankclaw
|
||||
runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770:
|
||||
kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF Q8_0_ROCMFPX_AGENT, a ftype-115
|
||||
file only that fork can read, with the repo's MTP head drafting 4 tokens, ONE
|
||||
session at 262,144, f16 KV, vision via the bundled projector)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q5_K_M
|
||||
GGUF, 2 x 262,144, asymmetric KV f16 K + q8_0 V, on the RTX 3090)**.
|
||||
GGUF, 3 x 180,224, asymmetric KV f16 K + q8_0 V, on the RTX 3090)**.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -31,9 +32,9 @@ graph LR
|
||||
gfx1201, 32 GiB each]
|
||||
B2[tcclaviger RDNA4 vLLM fork, digest-pinned
|
||||
podman run, frank-primary, port 7770]
|
||||
C[Qwen3.8-Flash-Next FP8 bf16 no-spec
|
||||
TP=2, expert offload automatic, 4 x 262144 ctx,
|
||||
fp8 KV, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
C[Qwen3.8-Flash-Next FP8, no spec decode
|
||||
TP=2, expert offload automatic, 3 x 200704 ctx,
|
||||
fp8 KV pool ~2 full-length, MNBT 1024, gather 32x64, qwen3 parsers]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
@@ -46,7 +47,7 @@ frank-large, port 7770]
|
||||
G[llama.cpp master CUDA sm_86
|
||||
frank-small, port 7771]
|
||||
H[MiMo-V2.6 Q5_K_M GGUF
|
||||
2 x 262144 ctx, f16 K + q8_0 V, flash-attn]
|
||||
3 x 180224 ctx, f16 K + q8_0 V, flash-attn, mmproj vision]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -79,10 +80,17 @@ frank-small, port 7771]
|
||||
THREE slots of 180,224 tokens (pool 540,672) with K f16 + V q8_0 and the
|
||||
mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free.
|
||||
The KV arithmetic, the flash-attn dependency, rollback to Q6_K
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since
|
||||
2026-10-02 (shape D)** (root systemd unit, port 7770): the tcclaviger RDNA4
|
||||
vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
|
||||
R9700s, TP=2 with automatic expert offload — `--max-model-len 200704` x 3
|
||||
slots, fp8 KV, pool 3,150,000,000 B/rank (measured 398,658 tokens = 1.99x),
|
||||
MNBT 1024, gather 32x64, no spec decode, prefetch off. Why `--max-num-seqs`
|
||||
must stay 3 (SEQS 2 cannot compile this model), why the arena lever is
|
||||
`max_model_len` rather than the KV pool, and the measured envelope
|
||||
- [[frank-primary-glm53-flash-w4a16]] — RETIRED 2026-09-30 → 10-01 (the
|
||||
- failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
|
||||
(2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
|
||||
(2026-09-30, root systemd unit, port 7770): the tcclaviger RDNA4 vLLM fork
|
||||
failed GLM experiment; frank-primary runs Qwen FP8 again, see the intro):
|
||||
the tcclaviger RDNA4 vLLM fork
|
||||
(digest-pinned `tcclaviger/vllm@sha256:da4029666…`, plain `podman run`,
|
||||
no shim) serving Intel/GLM-5.3-Flash-W4A16-AutoRound — AutoRound symmetric
|
||||
int4 group 128, ~170 GiB, 34 shards + 2 extras, 113,074 tensors — on BOTH
|
||||
|
||||
Reference in New Issue
Block a user