From bacabdcd49c37594ac7ad214ec4ff95a3373ca76 Mon Sep 17 00:00:00 2001 From: Frank Date: Sat, 26 Sep 2026 20:55:02 -0700 Subject: [PATCH] frank-primary: record measured battery numbers and the drafter-placement law Battery 6/6: alias, reasoning_content, tool_calls, vision (probe image described exactly), reasoning_effort high -> 200. Decode 11.8 t/s at depth-0 prose (draft 109/accepted 72), 0.864 mean len 2.73 on tool traffic. VRAM 29%/36% (output-head card carries the drafter). Also records the ROCm1 drafter law in the runbook body (borrowed output.weight + n_copies==1 abort) and the two cutover traps: foreign-process health false green, and llama-server SIGTERM cleanup hang. --- ...k-primary-llamacpp-qwen38-flash-next-q8.md | 27 +++++++++++++++++-- 1 file changed, 25 insertions(+), 2 deletions(-) diff --git a/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md b/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md index 6e6aed7..ffc112e 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md +++ b/serverAppConfigs-docs/LLM/frank-primary-llamacpp-qwen38-flash-next-q8.md @@ -72,7 +72,14 @@ Lucebox DS4.1 quadlet that held the port for hours earlier the same day draft load with `invalid vector subscript`). `ngram-mod` is draftless, so it chains legally behind the drafter. Depth 2: each longer verification round re-batches the CPU expert reads, so depth buys less - than bandwidth here. + than bandwidth here. **The drafter must sit on ROCm1** — the head also + borrows the target's `output.weight`, which a `-ts 1,1` split places on + the last-layers card; with tensor overrides set the draft scheduler runs + `n_copies==1`, and a draft context elsewhere aborts at load + (`ggml-backend.cpp:941`, "pre-allocated tensor (output.weight) in a + buffer (ROCm1) that cannot run the operation"). The 2026-09-26 cutover + crash-looped 4× on `ROCm0` before the probe confirmed `ROCm1` loads — + the frank-medium precedent (drafter on the last-layers card) was right. - **Vision, explicitly.** This build never autoloads projectors — without `--mmproj` the server comes up healthy and text-only and answers image requests with 500 (the frank-medium 2026-09-17 trap). The BF16 projector @@ -118,7 +125,23 @@ truly free, relinks + enables + restarts the root unit, and waits up to - VRAM/RAM headroom: `rocm-smi` per card + `free -g` after the first long completion. -_Measured numbers land here after the deploy battery._ +_**Measured (2026-09-26 battery, 6/6 pass).** Load: ~7 s warm (mmap from +page cache; the fetch's own verify pass pre-warms it — a cold first load +pages in 175 GiB). Decode: **11.8 t/s** `predicted_per_second` at +depth-0 prose, 128-token generation, draft_n 109 / accepted 72 (0.66 on +reasoning-heavy prose; **0.864 mean len 2.73** on the tool-call task). +Vision: the probe image (green square centered on a red field) described +exactly. VRAM: GPU0 29% / GPU1 36% of 32 GiB (~9.3 / ~11.5 GiB — the +output-head card carries the drafter, as designed; ~22 GiB free per +card). RAM: ~11 GiB used, ~226 GiB page cache (the payload itself), +available ~235 of 246 GiB. `reasoning_effort: high` → 200. Two cutover +traps for the record: a LAST-ACK tail from a LAN client held +`wait_port_free` until it cleared (a foreign process on the port gates +the deploy correctly — but the deployer's `/health` wait cannot tell WHO +answers; after any aborted deploy, confirm the responder is the unit's +own MainPID), and llama-server's SIGTERM cleanup hung after +spec-decode service (logged `cleaning up before exit` and never exited — +`kill -9` was needed; `systemctl stop` relies on TimeoutStopSec)._ ## Rollback