dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: frank-large cache pin verified warm (16x prefill) + depth law

Browse Source 
Probe on the live endpoint (2026-10-06): 17.3k cold prefill 27.0 s ->
warm turn resumes at cache_n 16,384, prefill 1.7 s. The boot WARNING
'smaller than ONE full-context entry' sizes against the native 262,144
window, not our 200,700 slot — documented as expected in the checklist.
Tuning notes record the depth law (decode 5.5 t/s @ 46k -> 3.3-3.8 @ 80k
vs 30-50 shallow; KV reads ~8 GB/s effective; no supported flag touches
decode) and the AGENTS.md unit paragraph regains the unit-facts
parenthetical the reflow dropped.
This commit is contained in:
Frank
2026-10-06 00:34:59 -07:00
parent 3eb4fa2b62
commit d39424ffba
3 changed files with 89 additions and 11 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+23 -5
View File
@@ -323,11 +323,16 @@ audio encoder, and left again with the Gemma4 retirement on 2026-10-01
The **halogen engine** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`,
digest-pinned, published tag 0.1.4) as a podman container driven by
`scripts/serve-frank-large.sh` under a **root systemd unit**
(`scripts/frank-large.service`, `User=dank`, `Restart=always`, `ExecStop=podman
stop -t 120 frank-large-halogen`, the script `exec podman wait` so the restart
policy is real) on port 7770 from the Strix Halo iGPU. Engine stdout goes to
`/home/dank/logs/frank_large.log` — **the boot ledger (KV pool, drafter state,
any clamp) lives there, not in the journal**. Serving
(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
`ExecStop=podman
stop -t 120 frank-large-halogen`, the script runs `podman run` FOREGROUND —
the container's exit code is the service's exit so `Restart=always` stays
real, and the engine ledger lands on disk) on port 7770 from the Strix Halo
iGPU. Engine stdout goes to `/home/dank/logs/frank_large.log` — **the boot
ledger (KV pool, prompt cache, drafter state, any clamp) lives there, not in
the journal** (the 2026-10-05 `-d` + `podman wait` shape captured only the
container ID — fixed 2026-10-06).
Serving
`peonist-ai/halogen-qwen3.8-27b`: `qwen3.8-27b-p1w4d-d2.hgn` (35,865,565,184 B
≈ 33.4 GiB, sha256 `274c3fc7…`, 1352 tensors; p1 weights / w4 activations /
d2 KV) + the flat `tokenizer/` directory (5 files). Cutover 2026-10-05 from the ROCmFPX llama.cpp fork
@@ -361,6 +366,19 @@ temperature is greedy, so the thinking recipe exists only in `models.yml`
Thinking lands in **`reasoning_content`** (harness field unchanged); the same
Qwen3.8 template rejects `reasoning_effort` outside `xhigh|medium|low`, so the
`models.yml` fold stays mandatory.
`HALOGEN_CACHE_MB=16384` pins the prompt cache (2026-10-06): auto is OFF
on this box — the floor is 2 full-context entries (36.9 GB) and the boot
ledger had 35.7 GB free — so every deep request re-prefilled its whole
79-81k context (84.9-179.8 s per request for ~500-650 new tokens per
turn) until the pin. Probe-verified: a 17.3k cold prefill of 27.0 s
resumes in 1.7 s on the next turn (cache_n 16,384). Decode-at-depth is
structural: measured 5.5 t/s at 46k → 3.3-3.8 at 80k vs 30-50 shallow
(the KV pool reads at ~8 GB/s effective vs ~180-240 for the weights);
`docs/FLAGS.md` documents no decode-affecting lever and 0.1.4 is the
latest engine. The boot WARNING "smaller than ONE full-context entry
(18.4 GB at kMaxCtx=262144)" is a false positive for our slot: the
deepest possible entry is 200,700 × ~70 KB ≈ 14 GB, inside the pin, and
the 262k entry it fears cannot exist (over-slot is a 400).
Fetch: `scripts/download-frank-large-halogen.sh` into
`/home/dank/models/halogen-qwen3.8-27b/` (`-f` + byte count + LFS sha256 +