diff --git a/AGENTS.md b/AGENTS.md index c50021b..a231c23 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -323,11 +323,16 @@ audio encoder, and left again with the Gemma4 retirement on 2026-10-01 The **halogen engine** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, published tag 0.1.4) as a podman container driven by `scripts/serve-frank-large.sh` under a **root systemd unit** -(`scripts/frank-large.service`, `User=dank`, `Restart=always`, `ExecStop=podman -stop -t 120 frank-large-halogen`, the script `exec podman wait` so the restart -policy is real) on port 7770 from the Strix Halo iGPU. Engine stdout goes to -`/home/dank/logs/frank_large.log` — **the boot ledger (KV pool, drafter state, -any clamp) lives there, not in the journal**. Serving +(`scripts/frank-large.service`, `User=dank`, `Restart=always`, +`ExecStop=podman +stop -t 120 frank-large-halogen`, the script runs `podman run` FOREGROUND — +the container's exit code is the service's exit so `Restart=always` stays +real, and the engine ledger lands on disk) on port 7770 from the Strix Halo +iGPU. Engine stdout goes to `/home/dank/logs/frank_large.log` — **the boot +ledger (KV pool, prompt cache, drafter state, any clamp) lives there, not in +the journal** (the 2026-10-05 `-d` + `podman wait` shape captured only the +container ID — fixed 2026-10-06). +Serving `peonist-ai/halogen-qwen3.8-27b`: `qwen3.8-27b-p1w4d-d2.hgn` (35,865,565,184 B ≈ 33.4 GiB, sha256 `274c3fc7…`, 1352 tensors; p1 weights / w4 activations / d2 KV) + the flat `tokenizer/` directory (5 files). Cutover 2026-10-05 from the ROCmFPX llama.cpp fork @@ -361,6 +366,19 @@ temperature is greedy, so the thinking recipe exists only in `models.yml` Thinking lands in **`reasoning_content`** (harness field unchanged); the same Qwen3.8 template rejects `reasoning_effort` outside `xhigh|medium|low`, so the `models.yml` fold stays mandatory. +`HALOGEN_CACHE_MB=16384` pins the prompt cache (2026-10-06): auto is OFF +on this box — the floor is 2 full-context entries (36.9 GB) and the boot +ledger had 35.7 GB free — so every deep request re-prefilled its whole +79-81k context (84.9-179.8 s per request for ~500-650 new tokens per +turn) until the pin. Probe-verified: a 17.3k cold prefill of 27.0 s +resumes in 1.7 s on the next turn (cache_n 16,384). Decode-at-depth is +structural: measured 5.5 t/s at 46k → 3.3-3.8 at 80k vs 30-50 shallow +(the KV pool reads at ~8 GB/s effective vs ~180-240 for the weights); +`docs/FLAGS.md` documents no decode-affecting lever and 0.1.4 is the +latest engine. The boot WARNING "smaller than ONE full-context entry +(18.4 GB at kMaxCtx=262144)" is a false positive for our slot: the +deepest possible entry is 200,700 × ~70 KB ≈ 14 GB, inside the pin, and +the 262k entry it fears cannot exist (over-slot is a 400). Fetch: `scripts/download-frank-large-halogen.sh` into `/home/dank/models/halogen-qwen3.8-27b/` (`-f` + byte count + LFS sha256 + diff --git a/serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md b/serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md index 1d998cb..5d5124d 100644 --- a/serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md +++ b/serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md @@ -54,6 +54,15 @@ bearing numbers: set + host baseline on the 121.2 GiB box. The engine sizes the pool itself and **prints a boot ledger naming any clamp** — the post-deploy check reads that ledger, not a formula. +- **`HALOGEN_CACHE_MB=16384` (pinned 2026-10-06)** — auto is OFF on this + box: the floor is 2 x a full-context entry (36.9 GB at 262k; one entry + is ~13.7 GB at our 200,700-slot x ~70 KB/token) and the ledger had + 35.7 GB free, so the engine refused and every deep agent request + re-prefilled its whole 79-81k context — 84.9-179.8 s measured + (serve_api lines, 2026-10-06) for ~500-650 new tokens per turn. The + README prices the cache at ~20x time-to-first-token on a warm turn; + entries evict LRU by bytes; the reserve (HALOGEN_CACHE_RESERVE_MB, + default 8192) is re-checked against live MemAvailable on every store. - **No server-side sampler:** an absent temperature is GREEDY. The card's thinking recipe (temp 1.0 / top_p 0.95) lives in `models.yml` `extraBody` — on the wire only. The template defaults `reasoning_effort` to **xhigh** @@ -94,10 +103,14 @@ bearing numbers: those links dangle inside the container mount. - **Unit:** `scripts/frank-large.service` (root, `User=dank`, relinked into `/etc/systemd/system/`), `ExecStart` = serve script, `ExecStop` = `podman - stop -t 120 frank-large-halogen`, script `exec podman wait` so - `Restart=always` is real. Engine stdout goes to - `/home/dank/logs/frank_large.log` (the boot ledger lives there, not in the - journal). + stop -t 120 frank-large-halogen`, script runs `podman run` FOREGROUND + (2026-10-06: the 2026-10-05 `-d` + `exec podman wait` shape captured + only the container ID — the engine ledger existed nowhere on disk and + the post-deploy grep below read a one-line hex file; foreground run + returns the container's exit code so `Restart=always` is real and the + ledger lands in the log). Engine stdout goes to + `/home/dank/logs/frank_large.log` (the boot ledger lives there, not in + the journal). - **Cutover:** `deploy_llm_large` in `scripts/deploy-remote.sh` sweeps the legacy bare `llama-server --model /home/dank/models/kingjones-*` process and the legacy `frank-large` container name by NAME (never `pkill -f @@ -109,8 +122,15 @@ bearing numbers: ## Verification (post-deploy checklist) -1. `grep -iE 'clamp|kv|pool|drafter|dflash' /home/dank/logs/frank_large.log` - — ledger shows the pool at the asked shape, DFlash2 active, NO clamp line. +1. `grep -iE 'clamp|kv|pool|drafter|dflash|cache' /home/dank/logs/frank_large.log` + — ledger shows the pool at the asked shape, DFlash2 active, the + prompt cache line naming the pinned budget, NO clamp line. EXPECT the + WARNING "smaller than ONE full-context entry (18.4 GB at + kMaxCtx=262144)": it sizes against the NATIVE window, not our + 200,700 slot — the deepest possible entry (~14 GB at ~70 KB/token) + fits the 16 GB pin, and the 262k entry it fears cannot exist (over + slot is a 400). Probe-proven 2026-10-06: the warm turn hit with + cache_n=16,384. 2. `curl -s http://192.168.0.143:7770/health` -> 200. 3. Real completion with `reasoning_effort` + temp/top_p from the wire recipe; `reasoning_content` populated; decode rate from the timings. @@ -120,6 +140,38 @@ bearing numbers: incl. a tool call (V1TOOLS). 6. Prompt > 200,700 tokens returns the 400 naming the limit (contract test). +## Tuning notes (2026-10-06) + +Measured on real agent traffic (serve_api lines) the day after cutover: + +- **Decode is depth-bound, and the KV path is the reason.** Shallow + prompts (< 1k tokens) decode 30-50 t/s at commit 4-6.3/round; 46k runs + 5.5, 63-67k runs 4.1-6.5, 79-81k runs 3.3-3.8. The round-time slope is + ~8 us/token of context: the KV pool reads at ~8 GB/s effective while + the weights stream ~180-240 GB/s (the engine's README puts batch-1 + weight streaming at the 240 GB/s hardware wall). Commit also falls + from 4-6 to 2.5-3 accepted/round as the conversation grows. +- **No supported flag touches decode.** `docs/FLAGS.md`: HALOGEN_W4A4 / + W4A4_EXCL are prefill-only ("decode is unaffected either way"); the + undocumented kernel levers are declared unsupported and their defaults + are the measured winners. 0.1.4 is the latest release (CHANGELOG + head; kernels unchanged since 0.1.0). The publisher's tg bench never + measured depth — its tg rows run over the ten SHORT eval prompt + shapes — so the README's "decode barely degrades at depth" claim was + never exercised at 46-80k. +- **The lever that existed was the prompt cache, and it was OFF** (see + the shape bullet). The pin turns the 85-180 s re-prefill per deep + turn into a resume; decode itself is expected UNCHANGED by the pin — + the cache resumes prefill, it does not touch the round loop. + Probe-proven (two synthetic 17.3k turns, 2026-10-06): cold prefill + 27.0 s (640 t/s, cache_n 0) -> warm turn cache_n 16,384, prompt_n 946 + (the extension + one 2048-aligned chunk), prefill 1.7 s — 16x on the + pair, answer correct warm. +- **Everything else stays put.** dflash2 over mtp (publisher 31.71 vs + 26.88 mean; at depth both pay the same KV term and dflash2's higher + commit wins), KV_SLOTS=1 (batch XOR speculation), SLOT_CTX=200700 + (Frank's window), W4A4 shipped (+9 % prefill, decode unharmed). + ## Rollback Revert the frank-large swap commit and redeploy `dankclaw --service frank-large`: diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 52c60f6..428b0b5 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -81,6 +81,14 @@ frank-small, port 7771] No server-side sampler — the thinking recipe rides the wire in models.yml. Rollback: revert the swap commit; the ROCmFPX payload + fork build + `build-rocmfpx.sh` pin all remain on disk + Tuned 2026-10-06: `HALOGEN_CACHE_MB=16384` pins the prompt cache — + auto was OFF at the 36.9 GB two-entry floor, so every deep request + re-prefilled its 79-81k context (85-180 s each); a 17.3k cold prefill + of 27.0 s now resumes in 1.7 s on the warm turn (cache_n 16,384). The + serve script captures the engine ledger on disk (foreground `podman + run`; the `-d` era wrote only the container ID). Decode-at-depth is + structural: 3.3-3.8 t/s at 80k vs 30-50 shallow; no supported flag + touches it. - [[frank-small-llama-swift15-iq3s-mtp-131072]] — **ACTIVE since 2026-10-05 (evening); 2 × 131,072 no-spec since 2026-10-06** (root systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving ukisai/Swift-1.5-