frank-large: document the ik_llama.cpp deployment and the decode performance campaign

- Runbook with the measured envelope (prefill ~192 t/s at the VNNI ceiling;
  decode ~3.8-4.3 t/s gated by 100 backend splits/token, GPU 95% idle), the
  root-cause evidence chain, tuning levers, and co-tenancy traps
- AGENTS.md: fleet topology row + Active Services section for the ik
  deployment; halogen demoted to rollback; bench policy unit type fixed
- LLM + root indexes relinked (frank-large-ik-uncensored ACTIVE,
  halogen-flash-server retired)
This commit is contained in:
Frank committed 2026-09-16 02:42:41 -07:00
1 parent 0c1e10a5ef
commit 795b0b3a5e
4 files changed
+164 -43

No files matched your search

+23 -32
View File
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-medium` | charlie12345/ROCmFPX llama.cpp (system unit) | kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF, 2x Radeon AI PRO R9700 layer split, 131072 ctx / 1 slot, MTP draft on ROCm1 | 7770 |
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai/halogen-qwen3.8-flash-next w4b .hgn, 262144 ctx / 3 slots, vision tower on | 7770 |
| Dankclaw | `frank-large` | ik_llama.cpp master d85a1d7 (root systemd unit) | mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF, RTX 3090 (dense/attn/KV) + LPDDR (experts/PLE) hybrid, 262144 ctx / 1 slot, vision via mmproj | 7770 |
Retired endpoints (`frank-npu`, `frank-small`, and every prior runtime) are in
the [runbook index](#runbook-index). The full swap history lives in
@@ -192,8 +192,8 @@ test process cannot share the single GPU anyway; validate against the service
itself.
1. Stop and mask (self-healing would restart it):
- Dankclaw `frank-large` (rootless — no sudo):
`systemctl --user stop frank-large && systemctl --user mask frank-large`
- Dankclaw `frank-large` (root unit since the 2026-09-15 ik cutover):
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
- Dankserver `frank-medium` (system unit):
`sudo systemctl stop frank-medium && sudo systemctl mask frank-medium`
2. Bench.
@@ -224,38 +224,29 @@ input anywhere in the fleet** (left with `frank-npu`).
## Active Services
### frank-large — halogen-flash-server (Dankclaw)
### frank-large — ik_llama.cpp (Dankclaw)
`ghcr.io/peonist-ai/halogen-flash-server:0.5.8` as a rootless Quadlet
(`dankclaw/frank-large/quadlet/frank-large.container` → `frank-large.service`
in the user manager), serving `peonist-ai/halogen-qwen3.8-flash-next`.
One container, two halves: engine on container loopback :8730 (unauthenticated,
NEVER published) and OpenAI front-end on :8731 published as host **7770**.
ik_llama.cpp master `d85a1d7` as a **root systemd unit** (repo
`scripts/frank-large.service` symlinked into `/etc/systemd/system/`;
`Restart=always`), serving `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`
(111 GiB GGUF + mmproj-f16) on port 7770 with 262144 ctx / 1 slot, greedy
default, reasoning on (deepseek), vision via mmproj. Topology: the RTX 3090
(PCIe x4 in the M.2 dock) carries dense/attention/KV; ALL MoE expert tensors
and the ~33 GiB PLE table stay in LPDDR via CPU buffers. Deployed flags in
`scripts/frank-large.env` / `scripts/serve-frank-large.sh`; the serve script
refuses to start without `GGML_CUDA_NO_PINNED_WEIGHTS=1` (the pinned-host
buffer incident starved co-tenants into unreachability until a power cycle).
Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=401408`,
`HALOGEN_KV_SLOTS=3`, `HALOGEN_MAX_TOK=32768`, `HALOGEN_HOST_RESERVE_GIB=15`,
`HALOGEN_VISION_TOWER=1`. Everything else upstream default. Sizing history
and the flag-by-flag rationale live in the runbook — do not re-derive here.
Measured envelope (2026-09-15/16 campaign): prefill ~192 tok/s (1.8k-32k
prompts, CPU VNNI compute ceiling), decode ~3.8-4.3 tok/s — gated by ~100
CPU↔GPU backend splits per token (GPU 95% idle, launch/sync-bound; thread
count irrelevant). The halogen-era 30 tok/s decode bar is out of reach on
this model+hardware; the runbook carries the full evidence chain, the
mainline-llama.cpp comparison (1.0 tok/s — ruled out), and the tuning levers.
Invariants that survive any config change:
- **Pool and reserve are a matched pair.** Budget = `121.2 host − 67.7
weights − reserve`; that one budget carries KV pool + prefill arena + the
1.97 GiB vision tower. The reserve is **not an allocation** — raising it
cannot free host RAM, it can only shrink the pool. To free host RAM, cut
`HALOGEN_KV_POOL_POSITIONS` or `HALOGEN_MAX_TOK`.
- **`free` lies on this host.** The ~68 GiB pinned weights count as
reclaimable page cache, so `free`/`MemAvailable`/Grafana overstate free
memory by the model size. Believe the server's startup ledger. A co-tenant
running the box dry looks like minutes of full-CPU stalls with no disk
activity — not a crash.
- **Client-visible contract:** no server-side sampler defaults (no
`temperature` ⇒ greedy); per-request budget defaults to 8192 and **includes
thinking** (running out mid-thought ⇒ `finish_reason: "length"`, empty
content); cap 65536; a request reserving prompt + `max_tokens` beyond the
pool queues, then fails `503 engine_busy` after the 3600 s timeout.
Runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`.
Runbook: `serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md`.
Rollback: [[halogen-flash-server]] (quadlet archived in
`~/.AGENTTRASH/20260915-ik-swap/` on dankclaw).
### frank-medium — ROCmFPX llama.cpp (Dankserver)