frank-large: document the ik_llama.cpp deployment and the decode performance campaign
- Runbook with the measured envelope (prefill ~192 t/s at the VNNI ceiling; decode ~3.8-4.3 t/s gated by 100 backend splits/token, GPU 95% idle), the root-cause evidence chain, tuning levers, and co-tenancy traps - AGENTS.md: fleet topology row + Active Services section for the ik deployment; halogen demoted to rollback; bench policy unit type fixed - LLM + root indexes relinked (frank-large-ik-uncensored ACTIVE, halogen-flash-server retired)
This commit is contained in:
1 parent
0c1e10a5ef
commit
795b0b3a5e
4 files changed
+164
-43
No files matched your search
@@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-medium` | charlie12345/ROCmFPX llama.cpp (system unit) | kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF, 2x Radeon AI PRO R9700 layer split, 131072 ctx / 1 slot, MTP draft on ROCm1 | 7770 |
|
||||
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai/halogen-qwen3.8-flash-next w4b .hgn, 262144 ctx / 3 slots, vision tower on | 7770 |
|
||||
| Dankclaw | `frank-large` | ik_llama.cpp master d85a1d7 (root systemd unit) | mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF, RTX 3090 (dense/attn/KV) + LPDDR (experts/PLE) hybrid, 262144 ctx / 1 slot, vision via mmproj | 7770 |
|
||||
|
||||
Retired endpoints (`frank-npu`, `frank-small`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -192,8 +192,8 @@ test process cannot share the single GPU anyway; validate against the service
|
||||
itself.
|
||||
|
||||
1. Stop and mask (self-healing would restart it):
|
||||
- Dankclaw `frank-large` (rootless — no sudo):
|
||||
`systemctl --user stop frank-large && systemctl --user mask frank-large`
|
||||
- Dankclaw `frank-large` (root unit since the 2026-09-15 ik cutover):
|
||||
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
|
||||
- Dankserver `frank-medium` (system unit):
|
||||
`sudo systemctl stop frank-medium && sudo systemctl mask frank-medium`
|
||||
2. Bench.
|
||||
@@ -224,38 +224,29 @@ input anywhere in the fleet** (left with `frank-npu`).
|
||||
|
||||
## Active Services
|
||||
|
||||
### frank-large — halogen-flash-server (Dankclaw)
|
||||
### frank-large — ik_llama.cpp (Dankclaw)
|
||||
|
||||
`ghcr.io/peonist-ai/halogen-flash-server:0.5.8` as a rootless Quadlet
|
||||
(`dankclaw/frank-large/quadlet/frank-large.container` → `frank-large.service`
|
||||
in the user manager), serving `peonist-ai/halogen-qwen3.8-flash-next`.
|
||||
One container, two halves: engine on container loopback :8730 (unauthenticated,
|
||||
NEVER published) and OpenAI front-end on :8731 published as host **7770**.
|
||||
ik_llama.cpp master `d85a1d7` as a **root systemd unit** (repo
|
||||
`scripts/frank-large.service` symlinked into `/etc/systemd/system/`;
|
||||
`Restart=always`), serving `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`
|
||||
(111 GiB GGUF + mmproj-f16) on port 7770 with 262144 ctx / 1 slot, greedy
|
||||
default, reasoning on (deepseek), vision via mmproj. Topology: the RTX 3090
|
||||
(PCIe x4 in the M.2 dock) carries dense/attention/KV; ALL MoE expert tensors
|
||||
and the ~33 GiB PLE table stay in LPDDR via CPU buffers. Deployed flags in
|
||||
`scripts/frank-large.env` / `scripts/serve-frank-large.sh`; the serve script
|
||||
refuses to start without `GGML_CUDA_NO_PINNED_WEIGHTS=1` (the pinned-host
|
||||
buffer incident starved co-tenants into unreachability until a power cycle).
|
||||
|
||||
Deployed flags: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=401408`,
|
||||
`HALOGEN_KV_SLOTS=3`, `HALOGEN_MAX_TOK=32768`, `HALOGEN_HOST_RESERVE_GIB=15`,
|
||||
`HALOGEN_VISION_TOWER=1`. Everything else upstream default. Sizing history
|
||||
and the flag-by-flag rationale live in the runbook — do not re-derive here.
|
||||
Measured envelope (2026-09-15/16 campaign): prefill ~192 tok/s (1.8k-32k
|
||||
prompts, CPU VNNI compute ceiling), decode ~3.8-4.3 tok/s — gated by ~100
|
||||
CPU↔GPU backend splits per token (GPU 95% idle, launch/sync-bound; thread
|
||||
count irrelevant). The halogen-era 30 tok/s decode bar is out of reach on
|
||||
this model+hardware; the runbook carries the full evidence chain, the
|
||||
mainline-llama.cpp comparison (1.0 tok/s — ruled out), and the tuning levers.
|
||||
|
||||
Invariants that survive any config change:
|
||||
|
||||
- **Pool and reserve are a matched pair.** Budget = `121.2 host − 67.7
|
||||
weights − reserve`; that one budget carries KV pool + prefill arena + the
|
||||
1.97 GiB vision tower. The reserve is **not an allocation** — raising it
|
||||
cannot free host RAM, it can only shrink the pool. To free host RAM, cut
|
||||
`HALOGEN_KV_POOL_POSITIONS` or `HALOGEN_MAX_TOK`.
|
||||
- **`free` lies on this host.** The ~68 GiB pinned weights count as
|
||||
reclaimable page cache, so `free`/`MemAvailable`/Grafana overstate free
|
||||
memory by the model size. Believe the server's startup ledger. A co-tenant
|
||||
running the box dry looks like minutes of full-CPU stalls with no disk
|
||||
activity — not a crash.
|
||||
- **Client-visible contract:** no server-side sampler defaults (no
|
||||
`temperature` ⇒ greedy); per-request budget defaults to 8192 and **includes
|
||||
thinking** (running out mid-thought ⇒ `finish_reason: "length"`, empty
|
||||
content); cap 65536; a request reserving prompt + `max_tokens` beyond the
|
||||
pool queues, then fails `503 engine_busy` after the 3600 s timeout.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md`.
|
||||
Rollback: [[halogen-flash-server]] (quadlet archived in
|
||||
`~/.AGENTTRASH/20260915-ik-swap/` on dankclaw).
|
||||
|
||||
### frank-medium — ROCmFPX llama.cpp (Dankserver)
|
||||
|
||||
|
||||
Reference in new issue
Block a user