dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: the vault and AGENTS.md describe two Dankclaw endpoints

Browse Source 
halogen-flash-server is ACTIVE again with its 2026-09-16 measured ledger (budget
33.5 GiB against a need of 27.2 — no clamp — and 14.7 GiB left for everything
else), ik_llama.cpp is retired with its measurements kept as the record of why,
and the new frank-small runbook covers the -mtp file, the 64 KiB/token KV
arithmetic, the MTP contract, and the reasoning_effort trap that cost me every
subagent on the endpoint.

AGENTS.md: topology, Active Services, runbook index, and three new llama.cpp
lessons (MTP is a property of the file; --jinja is what `tools` needs; a model's
template can reject the harness's thinking level). Three pre-existing defects
found while editing it: a truncated sentence mid-file, a harness path that does
not exist on either server, and the benchmarking stop/mask recipe still calling
frank-large a root unit.

Truthfulness notes worth keeping: the co-tenant stall NOTE prints again, so the
claim that it "no longer prints" is gone and the three observed block counts are
recorded instead of a guessed threshold; and the Observability page says out
loud why frank-large has no scrape job rather than leaving a silent absence.
This commit is contained in:
Frank committed 2026-09-16 14:52:51 -07:00
1 parent 40850bbc8f
commit 49e8dfff66
8 files changed
+565 -153
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+102 -41
View File
@@ -12,8 +12,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
Calibre-Web, Jellyfin, Plex) plus the `frank-medium` LLM service.
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
Alloy, Pi-hole) plus the `frank-large` LLM service. Caddy on Dankclaw
reverse-proxies both hosts.
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen-flash-server on
the gfx1151 iGPU) and `frank-small` (llama.cpp on the RTX 3090). Caddy on
Dankclaw reverse-proxies both hosts.
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
(Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
its podman machine; all egress rides the Gluetun WireGuard netns. Deployed
@@ -24,9 +25,10 @@ merge absorbed the former `vllmScripts` repo, history preserved).
| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-medium` | vllm-radiance 0.9.3 (ggz14/radiance-vllm-mxfp4 patches), 2x single-GPU rootless Quadlets behind an adaptive-routing proxy | just1moremodel Qwen3.8-27B-Uncensored-MXFP4-awq + Qwen3.8-27B-DFlash2-FP8 drafter, 2x Radeon AI PRO R9700 (one card per replica, TP=1), 114688 ctx / 4 seqs each | 7770 |
| Dankclaw | `frank-large` | ik_llama.cpp master d85a1d7 (root systemd unit) | mradermacher Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M GGUF, RTX 3090 (dense/attn/KV) + LPDDR (experts/PLE) hybrid, 262144 ctx / 1 slot, vision via mmproj | 7770 |
| Dankclaw | `frank-large` | halogen-flash-server 0.5.8 (rootless Quadlet) | peonist-ai Qwen3.8-Flash-Next w4b `.hgn` on the gfx1151 iGPU (Strix Halo LPDDR), 200192 ctx per conversation over one 400384-position KV pool / 2 slots, text only | 7770 |
| Dankclaw | `frank-small` | mainline llama.cpp master 37b53fd (root systemd unit) | ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`-mtp` + BF16 mmproj) on the RTX 3090, 131072 ctx / 1 slot, MTP speculative decoding, the fleet's vision endpoint | 7771 |
Retired endpoints (`frank-npu`, `frank-small`, and every prior runtime) are in
Retired endpoints (`frank-npu`, and every prior runtime) are in
the [runbook index](#runbook-index). The full swap history lives in
`serverAppConfigs-docs/LLM/index.md` — do not grow it here.
@@ -66,9 +68,10 @@ All edits flow through git. Never edit files on a remote machine directly.
The planner maps changed paths to services; the remote runner restarts only
the selected service and records its revision outside the checkout.
Documentation-only changes are a no-op. LLM services are selected only by
their own unit/env/serve files or the shared chat template — unrelated
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
runs on a host without passwordless sudo; the value never enters the repo.
their own files — frank-large by `dankclaw/frank-large/quadlet/`, frank-small
by its unit/env/serve trio in `scripts/` — so unrelated container changes
cannot interrupt inference. Set `FRANK_SUDO_PW` for headless runs on a host
without passwordless sudo; the value never enters the repo.
### Deployment completeness (MANDATORY)
@@ -194,8 +197,9 @@ archive.
**STOP the production service BEFORE benchmarking** — these servers serve
real users, and a bench beside a live service starves the GPU and corrupts
results. Use the runtime's own harness: `llama-bench` (llama.cpp), `vllm
bench` (vLLM), llama-bench for ik_llama.cpp (the halogen `bench`/`sweep` modes
belong to a retired runtime). Rolling your own bench requires asking first.
bench` (vLLM), and halogen-flash-server's own `bench`/`sweep` modes for
frank-large, which is the active runtime there again. Rolling your own bench
requires asking first.
Scratch ports are not used on LLM hosts — a
test process cannot share the single GPU anyway; validate against the service
itself.
@@ -212,8 +216,13 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
merely wedged kernel usually still answers ARP.
1. Stop and mask (self-healing would restart it):
- Dankclaw `frank-large` (root unit since the 2026-09-15 ik cutover):
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
- Dankclaw `frank-large` (rootless Quadlet again since 2026-09-16):
`systemctl --user stop frank-large`, then move its
`~/.config/containers/systemd/frank-large.container` aside and
`systemctl --user daemon-reload` — the quadlet generator re-creates that
unit from the file on every reload, so the source is the lever, not `mask`.
- Dankclaw `frank-small` (root unit, llama.cpp on the 3090):
`sudo systemctl stop frank-small && sudo systemctl mask frank-small`
- Dankserver `frank-medium` (system unit):
`sudo systemctl stop frank-medium && sudo systemctl mask frank-medium`
2. Bench.
@@ -225,47 +234,71 @@ deletes the unit file from `/etc/systemd/system/`, not just the symlink.
## Model Config (harness)
The harness reaches the fleet through `/home/dank/.omp/agent/models.yml`
(outside the repo). Each provider is one host endpoint; each model `id` is
the served alias: `frank-medium` → `192.168.0.235:7770`, `frank-large` →
`192.168.0.143:7770`. Per-role routing lives in
`/home/dank/.omp/agent/config.yml` under `modelRoles`.
The harness reaches the fleet through `.omp/agent/models.yml` in the
operator's home — `/home/blank/.omp/agent/models.yml` on the dev box, which is
where the harness runs; there is no copy on the servers. Each provider is one
HOST:PORT; each model `id` is the served alias: `frank-medium` →
`192.168.0.235:7770`, `frank-large` → `192.168.0.143:7770`, `frank-small` →
`192.168.0.143:7771`. Per-role routing lives beside it in `config.yml` under
`modelRoles`.
Resolution rules: the default model resolves as a `defaultProvider` +
`defaultModel` pair (the provider must serve the model); a subagent override
uses a bare model `id`, which resolves only when unique across providers.
uses a bare model `id`, which resolves only when unique across providers. Two
ports on one host need two providers — the file cannot express "same host,
different port".
When renaming a service, retiring an endpoint, or moving a model: update the
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
a role pointing at a removed provider fails at call time. `frank-large`
declares `input: [text, image]` (its mmproj vision tower). There is **no audio
a role pointing at a provider the file does not define fails at call time. As
of 2026-09-16 the image path is `frank-small` (`input: [text, image]`), and
`frank-large` is text-only because its halogen deployment loads no vision
tower. There is **no audio** input anywhere in the fleet: that capability left
with the FLM NPU endpoint on 2026-09-11.
## Active Services
### frank-large — ik_llama.cpp (Dankclaw)
### frank-large — halogen-flash-server (Dankclaw, ACTIVE)
ik_llama.cpp master `d85a1d7` as a **root systemd unit** (repo
`scripts/frank-large.service` symlinked into `/etc/systemd/system/`;
`Restart=always`), serving `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`
(111 GiB GGUF + mmproj-f16) on port 7770 with 262144 ctx / 1 slot, greedy
default, reasoning on (deepseek), vision via mmproj. Topology: the RTX 3090
(PCIe x4 in the M.2 dock) carries dense/attention/KV; ALL MoE expert tensors
and the ~33 GiB PLE table stay in LPDDR via CPU buffers. Deployed flags in
`scripts/frank-large.env` / `scripts/serve-frank-large.sh`; the serve script
refuses to start without `GGML_CUDA_NO_PINNED_WEIGHTS=1` (the pinned-host
buffer incident starved co-tenants into unreachability until a power cycle).
halogen-flash-server 0.5.8 as a **rootless Quadlet again since 2026-09-16**
(`dankclaw/frank-large/quadlet/frank-large.container`), serving
`peonist-ai/halogen-qwen3.8-flash-next` (115.55 GiB w4b `.hgn` checkpoint + the
2.31 GiB quality overlay) on port 7770 from the gfx1151 iGPU. Two conversations
of 200,192 tokens resident (pool 400,384, 2 slots), greedy default, reasoning
extracted to `reasoning_content`, and **no vision tower** — that moved to
frank-small. It replaced ik_llama.cpp, which had replaced it for two days.
Measured envelope (2026-09-15/16 campaign): prefill ~192 tok/s (1.8k-32k
prompts, CPU VNNI compute ceiling), decode ~3.8-4.3 tok/s — gated by ~100
CPU↔GPU backend splits per token (GPU 95% idle, launch/sync-bound; thread
count irrelevant). The halogen-era 30 tok/s decode bar is out of reach on
this model+hardware; the runbook carries the full evidence chain, the
mainline-llama.cpp comparison (1.0 tok/s — ruled out), and the tuning levers.
Measured boot ledger (2026-09-16, `HALOGEN_VERBOSE=1` ships in the unit):
device budget `121 − 67.7 − 20 = 33.5 GiB` against a need of `~27.2 GiB`, so
the configured pool was **not clamped**; 68.0 GiB of weights locked + 11.0 GiB
of KV pool + 16.5 GiB of working memory = 95.4 GiB in all, leaving 14.7 GiB
(475 contiguous 2 MiB blocks) for everything else: frank-small on the 3090 and
~28 co-tenant containers. The engine printed its co-tenant stall NOTE and
logged 4,350 compaction stalls (285 failed) while reserving the pool, so read
that as "tight and fragmented", not "out of memory".
Runbook: `serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md`.
Rollback: [[halogen-flash-server]] (quadlet archived in
`~/.AGENTTRASH/20260915-ik-swap/` on dankclaw).
Runbook: `serverAppConfigs-docs/LLM/halogen-flash-server.md`.
Rollback: [[frank-large-ik-uncensored]].
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
Mainline llama.cpp master `37b53fd` as a **root systemd unit** (repo
`scripts/frank-small.service`, `relink_unit`ed into `/etc/systemd/system/`),
serving `ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF` at IQ3_S — the `-mtp` build, so
the model's own Multi-Token Prediction head drives speculative decoding — with
the BF16 mmproj, on port 7771, 131,072 ctx / 1 slot, f16 KV, greedy default.
Deployed flags are in `scripts/frank-small.env`; the serve script refuses to
start if the weights, the projector or the binary are missing.
Verified at deploy (2026-09-16): `/v1/models` reports `frank-small`; a chat
completion returns content plus `reasoning_content`; a tool call returns
`tool_calls`; the slot timing lines show `draft acceptance = 0.66–0.94, mean
len ≈ 3` (MTP engaged); an image request is answered from the projector while
the halogen side refuses images with a named reason; 22,362 MiB of the card's
24,576 MiB is in use; `n_slots = 1, n_ctx_slot = 131072`.
Runbook: `serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md`.
### frank-medium — vllm-radiance MXFP4 (Dankserver, ACTIVE)
@@ -400,6 +433,33 @@ linked runbooks.
- **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch;
stock binaries fail the draft load with a tensor-shape mismatch.
**llama.cpp serving**
- **MTP is a property of the FILE, not of the flags.** The `-mtp` GGUF build
carries the head as `blk.<n>.nextn.*` (`qwen35.nextn_predict_layers = 1`);
`--spec-type draft-mtp` then sets `load_mtp`, and the draft context runs on
the MAIN model's weights (`shares_model`), so there is no second copy of a
12 GiB model in VRAM and no `-md` argument. The plain build has no MTP
tensors at all, and pointing `--model-draft` at the file would load it twice.
- **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
`500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
then served frank-large: every tool-using agent on that endpoint was broken
and the failure read like a model fault. Mainline llama.cpp 0.4.1 defaults
`--jinja` on and the serve script pins it; halogen also accepts `tools` and
returns `tool_calls`. Check tool support before routing a role at an endpoint.
- **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same
option and the same ~12x prefill collapse described in the ROCm note above.
f16 KV needs nothing, which is why frank-small runs f16.
- **A model's own template can REJECT the harness's thinking level.** Qwen3.8's
template accepts only `xhigh`, `medium` and `low` for `reasoning_effort` and
raises on anything else — HTTP 500, not a clamp — so the harness's levels
must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`).
A role whose level is unmapped fails on its FIRST request, and the failure
names Jinja, not the model: every subagent on `frank-small` died with
`Unexpected reasoning effort high` until `high: xhigh` was added on
2026-09-16. Halogen accepts the wider set, so the two entries legitimately
differ.
**Ops**
- Never `pkill -f` a pattern that appears in your own command line — it kills
@@ -430,7 +490,8 @@ Active:
| Service | Runbook |
|---------|---------|
| frank-large (ik_llama.cpp, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md` |
| frank-large (halogen-flash-server, Dankclaw) | `serverAppConfigs-docs/LLM/halogen-flash-server.md` |
| frank-small (llama.cpp Qwen3.8-27B GSQ-RCO, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-qwen38-gsq-rco.md` |
| frank-medium (vllm-radiance MXFP4, Dankserver) | `serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
@@ -440,7 +501,7 @@ Retired (rollback targets and history):
| Runtime | Host | Window | Runbook |
|---------|------|--------|---------|
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
| halogen-flash-server (`frank-large`) | Dankclaw | → 2026-09-15 | `halogen-flash-server.md` |
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |