dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: frank-medium runbook, fleet topology and client cutover

Browse Source 
New runbook page serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md records
the hardware swap, the self-contained gfx1201 build, the measured layer-vs-
tensor decision (llama-bench multi-device syntax gotcha included), the
tensor-mode draft abort, served-path numbers, and the post-cleanup host ROCm
posture. AGENTS.md, README, docs indexes, Dankserver System Specs,
deployment/Hermes/OpenWebUI/Plex/Stack pages follow the rename;
freetoken-frank-small is marked RETIRED and kingjones-q8agent SUPERSEDED.
Work log entry added.
This commit is contained in:
Frank committed 2026-09-15 19:27:04 -07:00
1 parent d53962fec5
commit d4024a9180
15 files changed
+426 -80
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+45 -14
View File
@@ -7,7 +7,7 @@ and the FrankBook-Pro Mac mini. It tracks the Podman container stacks and
the LLM inference fleet. One repo, one checkout per host, one deploy script.
- **Dankserver** (192.168.0.235) — media/library stack (RomM, Copyparty,
Calibre-Web, Jellyfin, Plex) as bare containers, plus the `frank-small`
Calibre-Web, Jellyfin, Plex) as bare containers, plus the `frank-medium`
LLM service.
- **Dankclaw** (192.168.0.143) — AI/search/development/observability stack
(SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana,
@@ -27,11 +27,11 @@ prefix, reorganized in follow-up commits). The retired target checkouts
archived. All services now run from `/home/dank/serverAppConfigs` on each
host.
### LLM fleet topology (2026-09-11)
### LLM fleet topology (2026-09-15)
| Target | Runtime | Service | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | FreeToken v0.1.2 | frank-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 |
| Dankserver | charlie12345/ROCmFPX llama.cpp | frank-medium | kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF (Q8_0_ROCMFPX_AGENT ftype 115, 26.28 GiB, 8.39 bpw), 2x Radeon AI PRO R9700 layer split, 131072 ctx / 1 slot, f16 KV, MTP draft head on ROCm1, reasoning on | 7770 |
|Dankclaw|halogen-flash-server 0.5.8 (rootless Quadlet container)|frank-large|peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 3 slots, pool 401408 / max_tok 32768 / reserve 15, vision tower on, greedy for silent clients)|7770|
| Dankclaw | *(retired 2026-09-11)* FastFlowLM v1.0.1 on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio). Retired with the `amd_iommu=off` decision: the driver's SVA/PASID bind needs an enabled IOMMU | 7771 |
@@ -58,7 +58,12 @@ llama.cpp deployment gave way to halogen-flash-server 0.5.8 serving
peonist-ai/halogen-qwen3.8-flash-next, now a rootless Quadlet container, and
the kernel cmdline moved from `iommu=pt` to `amd_iommu=off` for 13–16 % of
prefill — which retired `frank-npu` for good (see the Runtime &
Troubleshooting section and its runbook). Runbooks:
Troubleshooting section and its runbook). On 2026-09-15 the RTX 3090 left
Dankserver for two Radeon AI PRO R9700 cards (gfx1201): `frank-small`
(FreeToken/CUDA) retired and `frank-medium` (charlie12345/ROCmFPX llama.cpp,
kingjones Q8_0_ROCMFPX-AGENT) took the :7770 endpoint — layer split, not
tensor, because the RCCL-less build's butterfly allreduce collapses prefill
7x (see the runbook for the measured table). Runbooks:
[serverAppConfigs-docs/LLM/index.md](serverAppConfigs-docs/LLM/index.md).
Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`.
@@ -127,7 +132,7 @@ selected service and records its successful revision outside the checkout.
./scripts/deploy.sh dankclaw
./scripts/deploy.sh dankserver
./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-small
./scripts/deploy.sh dankserver --service frank-medium
./scripts/deploy.sh dankclaw --dry-run
```
@@ -306,11 +311,11 @@ DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE
1. Stop the service before any benchmark:
`systemctl --user stop frank-large` (Dankclaw — a **rootless** Quadlet
service, so drop the `sudo`) or `sudo systemctl stop frank-small`
service, so drop the `sudo`) or `sudo systemctl stop frank-medium`
(Dankserver — still a system unit).
Then mask it to prevent the self-healing unit from restarting:
`systemctl --user mask frank-large` (Dankclaw) or
`sudo systemctl mask frank-small` (Dankserver).
`sudo systemctl mask frank-medium` (Dankserver).
**CRITICAL WARNING:** Do NOT use `systemctl disable` on the **system** units.
On these hosts, `systemctl disable` doesn't just remove symlinks — it **deletes
the unit file itself** from `/etc/systemd/system/`. You'd have to manually
@@ -326,8 +331,8 @@ DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE
against the service itself.
3. Restart the service when the bench completes: unmask, then start:
`systemctl --user unmask frank-large && systemctl --user start frank-large`
(Dankclaw) or `sudo systemctl unmask frank-small && sudo systemctl start
frank-small` (Dankserver).
(Dankclaw) or `sudo systemctl unmask frank-medium && sudo systemctl start
frank-medium` (Dankserver).
4. Verify health before reporting:
`curl http://192.168.0.235:7770/health` or
`curl http://192.168.0.143:7770/health`.
@@ -342,7 +347,7 @@ served alias:
| alias | endpoint |
|---|---|
| `frank-small` | Dankserver `192.168.0.235:7770` |
| `frank-medium` | Dankserver `192.168.0.235:7770` |
| `frank-large` | Dankclaw `192.168.0.143:7770` |
| `gemma4-it:e4b` | *(removed 2026-09-11 with `frank-npu`)* |
@@ -375,8 +380,8 @@ driver cannot bind SVA/PASID, so every open of `/dev/accel/accel0` fails with
from both the harness and Hermes. **Audio input left the fleet with it and
has no replacement**; image input survives on `frank-large` through halogen's
vision tower. Route map: Hermes `vision` → `frank-large`, Hermes
`title_generation` → `frank-small`, harness `vision` → `frank-large`, harness
`tiny` → `frank-small`, harness `task` → `frank-large` at medium effort.
`title_generation` → `frank-medium`, harness `vision` → `frank-large`, harness
`tiny` → `frank-medium`, harness `task` → `frank-large` at medium effort.
The retired runbook keeps the full record, including the original IOMMU/SVA
root cause: [serverAppConfigs-docs/LLM/frank-npu.md](serverAppConfigs-docs/LLM/frank-npu.md).
@@ -491,7 +496,33 @@ load on any current loader (require full per-layer
3x). Binary and payload retained. Runbook:
[serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md](serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md).
### Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on FreeToken (Dankserver, ACTIVE)
### kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT on ROCmFPX llama.cpp (Dankserver, ACTIVE since 2026-09-15)
The medium service serves `kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF`
(public, 26.28 GiB, 8.39 bpw, `Q8_0_ROCMFPX_AGENT` ftype 115 — stock
llama.cpp cannot load it) from
`/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` with
charlie12345/ROCmFPX llama.cpp `c49ebdb`, built for **gfx1201** with the
fork's self-contained ROCm 7.14 toolchain
(`env JOBS=8 ./scripts/build-rocmfp4-rocm714-local.sh` in
`/home/dank/GitPackages/charlie-ROCmFPX`; the binary bundles its ROCm runtime
via RPATH, so the host's ROCm packages never participate). `--split-mode
layer` across both R9700s — tensor parallel loses 7x prefill on this
RCCL-less build — 131072 context / 1 slot, f16 KV, flash-attn on, MTP draft
head (n_max 6, p_min 0.6, `--spec-draft-device ROCm1`), reasoning on
(deepseek format), sampler defaults from the checkpoint's instruct table.
Measured 2026-09-15: llama-bench pp4096 1991 t/s / tg128 18.8 t/s (tensor:
271.8 / 26.26; single card: 1160.9 / 19.42); served with MTP 35.7–46.2 tok/s
and 60–69 % draft acceptance; prefix caching verified working with the draft
head. Env `scripts/frank-medium.env`, serve `scripts/serve-frank-medium.sh`.
Runbook:
[serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md](serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md).
### Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on FreeToken (Dankserver, RETIRED 2026-09-15)
Retired with the RTX 3090: FreeToken/CUDA cannot run on the replacement
R9700 cards, and the FreeToken-era unit was archived to
`~/.AGENTTRASH/20260915-frank-small-retire/host-units/`. Historical record:
The small service serves `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8`
(public, ~35 GB, 26 safetensors shards) from
@@ -623,7 +654,7 @@ graph TD
flags. A different outcome isolates the env file as the cause.
3. Run gdb for the backtrace with this command:
```
gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-small.sh
gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-medium.sh
```
`llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost.
Trust gdb, not the log.