serverAppConfigs/AGENTS.md

AGENTS.md — serverAppConfigs Agent Rules

Overview

This repository is the single source of truth for the two homelab servers and the FrankBook-Pro Mac mini. It tracks the Podman container stacks and the LLM inference fleet. One repo, one checkout per host, one deploy script. All services run from /home/dank/serverAppConfigs on each host (the 2026-08-22 merge absorbed the former vllmScripts repo, history preserved).

• Dankserver (192.168.0.235) — media/library stack (RomM, Copyparty, Calibre-Web, Jellyfin, Plex) plus the frank-archon LLM service.
• Dankclaw (192.168.0.143) — AI/search/dev/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki, Alloy, Pi-hole) plus TWO LLM services: frank-dragoon (flash-server container on the gfx1151 iGPU) and frank-zealot (llama.cpp CUDA on the RTX 3090). Caddy on Dankclaw reverse-proxies both hosts.
• FrankBook-Pro (192.168.0.112) — macOS mini running the *arr stack (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside its podman machine; all egress rides the Gluetun WireGuard netns. Deployed via the frankbook-pro handler.

Fleet topology (ACTIVE)

Target	Service	Runtime	Model	Port
Dankserver	frank-archon	radiance (StillDeadcode, pin 1.0.13 digest-pinned stilldeadcode/radiance@sha256:4c3f484564…, plain podman run, root systemd unit)	Publisher payload (swap 2026-10-15, MODEL_FILE-only): StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe → qwen3.8-next-flash-fp8-iq4r-moe.rad (121,969,901,568 B = 116,322 MiB, Hub-LFS sha256 0af5e962… verified at fetch by scripts/download-frank-archon-iq4r.sh) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as 4-bit codes into a 16-level non-uniform codebook (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), 10 whole bf16-protected experts/layer, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), MTP head + bf16 vision tower in the container; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. Base-alignment (censored) model — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: 3 × 200,000 fp8-KV (--max-num-seqs 3), 25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the w4dn8 record (the bar to beat; new-payload probes pending). Rollback: re-download the w4dn8 .rad from the HF mirror frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700 (LFS sha256 09dd9306…) + revert the swap commit — the local copy was purged 2026-10-17 (it was the last non-iq4r model on the host; the GLM-4.7 GGUF, the radiance kvcache tiers, and every stale compile cache went in the same purge, 763 GiB free). The fp8-native .rad + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts).	7770
Dankclaw	frank-dragoon	halogen-flash-server 0.16.2 (ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…, digest-pinned, plain podman run via scripts/serve-frank-dragoon.sh, root systemd unit)	Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 b5bd925f… verified at fetch — the NATIVE 2026-10-11 v2 conversion of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 .hgn format loads in NO other engine), + REQUIRED qwen38-flash-next-ngram.hgn (51,200,246,144 B, peonist @5cc17cea) + vision tower (897,916,416 B, @e053f488) — both hardlinked from the swift15 payload dir (the release publishes only the main file; the engine's own niah 18/18 + ppl 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), vision ON (image_url parts only, ≥256 px probes), MTP spec decode ALWAYS-ON (HALOGEN_SPEC_ADAPT=0 — the adaptive default disables the drafter per request and HALVES decode), MTP head INSIDE the checkpoint (HALOGEN_MTP_HEAD unset), TWO slots over ONE HALOGEN_KV_POOL_POSITIONS=401408 pool (= 2 × ctx) at HALOGEN_CTX=200704 (pool RAISED from 200,704 on 2026-10-18 so BOTH slots can hold a maximum-length request at once; the parity play was frank-zealot's then-401,408 total in-flight context — zealot moved to 786,432 the same evening and dragoon's pool was NOT raised again, because every position is page cache the n-gram table loses; ~10.8 GiB reserved where 200,704 cost 5.5, 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; the ceiling does NOT move with it — 262,144 is the native wall and the engine refuses more without HALOGEN_ROPE_YARN), harness window 155,648 = ceiling − 45k estimate headroom, server-side sampler 1.0/0.95/20 (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), n-gram table disk-backed (mmap page cache, Pin::None); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9	7770
Dankclaw	frank-zealot	llama.cpp PrismML fork build (vendor release 0.2.0-dev build 10777, commit e3de71522; root systemd unit, relinked — the ONLY loader for this payload, and since 2026-10-18 also the minimum build that pairs MTP with a quantized V cache)	prism-ml/Ternary-Bonsai-2-9B-gguf PQ2_0 carrying the MTP head in the file (2,675,410,720 B, Hub-LFS sha256 c79fcdfa… verified at fetch by scripts/download-frank-zealot-bonsai9.sh; qwen35 hybrid: block_count 33 = 32 trunk layers (8 full-attention + 24 Gated DeltaNet at full_attention_interval 4) + the blk.32 MTP block, 4 KV heads × 256 dim, native 262,144; general.file_type 141 with 202 custom-type PQ2_0 tensors of 442 + prism.hadamard.* dequant metadata, which is what locks the runtime) + the publisher's own Q8_0 mmproj (624,229,664 B, sha256 e0cb6f69…; Q8_0 is the publisher's requirement for concurrent image requests — the BF16 tower can crash the server, and this endpoint runs three slots) for vision, on the RTX 3090, THREE 262,144-token slots (--ctx-size 786432 --parallel 3 = every conversation at the trained window), q8_0 K + q8_0 V under flash attention (17 KiB/token over the 8 full layers = 13,056 MiB pool, ledger-exact; f16 KV needs 26,112 MiB and cannot boot this card; +452 MiB of recurrent state for the three DeltaNet cells), batch 4096 / ubatch 2048 (measured CUDA optimum), --log-verbosity 4, MTP OFF — measured, not because the head is absent: the draft context bills its own f16 KV over the WHOLE pool (3,072 MiB) and at ubatch 2048 that squeeze halves decode (167.2 → 109.2 t/s); the documented MTP arm is SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 → 213.5 t/s. Boot 17,876 used / 6,249 free; 167.2 t/s single-stream decode, 280.2 t/s aggregate across three concurrent streams, 2,877 t/s prefill at 106,418. Cutover 2026-10-18 from Ternary-Bonsai-2-27B PQ2_0 at 2 × 200,704 (payload + build 10754 tree on disk as rollback).	7771

Retired endpoints (frank-npu, and every prior runtime) are in the runbook index. The full swap history lives in serverAppConfigs-docs/LLM/index.md — do not grow it here.

Repository Layout

• dankserver/ — compose.yaml + Caddyfile + .env.example for the media stack
• dankclaw/ — compose.yaml + Caddyfile + .env.example for the AI stack
• shared/ — homepage, Prometheus, Loki, Alloy, SearXNG configs
• scripts/ — deploy.sh / deploy-plan.py / deploy-remote.sh, LLM serve scripts, systemd units, env tunables, bench harnesses
• old/ — retired LLM scripts (reference only)
• serverAppConfigs-docs/ — knowledge base (Obsidian vault, OKF format). The managed skill library is distilled there one page per skill-cluster: LLM/index.md → Cross-Cutting References, plus Operations/index.md, Agent/index.md, and the Infrastructure/Dankclaw/FrankBook-Pro service pages (serverAppConfigs-docs/log.md carries the wave plan and results)
• tests/ — bats suite + Playwright e2e

Git Workflow

All edits flow through git. Never edit files on a remote machine directly.

1. Source the live config from the target machine
2. Commit the fetched version here (redact secrets into .env)
3. Edit locally
4. Test — ./tests/run.sh must pass
5. Commit and push — origin is Gitea on Dankclaw
6. Pull on the target (deploy.sh does this)
7. Deploy — scripts/deploy.sh plans and dispatches only changed services

Deployment

./scripts/deploy.sh dankclaw
./scripts/deploy.sh dankserver
./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-archon
./scripts/deploy.sh dankclaw --dry-run

The planner maps changed paths to services; the remote runner restarts only the selected service and records its revision outside the checkout. Documentation-only changes are a no-op. LLM services are selected only by their own files — every endpoint (frank-dragoon/frank-zealot/frank-archon) by its unit/env/serve trio in scripts/ — so unrelated container changes cannot interrupt inference. Set FRANK_SUDO_PW for headless runs on a host without passwordless sudo; the value never enters the repo.

deploy.sh <target> with no --service is a fleet-wide DRAIN, not a sync. It also walks every per-service revision marker under the deploy-state dir and restarts each service whose marker is behind HEAD and whose paths changed since that marker — so a backlog accumulated by service-scoped deploys fires all at once, and one failing health gate aborts the run with every later service left pending. Name the service unless a drain is what you want (./scripts/deploy.sh dankserver --service frank-archon).

Deployment completeness (MANDATORY)

A deploy is not complete when deploy.sh exits green. It requires:

1. ./tests/run.sh — all seams green, and
2. the Playwright e2e suite (tests/e2e/run.sh) — runs automatically as the [3/3] gate at the end of deploy.sh against the LIVE fleet; a behavior-affecting deploy that fails it exits DEPLOY INCOMPLETE. Bypass only with --skip-e2e in an emergency. The gate runs the conditional frankbook Playwright project (the *arr/qBittorrent walk, tests/e2e/specs/19-frankbook.spec.ts) ONLY when a frankbook blast-radius path changed since origin/main — that spec, frankbook-pro/**, shared/homepage/services.json, or the dankclaw/Caddyfile serving the arr vhosts — so a sleeping FrankBook-Pro never fails an unrelated deploy. The default project set excludes it everywhere else (E2E_INCLUDE_FRANKBOOK=1 opts in; see serverAppConfigs-docs/Architecture/E2E Suite.md).

graph LR
    A[Edit locally] --> B[Tests pass] --> C[Commit + push]
    C --> D[deploy.sh plan] --> E[Target pulls ff-only]
    E --> F[Dispatch selected service] --> G[Per-service health]
    G --> H[Playwright e2e gate] --> I{green?}
    I -->|yes| J[Deploy complete]
    I -->|no| K[DEPLOY INCOMPLETE]

Boot Persistence & Self-Healing (MANDATORY)

Every deployed, working Podman container MUST have a self-healing, boot-persistent Quadlet unit. A container that runs only from compose.yaml or a manual podman run dies silently at reboot — rootless Podman does not auto-start bare containers, and restart: unless-stopped does not survive the explicit stop that shutdown performs. Never report such a service as "up".

Unit requirements:

• Boot persistence — [Install] WantedBy=default.target, installed to ~/.config/containers/systemd/<name>.container, enabled in the user manager, with linger on for the deploy user (loginctl show-user dank -p Linger → Linger=yes).
• Self-healing — Restart=always (or on-failure) with a short RestartSec; quadlet health check where the service has one.
• Repo provenance — the .container file lives in this repo (<host>/<svc>/quadlet/), is installed by the deployer, and is covered by the quadlet test seam. The host-side unit is generated state; the repo file is the source of truth.

Before declaring a service deployed: systemctl --user is-enabled <name> → enabled, is-active → active, and the container answers its health check. A running container with no unit is a gap: write the unit, test it, deploy it.

This rule exists because reboots found gaps twice (2026-08-26: Caddy as a bare compose container never returned; 2026-09-11: five observability services had no units). The observability services now have units under dankclaw/observability/quadlet/ and their compose definitions were removed — one owner per container name. The stack is configured and delivering as of 2026-09-17: a Grafana datasource and the llm-fleet dashboard are provisioned from shared/grafana/provisioning/, shared/prometheus/rules/ llm-fleet.yml loads 5 alert rules + 1 recording rule, and firing rules are delivered: Prometheus → Alertmanager (dankclaw/observability/quadlet/ alertmanager.container, loopback :9093) → host-side Signal relay (scripts/signal-relay.py, user unit, 192.168.0.143:18093 — the host LAN IP, the one bind a container can reach as host.containers.internal; iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311. Prometheus targets read down for structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is a blackbox exporter. frank-zealot was down for a different reason until 2026-10-18: llama.cpp will not serve /metrics at all without --metrics. See serverAppConfigs-docs/Infrastructure/Observability.md.

Testing

Tests are mandatory before commit (bats required on the dev machine):

./tests/run.sh

Seams: compose validity, env consistency (no leaked secrets), Caddyfile validity, quadlet units, docs completeness including vault link integrity, script syntax (shell + python), LLM unit integrity, LLM secret hygiene, deployment flow, service-scoped planning, homepage config/pages/deploy/checker, and e2e_scaffold — which fails if a Playwright spec file's number prefix is in no project, because such a spec silently never runs. See serverAppConfigs-docs/Architecture/Testing Policy.md.

Secret Rules

• Never commit real secrets. The live Gitea DB password and sudo password are test-enforced leak checks.
• .env files are git-ignored; commit only .env.example with placeholders. Exception: scripts/*.env LLM tunables (paths and numbers, never credentials).
• Do not document passwords anywhere in the repo. SSH keys are the auth mechanism; FRANK_SUDO_PW covers headless sudo and never enters the repo.
• A token passed as curl -H "Authorization: Bearer …" is world-readable. It lands in /proc/<pid>/cmdline, so any ps/pgrep -a on the host prints it — that leaked an HF token into a session transcript on 2026-09-16, while merely checking whether a download was still running. Pass secrets out of band (curl -K <mode-600 file> carrying the header, or HF_TOKEN in the environment for hf/hf_transfer) and never pgrep -a a fetch command that carries an auth header.

External Exposure Policy (MANDATORY)

Every Caddy-proxied service is either behind the auth chain (Anubis :8923 → Authentik embedded outpost) or explicitly internal-only — no public DNS name, no vhost. New services default to internal-only until guarded.

When guarding a new vhost — provider first, then route:

1. Create the outpost proxy provider FIRST (external_host = exact vhost URL with trailing slash; internal_host = http://caddy:8017). If the vhost flips first, authenticated traffic hits the outpost error page.
2. Flip the vhost to reverse_proxy anubis:8923 with header_up X-Real-Ip {remote_host}, and add the post-auth @<svc> host <vhost> handler on the http://:8017 listener routing to the real upstream.
3. Anubis guard set, Caddy TLS edge, and outpost provider list change together in one deploy.
4. Service-to-service traffic targets internal addresses (loki:3100), never the public hostname — non-browser clients cannot solve challenges.

Exempt categories (document the reason in the vhost comment): the IdP itself (auth.), git smart-HTTP endpoints, services with their own complete auth.

Deletion & Trash Policy (MANDATORY)

Never rm. Move deletions into trash so they stay recoverable:

1. In-repo files → .AGENTTRASH/ beside the item's directory.
2. Elsewhere → dated staging dir ~/.AGENTTRASH/YYYYMMDD-topic.
3. List staged items and sizes; wait for explicit user approval to purge.

rm, rm -rf, and git clean -f are forbidden without explicit permission. git * --no-verify is forbidden. The trash is a temporary hold, not an archive.

Benchmarking Policy (MANDATORY)

STOP the production service BEFORE benchmarking — these servers serve real users, and a bench beside a live service starves the GPU and corrupts results. Use the runtime's own harness: llama-bench for the llama.cpp endpoints; a live llama.cpp server also reports honest per-request numbers in the /completion timings object (prefill/decode rates, draft_n, draft_n_accepted), which is how frank-dragoon and frank-zealot were measured. Rolling your own bench requires asking first. Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service itself.

(A gufo-era law; frank-dragoon ran ROCmFPX llama.cpp 2026-10-02 → 10-05, the halogen 27B container 2026-10-05 → 10-06, and is flash-server since, but the rollback tags keep it live.) gufo bench cannot rank speculative-decode settings, and will invert the ranking. Measured 2026-09-28 on frank-dragoon at depth 32768: its corpus reports acceptance=0.933 ± 0.000 with an identical trace sha256 across draft widths, i.e. the same highly-predictable text every run, and 93 % of every proposal accepted. It reported 29–37 t/s decode where real traffic gives ~15 t/s, and it made --draft-tokens 16 look 6 % faster than 7 when real text made it 14 % slower. Use it for shape, not for A/B. For an A/B, replay real text through the live server and read the Server-Timing header — but note the unit rides in the name half (ttft;dur=402.485), so splitting on = alone keys the dict ttft;dur and every lookup silently returns nothing. Generate at least 128 tokens for the measurement: at 8 tokens, inter_token is first-token noise and the run-to-run spread swamps every real difference.

This covers any GPU experiment, not just benchmarks — including a bare 2-rank RCCL/torch.distributed probe. On 2026-09-16 one such probe was run against the live service "because it only needs 200 MB of VRAM"; the host went hard down ~20 minutes later (no ARP reply, 100% ICMP loss from two subnets, no recovery — a power cycle was required). A hung AQL dispatch wedging the GPU driver is the plausible mechanism, and this host has precedent for going unreachable until a power cycle. If an experiment needs GPUs, stop and mask first, or use a host that serves nobody. Telling an off machine from a wedged one: ip neigh show <ip> from a peer — FAILED means off/NIC dead, while a merely wedged kernel usually still answers ARP.

1. Stop and mask (self-healing would restart it):
   • Dankclaw frank-dragoon (root unit, halogen podman on the iGPU): sudo systemctl stop frank-dragoon && sudo systemctl mask frank-dragoon
   • Dankclaw frank-zealot (root unit, llama.cpp on the 3090): sudo systemctl stop frank-zealot && sudo systemctl mask frank-zealot

• Dankserver frank-archon (root unit, radiance on both R9700s, TP=2): sudo systemctl stop frank-archon && sudo systemctl mask frank-archon

2. Bench. NOTE: systemctl mask FAILS on a relinked unit (the unit file in /etc/systemd/system/ is itself a symlink into the checkout — mask refuses to replace it). Stop is enough for the bench window; the deploy restores the unit, so re-deploy after benching instead of unmasking.
3. Unmask and start; verify curl http://192.168.0.<host>:7770/health.

CRITICAL: never systemctl disable a system unit on these hosts — it deletes the unit file from /etc/systemd/system/, not just the symlink. mask symlinks to /dev/null and is safe/reversible.

Model Config (harness)

The harness reaches the fleet through .omp/agent/models.yml in the operator's home — /home/blank/.omp/agent/models.yml on the dev box, which is where the harness runs; there is no copy on the servers. Each provider is one HOST:PORT; each model id is the served alias. Official model ids are the StarCraft unit names (2026-10-08), on the wire since 2026-10-09, and since 2026-10-10 they are also the SERVICE names: frank-archon → 192.168.0.235:7770 (FRANK_ARCHON_SERVED_NAME), frank-dragoon → 192.168.0.143:7770 (HALOGEN_MODEL_ID), frank-zealot → 192.168.0.143:7771 (ALIAS). The 2026-10-10 cutover renamed the units, scripts, env files, containers, deploy services and runbooks to match — the three former service names are fully retired and exist nowhere in the repo; historical text in runbooks uses the current names. Verify a rename on /v1/models, never on a chat response: radiance echoes the requested model name, so a 200 proves nothing about the served id (llama.cpp and halogen ignore it).

Resolution rules: the default model resolves as a defaultProvider + defaultModel pair (the provider must serve the model); a subagent override uses a bare model id, which resolves only when unique across providers. Two ports on one host need two providers — the file cannot express "same host, different port".

When renaming a service, retiring an endpoint, or moving a model: update the provider block, model id, contextWindow, maxTokens, input in models.yml, then fix every modelRoles entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As of 2026-10-06 all three endpoints accept images (input: [text, image]) — frank-dragoon returned to vision with the flash-server v2 swap (the v2 vision tower loads beside the checkpoint; the text-only halogen 27B window closed 2026-10-06) — while frank-archon takes them natively (radiance runs the checkpoint's 27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary are the engine core; --mm-max-patches available), and frank-zealot through the publisher's own Q8_0 mmproj passed as --mmproj — Q8_0 is the publisher's requirement for concurrent image requests (the card states the BF16 tower can crash llama-server when several image requests run at once, and this endpoint runs three slots). frank-zealot's contextWindow is 262144 — the PER-SLOT window (--ctx-size 786432 --parallel 3 since the 2026-10-18 Bonsai-2-9B swap), which since that swap EQUALS the model's native window; llama.cpp answers a longer prompt with a 400 naming the limit, never a truncation, so the harness must compact before it. Note also that the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3 parser, live-verified) emit reasoning_content, vLLM emits reasoning — models.yml sets reasoningContentField per endpoint to match. There is no audio input anywhere in the fleet: that capability left with the FLM NPU endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's audio encoder, and left again with the Gemma4 retirement on 2026-10-01 (neither the MiMo nor the Qwen3.5 tower carries audio).

frank-dragoon — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 NATIVE abliterated (Dankclaw, ACTIVE)

The flash-server engine (ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…, digest-pinned, published tag 0.16.2, 3.6 GB image) as a podman container driven by scripts/serve-frank-dragoon.sh under a root systemd unit (scripts/frank-dragoon.service, User=dank, Restart=always, ExecStop=podman stop -t 60 frank-dragoon-halogen, the script runs podman run FOREGROUND — the container's exit code is the service's exit so Restart=always stays real, and the engine ledger lands on disk) on port 7770 from the Strix Halo iGPU. Engine stdout goes to /home/dank/logs/frank_dragoon.log — the boot ledger (weights resident, KV pool, MTP head, prompt cache, any clamp) lives there, not in the journal. Cutover 2026-10-17 from the Swift1.5-finetune shape (frank-dragoon-flash-server-v2-swift15, retired the same window; it itself cut over the halogen 27B-dense era on 2026-10-06).

Serving Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated: qwen38-flash-next-v2-abliterated.hgn (66,687,678,432 B ≈ 62.1 GiB, sha256 b5bd925f… — the NATIVE 2026-10-11 v2 conversion of orcarouter's pinned BF16 checkpoint, abliterated; NOT the writer transplant that carried this filename, which measured 66,687,678,464 B — 32 bytes apart, same filename family, different model) + the REQUIRED n-gram lookup table qwen38-flash-next-ngram.hgn (51,200,246,144 B, peonist @5cc17cea) + the vision tower qwen38-flash-next-vision.hgn (897,916,416 B, @e053f488) + the flat tokenizer/ directory (6 files) — the three companions hardlinked out of the swift15 payload dir, because the release publishes only the main file and no source-matched pair exists. Cutover 2026-10-17 from the Swift1.5 finetune, chosen by Frank for the base checkpoint itself plus the card's memory-saving pool.

The format is the lock. The v2 .hgn loads ONLY in the flash-server line — the 27B-era halogen 0.1.4 container refuses it, so the engine moved with the model. The MTP head is INSIDE the checkpoint (HALOGEN_MTP_HEAD stays unset; the separate qwen38-flash-next-mtp.hgn sidecar is deliberately not fetched).

Product facts vs the 27B era, both regressions FIXED: the base is abliterated (the 27B was a censored-base finetune; refusals were an accepted trade there) and vision is ON (the 27B engine silently ignored its encoder — frank-zealot owned images for exactly one day at this endpoint).

Engine laws that shape the config (all in scripts/frank-dragoon.env with evidence): HALOGEN_SPEC_ADAPT=0 is MANDATORY — the default adaptive policy disables the MTP head for whole requests after low acceptance and HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance for it (accepted). HALOGEN_CTX=200704 is the PER-REQUEST hard ceiling (400 naming the limit, never truncation) — the checkpoint's native window is 262,144 and was served until 2026-10-17, when Frank lowered it for RAM. The 400-zone law is invariant to the value: the harness clamps max_tokens to min(45056, contextWindow − est − 4096), its chars/4 est UNDER-counts the rendered prompt by ~3 % (measured: est 152,938 → server 157,388, request 43,158 vs room 42,804), so a harness window == engine ceiling makes every long request a guaranteed 400. models.yml contextWindow: 155648 (= ceiling − 45056) keeps that gap (worst-case request ≈ 156.2k ≤ 200,704); compaction fires at 85 % of the harness window = 132,301 est ≈ 136k measured (the 180k target died with the 262,144 window). HALOGEN_KV_SLOTS=2 is LEGAL here — the 27B "slots>1 disables the drafter" law does NOT carry over; batching and speculation coexist (the card's quickstart ships 2 slots with the head), so 2 is concurrent capacity. HALOGEN_KV_POOL_POSITIONS=401408 — 2 × ctx, RAISED from 200,704 on 2026-10-18 (Frank: dragoon's total in-flight context was HALF frank-zealot's 401,408 pool — doubling the POOL is what fixes that, not doubling the window) — is ONE pool behind both slots, ledger form 2 slots over ONE <pool>-position KV pool, each request up to HALOGEN_CTX, and at pool == slots × ctx two MAXIMUM-length conversations DO coexist. The PER-REQUEST ceiling stays HALOGEN_CTX=200704: the checkpoint's native window is 262,144 and the engine's own 400 text refuses to exceed it without HALOGEN_ROPE_YARN, so "more context" bought here is CONCURRENCY, not a longer single conversation. ~28 KiB/position ⇒ the ledger reserves "about 10.8 GiB", where 200,704 cost 5.5, 262,144 cost 7.2 and the 2026-10-06 shape's 524,288 measured 14.6; the engine's preflight FORMULA bills slightly more than the reservation (block scratch + ~115 MiB/slot of O(1) state) — quote the reservation. HALOGEN_MAX_TOK=8192 is only the per-request DEFAULT (the harness sends at most its maxTokens 45056); HALOGEN_MAX_TOKENS_CAP=65536 + HALOGEN_QUEUE_TIMEOUT=7200 are a coupled pair (FLAGS.md table). A server-side sampler EXISTS (HALOGEN_TEMPERATURE/TOP_P/TOP_K = 1.0/0.95/20, the card's thinking recipe) — an absent temperature is no longer greedy; models.yml extraBody carries the same values, consistent not load-bearing. Prompt cache is ON by default (PROMPT_CACHE=2): a warm 100k follow-up costs ~2 s vs ~88 s cold — the 27B engine's manual HALOGEN_CACHE_MB pin era is over. response_format is unsupported. Thinking lands in reasoning_content; the v2 chat template rejects reasoning_effort outside xhigh|medium|low, so the models.yml fold stays mandatory. Thinking tokens COUNT against the budget (empty content + finish_reason: "length" when exhausted mid-thought). Vision: OpenAI image_url content parts only; ≥256 px probe images or the model answers "video".

Memory ledger (121.2 GiB unified; the referee is the boot ledger): weights 62.9 (trunk 62.11 + vision 0.84) + KV ~10.8 + working 5.7 (3.8 + 8.4 × MAX_TOK/16384) ≈ 79.4 GiB in all, the ~1.0 GiB prompt cache booked separately by the ledger, + HALOGEN_HOST_RESERVE_GIB=20 → ~22 GiB margin over baseline co-tenancy (the pool-equals-ctx boot printed 74.1 GiB with ~30 GiB of margin). The 51.2 GB n-gram table rides as mmap'd page cache beside the checkpoint (ledger: mapped 51.2 GB, NOT pinned (Pin::None)) — which is why every pool position is a double cost: RAM the pool takes is RAM that table loses, so the warm prompt-cache ratio is re-measured after a raise. HALOGEN_FLASH_PIN_TRUNK=0 is REJECTED by the v2 loader, so disk-backing the trunk is not a lever on this format. READ the boot ledger and re-size if the engine prints a clamp.

Fetch: scripts/download-frank-dragoon-v2abl.sh into /home/dank/models/halogen-v2-abliterated/ (-f + exact byte counts + LFS sha256 + --verify-only, no token on the command line; resume goes against the RESOLVED CDN URL because curl drops Range across HF's 302) — the main file is fetched, the three companions are HARDLINKED from the verified swift15 dir (a hardlink is a regular file, so the no-symlink serve gate passes without a 47.7 GiB copy). The serve gates mirror the deployer EXACTLY (byte-exact checkpoint + n-gram + vision tower, flat tokenizer with all six files as regular files) — the 2026-10-01 crash-loop lesson.

Runbook: serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md. Rollback: revert the swap commit and redeploy — the Swift1.5 payload (/home/dank/models/halogen-flashnext-v2/, re-verified by scripts/download-frank-dragoon-flashnext.sh), the 27B payload (/home/dank/models/halogen-qwen3.8-27b/) and the halogen 0.1.4 image digest all remain on disk; the ROCmFPX door behind that (scripts/build-rocmfpx.sh, pin 36e9acd) is untouched.

frank-zealot — llama.cpp (PrismML fork) on the RTX 3090 (Dankclaw, ACTIVE)

The PrismML fork build of llama.cpp (vendor release, --version = 0.2.0-dev (build 10777, commit e3de71522); /home/dank/GitPackages/llama-prism-b10777/llama-server) as a root systemd unit (scripts/frank-zealot.service, relink_united into /etc/systemd/system/), serving prism-ml/Ternary-Bonsai-2-9B-gguf at PQ2_0 with the MTP head IN the file (2,675,410,720 B, Hub-LFS sha256 c79fcdfa… verified at fetch) plus the publisher's own Q8_0 mmproj (624,229,664 B, sha256 e0cb6f69…), both /home/dank/models/Bonsai2-9B-MTP/, on port 7771. Cutover 2026-10-18 from the Bonsai-2-27B PQ2_0 era (2 × 200,704, same card, same fork family).

The runtime is part of the payload identity — twice over. The file is general.file_type 141 — 202 of the 442 tensors carry the fork's PQ2_0 type (the rest are f32 norms/scalars plus 8 q8_0 in the MTP block) — beside prism.hadamard.* metadata (normalized Sylvester–Walsh transform): stock llama.cpp refuses the custom type at load, and a loader that ignored the Hadamard metadata would dequant WRONG — same lock class as the ROCmFPX era. On top of that lock, the fork gates MTP beside a quantized V cache on build prism-b10776-d00e252 or newer, which is why the prebuilt moved 10754 → 10777; the 10754 tree stays on disk as the 27B rollback runtime. Retargeting the payload without the fork ships an endpoint that cannot load, so the seam pins both paths.

Architecture: qwen35 hybrid — read from the GGUF's own header: block_count 33 = 32 trunk layers + the blk.32 MTP block (the MTP block has its own attention + FFN in q8_0, which is why the draft context bills real KV), and qwen35.full_attention_interval = 4 ⇒ 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, native 262,144. KV accounting: the pool scales with the 8 full layers — 8 × 4 × 256 = 8,192 elements/token per cache, so q8_0 K + q8_0 V bills 17,408 B/token ≈ 17 KiB = 13,056.00 MiB at the 786,432 pool (ledger-exact) — exactly HALF the 27B's 34,816 B/token, which is the whole reason three native windows fit where the 27B afforded only two shortened ones. f16 K would bill 26,112 MiB and cannot boot on a 24,123 MiB card. The 24 DeltaNet layers bill a fixed recurrent state per SLOT — RS buffer size = 452.25 MiB (3 cells, 32 layers, 3 seqs) ≈ 150 MiB each — so pool arithmetic alone under-prices the shape.

The shape (2026-10-18): --ctx-size 786432 --parallel 3 = THREE conversations of 262,144, every one at the model's TRAINED window (Frank: "if you can possibly fit a third context, do it"), --flash-attn on, q8_0 K + q8_0 V, batch 4096 / ubatch 2048 (the measured CUDA optimum carried from the MiMo/Swift A/Bs on this card; it also sets the compute-buffer size, which is what decides whether MTP fits), --log-verbosity 4, --metrics, --jinja, --mmproj for vision, speculative decoding OFF (SPEC_TYPE=none). Boot: 17,876 MiB used / 6,249 free; compute 1,409 MiB CUDA + 1,201 MiB CUDA_Host, projector 595 + 248 MiB. Measured on the same boot: 167.2 t/s greedy decode, 158.9 t/s single-stream chat, 280.2 t/s aggregate across three concurrent streams, 2,877 t/s prefill at 106,418 tokens — the 27B record (76.6 decode / 109.4 aggregate / 691.7 prefill at 166k) beaten on every axis. The first request after boot is a warmup artifact (37.7 t/s class) — never report it.

Why MTP is off even though the head is in the file. The drafter is GOOD — acceptance 0.798 / 0.617 per position, mean accepted length 2.41 — and the card recommends depth 2. It still loses AT THIS SHAPE because the draft context bills its OWN f16 KV over the WHOLE pool (4,096 B/position × 786,432 = 3,072.00 MiB, 4,830 MiB with its compute buffers) and ubatch 2048 refuses to give it up. Booted arms, same card, same traffic (2026-10-18):

slots	spec	ubatch	used / free	decode
3 × 262,144	off	2048	17,876 / 6,249	167.2
3 × 262,144	on	2048	22,706 / 1,419	109.2
3 × 262,144	on	1024	21,400 / 2,725	213.5
2 × 262,144	on	2048	17,178 / 6,947	213.2
1 × 262,144	on	2048	11,652 / 12,473	214.2

MTP is worth +28 % decode whenever the card has room; the squeeze, not the drafter, is what halves it at three windows. The arm that WOULD run it at three windows is SPEC_TYPE=draft-mtp + UBATCH_SIZE=1024 (213.5 t/s single-stream, 250.8 three-stream aggregate) and pays prefill −17 % and 2.7 GiB of headroom on a card whose projector worst-case lands AFTER the engine's own fit check. This endpoint is the fleet's compaction hub — prefill-bound, 2–3 concurrent subagents — so the shipped arm takes the +17 % prefill (2,877 vs 2,395 t/s) and the +12 % three-stream aggregate (280.2 vs 250.8) over single-stream speed, and keeps 6.2 GiB free. SPEC_TYPE=none strips the WHOLE spec set (--spec-type and every --spec-draft-*) — flags move as a set or the arm measures a crash instead of an absence.

Sampler: the recipe lives in the checkpoint (general.sampling.* = temperature 1.0 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.0, confirmed in the engine's own /props) and llama.cpp reads it itself. Presence penalty is the one recipe value with no GGUF metadata key, so PRESENCE_PENALTY=1.0 must reach the argv; "" is the documented opt-out, and the serve script therefore uses ${PRESENCE_PENALTY=1.0} (NOT :=, which would silently re-add the flag on an explicit empty value).

Vision: the publisher's own Q8_0 mmproj via --mmproj — Q8_0 is MANDATORY, not a preference: the card says the BF16 tower "can crash llama-server when several image requests run at once", and --parallel 3 is exactly that shape. Probe traps carry over: ≥256 px probe images or the model answers "video", only the OpenAI image_url content part reaches the model, and thinking tokens count against max_tokens — budget ≥512 or the probe returns EMPTY content with finish_reason: "length" and the endpoint looks blind while it is only mid-thought.

Base alignment: Bonsai-2 is the censored BASE lineage. Unlike the 27B, the CRACK re-export on disk (/home/dank/models/Bonsai2-CRACK/) is byte-identical geometry to the 27B it matched, NOT to this 9B file — if a refusal ever surfaces there is no one-line MODEL_FILE swap; a 9B dealigned export would have to be fetched first.

Compaction hub: models.yml routes compactionModel for frank-archon, frank-dragoon and frank-zealot (the 3090 endpoint) here — 262,144 (since the 2026-10-18 reslice) is the widest window in the fleet and the only one that comfortably fits a worst-case compaction input (~170k tokens). With --parallel 3 a compaction request no longer serializes behind a live generation (the Gemma4 era's --parallel 1 law). Retrieval through a 25,960-token prompt was verified exact after the swap.

Contract law learned at verbosity 4: a raw /completion request prints failed to parse reasoning content (… parse_reasoning=true) whenever the model closes a bare prompt with `` — cosmetic (probe_request.cpp:789); the /v1/chat/completions path parses reasoning_content correctly, which is the only path consumers use. Do not lower VERBOSITY to silence it: the KV/buffer ledger is what VRAM forensics read. --metrics is required or the Prometheus scrape reads 501 = DOWN.

Runbook: serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md. Rollback: revert the swap commit — the 27B payload (/home/dank/models/Bonsai2-PQ2_0/) with the 10754 fork tree (~/GitPackages/llama-prism/), the Qwen3.5-9B Q6_K payload (/home/dank/models/Qwen35-9B-HauhauCS-Q6K/) with the master CUDA build (~/GitPackages/llama-master/build-cuda/), the Gemma4 payload (/home/dank/models/Gemma4-26B-A4B/), the MiMo Q6_K payload (/home/dank/models/MiMo-Q6K/), the Swift payload (/home/dank/models/Swift15-UncIQ3XXS/) and the Ornith Q8_0 payload (/home/dank/models/Ornith-Q8/) all remain on disk.

frank-archon — Radiance (StillDeadcode), publisher Qwen3.8-Flash-Next iq4r, TP=2 (Dankserver, ACTIVE)

The radiance engine (StillDeadcode, Apache-2.0; image stilldeadcode/radiance@sha256:4c3f484564… digest-pinned, pin 1.0.13 for serve; libr4d gfx1201 kernels, plain podman run, root systemd unit, container name = unit name) serving the publisher's container of the official Qwen/Qwen3.8-Flash-Next: StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe → qwen3.8-next-flash-fp8-iq4r-moe.rad (116,322 MiB, Hub-LFS sha256 verified at fetch by scripts/download-frank-archon-iq4r.sh; recipe qwen4exp-w4nl64-i8-hc8m): routed experts as 4-bit codes into a 16-level non-uniform codebook after a 128-point Walsh–Hadamard rotation (E4M3 scale per 64 weights; codes by GPTQ on a 10M-token calibration with per-expert down-proj Hessians, refined by 3 coordinate-descent sweeps), 10 whole bf16-protected experts/layer, int8 W8A8 trunk/lm_head (scale per 128 columns searched), fp8 HC mixtures, PLE n-gram table at fp8 fixed scale (47.7 GiB), MTP head and bf16 vision tower in the container. Card quality (full-vocab teacher-forced KL vs bf16, ~75K positions): mean 0.0789 all-tokens / 0.0239 assistant turns, top-1 92.85 %. MODEL_FILE is the only serving argument that moved (2026-10-15); every number of the 2026-10-13 shape below carries verbatim. The plugin's one-table-one-encoding law (fleet-measured at rad-convert declare): one expert table takes ONE encoding; only whole bf16 experts are legal beside it. Rollback: revert the swap commit AND re-download the in-house w4dn8 .rad (KLD-gated Swift1.5 conversion) — it is no longer on disk (purged 2026-10-17); the byte-exact copy is published as frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700 (LFS sha256 09dd9306…). The fp8-native .rad was deleted 2026-10-15 (rebuild: scripts/download-frank-archon.sh + scripts/build-frank-archon-rad.sh).

Shape (both R9700s, TP=2): --tp-wire wht6, --max-num-seqs 3, --max-model-len 200000 (the harness contextWindow must quote this), --placement expert_tiered --host-pool-mib 44928 + explicit VRAM partition --vram-weights-mib 25600 --vram-kv-mib 5120 (the vram-pair replaced --expert-vs-cache-ratio; the two are ONE budget — weights + kv ≤ 30,969 MiB, derived from the engine's own budget table: card 32,624 MiB, the pair at 29,120 settling 2,058 MiB free against the engine's 209.19 MiB floor (--gpu-headroom-mib 96 + 113.19 static), so 1,849 MiB is claimable and the old 29,120 bound was my hand-picked 1 GiB reserve, not a hardware wall), --kv-cache-dtype fp8 — quantized KV, restored 2026-10-13 because it is the only dtype that funds the slot count at this window, --num-speculative-tokens 2, --max-num-batched-tokens 2048, prefix cache 8 GiB pinned host + 64 GiB disk at /home/dank/radiance-state, --ngram-placement ram, Froggeric v22.5 chat template vendored (scripts/qwen38-froggeric-v22.5.jinja, --override-chat-template). Boot ledger (2026-10-13, pair 25600/5120): kv cache 5.00 GiB … backs 3 sequences all at 200000 tokens at once, which is the worst case — kv_attn page 26.00 KiB (fp8) at 607,740 tokens (the sizing law predicted 608,027), kv_qsa_bkey 3.25 KiB at the same token count, kv_gdn_state 54 MiB × 6 slots (3 sequences × 2 copies) = 324 MiB; the sampler prints 9 rows (3 × (1+2), inside the 25-row wht6 exactness bound); the slab filled to 563 KiB of slack with 17,548 slots (16,235 at 24,000, 14,753 at 22,208 — every 1,792 MiB buys ~1,300 resident experts, and the pinned host pool hands the same experts back: 8,893 → 7,580 slots); unclaimed 477.30 MiB, cards settled ~336 MiB free against the engine's 209.19 MiB floor, KV lending 4.22 GiB to the expert cache. Prefill tracks this slab only up to a threshold: 22,208 → 3,397 and 24,000 → 3,290 t/s at 43.5k are one band (±3 %), and the 25,600 arm measured 3,290 — the third step bought nothing (the 3,584 arm did not reproduce at 3 slots); the cliff is at ~26 GiB (26,071 → 4,379 on 2026-10-11's arm), so below that the slab is a threshold, not a slope.

Decode ladder (2026-10-13, 512 tokens, greedy, this shape): 99.0 t/s single-stream at 43.5k, 93.3 at 197k — decode is nearly window-free (−6 % for 4.5× depth); 3 concurrent streams give 113.9/43.3/10.9 t/s = 18.4 t/s aggregate, so the slot count is paid in decode concurrency (a fused-step lottery), not in prefill; leg D of probe-frank-archon-kv.py measures it.

Shape notes carried from the fp8-native era and unchanged: wht6 gains, MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk prefix cache, and the rule that a prompt over --max-model-len is refused, never truncated. What changed: the fp8-native era's NVMe expert streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the host pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s on the fp8 shape (512-token streamed probe). The campaign also measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth, accept 0.7246).

The KV dtype law (measured 2026-10-11, applied at 4 slots 2026-10-13): --kv-cache-dtype takes ONLY bf16|fp8 — one dtype for the whole cache, so llama.cpp's exact-K/quantized-V pair (the 9B-era frank-zealot ran it; the Bonsai-2 swap runs q8_0/q8_0) has NO radiance equivalent; the rung under bf16 is fp8 on both halves. Measured per rank from the two boot ledgers: kv_attn (13 layers, 4-token blocks) pages 26.00 → 52.00 KiB, while kv_qsa_bkey (13 layers) stays 3.25 KiB at BOTH dtypes (the QSA block key is an index, not attention state) — so a token costs 7.3125 KiB at fp8 and 13.8125 KiB at bf16: 1.89×, not 2×. Pool sizing must also pay the dtype-independent terms that live in the same pool (~110 MiB a sequence of recurrent state + 448 MiB of prefix checkpoints): (vram_kv − 110·seqs − 448) MiB / rate — bf16 at 6,912 gives 462,946 where the engine reported 462,752 (0.04 % page rounding); fp8 at 6,912 with four slots gives 843,566, i.e. the deployed capacity contract 4 × 200,000. That is why the dtype is fp8: the pool must back seqs × max-model-len as a hard limit, and bf16 at four slots costs 11,679 MiB — the slab would fall to 17,441 MiB (−33 %), where fp8 leaves it at 22,208. The cost is paid twice on this engine, and half-refunded: idle KV VRAM is also the expert cache's spillover, and the bigger pool lends 6.08 GiB to the slab where the 3,543 MiB fp8 pool lent 2.69 GiB. --max-num-seqs rides 4 × (1 + spec) = 12 fused-all-reduce rows, under the 25-row bound where wht6 is bit-exact (the "seqs 2 cannot boot" law belongs to the retired tcclaviger fork's torch.compile, not radiance). Disk-tier sizing law: the engine forces ~9.6 GiB of disk-tier weight buffers + arena onto the weights side, so weights + kv is capped near 29.6 GiB/card; boot with the pair, never --kv-cache-size-mib.

Shape notes carried from the fp8-native era and unchanged: wht6 gains, MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk prefix cache, and the rule that a prompt over --max-model-len is refused, never truncated. What changed: the fp8-native era's NVMe expert streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the host pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s on the fp8 shape (512-token streamed probe). The campaign also measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth, accept 0.7246).

The pinned-RAM ledger is why this is a root unit: 2 × 44 GiB expert pools + the 47.7 GiB n-gram copy (--ngram-placement ram is pinned RAM the card reads) + 8 GiB prefix host ≈ 147 GiB charged against RLIMIT_MEMLOCK — the TTM floor is set just above it. ttm.pages_limit/pages_max cap GPU-mappable pinned pages, and the knob is runtime state whose default is half of RAM: scripts/ttm-preflight.sh (sourced by the serve script) therefore RAISES it to FRANK_ARCHON_TTM_TARGET_MIB=204800 before refusing, because the 2026-10-12 power failure reset it to 125,988 MiB and the preflight re-refused 6,919 times. A manual sysfs write in the stop window is a convenience, not the repair; kernel cmdline persistence (ttm.pages_limit=52428800) is the belt to it.

Contract facts (source-verified): the HTTP listener binds AFTER the full load, so a bare curl /health gate is honest (/health == /ping, 200 serving / 503 stopping); sampling params temperature…preserve_thinking accepted, unknown params 400 (models.yml's extraBody set is fully inside it); response_format supported; logprobs REFUSED; reasoning in reasoning_content and tool calls in the qwen3.8 dialect via the auto chat format — no parser flags; vision native (the 27-layer Qwen-VL tower, --mm-max-patches); /metrics carries vLLM-compatible names + radiance:* families — the Dankclaw Prometheus scrape is re-added.

Runbook: serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md. Retired Swift1.5 w4dn8 era (payload purged 2026-10-17; rollback = HF re-download frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700): serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md. Retired fp8-native era (payload deleted 2026-10-15, rebuildable): serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md. Retired FreeToken era: serverAppConfigs-docs/LLM/frank-archon-freetoken-pr587.md. Retired vLLM era: serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md. Retired GLM experiment: serverAppConfigs-docs/LLM/frank-archon-glm53-flash-w4a16.md.

Key Lessons Learned

Durable operational lessons from the fleet's history. Full detail in the linked runbooks.

PCIe / multi-GPU

• RCCL collectives need PCIe atomics on every participating GPU. RCCL's kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch them when the device has no atomics, and the failure surfaces as hipErrorIllegalState ("the operation cannot be performed in the present state") from hipLaunchKernel, not as anything that names atomics. The value is consumed at driver probe time, so writing the AtomicOp Requester bit with setpci at runtime changes nothing. Check the chain: lspci -vv -s <gpu> and every upstream bridge must show AtomicOpsCap: Routing+ (a Routing- bridge is a hardware wall), and the endpoint's Device Control 2 (setpci -s <gpu> CAP_EXP+28.w) must have bit 6 set. AMD desktop platforms put one card on CPU root ports (atomics work) and the rest behind the chipset (usually not) — that asymmetry is what blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL short-circuits at world_size=1.
• That wall was physical, and it was fixed on 2026-09-20 by a board swap. Both paths that put two cards on CPU root ports were: an M.2→PCIe x4 riser into a CPU-attached M.2 slot, or a board whose CPU lanes bifurcate to x8/x8. dankserver took the second: ASUS ProArt B850-Creator WiFi NEO, PCIEX16_1 + PCIEX16_2 at x8/x8 Gen5 on root ports 00:01.1 / 00:01.3, every bridge now Routing+ and both endpoints ReqEn+ — the exact precondition the old test plan named. Bandwidth was never the constraint (a 27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4 and under 0.4 ms at Gen5 x8). Proven on the same day. vLLM's R4D one-shot allreduce runs TP=2 here and frank-medium serves on it; llama.cpp's RCCL path was built too (GGML_HIP_RCCL=ON) and its tensor mode measured — real collective, slower shape. Note the collective choice matters: TheRock's RCCL builds lack gfx1201 device code (invalid device function), which is why R4D is the path vLLM uses at all. Evidence matrix and the measured three-arm verdict: Dual-R9700-Cooperation.
• When a GPU collective dies, read the runtime, not the framework. The message that identified the cause came from AMD_LOG_LEVEL=3 (Pcie atomics not enabled, hostcall not supported → AQL dispatch failed!), not from vLLM or RCCL. AMD_LOG_LEVEL costs one env var and prints the failing HIP call with its return code.
• The software path that puts both cards on ONE model without a collective: llama.cpp's layer split. On the old board vLLM could not do it in any of its parallelisms — TP needs an allreduce, PP needs the same p2p primitive, and both died on the card that could not dispatch an RCCL kernel. --split-mode layer in llama.cpp needs no collective: layers are distributed in order and only the boundary activation crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill / 18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly fallback). This was what frank-medium ran until 2026-09-20, when the board swap made collectives legal and radiance TP=2 beat it on every axis (95.8 vs 45.7 t/s decode, 0.21 vs 0.72 s TTFT, 259 vs 72 t/s burst-4) — see frank-medium-radiance-mxfp4; the llama shape is the rollback target frank-medium-llamacpp-dflash2.
• Layer split is also the cross-vendor answer — and -ot is what cannot be combined with it. On Dankclaw (2026-09-16) a CUDA 3090 + Vulkan iGPU layer split ran a real two-device graph: -dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85 held 14,994 MiB on the 3090 and gave 27.2 tok/s with 268.5 t/s prefill at 64k depth (iGPU alone: 25.8 / 231.8). What does NOT work is per-tensor placement across devices: --override-tensor <group>=<device> sets has_tensor_overrides(), a REQUIRED term of the scheduler's pipeline_parallel flag (llama-context.cpp:433 — the only place in the tree that reads it), and that flag is what sets n_copies in ggml_backend_sched_new; with n_copies == 1 a pre-allocated weight on a non-primary device has no backend and the graph aborts (ggml-backend.cpp:941, reported with the leaf's own op name (NONE)). =CPU is the only override target that works, because the CPU backend is last in the list. Pure Vulkan changes nothing here — the gate is the override, not the driver mix. See x4-pair-experiment.
• A host↔3090 link at x4 measures 6.72 GB/s, and that is a design constraint, not a detail. pcie-link-test.cu on Dankclaw (2026-09-16, services stopped): pinned H2D 6.72 GB/s, D2H 6.60 GB/s — 85 % of the Gen4 x4 ceiling — with async no better than sync and pageable within 1 % of pinned, i.e. the copies are link-limited rather than staging-limited (the usual "pageable costs 2x" rule does not apply here). The card is x16-capable and the Strix Halo port beneath it is x8-capable (LnkCap: 16GT/s, Width x8 on 00:03.1), so the x4 lives in the adapter or cable; the link idles at 2.5 GT/s x4 and trains up under load. Consequence for any engine that streams weights across it: a 3 MiB slab costs 474 us, and a 111 GiB A3B model needs ~1.70 GiB of active experts per token ⇒ 3.7 tok/s (4.6 with a 20 % resident cache), against 20.46 tok/s for the same model with the experts computed in host RAM by llama.cpp (-ot exps=CPU, whence the link carries activations only) and 25–26.5 on the iGPU. Streaming over PCIe loses; computing in RAM does not care. FreeToken — expert LRU in host RAM, exactly this streaming shape — also loads only modelopt block-FP8 safetensors for qwen3_5_moe, so a Q4_K_M GGUF is unloadable there before bandwidth is even the question. See x4-pair-experiment.

ROCm / GPU

• Layer split was the deployed shape on both multi-device llama.cpp services, for two different reasons. On Dankserver it was the only way both R9700s served one model (no collective could be dispatched on card #2) until the 2026-09-20 board swap; frank-medium now runs radiance TP=2 instead. On a RCCL-less fork build it is also the fast one: tensor parallel falls back to a butterfly allreduce and collapses prefill ~7x (measured 271.8 vs 1991 t/s). Pin the spec-decode drafter to one device — an unpinned draft under tensor mode aborts the KV build, and on a layer split the natural home is the card that ends up holding the target's last layers.

• **Bundling a ROCm runtime next to a llama.cpp binary: cp -r SRC DST/ is a

  trap.** With a missing DST, cp creates DST as a copy of SRC and drops

  one directory level. rocBLAS and hipBLASLt look for their device code at a

  fixed depth (../lib/rocblas/library, ../lib/hipblaslt/library/<arch>),

  and the flattened copy fails at request time, not load time: the engine

  reads the weights, answers /health and dies on the first completion with

  `Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch

  gfx1201`. Declare the destination, then copy the contents.

• mmap must stay on for the big-PLE models: --no-mmap forces the ~50 GiB PLE table into anonymous memory → silent cgroup kill, dmesg is the only evidence. mmap pages ARE page cache. Never pin the PLE tensor with -ot (23 → 13.4 t/s).
• HIP reads bare env vars TEMP, HOST, GPU_MAX_HEAP_SIZE. Serve scripts export env files with set -a, so a tunable named TEMP leaks into HIP (TEMP=1.0 sent the JIT cache to path 1.0 and segfaulted every load, fixed 2026-08-13 by renaming to TEMPERATURE). Never name a tunable these; check with strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE".
• -DGGML_CUDA_FA_ALL_QUANTS=ON is required for q8_0 KV + flash attention on CUDA builds; without it prefill collapses ~12x (95 vs 1300+ t/s, measured 2026-08-17).
• Flash-Next MTP heads do not load on any current loader — they need full per-layer blk.<il>.hc_attn_norm.weight and ship only blk.48 (verified 3x). Don't re-attempt without new loader support.
• QSA flash-attn assert: GGML_ASSERT(a-type == GGML_TYPE_F32) on qwen4exp needs scripts/patches/qwen4exp-qsa-f32-mask.patch — but only on the agentionai FAST layer set; the kingjones STRIX set works unpatched.
• GPU budget on Dankclaw is the GTT aperture (~114 GiB), not the rocm-smi "dedicated" figure. The Radeon 8060S (gfx1151) reports 116,732 MiB (llama-server --list-devices; gtt_total=122402963456 in /sys/class/drm/card1/device/). The older "64 GiB, SVM-backed" lesson was already stale — the CIRU Orca predecessor held 92.6 GiB of weights on the iGPU. free/dmidecode show the CPU-side carve-out — intentional, do not change.

Load / crash diagnosis

• Model load is slow (minutes cold); health returns 503 while loading; do not kill the process. deploy.sh waits up to 20 minutes (LLM_WAIT). On halogen, the FIRST request after a (re)start can go silent ~45–60 s (health engine_unresponsive, 503s) and then self-recovers — do not restart over it.
• Crash-loop procedure: (1) journalctl -u <svc> -n 100 + kernel line via journalctl -k (segfault at address 8 in libamdhip64 = NULL unref in HIP); (2) run serve script, then raw binary — different outcome isolates the env file; (3) gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-<svc>.sh — llama-server buffers stdout and loses the log tail on SIGSEGV; trust gdb, not the log; (4) amdgpu: SVM mapping failed spam is a symptom, not a root cause.
• Journal signatures: client abort logs W srv stop: cancel task, id_task = N; natural stop logs only slot release with truncated = 0. The server never preempts a running generation; excess tasks queue when slots are busy.

Host memory / swap

• Swap resize ordering: create and swapon the replacement BEFORE swapoff of the old one, or live pages must land in RAM that MemAvailable overstates. Dankclaw swap (64 GiB at /swap.img, fstab-driven) protects the co-tenants, not frank-dragoon — pinned weights don't swap, and paging the KV pool is worse than the OOM it prevents.
• Host swap state is invisible to deploy.sh (no repo file) — check swapon --show before reasoning about headroom.
• A sysfs raise is a countdown to the next reboot. ttm.pages_limit defaults to half of RAM, so the manual raise to 200 GiB (2026-10-05) vanished with the 2026-10-12 23:17 POWER FAILURE on dankserver: the cap fell to 125,988 MiB, frank-archon's pinned-ledger preflight (floor 150,000 MiB) refused the start, and Restart=always re-ran that refusal 6,919 times in ~20 h — a unit in activating (auto-restart) that exits 1 in 50 ms with no container and no engine log is a PREFLIGHT refusal, not an engine fault. Repo law now: scripts/ttm-preflight.sh (sourced by serve-frank-archon.sh) raises the knob to FRANK_ARCHON_TTM_TARGET_MIB itself, re-reads, and refuses by name only after a write the kernel rejected. Never prescribe a manual sysfs write as the recovery procedure for a service that can boot before a human reaches it.
• Silent RAM corruption outranks every software theory. frank-archon's ~40k-token output corruption (2026-10-04/05, repetition/garbage in long agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed by 4600. Consumer non-ECC memory raises no MCE for a silent flip, and a short in-band pattern check (60 s, 8 GiB of 246 GiB) is not a stability proof — combined, those two instruments said "ruled out" while the RAM was the cause. "4800 is the JEDEC floor on 5600-rated DIMMs" is a rating, not a margin guarantee. When the fault is data-shaped (NaN states, garbage tokens) and a software revert doesn't move it, price in the RAM before the runtime — and remember a revert tested under the same marginal hardware is itself confounded. See the runbook's 2026-10-05 section.

Runtime-specific guards

• Rootless podman cannot RAISE rlimits, and --ulimit nofile=1048576 dies at container create, not at runtime. runc reports error setting rlimit type 7: operation not permitted (type 7 = RLIMIT_NOFILE) when the requested hard limit exceeds the host process's hard limit — on Dankclaw that ceiling is 524288. Inherited-equal limits are fine, so memlock=-1:-1 works iff the unit grants LimitMEMLOCK=infinity (systemd is privileged; the value flows through podman to runc without any raise). Upstream halogen's own run lines pass NO --ulimit at all — copy vendor run lines before adding flags (measured 2026-10-05 on the frank-dragoon cutover).
• Kairic Edge (if ever re-deployed): the pinned CK commit needs third_party/composable_kernel/include/ck/config.h generated manually (untracked; git clean removes it), and its greedy-argmax fastpath rejects non-greedy requests and reasoning budgets. Re-test chat before production.
• FastMTP sidecar: build must be commit 4df29be4 + the HauhauCS patch; stock binaries fail the draft load with a tensor-shape mismatch.
• A GGUF's custom tensor type picks the runtime — read the histogram before promising a file to a loader. Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf is ftype 115 with 311 tensors of type 103 (GGML_TYPE_Q8_0_ROCMFPX); stock llama.cpp and gufo refuse it at load, so ROCmFPX was not a preference but the only door. Counting tensor types out of the GGUF KV section takes one python read and settles a runtime argument in minutes. The build identity matters as much as the fork: the 2026-08-25 ROCmFPX builds SIGSEGV in ggml_compute_forward_set_rows on this same file, so scripts/build-rocmfpx.sh pins commit 36e9acd by SHA.
• Speculative-decode flags move as a SET. Handing the ROCmFPX fork --model-draft with no --spec-type (the naive way to build a "no MTP" bench arm) segfaulted at load, core dumped, no diagnostic (2026-10-02). A baseline arm must strip --model-draft AND every --spec-draft-* AND --spec-type, or it measures a crash instead of the absence of drafting.
• An orphaned scratch server taxes the whole host. A bench child left alive by a client-side exception kept ~26 GiB of weights resident on a 121 GiB box, pushed available to 3 GiB, and every later rate was suspect. Record the child pid and kill it in a finally, then prove free came back before believing a number. Kill by pid — never pkill -f llama-server, which also hits frank-zealot.

llama.cpp serving

• MTP is a property of the FILE, not of the flags. The -mtp GGUF build carries the head as blk.<n>.nextn.* (qwen35.nextn_predict_layers = 1); --spec-type draft-mtp then sets load_mtp, and the draft context runs on the MAIN model's weights (shares_model), so there is no second copy of a 12 GiB model in VRAM and no -md argument. The plain build has no MTP tensors at all, and pointing --model-draft at the file would load it twice.
• What can be chained, what cannot, and what a chain costs on a hybrid. --spec-type a,b builds every enabled implementation and chains drafting through them, in a fixed internal order (ngram types first, then the drafters — CLI order is irrelevant). The gate is ctx_dft: draftless types (the ngram family) are added unconditionally, while draft-mtp, draft-dflash, draft-dspark and draft-eagle3 all require a draft CONTEXT. So a chain may hold one context-backed drafter plus any number of ngram types — draft-dflash,ngram-mod is legal and measured (2.6x the drafter alone on prompt-echo traffic, byte-identical to it on prose at the fleet's 48/64 window) — but draft-mtp,draft-dflash dies at load: "context type MTP requested but model doesn't contain MTP layers", because MTP builds its context against whatever model the draft context holds. Two context-backed types cannot share one process, and MTP's head only loads when MTP is the draft model. A LEGAL chain can still lose on a recurrent model: draft-mtp,ngram-mod on frank-dragoon's qwen35 hybrid measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer logs recurrent state rollback is not compatible with 'ngram-mod' - disabling rollback support, and MTP verification depends on that rollback, so acceptance drops on ALL traffic even where the matcher fires. Measure chains per architecture, never by analogy.
• A drafter's KV scales with the TARGET's context, and its window policy decides how much that costs. The draft context is built with cparams.n_ctx = llama_n_ctx(ctx_tgt) (common/speculative.cpp) — there is no flag to cap it — so a drafter that declares NO sliding window caches the whole window on whatever device --spec-draft-device names. Swapping DFlash2 (dflash.attention.sliding_window = 2048, ~40 MiB of draft KV) for DSpark (no window; 5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per position, so 5.0 GiB at 262144) put ROCm1 at 33.9 GB of 34.2 GB and killed the FIRST draft graph capture with ROCm error: out of memory — after the model had loaded and /health had answered. Read sliding_window out of the drafter's GGUF header before sizing any spec-decode deployment, and prefer a skewed -ts over a context cut when the full window matters. The pair was swapped DFlash2 -> DSpark -> DFlash2 in one day (2026-09-16) and the round trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 -> 36:29), which is the practical size of the effect. See frank-medium-llamacpp-dflash2.
• A draft context bills its own f16 KV over the WHOLE POOL, and on a single-card endpoint that squeeze can cost more than the drafter wins. The draft context is built at the target's n_ctx, so with --parallel N the drafter pays the POOL, not the slot: on frank-zealot's 786,432-position qwen35 pool, --spec-type draft-mtp added 3,072.00 MiB of f16 draft KV (4,096 B/position — the draft cache is never quantized) plus its own compute buffers, 4,830 MiB all told. Booted arms, same card, same traffic (2026-10-18): 3 slots + ubatch 2048 → 167.2 t/s decode with spec OFF, 109.2 with spec ON; dropping UBATCH_SIZE to 1024 hands the buffers back and spec ON gives 213.5 (+28 % over the spec-OFF arm), and at 1–2 slots the SAME card runs MTP at ubatch 2048 and still gains ~28 %. So the MTP question is never "does the file carry the head" — it is "what do draft KV
  • buffers cost against whatever the batch knobs keep for compute buffers", and that answer is per-shape. Measure the arm, never the flag. See frank-zealot-llama-prism-bonsai9-mtp.
• --ubatch-size has an OPTIMUM on the 2x R9700 layer split, not a plateau — and the curve is BACKEND-SPECIFIC, so never carry it across (next bullet). Its compute buffers land about TWICE as heavy on the card that holds the output layers and the drafter. Measured on frank-medium 2026-09-16 at the deployed shape (262144 over two slots, a 64k prefill measured at 64k depth, scripts/bench-batch-ubatch.sh): prefill runs 1046 tok/s at ubatch 256, 1068 at 384, 1193 at 512 (the peak), 1135 at 768, 1152 at 1024, 1151 at 1280, 1165 at 1536, 1148 at 1792 and 1078 at 2048. Both ends lose — below 512 the SMs are not saturated, and 2048 is 10 % off the peak — and 512 -> 1536 costs ROCm0 1.30 GiB and ROCm1 2.41 GiB to go 2.4 % SLOWER. The logical --batch-size does not matter: 2048 through 8192 land within 2 % of each other at a fixed ubatch and use identical VRAM. Decode never moves at all (15.88 tok/s at depth 65536 on every cell). Because the buffers are asymmetric, a split balanced at one batch is unbalanced at another: 512 -> 1536 shifted frank-medium's balance by a layer, so 36:29 became 35:30. Re-measure the balance after changing the batch. (8192/2048 failed to allocate its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had 0.08 GiB free. That was a CONTIGUITY failure, not a total-bytes one, and no totals model — the sweep's or the seam's — can see it.)
• On the CUDA 3090 the same knob has NO interior optimum up to 2048, so frank-zealot ships 4096/2048. Measured 2026-10-05 by boot-pair A/B on the live service: at 8,928 prompt tokens ubatch 1024 gives 3,972 / 4,077 t/s and ubatch 2048 gives 4,207 / 4,212 — +4.6 % — for 616 MiB (free 2,849 → 2,279). Decode did not move (91.9–93.0 vs 93.8–94.3, i.e. noise), and a 279-token prompt, which is below either ubatch, came back identical — that is the mechanical proof the knob was the only variable, since a prompt shorter than the ubatch cannot see it. The logical --batch-size was again free. And the predecessor's "ubatch 2048 failed the scratch boot" was a shape observation, not a law: the same file boots 2048 clean at 2 × 200,704. Bench it by restarting the SERVICE through a transient FRANK_ZEALOT_ENV_FILE drop-in, never a scratch port — one card holds one server. See frank-zealot-llama-mimo-q6k-2x200704.
• One -ts step is one LAYER, and a hybrid stack's layers are not the same size. qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one (0.45 GiB of weights plus ~1.0 GiB of its own f16 KV at 262144) — both measured on frank-medium. Balance by measurement, never by interpolating between two splits; an affine fit reproduces the deployed point and overstates the gaps in between. tests/frank_medium_unit.bats encodes that fit as a gate. See frank-medium-llamacpp-dflash2.
• ngram-mod prints no startup banner. The load log shows the drafter's adding speculative implementation 'draft-dflash' and nothing at all for the matcher, so a chain can look unengaged while it is working. Identify it behaviourally (draft_n > 0 only where text repeats) or by the per-task draft acceptance = ... mean len = ... line. It costs no VRAM (a 64 MiB host-RAM hash table) and its window decides which traffic it can see: too short and it proposes on ordinary prose fragments, which costs verification without acceptance.
• --parallel divides the pool, so MORE conversations cost WINDOW, and the per-slot number is the contract — not ctx-size. n_ctx_slot = CTX_SIZE / PARALLEL, and llama.cpp refuses a prompt longer than the SLOT (400 exceed_context_size_error: request (184001 tokens) exceeds the available context size (180224 tokens)) even when the pool has room. So frank-zealot's 2 → 3 reslice moved each conversation from the model's native 262,144 down to 180,224, and models.yml contextWindow had to follow or the harness would discover the ceiling as a 400 on a long conversation. Side effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs 100.4 tok/s) — less attention to scan and one fewer DeltaNet state resident. The 2026-10-03 reslice went the other way (3 → 2 slots, 180,224 → 200,704, pool 401,408) to fund unquantized f16 V, and contextWindow followed again; decode landed 104.8 tok/s — the ~2 % off 106.8 is the exact-V tax. The same day the freed space also bought the Q6_K rung (now 94.4 tok/s — the rung itself is a further measured −10 %, weight-bandwidth, not slot geometry). All moves measured; both slot directions cost or buy window, never pool. See frank-zealot-llama-mimo-q6k-2x200704.
• Price a VRAM budget from two boots, never from a formula — and an mmproj costs ~1.5x its file size. The frank-zealot 3090 fit was solved from two measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒ non-KV 7,390), 3 × 196,608 with the f16 projector = 22,972 used (KV 14,113 ⇒ non-KV 8,859). The difference is the vision tower: a 918 MB file that bills ~1.4 GiB, because the encoder's activation and compute buffers ride with it. Subtracting the two rows is exact; extrapolating weights+KV+ "a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then solve KV ≤ card − non-KV − headroom and round to a multiple of PARALLEL × 4096. Recorded in scripts/frank-zealot.env so the next resize starts from measurement.
• images and image_url are NOT interchangeable: only the content part reaches the model. --mmproj loads, usage.prompt_tokens rises, and the model still invents an answer if the client used the legacy top-level images field on /v1/chat/completions — frank-zealot answered "Brown." for a solid-red PNG that way and "Red." through the OpenAI image_url part. The failure is silent because a hallucinated color is a plausible answer. Probe vision with a solid color and expect an exact word.
• --jinja is what tools needs. On 2026-09-16 a subagent died with 500 tools param requires --jinja flag, from the ik_llama.cpp runtime that then served frank-dragoon: every tool-using agent on that endpoint was broken and the failure read like a model fault. Mainline llama.cpp 0.4.1 defaults --jinja on and the serve script pins it; halogen also accepts tools and returns tool_calls. Check tool support before routing a role at an endpoint.
• Quantized KV on CUDA needs GGML_CUDA_FA_ALL_QUANTS=ON — the same option and the same ~12x prefill collapse described in the ROCm note above. f16 KV needs nothing; frank-zealot ran f16 for that reason until 2026-09-17, when build-cuda was rebuilt with the option ON so the endpoint could take q8_0 KV.
• llama-bench separates a LIST with / and SWEEP values with ,. Both -dev and -ts follow this: -dev CUDA0,Vulkan1 is two single-device configurations, never one two-device run, and -ts 0.85,0.15 is two sweep values each becoming [0.85, 0, …] / [0.15, 0, …], i.e. device 0 takes everything. Write -dev A/B -ts x/y. This artifact produced a wrong fleet conclusion on 2026-09-16 — "CUDA + Vulkan cannot serve one model", with two bogus OOMs that were really "all 111 GiB on the 24 GiB card" — and the bench table's own dev column was printing the truth the whole time. Any placement-dependent number needs an independent check: sample the other device's memory (nvidia-smi) while the run is loaded.
• A model's own template can REJECT the harness's thinking level. Qwen3.8's template accepts only xhigh, medium and low for reasoning_effort and raises on anything else — HTTP 500, not a clamp — so the harness's levels must be folded onto that set in models.yml (compat.reasoningEffortMap). A role whose level is unmapped fails on its FIRST request, and the failure names Jinja, not the model: every subagent on frank-zealot died with Unexpected reasoning effort high until high: xhigh was added on 2026-09-16. Halogen accepts the wider set, so the two entries legitimately differ.
• The vision projector is NOT billed in llama.cpp's own fit check. The engine prints [mtmd] estimated worst-case memory usage of mmproj is 873.10 MiB and then fits WITHOUT it: frank-zealot's Bonsai-2 boot projected 21,439 MiB and settled at 22,664 (2026-10-17). Price a 3090-class shape from settled nvidia-smi after a real boot, never from model + context + compute — the projector (and any drafter context) lands after /health would already have answered. Same failure class as the DSpark drafter OOM.
• The FIRST request after a boot is a warmup artifact. Measured on Bonsai-2/3090: 2.6 t/s prefill and 37.7 t/s decode on request #1, then 1,258 t/s prefill and 54.7 t/s (per stream, two concurrent) on the next — a 3× spread on the same shape. Benchmark the second request or later, or the graph-capture cost reads as a model fault.
• Thinking tokens count against max_tokens, so a short-budget vision probe reads as blindness. The Bonsai-2/image_url probe at max_tokens: 64 returned content: "" with finish_reason: "length" and half a thought in reasoning_content — the image HAD reached the model (322 prompt tokens vs 42 for the text alone). At 512 tokens the same image answers "Red". Budget ≥512 for any thinking model you probe for vision.
• Quantized KV is not the only dtype law a fork build can hide. Vendor builds carry their own compile options: frank-zealot's PrismML prebuilt served --cache-type-v q8_0 under flash attention at the exact 34,816 B/token (the GGML_CUDA_FA_ALL_QUANTS door), so a build decision must be proven by the KV buffer line in the boot ledger, not by the build flags you remember passing.
• A llama.cpp endpoint is invisible to Prometheus unless --metrics is passed. Without it the engine answers 501 Not Implemented on /metrics, so the scrape fails while /health answers 200 and LlmEndpointDown fires beside a healthy service — that is how frank-zealot sat red across four payload generations (the oldest retained sample of up{job="frank-zealot"}, 2026-10-11, is already 0). halogen serves /metrics natively, so the asymmetry reads as "only the 3090 is broken". Diagnose with lastError from /api/v1/targets, not the alert list, and note Alertmanager has no persistence: a reboot re-stamps startsAt on an alert that has been firing for weeks.

Ops

• Never pkill -f a pattern that appears in your own command line — it kills your ssh session. Use pkill -x <exact-name>.
• Both hosts allow passwordless sudo for the deploy user. Use plain sudo in committed scripts (no echo pw | sudo -S); FRANK_SUDO_PW covers hosts without it.
• ComfyUI on Dankclaw runs with GPU access (/dev/kfd + /dev/dri via AddDevice, GroupAdd=keep-groups, plus the udev rule that makes those nodes world-writable — user-unit spawns carry no supplementary groups). It must NOT pass --cpu: the gfx1151 ROCm wheels exist and the unit's --cpu era ended 2026-09-16, when a stale device-less container was found restarted 9,146 times (RuntimeError: No HIP GPUs are available).
• SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its ports off the gluetun service.
• Dankclaw boots iommu=pt since 2026-10-18. The boot line is tracked in dankclaw/host/grub and moves ONLY through scripts/set-dankclaw-bootline.sh (diff, install, update-grub, verify grub.cfg, --reboot) — never by hand-editing /etc/default/grub. It replaced amd_iommu=off (2026-09-11), adopted on the vendor's 13–16 % prefill figure and never measured here: the local A/B on byte-identical traffic says identity mode costs nothing (+2.0 %, 1,616 t/s cold prefill), so the XDNA2 NPU returned for free — amdxdna binds c6:00.1 (PASID width 10, identity group 26) and the old err=-19 only ever asked for an enabled IOMMU. AMD PerfOpt (iommu tree 2026-09-24, Linux 7.4) still skips the page walk outright for the privileged iGPU, but it arms only in the identity domain and 26.04 caps at 7.0.0-38.
• Rootless Quadlet containers run with --replace --rm: podman logs covers only the CURRENT instance; history across recreations lives in journalctl --user -u <name>.service. Read rootless storage WITHOUT sudo (per-user storage — sudo podman logs finds nothing).
• A dead rootless podman pause process breaks ALL container management on the host, and podman system migrate panics in 5.7.0. Symptoms: every podman command fails with invalid internal status … could not find any running process, every container start fails binding podman's bridge addresses (Error starting server failed to bind udp listener on 10.89.x.1:53: Cannot assign requested address), and every Restart=always quadlet loops. Container storage is intact; what is gone is /run/user/<uid>/libpod/tmp/alive (a UUID token, not a pid) and the rootless netns holding the bridges — deleting the stale token is not enough, and podman system migrate segfaults in teardownNetworkBackend. Repair: reboot (/run/user clears and all 28 quadlets return on their own — check Linger=yes and WantedBy=default.target first). Seen 2026-09-16, hours into heavy GPU work on the same host — but the trigger is NOT established (journalctl -k recorded no OOM kill; the kernel had no other word on it). Treat the pause process as host state that can die for its own reasons, and reach for the reboot when it does.
• A container cannot reach a Network=host service at the host's LAN IP. netavark DNATs published ports only: from a bridge container 192.168.0.143:<published> answers while 192.168.0.143:<host bind> and the gateway 10.89.1.1 are both refused. Use host.containers.internal (169.254.1.2). This is what 502'd hermes.franksdank.cloud until the Caddyfile upstream was changed on 2026-09-16 (its vhost comment had claimed the LAN IP was open from that bridge — it never was).
• A *.gguf filter misses mradermacher's payloads: large quants ship as .partNofM fragments. mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF carries Q4_K_M (37.00 + 37.00 + 36.97 = 110.97 GiB) and Q4_K_S (35.00 + 35.00
  • 34.09 = 104.09 GiB) as parts, while the smaller rungs (Q2_K, Q3_K_, IQ3_, IQ4_XS) are single files. Fetch the parts, verify each against the Hub's LFS sha256, concatenate in numeric order — that is how the deployed 111 GiB file was assembled. Listing the Hub tree and filtering endswith(".gguf") makes Q4_K_S look like it does not exist.
• A fetch script must -f and verify, or it reports success on an access error. curl -o FILE URL exits 0 on HTTP 401/403 and writes the error body as the "artifact": a gated repo yielded three 153-byte shards with identical sha256 values and a cheerful DOWNLOAD_DONE in 20 seconds for 104 GiB. Use -f/--fail-with-body, compare each file's byte count to the Hub's declared size, and check the LFS sha256 — scripts/download-frank-medium-gguf.sh already does all three; ad-hoc fetches tend not to. Gated repos (e.g. orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF, same 104.09 GiB Q4_K_S) return 403 even with the host's HF token until that account accepts the gate.

Runbook Index

Active:

Service	Runbook
frank-dragoon (flash-server 0.16.2, NATIVE v2 abliterated .hgn + MTP always-on, 2 slots over ONE 401,408 pool at ctx 200704, vision ON, Dankclaw)	serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-native.md
frank-zealot (llama.cpp PrismML fork build 10777, Bonsai-2-9B PQ2_0 with the MTP head, 3×262,144 q8_0 KV, MTP off by measurement, vision ON via Q8_0 mmproj, Dankclaw)	serverAppConfigs-docs/LLM/frank-zealot-llama-prism-bonsai9-mtp.md
frank-archon (radiance StillDeadcode, publisher Qwen3.8-Flash-Next iq4r .rad, 2×R9700 TP=2, Dankserver)	serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md
Dankclaw kernel posture (iommu=pt since 2026-10-18, was amd_iommu=off)	serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md
Deployment flow + incidents	serverAppConfigs-docs/LLM/deployment.md

Retired (rollback targets and history):

Runtime	Host	Window	Runbook
llama.cpp PrismML fork build 10754, Ternary-Bonsai-2-27B PQ2_0, 2×200,704 q8_0 K+V, MTP off (no head in the file), vision ON (frank-zealot)	Dankclaw	2026-10-17 → 10-18 (rollback target — payload + 10754 fork tree on disk; owns the 34,816 B/token 16-full-layer KV law, the 25,088 MiB f16 OOM wall and the two-boot VRAM pricing)	frank-zealot-llama-prism-bonsai2-pq20.md
radiance Swift1.5-Qwen3.8-Flash-Next GPTQ w4dn8, 3×200,000 fp8-KV slab 25600 (frank-archon)	Dankserver	2026-10-08 → 10-15 (rollback target — .rad purged 2026-10-17, byte-exact re-download frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700; owns the slab-slope ladder, KV dtype law, KLD campaign, and every shape law the iq4r swap carried verbatim)	frank-archon-radiance-swift15-w4dn8.md
llama.cpp CUDA (master) Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off (frank-zealot)	Dankclaw	2026-10-07 → 10-17 (rollback target — payload + master CUDA build on disk; owns the 32 KiB/token 8-full-layer KV law and the 2→3-slot reslice law)	frank-zealot-llama-qwen35-9b-q6k-2x200704.md
llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (frank-zealot)	Dankclaw	2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the attention_k_eq_v pair-billing KV law and the gemma4-assistant drafter-cost boots)	frank-zealot-llama-gemma4-q4km-200704.md
halogen-flash-server 0.16.2 abliterated Swift1.5 v2 .hgn + MTP always-on, TWO slots at ctx 262,144 over a 524,288 pool (frank-dragoon)	Dankclaw	2026-10-06 → 10-17 (rollback target — payload + fetch script on disk; owns the 400-zone law and the sampler/prompt-cache contract)	frank-dragoon-flash-server-v2-swift15.md
ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (frank-dragoon)	Dankclaw	2026-10-03 → 10-05 (rollback target, payload + fork build on disk)	frank-dragoon-rocmfpx-q6agent-uncensored-mtp.md
tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (frank-archon)	Dankserver	2026-09-29 → 10-05 (rollback target — tag frank-archon-vllm-shapeG-20261003; trio + env lore in the tag, image in rootless storage)	dankserver-qwen38-fp8-expert-offload-tuning.md
FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (frank-archon)	Dankserver	2026-10-05 → 10-05 (rollback target — tag frank-archon-freetoken-pr587-20261005; trio + Dockerfile in the tag, image localhost/frank-archon-freetoken:pr587-77a7201 and orcarouter payload on disk)	frank-archon-freetoken-pr587.md
llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (frank-zealot)	Dankclaw	2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced)	frank-zealot-llama-mimo-q6k-2x200704.md
llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (frank-zealot)	Dankclaw	2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk)	trail in frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md
llama.cpp CUDA Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×114,688 draft-mtp q8_0 KV (frank-zealot)	Dankclaw	2026-10-06 → 2026-10-07 (rollback target — payload on disk; owns the 3090 ubatch ladder A/B and the 34 KiB/token qwen35 KV law)	frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md
ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (frank-dragoon)	Dankclaw	2026-10-02 → 10-03 (rollback target, payload on disk)	frank-dragoon-rocmfpx-q8agent-mtp.md
gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (frank-dragoon)	Dankclaw	2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download)	frank-dragoon-gufo-qwen38-27b.md
vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (frank-dragoon)	Dankclaw	2026-10-01 (same-day rollback to gufo)	frank-dragoon-vllm-mimo-v26-bf16.md
llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (frank-zealot)	Dankclaw	2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — frank-zealot-llama-mimo-q6k-2x200704.md)	frank-zealot-llama-mimo-q6k.md
llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (frank-zealot)	Dankclaw	2026-10-02 → 10-03 (rollback target — one MODEL_FILE line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table)	frank-zealot-llama-mimo-q5km.md
SGLang MiMo-V2.6 INT8-W8A16-AutoRound (frank-zealot)	Dankclaw	2026-10-01 (same-day swap to the GGUF path)	frank-zealot-sglang-mimo-v26.md
tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (frank-archon)	Dankserver	2026-09-30 → 10-01	frank-archon-glm53-flash-w4a16.md
llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (frank-dragoon)	Dankclaw	2026-09-21 → 09-28	frank-dragoon-unsloth-qwen38-27b-q8xl.md
R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (frank-archon)	Dankserver	2026-09-27 (same-day swap to Q8_0)	frank-archon-r9v-radiance-qwen38-uncensored.md
llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (frank-archon)	Dankserver	2026-09-26 → 09-27	frank-archon-llamacpp-qwen38-flash-next-q8.md
ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (frank-dragoon)	Dankclaw	2026-09-15 → 09-16	frank-dragoon-ik-uncensored.md
vllm-radiance TP=2 MXFP4 + DFlash2 (frank-medium)	Dankserver	2026-09-20 → 09-26	frank-medium-radiance-mxfp4.md
Lucebox DS4.1 DeepSeek-V4.1-Flash ROCmFP23 (frank-archon)	Dankserver	2026-09-26 (same-day swap)	frank-archon-lucebox-ds4.md
kingjones STRIX Qwen3.8-Flash-Next ROCmFP4	Dankclaw	2026-08-30 → 09-11	kingjones-strix-rocmfp4.md
halogen-flash-server Qwen3.8-Flash-Next w4b (frank-dragoon)	Dankclaw	2026-09-16 → 09-19	halogen-flash-server.md
CIRU v4.4.1 Orca Qwen3.8-Flash (frank-dragoon)	Dankclaw	2026-09-19 → 09-21	frank-dragoon-ciru-orca.md
unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (frank-dragoon)	Dankclaw	2026-09-21 (same-day swap)	frank-dragoon-unsloth-qwen38-flash-next.md
Kairic Edge Qwen3.8-27B IU4 W4A4 (frank-dragoon)	Dankclaw	2026-09-21 (same-day swap)	frank-dragoon-kairic-edge-qwen38-27b.md
ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (frank-zealot)	Dankclaw	2026-09-16 → 09-17	frank-zealot-qwen38-gsq-rco.md
vLLM Gemma4-12B w4a16 (frank-zealot)	Dankclaw	2026-09-19 (same-day revert)	frank-zealot-vllm-gemma4.md
llama.cpp Gemma4-12B QAT Q4_K_M (frank-zealot)	Dankclaw	2026-09-17 → 10-01	frank-zealot-gemma4-12b.md
agentionai ROCmFP4-FAST imatrix	Dankclaw	2026-08-29 → 08-30	agentionai-fast-imatrix.md
eemin Qwen3.8-27B ROCmI4 W4A4 MTP	Dankclaw	2026-08-18 → 08-25	eemin-rocmi4-mtp.md
Kairic Edge IU4	Dankclaw	2026-08-22 (same-day revert)	kairic-edge.md
CIRU ActiveFPX v3 / PromptForge	Dankclaw	through 2026-08-22	ciru-activefpx-promptforge.md
DFlash2 Vulkan	Dankclaw	2026-08-21 (same-day revert)	dflash2-large-vulkan.md
FastFlowLM NPU (frank-npu)	Dankclaw	twice, retired 2026-09-11	frank-npu.md, FastFlowLM NPU Inference.md
FreeToken (official + Huihui FP8, frank-zealot)	Dankserver	2026-09-02 → 09-15	freetoken-frank-small.md
FastMTP patched llama.cpp	Dankserver	retired 2026-08-18	qwen3.8-spec-matrix.md
ik_llama.cpp Ornith CRACK	Dankserver	retired 2026-09-02	ikllama-ornith-dankserver.md
llama.cpp layer split + DFlash2 (frank-medium)	Dankserver	2026-09-16 → 09-20	frank-medium-llamacpp-dflash2.md

Retired runbooks are under serverAppConfigs-docs/LLM/. Retired scripts live in old/ and .AGENTTRASH/.

Why

Direct edits on the remote drift from version control, and future agents won't know what changed. Git is the single source of truth. The two workloads share two hosts, one Caddy, and one Gitea; one repo keeps them in one history.

Summary

Context	Action
Editing a compose/Caddyfile/serve script	Through git — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote.
Config exists only on remote	Pull it into the repo first — commit the fetched version (redacted) before editing.
Unstaged local changes	Commit or discard — check git status before starting work; ask if unsure.
Deploying a committed change	scripts/deploy.sh <target> — pushes, pulls ff-only, plans, dispatches, health-checks.
Container without a Quadlet unit	Give it one — boot-persistent, self-healing .container via git.
Benchmarking	Stop and mask the service first, bench, unmask, restart, verify health.
Deleting a file	Move it to trash — .AGENTTRASH/ (in-repo) or ~/.AGENTTRASH/YYYYMMDD-topic. Never rm.
Tests exist	Run them — verify tests pass after every change. Do not commit failing tests.
Real secrets	Never commit — host .env files; .env.example holds placeholders only.