serverAppConfigs/AGENTS.md

# AGENTS.md — serverAppConfigs Agent Rules

## Overview

This repository is the single source of truth for the two homelab servers and
the FrankBook-Pro Mac mini. It tracks the Podman container stacks and the LLM
inference fleet. One repo, one checkout per host, one deploy script. All
services run from `/home/dank/serverAppConfigs` on each host (the 2026-08-22
merge absorbed the former `vllmScripts` repo, history preserved).

- **Dankserver** (192.168.0.235) — media/library stack (RomM, Copyparty,
  Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
  via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
  Alloy, Pi-hole) plus TWO LLM services: `frank-large` (flash-server
  container on the gfx1151 iGPU) and `frank-secondary` (llama.cpp CUDA on the
  RTX 3090). Caddy on
  Dankclaw reverse-proxies both hosts.
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
  (Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
  its podman machine; all egress rides the Gluetun WireGuard netns. Deployed
  via the `frankbook-pro` handler.

## Fleet topology (ACTIVE)

| Target | Service | Runtime | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *…
| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |

Retired endpoints (`frank-npu`, and every prior runtime) are in
the [runbook index](#runbook-index). The full swap history lives in
`serverAppConfigs-docs/LLM/index.md` — do not grow it here.

## Repository Layout

- `dankserver/` — compose.yaml + Caddyfile + .env.example for the media stack
- `dankclaw/` — compose.yaml + Caddyfile + .env.example for the AI stack
- `shared/` — homepage, Prometheus, Loki, Alloy, SearXNG configs
- `scripts/` — `deploy.sh` / `deploy-plan.py` / `deploy-remote.sh`, LLM serve
  scripts, systemd units, env tunables, bench harnesses
- `old/` — retired LLM scripts (reference only)
- `serverAppConfigs-docs/` — knowledge base (Obsidian vault, OKF format).
  The managed skill library is distilled there one page per skill-cluster:
  `LLM/index.md` → *Cross-Cutting References*, plus `Operations/index.md`,
  `Agent/index.md`, and the Infrastructure/Dankclaw/FrankBook-Pro service
  pages (`serverAppConfigs-docs/log.md` carries the wave plan and results)
- `tests/` — bats suite + Playwright e2e

## Git Workflow

All edits flow through git. Never edit files on a remote machine directly.

1. **Source** the live config from the target machine
2. **Commit** the fetched version here (redact secrets into `.env`)
3. **Edit** locally
4. **Test** — `./tests/run.sh` must pass
5. **Commit and push** — origin is Gitea on Dankclaw
6. **Pull** on the target (`deploy.sh` does this)
7. **Deploy** — `scripts/deploy.sh` plans and dispatches only changed services

## Deployment

```
./scripts/deploy.sh dankclaw
./scripts/deploy.sh dankserver
./scripts/deploy.sh dankclaw --service searxng
./scripts/deploy.sh dankserver --service frank-primary
./scripts/deploy.sh dankclaw --dry-run
```

The planner maps changed paths to services; the remote runner restarts only
the selected service and records its revision outside the checkout.
Documentation-only changes are a no-op. LLM services are selected only by
their own files — every endpoint (`frank-large`/`frank-secondary`/`frank-primary`)
by its unit/env/serve trio in `scripts/` — so
unrelated
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
runs on a host without passwordless sudo; the value never enters the repo.

**`deploy.sh <target>` with no `--service` is a fleet-wide DRAIN, not a sync.**
It also walks every per-service revision marker under the deploy-state dir and
restarts each service whose marker is behind HEAD *and* whose paths changed
since that marker — so a backlog accumulated by service-scoped deploys fires
all at once, and one failing health gate aborts the run with every later
service left pending. Name the service unless a drain is what you want
(`./scripts/deploy.sh dankserver --service frank-primary`).

### Deployment completeness (MANDATORY)

A deploy is **not complete** when `deploy.sh` exits green. It requires:

1. `./tests/run.sh` — all seams green, and
2. the Playwright e2e suite (`tests/e2e/run.sh`) — runs **automatically** as
   the `[3/3]` gate at the end of `deploy.sh` against the LIVE fleet; a
   behavior-affecting deploy that fails it exits `DEPLOY INCOMPLETE`. Bypass
   only with `--skip-e2e` in an emergency. The gate runs the conditional
   `frankbook` Playwright project (the `*arr`/qBittorrent walk,
   `tests/e2e/specs/19-frankbook.spec.ts`) ONLY when a frankbook blast-radius
   path changed since origin/main — that spec, `frankbook-pro/**`,
   `shared/homepage/services.json`, or the `dankclaw/Caddyfile` serving the
   arr vhosts — so a sleeping FrankBook-Pro never fails an unrelated deploy.
   The default project set excludes it everywhere else (`E2E_INCLUDE_FRANKBOOK=1`
   opts in; see `serverAppConfigs-docs/Architecture/E2E Suite.md`).

```mermaid
graph LR
    A[Edit locally] --> B[Tests pass] --> C[Commit + push]
    C --> D[deploy.sh plan] --> E[Target pulls ff-only]
    E --> F[Dispatch selected service] --> G[Per-service health]
    G --> H[Playwright e2e gate] --> I{green?}
    I -->|yes| J[Deploy complete]
    I -->|no| K[DEPLOY INCOMPLETE]
```

## Boot Persistence & Self-Healing (MANDATORY)

Every deployed, working Podman container MUST have a self-healing,
boot-persistent Quadlet unit. A container that runs only from `compose.yaml`
or a manual `podman run` dies silently at reboot — rootless Podman does not
auto-start bare containers, and `restart: unless-stopped` does not survive the
explicit stop that shutdown performs. Never report such a service as "up".

Unit requirements:

- **Boot persistence** — `[Install] WantedBy=default.target`, installed to
  `~/.config/containers/systemd/<name>.container`, enabled in the user
  manager, with linger on for the deploy user (`loginctl show-user dank -p
  Linger` → `Linger=yes`).
- **Self-healing** — `Restart=always` (or `on-failure`) with a short
  `RestartSec`; quadlet health check where the service has one.
- **Repo provenance** — the `.container` file lives in this repo
  (`<host>/<svc>/quadlet/`), is installed by the deployer, and is covered by
  the quadlet test seam. The host-side unit is generated state; the repo file
  is the source of truth.

Before declaring a service deployed: `systemctl --user is-enabled <name>` →
`enabled`, `is-active` → `active`, and the container answers its health check.
A running container with no unit is a gap: write the unit, test it, deploy it.

This rule exists because reboots found gaps twice (2026-08-26: Caddy as a
bare compose container never returned; 2026-09-11: five observability
services had no units). The observability services now have units under
`dankclaw/observability/quadlet/` and their compose definitions were removed —
one owner per container name. The stack is **configured and delivering as of
2026-09-17**: a Grafana datasource and the `llm-fleet` dashboard are
provisioned from `shared/grafana/provisioning/`, `shared/prometheus/rules/
llm-fleet.yml` loads 5 alert rules + 1 recording rule, and firing rules are
delivered: Prometheus → Alertmanager (`dankclaw/observability/quadlet/
alertmanager.container`, loopback :9093) → host-side Signal relay
(`scripts/signal-relay.py`, user unit, 192.168.0.143:18093 — the host LAN
IP, the one bind a container can reach as `host.containers.internal`;
iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.
Prometheus targets read down for structural reasons (HTML-scraped "probes",
Gitea metrics off); the remedy is a blackbox exporter. See
`serverAppConfigs-docs/Infrastructure/Observability.md`.

## Testing

Tests are mandatory before commit (`bats` required on the dev machine):

```
./tests/run.sh
```

Seams: compose validity, env consistency (no leaked secrets), Caddyfile
validity, quadlet units, docs completeness **including vault link integrity**,
script syntax (shell + python), LLM unit integrity, LLM secret hygiene,
deployment flow, service-scoped planning, homepage config/pages/deploy/checker,
and `e2e_scaffold` — which fails if a Playwright spec file's number prefix is in
no project, because such a spec silently never runs.
See `serverAppConfigs-docs/Architecture/Testing Policy.md`.

## Secret Rules

- Never commit real secrets. The live Gitea DB password and sudo password are
  test-enforced leak checks.
- `.env` files are git-ignored; commit only `.env.example` with placeholders.
  Exception: `scripts/*.env` LLM tunables (paths and numbers, never
  credentials).
- Do not document passwords anywhere in the repo. SSH keys are the auth
  mechanism; `FRANK_SUDO_PW` covers headless sudo and never enters the repo.
- **A token passed as `curl -H "Authorization: Bearer …"` is world-readable.**
  It lands in `/proc/<pid>/cmdline`, so any `ps`/`pgrep -a` on the host prints
  it — that leaked an HF token into a session transcript on 2026-09-16, while
  merely checking whether a download was still running. Pass secrets out of
  band (`curl -K <mode-600 file>` carrying the header, or `HF_TOKEN` in the
  environment for `hf`/`hf_transfer`) and never `pgrep -a` a fetch command that
  carries an auth header.

## External Exposure Policy (MANDATORY)

Every Caddy-proxied service is either behind the auth chain (Anubis :8923 →
Authentik embedded outpost) or explicitly internal-only — no public DNS name,
no vhost. New services default to internal-only until guarded.

When guarding a new vhost — provider first, then route:

1. Create the outpost proxy provider FIRST (`external_host` = exact vhost URL
   with trailing slash; `internal_host = http://caddy:8017`). If the vhost
   flips first, authenticated traffic hits the outpost error page.
2. Flip the vhost to `reverse_proxy anubis:8923` with
   `header_up X-Real-Ip {remote_host}`, and add the post-auth
   `@<svc> host <vhost>` handler on the `http://:8017` listener routing to
   the real upstream.
3. Anubis guard set, Caddy TLS edge, and outpost provider list change
   together in one deploy.
4. Service-to-service traffic targets internal addresses (`loki:3100`), never
   the public hostname — non-browser clients cannot solve challenges.

Exempt categories (document the reason in the vhost comment): the IdP itself
(`auth.`), git smart-HTTP endpoints, services with their own complete auth.

## Deletion & Trash Policy (MANDATORY)

Never `rm`. Move deletions into trash so they stay recoverable:

1. In-repo files → `.AGENTTRASH/` beside the item's directory.
2. Elsewhere → dated staging dir `~/.AGENTTRASH/YYYYMMDD-topic`.
3. List staged items and sizes; wait for explicit user approval to purge.

`rm`, `rm -rf`, and `git clean -f` are forbidden without explicit permission.
`git * --no-verify` is forbidden. The trash is a temporary hold, not an
archive.

## Benchmarking Policy (MANDATORY)

**STOP the production service BEFORE benchmarking** — these servers serve
real users, and a bench beside a live service starves the GPU and corrupts
results. Use the runtime's own harness: `llama-bench` for the llama.cpp
endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-secondary were measured. Rolling your own bench
requires asking first.
Scratch ports are not used on LLM hosts — a
test process cannot share the single GPU anyway; validate against the service
itself.

(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05, the
halogen 27B container 2026-10-05 → 10-06, and is flash-server since, but the
rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode
settings, and will invert the ranking.**
Measured 2026-09-28 on frank-large at depth 32768: its corpus
reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
draft widths, i.e. the same highly-predictable text every run, and 93 % of
every proposal accepted. It reported 29–37 t/s decode where real traffic
gives ~15 t/s, and it made `--draft-tokens 16` look 6 % faster than 7 when
real text made it 14 % *slower*. Use it for shape, not for A/B. For an A/B,
replay **real** text through the live server and read the `Server-Timing`
header — but note the unit rides in the **name** half (`ttft;dur=402.485`),
so splitting on `=` alone keys the dict `ttft;dur` and every lookup silently
returns nothing. Generate **at least 128 tokens** for the measurement: at 8
tokens, `inter_token` is first-token noise and the run-to-run spread swamps
every real difference.

This covers **any GPU experiment, not just benchmarks** — including a bare
2-rank RCCL/`torch.distributed` probe. On 2026-09-16 one such probe was run
against the live service "because it only needs 200 MB of VRAM"; the host went
hard down ~20 minutes later (no ARP reply, 100% ICMP loss from two subnets, no
recovery — a power cycle was required). A hung AQL dispatch wedging the GPU
driver is the plausible mechanism, and this host has precedent for going
unreachable until a power cycle. If an experiment needs GPUs, stop and mask
first, or use a host that serves nobody. Telling an off machine from a wedged
one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
merely wedged kernel usually still answers ARP.

1. Stop and mask (self-healing would restart it):
   - Dankclaw `frank-large` (root unit, halogen podman on the iGPU):
     `sudo systemctl stop frank-large && sudo systemctl mask frank-large`
   - Dankclaw `frank-secondary` (root unit, llama.cpp on the 3090):
     `sudo systemctl stop frank-secondary && sudo systemctl mask frank-secondary`
  - Dankserver `frank-primary` (root unit, radiance on both R9700s, TP=2):
     `sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
2. Bench.
   NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
   `/etc/systemd/system/` is itself a symlink into the checkout — mask
   refuses to replace it). Stop is enough for the bench window; the deploy
   restores the unit, so re-deploy after benching instead of unmasking.
3. Unmask and start; verify `curl http://192.168.0.<host>:7770/health`.

**CRITICAL:** never `systemctl disable` a system unit on these hosts — it
deletes the unit file from `/etc/systemd/system/`, not just the symlink.
`mask` symlinks to `/dev/null` and is safe/reversible.

## Model Config (harness)

The harness reaches the fleet through `.omp/agent/models.yml` in the
operator's home — `/home/blank/.omp/agent/models.yml` on the dev box, which is
where the harness runs; there is no copy on the servers. Each provider is one
HOST:PORT; each model `id` is the served alias: `frank-primary` →
`192.168.0.235:7770`, `frank-large` → `192.168.0.143:7770`, `frank-secondary` →
`modelRoles`.

Resolution rules: the default model resolves as a `defaultProvider` +
`defaultModel` pair (the provider must serve the model); a subagent override
uses a bare model `id`, which resolves only when unique across providers. Two
ports on one host need two providers — the file cannot express "same host,
different port".

When renaming a service, retiring an endpoint, or moving a model: update the
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
a role pointing at a provider the file does not define fails at call time. As
of 2026-10-06 **all three endpoints accept images** (`input: [text,
image]`) — `frank-large` returned to vision with the flash-server v2 swap
(the v2 vision tower loads beside the checkpoint; the text-only halogen
27B window closed 2026-10-06) — while
`frank-primary` takes them natively (radiance runs the checkpoint's
27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary
are the engine core; `--mm-max-patches` available),
and `frank-secondary` through
the publisher's own BF16 mmproj passed as `--mmproj` — the model is natively
multimodal but the Q6_K GGUF carries no vision tensors (same arrangement the
Gemma4 era used). `frank-secondary`'s `contextWindow` is
**200704** — the PER-SLOT window (`--ctx-size 401408 --parallel 2` since the
2026-10-07 Qwen3.5-9B swap), not the pool and not the model's native 262144:
the harness must compact before llama.cpp answers a longer prompt with a 400.
Note also
that
the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
models.yml sets `reasoningContentField` per endpoint to match.
There is **no audio**
input anywhere in the fleet: that capability left with the FLM NPU
endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
audio encoder, and left again with the Gemma4 retirement on 2026-10-01
(neither the MiMo nor the Qwen3.5 tower carries audio).

### frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, ACTIVE)

The **flash-server engine** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`,
digest-pinned, published tag 0.16.2, 3.6 GB image) as a podman container
driven by `scripts/serve-frank-large.sh` under a **root systemd unit**
(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
`ExecStop=podman stop -t 60 frank-large-halogen`, the script runs `podman
run` FOREGROUND — the container's exit code is the service's exit so
`Restart=always` stays real, and the engine ledger lands on disk) on port
7770 from the Strix Halo iGPU. Engine stdout goes to
`/home/dank/logs/frank_large.log` — **the boot ledger (weights resident, KV
pool, MTP head, prompt cache, any clamp) lives there, not in the journal**.
Cutover 2026-10-06 from the halogen 27B-dense era
([[frank-large-halogen27b-dflash2]], retired the same window).

Serving `Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`:
`qwen38-flash-next-v2-swift15-abliterated.hgn` (66,687,678,464 B ≈ 62.1
GiB, sha256 `9c880b8b…` — a compacted r2 of the card's 89,775,868,032 B
append-only original; the abliteration workflow appends and the repo's
tooling drops dead padding, live tensors byte-identical) + the **REQUIRED**
n-gram lookup table `qwen38-flash-next-ngram.hgn` (51,200,246,144 B,
peonist @5cc17cea) + the vision tower `qwen38-flash-next-vision.hgn`
(897,916,416 B, @e053f488) + the flat `tokenizer/` directory (6 files,
fetched fresh from peonist). Cutover 2026-10-06 from the halogen
27B-dense shape, chosen by Frank for the ABLITERATED base.

**The format is the lock.** The v2 `.hgn` loads ONLY in the
flash-server line — the 27B-era halogen 0.1.4 container refuses it, so the
engine moved with the model. The MTP head is INSIDE the checkpoint
(`HALOGEN_MTP_HEAD` stays unset; the separate `qwen38-flash-next-mtp.hgn`
sidecar is deliberately not fetched).

**Product facts vs the 27B era, both regressions FIXED:** the base is
**abliterated** (the 27B was a censored-base finetune; refusals were an
accepted trade there) and **vision is ON** (the 27B engine silently
ignored its encoder — frank-secondary owned images for exactly one day at this
endpoint).

**Engine laws that shape the config** (all in `scripts/frank-large.env`
with evidence): `HALOGEN_SPEC_ADAPT=0` is MANDATORY — the default adaptive
policy disables the MTP head for whole requests after low acceptance and
HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance
for it (accepted). `HALOGEN_CTX=262144` is the PER-SLOT hard ceiling (400
naming the limit, never truncation) — RAISED from 200192 on 2026-10-07
under the 400-zone law: the harness clamps max_tokens to
`min(45056, contextWindow − est − 4096)`, its chars/4 est UNDER-counts
the rendered prompt by ~3 % (measured: est 152,938 → server 157,388,
request 43,158 vs room 42,804), so a harness window == engine ceiling
made every request over ~146k est a guaranteed 400. `models.yml`
`contextWindow: 217088` leaves the 45k gap that absorbs the estimate
error (worst-case request ≈ 219k ≤ 262144); compaction fires at 85 % of
the harness window = 184,524 est ≈ 190k measured (Frank's 180k target).
`HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter"
law does NOT carry over; batching and speculation coexist (the card's
quickstart ships 2 slots with the head), so 2 is concurrent capacity.
`HALOGEN_KV_POOL_POSITIONS=524288` (explicit 2 × ctx; ~28 KiB/position ⇒
14.0 GiB). `HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the
harness sends at most its `maxTokens` 45056); `HALOGEN_MAX_TOKENS_CAP=65536` +
`HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table).
**A server-side sampler EXISTS** (`HALOGEN_TEMPERATURE/TOP_P/TOP_K` =
1.0/0.95/20, the card's thinking recipe) — an absent temperature is no
longer greedy; `models.yml` extraBody carries the same values, consistent
not load-bearing. Prompt cache is ON by default (`PROMPT_CACHE=2`): a warm
100k follow-up costs ~2 s vs ~88 s cold — the 27B engine's manual
`HALOGEN_CACHE_MB` pin era is over. `response_format` is unsupported.
Thinking lands in **`reasoning_content`**; the v2 chat template rejects
`reasoning_effort` outside `xhigh|medium|low`, so the `models.yml` fold
stays mandatory. Thinking tokens COUNT against the budget (empty content +
`finish_reason: "length"` when exhausted mid-thought). Vision: OpenAI
`image_url` content parts only; ≥256 px probe images or the model answers
"video".

**Memory ledger** (121.2 GiB unified; the referee is the boot ledger):
weights ~62 + KV 14.0 + arena ~8 (`3.8 + 8.4 × MAX_TOK/16384`) + vision
~2 ≈ 86 GiB engine-side + `HALOGEN_HOST_RESERVE_GIB=20` → ~15 GiB
margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd
page cache beside the checkpoint — the card's quickstart arithmetic does
not bill it as resident, but READ the boot ledger and re-size if the
engine prints a clamp.

Fetch: `scripts/download-frank-large-flashnext.sh` into
`/home/dank/models/halogen-flashnext-v2/` (`-f` + exact byte counts + LFS
sha256 + `--verify-only`, no token on the command line; resume goes
against the RESOLVED CDN URL because curl drops `Range` across HF's 302).
The serve gates mirror the deployer EXACTLY (byte-exact checkpoint +
n-gram + vision tower, flat tokenizer with all six files as regular
files) — the 2026-10-01 crash-loop lesson.

Runbook:
`serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md`.
Rollback: revert the swap commit and redeploy — the 27B payload
(`/home/dank/models/halogen-qwen3.8-27b/`), its fetch script and the
halogen 0.1.4 image digest all remain on disk; the ROCmFPX door behind
that (`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched.

### frank-secondary — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)

**llama.cpp master** (0.5.0-dev, commit `5fc4f3c`; CUDA sm_86 build
`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`,
`GGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit**
(`scripts/frank-secondary.service`, `relink_unit`ed into
`/etc/systemd/system/`), serving
`HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive` at **Q6_K**
(7,359,259,008 B, sha256 `c0ba7beb…` computed locally — the Hub carries no
LFS oid) **plus the publisher's own BF16 mmproj** (921,704,448 B), both
`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`, on port 7771. Cutover
2026-10-07 (late) from the Gemma4-26B-A4B Q4_K_M 1×200,704 shape (same day).

**Architecture: qwen35 dense hybrid** — read from the GGUF's own header, not
the model card: 32 layers, `full_attention_interval = 4` ⇒ **8 full-attention
layers (3, 7, 11, …, 31) + 24 Gated DeltaNet linear layers**, 16 attention
heads / **4 KV heads** × 256 dim, `ssm` conv_kernel 4 / state_size 128 /
group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144, vocab
248,320. Tensor types are Q6_K × 250 + F32 × 177 — standard quants only, so
the stock master build loads it (arch `qwen35` is registered in this build's
`src/llama-arch.cpp`). **KV accounting:** the pool scales with the 8 full
layers only — 8 × 4 × 256 = 8,192 elements/token per cache, so f16 K + f16 V
bills **32 KiB/token** = 12.25 GiB at the 401,408 pool; the 24 DeltaNet
layers bill a fixed recurrent state per SLOT, not per token, which is exactly
why two 200k slots fit where the 26B MoE fit one.

**The shape (2026-10-07):** `--ctx-size 401408 --parallel 2` = TWO
conversations of 200,704 (Frank's cap; the harness `contextWindow` quotes
the per-slot number), `--flash-attn on`, **f16 K + f16 V** (exact KV),
batch 4096 / **ubatch 2048** (the measured CUDA optimum: +4.6 % prefill,
decode flat, 616 MiB — MiMo-era A/B on this card), `--log-verbosity 4`,
`--mmproj` for vision, **speculative decoding OFF** (`SPEC_TYPE=none`).

**Why MTP is off**: it is not a choice — **this Q6_K file carries no
`nextn`/`mtp` tensors** (verified from the tensor-name list), and MTP is a
property of the file, not of the flags. The publisher card advertises "MTP
support" for the base architecture; the quantized export dropped the head.

**Measured boots** (nvidia-smi, the referee): the deployed boot is
**21,436 used / 2,689 free** (ubatch 2048) — the pricing boots were the
MiMo-V2.6-Distill-Qwen-9B pair at this exact geometry (**21,276 / 2,849** at
ubatch 1024, **21,846 / 2,279** at 2048), so the 415 MiB lighter weights
bought back what the projection predicted. The engine ledger confirms the KV
law to the MiB: `CUDA0 KV buffer size = 12544.00 MiB` (`K (f16): 6272.00`,
`V (f16): 6272.00`), compute buffers 1,136 MiB on CUDA0 + 848 MiB host, the
mmproj 879 MiB + 248 MiB. Decode is ubatch-invariant (Swift-era 3090 law). The
Q6_K rung costs ~10 % decode vs Q5_K_M (weight bandwidth; MiMo-measured 94.4
vs 104.8 t/s).

**Vision**: the publisher's own BF16 mmproj via `--mmproj` (the model is
natively multimodal; the Q6_K file carries no vision tensors). Probe traps
carry over: ≥256 px probe images or the model answers "video", and only the
OpenAI `image_url` content part reaches the model.

**Compaction hub**: `models.yml` routes `compactionModel` for
`frank-primary`, `frank-large` and `frank-secondary` here — 200,704 is the
only window in the fleet that fits a worst-case compaction input (~170k
tokens). With `--parallel 2` a compaction request no longer serializes behind
a live generation (the Gemma4 era's `--parallel 1` law).

Runbook:
`serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md`.
Rollback: revert the swap commit — the Gemma4 payload
(`/home/dank/models/Gemma4-26B-A4B/`), the MiMo Q6_K payload
(`/home/dank/models/MiMo-Q6K/`), the Swift payload
(`/home/dank/models/Swift15-UncIQ3XXS/`) and the Ornith Q8_0 payload
(`/home/dank/models/Ornith-Q8/`) all remain on disk.


### frank-primary — Radiance (StillDeadcode), Qwen3.8-Flash-Next FP8 native, TP=2 (Dankserver, ACTIVE)

The **radiance engine** (StillDeadcode, Apache-2.0; image
`stilldeadcode/radiance@sha256:4c3f484564bb2872784e4…` digest-pinned,
287 MiB, `libr4d@0.5.0` kernels prebuilt for gfx1201, engine as the image
entrypoint) serving the **official `Qwen/Qwen3.8-Flash-Next-FP8`
checkpoint at NATIVE precision** — modelopt block-FP8 throughout, no
requantisation, no calibration. Plain `podman run` through
`scripts/serve-frank-primary.sh` under the root systemd unit
(`frank-primary.service`, `User=dank`, `LimitMEMLOCK=infinity`); the
container name is the unit name (`frank-primary`). Cutover LIVE 2026-10-06
00:55 (chain built 2026-10-05/06);
rollback tag `frank-primary-freetoken-pr587-20261005` (the FreeToken trio
+ Dockerfile are in the tag; the image and the orcarouter payload stay on
disk until the window closes), older rollback `frank-primary-vllm-shapeG-20261003`.

**Payload chain**: official checkpoint → staging dir whose `config.json`
drops `quantization_config` → `stack-qwen38-official-fp8.py` (two
value-preserving reshapes: experts byte-stacked into the plugin's stacked
`gate_up_proj`/`down_proj` layout with `_scale_inv` at the same index — 49
stacks × 512 experts, no requantisation; PLE's 128 scaleless fp8 shards
decoded to one bf16 shard with the checkpoint's own bf16 scale
1.9931793212890625e-04) → `rad-convert` with ONE `--quant` rule — pure
copy for every weight, and the rtn rule re-encodes the bf16 PLE at that
same fixed scale, an exact round-trip that restores the original codes and
lands the plugin's one-element scale plane →
`/home/dank/models/rad/qwen4exp-fp8-native.rad` (177,109 MiB measured,
floor `FRANK_PRIMARY_RAD_MIN_MIB=177000`). The staging drop is the plugin
SELECTION law (spec §2.4): the descriptor is `(arch_id, quant)` matched
exactly, no shipped plugin claims `(qwen4exp, "fp8")`, the shipped
`qwen4exp_fp8` plugin claims `(qwen4exp, "")` — and the descriptor is only
the selection key, per-weight encodings are data, so nothing else is
requantised.
Raw serve AND unmodified convert both refuse; that is the design. The
checkpoint itself is sha256-verified against the Hub (131/131 shards,
2026-10-05). Built by `scripts/build-frank-primary-rad.sh` (`--plan-only`
first — its declare phase is the engine's own, so a resolving plan boots).

**Shape (both R9700s, TP=2)**: `--tp-wire wht6` (fused lossy all-reduce,
+5.6–6.3 % prefill / ~−1.3 % 8-stream decode at TP2), `--max-num-seqs 2`
(Frank's first boot; raise from the ledger + slot occupancy), `--max-model-len
200000` (native 262,144; the harness contextWindow must quote this — a longer
prompt is refused, never truncated), `--placement expert_tiered
--host-pool-mib 40960` (native FP8 experts are 112.8 GiB total, 56.4 GiB
per rank slice — the pinned pool is the price of native precision),
`--expert-vs-cache-ratio 0.82` (the measured 99 %-hit point; 0.87 collapses
to 0 %), `--kv-cache-dtype fp8` (**7.32 KiB/token**; a 200K sequence is
1.43 GiB), `--num-speculative-tokens 3` (the checkpoint's own MTP head —
3,101 `mtp.*` tensors, probe depth 3), `--max-num-batched-tokens 2048`,
prefix cache 4 GiB pinned host + 64 GiB disk at
`/home/dank/radiance-state`, and **`--ngram-placement ram`**: the 47.7 GiB
PLE n-gram table (51.2 GB = 29 % of all parameters) serves from pinned
host RAM. Froggeric v22.5 chat template vendored
(`scripts/qwen38-froggeric-v22.5.jinja`, `--override-chat-template`).

**The pinned-RAM ledger is why this is a root unit**: 2 × 40 GiB pools +
47.7 GiB PLE + 4 GiB prefix host ≈ **134 GiB** charged against
`RLIMIT_MEMLOCK`. `ttm.pages_limit`/`pages_max` cap GPU-mappable pinned
pages (pre-cutover cap 123 GiB — BELOW the plan); the serve script's
TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to start until
both knobs are raised in the stop window (sysfs write first, kernel
cmdline + reboot only if read-only).

**Contract facts** (source-verified): the HTTP listener binds AFTER the
full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
200 serving / 503 stopping); sampling params temperature…preserve_thinking
accepted, **unknown params 400** (models.yml's extraBody set is fully
inside it); `response_format` supported; `logprobs` REFUSED; reasoning in
`reasoning_content` and tool calls in the qwen3.8 dialect via the auto
chat format — no parser flags; vision native (the 27-layer Qwen-VL tower,
`--mm-max-patches`); `/metrics` carries vLLM-compatible names +
`radiance:*` families — the Dankclaw Prometheus scrape is re-added.

Runbook: `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md`.
Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.

## Key Lessons Learned
Durable operational lessons from the fleet's history. Full detail in the
linked runbooks.

**PCIe / multi-GPU**

- **RCCL collectives need PCIe atomics on every participating GPU.** RCCL's
  kernels carry a hidden hostcall-buffer parameter; ROCr refuses to dispatch
  them when the device has no atomics, and the failure surfaces as
  `hipErrorIllegalState` ("the operation cannot be performed in the present
  state") from `hipLaunchKernel`, not as anything that names atomics. The
  value is consumed at driver probe time, so writing the AtomicOp Requester
  bit with `setpci` at runtime changes nothing. Check the *chain*:
  `lspci -vv -s <gpu>` and every upstream bridge must show
  `AtomicOpsCap: Routing+` (a `Routing-` bridge is a hardware wall), and the
  endpoint's Device Control 2 (`setpci -s <gpu> CAP_EXP+28.w`) must have bit
  6 set. AMD desktop platforms put one card on CPU root ports (atomics work)
  and the rest behind the chipset (usually not) — that asymmetry is what
  blocked frank-medium's TP=2. A one-card collective proves nothing: NCCL
  short-circuits at `world_size=1`.
- **That wall was physical, and it was fixed on 2026-09-20 by a board swap.**
  Both paths that put two cards on CPU root ports were: an M.2→PCIe x4 riser
  into a *CPU-attached* M.2 slot, or a board whose CPU lanes bifurcate to
  x8/x8. dankserver took the second: **ASUS ProArt B850-Creator WiFi NEO**,
  `PCIEX16_1` + `PCIEX16_2` at x8/x8 Gen5 on root ports `00:01.1` / `00:01.3`,
  every bridge now `Routing+` and **both endpoints `ReqEn+`** — the exact
  precondition the old test plan named. Bandwidth was never the constraint (a
  27B TP=2 allreduce at batch 4 is ≈10 MB per token-step, ≈1.4 ms at Gen4 x4
  and under 0.4 ms at Gen5 x8).
  **Proven on the same day.** vLLM's R4D one-shot allreduce runs TP=2 here and
  frank-medium serves on it; llama.cpp's RCCL path was built too
  (`GGML_HIP_RCCL=ON`) and its tensor mode measured — real collective, slower
  shape. Note the collective choice matters: TheRock's RCCL builds lack gfx1201
  device code (`invalid device function`), which is why R4D is the path vLLM
  uses at all. Evidence matrix and the measured three-arm verdict:
  [[Dual-R9700-Cooperation]].
- **When a GPU collective dies, read the runtime, not the framework.** The
  message that identified the cause came from `AMD_LOG_LEVEL=3`
  (`Pcie atomics not enabled, hostcall not supported` → `AQL dispatch
  failed!`), not from vLLM or RCCL. `AMD_LOG_LEVEL` costs one env var and
  prints the failing HIP call with its return code.
- **The software path that puts both cards on ONE model without a collective:
  llama.cpp's layer split.** On the old board vLLM could not do it in any of its
  parallelisms — TP needs an allreduce, PP needs the same p2p primitive, and both
  died on the card that could not dispatch an RCCL kernel. `--split-mode
  layer` in llama.cpp needs no
  collective: layers are distributed in order and only the boundary activation
  crosses PCIe, once per forward pass. Measured here: 1991 t/s prefill /
  18.8 tok/s decode layer-split, against 272 / 26.3 tensor (butterfly
  fallback).
  This was what `frank-medium` ran until 2026-09-20, when the board swap made
  collectives legal and radiance TP=2 beat it on every axis (95.8 vs 45.7 t/s
  decode, 0.21 vs 0.72 s TTFT, 259 vs 72 t/s burst-4) — see
  [[frank-medium-radiance-mxfp4]]; the llama shape is the rollback target
  [[frank-medium-llamacpp-dflash2]].
- **Layer split is also the cross-vendor answer — and `-ot` is what cannot be
  combined with it.** On Dankclaw (2026-09-16) a CUDA 3090 + Vulkan iGPU layer
  split ran a real two-device graph: `-dev CUDA0/Vulkan1 -sm layer -ts 0.15/0.85`
  held 14,994 MiB on the 3090 and gave 27.2 tok/s with 268.5 t/s prefill at 64k
  depth (iGPU alone: 25.8 / 231.8). What does NOT work is per-tensor placement
  across devices: `--override-tensor <group>=<device>` sets
  `has_tensor_overrides()`, a REQUIRED term of the scheduler's
  `pipeline_parallel` flag (llama-context.cpp:433 — the only place in the tree
  that reads it), and that flag is what sets `n_copies` in
  `ggml_backend_sched_new`; with `n_copies == 1` a pre-allocated weight on a
  non-primary device has no backend and the graph aborts (`ggml-backend.cpp:941`,
  reported with the leaf's own op name `(NONE)`). `=CPU` is the only override
  target that works, because the CPU backend is last in the list. Pure Vulkan
  changes nothing here — the gate is the override, not the driver mix. See
  [[x4-pair-experiment]].
- **A host↔3090 link at x4 measures 6.72 GB/s, and that is a design
  constraint, not a detail.** `pcie-link-test.cu` on Dankclaw (2026-09-16,
  services stopped): pinned H2D 6.72 GB/s, D2H 6.60 GB/s — 85 % of the Gen4 x4
  ceiling — with async no better than sync and pageable within 1 % of pinned,
  i.e. the copies are link-limited rather than staging-limited (the usual
  "pageable costs 2x" rule does not apply here). The card is x16-capable and the
  Strix Halo port beneath it is x8-capable (`LnkCap: 16GT/s, Width x8` on
  `00:03.1`), so the x4 lives in the adapter or cable; the link idles at
  2.5 GT/s x4 and trains up under load. Consequence for any engine that
  *streams* weights across it: a 3 MiB slab costs 474 us, and a 111 GiB A3B
  model needs ~1.70 GiB of active experts per token ⇒ **3.7 tok/s** (4.6 with a
  20 % resident cache), against **20.46 tok/s** for the same model with the
  experts *computed* in host RAM by llama.cpp (`-ot exps=CPU`, whence the link
  carries activations only) and 25–26.5 on the iGPU. Streaming over PCIe loses;
  computing in RAM does not care. FreeToken — expert LRU in host RAM, exactly
  this streaming shape — also loads only modelopt block-FP8 safetensors for
  `qwen3_5_moe`, so a Q4_K_M GGUF is unloadable there before bandwidth is even
  the question. See [[x4-pair-experiment]].


**ROCm / GPU**

- **Layer split was the deployed shape on both multi-device llama.cpp services,
  for two different reasons.** On Dankserver it was the *only* way both R9700s
  served one model (no collective could be dispatched on card #2) until the
  2026-09-20 board swap; frank-medium now runs radiance TP=2 instead. On a
  RCCL-less fork build it is also the fast one: tensor parallel falls back to
  a butterfly allreduce and collapses prefill ~7x (measured 271.8 vs 1991 t/s).
  Pin the spec-decode drafter to one device — an unpinned draft under tensor
  mode aborts the KV build, and on a layer split the natural home is the card
  that ends up holding the target's last layers.
- **Bundling a ROCm runtime next to a llama.cpp binary: `cp -r SRC DST/` is a
  trap.** With a missing `DST`, cp creates `DST` as a copy of `SRC` and drops
  one directory level. rocBLAS and hipBLASLt look for their device code at a
  fixed depth (`../lib/rocblas/library`, `../lib/hipblaslt/library/<arch>`),
  and the flattened copy fails at *request* time, not load time: the engine
  reads the weights, answers `/health` and dies on the first completion with
  `Cannot read .../TensileLibrary.dat: No such file or directory for GPU arch
  : gfx1201`. Declare the destination, then copy the contents.
- **mmap must stay on** for the big-PLE models: `--no-mmap` forces the
  ~50 GiB PLE table into anonymous memory → silent cgroup kill, dmesg is the
  only evidence. mmap pages ARE page cache. Never pin the PLE tensor with
  `-ot` (23 → 13.4 t/s).
- **HIP reads bare env vars** `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE`. Serve
  scripts export env files with `set -a`, so a tunable named `TEMP` leaks
  into HIP (`TEMP=1.0` sent the JIT cache to path `1.0` and segfaulted every
  load, fixed 2026-08-13 by renaming to `TEMPERATURE`). Never name a tunable
  these; check with
  `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE"`.
- **`-DGGML_CUDA_FA_ALL_QUANTS=ON` is required** for q8_0 KV + flash
  attention on CUDA builds; without it prefill collapses ~12x (95 vs 1300+
  t/s, measured 2026-08-17).
- **Flash-Next MTP heads do not load on any current loader** — they need full
  per-layer `blk.<il>.hc_attn_norm.weight` and ship only `blk.48` (verified
  3x). Don't re-attempt without new loader support.
- **QSA flash-attn assert:** `GGML_ASSERT(a-type == GGML_TYPE_F32)` on
  qwen4exp needs `scripts/patches/qwen4exp-qsa-f32-mask.patch` — but only on
  the agentionai FAST layer set; the kingjones STRIX set works unpatched.
- **GPU budget on Dankclaw is the GTT aperture (~114 GiB), not the
  rocm-smi "dedicated" figure.** The Radeon 8060S (gfx1151) reports
  116,732 MiB (`llama-server --list-devices`;
  `gtt_total=122402963456` in `/sys/class/drm/card1/device/`). The older
  "64 GiB, SVM-backed" lesson was already stale — the CIRU Orca predecessor
  held 92.6 GiB of weights on the iGPU. `free`/`dmidecode` show the
  CPU-side carve-out — intentional, do not change.

**Load / crash diagnosis**

- Model load is slow (minutes cold); health returns 503 while loading; do
  not kill the process. `deploy.sh` waits up to 20 minutes (`LLM_WAIT`). On
  halogen, the FIRST request after a (re)start can go silent ~45–60 s
  (health `engine_unresponsive`, 503s) and then self-recovers — do not
  restart over it.
- Crash-loop procedure: (1) `journalctl -u <svc> -n 100` + kernel line via
  `journalctl -k` (segfault at address 8 in `libamdhip64` = NULL unref in
  HIP); (2) run serve script, then raw binary — different outcome isolates
  the env file; (3) `gdb -batch -ex run -ex "bt 25" --args bash
  scripts/serve-<svc>.sh` — llama-server buffers stdout and loses the log
  tail on SIGSEGV; trust gdb, not the log; (4) `amdgpu: SVM mapping failed`
  spam is a symptom, not a root cause.
- Journal signatures: client abort logs `W srv stop: cancel task, id_task =
  N`; natural stop logs only `slot release` with `truncated = 0`. The server
  never preempts a running generation; excess tasks queue when slots are
  busy.

**Host memory / swap**

- Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff`
  of the old one, or live pages must land in RAM that `MemAvailable`
  overstates. Dankclaw swap (64 GiB at `/swap.img`, fstab-driven) protects
  the co-tenants, not frank-large — pinned weights don't swap, and paging
  the KV pool is worse than the OOM it prevents.
- Host swap state is invisible to `deploy.sh` (no repo file) — check
  `swapon --show` before reasoning about headroom.
- **Silent RAM corruption outranks every software theory.** frank-primary's
  ~40k-token output corruption (2026-10-04/05, repetition/garbage in long
  agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
  by 4600. Consumer non-ECC memory raises **no MCE** for a silent flip, and
  a short in-band pattern check (60 s, 8 GiB of 246 GiB) is not a stability
  proof — combined, those two instruments said "ruled out" while the RAM
  was the cause. "4800 is the JEDEC floor on 5600-rated DIMMs" is a rating,
  not a margin guarantee. When the fault is data-shaped (NaN states,
  garbage tokens) and a software revert doesn't move it, price in the RAM
  before the runtime — and remember a revert tested under the same marginal
  hardware is itself confounded. See the runbook's 2026-10-05 section.

**Runtime-specific guards**

- **Rootless podman cannot RAISE rlimits, and `--ulimit nofile=1048576`
  dies at container create, not at runtime.** runc reports
  `error setting rlimit type 7: operation not permitted` (type 7 =
  RLIMIT_NOFILE) when the requested hard limit exceeds the host process's
  hard limit — on Dankclaw that ceiling is 524288. Inherited-equal limits
  are fine, so `memlock=-1:-1` works iff the unit grants
  `LimitMEMLOCK=infinity` (systemd is privileged; the value flows through
  podman to runc without any raise). Upstream halogen's own run lines pass
  NO `--ulimit` at all — copy vendor run lines before adding flags
  (measured 2026-10-05 on the frank-large cutover).
- **Kairic Edge** (if ever re-deployed): the pinned CK commit needs
  `third_party/composable_kernel/include/ck/config.h` generated manually
  (untracked; `git clean` removes it), and its greedy-argmax fastpath rejects
  non-greedy requests and reasoning budgets. Re-test chat before production.
- **FastMTP sidecar**: build must be commit `4df29be4` + the HauhauCS patch;
  stock binaries fail the draft load with a tensor-shape mismatch.
- **A GGUF's custom tensor type picks the runtime — read the histogram before
  promising a file to a loader.** `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` is
  ftype 115 with 311 tensors of type 103 (`GGML_TYPE_Q8_0_ROCMFPX`); stock
  llama.cpp and gufo refuse it at load, so ROCmFPX was not a preference but
  the only door. Counting tensor types out of the GGUF KV section takes one
  python read and settles a runtime argument in minutes. The build identity
  matters as much as the fork: the 2026-08-25 ROCmFPX builds SIGSEGV in
  `ggml_compute_forward_set_rows` on this same file, so
  `scripts/build-rocmfpx.sh` pins commit `36e9acd` by SHA.
- **Speculative-decode flags move as a SET.** Handing the ROCmFPX fork
  `--model-draft` with no `--spec-type` (the naive way to build a "no MTP"
  bench arm) segfaulted at load, core dumped, no diagnostic (2026-10-02). A
  baseline arm must strip `--model-draft` AND every `--spec-draft-*` AND
  `--spec-type`, or it measures a crash instead of the absence of drafting.
- **An orphaned scratch server taxes the whole host.** A bench child left
  alive by a client-side exception kept ~26 GiB of weights resident on a
  121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
  Record the child pid and kill it in a `finally`, then prove `free` came back
  before believing a number. Kill by pid — never `pkill -f llama-server`,
  which also hits frank-secondary.

**llama.cpp serving**

- **MTP is a property of the FILE, not of the flags.** The `-mtp` GGUF build
  carries the head as `blk.<n>.nextn.*` (`qwen35.nextn_predict_layers = 1`);
  `--spec-type draft-mtp` then sets `load_mtp`, and the draft context runs on
  the MAIN model's weights (`shares_model`), so there is no second copy of a
  12 GiB model in VRAM and no `-md` argument. The plain build has no MTP
  tensors at all, and pointing `--model-draft` at the file would load it twice.
- **What can be chained, what cannot, and what a chain costs on a hybrid.**
  `--spec-type a,b` builds every enabled implementation and chains drafting
  through them, in a fixed internal order (ngram types first, then the
  drafters — CLI order is irrelevant). The gate is `ctx_dft`: **draftless**
  types (the ngram family) are added unconditionally, while `draft-mtp`,
  `draft-dflash`, `draft-dspark` and `draft-eagle3` all require a draft
  CONTEXT. So a chain may hold one context-backed drafter plus any number of
  ngram types — `draft-dflash,ngram-mod` is legal and measured (2.6x the
  drafter alone on prompt-echo traffic, byte-identical to it on prose at the
  fleet's 48/64 window) — but `draft-mtp,draft-dflash` dies at load: *"context
  type MTP requested but model doesn't contain MTP layers"*, because MTP
  builds its context against whatever model the draft context holds. Two
  context-backed types cannot share one process, and MTP's head only loads
  when MTP **is** the draft model. **A LEGAL chain can still lose on a
  recurrent model:** `draft-mtp,ngram-mod` on frank-large's qwen35 hybrid
  measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer
  logs `recurrent state rollback is not compatible with 'ngram-mod' -
  disabling rollback support`, and MTP verification depends on that rollback,
  so acceptance drops on ALL traffic even where the matcher fires. Measure
  chains per architecture, never by analogy.
- **A drafter's KV scales with the TARGET's context, and its window policy
  decides how much that costs.** The draft context is built with
  `cparams.n_ctx = llama_n_ctx(ctx_tgt)` (`common/speculative.cpp`) — there is
  no flag to cap it — so a drafter that declares NO sliding window caches the
  whole window on whatever device `--spec-draft-device` names. Swapping DFlash2
  (`dflash.attention.sliding_window = 2048`, ~40 MiB of draft KV) for DSpark
  (no window; 5 layers x 2 x 8 kv heads x 128 dim x 2 B = 20 KiB per position,
  so **5.0 GiB at 262144**) put ROCm1 at 33.9 GB of 34.2 GB and killed the
  FIRST draft graph capture with `ROCm error: out of memory` — after the model
  had loaded and `/health` had answered. Read `sliding_window` out of the
  drafter's GGUF header before sizing any spec-decode deployment, and prefer a
  skewed `-ts` over a context cut when the full window matters. The pair was
  swapped DFlash2 -> DSpark -> DFlash2 in one day (2026-09-16) and the round
  trip moved the balance point FOUR layers (~4.3 GiB of headroom, 40:25 ->
  36:29), which is the practical size of the effect. See
  [[frank-medium-llamacpp-dflash2]].
- **`--ubatch-size` has an OPTIMUM on the 2x R9700 layer split, not a plateau —
  and the curve is BACKEND-SPECIFIC, so never carry it across (next bullet). Its
  compute buffers land about TWICE as heavy on the card that holds the output
  layers and the drafter.** Measured on frank-medium 2026-09-16 at the deployed
  shape (262144 over two slots, a 64k prefill measured at 64k depth,
  `scripts/bench-batch-ubatch.sh`): prefill runs 1046 tok/s at ubatch 256, 1068
  at 384, **1193 at 512 (the peak)**, 1135 at 768, 1152 at 1024, 1151 at 1280,
  1165 at 1536, 1148 at 1792 and 1078 at 2048. Both ends lose — below 512 the
  SMs are not saturated, and 2048 is 10 % off the peak — and 512 -> 1536 costs
  ROCm0 1.30 GiB and ROCm1 2.41 GiB to go 2.4 % SLOWER. The logical
  `--batch-size` does not matter: 2048 through 8192 land within 2 % of each
  other at a fixed ubatch and use identical VRAM. Decode never moves at all
  (15.88 tok/s at depth 65536 on every cell). Because the buffers are
  asymmetric, a split balanced at one batch is unbalanced at another: 512 ->
  1536 shifted frank-medium's balance by a layer, so 36:29 became 35:30.
  Re-measure the balance after changing the batch. (8192/2048 failed to allocate
  its 2020 MiB pp buffer here under the DSpark drafter, when ROCm1 had 0.08 GiB
  free. That was a CONTIGUITY failure, not a total-bytes one, and no totals
  model — the sweep's or the seam's — can see it.)
- **On the CUDA 3090 the same knob has NO interior optimum up to 2048, so
  frank-secondary ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the
  live service: at 8,928 prompt tokens ubatch 1024 gives 3,972 / 4,077 t/s and
  ubatch 2048 gives 4,207 / 4,212 — **+4.6 %** — for **616 MiB** (free 2,849 →
  2,279). Decode did not move (91.9–93.0 vs 93.8–94.3, i.e. noise), and a
  279-token prompt, which is below either ubatch, came back identical — that is
  the mechanical proof the knob was the only variable, since a prompt shorter
  than the ubatch cannot see it. The logical `--batch-size` was again free. And
  the predecessor's "ubatch 2048 failed the scratch boot" was a shape
  observation, not a law: the same file boots 2048 clean at 2 × 200,704. Bench
  it by restarting the SERVICE through a transient `FRANK_SECONDARY_ENV_FILE`
  drop-in, never a scratch port — one card holds one server. See
  [[frank-secondary-llama-mimo-q6k-2x200704]].
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
  size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
  worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one
  (0.45 GiB of weights plus ~1.0 GiB of its own f16 KV at 262144) — both
  measured on frank-medium. Balance by measurement, never by interpolating
  between two splits; an affine fit reproduces the deployed point and overstates
  the gaps in between. `tests/frank_medium_unit.bats` encodes that fit as a
  gate. See [[frank-medium-llamacpp-dflash2]].
- **ngram-mod prints no startup banner.** The load log shows the drafter's
  `adding speculative implementation 'draft-dflash'` and nothing at all for the
  matcher, so a chain can look unengaged while it is working. Identify it
  behaviourally (`draft_n` > 0 only where text repeats) or by the per-task
  `draft acceptance = ... mean len = ...` line. It costs no VRAM (a 64 MiB
  host-RAM hash table) and its window decides which traffic it can see: too
  short and it proposes on ordinary prose fragments, which costs verification
  without acceptance.
- **`--parallel` divides the pool, so MORE conversations cost WINDOW, and the
  per-slot number is the contract — not `ctx-size`.** `n_ctx_slot =
  CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT
  (`400 exceed_context_size_error: request (184001 tokens) exceeds the
  available context size (180224 tokens)`) even when the pool has room. So
  frank-secondary's 2 → 3 reslice moved each conversation from the model's native
  262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or
  the harness would discover the ceiling as a 400 on a long conversation. Side
  effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
  100.4 tok/s) — less attention to scan and one fewer DeltaNet state resident.
  The 2026-10-03 reslice went the other way (3 → 2 slots, 180,224 → 200,704,
  pool 401,408) to fund unquantized f16 V, and `contextWindow` followed again;
  decode landed 104.8 tok/s — the ~2 % off 106.8 is the exact-V tax. The same
  day the freed space also bought the Q6_K rung (now 94.4 tok/s — the rung
  itself is a further measured −10 %, weight-bandwidth, not slot geometry).
  All moves measured; both slot directions cost or buy window, never pool.
  See [[frank-secondary-llama-mimo-q6k-2x200704]].
- **Price a VRAM budget from two boots, never from a formula — and an mmproj
  costs ~1.5x its file size.** The frank-secondary 3090 fit was solved from two
  measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
  **non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV
  14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file
  that bills ~1.4 GiB, because the encoder's activation and compute buffers
  ride with it. Subtracting the two rows is exact; extrapolating weights+KV+
  "a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then
  solve `KV ≤ card − non-KV − headroom` and round to a multiple of
  `PARALLEL × 4096`. Recorded in `scripts/frank-secondary.env` so the next resize
  starts from measurement.
- **`images` and `image_url` are NOT interchangeable: only the content part
  reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
  model still invents an answer if the client used the legacy top-level
  `images` field on `/v1/chat/completions` — frank-secondary answered "Brown." for
  a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
  failure is silent because a hallucinated color is a plausible answer. Probe
  vision with a solid color and expect an exact word.
- **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
  `500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
  then served frank-large: every tool-using agent on that endpoint was broken
  and the failure read like a model fault. Mainline llama.cpp 0.4.1 defaults
  `--jinja` on and the serve script pins it; halogen also accepts `tools` and
  returns `tool_calls`. Check tool support before routing a role at an endpoint.
- **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same
  option and the same ~12x prefill collapse described in the ROCm note above.
  f16 KV needs nothing; frank-secondary ran f16 for that reason until
  2026-09-17, when `build-cuda` was rebuilt with the option ON so the
  endpoint could take q8_0 KV.
- **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both
  `-dev` and `-ts` follow this: `-dev CUDA0,Vulkan1` is two *single-device*
  configurations, never one two-device run, and `-ts 0.85,0.15` is two sweep
  values each becoming `[0.85, 0, …]` / `[0.15, 0, …]`, i.e. device 0 takes
  everything. Write `-dev A/B -ts x/y`. This artifact produced a wrong fleet
  conclusion on 2026-09-16 — "CUDA + Vulkan cannot serve one model", with two
  bogus OOMs that were really "all 111 GiB on the 24 GiB card" — and the bench
  table's own `dev` column was printing the truth the whole time. Any
  placement-dependent number needs an independent check: sample the other
  device's memory (`nvidia-smi`) while the run is loaded.
- **A model's own template can REJECT the harness's thinking level.** Qwen3.8's
  template accepts only `xhigh`, `medium` and `low` for `reasoning_effort` and
  raises on anything else — HTTP 500, not a clamp — so the harness's levels
  must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`).
  A role whose level is unmapped fails on its FIRST request, and the failure
  names Jinja, not the model: every subagent on `frank-secondary` died with
  `Unexpected reasoning effort high` until `high: xhigh` was added on
  2026-09-16. Halogen accepts the wider set, so the two entries legitimately
  differ.

**Ops**

- Never `pkill -f` a pattern that appears in your own command line — it kills
  your ssh session. Use `pkill -x <exact-name>`.
- Both hosts allow passwordless sudo for the deploy user. Use plain `sudo` in
  committed scripts (no `echo pw | sudo -S`); `FRANK_SUDO_PW` covers hosts
  without it.
- ComfyUI on Dankclaw runs with **GPU access** (`/dev/kfd` + `/dev/dri` via
  `AddDevice`, `GroupAdd=keep-groups`, plus the udev rule that makes those nodes
  world-writable — user-unit spawns carry no supplementary groups). It must NOT
  pass `--cpu`: the gfx1151 ROCm wheels exist and the unit's `--cpu` era ended
  2026-09-16, when a stale device-less container was found restarted 9,146
  times (`RuntimeError: No HIP GPUs are available`).
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its
  ports off the gluetun service.
- `amd_iommu=off` on Dankclaw (bought 13–16 % prefill for frank-large,
  2026-09-11) permanently retired the XDNA2 NPU: the driver's SVA/PASID bind
  needs an enabled IOMMU (`/dev/accel/accel0` fails `err=-19`). Revert path
  in `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md`.
- Rootless Quadlet containers run with `--replace --rm`: `podman logs` covers
  only the CURRENT instance; history across recreations lives in
  `journalctl --user -u <name>.service`. Read rootless storage WITHOUT sudo
  (per-user storage — `sudo podman logs` finds nothing).
- **A dead rootless podman pause process breaks ALL container management on the
  host, and `podman system migrate` panics in 5.7.0.** Symptoms: every `podman`
  command fails with `invalid internal status … could not find any running
  process`, every container start fails binding podman's bridge addresses
  (`Error starting server failed to bind udp listener on 10.89.x.1:53: Cannot
  assign requested address`), and every `Restart=always` quadlet loops. Container
  storage is intact; what is gone is `/run/user/<uid>/libpod/tmp/alive` (a UUID
  token, not a pid) and the rootless netns holding the bridges — deleting the
  stale token is not enough, and `podman system migrate` segfaults in
  `teardownNetworkBackend`. Repair: **reboot** (`/run/user` clears and all 28
  quadlets return on their own — check `Linger=yes` and `WantedBy=default.target`
  first). Seen 2026-09-16, hours into heavy GPU work on the same host — but the
  trigger is NOT established (`journalctl -k` recorded no OOM kill; the kernel
  had no other word on it). Treat the pause process as host state that can die
  for its own reasons, and reach for the reboot when it does.
- **A container cannot reach a `Network=host` service at the host's LAN IP.**
  netavark DNATs *published* ports only: from a bridge container
  `192.168.0.143:<published>` answers while `192.168.0.143:<host bind>` and the
  gateway `10.89.1.1` are both refused. Use `host.containers.internal`
  (169.254.1.2). This is what 502'd `hermes.franksdank.cloud` until the Caddyfile
  upstream was changed on 2026-09-16 (its vhost comment had claimed the LAN IP
  was open from that bridge — it never was).
- **A `*.gguf` filter misses mradermacher's payloads: large quants ship as
  `.partNofM` fragments.** `mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF`
  carries Q4_K_M (37.00 + 37.00 + 36.97 = 110.97 GiB) and Q4_K_S (35.00 + 35.00
  + 34.09 = 104.09 GiB) as parts, while the smaller rungs (Q2_K, Q3_K_*, IQ3_*,
  IQ4_XS) are single files. Fetch the parts, verify each against the Hub's LFS
  sha256, concatenate in numeric order — that is how the deployed 111 GiB file
  was assembled. Listing the Hub tree and filtering `endswith(".gguf")` makes
  Q4_K_S look like it does not exist.
- **A fetch script must `-f` and verify, or it reports success on an access
  error.** `curl -o FILE URL` exits 0 on HTTP 401/403 and writes the error body
  as the "artifact": a gated repo yielded three 153-byte shards with identical
  sha256 values and a cheerful `DOWNLOAD_DONE` in 20 seconds for 104 GiB. Use
  `-f`/`--fail-with-body`, compare each file's byte count to the Hub's declared
  size, and check the LFS sha256 — `scripts/download-frank-medium-gguf.sh`
  already does all three; ad-hoc fetches tend not to. Gated repos (e.g.
  `orcarouter/Qwen3.8-Flash-Next-Uncensored-GGUF`, same 104.09 GiB Q4_K_S)
  return 403 even with the host's HF token until that account accepts the gate.

## Runbook Index

Active:

| Service | Runbook |
|---------|---------|
| frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×262,144, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` |
| frank-secondary (llama.cpp CUDA, Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md` |
| frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` |
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |

Retired (rollback targets and history):

| Runtime | Host | Window | Runbook |
|---------|------|--------|---------|
| llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-secondary`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-secondary-llama-gemma4-q4km-200704.md` |
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` |
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-secondary-llama-mimo-q6k-2x200704.md` |
| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` |
| llama.cpp CUDA Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×114,688 draft-mtp q8_0 KV (`frank-secondary`) | Dankclaw | 2026-10-06 → 2026-10-07 (rollback target — payload on disk; owns the 3090 ubatch ladder A/B and the 34 KiB/token qwen35 KV law) | `frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` |
| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` |
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` |
| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-secondary`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-secondary-llama-mimo-q6k-2x200704.md`) | `frank-secondary-llama-mimo-q6k.md` |
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-secondary-llama-mimo-q5km.md` |
| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-secondary`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-secondary-sglang-mimo-v26.md` |
|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
| R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
| llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-primary`) | Dankserver | 2026-09-26 → 09-27 | `frank-primary-llamacpp-qwen38-flash-next-q8.md` |
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
| vllm-radiance TP=2 MXFP4 + DFlash2 (`frank-medium`) | Dankserver | 2026-09-20 → 09-26 | `frank-medium-radiance-mxfp4.md` |
| Lucebox DS4.1 DeepSeek-V4.1-Flash ROCmFP23 (`frank-primary`) | Dankserver | 2026-09-26 (same-day swap) | `frank-primary-lucebox-ds4.md` |
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` |
| unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` |
| Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` |
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-secondary`) | Dankclaw | 2026-09-16 → 09-17 | `frank-secondary-qwen38-gsq-rco.md` |
| vLLM Gemma4-12B w4a16 (`frank-secondary`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-secondary-vllm-gemma4.md` |
| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-secondary`) | Dankclaw | 2026-09-17 → 10-01 | `frank-secondary-gemma4-12b.md` |
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
| CIRU ActiveFPX v3 / PromptForge | Dankclaw | through 2026-08-22 | `ciru-activefpx-promptforge.md` |
| DFlash2 Vulkan | Dankclaw | 2026-08-21 (same-day revert) | `dflash2-large-vulkan.md` |
| FastFlowLM NPU (`frank-npu`) | Dankclaw | twice, retired 2026-09-11 | `frank-npu.md`, `FastFlowLM NPU Inference.md` |
| FreeToken (official + Huihui FP8, `frank-secondary`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
| FastMTP patched llama.cpp | Dankserver | retired 2026-08-18 | `qwen3.8-spec-matrix.md` |
| ik_llama.cpp Ornith CRACK | Dankserver | retired 2026-09-02 | `ikllama-ornith-dankserver.md` |
| llama.cpp layer split + DFlash2 (`frank-medium`) | Dankserver | 2026-09-16 → 09-20 | `frank-medium-llamacpp-dflash2.md` |

Retired runbooks are under `serverAppConfigs-docs/LLM/`. Retired scripts live
in `old/` and `.AGENTTRASH/`.

## Why

Direct edits on the remote drift from version control, and future agents
won't know what changed. Git is the single source of truth. The two workloads
share two hosts, one Caddy, and one Gitea; one repo keeps them in one
history.

## Summary

| Context | Action |
|---------|--------|
| Editing a compose/Caddyfile/serve script | **Through git** — fetch, commit, edit, test, push, pull, deploy. Never edit on the remote. |
| Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before editing. |
| Unstaged local changes | **Commit or discard** — check `git status` before starting work; ask if unsure. |
| Deploying a committed change | **`scripts/deploy.sh <target>`** — pushes, pulls ff-only, plans, dispatches, health-checks. |
| Container without a Quadlet unit | **Give it one** — boot-persistent, self-healing `.container` via git. |
| Benchmarking | **Stop and mask the service first**, bench, unmask, restart, verify health. |
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
| Real secrets | **Never commit** — host `.env` files; `.env.example` holds placeholders only. |