command-center/agent/memories/sharpshooter/wt-20261017-181502-6d1fc1f-1oq2osmoyw03k/architecture.md

Runtime Stack

llama.cpp is the inference runtime for all frank-zealot-class endpoints. The frank-zealot service runs llama.cpp master (0.5.0-dev, commit 5fc4f3c), CUDA sm_86 build, under a root systemd unit.

Model Inventory

frank-zealot (Dankclaw: 192.168.0.143:7771)

Ternary-Bonsai-2-27B-PQ2_0.gguf (prism-ml/Ternary-Bonsai-2-27B-gguf)

• Runtime: llama.cpp
• Format: PQ2_0 quantized GGUF
• 27B parameter class
• Vision: native BF16 mmproj projector for image input
• MTP: OFF (no nextn tensors in checkpoint)
• Context: 256k per-slot window (--ctx-size 401408 --parallel 2 = 200,704 tokens effective)
• Batch: 4096 / ubatch 2048 (CUDA optimum)
• Log verbosity: 4
• Ports: 7771 on Dankclaw

frank-dragoon (Dankclaw: 192.168.0.143:7770)

Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated (.hgn format)

• Runtime: halogen-flash-server 0.16.2 (digest-pinned ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…)
• Format: v2 .hgn (flash-server exclusive)
• MTP head: INSIDE checkpoint (HALOGEN_MTP_HEAD unset)
• Vision: ON (v2 tower loads beside checkpoint)
• Base: abliterated (27B-era was censored finetune)
• Slots: 2 × 262,144 ctx (KV pool 524,288 positions)
• Server sampler: 1.0/0.95/20
• Prompt cache: ON by default

frank-archon (Dankserver: 192.168.0.235:7770)

StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe (.rad format)

• Runtime: radiance 1.0.13 (digest-pinned stilldeadcode/radiance@sha256:4c3f484564…)
• Format: fp8-KV .rad (121,969,901,568 B = 116,322 MiB)
• MTP head: MTP depth 2, MNBT 2048, Froggeric v22.5 template
• Vision: 27-layer Qwen-VL tower native
• KV: 3 × 200,000 fp8-KV (--max-num-seqs 3)
• 30,720 MiB of 30,969 MiB claimable
• 44 GiB/rank pinned host pool, PLE pinned to RAM
• Base-alignment: censored (abliterated family lives on frank-dragoon)

Resolution Rules

Official model ids are StarCraft unit names, on the wire since 2026-10-09, and since 2026-10-10 they are also the SERVICE names: frank-archon → 192.168.0.235:7770, frank-dragoon → 192.168.0.143:7770, frank-zealot → 192.168.0.143:7771.

The default model resolves as a defaultProvider + defaultModel pair (provider must serve the model); a subagent override uses a bare model id, which resolves only when unique across providers. Two ports on one host need two providers — the file cannot express "same host, different port".

When renaming a service, retiring an endpoint, or moving a model: update the provider block, model id, contextWindow, maxTokens, input in models.yml, then fix every modelRoles entry that named the old endpoint — a role pointing at a provider the file does not define fails at call time. As of 2026-10-06 all three endpoints accept images (input: [text, image]) — frank-dragoon returned to vision with the flash-server v2 swap, while frank-archon takes them natively and frank-zealot through its BF16 mmproj.

Reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3 parser, live-verified) emit reasoning_content, vLLM emits reasoning — models.yml sets reasoningContentField per endpoint to match.

There is no audio input anywhere in the fleet: that capability left with the FLM NPU endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's audio encoder, and left again with the Gemma4 retirement on 2026-10-01.

Boot Persistence & Self-Healing (MANDATORY)

Every deployed, working Podman container MUST have a self-healing, boot-persistent Quadlet unit. A container that runs only from compose.yaml or a manual podman run dies silently at reboot — rootless Podman does not auto-start bare containers, and restart: unless-stopped does not survive the explicit stop that shutdown performs. Never report such a service as "up".

Unit requirements:

• Boot persistence — [Install] WantedBy=default.target, installed to ~/.config/containers/systemd/<name>.container, enabled in the user manager, with linger on for the deploy user (loginctl show-user dank -p Linger → Linger=yes).
• Self-healing — Restart=always (or on-failure) with a short RestartSec; quadlet health check where the service has one.
• Repo provenance — the .container file lives in this repo (<host>/<svc>/quadlet/), is installed by the deployer, and is covered by the quadlet test seam. The host-side unit is generated state; the repo file is the source of truth.

Before declaring a service deployed: systemctl --user is-enabled <name> → enabled, is-active → active, and the container answers its health check.

A running container with no unit is a gap: write the unit, test it, deploy it.

This rule exists because reboots found gaps twice (2026-08-26: Caddy as a bare compose container never returned; 2026-09-11: five observability services had no units). The observability services now have units under dankclaw/observability/quadlet/ and their compose definitions were removed — one owner per container name. The stack is configured and delivering as of 2026-09-17: a Grafana datasource and the llm-fleet dashboard are provisioned from shared/grafana/provisioning/, shared/prometheus/rules/llm-fleet.yml loads 5 alert rules + 1 recording rule, and firing rules are delivered: Prometheus → Alertmanager (dankclaw/observability/quadlet/alertmanager.container, loopback :9093) → host-side Signal relay (scripts/signal-relay.py, user unit, 192.168.0.143:18093 — the host LAN IP, the one bind a container can reach as host.containers.internal; iptables admits loopback, the host and the podman bridge subnet only) → signal-cli account 318311.

Prometheus targets read down for structural reasons (HTML-scraped "probes", Gitea metrics off); the remedy is a blackbox exporter. See serverAppConfigs-docs/Infrastructure/Observability.md.

Benchmarking Policy (MANDATORY)

STOP the production service BEFORE benchmarking — these servers serve real users, and a bench beside a live service starves the GPU and corrupts results. Use the runtime's own harness: llama-bench for the llama.cpp endpoints; a live llama.cpp server also reports honest per-request numbers in the /completion timings object (prefill/decode rates, draft_n, draft_n_accepted), which is how frank-dragoon and frank-zealot were measured. Rolling your own bench requires asking first.

Scratch ports are not used on LLM hosts — a test process cannot share the single GPU anyway; validate against the service itself.

A gufo-era law: gufo bench cannot rank speculative-decode settings, and will invert the ranking.

Measured 2026-09-28 on frank-dragoon at depth 32768: its corpus reports acceptance=0.933 ± 0.000 with an identical trace sha256 across draft widths, i.e. the same highly-predictable text every run, and 93 % of every proposal accepted. It reported 29–37 t/s decode where real traffic gives ~15 t/s, and it made --draft-tokens 16 look 6 % faster than 7 when real text made it 14 % slower. Use it for shape, not for A/B. For an A/B, replay real text through the live server and read the Server-Timing header — but note the unit rides in the name half (ttft;dur=402.485), so splitting on = alone keys the dict ttft;dur and every lookup silently returns nothing. Generate at least 128 tokens for the measurement: at 8 tokens, inter_token is first-token noise and the run-to-run spread swamps every real difference.

This covers any GPU experiment, not just benchmarks — including a bare 2-rank RCCL/torch.distributed probe. On 2026-09-16 one such probe was run against the live service "because it only needs 200 MB of VRAM"; the host went hard down ~20 minutes later (no ARP reply, 100% ICMP loss from two subnets, no recovery — a power cycle was required). A hung AQL dispatch wedging the GPU driver is the plausible mechanism, and this host has precedent for going unreachable until a power cycle. If an experiment needs GPUs, stop and mask first, or use a host that serves nobody. Telling an off machine from a wedged one: ip neigh show <ip> from a peer — FAILED means off/NIC dead, while a merely wedged kernel usually still answers ARP.

1. Stop and mask (self-healing would restart it):
   • Dankclaw frank-dragoon (root unit, halogen podman on the iGPU): sudo systemctl stop frank-dragoon && sudo systemctl mask frank-dragoon
   • Dankclaw frank-zealot (root unit, llama.cpp on the 3090): sudo systemctl stop frank-zealot && sudo systemctl mask frank-zealot
   • Dankserver frank-archon (root unit, radiance on both R9700s, TP=2): sudo systemctl stop frank-archon && sudo systemctl mask frank-archon
2. Bench. NOTE: systemctl mask FAILS on a relinked unit (the unit file in /etc/systemd/system/ is itself a symlink into the checkout — mask refuses to replace it). Stop is enough for the bench window; the deploy restores the unit, so re-deploy after benching instead of unmasking.
3. Unmask and start; verify curl http://192.168.0.<host>:7770/health.

CRITICAL: never systemctl disable a system unit on these hosts — it deletes the unit file from /etc/systemd/system/, not just the symlink. mask symlinks to /dev/null and is safe/reversible.

Subagent Lifecycle (MANDATORY)

A subagent that appears stuck must have an explicit recovery path defined before the subagent is considered deployed. "Stuck" includes: silent completion halt, repeated identical responses, or extended latency without token emission.

Recovery requirements:

• Explicit timeout — a hard timeout (≤30s) on the subagent's response channel, with a defined action (retry, fallback, or escalate).
• State checkpoint — the subagent maintains checkpoints at decision boundaries, allowing resumption from a known good state.
• Health signal — the subagent publishes a heartbeat (≤5s interval) that triggers the recovery path when missed.

A subagent without these three properties is a gap: write the timeout, add the checkpoint, wire the heartbeat, then deploy.

This rule exists because subagent hangs were observed in early deployments (2026-09-xx: silent completion halt; 2026-09-xx: repeated identical responses). The deployment script must verify the heartbeat channel is live before reporting "deployed".

Before declaring a subagent deployed: verify the heartbeat endpoint returns within the timeout window, and confirm the recovery path is wired (retry/fallback/escalation logic exists in the codepath).

A subagent without a heartbeat or recovery path is a gap: wire the heartbeat, add the recovery path, then deploy.