Files

T

Frank e7f0d93c86 docs(vault): fleet-wide fact re-verification pass (67 pages)

The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.

Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.

2026-10-18 13:15:43 -07:00

7.6 KiB

Raw Blame History

title, service, host, port, status, retired, superseded_by, supersedes, tags

title

service

host

port

status

retired

superseded_by

supersedes

frank-archon — R9V radiance-vLLM (dankserver, RETIRED)

RETIRED 2026-09-27 — superseded the same day by the Q8_0 cutover, frank-archon-r9v-radiance-qwen38-q8 (itself retired 2026-09-30; the live frank-archon runbook today, 2026-10-18, is frank-archon-radiance-qwen38-iq4r). The uncensored state dir and frozen config are still on disk (/home/dank/r9v-state/uncensored/ — setup.json, restore-config-20260927.env); the IQ4_XS payload is not — it was deleted in the 2026-10-17 cleanup. The rollback procedure lives in the Q8_0 page's Ops section, with a re-fetch as step one.

The Dankserver LLM endpoint was the R9V radiance-vLLM fork (drwolfen/radiance-vllm-qwen4exp, docker-pinned prebuilt) serving Dyluhn/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS — the abliterated Qwen3.8-Flash-Next at IQ4_XS (92.39 GiB, 3 GGUF shards) with the blockfp8 MTP4 head and the F16 mmproj — on both R9700s (gfx1201, BDFs 0000:03:00.0 / 0000:07:00.0) as frank-primary on :7770 (the frank-archon wire alias arrived 2026-10-09).

Replaced llama.cpp b11030 (Qwen3.8-Flash-Next Q_8) on 2026-09-27 at Frank's direction; downtime window ~1 h 20 min (killed 02:47, healthy 04:11 PDT). Rollback target: frank-archon-llamacpp-qwen38-flash-next-q8 — the llama.cpp trio (unit/env/serve) is at commit 4992e5a+ (the repo-side wiring; host unit was the relinked one); the GGUF payload is no longer on disk (Dankserver's /home/dank/models/ holds only StillDeadcode, rad, tpl.jinja as of 2026-10-18), so that rollback is a re-fetch plus a redeploy.

Production shape (repo-tracked)

The repo carries a thin wiring; the engine shape lives on-host:

Piece	File	Law
Unit	`scripts/frank-archon.service`	Root unit, `User=dank`, `LimitMEMLOCK=infinity`, `ExecStop=podman stop -t 120 frank-archon-r9v`, `Restart=on-failure`
Env	`scripts/frank-archon.env`	Production overrides ONLY: bind 0.0.0.0, :7770, alias `frank-primary` (the wire id became `frank-archon` on 2026-10-09), container `frank-archon-r9v`, GPU BDFs
Serve	`scripts/serve-frank-archon.sh`	Layers setup state under repo env → execs the fork's own `scripts/launch.sh` via the docker→podman shim → `docker wait`
Deployer	`scripts/deploy-remote.sh` `deploy_llm_archon`	Gates on setup state ready, image digest, PLE bytes, shard files; MainPID false-green check

Why the tuned shape lives on-host: the profile's setup run qualified a 104-key R9V_* config (tiered-expert placement, per-rank offload budgets 146.25/78.75 GiB, CED on, MTP4, parser wiring, MAX_MODEL_LEN=196608). It is written to /home/dank/r9v-state/uncensored/setup.json and the serve script layers it UNDER the repo env (repo keys win). Copying those values into the repo would drift from the qualification — don't.

Self-healing law: the launcher --detaches; the serve script therefore blocks on docker wait. Without the wait, a crashed engine is a green unit. docker wait cannot stop the container child, hence ExecStop.

Fork checkout + shim (on-host, outside the repo)

Checkout: /home/dank/GitPackages/radiance-vllm-qwen4exp
State: /home/dank/r9v-state/uncensored/setup.json (0600, ready: true)
Model dir: /home/dank/models/Qwen3.8-Flash-Next-Uncensored-R9V-IQ4_XS/ (92.39 GiB payload + r9v-data/per_layer_token_embd.iq4_nl.bin 26.82 GiB PLE, sha256 34fa36f8…)
Docker→podman shim: /home/dank/GitPackages/radiance-vllm-qwen4exp/.podman-shim/docker (PATH-prepended by the serve script). The R9V tooling is docker-pinned; the shim carries three laws:
1. Image ID law — the bundle is a containerd/OCI docker-save; docker keys the image by its OCI MANIFEST digest (sha256:2dac17a2…), podman by config digest but preserves the manifest digest in RepoDigests. The shim answers image inspect sha256:X from RepoDigests and rewrites bare sha256:X run refs to repo@sha256:X.
2. LogConfig law — podman inspect answers HostConfig.LogConfig.Config: null where docker returns the log-opts map (the doctor crashes on .get("Config", {}) == null). The shim records run --log-opt pairs and normalizes inspect output.
3. docker info law — podman lacks SecurityOptions/ServerVersion; the shim answers rootless + 27.5.1 (liveness probe).
venv: ~/.r9v-venv (huggingface_hub for fetch/verify)

Verified at cutover (2026-09-27)

Setup: PASS=28 WARN=3 FAIL=0 — 20/20 artifacts size+sha256 verified (~93 GiB), PLE derived at exact pinned size, CED projector verified.
First-start doctor battery: PASS=34 WARN=8 FAIL=0, including runtime-worker-transport: pinned UVA reads, H2D copies, TP all-reduce verified inside serving workers on both BDFs.
Wire: /v1/models → frank-archon; /health + /metrics 200 (Prometheus scraping); "Say PONG" (non-thinking) → 'PONG', stop; thinking requests populate reasoning (vLLM channel).
Memlock: the full tiered-expert load ran to healthy under a plain dank session before the unit existed — the pin path does NOT hard-require LimitMEMLOCK=infinity on this host, but the root unit keeps fleet parity and headroom for the pinned reserve.
e2e gate: 32 passed / 1 failed — the failure is 11-admin open links, which needs FrankBook-Pro (192.168.0.112) awake; the machine was down and WOL from both hosts failed. Suite minus that spec: 31 passed (the one-time 12-crawl flake re-passes in isolation).

Memory shape (measured)

Both cards ~29–30 GiB used of 31.86; engine budget CPU_OFFLOAD_GB_BY_DEVICE=146.25,78.75 (UVA host pin ≈ 43 GiB on rank 0
- 16 GiB PLE pinned reserve; PLE residency mode ssd mmap+trim).
MemAvailable floor: 56.3 GiB (R9V_MIN_HOST_AVAILABLE_BYTES).
Context envelope: 192,608 native window served as MAX_MODEL_LEN=196608; harness models.yml carries contextWindow: 196608, maxTokens: 57344 (compaction-gap math: 70% × 196,608 = 137,625 + 57,344 ≤ 196,608).

Ops

Restart: sudo systemctl restart frank-archon (root unit; the serve script re-gates everything, relaunch through the shim, waits).
First start after boot re-warms ~93 GiB of page cache; expect several minutes to /health; TimeoutStartSec=1800.
Scratch/probe: NEVER probe against the prod state dir. Copy it, set R9V_HOST_PORT to a free port and R9V_CONTAINER_NAME to a scratch name, and stop the probe before any deploy — a probe holding :7770 produces a false-green deploy (the 2026-09-26 lesson, now guarded by the MainPID check).
Qualification artifacts (kept on-host): /home/dank/r9v-state/uncensored/qualification-*, failure bundles failure-* (review before sharing — raw logs).
Models on the wire: served alias frank-archon; thinking channel is reasoning (vLLM), tool parser qwen3_coder (not qwen3_xml like the vllm030 wiring had planned).

Known costs (accepted at selection)

Abliterated weights (Dyluhn uncensored build) — not the reference model; answer-quality deltas are possible.
192K envelope — down from 262,144 on llama.cpp.
Docker-pinned fork — updates flow through image bundles + the shim, never a bare container run; the launcher stays the single argv owner.

7.6 KiB Raw Blame History Unescape Escape