Compare commits
15
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
2e86445fc5 | ||
|
|
cf6319aad7 | ||
|
|
a9d45e7012 | ||
|
|
2581b8d4b6 | ||
|
|
09a0251d2d | ||
|
|
066813ea95 | ||
|
|
7daa886dd6 | ||
|
|
bae78e7dae | ||
|
|
a6f55f4cc6 | ||
|
|
a9da6589a6 | ||
|
|
1a6a78ef73 | ||
|
|
c52ba781e3 | ||
|
|
0bf26b0741 | ||
|
|
394fd8535b | ||
|
|
55f87fcdce |
@@ -1,4 +1,4 @@
|
||||
# Tunables for frank-large (kingjones777/Qwen3.8-Flash-Next-ROCmFP4-
|
||||
# Tunables for frank-dragoon (kingjones777/Qwen3.8-Flash-Next-ROCmFP4-
|
||||
# STRIX-GGUF on the kingjones30/ROCmFPX llama.cpp runtime, Dankclaw,
|
||||
# Strix Halo gfx1151, unified memory).
|
||||
#
|
||||
@@ -56,7 +56,7 @@ CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
|
||||
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}'
|
||||
|
||||
# Alias + network.
|
||||
ALIAS=frank-large
|
||||
ALIAS=frank-dragoon
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7770
|
||||
|
||||
+3
-3
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=ROCmFPX (kingjones30 fork) llama.cpp Qwen3.8-Flash-Next ROCmFP4 STRIX 177B MoE (dankclaw, frank-large)
|
||||
Description=ROCmFPX (kingjones30 fork) llama.cpp Qwen3.8-Flash-Next ROCmFP4 STRIX 177B MoE (dankclaw, frank-dragoon)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
@@ -7,8 +7,8 @@ Wants=network-online.target
|
||||
Type=simple
|
||||
User=dank
|
||||
Group=dank
|
||||
Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
|
||||
Environment="FRANK_DRAGOON_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-dragoon.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-dragoon.sh
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
# is reachable on this port regardless of the boot tag. The NPU is a
|
||||
# single-lane device: generations serialize FIFO, and the first request after a
|
||||
# model swap pays the ~6-11 s load cost. The NPU draws from the unified RAM
|
||||
# pool shared with frank-large (iGPU).
|
||||
# pool shared with frank-dragoon (iGPU).
|
||||
FLM_MODEL=nuextract3:4b
|
||||
FLM_HOST=0.0.0.0
|
||||
FLM_PORT=7771
|
||||
|
||||
+3
-3
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF as
|
||||
# frank-large via the kingjones30/ROCmFPX llama.cpp runtime on
|
||||
# frank-dragoon via the kingjones30/ROCmFPX llama.cpp runtime on
|
||||
# Dankclaw (Strix Halo gfx1151, unified memory).
|
||||
#
|
||||
# Runtime: kingjones30/ROCmFPX (fork of charlie12345/ROCmFPX with the
|
||||
@@ -29,7 +29,7 @@
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
|
||||
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-dragoon.env}"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
@@ -44,7 +44,7 @@ fi
|
||||
: "${MMPROJ_PATH:=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf}"
|
||||
: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
|
||||
: "${CHAT_TEMPLATE_KWARGS:={}}"
|
||||
: "${ALIAS:=frank-large}"
|
||||
: "${ALIAS:=frank-dragoon}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=7770}"
|
||||
: "${CTX_SIZE:=401408}"
|
||||
@@ -4,7 +4,7 @@
|
||||
# Runs `flm serve <tag>` on the XDNA2 NPU (50 TOPS, Dankclaw). The NPU is a
|
||||
# single-lane device: generations serialize in FIFO order, so this is a
|
||||
# background/triage endpoint, not an interactive multi-user chat. The NPU
|
||||
# draws from the unified RAM pool shared with frank-large (the iGPU
|
||||
# draws from the unified RAM pool shared with frank-dragoon (the iGPU
|
||||
# llama.cpp service); see serverAppConfigs-docs/LLM/frank-npu.md
|
||||
# for the memory budget and the model-selection tradeoff.
|
||||
#
|
||||
|
||||
@@ -9,11 +9,11 @@ services run from `/home/dank/serverAppConfigs` on each host (the 2026-08-22
|
||||
merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
- **Dankserver** (192.168.0.235) — media/library stack (RomM, Copyparty,
|
||||
Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
|
||||
Calibre-Web, Jellyfin, Plex) plus the `frank-archon` LLM service.
|
||||
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
|
||||
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (flash-server
|
||||
container on the gfx1151 iGPU) and `frank-secondary` (llama.cpp CUDA on the
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-dragoon` (flash-server
|
||||
container on the gfx1151 iGPU) and `frank-zealot` (llama.cpp CUDA on the
|
||||
RTX 3090). Caddy on
|
||||
Dankclaw reverse-proxies both hosts.
|
||||
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
|
||||
@@ -25,9 +25,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 26071 --vram-kv-mib 3543`), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (inherited; pool 3.46 GiB), **MTP spec decode depth 2**, `--max-num-seqs 3`, `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8). Cutover 2026-10-08 from fp8-native (payload on disk = rollback). | 7770 |
|
||||
| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
|
||||
| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
|
||||
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **44 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the **30,969 MiB** the engine's own budget table makes claimable: card 32,624 MiB, settle floor 209 MiB), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (pool 5,120 MiB = the 3-slot contract + 1.3 % — the only dtype that funds 3 × 200,000 without gutting the slab, since bf16 bills 13.8 KiB/token), **MTP spec decode depth 2**, `--max-num-seqs 3` (the capacity contract is 3 × 200,000; 3 × (1+2) = 9 all-reduce rows, inside the 25-row wht6 exactness bound), `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8); prefill is **slab-limited and pays for it** — 22,208 → 3,397, 24,000 → 3,290, 25,600 → **3,584** t/s at 43.5k single-stream, 3-way 197k aggregate 3,051 → **3,589** t/s, with a cliff at ~26 GiB (26,071 → 4,379). Cutover 2026-10-08 from fp8-native (payload on disk = rollback); 3-slot fp8-KV shape 2026-10-13, re-cut twice in the slab's favour (22208/6912 at 4 slots → 24000/5120 → 25600/5120); the 4-slot shape (22208/6912, `--max-num-seqs 4`) is the documented rollback. | 7770 |
|
||||
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
|
||||
| Dankclaw | `frank-zealot` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
the [runbook index](#runbook-index). The full swap history lives in
|
||||
@@ -66,14 +66,14 @@ All edits flow through git. Never edit files on a remote machine directly.
|
||||
./scripts/deploy.sh dankclaw
|
||||
./scripts/deploy.sh dankserver
|
||||
./scripts/deploy.sh dankclaw --service searxng
|
||||
./scripts/deploy.sh dankserver --service frank-primary
|
||||
./scripts/deploy.sh dankserver --service frank-archon
|
||||
./scripts/deploy.sh dankclaw --dry-run
|
||||
```
|
||||
|
||||
The planner maps changed paths to services; the remote runner restarts only
|
||||
the selected service and records its revision outside the checkout.
|
||||
Documentation-only changes are a no-op. LLM services are selected only by
|
||||
their own files — every endpoint (`frank-large`/`frank-secondary`/`frank-primary`)
|
||||
their own files — every endpoint (`frank-dragoon`/`frank-zealot`/`frank-archon`)
|
||||
by its unit/env/serve trio in `scripts/` — so
|
||||
unrelated
|
||||
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
|
||||
@@ -85,7 +85,7 @@ restarts each service whose marker is behind HEAD *and* whose paths changed
|
||||
since that marker — so a backlog accumulated by service-scoped deploys fires
|
||||
all at once, and one failing health gate aborts the run with every later
|
||||
service left pending. Name the service unless a drain is what you want
|
||||
(`./scripts/deploy.sh dankserver --service frank-primary`).
|
||||
(`./scripts/deploy.sh dankserver --service frank-archon`).
|
||||
|
||||
### Deployment completeness (MANDATORY)
|
||||
|
||||
@@ -229,17 +229,17 @@ archive.
|
||||
**STOP the production service BEFORE benchmarking** — these servers serve
|
||||
real users, and a bench beside a live service starves the GPU and corrupts
|
||||
results. Use the runtime's own harness: `llama-bench` for the llama.cpp
|
||||
endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-secondary were measured. Rolling your own bench
|
||||
endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-dragoon and frank-zealot were measured. Rolling your own bench
|
||||
requires asking first.
|
||||
Scratch ports are not used on LLM hosts — a
|
||||
test process cannot share the single GPU anyway; validate against the service
|
||||
itself.
|
||||
|
||||
(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05, the
|
||||
(A gufo-era law; frank-dragoon ran ROCmFPX llama.cpp 2026-10-02 → 10-05, the
|
||||
halogen 27B container 2026-10-05 → 10-06, and is flash-server since, but the
|
||||
rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode
|
||||
settings, and will invert the ranking.**
|
||||
Measured 2026-09-28 on frank-large at depth 32768: its corpus
|
||||
Measured 2026-09-28 on frank-dragoon at depth 32768: its corpus
|
||||
reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
|
||||
draft widths, i.e. the same highly-predictable text every run, and 93 % of
|
||||
every proposal accepted. It reported 29–37 t/s decode where real traffic
|
||||
@@ -264,12 +264,12 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
|
||||
merely wedged kernel usually still answers ARP.
|
||||
|
||||
1. Stop and mask (self-healing would restart it):
|
||||
- Dankclaw `frank-large` (root unit, halogen podman on the iGPU):
|
||||
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
|
||||
- Dankclaw `frank-secondary` (root unit, llama.cpp on the 3090):
|
||||
`sudo systemctl stop frank-secondary && sudo systemctl mask frank-secondary`
|
||||
- Dankserver `frank-primary` (root unit, radiance on both R9700s, TP=2):
|
||||
`sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
|
||||
- Dankclaw `frank-dragoon` (root unit, halogen podman on the iGPU):
|
||||
`sudo systemctl stop frank-dragoon && sudo systemctl mask frank-dragoon`
|
||||
- Dankclaw `frank-zealot` (root unit, llama.cpp on the 3090):
|
||||
`sudo systemctl stop frank-zealot && sudo systemctl mask frank-zealot`
|
||||
- Dankserver `frank-archon` (root unit, radiance on both R9700s, TP=2):
|
||||
`sudo systemctl stop frank-archon && sudo systemctl mask frank-archon`
|
||||
2. Bench.
|
||||
NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
|
||||
`/etc/systemd/system/` is itself a symlink into the checkout — mask
|
||||
@@ -287,14 +287,15 @@ The harness reaches the fleet through `.omp/agent/models.yml` in the
|
||||
operator's home — `/home/blank/.omp/agent/models.yml` on the dev box, which is
|
||||
where the harness runs; there is no copy on the servers. Each provider is one
|
||||
HOST:PORT; each model `id` is the served alias. **Official model ids are
|
||||
the StarCraft unit names (2026-10-08) and since 2026-10-09 they are also the
|
||||
ids on the wire:** `frank-archon` → `192.168.0.235:7770`
|
||||
(`FRANK_PRIMARY_SERVED_NAME`), `frank-dragoon` → `192.168.0.143:7770`
|
||||
(`HALOGEN_MODEL_ID`), `frank-zealot` → `192.168.0.143:7771` (`ALIAS`). The
|
||||
SERVICES, units, scripts, env files and runbooks keep the old names
|
||||
(`frank-primary`, `frank-large`, `frank-secondary`); only the served alias,
|
||||
the `models.yml`/`config.yml` ids, the homepage cards, the OpenWebUI grants
|
||||
and the Hermes routing were renamed. Verify a rename on `/v1/models`, never
|
||||
the StarCraft unit names (2026-10-08), on the wire since 2026-10-09, and
|
||||
since 2026-10-10 they are also the SERVICE names:** `frank-archon` →
|
||||
`192.168.0.235:7770` (`FRANK_ARCHON_SERVED_NAME`), `frank-dragoon` →
|
||||
`192.168.0.143:7770` (`HALOGEN_MODEL_ID`), `frank-zealot` →
|
||||
`192.168.0.143:7771` (`ALIAS`). The 2026-10-10 cutover renamed the units,
|
||||
scripts, env files, containers, deploy services and runbooks to match —
|
||||
the three former service names are fully retired and exist nowhere in
|
||||
the repo; historical text in runbooks uses the current names. Verify a
|
||||
rename on `/v1/models`, never
|
||||
on a chat response: radiance echoes the **requested** model name, so a 200
|
||||
proves nothing about the served id (llama.cpp and halogen ignore it).
|
||||
|
||||
@@ -309,16 +310,16 @@ provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
|
||||
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
|
||||
a role pointing at a provider the file does not define fails at call time. As
|
||||
of 2026-10-06 **all three endpoints accept images** (`input: [text,
|
||||
image]`) — `frank-large` returned to vision with the flash-server v2 swap
|
||||
image]`) — `frank-dragoon` returned to vision with the flash-server v2 swap
|
||||
(the v2 vision tower loads beside the checkpoint; the text-only halogen
|
||||
27B window closed 2026-10-06) — while
|
||||
`frank-primary` takes them natively (radiance runs the checkpoint's
|
||||
`frank-archon` takes them natively (radiance runs the checkpoint's
|
||||
27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary
|
||||
are the engine core; `--mm-max-patches` available),
|
||||
and `frank-secondary` through
|
||||
and `frank-zealot` through
|
||||
the publisher's own BF16 mmproj passed as `--mmproj` — the model is natively
|
||||
multimodal but the Q6_K GGUF carries no vision tensors (same arrangement the
|
||||
Gemma4 era used). `frank-secondary`'s `contextWindow` is
|
||||
Gemma4 era used). `frank-zealot`'s `contextWindow` is
|
||||
**200704** — the PER-SLOT window (`--ctx-size 401408 --parallel 2` since the
|
||||
2026-10-07 Qwen3.5-9B swap), not the pool and not the model's native 262144:
|
||||
the harness must compact before llama.cpp answers a longer prompt with a 400.
|
||||
@@ -333,20 +334,20 @@ endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
|
||||
audio encoder, and left again with the Gemma4 retirement on 2026-10-01
|
||||
(neither the MiMo nor the Qwen3.5 tower carries audio).
|
||||
|
||||
### frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, ACTIVE)
|
||||
### frank-dragoon — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, ACTIVE)
|
||||
|
||||
The **flash-server engine** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`,
|
||||
digest-pinned, published tag 0.16.2, 3.6 GB image) as a podman container
|
||||
driven by `scripts/serve-frank-large.sh` under a **root systemd unit**
|
||||
(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
|
||||
`ExecStop=podman stop -t 60 frank-large-halogen`, the script runs `podman
|
||||
driven by `scripts/serve-frank-dragoon.sh` under a **root systemd unit**
|
||||
(`scripts/frank-dragoon.service`, `User=dank`, `Restart=always`,
|
||||
`ExecStop=podman stop -t 60 frank-dragoon-halogen`, the script runs `podman
|
||||
run` FOREGROUND — the container's exit code is the service's exit so
|
||||
`Restart=always` stays real, and the engine ledger lands on disk) on port
|
||||
7770 from the Strix Halo iGPU. Engine stdout goes to
|
||||
`/home/dank/logs/frank_large.log` — **the boot ledger (weights resident, KV
|
||||
`/home/dank/logs/frank_dragoon.log` — **the boot ledger (weights resident, KV
|
||||
pool, MTP head, prompt cache, any clamp) lives there, not in the journal**.
|
||||
Cutover 2026-10-06 from the halogen 27B-dense era
|
||||
([[frank-large-halogen27b-dflash2]], retired the same window).
|
||||
([[frank-dragoon-halogen27b-dflash2]], retired the same window).
|
||||
|
||||
Serving `Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`:
|
||||
`qwen38-flash-next-v2-swift15-abliterated.hgn` (66,687,678,464 B ≈ 62.1
|
||||
@@ -368,10 +369,10 @@ sidecar is deliberately not fetched).
|
||||
**Product facts vs the 27B era, both regressions FIXED:** the base is
|
||||
**abliterated** (the 27B was a censored-base finetune; refusals were an
|
||||
accepted trade there) and **vision is ON** (the 27B engine silently
|
||||
ignored its encoder — frank-secondary owned images for exactly one day at this
|
||||
ignored its encoder — frank-zealot owned images for exactly one day at this
|
||||
endpoint).
|
||||
|
||||
**Engine laws that shape the config** (all in `scripts/frank-large.env`
|
||||
**Engine laws that shape the config** (all in `scripts/frank-dragoon.env`
|
||||
with evidence): `HALOGEN_SPEC_ADAPT=0` is MANDATORY — the default adaptive
|
||||
policy disables the MTP head for whole requests after low acceptance and
|
||||
HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance
|
||||
@@ -413,7 +414,7 @@ page cache beside the checkpoint — the card's quickstart arithmetic does
|
||||
not bill it as resident, but READ the boot ledger and re-size if the
|
||||
engine prints a clamp.
|
||||
|
||||
Fetch: `scripts/download-frank-large-flashnext.sh` into
|
||||
Fetch: `scripts/download-frank-dragoon-flashnext.sh` into
|
||||
`/home/dank/models/halogen-flashnext-v2/` (`-f` + exact byte counts + LFS
|
||||
sha256 + `--verify-only`, no token on the command line; resume goes
|
||||
against the RESOLVED CDN URL because curl drops `Range` across HF's 302).
|
||||
@@ -422,18 +423,18 @@ n-gram + vision tower, flat tokenizer with all six files as regular
|
||||
files) — the 2026-10-01 crash-loop lesson.
|
||||
|
||||
Runbook:
|
||||
`serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md`.
|
||||
`serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md`.
|
||||
Rollback: revert the swap commit and redeploy — the 27B payload
|
||||
(`/home/dank/models/halogen-qwen3.8-27b/`), its fetch script and the
|
||||
halogen 0.1.4 image digest all remain on disk; the ROCmFPX door behind
|
||||
that (`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched.
|
||||
|
||||
### frank-secondary — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
### frank-zealot — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
**llama.cpp master** (0.5.0-dev, commit `5fc4f3c`; CUDA sm_86 build
|
||||
`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`,
|
||||
`GGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit**
|
||||
(`scripts/frank-secondary.service`, `relink_unit`ed into
|
||||
(`scripts/frank-zealot.service`, `relink_unit`ed into
|
||||
`/etc/systemd/system/`), serving
|
||||
`HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive` at **Q6_K**
|
||||
(7,359,259,008 B, sha256 `c0ba7beb…` computed locally — the Hub carries no
|
||||
@@ -489,7 +490,7 @@ tokens). With `--parallel 2` a compaction request no longer serializes behind
|
||||
a live generation (the Gemma4 era's `--parallel 1` law).
|
||||
|
||||
Runbook:
|
||||
`serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md`.
|
||||
`serverAppConfigs-docs/LLM/frank-zealot-llama-qwen35-9b-q6k-2x200704.md`.
|
||||
Rollback: revert the swap commit — the Gemma4 payload
|
||||
(`/home/dank/models/Gemma4-26B-A4B/`), the MiMo Q6_K payload
|
||||
(`/home/dank/models/MiMo-Q6K/`), the Swift payload
|
||||
@@ -497,7 +498,7 @@ Rollback: revert the swap commit — the Gemma4 payload
|
||||
(`/home/dank/models/Ornith-Q8/`) all remain on disk.
|
||||
|
||||
|
||||
### frank-primary — Radiance (StillDeadcode), Swift1.5-Qwen3.8-Flash-Next GPTQ w4, TP=2 (Dankserver, ACTIVE)
|
||||
### frank-archon — Radiance (StillDeadcode), Swift1.5-Qwen3.8-Flash-Next GPTQ w4, TP=2 (Dankserver, ACTIVE)
|
||||
|
||||
The **radiance engine** (StillDeadcode, Apache-2.0; image
|
||||
`stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned, **pin 1.0.13**
|
||||
@@ -526,39 +527,98 @@ Rollback: revert the swap commit — the fp8-native `.rad`
|
||||
|
||||
**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 3`,
|
||||
`--max-model-len 200000` (the harness contextWindow must quote this),
|
||||
`--placement expert_tiered --host-pool-mib 40960` + **explicit VRAM
|
||||
partition `--vram-weights-mib 26071 --vram-kv-mib 3543`** (the vram-pair
|
||||
replaced `--expert-vs-cache-ratio`; the pair sizes the fp8-KV shape at the
|
||||
deployed boot), `--kv-cache-dtype fp8` (inherited from the fp8-native
|
||||
env), `--num-speculative-tokens 2`,
|
||||
`--max-num-batched-tokens 2048`, prefix cache 4 GiB pinned host + 64 GiB
|
||||
disk at `/home/dank/radiance-state`, **`--ngram-placement ram`**, Froggeric
|
||||
v22.5 chat template vendored (`scripts/qwen38-froggeric-v22.5.jinja`,
|
||||
`--override-chat-template`). Boot ledger: KV pool 3.46 GiB (~480k fp8
|
||||
tokens), two expert pools 57.84 GiB, cards settled 1.41 GiB free.
|
||||
`--placement expert_tiered --host-pool-mib 44928` + **explicit VRAM
|
||||
partition `--vram-weights-mib 25600 --vram-kv-mib 5120`** (the vram-pair
|
||||
replaced `--expert-vs-cache-ratio`; the two are ONE budget —
|
||||
`weights + kv ≤ 30,969 MiB`, derived from the engine's own budget table: card
|
||||
32,624 MiB, the pair at 29,120 settling 2,058 MiB free against the engine's
|
||||
209.19 MiB floor (`--gpu-headroom-mib 96` + 113.19 static), so 1,849 MiB is
|
||||
claimable and the old 29,120 bound was my hand-picked 1 GiB reserve, not a
|
||||
hardware wall), `--kv-cache-dtype` **`fp8` — quantized KV, restored 2026-10-13
|
||||
because it is the only dtype that funds the slot count at this window**,
|
||||
`--num-speculative-tokens 2`, `--max-num-batched-tokens 2048`, prefix cache
|
||||
8 GiB pinned host + 64 GiB disk at `/home/dank/radiance-state`,
|
||||
**`--ngram-placement ram`**, Froggeric v22.5 chat template vendored
|
||||
(`scripts/qwen38-froggeric-v22.5.jinja`, `--override-chat-template`). Boot
|
||||
ledger (2026-10-13, pair 25600/5120): `kv cache 5.00 GiB … backs 3 sequences
|
||||
all at 200000 tokens at once, which is the worst case` — `kv_attn` page
|
||||
26.00 KiB (fp8) at 607,740 tokens (the sizing law predicted 608,027),
|
||||
`kv_qsa_bkey` 3.25 KiB at the same token count, `kv_gdn_state` 54 MiB × 6 slots
|
||||
(3 sequences × 2 copies) = 324 MiB; the sampler prints **9 rows**
|
||||
(3 × (1+2), inside the 25-row `wht6` exactness bound); the slab filled to
|
||||
563 KiB of slack with **17,548 slots** (16,235 at 24,000, 14,753 at 22,208 —
|
||||
every 1,792 MiB buys ~1,300 resident experts, and the pinned host pool hands
|
||||
the same experts back: 8,893 → 7,580 slots); `unclaimed 477.30 MiB`, cards
|
||||
settled ~336 MiB free against the engine's 209.19 MiB floor, KV lending
|
||||
4.22 GiB to the expert cache. Prefill tracks this slab only up to a
|
||||
**threshold**: 22,208 → 3,397 and 24,000 → 3,290 t/s at 43.5k are one band
|
||||
(±3 %), and the 25,600 arm measured **3,290** — the third step bought nothing
|
||||
(the 3,584 arm did not reproduce at 3 slots); the cliff is at ~26 GiB
|
||||
(26,071 → 4,379 on 2026-10-11's arm), so below that the slab is a threshold,
|
||||
not a slope.
|
||||
|
||||
**Decode ladder (2026-10-13, 512 tokens, greedy, this shape)**: 99.0 t/s
|
||||
single-stream at 43.5k, 93.3 at 197k — decode is nearly window-free (−6 % for
|
||||
4.5× depth); **3 concurrent streams give 113.9/43.3/10.9 t/s = 18.4 t/s
|
||||
aggregate**, so the slot count is paid in decode concurrency (a fused-step
|
||||
lottery), not in prefill; leg D of `probe-frank-archon-kv.py` measures it.
|
||||
|
||||
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
|
||||
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
|
||||
prefix cache, and the rule that a prompt over `--max-model-len` is
|
||||
refused, never truncated. What changed: the fp8-native era's NVMe expert
|
||||
streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the
|
||||
40 GiB pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
|
||||
on the deployed shape (512-token streamed probe). The campaign also
|
||||
host pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
|
||||
on the fp8 shape (512-token streamed probe). The campaign also
|
||||
measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
|
||||
accept 0.7246); production shipped the fp8-KV + spec-2 pair at cutover —
|
||||
the KLD gate scored at the fp8-KV shape, so the gate and the boot agree.
|
||||
Disk-tier sizing law: the engine forces ~9.6 GiB of disk-tier weight
|
||||
buffers + arena onto the weights side, so `weights + kv` is capped near
|
||||
29.6 GiB/card; boot with the pair, never `--kv-cache-size-mib`.
|
||||
accept 0.7246).
|
||||
|
||||
**The pinned-RAM ledger is why this is a root unit**: 2 × 40 GiB expert
|
||||
**The KV dtype law (measured 2026-10-11, applied at 4 slots 2026-10-13)**:
|
||||
`--kv-cache-dtype` takes ONLY `bf16|fp8` — one dtype for the whole cache, so
|
||||
llama.cpp's exact-K/quantized-V pair (what frank-zealot runs) has NO radiance
|
||||
equivalent; the rung under bf16 is fp8 on both halves. Measured per rank from
|
||||
the two boot ledgers: `kv_attn` (13 layers, 4-token blocks) pages 26.00 → 52.00
|
||||
KiB, while `kv_qsa_bkey` (13 layers) stays 3.25 KiB at BOTH dtypes (the QSA
|
||||
block key is an index, not attention state) — so a token costs 7.3125 KiB at fp8
|
||||
and 13.8125 KiB at bf16: **1.89×, not 2×**. Pool sizing must also pay the
|
||||
dtype-independent terms that live in the same pool (~110 MiB a sequence of
|
||||
recurrent state + 448 MiB of prefix checkpoints):
|
||||
`(vram_kv − 110·seqs − 448) MiB / rate` — bf16 at 6,912 gives 462,946 where the
|
||||
engine reported 462,752 (0.04 % page rounding); fp8 at 6,912 with four slots
|
||||
gives 843,566, i.e. the deployed capacity contract 4 × 200,000. **That is why
|
||||
the dtype is fp8: the pool must back `seqs × max-model-len` as a hard limit, and
|
||||
bf16 at four slots costs 11,679 MiB — the slab would fall to 17,441 MiB (−33 %),
|
||||
where fp8 leaves it at 22,208.** The cost is paid twice on this engine, and
|
||||
half-refunded: idle KV VRAM is also the expert cache's spillover, and the bigger
|
||||
pool *lends* 6.08 GiB to the slab where the 3,543 MiB fp8 pool lent 2.69 GiB.
|
||||
`--max-num-seqs` rides 4 × (1 + spec) = 12 fused-all-reduce rows, under the
|
||||
25-row bound where wht6 is bit-exact (the "seqs 2 cannot boot" law belongs to
|
||||
the retired tcclaviger fork's torch.compile, not radiance). Disk-tier sizing
|
||||
law: the engine forces ~9.6 GiB of disk-tier weight buffers + arena onto the
|
||||
weights side, so `weights + kv` is capped near 29.6 GiB/card; boot with the
|
||||
pair, never `--kv-cache-size-mib`.
|
||||
|
||||
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
|
||||
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
|
||||
prefix cache, and the rule that a prompt over `--max-model-len` is
|
||||
refused, never truncated. What changed: the fp8-native era's NVMe expert
|
||||
streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the
|
||||
host pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
|
||||
on the fp8 shape (512-token streamed probe). The campaign also
|
||||
measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
|
||||
accept 0.7246).
|
||||
|
||||
**The pinned-RAM ledger is why this is a root unit**: 2 × 44 GiB expert
|
||||
pools + the 47.7 GiB n-gram copy (`--ngram-placement ram` is pinned RAM
|
||||
the card reads) + 4 GiB prefix host ≈ **132 GiB** charged against
|
||||
the card reads) + 8 GiB prefix host ≈ **147 GiB** charged against
|
||||
`RLIMIT_MEMLOCK` — the TTM floor is set just above it.
|
||||
`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages; the serve
|
||||
script's TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to
|
||||
start until both knobs are raised in the stop window (sysfs write first,
|
||||
kernel cmdline + reboot only if read-only).
|
||||
`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages, and **the knob is
|
||||
runtime state whose default is half of RAM**: `scripts/ttm-preflight.sh` (sourced
|
||||
by the serve script) therefore RAISES it to `FRANK_ARCHON_TTM_TARGET_MIB=204800`
|
||||
before refusing, because the 2026-10-12 power failure reset it to 125,988 MiB
|
||||
and the preflight re-refused 6,919 times. A manual sysfs write in the stop
|
||||
window is a convenience, not the repair; kernel cmdline persistence
|
||||
(`ttm.pages_limit=52428800`) is the belt to it.
|
||||
|
||||
**Contract facts** (source-verified): the HTTP listener binds AFTER the
|
||||
full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
|
||||
@@ -570,11 +630,11 @@ chat format — no parser flags; vision native (the 27-layer Qwen-VL tower,
|
||||
`--mm-max-patches`); `/metrics` carries vLLM-compatible names +
|
||||
`radiance:*` families — the Dankclaw Prometheus scrape is re-added.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-primary-radiance-swift15-w4dn8.md`.
|
||||
Retired fp8-native era: `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md`.
|
||||
Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md`.
|
||||
Retired fp8-native era: `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md`.
|
||||
Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-archon-freetoken-pr587.md`.
|
||||
Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
|
||||
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
|
||||
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-archon-glm53-flash-w4a16.md`.
|
||||
|
||||
## Key Lessons Learned
|
||||
Durable operational lessons from the fleet's history. Full detail in the
|
||||
@@ -736,11 +796,22 @@ linked runbooks.
|
||||
- Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff`
|
||||
of the old one, or live pages must land in RAM that `MemAvailable`
|
||||
overstates. Dankclaw swap (64 GiB at `/swap.img`, fstab-driven) protects
|
||||
the co-tenants, not frank-large — pinned weights don't swap, and paging
|
||||
the co-tenants, not frank-dragoon — pinned weights don't swap, and paging
|
||||
the KV pool is worse than the OOM it prevents.
|
||||
- Host swap state is invisible to `deploy.sh` (no repo file) — check
|
||||
`swapon --show` before reasoning about headroom.
|
||||
- **Silent RAM corruption outranks every software theory.** frank-primary's
|
||||
- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults
|
||||
to half of RAM, so the manual raise to 200 GiB (2026-10-05) vanished with the
|
||||
2026-10-12 23:17 POWER FAILURE on dankserver: the cap fell to 125,988 MiB,
|
||||
frank-archon's pinned-ledger preflight (floor 150,000 MiB) refused the start,
|
||||
and `Restart=always` re-ran that refusal **6,919 times in ~20 h** — a unit in
|
||||
`activating (auto-restart)` that exits 1 in 50 ms with no container and no
|
||||
engine log is a PREFLIGHT refusal, not an engine fault. Repo law now:
|
||||
`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) raises the knob
|
||||
to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only
|
||||
after a write the kernel rejected. Never prescribe a manual sysfs write as the
|
||||
recovery procedure for a service that can boot before a human reaches it.
|
||||
- **Silent RAM corruption outranks every software theory.** frank-archon's
|
||||
~40k-token output corruption (2026-10-04/05, repetition/garbage in long
|
||||
agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
|
||||
by 4600. Consumer non-ECC memory raises **no MCE** for a silent flip, and
|
||||
@@ -763,7 +834,7 @@ linked runbooks.
|
||||
`LimitMEMLOCK=infinity` (systemd is privileged; the value flows through
|
||||
podman to runc without any raise). Upstream halogen's own run lines pass
|
||||
NO `--ulimit` at all — copy vendor run lines before adding flags
|
||||
(measured 2026-10-05 on the frank-large cutover).
|
||||
(measured 2026-10-05 on the frank-dragoon cutover).
|
||||
- **Kairic Edge** (if ever re-deployed): the pinned CK commit needs
|
||||
`third_party/composable_kernel/include/ck/config.h` generated manually
|
||||
(untracked; `git clean` removes it), and its greedy-argmax fastpath rejects
|
||||
@@ -789,7 +860,7 @@ linked runbooks.
|
||||
121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
|
||||
Record the child pid and kill it in a `finally`, then prove `free` came back
|
||||
before believing a number. Kill by pid — never `pkill -f llama-server`,
|
||||
which also hits frank-secondary.
|
||||
which also hits frank-zealot.
|
||||
|
||||
**llama.cpp serving**
|
||||
|
||||
@@ -813,7 +884,7 @@ linked runbooks.
|
||||
builds its context against whatever model the draft context holds. Two
|
||||
context-backed types cannot share one process, and MTP's head only loads
|
||||
when MTP **is** the draft model. **A LEGAL chain can still lose on a
|
||||
recurrent model:** `draft-mtp,ngram-mod` on frank-large's qwen35 hybrid
|
||||
recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35 hybrid
|
||||
measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer
|
||||
logs `recurrent state rollback is not compatible with 'ngram-mod' -
|
||||
disabling rollback support`, and MTP verification depends on that rollback,
|
||||
@@ -855,7 +926,7 @@ linked runbooks.
|
||||
free. That was a CONTIGUITY failure, not a total-bytes one, and no totals
|
||||
model — the sweep's or the seam's — can see it.)
|
||||
- **On the CUDA 3090 the same knob has NO interior optimum up to 2048, so
|
||||
frank-secondary ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the
|
||||
frank-zealot ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the
|
||||
live service: at 8,928 prompt tokens ubatch 1024 gives 3,972 / 4,077 t/s and
|
||||
ubatch 2048 gives 4,207 / 4,212 — **+4.6 %** — for **616 MiB** (free 2,849 →
|
||||
2,279). Decode did not move (91.9–93.0 vs 93.8–94.3, i.e. noise), and a
|
||||
@@ -864,9 +935,9 @@ linked runbooks.
|
||||
than the ubatch cannot see it. The logical `--batch-size` was again free. And
|
||||
the predecessor's "ubatch 2048 failed the scratch boot" was a shape
|
||||
observation, not a law: the same file boots 2048 clean at 2 × 200,704. Bench
|
||||
it by restarting the SERVICE through a transient `FRANK_SECONDARY_ENV_FILE`
|
||||
it by restarting the SERVICE through a transient `FRANK_ZEALOT_ENV_FILE`
|
||||
drop-in, never a scratch port — one card holds one server. See
|
||||
[[frank-secondary-llama-mimo-q6k-2x200704]].
|
||||
[[frank-zealot-llama-mimo-q6k-2x200704]].
|
||||
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
|
||||
size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
|
||||
worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one
|
||||
@@ -888,7 +959,7 @@ linked runbooks.
|
||||
CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT
|
||||
(`400 exceed_context_size_error: request (184001 tokens) exceeds the
|
||||
available context size (180224 tokens)`) even when the pool has room. So
|
||||
frank-secondary's 2 → 3 reslice moved each conversation from the model's native
|
||||
frank-zealot's 2 → 3 reslice moved each conversation from the model's native
|
||||
262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or
|
||||
the harness would discover the ceiling as a 400 on a long conversation. Side
|
||||
effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
|
||||
@@ -899,9 +970,9 @@ linked runbooks.
|
||||
day the freed space also bought the Q6_K rung (now 94.4 tok/s — the rung
|
||||
itself is a further measured −10 %, weight-bandwidth, not slot geometry).
|
||||
All moves measured; both slot directions cost or buy window, never pool.
|
||||
See [[frank-secondary-llama-mimo-q6k-2x200704]].
|
||||
See [[frank-zealot-llama-mimo-q6k-2x200704]].
|
||||
- **Price a VRAM budget from two boots, never from a formula — and an mmproj
|
||||
costs ~1.5x its file size.** The frank-secondary 3090 fit was solved from two
|
||||
costs ~1.5x its file size.** The frank-zealot 3090 fit was solved from two
|
||||
measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
|
||||
**non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV
|
||||
14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file
|
||||
@@ -909,24 +980,24 @@ linked runbooks.
|
||||
ride with it. Subtracting the two rows is exact; extrapolating weights+KV+
|
||||
"a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then
|
||||
solve `KV ≤ card − non-KV − headroom` and round to a multiple of
|
||||
`PARALLEL × 4096`. Recorded in `scripts/frank-secondary.env` so the next resize
|
||||
`PARALLEL × 4096`. Recorded in `scripts/frank-zealot.env` so the next resize
|
||||
starts from measurement.
|
||||
- **`images` and `image_url` are NOT interchangeable: only the content part
|
||||
reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
|
||||
model still invents an answer if the client used the legacy top-level
|
||||
`images` field on `/v1/chat/completions` — frank-secondary answered "Brown." for
|
||||
`images` field on `/v1/chat/completions` — frank-zealot answered "Brown." for
|
||||
a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
|
||||
failure is silent because a hallucinated color is a plausible answer. Probe
|
||||
vision with a solid color and expect an exact word.
|
||||
- **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
|
||||
`500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
|
||||
then served frank-large: every tool-using agent on that endpoint was broken
|
||||
then served frank-dragoon: every tool-using agent on that endpoint was broken
|
||||
and the failure read like a model fault. Mainline llama.cpp 0.4.1 defaults
|
||||
`--jinja` on and the serve script pins it; halogen also accepts `tools` and
|
||||
returns `tool_calls`. Check tool support before routing a role at an endpoint.
|
||||
- **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same
|
||||
option and the same ~12x prefill collapse described in the ROCm note above.
|
||||
f16 KV needs nothing; frank-secondary ran f16 for that reason until
|
||||
f16 KV needs nothing; frank-zealot ran f16 for that reason until
|
||||
2026-09-17, when `build-cuda` was rebuilt with the option ON so the
|
||||
endpoint could take q8_0 KV.
|
||||
- **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both
|
||||
@@ -944,7 +1015,7 @@ linked runbooks.
|
||||
raises on anything else — HTTP 500, not a clamp — so the harness's levels
|
||||
must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`).
|
||||
A role whose level is unmapped fails on its FIRST request, and the failure
|
||||
names Jinja, not the model: every subagent on `frank-secondary` died with
|
||||
names Jinja, not the model: every subagent on `frank-zealot` died with
|
||||
`Unexpected reasoning effort high` until `high: xhigh` was added on
|
||||
2026-09-16. Halogen accepts the wider set, so the two entries legitimately
|
||||
differ.
|
||||
@@ -964,7 +1035,7 @@ linked runbooks.
|
||||
times (`RuntimeError: No HIP GPUs are available`).
|
||||
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its
|
||||
ports off the gluetun service.
|
||||
- `amd_iommu=off` on Dankclaw (bought 13–16 % prefill for frank-large,
|
||||
- `amd_iommu=off` on Dankclaw (bought 13–16 % prefill for frank-dragoon,
|
||||
2026-09-11) permanently retired the XDNA2 NPU: the driver's SVA/PASID bind
|
||||
needs an enabled IOMMU (`/dev/accel/accel0` fails `err=-19`). Revert path
|
||||
in `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md`.
|
||||
@@ -1018,9 +1089,9 @@ Active:
|
||||
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×262,144, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` |
|
||||
| frank-secondary (llama.cpp CUDA, Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-qwen35-9b-q6k-2x200704.md` |
|
||||
| frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` |
|
||||
| frank-dragoon (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×262,144, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md` |
|
||||
| frank-zealot (llama.cpp CUDA, Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-zealot-llama-qwen35-9b-q6k-2x200704.md` |
|
||||
| frank-archon (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
|
||||
@@ -1028,41 +1099,41 @@ Retired (rollback targets and history):
|
||||
|
||||
| Runtime | Host | Window | Runbook |
|
||||
|---------|------|--------|---------|
|
||||
| llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-secondary`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-secondary-llama-gemma4-q4km-200704.md` |
|
||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
|
||||
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
||||
| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` |
|
||||
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-secondary-llama-mimo-q6k-2x200704.md` |
|
||||
| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` |
|
||||
| llama.cpp CUDA Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×114,688 draft-mtp q8_0 KV (`frank-secondary`) | Dankclaw | 2026-10-06 → 2026-10-07 (rollback target — payload on disk; owns the 3090 ubatch ladder A/B and the 34 KiB/token qwen35 KV law) | `frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` |
|
||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` |
|
||||
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` |
|
||||
| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
|
||||
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-secondary`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-secondary-llama-mimo-q6k-2x200704.md`) | `frank-secondary-llama-mimo-q6k.md` |
|
||||
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-secondary-llama-mimo-q5km.md` |
|
||||
| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-secondary`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-secondary-sglang-mimo-v26.md` |
|
||||
|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
|
||||
| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
|
||||
| R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
|
||||
| llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-primary`) | Dankserver | 2026-09-26 → 09-27 | `frank-primary-llamacpp-qwen38-flash-next-q8.md` |
|
||||
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
|
||||
| llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-zealot`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-zealot-llama-gemma4-q4km-200704.md` |
|
||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-dragoon`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-dragoon-rocmfpx-q6agent-uncensored-mtp.md` |
|
||||
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-archon`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-archon-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
||||
| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-archon`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-archon-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-archon-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-archon-freetoken-pr587.md` |
|
||||
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-zealot`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-zealot-llama-mimo-q6k-2x200704.md` |
|
||||
| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-zealot`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md` |
|
||||
| llama.cpp CUDA Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×114,688 draft-mtp q8_0 KV (`frank-zealot`) | Dankclaw | 2026-10-06 → 2026-10-07 (rollback target — payload on disk; owns the 3090 ubatch ladder A/B and the 34 KiB/token qwen35 KV law) | `frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md` |
|
||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-dragoon`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-dragoon-rocmfpx-q8agent-mtp.md` |
|
||||
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-dragoon`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-dragoon-gufo-qwen38-27b.md` |
|
||||
| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-dragoon`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-dragoon-vllm-mimo-v26-bf16.md` |
|
||||
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-zealot`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-zealot-llama-mimo-q6k-2x200704.md`) | `frank-zealot-llama-mimo-q6k.md` |
|
||||
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-zealot`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-zealot-llama-mimo-q5km.md` |
|
||||
| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-zealot`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-zealot-sglang-mimo-v26.md` |
|
||||
|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-archon`)|Dankserver|2026-09-30 → 10-01|`frank-archon-glm53-flash-w4a16.md`|
|
||||
| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-dragoon`) | Dankclaw | 2026-09-21 → 09-28 | `frank-dragoon-unsloth-qwen38-27b-q8xl.md` |
|
||||
| R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-archon`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-archon-r9v-radiance-qwen38-uncensored.md` |
|
||||
| llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-archon`) | Dankserver | 2026-09-26 → 09-27 | `frank-archon-llamacpp-qwen38-flash-next-q8.md` |
|
||||
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-dragoon`) | Dankclaw | 2026-09-15 → 09-16 | `frank-dragoon-ik-uncensored.md` |
|
||||
| vllm-radiance TP=2 MXFP4 + DFlash2 (`frank-medium`) | Dankserver | 2026-09-20 → 09-26 | `frank-medium-radiance-mxfp4.md` |
|
||||
| Lucebox DS4.1 DeepSeek-V4.1-Flash ROCmFP23 (`frank-primary`) | Dankserver | 2026-09-26 (same-day swap) | `frank-primary-lucebox-ds4.md` |
|
||||
| Lucebox DS4.1 DeepSeek-V4.1-Flash ROCmFP23 (`frank-archon`) | Dankserver | 2026-09-26 (same-day swap) | `frank-archon-lucebox-ds4.md` |
|
||||
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
|
||||
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
|
||||
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` |
|
||||
| unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` |
|
||||
| Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` |
|
||||
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-secondary`) | Dankclaw | 2026-09-16 → 09-17 | `frank-secondary-qwen38-gsq-rco.md` |
|
||||
| vLLM Gemma4-12B w4a16 (`frank-secondary`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-secondary-vllm-gemma4.md` |
|
||||
| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-secondary`) | Dankclaw | 2026-09-17 → 10-01 | `frank-secondary-gemma4-12b.md` |
|
||||
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-dragoon`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
|
||||
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-dragoon`) | Dankclaw | 2026-09-19 → 09-21 | `frank-dragoon-ciru-orca.md` |
|
||||
| unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-dragoon`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-dragoon-unsloth-qwen38-flash-next.md` |
|
||||
| Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-dragoon`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-dragoon-kairic-edge-qwen38-27b.md` |
|
||||
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-zealot`) | Dankclaw | 2026-09-16 → 09-17 | `frank-zealot-qwen38-gsq-rco.md` |
|
||||
| vLLM Gemma4-12B w4a16 (`frank-zealot`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-zealot-vllm-gemma4.md` |
|
||||
| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-zealot`) | Dankclaw | 2026-09-17 → 10-01 | `frank-zealot-gemma4-12b.md` |
|
||||
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
|
||||
| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
|
||||
| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
|
||||
| CIRU ActiveFPX v3 / PromptForge | Dankclaw | through 2026-08-22 | `ciru-activefpx-promptforge.md` |
|
||||
| DFlash2 Vulkan | Dankclaw | 2026-08-21 (same-day revert) | `dflash2-large-vulkan.md` |
|
||||
| FastFlowLM NPU (`frank-npu`) | Dankclaw | twice, retired 2026-09-11 | `frank-npu.md`, `FastFlowLM NPU Inference.md` |
|
||||
| FreeToken (official + Huihui FP8, `frank-secondary`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
|
||||
| FreeToken (official + Huihui FP8, `frank-zealot`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
|
||||
| FastMTP patched llama.cpp | Dankserver | retired 2026-08-18 | `qwen3.8-spec-matrix.md` |
|
||||
| ik_llama.cpp Ornith CRACK | Dankserver | retired 2026-09-02 | `ikllama-ornith-dankserver.md` |
|
||||
| llama.cpp layer split + DFlash2 (`frank-medium`) | Dankserver | 2026-09-16 → 09-20 | `frank-medium-llamacpp-dflash2.md` |
|
||||
|
||||
@@ -13,7 +13,7 @@ one deploy script.
|
||||
| Host | User | IP | Podman stack | LLM service | LLM port |
|
||||
|------|------|----|--------------|-------------|----------|
|
||||
| dankserver | dank | 192.168.0.235 | media and library | frank-medium (ROCmFPX llama.cpp, Qwen3.8-27B AGENT) | 7770 |
|
||||
| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-large (gufo, Qwen3.8-27B UD-Q8_K_XL) | 7770 |
|
||||
| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-dragoon (gufo, Qwen3.8-27B UD-Q8_K_XL) | 7770 |
|
||||
|
||||
## Active Services
|
||||
|
||||
@@ -30,7 +30,7 @@ one deploy script.
|
||||
| Gitea | dankclaw | gitea.dankclaw.local | 3001 |
|
||||
| Grafana | dankclaw | grafana.dankclaw.local | 3002 |
|
||||
| Prometheus | dankclaw | prometheus.dankclaw.local | 9090 |
|
||||
| frank-large | dankclaw | — (bare endpoint) | 7770 |
|
||||
| frank-dragoon | dankclaw | — (bare endpoint) | 7770 |
|
||||
|
||||
## Structure
|
||||
|
||||
|
||||
+9
-9
@@ -1,26 +1,26 @@
|
||||
#!/usr/bin/env bash
|
||||
# autoresearch.sh — canonical frank-primary benchmark entry point (dev box).
|
||||
# autoresearch.sh — canonical frank-archon benchmark entry point (dev box).
|
||||
#
|
||||
# The service runs on dankserver (192.168.0.235:7770), the two R9700s live
|
||||
# there, and the benchmark must own those cards, so the whole stop -> boot
|
||||
# scratch -> bench -> restore sequence runs ON dankserver (see
|
||||
# scripts/autoresearch-frank-primary.sh). This wrapper:
|
||||
# 1. copies the candidate config (the repo's scripts/frank-primary.env, the
|
||||
# scripts/autoresearch-frank-archon.sh). This wrapper:
|
||||
# 1. copies the candidate config (the repo's scripts/frank-archon.env, the
|
||||
# file the autoresearch loop edits) and the bench script to dankserver,
|
||||
# 2. runs the bench over ssh, relaying its stdout (the METRIC lines) and
|
||||
# exit code back here verbatim.
|
||||
# It deliberately does NOT push to origin: the scp dirties dankserver's
|
||||
# working tree, and the loop's uncommitted candidate config must not be
|
||||
# pushed. Only this file and scripts/autoresearch-frank-primary.sh are
|
||||
# pushed. Only this file and scripts/autoresearch-frank-archon.sh are
|
||||
# committed; the candidate env rides over ssh each run.
|
||||
set -uo pipefail
|
||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REMOTE=dankserver
|
||||
REMOTE_REPO=/home/dank/serverAppConfigs
|
||||
|
||||
scp -q "$DIR/scripts/frank-primary.env" \
|
||||
"$REMOTE:$REMOTE_REPO/scripts/frank-primary.env" || { echo "scp env failed" >&2; exit 1; }
|
||||
scp -q "$DIR/scripts/autoresearch-frank-primary.sh" \
|
||||
"$REMOTE:$REMOTE_REPO/scripts/autoresearch-frank-primary.sh" || { echo "scp script failed" >&2; exit 1; }
|
||||
scp -q "$DIR/scripts/frank-archon.env" \
|
||||
"$REMOTE:$REMOTE_REPO/scripts/frank-archon.env" || { echo "scp env failed" >&2; exit 1; }
|
||||
scp -q "$DIR/scripts/autoresearch-frank-archon.sh" \
|
||||
"$REMOTE:$REMOTE_REPO/scripts/autoresearch-frank-archon.sh" || { echo "scp script failed" >&2; exit 1; }
|
||||
|
||||
ssh "$REMOTE" "bash $REMOTE_REPO/scripts/autoresearch-frank-primary.sh $REMOTE_REPO/scripts/frank-primary.env"
|
||||
ssh "$REMOTE" "bash $REMOTE_REPO/scripts/autoresearch-frank-archon.sh $REMOTE_REPO/scripts/frank-archon.env"
|
||||
|
||||
+1
-1
File diff suppressed because one or more lines are too long
@@ -1,9 +1,9 @@
|
||||
#!/usr/bin/env bash
|
||||
# download-frank-primary-fp8.sh — fetch/verify the Qwen3.8-Flash-Next-FP8
|
||||
# download-frank-archon-fp8.sh — fetch/verify the Qwen3.8-Flash-Next-FP8
|
||||
# payload (131 safetensors shards, 172.78 GiB total, plus aux files) for the
|
||||
# vLLM frank-primary deployment on dankserver.
|
||||
# vLLM frank-archon deployment on dankserver.
|
||||
#
|
||||
# bash scripts/download-frank-primary-fp8.sh [dest_dir]
|
||||
# bash scripts/download-frank-archon-fp8.sh [dest_dir]
|
||||
#
|
||||
# The manifest (scripts/manifests/qwen3.8-flash-next-fp8.manifest) carries
|
||||
# per-file sizes + LFS sha256s from the Hub; hf-fetch.sh is idempotent — an
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the z-lab Qwen3.5-9B-DFlash drafter for frank-large's DFlash
|
||||
# Download the z-lab Qwen3.5-9B-DFlash drafter for frank-dragoon's DFlash
|
||||
# speculative-decode arm (vLLM --speculative-config method=dflash). Public
|
||||
# repo — no auth header on the command line.
|
||||
#
|
||||
@@ -10,7 +10,7 @@
|
||||
# the draft for that target.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/download-frank-large-dflash.sh [DEST_DIR] [--verify-only]
|
||||
# scripts/download-frank-dragoon-dflash.sh [DEST_DIR] [--verify-only]
|
||||
set -euo pipefail
|
||||
|
||||
REPO="z-lab/Qwen3.5-9B-DFlash"
|
||||
@@ -1,14 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the BF16 MiMo-V2.6-Distill-Qwen-9B payload for frank-large
|
||||
# Download the BF16 MiMo-V2.6-Distill-Qwen-9B payload for frank-dragoon
|
||||
# (vLLM/kyuz0 gfx1151, Strix Halo iGPU). Public repo — no auth header on the
|
||||
# command line (a Bearer token in argv is world-readable via /proc).
|
||||
#
|
||||
# Per-file FRESH downloads, no resume: the house rule (see
|
||||
# download-frank-secondary.sh); -f/--fail-with-body plus size+sha256 against the
|
||||
# download-frank-zealot.sh); -f/--fail-with-body plus size+sha256 against the
|
||||
# Hub LFS oid is what makes an access error fail instead of "succeeding".
|
||||
#
|
||||
# Usage:
|
||||
# scripts/download-frank-large-mimo.sh [DEST_DIR] [--verify-only]
|
||||
# scripts/download-frank-dragoon-mimo.sh [DEST_DIR] [--verify-only]
|
||||
#
|
||||
# The verified set below is exactly what the 2026-10-01 bench window served:
|
||||
# 17.53 GiB of BF16 weights (4 shards, 18,819,720,848 B total with the rest).
|
||||
@@ -1,14 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the source Qwen3.5-9B checkpoint — the ONLY place this model
|
||||
# family's MTP head (15 mtp.* tensors) exists. frank-large runs it as the
|
||||
# family's MTP head (15 mtp.* tensors) exists. frank-dragoon runs it as the
|
||||
# vLLM cross-model MTP draft (MiMo-V2.6-Distill-Qwen-9B itself ships zero
|
||||
# mtp tensors; that is why SGLang's NEXTN on frank-secondary draws 0.00
|
||||
# mtp tensors; that is why SGLang's NEXTN on frank-zealot draws 0.00
|
||||
# acceptance). Public repo — no auth header on the command line.
|
||||
#
|
||||
# NOTE the odd upstream shard names (model.safetensors-0000N-of-00004...).
|
||||
#
|
||||
# Usage:
|
||||
# scripts/download-frank-large-qwen35.sh [DEST_DIR] [--verify-only]
|
||||
# scripts/download-frank-dragoon-qwen35.sh [DEST_DIR] [--verify-only]
|
||||
set -euo pipefail
|
||||
|
||||
REPO="Qwen/Qwen3.5-9B"
|
||||
+2
-2
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-secondary
|
||||
# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-zealot
|
||||
# (SGLang, RTX 3090). Public repo — no auth header on the command line.
|
||||
#
|
||||
# Per-file FRESH downloads, no resume: `curl -C -` once produced a 213 MB
|
||||
@@ -7,7 +7,7 @@
|
||||
# the house rule.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only]
|
||||
# scripts/download-frank-zealot.sh [DEST_DIR] [--verify-only]
|
||||
#
|
||||
# LFS shards are verified against the Hub LFS sha256; non-LFS text files
|
||||
# against their declared byte size. Idempotent: complete files are skipped.
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# bench-frank-large.sh — 64k-context benchmark for the frank-default-large stack.
|
||||
# bench-frank-dragoon.sh — 64k-context benchmark for the frank-default-large stack.
|
||||
#
|
||||
# Measures generation throughput of Qwen3.6-27B Q8_K_P + DFlash + ngram-mod on
|
||||
# a single backend at 64k context. It starts llama-server on a scratch port,
|
||||
@@ -7,15 +7,15 @@
|
||||
# prompt, and prints tokens/sec.
|
||||
#
|
||||
# Usage:
|
||||
# ./scripts/bench-frank-large.sh <ROCm0|Vulkan0> [--generations N]
|
||||
# ./scripts/bench-frank-dragoon.sh <ROCm0|Vulkan0> [--generations N]
|
||||
#
|
||||
# Example:
|
||||
# ./scripts/bench-frank-large.sh ROCm0
|
||||
# ./scripts/bench-frank-large.sh Vulkan0 --generations 3
|
||||
# ./scripts/bench-frank-dragoon.sh ROCm0
|
||||
# ./scripts/bench-frank-dragoon.sh Vulkan0 --generations 3
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
DEVICE="${1:?usage: bench-frank-large.sh <ROCm0|Vulkan0> [--generations N]}"
|
||||
DEVICE="${1:?usage: bench-frank-dragoon.sh <ROCm0|Vulkan0> [--generations N]}"
|
||||
GENERATIONS=1
|
||||
if [ "${2:-}" = "--generations" ]; then
|
||||
GENERATIONS="${3:?--generations needs a count}"
|
||||
@@ -41,7 +41,7 @@ SERVICE_HEALTH[dankclaw]="http://127.0.0.1:7770/health"
|
||||
# Drop page caches before restart: with the model's pages fully cached, the
|
||||
# KV/graph allocation stalls at load (VRAM stuck at 28 GiB). See AGENTS.md.
|
||||
DROP_CACHES[dankclaw]=1
|
||||
# Passwordless restart helper (root-owned, installed via /etc/sudoers.d/frank-large).
|
||||
# Passwordless restart helper (root-owned, installed via /etc/sudoers.d/frank-dragoon).
|
||||
# When present, deploy.sh uses it instead of inline sudo commands.
|
||||
HELPER[dankclaw]="/usr/local/sbin/frank-restart-large.sh"
|
||||
# Health wait in iterations of 2s. Cold loads after the cache drop measured
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Tunables for the Ling-3.0-flash TRAIL on dankclaw (frank-large-ling).
|
||||
# Tunables for the Ling-3.0-flash TRAIL on dankclaw (frank-dragoon-ling).
|
||||
# This REPLACES frank-default-large (Qwen3.6-27B) temporarily. To switch
|
||||
# back: systemctl stop frank-large-ling && systemctl start frank-default-large.
|
||||
# back: systemctl stop frank-dragoon-ling && systemctl start frank-default-large.
|
||||
# Runtime: llama.cpp FORK bailingmoe3-support; Vulkan + AMD GTT.
|
||||
#
|
||||
# Memory model (amdgpu Strix Halo gfx1151, 8x16GB LPDDR5 = 128 GiB physical):
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=llama.cpp Ling-3.0-flash Q6_K Vulkan+MTP (frank-large-ling TRIAL)
|
||||
Description=llama.cpp Ling-3.0-flash Q6_K Vulkan+MTP (frank-dragoon-ling TRIAL)
|
||||
After=network.target
|
||||
|
||||
[Service]
|
||||
@@ -8,7 +8,7 @@
|
||||
# Vulkan exposes as an ~111 GiB heap). MTP drafter is bundled in the same
|
||||
# GGUF (block 42) - no separate --spec-draft-model needed.
|
||||
#
|
||||
# Tie-back: to restore the Qwen3.6-27B service, stop frank-large-ling and
|
||||
# Tie-back: to restore the Qwen3.6-27B service, stop frank-dragoon-ling and
|
||||
# start frank-default-large (its env/unit are committed and unchanged).
|
||||
#
|
||||
# Tunables come from frank-default-large.ling.env in this directory unless
|
||||
|
||||
@@ -4,11 +4,11 @@
|
||||
# Runtime: llama.cpp llama-server (>= b10411).
|
||||
#
|
||||
# Tunables come from frank-default-large.env in this directory unless the
|
||||
# FRANK_LARGE_ENV_FILE environment variable overrides that path.
|
||||
# FRANK_DRAGOON_ENV_FILE environment variable overrides that path.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-default-large.env}"
|
||||
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-default-large.env}"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
+1
-1
File diff suppressed because one or more lines are too long
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Decode throughput benchmark for frank-large.
|
||||
Decode throughput benchmark for frank-dragoon.
|
||||
Starts the server with a given --threads-batch value, sends an 8K+ token prompt,
|
||||
generates 256 tokens, measures generation tok/s.
|
||||
|
||||
@@ -13,12 +13,12 @@ SERVE = "/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server"
|
||||
PORT = 7770
|
||||
N_GEN = 256
|
||||
PROMPT_FILE = "/tmp/bench-prompts/prompt_code.txt"
|
||||
ENV_FILE = os.path.join(os.path.dirname(__file__), "frank-large.env")
|
||||
ENV_FILE = os.path.join(os.path.dirname(__file__), "frank-dragoon.env")
|
||||
SERVER_DIR = os.path.dirname(SERVE)
|
||||
ROCM_PATH = "/opt/rocm"
|
||||
|
||||
def make_env():
|
||||
"""Build the same env as serve-frank-large.sh."""
|
||||
"""Build the same env as serve-frank-dragoon.sh."""
|
||||
env = {**os.environ, "HSA_OVERRIDE_GFX_VERSION": "11.5.1", "HIP_VISIBLE_DEVICES": "0"}
|
||||
|
||||
# Source env file for tunables
|
||||
@@ -59,7 +59,7 @@ def wait_ready():
|
||||
def send_request(prompt, max_tokens=N_GEN):
|
||||
"""Send chat completion, measure generation throughput."""
|
||||
body = json.dumps({
|
||||
"model": "frank-large",
|
||||
"model": "frank-dragoon",
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"max_tokens": max_tokens,
|
||||
"stream": False,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Benchmark n-gram speculative decoding vs no-spec for frank-large.
|
||||
Benchmark n-gram speculative decoding vs no-spec for frank-dragoon.
|
||||
|
||||
Tests prompt-processing throughput with 16K token prompts (manageable request size).
|
||||
Code-heavy prompt (high n-gram redundancy) vs prose prompt (low redundancy).
|
||||
@@ -103,7 +103,7 @@ def start_server(port, spec_type):
|
||||
def run_test(prompt_text, spec_type, port):
|
||||
"""Send a request and extract prompt processing metrics."""
|
||||
data = json.dumps({
|
||||
"model": "frank-large",
|
||||
"model": "frank-dragoon",
|
||||
"messages": [{"role": "user", "content": prompt_text}],
|
||||
"max_tokens": TEST_TOKENS,
|
||||
"stream": False,
|
||||
@@ -189,7 +189,7 @@ def main():
|
||||
print(f" Prose: {prose_prompt_file.stat().st_size:,} bytes")
|
||||
|
||||
print("=" * 60)
|
||||
print(" N-gram Mod Benchmark — frank-large")
|
||||
print(" N-gram Mod Benchmark — frank-dragoon")
|
||||
print(f" STRIX ROCmFP4 · 64K context · 16K prompts")
|
||||
print("=" * 60)
|
||||
|
||||
|
||||
@@ -1,19 +1,19 @@
|
||||
#!/usr/bin/env bash
|
||||
# Benchmark n-gram speculative decoding vs no-spec for frank-large.
|
||||
# Benchmark n-gram speculative decoding vs no-spec for frank-dragoon.
|
||||
#
|
||||
# This tests prompt-processing throughput at 64k context with two distinct
|
||||
# prompts: one code-heavy and one prose-heavy. Uses llama-server on scratch
|
||||
# ports with the OpenAI-compatible API to measure prompt_eval_duration_ms.
|
||||
#
|
||||
# Run on Dankclaw after stopping the production service:
|
||||
# sudo systemctl stop frank-large
|
||||
# sudo systemctl stop frank-dragoon
|
||||
# ./scripts/bench-ngram-mod.sh
|
||||
# sudo systemctl start frank-large
|
||||
# sudo systemctl start frank-dragoon
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
|
||||
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-dragoon.env}"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
set -a; source "$ENV_FILE"; set +a
|
||||
fi
|
||||
@@ -184,7 +184,7 @@ run_single_test() {
|
||||
response=$(curl -s -X POST "http://127.0.0.1:$port/v1/completions" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{
|
||||
\"model\": \"frank-large\",
|
||||
\"model\": \"frank-dragoon\",
|
||||
\"prompt\": \"$prompt_text\",
|
||||
\"max_tokens\": 128,
|
||||
\"temperature\": 0,
|
||||
@@ -220,7 +220,7 @@ run_single_test() {
|
||||
}
|
||||
|
||||
echo "╔══════════════════════════════════════════════════════════╗"
|
||||
echo "║ N-gram Mod Benchmark — frank-large ║"
|
||||
echo "║ N-gram Mod Benchmark — frank-dragoon ║"
|
||||
echo "║ STRIX ROCmFP4 · 64k context · two prompts ║"
|
||||
echo "╚══════════════════════════════════════════════════════════╝"
|
||||
echo ""
|
||||
|
||||
@@ -15,9 +15,9 @@ import time
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
|
||||
SERVICE = "frank-large"
|
||||
ENV_FILE = "/home/dank/serverAppConfigs/scripts/frank-large.env"
|
||||
ALIAS = "frank-large"
|
||||
SERVICE = "frank-dragoon"
|
||||
ENV_FILE = "/home/dank/serverAppConfigs/scripts/frank-dragoon.env"
|
||||
ALIAS = "frank-dragoon"
|
||||
DEFAULT_HOST = "localhost"
|
||||
DEFAULT_PORT = 7770
|
||||
DEFAULT_MAX_TOKENS = 16384
|
||||
|
||||
+2
-2
@@ -1,4 +1,4 @@
|
||||
# Tunables for frank-large (mradermacher i1-Q4_K_M of
|
||||
# Tunables for frank-dragoon (mradermacher i1-Q4_K_M of
|
||||
# orcarouter/Qwen3.8-Flash-Next-Uncensored on ik_llama.cpp, Dankclaw,
|
||||
# RTX 3090 CUDA + Strix Halo CPU path, since 2026-09-15).
|
||||
#
|
||||
@@ -65,7 +65,7 @@ MMPROJ_PATH=/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K
|
||||
CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
|
||||
|
||||
# Alias + network.
|
||||
ALIAS=frank-large
|
||||
ALIAS=frank-dragoon
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=7770
|
||||
|
||||
+3
-3
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M 177B MoE (dankclaw, frank-large)
|
||||
Description=ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M 177B MoE (dankclaw, frank-dragoon)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
@@ -12,8 +12,8 @@ Group=dank
|
||||
# budget still needs room. Parity with the halogen quadlet's
|
||||
# Ulimit=memlock=-1:-1.
|
||||
LimitMEMLOCK=infinity
|
||||
Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
|
||||
Environment="FRANK_DRAGOON_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-dragoon.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-dragoon.sh
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
+3
-3
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve mradermacher/Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M as frank-large
|
||||
# Serve mradermacher/Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M as frank-dragoon
|
||||
# via ik_llama.cpp on Dankclaw.
|
||||
#
|
||||
# Topology (ikawrakow's hybrid recipe; upstream issue #2402 for the PLE):
|
||||
@@ -26,7 +26,7 @@
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
|
||||
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-dragoon.env}"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
@@ -66,7 +66,7 @@ fi
|
||||
# CPU-overridden weights mmapped (src/llama-load-tensors.cpp); the env file
|
||||
# sets it. Refuse to run without it.
|
||||
if [[ "${GGML_CUDA_NO_PINNED_WEIGHTS:-}" != "1" ]]; then
|
||||
echo "frank-large: refusing to start — GGML_CUDA_NO_PINNED_WEIGHTS=1 is mandatory on this host" >&2
|
||||
echo "frank-dragoon: refusing to start — GGML_CUDA_NO_PINNED_WEIGHTS=1 is mandatory on this host" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -112,7 +112,7 @@ fi
|
||||
: "${SPEC_DRAFT_POLL_BATCH:=1}"
|
||||
|
||||
# Text only: the server must not try to autoload the vision tower. Vision
|
||||
# stays on frank-large (Dankclaw), which is where the fleet routes images.
|
||||
# stays on frank-dragoon (Dankclaw), which is where the fleet routes images.
|
||||
NO_MMPROJ="${NO_MMPROJ:-1}"
|
||||
|
||||
export HIP_VISIBLE_DEVICES
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""autoresearch-bench.py — deterministic greedy battery for frank-large.
|
||||
"""autoresearch-bench.py — deterministic greedy battery for frank-dragoon.
|
||||
|
||||
Runs ON Dankclaw (no LAN jitter) against the candidate container on :7770.
|
||||
Fixed embedded corpus, temperature 0, streaming so decode speed is measured
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
#!/usr/bin/env bash
|
||||
# autoresearch-frank-primary.sh — RUNS ON DANKSERVER, driven over ssh by the
|
||||
# dev-box autoresearch.sh. It is the canonical frank-primary benchmark entry
|
||||
# autoresearch-frank-archon.sh — RUNS ON DANKSERVER, driven over ssh by the
|
||||
# dev-box autoresearch.sh. It is the canonical frank-archon benchmark entry
|
||||
# point for the autoresearch loop.
|
||||
#
|
||||
# Method (mandated by the Benchmarking Policy): the production service serves
|
||||
# real users and shares the two R9700s, so a bench beside it starves the GPUs
|
||||
# and corrupts results. This script therefore:
|
||||
# 1. stops the production frank-primary unit (root, relink symlink: stop,
|
||||
# 1. stops the production frank-archon unit (root, relink symlink: stop,
|
||||
# NOT mask — mask refuses to replace a symlinked unit file),
|
||||
# 2. boots a scratch radiance container named `arbench` on the SAME two
|
||||
# cards with the candidate engine argv (the env file the loop edits),
|
||||
@@ -18,7 +18,7 @@
|
||||
# prefill and decode rates for that single request.
|
||||
# 5. prints METRIC lines and tears the scratch down,
|
||||
# 6. restores production (systemctl start), via a trap so a failed bench
|
||||
# never leaves frank-primary down (the stale-probe-tail lesson).
|
||||
# never leaves frank-archon down (the stale-probe-tail lesson).
|
||||
#
|
||||
# Determinism: the prompt is built from a fixed, sorted corpus of repo files
|
||||
# with a fixed random seed, so prompt_n is stable across runs. Sampling is
|
||||
@@ -32,10 +32,10 @@
|
||||
# /dev/dri, the .rad ro, the state dir rw, the tune cache mount. The ONLY
|
||||
# differences: the container name (arbench, so prod's identity is untouched)
|
||||
# and a FRESH prefix-cache disk tier (arbench/, wiped per run) so the cold
|
||||
# prefill is cold — the prod tier at kvcache/frank-primary is left intact.
|
||||
# prefill is cold — the prod tier at kvcache/frank-archon is left intact.
|
||||
set -uo pipefail
|
||||
|
||||
ENV_FILE="${1:-/home/dank/serverAppConfigs/scripts/frank-primary.env}"
|
||||
ENV_FILE="${1:-/home/dank/serverAppConfigs/scripts/frank-archon.env}"
|
||||
SCRATCH=arbench
|
||||
PORT=7770
|
||||
HEALTH="http://127.0.0.1:$PORT/health"
|
||||
@@ -45,7 +45,7 @@ LOG=/home/dank/logs/arbench.log
|
||||
|
||||
restore() {
|
||||
podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
|
||||
sudo systemctl start frank-primary >/dev/null 2>&1 || true
|
||||
sudo systemctl start frank-archon >/dev/null 2>&1 || true
|
||||
}
|
||||
trap restore EXIT
|
||||
|
||||
@@ -57,13 +57,13 @@ fi
|
||||
set -a; . "$ENV_FILE"; set +a
|
||||
|
||||
# --- stop production; the scratch needs both cards to itself ----------------
|
||||
sudo systemctl stop frank-primary >/dev/null 2>&1 || true
|
||||
sudo systemctl stop frank-archon >/dev/null 2>&1 || true
|
||||
podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
|
||||
|
||||
# --- fresh prefix-cache tier for a cold prefill (prod's tier untouched) ------
|
||||
FRESH_KV="$FRANK_PRIMARY_STATE_DIR/kvcache/$SCRATCH"
|
||||
FRESH_KV="$FRANK_ARCHON_STATE_DIR/kvcache/$SCRATCH"
|
||||
rm -rf "$FRESH_KV"; mkdir -p "$FRESH_KV"
|
||||
mkdir -p "$FRANK_PRIMARY_STATE_DIR/tune"
|
||||
mkdir -p "$FRANK_ARCHON_STATE_DIR/tune"
|
||||
|
||||
# --- boot the scratch: the production run line, scratch name + fresh KV ------
|
||||
podman run -d --name "$SCRATCH" \
|
||||
@@ -72,31 +72,31 @@ podman run -d --name "$SCRATCH" \
|
||||
--security-opt seccomp=unconfined \
|
||||
--ulimit memlock=-1:-1 \
|
||||
--device /dev/kfd --device /dev/dri \
|
||||
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
|
||||
-v "$FRANK_PRIMARY_STATE_DIR":/data \
|
||||
-v "$FRANK_PRIMARY_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
|
||||
-v "$FRANK_PRIMARY_TEMPLATE":/templates/template.jinja:ro \
|
||||
"$FRANK_PRIMARY_IMAGE" \
|
||||
--model "/models/$FRANK_PRIMARY_MODEL_FILE" \
|
||||
-v "$FRANK_ARCHON_MODEL_DIR":/models:ro \
|
||||
-v "$FRANK_ARCHON_STATE_DIR":/data \
|
||||
-v "$FRANK_ARCHON_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
|
||||
-v "$FRANK_ARCHON_TEMPLATE":/templates/template.jinja:ro \
|
||||
"$FRANK_ARCHON_IMAGE" \
|
||||
--model "/models/$FRANK_ARCHON_MODEL_FILE" \
|
||||
--host :: --port "$PORT" \
|
||||
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" \
|
||||
--served-model-name "$FRANK_ARCHON_SERVED_NAME" \
|
||||
--override-chat-template /templates/template.jinja \
|
||||
--tp "$FRANK_PRIMARY_TP" --tp-wire "$FRANK_PRIMARY_TP_WIRE" \
|
||||
--max-num-seqs "$FRANK_PRIMARY_MAX_SEQS" \
|
||||
--max-model-len "$FRANK_PRIMARY_MAX_MODEL_LEN" \
|
||||
--placement "$FRANK_PRIMARY_PLACEMENT" \
|
||||
--host-pool-mib "$FRANK_PRIMARY_HOST_POOL_MIB" \
|
||||
--gpu-headroom-mib "$FRANK_PRIMARY_GPU_HEADROOM_MIB" \
|
||||
--vram-weights-mib "$FRANK_PRIMARY_VRAM_WEIGHTS_MIB" \
|
||||
--vram-kv-mib "$FRANK_PRIMARY_VRAM_KV_MIB" \
|
||||
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
|
||||
--prefix-cache-host-mib "$FRANK_PRIMARY_PREFIX_HOST_MIB" \
|
||||
--tp "$FRANK_ARCHON_TP" --tp-wire "$FRANK_ARCHON_TP_WIRE" \
|
||||
--max-num-seqs "$FRANK_ARCHON_MAX_SEQS" \
|
||||
--max-model-len "$FRANK_ARCHON_MAX_MODEL_LEN" \
|
||||
--placement "$FRANK_ARCHON_PLACEMENT" \
|
||||
--host-pool-mib "$FRANK_ARCHON_HOST_POOL_MIB" \
|
||||
--gpu-headroom-mib "$FRANK_ARCHON_GPU_HEADROOM_MIB" \
|
||||
--vram-weights-mib "$FRANK_ARCHON_VRAM_WEIGHTS_MIB" \
|
||||
--vram-kv-mib "$FRANK_ARCHON_VRAM_KV_MIB" \
|
||||
--kv-cache-dtype "$FRANK_ARCHON_KV_DTYPE" \
|
||||
--prefix-cache-host-mib "$FRANK_ARCHON_PREFIX_HOST_MIB" \
|
||||
--prefix-cache-dir "/data/kvcache/$SCRATCH" \
|
||||
--prefix-cache-disk-mib "$FRANK_PRIMARY_PREFIX_DISK_MIB" \
|
||||
--ngram-placement "$FRANK_PRIMARY_NGRAM_PLACEMENT" \
|
||||
--num-speculative-tokens "$FRANK_PRIMARY_SPEC_TOKENS" \
|
||||
--max-num-batched-tokens "$FRANK_PRIMARY_MAX_BATCHED_TOKENS" \
|
||||
${FRANK_PRIMARY_EXTRA_ARGS:+$FRANK_PRIMARY_EXTRA_ARGS} \
|
||||
--prefix-cache-disk-mib "$FRANK_ARCHON_PREFIX_DISK_MIB" \
|
||||
--ngram-placement "$FRANK_ARCHON_NGRAM_PLACEMENT" \
|
||||
--num-speculative-tokens "$FRANK_ARCHON_SPEC_TOKENS" \
|
||||
--max-num-batched-tokens "$FRANK_ARCHON_MAX_BATCHED_TOKENS" \
|
||||
${FRANK_ARCHON_EXTRA_ARGS:+$FRANK_ARCHON_EXTRA_ARGS} \
|
||||
> "$LOG" 2>&1
|
||||
if [ $? -ne 0 ]; then echo "scratch boot failed; see $LOG" >&2; exit 1; fi
|
||||
|
||||
+32
-32
@@ -1,19 +1,19 @@
|
||||
#!/usr/bin/env bash
|
||||
# frank-primary-arms.sh — run a SEQUENCE of frank-primary bench arms and keep
|
||||
# frank-archon-arms.sh — run a SEQUENCE of frank-archon bench arms and keep
|
||||
# every arm's evidence. Runs on the operator box; each arm is one full window
|
||||
# of ./autoresearch.sh (stop+mask -> boot the arm as the service -> measure ->
|
||||
# restore production and verify), so the fleet is only ever down for one arm at
|
||||
# a time and the production unit is back up between arms.
|
||||
#
|
||||
# An arm is the repo's own frank-primary.env plus an OVERLAY of engine knobs
|
||||
# An arm is the repo's own frank-archon.env plus an OVERLAY of engine knobs
|
||||
# appended after it (later assignment wins when the serve script sources it),
|
||||
# staged through FRANK_PRIMARY_AR_ENV_FILE so the checkout is never mutated and
|
||||
# staged through FRANK_ARCHON_AR_ENV_FILE so the checkout is never mutated and
|
||||
# each arm gets its own fingerprint (the memo can never answer one arm with
|
||||
# another arm's numbers).
|
||||
#
|
||||
# Usage: scripts/autoresearch/frank-primary-arms.sh [arm ...]
|
||||
# scripts/autoresearch/frank-primary-arms.sh --list
|
||||
# Output, under /tmp/frank-primary-arms-<UTC stamp>/ (override with ARMS_DIR):
|
||||
# Usage: scripts/autoresearch/frank-archon-arms.sh [arm ...]
|
||||
# scripts/autoresearch/frank-archon-arms.sh --list
|
||||
# Output, under /tmp/frank-archon-arms-<UTC stamp>/ (override with ARMS_DIR):
|
||||
#
|
||||
# <arm>.out full autoresearch output (metrics + engine evidence)
|
||||
# <arm>.env the exact staged env
|
||||
@@ -26,8 +26,8 @@ set -uo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO="$(cd -- "$SCRIPT_DIR/../.." && pwd)"
|
||||
BASE_ENV="$REPO/scripts/frank-primary.env"
|
||||
ARMS_DIR="${ARMS_DIR:-/tmp/frank-primary-arms-$(date -u +%Y%m%dT%H%M%SZ)}"
|
||||
BASE_ENV="$REPO/scripts/frank-archon.env"
|
||||
ARMS_DIR="${ARMS_DIR:-/tmp/frank-archon-arms-$(date -u +%Y%m%dT%H%M%SZ)}"
|
||||
|
||||
# --- arms ------------------------------------------------------------------
|
||||
# baseline: the shipped shape E, measured with THIS harness so the comparison is
|
||||
@@ -45,18 +45,18 @@ LADDER3='{"cudagraph_capture_sizes":[4,8,12],"max_cudagraph_capture_size":12,"in
|
||||
# the engine as {temperature:0.7}.
|
||||
GEN07='{"temperature":0.7}'
|
||||
OVER_B=(
|
||||
"FRANK_PRIMARY_SPEC='$SPEC3'"
|
||||
"FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
|
||||
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
|
||||
"FRANK_ARCHON_SPEC='$SPEC3'"
|
||||
"FRANK_ARCHON_COMPCONFIG='$LADDER3'"
|
||||
'FRANK_ARCHON_GEN_CONFIG='"'$GEN07'"
|
||||
)
|
||||
# B2: identical to B with the default (7.60 GiB) overhead, i.e. precise memory
|
||||
# OFF. It is the fallback if B dies at graph capture, and it prices the arena
|
||||
# that precise memory buys (7,522 slots -> ~6,496).
|
||||
OVER_B2=(
|
||||
"FRANK_PRIMARY_SPEC='$SPEC3'"
|
||||
"FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
|
||||
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
|
||||
'FRANK_PRIMARY_PRECISE_MEMORY='
|
||||
"FRANK_ARCHON_SPEC='$SPEC3'"
|
||||
"FRANK_ARCHON_COMPCONFIG='$LADDER3'"
|
||||
'FRANK_ARCHON_GEN_CONFIG='"'$GEN07'"
|
||||
'FRANK_ARCHON_PRECISE_MEMORY='
|
||||
)
|
||||
# C: the vendor's concurrency shape — SEQS 16 with the [4..32] ladder and util
|
||||
# 0.95. This is also the direct test of the theory that pinning the capture
|
||||
@@ -64,12 +64,12 @@ OVER_B2=(
|
||||
# the constant 3, which is NOT in the default ladder; 4 is).
|
||||
LADDER16='{"cudagraph_capture_sizes":[4,8,12,16,20,24,28,32],"max_cudagraph_capture_size":32,"inductor_compile_config":{"combo_kernels":false,"benchmark_combo_kernel":false}}'
|
||||
OVER_C=(
|
||||
"FRANK_PRIMARY_SPEC='$SPEC3'"
|
||||
"FRANK_PRIMARY_COMPCONFIG='$LADDER16'"
|
||||
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
|
||||
'FRANK_PRIMARY_SEQS=16'
|
||||
'FRANK_PRIMARY_MNBT=4096'
|
||||
'FRANK_PRIMARY_UTIL=0.95'
|
||||
"FRANK_ARCHON_SPEC='$SPEC3'"
|
||||
"FRANK_ARCHON_COMPCONFIG='$LADDER16'"
|
||||
'FRANK_ARCHON_GEN_CONFIG='"'$GEN07'"
|
||||
'FRANK_ARCHON_SEQS=16'
|
||||
'FRANK_ARCHON_MNBT=4096'
|
||||
'FRANK_ARCHON_UTIL=0.95'
|
||||
)
|
||||
|
||||
# D/E: the CONTEXT arms. Measured 2026-10-02 (arms run 20261002T233855Z): the
|
||||
@@ -83,11 +83,11 @@ OVER_C=(
|
||||
# D is arm B with EXACTLY ONE variable changed (the KV bytes), so its result
|
||||
# says whether the 4k-depth collapse (B 23.09 vs baseline 33.99) was KV or the
|
||||
# batch shape.
|
||||
KV401K='FRANK_PRIMARY_KV_BYTES=4050000000'
|
||||
KV401K='FRANK_ARCHON_KV_BYTES=4050000000'
|
||||
OVER_D=(
|
||||
"FRANK_PRIMARY_SPEC='$SPEC3'"
|
||||
"FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
|
||||
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
|
||||
"FRANK_ARCHON_SPEC='$SPEC3'"
|
||||
"FRANK_ARCHON_COMPCONFIG='$LADDER3'"
|
||||
'FRANK_ARCHON_GEN_CONFIG='"'$GEN07'"
|
||||
"$KV401K"
|
||||
)
|
||||
# E: D with C's max_num_batched_tokens (4096 — C is the only arm that did NOT
|
||||
@@ -97,12 +97,12 @@ OVER_D=(
|
||||
# only 0.32 GiB above the allowance while the CUDA-graph cost is an ANALYTIC
|
||||
# estimate (C's was 1.47 GiB over 24 graphs), so it is measured, not assumed.
|
||||
OVER_E=(
|
||||
"FRANK_PRIMARY_SPEC='$SPEC3'"
|
||||
"FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
|
||||
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
|
||||
"FRANK_ARCHON_SPEC='$SPEC3'"
|
||||
"FRANK_ARCHON_COMPCONFIG='$LADDER3'"
|
||||
'FRANK_ARCHON_GEN_CONFIG='"'$GEN07'"
|
||||
"$KV401K"
|
||||
'FRANK_PRIMARY_MNBT=4096'
|
||||
'FRANK_PRIMARY_UTIL=0.99'
|
||||
'FRANK_ARCHON_MNBT=4096'
|
||||
'FRANK_ARCHON_UTIL=0.99'
|
||||
)
|
||||
|
||||
arm_overlay() {
|
||||
@@ -142,7 +142,7 @@ for arm in "${ARMS[@]}"; do
|
||||
# holds JSON must still be one valid JSON object after the file is SOURCED
|
||||
# (that is the step that silently eats unquoted braces).
|
||||
if ! bash -c 'set -a; . "$1" >/dev/null; set +a
|
||||
for k in FRANK_PRIMARY_SPEC FRANK_PRIMARY_COMPCONFIG FRANK_PRIMARY_GEN_CONFIG; do
|
||||
for k in FRANK_ARCHON_SPEC FRANK_ARCHON_COMPCONFIG FRANK_ARCHON_GEN_CONFIG; do
|
||||
v="${!k:-}"
|
||||
[ -n "$v" ] || continue
|
||||
case "$v" in
|
||||
@@ -158,7 +158,7 @@ for arm in "${ARMS[@]}"; do
|
||||
fi
|
||||
|
||||
echo "[arms] === $arm: opening window $(date -u +%H:%M:%S)Z ===" >&2
|
||||
FRANK_PRIMARY_AR_ENV_FILE="$ARMS_DIR/$arm.env" \
|
||||
FRANK_ARCHON_AR_ENV_FILE="$ARMS_DIR/$arm.env" \
|
||||
"$REPO/autoresearch.sh" >"$ARMS_DIR/$arm.out" 2>&1
|
||||
rc=$?
|
||||
echo "$rc" >"$ARMS_DIR/$arm.rc"
|
||||
+2
-2
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""frank-primary decode measurement — runs INSIDE the engine container.
|
||||
"""frank-archon decode measurement — runs INSIDE the engine container.
|
||||
|
||||
The method is the fleet's established one (the 2026-09-29 / 2026-10-01
|
||||
tuning campaigns, /home/dank/mtp_measure.py): SSE chunk counting is invalid
|
||||
@@ -28,7 +28,7 @@ Workload (fixed, deterministic, offline):
|
||||
|
||||
Env:
|
||||
AR_BASE endpoint (default http://127.0.0.1:7770)
|
||||
AR_MODEL served name (default frank-primary)
|
||||
AR_MODEL served name (default frank-archon)
|
||||
AR_PROMPT_CEILING hard prompt ceiling in tokens (default 100000)
|
||||
AR_TEMPS comma-separated sampling temperatures (default "0,0.7")
|
||||
|
||||
+14
-14
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# frank-primary-window.sh — one full bench window for frank-primary, run ON
|
||||
# frank-archon-window.sh — one full bench window for frank-archon, run ON
|
||||
# DANKSERVER as the deploy user (invoked over ssh by ./autoresearch.sh).
|
||||
#
|
||||
# Discipline (AGENTS.md Benchmarking Policy): the production unit is
|
||||
@@ -12,9 +12,9 @@
|
||||
#
|
||||
# Staging layout (created by the orchestrator, ephemeral — never a config
|
||||
# source; the repo checkout is):
|
||||
# /home/dank/.autoresearch-frank-primary/
|
||||
# /home/dank/.autoresearch-frank-archon/
|
||||
# serve.sh staged candidate serve script
|
||||
# frank-primary.env staged candidate env (the candidate knobs)
|
||||
# frank-archon.env staged candidate env (the candidate knobs)
|
||||
# measure.py the in-container measure client
|
||||
# hash candidate fingerprint
|
||||
# run.json result (metrics + boot + kv + fingerprint)
|
||||
@@ -27,16 +27,16 @@
|
||||
# outcome; a failed restore forces exit 1).
|
||||
set -uo pipefail
|
||||
|
||||
STAGE="${FRANK_PRIMARY_AR_STAGE:-/home/dank/.autoresearch-frank-primary}"
|
||||
UNIT="${FRANK_PRIMARY_AR_UNIT:-frank-primary}"
|
||||
CTR="${FRANK_PRIMARY_AR_CTR:-frank-primary-qwen}"
|
||||
STAGE="${FRANK_ARCHON_AR_STAGE:-/home/dank/.autoresearch-frank-archon}"
|
||||
UNIT="${FRANK_ARCHON_AR_UNIT:-frank-archon}"
|
||||
CTR="${FRANK_ARCHON_AR_CTR:-frank-archon-qwen}"
|
||||
HEALTH_URL="http://127.0.0.1:7770/health"
|
||||
MODELS_URL="http://127.0.0.1:7770/v1/models"
|
||||
BOOT_TIMEOUT="${FRANK_PRIMARY_AR_BOOT_TIMEOUT:-1200}"
|
||||
RESTORE_TIMEOUT="${FRANK_PRIMARY_AR_RESTORE_TIMEOUT:-1200}"
|
||||
MEASURE_TIMEOUT="${FRANK_PRIMARY_AR_MEASURE_TIMEOUT:-1800}"
|
||||
SERVED="$(sed -n 's/^FRANK_PRIMARY_SERVED_NAME=//p' "$STAGE/frank-primary.env" 2>/dev/null | head -1)"
|
||||
SERVED="${SERVED:-frank-primary}"
|
||||
BOOT_TIMEOUT="${FRANK_ARCHON_AR_BOOT_TIMEOUT:-1200}"
|
||||
RESTORE_TIMEOUT="${FRANK_ARCHON_AR_RESTORE_TIMEOUT:-1200}"
|
||||
MEASURE_TIMEOUT="${FRANK_ARCHON_AR_MEASURE_TIMEOUT:-1800}"
|
||||
SERVED="$(sed -n 's/^FRANK_ARCHON_SERVED_NAME=//p' "$STAGE/frank-archon.env" 2>/dev/null | head -1)"
|
||||
SERVED="${SERVED:-frank-archon}"
|
||||
|
||||
RESTORED=0
|
||||
|
||||
@@ -72,7 +72,7 @@ capture_bench_failure() {
|
||||
echo "=== staged serve.out (tail 60) ==="
|
||||
tail -60 "$STAGE/serve.out" 2>/dev/null || true
|
||||
echo "=== engine log (tail 60) ==="
|
||||
tail -60 /home/dank/logs/frank_primary.log 2>/dev/null || true
|
||||
tail -60 /home/dank/logs/frank_archon.log 2>/dev/null || true
|
||||
} >"$STAGE/fail.log" 2>&1 || true
|
||||
grep -E 'OutOfMemory|Tried to allocate|Engine core initialization failed|Error|error' \
|
||||
"$STAGE/fail.log" 2>/dev/null | tail -25 >&2 || true
|
||||
@@ -114,7 +114,7 @@ trap 'restore; rc=$?; [ "$rc" -ne 0 ] && exit 1' EXIT
|
||||
|
||||
main() {
|
||||
local f
|
||||
for f in serve.sh frank-primary.env measure.py hash; do
|
||||
for f in serve.sh frank-archon.env measure.py hash; do
|
||||
[ -f "$STAGE/$f" ] || { log "staged file missing: $STAGE/$f"; exit 1; }
|
||||
done
|
||||
mkdir -p "$STAGE/last"
|
||||
@@ -133,7 +133,7 @@ main() {
|
||||
|
||||
t0="$(date +%s)"
|
||||
log "starting bench instance from staged config (fingerprint $(tr -d '[:space:]' <"$STAGE/hash"))"
|
||||
FRANK_PRIMARY_ENV_FILE="$STAGE/frank-primary.env" \
|
||||
FRANK_ARCHON_ENV_FILE="$STAGE/frank-archon.env" \
|
||||
setsid nohup bash "$STAGE/serve.sh" >"$STAGE/serve.out" 2>&1 </dev/null &
|
||||
echo "$!" >"$STAGE/serve.pid"
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# Decode throughput benchmark for frank-large.
|
||||
# Decode throughput benchmark for frank-dragoon.
|
||||
# Measures generation tok/s with 8K+ token prompts and 256 generated tokens.
|
||||
set -euo pipefail
|
||||
|
||||
@@ -75,7 +75,7 @@ print(json.dumps(' '.join(words)))
|
||||
t0=$(date +%s%N)
|
||||
resp=$(curl -s -m 600 http://127.0.0.1:$PORT/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"model\":\"frank-large\",\"messages\":[{\"role\":\"user\",\"content\":$prompt}],\"max_tokens\":$N_GEN,\"stream\":false}" 2>&1)
|
||||
-d "{\"model\":\"frank-dragoon\",\"messages\":[{\"role\":\"user\",\"content\":$prompt}],\"max_tokens\":$N_GEN,\"stream\":false}" 2>&1)
|
||||
t1=$(date +%s%N)
|
||||
|
||||
if ! echo "$resp" | python3 -c "import sys,json; json.load(sys.stdin)" 2>/dev/null; then
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
HTTP-based prefill-throughput sweep for frank-large.
|
||||
HTTP-based prefill-throughput sweep for frank-dragoon.
|
||||
Starts the server with a given --threads-batch value, sends a 64K-prompt via
|
||||
the chat API (max_tokens=0), measures prompt_tokens/wall_time.
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# Threads-batch sweep for frank-large on dankclaw
|
||||
# Threads-batch sweep for frank-dragoon on dankclaw
|
||||
# Benches prefill throughput at 65536 prompt tokens across threads-batch values.
|
||||
set -uo pipefail
|
||||
|
||||
@@ -10,7 +10,7 @@ TB_LIST=(16 32 48 64 80 96 128)
|
||||
ENV_FILE="/home/dank/GitPackages/ROCmFPX/build-qwen38/bin"
|
||||
|
||||
echo "========================================"
|
||||
echo "THREADS-BATCH SWEEP — frank-large"
|
||||
echo "THREADS-BATCH SWEEP — frank-dragoon"
|
||||
echo "Model: $(basename "$MODEL") ($(du -h "$MODEL" | cut -f1))"
|
||||
echo "Threads-batch values: ${TB_LIST[*]}"
|
||||
echo "Prompt: 65536 tokens, n-gen: 0 (prefill-only)"
|
||||
|
||||
@@ -1,14 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
# Benchmark ubatch with llama-bench for frank-large.
|
||||
# Benchmark ubatch with llama-bench for frank-dragoon.
|
||||
#
|
||||
# This is intentionally a ubatch-only benchmark. It uses one llama-bench
|
||||
# process, an 8K prompt workload, and equal batch/ubatch values. No server
|
||||
# process, HTTP requests, or log scraping are used.
|
||||
#
|
||||
# Run on Dankclaw after stopping frank-large:
|
||||
# sudo systemctl stop frank-large
|
||||
# Run on Dankclaw after stopping frank-dragoon:
|
||||
# sudo systemctl stop frank-dragoon
|
||||
# ./scripts/bench-ubatch.sh
|
||||
# sudo systemctl start frank-large
|
||||
# sudo systemctl start frank-dragoon
|
||||
set -euo pipefail
|
||||
|
||||
: "${LLAMA_BENCH:=/home/dank/GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-bench}"
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
# build-frank-primary-rad.sh — build frank-primary's .rad container from the
|
||||
# build-frank-archon-rad.sh — build frank-archon's .rad container from the
|
||||
# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint, at NATIVE precision.
|
||||
#
|
||||
# bash scripts/build-frank-primary-rad.sh # convert
|
||||
# bash scripts/build-frank-primary-rad.sh --plan-only # declare, no I/O
|
||||
# bash scripts/build-frank-primary-rad.sh --restage # redo staging
|
||||
# bash scripts/build-frank-primary-rad.sh --force # rebuild existing
|
||||
# bash scripts/build-frank-archon-rad.sh # convert
|
||||
# bash scripts/build-frank-archon-rad.sh --plan-only # declare, no I/O
|
||||
# bash scripts/build-frank-archon-rad.sh --restage # redo staging
|
||||
# bash scripts/build-frank-archon-rad.sh --force # rebuild existing
|
||||
#
|
||||
# WHY A STAGING DIR (the one non-obvious step): rad-convert selects its
|
||||
# architecture plugin by the checkpoint's quantisation DESCRIPTOR —
|
||||
@@ -55,8 +55,8 @@ OUT="$OUT_DIR/qwen4exp-fp8-native.rad"
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "$SCRIPT_DIR/frank-primary.env"
|
||||
IMAGE="$FRANK_PRIMARY_IMAGE"
|
||||
source "$SCRIPT_DIR/frank-archon.env"
|
||||
IMAGE="$FRANK_ARCHON_IMAGE"
|
||||
|
||||
plan_only=0 restage=0 force=0
|
||||
for a in "$@"; do
|
||||
@@ -69,11 +69,11 @@ for a in "$@"; do
|
||||
done
|
||||
|
||||
if [ ! -f "$SRC/config.json" ]; then
|
||||
echo "ERROR: raw checkpoint missing: $SRC (run scripts/download-frank-primary.sh --fetch)" >&2
|
||||
echo "ERROR: raw checkpoint missing: $SRC (run scripts/download-frank-archon.sh --fetch)" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "verifying raw checkpoint first..."
|
||||
bash "$SCRIPT_DIR/download-frank-primary.sh"
|
||||
bash "$SCRIPT_DIR/download-frank-archon.sh"
|
||||
|
||||
if [ -f "$OUT" ] && [ "$plan_only" -eq 0 ] && [ "$force" -eq 0 ]; then
|
||||
echo "container already exists: $OUT"
|
||||
@@ -83,7 +83,7 @@ fi
|
||||
|
||||
# ---- staging ----------------------------------------------------------------
|
||||
if [ "$restage" -eq 1 ] && [ -e "$STAGE" ]; then
|
||||
TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-primary-stage"
|
||||
TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-archon-stage"
|
||||
mkdir -p "$TRASH"
|
||||
mv "$STAGE" "$TRASH/stage-$(date +%H%M%S)"
|
||||
echo "old staging dir moved to $TRASH"
|
||||
@@ -138,7 +138,7 @@ CONVERT_ARGS=(/models/stage-qwen4exp-fp8 -o /models/rad/qwen4exp-fp8-native.rad
|
||||
# exactly and the plugin's one-element scale plane is produced.
|
||||
CONVERT_ARGS+=(--quant "blk.*.ple_ngram.weight=rtn:codes=fp8_e4m3,block=*x*,scale=bf16,rule=fixed,scale_value=$PLE_SCALE")
|
||||
[ "$force" -eq 1 ] && [ -f "$OUT" ] && {
|
||||
TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-primary-rad"
|
||||
TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-archon-rad"
|
||||
mkdir -p "$TRASH"
|
||||
mv "$OUT" "$TRASH/qwen4exp-fp8-native.rad.$(date +%H%M%S)"
|
||||
echo "previous container moved to $TRASH"
|
||||
@@ -158,8 +158,8 @@ echo "rad-info summary:"
|
||||
podman run --rm --entrypoint rad-info -v /home/dank/models:/models:ro \
|
||||
"$IMAGE" "/models/rad/qwen4exp-fp8-native.rad" | head -20
|
||||
MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
|
||||
echo "container: $OUT (${MIB} MiB; env floor $FRANK_PRIMARY_RAD_MIN_MIB MiB)"
|
||||
if [ "$MIB" -lt "$FRANK_PRIMARY_RAD_MIN_MIB" ]; then
|
||||
echo "container: $OUT (${MIB} MiB; env floor $FRANK_ARCHON_RAD_MIN_MIB MiB)"
|
||||
if [ "$MIB" -lt "$FRANK_ARCHON_RAD_MIN_MIB" ]; then
|
||||
echo "ERROR: container below the env floor — the convert was truncated" >&2
|
||||
exit 1
|
||||
fi
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build the gufo runtime that serves frank-large on Dankclaw.
|
||||
# Build the gufo runtime that serves frank-dragoon on Dankclaw.
|
||||
#
|
||||
# bash scripts/build-gufo.sh # build + install the pinned tag
|
||||
# bash scripts/build-gufo.sh --verify # only check the installed tree
|
||||
@@ -93,4 +93,4 @@ cmake --install build/release
|
||||
|
||||
echo "== verify =="
|
||||
"$PREFIX/bin/gufo" diagnose
|
||||
echo "frank-large runtime ready: $PREFIX/bin/gufo ($GUFO_TAG @ $commit)"
|
||||
echo "frank-dragoon runtime ready: $PREFIX/bin/gufo ($GUFO_TAG @ $commit)"
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build the ROCmFPX runtime that serves frank-large on Dankclaw.
|
||||
# Build the ROCmFPX runtime that serves frank-dragoon on Dankclaw.
|
||||
#
|
||||
# bash scripts/build-rocmfpx.sh # clone/fetch + build the pin
|
||||
# bash scripts/build-rocmfpx.sh --verify # only check the installed tree
|
||||
#
|
||||
# Runtime: kingjones30/ROCmFPX — charlie12345/ROCmFPX plus the model-card
|
||||
# patches, i.e. the fork that owns the ROCmFPx tensor formats. frank-large's
|
||||
# patches, i.e. the fork that owns the ROCmFPx tensor formats. frank-dragoon's
|
||||
# payload is ftype 115 (`Q8_0_ROCMFPX_AGENT`): 311 tensors of
|
||||
# GGML_TYPE_Q8_0_ROCMFPX (103), a type that exists ONLY in this fork, so
|
||||
# stock llama.cpp, gufo and every other loader on the host reject the file.
|
||||
@@ -30,7 +30,7 @@
|
||||
set -euo pipefail
|
||||
|
||||
ROCMFPX_REPO="${ROCMFPX_REPO:-https://github.com/kingjones30/ROCmFPX.git}"
|
||||
# The commit frank-large was qualified with (measured 2026-10-02).
|
||||
# The commit frank-dragoon was qualified with (measured 2026-10-02).
|
||||
ROCMFPX_COMMIT="${ROCMFPX_COMMIT:-36e9acd40e10a87cd3c3ef8ec734668757dc8520}"
|
||||
SRC_DIR="${ROCMFPX_SRC_DIR:-/home/dank/GitPackages/kingjones-ROCmFPX}"
|
||||
BUILD_DIR="${ROCMFPX_BUILD_DIR:-$SRC_DIR/build}"
|
||||
@@ -86,4 +86,4 @@ cmake --build "$BUILD_DIR" --target llama-server llama-quantize -j "$JOBS"
|
||||
|
||||
echo "== verify =="
|
||||
"$BUILD_DIR/bin/llama-server" --version | head -1
|
||||
echo "frank-large runtime ready: $BUILD_DIR/bin/llama-server ($ROCMFPX_COMMIT)"
|
||||
echo "frank-dragoon runtime ready: $BUILD_DIR/bin/llama-server ($ROCMFPX_COMMIT)"
|
||||
|
||||
@@ -27,7 +27,8 @@ CALIB=/home/dank/corpora/calib
|
||||
OUT=/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad
|
||||
LOG=/home/dank/logs/swift15_convert.log
|
||||
# gate_up ~37.6 + down ~18.8 + PLE 47.7 + trunk/embeddings/head ~10 GiB (all-w4 expert table;
|
||||
# the plugin's one-encoding law dropped the block-fp8 down tier -- see the recipe header).
|
||||
# the format assigns ONE encoding per expert table, so a per-tensor block-fp8
|
||||
# down tier cannot be served beside a w4 gate_up tier -- see the recipe header).
|
||||
MIN_MIB=${FRANK_SWIFT15_RAD_MIN_MIB:-108000}
|
||||
|
||||
# Gate 1: inputs. A missing Gram directory converts UNCALIBRATED with a
|
||||
|
||||
@@ -57,9 +57,9 @@ else
|
||||
echo " none"
|
||||
fi
|
||||
|
||||
section "frank-primary unit"
|
||||
printf ' active: %s\n' "$(systemctl is-active frank-primary 2>/dev/null | head -1)"
|
||||
printf ' enabled: %s\n' "$(systemctl is-enabled frank-primary 2>/dev/null | head -1)"
|
||||
section "frank-archon unit"
|
||||
printf ' active: %s\n' "$(systemctl is-active frank-archon 2>/dev/null | head -1)"
|
||||
printf ' enabled: %s\n' "$(systemctl is-enabled frank-archon 2>/dev/null | head -1)"
|
||||
|
||||
section "glm serve tail"
|
||||
if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qx "$GLM_CTR"; then
|
||||
|
||||
+23
-23
@@ -61,9 +61,9 @@ SERVICE_ORDER = (
|
||||
"sonarr",
|
||||
"lidarr",
|
||||
"readarr",
|
||||
"frank-large",
|
||||
"frank-secondary",
|
||||
"frank-primary",
|
||||
"frank-dragoon",
|
||||
"frank-zealot",
|
||||
"frank-archon",
|
||||
)
|
||||
|
||||
DANKCLAW_SERVICES = {
|
||||
@@ -93,8 +93,8 @@ DANKCLAW_SERVICES = {
|
||||
"authentik-outpost-ldap",
|
||||
"caddy",
|
||||
"homepage",
|
||||
"frank-large",
|
||||
"frank-secondary",
|
||||
"frank-dragoon",
|
||||
"frank-zealot",
|
||||
}
|
||||
|
||||
DANKSERVER_SERVICES = {
|
||||
@@ -113,7 +113,7 @@ DANKSERVER_SERVICES = {
|
||||
"sonarr",
|
||||
"lidarr",
|
||||
"readarr",
|
||||
"frank-primary",
|
||||
"frank-archon",
|
||||
"node-exporter",
|
||||
}
|
||||
|
||||
@@ -316,7 +316,7 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
if len(parts) >= 2 and parts[1] == "open-webui":
|
||||
_add(services, "open-webui")
|
||||
return services
|
||||
# frank-large left the quadlet shape on 2026-09-19 (CIRU v4.4.1 host
|
||||
# frank-dragoon left the quadlet shape on 2026-09-19 (CIRU v4.4.1 host
|
||||
# build, root unit): its files live in scripts/ and map there.
|
||||
if len(parts) >= 2 and parts[1] == "observability":
|
||||
# The five observability containers became Quadlet units on
|
||||
@@ -371,10 +371,10 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
if name == "node-exporter":
|
||||
_add(services, "node-exporter")
|
||||
return services
|
||||
# frank-primary's quadlet-era directory left the tree on 2026-09-26
|
||||
# frank-archon's quadlet-era directory left the tree on 2026-09-26
|
||||
# (the Lucebox DS4.1 unit was retired for the llama.cpp root unit —
|
||||
# the trio in scripts/ now maps it). Nothing lives under
|
||||
# dankserver/frank-primary/ except dated trash, and trash plans
|
||||
# dankserver/frank-archon/ except dated trash, and trash plans
|
||||
# nothing (the retire rule above the service blocks).
|
||||
return services
|
||||
|
||||
@@ -400,32 +400,32 @@ def services_for_path(path: str, target: str) -> set[str]:
|
||||
# change to it redeploying alertmanager is the only path that
|
||||
# reaches it on the host.
|
||||
_add(services, "alertmanager")
|
||||
if name in {"frank-secondary.env", "frank-secondary.service", "serve-frank-secondary.sh"}:
|
||||
# llama.cpp frank-secondary (2026-09-16) keeps its unit, env and serve
|
||||
# files here; each one selects frank-secondary only. The ik_llama.cpp
|
||||
if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
|
||||
# llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
|
||||
# files here; each one selects frank-zealot only. The ik_llama.cpp
|
||||
# trio that lived here for two days is in scripts/.AGENTTRASH/,
|
||||
# which is archive material and plans nothing.
|
||||
_add(services, "frank-secondary")
|
||||
if name in {"frank-large.env", "frank-large.service", "serve-frank-large.sh"}:
|
||||
# CIRU v4.4.1 frank-large (2026-09-19) keeps its unit, env and
|
||||
_add(services, "frank-zealot")
|
||||
if name in {"frank-dragoon.env", "frank-dragoon.service", "serve-frank-dragoon.sh"}:
|
||||
# CIRU v4.4.1 frank-dragoon (2026-09-19) keeps its unit, env and
|
||||
# serve files here the same way: a root unit on a host-built
|
||||
# runtime. The download script fetches the payload, which is
|
||||
# host-side state, so it selects nothing.
|
||||
_add(services, "frank-large")
|
||||
_add(services, "frank-dragoon")
|
||||
if name == "qwen-sharp-v22.4.0.jinja":
|
||||
# The Qwen-Sharp chat template frank-large loads via
|
||||
# The Qwen-Sharp chat template frank-dragoon loads via
|
||||
# --chat-template-file. It lived under dankserver/frank-medium/ until
|
||||
# the 2026-09-26 cutover retired that directory; editing it changes
|
||||
# what the live frank-large endpoint renders, so it plans
|
||||
# frank-large.
|
||||
_add(services, "frank-large")
|
||||
if name in {"frank-primary.env", "frank-primary.service", "serve-frank-primary.sh"}:
|
||||
# llama.cpp frank-primary (2026-09-26, replacing the Lucebox
|
||||
# what the live frank-dragoon endpoint renders, so it plans
|
||||
# frank-dragoon.
|
||||
_add(services, "frank-dragoon")
|
||||
if name in {"frank-archon.env", "frank-archon.service", "serve-frank-archon.sh"}:
|
||||
# llama.cpp frank-archon (2026-09-26, replacing the Lucebox
|
||||
# DS4.1 quadlet) keeps its unit, env and serve files here the
|
||||
# same way: a root unit on the unsloth b11030-mix prebuilt
|
||||
# serving Qwen3.8-Flash-Next Q8_0. The download script fetches
|
||||
# the payload, which is host-side state, so it selects nothing.
|
||||
_add(services, "frank-primary")
|
||||
_add(services, "frank-archon")
|
||||
# Deployment/test/benchmark scripts and other repo tooling do not
|
||||
# restart a runtime service.
|
||||
return services
|
||||
|
||||
+115
-114
@@ -90,12 +90,12 @@ service_is_allowed() {
|
||||
dankclaw:authentik-postgres|dankclaw:authentik-server|\
|
||||
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
|
||||
dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
|
||||
dankclaw:frank-large|dankclaw:frank-secondary|\
|
||||
dankclaw:frank-dragoon|dankclaw:frank-zealot|\
|
||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
||||
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
|
||||
dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
|
||||
dankserver:frank-primary|dankserver:node-exporter|\
|
||||
dankserver:frank-archon|dankserver:node-exporter|\
|
||||
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
|
||||
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
|
||||
frankbook-pro:readarr)
|
||||
@@ -192,7 +192,7 @@ add_pending_services() {
|
||||
continue
|
||||
fi
|
||||
[[ -n "$recorded" && "$recorded" != "$REVISION" ]] || continue
|
||||
if [[ "$TARGET" == dankserver && "$service" == frank-large ]]; then continue; fi
|
||||
if [[ "$TARGET" == dankserver && "$service" == frank-dragoon ]]; then continue; fi
|
||||
if ! changed="$(python3 scripts/deploy-plan.py --target "$TARGET" \
|
||||
--repo "$REMOTE_DIR" --base "$recorded" --head "$REVISION")"; then
|
||||
echo "WARN: unable to compare state for $service; retaining it as pending" >&2
|
||||
@@ -947,7 +947,7 @@ deploy_plex() {
|
||||
# accepted and then answered with nothing (curl 52, Empty reply) against a
|
||||
# perfectly healthy server. Measured 2026-10-02 — the gate burned its full
|
||||
# 600s and failed a plex whose https://127.0.0.1:32400/identity answered
|
||||
# 200 in 44 ms the whole time. The same law is why the Caddyfile dials the
|
||||
# 200 in 44 ms the whole time. The same requirement is why the Caddyfile dials the
|
||||
# plex upstream over TLS (tests/caddyfile.bats pins it).
|
||||
# -k: PMS presents its own self-signed cert on 32400.
|
||||
#
|
||||
@@ -1092,21 +1092,21 @@ deploy_anubis() {
|
||||
wait_container_healthy anubis 60 || return 1
|
||||
}
|
||||
|
||||
# frank-large is the halogen-flash-server CONTAINER (0.16.2, digest-pinned
|
||||
# through scripts/frank-large.env) serving the Quat3rnion ABLITERATED
|
||||
# frank-dragoon is the halogen-flash-server CONTAINER (0.16.2, digest-pinned
|
||||
# through scripts/frank-dragoon.env) serving the Quat3rnion ABLITERATED
|
||||
# Swift1.5 finetune of peonist's Qwen3.8-Flash-Next v2 checkpoint —
|
||||
# qwen38-flash-next-v2-swift15-abliterated.hgn (66,687,678,464 B, ~62 GiB
|
||||
# resident) + the REQUIRED n-gram lookup table + the vision tower —
|
||||
# MTP spec decode (HALOGEN_SPEC_ADAPT=0), 2 slots x 200192, image input —
|
||||
# on the Strix Halo iGPU. It stays a ROOT unit because the serve script
|
||||
# drives `podman run` directly (the frank-primary pattern): relink_unit
|
||||
# points /etc/systemd/system/frank-large.service at the repo file, boot
|
||||
# drives `podman run` directly (the frank-archon pattern): relink_unit
|
||||
# points /etc/systemd/system/frank-dragoon.service at the repo file, boot
|
||||
# persistence is that unit's [Install], and LimitMEMLOCK=infinity is
|
||||
# fleet parity. 2026-10-06 cutover from the halogen 27B-dense shape —
|
||||
# rollback = revert the swap commit's trio; the 27B payload stays on
|
||||
# disk (~/models/halogen-qwen3.8-27b).
|
||||
#
|
||||
# The gates mirror serve-frank-large.sh's predicates EXACTLY (the
|
||||
# The gates mirror serve-frank-dragoon.sh's predicates EXACTLY (the
|
||||
# 2026-10-01 crash-loop lesson): image digest in dank's podman storage,
|
||||
# each .hgn payload (checkpoint, n-gram table, vision tower) at the exact
|
||||
# byte count, the tokenizer directory holding every named file as a
|
||||
@@ -1114,23 +1114,23 @@ deploy_anubis() {
|
||||
# artifact would leave a restart-looping unit and a dark endpoint. The
|
||||
# port gate is the same as every cutover since 2026-09-16: nothing
|
||||
# starts until 7770 is released.
|
||||
LLM_LARGE_IMAGE="${LLM_LARGE_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
|
||||
LLM_LARGE_CKPT="${LLM_LARGE_CKPT:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-v2-swift15-abliterated.hgn}"
|
||||
LLM_LARGE_CKPT_BYTES="${LLM_LARGE_CKPT_BYTES:-66687678464}"
|
||||
LLM_LARGE_NGRAM="${LLM_LARGE_NGRAM:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-ngram.hgn}"
|
||||
LLM_LARGE_NGRAM_BYTES="${LLM_LARGE_NGRAM_BYTES:-51200246144}"
|
||||
LLM_LARGE_VISION="${LLM_LARGE_VISION:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-vision.hgn}"
|
||||
LLM_LARGE_VISION_BYTES="${LLM_LARGE_VISION_BYTES:-897916416}"
|
||||
LLM_LARGE_TOK_DIR="${LLM_LARGE_TOK_DIR:-/home/dank/models/halogen-flashnext-v2/tokenizer}"
|
||||
LLM_LARGE_TOK_FILES="${LLM_LARGE_TOK_FILES:-tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json}"
|
||||
LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
|
||||
LLM_DRAGOON_CKPT="${LLM_DRAGOON_CKPT:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-v2-swift15-abliterated.hgn}"
|
||||
LLM_DRAGOON_CKPT_BYTES="${LLM_DRAGOON_CKPT_BYTES:-66687678464}"
|
||||
LLM_DRAGOON_NGRAM="${LLM_DRAGOON_NGRAM:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-ngram.hgn}"
|
||||
LLM_DRAGOON_NGRAM_BYTES="${LLM_DRAGOON_NGRAM_BYTES:-51200246144}"
|
||||
LLM_DRAGOON_VISION="${LLM_DRAGOON_VISION:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-vision.hgn}"
|
||||
LLM_DRAGOON_VISION_BYTES="${LLM_DRAGOON_VISION_BYTES:-897916416}"
|
||||
LLM_DRAGOON_TOK_DIR="${LLM_DRAGOON_TOK_DIR:-/home/dank/models/halogen-flashnext-v2/tokenizer}"
|
||||
LLM_DRAGOON_TOK_FILES="${LLM_DRAGOON_TOK_FILES:-tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json}"
|
||||
|
||||
retire_halogen_large_quadlet() {
|
||||
retire_halogen_dragoon_quadlet() {
|
||||
local trash="$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet"
|
||||
systemctl --user stop frank-large.service 2>/dev/null || true
|
||||
if [[ -e "$quadlet_unit_dir/frank-large.container" ]]; then
|
||||
systemctl --user stop frank-dragoon.service 2>/dev/null || true
|
||||
if [[ -e "$quadlet_unit_dir/frank-dragoon.container" ]]; then
|
||||
mkdir -p "$trash"
|
||||
mv -- "$quadlet_unit_dir/frank-large.container" "$trash/frank-large.container"
|
||||
echo "retired quadlet frank-large.container -> $trash"
|
||||
mv -- "$quadlet_unit_dir/frank-dragoon.container" "$trash/frank-dragoon.container"
|
||||
echo "retired quadlet frank-dragoon.container -> $trash"
|
||||
fi
|
||||
systemctl --user daemon-reload
|
||||
}
|
||||
@@ -1154,165 +1154,166 @@ wait_port_free() {
|
||||
return 1
|
||||
}
|
||||
|
||||
deploy_llm_large() {
|
||||
deploy_llm_dragoon() {
|
||||
# Refuse rather than restart-loop; every gate below has a twin in
|
||||
# serve-frank-large.sh and they must agree (2026-10-01 lesson).
|
||||
# serve-frank-dragoon.sh and they must agree (2026-10-01 lesson).
|
||||
# Capture podman output before grepping: `podman images | grep -q`
|
||||
# races under pipefail (SIGPIPE 141 becomes "no match", 2026-09-30).
|
||||
if ! grep -qF "${LLM_LARGE_IMAGE#*@}" <<<"$(podman images --format '{{.RepoDigests}}' 2>/dev/null)"; then
|
||||
echo "ERROR: halogen image absent: $LLM_LARGE_IMAGE" >&2
|
||||
echo "frank-large: pull it: podman pull ${LLM_LARGE_IMAGE%@*}" >&2
|
||||
if ! grep -qF "${LLM_DRAGOON_IMAGE#*@}" <<<"$(podman images --format '{{.RepoDigests}}' 2>/dev/null)"; then
|
||||
echo "ERROR: halogen image absent: $LLM_DRAGOON_IMAGE" >&2
|
||||
echo "frank-dragoon: pull it: podman pull ${LLM_DRAGOON_IMAGE%@*}" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ ! -f "$LLM_LARGE_CKPT" ]]; then
|
||||
echo "ERROR: halogen checkpoint absent at $LLM_LARGE_CKPT" >&2
|
||||
echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
|
||||
if [[ ! -f "$LLM_DRAGOON_CKPT" ]]; then
|
||||
echo "ERROR: halogen checkpoint absent at $LLM_DRAGOON_CKPT" >&2
|
||||
echo "frank-dragoon: fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ "$(stat -c %s "$LLM_LARGE_CKPT")" != "$LLM_LARGE_CKPT_BYTES" ]]; then
|
||||
echo "ERROR: halogen checkpoint is not $LLM_LARGE_CKPT_BYTES bytes (partial download?)" >&2
|
||||
echo "frank-large: re-check: bash scripts/download-frank-large-flashnext.sh --verify-only" >&2
|
||||
if [[ "$(stat -c %s "$LLM_DRAGOON_CKPT")" != "$LLM_DRAGOON_CKPT_BYTES" ]]; then
|
||||
echo "ERROR: halogen checkpoint is not $LLM_DRAGOON_CKPT_BYTES bytes (partial download?)" >&2
|
||||
echo "frank-dragoon: re-check: bash scripts/download-frank-dragoon-flashnext.sh --verify-only" >&2
|
||||
return 1
|
||||
fi
|
||||
for gate in "$LLM_LARGE_NGRAM:$LLM_LARGE_NGRAM_BYTES:n-gram table" \
|
||||
"$LLM_LARGE_VISION:$LLM_LARGE_VISION_BYTES:vision tower"; do
|
||||
for gate in "$LLM_DRAGOON_NGRAM:$LLM_DRAGOON_NGRAM_BYTES:n-gram table" \
|
||||
"$LLM_DRAGOON_VISION:$LLM_DRAGOON_VISION_BYTES:vision tower"; do
|
||||
IFS=: read -r gpath gbytes gname <<<"$gate"
|
||||
if [[ ! -f "$gpath" ]]; then
|
||||
echo "ERROR: halogen $gname absent at $gpath" >&2
|
||||
echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
|
||||
echo "frank-dragoon: fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ "$(stat -c %s "$gpath")" != "$gbytes" ]]; then
|
||||
echo "ERROR: halogen $gname is not $gbytes bytes (partial download?)" >&2
|
||||
echo "frank-large: re-check: bash scripts/download-frank-large-flashnext.sh --verify-only" >&2
|
||||
echo "frank-dragoon: re-check: bash scripts/download-frank-dragoon-flashnext.sh --verify-only" >&2
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
if [[ -L "$LLM_LARGE_TOK_DIR" || ! -d "$LLM_LARGE_TOK_DIR" ]]; then
|
||||
echo "ERROR: tokenizer dir missing or a symlink at $LLM_LARGE_TOK_DIR" >&2
|
||||
echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
|
||||
if [[ -L "$LLM_DRAGOON_TOK_DIR" || ! -d "$LLM_DRAGOON_TOK_DIR" ]]; then
|
||||
echo "ERROR: tokenizer dir missing or a symlink at $LLM_DRAGOON_TOK_DIR" >&2
|
||||
echo "frank-dragoon: fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
for tf in $LLM_LARGE_TOK_FILES; do
|
||||
if [[ ! -f "$LLM_LARGE_TOK_DIR/$tf" || -L "$LLM_LARGE_TOK_DIR/$tf" ]]; then
|
||||
echo "ERROR: tokenizer file missing or a symlink: $LLM_LARGE_TOK_DIR/$tf" >&2
|
||||
echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
|
||||
for tf in $LLM_DRAGOON_TOK_FILES; do
|
||||
if [[ ! -f "$LLM_DRAGOON_TOK_DIR/$tf" || -L "$LLM_DRAGOON_TOK_DIR/$tf" ]]; then
|
||||
echo "ERROR: tokenizer file missing or a symlink: $LLM_DRAGOON_TOK_DIR/$tf" >&2
|
||||
echo "frank-dragoon: fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
|
||||
retire_halogen_large_quadlet
|
||||
retire_halogen_dragoon_quadlet
|
||||
# The predecessor on a re-deploy is THIS unit: stop it (ExecStop stops
|
||||
# the container) before waiting on the port. The stray sweep is keyed
|
||||
# on the FULL container name, which appears in the argv of a manual
|
||||
# `podman run --name frank-large-halogen` / `podman wait` but NOT in
|
||||
# `podman run --name frank-dragoon-halogen` / `podman wait` but NOT in
|
||||
# this deployer's own command line (`bash deploy-remote.sh
|
||||
# frank-large`) — so it cannot shoot itself in the foot (the fleet
|
||||
# pkill law), and it can never match frank-secondary's llama-server.
|
||||
run_privileged systemctl stop frank-large.service 2>/dev/null || true
|
||||
pkill -f 'frank-large-halogen' 2>/dev/null || true
|
||||
podman rm -f frank-large frank-large-halogen >/dev/null 2>&1 || true
|
||||
# frank-dragoon`) — so the sweep can never match the deployer's own argv
|
||||
# (`pkill -f` kills the invoker when the pattern appears in the invoker's
|
||||
# own command line), and it can never match frank-zealot's llama-server.
|
||||
run_privileged systemctl stop frank-dragoon.service 2>/dev/null || true
|
||||
pkill -f 'frank-dragoon-halogen' 2>/dev/null || true
|
||||
podman rm -f frank-dragoon frank-dragoon-halogen >/dev/null 2>&1 || true
|
||||
# Nothing may start until the predecessor has released the port
|
||||
# (2026-09-16: "Deactivated successfully" was not "port free").
|
||||
wait_port_free 7770 || return 1
|
||||
relink_unit frank-large
|
||||
relink_unit frank-dragoon
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-large.service
|
||||
run_privileged systemctl restart frank-large.service
|
||||
wait_for_health frank-large http://127.0.0.1:7770/health 1200
|
||||
run_privileged systemctl enable frank-dragoon.service
|
||||
run_privileged systemctl restart frank-dragoon.service
|
||||
wait_for_health frank-dragoon http://127.0.0.1:7770/health 1200
|
||||
}
|
||||
|
||||
# deploy_llm_secondary — frank-secondary is llama.cpp (master build, CUDA sm_86) on
|
||||
# deploy_llm_zealot — frank-zealot is llama.cpp (master build, CUDA sm_86) on
|
||||
# the RTX 3090 as a ROOT unit: the runtime is a native binary tree under
|
||||
# ~/GitPackages/llama-master/build-cuda, so there is no container image and
|
||||
# no generated unit to avoid. relink_unit points
|
||||
# /etc/systemd/system/frank-secondary.service at the repo file and boot
|
||||
# /etc/systemd/system/frank-zealot.service at the repo file and boot
|
||||
# persistence is that unit's [Install] section.
|
||||
#
|
||||
# It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
|
||||
# not shared: frank-large's engine runs on the gfx1151 iGPU.
|
||||
LLM_SECONDARY_MODEL_FILE="${LLM_SECONDARY_MODEL_FILE:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
|
||||
LLM_SECONDARY_RUNTIME="${LLM_SECONDARY_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
|
||||
# It is a SEPARATE endpoint on port 7771; frank-dragoon keeps 7770. The 3090 is
|
||||
# not shared: frank-dragoon's engine runs on the gfx1151 iGPU.
|
||||
LLM_ZEALOT_MODEL_FILE="${LLM_ZEALOT_MODEL_FILE:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
|
||||
LLM_ZEALOT_RUNTIME="${LLM_ZEALOT_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
|
||||
# The projector IS the vision capability here: the model is natively
|
||||
# multimodal and the Q6_K file carries no vision tensors — the publisher ships
|
||||
# its own BF16 projector in the same repo, so a deploy that starts without
|
||||
# this file silently ships a text-only endpoint. Checked unconditionally, like
|
||||
# frank-large's.
|
||||
LLM_SECONDARY_MMPROJ="${LLM_SECONDARY_MMPROJ:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
|
||||
# frank-dragoon's.
|
||||
LLM_ZEALOT_MMPROJ="${LLM_ZEALOT_MMPROJ:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
|
||||
|
||||
deploy_llm_secondary() {
|
||||
deploy_llm_zealot() {
|
||||
# Refuse rather than restart-loop: a unit whose runtime or payload is
|
||||
# missing never becomes healthy, and a green deploy would leave the
|
||||
# endpoint dark with nothing in the log to say why.
|
||||
if [[ ! -x "$LLM_SECONDARY_RUNTIME" ]]; then
|
||||
echo "ERROR: llama.cpp runtime absent at $LLM_SECONDARY_RUNTIME" >&2
|
||||
echo "frank-secondary: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
|
||||
echo "frank-secondary: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
|
||||
echo "frank-secondary: && cmake --build build-cuda --target llama-server) first." >&2
|
||||
if [[ ! -x "$LLM_ZEALOT_RUNTIME" ]]; then
|
||||
echo "ERROR: llama.cpp runtime absent at $LLM_ZEALOT_RUNTIME" >&2
|
||||
echo "frank-zealot: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
|
||||
echo "frank-zealot: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
|
||||
echo "frank-zealot: && cmake --build build-cuda --target llama-server) first." >&2
|
||||
return 1
|
||||
fi
|
||||
for f in "$LLM_SECONDARY_MODEL_FILE" "$LLM_SECONDARY_MMPROJ"; do
|
||||
for f in "$LLM_ZEALOT_MODEL_FILE" "$LLM_ZEALOT_MMPROJ"; do
|
||||
if [[ ! -f "$f" ]]; then
|
||||
echo "ERROR: frank-secondary payload file absent at $f" >&2
|
||||
echo "frank-secondary: run scripts/download-frank-secondary.sh first;" >&2
|
||||
echo "frank-secondary: refusing to start a unit that cannot load." >&2
|
||||
echo "ERROR: frank-zealot payload file absent at $f" >&2
|
||||
echo "frank-zealot: run scripts/download-frank-zealot.sh first;" >&2
|
||||
echo "frank-zealot: refusing to start a unit that cannot load." >&2
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
relink_unit frank-secondary
|
||||
relink_unit frank-zealot
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-secondary.service
|
||||
run_privileged systemctl restart frank-secondary.service
|
||||
wait_for_health frank-secondary http://127.0.0.1:7771/health 900 || return 1
|
||||
# False-green hardening (frank-large's lesson): the port answering
|
||||
run_privileged systemctl enable frank-zealot.service
|
||||
run_privileged systemctl restart frank-zealot.service
|
||||
wait_for_health frank-zealot http://127.0.0.1:7771/health 900 || return 1
|
||||
# False-green hardening (frank-dragoon's lesson): the port answering
|
||||
# proves nothing about WHO answers — the unit must be active with a
|
||||
# live MainPID, or a stray SGLang holdover is masquerading as deployed.
|
||||
local main_pid
|
||||
main_pid="$(run_privileged systemctl show -p MainPID --value frank-secondary.service)"
|
||||
main_pid="$(run_privileged systemctl show -p MainPID --value frank-zealot.service)"
|
||||
if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
|
||||
echo "FAIL: :7771 answered but frank-secondary.service has no MainPID — a stray process holds the port" >&2
|
||||
echo "FAIL: :7771 answered but frank-zealot.service has no MainPID — a stray process holds the port" >&2
|
||||
return 1
|
||||
fi
|
||||
echo "frank-secondary: health answered by unit MainPID $main_pid"
|
||||
echo "frank-zealot: health answered by unit MainPID $main_pid"
|
||||
}
|
||||
|
||||
# deploy_llm_primary — frank-primary is the radiance engine (StillDeadcode,
|
||||
# deploy_llm_archon — frank-archon is the radiance engine (StillDeadcode,
|
||||
# digest-pinned stilldeadcode/radiance image) serving the OFFICIAL
|
||||
# Qwen/Qwen3.8-Flash-Next-FP8 at native block-FP8 precision from the
|
||||
# rad-convert container qwen4exp-fp8-native.rad, on BOTH R9700s (TP=2) as a
|
||||
# ROOT unit whose serve script drives `podman run` directly (the engine is
|
||||
# the image entrypoint; the image lives in dank's rootless podman storage).
|
||||
# 2026-10-05 cutover from FreeToken (pre-cutover tag
|
||||
# frank-primary-freetoken-pr587-20261005); before that, the tcclaviger vLLM
|
||||
# fork (tag frank-primary-vllm-shapeG-20261003). Same port and alias; the
|
||||
# frank-archon-freetoken-pr587-20261005); before that, the tcclaviger vLLM
|
||||
# fork (tag frank-archon-vllm-shapeG-20261003). Same port and alias; the
|
||||
# harness models.yml id is unchanged.
|
||||
#
|
||||
# Gates: the image must exist under its digest ref in dank's rootless
|
||||
# storage (FRANK_PRIMARY_IMAGE), the .rad container file must exist at the
|
||||
# env's size floor (FRANK_PRIMARY_MODEL_FILE >= FRANK_PRIMARY_RAD_MIN_MIB —
|
||||
# storage (FRANK_ARCHON_IMAGE), the .rad container file must exist at the
|
||||
# env's size floor (FRANK_ARCHON_MODEL_FILE >= FRANK_ARCHON_RAD_MIN_MIB —
|
||||
# a truncated convert is a guaranteed long-timeout boot), and the unit env
|
||||
# must be complete. Per-file sha256 verification is the fetch script's job
|
||||
# at download time and rad-convert/rad-info's job at build time, never
|
||||
# re-hashed here.
|
||||
LLM_PRIMARY_CONTAINER=frank-primary
|
||||
LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
|
||||
LLM_ARCHON_CONTAINER=frank-archon
|
||||
LLM_ARCHON_LEGACY_CONTAINERS="frank-archon-qwen frank-archon-r9v qwen-fp8 glm-probe"
|
||||
|
||||
deploy_llm_primary() {
|
||||
deploy_llm_archon() {
|
||||
# Refuse rather than restart-loop: a unit whose image or payload is
|
||||
# missing never becomes healthy, and a green deploy would leave the
|
||||
# endpoint dark with nothing in the log to say why.
|
||||
local env_file="/home/dank/serverAppConfigs/scripts/frank-primary.env"
|
||||
local env_file="/home/dank/serverAppConfigs/scripts/frank-archon.env"
|
||||
local image model_dir model_file rad_min mib
|
||||
if [[ ! -f "$env_file" ]]; then
|
||||
echo "ERROR: frank-primary env file absent: $env_file" >&2
|
||||
echo "ERROR: frank-archon env file absent: $env_file" >&2
|
||||
return 1
|
||||
fi
|
||||
image="$(sed -n 's/^FRANK_PRIMARY_IMAGE=//p' "$env_file" | tail -1)"
|
||||
model_dir="$(sed -n 's/^FRANK_PRIMARY_MODEL_DIR=//p' "$env_file" | tail -1)"
|
||||
model_file="$(sed -n 's/^FRANK_PRIMARY_MODEL_FILE=//p' "$env_file" | tail -1)"
|
||||
rad_min="$(sed -n 's/^FRANK_PRIMARY_RAD_MIN_MIB=//p' "$env_file" | tail -1)"
|
||||
image="$(sed -n 's/^FRANK_ARCHON_IMAGE=//p' "$env_file" | tail -1)"
|
||||
model_dir="$(sed -n 's/^FRANK_ARCHON_MODEL_DIR=//p' "$env_file" | tail -1)"
|
||||
model_file="$(sed -n 's/^FRANK_ARCHON_MODEL_FILE=//p' "$env_file" | tail -1)"
|
||||
rad_min="$(sed -n 's/^FRANK_ARCHON_RAD_MIN_MIB=//p' "$env_file" | tail -1)"
|
||||
if [[ -z "$image" || -z "$model_dir" || -z "$model_file" || -z "$rad_min" ]]; then
|
||||
echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_FILE/RAD_MIN_MIB" >&2
|
||||
echo "ERROR: frank-archon.env missing FRANK_ARCHON_IMAGE/MODEL_DIR/MODEL_FILE/RAD_MIN_MIB" >&2
|
||||
return 1
|
||||
fi
|
||||
# The image gate is the digest ref itself: a name@sha256 ref exists in
|
||||
@@ -1320,18 +1321,18 @@ deploy_llm_primary() {
|
||||
# pipefail/grep race (the 2026-09-30 lesson).
|
||||
if ! podman image exists "$image"; then
|
||||
echo "ERROR: runtime image absent from storage: $image" >&2
|
||||
echo "frank-primary: podman pull docker://stilldeadcode/radiance:latest" >&2
|
||||
echo "frank-archon: podman pull docker://stilldeadcode/radiance:latest" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ ! -f "$model_dir/$model_file" ]]; then
|
||||
echo "ERROR: .rad container missing: $model_dir/$model_file" >&2
|
||||
echo "frank-primary: run scripts/build-frank-primary-rad.sh" >&2
|
||||
echo "frank-archon: run scripts/build-frank-archon-rad.sh" >&2
|
||||
return 1
|
||||
fi
|
||||
mib=$(( $(stat -c %s "$model_dir/$model_file") / 1048576 ))
|
||||
if [[ "$mib" -lt "$rad_min" ]]; then
|
||||
echo "ERROR: .rad container too small: $model_dir/$model_file is ${mib} MiB, want >= $rad_min" >&2
|
||||
echo "frank-primary: rebuild it (scripts/build-frank-primary-rad.sh --force)" >&2
|
||||
echo "frank-archon: rebuild it (scripts/build-frank-archon-rad.sh --force)" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
@@ -1343,19 +1344,19 @@ deploy_llm_primary() {
|
||||
# The legacy identities (the retired R9V container, the ad hoc
|
||||
# experiment container qwen-fp8, the retired glm-probe) hold the same
|
||||
# GPUs and die here too.
|
||||
run_privileged systemctl stop frank-primary.service 2>/dev/null || true
|
||||
run_privileged systemctl stop frank-archon.service 2>/dev/null || true
|
||||
local ctr
|
||||
for ctr in $LLM_PRIMARY_LEGACY_CONTAINERS "$LLM_PRIMARY_CONTAINER"; do
|
||||
for ctr in $LLM_ARCHON_LEGACY_CONTAINERS "$LLM_ARCHON_CONTAINER"; do
|
||||
podman rm -f "$ctr" >/dev/null 2>&1 || true
|
||||
done
|
||||
# Nothing starts until the port is released: the lucebox cutover
|
||||
# (2026-09-26) and the frank-large cutover (2026-09-16) both failed
|
||||
# (2026-09-26) and the frank-dragoon cutover (2026-09-16) both failed
|
||||
# exactly here when the next engine bound 7770 a second too early.
|
||||
wait_port_free 7770 || return 1
|
||||
relink_unit frank-primary
|
||||
relink_unit frank-archon
|
||||
run_privileged systemctl daemon-reload
|
||||
run_privileged systemctl enable frank-primary.service
|
||||
run_privileged systemctl restart frank-primary.service
|
||||
run_privileged systemctl enable frank-archon.service
|
||||
run_privileged systemctl restart frank-archon.service
|
||||
# Cold start: .rad load + PLE pin + expert host-pool upload (the
|
||||
# publisher's compose healthcheck allows a 15-minute start_period).
|
||||
# Parity with every predecessor on this port. The HTTP listener binds
|
||||
@@ -1366,14 +1367,14 @@ deploy_llm_primary() {
|
||||
local i ok=0
|
||||
for ((i=1; i<=900; i++)); do
|
||||
if curl -fsS http://127.0.0.1:7770/health 2>/dev/null >/dev/null; then
|
||||
echo "frank-primary engine healthy after approximately $((i * 2))s"
|
||||
echo "frank-archon engine healthy after approximately $((i * 2))s"
|
||||
ok=1
|
||||
break
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
if [[ "$ok" != 1 ]]; then
|
||||
echo "FAIL: frank-primary /health never answered 200 in 1800s" >&2
|
||||
echo "FAIL: frank-archon /health never answered 200 in 1800s" >&2
|
||||
return 1
|
||||
fi
|
||||
# False-green hardening (the 2026-09-26 lesson): a bare curl against the
|
||||
@@ -1381,12 +1382,12 @@ deploy_llm_primary() {
|
||||
# it while the unit is stopped. The deploy is only green if the unit
|
||||
# itself is active with a live MainPID behind the health answer.
|
||||
local main_pid
|
||||
main_pid="$(run_privileged systemctl show -p MainPID --value frank-primary.service)"
|
||||
main_pid="$(run_privileged systemctl show -p MainPID --value frank-archon.service)"
|
||||
if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
|
||||
echo "FAIL: :7770 answered but frank-primary.service has no MainPID — a stray process holds the port" >&2
|
||||
echo "FAIL: :7770 answered but frank-archon.service has no MainPID — a stray process holds the port" >&2
|
||||
return 1
|
||||
fi
|
||||
echo "frank-primary: health answered by unit MainPID $main_pid"
|
||||
echo "frank-archon: health answered by unit MainPID $main_pid"
|
||||
}
|
||||
|
||||
wait_for_health() {
|
||||
@@ -1573,9 +1574,9 @@ deploy_service() {
|
||||
anubis) deploy_anubis ;;
|
||||
seerr) deploy_seerr ;;
|
||||
marreta) deploy_marreta ;;
|
||||
frank-large) deploy_llm_large ;;
|
||||
frank-secondary) deploy_llm_secondary ;;
|
||||
frank-primary) deploy_llm_primary ;;
|
||||
frank-dragoon) deploy_llm_dragoon ;;
|
||||
frank-zealot) deploy_llm_zealot ;;
|
||||
frank-archon) deploy_llm_archon ;;
|
||||
hermes|hermes-dashboard|hermes-webui) deploy_hermes ;;
|
||||
romm|romm-mariadb|romm-redis|mariadb|redis) deploy_romm_stack "$service" ;;
|
||||
jellyfin) deploy_jellyfin ;;
|
||||
|
||||
+3
-3
@@ -35,7 +35,7 @@ Options:
|
||||
Examples:
|
||||
./scripts/deploy.sh dankclaw
|
||||
./scripts/deploy.sh dankclaw --service searxng
|
||||
./scripts/deploy.sh dankserver --service frank-primary
|
||||
./scripts/deploy.sh dankserver --service frank-archon
|
||||
./scripts/deploy.sh dankclaw --dry-run
|
||||
EOF
|
||||
}
|
||||
@@ -141,12 +141,12 @@ valid_service() {
|
||||
dankclaw:alertmanager|\
|
||||
dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
|
||||
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
|
||||
dankclaw:frank-large|dankclaw:frank-secondary|\
|
||||
dankclaw:frank-dragoon|dankclaw:frank-zealot|\
|
||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
||||
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
|
||||
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
|
||||
dankserver:frank-primary|\
|
||||
dankserver:frank-archon|\
|
||||
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
|
||||
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
|
||||
frankbook-pro:readarr)
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"""derive-protected-experts.py — rewrite the recipe's protected-expert section
|
||||
from a calibration recording. RUNS ON DANKSERVER (venv: numpy).
|
||||
|
||||
The law (docs/MOE-W4.md "Protected experts", restated in the recipe):
|
||||
The protection rule (docs/MOE-W4.md "Protected experts", restated in the recipe):
|
||||
an expert whose share of its layer's down-input energy is at least 3% AND
|
||||
at least three times its share of the routing is kept bf16; the count must
|
||||
be EVEN on each rank's half of the layer (the four-bit GEMM takes the rest
|
||||
|
||||
+5
-5
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# download-frank-primary-swift15.sh — payload fetch/verify for the planned
|
||||
# frank-primary swap to the Swift 1.5 finetune: ukisai/Swift1.5-Qwen3.8-Flash-Next
|
||||
# download-frank-archon-swift15.sh — payload fetch/verify for the planned
|
||||
# frank-archon swap to the Swift 1.5 finetune: ukisai/Swift1.5-Qwen3.8-Flash-Next
|
||||
# (BF16 source, 149 files, 360,037,355,038 bytes = 335.3 GiB: 131 safetensors
|
||||
# shards + weight-map index + tokenizer + vision preprocessing configs + chat
|
||||
# template + licenses). Key set verified identical to the official
|
||||
@@ -8,13 +8,13 @@
|
||||
# apply verbatim; the checkpoint ships BF16 with NO quantization_config.
|
||||
# Repo is UNGATED — no token needed (never put one on the command line).
|
||||
#
|
||||
# bash scripts/download-frank-primary-swift15.sh --fetch resume/complete
|
||||
# bash scripts/download-frank-primary-swift15.sh verify only
|
||||
# bash scripts/download-frank-archon-swift15.sh --fetch resume/complete
|
||||
# bash scripts/download-frank-archon-swift15.sh verify only
|
||||
#
|
||||
# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
|
||||
# listing/resume downloader, curl+jq only): per-file size checks against
|
||||
# the Hub listing, `-C -` resume, pickle-format duplicates skipped.
|
||||
# Verify (default) is the AGENTS.md law in three parts: every listed file
|
||||
# Verify (default) enforces the download contract in three parts: every listed file
|
||||
# present with the listed size, the shard count matches the weight-map
|
||||
# index, and every shard's sha256 equals the Hub's LFS oid. Verify needs
|
||||
# only the public API.
|
||||
@@ -1,18 +1,18 @@
|
||||
#!/usr/bin/env bash
|
||||
# download-frank-primary.sh — payload fetch/verify for frank-primary:
|
||||
# download-frank-archon.sh — payload fetch/verify for frank-archon:
|
||||
# the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint (131 safetensors
|
||||
# shards + weight-map index + tokenizer + vision preprocessing configs,
|
||||
# ~172.8 GB). 2026-10-05: replaces the orcarouter uncensored payload of the
|
||||
# FreeToken era (that script lived here; its repo was gated and needed
|
||||
# HF_TOKEN — this one is UNGATED and needs no token at all).
|
||||
#
|
||||
# bash scripts/download-frank-primary.sh --fetch resume/complete
|
||||
# bash scripts/download-frank-primary.sh verify only
|
||||
# bash scripts/download-frank-archon.sh --fetch resume/complete
|
||||
# bash scripts/download-frank-archon.sh verify only
|
||||
#
|
||||
# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
|
||||
# listing/resume downloader, curl+jq only): per-file size checks against
|
||||
# the Hub listing, `-C -` resume, pickle-format duplicates skipped.
|
||||
# Verify (default) is the AGENTS.md law in three parts: every listed file
|
||||
# Verify (default) enforces the download contract in three parts: every listed file
|
||||
# present with the listed size, the shard count matches the weight-map
|
||||
# index, and every shard's sha256 equals the Hub's LFS oid. Verify needs
|
||||
# only the public API.
|
||||
+7
-5
@@ -1,12 +1,13 @@
|
||||
#!/usr/bin/env bash
|
||||
# download-frank-large-flashnext.sh — fetch frank-large's flash-server v2
|
||||
# download-frank-dragoon-flashnext.sh — fetch frank-dragoon's flash-server v2
|
||||
# payload: the Quat3rnion abliterated Swift1.5 checkpoint of peonist's v2
|
||||
# .hgn, the REQUIRED lookup table and the optional vision tower from
|
||||
# peonist-ai/halogen-qwen3.8-flash-next (both at the revisions the model
|
||||
# card pins), and the flat tokenizer the two repos share. All public, no
|
||||
# token — and per the fleet secret law, none would ever touch argv.
|
||||
# token — and a token never rides a command line: argv is readable by every
|
||||
# process on the host through /proc, so a key placed there is a key exposed.
|
||||
#
|
||||
# The fetch law, same as every sibling script: curl -f (an access error
|
||||
# The fetch discipline, same as every sibling script: curl -f (an access error
|
||||
# must not become a "successful" 153-byte artifact), exact byte count from
|
||||
# the Hub tree API, LFS sha256 verification, --verify-only to re-check an
|
||||
# existing payload without downloading.
|
||||
@@ -26,10 +27,11 @@
|
||||
# tokenizer/* @e053f488 — byte-identical to the 27B-era tokenizer
|
||||
# already on disk (tokenizer.json LFS
|
||||
# 0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3);
|
||||
# fetched fresh rather than symlinked, per the no-symlink law.
|
||||
# fetched fresh rather than symlinked, so the payload dir is
|
||||
# self-contained — every gate and every mount sees real files inside it.
|
||||
set -euo pipefail
|
||||
|
||||
DEST="${FRANK_LARGE_MODEL_DIR:-$HOME/models/halogen-flashnext-v2}"
|
||||
DEST="${FRANK_DRAGOON_MODEL_DIR:-$HOME/models/halogen-flashnext-v2}"
|
||||
ABLIT="https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated/resolve/main"
|
||||
PEONIST="https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next/resolve"
|
||||
NGRAM_REV="5cc17cea1a10b502b4a5db41d8d2b1a0f22dba26"
|
||||
@@ -1,16 +1,17 @@
|
||||
#!/usr/bin/env bash
|
||||
# download-frank-large-halogen.sh — fetch frank-large's halogen payload:
|
||||
# download-frank-dragoon-halogen.sh — fetch frank-dragoon's halogen payload:
|
||||
# the qwen3.8-27b .hgn checkpoint and the export tokenizer, from the
|
||||
# PUBLIC peonist-ai/halogen-qwen3.8-27b repo (no token — and per the fleet
|
||||
# secret law, none would ever touch argv).
|
||||
#
|
||||
# The fetch law, same as every sibling script: curl -f (an access error
|
||||
# PUBLIC peonist-ai/halogen-qwen3.8-27b repo (no token — and a token never
|
||||
# rides a command line: argv is readable by every process on the host
|
||||
# through /proc, so a key placed there is a key exposed).
|
||||
#
|
||||
# The fetch discipline, same as every sibling script: curl -f (an access error
|
||||
# must not become a "successful" 153-byte artifact), exact byte count from
|
||||
# the model card, LFS sha256 verification, --verify-only to re-check an
|
||||
# existing payload without downloading.
|
||||
set -euo pipefail
|
||||
|
||||
DEST="${FRANK_LARGE_MODEL_DIR:-$HOME/models/halogen-qwen3.8-27b}"
|
||||
DEST="${FRANK_DRAGOON_MODEL_DIR:-$HOME/models/halogen-qwen3.8-27b}"
|
||||
BASE="https://huggingface.co/peonist-ai/halogen-qwen3.8-27b/resolve/main"
|
||||
|
||||
FILES=(qwen3.8-27b-p1w4d-d2.hgn
|
||||
+3
-3
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF for
|
||||
# frank-large (Dankclaw iGPU). Public repo — no auth header, and never a token
|
||||
# frank-dragoon (Dankclaw iGPU). Public repo — no auth header, and never a token
|
||||
# on the command line (it is world-readable in /proc/<pid>/cmdline).
|
||||
#
|
||||
# The payload is the repo's Q6 AGENT rung:
|
||||
@@ -10,7 +10,7 @@
|
||||
# names for MTP serving ("Take Q6 AGENT (114) if you want protected
|
||||
# heads"). Like every ROCmFPX file it loads on nothing but the fork.
|
||||
#
|
||||
# The other two files frank-large needs are NOT re-downloaded:
|
||||
# The other two files frank-dragoon needs are NOT re-downloaded:
|
||||
# * mtp-Qwen3.8-27B-Q4_0.gguf — the uncensored repo redistributes the
|
||||
# ALIGNED repo's draft head byte for byte (identical Hub LFS oid
|
||||
# 051a1764…), so it is hardlinked out of the aligned payload dir when that
|
||||
@@ -25,7 +25,7 @@
|
||||
# byte count and the Hub LFS sha256. Idempotent: complete files are skipped.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/download-frank-large-kingjones-uncensored.sh [DEST_DIR] [--verify-only]
|
||||
# scripts/download-frank-dragoon-kingjones-uncensored.sh [DEST_DIR] [--verify-only]
|
||||
set -euo pipefail
|
||||
|
||||
REPO="kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF"
|
||||
+2
-2
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF for frank-large
|
||||
# Download kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF for frank-dragoon
|
||||
# (Dankclaw iGPU). Public repo — no auth header, and never a token on the
|
||||
# command line (it is world-readable in /proc/<pid>/cmdline).
|
||||
#
|
||||
@@ -24,7 +24,7 @@
|
||||
# come from the Hub LFS oids (api/models/<repo>/tree/main).
|
||||
#
|
||||
# Usage:
|
||||
# scripts/download-frank-large-kingjones.sh [DEST_DIR] [--verify-only]
|
||||
# scripts/download-frank-dragoon-kingjones.sh [DEST_DIR] [--verify-only]
|
||||
set -euo pipefail
|
||||
|
||||
REPO="kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF"
|
||||
@@ -1,9 +1,9 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download and assemble jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca (the frank-large
|
||||
# Download and assemble jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca (the frank-dragoon
|
||||
# model) on Dankclaw.
|
||||
#
|
||||
# bash scripts/download-frank-large-orca.sh
|
||||
# bash scripts/download-frank-large-orca.sh --verify-only
|
||||
# bash scripts/download-frank-dragoon-orca.sh
|
||||
# bash scripts/download-frank-dragoon-orca.sh --verify-only
|
||||
#
|
||||
# The package is 136 GB of Hub-sized transport parts (HF caps new uploads at
|
||||
# 50 GB, so the 79 GB main GGUF, the 52 GB PLE payload and the 4 GB MTP head
|
||||
@@ -105,4 +105,4 @@ fi
|
||||
echo "verifying assembled files against SHA256SUMS"
|
||||
(cd "$MODEL_DIR" && sha256sum -c SHA256SUMS)
|
||||
(cd "$MODEL_DIR/vision" && sha256sum -c SHA256SUMS)
|
||||
echo "frank-large payload complete: $MODEL_DIR"
|
||||
echo "frank-dragoon payload complete: $MODEL_DIR"
|
||||
@@ -1,9 +1,9 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the Q4_K_XL frank-large payload on Dankclaw:
|
||||
# Download the Q4_K_XL frank-dragoon payload on Dankclaw:
|
||||
# unsloth/Qwen3.8-27B-GGUF -> Qwen3.8-27B-UD-Q4_K_XL.gguf
|
||||
#
|
||||
# bash scripts/download-frank-large-q4.sh
|
||||
# bash scripts/download-frank-large-q4.sh --verify-only
|
||||
# bash scripts/download-frank-dragoon-q4.sh
|
||||
# bash scripts/download-frank-dragoon-q4.sh --verify-only
|
||||
#
|
||||
# Integrity: the file is checked against its declared byte count and sha256
|
||||
# (the LFS oid from the Hub tree API, 2026-10-01). NO AUTH HEADER IS USED:
|
||||
@@ -21,12 +21,12 @@
|
||||
# x 2 (K,V) x 2 B), so 4 sessions x 163840 ctx = 640,000 tokens = 39.06 GiB.
|
||||
#
|
||||
# The speculative drafter is UNCHANGED and still fetched by its own script
|
||||
# (scripts/download-frank-large.sh, z-lab's Hub repo). That name must not
|
||||
# appear here at all — the frank-large seam greps THIS file for it to prove
|
||||
# (scripts/download-frank-dragoon.sh, z-lab's Hub repo). That name must not
|
||||
# appear here at all — the frank-dragoon seam greps THIS file for it to prove
|
||||
# the drafter was not folded into the payload fetch.
|
||||
#
|
||||
# NOTE ON MTP: this payload has no baked-in MTP head (the Hub repo ships one
|
||||
# separately as MTP/mtp-Qwen3.8-27B-Q4_0.gguf). frank-large does not need it —
|
||||
# separately as MTP/mtp-Qwen3.8-27B-Q4_0.gguf). frank-dragoon does not need it —
|
||||
# spec decode here is the block-diffusion drafter, not MTP.
|
||||
set -euo pipefail
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the Swift-1.5 frank-large candidate on Dankclaw:
|
||||
# Download the Swift-1.5 frank-dragoon candidate on Dankclaw:
|
||||
# ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF
|
||||
# Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
|
||||
#
|
||||
# bash scripts/download-frank-large-swift.sh
|
||||
# bash scripts/download-frank-large-swift.sh --verify-only
|
||||
# bash scripts/download-frank-dragoon-swift.sh
|
||||
# bash scripts/download-frank-dragoon-swift.sh --verify-only
|
||||
#
|
||||
# Integrity: the file is checked against its declared byte count and sha256
|
||||
# (the LFS oid from the Hub, 2026-09-28). NO AUTH HEADER IS USED: the repo is
|
||||
@@ -15,9 +15,9 @@
|
||||
# Layout after a successful run (MODEL_DIR):
|
||||
# Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf 31457990272
|
||||
#
|
||||
# This is the CANDIDATE payload, not the deployed one. frank-large still
|
||||
# serves scripts/download-frank-large.sh's output; do not repoint
|
||||
# frank-large.env at MODEL_DIR until the gufo load test passes.
|
||||
# This is the CANDIDATE payload, not the deployed one. frank-dragoon still
|
||||
# serves scripts/download-frank-dragoon.sh's output; do not repoint
|
||||
# frank-dragoon.env at MODEL_DIR until the gufo load test passes.
|
||||
set -euo pipefail
|
||||
|
||||
MODEL_DIR="/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL"
|
||||
@@ -1,9 +1,9 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (the
|
||||
# frank-large model) on Dankclaw.
|
||||
# frank-dragoon model) on Dankclaw.
|
||||
#
|
||||
# bash scripts/download-frank-large-unsloth.sh
|
||||
# bash scripts/download-frank-large-unsloth.sh --verify-only
|
||||
# bash scripts/download-frank-dragoon-unsloth.sh
|
||||
# bash scripts/download-frank-dragoon-unsloth.sh --verify-only
|
||||
#
|
||||
# The UD-IQ4_XS GGUF ships as three llama.cpp split parts (HF caps uploads at
|
||||
# 50 GB). The parts ARE the canonical payload: the serve script points -m at
|
||||
@@ -112,4 +112,4 @@ if [[ "$mode" == "download" ]]; then
|
||||
[[ -e "$MODEL_DIR/vision/mmproj-F16.gguf" ]] || mv "$MODEL_DIR/mmproj-F16.gguf" "$MODEL_DIR/vision/"
|
||||
fi
|
||||
|
||||
echo "frank-large payload complete: $MODEL_DIR"
|
||||
echo "frank-dragoon payload complete: $MODEL_DIR"
|
||||
@@ -1,10 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the frank-large payload on Dankclaw:
|
||||
# Download the frank-dragoon payload on Dankclaw:
|
||||
# - unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (the 27B dense main GGUF)
|
||||
# - z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M (the DFlash2 drafter)
|
||||
#
|
||||
# bash scripts/download-frank-large.sh
|
||||
# bash scripts/download-frank-large.sh --verify-only
|
||||
# bash scripts/download-frank-dragoon.sh
|
||||
# bash scripts/download-frank-dragoon.sh --verify-only
|
||||
#
|
||||
# Integrity: every file is checked against its declared byte count and sha256
|
||||
# (the values below are the LFS oids from the Hub, 2026-09-21). An HF token
|
||||
@@ -86,4 +86,4 @@ if [[ "$mode" == "download" ]]; then
|
||||
done
|
||||
fi
|
||||
|
||||
echo "frank-large payload complete: $MODEL_DIR"
|
||||
echo "frank-dragoon payload complete: $MODEL_DIR"
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the frank-secondary payload for llama.cpp on the RTX 3090:
|
||||
# Download the frank-zealot payload for llama.cpp on the RTX 3090:
|
||||
# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at **Q6_K** plus the
|
||||
# PUBLISHER's OWN vision projector (the BF16 mmproj — the model is natively
|
||||
# multimodal, so this file is the endpoint's eyes). Public repo — no auth
|
||||
@@ -21,7 +21,7 @@
|
||||
# replaced), Swift15-UncIQ3XXS/, MiMo-Q6K/, MiMo-Q5KM/, Ornith-Q8/.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only]
|
||||
# scripts/download-frank-zealot.sh [DEST_DIR] [--verify-only]
|
||||
set -euo pipefail
|
||||
|
||||
DEST=""
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Measure per-block / per-expert expert-routing activity on live frank-primary.
|
||||
"""Measure per-block / per-expert expert-routing activity on live frank-archon.
|
||||
|
||||
Method: AMD IBS op sampling (data addresses, `perf record -e ibs_op// --data`)
|
||||
on the llama-server process while concurrent generations of mixed content run.
|
||||
@@ -11,8 +11,8 @@ Tensor extents are DERIVED from consecutive GGUF header offsets — no quant
|
||||
type-size table (those are easy to get subtly wrong; a wrong per-element size
|
||||
silently under-covers each tensor and dumps tail samples into a junk bucket).
|
||||
|
||||
Fleet law this instrument established (2026-09-26, 75 s window, 4 mixed
|
||||
streams, 2.7M expert-tensor samples): per-BLOCK read volume is FLAT
|
||||
What this instrument shows (75 s window, 4 mixed streams, 2.7M
|
||||
expert-tensor samples): per-BLOCK read volume is FLAT
|
||||
(0.93x-1.08x of mean across all 40 CPU-resident blocks) because MoE top-k
|
||||
routing fires every block every token — there are no "most active layers" at
|
||||
`-ot` granularity. The real skew is per-EXPERT (top 10% of (block,expert)
|
||||
@@ -0,0 +1,286 @@
|
||||
# frank-archon.env — radiance (StillDeadcode, Apache-2.0) serving
|
||||
# ukisai/Swift1.5-Qwen3.8-Flash-Next converted to GPTQ w4 (MODEL_FILE below,
|
||||
# built by scripts/convert-swift15-w4dn8.sh) on BOTH R9700s (TP=2),
|
||||
# dankserver :7770.
|
||||
#
|
||||
# SHAPE (2026-10-13): THREE decode slots x 200,000 tokens, fp8 (quantized) KV,
|
||||
# with the per-card budget spent on resident experts: 25,600 MiB of slab +
|
||||
# 5,120 MiB of pool = 30,720 MiB, inside the 30,969 MiB ceiling the engine's own
|
||||
# budget table allows (card 32,624 MiB, settle floor 209 MiB). The pool is sized
|
||||
# to the contract and nothing more (3 x 200,000 needs 5,063 MiB at fp8; 5,120
|
||||
# predicts 608,027 tokens, 1.3 % slack), because every MiB taken from the slab
|
||||
# is prefill and TTFT (2026-10-11 A/B: slab -14.8 % cost prefill -29 %, TTFT
|
||||
# +25 % at the full window) while KV beyond the contract buys nothing.
|
||||
# Consequence, deliberate: a FOURTH full-height conversation does not fit this
|
||||
# pair, and the engine refuses it — it never truncates. See the KV block for the
|
||||
# arithmetic and the measured per-dtype cost. The fp8-native payload this file was
|
||||
# written for is the rollback target and keeps its own trio under the
|
||||
# fp8-native runbook.
|
||||
#
|
||||
# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
|
||||
# pinned host RAM; git tag frank-archon-freetoken-pr587-20261005 is the
|
||||
# pre-cutover state — its unit/env/serve trio and Dockerfile are all in the
|
||||
# tag and rebuildable). Before that: tcclaviger vLLM fork (tag
|
||||
# frank-archon-vllm-shapeG-20261003).
|
||||
#
|
||||
# WHY RADIANCE: the engine is C++/HIP with libr4d kernels built for
|
||||
# gfx1201, drives `podman run` directly from the digest-pinned
|
||||
# stilldeadcode/radiance image (287 MiB; host needs only the amdgpu driver),
|
||||
# and its qwen4exp plugin reads the modelopt FP8 encoding NATIVELY
|
||||
# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram
|
||||
# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring
|
||||
# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram."
|
||||
# The 4-bit-codebook publisher shape (qwen3.8-next-flash-fp8-iq4r-moe.rad,
|
||||
# --host-pool-mib 12288) remains the documented fallback if the native
|
||||
# container proves unbootable.
|
||||
#
|
||||
# PAYLOAD: /home/dank/models/rad/qwen4exp-fp8-native.rad — a rad-convert
|
||||
# NO-RECIPE pure copy of the official 131-shard checkpoint (every weight
|
||||
# keeps its block-FP8 encoding; ~161 GiB). The convert runs against a
|
||||
# staging dir whose config.json drops `quantization_config`: the descriptor
|
||||
# string is only the plugin SELECTION key (spec §2.4 — exact match, no
|
||||
# fallback), and no shipped plugin claims (qwen4exp, "fp8"); the shipped
|
||||
# qwen4exp plugin claims (qwen4exp, "") which is what recipe-built
|
||||
# containers carry. Per-weight encodings are data, not the descriptor —
|
||||
# removing the key changes nothing about what the weights hold. Built by
|
||||
# scripts/build-frank-archon-rad.sh (staging + convert + rad-info).
|
||||
#
|
||||
# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml):
|
||||
# - The HTTP listener binds AFTER the full load (main.cpp: engine.start()
|
||||
# then engine.serve()) — during load the port refuses connections, so a
|
||||
# bare curl /health gate is honest. /health == /ping: 200 JSON while
|
||||
# serving, 503 only while stopping (core/server/admin.cpp:385).
|
||||
# - Reasoning: --reasoning-format defaults to auto, emitting thinking in
|
||||
# `reasoning_content` (config.cpp). Tool calls: the qwen3.8 dialect, from
|
||||
# the chat format. Both need no parser flags.
|
||||
# - Sampling: the OpenAI endpoint accepts temperature, top_p, top_k, min_p,
|
||||
# typical_p, presence_penalty, frequency_penalty, repetition_penalty,
|
||||
# repeat_last_n, chat_template_kwargs, reasoning_effort,
|
||||
# enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) —
|
||||
# models.yml's extraBody set is fully inside it. Unknown params are
|
||||
# refused (400), which is why nothing unverified rides the wire.
|
||||
# - MTP: the official checkpoint carries the mtp.* head (3,101 tensors,
|
||||
# mtp_num_hidden_layers=1) and the plugin probes draft depth 3 — the
|
||||
# publisher's throughput peak for this model. Stated explicitly with
|
||||
# --num-speculative-tokens 3.
|
||||
# - The fused TP all-reduce is declared over max-num-seqs x
|
||||
# (1 + --num-speculative-tokens) rows; 2 x 3 = 6 rows, under the 25-row
|
||||
# exactness bound of the wht6 wire.
|
||||
# Engine pin: Codeberg tag v1.0.13 (commit 53282fa), Docker Hub tag 1.0.13,
|
||||
# index digest below. REJECTED UPGRADE: 1.2.0 (sha256:665775ed1d90de6dbb
|
||||
# 25558e07fc60d20328f1346f96f3c6e00845eb763f01fa, tag 1.2.0, commit
|
||||
# e95d397) was benched 2026-10-06 on this exact shape (fp8-native rad,
|
||||
# vram pair, spec 2, seqs 3, 43.6k greedy probe): decode 29.21 t/s and
|
||||
# draft acceptance 0.7123 vs the 1.0.13 control's 30.08 and 0.7549.
|
||||
# Acceptance on the greedy harness is deterministic (the control
|
||||
# reproduced the recorded 0.755 exactly), so the drop is a real numerics
|
||||
# change — the 1.2.0 MoE weight-rotation change (92b8aaf) is the named
|
||||
# suspect; decode loss tracks the acceptance ratio (0.943 vs 0.945).
|
||||
# The vram-pair translation itself is exact (1.2.0 boot ledger took
|
||||
# vram_weights 25.46 GiB / vram_kv 3.46 GiB per card byte-identical to
|
||||
# the 1.0.13 ledger at EVC 0.82). Re-test 1.2.0+ when the rotation change
|
||||
# is untangled; 1.2.0 also REMOVED --expert-vs-cache-ratio in favour of
|
||||
# the --vram-weights-mib / --vram-kv-mib pair (core/config.cpp) — this
|
||||
# file already uses the pair, which BOTH pins accept (rollback symmetry).
|
||||
FRANK_ARCHON_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
|
||||
# The .rad container FILE (mounted under /models) and the state dir for the
|
||||
# prefix-cache disk tier (mounted rw at /data). The model dir holds ONLY
|
||||
# .rad files; the raw HF download and its staging dir live beside it and
|
||||
# are never mounted.
|
||||
FRANK_ARCHON_MODEL_DIR=/home/dank/models/rad
|
||||
FRANK_ARCHON_MODEL_FILE=qwen4exp-swift15-w4dn8-i8-hc8m.rad
|
||||
FRANK_ARCHON_STATE_DIR=/home/dank/radiance-state
|
||||
# The state dir's tune/ subdirectory holds rad-tune's kernel-selection
|
||||
# cache (the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune
|
||||
# at boot; $RADIANCE_HOME defaults to /opt/radiance, in the image, so the
|
||||
# cache must ride a host mount — serve script mounts it, see the runbook's
|
||||
# "Tuning levers").
|
||||
# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
|
||||
# name across systemctl and podman ps. The runtime changes; the name does
|
||||
# not. The legacy list retires every predecessor identity that could hold
|
||||
# the GPUs or the port.
|
||||
FRANK_ARCHON_CONTAINER=frank-archon
|
||||
FRANK_ARCHON_LEGACY_CONTAINERS="frank-archon-qwen frank-archon-r9v qwen-fp8 glm-probe"
|
||||
# Served model id — Protoss name (wire rename 2026-10-09). The container,
|
||||
# unit, scripts and this file keep the frank-archon name above; only the
|
||||
# alias on the wire moved. Radiance validates the requested model name, so
|
||||
# every client (harness, bench scripts, probes) MUST send this value.
|
||||
FRANK_ARCHON_SERVED_NAME=frank-archon
|
||||
FRANK_ARCHON_PORT=7770
|
||||
# Frank's chat-template call: froggeric's fixed Qwen3.8 template (the
|
||||
# repo's root chat_template.jinja, vendored here; the checkpoint's own
|
||||
# template is baked into the .rad by rad-convert and this overrides it).
|
||||
# Radiance derives the reasoning/tool-call reply format from it.
|
||||
FRANK_ARCHON_TEMPLATE=/home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja
|
||||
|
||||
# --- engine shape (the boot ledger arbitrates every number below) ---------
|
||||
# Tensor parallel: BOTH R9700s. The board swap (2026-09-20, ProArt B850
|
||||
# x8/x8 on root ports) made collectives legal on this host; radiance's TP
|
||||
# wire is libr4d's own, not RCCL.
|
||||
FRANK_ARCHON_TP=2
|
||||
# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice:
|
||||
# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
|
||||
# engine default.
|
||||
FRANK_ARCHON_TP_WIRE=wht6
|
||||
# Decode slots: THREE, so the served window is a CAPACITY CONTRACT — three
|
||||
# concurrent conversations of --max-model-len fit the KV pool (see the KV
|
||||
# block below). A runtime that compiles its kernels specialized to the exact
|
||||
# batch shape can refuse a slot count it has never compiled for; a
|
||||
# shape-agnostic engine accepts any count that fits memory. Radiance is the
|
||||
# latter: seqs 2 boots, and measured decode at seqs 2 and 3 is statistically
|
||||
# tied (30.9-33.1 t/s, inside the +-3% noise floor) — slot count buys
|
||||
# concurrency, not speed, so a slot costs KV, never decode.
|
||||
# Cost of each extra slot: 110 MiB of per-slot recurrent state inside the KV
|
||||
# pool, and the fused TP all-reduce is declared over seqs x (1 + spec) rows —
|
||||
# 3 x 3 = 9 here, under the 25-row bound at which the wht6 wire stays EXACT
|
||||
# (single-sequence decode and any step of <= 25 rows is bit-exact; only busier
|
||||
# steps are lossy).
|
||||
# The pool is sized to this contract and nothing more (see the KV block):
|
||||
# 3 x 200,000 at fp8 = 5,063 MiB, pool 5,120, and the freed 1,792 MiB went into
|
||||
# the resident expert slab. The trade is explicit and it is Frank's call:
|
||||
# a FOURTH full-height conversation no longer fits (5,120 MiB of pool backs
|
||||
# ~608k tokens; 4 x 200,000 needs 6,601), so a fourth 200k request is refused
|
||||
# as too large to fit — it is NOT squeezed, paged, or truncated. Third-party
|
||||
# traffic that wants four full-height sessions is a --vram-kv-mib change, not
|
||||
# a code change.
|
||||
FRANK_ARCHON_MAX_SEQS=3
|
||||
# Served window. Native is 262,144; 200,000 is the publisher's production
|
||||
# value (compose) and what models.yml's contextWindow must quote — a longer
|
||||
# prompt is refused by the engine, never truncated. The engine treats
|
||||
# `kv group tokens >= max-model-len` as a HARD limit ("ONE request longer
|
||||
# than N tokens is accepted by the API and then refused as too large to ever
|
||||
# fit — this is a hard limit, not a backing ratio"), so the pool is sized
|
||||
# for seqs x maxlen, not for one sequence.
|
||||
FRANK_ARCHON_MAX_MODEL_LEN=200000
|
||||
FRANK_ARCHON_PLACEMENT=expert_tiered
|
||||
# Pinned host pool for the expert rows that do not fit VRAM. vram_weights and
|
||||
# this pool serve the per-rank resident set TOGETHER: at the fp8-KV pair
|
||||
# (26071/3543) the ledger showed the VRAM slab filled to 259 KiB of slack,
|
||||
# i.e. the pair was exactly sized. Cutting vram-weights by 3.8 GiB to pay for
|
||||
# the bf16 KV pool SPILLS ~3.8 GiB/rank here, so the pool moves with it —
|
||||
# under-sizing it is a boot failure (36864 refused the boot in the fp8 era).
|
||||
# The boot ledger's coverage line remains the referee, not this arithmetic.
|
||||
FRANK_ARCHON_HOST_POOL_MIB=44928
|
||||
# VRAM left unclaimed per card (publisher default 96 MiB).
|
||||
FRANK_ARCHON_GPU_HEADROOM_MIB=96
|
||||
# VRAM split of claimed memory per card, MiB (0 = derive from the card).
|
||||
# THE CEILING, read off this engine's own VRAM-budget table (boot 2026-10-13,
|
||||
# pair 24000/5120): card 32,624 MiB; budgets claimed 29,120; `3.42 GiB is
|
||||
# neither budgeted nor accounted — the driver's context, the activation arena
|
||||
# and --gpu-headroom-mib come out of it; anything past those is throughput on
|
||||
# the floor`; and `card 0 settled at 2.01 GiB free against 209.19 MiB (the
|
||||
# 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)`.
|
||||
# So the reserve above the ENGINE's floor is 2,058 - 209 = 1,849 MiB and the
|
||||
# ceiling is 29,120 + 1,849 = 30,969 MiB. The 2026-10-11 bound of 29,120
|
||||
# ("31,856 - 1,710 outside both budgets - keep 1 GiB free") was MY reserve, not
|
||||
# the engine's; the pair below spends the difference and lands 458 MiB above
|
||||
# the engine's floor. `weights + kv <= 30,969` is the gate.
|
||||
# THE PRICE, and it is paid twice: on this engine idle KV VRAM is ALSO the
|
||||
# expert cache's spillover (`mover: 2275 slots can be lent out of the KV
|
||||
# cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <= 256 tokens`),
|
||||
# so a bigger KV pool steals weight residency from the slab AND from the
|
||||
# lendable cache. Priors on this card/model: stealing 2.14 GiB of weights for
|
||||
# a bigger pool (MAX_MODEL_LEN 262144) cost 2.5% decode; the EVC ladder's
|
||||
# other direction (0.82 -> 0.84, +2.5 GiB weights) cost 8.9%. The 2026-10-11
|
||||
# exact-KV A/B measured the slab's own price (22208 vs the fp8-era 26071):
|
||||
# prefill -29 %, TTFT at the full window +25 %, decode inside the noise band on
|
||||
# 2 of 3 passes, draft acceptance UP 0.661 -> 0.740.
|
||||
# 2026-10-13 (late): THREE slots, so the pool is cut to the contract (5,120)
|
||||
# and every spare MiB goes to the slab: 22208 -> 24000 (the third slot's
|
||||
# surplus KV), then 24000 -> 25600 (the card's settling slack). The direction
|
||||
# the A/B data points: prefill is slab-limited, KV beyond the contract is not.
|
||||
# The margin is thin by design (608,027 predicted tokens against the 600,000
|
||||
# contract), so the boot ledger's `backs N sequences all at 200000 tokens at
|
||||
# once` line is the referee for THIS pair and the concurrent probe is its
|
||||
# regression test.
|
||||
# VERDICT on the 25,600 arm (measured 2026-10-13, scripts/probe-frank-archon-kv.py):
|
||||
# the threshold claim is FALSIFIED — the slab is a SLOPE at this end. A-cold
|
||||
# 43.5k single-stream: 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,584 t/s;
|
||||
# the 3-way 197k arm's aggregate prefill 3,051 -> 3,589 t/s (wall
|
||||
# 193.7 -> 164.5 s). So +1,600 MiB of slab = +8.9 % single-stream and
|
||||
# +17.6 % concurrent, both outside the +-3 % noise floor. The 22,208/24,000
|
||||
# pair landed inside noise of EACH OTHER, which is what made it read as a
|
||||
# plateau; two points are not a curve.
|
||||
# The cliff that remains is between 25,600 (3,584) and 26,071 (4,379, the
|
||||
# 2026-10-11 fp8 control at seqs 3 / pool 3,543) — +22 % for +471 MiB. At THREE
|
||||
# slots that is only reachable by breaking the ceiling (26,112 + 5,120 = 31,232
|
||||
# vs 30,969, ~214 MiB above the engine's floor); at TWO slots it is free:
|
||||
# pool 3,072 + slab 27,897 = exactly the ceiling, slab above the cliff.
|
||||
FRANK_ARCHON_VRAM_WEIGHTS_MIB=25600
|
||||
FRANK_ARCHON_VRAM_KV_MIB=5120
|
||||
# KV cache QUANTIZED (fp8). This engine's flag takes ONLY bf16|fp8 — there is
|
||||
# ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's
|
||||
# --cache-type-k/--cache-type-v pair, which is what frank-zealot runs) is
|
||||
# UNSERVABLE here: the rung below bf16 is fp8 on both halves.
|
||||
# Measured per-rank cost, read out of the boot ledger's group table (fp8 boot
|
||||
# 2026-10-08, bf16 boot 2026-10-11):
|
||||
# kv_attn 13 layers, 4-token blocks, page 26.00 KiB (fp8) -> 52.00 KiB
|
||||
# kv_qsa_bkey 13 layers, 4-token blocks, page 3.25 KiB at BOTH dtypes —
|
||||
# the QSA block key is an index, not attention state, and does
|
||||
# NOT follow --kv-cache-dtype
|
||||
# => 7.3125 KiB/token fp8, 13.8125 KiB/token bf16: bf16 costs 1.89x fp8,
|
||||
# not 2x, precisely because the bkey plane is dtype-fixed.
|
||||
# The dtype-INDEPENDENT terms also live in the pool: kv_gdn_state 54 MiB x 2
|
||||
# copies per sequence (+ conv/tail/ple ~= 110 MiB a sequence) and the prefix
|
||||
# checkpoint region 8 x 55.99 = 448 MiB. So:
|
||||
# pool_mib >= tokens x 7.3125 KiB + 110 x seqs + 448
|
||||
# 3 x 200,000 at fp8 needs 4,285 + 330 + 448 = 5,063 MiB; the pool is 5,120,
|
||||
# and it predicts (5120 - 330 - 448) x 1024 / 7.3125 = 608,027 tokens: the
|
||||
# three-slot contract with 1.3 % slack, every MiB above 5,063 having been
|
||||
# handed to the expert slab instead. The ledger line is the referee ("backs N
|
||||
# sequences all at 200000 tokens at once") — a pair this tight is read off the
|
||||
# boot, never assumed.
|
||||
# At bf16 the same three slots cost 8,093 + 778 = 8,871 MiB, which would cut
|
||||
# the slab to 20,249 MiB (-9 %); at FOUR slots bf16 costs 10,791 + 888 =
|
||||
# 11,679 MiB and the slab falls to 17,441 (-33 %). fp8 keeps 22,208 at both
|
||||
# slot counts. The 2026-10-11 bf16 shape (6,912 / seqs 2) backed 462,752
|
||||
# tokens = 2 x 200,000; the historical fp8 pool of 3,543 gave 386,904 (1.93
|
||||
# sessions), and the four-slot fp8 boot of 2026-10-13 19:21 backed 843,276.
|
||||
# Secondary saving: the prefix-cache entry shrinks with the attention plane
|
||||
# (one entry 110.50 KiB at bf16 -> 58.50 KiB at fp8, measured on the two boot
|
||||
# ledgers' `kv tier transfer` lines), so the host tier below parks twice as
|
||||
# many conversations as it did at bf16 — no re-size needed to park three slots.
|
||||
FRANK_ARCHON_KV_DTYPE=fp8
|
||||
# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
|
||||
# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
|
||||
# flag states it so the ledger records the intent).
|
||||
FRANK_ARCHON_SPEC_TOKENS=2
|
||||
FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
|
||||
# Prefix cache tiers (publisher pattern). The host tier is sized in ENTRIES,
|
||||
# and one entry tracks the cache dtype (58.50 KiB at fp8, 117 KiB at bf16, per
|
||||
# the ledger's `kv tier transfer` line): at fp8, 8192 MiB parks ~5.8 sessions
|
||||
# of 200k (at bf16 the same tier parked ~2.9), so four live slots plus a full
|
||||
# parked set fits with the tier as it stands — at RAM the host has (104 GiB
|
||||
# available with the service loaded).
|
||||
FRANK_ARCHON_PREFIX_HOST_MIB=8192
|
||||
FRANK_ARCHON_PREFIX_DISK_MIB=65536
|
||||
# The PLE n-gram table (47.7 GiB) serves from pinned host RAM the cards
|
||||
# read directly, so it cannot be evicted or page-faulted mid-token. `ram`
|
||||
# placement REFUSES the start if the memlock limit is too small, and the
|
||||
# unit's LimitMEMLOCK=infinity is the matching half of that pairing.
|
||||
FRANK_ARCHON_NGRAM_PLACEMENT=ram
|
||||
# TTM preflight: the pinned ledger (2 x 44 GiB pools + 47.7 GiB PLE + 8 GiB
|
||||
# prefix host = ~147 GiB) rides TTM's shared pages, capped by ttm's
|
||||
# pages_limit (the current kernel exposes NO pages_max; the cap function
|
||||
# takes the min over the knobs that exist). Below the plan the load dies
|
||||
# mid-pin with an allocator error instead of this named refusal.
|
||||
# THE CAP IS A RUNTIME KNOB — the kernel default is half of RAM, so every
|
||||
# raise dies with a reboot. Measured the hard way: the 2026-10-05 manual raise
|
||||
# to 52,428,800 pages (200 GiB) was gone after the 2026-10-12 23:17 POWER
|
||||
# FAILURE, the cap fell back to 125,988 MiB, and the preflight refused the
|
||||
# start 6,919 times in ~20 h while Restart=always re-ran it. The gate in
|
||||
# scripts/ttm-preflight.sh now RAISES the knob itself before refusing (the
|
||||
# root unit is privileged; the knob is a 0644 root module parameter), so the
|
||||
# recovery is a boot, not a visit to the machine.
|
||||
FRANK_ARCHON_TTM_MIN_MIB=150000
|
||||
FRANK_ARCHON_TTM_TARGET_MIB=204800
|
||||
# Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
|
||||
# multi-flag values as one string; the serve script expands it unquoted on
|
||||
# purpose.
|
||||
FRANK_ARCHON_EXTRA_ARGS=
|
||||
# The .rad's minimum size in MiB (deployer gate; a truncated convert is a
|
||||
# guaranteed long-timeout boot instead of a fast failure). The converted
|
||||
# container measured 177,109 MiB (1,857,127,997,440 B = 173.0 GiB; the
|
||||
# plan-only estimate 172.76 GiB covers the data planes but not the metadata
|
||||
# tail) - floor sits just under the payload.
|
||||
FRANK_ARCHON_RAD_MIN_MIB=110000
|
||||
@@ -1,8 +1,8 @@
|
||||
# frank-primary.service — radiance (StillDeadcode, Apache-2.0) endpoint
|
||||
# frank-archon.service — radiance (StillDeadcode, Apache-2.0) endpoint
|
||||
# (OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 at NATIVE block-FP8 precision, the
|
||||
# rad-convert container qwen4exp-fp8-native.rad) on BOTH R9700s (TP=2,
|
||||
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (git tag
|
||||
# frank-primary-freetoken-pr587-20261005 is the pre-cutover state; the
|
||||
# frank-archon-freetoken-pr587-20261005 is the pre-cutover state; the
|
||||
# retired unit/env/serve trio and Dockerfile are in that tag).
|
||||
#
|
||||
# Same port, same served alias, same container identity as the unit name;
|
||||
@@ -19,7 +19,7 @@
|
||||
# unit grants LimitMEMLOCK=infinity to the dank process via PID1; the
|
||||
# engine still runs out of dank's rootless podman storage.
|
||||
[Unit]
|
||||
Description=frank-primary — radiance Qwen3.8-Flash-Next-FP8 (native) on 2x R9700 TP=2
|
||||
Description=frank-archon — radiance Qwen3.8-Flash-Next-FP8 (native) on 2x R9700 TP=2
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
# Never let the start-rate limiter give up after a crash loop (a failed LLM
|
||||
@@ -33,14 +33,21 @@ Group=dank
|
||||
# RLIMIT_MEMLOCK smaller than the pinned set fails the load mid-start.
|
||||
# PID1 grants infinity here — parity with the fleet's other LLM units.
|
||||
LimitMEMLOCK=infinity
|
||||
Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
|
||||
Environment="FRANK_ARCHON_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-archon.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-archon.sh
|
||||
# The `+` prefix runs this as root even though the service is User=dank — the
|
||||
# ttm pages cap is a root-owned knob and the pinned-RAM ledger (~147 GiB) is
|
||||
# unusable below it. After a reboot the knob is back at its default (half of
|
||||
# RAM), so the raise must be part of the start, not a note in a runbook: the
|
||||
# 2026-10-12 power failure reset it and the endpoint refused to boot 6,919
|
||||
# times. The serve script re-verifies the cap after this hook.
|
||||
ExecStartPre=+/home/dank/serverAppConfigs/scripts/ttm-preflight.sh --raise
|
||||
# systemd stop: wait alone cannot stop the container child; stop it
|
||||
# explicitly. No graceful drain contract is claimed (the OpenAI front end
|
||||
# holds no queue to drain — in-flight decodes are lost, which a stop
|
||||
# during a bench window accepts; radiance takes SIGTERM on the engine
|
||||
# process and podman reaps the container).
|
||||
ExecStop=/usr/bin/podman stop -t 120 frank-primary
|
||||
ExecStop=/usr/bin/podman stop -t 120 frank-archon
|
||||
# Boot cost: .rad load (weights + PLE upload) + expert host-pool pin. The
|
||||
# publisher's compose healthcheck allows a 15-minute start_period; the
|
||||
# vLLM-era budget (1800 s) carried forward.
|
||||
@@ -1,11 +1,11 @@
|
||||
# frank-large.env — tunables for the frank-large endpoint on Dankclaw's
|
||||
# frank-dragoon.env — tunables for the frank-dragoon endpoint on Dankclaw's
|
||||
# Strix Halo iGPU (gfx1151), host port 7770.
|
||||
#
|
||||
# RUNTIME (2026-10-06, Frank): halogen-flash-server 0.16.2 — the
|
||||
# closed-source C++/HIP engine (image ghcr.io/peonist-ai/halogen-flash-server,
|
||||
# digest-pinned below) serving the Quat3rnion ABLITERATED Swift1.5 finetune
|
||||
# of peonist's Qwen3.8-Flash-Next v2 checkpoint, as a podman container under
|
||||
# a root unit, driven by scripts/serve-frank-large.sh. This CUTS OVER the
|
||||
# a root unit, driven by scripts/serve-frank-dragoon.sh. This CUTS OVER the
|
||||
# 2026-10-05 halogen 27B-dense shape (censored base, DFlash2, 31.7 t/s):
|
||||
# rollback = revert the swap commit's trio; the 27B payload stays on disk at
|
||||
# /home/dank/models/halogen-qwen3.8-27b/.
|
||||
@@ -36,23 +36,23 @@
|
||||
# target). Worst-case request: prompt + budget <= 217088 − 4096 + 3 %
|
||||
# est-error ≈ 219k <= 262144 — the 45k gap between harness window and
|
||||
# engine ceiling absorbs the estimate error with ~7x margin (robust to a
|
||||
# 10 % under-count). Compaction rides on frank-secondary: input ≈
|
||||
# 10 % under-count). Compaction rides on frank-zealot: input ≈
|
||||
# threshold − keepRecent(20k) ≈ 170k measured, inside its 200,704 slot.
|
||||
# HALOGEN_KV_POOL_POSITIONS=524288 = KV_SLOTS(2) x CTX — the explicit
|
||||
# form of the engine's default, so the seam test can pin the product.
|
||||
# KV costs ~28 KiB/position (FLAGS.md) => 524,288 x 28 KiB = 14.0 GiB.
|
||||
# HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
|
||||
# drafter is NOT slot-locked on this engine (batching and speculation
|
||||
# coexist — DFlash-era slots>1 law does NOT carry over; the card's own
|
||||
# quickstart ships 2 slots WITH the head), so the second slot is
|
||||
# concurrent capacity, not a speculation tax.
|
||||
# drafter is NOT slot-locked on this engine: some engines silently
|
||||
# disable drafting the moment a second slot is enabled, this one does not
|
||||
# (the card's own quickstart ships 2 slots WITH the head), so the second
|
||||
# slot is concurrent capacity, not a speculation tax.
|
||||
# Memory ledger against the box's 121.2 GiB unified (the n-gram table is
|
||||
# mmap'd page cache beside the checkpoint, not a pinned charge — the
|
||||
# card's own quickstart arithmetic does not bill it):
|
||||
# weights ~62 GiB resident + KV 14.0 + arena (HALOGEN_MAX_TOK=8192 =>
|
||||
# ~8 GiB per FLAGS.md arena formula 3.8+8.4*(MT/16384)) + vision ~2
|
||||
# = ~86 GiB engine-side; HALOGEN_HOST_RESERVE_GIB=20 keeps the host
|
||||
# honest; baseline co-tenancy (frank-secondary, ComfyUI, Caddy/observ)
|
||||
# honest; baseline co-tenancy (frank-zealot, ComfyUI, Caddy/observ)
|
||||
# ~10-15. ~15 GiB margin. The boot ledger is the referee — read it
|
||||
# after deploy and re-size if the engine prints a clamp.
|
||||
# HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy
|
||||
@@ -88,50 +88,51 @@
|
||||
# before. Kernel needs CONFIG_HSA_AMD_SVM (CachyOS ships it) and
|
||||
# amdgpu.noretry=0 (check at deploy).
|
||||
|
||||
# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
|
||||
# is what was qualified). 0.16.2 = the version the model card names as its
|
||||
# Image: digest-pinned (a tag is a mutable pointer — the same tag can
|
||||
# silently deliver different bits — so the digest pins exactly the bits
|
||||
# that were qualified). 0.16.2 = the version the model card names as its
|
||||
# quickstart pin; RepoDigest captured at pull on 2026-10-06.
|
||||
FRANK_LARGE_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
|
||||
FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
|
||||
# Container identity: kept from the 27B era (unit ExecStop names it).
|
||||
FRANK_LARGE_CONTAINER=frank-large-halogen
|
||||
FRANK_DRAGOON_CONTAINER=frank-dragoon-halogen
|
||||
# Old quadlet-era container identity; the serve script clears stale names.
|
||||
FRANK_LARGE_LEGACY_CONTAINERS=frank-large
|
||||
FRANK_DRAGOON_LEGACY_CONTAINERS=frank-dragoon
|
||||
# host:container — the frontend listens on 8731 inside (HALOGEN_API_PORT
|
||||
# default); the engine port (8730) is never published.
|
||||
FRANK_LARGE_PUBLISH=7770:8731
|
||||
FRANK_DRAGOON_PUBLISH=7770:8731
|
||||
|
||||
FRANK_LARGE_MODEL_DIR=/home/dank/models/halogen-flashnext-v2
|
||||
FRANK_DRAGOON_MODEL_DIR=/home/dank/models/halogen-flashnext-v2
|
||||
# The checkpoint: Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated
|
||||
# qwen38-flash-next-v2-swift15-abliterated.hgn — compacted r2 of the card's
|
||||
# append-only original (byte-identical live tensors, padding dropped:
|
||||
# 66,687,678,464 B vs 89,775,868,032 B). LFS sha256 verified at fetch.
|
||||
FRANK_LARGE_CHECKPOINT=qwen38-flash-next-v2-swift15-abliterated.hgn
|
||||
FRANK_LARGE_CHECKPOINT_BYTES=66687678464
|
||||
FRANK_LARGE_CHECKPOINT_SHA256=9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
|
||||
FRANK_DRAGOON_CHECKPOINT=qwen38-flash-next-v2-swift15-abliterated.hgn
|
||||
FRANK_DRAGOON_CHECKPOINT_BYTES=66687678464
|
||||
FRANK_DRAGOON_CHECKPOINT_SHA256=9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
|
||||
# The n-gram lookup table: REQUIRED (the engine reads it beside the
|
||||
# checkpoint; the card's quickstart names it explicitly). From
|
||||
# peonist-ai/halogen-qwen3.8-flash-next @ 5cc17cea.
|
||||
FRANK_LARGE_NGRAM_TABLE=qwen38-flash-next-ngram.hgn
|
||||
FRANK_LARGE_NGRAM_BYTES=51200246144
|
||||
FRANK_LARGE_NGRAM_SHA256=3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
|
||||
FRANK_DRAGOON_NGRAM_TABLE=qwen38-flash-next-ngram.hgn
|
||||
FRANK_DRAGOON_NGRAM_BYTES=51200246144
|
||||
FRANK_DRAGOON_NGRAM_SHA256=3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
|
||||
# The vision tower: OPTIONAL on the engine, ENABLED here (image input).
|
||||
# From peonist-ai/halogen-qwen3.8-flash-next @ e053f488.
|
||||
FRANK_LARGE_VISION_TOWER=qwen38-flash-next-vision.hgn
|
||||
FRANK_LARGE_VISION_BYTES=897916416
|
||||
FRANK_LARGE_VISION_SHA256=d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
|
||||
# The tokenizer is a FLAT DIRECTORY, $FRANK_LARGE_MODEL_DIR/tokenizer,
|
||||
FRANK_DRAGOON_VISION_TOWER=qwen38-flash-next-vision.hgn
|
||||
FRANK_DRAGOON_VISION_BYTES=897916416
|
||||
FRANK_DRAGOON_VISION_SHA256=d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
|
||||
# The tokenizer is a FLAT DIRECTORY, $FRANK_DRAGOON_MODEL_DIR/tokenizer,
|
||||
# mounted whole at /tokenizer. The two repos ship the same one; fetched
|
||||
# fresh from peonist @ e053f488 (tokenizer.json is LFS, byte-identical to
|
||||
# the 27B-era copy; the rest are plain git blobs — size-checked only). The
|
||||
# serve gate and the deployer refuse a missing file or ANY symlink: an HF
|
||||
# cache snapshot symlinks each file into blobs/, and those links dangle
|
||||
# inside the container mount.
|
||||
FRANK_LARGE_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json"
|
||||
FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json"
|
||||
|
||||
# --- engine flags (docs/FLAGS.md of the flash-server repo; anything unset
|
||||
# stays at the image default, which is what was qualified) ------------------
|
||||
# Served model id — Protoss name (wire rename 2026-10-09); the service, unit,
|
||||
# scripts and this file keep the frank-large name. OpenWebUI api_configs
|
||||
# scripts and this file keep the frank-dragoon name. OpenWebUI api_configs
|
||||
# model_ids for the :7770 backend MUST name this value.
|
||||
HALOGEN_MODEL_ID=frank-dragoon
|
||||
HALOGEN_CTX=262144
|
||||
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=halogen-flash-server container (Quat3rnion abliterated Swift1.5 of Qwen3.8-Flash-Next v2, MTP spec decode, 2 slots x 200192, vision ON), root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
|
||||
Description=halogen-flash-server container (Quat3rnion abliterated Swift1.5 of Qwen3.8-Flash-Next v2, MTP spec decode, 2 slots x 200192, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
@@ -11,11 +11,11 @@ Group=dank
|
||||
# RLIMIT smaller than the pinned set cannot fail the load. Parity with the
|
||||
# other LLM units on this fleet.
|
||||
LimitMEMLOCK=infinity
|
||||
Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
|
||||
Environment="FRANK_DRAGOON_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-dragoon.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-dragoon.sh
|
||||
# systemd stop alone cannot stop the container child; stop it explicitly
|
||||
# (frank-primary pattern). The engine drains on SIGTERM.
|
||||
ExecStop=/usr/bin/podman stop -t 60 frank-large-halogen
|
||||
# (frank-archon pattern). The engine drains on SIGTERM.
|
||||
ExecStop=/usr/bin/podman stop -t 60 frank-dragoon-halogen
|
||||
TimeoutStartSec=1200
|
||||
TimeoutStopSec=120
|
||||
# always, not on-failure: an engine that exits 0 after an internal fault
|
||||
@@ -23,7 +23,7 @@ TimeoutStopSec=120
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
# The engine pins ~62 GiB of v2 weights and keeps a multi-GiB working set
|
||||
# on a box that also hosts frank-secondary, ComfyUI and the Caddy/observability
|
||||
# on a box that also hosts frank-zealot, ComfyUI and the Caddy/observability
|
||||
# stack; make it a late OOM candidate, not the first.
|
||||
OOMScoreAdjust=-500
|
||||
|
||||
@@ -1,168 +0,0 @@
|
||||
# frank-primary.env — radiance (StillDeadcode, Apache-2.0) serving the
|
||||
# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at NATIVE precision
|
||||
# (modelopt block-FP8 throughout, no requantisation) on BOTH R9700s (TP=2),
|
||||
# dankserver :7770.
|
||||
#
|
||||
# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
|
||||
# pinned host RAM; git tag frank-primary-freetoken-pr587-20261005 is the
|
||||
# pre-cutover state — its unit/env/serve trio and Dockerfile are all in the
|
||||
# tag and rebuildable). Before that: tcclaviger vLLM fork (tag
|
||||
# frank-primary-vllm-shapeG-20261003).
|
||||
#
|
||||
# WHY RADIANCE: the engine is C++/HIP with libr4d kernels built for
|
||||
# gfx1201, drives `podman run` directly from the digest-pinned
|
||||
# stilldeadcode/radiance image (287 MiB; host needs only the amdgpu driver),
|
||||
# and its qwen4exp plugin reads the modelopt FP8 encoding NATIVELY
|
||||
# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram
|
||||
# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring
|
||||
# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram."
|
||||
# The 4-bit-codebook publisher shape (qwen3.8-next-flash-fp8-iq4r-moe.rad,
|
||||
# --host-pool-mib 12288) remains the documented fallback if the native
|
||||
# container proves unbootable.
|
||||
#
|
||||
# PAYLOAD: /home/dank/models/rad/qwen4exp-fp8-native.rad — a rad-convert
|
||||
# NO-RECIPE pure copy of the official 131-shard checkpoint (every weight
|
||||
# keeps its block-FP8 encoding; ~161 GiB). The convert runs against a
|
||||
# staging dir whose config.json drops `quantization_config`: the descriptor
|
||||
# string is only the plugin SELECTION key (spec §2.4 — exact match, no
|
||||
# fallback), and no shipped plugin claims (qwen4exp, "fp8"); the shipped
|
||||
# qwen4exp plugin claims (qwen4exp, "") which is what recipe-built
|
||||
# containers carry. Per-weight encodings are data, not the descriptor —
|
||||
# removing the key changes nothing about what the weights hold. Built by
|
||||
# scripts/build-frank-primary-rad.sh (staging + convert + rad-info).
|
||||
#
|
||||
# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml):
|
||||
# - The HTTP listener binds AFTER the full load (main.cpp: engine.start()
|
||||
# then engine.serve()) — during load the port refuses connections, so a
|
||||
# bare curl /health gate is honest. /health == /ping: 200 JSON while
|
||||
# serving, 503 only while stopping (core/server/admin.cpp:385).
|
||||
# - Reasoning: --reasoning-format defaults to auto, emitting thinking in
|
||||
# `reasoning_content` (config.cpp). Tool calls: the qwen3.8 dialect, from
|
||||
# the chat format. Both need no parser flags.
|
||||
# - Sampling: the OpenAI endpoint accepts temperature, top_p, top_k, min_p,
|
||||
# typical_p, presence_penalty, frequency_penalty, repetition_penalty,
|
||||
# repeat_last_n, chat_template_kwargs, reasoning_effort,
|
||||
# enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) —
|
||||
# models.yml's extraBody set is fully inside it. Unknown params are
|
||||
# refused (400), which is why nothing unverified rides the wire.
|
||||
# - MTP: the official checkpoint carries the mtp.* head (3,101 tensors,
|
||||
# mtp_num_hidden_layers=1) and the plugin probes draft depth 3 — the
|
||||
# publisher's throughput peak for this model. Stated explicitly with
|
||||
# --num-speculative-tokens 3.
|
||||
# - The fused TP all-reduce is declared over max-num-seqs x
|
||||
# (1 + --num-speculative-tokens) rows; 3 x 3 = 9 rows fits either wire.
|
||||
# Engine pin: Codeberg tag v1.0.13 (commit 53282fa), Docker Hub tag 1.0.13,
|
||||
# index digest below. REJECTED UPGRADE: 1.2.0 (sha256:665775ed1d90de6dbb
|
||||
# 25558e07fc60d20328f1346f96f3c6e00845eb763f01fa, tag 1.2.0, commit
|
||||
# e95d397) was benched 2026-10-06 on this exact shape (fp8-native rad,
|
||||
# vram pair, spec 2, seqs 3, 43.6k greedy probe): decode 29.21 t/s and
|
||||
# draft acceptance 0.7123 vs the 1.0.13 control's 30.08 and 0.7549.
|
||||
# Acceptance on the greedy harness is deterministic (the control
|
||||
# reproduced the recorded 0.755 exactly), so the drop is a real numerics
|
||||
# change — the 1.2.0 MoE weight-rotation change (92b8aaf) is the named
|
||||
# suspect; decode loss tracks the acceptance ratio (0.943 vs 0.945).
|
||||
# The vram-pair translation itself is exact (1.2.0 boot ledger took
|
||||
# vram_weights 25.46 GiB / vram_kv 3.46 GiB per card byte-identical to
|
||||
# the 1.0.13 ledger at EVC 0.82). Re-test 1.2.0+ when the rotation change
|
||||
# is untangled; 1.2.0 also REMOVED --expert-vs-cache-ratio in favour of
|
||||
# the --vram-weights-mib / --vram-kv-mib pair (core/config.cpp) — this
|
||||
# file already uses the pair, which BOTH pins accept (rollback symmetry).
|
||||
FRANK_PRIMARY_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
|
||||
# The .rad container FILE (mounted under /models) and the state dir for the
|
||||
# prefix-cache disk tier (mounted rw at /data). The model dir holds ONLY
|
||||
# .rad files; the raw HF download and its staging dir live beside it and
|
||||
# are never mounted.
|
||||
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/rad
|
||||
FRANK_PRIMARY_MODEL_FILE=qwen4exp-swift15-w4dn8-i8-hc8m.rad
|
||||
FRANK_PRIMARY_STATE_DIR=/home/dank/radiance-state
|
||||
# The state dir's tune/ subdirectory holds rad-tune's kernel-selection
|
||||
# cache (the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune
|
||||
# at boot; $RADIANCE_HOME defaults to /opt/radiance, in the image, so the
|
||||
# cache must ride a host mount — serve script mounts it, see the runbook's
|
||||
# "Tuning levers").
|
||||
# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
|
||||
# name across systemctl and podman ps. The runtime changes; the name does
|
||||
# not. The legacy list retires every predecessor identity that could hold
|
||||
# the GPUs or the port.
|
||||
FRANK_PRIMARY_CONTAINER=frank-primary
|
||||
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
|
||||
# Served model id — Protoss name (wire rename 2026-10-09). The container,
|
||||
# unit, scripts and this file keep the frank-primary name above; only the
|
||||
# alias on the wire moved. Radiance validates the requested model name, so
|
||||
# every client (harness, bench scripts, probes) MUST send this value.
|
||||
FRANK_PRIMARY_SERVED_NAME=frank-archon
|
||||
FRANK_PRIMARY_PORT=7770
|
||||
# Frank's chat-template call: froggeric's fixed Qwen3.8 template (the
|
||||
# repo's root chat_template.jinja, vendored here; the checkpoint's own
|
||||
# template is baked into the .rad by rad-convert and this overrides it).
|
||||
# Radiance derives the reasoning/tool-call reply format from it.
|
||||
FRANK_PRIMARY_TEMPLATE=/home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja
|
||||
|
||||
# --- engine shape (first-boot conservative; the boot ledger arbitrates) --
|
||||
# Tensor parallel: BOTH R9700s. The board swap (2026-09-20, ProArt B850
|
||||
# x8/x8 on root ports) made collectives legal on this host; radiance's TP
|
||||
# wire is libr4d's own, not RCCL.
|
||||
FRANK_PRIMARY_TP=2
|
||||
# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice:
|
||||
# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
|
||||
# engine default.
|
||||
FRANK_PRIMARY_TP_WIRE=wht6
|
||||
# Decode slots. 2 is Frank's first-boot value ("use parallelism = 2 to
|
||||
# start"); raise only after a real-traffic look at slot occupancy and the
|
||||
# KV pool's headroom line. The recurrent-state cost scales with it (54
|
||||
# MiB/copy/card, 2 copies with speculation).
|
||||
FRANK_PRIMARY_MAX_SEQS=3
|
||||
# Served window. Native is 262,144; 200,000 is the publisher's production
|
||||
# value (compose) and what models.yml's contextWindow must quote — a longer
|
||||
# prompt is refused by the engine, never truncated.
|
||||
FRANK_PRIMARY_MAX_MODEL_LEN=200000
|
||||
FRANK_PRIMARY_PLACEMENT=expert_tiered
|
||||
# Pinned host pool for the expert rows that do not fit VRAM. Native FP8
|
||||
# experts are 112.8 GiB total (56.4 GiB per rank slice) vs the 4-bit
|
||||
# publisher shape's ~29 GiB — this pool is the cost of native precision.
|
||||
# The boot ledger prints the pool's coverage ("the pool backs N% of the
|
||||
# worst case ... M full-length sessions"); resize from that line, not from
|
||||
# arithmetic.
|
||||
FRANK_PRIMARY_HOST_POOL_MIB=40960
|
||||
# VRAM left unclaimed per card (publisher default 96 MiB).
|
||||
FRANK_PRIMARY_GPU_HEADROOM_MIB=96
|
||||
# VRAM split of claimed memory per card, MiB (0 = derive from the card).
|
||||
# Engine 1.2.0 REMOVED --expert-vs-cache-ratio; these two knobs replace
|
||||
# it. The values transcribe the live 1.0.13 boot ledger at ratio 0.82:
|
||||
# vram_weights 25.46 GiB, vram_kv 3.46 GiB per card -> 26071 / 3543.
|
||||
# Bracket arms scale the KV term linearly in (1-r) from the 0.82 anchor
|
||||
# with weights fixed: ratio 0.90 -> KV 1968, ratio 0.95 -> KV 984. The
|
||||
# boot ledger arbitrates; re-derive after any payload change.
|
||||
FRANK_PRIMARY_VRAM_WEIGHTS_MIB=26071
|
||||
FRANK_PRIMARY_VRAM_KV_MIB=3543
|
||||
# KV cache in FP8 (7.32 KiB/token at this model's 12 full-attention layers,
|
||||
# 2 KV heads, 256 head_dim, K+V).
|
||||
FRANK_PRIMARY_KV_DTYPE=fp8
|
||||
# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
|
||||
# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
|
||||
# flag states it so the ledger records the intent).
|
||||
FRANK_PRIMARY_SPEC_TOKENS=2
|
||||
FRANK_PRIMARY_MAX_BATCHED_TOKENS=2048
|
||||
# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
|
||||
FRANK_PRIMARY_PREFIX_HOST_MIB=4096
|
||||
FRANK_PRIMARY_PREFIX_DISK_MIB=65536
|
||||
# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
|
||||
# RAM the cards read. `ram` REFUSES the start if the memlock limit is too
|
||||
# small — the unit's LimitMEMLOCK=infinity is the other half of this law.
|
||||
FRANK_PRIMARY_NGRAM_PLACEMENT=ram
|
||||
# TTM preflight: the pinned ledger (2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
|
||||
# prefix host) is ~132 GiB. ttm's pages_limit/pages_max cap the GPU-mappable
|
||||
# pinned pages; the current cap (32,252,917 pages = ~123 GiB) is BELOW the
|
||||
# plan, so the serve script refuses to start until both are raised. This is
|
||||
# the floor it enforces (in MiB), with ~8 GiB of margin.
|
||||
FRANK_PRIMARY_TTM_MIN_MIB=140000
|
||||
# Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
|
||||
# multi-flag values as one string; the serve script expands it unquoted on
|
||||
# purpose.
|
||||
FRANK_PRIMARY_EXTRA_ARGS=
|
||||
# The .rad's minimum size in MiB (deployer gate; a truncated convert is a
|
||||
# guaranteed long-timeout boot instead of a fast failure). The converted
|
||||
# container measured 177,109 MiB (1,857,127,997,440 B = 173.0 GiB; the
|
||||
# plan-only estimate 172.76 GiB covers the data planes but not the metadata
|
||||
# tail) - floor sits just under the payload.
|
||||
FRANK_PRIMARY_RAD_MIN_MIB=110000
|
||||
@@ -1,4 +1,4 @@
|
||||
# frank-secondary — llama.cpp (master, CUDA sm_86) serving
|
||||
# frank-zealot — llama.cpp (master, CUDA sm_86) serving
|
||||
# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090,
|
||||
# :7771.
|
||||
#
|
||||
@@ -16,7 +16,7 @@
|
||||
# custom-quant lock-in like the ROCmFPX era). NO nextn/mtp tensors exist, so
|
||||
# in-file MTP is impossible here; SPEC_TYPE=none is forced by the payload.
|
||||
#
|
||||
# KV rate law: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per
|
||||
# KV rate arithmetic: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per
|
||||
# token per cache. f16 K + f16 V = 32,768 B/token = 32 KiB/token, so the
|
||||
# 401,408-token pool is 12.25 GiB. The 24 DeltaNet layers cost a fixed
|
||||
# recurrent state per SLOT (not per token) — that is why two 200k slots fit
|
||||
@@ -38,7 +38,7 @@ MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensore
|
||||
# Served model id — the harness/family name (Protoss rename completed to the
|
||||
# wire 2026-10-09: harness ids + homepage cards 2026-10-08, /v1/models aliases
|
||||
# this day). The SERVICE, unit, scripts and this env file keep the
|
||||
# frank-secondary name; only the alias on the wire changed. Consumers that
|
||||
# frank-zealot name; only the alias on the wire changed. Consumers that
|
||||
# key on the served id: OpenWebUI (model row, access_grant, api_configs
|
||||
# model_ids) and the harness models.yml.
|
||||
ALIAS=frank-zealot
|
||||
@@ -1,5 +1,5 @@
|
||||
[Unit]
|
||||
Description=llama.cpp Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K GGUF (qwen35 hybrid, 2x200704 slots, f16 K+V, MTP off, mmproj vision) on the RTX 3090 (dankclaw, frank-secondary)
|
||||
Description=llama.cpp Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K GGUF (qwen35 hybrid, 2x200704 slots, f16 K+V, MTP off, mmproj vision) on the RTX 3090 (dankclaw, frank-zealot)
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
@@ -10,8 +10,8 @@ Group=dank
|
||||
# CUDA pins its staging buffers, so the process needs the memlock budget even
|
||||
# though the weights themselves are copied into VRAM.
|
||||
LimitMEMLOCK=infinity
|
||||
Environment="FRANK_SECONDARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-secondary.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-secondary.sh
|
||||
Environment="FRANK_ZEALOT_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-zealot.env"
|
||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-zealot.sh
|
||||
Restart=on-failure
|
||||
RestartSec=10
|
||||
|
||||
@@ -24,7 +24,7 @@ KLD_OUT=/home/dank/radiance-state/kld-gate.json
|
||||
: > "$KLD_OUT"
|
||||
cp -f /home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja \
|
||||
/home/dank/models/tpl.jinja 2>/dev/null || true
|
||||
sudo systemctl stop frank-primary >/dev/null 2>&1 || true
|
||||
sudo systemctl stop frank-archon >/dev/null 2>&1 || true
|
||||
podman rm -f "$C" >/dev/null 2>&1 || true
|
||||
|
||||
podman run --rm --name "$C" \
|
||||
|
||||
@@ -44,17 +44,17 @@ JSONL_EXTS = (".jsonl", ".json")
|
||||
DOC_DIR = "serverAppConfigs-docs"
|
||||
|
||||
FLEET_CMDS = [
|
||||
"systemctl status frank-primary --no-pager",
|
||||
"journalctl -u frank-primary -n 100 --no-pager",
|
||||
"systemctl status frank-archon --no-pager",
|
||||
"journalctl -u frank-archon -n 100 --no-pager",
|
||||
"podman ps --format '{{.Names}} {{.Status}}'",
|
||||
"podman logs --tail 50 frank-primary",
|
||||
"podman logs --tail 50 frank-archon",
|
||||
"curl -s http://192.168.0.235:7770/health",
|
||||
"curl -s http://192.168.0.235:7770/metrics | grep radiance_",
|
||||
"df -h /home/dank | tail -1",
|
||||
"free -g",
|
||||
"sudo systemctl restart frank-primary",
|
||||
"sudo systemctl restart frank-archon",
|
||||
"cd /home/dank/serverAppConfigs && git pull --ff-only",
|
||||
"./scripts/deploy.sh dankserver --service frank-primary",
|
||||
"./scripts/deploy.sh dankserver --service frank-archon",
|
||||
"./tests/run.sh 2>&1 | tail -5",
|
||||
"jq . /home/dank/models/rad/index.json | head -40",
|
||||
"python3 -c \"import json; print(json.load(open('config.json'))['num_hidden_layers'])\"",
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
#
|
||||
# STATUS: WIP — first of three patches in the "compile the GLM trunk" changelist. The
|
||||
# set does NOT yet boot the engine (see 03 for the next blocker), so it is NOT part of
|
||||
# the deployed frank-primary shape. Nothing in scripts/ mounts these files.
|
||||
# the deployed frank-archon shape. Nothing in scripts/ mounts these files.
|
||||
#
|
||||
# WHY THIS IS THE RIGHT LEVER (measured 2026-09-30, not inferred):
|
||||
#
|
||||
|
||||
@@ -29,7 +29,8 @@
|
||||
# its base SupportsEagleBase (has_own_lm_head, has_own_embed_tokens) -- and
|
||||
# getattr_static()s each one off the instance, breaking on the first miss.
|
||||
# So BOTH wrappers need all three attributes.
|
||||
# * MRO law: INHERITING SupportsEagle3 as a base dies at class creation with
|
||||
# * Method-resolution order forbids the obvious inheritance: adding
|
||||
# SupportsEagle3 as a base dies at class creation with
|
||||
# "Cannot create a consistent method resolution order (MRO) for bases ..."
|
||||
# (GLM's wrappers already carry four/five Protocol-derived bases). The
|
||||
# attribute route avoids the MRO entirely. Also proven by simulation:
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# gpu_worker-kv-cap.patch — fork-local fix for the expert-offload KV cap.
|
||||
#
|
||||
# STATUS: NOT part of the deployed frank-primary shape. It exists because the
|
||||
# STATUS: NOT part of the deployed frank-archon shape. It exists because the
|
||||
# fork's plan under-models this model's KV, and it is the enabler for a future
|
||||
# MTP attempt (the production GLM shape runs without MTP and boots fine on the
|
||||
# plan's own reserve).
|
||||
|
||||
Executable
+202
@@ -0,0 +1,202 @@
|
||||
#!/usr/bin/env python3
|
||||
"""probe-frank-archon-kv.py — verify the seqs x max-model-len KV contract on a
|
||||
frank-archon (radiance) service. Run it ON dankserver: it talks to
|
||||
127.0.0.1:7770 and builds its prompts from the repo checkout there.
|
||||
No scratch container, no mask: the service under test is the service.
|
||||
|
||||
ssh dankserver 'python3 /home/dank/serverAppConfigs/scripts/probe-frank-archon-kv.py'
|
||||
|
||||
Four probes, in increasing cost:
|
||||
A decode at depth — one ~51k-token chat completion, 512 tokens, greedy.
|
||||
Reports the engine's own `timings` (prefill, decode, draft acceptance).
|
||||
Measurement discipline: generate at least 128 tokens (a short run
|
||||
measures first-token noise, not the server), greedy, real text.
|
||||
B the capacity contract — as many CONCURRENT chats of ~197k prompt tokens as
|
||||
the deployed `FRANK_ARCHON_MAX_SEQS` (read from the repo env file on this
|
||||
host), each carrying a unique marker at its head. All must answer their own
|
||||
marker. This is what the endpoint owes: N full-height conversations
|
||||
resident at once, exact retrieval in every one of them.
|
||||
C prefix-cache restore — probe A's cold prompt again, so the disk tier has
|
||||
to give the blocks back (cache_n > 0, TTFT collapses).
|
||||
D concurrency decode — `seqs` CONCURRENT 512-token generations at ~25 % of
|
||||
the window. This is the cost a slot count imposes on DECODE, which no
|
||||
prefill number shows: the fused-step schedule is unfair, so every stream is
|
||||
printed plus the aggregate. Measured 2026-10-13 at 3 x 200,000 fp8:
|
||||
single-stream 99.0 t/s (43.5k) and 93.3 (197k) — decode is nearly
|
||||
window-free — against 3-stream 113.9/43.3/10.9 t/s = 18.4 t/s aggregate.
|
||||
Concurrency, not the slab, is what decode pays for slots. The 2026-10-13
|
||||
shape hunt missed this for a whole day because leg B generates 24 tokens.
|
||||
PROBE LAW (learned the hard way, twice): radiance routes the thinking block to
|
||||
`reasoning_content`, so a chat request whose budget is spent inside
|
||||
`<think>` returns EMPTY `content` with `finish_reason: stop`. A probe that
|
||||
reads `content` alone scores 0/2 on a healthy server. Every request here sets
|
||||
chat_template_kwargs.enable_thinking=false and reads both fields.
|
||||
|
||||
A/B method for a shape change (no scratch container): write a variant env file
|
||||
on the host, point a transient systemd drop-in at it
|
||||
(`Environment=FRANK_ARCHON_ENV_FILE=...` — drop-ins are read AFTER the unit's
|
||||
own Environment= lines, so they win), restart, probe, delete the drop-in,
|
||||
daemon-reload, restart. `systemctl mask` is not available: the unit file in
|
||||
/etc/systemd/system is a relink symlink into the checkout.
|
||||
"""
|
||||
import json
|
||||
import glob
|
||||
import time
|
||||
import urllib.request
|
||||
import concurrent.futures
|
||||
|
||||
BASE = "http://127.0.0.1:7770"
|
||||
MODEL = "frank-archon"
|
||||
REPO = "/home/dank/serverAppConfigs"
|
||||
GLOBS = [f"{REPO}/scripts/*.sh", f"{REPO}/serverAppConfigs-docs/**/*.md"]
|
||||
|
||||
|
||||
def post(payload, timeout=1800):
|
||||
req = urllib.request.Request(
|
||||
BASE + "/v1/chat/completions",
|
||||
data=json.dumps(payload).encode(),
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
t0 = time.time()
|
||||
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||
body = json.loads(r.read().decode())
|
||||
hdr = dict(r.headers.items())
|
||||
return body, time.time() - t0, hdr
|
||||
|
||||
|
||||
def get(path, timeout=60):
|
||||
with urllib.request.urlopen(BASE + path, timeout=timeout) as r:
|
||||
return json.loads(r.read().decode())
|
||||
|
||||
|
||||
def corpus(min_chars):
|
||||
files = []
|
||||
for g in GLOBS:
|
||||
files.extend(sorted(glob.glob(g, recursive=True)))
|
||||
out, total = [], 0
|
||||
for f in files:
|
||||
try:
|
||||
t = open(f, encoding="utf-8", errors="replace").read()
|
||||
except OSError:
|
||||
continue
|
||||
out.append(t)
|
||||
total += len(t)
|
||||
if total > min_chars:
|
||||
break
|
||||
return "".join(text for text in out)
|
||||
|
||||
|
||||
def prompt(text, chars, salt, fact):
|
||||
"""Real repo text, salted so the prefix cache cannot answer, marker planted
|
||||
near the head, question at the tail."""
|
||||
head = f"\n[SALT {salt}] confidential marker: {fact}\n"
|
||||
body = text + text
|
||||
p = (head + body * max(1, chars // len(body)))[:chars]
|
||||
return p + "\n\nQuestion: what is the confidential marker in the " \
|
||||
"SALT block near the top of this document? Answer with the marker " \
|
||||
"alone, no other words.\n"
|
||||
|
||||
|
||||
def chat(text, chars, salt, fact, max_tokens):
|
||||
return {"model": MODEL,
|
||||
"messages": [{"role": "user",
|
||||
"content": prompt(text, chars, salt, fact)}],
|
||||
"max_tokens": max_tokens, "temperature": 0.0,
|
||||
"chat_template_kwargs": {"enable_thinking": False}}
|
||||
|
||||
|
||||
def show(tag, j, wall, expect=None):
|
||||
tim, u = j.get("timings") or {}, j.get("usage", {})
|
||||
ch = (j.get("choices") or [{}])[0]
|
||||
msg = ch.get("message") or {}
|
||||
txt = (msg.get("content") or msg.get("reasoning_content") or "").strip()
|
||||
acc = (tim["draft_n_accepted"] / tim["draft_n"]) if tim.get("draft_n") else None
|
||||
line = (f"[{tag}] prompt={u.get('prompt_tokens')} gen={u.get('completion_tokens')} "
|
||||
f"wall={wall:.1f}s prefill={tim.get('prompt_per_second', 0):.0f} t/s "
|
||||
f"decode={tim.get('predicted_per_second', 0):.1f} t/s "
|
||||
f"draft={tim.get('draft_n')}/{tim.get('draft_n_accepted')} "
|
||||
f"accept={acc if acc is None else round(acc, 4)} "
|
||||
f"cache_n={tim.get('cache_n')} finish={ch.get('finish_reason')}")
|
||||
if expect:
|
||||
hit = expect.split("-")[0] in txt.lower()
|
||||
line += f" expected={expect} answered={txt[:80]!r} HIT={hit}"
|
||||
print(line)
|
||||
|
||||
|
||||
def deployed_shape():
|
||||
"""The contract is the deployed env, not a constant: slot count and window
|
||||
come from the checkout on this host, so the probe cannot drift from the
|
||||
service it is testing (the 2026-10-13 shape is 4 x 200,000; before that
|
||||
2 x 200,000, and the fp8 era 3 x 200,000)."""
|
||||
shape = {}
|
||||
with open(f"{REPO}/scripts/frank-archon.env") as fh:
|
||||
for line in fh:
|
||||
line = line.strip()
|
||||
if line.startswith("FRANK_ARCHON_") and "=" in line:
|
||||
key, _, val = line.partition("=")
|
||||
shape[key] = val.strip().strip('"')
|
||||
return (int(shape["FRANK_ARCHON_MAX_SEQS"]),
|
||||
int(shape["FRANK_ARCHON_MAX_MODEL_LEN"]),
|
||||
shape["FRANK_ARCHON_KV_DTYPE"])
|
||||
|
||||
|
||||
def main():
|
||||
seqs, maxlen, dtype = deployed_shape()
|
||||
print(f"shape: {seqs} x {maxlen:,} tokens, kv {dtype}")
|
||||
print("health:", json.dumps(get("/health")))
|
||||
print("models:", [m.get("id") for m in get("/v1/models").get("data", [])])
|
||||
deep = corpus(160_000)
|
||||
wide = corpus(1_100_000)
|
||||
|
||||
# 3.28 chars/token is the measured ratio of this corpus through the
|
||||
# froggeric template (646,000 chars -> 196,919 tokens at the 2026-10-11
|
||||
# deploy); 98.5 % of the window leaves the response room under the ceiling.
|
||||
chars = int(0.985 * maxlen * 3.28)
|
||||
# Each marker's FIRST hyphen-separated word is unique, because show()
|
||||
# scores the hit on that word — a shared prefix would score every slot HIT
|
||||
# on any answer that says the word.
|
||||
words = ["cobalt", "quartz", "basalt", "ember", "willow", "cobweb", "flint"]
|
||||
facts = {f"B{i+1}": f"{words[i % len(words)]}-flag-{(i + 1) * 137}"
|
||||
for i in range(seqs)}
|
||||
bodies = {s: chat(wide, chars, s, f, 24) for s, f in facts.items()}
|
||||
print(f"[B] {len(bodies['B1']['messages'][0]['content']):,} chars per "
|
||||
f"request, x{seqs} concurrent")
|
||||
|
||||
# A + C first (cheap), B last (the expensive set).
|
||||
for tag, salt in (("A-cold", "A1-9137"), ):
|
||||
j, wall, _ = post(chat(deep, 140_000, salt, "zephyr-lantern-7", 512))
|
||||
show(tag, j, wall, "zephyr-lantern-7")
|
||||
t0 = time.time()
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=seqs) as ex:
|
||||
futs = {s: ex.submit(post, b) for s, b in bodies.items()}
|
||||
for s, fut in futs.items():
|
||||
try:
|
||||
j, wall, _ = fut.result()
|
||||
show(f"B-{s}", j, wall, facts[s])
|
||||
except Exception as e: # a preemption/timeout lands here
|
||||
print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
|
||||
print(f"[B] total {time.time() - t0:.1f}s")
|
||||
# D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth.
|
||||
dchars = int(0.25 * maxlen * 3.28)
|
||||
dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512)
|
||||
for i in range(seqs)]
|
||||
print(f"[D] {seqs} concurrent streams x 512 tokens, "
|
||||
f"{dchars:,} chars each")
|
||||
t0 = time.time()
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=seqs) as ex:
|
||||
futs = [ex.submit(post, b) for b in dbodies]
|
||||
for i, fut in enumerate(futs, 1):
|
||||
try:
|
||||
j, wall, _ = fut.result()
|
||||
show(f"D-s{i}", j, wall)
|
||||
except Exception as e: # a preemption/timeout lands here
|
||||
print(f"[D-s{i}] FAILED after {time.time() - t0:.1f}s: {e!r}")
|
||||
tot = time.time() - t0
|
||||
print(f"[D] {seqs} x 512 tokens in {tot:.1f}s = "
|
||||
f"{seqs * 512 / tot:.1f} t/s aggregate decode")
|
||||
j, wall, _ = post(chat(deep, 140_000, "A1-9137", "zephyr-lantern-7", 512))
|
||||
show("C-prefix-hit", j, wall, "zephyr-lantern-7")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,9 +1,9 @@
|
||||
# qwen4exp-swift15-w4dn8-i8-hc8m.recipe -- the Swift 1.5 finetune of Qwen3.8-Flash-Next
|
||||
# (ukisai/Swift1.5-Qwen3.8-Flash-Next, BF16 source, 335.3 GiB, key set identical to the official
|
||||
# base) as a radiance container for frank-primary: EVERY routed expert -- gate_up AND down --
|
||||
# through GPTQ into the w4nl64a8h table. The plugin's one-table-one-encoding law
|
||||
# base) as a radiance container for frank-archon: EVERY routed expert -- gate_up AND down --
|
||||
# through GPTQ into the w4nl64a8h table. The format assigns exactly ONE encoding per table
|
||||
# ("one table of experts takes one encoding, and only a whole expert kept plain bf16 is served
|
||||
# beside it", qwen4exp_fp8 rad_arch_declare) forbids the original split tier (down block-fp8 +
|
||||
# beside it", qwen4exp_fp8 rad_arch_declare — so the original split tier (down block-fp8 +
|
||||
# gate_up w4): the served kernel is w4nl64a8h for the whole expert set, bf16 experts beside it.
|
||||
# The rest is qwen4exp-w5dn64-i8-hc8m.recipe verbatim: int8 trunk linears, block-fp8 hc mixers
|
||||
# and MTP fc, the fixed-scale n-gram table, the 2-bit draft head.
|
||||
@@ -26,8 +26,8 @@
|
||||
# The protected experts, cast to bf16 ahead of both expert rules. THE NAMES BELOW ARE THE BASE
|
||||
# CHECKPOINT'S (from its 1M-token recording, qwen4exp-w5dn64-i8-hc8m.recipe) -- a warm start only.
|
||||
# The Swift 1.5 finetune's routing distribution is not assumed equal: the derivation step rewrites
|
||||
# this section from Swift's OWN calibration recording before this recipe is ever run, by the base's
|
||||
# law -- an expert whose share of its layer's down-input energy is at least 3% and at least three
|
||||
# this section from Swift's OWN calibration recording before this recipe is ever run, by the
|
||||
# protection rule -- an expert whose share of its layer's down-input energy is at least 3% and at least three
|
||||
# times its share of the routing, an even count on each rank's half of the layer (the four-bit GEMM
|
||||
# takes the rest as two equal tables by parity) by adding that half's next most energetic expert.
|
||||
# PROTECTED-BEGIN (rewritten from Swift's OWN grams by derive-protected-experts.py)
|
||||
|
||||
@@ -30,9 +30,9 @@ KLD_REF_DIR=/home/dank/corpora/kld-ref
|
||||
CALIB_DIR=/home/dank/corpora/calib
|
||||
KLD_CORPUS=/home/dank/corpora/out/kld.jsonl
|
||||
|
||||
restore() { sudo systemctl start frank-primary >/dev/null 2>&1 || true; }
|
||||
restore() { sudo systemctl start frank-archon >/dev/null 2>&1 || true; }
|
||||
trap restore EXIT
|
||||
sudo systemctl stop frank-primary >/dev/null 2>&1 || true
|
||||
sudo systemctl stop frank-archon >/dev/null 2>&1 || true
|
||||
podman rm -f "$C" >/dev/null 2>&1 || true
|
||||
|
||||
ARGS=(--model /models/rad/qwen4exp-swift15-bf16.rad
|
||||
|
||||
Executable
+196
@@ -0,0 +1,196 @@
|
||||
#!/usr/bin/env bash
|
||||
# serve-frank-archon.sh — frank-archon endpoint: radiance (StillDeadcode,
|
||||
# Apache-2.0) serving the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at
|
||||
# NATIVE block-FP8 precision from a rad-convert container
|
||||
# (/home/dank/models/rad/qwen4exp-fp8-native.rad), on BOTH R9700s (TP=2,
|
||||
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (PR #587,
|
||||
# one card; git tag frank-archon-freetoken-pr587-20261005 is the
|
||||
# pre-cutover state).
|
||||
#
|
||||
# The run line below IS the engine argv: the image's entrypoint is the
|
||||
# engine and its tools sit beside it on PATH. Unlike every predecessor
|
||||
# there is no per-boot VRAM planner to second-guess: the boot ledger in the
|
||||
# log states every budget before anything is allocated (weights vs KV
|
||||
# split, the expert host pool's coverage of the worst case, the PLE pin,
|
||||
# the P2P answer per pair) — read it after every resize.
|
||||
# Deployed shape (2026-10-13): THREE decode slots x 200,000 tokens, fp8
|
||||
# (quantized) KV in a 5,120 MiB pool (contract 3 x 200,000 + 1.3 %) and a
|
||||
# 25,600 MiB resident expert slab — 30,720 MiB of the 30,969 MiB the engine's
|
||||
# own budget table makes claimable, spent on the slab because prefill is
|
||||
# slab-limited. 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache host
|
||||
# 8 GiB + disk 64 GiB, PLE pinned in RAM.
|
||||
#
|
||||
# The root unit (scripts/frank-archon.service, User=dank,
|
||||
# LimitMEMLOCK=infinity) grants the pin budget: 2 x 44 GiB expert pools +
|
||||
# the 47.7 GiB PLE table + 8 GiB prefix host tier — ~147 GiB of pinned
|
||||
# pages, which is ALSO why this script refuses to start when ttm's
|
||||
# pages_limit/pages_max cap is under FRANK_ARCHON_TTM_MIN_MIB (see below).
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_ARCHON_ENV_FILE:-$SCRIPT_DIR/frank-archon.env}"
|
||||
|
||||
if [ ! -f "$ENV_FILE" ]; then
|
||||
echo "ERROR: frank-archon env file absent: $ENV_FILE" >&2
|
||||
exit 1
|
||||
fi
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
source "$ENV_FILE"
|
||||
set +a
|
||||
|
||||
: "${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
|
||||
: "${FRANK_ARCHON_MODEL_DIR:?FRANK_ARCHON_MODEL_DIR must name the .rad dir}"
|
||||
: "${FRANK_ARCHON_MODEL_FILE:?FRANK_ARCHON_MODEL_FILE must name the .rad file}"
|
||||
: "${FRANK_ARCHON_STATE_DIR:?FRANK_ARCHON_STATE_DIR must name the /data state dir}"
|
||||
: "${FRANK_ARCHON_CONTAINER:?FRANK_ARCHON_CONTAINER must name the container}"
|
||||
: "${FRANK_ARCHON_SERVED_NAME:?FRANK_ARCHON_SERVED_NAME must name the alias}"
|
||||
: "${FRANK_ARCHON_PORT:?FRANK_ARCHON_PORT must be the port}"
|
||||
: "${FRANK_ARCHON_TEMPLATE:?FRANK_ARCHON_TEMPLATE must name the chat template}"
|
||||
: "${FRANK_ARCHON_TP:?FRANK_ARCHON_TP must be set}"
|
||||
: "${FRANK_ARCHON_TP_WIRE:?FRANK_ARCHON_TP_WIRE must be set}"
|
||||
: "${FRANK_ARCHON_MAX_SEQS:?FRANK_ARCHON_MAX_SEQS must be set}"
|
||||
: "${FRANK_ARCHON_MAX_MODEL_LEN:?FRANK_ARCHON_MAX_MODEL_LEN must be set}"
|
||||
: "${FRANK_ARCHON_PLACEMENT:?FRANK_ARCHON_PLACEMENT must be set}"
|
||||
: "${FRANK_ARCHON_HOST_POOL_MIB:?FRANK_ARCHON_HOST_POOL_MIB must be set}"
|
||||
: "${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB must be set}"
|
||||
: "${FRANK_ARCHON_VRAM_WEIGHTS_MIB:?FRANK_ARCHON_VRAM_WEIGHTS_MIB must be set}"
|
||||
: "${FRANK_ARCHON_VRAM_KV_MIB:?FRANK_ARCHON_VRAM_KV_MIB must be set}"
|
||||
: "${FRANK_ARCHON_KV_DTYPE:?FRANK_ARCHON_KV_DTYPE must be set}"
|
||||
: "${FRANK_ARCHON_SPEC_TOKENS:?FRANK_ARCHON_SPEC_TOKENS must be set}"
|
||||
: "${FRANK_ARCHON_MAX_BATCHED_TOKENS:?FRANK_ARCHON_MAX_BATCHED_TOKENS must be set}"
|
||||
: "${FRANK_ARCHON_PREFIX_HOST_MIB:?FRANK_ARCHON_PREFIX_HOST_MIB must be set}"
|
||||
: "${FRANK_ARCHON_PREFIX_DISK_MIB:?FRANK_ARCHON_PREFIX_DISK_MIB must be set}"
|
||||
: "${FRANK_ARCHON_NGRAM_PLACEMENT:?FRANK_ARCHON_NGRAM_PLACEMENT must be set}"
|
||||
: "${FRANK_ARCHON_TTM_MIN_MIB:?FRANK_ARCHON_TTM_MIN_MIB must be set}"
|
||||
: "${FRANK_ARCHON_TTM_TARGET_MIB:?FRANK_ARCHON_TTM_TARGET_MIB must be set}"
|
||||
: "${FRANK_ARCHON_RAD_MIN_MIB:?FRANK_ARCHON_RAD_MIN_MIB must be set}"
|
||||
|
||||
# The image gate is the digest-pinned ref itself: a name@sha256 ref exists
|
||||
# in storage only if THAT manifest was pulled — a retagged look-alike or a
|
||||
# newer latest does not satisfy it. `podman image exists` takes a
|
||||
# name@digest ref in one command — no pipefail/grep race (the 2026-09-30
|
||||
# lesson). Runs as dank: the unit's process uses dank's ROOTLESS podman
|
||||
# storage (root unit, User=dank — the fleet's frank-archon shape).
|
||||
if ! podman image exists "$FRANK_ARCHON_IMAGE"; then
|
||||
echo "ERROR: radiance image absent: $FRANK_ARCHON_IMAGE" >&2
|
||||
echo "frank-archon: podman pull docker://stilldeadcode/radiance:latest (as dank)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# The payload gate: the .rad container file, sized. A truncated convert
|
||||
# would otherwise buy a guaranteed long-timeout boot instead of this fast
|
||||
# failure. The staging dir / raw checkpoint are NOT this script's concern
|
||||
# (scripts/build-frank-archon-rad.sh built the file; its rad-info output
|
||||
# is the record).
|
||||
RAD="$FRANK_ARCHON_MODEL_DIR/$FRANK_ARCHON_MODEL_FILE"
|
||||
if [ ! -f "$RAD" ]; then
|
||||
echo "ERROR: .rad container missing: $RAD" >&2
|
||||
echo "frank-archon: run scripts/build-frank-archon-rad.sh" >&2
|
||||
exit 1
|
||||
fi
|
||||
RAD_MIB=$(( $(stat -c %s "$RAD") / 1048576 ))
|
||||
if [ "$RAD_MIB" -lt "$FRANK_ARCHON_RAD_MIN_MIB" ]; then
|
||||
echo "ERROR: .rad container too small: $RAD is ${RAD_MIB} MiB, want >= $FRANK_ARCHON_RAD_MIN_MIB" >&2
|
||||
echo "frank-archon: the convert was truncated — rebuild it" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$FRANK_ARCHON_TEMPLATE" ]; then
|
||||
echo "ERROR: chat template missing: $FRANK_ARCHON_TEMPLATE" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$FRANK_ARCHON_STATE_DIR/kvcache/frank-archon"
|
||||
# The engine's create is NOT recursive: with only the state dir mounted, its
|
||||
# "cannot create /data/kvcache/frank-archon: No such file or directory"
|
||||
# left the SSD prefix tier off (measured 2026-10-06 boot ledger). Pre-create
|
||||
# the full path on the host so the tier opens on first boot.
|
||||
|
||||
mkdir -p "$FRANK_ARCHON_STATE_DIR/tune"
|
||||
# rad-tune's kernel-selection cache (see the runbook's "Tuning levers"):
|
||||
# the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune at boot;
|
||||
# $RADIANCE_HOME defaults to /opt/radiance, which lives in the image, so
|
||||
# the cache must ride a host mount or it dies with the --rm container
|
||||
# that wrote it. The dir is pre-created for the same reason as kvcache
|
||||
# (the engine's create is not recursive).
|
||||
|
||||
if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
|
||||
echo "ERROR: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# TTM preflight: the pinned ledger (2 x FRANK_ARCHON_HOST_POOL_MIB + the
|
||||
# 47.7 GiB PLE table + the prefix host tier, ~147 GiB) rides TTM's shared
|
||||
# pages, capped by ttm's pages_limit (the current kernel exposes NO
|
||||
# pages_max; the cap function takes the min over the knobs that exist).
|
||||
# Below FRANK_ARCHON_TTM_MIN_MIB the load dies mid-pin with an allocator
|
||||
# error, so the gate refuses first — and RAISES the knob before refusing:
|
||||
# the cap is a runtime knob whose default is half of RAM, so a power failure
|
||||
# resets it (measured 2026-10-12 23:17 on this host: 200 GiB -> 125,988 MiB,
|
||||
# and Restart=always re-ran the refusal 6,919 times over ~20 h).
|
||||
# FRANK_ARCHON_TTM_TARGET_MIB is the recorded host ceiling, not the floor.
|
||||
# shellcheck source=/dev/null
|
||||
source "$SCRIPT_DIR/ttm-preflight.sh"
|
||||
ttm_ensure_cap || exit 1
|
||||
|
||||
# A crashed/killed unit can leave the container name registered (systemd
|
||||
# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
|
||||
# identities (the FreeToken-era frank-archon-qwen, the retired R9V
|
||||
# container and the ad hoc experiment containers all hold this GPU); all
|
||||
# idempotent.
|
||||
for CTR in $FRANK_ARCHON_LEGACY_CONTAINERS "$FRANK_ARCHON_CONTAINER"; do
|
||||
podman rm -f "$CTR" >/dev/null 2>&1 || true
|
||||
done
|
||||
|
||||
LOG=/home/dank/logs/frank_archon.log
|
||||
mkdir -p "$(dirname "$LOG")"
|
||||
|
||||
# Publisher-vetted container contract (docs/DOCKER.md, compose/common.yaml):
|
||||
# --network host (the server binds the host's addresses directly, no NAT),
|
||||
# --init (the engine releases the cards on SIGTERM), seccomp=unconfined
|
||||
# (Docker/podman's default profile refuses io_uring, which the n-gram
|
||||
# table's reads go through, and the NUMA binding calls the ROCm runtime
|
||||
# makes for host memory), memlock unbounded (the pinned ledger — legal
|
||||
# because the root unit grants LimitMEMLOCK=infinity to this dank process),
|
||||
# /dev/kfd + /dev/dri (render/video groups are moot for a root container).
|
||||
# The template rides a single-file mount; the .rad dir is ro; /data holds
|
||||
# the prefix-cache disk tier.
|
||||
podman run -d --name "$FRANK_ARCHON_CONTAINER" \
|
||||
--network host \
|
||||
--init \
|
||||
--security-opt seccomp=unconfined \
|
||||
--ulimit memlock=-1:-1 \
|
||||
--device /dev/kfd --device /dev/dri \
|
||||
-v "$FRANK_ARCHON_MODEL_DIR":/models:ro \
|
||||
-v "$FRANK_ARCHON_STATE_DIR":/data \
|
||||
-v "$FRANK_ARCHON_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
|
||||
-v "$FRANK_ARCHON_TEMPLATE":/templates/template.jinja:ro \
|
||||
"$FRANK_ARCHON_IMAGE" \
|
||||
--model "/models/$FRANK_ARCHON_MODEL_FILE" \
|
||||
--host :: --port "$FRANK_ARCHON_PORT" \
|
||||
--served-model-name "$FRANK_ARCHON_SERVED_NAME" \
|
||||
--override-chat-template /templates/template.jinja \
|
||||
--tp "$FRANK_ARCHON_TP" --tp-wire "$FRANK_ARCHON_TP_WIRE" \
|
||||
--max-num-seqs "$FRANK_ARCHON_MAX_SEQS" \
|
||||
--max-model-len "$FRANK_ARCHON_MAX_MODEL_LEN" \
|
||||
--placement "$FRANK_ARCHON_PLACEMENT" \
|
||||
--host-pool-mib "$FRANK_ARCHON_HOST_POOL_MIB" \
|
||||
--gpu-headroom-mib "$FRANK_ARCHON_GPU_HEADROOM_MIB" \
|
||||
--vram-weights-mib "$FRANK_ARCHON_VRAM_WEIGHTS_MIB" \
|
||||
--vram-kv-mib "$FRANK_ARCHON_VRAM_KV_MIB" \
|
||||
--kv-cache-dtype "$FRANK_ARCHON_KV_DTYPE" \
|
||||
--prefix-cache-host-mib "$FRANK_ARCHON_PREFIX_HOST_MIB" \
|
||||
--prefix-cache-dir /data/kvcache/frank-archon \
|
||||
--prefix-cache-disk-mib "$FRANK_ARCHON_PREFIX_DISK_MIB" \
|
||||
--ngram-placement "$FRANK_ARCHON_NGRAM_PLACEMENT" \
|
||||
--num-speculative-tokens "$FRANK_ARCHON_SPEC_TOKENS" \
|
||||
--max-num-batched-tokens "$FRANK_ARCHON_MAX_BATCHED_TOKENS" \
|
||||
${FRANK_ARCHON_EXTRA_ARGS:+$FRANK_ARCHON_EXTRA_ARGS} \
|
||||
> "$LOG" 2>&1
|
||||
|
||||
# From here on, this script's lifetime IS the container's: wait returns the
|
||||
# container's exit code, so a crash is a nonzero ExecStart and the unit's
|
||||
# Restart=always is real. (systemd stop goes through ExecStop in the unit —
|
||||
# podman wait alone cannot stop the container child.)
|
||||
exec podman wait "$FRANK_ARCHON_CONTAINER"
|
||||
@@ -1,13 +1,13 @@
|
||||
#!/usr/bin/env bash
|
||||
# serve-frank-large.sh — frank-large endpoint: halogen-flash-server 0.16.2
|
||||
# serve-frank-dragoon.sh — frank-dragoon endpoint: halogen-flash-server 0.16.2
|
||||
# (closed-source engine, image ghcr.io/peonist-ai/halogen-flash-server,
|
||||
# digest-pinned in frank-large.env) serving the Quat3rnion abliterated
|
||||
# digest-pinned in frank-dragoon.env) serving the Quat3rnion abliterated
|
||||
# Swift1.5 finetune of peonist's Qwen3.8-Flash-Next v2 checkpoint on
|
||||
# Dankclaw's Strix Halo iGPU, host :7770. 2026-10-06 cutover from the
|
||||
# halogen 27B-dense shape — revert the swap commit's trio to roll back
|
||||
# (the 27B payload stays on disk).
|
||||
#
|
||||
# The engine is driven by `podman run` DIRECTLY (the fleet's frank-primary
|
||||
# The engine is driven by `podman run` DIRECTLY (the fleet's frank-archon
|
||||
# pattern): the image starts engine + OpenAI frontend in one container.
|
||||
# The frontend listens on 8731 inside (HALOGEN_API_PORT default) and is
|
||||
# published as host 7770; the engine's own token port (8730, no auth)
|
||||
@@ -26,13 +26,13 @@
|
||||
# Why containers instead of a bare binary: the payload is .hgn and only
|
||||
# the flash-server line reads it; it ships as a container; the root unit
|
||||
# keeps LimitMEMLOCK=infinity for parity and to outlive the login session,
|
||||
# exactly like frank-primary's.
|
||||
# exactly like frank-archon's.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
|
||||
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-dragoon.env}"
|
||||
if [ ! -f "$ENV_FILE" ]; then
|
||||
echo "frank-large: env file not found: $ENV_FILE" >&2
|
||||
echo "frank-dragoon: env file not found: $ENV_FILE" >&2
|
||||
exit 1
|
||||
fi
|
||||
set -a
|
||||
@@ -40,17 +40,17 @@ set -a
|
||||
source "$ENV_FILE"
|
||||
set +a
|
||||
|
||||
: "${FRANK_LARGE_IMAGE:?FRANK_LARGE_IMAGE must name the digest-pinned image}"
|
||||
: "${FRANK_LARGE_CONTAINER:?FRANK_LARGE_CONTAINER must name the container}"
|
||||
: "${FRANK_LARGE_PUBLISH:?FRANK_LARGE_PUBLISH must be host:container}"
|
||||
: "${FRANK_LARGE_MODEL_DIR:?FRANK_LARGE_MODEL_DIR must name the payload dir}"
|
||||
: "${FRANK_LARGE_CHECKPOINT:?FRANK_LARGE_CHECKPOINT must name the .hgn inside it}"
|
||||
: "${FRANK_LARGE_CHECKPOINT_BYTES:?FRANK_LARGE_CHECKPOINT_BYTES must pin the size}"
|
||||
: "${FRANK_LARGE_NGRAM_TABLE:?FRANK_LARGE_NGRAM_TABLE must name the lookup table}"
|
||||
: "${FRANK_LARGE_NGRAM_BYTES:?FRANK_LARGE_NGRAM_BYTES must pin the table size}"
|
||||
: "${FRANK_LARGE_VISION_TOWER:?FRANK_LARGE_VISION_TOWER must name the tower}"
|
||||
: "${FRANK_LARGE_VISION_BYTES:?FRANK_LARGE_VISION_BYTES must pin the tower size}"
|
||||
: "${FRANK_LARGE_TOKENIZER_FILES:?FRANK_LARGE_TOKENIZER_FILES must list the tokenizer dir contents}"
|
||||
: "${FRANK_DRAGOON_IMAGE:?FRANK_DRAGOON_IMAGE must name the digest-pinned image}"
|
||||
: "${FRANK_DRAGOON_CONTAINER:?FRANK_DRAGOON_CONTAINER must name the container}"
|
||||
: "${FRANK_DRAGOON_PUBLISH:?FRANK_DRAGOON_PUBLISH must be host:container}"
|
||||
: "${FRANK_DRAGOON_MODEL_DIR:?FRANK_DRAGOON_MODEL_DIR must name the payload dir}"
|
||||
: "${FRANK_DRAGOON_CHECKPOINT:?FRANK_DRAGOON_CHECKPOINT must name the .hgn inside it}"
|
||||
: "${FRANK_DRAGOON_CHECKPOINT_BYTES:?FRANK_DRAGOON_CHECKPOINT_BYTES must pin the size}"
|
||||
: "${FRANK_DRAGOON_NGRAM_TABLE:?FRANK_DRAGOON_NGRAM_TABLE must name the lookup table}"
|
||||
: "${FRANK_DRAGOON_NGRAM_BYTES:?FRANK_DRAGOON_NGRAM_BYTES must pin the table size}"
|
||||
: "${FRANK_DRAGOON_VISION_TOWER:?FRANK_DRAGOON_VISION_TOWER must name the tower}"
|
||||
: "${FRANK_DRAGOON_VISION_BYTES:?FRANK_DRAGOON_VISION_BYTES must pin the tower size}"
|
||||
: "${FRANK_DRAGOON_TOKENIZER_FILES:?FRANK_DRAGOON_TOKENIZER_FILES must list the tokenizer dir contents}"
|
||||
: "${HALOGEN_MODEL_ID:?HALOGEN_MODEL_ID must be the served alias (frank-dragoon)}"
|
||||
: "${HALOGEN_CTX:?HALOGEN_CTX must be set (per-slot window)}"
|
||||
: "${HALOGEN_KV_POOL_POSITIONS:?HALOGEN_KV_POOL_POSITIONS must be set}"
|
||||
@@ -62,57 +62,59 @@ set +a
|
||||
# podman dies SIGPIPE 141, pipefail turns the 141 into "no match" — the
|
||||
# 2026-09-30 deploy refusal).
|
||||
DIGESTS="$(podman images --format '{{.RepoDigests}}')"
|
||||
if ! grep -qF "${FRANK_LARGE_IMAGE#*@}" <<<"$DIGESTS"; then
|
||||
echo "frank-large: image $FRANK_LARGE_IMAGE not in podman storage;" >&2
|
||||
echo " pull it: podman pull ${FRANK_LARGE_IMAGE%@*}" >&2
|
||||
if ! grep -qF "${FRANK_DRAGOON_IMAGE#*@}" <<<"$DIGESTS"; then
|
||||
echo "frank-dragoon: image $FRANK_DRAGOON_IMAGE not in podman storage;" >&2
|
||||
echo " pull it: podman pull ${FRANK_DRAGOON_IMAGE%@*}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Size-exact gates on all three payload files: a truncated curl exits 0 on
|
||||
# some failures and a partial .hgn loads far enough to look alive, so the
|
||||
# byte counts from the Hub tree API are the contract (fleet fetch law).
|
||||
# byte counts from the Hub tree API are the contract: curl exits 0 on an
|
||||
# HTTP error and writes the error body as the file, so a truncated download
|
||||
# is caught by size here and never boots.
|
||||
ck() {
|
||||
# ck <env-name> <path> <expected-bytes>
|
||||
if [ ! -f "$2" ]; then
|
||||
echo "frank-large: $1 $2 missing;" >&2
|
||||
echo " fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
|
||||
echo "frank-dragoon: $1 $2 missing;" >&2
|
||||
echo " fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||
exit 1
|
||||
fi
|
||||
local sz
|
||||
sz="$(stat -c %s "$2")"
|
||||
if [ "$sz" != "$3" ]; then
|
||||
echo "frank-large: $1 is $sz bytes, expected $3 (partial download?)" >&2
|
||||
echo "frank-dragoon: $1 is $sz bytes, expected $3 (partial download?)" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
ck checkpoint "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_CHECKPOINT" "$FRANK_LARGE_CHECKPOINT_BYTES"
|
||||
ck n-gram-table "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_NGRAM_TABLE" "$FRANK_LARGE_NGRAM_BYTES"
|
||||
ck vision-tower "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_VISION_TOWER" "$FRANK_LARGE_VISION_BYTES"
|
||||
ck checkpoint "$FRANK_DRAGOON_MODEL_DIR/$FRANK_DRAGOON_CHECKPOINT" "$FRANK_DRAGOON_CHECKPOINT_BYTES"
|
||||
ck n-gram-table "$FRANK_DRAGOON_MODEL_DIR/$FRANK_DRAGOON_NGRAM_TABLE" "$FRANK_DRAGOON_NGRAM_BYTES"
|
||||
ck vision-tower "$FRANK_DRAGOON_MODEL_DIR/$FRANK_DRAGOON_VISION_TOWER" "$FRANK_DRAGOON_VISION_BYTES"
|
||||
|
||||
# The payload ships the tokenizer as a FLAT DIRECTORY. A HuggingFace cache
|
||||
# snapshot symlinks each file into a sibling blobs/, and those symlinks
|
||||
# dangle inside the container mount — so the directory must be real and
|
||||
# every named file must be a regular file, not a link.
|
||||
TOK_DIR="$FRANK_LARGE_MODEL_DIR/tokenizer"
|
||||
TOK_DIR="$FRANK_DRAGOON_MODEL_DIR/tokenizer"
|
||||
if [ -L "$TOK_DIR" ] || [ ! -d "$TOK_DIR" ]; then
|
||||
echo "frank-large: tokenizer dir $TOK_DIR missing or a symlink (HF cache snapshot?);" >&2
|
||||
echo " fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
|
||||
echo "frank-dragoon: tokenizer dir $TOK_DIR missing or a symlink (HF cache snapshot?);" >&2
|
||||
echo " fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||
exit 1
|
||||
fi
|
||||
for TF in $FRANK_LARGE_TOKENIZER_FILES; do
|
||||
for TF in $FRANK_DRAGOON_TOKENIZER_FILES; do
|
||||
if [ ! -f "$TOK_DIR/$TF" ] || [ -L "$TOK_DIR/$TF" ]; then
|
||||
echo "frank-large: tokenizer file $TOK_DIR/$TF missing or a symlink;" >&2
|
||||
echo " fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
|
||||
echo "frank-dragoon: tokenizer file $TOK_DIR/$TF missing or a symlink;" >&2
|
||||
echo " fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
# GPU access is mandatory but the node paths are variables so the seam can
|
||||
# drive the gate on a box without an AMD GPU.
|
||||
DEV_KFD="${FRANK_LARGE_DEV_KFD:-/dev/kfd}"
|
||||
DEV_DRI="${FRANK_LARGE_DEV_DRI:-/dev/dri}"
|
||||
DEV_KFD="${FRANK_DRAGOON_DEV_KFD:-/dev/kfd}"
|
||||
DEV_DRI="${FRANK_DRAGOON_DEV_DRI:-/dev/dri}"
|
||||
if [ ! -e "$DEV_KFD" ] || [ ! -d "$DEV_DRI" ]; then
|
||||
echo "frank-large: $DEV_KFD or $DEV_DRI missing — no gfx1151 to serve on" >&2
|
||||
echo "frank-dragoon: $DEV_KFD or $DEV_DRI missing — no gfx1151 to serve on" >&2
|
||||
exit 1
|
||||
fi
|
||||
# Host-RAM preflight — the engine's pin gate, checked BEFORE podman run.
|
||||
@@ -125,15 +127,15 @@ fi
|
||||
# self-resolved in ~2 min when the dying 27B engine freed RAM. Check here and
|
||||
# WAIT for the RAM to clear, so a transient dip boots once instead of
|
||||
# crash-looping; if it never clears, refuse loudly.
|
||||
MEMINFO="${FRANK_LARGE_MEMINFO:-/proc/meminfo}"
|
||||
PIN_KB=$(( FRANK_LARGE_CHECKPOINT_BYTES / 1024 ))
|
||||
MEMINFO="${FRANK_DRAGOON_MEMINFO:-/proc/meminfo}"
|
||||
PIN_KB=$(( FRANK_DRAGOON_CHECKPOINT_BYTES / 1024 ))
|
||||
NEED_KB=$(( PIN_KB + 16 * 1048576 ))
|
||||
ram_avail_kb() { awk '/^MemAvailable:/{print $2; exit}' "$MEMINFO"; }
|
||||
MEMAVAIL_KB="$(ram_avail_kb)" || MEMAVAIL_KB=""
|
||||
if [ -n "$MEMAVAIL_KB" ] && [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
|
||||
WAIT_S="${FRANK_LARGE_RAM_WAIT:-600}"
|
||||
WAIT_S="${FRANK_DRAGOON_RAM_WAIT:-600}"
|
||||
GIB() { awk -v kb="$1" 'BEGIN{printf "%.2f", kb/1048576}'; }
|
||||
echo "frank-large: host RAM below the pin gate — MemAvailable $(GIB "$MEMAVAIL_KB") GiB < needed $(GIB "$NEED_KB") GiB ($(GIB "$PIN_KB") GiB checkpoint + 16.00 GiB floor); waiting up to ${WAIT_S}s for host RAM to clear" >&2
|
||||
echo "frank-dragoon: host RAM below the pin gate — MemAvailable $(GIB "$MEMAVAIL_KB") GiB < needed $(GIB "$NEED_KB") GiB ($(GIB "$PIN_KB") GiB checkpoint + 16.00 GiB floor); waiting up to ${WAIT_S}s for host RAM to clear" >&2
|
||||
deadline=$(( $(date +%s) + WAIT_S ))
|
||||
while [ "$(date +%s)" -lt "$deadline" ]; do
|
||||
sleep 5
|
||||
@@ -143,16 +145,16 @@ if [ -n "$MEMAVAIL_KB" ] && [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
|
||||
fi
|
||||
done
|
||||
if [ -z "$MEMAVAIL_KB" ] || [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
|
||||
echo "frank-large: MemAvailable $(GIB "${MEMAVAIL_KB:-0}") GiB after ${WAIT_S}s; refusing to start (the engine would refuse the pin)" >&2
|
||||
echo "frank-dragoon: MemAvailable $(GIB "${MEMAVAIL_KB:-0}") GiB after ${WAIT_S}s; refusing to start (the engine would refuse the pin)" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "frank-large: MemAvailable cleared to $(GIB "$MEMAVAIL_KB") GiB; starting" >&2
|
||||
echo "frank-dragoon: MemAvailable cleared to $(GIB "$MEMAVAIL_KB") GiB; starting" >&2
|
||||
fi
|
||||
for CTR in ${FRANK_LARGE_LEGACY_CONTAINERS:-} "$FRANK_LARGE_CONTAINER"; do
|
||||
for CTR in ${FRANK_DRAGOON_LEGACY_CONTAINERS:-} "$FRANK_DRAGOON_CONTAINER"; do
|
||||
podman rm -f "$CTR" >/dev/null 2>&1 || true
|
||||
done
|
||||
|
||||
LOG="${FRANK_LARGE_LOG:-/home/dank/logs/frank_large.log}"
|
||||
LOG="${FRANK_DRAGOON_LOG:-/home/dank/logs/frank_dragoon.log}"
|
||||
mkdir -p "$(dirname "$LOG")"
|
||||
# --ipc=host for the udev-granted supplementary group, seccomp unconfined
|
||||
# for the ROCm ioctl surface, and unbounded memlock (the unit's
|
||||
@@ -163,17 +165,17 @@ mkdir -p "$(dirname "$LOG")"
|
||||
# and both sibling serve scripts pass no nofile either. No comment lines
|
||||
# inside the backslash chain — a mid-chain # terminates the continuation
|
||||
# (the 2026-10-05 exit-127 crash loop on the sibling serve script).
|
||||
podman run --name "$FRANK_LARGE_CONTAINER" \
|
||||
--publish "$FRANK_LARGE_PUBLISH" \
|
||||
podman run --name "$FRANK_DRAGOON_CONTAINER" \
|
||||
--publish "$FRANK_DRAGOON_PUBLISH" \
|
||||
--ipc=host \
|
||||
--ulimit memlock=-1:-1 \
|
||||
--device "$DEV_KFD" --device "$DEV_DRI" \
|
||||
--group-add keep-groups \
|
||||
--security-opt seccomp=unconfined \
|
||||
-e HALOGEN_MODEL_ID="$HALOGEN_MODEL_ID" \
|
||||
-e HALOGEN_CHECKPOINT="/models/$FRANK_LARGE_CHECKPOINT" \
|
||||
-e HALOGEN_NGRAM_TABLE="/models/$FRANK_LARGE_NGRAM_TABLE" \
|
||||
-e HALOGEN_VISION_TOWER="/models/$FRANK_LARGE_VISION_TOWER" \
|
||||
-e HALOGEN_CHECKPOINT="/models/$FRANK_DRAGOON_CHECKPOINT" \
|
||||
-e HALOGEN_NGRAM_TABLE="/models/$FRANK_DRAGOON_NGRAM_TABLE" \
|
||||
-e HALOGEN_VISION_TOWER="/models/$FRANK_DRAGOON_VISION_TOWER" \
|
||||
-e HALOGEN_TOKENIZER=/tokenizer \
|
||||
-e HALOGEN_CTX="$HALOGEN_CTX" \
|
||||
-e HALOGEN_KV_POOL_POSITIONS="$HALOGEN_KV_POOL_POSITIONS" \
|
||||
@@ -187,9 +189,9 @@ podman run --name "$FRANK_LARGE_CONTAINER" \
|
||||
-e HALOGEN_TOP_P="${HALOGEN_TOP_P:-0.95}" \
|
||||
-e HALOGEN_TOP_K="${HALOGEN_TOP_K:-20}" \
|
||||
${HALOGEN_WEIGHTS_LOCK:+-e HALOGEN_WEIGHTS_LOCK="$HALOGEN_WEIGHTS_LOCK"} \
|
||||
-v "$FRANK_LARGE_MODEL_DIR:/models:ro" \
|
||||
-v "$FRANK_LARGE_MODEL_DIR/tokenizer:/tokenizer:ro" \
|
||||
"$FRANK_LARGE_IMAGE" \
|
||||
-v "$FRANK_DRAGOON_MODEL_DIR:/models:ro" \
|
||||
-v "$FRANK_DRAGOON_MODEL_DIR/tokenizer:/tokenizer:ro" \
|
||||
"$FRANK_DRAGOON_IMAGE" \
|
||||
> "$LOG" 2>&1
|
||||
|
||||
# FOREGROUND `podman run`: the engine's stdout/stderr are this script's
|
||||
@@ -1,206 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# serve-frank-primary.sh — frank-primary endpoint: radiance (StillDeadcode,
|
||||
# Apache-2.0) serving the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at
|
||||
# NATIVE block-FP8 precision from a rad-convert container
|
||||
# (/home/dank/models/rad/qwen4exp-fp8-native.rad), on BOTH R9700s (TP=2,
|
||||
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (PR #587,
|
||||
# one card; git tag frank-primary-freetoken-pr587-20261005 is the
|
||||
# pre-cutover state).
|
||||
#
|
||||
# The run line below IS the engine argv: the image's entrypoint is the
|
||||
# engine and its tools sit beside it on PATH. Unlike every predecessor
|
||||
# there is no per-boot VRAM planner to second-guess: the boot ledger in the
|
||||
# log states every budget before anything is allocated (weights vs KV
|
||||
# split, the expert host pool's coverage of the worst case, the PLE pin,
|
||||
# the P2P answer per pair) — read it after every resize. First-boot shape:
|
||||
# 2 decode slots (Frank's call), 40 GiB/rank pinned expert pool, fp8 KV,
|
||||
# MTP depth 3, prefix cache host 4 GiB + disk 64 GiB, PLE pinned in RAM.
|
||||
#
|
||||
# The root unit (scripts/frank-primary.service, User=dank,
|
||||
# LimitMEMLOCK=infinity) grants the pin budget: 2 x 40 GiB expert pools +
|
||||
# the 47.7 GiB PLE table + 4 GiB prefix host tier — ~132 GiB of pinned
|
||||
# pages, which is ALSO why this script refuses to start when ttm's
|
||||
# pages_limit/pages_max cap is under FRANK_PRIMARY_TTM_MIN_MIB (see below).
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_PRIMARY_ENV_FILE:-$SCRIPT_DIR/frank-primary.env}"
|
||||
|
||||
if [ ! -f "$ENV_FILE" ]; then
|
||||
echo "ERROR: frank-primary env file absent: $ENV_FILE" >&2
|
||||
exit 1
|
||||
fi
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
source "$ENV_FILE"
|
||||
set +a
|
||||
|
||||
: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the radiance image}"
|
||||
: "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the .rad dir}"
|
||||
: "${FRANK_PRIMARY_MODEL_FILE:?FRANK_PRIMARY_MODEL_FILE must name the .rad file}"
|
||||
: "${FRANK_PRIMARY_STATE_DIR:?FRANK_PRIMARY_STATE_DIR must name the /data state dir}"
|
||||
: "${FRANK_PRIMARY_CONTAINER:?FRANK_PRIMARY_CONTAINER must name the container}"
|
||||
: "${FRANK_PRIMARY_SERVED_NAME:?FRANK_PRIMARY_SERVED_NAME must name the alias}"
|
||||
: "${FRANK_PRIMARY_PORT:?FRANK_PRIMARY_PORT must be the port}"
|
||||
: "${FRANK_PRIMARY_TEMPLATE:?FRANK_PRIMARY_TEMPLATE must name the chat template}"
|
||||
: "${FRANK_PRIMARY_TP:?FRANK_PRIMARY_TP must be set}"
|
||||
: "${FRANK_PRIMARY_TP_WIRE:?FRANK_PRIMARY_TP_WIRE must be set}"
|
||||
: "${FRANK_PRIMARY_MAX_SEQS:?FRANK_PRIMARY_MAX_SEQS must be set}"
|
||||
: "${FRANK_PRIMARY_MAX_MODEL_LEN:?FRANK_PRIMARY_MAX_MODEL_LEN must be set}"
|
||||
: "${FRANK_PRIMARY_PLACEMENT:?FRANK_PRIMARY_PLACEMENT must be set}"
|
||||
: "${FRANK_PRIMARY_HOST_POOL_MIB:?FRANK_PRIMARY_HOST_POOL_MIB must be set}"
|
||||
: "${FRANK_PRIMARY_GPU_HEADROOM_MIB:?FRANK_PRIMARY_GPU_HEADROOM_MIB must be set}"
|
||||
: "${FRANK_PRIMARY_VRAM_WEIGHTS_MIB:?FRANK_PRIMARY_VRAM_WEIGHTS_MIB must be set}"
|
||||
: "${FRANK_PRIMARY_VRAM_KV_MIB:?FRANK_PRIMARY_VRAM_KV_MIB must be set}"
|
||||
: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set}"
|
||||
: "${FRANK_PRIMARY_SPEC_TOKENS:?FRANK_PRIMARY_SPEC_TOKENS must be set}"
|
||||
: "${FRANK_PRIMARY_MAX_BATCHED_TOKENS:?FRANK_PRIMARY_MAX_BATCHED_TOKENS must be set}"
|
||||
: "${FRANK_PRIMARY_PREFIX_HOST_MIB:?FRANK_PRIMARY_PREFIX_HOST_MIB must be set}"
|
||||
: "${FRANK_PRIMARY_PREFIX_DISK_MIB:?FRANK_PRIMARY_PREFIX_DISK_MIB must be set}"
|
||||
: "${FRANK_PRIMARY_NGRAM_PLACEMENT:?FRANK_PRIMARY_NGRAM_PLACEMENT must be set}"
|
||||
: "${FRANK_PRIMARY_TTM_MIN_MIB:?FRANK_PRIMARY_TTM_MIN_MIB must be set}"
|
||||
: "${FRANK_PRIMARY_RAD_MIN_MIB:?FRANK_PRIMARY_RAD_MIN_MIB must be set}"
|
||||
|
||||
# The image gate is the digest-pinned ref itself: a name@sha256 ref exists
|
||||
# in storage only if THAT manifest was pulled — a retagged look-alike or a
|
||||
# newer latest does not satisfy it. `podman image exists` takes a
|
||||
# name@digest ref in one command — no pipefail/grep race (the 2026-09-30
|
||||
# lesson). Runs as dank: the unit's process uses dank's ROOTLESS podman
|
||||
# storage (root unit, User=dank — the fleet's frank-primary shape).
|
||||
if ! podman image exists "$FRANK_PRIMARY_IMAGE"; then
|
||||
echo "ERROR: radiance image absent: $FRANK_PRIMARY_IMAGE" >&2
|
||||
echo "frank-primary: podman pull docker://stilldeadcode/radiance:latest (as dank)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# The payload gate: the .rad container file, sized. A truncated convert
|
||||
# would otherwise buy a guaranteed long-timeout boot instead of this fast
|
||||
# failure. The staging dir / raw checkpoint are NOT this script's concern
|
||||
# (scripts/build-frank-primary-rad.sh built the file; its rad-info output
|
||||
# is the record).
|
||||
RAD="$FRANK_PRIMARY_MODEL_DIR/$FRANK_PRIMARY_MODEL_FILE"
|
||||
if [ ! -f "$RAD" ]; then
|
||||
echo "ERROR: .rad container missing: $RAD" >&2
|
||||
echo "frank-primary: run scripts/build-frank-primary-rad.sh" >&2
|
||||
exit 1
|
||||
fi
|
||||
RAD_MIB=$(( $(stat -c %s "$RAD") / 1048576 ))
|
||||
if [ "$RAD_MIB" -lt "$FRANK_PRIMARY_RAD_MIN_MIB" ]; then
|
||||
echo "ERROR: .rad container too small: $RAD is ${RAD_MIB} MiB, want >= $FRANK_PRIMARY_RAD_MIN_MIB" >&2
|
||||
echo "frank-primary: the convert was truncated — rebuild it" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ ! -f "$FRANK_PRIMARY_TEMPLATE" ]; then
|
||||
echo "ERROR: chat template missing: $FRANK_PRIMARY_TEMPLATE" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mkdir -p "$FRANK_PRIMARY_STATE_DIR/kvcache/frank-primary"
|
||||
# The engine's create is NOT recursive: with only the state dir mounted, its
|
||||
# "cannot create /data/kvcache/frank-primary: No such file or directory"
|
||||
# left the SSD prefix tier off (measured 2026-10-06 boot ledger). Pre-create
|
||||
# the full path on the host so the tier opens on first boot.
|
||||
|
||||
mkdir -p "$FRANK_PRIMARY_STATE_DIR/tune"
|
||||
# rad-tune's kernel-selection cache (see the runbook's "Tuning levers"):
|
||||
# the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune at boot;
|
||||
# $RADIANCE_HOME defaults to /opt/radiance, which lives in the image, so
|
||||
# the cache must ride a host mount or it dies with the --rm container
|
||||
# that wrote it. The dir is pre-created for the same reason as kvcache
|
||||
# (the engine's create is not recursive).
|
||||
|
||||
if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
|
||||
echo "ERROR: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# TTM preflight: the pinned ledger (~2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
|
||||
# prefix host = ~132 GiB) rides TTM's shared pages, capped by ttm's
|
||||
# pages_limit (the current kernel exposes NO pages_max; the cap function
|
||||
# takes the min over the knobs that exist). Below FRANK_PRIMARY_TTM_MIN_MIB
|
||||
# the load dies mid-pin with an allocator error instead of this named
|
||||
# refusal — raise the knob first (runtime sysfs write; kernel cmdline
|
||||
# fallback + one reboot only if the file is read-only), see the runbook.
|
||||
# RAISED 2026-10-05 pre-cutover to 52,428,800 pages = 200 GiB.
|
||||
ttm_cap_mib() {
|
||||
# min over the knobs THIS kernel exposes — the current kernel has
|
||||
# pages_limit only (no pages_max); a knob that is absent is skipped,
|
||||
# not a zero (a zero would refuse a healthy cap).
|
||||
local v cur=""
|
||||
for p in /sys/module/ttm/parameters/pages_limit /sys/module/ttm/parameters/pages_max; do
|
||||
[ -r "$p" ] || continue
|
||||
v="$(cat "$p" 2>/dev/null)" || continue
|
||||
[ -n "$v" ] || continue
|
||||
v=$(( v * 4 / 1024 ))
|
||||
if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
|
||||
done
|
||||
echo "${cur:-0}"
|
||||
}
|
||||
TTM_CAP="$(ttm_cap_mib)"
|
||||
if [ "$TTM_CAP" -lt "$FRANK_PRIMARY_TTM_MIN_MIB" ]; then
|
||||
echo "ERROR: ttm pages cap is ${TTM_CAP} MiB, the pinned ledger needs >= $FRANK_PRIMARY_TTM_MIN_MIB" >&2
|
||||
echo "frank-primary: raise /sys/module/ttm/parameters/pages_limit (and pages_max if the kernel exposes one) to >= $FRANK_PRIMARY_TTM_MIN_MIB in 4 KiB pages" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# A crashed/killed unit can leave the container name registered (systemd
|
||||
# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
|
||||
# identities (the FreeToken-era frank-primary-qwen, the retired R9V
|
||||
# container and the ad hoc experiment containers all hold this GPU); all
|
||||
# idempotent.
|
||||
for CTR in $FRANK_PRIMARY_LEGACY_CONTAINERS "$FRANK_PRIMARY_CONTAINER"; do
|
||||
podman rm -f "$CTR" >/dev/null 2>&1 || true
|
||||
done
|
||||
|
||||
LOG=/home/dank/logs/frank_primary.log
|
||||
mkdir -p "$(dirname "$LOG")"
|
||||
|
||||
# Publisher-vetted container contract (docs/DOCKER.md, compose/common.yaml):
|
||||
# --network host (the server binds the host's addresses directly, no NAT),
|
||||
# --init (the engine releases the cards on SIGTERM), seccomp=unconfined
|
||||
# (Docker/podman's default profile refuses io_uring, which the n-gram
|
||||
# table's reads go through, and the NUMA binding calls the ROCm runtime
|
||||
# makes for host memory), memlock unbounded (the pinned ledger — legal
|
||||
# because the root unit grants LimitMEMLOCK=infinity to this dank process),
|
||||
# /dev/kfd + /dev/dri (render/video groups are moot for a root container).
|
||||
# The template rides a single-file mount; the .rad dir is ro; /data holds
|
||||
# the prefix-cache disk tier.
|
||||
podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
||||
--network host \
|
||||
--init \
|
||||
--security-opt seccomp=unconfined \
|
||||
--ulimit memlock=-1:-1 \
|
||||
--device /dev/kfd --device /dev/dri \
|
||||
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
|
||||
-v "$FRANK_PRIMARY_STATE_DIR":/data \
|
||||
-v "$FRANK_PRIMARY_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
|
||||
-v "$FRANK_PRIMARY_TEMPLATE":/templates/template.jinja:ro \
|
||||
"$FRANK_PRIMARY_IMAGE" \
|
||||
--model "/models/$FRANK_PRIMARY_MODEL_FILE" \
|
||||
--host :: --port "$FRANK_PRIMARY_PORT" \
|
||||
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" \
|
||||
--override-chat-template /templates/template.jinja \
|
||||
--tp "$FRANK_PRIMARY_TP" --tp-wire "$FRANK_PRIMARY_TP_WIRE" \
|
||||
--max-num-seqs "$FRANK_PRIMARY_MAX_SEQS" \
|
||||
--max-model-len "$FRANK_PRIMARY_MAX_MODEL_LEN" \
|
||||
--placement "$FRANK_PRIMARY_PLACEMENT" \
|
||||
--host-pool-mib "$FRANK_PRIMARY_HOST_POOL_MIB" \
|
||||
--gpu-headroom-mib "$FRANK_PRIMARY_GPU_HEADROOM_MIB" \
|
||||
--vram-weights-mib "$FRANK_PRIMARY_VRAM_WEIGHTS_MIB" \
|
||||
--vram-kv-mib "$FRANK_PRIMARY_VRAM_KV_MIB" \
|
||||
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
|
||||
--prefix-cache-host-mib "$FRANK_PRIMARY_PREFIX_HOST_MIB" \
|
||||
--prefix-cache-dir /data/kvcache/frank-primary \
|
||||
--prefix-cache-disk-mib "$FRANK_PRIMARY_PREFIX_DISK_MIB" \
|
||||
--ngram-placement "$FRANK_PRIMARY_NGRAM_PLACEMENT" \
|
||||
--num-speculative-tokens "$FRANK_PRIMARY_SPEC_TOKENS" \
|
||||
--max-num-batched-tokens "$FRANK_PRIMARY_MAX_BATCHED_TOKENS" \
|
||||
${FRANK_PRIMARY_EXTRA_ARGS:+$FRANK_PRIMARY_EXTRA_ARGS} \
|
||||
> "$LOG" 2>&1
|
||||
|
||||
# From here on, this script's lifetime IS the container's: wait returns the
|
||||
# container's exit code, so a crash is a nonzero ExecStart and the unit's
|
||||
# Restart=always is real. (systemd stop goes through ExecStop in the unit —
|
||||
# podman wait alone cannot stop the container child.)
|
||||
exec podman wait "$FRANK_PRIMARY_CONTAINER"
|
||||
@@ -1,6 +1,6 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive (Q6_K) as
|
||||
# frank-secondary on Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
|
||||
# frank-zealot on Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
|
||||
#
|
||||
# 2026-10-07 (Frank): Qwen3.5-9B swap at TWO 200,704-token conversations
|
||||
# (--ctx-size 401408 --parallel 2) with EXACT f16 KV — the 9B rung frees the
|
||||
@@ -24,11 +24,11 @@
|
||||
# Sampling: llama.cpp takes it per request (the fleet harness sends the
|
||||
# publisher recipe from models.yml: thinking temp 0.6 / top_p 0.95 / top_k 20).
|
||||
#
|
||||
# Tunables come from frank-secondary.env unless FRANK_SECONDARY_ENV_FILE overrides.
|
||||
# Tunables come from frank-zealot.env unless FRANK_ZEALOT_ENV_FILE overrides.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_SECONDARY_ENV_FILE:-$SCRIPT_DIR/frank-secondary.env}"
|
||||
ENV_FILE="${FRANK_ZEALOT_ENV_FILE:-$SCRIPT_DIR/frank-zealot.env}"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
@@ -63,8 +63,9 @@ fi
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
# Speculative decoding is OFF: this Q6_K file carries NO nextn/mtp tensors, so
|
||||
# there is no in-file head (the law: MTP is a property of the file, not of the
|
||||
# flags). SPEC_TYPE=none strips BOTH speculative flags (flags move as a set);
|
||||
# there is no in-file head — this kind of drafter exists only as tensors
|
||||
# inside the checkpoint file, and no flag can supply one the file lacks.
|
||||
# SPEC_TYPE=none strips BOTH speculative flags (they move as a set);
|
||||
# naming a drafter requires SPEC_DRAFT_FILE alongside SPEC_TYPE — the Gemma4
|
||||
# assistant on disk is arch gemma4-assistant and cannot drafter-match qwen35.
|
||||
: "${SPEC_TYPE:=none}"
|
||||
@@ -83,11 +84,11 @@ fi
|
||||
# has not finished, and Restart=always would turn it into a crash loop that
|
||||
# reads like a runtime fault.
|
||||
for path in "$LLAMA_BIN" "$MODEL_FILE"; do
|
||||
[[ -e "$path" ]] || { echo "frank-secondary: missing $path" >&2; exit 1; }
|
||||
[[ -e "$path" ]] || { echo "frank-zealot: missing $path" >&2; exit 1; }
|
||||
done
|
||||
EXTRA_ARGS=()
|
||||
if [[ -n "$MMPROJ_FILE" ]]; then
|
||||
[[ -e "$MMPROJ_FILE" ]] || { echo "frank-secondary: missing $MMPROJ_FILE" >&2; exit 1; }
|
||||
[[ -e "$MMPROJ_FILE" ]] || { echo "frank-zealot: missing $MMPROJ_FILE" >&2; exit 1; }
|
||||
EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
|
||||
fi
|
||||
|
||||
@@ -97,7 +98,7 @@ fi
|
||||
if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
|
||||
EXTRA_ARGS+=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
|
||||
if [[ -n "$SPEC_DRAFT_FILE" ]]; then
|
||||
[[ -e "$SPEC_DRAFT_FILE" ]] || { echo "frank-secondary: missing $SPEC_DRAFT_FILE" >&2; exit 1; }
|
||||
[[ -e "$SPEC_DRAFT_FILE" ]] || { echo "frank-zealot: missing $SPEC_DRAFT_FILE" >&2; exit 1; }
|
||||
EXTRA_ARGS+=(--spec-draft-model "$SPEC_DRAFT_FILE")
|
||||
fi
|
||||
fi
|
||||
@@ -108,7 +109,7 @@ nvidia-modprobe -u -c=0 2>/dev/null || true
|
||||
|
||||
# Flash attention stays pinned on: it is REQUIRED for quantized V (q8_0),
|
||||
# and the build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17), so
|
||||
# the q8_0 KV contract in frank-secondary.env is a real path — without this
|
||||
# the q8_0 KV contract in frank-zealot.env is a real path — without this
|
||||
# flag or that build option the engine silently falls back and the
|
||||
# contract becomes a lie.
|
||||
ARGS=(
|
||||
@@ -298,7 +298,7 @@ def main():
|
||||
and list(dssh) == [(dsh[0] + 127) // 128, (dsh[1] + 127) // 128])
|
||||
if not scale_law:
|
||||
log(f"ERROR: unexpected expert shapes: gate {gdt}{gsh} scale {gsd}{gssh} "
|
||||
f"down {ddt}{dsh} scale {dsd}{dssh} (scale law 128x128)")
|
||||
f"down {ddt}{dsh} scale {dsd}{dssh} (128x128 block scales)")
|
||||
return 2
|
||||
shapes = {"ffn2_h": (gsh[0] * 2, gsh[1]), "ffn": (dsh[1],),
|
||||
"gs": tuple(gssh), "ds": tuple(dssh)}
|
||||
|
||||
+11
-11
@@ -145,12 +145,12 @@ for path in "${CHANGED[@]}"; do
|
||||
add_test tests/homepage_caddy.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
;;
|
||||
# frank-large is a rootless quadlet again since 2026-09-16 (the
|
||||
# frank-dragoon is a rootless quadlet again since 2026-09-16 (the
|
||||
# halogen-flash-server restore). Its unit carries the fleet's GPU and
|
||||
# podman posture, so it gets its own seam rather than the generic
|
||||
# fallback below.
|
||||
dankclaw/frank-large/*)
|
||||
add_test tests/frank_large_unit.bats
|
||||
dankclaw/frank-dragoon/*)
|
||||
add_test tests/frank_dragoon_unit.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/docs_completeness.bats
|
||||
;;
|
||||
@@ -187,29 +187,29 @@ for path in "${CHANGED[@]}"; do
|
||||
# as additions, with "no focused-test mapping".
|
||||
.AGENTTRASH/*|*/.AGENTTRASH/*)
|
||||
;;
|
||||
# frank-primary is llama.cpp (the unsloth b11030-mix prebuilt) on the
|
||||
# frank-archon is llama.cpp (the unsloth b11030-mix prebuilt) on the
|
||||
# two R9700s as a ROOT unit — the 2026-09-26 swap that replaced the
|
||||
# Lucebox DS4.1 quadlet on :7770. Its env, unit and serve script all
|
||||
# live in scripts/, which is why this case has to precede the generic
|
||||
# scripts/*.service and scripts/* fallbacks below.
|
||||
scripts/frank-primary.*|scripts/serve-frank-primary.sh)
|
||||
add_test tests/frank_primary_unit.bats
|
||||
scripts/frank-archon.*|scripts/serve-frank-archon.sh)
|
||||
add_test tests/frank_archon_unit.bats
|
||||
add_test tests/llm_units.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
;;
|
||||
# frank-large's chat template lives in scripts/ (moved there on
|
||||
# frank-dragoon's chat template lives in scripts/ (moved there on
|
||||
# 2026-09-26 when its old home was retired): a template edit changes
|
||||
# what the live endpoint renders, so it selects the large unit seam.
|
||||
scripts/qwen-sharp-v22.4.0.jinja)
|
||||
add_test tests/frank_large_unit.bats
|
||||
add_test tests/frank_dragoon_unit.bats
|
||||
;;
|
||||
# frank-secondary is llama.cpp on the 3090 as a ROOT unit: a native CUDA
|
||||
# frank-zealot is llama.cpp on the 3090 as a ROOT unit: a native CUDA
|
||||
# binary, so no generated quadlet is involved. Its env, unit and serve
|
||||
# script all live in scripts/, which is why this case has to precede
|
||||
# the generic scripts/*.service and scripts/* fallbacks below.
|
||||
scripts/frank-secondary.*|scripts/serve-frank-secondary.sh)
|
||||
add_test tests/frank_secondary_unit.bats
|
||||
scripts/frank-zealot.*|scripts/serve-frank-zealot.sh)
|
||||
add_test tests/frank_zealot_unit.bats
|
||||
add_test tests/llm_units.bats
|
||||
add_test tests/deployment_scoping.bats
|
||||
add_test tests/scripts_syntax.bats
|
||||
|
||||
Executable
+104
@@ -0,0 +1,104 @@
|
||||
#!/usr/bin/env bash
|
||||
# ttm-preflight.sh — the TTM pages-cap gate for engines that PIN host RAM.
|
||||
#
|
||||
# Sourced, never executed. An engine that pins tens of GiB of host RAM (the
|
||||
# radiance expert pools + the PLE n-gram table) has its pinned pages charged
|
||||
# against the kernel's ttm page cap: /sys/module/ttm/parameters/pages_limit
|
||||
# (and pages_max on kernels that expose one). The cap function below takes the
|
||||
# MIN over the knobs the running kernel actually exposes — an absent knob is
|
||||
# skipped, never read as zero, because a zero would refuse a healthy cap.
|
||||
#
|
||||
# THE LAW THIS FILE EXISTS FOR: the cap is a RUNTIME knob. The kernel default is
|
||||
# half of RAM, so every raise is lost on reboot. On 2026-10-12 23:17 a power
|
||||
# failure on dankserver reset it 200 GiB -> 125,988 MiB, and frank-archon's
|
||||
# preflight refused the start — 6,919 crash-restarts over ~20 hours, with
|
||||
# `Restart=always` cheerfully re-running a refusal. A manual sysfs write is
|
||||
# therefore not an acceptable repair procedure: the gate raises the knob itself
|
||||
# (the units that need it run as root, and the knob is a 0644 root-owned module
|
||||
# parameter), re-reads, and only then refuses.
|
||||
#
|
||||
# Contract (set by the sourcing script's env file):
|
||||
# FRANK_ARCHON_TTM_MIN_MIB floor the pinned ledger needs; refuse below it
|
||||
# FRANK_ARCHON_TTM_TARGET_MIB value to raise the cap to when it is below the
|
||||
# floor (the recorded host ceiling — writing the
|
||||
# floor itself would leave the ledger no slack)
|
||||
# FRANK_ARCHON_TTM_SYS_DIR knob directory; defaults to the real sysfs
|
||||
# path, a test points it at a temp dir.
|
||||
# Pages are 4 KiB: pages = MiB x 256.
|
||||
|
||||
: "${FRANK_ARCHON_TTM_SYS_DIR:=/sys/module/ttm/parameters}"
|
||||
|
||||
# Prints the effective cap in MiB, 0 when no knob is readable.
|
||||
ttm_cap_mib() {
|
||||
local p v cur=""
|
||||
for p in "$FRANK_ARCHON_TTM_SYS_DIR/pages_limit" \
|
||||
"$FRANK_ARCHON_TTM_SYS_DIR/pages_max"; do
|
||||
[ -r "$p" ] || continue
|
||||
v="$(cat "$p" 2>/dev/null)" || continue
|
||||
[ -n "$v" ] || continue
|
||||
v=$(( v * 4 / 1024 ))
|
||||
if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
|
||||
done
|
||||
echo "${cur:-0}"
|
||||
}
|
||||
|
||||
# Writes $1 MiB (as 4 KiB pages) into every knob present. Failures are
|
||||
# swallowed: the caller re-reads the cap, and a write the kernel refused shows
|
||||
# up there (ttm refuses a limit below the pages it has already pinned).
|
||||
#
|
||||
# PRIVILEGE, measured the hard way: the knob is root-owned 0644 and the service
|
||||
# process runs as User=dank, so `-w` is FALSE there and a plain echo fails. The
|
||||
# canonical privileged path is the unit's ExecStartPre with the `+` prefix
|
||||
# (systemd runs it as root even under User=); `sudo -n` is the fallback for a
|
||||
# human running the gate from a shell (passwordless sudo on both hosts).
|
||||
ttm_raise_cap() {
|
||||
local pages=$(( $1 * 256 )) p
|
||||
for p in "$FRANK_ARCHON_TTM_SYS_DIR/pages_limit" \
|
||||
"$FRANK_ARCHON_TTM_SYS_DIR/pages_max"; do
|
||||
[ -e "$p" ] || continue
|
||||
if [ -w "$p" ]; then
|
||||
echo "$pages" > "$p" 2>/dev/null || true
|
||||
elif command -v sudo >/dev/null 2>&1; then
|
||||
sudo -n sh -c "echo $pages > '$p'" >/dev/null 2>&1 || true
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
# Returns 0 when the cap clears FRANK_ARCHON_TTM_MIN_MIB, raising it first if
|
||||
# needed; returns 1 with a named refusal when it cannot be made to.
|
||||
ttm_ensure_cap() {
|
||||
local cap
|
||||
cap="$(ttm_cap_mib)"
|
||||
if [ "$cap" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
|
||||
echo "frank-archon: ttm pages cap is ${cap} MiB, below the pinned-ledger" \
|
||||
"floor $FRANK_ARCHON_TTM_MIN_MIB — raising the knob to" \
|
||||
"$FRANK_ARCHON_TTM_TARGET_MIB MiB ($(( FRANK_ARCHON_TTM_TARGET_MIB * 256 )) pages)" >&2
|
||||
ttm_raise_cap "$FRANK_ARCHON_TTM_TARGET_MIB"
|
||||
cap="$(ttm_cap_mib)"
|
||||
fi
|
||||
if [ "$cap" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
|
||||
echo "ERROR: ttm pages cap is ${cap} MiB, the pinned ledger needs >= $FRANK_ARCHON_TTM_MIN_MIB" >&2
|
||||
echo "frank-archon: raise $FRANK_ARCHON_TTM_SYS_DIR/pages_limit (and pages_max if the kernel exposes one) to >= $(( FRANK_ARCHON_TTM_MIN_MIB * 256 )) in 4 KiB pages; a read-only knob needs the module option (options ttm pages_limit=N) and a reboot" >&2
|
||||
return 1
|
||||
fi
|
||||
echo "frank-archon: ttm pages cap ${cap} MiB clears the pinned-ledger floor $FRANK_ARCHON_TTM_MIN_MIB" >&2
|
||||
}
|
||||
|
||||
# Executed (not sourced) it is the privileged pre-start hook: systemd runs this
|
||||
# as root from the unit's `ExecStartPre=+…`, where the knob write is legal and
|
||||
# the `Restart=always` loop can never outrun it.
|
||||
if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
|
||||
set -euo pipefail
|
||||
case "${1:-}" in
|
||||
--raise)
|
||||
TTM_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=/dev/null
|
||||
source "${FRANK_ARCHON_ENV_FILE:-$TTM_DIR/frank-archon.env}"
|
||||
ttm_ensure_cap
|
||||
;;
|
||||
*)
|
||||
echo "usage: ${0##*/} --raise (raise the ttm pages cap, then verify)" >&2
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
@@ -16,13 +16,13 @@ this file is repo-managed; edits are harness-side only.
|
||||
**Model ids are the StarCraft unit names (official 2026-10-08, on the wire
|
||||
since 2026-10-09):** `frank-archon` (dankserver :7770, radiance),
|
||||
`frank-dragoon` (dankclaw :7770, flash-server), `frank-zealot` (dankclaw
|
||||
:7771, the 3090 compaction hub) — formerly `frank-primary`, `frank-large`,
|
||||
`frank-secondary`. The SERVICES keep the old names (units, scripts, env
|
||||
:7771, the 3090 compaction hub) — formerly `frank-archon`, `frank-dragoon`,
|
||||
`frank-zealot`. The SERVICES keep the old names (units, scripts, env
|
||||
files, runbook pages); pages written in the old era name the service,
|
||||
the harness names the model. The wire now matches: each endpoint's
|
||||
`/v1/models` answers with the Protoss name, so `models.yml`, the OpenWebUI
|
||||
picker/grants, the homepage docs and the raw API all use one id per model
|
||||
(see [[OpenWebUI]] for the grant re-key and the [[frank-primary-radiance-swift15-w4dn8]]
|
||||
(see [[OpenWebUI]] for the grant re-key and the [[frank-archon-radiance-swift15-w4dn8]]
|
||||
note on radiance echoing the requested name).
|
||||
|
||||
## Compaction chain arithmetic (verify before ANY window change)
|
||||
@@ -45,14 +45,14 @@ The arithmetic (the coupled contract every endpoint's maxTokens comment
|
||||
encodes):
|
||||
|
||||
1. **Source trigger** = `thresholdPercent × source contextWindow`
|
||||
(0.70 × 200,704 = 140,492 for frank-primary on the current shape)
|
||||
(0.70 × 200,704 = 140,492 for frank-archon on the current shape)
|
||||
2. **Output budget** = capped at the target model's `maxTokens`
|
||||
(32,768 for frank-secondary; 36,864 frank-primary; 57,344 frank-large — the
|
||||
(32,768 for frank-zealot; 36,864 frank-archon; 57,344 frank-dragoon — the
|
||||
largest, because halogen's `HALOGEN_MAX_TOKENS_CAP` must stay ≥ it)
|
||||
3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
|
||||
window. Worked pairs from the live registry: 140,492 + 36,864 = 177,356 ≤
|
||||
200,704 (frank-primary); 140,490 + 57,344 = 197,834 ≤ 200,700
|
||||
(frank-large); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-secondary).
|
||||
200,704 (frank-archon); 140,490 + 57,344 = 197,834 ≤ 200,700
|
||||
(frank-dragoon); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-zealot).
|
||||
|
||||
Procedure (recompute from the LIVE files every time — windows rotate with
|
||||
every swap; the numbers above are the 2026-10-05 state):
|
||||
@@ -112,19 +112,19 @@ Use **wire names**, not OMP camelCase. Two traps:
|
||||
`qwenPreserveThinking` is auto-derived in `buildOpenAICompat` from the
|
||||
*auto-detected* `thinkingFormat` + local baseUrl, **before**
|
||||
`applyCompatOverrides`. Setting `compat.thinkingFormat: qwen` in
|
||||
models.yml does **not** enable it. Custom-named models (`frank-large`)
|
||||
models.yml does **not** enable it. Custom-named models (`frank-dragoon`)
|
||||
match no Qwen id pattern → use `extraBody: {preserve_thinking: true}`.
|
||||
2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy
|
||||
also sets it — `Object.assign` is shallow and replaces the whole object,
|
||||
silently dropping the `reasoning_effort` kwarg the compat policy set.
|
||||
Top-level `preserve_thinking` is honoured by halogen. (frank-primary's
|
||||
Top-level `preserve_thinking` is honoured by halogen. (frank-archon's
|
||||
FreeToken entry currently carries a `chat_template_kwargs` extraBody with
|
||||
a comment asking to verify-and-prune — the exact trap to check at that
|
||||
endpoint's next swap.)
|
||||
|
||||
**Gotcha: silent greedy decode.** halogen has **no server-side sampler
|
||||
defaults** — `temperature` absent or `0` = greedy. If OMP sends nothing, the
|
||||
endpoint decodes greedy while looking configured. This is why frank-large's
|
||||
endpoint decodes greedy while looking configured. This is why frank-dragoon's
|
||||
entry carries the recipe on the wire and a comment saying "do not drop it".
|
||||
|
||||
**Reasoning-effort folding.** Qwen3.8's template accepts only
|
||||
|
||||
@@ -23,7 +23,7 @@ marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
|
||||
|
||||
**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
|
||||
whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of
|
||||
a 100+ GiB payload — and frank-primary (dankserver:7770) is also the
|
||||
a 100+ GiB payload — and frank-archon (dankserver:7770) is also the
|
||||
harness's own model backend, so a blind deploy is a self-inflicted outage for
|
||||
possibly zero behavioral change.
|
||||
|
||||
|
||||
@@ -62,8 +62,8 @@ service on the other host.
|
||||
|---|---|---|
|
||||
| Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
|
||||
| SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. |
|
||||
| `frank-large` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
|
||||
| `frank-secondary` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
|
||||
| `frank-dragoon` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
|
||||
| `frank-zealot` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
|
||||
| `frank-medium` | rootless Quadlet + `systemctl --user` | Installs `dankserver/frank-medium/quadlet/frank-medium.container`; on first deploy retires the llama.cpp root unit and the vLLM replica quadlets into the home trash, then restarts the generated unit. Waits up to 20 minutes for `/health`. |
|
||||
| Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
|
||||
|
||||
|
||||
@@ -95,7 +95,7 @@ host-side `workflows/` dir and the repo copy.
|
||||
## Recommended additions
|
||||
|
||||
Quantized preferred, sized for the 8060S iGPU (GTT-backed; the RTX 3090 is
|
||||
frank-secondary's). All repos verified 2026-10-05. None are on disk yet.
|
||||
frank-zealot's). All repos verified 2026-10-05. None are on disk yet.
|
||||
|
||||
| # | What | Repo | Grab | Size | Why |
|
||||
|---|---|---|---|---|---|
|
||||
|
||||
@@ -13,7 +13,7 @@ timestamp: 2026-08-26
|
||||
# Dankclaw Stack
|
||||
|
||||
Dankclaw (192.168.0.143) runs the AI, search, development, and observability
|
||||
stack. Its LLM inference endpoint `frank-large` (`:7770`) is a ROCmFPX
|
||||
stack. Its LLM inference endpoint `frank-dragoon` (`:7770`) is a ROCmFPX
|
||||
llama.cpp fork serving the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF on
|
||||
the card's pinned runtime; an NPU endpoint (`frank-npu`, `:7771`,
|
||||
FLM v1.0.1) serves sight+sound models. The completion endpoint
|
||||
@@ -30,7 +30,7 @@ graph LR
|
||||
S --> V[Gluetun VPN]
|
||||
O --> L[llama.cpp dankserver :7770]
|
||||
O --> H[Hermes]
|
||||
O --> E[frank-large :7770]
|
||||
O --> E[frank-dragoon :7770]
|
||||
```
|
||||
|
||||
## Services
|
||||
@@ -48,14 +48,14 @@ graph LR
|
||||
| Seerr | `seerr.franksdank.cloud` (guarded) | Caddy routes post-auth to `seerr:5055` | [[Seerr]] |
|
||||
| Homepage | `franksdank.cloud` + `www.` (root site; guarded for admin actions) | served by Caddy itself from `/srv/homepage` | — |
|
||||
| Pi-hole | `pihole.franksdank.cloud` (basicauth edge, not the guard chain) | admin UI `8083`; DNS on `:53` LAN-wide | [[Pi-hole]] |
|
||||
| frank-large (host service) | — | `7770` | [[deployment]] |
|
||||
| frank-dragoon (host service) | — | `7770` | [[deployment]] |
|
||||
| frank-npu (host service) | — | `7771` (`/v1/version` health) | [[frank-npu]] |
|
||||
| frank-medium (host service, Dankserver) | — | `7770` | [[frank-medium-rocmfpx]] |
|
||||
|
||||
The LLM rows are bare-metal systemd services, not containers. This
|
||||
repo owns their scripts, units, and knowledge base (see
|
||||
[LLM fleet knowledge base](../LLM/index.md)). Current mapping (2026-09-03):
|
||||
`frank-large` runs the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF
|
||||
`frank-dragoon` runs the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF
|
||||
on the card's pinned `kingjones30/ROCmFPX` llama.cpp build;
|
||||
`frank-npu` runs FLM v1.0.1 on the XDNA2 NPU;
|
||||
`frank-medium` runs charlie12345/ROCmFPX llama.cpp serving
|
||||
|
||||
@@ -47,7 +47,7 @@ is not durable. The tracked, sanitized configuration baseline is
|
||||
`0.0.0.0:8080`, so the Signal HTTP API is reachable from the whole LAN.
|
||||
- The data volume persists skills, memory, and session history.
|
||||
- Hermes has two named local providers: `dankclaw-large` reaches
|
||||
`http://192.168.0.143:7770/v1` and exposes `frank-large`;
|
||||
`http://192.168.0.143:7770/v1` and exposes `frank-dragoon`;
|
||||
`dankserver-small` reaches `http://192.168.0.235:7770/v1` and exposes
|
||||
`frank-medium` (renamed from `frank-small` on 2026-09-15).
|
||||
- The live `config.yaml` contains secrets and user state and is not committed;
|
||||
@@ -64,7 +64,7 @@ is not durable. The tracked, sanitized configuration baseline is
|
||||
## Gotchas
|
||||
|
||||
- Hermes is under active development; pin the image tag for reproducibility.
|
||||
- Model choice drives quality; the local `frank-large` (Dankclaw, 7770) is the intended backend.
|
||||
- Model choice drives quality; the local `frank-dragoon` (Dankclaw, 7770) is the intended backend.
|
||||
- Do not expose the gateway without auth — put it behind [[Caddy]] with basic auth.
|
||||
|
||||
## Browser automation (Chrome DevTools Protocol daemon)
|
||||
@@ -189,7 +189,7 @@ What actually breaks, and the fix:
|
||||
auxiliary:
|
||||
title_generation:
|
||||
provider: dankserver-small # names track the live config; diff first
|
||||
model: frank-secondary
|
||||
model: frank-zealot
|
||||
api_mode: codex_responses # real per-task key
|
||||
extra_body:
|
||||
reasoning:
|
||||
|
||||
@@ -24,7 +24,7 @@ https://github.com/open-webui/open-webui
|
||||
| Upstream latest | 0.11.4 (2026-09-21) — security/access-control advisory; `:main` is the rolling image, not a stable pin. Imagegen endpoint is `POST /api/v1/images/generations` (the `comfyui` engine is still upstream). |
|
||||
| Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. |
|
||||
| Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` |
|
||||
| Backends | Picker exposes exactly one model — `frank-secondary` (host.containers.internal:7771). The `frank-large` and `frank-primary` connections remain configured in the DB, disabled (2026-10-07) — see [The LLM backends](#the-llm-backends) |
|
||||
| Backends | Picker exposes exactly one model — `frank-zealot` (host.containers.internal:7771). The `frank-dragoon` and `frank-archon` connections remain configured in the DB, disabled (2026-10-07) — see [The LLM backends](#the-llm-backends) |
|
||||
| Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) |
|
||||
| Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` |
|
||||
| Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` |
|
||||
@@ -34,15 +34,15 @@ https://github.com/open-webui/open-webui
|
||||
The picker is built from the `model` table in `webui.db` (one row per model)
|
||||
plus the `model_ids` of each **enabled** OpenAI connection, NOT from live
|
||||
connection discovery — so the picker shows exactly what is seeded there.
|
||||
As of 2026-10-07 the picker exposes **exactly one model: `frank-secondary`**.
|
||||
As of 2026-10-07 the picker exposes **exactly one model: `frank-zealot`**.
|
||||
|
||||
| Connection (`openai.api_configs` idx) | URL | Enabled | `model_ids` | Runtime | Host |
|
||||
|---|---|---|---|---|---|
|
||||
| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-large` | halogen flash-server (Strix Halo iGPU) | Dankclaw |
|
||||
| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-primary` | radiance TP=2 (2× R9700) | Dankserver |
|
||||
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-secondary` | llama.cpp CUDA (RTX 3090, 2×200,704) | Dankclaw |
|
||||
| 0 | `http://host.containers.internal:7770/v1` | no (2026-10-07) | `frank-dragoon` | halogen flash-server (Strix Halo iGPU) | Dankclaw |
|
||||
| 1 | `http://192.168.0.235:7770/v1` | no (2026-10-07) | `frank-archon` | radiance TP=2 (2× R9700) | Dankserver |
|
||||
| 2 | `http://host.containers.internal:7771/v1` | yes | `frank-zealot` | llama.cpp CUDA (RTX 3090, 2×200,704) | Dankclaw |
|
||||
|
||||
The `model` table carries exactly one decoration row, `frank-secondary`
|
||||
The `model` table carries exactly one decoration row, `frank-zealot`
|
||||
(context length 200,704 = the per-slot window; description: Qwen3.5-9B
|
||||
Q6_K + BF16 vision mmproj).
|
||||
|
||||
@@ -62,7 +62,7 @@ Two gotchas make the wiring non-obvious:
|
||||
- **llama.cpp `/v1/models` is Ollama-shaped.** llama.cpp returns
|
||||
`{"models":[{"name":...}]}`; OpenWebUI's `extract_data()` only reads the
|
||||
OpenAI `{"data":[{"id":...}]}` shape, so connection discovery silently misses
|
||||
the llama.cpp endpoints (frank-large, frank-secondary). The fix is to set
|
||||
the llama.cpp endpoints (frank-dragoon, frank-zealot). The fix is to set
|
||||
`model_ids` in each connection's `api_config` — when non-empty, OpenWebUI
|
||||
builds the model list directly and never fetches `/v1/models`. frank-medium
|
||||
(vLLM) serves the OpenAI shape and would discover fine, but uses `model_ids`
|
||||
@@ -116,7 +116,7 @@ The Ollama connection is disabled (`ollama.enable=false`).
|
||||
models. Response key is `items`, not `data`.
|
||||
2. `POST /api/chat/completions` per model with a trivial prompt ("Reply
|
||||
with exactly one word: pong") → each returns content `pong`. Reasoning
|
||||
models need a generous budget: frank-secondary (Qwen3.5 template) returns
|
||||
models need a generous budget: frank-zealot (Qwen3.5 template) returns
|
||||
empty content + `finish_reason: "length"` at `max_tokens 200` (the
|
||||
thinking block eats it) and answers `pong` at 1500.
|
||||
3. A 500 "Server Connection Error" → `podman logs open-webui` for
|
||||
@@ -191,8 +191,8 @@ browser-driven and only touches the two exempt vhosts (`ai.` and `auth.`).
|
||||
run with host `python3` over ssh against
|
||||
`/home/dank/Containers/open-webui/data/webui.db` (no `sqlite3` CLI on
|
||||
Dankclaw; table is `user`, singular; no `active` column). No per-user model
|
||||
grant is needed: the picker model `frank-secondary` carries the global
|
||||
`access_grant` row `(model, frank-secondary, user, *, read)`, so any
|
||||
grant is needed: the picker model `frank-zealot` carries the global
|
||||
`access_grant` row `(model, frank-zealot, user, *, read)`, so any
|
||||
`role='user'` account sees it. Flip on FIRST-LOGIN SIGHTING — the row does
|
||||
not exist until the account signs in (2026-10-09: five provisioned accounts
|
||||
had no row).
|
||||
@@ -271,18 +271,18 @@ on large sizes ⇒ check ComfyUI resources; 401 ⇒ the token is bad.
|
||||
|
||||
- **2026-08-22 — slirp4netns → pinned bridge.** The unit ran on `Network=slirp4netns`. The slirp4netns helper process died (no OOM, no kill entry, no crash log found), and the container kept running with only `lo` in its netns: published port 3005 accepted TCP and reset every request, so chat served empty responses for ~2 days. Nothing logged an error. Moving to a kernel bridge removes the per-container helper entirely, and the healthcheck's interface check makes the zombie state visible if it ever returns.
|
||||
|
||||
- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-large`/`frank-medium`/`frank-secondary` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`.
|
||||
- **2026-09-21 — model picker rebuilt for the 3-model fleet.** The picker carried 39 stale rows (37 dead Ollama models + 2 `frank-default-*`) and was missing `frank-medium`. Rebuilt the `model` table to exactly `frank-dragoon`/`frank-medium`/`frank-zealot` and set `model_ids` on each connection (llama.cpp's Ollama-shaped `/v1/models` is invisible to OpenWebUI's discovery). Pointed the two on-host endpoints at `host.containers.internal` — the bridge refuses the host LAN IP for non-published ports. All three verified end-to-end via `/openai/chat/completions`.
|
||||
|
||||
- **2026-09-24 — SSO via authentik.** OIDC authorization-code flow wired per the [SSO (authentik)](#sso-authentik) section: application/provider/entitlements created in authentik, client secret in the host env file, roles from the `roles` scope claim. E2E coverage in `tests/e2e/specs/18-openwebui-sso.spec.ts`.
|
||||
|
||||
- **2026-09-24 — flipped to SSO-only.** The OIDC round-trip verified in a real browser (login view renders no password form, only "Continue with authentik", and the flow lands on a signed-in session), so the local username/password path was retired: `ENABLE_LOGIN_FORM=false` + `ENABLE_PASSWORD_AUTH=false` in the quadlet and `ui.enable_login_form` set to `false` in the config DB (the DB wins over env, so both were changed together). `qa-e2e-admin` was demoted out of the `authentik Admins` group (it keeps `openwebui_users` / `homepage-admins` for app access).
|
||||
|
||||
- **2026-10-07 — picker narrowed to `frank-secondary` only.** Per Frank: the only model accessible in OpenWebUI is `frank-secondary`. Rebuilt the `model` table to a single `frank-secondary` row (the old rows `frank-large`/`frank-primary`/`frank-small` are deleted), set `openai.api_configs` indices 0 (frank-large) and 1 (frank-primary) to `enable: false`, and retargeted index 2's `model_ids` from the stale `frank-small` label to `frank-secondary` — llama.cpp serves by alias, so the old label addressed a model name the server does not know. Pre-change snapshot: `/home/dank/Containers/open-webui/data/webui.db.bak-20261007` (+ `-wal`/`-shm`). Verified: `/api/v1/models/list` returns exactly `["frank-secondary"]`, chat completion answers `pong` (needs `max_tokens≥1500` — the Qwen3.5 thinking block consumes a 200-token budget), and anonymous API probes are all 401 (see [API access](#api-access-auth-gated)).
|
||||
- **2026-10-07 — picker narrowed to `frank-zealot` only.** Per Frank: the only model accessible in OpenWebUI is `frank-zealot`. Rebuilt the `model` table to a single `frank-zealot` row (the old rows `frank-dragoon`/`frank-archon`/`frank-small` are deleted), set `openai.api_configs` indices 0 (frank-dragoon) and 1 (frank-archon) to `enable: false`, and retargeted index 2's `model_ids` from the stale `frank-small` label to `frank-zealot` — llama.cpp serves by alias, so the old label addressed a model name the server does not know. Pre-change snapshot: `/home/dank/Containers/open-webui/data/webui.db.bak-20261007` (+ `-wal`/`-shm`). Verified: `/api/v1/models/list` returns exactly `["frank-zealot"]`, chat completion answers `pong` (needs `max_tokens≥1500` — the Qwen3.5 thinking block consumes a 200-token budget), and anonymous API probes are all 401 (see [API access](#api-access-auth-gated)).
|
||||
|
||||
- **2026-10-09 — re-keyed to the Protoss ids (wire rename).** The endpoints
|
||||
began serving `frank-zealot` / `frank-dragoon` / `frank-archon`, so three
|
||||
id-keyed objects had to move in the same window as the alias flip:
|
||||
`model.id`+`name` (`frank-secondary` → `frank-zealot`, `params`
|
||||
`model.id`+`name` (`frank-zealot` → `frank-zealot`, `params`
|
||||
`{"context":{"length":200704}}` and the `meta.description` kept),
|
||||
`access_grant` (`resource_type='model'`, `resource_id`, principal
|
||||
`user`/`*`, permission `read`), and `openai.api_configs` per-backend
|
||||
|
||||
@@ -33,10 +33,10 @@ it is not invisible.
|
||||
|
||||
## Why 64 GiB
|
||||
|
||||
Dankclaw is shared. `frank-large` alone holds **103.5 GiB of the 121.2 GiB**
|
||||
Dankclaw is shared. `frank-dragoon` alone holds **103.5 GiB of the 121.2 GiB**
|
||||
host (`68.0 weights + 14.4 KV pool + 21.1 working`), leaving roughly
|
||||
**7.1 GiB** for 28 co-tenant containers whose working set is ~4.3 GiB. The
|
||||
engine itself warns that when that runs out, frank-large and its neighbours
|
||||
engine itself warns that when that runs out, frank-dragoon and its neighbours
|
||||
"can stall for minutes at a time at full CPU with no disk activity and no
|
||||
output". Swap is the valve that turns an OOM kill into slow paging.
|
||||
|
||||
@@ -96,20 +96,20 @@ Then confirm the inference service is unaffected — a completion, not just
|
||||
```bash
|
||||
curl -s http://127.0.0.1:7770/v1/chat/completions \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"model":"frank-large","messages":[{"role":"user","content":"Reply with exactly: SWAP64-OK"}],"max_tokens":48,"reasoning_effort":"low"}'
|
||||
-d '{"model":"frank-dragoon","messages":[{"role":"user","content":"Reply with exactly: SWAP64-OK"}],"max_tokens":48,"reasoning_effort":"low"}'
|
||||
```
|
||||
|
||||
As run 2026-09-12: `finish_reason: stop`, `SWAP64-OK`, 28 containers up,
|
||||
frank-large `health=200` throughout.
|
||||
frank-dragoon `health=200` throughout.
|
||||
|
||||
## Caveats
|
||||
|
||||
- **Swap cannot help the pinned weights.** frank-large's ~68 GiB of weights are
|
||||
- **Swap cannot help the pinned weights.** frank-dragoon's ~68 GiB of weights are
|
||||
held deliberately; paging them is a stall, not a relief. More swap does not
|
||||
make the doubled KV pool affordable — see the reserve/pool ceiling in
|
||||
[[halogen-flash-server]].
|
||||
- **`vm.swappiness=60` is untested against this workload.** Under real
|
||||
pressure the kernel will happily page anonymous memory, and frank-large's KV
|
||||
pressure the kernel will happily page anonymous memory, and frank-dragoon's KV
|
||||
pool and working set *are* anonymous. Swapping the KV pool would be far worse
|
||||
than the OOM kill it prevents. If the box ever starts thrashing, lowering
|
||||
swappiness (or capping the engine's cgroup) is the knob, not more swap.
|
||||
|
||||
@@ -22,8 +22,8 @@ The homepage is live on Dankclaw. `./scripts/deploy.sh dankclaw` copies the stat
|
||||
**Docs page shipped (2026-09-17):** `/docs` serves a static user documentation
|
||||
page (`shared/homepage/docs.html`) — access walkthrough, the running-model
|
||||
guide, media how-tos, and troubleshooting. Per-service `/docs/<svc>/` pages
|
||||
remain a separate thread. The three LLM cards (`frank-large`, `frank-medium`,
|
||||
`frank-secondary`) moved from admin-only to the **user** page with live model
|
||||
remain a separate thread. The three LLM cards (`frank-dragoon`, `frank-medium`,
|
||||
`frank-zealot`) moved from admin-only to the **user** page with live model
|
||||
descriptions and a `docs` link to `/docs#models`.
|
||||
|
||||
**Sections + WIP split (2026-09-21):** both pages now group their panels
|
||||
@@ -48,23 +48,23 @@ exact bug bit the screenshots on 2026-10-08). The docs page gained a full
|
||||
(`/docs#api-access`): per-user OpenWebUI API keys (User menu → Settings →
|
||||
Account → API keys), the base URL `https://ai.franksdank.cloud/api/v1`, and
|
||||
copy-paste configs for **Opencode** (`~/.config/opencode/opencode.json`) and
|
||||
**Pi** (`~/.pi/agent/models.json`) pointing at `frank-secondary`. Shipped
|
||||
**Pi** (`~/.pi/agent/models.json`) pointing at `frank-zealot`. Shipped
|
||||
together with the OpenWebUI side: `features.api_keys=true` for regular users
|
||||
and the restored `frank-secondary` model row + public `read` access grant
|
||||
and the restored `frank-zealot` model row + public `read` access grant
|
||||
without which the picker showed "No results found" for every family account
|
||||
(see [[OpenWebUI]]).
|
||||
|
||||
**Capability equivalence notes (2026-10-08):** each fleet card (and the
|
||||
`/docs#models` section) carries a `details.equiv` line — frank-large
|
||||
`/docs#models` section) carries a `details.equiv` line — frank-dragoon
|
||||
"equivalent to Claude Sonnet 5 / OpenAI-Luna (Summer 2026 frontier
|
||||
intelligence)", frank-primary "equivalent to Claude Opus 4.8 / OpenAI-Terra
|
||||
(same era)", frank-secondary "equivalent to last year's frontier level
|
||||
intelligence)", frank-archon "equivalent to Claude Opus 4.8 / OpenAI-Terra
|
||||
(same era)", frank-zealot "equivalent to last year's frontier level
|
||||
(2025)". Rendered by `descHTML()` as `.desc-equiv` (head color, italic);
|
||||
pinned in `homepage_config.bats`.
|
||||
|
||||
**Protoss rename + Model Information component (2026-10-08):** the model
|
||||
cards became the StarCraft trio — `frank-large` → **frank-dragoon**
|
||||
(iGPU), `frank-primary` → **frank-archon** (Dankserver), `frank-secondary`
|
||||
cards became the StarCraft trio — `frank-dragoon` → **frank-dragoon**
|
||||
(iGPU), `frank-archon` → **frank-archon** (Dankserver), `frank-zealot`
|
||||
→ **frank-zealot** (3090). The endpoint/API ids did NOT change in this wave; only the
|
||||
family-facing names. The three cards fused into one **Model Information**
|
||||
component (`app.js` `renderModelInfo()`, `.model-info`/`.mi-*` styles):
|
||||
@@ -201,8 +201,8 @@ the pixels.
|
||||
**Model ids official (2026-10-08):** the card ids `frank-zealot`,
|
||||
`frank-dragoon`, `frank-archon` are now the harness's official model ids
|
||||
in `models.yml`/`config.yml` (see [[OMP Harness]]); the
|
||||
systemd services keep the `frank-primary`/`frank-large`/
|
||||
`frank-secondary` names. Homepage, docs page and harness now agree on
|
||||
systemd services keep the `frank-archon`/`frank-dragoon`/
|
||||
`frank-zealot` names. Homepage, docs page and harness now agree on
|
||||
one name per model.
|
||||
|
||||
**Api ids renamed on the wire (2026-10-09):** the promise in
|
||||
@@ -316,7 +316,7 @@ flowchart TD
|
||||
## Service classification
|
||||
|
||||
**User services** (root page), in section order:
|
||||
- **AI** — OpenWebUI, and the three LLM fleet cards (frank-large, frank-primary, frank-secondary — status + docs link, no Open link).
|
||||
- **AI** — OpenWebUI, and the three LLM fleet cards (frank-dragoon, frank-archon, frank-zealot — status + docs link, no Open link).
|
||||
- **Web Apps** — SearXNG, Gitea, Copyparty, Marreta.
|
||||
- **Media Library Apps** — Calibre-Web, RomM, Plex, Jellyfin, Seerr.
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ Authentik; post-auth the Caddy vhost routes straight to `prometheus:9090`) —
|
||||
| `node-dankserver` | `192.168.0.235:9100` | host metrics from [[Node-Exporter]] — a different machine, so the LAN address works |
|
||||
| `grafana`, `loki` | `grafana:3000`, `loki:3100` | container names on the `dankclaw_default` network; both serve real `/metrics` |
|
||||
| `frank-medium` | `192.168.0.235:7770` | the LLM fleet — `vllm:` series; a different machine, so the LAN address works |
|
||||
| `frank-large`, `frank-secondary` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |
|
||||
| `frank-dragoon`, `frank-zealot` | `host.containers.internal:7770`, `:7771` | the LLM fleet — `llamacpp:` series; same-host net=host binds, so via the gateway |
|
||||
|
||||
**Three jobs read down, and they are not outages** (audited 2026-09-16):
|
||||
|
||||
|
||||
@@ -115,7 +115,7 @@ unconditionally, while **context-backed** types (`draft-mtp`, `draft-dflash`,
|
||||
- Internal order is fixed (ngram types first, then drafters); CLI order is
|
||||
irrelevant.
|
||||
- **A legal chain can still lose on a recurrent model.** `draft-mtp,ngram-mod`
|
||||
on the qwen35 hybrid (frank-large era) measured −15 % prose / −7 %
|
||||
on the qwen35 hybrid (frank-dragoon era) measured −15 % prose / −7 %
|
||||
block-copy (2026-10-03): adding the ngram layer logs `recurrent state
|
||||
rollback is not compatible with 'ngram-mod' - disabling rollback support`,
|
||||
and MTP verification depends on that rollback, so acceptance drops on ALL
|
||||
@@ -215,9 +215,9 @@ The DFlash draft is not a standalone model. It must pair with its target model.
|
||||
|
||||
## Deployment on Dankserver (NVIDIA RTX 3090)
|
||||
|
||||
The Dankserver llama.cpp deployment (`frank-secondary`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
|
||||
The Dankserver llama.cpp deployment (`frank-zealot`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
|
||||
|
||||
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-secondary` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
|
||||
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-zealot` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
|
||||
|
||||
The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:
|
||||
|
||||
@@ -250,7 +250,7 @@ docker run --rm --name vllm-qwen36-dflash \
|
||||
|
||||
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
|
||||
|
||||
`frank-large` on Dankclaw ran the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec was the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. This page is a historical record of that era (mid-August 2026); for the current deployment see [[kingjones-strix-rocmfp4]].
|
||||
`frank-dragoon` on Dankclaw ran the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec was the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. This page is a historical record of that era (mid-August 2026); for the current deployment see [[kingjones-strix-rocmfp4]].
|
||||
|
||||
History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
|
||||
|
||||
@@ -315,7 +315,7 @@ not switch without re-measuring".
|
||||
|
||||
The A/B vehicle for ANY single-GPU llama.cpp knob (batch/ubatch, KV type,
|
||||
slot split, quant rung — not just spec flags) is the transient systemd
|
||||
drop-in procedure in [[frank-large-benchmark]] and [[frank-secondary-benchmark]].
|
||||
drop-in procedure in [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]].
|
||||
|
||||
## Is it worth it? (arithmetic before deployment)
|
||||
|
||||
@@ -419,4 +419,4 @@ measurement that would settle it.
|
||||
- [[GGUF Compatibility]] — header parsing and custom tensor-type lock-in
|
||||
- [[Llama Cpp Serving Sizing]] — slot-pool and drafter KV sizing arithmetic
|
||||
- [[Llama Cpp Multi-Device Splits]] — where the drafter's KV actually lands
|
||||
- [[frank-large-benchmark]] and [[frank-secondary-benchmark]] — the single-GPU env drop-in A/B vehicle
|
||||
- [[frank-dragoon-benchmark]] and [[frank-zealot-benchmark]] — the single-GPU env drop-in A/B vehicle
|
||||
@@ -22,7 +22,7 @@ marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
|
||||
(b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx
|
||||
Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs
|
||||
302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not
|
||||
hold on the 8060S. See [[frank-large-benchmark]].
|
||||
hold on the 8060S. See [[frank-dragoon-benchmark]].
|
||||
- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
|
||||
does not exist on this machine (checked 2026-08-15). GPU DPM is already
|
||||
`performance`, forced level `auto`. This item is N/A.
|
||||
@@ -48,7 +48,7 @@ lifted to 114 GiB. ROCm/SMI still reports a 96 GiB VRAM carve-out (VGM) — a
|
||||
different, driver-reported figure that has always been larger than the kernel
|
||||
cap.
|
||||
This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory
|
||||
limits documented in [[frank-large-benchmark]].
|
||||
limits documented in [[frank-dragoon-benchmark]].
|
||||
|
||||
## 1. Self-Hosted Agentic Tools
|
||||
|
||||
@@ -106,7 +106,7 @@ All require root and a reboot for the modprobe change.
|
||||
|
||||
## Sources
|
||||
|
||||
- [[frank-large-benchmark]] — measured backend/spec numbers
|
||||
- [[frank-dragoon-benchmark]] — measured backend/spec numbers
|
||||
- [[deployment]] — deploy and update process
|
||||
- AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
|
||||
- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`,
|
||||
@@ -115,7 +115,7 @@ All require root and a reboot for the modprobe change.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[frank-large-benchmark]]
|
||||
- [[frank-dragoon-benchmark]]
|
||||
- [[DFlash DSpark Speculative Decoding]]
|
||||
- [[deployment]]
|
||||
- [[index]]
|
||||
|
||||
@@ -46,7 +46,7 @@ Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host run
|
||||
| Runtime | FastFlowLM 0.9.45 + XRT (installed 2026-08-16; **retired 2026-09-11** with the `amd_iommu=off` decision) |
|
||||
| Status | Not in service. Research only. See [[FastFlowLM NPU Inference]] |
|
||||
|
||||
The NPU is retired (2026-09-11). The `frank-secondary` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-secondary-gemma4-12b]]).
|
||||
The NPU is retired (2026-09-11). The `frank-zealot` name now serves Gemma4-12B QAT on the RTX 3090 via llama.cpp (see [[frank-zealot-gemma4-12b]]).
|
||||
|
||||
### ROCm
|
||||
|
||||
@@ -101,7 +101,7 @@ cable between claw and server is required for 10G.
|
||||
|
||||
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
|
||||
- The GPU budget is the GTT aperture: ~114 GiB (`gtt_total` in `/sys/class/drm/card1/device/`; `llama-server --list-devices` reports 116,732 MiB). rocm-smi's smaller SVM/dedicated carve-out is **not** the budget — the 2026-08 "64 GiB rocm-smi" lesson is stale. Do not use `free` or `dmidecode` for the GPU budget.
|
||||
- The active service is `frank-large` on port 7770: **gufo** (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source at `/home/dank/GitPackages/gufo-v0.1.1/`, serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, DFlash2 drafter). See [[frank-large-gufo-qwen38-27b]].
|
||||
- The active service is `frank-dragoon` on port 7770: **gufo** (gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from source at `/home/dank/GitPackages/gufo-v0.1.1/`, serving unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense, DFlash2 drafter). See [[frank-dragoon-gufo-qwen38-27b]].
|
||||
- The active runtime **links against the host ROCm** (`/opt/rocm` → `/etc/alternatives/rocm` → 7.2.4) through its RPATH, so it needs the host HIP stack rather than a bundled one — the opposite of the retired llama.cpp prebuilt, which carried its own. Building it needs the ROCm dev packages that are installed, and two workarounds recorded in the runbook (drop `librocwmma-dev`; pin `--gcc-install-dir` to GCC 15). The retired CIRU venv's TheRock 7.15 SDK (`LD_LIBRARY_PATH`) is not in the serving path.
|
||||
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
|
||||
- 1.1 TB free disk space is ample for model storage.
|
||||
|
||||
@@ -17,7 +17,7 @@ timestamp: 2026-09-20
|
||||
|
||||
Dankserver is a media/library server and the `frank-medium` LLM host. The
|
||||
host runs Ubuntu 26.04 (Resolute Raccoon). The RTX 3090 that served the
|
||||
FreeToken/CUDA `frank-secondary` era was removed on 2026-09-15 and replaced with
|
||||
FreeToken/CUDA `frank-zealot` era was removed on 2026-09-15 and replaced with
|
||||
two AMD Radeon AI PRO R9700 cards.
|
||||
|
||||
## Hardware Specifications
|
||||
@@ -107,8 +107,8 @@ hijack `libhsa-runtime64` were purged on 2026-09-15 — see
|
||||
The active inference service is `frank-medium.service`: mainline llama-server on
|
||||
port 7770, layer-split 32:33 across both R9700s with the DFlash2 drafter pinned to
|
||||
ROCm1. See [[frank-medium-llamacpp-dflash2]]. The former
|
||||
`frank-secondary.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
|
||||
its unit is archived in `~/.AGENTTRASH/20260915-frank-secondary-retire/host-units/`,
|
||||
`frank-zealot.service` (FreeToken/CUDA) was retired with the 2026-09-15 GPU swap;
|
||||
its unit is archived in `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/`,
|
||||
and the vllm-radiance replica trio in `~/.AGENTTRASH/20260916-vllm-radiance-retire/`.
|
||||
See [[deployment]].
|
||||
|
||||
|
||||
@@ -13,12 +13,12 @@ timestamp: 2026-09-03
|
||||
|
||||
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
|
||||
> See [[frank-redo-2026-08-17]] and [[deployment]] for the active configuration.
|
||||
> The `frank-secondary` name and port 52625 then belonged to the vLLM
|
||||
> The `frank-zealot` name and port 52625 then belonged to the vLLM
|
||||
> service on Dankclaw (nameistoken Quark W8A8); this page is archive.
|
||||
|
||||
# FastFlowLM NPU Inference
|
||||
|
||||
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-secondary` (port 52625) serves in production. See Promotion.
|
||||
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-zealot` (port 52625) serves in production. See Promotion.
|
||||
|
||||
## Verdict (2026-08-16)
|
||||
|
||||
@@ -167,7 +167,7 @@ Proposed role: a small, always-on, low-power endpoint for triage, summarization,
|
||||
- `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`.
|
||||
- `qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via `flm-add.py` (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry `~/.config/flm/model_list.json` (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked `~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2`. The service env exports `FLM_CONFIG_PATH` and `FLM_XCLBIN_PATH` so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below).
|
||||
- `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot.
|
||||
- The `llama-server` (frank-large, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.
|
||||
- The `llama-server` (frank-dragoon, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.
|
||||
|
||||
## Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU)
|
||||
|
||||
@@ -185,7 +185,7 @@ Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens)
|
||||
|
||||
## Model Bakeoff (measured 2026-08-16, 4 models)
|
||||
|
||||
All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-large) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).
|
||||
All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-dragoon) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).
|
||||
|
||||
| Model | Tag | Weights | Decode t/s (short) | 8k prefill t/s | 8k decode t/s | Vision decode t/s | Vision accuracy | Max ctx (registry) |
|
||||
|-------|-----|---------|-------------------:|---------------:|--------------:|------------------:|----------------|--------------------|
|
||||
@@ -203,21 +203,21 @@ Notes:
|
||||
|
||||
## Promotion (2026-08-16)
|
||||
|
||||
`frank-secondary` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-secondary.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-secondary.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
|
||||
`frank-zealot` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-zealot.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-zealot.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
|
||||
|
||||
### Boot policy (2026-08-17)
|
||||
|
||||
The `frank-secondary` unit is disabled for boot. Only the PP pair (`frank-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
|
||||
The `frank-zealot` unit is disabled for boot. Only the PP pair (`frank-dragoon` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
|
||||
|
||||
```
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-secondary.service /etc/systemd/system/frank-secondary.service
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-zealot.service /etc/systemd/system/frank-zealot.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable frank-secondary
|
||||
sudo systemctl enable frank-zealot
|
||||
```
|
||||
|
||||
Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).
|
||||
|
||||
The PP split was renamed `frank-pp-head` → `frank-large` (2026-08-16, commit `48142f5`).
|
||||
The PP split was renamed `frank-pp-head` → `frank-dragoon` (2026-08-16, commit `48142f5`).
|
||||
|
||||
Model swap (2026-08-16, commit `50c55ee`): `FLM_MODEL=qwen3.6-moe:35b-a3b`, `FLM_CTX_LEN=262144` (native max), `FLM_PMODE=turbo`, `FLM_ASR=1` kept. Decode measured 12.5-13.1 t/s. The Gemma4 live config above (131072 ctx) applied to `gemma4-it:e4b`. Swap `FLM_MODEL` and `FLM_CTX_LEN` back to restore it.
|
||||
|
||||
@@ -353,7 +353,7 @@ installed) against `/opt/fastflowlm/share/flm/xclbins/` (available).
|
||||
Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.
|
||||
|
||||
1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`.
|
||||
2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-secondary` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-secondary' is not supported.` Each such request also triggered the unknown-tag fallback below.
|
||||
2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-zealot` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-zealot' is not supported.` Each such request also triggered the unknown-tag fallback below.
|
||||
3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load.
|
||||
|
||||
Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16.
|
||||
@@ -390,7 +390,7 @@ pi sessions failed against this endpoint with `Stream ended without finish_reaso
|
||||
- **The FLM serve process is not under systemd.** A reboot of Dankclaw kills it. The PP head service is enabled and returns. The FLM process must be restarted by hand. A unit file is a natural next step.
|
||||
- **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this.
|
||||
- **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
|
||||
- **NPU production role (2026-08-16).** `frank-secondary` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
|
||||
- **NPU production role (2026-08-16).** `frank-zealot` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
|
||||
|
||||
## Citations
|
||||
|
||||
|
||||
@@ -23,7 +23,7 @@ read the returned content:
|
||||
```python
|
||||
import json, urllib.request
|
||||
req = urllib.request.Request("http://<host>:<port>/v1/chat/completions",
|
||||
json.dumps({"model": "frank-secondary", "messages": [{"role": "user",
|
||||
json.dumps({"model": "frank-zealot", "messages": [{"role": "user",
|
||||
"content": "Hi"}]}).encode(), {"Content-Type": "application/json"})
|
||||
print(json.load(urllib.request.urlopen(req, timeout=600)))
|
||||
```
|
||||
|
||||
@@ -119,5 +119,5 @@ re-download.
|
||||
|
||||
- [[Llama Cpp Runtimes]] — which fork ships which custom types
|
||||
- [[Llama Cpp Code Diff]] — the enum tables and code-level differences
|
||||
- [[frank-large-rocmfpx-q8agent-mtp]] — the ftype-115 lock-in in production
|
||||
- [[frank-dragoon-rocmfpx-q8agent-mtp]] — the ftype-115 lock-in in production
|
||||
- [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads
|
||||
|
||||
@@ -13,7 +13,7 @@ Fleet law first: **STOP the production service before any GPU experiment**
|
||||
GPU and corrupts results; an unisolated GPU experiment has precedent for
|
||||
wedging a host until power cycle. Reference harnesses:
|
||||
`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`,
|
||||
`scripts/autoresearch/frank-primary-arms.sh`.
|
||||
`scripts/autoresearch/frank-archon-arms.sh`.
|
||||
|
||||
## 1. The window structure (one config per arm, a driver owns the window)
|
||||
|
||||
@@ -27,7 +27,7 @@ wedging a host until power cycle. Reference harnesses:
|
||||
with `container state improper`.
|
||||
2. **Boot the arm as the service, not a scratch port** (no scratch ports on
|
||||
LLM hosts — one card, one server). The serve script takes its config via
|
||||
env-file override (`FRANK_SECONDARY_ENV_FILE`, `FRANK_PRIMARY_AR_ENV_FILE`)
|
||||
env-file override (`FRANK_ZEALOT_ENV_FILE`, `FRANK_ARCHON_AR_ENV_FILE`)
|
||||
pointing at a staged copy; the checkout is never mutated.
|
||||
3. **Fast-fail the health wait**: poll `podman ps | grep -qx <name>` inside
|
||||
the loop — a refused boot dies in one poll instead of 20 min. Capture the
|
||||
@@ -245,17 +245,17 @@ nearly shipped the wrong setting.
|
||||
the comment); verify multi-line command construction with `bash -n` AND a
|
||||
render test.
|
||||
|
||||
## 8. Results location and recovery (frank-primary arms)
|
||||
## 8. Results location and recovery (frank-archon arms)
|
||||
|
||||
| copy | path |
|
||||
|---|---|
|
||||
| per-arm evidence | `$ARMS_DIR` = `~/.jobs/<run-stamp>/` (`.rc`, `.done` beside it) |
|
||||
| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `<arm>.out/.env/.overlay/.rc/.metrics` |
|
||||
| server-side last window | `/home/dank/.autoresearch-frank-primary/run.json` |
|
||||
| server-side last window | `/home/dank/.autoresearch-frank-archon/run.json` |
|
||||
| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) |
|
||||
| fallback | `~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log` (permanent) |
|
||||
|
||||
`/tmp/frank-primary-arms-*` and `/tmp/armstub/` are stub self-tests —
|
||||
`/tmp/frank-archon-arms-*` and `/tmp/armstub/` are stub self-tests —
|
||||
constants 99.9/88.8 identical across arms by construction; **identical
|
||||
numbers across arms == stub or bug, never a measurement.** After compaction,
|
||||
re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user