Compare commits
101
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f96898d4c5 | ||
|
|
3e140cd077 | ||
|
|
c5ac0725f7 | ||
|
|
f44c39812f | ||
|
|
6dcf4a5913 | ||
|
|
a9d45e7012 | ||
|
|
2581b8d4b6 | ||
|
|
09a0251d2d | ||
|
|
066813ea95 | ||
|
|
7daa886dd6 | ||
|
|
bae78e7dae | ||
|
|
a6f55f4cc6 | ||
|
|
a9da6589a6 | ||
|
|
1a6a78ef73 | ||
|
|
c52ba781e3 | ||
|
|
0bf26b0741 | ||
|
|
394fd8535b | ||
|
|
55f87fcdce | ||
|
|
5ced474eb9 | ||
|
|
2992a7ddd6 | ||
|
|
7c77326c33 | ||
|
|
65dec6a80e | ||
|
|
f99b698d25 | ||
|
|
757a90057f | ||
|
|
009e8a885d | ||
|
|
6b677139ad | ||
|
|
184a5c665a | ||
|
|
958b10545e | ||
|
|
c4dca68a03 | ||
|
|
f29dbdf7c7 | ||
|
|
98f11a19c1 | ||
|
|
a30ac9457a | ||
|
|
75695c312a | ||
|
|
c507f91438 | ||
|
|
b175f4afd1 | ||
|
|
971e23a25c | ||
|
|
3ca64e14fc | ||
|
|
f3dfea13ad | ||
|
|
02fa1e22ae | ||
|
|
1aef7418ad | ||
|
|
fa5a444b64 | ||
|
|
90f678582e | ||
|
|
cc0fdd15a1 | ||
|
|
f148ecab4a | ||
|
|
c2f3411342 | ||
|
|
8c5bde7034 | ||
|
|
22a7331100 | ||
|
|
c1e192df00 | ||
|
|
8375567de3 | ||
|
|
79c2e64c2c | ||
|
|
f20da6a6ff | ||
|
|
058ea6cca8 | ||
|
|
3f8531179e | ||
|
|
3c5d77d1fa | ||
|
|
203b0be145 | ||
|
|
74da746c51 | ||
|
|
7066accc25 | ||
|
|
76aaa2543b | ||
|
|
8220030ba1 | ||
|
|
5191c49967 | ||
|
|
f9330fa201 | ||
|
|
4b71c014e9 | ||
|
|
cee90eea52 | ||
|
|
73c37f4301 | ||
|
|
6632d0a2f6 | ||
|
|
caf5e1e0ca | ||
|
|
5278f5be26 | ||
|
|
ec6184de9a | ||
|
|
1d062f06a3 | ||
|
|
b51e078e36 | ||
|
|
68a525740d | ||
|
|
5b7b3ccb90 | ||
|
|
2137080d1c | ||
|
|
f747cb0dab | ||
|
|
db8faa68ff | ||
|
|
cf6c62d44a | ||
|
|
0aafad5f2c | ||
|
|
a3f45ce028 | ||
|
|
45de2fd47f | ||
|
|
fa858f4c10 | ||
|
|
07af1a9366 | ||
|
|
f0d0614ce6 | ||
|
|
9aed9987b6 | ||
|
|
4048dc8630 | ||
|
|
9bc40fd899 | ||
|
|
bb60e8524f | ||
|
|
319dd0f7da | ||
|
|
650dfff5a2 | ||
|
|
de70320cde | ||
|
|
910c54b6ff | ||
|
|
8c887e6fc9 | ||
|
|
49dd2be4cd | ||
|
|
a11f703c49 | ||
|
|
6cdf88f611 | ||
|
|
259fc5505d | ||
|
|
3e46d51df8 | ||
|
|
d062a376b2 | ||
|
|
95219cf4c1 | ||
|
|
fee66187a1 | ||
|
|
0258ae0897 | ||
|
|
9aee00f2b3 |
No files matched your search
@@ -1,4 +1,4 @@
|
|||||||
# Tunables for frank-large (kingjones777/Qwen3.8-Flash-Next-ROCmFP4-
|
# Tunables for frank-dragoon (kingjones777/Qwen3.8-Flash-Next-ROCmFP4-
|
||||||
# STRIX-GGUF on the kingjones30/ROCmFPX llama.cpp runtime, Dankclaw,
|
# STRIX-GGUF on the kingjones30/ROCmFPX llama.cpp runtime, Dankclaw,
|
||||||
# Strix Halo gfx1151, unified memory).
|
# Strix Halo gfx1151, unified memory).
|
||||||
#
|
#
|
||||||
@@ -56,7 +56,7 @@ CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
|
|||||||
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}'
|
CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}'
|
||||||
|
|
||||||
# Alias + network.
|
# Alias + network.
|
||||||
ALIAS=frank-large
|
ALIAS=frank-dragoon
|
||||||
BIND_ADDR=0.0.0.0
|
BIND_ADDR=0.0.0.0
|
||||||
LISTEN_PORT=7770
|
LISTEN_PORT=7770
|
||||||
|
|
||||||
+3
-3
@@ -1,5 +1,5 @@
|
|||||||
[Unit]
|
[Unit]
|
||||||
Description=ROCmFPX (kingjones30 fork) llama.cpp Qwen3.8-Flash-Next ROCmFP4 STRIX 177B MoE (dankclaw, frank-large)
|
Description=ROCmFPX (kingjones30 fork) llama.cpp Qwen3.8-Flash-Next ROCmFP4 STRIX 177B MoE (dankclaw, frank-dragoon)
|
||||||
After=network-online.target
|
After=network-online.target
|
||||||
Wants=network-online.target
|
Wants=network-online.target
|
||||||
|
|
||||||
@@ -7,8 +7,8 @@ Wants=network-online.target
|
|||||||
Type=simple
|
Type=simple
|
||||||
User=dank
|
User=dank
|
||||||
Group=dank
|
Group=dank
|
||||||
Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
|
Environment="FRANK_DRAGOON_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-dragoon.env"
|
||||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
|
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-dragoon.sh
|
||||||
Restart=on-failure
|
Restart=on-failure
|
||||||
RestartSec=10
|
RestartSec=10
|
||||||
|
|
||||||
@@ -6,7 +6,7 @@
|
|||||||
# is reachable on this port regardless of the boot tag. The NPU is a
|
# is reachable on this port regardless of the boot tag. The NPU is a
|
||||||
# single-lane device: generations serialize FIFO, and the first request after a
|
# single-lane device: generations serialize FIFO, and the first request after a
|
||||||
# model swap pays the ~6-11 s load cost. The NPU draws from the unified RAM
|
# model swap pays the ~6-11 s load cost. The NPU draws from the unified RAM
|
||||||
# pool shared with frank-large (iGPU).
|
# pool shared with frank-dragoon (iGPU).
|
||||||
FLM_MODEL=nuextract3:4b
|
FLM_MODEL=nuextract3:4b
|
||||||
FLM_HOST=0.0.0.0
|
FLM_HOST=0.0.0.0
|
||||||
FLM_PORT=7771
|
FLM_PORT=7771
|
||||||
|
|||||||
+3
-3
@@ -1,6 +1,6 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Serve kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF as
|
# Serve kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF as
|
||||||
# frank-large via the kingjones30/ROCmFPX llama.cpp runtime on
|
# frank-dragoon via the kingjones30/ROCmFPX llama.cpp runtime on
|
||||||
# Dankclaw (Strix Halo gfx1151, unified memory).
|
# Dankclaw (Strix Halo gfx1151, unified memory).
|
||||||
#
|
#
|
||||||
# Runtime: kingjones30/ROCmFPX (fork of charlie12345/ROCmFPX with the
|
# Runtime: kingjones30/ROCmFPX (fork of charlie12345/ROCmFPX with the
|
||||||
@@ -29,7 +29,7 @@
|
|||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
|
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-dragoon.env}"
|
||||||
if [[ -f "$ENV_FILE" ]]; then
|
if [[ -f "$ENV_FILE" ]]; then
|
||||||
set -a
|
set -a
|
||||||
# shellcheck disable=SC1090
|
# shellcheck disable=SC1090
|
||||||
@@ -44,7 +44,7 @@ fi
|
|||||||
: "${MMPROJ_PATH:=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf}"
|
: "${MMPROJ_PATH:=/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/mmproj/mmproj-Qwen3.8-Flash-Next-f16.gguf}"
|
||||||
: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
|
: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
|
||||||
: "${CHAT_TEMPLATE_KWARGS:={}}"
|
: "${CHAT_TEMPLATE_KWARGS:={}}"
|
||||||
: "${ALIAS:=frank-large}"
|
: "${ALIAS:=frank-dragoon}"
|
||||||
: "${BIND_ADDR:=0.0.0.0}"
|
: "${BIND_ADDR:=0.0.0.0}"
|
||||||
: "${LISTEN_PORT:=7770}"
|
: "${LISTEN_PORT:=7770}"
|
||||||
: "${CTX_SIZE:=401408}"
|
: "${CTX_SIZE:=401408}"
|
||||||
@@ -4,7 +4,7 @@
|
|||||||
# Runs `flm serve <tag>` on the XDNA2 NPU (50 TOPS, Dankclaw). The NPU is a
|
# Runs `flm serve <tag>` on the XDNA2 NPU (50 TOPS, Dankclaw). The NPU is a
|
||||||
# single-lane device: generations serialize in FIFO order, so this is a
|
# single-lane device: generations serialize in FIFO order, so this is a
|
||||||
# background/triage endpoint, not an interactive multi-user chat. The NPU
|
# background/triage endpoint, not an interactive multi-user chat. The NPU
|
||||||
# draws from the unified RAM pool shared with frank-large (the iGPU
|
# draws from the unified RAM pool shared with frank-dragoon (the iGPU
|
||||||
# llama.cpp service); see serverAppConfigs-docs/LLM/frank-npu.md
|
# llama.cpp service); see serverAppConfigs-docs/LLM/frank-npu.md
|
||||||
# for the memory budget and the model-selection tradeoff.
|
# for the memory budget and the model-selection tradeoff.
|
||||||
#
|
#
|
||||||
|
|||||||
@@ -9,11 +9,11 @@ services run from `/home/dank/serverAppConfigs` on each host (the 2026-08-22
|
|||||||
merge absorbed the former `vllmScripts` repo, history preserved).
|
merge absorbed the former `vllmScripts` repo, history preserved).
|
||||||
|
|
||||||
- **Dankserver** (192.168.0.235) — media/library stack (RomM, Copyparty,
|
- **Dankserver** (192.168.0.235) — media/library stack (RomM, Copyparty,
|
||||||
Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
|
Calibre-Web, Jellyfin, Plex) plus the `frank-archon` LLM service.
|
||||||
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
|
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
|
||||||
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
|
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
|
||||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (flash-server
|
Alloy, Pi-hole) plus TWO LLM services: `frank-dragoon` (flash-server
|
||||||
container on the gfx1151 iGPU) and `frank-secondary` (llama.cpp CUDA on the
|
container on the gfx1151 iGPU) and `frank-zealot` (llama.cpp CUDA on the
|
||||||
RTX 3090). Caddy on
|
RTX 3090). Caddy on
|
||||||
Dankclaw reverse-proxies both hosts.
|
Dankclaw reverse-proxies both hosts.
|
||||||
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
|
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
|
||||||
@@ -25,9 +25,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
|||||||
|
|
||||||
| Target | Service | Runtime | Model | Port |
|
| Target | Service | Runtime | Model | Port |
|
||||||
|--------|---------|---------|-------|------|
|
|--------|---------|---------|-------|------|
|
||||||
| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *…
|
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | **Publisher payload** (swap 2026-10-15, MODEL_FILE-only): `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` → **`qwen3.8-next-flash-fp8-iq4r-moe.rad`** (121,969,901,568 B = **116,322 MiB**, Hub-LFS sha256 `0af5e962…` verified at fetch by `scripts/download-frank-archon-iq4r.sh`) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as **4-bit codes into a 16-level non-uniform codebook** (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), **10 whole bf16-protected experts/layer**, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), **MTP head + bf16 vision tower in the container**; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. **Base-alignment (censored) model** — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: **3 × 200,000 fp8-KV** (`--max-num-seqs 3`), **25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable**, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the **w4dn8 record** (the bar to beat; new-payload probes pending). Rollback: revert the swap commit (w4dn8 `.rad` on disk); the fp8-native `.rad` + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts). | 7770 |
|
||||||
| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the head per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 200,192 ctx** (KV pool 400,384 positions ≈ 10.7 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold) | 7770 |
|
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
|
||||||
| Dankclaw | `frank-secondary` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp** (27B dense qwen3_5 hybrid, 10,442,827,392 B; a per-tensor allocation mix of stock types; the MTP head in the file **drafted** — **spec ON** (draft-mtp depth 3) since 2026-10-06 evening, ngram skipped on the recurrent-rollback law) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **2 × 114,688** (Frank's 112k cap each since the 2026-10-06 late reshape, cut from 131,072 to fund the **ubatch ladder 512 → 1024 → 2048** — **1,005 MiB free at boot, 763 settled after a vision probe** (encoder buffers resident)) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / **ubatch 2048** (the ceiling for this shape — 4096 would not fit); **prefill 1,209 t/s @ 3,760 tokens; ~52–58 t/s recipe decode** (MTP acceptance 34–54 %, temp-1.0 noise; decode is ubatch-invariant). Renamed from `frank-small` 2026-10-06 (87721ab). Supersedes the same-morning IQ3_S-mtp payloads: the two-slot no-spec shape and the one-slot MTP shape (both 2026-10-06; payload on disk).|7771|
|
| Dankclaw | `frank-zealot` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
|
||||||
|
|
||||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||||
the [runbook index](#runbook-index). The full swap history lives in
|
the [runbook index](#runbook-index). The full swap history lives in
|
||||||
@@ -66,14 +66,14 @@ All edits flow through git. Never edit files on a remote machine directly.
|
|||||||
./scripts/deploy.sh dankclaw
|
./scripts/deploy.sh dankclaw
|
||||||
./scripts/deploy.sh dankserver
|
./scripts/deploy.sh dankserver
|
||||||
./scripts/deploy.sh dankclaw --service searxng
|
./scripts/deploy.sh dankclaw --service searxng
|
||||||
./scripts/deploy.sh dankserver --service frank-primary
|
./scripts/deploy.sh dankserver --service frank-archon
|
||||||
./scripts/deploy.sh dankclaw --dry-run
|
./scripts/deploy.sh dankclaw --dry-run
|
||||||
```
|
```
|
||||||
|
|
||||||
The planner maps changed paths to services; the remote runner restarts only
|
The planner maps changed paths to services; the remote runner restarts only
|
||||||
the selected service and records its revision outside the checkout.
|
the selected service and records its revision outside the checkout.
|
||||||
Documentation-only changes are a no-op. LLM services are selected only by
|
Documentation-only changes are a no-op. LLM services are selected only by
|
||||||
their own files — every endpoint (`frank-large`/`frank-secondary`/`frank-primary`)
|
their own files — every endpoint (`frank-dragoon`/`frank-zealot`/`frank-archon`)
|
||||||
by its unit/env/serve trio in `scripts/` — so
|
by its unit/env/serve trio in `scripts/` — so
|
||||||
unrelated
|
unrelated
|
||||||
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
|
container changes cannot interrupt inference. Set `FRANK_SUDO_PW` for headless
|
||||||
@@ -85,7 +85,7 @@ restarts each service whose marker is behind HEAD *and* whose paths changed
|
|||||||
since that marker — so a backlog accumulated by service-scoped deploys fires
|
since that marker — so a backlog accumulated by service-scoped deploys fires
|
||||||
all at once, and one failing health gate aborts the run with every later
|
all at once, and one failing health gate aborts the run with every later
|
||||||
service left pending. Name the service unless a drain is what you want
|
service left pending. Name the service unless a drain is what you want
|
||||||
(`./scripts/deploy.sh dankserver --service frank-primary`).
|
(`./scripts/deploy.sh dankserver --service frank-archon`).
|
||||||
|
|
||||||
### Deployment completeness (MANDATORY)
|
### Deployment completeness (MANDATORY)
|
||||||
|
|
||||||
@@ -229,17 +229,17 @@ archive.
|
|||||||
**STOP the production service BEFORE benchmarking** — these servers serve
|
**STOP the production service BEFORE benchmarking** — these servers serve
|
||||||
real users, and a bench beside a live service starves the GPU and corrupts
|
real users, and a bench beside a live service starves the GPU and corrupts
|
||||||
results. Use the runtime's own harness: `llama-bench` for the llama.cpp
|
results. Use the runtime's own harness: `llama-bench` for the llama.cpp
|
||||||
endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-large and frank-secondary were measured. Rolling your own bench
|
endpoints; a live llama.cpp server also reports honest per-request numbers in the `/completion` `timings` object (prefill/decode rates, `draft_n`, `draft_n_accepted`), which is how frank-dragoon and frank-zealot were measured. Rolling your own bench
|
||||||
requires asking first.
|
requires asking first.
|
||||||
Scratch ports are not used on LLM hosts — a
|
Scratch ports are not used on LLM hosts — a
|
||||||
test process cannot share the single GPU anyway; validate against the service
|
test process cannot share the single GPU anyway; validate against the service
|
||||||
itself.
|
itself.
|
||||||
|
|
||||||
(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05, the
|
(A gufo-era law; frank-dragoon ran ROCmFPX llama.cpp 2026-10-02 → 10-05, the
|
||||||
halogen 27B container 2026-10-05 → 10-06, and is flash-server since, but the
|
halogen 27B container 2026-10-05 → 10-06, and is flash-server since, but the
|
||||||
rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode
|
rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode
|
||||||
settings, and will invert the ranking.**
|
settings, and will invert the ranking.**
|
||||||
Measured 2026-09-28 on frank-large at depth 32768: its corpus
|
Measured 2026-09-28 on frank-dragoon at depth 32768: its corpus
|
||||||
reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
|
reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
|
||||||
draft widths, i.e. the same highly-predictable text every run, and 93 % of
|
draft widths, i.e. the same highly-predictable text every run, and 93 % of
|
||||||
every proposal accepted. It reported 29–37 t/s decode where real traffic
|
every proposal accepted. It reported 29–37 t/s decode where real traffic
|
||||||
@@ -264,12 +264,12 @@ one: `ip neigh show <ip>` from a peer — FAILED means off/NIC dead, while a
|
|||||||
merely wedged kernel usually still answers ARP.
|
merely wedged kernel usually still answers ARP.
|
||||||
|
|
||||||
1. Stop and mask (self-healing would restart it):
|
1. Stop and mask (self-healing would restart it):
|
||||||
- Dankclaw `frank-large` (root unit, halogen podman on the iGPU):
|
- Dankclaw `frank-dragoon` (root unit, halogen podman on the iGPU):
|
||||||
`sudo systemctl stop frank-large && sudo systemctl mask frank-large`
|
`sudo systemctl stop frank-dragoon && sudo systemctl mask frank-dragoon`
|
||||||
- Dankclaw `frank-secondary` (root unit, llama.cpp on the 3090):
|
- Dankclaw `frank-zealot` (root unit, llama.cpp on the 3090):
|
||||||
`sudo systemctl stop frank-secondary && sudo systemctl mask frank-secondary`
|
`sudo systemctl stop frank-zealot && sudo systemctl mask frank-zealot`
|
||||||
- Dankserver `frank-primary` (root unit, radiance on both R9700s, TP=2):
|
- Dankserver `frank-archon` (root unit, radiance on both R9700s, TP=2):
|
||||||
`sudo systemctl stop frank-primary && sudo systemctl mask frank-primary`
|
`sudo systemctl stop frank-archon && sudo systemctl mask frank-archon`
|
||||||
2. Bench.
|
2. Bench.
|
||||||
NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
|
NOTE: `systemctl mask` FAILS on a relinked unit (the unit file in
|
||||||
`/etc/systemd/system/` is itself a symlink into the checkout — mask
|
`/etc/systemd/system/` is itself a symlink into the checkout — mask
|
||||||
@@ -286,9 +286,18 @@ deletes the unit file from `/etc/systemd/system/`, not just the symlink.
|
|||||||
The harness reaches the fleet through `.omp/agent/models.yml` in the
|
The harness reaches the fleet through `.omp/agent/models.yml` in the
|
||||||
operator's home — `/home/blank/.omp/agent/models.yml` on the dev box, which is
|
operator's home — `/home/blank/.omp/agent/models.yml` on the dev box, which is
|
||||||
where the harness runs; there is no copy on the servers. Each provider is one
|
where the harness runs; there is no copy on the servers. Each provider is one
|
||||||
HOST:PORT; each model `id` is the served alias: `frank-primary` →
|
HOST:PORT; each model `id` is the served alias. **Official model ids are
|
||||||
`192.168.0.235:7770`, `frank-large` → `192.168.0.143:7770`, `frank-secondary` →
|
the StarCraft unit names (2026-10-08), on the wire since 2026-10-09, and
|
||||||
`modelRoles`.
|
since 2026-10-10 they are also the SERVICE names:** `frank-archon` →
|
||||||
|
`192.168.0.235:7770` (`FRANK_ARCHON_SERVED_NAME`), `frank-dragoon` →
|
||||||
|
`192.168.0.143:7770` (`HALOGEN_MODEL_ID`), `frank-zealot` →
|
||||||
|
`192.168.0.143:7771` (`ALIAS`). The 2026-10-10 cutover renamed the units,
|
||||||
|
scripts, env files, containers, deploy services and runbooks to match —
|
||||||
|
the three former service names are fully retired and exist nowhere in
|
||||||
|
the repo; historical text in runbooks uses the current names. Verify a
|
||||||
|
rename on `/v1/models`, never
|
||||||
|
on a chat response: radiance echoes the **requested** model name, so a 200
|
||||||
|
proves nothing about the served id (llama.cpp and halogen ignore it).
|
||||||
|
|
||||||
Resolution rules: the default model resolves as a `defaultProvider` +
|
Resolution rules: the default model resolves as a `defaultProvider` +
|
||||||
`defaultModel` pair (the provider must serve the model); a subagent override
|
`defaultModel` pair (the provider must serve the model); a subagent override
|
||||||
@@ -301,18 +310,20 @@ provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
|
|||||||
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
|
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
|
||||||
a role pointing at a provider the file does not define fails at call time. As
|
a role pointing at a provider the file does not define fails at call time. As
|
||||||
of 2026-10-06 **all three endpoints accept images** (`input: [text,
|
of 2026-10-06 **all three endpoints accept images** (`input: [text,
|
||||||
image]`) — `frank-large` returned to vision with the flash-server v2 swap
|
image]`) — `frank-dragoon` returned to vision with the flash-server v2 swap
|
||||||
(the v2 vision tower loads beside the checkpoint; the text-only halogen
|
(the v2 vision tower loads beside the checkpoint; the text-only halogen
|
||||||
27B window closed 2026-10-06) — while
|
27B window closed 2026-10-06) — while
|
||||||
`frank-primary` takes them natively (radiance runs the checkpoint's
|
`frank-archon` takes them natively (radiance runs the checkpoint's
|
||||||
27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary
|
27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary
|
||||||
are the engine core; `--mm-max-patches` available),
|
are the engine core; `--mm-max-patches` available),
|
||||||
and `frank-secondary` through
|
and `frank-zealot` through
|
||||||
the publisher's own `mmproj-BF16.gguf` passed as `--mmproj` — the Swift
|
the publisher's own BF16 mmproj passed as `--mmproj` — the model is natively
|
||||||
GGUF itself carries no vision tensors. `frank-secondary`'s `contextWindow` is
|
multimodal but the Q6_K GGUF carries no vision tensors (same arrangement the
|
||||||
**114688** (cut from 131072 on 2026-10-06 late to fund the ubatch ladder to 2048), the
|
Gemma4 era used). `frank-zealot`'s `contextWindow` is
|
||||||
per-slot window, not the model's native 262144: the harness must compact
|
**200704** — the PER-SLOT window (`--ctx-size 401408 --parallel 2` since the
|
||||||
before llama.cpp answers a longer prompt with a 400. Note also
|
2026-10-07 Qwen3.5-9B swap), not the pool and not the model's native 262144:
|
||||||
|
the harness must compact before llama.cpp answers a longer prompt with a 400.
|
||||||
|
Note also
|
||||||
that
|
that
|
||||||
the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
|
the reasoning channel differs by runtime: llama.cpp AND SGLang (qwen3
|
||||||
parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
|
parser, live-verified) emit `reasoning_content`, vLLM emits `reasoning` —
|
||||||
@@ -321,22 +332,22 @@ There is **no audio**
|
|||||||
input anywhere in the fleet: that capability left with the FLM NPU
|
input anywhere in the fleet: that capability left with the FLM NPU
|
||||||
endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
|
endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
|
||||||
audio encoder, and left again with the Gemma4 retirement on 2026-10-01
|
audio encoder, and left again with the Gemma4 retirement on 2026-10-01
|
||||||
(the MiMo tower is vision-only).
|
(neither the MiMo nor the Qwen3.5 tower carries audio).
|
||||||
|
|
||||||
### frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, ACTIVE)
|
### frank-dragoon — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, ACTIVE)
|
||||||
|
|
||||||
The **flash-server engine** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`,
|
The **flash-server engine** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`,
|
||||||
digest-pinned, published tag 0.16.2, 3.6 GB image) as a podman container
|
digest-pinned, published tag 0.16.2, 3.6 GB image) as a podman container
|
||||||
driven by `scripts/serve-frank-large.sh` under a **root systemd unit**
|
driven by `scripts/serve-frank-dragoon.sh` under a **root systemd unit**
|
||||||
(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
|
(`scripts/frank-dragoon.service`, `User=dank`, `Restart=always`,
|
||||||
`ExecStop=podman stop -t 60 frank-large-halogen`, the script runs `podman
|
`ExecStop=podman stop -t 60 frank-dragoon-halogen`, the script runs `podman
|
||||||
run` FOREGROUND — the container's exit code is the service's exit so
|
run` FOREGROUND — the container's exit code is the service's exit so
|
||||||
`Restart=always` stays real, and the engine ledger lands on disk) on port
|
`Restart=always` stays real, and the engine ledger lands on disk) on port
|
||||||
7770 from the Strix Halo iGPU. Engine stdout goes to
|
7770 from the Strix Halo iGPU. Engine stdout goes to
|
||||||
`/home/dank/logs/frank_large.log` — **the boot ledger (weights resident, KV
|
`/home/dank/logs/frank_dragoon.log` — **the boot ledger (weights resident, KV
|
||||||
pool, MTP head, prompt cache, any clamp) lives there, not in the journal**.
|
pool, MTP head, prompt cache, any clamp) lives there, not in the journal**.
|
||||||
Cutover 2026-10-06 from the halogen 27B-dense era
|
Cutover 2026-10-06 from the halogen 27B-dense era
|
||||||
([[frank-large-halogen27b-dflash2]], retired the same window).
|
([[frank-dragoon-halogen27b-dflash2]], retired the same window).
|
||||||
|
|
||||||
Serving `Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`:
|
Serving `Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`:
|
||||||
`qwen38-flash-next-v2-swift15-abliterated.hgn` (66,687,678,464 B ≈ 62.1
|
`qwen38-flash-next-v2-swift15-abliterated.hgn` (66,687,678,464 B ≈ 62.1
|
||||||
@@ -358,22 +369,29 @@ sidecar is deliberately not fetched).
|
|||||||
**Product facts vs the 27B era, both regressions FIXED:** the base is
|
**Product facts vs the 27B era, both regressions FIXED:** the base is
|
||||||
**abliterated** (the 27B was a censored-base finetune; refusals were an
|
**abliterated** (the 27B was a censored-base finetune; refusals were an
|
||||||
accepted trade there) and **vision is ON** (the 27B engine silently
|
accepted trade there) and **vision is ON** (the 27B engine silently
|
||||||
ignored its encoder — frank-secondary owned images for exactly one day at this
|
ignored its encoder — frank-zealot owned images for exactly one day at this
|
||||||
endpoint).
|
endpoint).
|
||||||
|
|
||||||
**Engine laws that shape the config** (all in `scripts/frank-large.env`
|
**Engine laws that shape the config** (all in `scripts/frank-dragoon.env`
|
||||||
with evidence): `HALOGEN_SPEC_ADAPT=0` is MANDATORY — the default adaptive
|
with evidence): `HALOGEN_SPEC_ADAPT=0` is MANDATORY — the default adaptive
|
||||||
policy disables the MTP head for whole requests after low acceptance and
|
policy disables the MTP head for whole requests after low acceptance and
|
||||||
HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance
|
HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance
|
||||||
for it (accepted). `HALOGEN_CTX=200192` is the PER-SLOT hard ceiling (400
|
for it (accepted). `HALOGEN_CTX=262144` is the PER-SLOT hard ceiling (400
|
||||||
naming the limit, never truncation); `models.yml` `contextWindow: 200192`
|
naming the limit, never truncation) — RAISED from 200192 on 2026-10-07
|
||||||
must match, and compaction at 70 % = 140,134 + 57,344 budget = 197,478 fits.
|
under the 400-zone law: the harness clamps max_tokens to
|
||||||
|
`min(45056, contextWindow − est − 4096)`, its chars/4 est UNDER-counts
|
||||||
|
the rendered prompt by ~3 % (measured: est 152,938 → server 157,388,
|
||||||
|
request 43,158 vs room 42,804), so a harness window == engine ceiling
|
||||||
|
made every request over ~146k est a guaranteed 400. `models.yml`
|
||||||
|
`contextWindow: 217088` leaves the 45k gap that absorbs the estimate
|
||||||
|
error (worst-case request ≈ 219k ≤ 262144); compaction fires at 85 % of
|
||||||
|
the harness window = 184,524 est ≈ 190k measured (Frank's 180k target).
|
||||||
`HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter"
|
`HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter"
|
||||||
law does NOT carry over; batching and speculation coexist (the card's
|
law does NOT carry over; batching and speculation coexist (the card's
|
||||||
quickstart ships 2 slots with the head), so 2 is concurrent capacity.
|
quickstart ships 2 slots with the head), so 2 is concurrent capacity.
|
||||||
`HALOGEN_KV_POOL_POSITIONS=400384` (explicit 2 × ctx; ~28 KiB/position ⇒
|
`HALOGEN_KV_POOL_POSITIONS=524288` (explicit 2 × ctx; ~28 KiB/position ⇒
|
||||||
~10.7 GiB). `HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the
|
14.0 GiB). `HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the
|
||||||
harness always sends 57,344); `HALOGEN_MAX_TOKENS_CAP=65536` +
|
harness sends at most its `maxTokens` 45056); `HALOGEN_MAX_TOKENS_CAP=65536` +
|
||||||
`HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table).
|
`HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table).
|
||||||
**A server-side sampler EXISTS** (`HALOGEN_TEMPERATURE/TOP_P/TOP_K` =
|
**A server-side sampler EXISTS** (`HALOGEN_TEMPERATURE/TOP_P/TOP_K` =
|
||||||
1.0/0.95/20, the card's thinking recipe) — an absent temperature is no
|
1.0/0.95/20, the card's thinking recipe) — an absent temperature is no
|
||||||
@@ -389,14 +407,14 @@ stays mandatory. Thinking tokens COUNT against the budget (empty content +
|
|||||||
"video".
|
"video".
|
||||||
|
|
||||||
**Memory ledger** (121.2 GiB unified; the referee is the boot ledger):
|
**Memory ledger** (121.2 GiB unified; the referee is the boot ledger):
|
||||||
weights ~62 + KV ~10.7 + arena ~8 (`3.8 + 8.4 × MAX_TOK/16384`) + vision
|
weights ~62 + KV 14.0 + arena ~8 (`3.8 + 8.4 × MAX_TOK/16384`) + vision
|
||||||
~2 ≈ 82.7 GiB engine-side + `HALOGEN_HOST_RESERVE_GIB=20` → ~18 GiB
|
~2 ≈ 86 GiB engine-side + `HALOGEN_HOST_RESERVE_GIB=20` → ~15 GiB
|
||||||
margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd
|
margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd
|
||||||
page cache beside the checkpoint — the card's quickstart arithmetic does
|
page cache beside the checkpoint — the card's quickstart arithmetic does
|
||||||
not bill it as resident, but READ the boot ledger and re-size if the
|
not bill it as resident, but READ the boot ledger and re-size if the
|
||||||
engine prints a clamp.
|
engine prints a clamp.
|
||||||
|
|
||||||
Fetch: `scripts/download-frank-large-flashnext.sh` into
|
Fetch: `scripts/download-frank-dragoon-flashnext.sh` into
|
||||||
`/home/dank/models/halogen-flashnext-v2/` (`-f` + exact byte counts + LFS
|
`/home/dank/models/halogen-flashnext-v2/` (`-f` + exact byte counts + LFS
|
||||||
sha256 + `--verify-only`, no token on the command line; resume goes
|
sha256 + `--verify-only`, no token on the command line; resume goes
|
||||||
against the RESOLVED CDN URL because curl drops `Range` across HF's 302).
|
against the RESOLVED CDN URL because curl drops `Range` across HF's 302).
|
||||||
@@ -405,164 +423,203 @@ n-gram + vision tower, flat tokenizer with all six files as regular
|
|||||||
files) — the 2026-10-01 crash-loop lesson.
|
files) — the 2026-10-01 crash-loop lesson.
|
||||||
|
|
||||||
Runbook:
|
Runbook:
|
||||||
`serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md`.
|
`serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md`.
|
||||||
Rollback: revert the swap commit and redeploy — the 27B payload
|
Rollback: revert the swap commit and redeploy — the 27B payload
|
||||||
(`/home/dank/models/halogen-qwen3.8-27b/`), its fetch script and the
|
(`/home/dank/models/halogen-qwen3.8-27b/`), its fetch script and the
|
||||||
halogen 0.1.4 image digest all remain on disk; the ROCmFPX door behind
|
halogen 0.1.4 image digest all remain on disk; the ROCmFPX door behind
|
||||||
that (`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched.
|
that (`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched.
|
||||||
|
|
||||||
### frank-secondary — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
### frank-zealot — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||||
|
|
||||||
**llama.cpp master** (0.5.0-dev, commit `5fc4f3c`; CUDA sm_86 build
|
**llama.cpp master** (0.5.0-dev, commit `5fc4f3c`; CUDA sm_86 build
|
||||||
`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`,
|
`/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server`,
|
||||||
`GGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit**
|
`GGML_CUDA_FA_ALL_QUANTS=ON`) as a **root systemd unit**
|
||||||
(`scripts/frank-secondary.service`, `relink_unit`ed into
|
(`scripts/frank-zealot.service`, `relink_unit`ed into
|
||||||
`/etc/systemd/system/`), serving `ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`
|
`/etc/systemd/system/`), serving
|
||||||
at **RCO-IQ3_XXS -mtp** (10,442,827,392 B, sha256 `e13d3e74…`) **plus
|
`HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive` at **Q6_K**
|
||||||
the publisher's own `mmproj-BF16.gguf`** (931,145,888 B, sha256
|
(7,359,259,008 B, sha256 `c0ba7beb…` computed locally — the Hub carries no
|
||||||
`19acd7fc…`), both `/home/dank/models/Swift15-UncIQ3XXS/`, on port 7771.
|
LFS oid) **plus the publisher's own BF16 mmproj** (921,704,448 B), both
|
||||||
Cutover 2026-10-05 (evening) on Frank's explicit contract: **128k
|
`/home/dank/models/Qwen35-9B-HauhauCS-Q6K/`, on port 7771. Cutover
|
||||||
context cap, KV at FP8, vision on, as many 128k contexts as fit**;
|
2026-10-07 (late) from the Gemma4-26B-A4B Q4_K_M 1×200,704 shape (same day).
|
||||||
reshaped three times on 2026-10-06 — morning to two 131,072 slots
|
|
||||||
no-spec, evening to this UNCERSED RCO-IQ3_XXS payload with MTP back
|
|
||||||
on, late the RENAME to **frank-secondary** (87721ab) at two 114,688
|
|
||||||
slots with the ubatch ladder climbing 512 → 1024 → 2048 (f2575a1,
|
|
||||||
d8da7a9). The
|
|
||||||
same-day payloads it supersedes: the LIVE MiMo-V2.6 Q6_K shape and the
|
|
||||||
Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never
|
|
||||||
deployed (blocked at the docs gate; the docs debt is what this section
|
|
||||||
settles).
|
|
||||||
|
|
||||||
**Architecture: qwen3_5 hybrid, bigger** — block_count 65 (64 real layers +
|
**Architecture: qwen35 dense hybrid** — read from the GGUF's own header, not
|
||||||
the blk.64 MTP head), `full_attention_interval 4` ⇒ **16 full-attention +
|
the model card: 32 layers, `full_attention_interval = 4` ⇒ **8 full-attention
|
||||||
48 Gated DeltaNet**, native 262,144 window, dense. KV exists only in the 16
|
layers (3, 7, 11, …, 31) + 24 Gated DeltaNet linear layers**, 16 attention
|
||||||
attention layers (4 KV heads × 256 dim): 64 KiB/token at f16, **34 KiB/token
|
heads / **4 KV heads** × 256 dim, `ssm` conv_kernel 4 / state_size 128 /
|
||||||
at q8_0**. The DeltaNet state is a fixed per-slot buffer (~150 MiB/slot at
|
group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144, vocab
|
||||||
48 linear layers × 48 value heads). Header facts were read from the Hub
|
248,320. Tensor types are Q6_K × 250 + F32 × 177 — standard quants only, so
|
||||||
file by HTTP range + struct parse before anything was downloaded.
|
the stock master build loads it (arch `qwen35` is registered in this build's
|
||||||
|
`src/llama-arch.cpp`). **KV accounting:** the pool scales with the 8 full
|
||||||
|
layers only — 8 × 4 × 256 = 8,192 elements/token per cache, so f16 K + f16 V
|
||||||
|
bills **32 KiB/token** = 12.25 GiB at the 401,408 pool; the 24 DeltaNet
|
||||||
|
layers bill a fixed recurrent state per SLOT, not per token, which is exactly
|
||||||
|
why two 200k slots fit where the 26B MoE fit one.
|
||||||
|
|
||||||
**The shape (2026-10-06 midnight; Frank's tight-fit order, MTP back on
|
**The shape (2026-10-07):** `--ctx-size 401408 --parallel 2` = TWO
|
||||||
in the evening, per-slot cut + ubatch climbed to 2048):** `--ctx-size
|
conversations of 200,704 (Frank's cap; the harness `contextWindow` quotes
|
||||||
229376 --parallel 2` = **TWO 114,688-token conversations** (Frank's
|
the per-slot number), `--flash-attn on`, **f16 K + f16 V** (exact KV),
|
||||||
112k cap each; `n_ctx_slot = CTX_SIZE / PARALLEL`; cut from 131,072 to
|
batch 4096 / **ubatch 2048** (the measured CUDA optimum: +4.6 % prefill,
|
||||||
fund the ubatch rungs), `--flash-attn on` pinned, **q8_0 K + q8_0 V**
|
decode flat, 616 MiB — MiMo-era A/B on this card), `--log-verbosity 4`,
|
||||||
(llama.cpp's FP8 class — no true fp8 KV dtype exists; 8.5 effective
|
`--mmproj` for vision, **speculative decoding OFF** (`SPEC_TYPE=none`).
|
||||||
bits; supersedes the same-day f16/f16 rule written for the Ornith 9B
|
|
||||||
fit), all layers on the GPU, `--mmproj` for vision, batch 4096 /
|
|
||||||
ubatch 2048. **Speculative decoding is ON** (draft-mtp, depth 3): the
|
|
||||||
-mtp file's head is drafted against the main model's own weights;
|
|
||||||
ngram-mod skipped on purpose — on a qwen35 hybrid ngram disables
|
|
||||||
recurrent-state rollback, so a rejected draft corrupts the DeltaNet
|
|
||||||
state.
|
|
||||||
|
|
||||||
**Measured at the midnight boot (2026-10-06, 2 × 114,688 / ubatch
|
**Why MTP is off**: it is not a choice — **this Q6_K file carries no
|
||||||
2048): 23,120 used / 1,005 MiB free, settling 763 MiB after a vision
|
`nextn`/`mtp` tensors** (verified from the tensor-name list), and MTP is a
|
||||||
probe** (`nvidia-smi` free; the encoder buffers stay resident on this
|
property of the file, not of the flags. The publisher card advertises "MTP
|
||||||
boot) — the per-slot cut freed ~1.05 GiB of KV and both ubatch rungs
|
support" for the base architecture; the quantized export dropped the head.
|
||||||
spent it back (~0.3 + ~0.75 GiB measured; the 1024 rung had landed
|
|
||||||
22,374/1,751, the 2048 rung cost 746 MiB, ~130 over the 616 the rung
|
|
||||||
measured on the MiMo shape). The MTP draft ctx costs **~2.4 GiB
|
|
||||||
measured** (23,636 − 22,728 + 1,561 at the 262,144 draft window) — ~6×
|
|
||||||
the ~0.4 GiB the one-slot era assumed, because the drafter's KV is
|
|
||||||
built against the target's FULL context, not the 3-token proposal
|
|
||||||
depth; the lighter RCO-IQ3_XXS quant (−1.56 GiB vs IQ3_S-mtp) funds it.
|
|
||||||
**Prefill 1,209 t/s @ 3,760 tokens; decode 52.0–58.2 tok/s (recipe
|
|
||||||
temp 1.0) / 66.9 (temp 0)**, acceptance 34.2–54.4 % — decode is
|
|
||||||
ubatch-invariant, the spread is temp-1.0 acceptance noise.
|
|
||||||
`reasoning_effort` probe unchanged: `xhigh` 200, `high` → 500 Jinja
|
|
||||||
raise (the models.yml fold stays mandatory).
|
|
||||||
|
|
||||||
**Vision with the publisher's own projector**: ajgazin's repo SHIPS
|
**Measured boots** (nvidia-smi, the referee): the deployed boot is
|
||||||
`mmproj-BF16.gguf` in the same repo (the IQ3_S era's unsloth mmproj was
|
**21,436 used / 2,689 free** (ubatch 2048) — the pricing boots were the
|
||||||
built from the SHARED BASE `Qwen/Qwen3.8-27B`, legal because Swift-1.5's
|
MiMo-V2.6-Distill-Qwen-9B pair at this exact geometry (**21,276 / 2,849** at
|
||||||
vision stack was hash-proven byte-identical to the base). The probe
|
ubatch 1024, **21,846 / 2,279** at 2048), so the 415 MiB lighter weights
|
||||||
traps carry over: ≥256 px probe images or the model answers "video";
|
bought back what the projection predicted. The engine ledger confirms the KV
|
||||||
only the OpenAI `image_url` content part reaches the model. **New on
|
law to the MiB: `CUDA0 KV buffer size = 12544.00 MiB` (`K (f16): 6272.00`,
|
||||||
this payload: the uncensored model thinks more than the censored
|
`V (f16): 6272.00`), compute buffers 1,136 MiB on CUDA0 + 848 MiB host, the
|
||||||
Swift 1.5** — a 64-token budget is eaten entirely by
|
mmproj 879 MiB + 248 MiB. Decode is ubatch-invariant (Swift-era 3090 law). The
|
||||||
`reasoning_content` (empty `content`); 128 tokens clears the vision
|
Q6_K rung costs ~10 % decode vs Q5_K_M (weight bandwidth; MiMo-measured 94.4
|
||||||
probe.
|
vs 104.8 t/s).
|
||||||
|
|
||||||
The RCO-IQ3_XXS tier is a **per-tensor allocation mix** of stock quant
|
**Vision**: the publisher's own BF16 mmproj via `--mmproj` (the model is
|
||||||
types (IQ3_S 97 / BF16 96 / IQ3_XXS 83 / IQ2_S 72 / IQ4_XS 33 / IQ2_XS 32
|
natively multimodal; the Q6_K file carries no vision tensors). Probe traps
|
||||||
/ IQ2_XXS 30 / Q2_K 28 / Q4_K 19 / Q6_K 8 / IQ1_S 4 / IQ1_M 4) — stock
|
carry over: ≥256 px probe images or the model answers "video", and only the
|
||||||
llama.cpp loads it; the tensor-type histogram is the law (see Key
|
OpenAI `image_url` content part reaches the model.
|
||||||
Lessons). Publisher KLD vs Swift BF16: mean 0.002631 / 90th 0.042136 /
|
|
||||||
max 34.49 — RCO is the publisher's "more experimental" family, and the
|
**Compaction hub**: `models.yml` routes `compactionModel` for
|
||||||
payload is the UNCECERSED Swift 1.5 (23/100 refusals vs Swift 1.5's
|
`frank-archon`, `frank-dragoon` and `frank-zealot` (the 3090 endpoint) here — 200,704 is the
|
||||||
98/100, KL 0.0884).
|
only window in the fleet that fits a worst-case compaction input (~170k
|
||||||
|
tokens). With `--parallel 2` a compaction request no longer serializes behind
|
||||||
|
a live generation (the Gemma4 era's `--parallel 1` law).
|
||||||
|
|
||||||
Runbook:
|
Runbook:
|
||||||
`serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md`.
|
`serverAppConfigs-docs/LLM/frank-zealot-llama-qwen35-9b-q6k-2x200704.md`.
|
||||||
Rollback: `/home/dank/models/Swift15-IQ3S/` (the IQ3_S-mtp payload live
|
Rollback: revert the swap commit — the Gemma4 payload
|
||||||
until the 2026-10-06 evening swap — revert da36fa3 for the two-slot
|
(`/home/dank/models/Gemma4-26B-A4B/`), the MiMo Q6_K payload
|
||||||
no-spec shape) or `/home/dank/models/MiMo-Q6K/` (revert to the
|
(`/home/dank/models/MiMo-Q6K/`), the Swift payload
|
||||||
2×200,704 f16 shape — [[frank-secondary-llama-mimo-q6k-2x200704]]) or
|
(`/home/dank/models/Swift15-UncIQ3XXS/`) and the Ornith Q8_0 payload
|
||||||
`/home/dank/models/Ornith-Q8/` (the committed-never-deployed Q8_0;
|
(`/home/dank/models/Ornith-Q8/`) all remain on disk.
|
||||||
revert to 380aafa's tree).
|
|
||||||
|
|
||||||
### frank-primary — Radiance (StillDeadcode), Qwen3.8-Flash-Next FP8 native, TP=2 (Dankserver, ACTIVE)
|
|
||||||
|
### frank-archon — Radiance (StillDeadcode), publisher Qwen3.8-Flash-Next iq4r, TP=2 (Dankserver, ACTIVE)
|
||||||
|
|
||||||
The **radiance engine** (StillDeadcode, Apache-2.0; image
|
The **radiance engine** (StillDeadcode, Apache-2.0; image
|
||||||
`stilldeadcode/radiance@sha256:4c3f484564bb2872784e4…` digest-pinned,
|
`stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned, **pin 1.0.13**
|
||||||
287 MiB, `libr4d@0.5.0` kernels prebuilt for gfx1201, engine as the image
|
for serve; `libr4d` gfx1201 kernels, plain `podman run`, root
|
||||||
entrypoint) serving the **official `Qwen/Qwen3.8-Flash-Next-FP8`
|
systemd unit, container name = unit name) serving the
|
||||||
checkpoint at NATIVE precision** — modelopt block-FP8 throughout, no
|
**publisher's container of the official Qwen/Qwen3.8-Flash-Next**:
|
||||||
requantisation, no calibration. Plain `podman run` through
|
`StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` →
|
||||||
`scripts/serve-frank-primary.sh` under the root systemd unit
|
`qwen3.8-next-flash-fp8-iq4r-moe.rad` (116,322 MiB, Hub-LFS sha256
|
||||||
(`frank-primary.service`, `User=dank`, `LimitMEMLOCK=infinity`); the
|
verified at fetch by `scripts/download-frank-archon-iq4r.sh`; recipe
|
||||||
container name is the unit name (`frank-primary`). Cutover LIVE 2026-10-06
|
`qwen4exp-w4nl64-i8-hc8m`): **routed experts as 4-bit codes into a
|
||||||
00:55 (chain built 2026-10-05/06);
|
16-level non-uniform codebook** after a 128-point Walsh–Hadamard rotation
|
||||||
rollback tag `frank-primary-freetoken-pr587-20261005` (the FreeToken trio
|
(E4M3 scale per 64 weights; codes by GPTQ on a 10M-token calibration with
|
||||||
+ Dockerfile are in the tag; the image and the orcarouter payload stay on
|
per-expert down-proj Hessians, refined by 3 coordinate-descent sweeps),
|
||||||
disk until the window closes), older rollback `frank-primary-vllm-shapeG-20261003`.
|
**10 whole bf16-protected experts/layer**, **int8 W8A8 trunk/lm_head**
|
||||||
|
(scale per 128 columns searched), **fp8 HC mixtures**, PLE n-gram table at
|
||||||
|
fp8 fixed scale (47.7 GiB), **MTP head and bf16 vision tower in the
|
||||||
|
container**. Card quality (full-vocab teacher-forced KL vs bf16, ~75K
|
||||||
|
positions): mean 0.0789 all-tokens / 0.0239 assistant turns, top-1 92.85 %.
|
||||||
|
**MODEL_FILE is the only serving argument that moved** (2026-10-15); every
|
||||||
|
number of the 2026-10-13 shape below carries verbatim. The plugin's
|
||||||
|
**one-table-one-encoding law** (fleet-measured at rad-convert declare):
|
||||||
|
one expert table takes ONE encoding; only whole bf16 experts are legal
|
||||||
|
beside it. Rollback: revert the swap commit — the in-house w4dn8 `.rad`
|
||||||
|
(KLD-gated Swift1.5 conversion) stays on disk; the fp8-native `.rad` was
|
||||||
|
deleted 2026-10-15 (rebuild: `scripts/download-frank-archon.sh` +
|
||||||
|
`scripts/build-frank-archon-rad.sh`).
|
||||||
|
|
||||||
**Payload chain**: official checkpoint → staging dir whose `config.json`
|
**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 3`,
|
||||||
drops `quantization_config` → `stack-qwen38-official-fp8.py` (two
|
`--max-model-len 200000` (the harness contextWindow must quote this),
|
||||||
value-preserving reshapes: experts byte-stacked into the plugin's stacked
|
`--placement expert_tiered --host-pool-mib 44928` + **explicit VRAM
|
||||||
`gate_up_proj`/`down_proj` layout with `_scale_inv` at the same index — 49
|
partition `--vram-weights-mib 25600 --vram-kv-mib 5120`** (the vram-pair
|
||||||
stacks × 512 experts, no requantisation; PLE's 128 scaleless fp8 shards
|
replaced `--expert-vs-cache-ratio`; the two are ONE budget —
|
||||||
decoded to one bf16 shard with the checkpoint's own bf16 scale
|
`weights + kv ≤ 30,969 MiB`, derived from the engine's own budget table: card
|
||||||
1.9931793212890625e-04) → `rad-convert` with ONE `--quant` rule — pure
|
32,624 MiB, the pair at 29,120 settling 2,058 MiB free against the engine's
|
||||||
copy for every weight, and the rtn rule re-encodes the bf16 PLE at that
|
209.19 MiB floor (`--gpu-headroom-mib 96` + 113.19 static), so 1,849 MiB is
|
||||||
same fixed scale, an exact round-trip that restores the original codes and
|
claimable and the old 29,120 bound was my hand-picked 1 GiB reserve, not a
|
||||||
lands the plugin's one-element scale plane →
|
hardware wall), `--kv-cache-dtype` **`fp8` — quantized KV, restored 2026-10-13
|
||||||
`/home/dank/models/rad/qwen4exp-fp8-native.rad` (177,109 MiB measured,
|
because it is the only dtype that funds the slot count at this window**,
|
||||||
floor `FRANK_PRIMARY_RAD_MIN_MIB=177000`). The staging drop is the plugin
|
`--num-speculative-tokens 2`, `--max-num-batched-tokens 2048`, prefix cache
|
||||||
SELECTION law (spec §2.4): the descriptor is `(arch_id, quant)` matched
|
8 GiB pinned host + 64 GiB disk at `/home/dank/radiance-state`,
|
||||||
exactly, no shipped plugin claims `(qwen4exp, "fp8")`, the shipped
|
**`--ngram-placement ram`**, Froggeric v22.5 chat template vendored
|
||||||
`qwen4exp_fp8` plugin claims `(qwen4exp, "")` — and the descriptor is only
|
(`scripts/qwen38-froggeric-v22.5.jinja`, `--override-chat-template`). Boot
|
||||||
the selection key, per-weight encodings are data, so nothing else is
|
ledger (2026-10-13, pair 25600/5120): `kv cache 5.00 GiB … backs 3 sequences
|
||||||
requantised.
|
all at 200000 tokens at once, which is the worst case` — `kv_attn` page
|
||||||
Raw serve AND unmodified convert both refuse; that is the design. The
|
26.00 KiB (fp8) at 607,740 tokens (the sizing law predicted 608,027),
|
||||||
checkpoint itself is sha256-verified against the Hub (131/131 shards,
|
`kv_qsa_bkey` 3.25 KiB at the same token count, `kv_gdn_state` 54 MiB × 6 slots
|
||||||
2026-10-05). Built by `scripts/build-frank-primary-rad.sh` (`--plan-only`
|
(3 sequences × 2 copies) = 324 MiB; the sampler prints **9 rows**
|
||||||
first — its declare phase is the engine's own, so a resolving plan boots).
|
(3 × (1+2), inside the 25-row `wht6` exactness bound); the slab filled to
|
||||||
|
563 KiB of slack with **17,548 slots** (16,235 at 24,000, 14,753 at 22,208 —
|
||||||
|
every 1,792 MiB buys ~1,300 resident experts, and the pinned host pool hands
|
||||||
|
the same experts back: 8,893 → 7,580 slots); `unclaimed 477.30 MiB`, cards
|
||||||
|
settled ~336 MiB free against the engine's 209.19 MiB floor, KV lending
|
||||||
|
4.22 GiB to the expert cache. Prefill tracks this slab only up to a
|
||||||
|
**threshold**: 22,208 → 3,397 and 24,000 → 3,290 t/s at 43.5k are one band
|
||||||
|
(±3 %), and the 25,600 arm measured **3,290** — the third step bought nothing
|
||||||
|
(the 3,584 arm did not reproduce at 3 slots); the cliff is at ~26 GiB
|
||||||
|
(26,071 → 4,379 on 2026-10-11's arm), so below that the slab is a threshold,
|
||||||
|
not a slope.
|
||||||
|
|
||||||
**Shape (both R9700s, TP=2)**: `--tp-wire wht6` (fused lossy all-reduce,
|
**Decode ladder (2026-10-13, 512 tokens, greedy, this shape)**: 99.0 t/s
|
||||||
+5.6–6.3 % prefill / ~−1.3 % 8-stream decode at TP2), `--max-num-seqs 2`
|
single-stream at 43.5k, 93.3 at 197k — decode is nearly window-free (−6 % for
|
||||||
(Frank's first boot; raise from the ledger + slot occupancy), `--max-model-len
|
4.5× depth); **3 concurrent streams give 113.9/43.3/10.9 t/s = 18.4 t/s
|
||||||
200000` (native 262,144; the harness contextWindow must quote this — a longer
|
aggregate**, so the slot count is paid in decode concurrency (a fused-step
|
||||||
prompt is refused, never truncated), `--placement expert_tiered
|
lottery), not in prefill; leg D of `probe-frank-archon-kv.py` measures it.
|
||||||
--host-pool-mib 40960` (native FP8 experts are 112.8 GiB total, 56.4 GiB
|
|
||||||
per rank slice — the pinned pool is the price of native precision),
|
|
||||||
`--expert-vs-cache-ratio 0.82` (the measured 99 %-hit point; 0.87 collapses
|
|
||||||
to 0 %), `--kv-cache-dtype fp8` (**7.32 KiB/token**; a 200K sequence is
|
|
||||||
1.43 GiB), `--num-speculative-tokens 3` (the checkpoint's own MTP head —
|
|
||||||
3,101 `mtp.*` tensors, probe depth 3), `--max-num-batched-tokens 2048`,
|
|
||||||
prefix cache 4 GiB pinned host + 64 GiB disk at
|
|
||||||
`/home/dank/radiance-state`, and **`--ngram-placement ram`**: the 47.7 GiB
|
|
||||||
PLE n-gram table (51.2 GB = 29 % of all parameters) serves from pinned
|
|
||||||
host RAM. Froggeric v22.5 chat template vendored
|
|
||||||
(`scripts/qwen38-froggeric-v22.5.jinja`, `--override-chat-template`).
|
|
||||||
|
|
||||||
**The pinned-RAM ledger is why this is a root unit**: 2 × 40 GiB pools +
|
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
|
||||||
47.7 GiB PLE + 4 GiB prefix host ≈ **134 GiB** charged against
|
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
|
||||||
`RLIMIT_MEMLOCK`. `ttm.pages_limit`/`pages_max` cap GPU-mappable pinned
|
prefix cache, and the rule that a prompt over `--max-model-len` is
|
||||||
pages (pre-cutover cap 123 GiB — BELOW the plan); the serve script's
|
refused, never truncated. What changed: the fp8-native era's NVMe expert
|
||||||
TTM preflight (`FRANK_PRIMARY_TTM_MIN_MIB=140000`) refuses to start until
|
streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the
|
||||||
both knobs are raised in the stop window (sysfs write first, kernel
|
host pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
|
||||||
cmdline + reboot only if read-only).
|
on the fp8 shape (512-token streamed probe). The campaign also
|
||||||
|
measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
|
||||||
|
accept 0.7246).
|
||||||
|
|
||||||
|
**The KV dtype law (measured 2026-10-11, applied at 4 slots 2026-10-13)**:
|
||||||
|
`--kv-cache-dtype` takes ONLY `bf16|fp8` — one dtype for the whole cache, so
|
||||||
|
llama.cpp's exact-K/quantized-V pair (what frank-zealot runs) has NO radiance
|
||||||
|
equivalent; the rung under bf16 is fp8 on both halves. Measured per rank from
|
||||||
|
the two boot ledgers: `kv_attn` (13 layers, 4-token blocks) pages 26.00 → 52.00
|
||||||
|
KiB, while `kv_qsa_bkey` (13 layers) stays 3.25 KiB at BOTH dtypes (the QSA
|
||||||
|
block key is an index, not attention state) — so a token costs 7.3125 KiB at fp8
|
||||||
|
and 13.8125 KiB at bf16: **1.89×, not 2×**. Pool sizing must also pay the
|
||||||
|
dtype-independent terms that live in the same pool (~110 MiB a sequence of
|
||||||
|
recurrent state + 448 MiB of prefix checkpoints):
|
||||||
|
`(vram_kv − 110·seqs − 448) MiB / rate` — bf16 at 6,912 gives 462,946 where the
|
||||||
|
engine reported 462,752 (0.04 % page rounding); fp8 at 6,912 with four slots
|
||||||
|
gives 843,566, i.e. the deployed capacity contract 4 × 200,000. **That is why
|
||||||
|
the dtype is fp8: the pool must back `seqs × max-model-len` as a hard limit, and
|
||||||
|
bf16 at four slots costs 11,679 MiB — the slab would fall to 17,441 MiB (−33 %),
|
||||||
|
where fp8 leaves it at 22,208.** The cost is paid twice on this engine, and
|
||||||
|
half-refunded: idle KV VRAM is also the expert cache's spillover, and the bigger
|
||||||
|
pool *lends* 6.08 GiB to the slab where the 3,543 MiB fp8 pool lent 2.69 GiB.
|
||||||
|
`--max-num-seqs` rides 4 × (1 + spec) = 12 fused-all-reduce rows, under the
|
||||||
|
25-row bound where wht6 is bit-exact (the "seqs 2 cannot boot" law belongs to
|
||||||
|
the retired tcclaviger fork's torch.compile, not radiance). Disk-tier sizing
|
||||||
|
law: the engine forces ~9.6 GiB of disk-tier weight buffers + arena onto the
|
||||||
|
weights side, so `weights + kv` is capped near 29.6 GiB/card; boot with the
|
||||||
|
pair, never `--kv-cache-size-mib`.
|
||||||
|
|
||||||
|
Shape notes carried from the fp8-native era and unchanged: wht6 gains,
|
||||||
|
MNDT 2048 law (2560 collapses decode to 2.6–7.7 t/s), the 64 GiB disk
|
||||||
|
prefix cache, and the rule that a prompt over `--max-model-len` is
|
||||||
|
refused, never truncated. What changed: the fp8-native era's NVMe expert
|
||||||
|
streaming (the 3.5 GB/s wall) is gone — the w4 resident set fits the
|
||||||
|
host pool, so decode ran 122–173 t/s in the campaign arms and ~190 t/s
|
||||||
|
on the fp8 shape (512-token streamed probe). The campaign also
|
||||||
|
measured the int8-KV + spec-3 arm (136.0 t/s median at 44.5k depth,
|
||||||
|
accept 0.7246).
|
||||||
|
|
||||||
|
**The pinned-RAM ledger is why this is a root unit**: 2 × 44 GiB expert
|
||||||
|
pools + the 47.7 GiB n-gram copy (`--ngram-placement ram` is pinned RAM
|
||||||
|
the card reads) + 8 GiB prefix host ≈ **147 GiB** charged against
|
||||||
|
`RLIMIT_MEMLOCK` — the TTM floor is set just above it.
|
||||||
|
`ttm.pages_limit`/`pages_max` cap GPU-mappable pinned pages, and **the knob is
|
||||||
|
runtime state whose default is half of RAM**: `scripts/ttm-preflight.sh` (sourced
|
||||||
|
by the serve script) therefore RAISES it to `FRANK_ARCHON_TTM_TARGET_MIB=204800`
|
||||||
|
before refusing, because the 2026-10-12 power failure reset it to 125,988 MiB
|
||||||
|
and the preflight re-refused 6,919 times. A manual sysfs write in the stop
|
||||||
|
window is a convenience, not the repair; kernel cmdline persistence
|
||||||
|
(`ttm.pages_limit=52428800`) is the belt to it.
|
||||||
|
|
||||||
**Contract facts** (source-verified): the HTTP listener binds AFTER the
|
**Contract facts** (source-verified): the HTTP listener binds AFTER the
|
||||||
full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
|
full load, so a bare `curl /health` gate is honest (`/health` == `/ping`,
|
||||||
@@ -574,10 +631,12 @@ chat format — no parser flags; vision native (the 27-layer Qwen-VL tower,
|
|||||||
`--mm-max-patches`); `/metrics` carries vLLM-compatible names +
|
`--mm-max-patches`); `/metrics` carries vLLM-compatible names +
|
||||||
`radiance:*` families — the Dankclaw Prometheus scrape is re-added.
|
`radiance:*` families — the Dankclaw Prometheus scrape is re-added.
|
||||||
|
|
||||||
Runbook: `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md`.
|
Runbook: `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md`.
|
||||||
Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-primary-freetoken-pr587.md`.
|
Retired Swift1.5 w4dn8 era (payload on disk = rollback): `serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md`.
|
||||||
|
Retired fp8-native era (payload deleted 2026-10-15, rebuildable): `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md`.
|
||||||
|
Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-archon-freetoken-pr587.md`.
|
||||||
Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
|
Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
|
||||||
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md`.
|
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-archon-glm53-flash-w4a16.md`.
|
||||||
|
|
||||||
## Key Lessons Learned
|
## Key Lessons Learned
|
||||||
Durable operational lessons from the fleet's history. Full detail in the
|
Durable operational lessons from the fleet's history. Full detail in the
|
||||||
@@ -739,11 +798,22 @@ linked runbooks.
|
|||||||
- Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff`
|
- Swap resize ordering: create and `swapon` the replacement BEFORE `swapoff`
|
||||||
of the old one, or live pages must land in RAM that `MemAvailable`
|
of the old one, or live pages must land in RAM that `MemAvailable`
|
||||||
overstates. Dankclaw swap (64 GiB at `/swap.img`, fstab-driven) protects
|
overstates. Dankclaw swap (64 GiB at `/swap.img`, fstab-driven) protects
|
||||||
the co-tenants, not frank-large — pinned weights don't swap, and paging
|
the co-tenants, not frank-dragoon — pinned weights don't swap, and paging
|
||||||
the KV pool is worse than the OOM it prevents.
|
the KV pool is worse than the OOM it prevents.
|
||||||
- Host swap state is invisible to `deploy.sh` (no repo file) — check
|
- Host swap state is invisible to `deploy.sh` (no repo file) — check
|
||||||
`swapon --show` before reasoning about headroom.
|
`swapon --show` before reasoning about headroom.
|
||||||
- **Silent RAM corruption outranks every software theory.** frank-primary's
|
- **A sysfs raise is a countdown to the next reboot.** `ttm.pages_limit` defaults
|
||||||
|
to half of RAM, so the manual raise to 200 GiB (2026-10-05) vanished with the
|
||||||
|
2026-10-12 23:17 POWER FAILURE on dankserver: the cap fell to 125,988 MiB,
|
||||||
|
frank-archon's pinned-ledger preflight (floor 150,000 MiB) refused the start,
|
||||||
|
and `Restart=always` re-ran that refusal **6,919 times in ~20 h** — a unit in
|
||||||
|
`activating (auto-restart)` that exits 1 in 50 ms with no container and no
|
||||||
|
engine log is a PREFLIGHT refusal, not an engine fault. Repo law now:
|
||||||
|
`scripts/ttm-preflight.sh` (sourced by `serve-frank-archon.sh`) raises the knob
|
||||||
|
to `FRANK_ARCHON_TTM_TARGET_MIB` itself, re-reads, and refuses by name only
|
||||||
|
after a write the kernel rejected. Never prescribe a manual sysfs write as the
|
||||||
|
recovery procedure for a service that can boot before a human reaches it.
|
||||||
|
- **Silent RAM corruption outranks every software theory.** frank-archon's
|
||||||
~40k-token output corruption (2026-10-04/05, repetition/garbage in long
|
~40k-token output corruption (2026-10-04/05, repetition/garbage in long
|
||||||
agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
|
agent conversations) was marginal DDR5 at 4800 MT/s on four DIMMs, fixed
|
||||||
by 4600. Consumer non-ECC memory raises **no MCE** for a silent flip, and
|
by 4600. Consumer non-ECC memory raises **no MCE** for a silent flip, and
|
||||||
@@ -766,7 +836,7 @@ linked runbooks.
|
|||||||
`LimitMEMLOCK=infinity` (systemd is privileged; the value flows through
|
`LimitMEMLOCK=infinity` (systemd is privileged; the value flows through
|
||||||
podman to runc without any raise). Upstream halogen's own run lines pass
|
podman to runc without any raise). Upstream halogen's own run lines pass
|
||||||
NO `--ulimit` at all — copy vendor run lines before adding flags
|
NO `--ulimit` at all — copy vendor run lines before adding flags
|
||||||
(measured 2026-10-05 on the frank-large cutover).
|
(measured 2026-10-05 on the frank-dragoon cutover).
|
||||||
- **Kairic Edge** (if ever re-deployed): the pinned CK commit needs
|
- **Kairic Edge** (if ever re-deployed): the pinned CK commit needs
|
||||||
`third_party/composable_kernel/include/ck/config.h` generated manually
|
`third_party/composable_kernel/include/ck/config.h` generated manually
|
||||||
(untracked; `git clean` removes it), and its greedy-argmax fastpath rejects
|
(untracked; `git clean` removes it), and its greedy-argmax fastpath rejects
|
||||||
@@ -792,7 +862,7 @@ linked runbooks.
|
|||||||
121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
|
121 GiB box, pushed `available` to 3 GiB, and every later rate was suspect.
|
||||||
Record the child pid and kill it in a `finally`, then prove `free` came back
|
Record the child pid and kill it in a `finally`, then prove `free` came back
|
||||||
before believing a number. Kill by pid — never `pkill -f llama-server`,
|
before believing a number. Kill by pid — never `pkill -f llama-server`,
|
||||||
which also hits frank-secondary.
|
which also hits frank-zealot.
|
||||||
|
|
||||||
**llama.cpp serving**
|
**llama.cpp serving**
|
||||||
|
|
||||||
@@ -816,7 +886,7 @@ linked runbooks.
|
|||||||
builds its context against whatever model the draft context holds. Two
|
builds its context against whatever model the draft context holds. Two
|
||||||
context-backed types cannot share one process, and MTP's head only loads
|
context-backed types cannot share one process, and MTP's head only loads
|
||||||
when MTP **is** the draft model. **A LEGAL chain can still lose on a
|
when MTP **is** the draft model. **A LEGAL chain can still lose on a
|
||||||
recurrent model:** `draft-mtp,ngram-mod` on frank-large's qwen35 hybrid
|
recurrent model:** `draft-mtp,ngram-mod` on frank-dragoon's qwen35 hybrid
|
||||||
measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer
|
measured −15 % prose / −7 % block-copy (2026-10-03) — adding the ngram layer
|
||||||
logs `recurrent state rollback is not compatible with 'ngram-mod' -
|
logs `recurrent state rollback is not compatible with 'ngram-mod' -
|
||||||
disabling rollback support`, and MTP verification depends on that rollback,
|
disabling rollback support`, and MTP verification depends on that rollback,
|
||||||
@@ -858,7 +928,7 @@ linked runbooks.
|
|||||||
free. That was a CONTIGUITY failure, not a total-bytes one, and no totals
|
free. That was a CONTIGUITY failure, not a total-bytes one, and no totals
|
||||||
model — the sweep's or the seam's — can see it.)
|
model — the sweep's or the seam's — can see it.)
|
||||||
- **On the CUDA 3090 the same knob has NO interior optimum up to 2048, so
|
- **On the CUDA 3090 the same knob has NO interior optimum up to 2048, so
|
||||||
frank-secondary ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the
|
frank-zealot ships 4096/2048.** Measured 2026-10-05 by boot-pair A/B on the
|
||||||
live service: at 8,928 prompt tokens ubatch 1024 gives 3,972 / 4,077 t/s and
|
live service: at 8,928 prompt tokens ubatch 1024 gives 3,972 / 4,077 t/s and
|
||||||
ubatch 2048 gives 4,207 / 4,212 — **+4.6 %** — for **616 MiB** (free 2,849 →
|
ubatch 2048 gives 4,207 / 4,212 — **+4.6 %** — for **616 MiB** (free 2,849 →
|
||||||
2,279). Decode did not move (91.9–93.0 vs 93.8–94.3, i.e. noise), and a
|
2,279). Decode did not move (91.9–93.0 vs 93.8–94.3, i.e. noise), and a
|
||||||
@@ -867,9 +937,9 @@ linked runbooks.
|
|||||||
than the ubatch cannot see it. The logical `--batch-size` was again free. And
|
than the ubatch cannot see it. The logical `--batch-size` was again free. And
|
||||||
the predecessor's "ubatch 2048 failed the scratch boot" was a shape
|
the predecessor's "ubatch 2048 failed the scratch boot" was a shape
|
||||||
observation, not a law: the same file boots 2048 clean at 2 × 200,704. Bench
|
observation, not a law: the same file boots 2048 clean at 2 × 200,704. Bench
|
||||||
it by restarting the SERVICE through a transient `FRANK_SECONDARY_ENV_FILE`
|
it by restarting the SERVICE through a transient `FRANK_ZEALOT_ENV_FILE`
|
||||||
drop-in, never a scratch port — one card holds one server. See
|
drop-in, never a scratch port — one card holds one server. See
|
||||||
[[frank-secondary-llama-mimo-q6k-2x200704]].
|
[[frank-zealot-llama-mimo-q6k-2x200704]].
|
||||||
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
|
- **One `-ts` step is one LAYER, and a hybrid stack's layers are not the same
|
||||||
size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
|
size.** qwen35 is 16 x (3 Gated DeltaNet + 1 Gated Attention), so a step is
|
||||||
worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one
|
worth ~0.43 GiB on a DeltaNet boundary and ~1.43 GiB on a full-attention one
|
||||||
@@ -891,7 +961,7 @@ linked runbooks.
|
|||||||
CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT
|
CTX_SIZE / PARALLEL`, and llama.cpp refuses a prompt longer than the SLOT
|
||||||
(`400 exceed_context_size_error: request (184001 tokens) exceeds the
|
(`400 exceed_context_size_error: request (184001 tokens) exceeds the
|
||||||
available context size (180224 tokens)`) even when the pool has room. So
|
available context size (180224 tokens)`) even when the pool has room. So
|
||||||
frank-secondary's 2 → 3 reslice moved each conversation from the model's native
|
frank-zealot's 2 → 3 reslice moved each conversation from the model's native
|
||||||
262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or
|
262,144 down to 180,224, and `models.yml` `contextWindow` had to follow or
|
||||||
the harness would discover the ceiling as a 400 on a long conversation. Side
|
the harness would discover the ceiling as a 400 on a long conversation. Side
|
||||||
effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
|
effect worth knowing: the shorter slot made decode ~5 % FASTER (106.8 vs
|
||||||
@@ -902,9 +972,9 @@ linked runbooks.
|
|||||||
day the freed space also bought the Q6_K rung (now 94.4 tok/s — the rung
|
day the freed space also bought the Q6_K rung (now 94.4 tok/s — the rung
|
||||||
itself is a further measured −10 %, weight-bandwidth, not slot geometry).
|
itself is a further measured −10 %, weight-bandwidth, not slot geometry).
|
||||||
All moves measured; both slot directions cost or buy window, never pool.
|
All moves measured; both slot directions cost or buy window, never pool.
|
||||||
See [[frank-secondary-llama-mimo-q6k-2x200704]].
|
See [[frank-zealot-llama-mimo-q6k-2x200704]].
|
||||||
- **Price a VRAM budget from two boots, never from a formula — and an mmproj
|
- **Price a VRAM budget from two boots, never from a formula — and an mmproj
|
||||||
costs ~1.5x its file size.** The frank-secondary 3090 fit was solved from two
|
costs ~1.5x its file size.** The frank-zealot 3090 fit was solved from two
|
||||||
measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
|
measurements: 2 × 262,144 with no projector = 20,190 MiB used (KV 12,800 ⇒
|
||||||
**non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV
|
**non-KV 7,390**), 3 × 196,608 with the f16 projector = 22,972 used (KV
|
||||||
14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file
|
14,113 ⇒ **non-KV 8,859**). The difference is the vision tower: a 918 MB file
|
||||||
@@ -912,24 +982,24 @@ linked runbooks.
|
|||||||
ride with it. Subtracting the two rows is exact; extrapolating weights+KV+
|
ride with it. Subtracting the two rows is exact; extrapolating weights+KV+
|
||||||
"a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then
|
"a bit for buffers" is what leaves 1.1 GiB free when you meant 2.7. Then
|
||||||
solve `KV ≤ card − non-KV − headroom` and round to a multiple of
|
solve `KV ≤ card − non-KV − headroom` and round to a multiple of
|
||||||
`PARALLEL × 4096`. Recorded in `scripts/frank-secondary.env` so the next resize
|
`PARALLEL × 4096`. Recorded in `scripts/frank-zealot.env` so the next resize
|
||||||
starts from measurement.
|
starts from measurement.
|
||||||
- **`images` and `image_url` are NOT interchangeable: only the content part
|
- **`images` and `image_url` are NOT interchangeable: only the content part
|
||||||
reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
|
reaches the model.** `--mmproj` loads, `usage.prompt_tokens` rises, and the
|
||||||
model still invents an answer if the client used the legacy top-level
|
model still invents an answer if the client used the legacy top-level
|
||||||
`images` field on `/v1/chat/completions` — frank-secondary answered "Brown." for
|
`images` field on `/v1/chat/completions` — frank-zealot answered "Brown." for
|
||||||
a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
|
a solid-red PNG that way and "Red." through the OpenAI `image_url` part. The
|
||||||
failure is silent because a hallucinated color is a plausible answer. Probe
|
failure is silent because a hallucinated color is a plausible answer. Probe
|
||||||
vision with a solid color and expect an exact word.
|
vision with a solid color and expect an exact word.
|
||||||
- **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
|
- **`--jinja` is what `tools` needs.** On 2026-09-16 a subagent died with
|
||||||
`500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
|
`500 tools param requires --jinja flag`, from the ik_llama.cpp runtime that
|
||||||
then served frank-large: every tool-using agent on that endpoint was broken
|
then served frank-dragoon: every tool-using agent on that endpoint was broken
|
||||||
and the failure read like a model fault. Mainline llama.cpp 0.4.1 defaults
|
and the failure read like a model fault. Mainline llama.cpp 0.4.1 defaults
|
||||||
`--jinja` on and the serve script pins it; halogen also accepts `tools` and
|
`--jinja` on and the serve script pins it; halogen also accepts `tools` and
|
||||||
returns `tool_calls`. Check tool support before routing a role at an endpoint.
|
returns `tool_calls`. Check tool support before routing a role at an endpoint.
|
||||||
- **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same
|
- **Quantized KV on CUDA needs `GGML_CUDA_FA_ALL_QUANTS=ON`** — the same
|
||||||
option and the same ~12x prefill collapse described in the ROCm note above.
|
option and the same ~12x prefill collapse described in the ROCm note above.
|
||||||
f16 KV needs nothing; frank-secondary ran f16 for that reason until
|
f16 KV needs nothing; frank-zealot ran f16 for that reason until
|
||||||
2026-09-17, when `build-cuda` was rebuilt with the option ON so the
|
2026-09-17, when `build-cuda` was rebuilt with the option ON so the
|
||||||
endpoint could take q8_0 KV.
|
endpoint could take q8_0 KV.
|
||||||
- **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both
|
- **llama-bench separates a LIST with `/` and SWEEP values with `,`.** Both
|
||||||
@@ -947,7 +1017,7 @@ linked runbooks.
|
|||||||
raises on anything else — HTTP 500, not a clamp — so the harness's levels
|
raises on anything else — HTTP 500, not a clamp — so the harness's levels
|
||||||
must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`).
|
must be folded onto that set in `models.yml` (`compat.reasoningEffortMap`).
|
||||||
A role whose level is unmapped fails on its FIRST request, and the failure
|
A role whose level is unmapped fails on its FIRST request, and the failure
|
||||||
names Jinja, not the model: every subagent on `frank-secondary` died with
|
names Jinja, not the model: every subagent on `frank-zealot` died with
|
||||||
`Unexpected reasoning effort high` until `high: xhigh` was added on
|
`Unexpected reasoning effort high` until `high: xhigh` was added on
|
||||||
2026-09-16. Halogen accepts the wider set, so the two entries legitimately
|
2026-09-16. Halogen accepts the wider set, so the two entries legitimately
|
||||||
differ.
|
differ.
|
||||||
@@ -967,7 +1037,7 @@ linked runbooks.
|
|||||||
times (`RuntimeError: No HIP GPUs are available`).
|
times (`RuntimeError: No HIP GPUs are available`).
|
||||||
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its
|
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its
|
||||||
ports off the gluetun service.
|
ports off the gluetun service.
|
||||||
- `amd_iommu=off` on Dankclaw (bought 13–16 % prefill for frank-large,
|
- `amd_iommu=off` on Dankclaw (bought 13–16 % prefill for frank-dragoon,
|
||||||
2026-09-11) permanently retired the XDNA2 NPU: the driver's SVA/PASID bind
|
2026-09-11) permanently retired the XDNA2 NPU: the driver's SVA/PASID bind
|
||||||
needs an enabled IOMMU (`/dev/accel/accel0` fails `err=-19`). Revert path
|
needs an enabled IOMMU (`/dev/accel/accel0` fails `err=-19`). Revert path
|
||||||
in `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md`.
|
in `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md`.
|
||||||
@@ -1021,9 +1091,9 @@ Active:
|
|||||||
|
|
||||||
| Service | Runbook |
|
| Service | Runbook |
|
||||||
|---------|---------|
|
|---------|---------|
|
||||||
| frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×200,192, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` |
|
| frank-dragoon (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×262,144, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md` |
|
||||||
| frank-secondary (llama.cpp CUDA, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×114,688 draft-mtp q8_0 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` |
|
| frank-zealot (llama.cpp CUDA, Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-zealot-llama-qwen35-9b-q6k-2x200704.md` |
|
||||||
| frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` |
|
| frank-archon (radiance StillDeadcode, publisher Qwen3.8-Flash-Next iq4r .rad, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md` |
|
||||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||||
|
|
||||||
@@ -1031,40 +1101,42 @@ Retired (rollback targets and history):
|
|||||||
|
|
||||||
| Runtime | Host | Window | Runbook |
|
| Runtime | Host | Window | Runbook |
|
||||||
|---------|------|--------|---------|
|
|---------|------|--------|---------|
|
||||||
| halogen 0.1.4 Qwen3.8-27B p1w4d-d2 + DFlash2 (`frank-large`) | Dankclaw | 2026-10-05 → 10-06 (rollback target — payload + fetch script + image digest on disk; owns the prompt-cache pin and FA-tile lessons) | `frank-large-halogen27b-dflash2.md` |
|
| radiance Swift1.5-Qwen3.8-Flash-Next GPTQ w4dn8, 3×200,000 fp8-KV slab 25600 (`frank-archon`) | Dankserver | 2026-10-08 → 10-15 (rollback target — `.rad` on disk; owns the slab-slope ladder, KV dtype law, KLD campaign, and every shape law the iq4r swap carried verbatim) | `frank-archon-radiance-swift15-w4dn8.md` |
|
||||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
|
| llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-zealot`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-zealot-llama-gemma4-q4km-200704.md` |
|
||||||
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-dragoon`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-dragoon-rocmfpx-q6agent-uncensored-mtp.md` |
|
||||||
| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` |
|
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-archon`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-archon-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
||||||
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-secondary-llama-mimo-q6k-2x200704.md` |
|
| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-archon`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-archon-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-archon-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-archon-freetoken-pr587.md` |
|
||||||
| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-secondary-llama-swift15-unc-iq3xxs-mtp-114688.md` |
|
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K 2×200,704 f16 KV (`frank-zealot`) | Dankclaw | 2026-10-03 → 10-05 (rollback target — payload on disk; owns the ubatch-2048 A/B; the Ornith swap of 10-05 was committed but never deployed, so this was the LIVE payload Swift replaced) | `frank-zealot-llama-mimo-q6k-2x200704.md` |
|
||||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-large-rocmfpx-q8agent-mtp.md` |
|
| llama.cpp Ornith-1.5-9B Q8_0, ONE 262,144 f16 KV (`frank-zealot`) | Dankclaw | 2026-10-05 (committed 77decb5→380aafa, never deployed — superseded by Swift the same evening; payload on disk) | trail in `frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md` |
|
||||||
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-large-gufo-qwen38-27b.md` |
|
| llama.cpp CUDA Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×114,688 draft-mtp q8_0 KV (`frank-zealot`) | Dankclaw | 2026-10-06 → 2026-10-07 (rollback target — payload on disk; owns the 3090 ubatch ladder A/B and the 34 KiB/token qwen35 KV law) | `frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md` |
|
||||||
| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-large`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-large-vllm-mimo-v26-bf16.md` |
|
| ROCmFPX llama.cpp fork Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP (`frank-dragoon`) | Dankclaw | 2026-10-02 → 10-03 (rollback target, payload on disk) | `frank-dragoon-rocmfpx-q8agent-mtp.md` |
|
||||||
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-secondary`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-secondary-llama-mimo-q6k-2x200704.md`) | `frank-secondary-llama-mimo-q6k.md` |
|
| gufo v0.1.1 Qwen3.8-27B UD-Q8_K_XL/UD-Q4_K_XL + DFlash2 (`frank-dragoon`) | Dankclaw | 2026-09-28 → 10-01, again 10-01 → 10-02 (payloads removed from disk 2026-10-03; rollback = re-download) | `frank-dragoon-gufo-qwen38-27b.md` |
|
||||||
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-secondary`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-secondary-llama-mimo-q5km.md` |
|
| vLLM TheRock MiMo-V2.6-Distill-Qwen-9B BF16 + cross-model MTP (`frank-dragoon`) | Dankclaw | 2026-10-01 (same-day rollback to gufo) | `frank-dragoon-vllm-mimo-v26-bf16.md` |
|
||||||
| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-secondary`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-secondary-sglang-mimo-v26.md` |
|
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q6_K (`frank-zealot`) | Dankclaw | 2026-10-01 → 10-02 (the rung returned ACTIVE 2026-10-03 at the 2×200,704 shape — `frank-zealot-llama-mimo-q6k-2x200704.md`) | `frank-zealot-llama-mimo-q6k.md` |
|
||||||
|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-primary`)|Dankserver|2026-09-30 → 10-01|`frank-primary-glm53-flash-w4a16.md`|
|
| llama.cpp master MiMo-V2.6-Distill-Qwen-9B Q5_K_M 2×200,704 f16 KV (`frank-zealot`) | Dankclaw | 2026-10-02 → 10-03 (rollback target — one `MODEL_FILE` line, buys back 875 MiB + the −10 % rung tax; owns the boot-pair table) | `frank-zealot-llama-mimo-q5km.md` |
|
||||||
| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-large`) | Dankclaw | 2026-09-21 → 09-28 | `frank-large-unsloth-qwen38-27b-q8xl.md` |
|
| SGLang MiMo-V2.6 INT8-W8A16-AutoRound (`frank-zealot`) | Dankclaw | 2026-10-01 (same-day swap to the GGUF path) | `frank-zealot-sglang-mimo-v26.md` |
|
||||||
| R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-primary`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-primary-r9v-radiance-qwen38-uncensored.md` |
|
|tcclaviger vLLM GLM-5.3-Flash W4A16 int4, +/- DFlash2 drafter (`frank-archon`)|Dankserver|2026-09-30 → 10-01|`frank-archon-glm53-flash-w4a16.md`|
|
||||||
| llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-primary`) | Dankserver | 2026-09-26 → 09-27 | `frank-primary-llamacpp-qwen38-flash-next-q8.md` |
|
| llama.cpp b11030 Qwen3.8-27B UD-Q8_K_XL + DFlash2 (`frank-dragoon`) | Dankclaw | 2026-09-21 → 09-28 | `frank-dragoon-unsloth-qwen38-27b-q8xl.md` |
|
||||||
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-large`) | Dankclaw | 2026-09-15 → 09-16 | `frank-large-ik-uncensored.md` |
|
| R9V radiance-vLLM Qwen3.8-Flash-Next-Uncensored IQ4_XS (`frank-archon`) | Dankserver | 2026-09-27 (same-day swap to Q8_0) | `frank-archon-r9v-radiance-qwen38-uncensored.md` |
|
||||||
|
| llama.cpp b11030 Qwen3.8-Flash-Next Q8_0 + shared MTP (`frank-archon`) | Dankserver | 2026-09-26 → 09-27 | `frank-archon-llamacpp-qwen38-flash-next-q8.md` |
|
||||||
|
| ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M (`frank-dragoon`) | Dankclaw | 2026-09-15 → 09-16 | `frank-dragoon-ik-uncensored.md` |
|
||||||
| vllm-radiance TP=2 MXFP4 + DFlash2 (`frank-medium`) | Dankserver | 2026-09-20 → 09-26 | `frank-medium-radiance-mxfp4.md` |
|
| vllm-radiance TP=2 MXFP4 + DFlash2 (`frank-medium`) | Dankserver | 2026-09-20 → 09-26 | `frank-medium-radiance-mxfp4.md` |
|
||||||
| Lucebox DS4.1 DeepSeek-V4.1-Flash ROCmFP23 (`frank-primary`) | Dankserver | 2026-09-26 (same-day swap) | `frank-primary-lucebox-ds4.md` |
|
| Lucebox DS4.1 DeepSeek-V4.1-Flash ROCmFP23 (`frank-archon`) | Dankserver | 2026-09-26 (same-day swap) | `frank-archon-lucebox-ds4.md` |
|
||||||
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
|
| kingjones STRIX Qwen3.8-Flash-Next ROCmFP4 | Dankclaw | 2026-08-30 → 09-11 | `kingjones-strix-rocmfp4.md` |
|
||||||
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-large`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
|
| halogen-flash-server Qwen3.8-Flash-Next w4b (`frank-dragoon`) | Dankclaw | 2026-09-16 → 09-19 | `halogen-flash-server.md` |
|
||||||
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-large`) | Dankclaw | 2026-09-19 → 09-21 | `frank-large-ciru-orca.md` |
|
| CIRU v4.4.1 Orca Qwen3.8-Flash (`frank-dragoon`) | Dankclaw | 2026-09-19 → 09-21 | `frank-dragoon-ciru-orca.md` |
|
||||||
| unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-unsloth-qwen38-flash-next.md` |
|
| unsloth llama.cpp b11030 Qwen3.8-Flash-Next UD-IQ4_XS (`frank-dragoon`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-dragoon-unsloth-qwen38-flash-next.md` |
|
||||||
| Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-large`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-large-kairic-edge-qwen38-27b.md` |
|
| Kairic Edge Qwen3.8-27B IU4 W4A4 (`frank-dragoon`) | Dankclaw | 2026-09-21 (same-day swap) | `frank-dragoon-kairic-edge-qwen38-27b.md` |
|
||||||
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-secondary`) | Dankclaw | 2026-09-16 → 09-17 | `frank-secondary-qwen38-gsq-rco.md` |
|
| ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S (`frank-zealot`) | Dankclaw | 2026-09-16 → 09-17 | `frank-zealot-qwen38-gsq-rco.md` |
|
||||||
| vLLM Gemma4-12B w4a16 (`frank-secondary`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-secondary-vllm-gemma4.md` |
|
| vLLM Gemma4-12B w4a16 (`frank-zealot`) | Dankclaw | 2026-09-19 (same-day revert) | `frank-zealot-vllm-gemma4.md` |
|
||||||
| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-secondary`) | Dankclaw | 2026-09-17 → 10-01 | `frank-secondary-gemma4-12b.md` |
|
| llama.cpp Gemma4-12B QAT Q4_K_M (`frank-zealot`) | Dankclaw | 2026-09-17 → 10-01 | `frank-zealot-gemma4-12b.md` |
|
||||||
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
|
| agentionai ROCmFP4-FAST imatrix | Dankclaw | 2026-08-29 → 08-30 | `agentionai-fast-imatrix.md` |
|
||||||
| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
|
| eemin Qwen3.8-27B ROCmI4 W4A4 MTP | Dankclaw | 2026-08-18 → 08-25 | `eemin-rocmi4-mtp.md` |
|
||||||
| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
|
| Kairic Edge IU4 | Dankclaw | 2026-08-22 (same-day revert) | `kairic-edge.md` |
|
||||||
| CIRU ActiveFPX v3 / PromptForge | Dankclaw | through 2026-08-22 | `ciru-activefpx-promptforge.md` |
|
| CIRU ActiveFPX v3 / PromptForge | Dankclaw | through 2026-08-22 | `ciru-activefpx-promptforge.md` |
|
||||||
| DFlash2 Vulkan | Dankclaw | 2026-08-21 (same-day revert) | `dflash2-large-vulkan.md` |
|
| DFlash2 Vulkan | Dankclaw | 2026-08-21 (same-day revert) | `dflash2-large-vulkan.md` |
|
||||||
| FastFlowLM NPU (`frank-npu`) | Dankclaw | twice, retired 2026-09-11 | `frank-npu.md`, `FastFlowLM NPU Inference.md` |
|
| FastFlowLM NPU (`frank-npu`) | Dankclaw | twice, retired 2026-09-11 | `frank-npu.md`, `FastFlowLM NPU Inference.md` |
|
||||||
| FreeToken (official + Huihui FP8, `frank-secondary`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
|
| FreeToken (official + Huihui FP8, `frank-zealot`) | Dankserver | 2026-09-02 → 09-15 | `freetoken-frank-small.md` |
|
||||||
| FastMTP patched llama.cpp | Dankserver | retired 2026-08-18 | `qwen3.8-spec-matrix.md` |
|
| FastMTP patched llama.cpp | Dankserver | retired 2026-08-18 | `qwen3.8-spec-matrix.md` |
|
||||||
| ik_llama.cpp Ornith CRACK | Dankserver | retired 2026-09-02 | `ikllama-ornith-dankserver.md` |
|
| ik_llama.cpp Ornith CRACK | Dankserver | retired 2026-09-02 | `ikllama-ornith-dankserver.md` |
|
||||||
| llama.cpp layer split + DFlash2 (`frank-medium`) | Dankserver | 2026-09-16 → 09-20 | `frank-medium-llamacpp-dflash2.md` |
|
| llama.cpp layer split + DFlash2 (`frank-medium`) | Dankserver | 2026-09-16 → 09-20 | `frank-medium-llamacpp-dflash2.md` |
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ one deploy script.
|
|||||||
| Host | User | IP | Podman stack | LLM service | LLM port |
|
| Host | User | IP | Podman stack | LLM service | LLM port |
|
||||||
|------|------|----|--------------|-------------|----------|
|
|------|------|----|--------------|-------------|----------|
|
||||||
| dankserver | dank | 192.168.0.235 | media and library | frank-medium (ROCmFPX llama.cpp, Qwen3.8-27B AGENT) | 7770 |
|
| dankserver | dank | 192.168.0.235 | media and library | frank-medium (ROCmFPX llama.cpp, Qwen3.8-27B AGENT) | 7770 |
|
||||||
| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-large (gufo, Qwen3.8-27B UD-Q8_K_XL) | 7770 |
|
| dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-dragoon (gufo, Qwen3.8-27B UD-Q8_K_XL) | 7770 |
|
||||||
|
|
||||||
## Active Services
|
## Active Services
|
||||||
|
|
||||||
@@ -30,7 +30,7 @@ one deploy script.
|
|||||||
| Gitea | dankclaw | gitea.dankclaw.local | 3001 |
|
| Gitea | dankclaw | gitea.dankclaw.local | 3001 |
|
||||||
| Grafana | dankclaw | grafana.dankclaw.local | 3002 |
|
| Grafana | dankclaw | grafana.dankclaw.local | 3002 |
|
||||||
| Prometheus | dankclaw | prometheus.dankclaw.local | 9090 |
|
| Prometheus | dankclaw | prometheus.dankclaw.local | 9090 |
|
||||||
| frank-large | dankclaw | — (bare endpoint) | 7770 |
|
| frank-dragoon | dankclaw | — (bare endpoint) | 7770 |
|
||||||
|
|
||||||
## Structure
|
## Structure
|
||||||
|
|
||||||
|
|||||||
+9
-9
@@ -1,26 +1,26 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# autoresearch.sh — canonical frank-primary benchmark entry point (dev box).
|
# autoresearch.sh — canonical frank-archon benchmark entry point (dev box).
|
||||||
#
|
#
|
||||||
# The service runs on dankserver (192.168.0.235:7770), the two R9700s live
|
# The service runs on dankserver (192.168.0.235:7770), the two R9700s live
|
||||||
# there, and the benchmark must own those cards, so the whole stop -> boot
|
# there, and the benchmark must own those cards, so the whole stop -> boot
|
||||||
# scratch -> bench -> restore sequence runs ON dankserver (see
|
# scratch -> bench -> restore sequence runs ON dankserver (see
|
||||||
# scripts/autoresearch-frank-primary.sh). This wrapper:
|
# scripts/autoresearch-frank-archon.sh). This wrapper:
|
||||||
# 1. copies the candidate config (the repo's scripts/frank-primary.env, the
|
# 1. copies the candidate config (the repo's scripts/frank-archon.env, the
|
||||||
# file the autoresearch loop edits) and the bench script to dankserver,
|
# file the autoresearch loop edits) and the bench script to dankserver,
|
||||||
# 2. runs the bench over ssh, relaying its stdout (the METRIC lines) and
|
# 2. runs the bench over ssh, relaying its stdout (the METRIC lines) and
|
||||||
# exit code back here verbatim.
|
# exit code back here verbatim.
|
||||||
# It deliberately does NOT push to origin: the scp dirties dankserver's
|
# It deliberately does NOT push to origin: the scp dirties dankserver's
|
||||||
# working tree, and the loop's uncommitted candidate config must not be
|
# working tree, and the loop's uncommitted candidate config must not be
|
||||||
# pushed. Only this file and scripts/autoresearch-frank-primary.sh are
|
# pushed. Only this file and scripts/autoresearch-frank-archon.sh are
|
||||||
# committed; the candidate env rides over ssh each run.
|
# committed; the candidate env rides over ssh each run.
|
||||||
set -uo pipefail
|
set -uo pipefail
|
||||||
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
REMOTE=dankserver
|
REMOTE=dankserver
|
||||||
REMOTE_REPO=/home/dank/serverAppConfigs
|
REMOTE_REPO=/home/dank/serverAppConfigs
|
||||||
|
|
||||||
scp -q "$DIR/scripts/frank-primary.env" \
|
scp -q "$DIR/scripts/frank-archon.env" \
|
||||||
"$REMOTE:$REMOTE_REPO/scripts/frank-primary.env" || { echo "scp env failed" >&2; exit 1; }
|
"$REMOTE:$REMOTE_REPO/scripts/frank-archon.env" || { echo "scp env failed" >&2; exit 1; }
|
||||||
scp -q "$DIR/scripts/autoresearch-frank-primary.sh" \
|
scp -q "$DIR/scripts/autoresearch-frank-archon.sh" \
|
||||||
"$REMOTE:$REMOTE_REPO/scripts/autoresearch-frank-primary.sh" || { echo "scp script failed" >&2; exit 1; }
|
"$REMOTE:$REMOTE_REPO/scripts/autoresearch-frank-archon.sh" || { echo "scp script failed" >&2; exit 1; }
|
||||||
|
|
||||||
ssh "$REMOTE" "bash $REMOTE_REPO/scripts/autoresearch-frank-primary.sh $REMOTE_REPO/scripts/frank-primary.env"
|
ssh "$REMOTE" "bash $REMOTE_REPO/scripts/autoresearch-frank-archon.sh $REMOTE_REPO/scripts/frank-archon.env"
|
||||||
+10
-1
@@ -337,7 +337,16 @@ http://:8017 {
|
|||||||
try_files /docs.html
|
try_files /docs.html
|
||||||
file_server
|
file_server
|
||||||
}
|
}
|
||||||
@data path /services.json /status.json /app.js /style.css /deployed.js
|
# /whoami echoes the outpost-injected identity headers so the page
|
||||||
|
# can gate the admin button client-side (the /admin* 403 above stays
|
||||||
|
# the real gate; this only decides what is shown). The outpost strips
|
||||||
|
# client-supplied X-Authentik-* before injecting, so this is spoof-proof.
|
||||||
|
handle /whoami {
|
||||||
|
templates
|
||||||
|
header Content-Type application/json
|
||||||
|
respond "{\"user\":\"{header.X-Authentik-Username}\",\"groups\":\"{header.X-Authentik-Groups}\"}"
|
||||||
|
}
|
||||||
|
@data path /services.json /status.json /app.js /style.css /deployed.js /img/* /sfx/*
|
||||||
handle @data {
|
handle @data {
|
||||||
file_server
|
file_server
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -10,15 +10,15 @@ model:
|
|||||||
provider: dankclaw-large
|
provider: dankclaw-large
|
||||||
base_url: http://192.168.0.143:7770/v1
|
base_url: http://192.168.0.143:7770/v1
|
||||||
api_key: not-needed
|
api_key: not-needed
|
||||||
default: frank-large
|
default: frank-dragoon
|
||||||
|
|
||||||
providers:
|
providers:
|
||||||
dankclaw-large:
|
dankclaw-large:
|
||||||
api: http://192.168.0.143:7770/v1
|
api: http://192.168.0.143:7770/v1
|
||||||
discover_models: false
|
discover_models: false
|
||||||
models:
|
models:
|
||||||
frank-large:
|
frank-dragoon:
|
||||||
context_length: 114688
|
context_length: 262144
|
||||||
dankserver-small:
|
dankserver-small:
|
||||||
api: http://192.168.0.235:7770/v1
|
api: http://192.168.0.235:7770/v1
|
||||||
discover_models: false
|
discover_models: false
|
||||||
@@ -29,7 +29,7 @@ providers:
|
|||||||
# Auxiliary routing from the live configuration. Named providers route via
|
# Auxiliary routing from the live configuration. Named providers route via
|
||||||
# their `api:` URL; `custom` entries route by their explicit `base_url`.
|
# their `api:` URL; `custom` entries route by their explicit `base_url`.
|
||||||
auxiliary:
|
auxiliary:
|
||||||
vision: {provider: dankclaw-large, model: frank-large}
|
vision: {provider: dankclaw-large, model: frank-dragoon}
|
||||||
web_extract: {provider: dankserver-small, model: frank-medium}
|
web_extract: {provider: dankserver-small, model: frank-medium}
|
||||||
compression: {provider: dankserver-small, model: frank-medium}
|
compression: {provider: dankserver-small, model: frank-medium}
|
||||||
skills_hub: {provider: dankserver-small, model: frank-medium}
|
skills_hub: {provider: dankserver-small, model: frank-medium}
|
||||||
@@ -50,8 +50,8 @@ auxiliary:
|
|||||||
effort: none
|
effort: none
|
||||||
memory_query_rewrite: {provider: custom, model: frank-medium, base_url: http://192.168.0.235:7770/v1}
|
memory_query_rewrite: {provider: custom, model: frank-medium, base_url: http://192.168.0.235:7770/v1}
|
||||||
tts_audio_tags: {provider: custom, model: frank-medium, base_url: http://192.168.0.235:7770/v1}
|
tts_audio_tags: {provider: custom, model: frank-medium, base_url: http://192.168.0.235:7770/v1}
|
||||||
triage_specifier: {provider: dankclaw-large, model: frank-large}
|
triage_specifier: {provider: dankclaw-large, model: frank-dragoon}
|
||||||
kanban_decomposer: {provider: dankclaw-large, model: frank-large}
|
kanban_decomposer: {provider: dankclaw-large, model: frank-dragoon}
|
||||||
profile_describer: {provider: dankserver-small, model: frank-medium}
|
profile_describer: {provider: dankserver-small, model: frank-medium}
|
||||||
curator: {provider: dankserver-small, model: frank-medium}
|
curator: {provider: dankserver-small, model: frank-medium}
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -1 +1 @@
|
|||||||
{"model": "frank-large", "prompt": "Below is a telemetry API schema, labeled SCHEMA A. Later in this message the same schema is repeated word for word, labeled SCHEMA B.\nTask: output SCHEMA B verbatim, with no commentary and no reformatting, starting immediately with the opening brace.\n\nSCHEMA A:\n{\"version\":\"2.3.1\",\"endpoints\":[{\"id\":\"ep-000\",\"path\":\"/v2/telemetry/gw5978x49\",\"method\":\"GET\",\"latency_ms\":219,\"owner\":\"team-0\",\"limit\":{\"burst\":221,\"sustained\":149}},{\"id\":\"ep-001\",\"path\":\"/v2/telemetry/gw6279x77\",\"method\":\"POST\",\"latency_ms\":456,\"owner\":\"team-1\",\"limit\":{\"burst\":227,\"sustained\":173}},{\"id\":\"ep-002\",\"path\":\"/v2/telemetry/gw4501x82\",\"method\":\"PUT\",\"latency_ms\":488,\"owner\":\"team-2\",\"limit\":{\"burst\":407,\"sustained\":189}},{\"id\":\"ep-003\",\"path\":\"/v2/telemetry/gw3780xA8\",\"method\":\"DELETE\",\"latency_ms\":90,\"owner\":\"team-3\",\"limit\":{\"burst\":146,\"sustained\":132}},{\"id\":\"ep-004\",\"path\":\"/v2/telemetry/gw6491x17\",\"method\":\"GET\",\"latency_ms\":842,\"owner\":\"team-4\",\"limit\":{\"burst\":488,\"sustained\":193}},{\"id\":\"ep-005\",\"path\":\"/v2/telemetry/gw4848xAC\",\"method\":\"POST\",\"latency_ms\":321,\"owner\":\"team-5\",\"limit\":{\"burst\":116,\"sustained\":51}},{\"id\":\"ep-006\",\"path\":\"/v2/telemetry/gw3414x06\",\"method\":\"PUT\",\"latency_ms\":51,\"owner\":\"team-6\",\"limit\":{\"burst\":126,\"sustained\":122}},{\"id\":\"ep-007\",\"path\":\"/v2/telemetry/gw2193x74\",\"method\":\"DELETE\",\"latency_ms\":428,\"owner\":\"team-0\",\"limit\":{\"burst\":454,\"sustained\":162}},{\"id\":\"ep-008\",\"path\":\"/v2/telemetry/gw4185xB7\",\"method\":\"GET\",\"latency_ms\":717,\"owner\":\"team-1\",\"limit\":{\"burst\":197,\"sustained\":127}},{\"id\":\"ep-009\",\"path\":\"/v2/telemetry/gw7547x3E\",\"method\":\"POST\",\"latency_ms\":155,\"owner\":\"team-2\",\"limit\":{\"burst\":336,\"sustained\":177}},{\"id\":\"ep-010\",\"path\":\"/v2/telemetry/gw1090xE4\",\"method\":\"PUT\",\"latency_ms\":772,\"owner\":\"team-3\",\"limit\":{\"burst\":441,\"sustained\":198}},{\"id\":\"ep-011\",\"path\":\"/v2/telemetry/gw2746xC7\",\"method\":\"DELETE\",\"latency_ms\":439,\"owner\":\"team-4\",\"limit\":{\"burst\":113,\"sustained\":46}},{\"id\":\"ep-012\",\"path\":\"/v2/telemetry/gw9486x76\",\"method\":\"GET\",\"latency_ms\":55,\"owner\":\"team-5\",\"limit\":{\"burst\":286,\"sustained\":64}},{\"id\":\"ep-013\",\"path\":\"/v2/telemetry/gw2988x74\",\"method\":\"POST\",\"latency_ms\":140,\"owner\":\"team-6\",\"limit\":{\"burst\":411,\"sustained\":119}},{\"id\":\"ep-014\",\"path\":\"/v2/telemetry/gw9702xFC\",\"method\":\"PUT\",\"latency_ms\":576,\"owner\":\"team-0\",\"limit\":{\"burst\":305,\"sustained\":82}},{\"id\":\"ep-015\",\"path\":\"/v2/telemetry/gw8251x9C\",\"method\":\"DELETE\",\"latency_ms\":838,\"owner\":\"team-1\",\"limit\":{\"burst\":369,\"sustained\":130}},{\"id\":\"ep-016\",\"path\":\"/v2/telemetry/gw7991xD4\",\"method\":\"GET\",\"latency_ms\":565,\"owner\":\"team-2\",\"limit\":{\"burst\":229,\"sustained\":41}},{\"id\":\"ep-017\",\"path\":\"/v2/telemetry/gw8777x73\",\"method\":\"POST\",\"latency_ms\":269,\"owner\":\"team-3\",\"limit\":{\"burst\":385,\"sustained\":64}},{\"id\":\"ep-018\",\"path\":\"/v2/telemetry/gw5543xC4\",\"method\":\"PUT\",\"latency_ms\":800,\"owner\":\"team-4\",\"limit\":{\"burst\":267,\"sustained\":125}},{\"id\":\"ep-019\",\"path\":\"/v2/telemetry/gw4919x46\",\"method\":\"DELETE\",\"latency_ms\":454,\"owner\":\"team-5\",\"limit\":{\"burst\":40,\"sustained\":183}},{\"id\":\"ep-020\",\"path\":\"/v2/telemetry/gw5681x3C\",\"method\":\"GET\",\"latency_ms\":282,\"owner\":\"team-6\",\"limit\":{\"burst\":172,\"sustained\":82}},{\"id\":\"ep-021\",\"path\":\"/v2/telemetry/gw9849x14\",\"method\":\"POST\",\"latency_ms\":145,\"owner\":\"team-0\",\"limit\":{\"burst\":78,\"sustained\":60}},{\"id\":\"ep-022\",\"path\":\"/v2/telemetry/gw7275xB1\",\"method\":\"PUT\",\"latency_ms\":160,\"owner\":\"team-1\",\"limit\":{\"burst\":362,\"sustained\":55}},{\"id\":\"ep-023\",\"path\":\"/v2/telemetry/gw2052x6A\",\"method\":\"DELETE\",\"latency_ms\":421,\"owner\":\"team-2\",\"limit\":{\"burst\":170,\"sustained\":139}},{\"id\":\"ep-024\",\"path\":\"/v2/telemetry/gw8633x6A\",\"method\":\"GET\",\"latency_ms\":67,\"owner\":\"team-3\",\"limit\":{\"burst\":106,\"sustained\":108}},{\"id\":\"ep-025\",\"path\":\"/v2/telemetry/gw7381xE7\",\"method\":\"POST\",\"latency_ms\":792,\"owner\":\"team-4\",\"limit\":{\"burst\":300,\"sustained\":179}},{\"id\":\"ep-026\",\"path\":\"/v2/telemetry/gw1320xDB\",\"method\":\"PUT\",\"latency_ms\":787,\"owner\":\"team-5\",\"limit\":{\"burst\":295,\"sustained\":98}},{\"id\":\"ep-027\",\"path\":\"/v2/telemetry/gw8814x01\",\"method\":\"DELETE\",\"latency_ms\":364,\"owner\":\"team-6\",\"limit\":{\"burst\":153,\"sustained\":193}},{\"id\":\"ep-028\",\"path\":\"/v2/telemetry/gw7389xDA\",\"method\":\"GET\",\"latency_ms\":859,\"owner\":\"team-0\",\"limit\":{\"burst\":215,\"sustained\":138}},{\"id\"Line truncated
|
{"model": "frank-dragoon", "prompt": "Below is a telemetry API schema, labeled SCHEMA A. Later in this message the same schema is repeated word for word, labeled SCHEMA B.\nTask: output SCHEMA B verbatim, with no commentary and no reformatting, starting immediately with the opening brace.\n\nSCHEMA A:\n{\"version\":\"2.3.1\",\"endpoints\":[{\"id\":\"ep-000\",\"path\":\"/v2/telemetry/gw5978x49\",\"method\":\"GET\",\"latency_ms\":219,\"owner\":\"team-0\",\"limit\":{\"burst\":221,\"sustained\":149}},{\"id\":\"ep-001\",\"path\":\"/v2/telemetry/gw6279x77\",\"method\":\"POST\",\"latency_ms\":456,\"owner\":\"team-1\",\"limit\":{\"burst\":227,\"sustained\":173}},{\"id\":\"ep-002\",\"path\":\"/v2/telemetry/gw4501x82\",\"method\":\"PUT\",\"latency_ms\":488,\"owner\":\"team-2\",\"limit\":{\"burst\":407,\"sustained\":189}},{\"id\":\"ep-003\",\"path\":\"/v2/telemetry/gw3780xA8\",\"method\":\"DELETE\",\"latency_ms\":90,\"owner\":\"team-3\",\"limit\":{\"burst\":146,\"sustained\":132}},{\"id\":\"ep-004\",\"path\":\"/v2/telemetry/gw6491x17\",\"method\":\"GET\",\"latency_ms\":842,\"owner\":\"team-4\",\"limit\":{\"burst\":488,\"sustained\":193}},{\"id\":\"ep-005\",\"path\":\"/v2/telemetry/gw4848xAC\",\"method\":\"POST\",\"latency_ms\":321,\"owner\":\"team-5\",\"limit\":{\"burst\":116,\"sustained\":51}},{\"id\":\"ep-006\",\"path\":\"/v2/telemetry/gw3414x06\",\"method\":\"PUT\",\"latency_ms\":51,\"owner\":\"team-6\",\"limit\":{\"burst\":126,\"sustained\":122}},{\"id\":\"ep-007\",\"path\":\"/v2/telemetry/gw2193x74\",\"method\":\"DELETE\",\"latency_ms\":428,\"owner\":\"team-0\",\"limit\":{\"burst\":454,\"sustained\":162}},{\"id\":\"ep-008\",\"path\":\"/v2/telemetry/gw4185xB7\",\"method\":\"GET\",\"latency_ms\":717,\"owner\":\"team-1\",\"limit\":{\"burst\":197,\"sustained\":127}},{\"id\":\"ep-009\",\"path\":\"/v2/telemetry/gw7547x3E\",\"method\":\"POST\",\"latency_ms\":155,\"owner\":\"team-2\",\"limit\":{\"burst\":336,\"sustained\":177}},{\"id\":\"ep-010\",\"path\":\"/v2/telemetry/gw1090xE4\",\"method\":\"PUT\",\"latency_ms\":772,\"owner\":\"team-3\",\"limit\":{\"burst\":441,\"sustained\":198}},{\"id\":\"ep-011\",\"path\":\"/v2/telemetry/gw2746xC7\",\"method\":\"DELETE\",\"latency_ms\":439,\"owner\":\"team-4\",\"limit\":{\"burst\":113,\"sustained\":46}},{\"id\":\"ep-012\",\"path\":\"/v2/telemetry/gw9486x76\",\"method\":\"GET\",\"latency_ms\":55,\"owner\":\"team-5\",\"limit\":{\"burst\":286,\"sustained\":64}},{\"id\":\"ep-013\",\"path\":\"/v2/telemetry/gw2988x74\",\"method\":\"POST\",\"latency_ms\":140,\"owner\":\"team-6\",\"limit\":{\"burst\":411,\"sustained\":119}},{\"id\":\"ep-014\",\"path\":\"/v2/telemetry/gw9702xFC\",\"method\":\"PUT\",\"latency_ms\":576,\"owner\":\"team-0\",\"limit\":{\"burst\":305,\"sustained\":82}},{\"id\":\"ep-015\",\"path\":\"/v2/telemetry/gw8251x9C\",\"method\":\"DELETE\",\"latency_ms\":838,\"owner\":\"team-1\",\"limit\":{\"burst\":369,\"sustained\":130}},{\"id\":\"ep-016\",\"path\":\"/v2/telemetry/gw7991xD4\",\"method\":\"GET\",\"latency_ms\":565,\"owner\":\"team-2\",\"limit\":{\"burst\":229,\"sustained\":41}},{\"id\":\"ep-017\",\"path\":\"/v2/telemetry/gw8777x73\",\"method\":\"POST\",\"latency_ms\":269,\"owner\":\"team-3\",\"limit\":{\"burst\":385,\"sustained\":64}},{\"id\":\"ep-018\",\"path\":\"/v2/telemetry/gw5543xC4\",\"method\":\"PUT\",\"latency_ms\":800,\"owner\":\"team-4\",\"limit\":{\"burst\":267,\"sustained\":125}},{\"id\":\"ep-019\",\"path\":\"/v2/telemetry/gw4919x46\",\"method\":\"DELETE\",\"latency_ms\":454,\"owner\":\"team-5\",\"limit\":{\"burst\":40,\"sustained\":183}},{\"id\":\"ep-020\",\"path\":\"/v2/telemetry/gw5681x3C\",\"method\":\"GET\",\"latency_ms\":282,\"owner\":\"team-6\",\"limit\":{\"burst\":172,\"sustained\":82}},{\"id\":\"ep-021\",\"path\":\"/v2/telemetry/gw9849x14\",\"method\":\"POST\",\"latency_ms\":145,\"owner\":\"team-0\",\"limit\":{\"burst\":78,\"sustained\":60}},{\"id\":\"ep-022\",\"path\":\"/v2/telemetry/gw7275xB1\",\"method\":\"PUT\",\"latency_ms\":160,\"owner\":\"team-1\",\"limit\":{\"burst\":362,\"sustained\":55}},{\"id\":\"ep-023\",\"path\":\"/v2/telemetry/gw2052x6A\",\"method\":\"DELETE\",\"latency_ms\":421,\"owner\":\"team-2\",\"limit\":{\"burst\":170,\"sustained\":139}},{\"id\":\"ep-024\",\"path\":\"/v2/telemetry/gw8633x6A\",\"method\":\"GET\",\"latency_ms\":67,\"owner\":\"team-3\",\"limit\":{\"burst\":106,\"sustained\":108}},{\"id\":\"ep-025\",\"path\":\"/v2/telemetry/gw7381xE7\",\"method\":\"POST\",\"latency_ms\":792,\"owner\":\"team-4\",\"limit\":{\"burst\":300,\"sustained\":179}},{\"id\":\"ep-026\",\"path\":\"/v2/telemetry/gw1320xDB\",\"method\":\"PUT\",\"latency_ms\":787,\"owner\":\"team-5\",\"limit\":{\"burst\":295,\"sustained\":98}},{\"id\":\"ep-027\",\"path\":\"/v2/telemetry/gw8814x01\",\"method\":\"DELETE\",\"latency_ms\":364,\"owner\":\"team-6\",\"limit\":{\"burst\":153,\"sustained\":193}},{\"id\":\"ep-028\",\"path\":\"/v2/telemetry/gw7389xDA\",\"method\":\"GET\",\"latency_ms\":859,\"owner\":\"team-0\",\"limit\":{\"burst\":215,\"sustained\":138}},{\"idLine truncated
|
||||||
@@ -1,9 +1,9 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# download-frank-primary-fp8.sh — fetch/verify the Qwen3.8-Flash-Next-FP8
|
# download-frank-archon-fp8.sh — fetch/verify the Qwen3.8-Flash-Next-FP8
|
||||||
# payload (131 safetensors shards, 172.78 GiB total, plus aux files) for the
|
# payload (131 safetensors shards, 172.78 GiB total, plus aux files) for the
|
||||||
# vLLM frank-primary deployment on dankserver.
|
# vLLM frank-archon deployment on dankserver.
|
||||||
#
|
#
|
||||||
# bash scripts/download-frank-primary-fp8.sh [dest_dir]
|
# bash scripts/download-frank-archon-fp8.sh [dest_dir]
|
||||||
#
|
#
|
||||||
# The manifest (scripts/manifests/qwen3.8-flash-next-fp8.manifest) carries
|
# The manifest (scripts/manifests/qwen3.8-flash-next-fp8.manifest) carries
|
||||||
# per-file sizes + LFS sha256s from the Hub; hf-fetch.sh is idempotent — an
|
# per-file sizes + LFS sha256s from the Hub; hf-fetch.sh is idempotent — an
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Download the z-lab Qwen3.5-9B-DFlash drafter for frank-large's DFlash
|
# Download the z-lab Qwen3.5-9B-DFlash drafter for frank-dragoon's DFlash
|
||||||
# speculative-decode arm (vLLM --speculative-config method=dflash). Public
|
# speculative-decode arm (vLLM --speculative-config method=dflash). Public
|
||||||
# repo — no auth header on the command line.
|
# repo — no auth header on the command line.
|
||||||
#
|
#
|
||||||
@@ -10,7 +10,7 @@
|
|||||||
# the draft for that target.
|
# the draft for that target.
|
||||||
#
|
#
|
||||||
# Usage:
|
# Usage:
|
||||||
# scripts/download-frank-large-dflash.sh [DEST_DIR] [--verify-only]
|
# scripts/download-frank-dragoon-dflash.sh [DEST_DIR] [--verify-only]
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
REPO="z-lab/Qwen3.5-9B-DFlash"
|
REPO="z-lab/Qwen3.5-9B-DFlash"
|
||||||
@@ -1,14 +1,14 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Download the BF16 MiMo-V2.6-Distill-Qwen-9B payload for frank-large
|
# Download the BF16 MiMo-V2.6-Distill-Qwen-9B payload for frank-dragoon
|
||||||
# (vLLM/kyuz0 gfx1151, Strix Halo iGPU). Public repo — no auth header on the
|
# (vLLM/kyuz0 gfx1151, Strix Halo iGPU). Public repo — no auth header on the
|
||||||
# command line (a Bearer token in argv is world-readable via /proc).
|
# command line (a Bearer token in argv is world-readable via /proc).
|
||||||
#
|
#
|
||||||
# Per-file FRESH downloads, no resume: the house rule (see
|
# Per-file FRESH downloads, no resume: the house rule (see
|
||||||
# download-frank-secondary.sh); -f/--fail-with-body plus size+sha256 against the
|
# download-frank-zealot.sh); -f/--fail-with-body plus size+sha256 against the
|
||||||
# Hub LFS oid is what makes an access error fail instead of "succeeding".
|
# Hub LFS oid is what makes an access error fail instead of "succeeding".
|
||||||
#
|
#
|
||||||
# Usage:
|
# Usage:
|
||||||
# scripts/download-frank-large-mimo.sh [DEST_DIR] [--verify-only]
|
# scripts/download-frank-dragoon-mimo.sh [DEST_DIR] [--verify-only]
|
||||||
#
|
#
|
||||||
# The verified set below is exactly what the 2026-10-01 bench window served:
|
# The verified set below is exactly what the 2026-10-01 bench window served:
|
||||||
# 17.53 GiB of BF16 weights (4 shards, 18,819,720,848 B total with the rest).
|
# 17.53 GiB of BF16 weights (4 shards, 18,819,720,848 B total with the rest).
|
||||||
@@ -1,14 +1,14 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Download the source Qwen3.5-9B checkpoint — the ONLY place this model
|
# Download the source Qwen3.5-9B checkpoint — the ONLY place this model
|
||||||
# family's MTP head (15 mtp.* tensors) exists. frank-large runs it as the
|
# family's MTP head (15 mtp.* tensors) exists. frank-dragoon runs it as the
|
||||||
# vLLM cross-model MTP draft (MiMo-V2.6-Distill-Qwen-9B itself ships zero
|
# vLLM cross-model MTP draft (MiMo-V2.6-Distill-Qwen-9B itself ships zero
|
||||||
# mtp tensors; that is why SGLang's NEXTN on frank-secondary draws 0.00
|
# mtp tensors; that is why SGLang's NEXTN on frank-zealot draws 0.00
|
||||||
# acceptance). Public repo — no auth header on the command line.
|
# acceptance). Public repo — no auth header on the command line.
|
||||||
#
|
#
|
||||||
# NOTE the odd upstream shard names (model.safetensors-0000N-of-00004...).
|
# NOTE the odd upstream shard names (model.safetensors-0000N-of-00004...).
|
||||||
#
|
#
|
||||||
# Usage:
|
# Usage:
|
||||||
# scripts/download-frank-large-qwen35.sh [DEST_DIR] [--verify-only]
|
# scripts/download-frank-dragoon-qwen35.sh [DEST_DIR] [--verify-only]
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
REPO="Qwen/Qwen3.5-9B"
|
REPO="Qwen/Qwen3.5-9B"
|
||||||
+2
-2
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-secondary
|
# Download MiMo-V2.6-Distill-Qwen-9B-INT8-W8A16-AutoRound for frank-zealot
|
||||||
# (SGLang, RTX 3090). Public repo — no auth header on the command line.
|
# (SGLang, RTX 3090). Public repo — no auth header on the command line.
|
||||||
#
|
#
|
||||||
# Per-file FRESH downloads, no resume: `curl -C -` once produced a 213 MB
|
# Per-file FRESH downloads, no resume: `curl -C -` once produced a 213 MB
|
||||||
@@ -7,7 +7,7 @@
|
|||||||
# the house rule.
|
# the house rule.
|
||||||
#
|
#
|
||||||
# Usage:
|
# Usage:
|
||||||
# scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only]
|
# scripts/download-frank-zealot.sh [DEST_DIR] [--verify-only]
|
||||||
#
|
#
|
||||||
# LFS shards are verified against the Hub LFS sha256; non-LFS text files
|
# LFS shards are verified against the Hub LFS sha256; non-LFS text files
|
||||||
# against their declared byte size. Idempotent: complete files are skipped.
|
# against their declared byte size. Idempotent: complete files are skipped.
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# bench-frank-large.sh — 64k-context benchmark for the frank-default-large stack.
|
# bench-frank-dragoon.sh — 64k-context benchmark for the frank-default-large stack.
|
||||||
#
|
#
|
||||||
# Measures generation throughput of Qwen3.6-27B Q8_K_P + DFlash + ngram-mod on
|
# Measures generation throughput of Qwen3.6-27B Q8_K_P + DFlash + ngram-mod on
|
||||||
# a single backend at 64k context. It starts llama-server on a scratch port,
|
# a single backend at 64k context. It starts llama-server on a scratch port,
|
||||||
@@ -7,15 +7,15 @@
|
|||||||
# prompt, and prints tokens/sec.
|
# prompt, and prints tokens/sec.
|
||||||
#
|
#
|
||||||
# Usage:
|
# Usage:
|
||||||
# ./scripts/bench-frank-large.sh <ROCm0|Vulkan0> [--generations N]
|
# ./scripts/bench-frank-dragoon.sh <ROCm0|Vulkan0> [--generations N]
|
||||||
#
|
#
|
||||||
# Example:
|
# Example:
|
||||||
# ./scripts/bench-frank-large.sh ROCm0
|
# ./scripts/bench-frank-dragoon.sh ROCm0
|
||||||
# ./scripts/bench-frank-large.sh Vulkan0 --generations 3
|
# ./scripts/bench-frank-dragoon.sh Vulkan0 --generations 3
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
DEVICE="${1:?usage: bench-frank-large.sh <ROCm0|Vulkan0> [--generations N]}"
|
DEVICE="${1:?usage: bench-frank-dragoon.sh <ROCm0|Vulkan0> [--generations N]}"
|
||||||
GENERATIONS=1
|
GENERATIONS=1
|
||||||
if [ "${2:-}" = "--generations" ]; then
|
if [ "${2:-}" = "--generations" ]; then
|
||||||
GENERATIONS="${3:?--generations needs a count}"
|
GENERATIONS="${3:?--generations needs a count}"
|
||||||
@@ -41,7 +41,7 @@ SERVICE_HEALTH[dankclaw]="http://127.0.0.1:7770/health"
|
|||||||
# Drop page caches before restart: with the model's pages fully cached, the
|
# Drop page caches before restart: with the model's pages fully cached, the
|
||||||
# KV/graph allocation stalls at load (VRAM stuck at 28 GiB). See AGENTS.md.
|
# KV/graph allocation stalls at load (VRAM stuck at 28 GiB). See AGENTS.md.
|
||||||
DROP_CACHES[dankclaw]=1
|
DROP_CACHES[dankclaw]=1
|
||||||
# Passwordless restart helper (root-owned, installed via /etc/sudoers.d/frank-large).
|
# Passwordless restart helper (root-owned, installed via /etc/sudoers.d/frank-dragoon).
|
||||||
# When present, deploy.sh uses it instead of inline sudo commands.
|
# When present, deploy.sh uses it instead of inline sudo commands.
|
||||||
HELPER[dankclaw]="/usr/local/sbin/frank-restart-large.sh"
|
HELPER[dankclaw]="/usr/local/sbin/frank-restart-large.sh"
|
||||||
# Health wait in iterations of 2s. Cold loads after the cache drop measured
|
# Health wait in iterations of 2s. Cold loads after the cache drop measured
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Tunables for the Ling-3.0-flash TRAIL on dankclaw (frank-large-ling).
|
# Tunables for the Ling-3.0-flash TRAIL on dankclaw (frank-dragoon-ling).
|
||||||
# This REPLACES frank-default-large (Qwen3.6-27B) temporarily. To switch
|
# This REPLACES frank-default-large (Qwen3.6-27B) temporarily. To switch
|
||||||
# back: systemctl stop frank-large-ling && systemctl start frank-default-large.
|
# back: systemctl stop frank-dragoon-ling && systemctl start frank-default-large.
|
||||||
# Runtime: llama.cpp FORK bailingmoe3-support; Vulkan + AMD GTT.
|
# Runtime: llama.cpp FORK bailingmoe3-support; Vulkan + AMD GTT.
|
||||||
#
|
#
|
||||||
# Memory model (amdgpu Strix Halo gfx1151, 8x16GB LPDDR5 = 128 GiB physical):
|
# Memory model (amdgpu Strix Halo gfx1151, 8x16GB LPDDR5 = 128 GiB physical):
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
[Unit]
|
[Unit]
|
||||||
Description=llama.cpp Ling-3.0-flash Q6_K Vulkan+MTP (frank-large-ling TRIAL)
|
Description=llama.cpp Ling-3.0-flash Q6_K Vulkan+MTP (frank-dragoon-ling TRIAL)
|
||||||
After=network.target
|
After=network.target
|
||||||
|
|
||||||
[Service]
|
[Service]
|
||||||
@@ -8,7 +8,7 @@
|
|||||||
# Vulkan exposes as an ~111 GiB heap). MTP drafter is bundled in the same
|
# Vulkan exposes as an ~111 GiB heap). MTP drafter is bundled in the same
|
||||||
# GGUF (block 42) - no separate --spec-draft-model needed.
|
# GGUF (block 42) - no separate --spec-draft-model needed.
|
||||||
#
|
#
|
||||||
# Tie-back: to restore the Qwen3.6-27B service, stop frank-large-ling and
|
# Tie-back: to restore the Qwen3.6-27B service, stop frank-dragoon-ling and
|
||||||
# start frank-default-large (its env/unit are committed and unchanged).
|
# start frank-default-large (its env/unit are committed and unchanged).
|
||||||
#
|
#
|
||||||
# Tunables come from frank-default-large.ling.env in this directory unless
|
# Tunables come from frank-default-large.ling.env in this directory unless
|
||||||
|
|||||||
@@ -4,11 +4,11 @@
|
|||||||
# Runtime: llama.cpp llama-server (>= b10411).
|
# Runtime: llama.cpp llama-server (>= b10411).
|
||||||
#
|
#
|
||||||
# Tunables come from frank-default-large.env in this directory unless the
|
# Tunables come from frank-default-large.env in this directory unless the
|
||||||
# FRANK_LARGE_ENV_FILE environment variable overrides that path.
|
# FRANK_DRAGOON_ENV_FILE environment variable overrides that path.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-default-large.env}"
|
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-default-large.env}"
|
||||||
if [[ -f "$ENV_FILE" ]]; then
|
if [[ -f "$ENV_FILE" ]]; then
|
||||||
set -a
|
set -a
|
||||||
# shellcheck disable=SC1090
|
# shellcheck disable=SC1090
|
||||||
+1
-1
@@ -1 +1 @@
|
|||||||
{"model": "frank-large", "prompt": "# Subsurface Aquifer Monitoring Network - System Design Document (Revision 7)\n\nThis document describes the deployment of a subsurface aquifer monitoring network for the Cascadia sedimentary basin. It is written for the operations crew and the regional water authority. Every sensor node, relay, aggregation service, calibration procedure, and failure mode below carries a stable alphanumeric identifier that appears in every table and in the field notes.\n\n## Sensor nodes\nNode SN-000-08B9 sits in a discharge lens at 359.0 m below datum and reports a salinity of 2.38 ppt with the fractured interval reading 5.40 degC at dawn and the wellhead porous 1.72 m3/d while the electrode pair returns 3157 uS/cm.\nNode SN-001-0A65 sits in a percolate lens at 25.9 m below datum and steps down to a salinity of 2.13 ppt with the diapir interval reading 8.55 degC at dawn and the wellhead silty 0.67 m3/d while the electrode pair returns 979 uS/cm.\nNode SN-002-0571 sits in a diapir lens at 287.8 m below datum and peaks at a salinity of 7.63 ppt with the karst interval reading 4.26 degC at dawn and the wellhead spooned 11.50 m3/d while the electrode pair returns 2987 uS/cm.\nNode SN-003-0CE9 sits in a confining lens at 169.5 m below datum and logs a salinity of 0.96 ppt with the alluvial interval reading 5.03 degC at dawn and the wellhead salinized 12.08 m3/d while the electrode pair returns 2607 uS/cm.\nNode SN-004-0E2A sits in a freshened lens at 70.4 m below datum and settles near a salinity of 8.68 ppt with the spooned interval reading 4.88 degC at dawn and the wellhead siphon 4.38 m3/d while the electrode pair returns 2707 uS/cm.\nNode SN-005-0DDD sits in a percolate lens at 44.6 m below datum and reports a salinity of 6.25 ppt with the alluvial interval reading 10.85 degC at dawn and the wellhead brackish 13.99 m3/d while the electrode pair returns 3618 uS/cm.\nNode SN-006-0EFC sits in a till lens at 309.5 m below datum and reports a salinity of 3.49 ppt with the lacustrine interval reading 7.38 degC at dawn and the wellhead spooned 3.21 m3/d while the electrode pair returns 1253 uS/cm.\nNode SN-007-0B02 sits in a recharge lens at 262.0 m below datum and steps down to a salinity of 2.38 ppt with the discharge interval reading 8.18 degC at dawn and the wellhead perched 4.05 m3/d while the electrode pair returns 3900 uS/cm.\nNode SN-008-0448 sits in a karst lens at 406.5 m below datum and reports a salinity of 7.32 ppt with the upconing interval reading 6.17 degC at dawn and the wellhead fluvial 0.75 m3/d while the electrode pair returns 1448 uS/cm.\nNode SN-009-043C sits in a alluvial lens at 159.3 m below datum and flags a salinity of 6.20 ppt with the percolate interval reading 7.60 degC at dawn and the wellhead brackish 13.01 m3/d while the electrode pair returns 2025 uS/cm.\nNode SN-010-0FF5 sits in a karst lens at 264.2 m below datum and flags a salinity of 7.05 ppt with the freshened interval reading 7.88 degC at dawn and the wellhead siphon 8.41 m3/d while the electrode pair returns 1635 uS/cm.\nNode SN-011-098A sits in a drain lens at 365.7 m below datum and reports a salinity of 8.11 ppt with the wedge interval reading 5.51 degC at dawn and the wellhead glacial 2.52 m3/d while the electrode pair returns 2916 uS/cm.\nNode SN-012-0AE8 sits in a till lens at 259.6 m below datum and steps down to a salinity of 9.13 ppt with the recharge interval reading 11.60 degC at dawn and the wellhead spooned 0.46 m3/d while the electrode pair returns 3077 uS/cm.\nNode SN-013-0B83 sits in a salinized lens at 171.2 m below datum and peaks at a salinity of 2.83 ppt with the lacustrine interval reading 5.56 degC at dawn and the wellhead discharge 0.35 m3/d while the electrode pair returns 3083 uS/cm.\nNode SN-014-032E sits in a drain lens at 249.6 m below datum and flags a salinity of 1.09 ppt with the salinized interval reading 9.58 degC at dawn and the wellhead perched 12.00 m3/d while the electrode pair returns 2221 uS/cm.\nNode SN-015-004F sits in a aeolian lens at 264.4 m below datum and reports a salinity of 7.34 ppt with the salinized interval reading 8.76 degC at dawn and the wellhead perched 0.31 m3/d while the electrode pair returns 1483 uS/cm.\nNode SN-016-07C6 sits in a aeolian lens at 155.9 m below datum and reports a salinity of 0.64 ppt with the mineralized interval reading 11.75 degC at dawn and the wellhead diluted 13.46 m3/d while the electrode pair returns 524 uS/cm.\nNode SN-017-06C5 sits in a salinized lens at 72.1 m below datum and averages a salinity of 4.52 ppt with the siphon interval reading 8.93 degC at dawn and the wellhead confining 3.98 m3/d while the electrode pair returns 3687 uS/cm.\nNode SN-018-0E64 sits in a siphon lens at 334.9 m below datum and steps down to a salinity of 6.73 ppt with the salinized interval reading 7.63 degC at dawn and the wellhead freshened 9.64 m3/d while the electrode pair returns 1681 uS/cm.\nNode SN-019-0122 sits in a spooned lens at 1Line truncated
|
{"model": "frank-dragoon", "prompt": "# Subsurface Aquifer Monitoring Network - System Design Document (Revision 7)\n\nThis document describes the deployment of a subsurface aquifer monitoring network for the Cascadia sedimentary basin. It is written for the operations crew and the regional water authority. Every sensor node, relay, aggregation service, calibration procedure, and failure mode below carries a stable alphanumeric identifier that appears in every table and in the field notes.\n\n## Sensor nodes\nNode SN-000-08B9 sits in a discharge lens at 359.0 m below datum and reports a salinity of 2.38 ppt with the fractured interval reading 5.40 degC at dawn and the wellhead porous 1.72 m3/d while the electrode pair returns 3157 uS/cm.\nNode SN-001-0A65 sits in a percolate lens at 25.9 m below datum and steps down to a salinity of 2.13 ppt with the diapir interval reading 8.55 degC at dawn and the wellhead silty 0.67 m3/d while the electrode pair returns 979 uS/cm.\nNode SN-002-0571 sits in a diapir lens at 287.8 m below datum and peaks at a salinity of 7.63 ppt with the karst interval reading 4.26 degC at dawn and the wellhead spooned 11.50 m3/d while the electrode pair returns 2987 uS/cm.\nNode SN-003-0CE9 sits in a confining lens at 169.5 m below datum and logs a salinity of 0.96 ppt with the alluvial interval reading 5.03 degC at dawn and the wellhead salinized 12.08 m3/d while the electrode pair returns 2607 uS/cm.\nNode SN-004-0E2A sits in a freshened lens at 70.4 m below datum and settles near a salinity of 8.68 ppt with the spooned interval reading 4.88 degC at dawn and the wellhead siphon 4.38 m3/d while the electrode pair returns 2707 uS/cm.\nNode SN-005-0DDD sits in a percolate lens at 44.6 m below datum and reports a salinity of 6.25 ppt with the alluvial interval reading 10.85 degC at dawn and the wellhead brackish 13.99 m3/d while the electrode pair returns 3618 uS/cm.\nNode SN-006-0EFC sits in a till lens at 309.5 m below datum and reports a salinity of 3.49 ppt with the lacustrine interval reading 7.38 degC at dawn and the wellhead spooned 3.21 m3/d while the electrode pair returns 1253 uS/cm.\nNode SN-007-0B02 sits in a recharge lens at 262.0 m below datum and steps down to a salinity of 2.38 ppt with the discharge interval reading 8.18 degC at dawn and the wellhead perched 4.05 m3/d while the electrode pair returns 3900 uS/cm.\nNode SN-008-0448 sits in a karst lens at 406.5 m below datum and reports a salinity of 7.32 ppt with the upconing interval reading 6.17 degC at dawn and the wellhead fluvial 0.75 m3/d while the electrode pair returns 1448 uS/cm.\nNode SN-009-043C sits in a alluvial lens at 159.3 m below datum and flags a salinity of 6.20 ppt with the percolate interval reading 7.60 degC at dawn and the wellhead brackish 13.01 m3/d while the electrode pair returns 2025 uS/cm.\nNode SN-010-0FF5 sits in a karst lens at 264.2 m below datum and flags a salinity of 7.05 ppt with the freshened interval reading 7.88 degC at dawn and the wellhead siphon 8.41 m3/d while the electrode pair returns 1635 uS/cm.\nNode SN-011-098A sits in a drain lens at 365.7 m below datum and reports a salinity of 8.11 ppt with the wedge interval reading 5.51 degC at dawn and the wellhead glacial 2.52 m3/d while the electrode pair returns 2916 uS/cm.\nNode SN-012-0AE8 sits in a till lens at 259.6 m below datum and steps down to a salinity of 9.13 ppt with the recharge interval reading 11.60 degC at dawn and the wellhead spooned 0.46 m3/d while the electrode pair returns 3077 uS/cm.\nNode SN-013-0B83 sits in a salinized lens at 171.2 m below datum and peaks at a salinity of 2.83 ppt with the lacustrine interval reading 5.56 degC at dawn and the wellhead discharge 0.35 m3/d while the electrode pair returns 3083 uS/cm.\nNode SN-014-032E sits in a drain lens at 249.6 m below datum and flags a salinity of 1.09 ppt with the salinized interval reading 9.58 degC at dawn and the wellhead perched 12.00 m3/d while the electrode pair returns 2221 uS/cm.\nNode SN-015-004F sits in a aeolian lens at 264.4 m below datum and reports a salinity of 7.34 ppt with the salinized interval reading 8.76 degC at dawn and the wellhead perched 0.31 m3/d while the electrode pair returns 1483 uS/cm.\nNode SN-016-07C6 sits in a aeolian lens at 155.9 m below datum and reports a salinity of 0.64 ppt with the mineralized interval reading 11.75 degC at dawn and the wellhead diluted 13.46 m3/d while the electrode pair returns 524 uS/cm.\nNode SN-017-06C5 sits in a salinized lens at 72.1 m below datum and averages a salinity of 4.52 ppt with the siphon interval reading 8.93 degC at dawn and the wellhead confining 3.98 m3/d while the electrode pair returns 3687 uS/cm.\nNode SN-018-0E64 sits in a siphon lens at 334.9 m below datum and steps down to a salinity of 6.73 ppt with the salinized interval reading 7.63 degC at dawn and the wellhead freshened 9.64 m3/d while the electrode pair returns 1681 uS/cm.\nNode SN-019-0122 sits in a spooned lens atLine truncated
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""
|
"""
|
||||||
Decode throughput benchmark for frank-large.
|
Decode throughput benchmark for frank-dragoon.
|
||||||
Starts the server with a given --threads-batch value, sends an 8K+ token prompt,
|
Starts the server with a given --threads-batch value, sends an 8K+ token prompt,
|
||||||
generates 256 tokens, measures generation tok/s.
|
generates 256 tokens, measures generation tok/s.
|
||||||
|
|
||||||
@@ -13,12 +13,12 @@ SERVE = "/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server"
|
|||||||
PORT = 7770
|
PORT = 7770
|
||||||
N_GEN = 256
|
N_GEN = 256
|
||||||
PROMPT_FILE = "/tmp/bench-prompts/prompt_code.txt"
|
PROMPT_FILE = "/tmp/bench-prompts/prompt_code.txt"
|
||||||
ENV_FILE = os.path.join(os.path.dirname(__file__), "frank-large.env")
|
ENV_FILE = os.path.join(os.path.dirname(__file__), "frank-dragoon.env")
|
||||||
SERVER_DIR = os.path.dirname(SERVE)
|
SERVER_DIR = os.path.dirname(SERVE)
|
||||||
ROCM_PATH = "/opt/rocm"
|
ROCM_PATH = "/opt/rocm"
|
||||||
|
|
||||||
def make_env():
|
def make_env():
|
||||||
"""Build the same env as serve-frank-large.sh."""
|
"""Build the same env as serve-frank-dragoon.sh."""
|
||||||
env = {**os.environ, "HSA_OVERRIDE_GFX_VERSION": "11.5.1", "HIP_VISIBLE_DEVICES": "0"}
|
env = {**os.environ, "HSA_OVERRIDE_GFX_VERSION": "11.5.1", "HIP_VISIBLE_DEVICES": "0"}
|
||||||
|
|
||||||
# Source env file for tunables
|
# Source env file for tunables
|
||||||
@@ -59,7 +59,7 @@ def wait_ready():
|
|||||||
def send_request(prompt, max_tokens=N_GEN):
|
def send_request(prompt, max_tokens=N_GEN):
|
||||||
"""Send chat completion, measure generation throughput."""
|
"""Send chat completion, measure generation throughput."""
|
||||||
body = json.dumps({
|
body = json.dumps({
|
||||||
"model": "frank-large",
|
"model": "frank-dragoon",
|
||||||
"messages": [{"role": "user", "content": prompt}],
|
"messages": [{"role": "user", "content": prompt}],
|
||||||
"max_tokens": max_tokens,
|
"max_tokens": max_tokens,
|
||||||
"stream": False,
|
"stream": False,
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""
|
"""
|
||||||
Benchmark n-gram speculative decoding vs no-spec for frank-large.
|
Benchmark n-gram speculative decoding vs no-spec for frank-dragoon.
|
||||||
|
|
||||||
Tests prompt-processing throughput with 16K token prompts (manageable request size).
|
Tests prompt-processing throughput with 16K token prompts (manageable request size).
|
||||||
Code-heavy prompt (high n-gram redundancy) vs prose prompt (low redundancy).
|
Code-heavy prompt (high n-gram redundancy) vs prose prompt (low redundancy).
|
||||||
@@ -103,7 +103,7 @@ def start_server(port, spec_type):
|
|||||||
def run_test(prompt_text, spec_type, port):
|
def run_test(prompt_text, spec_type, port):
|
||||||
"""Send a request and extract prompt processing metrics."""
|
"""Send a request and extract prompt processing metrics."""
|
||||||
data = json.dumps({
|
data = json.dumps({
|
||||||
"model": "frank-large",
|
"model": "frank-dragoon",
|
||||||
"messages": [{"role": "user", "content": prompt_text}],
|
"messages": [{"role": "user", "content": prompt_text}],
|
||||||
"max_tokens": TEST_TOKENS,
|
"max_tokens": TEST_TOKENS,
|
||||||
"stream": False,
|
"stream": False,
|
||||||
@@ -189,7 +189,7 @@ def main():
|
|||||||
print(f" Prose: {prose_prompt_file.stat().st_size:,} bytes")
|
print(f" Prose: {prose_prompt_file.stat().st_size:,} bytes")
|
||||||
|
|
||||||
print("=" * 60)
|
print("=" * 60)
|
||||||
print(" N-gram Mod Benchmark — frank-large")
|
print(" N-gram Mod Benchmark — frank-dragoon")
|
||||||
print(f" STRIX ROCmFP4 · 64K context · 16K prompts")
|
print(f" STRIX ROCmFP4 · 64K context · 16K prompts")
|
||||||
print("=" * 60)
|
print("=" * 60)
|
||||||
|
|
||||||
|
|||||||
@@ -1,19 +1,19 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Benchmark n-gram speculative decoding vs no-spec for frank-large.
|
# Benchmark n-gram speculative decoding vs no-spec for frank-dragoon.
|
||||||
#
|
#
|
||||||
# This tests prompt-processing throughput at 64k context with two distinct
|
# This tests prompt-processing throughput at 64k context with two distinct
|
||||||
# prompts: one code-heavy and one prose-heavy. Uses llama-server on scratch
|
# prompts: one code-heavy and one prose-heavy. Uses llama-server on scratch
|
||||||
# ports with the OpenAI-compatible API to measure prompt_eval_duration_ms.
|
# ports with the OpenAI-compatible API to measure prompt_eval_duration_ms.
|
||||||
#
|
#
|
||||||
# Run on Dankclaw after stopping the production service:
|
# Run on Dankclaw after stopping the production service:
|
||||||
# sudo systemctl stop frank-large
|
# sudo systemctl stop frank-dragoon
|
||||||
# ./scripts/bench-ngram-mod.sh
|
# ./scripts/bench-ngram-mod.sh
|
||||||
# sudo systemctl start frank-large
|
# sudo systemctl start frank-dragoon
|
||||||
|
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
|
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-dragoon.env}"
|
||||||
if [[ -f "$ENV_FILE" ]]; then
|
if [[ -f "$ENV_FILE" ]]; then
|
||||||
set -a; source "$ENV_FILE"; set +a
|
set -a; source "$ENV_FILE"; set +a
|
||||||
fi
|
fi
|
||||||
@@ -184,7 +184,7 @@ run_single_test() {
|
|||||||
response=$(curl -s -X POST "http://127.0.0.1:$port/v1/completions" \
|
response=$(curl -s -X POST "http://127.0.0.1:$port/v1/completions" \
|
||||||
-H "Content-Type: application/json" \
|
-H "Content-Type: application/json" \
|
||||||
-d "{
|
-d "{
|
||||||
\"model\": \"frank-large\",
|
\"model\": \"frank-dragoon\",
|
||||||
\"prompt\": \"$prompt_text\",
|
\"prompt\": \"$prompt_text\",
|
||||||
\"max_tokens\": 128,
|
\"max_tokens\": 128,
|
||||||
\"temperature\": 0,
|
\"temperature\": 0,
|
||||||
@@ -220,7 +220,7 @@ run_single_test() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
echo "╔══════════════════════════════════════════════════════════╗"
|
echo "╔══════════════════════════════════════════════════════════╗"
|
||||||
echo "║ N-gram Mod Benchmark — frank-large ║"
|
echo "║ N-gram Mod Benchmark — frank-dragoon ║"
|
||||||
echo "║ STRIX ROCmFP4 · 64k context · two prompts ║"
|
echo "║ STRIX ROCmFP4 · 64k context · two prompts ║"
|
||||||
echo "╚══════════════════════════════════════════════════════════╝"
|
echo "╚══════════════════════════════════════════════════════════╝"
|
||||||
echo ""
|
echo ""
|
||||||
|
|||||||
@@ -15,9 +15,9 @@ import time
|
|||||||
import urllib.request
|
import urllib.request
|
||||||
import urllib.error
|
import urllib.error
|
||||||
|
|
||||||
SERVICE = "frank-large"
|
SERVICE = "frank-dragoon"
|
||||||
ENV_FILE = "/home/dank/serverAppConfigs/scripts/frank-large.env"
|
ENV_FILE = "/home/dank/serverAppConfigs/scripts/frank-dragoon.env"
|
||||||
ALIAS = "frank-large"
|
ALIAS = "frank-dragoon"
|
||||||
DEFAULT_HOST = "localhost"
|
DEFAULT_HOST = "localhost"
|
||||||
DEFAULT_PORT = 7770
|
DEFAULT_PORT = 7770
|
||||||
DEFAULT_MAX_TOKENS = 16384
|
DEFAULT_MAX_TOKENS = 16384
|
||||||
|
|||||||
+2
-2
@@ -1,4 +1,4 @@
|
|||||||
# Tunables for frank-large (mradermacher i1-Q4_K_M of
|
# Tunables for frank-dragoon (mradermacher i1-Q4_K_M of
|
||||||
# orcarouter/Qwen3.8-Flash-Next-Uncensored on ik_llama.cpp, Dankclaw,
|
# orcarouter/Qwen3.8-Flash-Next-Uncensored on ik_llama.cpp, Dankclaw,
|
||||||
# RTX 3090 CUDA + Strix Halo CPU path, since 2026-09-15).
|
# RTX 3090 CUDA + Strix Halo CPU path, since 2026-09-15).
|
||||||
#
|
#
|
||||||
@@ -65,7 +65,7 @@ MMPROJ_PATH=/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K
|
|||||||
CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
|
CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja
|
||||||
|
|
||||||
# Alias + network.
|
# Alias + network.
|
||||||
ALIAS=frank-large
|
ALIAS=frank-dragoon
|
||||||
BIND_ADDR=0.0.0.0
|
BIND_ADDR=0.0.0.0
|
||||||
LISTEN_PORT=7770
|
LISTEN_PORT=7770
|
||||||
|
|
||||||
+3
-3
@@ -1,5 +1,5 @@
|
|||||||
[Unit]
|
[Unit]
|
||||||
Description=ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M 177B MoE (dankclaw, frank-large)
|
Description=ik_llama.cpp Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M 177B MoE (dankclaw, frank-dragoon)
|
||||||
After=network-online.target
|
After=network-online.target
|
||||||
Wants=network-online.target
|
Wants=network-online.target
|
||||||
|
|
||||||
@@ -12,8 +12,8 @@ Group=dank
|
|||||||
# budget still needs room. Parity with the halogen quadlet's
|
# budget still needs room. Parity with the halogen quadlet's
|
||||||
# Ulimit=memlock=-1:-1.
|
# Ulimit=memlock=-1:-1.
|
||||||
LimitMEMLOCK=infinity
|
LimitMEMLOCK=infinity
|
||||||
Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
|
Environment="FRANK_DRAGOON_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-dragoon.env"
|
||||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
|
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-dragoon.sh
|
||||||
Restart=on-failure
|
Restart=on-failure
|
||||||
RestartSec=10
|
RestartSec=10
|
||||||
|
|
||||||
+3
-3
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Serve mradermacher/Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M as frank-large
|
# Serve mradermacher/Qwen3.8-Flash-Next-Uncensored i1-Q4_K_M as frank-dragoon
|
||||||
# via ik_llama.cpp on Dankclaw.
|
# via ik_llama.cpp on Dankclaw.
|
||||||
#
|
#
|
||||||
# Topology (ikawrakow's hybrid recipe; upstream issue #2402 for the PLE):
|
# Topology (ikawrakow's hybrid recipe; upstream issue #2402 for the PLE):
|
||||||
@@ -26,7 +26,7 @@
|
|||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
|
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-dragoon.env}"
|
||||||
if [[ -f "$ENV_FILE" ]]; then
|
if [[ -f "$ENV_FILE" ]]; then
|
||||||
set -a
|
set -a
|
||||||
# shellcheck disable=SC1090
|
# shellcheck disable=SC1090
|
||||||
@@ -66,7 +66,7 @@ fi
|
|||||||
# CPU-overridden weights mmapped (src/llama-load-tensors.cpp); the env file
|
# CPU-overridden weights mmapped (src/llama-load-tensors.cpp); the env file
|
||||||
# sets it. Refuse to run without it.
|
# sets it. Refuse to run without it.
|
||||||
if [[ "${GGML_CUDA_NO_PINNED_WEIGHTS:-}" != "1" ]]; then
|
if [[ "${GGML_CUDA_NO_PINNED_WEIGHTS:-}" != "1" ]]; then
|
||||||
echo "frank-large: refusing to start — GGML_CUDA_NO_PINNED_WEIGHTS=1 is mandatory on this host" >&2
|
echo "frank-dragoon: refusing to start — GGML_CUDA_NO_PINNED_WEIGHTS=1 is mandatory on this host" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
@@ -112,7 +112,7 @@ fi
|
|||||||
: "${SPEC_DRAFT_POLL_BATCH:=1}"
|
: "${SPEC_DRAFT_POLL_BATCH:=1}"
|
||||||
|
|
||||||
# Text only: the server must not try to autoload the vision tower. Vision
|
# Text only: the server must not try to autoload the vision tower. Vision
|
||||||
# stays on frank-large (Dankclaw), which is where the fleet routes images.
|
# stays on frank-dragoon (Dankclaw), which is where the fleet routes images.
|
||||||
NO_MMPROJ="${NO_MMPROJ:-1}"
|
NO_MMPROJ="${NO_MMPROJ:-1}"
|
||||||
|
|
||||||
export HIP_VISIBLE_DEVICES
|
export HIP_VISIBLE_DEVICES
|
||||||
|
|||||||
Executable
+43
@@ -0,0 +1,43 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# apply-protected-cast.sh — RUNS ON DANKSERVER. Rewrite the recipe's
|
||||||
|
# PROTECTED-BEGIN/END section from Swift's OWN calibration grams: run
|
||||||
|
# derive-protected-experts.py over the calib dir, splice its cast lines into
|
||||||
|
# the marked section of the recipe in place, keep the pre-edit recipe in the
|
||||||
|
# campaign trash. Refuses to run twice (marker text says derived).
|
||||||
|
set -euo pipefail
|
||||||
|
CALIB=${1:-/home/dank/corpora/calib}
|
||||||
|
RECIPE=${2:-/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe}
|
||||||
|
PY=/home/dank/venvs/radcorpus/bin/python
|
||||||
|
DERIVE=/home/dank/serverAppConfigs-swift15/scripts/derive-protected-experts.py
|
||||||
|
TS=$(date +%Y%m%d-%H%M%S)
|
||||||
|
|
||||||
|
grep -q "derive-protected-experts.py rewrote" "$RECIPE" && {
|
||||||
|
echo "recipe already carries derived cast lines; refusing to re-derive" >&2; exit 1; }
|
||||||
|
grep -q "PROTECTED-BEGIN" "$RECIPE" || { echo "no PROTECTED markers in $RECIPE" >&2; exit 1; }
|
||||||
|
|
||||||
|
"$PY" "$DERIVE" "$CALIB" > "/tmp/protected-cast-$TS.txt"
|
||||||
|
N=$(grep -c 'cast dtype=bf16' "/tmp/protected-cast-$TS.txt")
|
||||||
|
[ "$N" -ge 2 ] || { echo "derive produced $N cast lines; refusing" >&2; exit 1; }
|
||||||
|
echo "derived $N protected experts -> /tmp/protected-cast-$TS.txt"
|
||||||
|
|
||||||
|
cp -f "$RECIPE" "/home/dank/.AGENTTRASH/$TS-qwen4exp-swift15-w4dn8-i8-hc8m.recipe"
|
||||||
|
"$PY" - "$RECIPE" "/tmp/protected-cast-$TS.txt" <<'PYEOF'
|
||||||
|
import sys
|
||||||
|
recipe, castfile = sys.argv[1], sys.argv[2]
|
||||||
|
lines = open(recipe).read().splitlines()
|
||||||
|
cast = [l for l in open(castfile).read().splitlines() if 'cast dtype=bf16' in l]
|
||||||
|
out, insec = [], False
|
||||||
|
for l in lines:
|
||||||
|
if 'PROTECTED-BEGIN' in l:
|
||||||
|
insec = True
|
||||||
|
out.append(l)
|
||||||
|
out.extend(cast)
|
||||||
|
elif 'PROTECTED-END' in l:
|
||||||
|
insec = False
|
||||||
|
out.append(l)
|
||||||
|
out.append('# (derive-protected-experts.py rewrote this section from Swift\'s own grams)')
|
||||||
|
elif not insec:
|
||||||
|
out.append(l)
|
||||||
|
open(recipe, 'w').write('\n'.join(out) + '\n')
|
||||||
|
print(f'spliced {len(cast)} cast lines into {recipe}')
|
||||||
|
PYEOF
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""autoresearch-bench.py — deterministic greedy battery for frank-large.
|
"""autoresearch-bench.py — deterministic greedy battery for frank-dragoon.
|
||||||
|
|
||||||
Runs ON Dankclaw (no LAN jitter) against the candidate container on :7770.
|
Runs ON Dankclaw (no LAN jitter) against the candidate container on :7770.
|
||||||
Fixed embedded corpus, temperature 0, streaming so decode speed is measured
|
Fixed embedded corpus, temperature 0, streaming so decode speed is measured
|
||||||
@@ -104,7 +104,7 @@ def stream_completion(url: str, model: str, prompt: str):
|
|||||||
def main():
|
def main():
|
||||||
ap = argparse.ArgumentParser()
|
ap = argparse.ArgumentParser()
|
||||||
ap.add_argument("--url", default="http://127.0.0.1:7770")
|
ap.add_argument("--url", default="http://127.0.0.1:7770")
|
||||||
ap.add_argument("--model", default="frank-large")
|
ap.add_argument("--model", default="frank-dragoon")
|
||||||
args = ap.parse_args()
|
args = ap.parse_args()
|
||||||
|
|
||||||
with urllib.request.urlopen(args.url.rstrip("/") + "/v1/models",
|
with urllib.request.urlopen(args.url.rstrip("/") + "/v1/models",
|
||||||
|
|||||||
@@ -1,12 +1,12 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# autoresearch-frank-primary.sh — RUNS ON DANKSERVER, driven over ssh by the
|
# autoresearch-frank-archon.sh — RUNS ON DANKSERVER, driven over ssh by the
|
||||||
# dev-box autoresearch.sh. It is the canonical frank-primary benchmark entry
|
# dev-box autoresearch.sh. It is the canonical frank-archon benchmark entry
|
||||||
# point for the autoresearch loop.
|
# point for the autoresearch loop.
|
||||||
#
|
#
|
||||||
# Method (mandated by the Benchmarking Policy): the production service serves
|
# Method (mandated by the Benchmarking Policy): the production service serves
|
||||||
# real users and shares the two R9700s, so a bench beside it starves the GPUs
|
# real users and shares the two R9700s, so a bench beside it starves the GPUs
|
||||||
# and corrupts results. This script therefore:
|
# and corrupts results. This script therefore:
|
||||||
# 1. stops the production frank-primary unit (root, relink symlink: stop,
|
# 1. stops the production frank-archon unit (root, relink symlink: stop,
|
||||||
# NOT mask — mask refuses to replace a symlinked unit file),
|
# NOT mask — mask refuses to replace a symlinked unit file),
|
||||||
# 2. boots a scratch radiance container named `arbench` on the SAME two
|
# 2. boots a scratch radiance container named `arbench` on the SAME two
|
||||||
# cards with the candidate engine argv (the env file the loop edits),
|
# cards with the candidate engine argv (the env file the loop edits),
|
||||||
@@ -18,7 +18,7 @@
|
|||||||
# prefill and decode rates for that single request.
|
# prefill and decode rates for that single request.
|
||||||
# 5. prints METRIC lines and tears the scratch down,
|
# 5. prints METRIC lines and tears the scratch down,
|
||||||
# 6. restores production (systemctl start), via a trap so a failed bench
|
# 6. restores production (systemctl start), via a trap so a failed bench
|
||||||
# never leaves frank-primary down (the stale-probe-tail lesson).
|
# never leaves frank-archon down (the stale-probe-tail lesson).
|
||||||
#
|
#
|
||||||
# Determinism: the prompt is built from a fixed, sorted corpus of repo files
|
# Determinism: the prompt is built from a fixed, sorted corpus of repo files
|
||||||
# with a fixed random seed, so prompt_n is stable across runs. Sampling is
|
# with a fixed random seed, so prompt_n is stable across runs. Sampling is
|
||||||
@@ -32,10 +32,10 @@
|
|||||||
# /dev/dri, the .rad ro, the state dir rw, the tune cache mount. The ONLY
|
# /dev/dri, the .rad ro, the state dir rw, the tune cache mount. The ONLY
|
||||||
# differences: the container name (arbench, so prod's identity is untouched)
|
# differences: the container name (arbench, so prod's identity is untouched)
|
||||||
# and a FRESH prefix-cache disk tier (arbench/, wiped per run) so the cold
|
# and a FRESH prefix-cache disk tier (arbench/, wiped per run) so the cold
|
||||||
# prefill is cold — the prod tier at kvcache/frank-primary is left intact.
|
# prefill is cold — the prod tier at kvcache/frank-archon is left intact.
|
||||||
set -uo pipefail
|
set -uo pipefail
|
||||||
|
|
||||||
ENV_FILE="${1:-/home/dank/serverAppConfigs/scripts/frank-primary.env}"
|
ENV_FILE="${1:-/home/dank/serverAppConfigs/scripts/frank-archon.env}"
|
||||||
SCRATCH=arbench
|
SCRATCH=arbench
|
||||||
PORT=7770
|
PORT=7770
|
||||||
HEALTH="http://127.0.0.1:$PORT/health"
|
HEALTH="http://127.0.0.1:$PORT/health"
|
||||||
@@ -45,7 +45,7 @@ LOG=/home/dank/logs/arbench.log
|
|||||||
|
|
||||||
restore() {
|
restore() {
|
||||||
podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
|
podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
|
||||||
sudo systemctl start frank-primary >/dev/null 2>&1 || true
|
sudo systemctl start frank-archon >/dev/null 2>&1 || true
|
||||||
}
|
}
|
||||||
trap restore EXIT
|
trap restore EXIT
|
||||||
|
|
||||||
@@ -57,13 +57,13 @@ fi
|
|||||||
set -a; . "$ENV_FILE"; set +a
|
set -a; . "$ENV_FILE"; set +a
|
||||||
|
|
||||||
# --- stop production; the scratch needs both cards to itself ----------------
|
# --- stop production; the scratch needs both cards to itself ----------------
|
||||||
sudo systemctl stop frank-primary >/dev/null 2>&1 || true
|
sudo systemctl stop frank-archon >/dev/null 2>&1 || true
|
||||||
podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
|
podman rm -f "$SCRATCH" >/dev/null 2>&1 || true
|
||||||
|
|
||||||
# --- fresh prefix-cache tier for a cold prefill (prod's tier untouched) ------
|
# --- fresh prefix-cache tier for a cold prefill (prod's tier untouched) ------
|
||||||
FRESH_KV="$FRANK_PRIMARY_STATE_DIR/kvcache/$SCRATCH"
|
FRESH_KV="$FRANK_ARCHON_STATE_DIR/kvcache/$SCRATCH"
|
||||||
rm -rf "$FRESH_KV"; mkdir -p "$FRESH_KV"
|
rm -rf "$FRESH_KV"; mkdir -p "$FRESH_KV"
|
||||||
mkdir -p "$FRANK_PRIMARY_STATE_DIR/tune"
|
mkdir -p "$FRANK_ARCHON_STATE_DIR/tune"
|
||||||
|
|
||||||
# --- boot the scratch: the production run line, scratch name + fresh KV ------
|
# --- boot the scratch: the production run line, scratch name + fresh KV ------
|
||||||
podman run -d --name "$SCRATCH" \
|
podman run -d --name "$SCRATCH" \
|
||||||
@@ -72,30 +72,31 @@ podman run -d --name "$SCRATCH" \
|
|||||||
--security-opt seccomp=unconfined \
|
--security-opt seccomp=unconfined \
|
||||||
--ulimit memlock=-1:-1 \
|
--ulimit memlock=-1:-1 \
|
||||||
--device /dev/kfd --device /dev/dri \
|
--device /dev/kfd --device /dev/dri \
|
||||||
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
|
-v "$FRANK_ARCHON_MODEL_DIR":/models:ro \
|
||||||
-v "$FRANK_PRIMARY_STATE_DIR":/data \
|
-v "$FRANK_ARCHON_STATE_DIR":/data \
|
||||||
-v "$FRANK_PRIMARY_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
|
-v "$FRANK_ARCHON_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
|
||||||
-v "$FRANK_PRIMARY_TEMPLATE":/templates/template.jinja:ro \
|
-v "$FRANK_ARCHON_TEMPLATE":/templates/template.jinja:ro \
|
||||||
"$FRANK_PRIMARY_IMAGE" \
|
"$FRANK_ARCHON_IMAGE" \
|
||||||
--model "/models/$FRANK_PRIMARY_MODEL_FILE" \
|
--model "/models/$FRANK_ARCHON_MODEL_FILE" \
|
||||||
--host :: --port "$PORT" \
|
--host :: --port "$PORT" \
|
||||||
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" \
|
--served-model-name "$FRANK_ARCHON_SERVED_NAME" \
|
||||||
--override-chat-template /templates/template.jinja \
|
--override-chat-template /templates/template.jinja \
|
||||||
--tp "$FRANK_PRIMARY_TP" --tp-wire "$FRANK_PRIMARY_TP_WIRE" \
|
--tp "$FRANK_ARCHON_TP" --tp-wire "$FRANK_ARCHON_TP_WIRE" \
|
||||||
--max-num-seqs "$FRANK_PRIMARY_MAX_SEQS" \
|
--max-num-seqs "$FRANK_ARCHON_MAX_SEQS" \
|
||||||
--max-model-len "$FRANK_PRIMARY_MAX_MODEL_LEN" \
|
--max-model-len "$FRANK_ARCHON_MAX_MODEL_LEN" \
|
||||||
--placement "$FRANK_PRIMARY_PLACEMENT" \
|
--placement "$FRANK_ARCHON_PLACEMENT" \
|
||||||
--host-pool-mib "$FRANK_PRIMARY_HOST_POOL_MIB" \
|
--host-pool-mib "$FRANK_ARCHON_HOST_POOL_MIB" \
|
||||||
--gpu-headroom-mib "$FRANK_PRIMARY_GPU_HEADROOM_MIB" \
|
--gpu-headroom-mib "$FRANK_ARCHON_GPU_HEADROOM_MIB" \
|
||||||
--expert-vs-cache-ratio "$FRANK_PRIMARY_EXPERT_VS_CACHE" \
|
--vram-weights-mib "$FRANK_ARCHON_VRAM_WEIGHTS_MIB" \
|
||||||
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
|
--vram-kv-mib "$FRANK_ARCHON_VRAM_KV_MIB" \
|
||||||
--prefix-cache-host-mib "$FRANK_PRIMARY_PREFIX_HOST_MIB" \
|
--kv-cache-dtype "$FRANK_ARCHON_KV_DTYPE" \
|
||||||
|
--prefix-cache-host-mib "$FRANK_ARCHON_PREFIX_HOST_MIB" \
|
||||||
--prefix-cache-dir "/data/kvcache/$SCRATCH" \
|
--prefix-cache-dir "/data/kvcache/$SCRATCH" \
|
||||||
--prefix-cache-disk-mib "$FRANK_PRIMARY_PREFIX_DISK_MIB" \
|
--prefix-cache-disk-mib "$FRANK_ARCHON_PREFIX_DISK_MIB" \
|
||||||
--ngram-placement "$FRANK_PRIMARY_NGRAM_PLACEMENT" \
|
--ngram-placement "$FRANK_ARCHON_NGRAM_PLACEMENT" \
|
||||||
--num-speculative-tokens "$FRANK_PRIMARY_SPEC_TOKENS" \
|
--num-speculative-tokens "$FRANK_ARCHON_SPEC_TOKENS" \
|
||||||
--max-num-batched-tokens "$FRANK_PRIMARY_MAX_BATCHED_TOKENS" \
|
--max-num-batched-tokens "$FRANK_ARCHON_MAX_BATCHED_TOKENS" \
|
||||||
${FRANK_PRIMARY_EXTRA_ARGS:+$FRANK_PRIMARY_EXTRA_ARGS} \
|
${FRANK_ARCHON_EXTRA_ARGS:+$FRANK_ARCHON_EXTRA_ARGS} \
|
||||||
> "$LOG" 2>&1
|
> "$LOG" 2>&1
|
||||||
if [ $? -ne 0 ]; then echo "scratch boot failed; see $LOG" >&2; exit 1; fi
|
if [ $? -ne 0 ]; then echo "scratch boot failed; see $LOG" >&2; exit 1; fi
|
||||||
|
|
||||||
@@ -145,7 +146,7 @@ while n < TGT:
|
|||||||
n += len(seg) + 1
|
n += len(seg) + 1
|
||||||
prompt = " ".join(parts)
|
prompt = " ".join(parts)
|
||||||
json.dump(
|
json.dump(
|
||||||
{"model": "frank-primary", "prompt": prompt,
|
{"model": "frank-archon", "prompt": prompt,
|
||||||
"max_tokens": 256, "temperature": 0.0, "stream": False,
|
"max_tokens": 256, "temperature": 0.0, "stream": False,
|
||||||
"stop": ["<|im_end|>"]},
|
"stop": ["<|im_end|>"]},
|
||||||
open(sys.argv[1], "w"),
|
open(sys.argv[1], "w"),
|
||||||
+32
-32
@@ -1,19 +1,19 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# frank-primary-arms.sh — run a SEQUENCE of frank-primary bench arms and keep
|
# frank-archon-arms.sh — run a SEQUENCE of frank-archon bench arms and keep
|
||||||
# every arm's evidence. Runs on the operator box; each arm is one full window
|
# every arm's evidence. Runs on the operator box; each arm is one full window
|
||||||
# of ./autoresearch.sh (stop+mask -> boot the arm as the service -> measure ->
|
# of ./autoresearch.sh (stop+mask -> boot the arm as the service -> measure ->
|
||||||
# restore production and verify), so the fleet is only ever down for one arm at
|
# restore production and verify), so the fleet is only ever down for one arm at
|
||||||
# a time and the production unit is back up between arms.
|
# a time and the production unit is back up between arms.
|
||||||
#
|
#
|
||||||
# An arm is the repo's own frank-primary.env plus an OVERLAY of engine knobs
|
# An arm is the repo's own frank-archon.env plus an OVERLAY of engine knobs
|
||||||
# appended after it (later assignment wins when the serve script sources it),
|
# appended after it (later assignment wins when the serve script sources it),
|
||||||
# staged through FRANK_PRIMARY_AR_ENV_FILE so the checkout is never mutated and
|
# staged through FRANK_ARCHON_AR_ENV_FILE so the checkout is never mutated and
|
||||||
# each arm gets its own fingerprint (the memo can never answer one arm with
|
# each arm gets its own fingerprint (the memo can never answer one arm with
|
||||||
# another arm's numbers).
|
# another arm's numbers).
|
||||||
#
|
#
|
||||||
# Usage: scripts/autoresearch/frank-primary-arms.sh [arm ...]
|
# Usage: scripts/autoresearch/frank-archon-arms.sh [arm ...]
|
||||||
# scripts/autoresearch/frank-primary-arms.sh --list
|
# scripts/autoresearch/frank-archon-arms.sh --list
|
||||||
# Output, under /tmp/frank-primary-arms-<UTC stamp>/ (override with ARMS_DIR):
|
# Output, under /tmp/frank-archon-arms-<UTC stamp>/ (override with ARMS_DIR):
|
||||||
#
|
#
|
||||||
# <arm>.out full autoresearch output (metrics + engine evidence)
|
# <arm>.out full autoresearch output (metrics + engine evidence)
|
||||||
# <arm>.env the exact staged env
|
# <arm>.env the exact staged env
|
||||||
@@ -26,8 +26,8 @@ set -uo pipefail
|
|||||||
|
|
||||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
REPO="$(cd -- "$SCRIPT_DIR/../.." && pwd)"
|
REPO="$(cd -- "$SCRIPT_DIR/../.." && pwd)"
|
||||||
BASE_ENV="$REPO/scripts/frank-primary.env"
|
BASE_ENV="$REPO/scripts/frank-archon.env"
|
||||||
ARMS_DIR="${ARMS_DIR:-/tmp/frank-primary-arms-$(date -u +%Y%m%dT%H%M%SZ)}"
|
ARMS_DIR="${ARMS_DIR:-/tmp/frank-archon-arms-$(date -u +%Y%m%dT%H%M%SZ)}"
|
||||||
|
|
||||||
# --- arms ------------------------------------------------------------------
|
# --- arms ------------------------------------------------------------------
|
||||||
# baseline: the shipped shape E, measured with THIS harness so the comparison is
|
# baseline: the shipped shape E, measured with THIS harness so the comparison is
|
||||||
@@ -45,18 +45,18 @@ LADDER3='{"cudagraph_capture_sizes":[4,8,12],"max_cudagraph_capture_size":12,"in
|
|||||||
# the engine as {temperature:0.7}.
|
# the engine as {temperature:0.7}.
|
||||||
GEN07='{"temperature":0.7}'
|
GEN07='{"temperature":0.7}'
|
||||||
OVER_B=(
|
OVER_B=(
|
||||||
"FRANK_PRIMARY_SPEC='$SPEC3'"
|
"FRANK_ARCHON_SPEC='$SPEC3'"
|
||||||
"FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
|
"FRANK_ARCHON_COMPCONFIG='$LADDER3'"
|
||||||
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
|
'FRANK_ARCHON_GEN_CONFIG='"'$GEN07'"
|
||||||
)
|
)
|
||||||
# B2: identical to B with the default (7.60 GiB) overhead, i.e. precise memory
|
# B2: identical to B with the default (7.60 GiB) overhead, i.e. precise memory
|
||||||
# OFF. It is the fallback if B dies at graph capture, and it prices the arena
|
# OFF. It is the fallback if B dies at graph capture, and it prices the arena
|
||||||
# that precise memory buys (7,522 slots -> ~6,496).
|
# that precise memory buys (7,522 slots -> ~6,496).
|
||||||
OVER_B2=(
|
OVER_B2=(
|
||||||
"FRANK_PRIMARY_SPEC='$SPEC3'"
|
"FRANK_ARCHON_SPEC='$SPEC3'"
|
||||||
"FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
|
"FRANK_ARCHON_COMPCONFIG='$LADDER3'"
|
||||||
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
|
'FRANK_ARCHON_GEN_CONFIG='"'$GEN07'"
|
||||||
'FRANK_PRIMARY_PRECISE_MEMORY='
|
'FRANK_ARCHON_PRECISE_MEMORY='
|
||||||
)
|
)
|
||||||
# C: the vendor's concurrency shape — SEQS 16 with the [4..32] ladder and util
|
# C: the vendor's concurrency shape — SEQS 16 with the [4..32] ladder and util
|
||||||
# 0.95. This is also the direct test of the theory that pinning the capture
|
# 0.95. This is also the direct test of the theory that pinning the capture
|
||||||
@@ -64,12 +64,12 @@ OVER_B2=(
|
|||||||
# the constant 3, which is NOT in the default ladder; 4 is).
|
# the constant 3, which is NOT in the default ladder; 4 is).
|
||||||
LADDER16='{"cudagraph_capture_sizes":[4,8,12,16,20,24,28,32],"max_cudagraph_capture_size":32,"inductor_compile_config":{"combo_kernels":false,"benchmark_combo_kernel":false}}'
|
LADDER16='{"cudagraph_capture_sizes":[4,8,12,16,20,24,28,32],"max_cudagraph_capture_size":32,"inductor_compile_config":{"combo_kernels":false,"benchmark_combo_kernel":false}}'
|
||||||
OVER_C=(
|
OVER_C=(
|
||||||
"FRANK_PRIMARY_SPEC='$SPEC3'"
|
"FRANK_ARCHON_SPEC='$SPEC3'"
|
||||||
"FRANK_PRIMARY_COMPCONFIG='$LADDER16'"
|
"FRANK_ARCHON_COMPCONFIG='$LADDER16'"
|
||||||
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
|
'FRANK_ARCHON_GEN_CONFIG='"'$GEN07'"
|
||||||
'FRANK_PRIMARY_SEQS=16'
|
'FRANK_ARCHON_SEQS=16'
|
||||||
'FRANK_PRIMARY_MNBT=4096'
|
'FRANK_ARCHON_MNBT=4096'
|
||||||
'FRANK_PRIMARY_UTIL=0.95'
|
'FRANK_ARCHON_UTIL=0.95'
|
||||||
)
|
)
|
||||||
|
|
||||||
# D/E: the CONTEXT arms. Measured 2026-10-02 (arms run 20261002T233855Z): the
|
# D/E: the CONTEXT arms. Measured 2026-10-02 (arms run 20261002T233855Z): the
|
||||||
@@ -83,11 +83,11 @@ OVER_C=(
|
|||||||
# D is arm B with EXACTLY ONE variable changed (the KV bytes), so its result
|
# D is arm B with EXACTLY ONE variable changed (the KV bytes), so its result
|
||||||
# says whether the 4k-depth collapse (B 23.09 vs baseline 33.99) was KV or the
|
# says whether the 4k-depth collapse (B 23.09 vs baseline 33.99) was KV or the
|
||||||
# batch shape.
|
# batch shape.
|
||||||
KV401K='FRANK_PRIMARY_KV_BYTES=4050000000'
|
KV401K='FRANK_ARCHON_KV_BYTES=4050000000'
|
||||||
OVER_D=(
|
OVER_D=(
|
||||||
"FRANK_PRIMARY_SPEC='$SPEC3'"
|
"FRANK_ARCHON_SPEC='$SPEC3'"
|
||||||
"FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
|
"FRANK_ARCHON_COMPCONFIG='$LADDER3'"
|
||||||
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
|
'FRANK_ARCHON_GEN_CONFIG='"'$GEN07'"
|
||||||
"$KV401K"
|
"$KV401K"
|
||||||
)
|
)
|
||||||
# E: D with C's max_num_batched_tokens (4096 — C is the only arm that did NOT
|
# E: D with C's max_num_batched_tokens (4096 — C is the only arm that did NOT
|
||||||
@@ -97,12 +97,12 @@ OVER_D=(
|
|||||||
# only 0.32 GiB above the allowance while the CUDA-graph cost is an ANALYTIC
|
# only 0.32 GiB above the allowance while the CUDA-graph cost is an ANALYTIC
|
||||||
# estimate (C's was 1.47 GiB over 24 graphs), so it is measured, not assumed.
|
# estimate (C's was 1.47 GiB over 24 graphs), so it is measured, not assumed.
|
||||||
OVER_E=(
|
OVER_E=(
|
||||||
"FRANK_PRIMARY_SPEC='$SPEC3'"
|
"FRANK_ARCHON_SPEC='$SPEC3'"
|
||||||
"FRANK_PRIMARY_COMPCONFIG='$LADDER3'"
|
"FRANK_ARCHON_COMPCONFIG='$LADDER3'"
|
||||||
'FRANK_PRIMARY_GEN_CONFIG='"'$GEN07'"
|
'FRANK_ARCHON_GEN_CONFIG='"'$GEN07'"
|
||||||
"$KV401K"
|
"$KV401K"
|
||||||
'FRANK_PRIMARY_MNBT=4096'
|
'FRANK_ARCHON_MNBT=4096'
|
||||||
'FRANK_PRIMARY_UTIL=0.99'
|
'FRANK_ARCHON_UTIL=0.99'
|
||||||
)
|
)
|
||||||
|
|
||||||
arm_overlay() {
|
arm_overlay() {
|
||||||
@@ -142,7 +142,7 @@ for arm in "${ARMS[@]}"; do
|
|||||||
# holds JSON must still be one valid JSON object after the file is SOURCED
|
# holds JSON must still be one valid JSON object after the file is SOURCED
|
||||||
# (that is the step that silently eats unquoted braces).
|
# (that is the step that silently eats unquoted braces).
|
||||||
if ! bash -c 'set -a; . "$1" >/dev/null; set +a
|
if ! bash -c 'set -a; . "$1" >/dev/null; set +a
|
||||||
for k in FRANK_PRIMARY_SPEC FRANK_PRIMARY_COMPCONFIG FRANK_PRIMARY_GEN_CONFIG; do
|
for k in FRANK_ARCHON_SPEC FRANK_ARCHON_COMPCONFIG FRANK_ARCHON_GEN_CONFIG; do
|
||||||
v="${!k:-}"
|
v="${!k:-}"
|
||||||
[ -n "$v" ] || continue
|
[ -n "$v" ] || continue
|
||||||
case "$v" in
|
case "$v" in
|
||||||
@@ -158,7 +158,7 @@ for arm in "${ARMS[@]}"; do
|
|||||||
fi
|
fi
|
||||||
|
|
||||||
echo "[arms] === $arm: opening window $(date -u +%H:%M:%S)Z ===" >&2
|
echo "[arms] === $arm: opening window $(date -u +%H:%M:%S)Z ===" >&2
|
||||||
FRANK_PRIMARY_AR_ENV_FILE="$ARMS_DIR/$arm.env" \
|
FRANK_ARCHON_AR_ENV_FILE="$ARMS_DIR/$arm.env" \
|
||||||
"$REPO/autoresearch.sh" >"$ARMS_DIR/$arm.out" 2>&1
|
"$REPO/autoresearch.sh" >"$ARMS_DIR/$arm.out" 2>&1
|
||||||
rc=$?
|
rc=$?
|
||||||
echo "$rc" >"$ARMS_DIR/$arm.rc"
|
echo "$rc" >"$ARMS_DIR/$arm.rc"
|
||||||
+3
-3
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""frank-primary decode measurement — runs INSIDE the engine container.
|
"""frank-archon decode measurement — runs INSIDE the engine container.
|
||||||
|
|
||||||
The method is the fleet's established one (the 2026-09-29 / 2026-10-01
|
The method is the fleet's established one (the 2026-09-29 / 2026-10-01
|
||||||
tuning campaigns, /home/dank/mtp_measure.py): SSE chunk counting is invalid
|
tuning campaigns, /home/dank/mtp_measure.py): SSE chunk counting is invalid
|
||||||
@@ -28,7 +28,7 @@ Workload (fixed, deterministic, offline):
|
|||||||
|
|
||||||
Env:
|
Env:
|
||||||
AR_BASE endpoint (default http://127.0.0.1:7770)
|
AR_BASE endpoint (default http://127.0.0.1:7770)
|
||||||
AR_MODEL served name (default frank-primary)
|
AR_MODEL served name (default frank-archon)
|
||||||
AR_PROMPT_CEILING hard prompt ceiling in tokens (default 100000)
|
AR_PROMPT_CEILING hard prompt ceiling in tokens (default 100000)
|
||||||
AR_TEMPS comma-separated sampling temperatures (default "0,0.7")
|
AR_TEMPS comma-separated sampling temperatures (default "0,0.7")
|
||||||
|
|
||||||
@@ -45,7 +45,7 @@ import urllib.error
|
|||||||
import urllib.request
|
import urllib.request
|
||||||
|
|
||||||
BASE = os.environ.get("AR_BASE", "http://127.0.0.1:7770")
|
BASE = os.environ.get("AR_BASE", "http://127.0.0.1:7770")
|
||||||
MODEL = os.environ.get("AR_MODEL", "frank-primary")
|
MODEL = os.environ.get("AR_MODEL", "frank-archon")
|
||||||
PROMPT_CEILING = int(os.environ.get("AR_PROMPT_CEILING", "100000"))
|
PROMPT_CEILING = int(os.environ.get("AR_PROMPT_CEILING", "100000"))
|
||||||
DEPTHS = (115, 1056, 4096)
|
DEPTHS = (115, 1056, 4096)
|
||||||
WARM_TOKENS = 64
|
WARM_TOKENS = 64
|
||||||
+14
-14
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# frank-primary-window.sh — one full bench window for frank-primary, run ON
|
# frank-archon-window.sh — one full bench window for frank-archon, run ON
|
||||||
# DANKSERVER as the deploy user (invoked over ssh by ./autoresearch.sh).
|
# DANKSERVER as the deploy user (invoked over ssh by ./autoresearch.sh).
|
||||||
#
|
#
|
||||||
# Discipline (AGENTS.md Benchmarking Policy): the production unit is
|
# Discipline (AGENTS.md Benchmarking Policy): the production unit is
|
||||||
@@ -12,9 +12,9 @@
|
|||||||
#
|
#
|
||||||
# Staging layout (created by the orchestrator, ephemeral — never a config
|
# Staging layout (created by the orchestrator, ephemeral — never a config
|
||||||
# source; the repo checkout is):
|
# source; the repo checkout is):
|
||||||
# /home/dank/.autoresearch-frank-primary/
|
# /home/dank/.autoresearch-frank-archon/
|
||||||
# serve.sh staged candidate serve script
|
# serve.sh staged candidate serve script
|
||||||
# frank-primary.env staged candidate env (the candidate knobs)
|
# frank-archon.env staged candidate env (the candidate knobs)
|
||||||
# measure.py the in-container measure client
|
# measure.py the in-container measure client
|
||||||
# hash candidate fingerprint
|
# hash candidate fingerprint
|
||||||
# run.json result (metrics + boot + kv + fingerprint)
|
# run.json result (metrics + boot + kv + fingerprint)
|
||||||
@@ -27,16 +27,16 @@
|
|||||||
# outcome; a failed restore forces exit 1).
|
# outcome; a failed restore forces exit 1).
|
||||||
set -uo pipefail
|
set -uo pipefail
|
||||||
|
|
||||||
STAGE="${FRANK_PRIMARY_AR_STAGE:-/home/dank/.autoresearch-frank-primary}"
|
STAGE="${FRANK_ARCHON_AR_STAGE:-/home/dank/.autoresearch-frank-archon}"
|
||||||
UNIT="${FRANK_PRIMARY_AR_UNIT:-frank-primary}"
|
UNIT="${FRANK_ARCHON_AR_UNIT:-frank-archon}"
|
||||||
CTR="${FRANK_PRIMARY_AR_CTR:-frank-primary-qwen}"
|
CTR="${FRANK_ARCHON_AR_CTR:-frank-archon-qwen}"
|
||||||
HEALTH_URL="http://127.0.0.1:7770/health"
|
HEALTH_URL="http://127.0.0.1:7770/health"
|
||||||
MODELS_URL="http://127.0.0.1:7770/v1/models"
|
MODELS_URL="http://127.0.0.1:7770/v1/models"
|
||||||
BOOT_TIMEOUT="${FRANK_PRIMARY_AR_BOOT_TIMEOUT:-1200}"
|
BOOT_TIMEOUT="${FRANK_ARCHON_AR_BOOT_TIMEOUT:-1200}"
|
||||||
RESTORE_TIMEOUT="${FRANK_PRIMARY_AR_RESTORE_TIMEOUT:-1200}"
|
RESTORE_TIMEOUT="${FRANK_ARCHON_AR_RESTORE_TIMEOUT:-1200}"
|
||||||
MEASURE_TIMEOUT="${FRANK_PRIMARY_AR_MEASURE_TIMEOUT:-1800}"
|
MEASURE_TIMEOUT="${FRANK_ARCHON_AR_MEASURE_TIMEOUT:-1800}"
|
||||||
SERVED="$(sed -n 's/^FRANK_PRIMARY_SERVED_NAME=//p' "$STAGE/frank-primary.env" 2>/dev/null | head -1)"
|
SERVED="$(sed -n 's/^FRANK_ARCHON_SERVED_NAME=//p' "$STAGE/frank-archon.env" 2>/dev/null | head -1)"
|
||||||
SERVED="${SERVED:-frank-primary}"
|
SERVED="${SERVED:-frank-archon}"
|
||||||
|
|
||||||
RESTORED=0
|
RESTORED=0
|
||||||
|
|
||||||
@@ -72,7 +72,7 @@ capture_bench_failure() {
|
|||||||
echo "=== staged serve.out (tail 60) ==="
|
echo "=== staged serve.out (tail 60) ==="
|
||||||
tail -60 "$STAGE/serve.out" 2>/dev/null || true
|
tail -60 "$STAGE/serve.out" 2>/dev/null || true
|
||||||
echo "=== engine log (tail 60) ==="
|
echo "=== engine log (tail 60) ==="
|
||||||
tail -60 /home/dank/logs/frank_primary.log 2>/dev/null || true
|
tail -60 /home/dank/logs/frank_archon.log 2>/dev/null || true
|
||||||
} >"$STAGE/fail.log" 2>&1 || true
|
} >"$STAGE/fail.log" 2>&1 || true
|
||||||
grep -E 'OutOfMemory|Tried to allocate|Engine core initialization failed|Error|error' \
|
grep -E 'OutOfMemory|Tried to allocate|Engine core initialization failed|Error|error' \
|
||||||
"$STAGE/fail.log" 2>/dev/null | tail -25 >&2 || true
|
"$STAGE/fail.log" 2>/dev/null | tail -25 >&2 || true
|
||||||
@@ -114,7 +114,7 @@ trap 'restore; rc=$?; [ "$rc" -ne 0 ] && exit 1' EXIT
|
|||||||
|
|
||||||
main() {
|
main() {
|
||||||
local f
|
local f
|
||||||
for f in serve.sh frank-primary.env measure.py hash; do
|
for f in serve.sh frank-archon.env measure.py hash; do
|
||||||
[ -f "$STAGE/$f" ] || { log "staged file missing: $STAGE/$f"; exit 1; }
|
[ -f "$STAGE/$f" ] || { log "staged file missing: $STAGE/$f"; exit 1; }
|
||||||
done
|
done
|
||||||
mkdir -p "$STAGE/last"
|
mkdir -p "$STAGE/last"
|
||||||
@@ -133,7 +133,7 @@ main() {
|
|||||||
|
|
||||||
t0="$(date +%s)"
|
t0="$(date +%s)"
|
||||||
log "starting bench instance from staged config (fingerprint $(tr -d '[:space:]' <"$STAGE/hash"))"
|
log "starting bench instance from staged config (fingerprint $(tr -d '[:space:]' <"$STAGE/hash"))"
|
||||||
FRANK_PRIMARY_ENV_FILE="$STAGE/frank-primary.env" \
|
FRANK_ARCHON_ENV_FILE="$STAGE/frank-archon.env" \
|
||||||
setsid nohup bash "$STAGE/serve.sh" >"$STAGE/serve.out" 2>&1 </dev/null &
|
setsid nohup bash "$STAGE/serve.sh" >"$STAGE/serve.out" 2>&1 </dev/null &
|
||||||
echo "$!" >"$STAGE/serve.pid"
|
echo "$!" >"$STAGE/serve.pid"
|
||||||
|
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Decode throughput benchmark for frank-large.
|
# Decode throughput benchmark for frank-dragoon.
|
||||||
# Measures generation tok/s with 8K+ token prompts and 256 generated tokens.
|
# Measures generation tok/s with 8K+ token prompts and 256 generated tokens.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
@@ -57,7 +57,7 @@ run_bench() {
|
|||||||
# Warmup (small request)
|
# Warmup (small request)
|
||||||
curl -s -m 60 http://127.0.0.1:$PORT/v1/chat/completions \
|
curl -s -m 60 http://127.0.0.1:$PORT/v1/chat/completions \
|
||||||
-H "Content-Type: application/json" \
|
-H "Content-Type: application/json" \
|
||||||
-d '{"model":"frank-large","messages":[{"role":"user","content":"Hi"}],"max_tokens":16,"stream":false}' > /dev/null 2>&1 || true
|
-d '{"model":"frank-dragoon","messages":[{"role":"user","content":"Hi"}],"max_tokens":16,"stream":false}' > /dev/null 2>&1 || true
|
||||||
|
|
||||||
# Build 8K prompt
|
# Build 8K prompt
|
||||||
local prompt
|
local prompt
|
||||||
@@ -75,7 +75,7 @@ print(json.dumps(' '.join(words)))
|
|||||||
t0=$(date +%s%N)
|
t0=$(date +%s%N)
|
||||||
resp=$(curl -s -m 600 http://127.0.0.1:$PORT/v1/chat/completions \
|
resp=$(curl -s -m 600 http://127.0.0.1:$PORT/v1/chat/completions \
|
||||||
-H "Content-Type: application/json" \
|
-H "Content-Type: application/json" \
|
||||||
-d "{\"model\":\"frank-large\",\"messages\":[{\"role\":\"user\",\"content\":$prompt}],\"max_tokens\":$N_GEN,\"stream\":false}" 2>&1)
|
-d "{\"model\":\"frank-dragoon\",\"messages\":[{\"role\":\"user\",\"content\":$prompt}],\"max_tokens\":$N_GEN,\"stream\":false}" 2>&1)
|
||||||
t1=$(date +%s%N)
|
t1=$(date +%s%N)
|
||||||
|
|
||||||
if ! echo "$resp" | python3 -c "import sys,json; json.load(sys.stdin)" 2>/dev/null; then
|
if ! echo "$resp" | python3 -c "import sys,json; json.load(sys.stdin)" 2>/dev/null; then
|
||||||
|
|||||||
Executable
+48
@@ -0,0 +1,48 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""bench-probe.py — RUNS ON DANKSERVER. The campaign's decode probe: one
|
||||||
|
/completion request, a prompt assembled from calib.jsonl to ~TARGET tokens,
|
||||||
|
temperature 0, max_tokens 512. Reports the engine's own timings: decode rate,
|
||||||
|
prefill rate, draft accept. Run 3x; the median is the arm's number."""
|
||||||
|
import json, sys, urllib.request
|
||||||
|
|
||||||
|
TARGET = 43600 # tokens
|
||||||
|
BYTES_PER_TOK = 3.8
|
||||||
|
URL = "http://localhost:7770/v1/completions"
|
||||||
|
|
||||||
|
def build_prompt():
|
||||||
|
want = int(TARGET * BYTES_PER_TOK)
|
||||||
|
parts = []
|
||||||
|
n = 0
|
||||||
|
with open("/home/dank/corpora/out/calib.jsonl") as f:
|
||||||
|
for line in f:
|
||||||
|
d = json.loads(line)
|
||||||
|
p = d.get("prompt") or d.get("messages", [{}])[0].get("content", "")
|
||||||
|
parts.append(p)
|
||||||
|
n += len(p.encode())
|
||||||
|
if n >= want:
|
||||||
|
break
|
||||||
|
return "".join(parts)
|
||||||
|
|
||||||
|
def probe(salt):
|
||||||
|
prompt = f"[probe {salt}]\n" + build_prompt()
|
||||||
|
body = json.dumps({
|
||||||
|
"model": "frank-archon",
|
||||||
|
"prompt": prompt, "temperature": 0.0, "max_tokens": 512,
|
||||||
|
}).encode()
|
||||||
|
req = urllib.request.Request(URL, data=body,
|
||||||
|
headers={"Content-Type": "application/json"})
|
||||||
|
with urllib.request.urlopen(req, timeout=1800) as r:
|
||||||
|
d = json.load(r)
|
||||||
|
t = d.get("timings", {})
|
||||||
|
return {
|
||||||
|
"prompt_n": t.get("prompt_n"),
|
||||||
|
"prefill_tps": round(t.get("prompt_per_second", 0), 1),
|
||||||
|
"pred_n": t.get("predicted_n"),
|
||||||
|
"decode_tps": round(t.get("predicted_per_second", 0), 2),
|
||||||
|
"draft_n": t.get("draft_n"),
|
||||||
|
"accept": (round(t["draft_n_accepted"] / t["draft_n"], 4)
|
||||||
|
if t.get("draft_n") else None),
|
||||||
|
}
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
print(json.dumps(probe(sys.argv[1] if len(sys.argv) > 1 else "a")))
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""
|
"""
|
||||||
HTTP-based prefill-throughput sweep for frank-large.
|
HTTP-based prefill-throughput sweep for frank-dragoon.
|
||||||
Starts the server with a given --threads-batch value, sends a 64K-prompt via
|
Starts the server with a given --threads-batch value, sends a 64K-prompt via
|
||||||
the chat API (max_tokens=0), measures prompt_tokens/wall_time.
|
the chat API (max_tokens=0), measures prompt_tokens/wall_time.
|
||||||
|
|
||||||
@@ -39,7 +39,7 @@ def send_prefill_request():
|
|||||||
with open(PROMPT_FILE, "r") as f:
|
with open(PROMPT_FILE, "r") as f:
|
||||||
prompt = f.read()
|
prompt = f.read()
|
||||||
body = json.dumps({
|
body = json.dumps({
|
||||||
"model": "frank-large",
|
"model": "frank-dragoon",
|
||||||
"messages": [{"role": "user", "content": prompt}],
|
"messages": [{"role": "user", "content": prompt}],
|
||||||
"max_tokens": 1,
|
"max_tokens": 1,
|
||||||
"stream": False
|
"stream": False
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Threads-batch sweep for frank-large on dankclaw
|
# Threads-batch sweep for frank-dragoon on dankclaw
|
||||||
# Benches prefill throughput at 65536 prompt tokens across threads-batch values.
|
# Benches prefill throughput at 65536 prompt tokens across threads-batch values.
|
||||||
set -uo pipefail
|
set -uo pipefail
|
||||||
|
|
||||||
@@ -10,7 +10,7 @@ TB_LIST=(16 32 48 64 80 96 128)
|
|||||||
ENV_FILE="/home/dank/GitPackages/ROCmFPX/build-qwen38/bin"
|
ENV_FILE="/home/dank/GitPackages/ROCmFPX/build-qwen38/bin"
|
||||||
|
|
||||||
echo "========================================"
|
echo "========================================"
|
||||||
echo "THREADS-BATCH SWEEP — frank-large"
|
echo "THREADS-BATCH SWEEP — frank-dragoon"
|
||||||
echo "Model: $(basename "$MODEL") ($(du -h "$MODEL" | cut -f1))"
|
echo "Model: $(basename "$MODEL") ($(du -h "$MODEL" | cut -f1))"
|
||||||
echo "Threads-batch values: ${TB_LIST[*]}"
|
echo "Threads-batch values: ${TB_LIST[*]}"
|
||||||
echo "Prompt: 65536 tokens, n-gen: 0 (prefill-only)"
|
echo "Prompt: 65536 tokens, n-gen: 0 (prefill-only)"
|
||||||
|
|||||||
@@ -1,14 +1,14 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Benchmark ubatch with llama-bench for frank-large.
|
# Benchmark ubatch with llama-bench for frank-dragoon.
|
||||||
#
|
#
|
||||||
# This is intentionally a ubatch-only benchmark. It uses one llama-bench
|
# This is intentionally a ubatch-only benchmark. It uses one llama-bench
|
||||||
# process, an 8K prompt workload, and equal batch/ubatch values. No server
|
# process, an 8K prompt workload, and equal batch/ubatch values. No server
|
||||||
# process, HTTP requests, or log scraping are used.
|
# process, HTTP requests, or log scraping are used.
|
||||||
#
|
#
|
||||||
# Run on Dankclaw after stopping frank-large:
|
# Run on Dankclaw after stopping frank-dragoon:
|
||||||
# sudo systemctl stop frank-large
|
# sudo systemctl stop frank-dragoon
|
||||||
# ./scripts/bench-ubatch.sh
|
# ./scripts/bench-ubatch.sh
|
||||||
# sudo systemctl start frank-large
|
# sudo systemctl start frank-dragoon
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
: "${LLAMA_BENCH:=/home/dank/GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-bench}"
|
: "${LLAMA_BENCH:=/home/dank/GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-bench}"
|
||||||
|
|||||||
@@ -1,11 +1,11 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# build-frank-primary-rad.sh — build frank-primary's .rad container from the
|
# build-frank-archon-rad.sh — build frank-archon's .rad container from the
|
||||||
# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint, at NATIVE precision.
|
# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint, at NATIVE precision.
|
||||||
#
|
#
|
||||||
# bash scripts/build-frank-primary-rad.sh # convert
|
# bash scripts/build-frank-archon-rad.sh # convert
|
||||||
# bash scripts/build-frank-primary-rad.sh --plan-only # declare, no I/O
|
# bash scripts/build-frank-archon-rad.sh --plan-only # declare, no I/O
|
||||||
# bash scripts/build-frank-primary-rad.sh --restage # redo staging
|
# bash scripts/build-frank-archon-rad.sh --restage # redo staging
|
||||||
# bash scripts/build-frank-primary-rad.sh --force # rebuild existing
|
# bash scripts/build-frank-archon-rad.sh --force # rebuild existing
|
||||||
#
|
#
|
||||||
# WHY A STAGING DIR (the one non-obvious step): rad-convert selects its
|
# WHY A STAGING DIR (the one non-obvious step): rad-convert selects its
|
||||||
# architecture plugin by the checkpoint's quantisation DESCRIPTOR —
|
# architecture plugin by the checkpoint's quantisation DESCRIPTOR —
|
||||||
@@ -55,8 +55,8 @@ OUT="$OUT_DIR/qwen4exp-fp8-native.rad"
|
|||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
# shellcheck disable=SC1091
|
# shellcheck disable=SC1091
|
||||||
source "$SCRIPT_DIR/frank-primary.env"
|
source "$SCRIPT_DIR/frank-archon.env"
|
||||||
IMAGE="$FRANK_PRIMARY_IMAGE"
|
IMAGE="$FRANK_ARCHON_IMAGE"
|
||||||
|
|
||||||
plan_only=0 restage=0 force=0
|
plan_only=0 restage=0 force=0
|
||||||
for a in "$@"; do
|
for a in "$@"; do
|
||||||
@@ -69,11 +69,11 @@ for a in "$@"; do
|
|||||||
done
|
done
|
||||||
|
|
||||||
if [ ! -f "$SRC/config.json" ]; then
|
if [ ! -f "$SRC/config.json" ]; then
|
||||||
echo "ERROR: raw checkpoint missing: $SRC (run scripts/download-frank-primary.sh --fetch)" >&2
|
echo "ERROR: raw checkpoint missing: $SRC (run scripts/download-frank-archon.sh --fetch)" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
echo "verifying raw checkpoint first..."
|
echo "verifying raw checkpoint first..."
|
||||||
bash "$SCRIPT_DIR/download-frank-primary.sh"
|
bash "$SCRIPT_DIR/download-frank-archon.sh"
|
||||||
|
|
||||||
if [ -f "$OUT" ] && [ "$plan_only" -eq 0 ] && [ "$force" -eq 0 ]; then
|
if [ -f "$OUT" ] && [ "$plan_only" -eq 0 ] && [ "$force" -eq 0 ]; then
|
||||||
echo "container already exists: $OUT"
|
echo "container already exists: $OUT"
|
||||||
@@ -83,7 +83,7 @@ fi
|
|||||||
|
|
||||||
# ---- staging ----------------------------------------------------------------
|
# ---- staging ----------------------------------------------------------------
|
||||||
if [ "$restage" -eq 1 ] && [ -e "$STAGE" ]; then
|
if [ "$restage" -eq 1 ] && [ -e "$STAGE" ]; then
|
||||||
TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-primary-stage"
|
TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-archon-stage"
|
||||||
mkdir -p "$TRASH"
|
mkdir -p "$TRASH"
|
||||||
mv "$STAGE" "$TRASH/stage-$(date +%H%M%S)"
|
mv "$STAGE" "$TRASH/stage-$(date +%H%M%S)"
|
||||||
echo "old staging dir moved to $TRASH"
|
echo "old staging dir moved to $TRASH"
|
||||||
@@ -138,7 +138,7 @@ CONVERT_ARGS=(/models/stage-qwen4exp-fp8 -o /models/rad/qwen4exp-fp8-native.rad
|
|||||||
# exactly and the plugin's one-element scale plane is produced.
|
# exactly and the plugin's one-element scale plane is produced.
|
||||||
CONVERT_ARGS+=(--quant "blk.*.ple_ngram.weight=rtn:codes=fp8_e4m3,block=*x*,scale=bf16,rule=fixed,scale_value=$PLE_SCALE")
|
CONVERT_ARGS+=(--quant "blk.*.ple_ngram.weight=rtn:codes=fp8_e4m3,block=*x*,scale=bf16,rule=fixed,scale_value=$PLE_SCALE")
|
||||||
[ "$force" -eq 1 ] && [ -f "$OUT" ] && {
|
[ "$force" -eq 1 ] && [ -f "$OUT" ] && {
|
||||||
TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-primary-rad"
|
TRASH="$HOME/.AGENTTRASH/$(date +%Y%m%d)-frank-archon-rad"
|
||||||
mkdir -p "$TRASH"
|
mkdir -p "$TRASH"
|
||||||
mv "$OUT" "$TRASH/qwen4exp-fp8-native.rad.$(date +%H%M%S)"
|
mv "$OUT" "$TRASH/qwen4exp-fp8-native.rad.$(date +%H%M%S)"
|
||||||
echo "previous container moved to $TRASH"
|
echo "previous container moved to $TRASH"
|
||||||
@@ -158,8 +158,8 @@ echo "rad-info summary:"
|
|||||||
podman run --rm --entrypoint rad-info -v /home/dank/models:/models:ro \
|
podman run --rm --entrypoint rad-info -v /home/dank/models:/models:ro \
|
||||||
"$IMAGE" "/models/rad/qwen4exp-fp8-native.rad" | head -20
|
"$IMAGE" "/models/rad/qwen4exp-fp8-native.rad" | head -20
|
||||||
MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
|
MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
|
||||||
echo "container: $OUT (${MIB} MiB; env floor $FRANK_PRIMARY_RAD_MIN_MIB MiB)"
|
echo "container: $OUT (${MIB} MiB; env floor $FRANK_ARCHON_RAD_MIN_MIB MiB)"
|
||||||
if [ "$MIB" -lt "$FRANK_PRIMARY_RAD_MIN_MIB" ]; then
|
if [ "$MIB" -lt "$FRANK_ARCHON_RAD_MIN_MIB" ]; then
|
||||||
echo "ERROR: container below the env floor — the convert was truncated" >&2
|
echo "ERROR: container below the env floor — the convert was truncated" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Build the gufo runtime that serves frank-large on Dankclaw.
|
# Build the gufo runtime that serves frank-dragoon on Dankclaw.
|
||||||
#
|
#
|
||||||
# bash scripts/build-gufo.sh # build + install the pinned tag
|
# bash scripts/build-gufo.sh # build + install the pinned tag
|
||||||
# bash scripts/build-gufo.sh --verify # only check the installed tree
|
# bash scripts/build-gufo.sh --verify # only check the installed tree
|
||||||
@@ -93,4 +93,4 @@ cmake --install build/release
|
|||||||
|
|
||||||
echo "== verify =="
|
echo "== verify =="
|
||||||
"$PREFIX/bin/gufo" diagnose
|
"$PREFIX/bin/gufo" diagnose
|
||||||
echo "frank-large runtime ready: $PREFIX/bin/gufo ($GUFO_TAG @ $commit)"
|
echo "frank-dragoon runtime ready: $PREFIX/bin/gufo ($GUFO_TAG @ $commit)"
|
||||||
@@ -1,11 +1,11 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Build the ROCmFPX runtime that serves frank-large on Dankclaw.
|
# Build the ROCmFPX runtime that serves frank-dragoon on Dankclaw.
|
||||||
#
|
#
|
||||||
# bash scripts/build-rocmfpx.sh # clone/fetch + build the pin
|
# bash scripts/build-rocmfpx.sh # clone/fetch + build the pin
|
||||||
# bash scripts/build-rocmfpx.sh --verify # only check the installed tree
|
# bash scripts/build-rocmfpx.sh --verify # only check the installed tree
|
||||||
#
|
#
|
||||||
# Runtime: kingjones30/ROCmFPX — charlie12345/ROCmFPX plus the model-card
|
# Runtime: kingjones30/ROCmFPX — charlie12345/ROCmFPX plus the model-card
|
||||||
# patches, i.e. the fork that owns the ROCmFPx tensor formats. frank-large's
|
# patches, i.e. the fork that owns the ROCmFPx tensor formats. frank-dragoon's
|
||||||
# payload is ftype 115 (`Q8_0_ROCMFPX_AGENT`): 311 tensors of
|
# payload is ftype 115 (`Q8_0_ROCMFPX_AGENT`): 311 tensors of
|
||||||
# GGML_TYPE_Q8_0_ROCMFPX (103), a type that exists ONLY in this fork, so
|
# GGML_TYPE_Q8_0_ROCMFPX (103), a type that exists ONLY in this fork, so
|
||||||
# stock llama.cpp, gufo and every other loader on the host reject the file.
|
# stock llama.cpp, gufo and every other loader on the host reject the file.
|
||||||
@@ -30,7 +30,7 @@
|
|||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
ROCMFPX_REPO="${ROCMFPX_REPO:-https://github.com/kingjones30/ROCmFPX.git}"
|
ROCMFPX_REPO="${ROCMFPX_REPO:-https://github.com/kingjones30/ROCmFPX.git}"
|
||||||
# The commit frank-large was qualified with (measured 2026-10-02).
|
# The commit frank-dragoon was qualified with (measured 2026-10-02).
|
||||||
ROCMFPX_COMMIT="${ROCMFPX_COMMIT:-36e9acd40e10a87cd3c3ef8ec734668757dc8520}"
|
ROCMFPX_COMMIT="${ROCMFPX_COMMIT:-36e9acd40e10a87cd3c3ef8ec734668757dc8520}"
|
||||||
SRC_DIR="${ROCMFPX_SRC_DIR:-/home/dank/GitPackages/kingjones-ROCmFPX}"
|
SRC_DIR="${ROCMFPX_SRC_DIR:-/home/dank/GitPackages/kingjones-ROCmFPX}"
|
||||||
BUILD_DIR="${ROCMFPX_BUILD_DIR:-$SRC_DIR/build}"
|
BUILD_DIR="${ROCMFPX_BUILD_DIR:-$SRC_DIR/build}"
|
||||||
@@ -86,4 +86,4 @@ cmake --build "$BUILD_DIR" --target llama-server llama-quantize -j "$JOBS"
|
|||||||
|
|
||||||
echo "== verify =="
|
echo "== verify =="
|
||||||
"$BUILD_DIR/bin/llama-server" --version | head -1
|
"$BUILD_DIR/bin/llama-server" --version | head -1
|
||||||
echo "frank-large runtime ready: $BUILD_DIR/bin/llama-server ($ROCMFPX_COMMIT)"
|
echo "frank-dragoon runtime ready: $BUILD_DIR/bin/llama-server ($ROCMFPX_COMMIT)"
|
||||||
Executable
+110
@@ -0,0 +1,110 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""calib-push.py — drive a calibration corpus through a radiance scratch
|
||||||
|
server's prefill so its tap writes layer Hessians (docs/MOE-W4.md).
|
||||||
|
|
||||||
|
Runs ON DANKSERVER next to the scratch server (the one autoresearch- /
|
||||||
|
serve-style scratch container booted with RADIANCE_CALIB_DIR set).
|
||||||
|
Reads the corpus JSONL produced by make-calib-corpus.py
|
||||||
|
({"prompt": "..."} a line), POSTs each prompt to /v1/completions with
|
||||||
|
max_tokens=1 (prefill-only cost, one token out), at N-way concurrency
|
||||||
|
matching the scratch server's --max-num-seqs budget.
|
||||||
|
|
||||||
|
Corpus rounds repeat until the engine prints the tap's completion line
|
||||||
|
("calibration wrote layer Hessians") in `podman logs <container>` — that
|
||||||
|
is the ground truth that RADIANCE_CALIB_TOKENS was reached. After that
|
||||||
|
line, further prompts are pure waste, so the driver exits.
|
||||||
|
|
||||||
|
Mechanics notes:
|
||||||
|
- requests carry no sampling params that matter (max_tokens=1, greedy).
|
||||||
|
- a per-request timeout of 30 min: at ~1k tok/s prefill a 12k-token
|
||||||
|
prompt is a few seconds; the timeout only catches a wedged engine.
|
||||||
|
- failures print and continue: the tap pools over tokens, a few lost
|
||||||
|
prompts do not bias the Hessians (the corpus is drawn round-robin over
|
||||||
|
source classes, so loss is class-neutral as long as failures are rare;
|
||||||
|
a systematic 4xx is loud in the log and must be fixed, not ignored).
|
||||||
|
|
||||||
|
Usage: calib-push.py [--port 7770] [--parallel 64] [--container arbench]
|
||||||
|
[--corpus /home/dank/corpora/out/calib.jsonl] [--max-rounds 4]
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import concurrent.futures
|
||||||
|
import json
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
DONE_MARK = "calibration wrote"
|
||||||
|
|
||||||
|
|
||||||
|
def tap_done(container):
|
||||||
|
"""True if the engine logged the tap's completion line."""
|
||||||
|
try:
|
||||||
|
out = subprocess.run(
|
||||||
|
["podman", "logs", "--tail", "50", container],
|
||||||
|
capture_output=True, text=True, timeout=30,
|
||||||
|
)
|
||||||
|
except Exception:
|
||||||
|
return False
|
||||||
|
return DONE_MARK in (out.stdout + out.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
def push_one(url, prompt):
|
||||||
|
body = json.dumps({
|
||||||
|
"model": "frank-archon",
|
||||||
|
"prompt": prompt,
|
||||||
|
"max_tokens": 1,
|
||||||
|
"temperature": 0,
|
||||||
|
"stream": False,
|
||||||
|
}).encode()
|
||||||
|
req = urllib.request.Request(
|
||||||
|
url, data=body, headers={"Content-Type": "application/json"})
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=1800) as r:
|
||||||
|
r.read()
|
||||||
|
return True
|
||||||
|
except Exception as e:
|
||||||
|
print(f"FAIL {str(e)[:120]}", flush=True)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("--port", type=int, default=7770)
|
||||||
|
ap.add_argument("--parallel", type=int, default=64)
|
||||||
|
ap.add_argument("--container", default="arbench")
|
||||||
|
ap.add_argument("--corpus", default="/home/dank/corpora/out/calib.jsonl")
|
||||||
|
ap.add_argument("--max-rounds", type=int, default=4)
|
||||||
|
a = ap.parse_args()
|
||||||
|
url = f"http://127.0.0.1:{a.port}/v1/completions"
|
||||||
|
|
||||||
|
prompts = []
|
||||||
|
with open(a.corpus) as f:
|
||||||
|
for line in f:
|
||||||
|
line = line.strip()
|
||||||
|
if line:
|
||||||
|
prompts.append(json.loads(line)["prompt"])
|
||||||
|
print(f"corpus: {len(prompts)} prompts", flush=True)
|
||||||
|
|
||||||
|
for rnd in range(1, a.max_rounds + 1):
|
||||||
|
if tap_done(a.container):
|
||||||
|
break
|
||||||
|
t0 = time.time()
|
||||||
|
ok = 0
|
||||||
|
with concurrent.futures.ThreadPoolExecutor(a.parallel) as ex:
|
||||||
|
for good in ex.map(lambda p: push_one(url, p), prompts):
|
||||||
|
ok += 1 if good else 0
|
||||||
|
dt = time.time() - t0
|
||||||
|
print(f"round {rnd}: {ok}/{len(prompts)} ok in {dt:.0f}s "
|
||||||
|
f"({sum(len(p) for p in prompts)/4/dt:.0f} chars/s est)",
|
||||||
|
flush=True)
|
||||||
|
if tap_done(a.container):
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
print("max rounds exhausted, tap not done", flush=True)
|
||||||
|
sys.exit(1)
|
||||||
|
print("TAP_DONE: calibration wrote layer Hessians", flush=True)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Executable
+73
@@ -0,0 +1,73 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# convert-swift15-w4dn8.sh — RUNS ON DANKSERVER. The final recipe convert of
|
||||||
|
# the Swift 1.5 checkpoint: gate_up experts through GPTQ against the 10M-token
|
||||||
|
# calibration Grams, down projections block-fp8, protected experts bf16,
|
||||||
|
# trunk int8, PLE fixed-scale fp8 (scripts/recipes/
|
||||||
|
# qwen4exp-swift15-w4dn8-i8-hc8m.recipe).
|
||||||
|
#
|
||||||
|
# The convert engine is 1.0.13 — the SAME pin that will serve the result, so
|
||||||
|
# the .rad crosses no version boundary. Only the Gram files cross versions
|
||||||
|
# (1.2.0 wrote them; the RADGRAM2 magic, gram.r%d.%s.bin naming and the
|
||||||
|
# calib_gram/calib_resolve/calib_stem symbols are byte-identical between the
|
||||||
|
# two libquant.so). rad-convert runs on the host backend (no GPU visible in
|
||||||
|
# the container) — production can stay up through this script.
|
||||||
|
#
|
||||||
|
# Prereqs (campaign order): calibration complete (calib-push printed TAP_DONE),
|
||||||
|
# the protected-expert cast lines rewritten into the recipe by
|
||||||
|
# derive-protected-experts.py, and the 335 GiB bf16 container deleted —
|
||||||
|
# the peak-disk arithmetic needs that space (checkpoint 335.3 + output ~132
|
||||||
|
# GiB on the volume; --reuse is NOT used, see the recipe header).
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
PIN_1013=sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
|
||||||
|
IMAGE="stilldeadcode/radiance@$PIN_1013"
|
||||||
|
CKPT=/home/dank/models/ukisai/Swift1.5-Qwen3.8-Flash-Next
|
||||||
|
RECIPE=/home/dank/serverAppConfigs-swift15/scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe
|
||||||
|
CALIB=/home/dank/corpora/calib
|
||||||
|
OUT=/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad
|
||||||
|
LOG=/home/dank/logs/swift15_convert.log
|
||||||
|
# gate_up ~37.6 + down ~18.8 + PLE 47.7 + trunk/embeddings/head ~10 GiB (all-w4 expert table;
|
||||||
|
# the plugin's one-encoding law dropped the block-fp8 down tier -- see the recipe header).
|
||||||
|
MIN_MIB=${FRANK_SWIFT15_RAD_MIN_MIB:-108000}
|
||||||
|
|
||||||
|
# Gate 1: inputs. A missing Gram directory converts UNCALIBRATED with a
|
||||||
|
# per-tensor stderr note, which is silently worse than failing here.
|
||||||
|
[ -d "$CKPT" ] || { echo "no checkpoint at $CKPT" >&2; exit 1; }
|
||||||
|
[ -f "$RECIPE" ] || { echo "no recipe at $RECIPE" >&2; exit 1; }
|
||||||
|
GRAMS=$(find "$CALIB" -maxdepth 1 -name 'gram.r*.bin' 2>/dev/null | wc -l)
|
||||||
|
[ "$GRAMS" -ge 48 ] || { echo "calib dir $CALIB holds $GRAMS gram files (<48)" >&2; exit 1; }
|
||||||
|
if [ -e "$OUT" ]; then echo "$OUT already exists; move it to trash first" >&2; exit 1; fi
|
||||||
|
if [ -e /home/dank/models/rad/qwen4exp-swift15-bf16.rad ]; then
|
||||||
|
echo "the bf16 container is still on disk; delete it first (peak disk)" >&2; exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Gate 2: disk. Output ~132 GiB plus convert staging, 20 GiB of slack.
|
||||||
|
FREE_MIB=$(df --output=avail -m /home/dank/models | tail -1 | tr -d ' ')
|
||||||
|
[ "$FREE_MIB" -ge $((MIN_MIB + 20480)) ] || {
|
||||||
|
echo "only ${FREE_MIB} MiB free on /home/dank/models" >&2; exit 1; }
|
||||||
|
|
||||||
|
# The recipe names $CALIB; rad-convert expands it from the environment
|
||||||
|
# (MOE-W4.md:275). The recipe's cast section must name THIS checkpoint's
|
||||||
|
# protected experts — derive-protected-experts.py rewrote it.
|
||||||
|
echo "convert: $CKPT -> $OUT (gptq calib=$CALIB, $(date -Is))" | tee "$LOG"
|
||||||
|
podman run --rm --name swift15-convert \
|
||||||
|
-v /home/dank/models:/models \
|
||||||
|
-v /home/dank/serverAppConfigs-swift15:/campaign:ro \
|
||||||
|
-v "$CALIB:/data/calib:ro" \
|
||||||
|
-e CALIB=/data/calib \
|
||||||
|
--entrypoint /opt/radiance/bin/rad-convert \
|
||||||
|
"$IMAGE" "/models/${CKPT#/home/dank/models/}" \
|
||||||
|
--recipe /campaign/"${RECIPE#/home/dank/serverAppConfigs-swift15/}" \
|
||||||
|
-o /models/rad/"$(basename "$OUT")" 2>&1 | tee -a "$LOG"
|
||||||
|
rc=${PIPESTATUS[0]}
|
||||||
|
[ "$rc" -eq 0 ] || { echo "rad-convert rc=$rc; log: $LOG" >&2; exit "$rc"; }
|
||||||
|
|
||||||
|
# Gate 3: the artifact. rad-info's recipe echo is the receipt that GPTQ ran.
|
||||||
|
SIZE_MIB=$(( $(stat -c %s "$OUT") / 1048576 ))
|
||||||
|
[ "$SIZE_MIB" -ge "$MIN_MIB" ] || {
|
||||||
|
echo "$OUT is ${SIZE_MIB} MiB < floor ${MIN_MIB}" >&2; exit 1; }
|
||||||
|
podman run --rm --entrypoint /opt/radiance/bin/rad-info \
|
||||||
|
-v /home/dank/models:/models:ro "$IMAGE" /models/rad/"$(basename "$OUT")" \
|
||||||
|
2>&1 | tee -a "$LOG" | grep -c 'gptq' >/dev/null \
|
||||||
|
|| echo "WARNING: rad-info shows no gptq rule in the recorded recipe" >&2
|
||||||
|
echo "CONVERT_DONE ${SIZE_MIB} MiB -> $OUT"
|
||||||
@@ -57,9 +57,9 @@ else
|
|||||||
echo " none"
|
echo " none"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
section "frank-primary unit"
|
section "frank-archon unit"
|
||||||
printf ' active: %s\n' "$(systemctl is-active frank-primary 2>/dev/null | head -1)"
|
printf ' active: %s\n' "$(systemctl is-active frank-archon 2>/dev/null | head -1)"
|
||||||
printf ' enabled: %s\n' "$(systemctl is-enabled frank-primary 2>/dev/null | head -1)"
|
printf ' enabled: %s\n' "$(systemctl is-enabled frank-archon 2>/dev/null | head -1)"
|
||||||
|
|
||||||
section "glm serve tail"
|
section "glm serve tail"
|
||||||
if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qx "$GLM_CTR"; then
|
if podman ps -a --format '{{.Names}}' 2>/dev/null | grep -qx "$GLM_CTR"; then
|
||||||
|
|||||||
Executable
+55
@@ -0,0 +1,55 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""deep-probe.py — one chat completion at ~43.6k prompt tokens, temp 0, 512
|
||||||
|
generated, streamed; decode = completion_tokens / (total - ttft)."""
|
||||||
|
import json, time, urllib.request
|
||||||
|
|
||||||
|
TARGET_BYTES = int(43600 * 3.8)
|
||||||
|
|
||||||
|
def build():
|
||||||
|
parts, n = [], 0
|
||||||
|
with open("/home/dank/corpora/out/calib.jsonl") as f:
|
||||||
|
for line in f:
|
||||||
|
p = json.loads(line)["prompt"]
|
||||||
|
parts.append(p)
|
||||||
|
n += len(p.encode())
|
||||||
|
if n >= TARGET_BYTES:
|
||||||
|
break
|
||||||
|
return "".join(parts)
|
||||||
|
|
||||||
|
def main(salt):
|
||||||
|
body = json.dumps({
|
||||||
|
"model": "frank-archon",
|
||||||
|
"messages": [{"role": "user", "content": f"[probe {salt}] Continue and summarize this material in English, 400 words.\n\n" + build()}],
|
||||||
|
"temperature": 0.0, "max_tokens": 512,
|
||||||
|
"stream": True, "stream_options": {"include_usage": True},
|
||||||
|
}).encode()
|
||||||
|
req = urllib.request.Request("http://localhost:7770/v1/chat/completions",
|
||||||
|
data=body, headers={"Content-Type": "application/json"})
|
||||||
|
t0 = time.time(); ttft = None; ctok = 0; usage = {}
|
||||||
|
with urllib.request.urlopen(req, timeout=2400) as r:
|
||||||
|
for raw in r:
|
||||||
|
line = raw.decode().strip()
|
||||||
|
if not line.startswith("data:"):
|
||||||
|
continue
|
||||||
|
d = line[5:].strip()
|
||||||
|
if d == "[DONE]":
|
||||||
|
break
|
||||||
|
j = json.loads(d)
|
||||||
|
if j.get("usage"):
|
||||||
|
usage = j["usage"]
|
||||||
|
for ch in j.get("choices", []):
|
||||||
|
delta = (ch.get("delta") or {})
|
||||||
|
piece = delta.get("content") or delta.get("reasoning_content") or ""
|
||||||
|
if piece and ttft is None:
|
||||||
|
ttft = time.time() - t0
|
||||||
|
ctok += 1
|
||||||
|
total = time.time() - t0
|
||||||
|
gen = usage.get("completion_tokens") or ctok
|
||||||
|
dec = gen / (total - (ttft or 0)) if total > (ttft or 0) else 0
|
||||||
|
print(json.dumps({"salt": salt, "prompt_tokens": usage.get("prompt_tokens"),
|
||||||
|
"completion_tokens": gen, "ttft_s": round(ttft or 0, 2),
|
||||||
|
"total_s": round(total, 1), "decode_tps": round(dec, 2)}))
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
import sys
|
||||||
|
main(sys.argv[1] if len(sys.argv) > 1 else "x")
|
||||||
+23
-23
@@ -61,9 +61,9 @@ SERVICE_ORDER = (
|
|||||||
"sonarr",
|
"sonarr",
|
||||||
"lidarr",
|
"lidarr",
|
||||||
"readarr",
|
"readarr",
|
||||||
"frank-large",
|
"frank-dragoon",
|
||||||
"frank-secondary",
|
"frank-zealot",
|
||||||
"frank-primary",
|
"frank-archon",
|
||||||
)
|
)
|
||||||
|
|
||||||
DANKCLAW_SERVICES = {
|
DANKCLAW_SERVICES = {
|
||||||
@@ -93,8 +93,8 @@ DANKCLAW_SERVICES = {
|
|||||||
"authentik-outpost-ldap",
|
"authentik-outpost-ldap",
|
||||||
"caddy",
|
"caddy",
|
||||||
"homepage",
|
"homepage",
|
||||||
"frank-large",
|
"frank-dragoon",
|
||||||
"frank-secondary",
|
"frank-zealot",
|
||||||
}
|
}
|
||||||
|
|
||||||
DANKSERVER_SERVICES = {
|
DANKSERVER_SERVICES = {
|
||||||
@@ -113,7 +113,7 @@ DANKSERVER_SERVICES = {
|
|||||||
"sonarr",
|
"sonarr",
|
||||||
"lidarr",
|
"lidarr",
|
||||||
"readarr",
|
"readarr",
|
||||||
"frank-primary",
|
"frank-archon",
|
||||||
"node-exporter",
|
"node-exporter",
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -316,7 +316,7 @@ def services_for_path(path: str, target: str) -> set[str]:
|
|||||||
if len(parts) >= 2 and parts[1] == "open-webui":
|
if len(parts) >= 2 and parts[1] == "open-webui":
|
||||||
_add(services, "open-webui")
|
_add(services, "open-webui")
|
||||||
return services
|
return services
|
||||||
# frank-large left the quadlet shape on 2026-09-19 (CIRU v4.4.1 host
|
# frank-dragoon left the quadlet shape on 2026-09-19 (CIRU v4.4.1 host
|
||||||
# build, root unit): its files live in scripts/ and map there.
|
# build, root unit): its files live in scripts/ and map there.
|
||||||
if len(parts) >= 2 and parts[1] == "observability":
|
if len(parts) >= 2 and parts[1] == "observability":
|
||||||
# The five observability containers became Quadlet units on
|
# The five observability containers became Quadlet units on
|
||||||
@@ -371,10 +371,10 @@ def services_for_path(path: str, target: str) -> set[str]:
|
|||||||
if name == "node-exporter":
|
if name == "node-exporter":
|
||||||
_add(services, "node-exporter")
|
_add(services, "node-exporter")
|
||||||
return services
|
return services
|
||||||
# frank-primary's quadlet-era directory left the tree on 2026-09-26
|
# frank-archon's quadlet-era directory left the tree on 2026-09-26
|
||||||
# (the Lucebox DS4.1 unit was retired for the llama.cpp root unit —
|
# (the Lucebox DS4.1 unit was retired for the llama.cpp root unit —
|
||||||
# the trio in scripts/ now maps it). Nothing lives under
|
# the trio in scripts/ now maps it). Nothing lives under
|
||||||
# dankserver/frank-primary/ except dated trash, and trash plans
|
# dankserver/frank-archon/ except dated trash, and trash plans
|
||||||
# nothing (the retire rule above the service blocks).
|
# nothing (the retire rule above the service blocks).
|
||||||
return services
|
return services
|
||||||
|
|
||||||
@@ -400,32 +400,32 @@ def services_for_path(path: str, target: str) -> set[str]:
|
|||||||
# change to it redeploying alertmanager is the only path that
|
# change to it redeploying alertmanager is the only path that
|
||||||
# reaches it on the host.
|
# reaches it on the host.
|
||||||
_add(services, "alertmanager")
|
_add(services, "alertmanager")
|
||||||
if name in {"frank-secondary.env", "frank-secondary.service", "serve-frank-secondary.sh"}:
|
if name in {"frank-zealot.env", "frank-zealot.service", "serve-frank-zealot.sh"}:
|
||||||
# llama.cpp frank-secondary (2026-09-16) keeps its unit, env and serve
|
# llama.cpp frank-zealot (2026-09-16) keeps its unit, env and serve
|
||||||
# files here; each one selects frank-secondary only. The ik_llama.cpp
|
# files here; each one selects frank-zealot only. The ik_llama.cpp
|
||||||
# trio that lived here for two days is in scripts/.AGENTTRASH/,
|
# trio that lived here for two days is in scripts/.AGENTTRASH/,
|
||||||
# which is archive material and plans nothing.
|
# which is archive material and plans nothing.
|
||||||
_add(services, "frank-secondary")
|
_add(services, "frank-zealot")
|
||||||
if name in {"frank-large.env", "frank-large.service", "serve-frank-large.sh"}:
|
if name in {"frank-dragoon.env", "frank-dragoon.service", "serve-frank-dragoon.sh"}:
|
||||||
# CIRU v4.4.1 frank-large (2026-09-19) keeps its unit, env and
|
# CIRU v4.4.1 frank-dragoon (2026-09-19) keeps its unit, env and
|
||||||
# serve files here the same way: a root unit on a host-built
|
# serve files here the same way: a root unit on a host-built
|
||||||
# runtime. The download script fetches the payload, which is
|
# runtime. The download script fetches the payload, which is
|
||||||
# host-side state, so it selects nothing.
|
# host-side state, so it selects nothing.
|
||||||
_add(services, "frank-large")
|
_add(services, "frank-dragoon")
|
||||||
if name == "qwen-sharp-v22.4.0.jinja":
|
if name == "qwen-sharp-v22.4.0.jinja":
|
||||||
# The Qwen-Sharp chat template frank-large loads via
|
# The Qwen-Sharp chat template frank-dragoon loads via
|
||||||
# --chat-template-file. It lived under dankserver/frank-medium/ until
|
# --chat-template-file. It lived under dankserver/frank-medium/ until
|
||||||
# the 2026-09-26 cutover retired that directory; editing it changes
|
# the 2026-09-26 cutover retired that directory; editing it changes
|
||||||
# what the live frank-large endpoint renders, so it plans
|
# what the live frank-dragoon endpoint renders, so it plans
|
||||||
# frank-large.
|
# frank-dragoon.
|
||||||
_add(services, "frank-large")
|
_add(services, "frank-dragoon")
|
||||||
if name in {"frank-primary.env", "frank-primary.service", "serve-frank-primary.sh"}:
|
if name in {"frank-archon.env", "frank-archon.service", "serve-frank-archon.sh"}:
|
||||||
# llama.cpp frank-primary (2026-09-26, replacing the Lucebox
|
# llama.cpp frank-archon (2026-09-26, replacing the Lucebox
|
||||||
# DS4.1 quadlet) keeps its unit, env and serve files here the
|
# DS4.1 quadlet) keeps its unit, env and serve files here the
|
||||||
# same way: a root unit on the unsloth b11030-mix prebuilt
|
# same way: a root unit on the unsloth b11030-mix prebuilt
|
||||||
# serving Qwen3.8-Flash-Next Q8_0. The download script fetches
|
# serving Qwen3.8-Flash-Next Q8_0. The download script fetches
|
||||||
# the payload, which is host-side state, so it selects nothing.
|
# the payload, which is host-side state, so it selects nothing.
|
||||||
_add(services, "frank-primary")
|
_add(services, "frank-archon")
|
||||||
# Deployment/test/benchmark scripts and other repo tooling do not
|
# Deployment/test/benchmark scripts and other repo tooling do not
|
||||||
# restart a runtime service.
|
# restart a runtime service.
|
||||||
return services
|
return services
|
||||||
|
|||||||
+130
-120
@@ -90,12 +90,12 @@ service_is_allowed() {
|
|||||||
dankclaw:authentik-postgres|dankclaw:authentik-server|\
|
dankclaw:authentik-postgres|dankclaw:authentik-server|\
|
||||||
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
|
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|\
|
||||||
dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
|
dankclaw:caddy|dankclaw:homepage|dankclaw:seerr|dankclaw:marreta|\
|
||||||
dankclaw:frank-large|dankclaw:frank-secondary|\
|
dankclaw:frank-dragoon|dankclaw:frank-zealot|\
|
||||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
||||||
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
|
dankserver:prowlarr|dankserver:qbittorrent|dankserver:radarr|\
|
||||||
dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
|
dankserver:sonarr|dankserver:lidarr|dankserver:readarr|\
|
||||||
dankserver:frank-primary|dankserver:node-exporter|\
|
dankserver:frank-archon|dankserver:node-exporter|\
|
||||||
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
|
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
|
||||||
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
|
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
|
||||||
frankbook-pro:readarr)
|
frankbook-pro:readarr)
|
||||||
@@ -192,7 +192,7 @@ add_pending_services() {
|
|||||||
continue
|
continue
|
||||||
fi
|
fi
|
||||||
[[ -n "$recorded" && "$recorded" != "$REVISION" ]] || continue
|
[[ -n "$recorded" && "$recorded" != "$REVISION" ]] || continue
|
||||||
if [[ "$TARGET" == dankserver && "$service" == frank-large ]]; then continue; fi
|
if [[ "$TARGET" == dankserver && "$service" == frank-dragoon ]]; then continue; fi
|
||||||
if ! changed="$(python3 scripts/deploy-plan.py --target "$TARGET" \
|
if ! changed="$(python3 scripts/deploy-plan.py --target "$TARGET" \
|
||||||
--repo "$REMOTE_DIR" --base "$recorded" --head "$REVISION")"; then
|
--repo "$REMOTE_DIR" --base "$recorded" --head "$REVISION")"; then
|
||||||
echo "WARN: unable to compare state for $service; retaining it as pending" >&2
|
echo "WARN: unable to compare state for $service; retaining it as pending" >&2
|
||||||
@@ -322,6 +322,12 @@ deploy_homepage() {
|
|||||||
if [[ -d shared/homepage/fonts ]]; then
|
if [[ -d shared/homepage/fonts ]]; then
|
||||||
run_privileged cp -r shared/homepage/fonts /srv/homepage/
|
run_privileged cp -r shared/homepage/fonts /srv/homepage/
|
||||||
fi
|
fi
|
||||||
|
if [[ -d shared/homepage/img ]]; then
|
||||||
|
run_privileged cp -r shared/homepage/img /srv/homepage/
|
||||||
|
fi
|
||||||
|
if [[ -d shared/homepage/sfx ]]; then
|
||||||
|
run_privileged cp -r shared/homepage/sfx /srv/homepage/
|
||||||
|
fi
|
||||||
local deployed_at
|
local deployed_at
|
||||||
deployed_at="$(git log -1 --format=%cI)"
|
deployed_at="$(git log -1 --format=%cI)"
|
||||||
printf 'window.DEPLOYED_AT = "%s";\n' "$deployed_at" > /tmp/deployed.js
|
printf 'window.DEPLOYED_AT = "%s";\n' "$deployed_at" > /tmp/deployed.js
|
||||||
@@ -1086,21 +1092,21 @@ deploy_anubis() {
|
|||||||
wait_container_healthy anubis 60 || return 1
|
wait_container_healthy anubis 60 || return 1
|
||||||
}
|
}
|
||||||
|
|
||||||
# frank-large is the halogen-flash-server CONTAINER (0.16.2, digest-pinned
|
# frank-dragoon is the halogen-flash-server CONTAINER (0.16.2, digest-pinned
|
||||||
# through scripts/frank-large.env) serving the Quat3rnion ABLITERATED
|
# through scripts/frank-dragoon.env) serving the Quat3rnion ABLITERATED
|
||||||
# Swift1.5 finetune of peonist's Qwen3.8-Flash-Next v2 checkpoint —
|
# Swift1.5 finetune of peonist's Qwen3.8-Flash-Next v2 checkpoint —
|
||||||
# qwen38-flash-next-v2-swift15-abliterated.hgn (66,687,678,464 B, ~62 GiB
|
# qwen38-flash-next-v2-swift15-abliterated.hgn (66,687,678,464 B, ~62 GiB
|
||||||
# resident) + the REQUIRED n-gram lookup table + the vision tower —
|
# resident) + the REQUIRED n-gram lookup table + the vision tower —
|
||||||
# MTP spec decode (HALOGEN_SPEC_ADAPT=0), 2 slots x 200192, image input —
|
# MTP spec decode (HALOGEN_SPEC_ADAPT=0), 2 slots x 200192, image input —
|
||||||
# on the Strix Halo iGPU. It stays a ROOT unit because the serve script
|
# on the Strix Halo iGPU. It stays a ROOT unit because the serve script
|
||||||
# drives `podman run` directly (the frank-primary pattern): relink_unit
|
# drives `podman run` directly (the frank-archon pattern): relink_unit
|
||||||
# points /etc/systemd/system/frank-large.service at the repo file, boot
|
# points /etc/systemd/system/frank-dragoon.service at the repo file, boot
|
||||||
# persistence is that unit's [Install], and LimitMEMLOCK=infinity is
|
# persistence is that unit's [Install], and LimitMEMLOCK=infinity is
|
||||||
# fleet parity. 2026-10-06 cutover from the halogen 27B-dense shape —
|
# fleet parity. 2026-10-06 cutover from the halogen 27B-dense shape —
|
||||||
# rollback = revert the swap commit's trio; the 27B payload stays on
|
# rollback = revert the swap commit's trio; the 27B payload stays on
|
||||||
# disk (~/models/halogen-qwen3.8-27b).
|
# disk (~/models/halogen-qwen3.8-27b).
|
||||||
#
|
#
|
||||||
# The gates mirror serve-frank-large.sh's predicates EXACTLY (the
|
# The gates mirror serve-frank-dragoon.sh's predicates EXACTLY (the
|
||||||
# 2026-10-01 crash-loop lesson): image digest in dank's podman storage,
|
# 2026-10-01 crash-loop lesson): image digest in dank's podman storage,
|
||||||
# each .hgn payload (checkpoint, n-gram table, vision tower) at the exact
|
# each .hgn payload (checkpoint, n-gram table, vision tower) at the exact
|
||||||
# byte count, the tokenizer directory holding every named file as a
|
# byte count, the tokenizer directory holding every named file as a
|
||||||
@@ -1108,23 +1114,23 @@ deploy_anubis() {
|
|||||||
# artifact would leave a restart-looping unit and a dark endpoint. The
|
# artifact would leave a restart-looping unit and a dark endpoint. The
|
||||||
# port gate is the same as every cutover since 2026-09-16: nothing
|
# port gate is the same as every cutover since 2026-09-16: nothing
|
||||||
# starts until 7770 is released.
|
# starts until 7770 is released.
|
||||||
LLM_LARGE_IMAGE="${LLM_LARGE_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
|
LLM_DRAGOON_IMAGE="${LLM_DRAGOON_IMAGE:-ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a}"
|
||||||
LLM_LARGE_CKPT="${LLM_LARGE_CKPT:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-v2-swift15-abliterated.hgn}"
|
LLM_DRAGOON_CKPT="${LLM_DRAGOON_CKPT:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-v2-swift15-abliterated.hgn}"
|
||||||
LLM_LARGE_CKPT_BYTES="${LLM_LARGE_CKPT_BYTES:-66687678464}"
|
LLM_DRAGOON_CKPT_BYTES="${LLM_DRAGOON_CKPT_BYTES:-66687678464}"
|
||||||
LLM_LARGE_NGRAM="${LLM_LARGE_NGRAM:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-ngram.hgn}"
|
LLM_DRAGOON_NGRAM="${LLM_DRAGOON_NGRAM:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-ngram.hgn}"
|
||||||
LLM_LARGE_NGRAM_BYTES="${LLM_LARGE_NGRAM_BYTES:-51200246144}"
|
LLM_DRAGOON_NGRAM_BYTES="${LLM_DRAGOON_NGRAM_BYTES:-51200246144}"
|
||||||
LLM_LARGE_VISION="${LLM_LARGE_VISION:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-vision.hgn}"
|
LLM_DRAGOON_VISION="${LLM_DRAGOON_VISION:-/home/dank/models/halogen-flashnext-v2/qwen38-flash-next-vision.hgn}"
|
||||||
LLM_LARGE_VISION_BYTES="${LLM_LARGE_VISION_BYTES:-897916416}"
|
LLM_DRAGOON_VISION_BYTES="${LLM_DRAGOON_VISION_BYTES:-897916416}"
|
||||||
LLM_LARGE_TOK_DIR="${LLM_LARGE_TOK_DIR:-/home/dank/models/halogen-flashnext-v2/tokenizer}"
|
LLM_DRAGOON_TOK_DIR="${LLM_DRAGOON_TOK_DIR:-/home/dank/models/halogen-flashnext-v2/tokenizer}"
|
||||||
LLM_LARGE_TOK_FILES="${LLM_LARGE_TOK_FILES:-tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json}"
|
LLM_DRAGOON_TOK_FILES="${LLM_DRAGOON_TOK_FILES:-tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json}"
|
||||||
|
|
||||||
retire_halogen_large_quadlet() {
|
retire_halogen_dragoon_quadlet() {
|
||||||
local trash="$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet"
|
local trash="$HOME/.AGENTTRASH/20260919-orca-cutover/quadlet"
|
||||||
systemctl --user stop frank-large.service 2>/dev/null || true
|
systemctl --user stop frank-dragoon.service 2>/dev/null || true
|
||||||
if [[ -e "$quadlet_unit_dir/frank-large.container" ]]; then
|
if [[ -e "$quadlet_unit_dir/frank-dragoon.container" ]]; then
|
||||||
mkdir -p "$trash"
|
mkdir -p "$trash"
|
||||||
mv -- "$quadlet_unit_dir/frank-large.container" "$trash/frank-large.container"
|
mv -- "$quadlet_unit_dir/frank-dragoon.container" "$trash/frank-dragoon.container"
|
||||||
echo "retired quadlet frank-large.container -> $trash"
|
echo "retired quadlet frank-dragoon.container -> $trash"
|
||||||
fi
|
fi
|
||||||
systemctl --user daemon-reload
|
systemctl --user daemon-reload
|
||||||
}
|
}
|
||||||
@@ -1148,164 +1154,168 @@ wait_port_free() {
|
|||||||
return 1
|
return 1
|
||||||
}
|
}
|
||||||
|
|
||||||
deploy_llm_large() {
|
deploy_llm_dragoon() {
|
||||||
# Refuse rather than restart-loop; every gate below has a twin in
|
# Refuse rather than restart-loop; every gate below has a twin in
|
||||||
# serve-frank-large.sh and they must agree (2026-10-01 lesson).
|
# serve-frank-dragoon.sh and they must agree (2026-10-01 lesson).
|
||||||
# Capture podman output before grepping: `podman images | grep -q`
|
# Capture podman output before grepping: `podman images | grep -q`
|
||||||
# races under pipefail (SIGPIPE 141 becomes "no match", 2026-09-30).
|
# races under pipefail (SIGPIPE 141 becomes "no match", 2026-09-30).
|
||||||
if ! grep -qF "${LLM_LARGE_IMAGE#*@}" <<<"$(podman images --format '{{.RepoDigests}}' 2>/dev/null)"; then
|
if ! grep -qF "${LLM_DRAGOON_IMAGE#*@}" <<<"$(podman images --format '{{.RepoDigests}}' 2>/dev/null)"; then
|
||||||
echo "ERROR: halogen image absent: $LLM_LARGE_IMAGE" >&2
|
echo "ERROR: halogen image absent: $LLM_DRAGOON_IMAGE" >&2
|
||||||
echo "frank-large: pull it: podman pull ${LLM_LARGE_IMAGE%@*}" >&2
|
echo "frank-dragoon: pull it: podman pull ${LLM_DRAGOON_IMAGE%@*}" >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
if [[ ! -f "$LLM_LARGE_CKPT" ]]; then
|
if [[ ! -f "$LLM_DRAGOON_CKPT" ]]; then
|
||||||
echo "ERROR: halogen checkpoint absent at $LLM_LARGE_CKPT" >&2
|
echo "ERROR: halogen checkpoint absent at $LLM_DRAGOON_CKPT" >&2
|
||||||
echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
|
echo "frank-dragoon: fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
if [[ "$(stat -c %s "$LLM_LARGE_CKPT")" != "$LLM_LARGE_CKPT_BYTES" ]]; then
|
if [[ "$(stat -c %s "$LLM_DRAGOON_CKPT")" != "$LLM_DRAGOON_CKPT_BYTES" ]]; then
|
||||||
echo "ERROR: halogen checkpoint is not $LLM_LARGE_CKPT_BYTES bytes (partial download?)" >&2
|
echo "ERROR: halogen checkpoint is not $LLM_DRAGOON_CKPT_BYTES bytes (partial download?)" >&2
|
||||||
echo "frank-large: re-check: bash scripts/download-frank-large-flashnext.sh --verify-only" >&2
|
echo "frank-dragoon: re-check: bash scripts/download-frank-dragoon-flashnext.sh --verify-only" >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
for gate in "$LLM_LARGE_NGRAM:$LLM_LARGE_NGRAM_BYTES:n-gram table" \
|
for gate in "$LLM_DRAGOON_NGRAM:$LLM_DRAGOON_NGRAM_BYTES:n-gram table" \
|
||||||
"$LLM_LARGE_VISION:$LLM_LARGE_VISION_BYTES:vision tower"; do
|
"$LLM_DRAGOON_VISION:$LLM_DRAGOON_VISION_BYTES:vision tower"; do
|
||||||
IFS=: read -r gpath gbytes gname <<<"$gate"
|
IFS=: read -r gpath gbytes gname <<<"$gate"
|
||||||
if [[ ! -f "$gpath" ]]; then
|
if [[ ! -f "$gpath" ]]; then
|
||||||
echo "ERROR: halogen $gname absent at $gpath" >&2
|
echo "ERROR: halogen $gname absent at $gpath" >&2
|
||||||
echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
|
echo "frank-dragoon: fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
if [[ "$(stat -c %s "$gpath")" != "$gbytes" ]]; then
|
if [[ "$(stat -c %s "$gpath")" != "$gbytes" ]]; then
|
||||||
echo "ERROR: halogen $gname is not $gbytes bytes (partial download?)" >&2
|
echo "ERROR: halogen $gname is not $gbytes bytes (partial download?)" >&2
|
||||||
echo "frank-large: re-check: bash scripts/download-frank-large-flashnext.sh --verify-only" >&2
|
echo "frank-dragoon: re-check: bash scripts/download-frank-dragoon-flashnext.sh --verify-only" >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
if [[ -L "$LLM_LARGE_TOK_DIR" || ! -d "$LLM_LARGE_TOK_DIR" ]]; then
|
if [[ -L "$LLM_DRAGOON_TOK_DIR" || ! -d "$LLM_DRAGOON_TOK_DIR" ]]; then
|
||||||
echo "ERROR: tokenizer dir missing or a symlink at $LLM_LARGE_TOK_DIR" >&2
|
echo "ERROR: tokenizer dir missing or a symlink at $LLM_DRAGOON_TOK_DIR" >&2
|
||||||
echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
|
echo "frank-dragoon: fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
for tf in $LLM_LARGE_TOK_FILES; do
|
for tf in $LLM_DRAGOON_TOK_FILES; do
|
||||||
if [[ ! -f "$LLM_LARGE_TOK_DIR/$tf" || -L "$LLM_LARGE_TOK_DIR/$tf" ]]; then
|
if [[ ! -f "$LLM_DRAGOON_TOK_DIR/$tf" || -L "$LLM_DRAGOON_TOK_DIR/$tf" ]]; then
|
||||||
echo "ERROR: tokenizer file missing or a symlink: $LLM_LARGE_TOK_DIR/$tf" >&2
|
echo "ERROR: tokenizer file missing or a symlink: $LLM_DRAGOON_TOK_DIR/$tf" >&2
|
||||||
echo "frank-large: fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
|
echo "frank-dragoon: fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
|
|
||||||
retire_halogen_large_quadlet
|
retire_halogen_dragoon_quadlet
|
||||||
# The predecessor on a re-deploy is THIS unit: stop it (ExecStop stops
|
# The predecessor on a re-deploy is THIS unit: stop it (ExecStop stops
|
||||||
# the container) before waiting on the port. The stray sweep is keyed
|
# the container) before waiting on the port. The stray sweep is keyed
|
||||||
# on the FULL container name, which appears in the argv of a manual
|
# on the FULL container name, which appears in the argv of a manual
|
||||||
# `podman run --name frank-large-halogen` / `podman wait` but NOT in
|
# `podman run --name frank-dragoon-halogen` / `podman wait` but NOT in
|
||||||
# this deployer's own command line (`bash deploy-remote.sh
|
# this deployer's own command line (`bash deploy-remote.sh
|
||||||
# frank-large`) — so it cannot shoot itself in the foot (the fleet
|
# frank-dragoon`) — so it cannot shoot itself in the foot (the fleet
|
||||||
# pkill law), and it can never match frank-secondary's llama-server.
|
# pkill law), and it can never match frank-zealot's llama-server.
|
||||||
run_privileged systemctl stop frank-large.service 2>/dev/null || true
|
run_privileged systemctl stop frank-dragoon.service 2>/dev/null || true
|
||||||
pkill -f 'frank-large-halogen' 2>/dev/null || true
|
pkill -f 'frank-dragoon-halogen' 2>/dev/null || true
|
||||||
podman rm -f frank-large frank-large-halogen >/dev/null 2>&1 || true
|
podman rm -f frank-dragoon frank-dragoon-halogen >/dev/null 2>&1 || true
|
||||||
# Nothing may start until the predecessor has released the port
|
# Nothing may start until the predecessor has released the port
|
||||||
# (2026-09-16: "Deactivated successfully" was not "port free").
|
# (2026-09-16: "Deactivated successfully" was not "port free").
|
||||||
wait_port_free 7770 || return 1
|
wait_port_free 7770 || return 1
|
||||||
relink_unit frank-large
|
relink_unit frank-dragoon
|
||||||
run_privileged systemctl daemon-reload
|
run_privileged systemctl daemon-reload
|
||||||
run_privileged systemctl enable frank-large.service
|
run_privileged systemctl enable frank-dragoon.service
|
||||||
run_privileged systemctl restart frank-large.service
|
run_privileged systemctl restart frank-dragoon.service
|
||||||
wait_for_health frank-large http://127.0.0.1:7770/health 1200
|
wait_for_health frank-dragoon http://127.0.0.1:7770/health 1200
|
||||||
}
|
}
|
||||||
|
|
||||||
# deploy_llm_secondary — frank-secondary is llama.cpp (master build, CUDA sm_86) on
|
# deploy_llm_zealot — frank-zealot is llama.cpp (master build, CUDA sm_86) on
|
||||||
# the RTX 3090 as a ROOT unit: the runtime is a native binary tree under
|
# the RTX 3090 as a ROOT unit: the runtime is a native binary tree under
|
||||||
# ~/GitPackages/llama-master/build-cuda, so there is no container image and
|
# ~/GitPackages/llama-master/build-cuda, so there is no container image and
|
||||||
# no generated unit to avoid. relink_unit points
|
# no generated unit to avoid. relink_unit points
|
||||||
# /etc/systemd/system/frank-secondary.service at the repo file and boot
|
# /etc/systemd/system/frank-zealot.service at the repo file and boot
|
||||||
# persistence is that unit's [Install] section.
|
# persistence is that unit's [Install] section.
|
||||||
#
|
#
|
||||||
# It is a SEPARATE endpoint on port 7771; frank-large keeps 7770. The 3090 is
|
# It is a SEPARATE endpoint on port 7771; frank-dragoon keeps 7770. The 3090 is
|
||||||
# not shared: frank-large's engine runs on the gfx1151 iGPU.
|
# not shared: frank-dragoon's engine runs on the gfx1151 iGPU.
|
||||||
LLM_SECONDARY_MODEL_FILE="${LLM_SECONDARY_MODEL_FILE:-/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf}"
|
LLM_ZEALOT_MODEL_FILE="${LLM_ZEALOT_MODEL_FILE:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
|
||||||
LLM_SECONDARY_RUNTIME="${LLM_SECONDARY_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
|
LLM_ZEALOT_RUNTIME="${LLM_ZEALOT_RUNTIME:-/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
|
||||||
# The projector IS the vision capability here: the GGUF carries no vision
|
# The projector IS the vision capability here: the model is natively
|
||||||
# tensors, and the publisher ships mmproj-BF16.gguf in the same repo, so a
|
# multimodal and the Q6_K file carries no vision tensors — the publisher ships
|
||||||
# deploy that starts without this file silently ships a text-only
|
# its own BF16 projector in the same repo, so a deploy that starts without
|
||||||
# endpoint. Checked unconditionally, like frank-large's.
|
# this file silently ships a text-only endpoint. Checked unconditionally, like
|
||||||
LLM_SECONDARY_MMPROJ="${LLM_SECONDARY_MMPROJ:-/home/dank/models/Swift15-UncIQ3XXS/mmproj-BF16.gguf}"
|
# frank-dragoon's.
|
||||||
|
LLM_ZEALOT_MMPROJ="${LLM_ZEALOT_MMPROJ:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
|
||||||
|
|
||||||
deploy_llm_secondary() {
|
deploy_llm_zealot() {
|
||||||
# Refuse rather than restart-loop: a unit whose runtime or payload is
|
# Refuse rather than restart-loop: a unit whose runtime or payload is
|
||||||
# missing never becomes healthy, and a green deploy would leave the
|
# missing never becomes healthy, and a green deploy would leave the
|
||||||
# endpoint dark with nothing in the log to say why.
|
# endpoint dark with nothing in the log to say why.
|
||||||
if [[ ! -x "$LLM_SECONDARY_RUNTIME" ]]; then
|
if [[ ! -x "$LLM_ZEALOT_RUNTIME" ]]; then
|
||||||
echo "ERROR: llama.cpp runtime absent at $LLM_SECONDARY_RUNTIME" >&2
|
echo "ERROR: llama.cpp runtime absent at $LLM_ZEALOT_RUNTIME" >&2
|
||||||
echo "frank-secondary: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
|
echo "frank-zealot: build it (cmake -B build-cuda -DGGML_CUDA=ON" >&2
|
||||||
echo "frank-secondary: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
|
echo "frank-zealot: -DCMAKE_CUDA_ARCHITECTURES=86 -DCMAKE_CUDA_HOST_COMPILER=/usr/bin/gcc-13" >&2
|
||||||
echo "frank-secondary: && cmake --build build-cuda --target llama-server) first." >&2
|
echo "frank-zealot: && cmake --build build-cuda --target llama-server) first." >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
for f in "$LLM_SECONDARY_MODEL_FILE" "$LLM_SECONDARY_MMPROJ"; do
|
for f in "$LLM_ZEALOT_MODEL_FILE" "$LLM_ZEALOT_MMPROJ"; do
|
||||||
if [[ ! -f "$f" ]]; then
|
if [[ ! -f "$f" ]]; then
|
||||||
echo "ERROR: frank-secondary payload file absent at $f" >&2
|
echo "ERROR: frank-zealot payload file absent at $f" >&2
|
||||||
echo "frank-secondary: run scripts/download-frank-secondary.sh first;" >&2
|
echo "frank-zealot: run scripts/download-frank-zealot.sh first;" >&2
|
||||||
echo "frank-secondary: refusing to start a unit that cannot load." >&2
|
echo "frank-zealot: refusing to start a unit that cannot load." >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
relink_unit frank-secondary
|
relink_unit frank-zealot
|
||||||
run_privileged systemctl daemon-reload
|
run_privileged systemctl daemon-reload
|
||||||
run_privileged systemctl enable frank-secondary.service
|
run_privileged systemctl enable frank-zealot.service
|
||||||
run_privileged systemctl restart frank-secondary.service
|
run_privileged systemctl restart frank-zealot.service
|
||||||
wait_for_health frank-secondary http://127.0.0.1:7771/health 900 || return 1
|
wait_for_health frank-zealot http://127.0.0.1:7771/health 900 || return 1
|
||||||
# False-green hardening (frank-large's lesson): the port answering
|
# False-green hardening (frank-dragoon's lesson): the port answering
|
||||||
# proves nothing about WHO answers — the unit must be active with a
|
# proves nothing about WHO answers — the unit must be active with a
|
||||||
# live MainPID, or a stray SGLang holdover is masquerading as deployed.
|
# live MainPID, or a stray SGLang holdover is masquerading as deployed.
|
||||||
local main_pid
|
local main_pid
|
||||||
main_pid="$(run_privileged systemctl show -p MainPID --value frank-secondary.service)"
|
main_pid="$(run_privileged systemctl show -p MainPID --value frank-zealot.service)"
|
||||||
if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
|
if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
|
||||||
echo "FAIL: :7771 answered but frank-secondary.service has no MainPID — a stray process holds the port" >&2
|
echo "FAIL: :7771 answered but frank-zealot.service has no MainPID — a stray process holds the port" >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
echo "frank-secondary: health answered by unit MainPID $main_pid"
|
echo "frank-zealot: health answered by unit MainPID $main_pid"
|
||||||
}
|
}
|
||||||
|
|
||||||
# deploy_llm_primary — frank-primary is the radiance engine (StillDeadcode,
|
# deploy_llm_archon — frank-archon is the radiance engine (StillDeadcode,
|
||||||
# digest-pinned stilldeadcode/radiance image) serving the OFFICIAL
|
# digest-pinned stilldeadcode/radiance image) serving the PUBLISHER's
|
||||||
# Qwen/Qwen3.8-Flash-Next-FP8 at native block-FP8 precision from the
|
# 4-bit-codebook container of the official Qwen/Qwen3.8-Flash-Next
|
||||||
# rad-convert container qwen4exp-fp8-native.rad, on BOTH R9700s (TP=2) as a
|
# (qwen3.8-next-flash-fp8-iq4r-moe.rad, fetched by
|
||||||
|
# scripts/download-frank-archon-iq4r.sh) on BOTH R9700s (TP=2) as a
|
||||||
# ROOT unit whose serve script drives `podman run` directly (the engine is
|
# ROOT unit whose serve script drives `podman run` directly (the engine is
|
||||||
# the image entrypoint; the image lives in dank's rootless podman storage).
|
# the image entrypoint; the image lives in dank's rootless podman storage).
|
||||||
|
# 2026-10-15 payload swap from the in-house w4dn8 (payload on disk =
|
||||||
|
# rollback).
|
||||||
# 2026-10-05 cutover from FreeToken (pre-cutover tag
|
# 2026-10-05 cutover from FreeToken (pre-cutover tag
|
||||||
# frank-primary-freetoken-pr587-20261005); before that, the tcclaviger vLLM
|
# frank-archon-freetoken-pr587-20261005); before that, the tcclaviger vLLM
|
||||||
# fork (tag frank-primary-vllm-shapeG-20261003). Same port and alias; the
|
# fork (tag frank-archon-vllm-shapeG-20261003). Same port and alias; the
|
||||||
# harness models.yml id is unchanged.
|
# harness models.yml id is unchanged.
|
||||||
#
|
#
|
||||||
# Gates: the image must exist under its digest ref in dank's rootless
|
# Gates: the image must exist under its digest ref in dank's rootless
|
||||||
# storage (FRANK_PRIMARY_IMAGE), the .rad container file must exist at the
|
# storage (FRANK_ARCHON_IMAGE), the .rad container file must exist at the
|
||||||
# env's size floor (FRANK_PRIMARY_MODEL_FILE >= FRANK_PRIMARY_RAD_MIN_MIB —
|
# env's size floor (FRANK_ARCHON_MODEL_FILE >= FRANK_ARCHON_RAD_MIN_MIB —
|
||||||
# a truncated convert is a guaranteed long-timeout boot), and the unit env
|
# a truncated fetch is a guaranteed long-timeout boot), and the unit env
|
||||||
# must be complete. Per-file sha256 verification is the fetch script's job
|
# must be complete. Per-file sha256 verification is the fetch script's job
|
||||||
# at download time and rad-convert/rad-info's job at build time, never
|
# at download time and rad-convert/rad-info's job at build time, never
|
||||||
# re-hashed here.
|
# re-hashed here.
|
||||||
LLM_PRIMARY_CONTAINER=frank-primary
|
LLM_ARCHON_CONTAINER=frank-archon
|
||||||
LLM_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
|
LLM_ARCHON_LEGACY_CONTAINERS="frank-archon-qwen frank-archon-r9v qwen-fp8 glm-probe"
|
||||||
|
|
||||||
deploy_llm_primary() {
|
deploy_llm_archon() {
|
||||||
# Refuse rather than restart-loop: a unit whose image or payload is
|
# Refuse rather than restart-loop: a unit whose image or payload is
|
||||||
# missing never becomes healthy, and a green deploy would leave the
|
# missing never becomes healthy, and a green deploy would leave the
|
||||||
# endpoint dark with nothing in the log to say why.
|
# endpoint dark with nothing in the log to say why.
|
||||||
local env_file="/home/dank/serverAppConfigs/scripts/frank-primary.env"
|
local env_file="/home/dank/serverAppConfigs/scripts/frank-archon.env"
|
||||||
local image model_dir model_file rad_min mib
|
local image model_dir model_file rad_min mib
|
||||||
if [[ ! -f "$env_file" ]]; then
|
if [[ ! -f "$env_file" ]]; then
|
||||||
echo "ERROR: frank-primary env file absent: $env_file" >&2
|
echo "ERROR: frank-archon env file absent: $env_file" >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
image="$(sed -n 's/^FRANK_PRIMARY_IMAGE=//p' "$env_file" | tail -1)"
|
image="$(sed -n 's/^FRANK_ARCHON_IMAGE=//p' "$env_file" | tail -1)"
|
||||||
model_dir="$(sed -n 's/^FRANK_PRIMARY_MODEL_DIR=//p' "$env_file" | tail -1)"
|
model_dir="$(sed -n 's/^FRANK_ARCHON_MODEL_DIR=//p' "$env_file" | tail -1)"
|
||||||
model_file="$(sed -n 's/^FRANK_PRIMARY_MODEL_FILE=//p' "$env_file" | tail -1)"
|
model_file="$(sed -n 's/^FRANK_ARCHON_MODEL_FILE=//p' "$env_file" | tail -1)"
|
||||||
rad_min="$(sed -n 's/^FRANK_PRIMARY_RAD_MIN_MIB=//p' "$env_file" | tail -1)"
|
rad_min="$(sed -n 's/^FRANK_ARCHON_RAD_MIN_MIB=//p' "$env_file" | tail -1)"
|
||||||
if [[ -z "$image" || -z "$model_dir" || -z "$model_file" || -z "$rad_min" ]]; then
|
if [[ -z "$image" || -z "$model_dir" || -z "$model_file" || -z "$rad_min" ]]; then
|
||||||
echo "ERROR: frank-primary.env missing FRANK_PRIMARY_IMAGE/MODEL_DIR/MODEL_FILE/RAD_MIN_MIB" >&2
|
echo "ERROR: frank-archon.env missing FRANK_ARCHON_IMAGE/MODEL_DIR/MODEL_FILE/RAD_MIN_MIB" >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
# The image gate is the digest ref itself: a name@sha256 ref exists in
|
# The image gate is the digest ref itself: a name@sha256 ref exists in
|
||||||
@@ -1313,18 +1323,18 @@ deploy_llm_primary() {
|
|||||||
# pipefail/grep race (the 2026-09-30 lesson).
|
# pipefail/grep race (the 2026-09-30 lesson).
|
||||||
if ! podman image exists "$image"; then
|
if ! podman image exists "$image"; then
|
||||||
echo "ERROR: runtime image absent from storage: $image" >&2
|
echo "ERROR: runtime image absent from storage: $image" >&2
|
||||||
echo "frank-primary: podman pull docker://stilldeadcode/radiance:latest" >&2
|
echo "frank-archon: podman pull docker://stilldeadcode/radiance:latest" >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
if [[ ! -f "$model_dir/$model_file" ]]; then
|
if [[ ! -f "$model_dir/$model_file" ]]; then
|
||||||
echo "ERROR: .rad container missing: $model_dir/$model_file" >&2
|
echo "ERROR: .rad container missing: $model_dir/$model_file" >&2
|
||||||
echo "frank-primary: run scripts/build-frank-primary-rad.sh" >&2
|
echo "frank-archon: run scripts/download-frank-archon-iq4r.sh --fetch" >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
mib=$(( $(stat -c %s "$model_dir/$model_file") / 1048576 ))
|
mib=$(( $(stat -c %s "$model_dir/$model_file") / 1048576 ))
|
||||||
if [[ "$mib" -lt "$rad_min" ]]; then
|
if [[ "$mib" -lt "$rad_min" ]]; then
|
||||||
echo "ERROR: .rad container too small: $model_dir/$model_file is ${mib} MiB, want >= $rad_min" >&2
|
echo "ERROR: .rad container too small: $model_dir/$model_file is ${mib} MiB, want >= $rad_min" >&2
|
||||||
echo "frank-primary: rebuild it (scripts/build-frank-primary-rad.sh --force)" >&2
|
echo "frank-archon: re-fetch it (scripts/download-frank-archon-iq4r.sh --fetch; delete the staged file first if its sha256 failed)" >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
@@ -1336,19 +1346,19 @@ deploy_llm_primary() {
|
|||||||
# The legacy identities (the retired R9V container, the ad hoc
|
# The legacy identities (the retired R9V container, the ad hoc
|
||||||
# experiment container qwen-fp8, the retired glm-probe) hold the same
|
# experiment container qwen-fp8, the retired glm-probe) hold the same
|
||||||
# GPUs and die here too.
|
# GPUs and die here too.
|
||||||
run_privileged systemctl stop frank-primary.service 2>/dev/null || true
|
run_privileged systemctl stop frank-archon.service 2>/dev/null || true
|
||||||
local ctr
|
local ctr
|
||||||
for ctr in $LLM_PRIMARY_LEGACY_CONTAINERS "$LLM_PRIMARY_CONTAINER"; do
|
for ctr in $LLM_ARCHON_LEGACY_CONTAINERS "$LLM_ARCHON_CONTAINER"; do
|
||||||
podman rm -f "$ctr" >/dev/null 2>&1 || true
|
podman rm -f "$ctr" >/dev/null 2>&1 || true
|
||||||
done
|
done
|
||||||
# Nothing starts until the port is released: the lucebox cutover
|
# Nothing starts until the port is released: the lucebox cutover
|
||||||
# (2026-09-26) and the frank-large cutover (2026-09-16) both failed
|
# (2026-09-26) and the frank-dragoon cutover (2026-09-16) both failed
|
||||||
# exactly here when the next engine bound 7770 a second too early.
|
# exactly here when the next engine bound 7770 a second too early.
|
||||||
wait_port_free 7770 || return 1
|
wait_port_free 7770 || return 1
|
||||||
relink_unit frank-primary
|
relink_unit frank-archon
|
||||||
run_privileged systemctl daemon-reload
|
run_privileged systemctl daemon-reload
|
||||||
run_privileged systemctl enable frank-primary.service
|
run_privileged systemctl enable frank-archon.service
|
||||||
run_privileged systemctl restart frank-primary.service
|
run_privileged systemctl restart frank-archon.service
|
||||||
# Cold start: .rad load + PLE pin + expert host-pool upload (the
|
# Cold start: .rad load + PLE pin + expert host-pool upload (the
|
||||||
# publisher's compose healthcheck allows a 15-minute start_period).
|
# publisher's compose healthcheck allows a 15-minute start_period).
|
||||||
# Parity with every predecessor on this port. The HTTP listener binds
|
# Parity with every predecessor on this port. The HTTP listener binds
|
||||||
@@ -1359,14 +1369,14 @@ deploy_llm_primary() {
|
|||||||
local i ok=0
|
local i ok=0
|
||||||
for ((i=1; i<=900; i++)); do
|
for ((i=1; i<=900; i++)); do
|
||||||
if curl -fsS http://127.0.0.1:7770/health 2>/dev/null >/dev/null; then
|
if curl -fsS http://127.0.0.1:7770/health 2>/dev/null >/dev/null; then
|
||||||
echo "frank-primary engine healthy after approximately $((i * 2))s"
|
echo "frank-archon engine healthy after approximately $((i * 2))s"
|
||||||
ok=1
|
ok=1
|
||||||
break
|
break
|
||||||
fi
|
fi
|
||||||
sleep 2
|
sleep 2
|
||||||
done
|
done
|
||||||
if [[ "$ok" != 1 ]]; then
|
if [[ "$ok" != 1 ]]; then
|
||||||
echo "FAIL: frank-primary /health never answered 200 in 1800s" >&2
|
echo "FAIL: frank-archon /health never answered 200 in 1800s" >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
# False-green hardening (the 2026-09-26 lesson): a bare curl against the
|
# False-green hardening (the 2026-09-26 lesson): a bare curl against the
|
||||||
@@ -1374,12 +1384,12 @@ deploy_llm_primary() {
|
|||||||
# it while the unit is stopped. The deploy is only green if the unit
|
# it while the unit is stopped. The deploy is only green if the unit
|
||||||
# itself is active with a live MainPID behind the health answer.
|
# itself is active with a live MainPID behind the health answer.
|
||||||
local main_pid
|
local main_pid
|
||||||
main_pid="$(run_privileged systemctl show -p MainPID --value frank-primary.service)"
|
main_pid="$(run_privileged systemctl show -p MainPID --value frank-archon.service)"
|
||||||
if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
|
if [[ -z "$main_pid" || "$main_pid" == "0" ]]; then
|
||||||
echo "FAIL: :7770 answered but frank-primary.service has no MainPID — a stray process holds the port" >&2
|
echo "FAIL: :7770 answered but frank-archon.service has no MainPID — a stray process holds the port" >&2
|
||||||
return 1
|
return 1
|
||||||
fi
|
fi
|
||||||
echo "frank-primary: health answered by unit MainPID $main_pid"
|
echo "frank-archon: health answered by unit MainPID $main_pid"
|
||||||
}
|
}
|
||||||
|
|
||||||
wait_for_health() {
|
wait_for_health() {
|
||||||
@@ -1566,9 +1576,9 @@ deploy_service() {
|
|||||||
anubis) deploy_anubis ;;
|
anubis) deploy_anubis ;;
|
||||||
seerr) deploy_seerr ;;
|
seerr) deploy_seerr ;;
|
||||||
marreta) deploy_marreta ;;
|
marreta) deploy_marreta ;;
|
||||||
frank-large) deploy_llm_large ;;
|
frank-dragoon) deploy_llm_dragoon ;;
|
||||||
frank-secondary) deploy_llm_secondary ;;
|
frank-zealot) deploy_llm_zealot ;;
|
||||||
frank-primary) deploy_llm_primary ;;
|
frank-archon) deploy_llm_archon ;;
|
||||||
hermes|hermes-dashboard|hermes-webui) deploy_hermes ;;
|
hermes|hermes-dashboard|hermes-webui) deploy_hermes ;;
|
||||||
romm|romm-mariadb|romm-redis|mariadb|redis) deploy_romm_stack "$service" ;;
|
romm|romm-mariadb|romm-redis|mariadb|redis) deploy_romm_stack "$service" ;;
|
||||||
jellyfin) deploy_jellyfin ;;
|
jellyfin) deploy_jellyfin ;;
|
||||||
|
|||||||
+3
-3
@@ -35,7 +35,7 @@ Options:
|
|||||||
Examples:
|
Examples:
|
||||||
./scripts/deploy.sh dankclaw
|
./scripts/deploy.sh dankclaw
|
||||||
./scripts/deploy.sh dankclaw --service searxng
|
./scripts/deploy.sh dankclaw --service searxng
|
||||||
./scripts/deploy.sh dankserver --service frank-primary
|
./scripts/deploy.sh dankserver --service frank-archon
|
||||||
./scripts/deploy.sh dankclaw --dry-run
|
./scripts/deploy.sh dankclaw --dry-run
|
||||||
EOF
|
EOF
|
||||||
}
|
}
|
||||||
@@ -141,12 +141,12 @@ valid_service() {
|
|||||||
dankclaw:alertmanager|\
|
dankclaw:alertmanager|\
|
||||||
dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
|
dankclaw:node-exporter|dankclaw:authentik-postgres|dankclaw:authentik-server|\
|
||||||
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
|
dankclaw:authentik-worker|dankclaw:authentik-outpost|dankclaw:authentik-outpost-ldap|dankclaw:anubis|dankclaw:seerr|\
|
||||||
dankclaw:frank-large|dankclaw:frank-secondary|\
|
dankclaw:frank-dragoon|dankclaw:frank-zealot|\
|
||||||
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
dankserver:romm-mariadb|dankserver:romm-redis|dankserver:romm|\
|
||||||
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
dankserver:copyparty|dankserver:jellyfin|dankserver:calibre-web|dankserver:calibre-server|dankserver:plex|dankserver:tautulli|\
|
||||||
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
|
dankserver:qbittorrent|dankserver:radarr|dankserver:sonarr|\
|
||||||
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
|
dankserver:lidarr|dankserver:readarr|dankserver:node-exporter|\
|
||||||
dankserver:frank-primary|\
|
dankserver:frank-archon|\
|
||||||
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
|
frankbook-pro:gluetun|frankbook-pro:qbittorrent|frankbook-pro:prowlarr|\
|
||||||
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
|
frankbook-pro:radarr|frankbook-pro:sonarr|frankbook-pro:lidarr|\
|
||||||
frankbook-pro:readarr)
|
frankbook-pro:readarr)
|
||||||
|
|||||||
Executable
+117
@@ -0,0 +1,117 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""derive-protected-experts.py — rewrite the recipe's protected-expert section
|
||||||
|
from a calibration recording. RUNS ON DANKSERVER (venv: numpy).
|
||||||
|
|
||||||
|
The law (docs/MOE-W4.md "Protected experts", restated in the recipe):
|
||||||
|
an expert whose share of its layer's down-input energy is at least 3% AND
|
||||||
|
at least three times its share of the routing is kept bf16; the count must
|
||||||
|
be EVEN on each rank's half of the layer (the four-bit GEMM takes the rest
|
||||||
|
as two equal tables by parity) — an odd half gains its next most energetic
|
||||||
|
non-qualifying expert.
|
||||||
|
|
||||||
|
Data: the calibration dir's per-expert down Grams,
|
||||||
|
gram.r<rank>.blk.L.ffn_down_exps.<e>.bin — 32-byte RADGRAM2 header
|
||||||
|
(magic, k, rank, rows, domain) then k*k f32, named with the checkpoint's
|
||||||
|
GLOBAL expert numbering (arch/common/rad_block_moe_fp8.h: ge =
|
||||||
|
expert_base + e). rows (header) is the expert's routing count; the trace
|
||||||
|
of the Gram is its down-input energy. Rank halves: 0-255, 256-511
|
||||||
|
(contiguous expert_base split at TP=2).
|
||||||
|
|
||||||
|
Output: the cast-rule lines (recipe format) on stdout, a report on stderr.
|
||||||
|
|
||||||
|
Usage: derive-protected-experts.py <calib dir> [--min-share 0.03]
|
||||||
|
[--min-route-mult 3.0]
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import glob
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import struct
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
|
NAME_RE = re.compile(r"^gram\.r(\d)\.blk\.(\d+)\.ffn_down_exps\.(\d+)\.bin$")
|
||||||
|
|
||||||
|
|
||||||
|
def read_gram(path):
|
||||||
|
"""(k, rank, rows, trace) from a RADGRAM2 file (full matrix on disk)."""
|
||||||
|
with open(path, "rb") as f:
|
||||||
|
hdr = f.read(32)
|
||||||
|
magic, k, rank, rows, domain = struct.unpack("<8sIIQI", hdr[:28])
|
||||||
|
if magic != b"RADGRAM2":
|
||||||
|
raise SystemExit(f"not a RADGRAM2 file: {path}")
|
||||||
|
n = k * k
|
||||||
|
a = np.frombuffer(f.read(n * 4), dtype=np.float32)
|
||||||
|
if a.size != n:
|
||||||
|
raise SystemExit(f"short gram file: {path}")
|
||||||
|
tr = float(a.reshape(k, k).diagonal().sum())
|
||||||
|
return k, rank, int(rows), tr
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
ap = argparse.ArgumentParser()
|
||||||
|
ap.add_argument("calib_dir")
|
||||||
|
ap.add_argument("--min-share", type=float, default=0.03)
|
||||||
|
ap.add_argument("--min-route-mult", type=float, default=3.0)
|
||||||
|
a = ap.parse_args()
|
||||||
|
|
||||||
|
# layer -> global expert -> (rank, rows, energy)
|
||||||
|
data = {}
|
||||||
|
for path in sorted(glob.glob(os.path.join(
|
||||||
|
a.calib_dir, "gram.r*.ffn_down_exps.*.bin"))):
|
||||||
|
m = NAME_RE.match(os.path.basename(path))
|
||||||
|
if not m:
|
||||||
|
continue
|
||||||
|
rank, layer, expert = int(m.group(1)), int(m.group(2)), int(m.group(3))
|
||||||
|
k, _, rows, tr = read_gram(path)
|
||||||
|
data.setdefault(layer, {})[expert] = (rank, rows, tr)
|
||||||
|
|
||||||
|
if not data:
|
||||||
|
raise SystemExit("no per-expert down grams found")
|
||||||
|
|
||||||
|
lines, report = [], []
|
||||||
|
total = 0
|
||||||
|
for layer in sorted(data):
|
||||||
|
experts = data[layer]
|
||||||
|
tot_e = sum(v[2] for v in experts.values())
|
||||||
|
tot_r = sum(v[1] for v in experts.values())
|
||||||
|
if tot_e <= 0 or tot_r <= 0:
|
||||||
|
report.append(f"blk.{layer}: empty recording, skipped")
|
||||||
|
continue
|
||||||
|
qual, pool = [], []
|
||||||
|
for e, (rank, rows, tr) in experts.items():
|
||||||
|
es, rs = tr / tot_e, rows / max(tot_r, 1)
|
||||||
|
rec = (e, rank, es, rs)
|
||||||
|
if es >= a.min_share and es >= a.min_route_mult * rs:
|
||||||
|
qual.append(rec)
|
||||||
|
elif es > 0:
|
||||||
|
pool.append(rec)
|
||||||
|
chosen = {r[0]: (r[2], r[3]) for r in qual}
|
||||||
|
for half in (0, 1): # rank halves by the contiguous expert_base split
|
||||||
|
n_half = sum(1 for e in chosen if e // 256 == half)
|
||||||
|
if n_half % 2:
|
||||||
|
pool.sort(key=lambda r: -r[2])
|
||||||
|
for cand in pool:
|
||||||
|
if cand[1] == half and cand[0] not in chosen:
|
||||||
|
chosen[cand[0]] = (cand[2], cand[3])
|
||||||
|
report.append(
|
||||||
|
f"blk.{layer}: half {half} odd -> added e{cand[0]} "
|
||||||
|
f"(energy {cand[2]*100:.2f}%)")
|
||||||
|
break
|
||||||
|
for e in sorted(chosen):
|
||||||
|
es, rs = chosen[e]
|
||||||
|
lines.append(f"blk.{layer}.ffn_*_exps.{e}.weight"
|
||||||
|
f" cast dtype=bf16")
|
||||||
|
report.append(f"blk.{layer} e{e:3d} (rank {e // 256}): "
|
||||||
|
f"energy {es*100:5.2f}%, routing {rs*100:5.3f}%, "
|
||||||
|
f"ratio {es/max(rs,1e-12):6.1f}")
|
||||||
|
total += len(chosen)
|
||||||
|
|
||||||
|
print("\n".join(lines))
|
||||||
|
print("\n".join(report), file=sys.stderr)
|
||||||
|
print(f"total protected: {total}", file=sys.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
Executable
+83
@@ -0,0 +1,83 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# download-frank-archon-iq4r.sh — payload fetch/verify for the frank-archon
|
||||||
|
# swap to the PUBLISHER's 4-bit-codebook container:
|
||||||
|
# StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe — the official
|
||||||
|
# Qwen/Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's own
|
||||||
|
# author into ONE .rad: routed experts as 4-bit codes into a 16-level
|
||||||
|
# non-uniform codebook (Walsh–Hadamard rotation, GPTQ + coordinate-descent,
|
||||||
|
# 10M-token calibration), ten bf16-protected experts per layer, int8 trunk +
|
||||||
|
# lm_head, fp8 HC mixtures, the PLE n-gram table at fp8 fixed scale, the MTP
|
||||||
|
# head, and the bf16 vision tower. File:
|
||||||
|
# qwen3.8-next-flash-fp8-iq4r-moe.rad — 121,969,901,568 B (113.6 GiB,
|
||||||
|
# 116,322 MiB), card KL 0.0789 all-tokens / top-1 92.85 % vs bf16.
|
||||||
|
# Repo is UNGATED — no token needed (never put one on the command line).
|
||||||
|
#
|
||||||
|
# bash scripts/download-frank-archon-iq4r.sh --fetch resume/complete
|
||||||
|
# bash scripts/download-frank-archon-iq4r.sh verify only
|
||||||
|
#
|
||||||
|
# Fetch uses the vendored scripts/hfget.sh (curl+jq only): per-file size
|
||||||
|
# checks against the Hub listing, `-C -` resume, disk-fit gate. The side
|
||||||
|
# files (LICENSE, README, the w4nl recipe) stay in the staging dir as
|
||||||
|
# provenance; the serve gate mounts ONLY /home/dank/models/rad, so the
|
||||||
|
# verified .rad alone moves into it.
|
||||||
|
#
|
||||||
|
# Verify (default) is the AGENTS.md law in two parts: the file lands at the
|
||||||
|
# final path with the Hub's EXACT byte count, and its sha256 equals the
|
||||||
|
# Hub's LFS oid. A truncated transfer is a guaranteed long-timeout boot
|
||||||
|
# without this gate.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
REPO="StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe"
|
||||||
|
RAD_NAME="qwen3.8-next-flash-fp8-iq4r-moe.rad"
|
||||||
|
STAGE="/home/dank/models/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe"
|
||||||
|
DEST="/home/dank/models/rad/$RAD_NAME"
|
||||||
|
SIZE=121969901568
|
||||||
|
SHA256="0af5e96244e80c21ac8edca1719dae49b8a201b94a2f3994c3e24026ceaa4d20"
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
mode="verify"
|
||||||
|
[[ "${1:-}" == "--fetch" ]] && mode="fetch"
|
||||||
|
|
||||||
|
if [[ "$mode" == "fetch" ]]; then
|
||||||
|
# HFGET_JOBS=4 is hfget's default (a few streams saturate the link).
|
||||||
|
bash "$SCRIPT_DIR/hfget.sh" "$REPO" "$STAGE" main
|
||||||
|
|
||||||
|
staged="$STAGE/$RAD_NAME"
|
||||||
|
if [[ ! -f "$staged" ]]; then
|
||||||
|
echo "FETCH FAILED: $staged absent after hfget" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
got=$(stat -c %s "$staged")
|
||||||
|
if [[ "$got" != "$SIZE" ]]; then
|
||||||
|
echo "FETCH FAILED: staged .rad is $got B, want $SIZE B" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "staged .rad byte-exact ($SIZE B); hashing (one full read, minutes)"
|
||||||
|
got_sha=$(sha256sum "$staged" | cut -d' ' -f1)
|
||||||
|
if [[ "$got_sha" != "$SHA256" ]]; then
|
||||||
|
echo "FETCH FAILED: sha256 $got_sha, want $SHA256 — DELETE the staged" \
|
||||||
|
"file before re-fetching (curl -C - cannot repair a corrupt body)" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
mkdir -p "$(dirname "$DEST")"
|
||||||
|
mv "$staged" "$DEST"
|
||||||
|
echo "payload placed: $DEST"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "verifying deployed payload: $DEST"
|
||||||
|
if [[ ! -f "$DEST" ]]; then
|
||||||
|
echo " BAD payload absent: $DEST" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
got=$(stat -c %s "$DEST")
|
||||||
|
if [[ "$got" != "$SIZE" ]]; then
|
||||||
|
echo " BAD payload is $got B, want $SIZE B" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "byte-exact ($SIZE B); hashing (one full read, minutes)"
|
||||||
|
got_sha=$(sha256sum "$DEST" | cut -d' ' -f1)
|
||||||
|
if [[ "$got_sha" != "$SHA256" ]]; then
|
||||||
|
echo " BAD sha256 $got_sha, want $SHA256" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "payload verified: $RAD_NAME, size and sha256 match the Hub LFS oid"
|
||||||
Executable
+113
@@ -0,0 +1,113 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# download-frank-archon-swift15.sh — payload fetch/verify for the planned
|
||||||
|
# frank-archon swap to the Swift 1.5 finetune: ukisai/Swift1.5-Qwen3.8-Flash-Next
|
||||||
|
# (BF16 source, 149 files, 360,037,355,038 bytes = 335.3 GiB: 131 safetensors
|
||||||
|
# shards + weight-map index + tokenizer + vision preprocessing configs + chat
|
||||||
|
# template + licenses). Key set verified identical to the official
|
||||||
|
# Qwen/Qwen3.8-Flash-Next base (0 added / 0 missing), so the qwen4exp recipes
|
||||||
|
# apply verbatim; the checkpoint ships BF16 with NO quantization_config.
|
||||||
|
# Repo is UNGATED — no token needed (never put one on the command line).
|
||||||
|
#
|
||||||
|
# bash scripts/download-frank-archon-swift15.sh --fetch resume/complete
|
||||||
|
# bash scripts/download-frank-archon-swift15.sh verify only
|
||||||
|
#
|
||||||
|
# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
|
||||||
|
# listing/resume downloader, curl+jq only): per-file size checks against
|
||||||
|
# the Hub listing, `-C -` resume, pickle-format duplicates skipped.
|
||||||
|
# Verify (default) is the AGENTS.md law in three parts: every listed file
|
||||||
|
# present with the listed size, the shard count matches the weight-map
|
||||||
|
# index, and every shard's sha256 equals the Hub's LFS oid. Verify needs
|
||||||
|
# only the public API.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
REPO="ukisai/Swift1.5-Qwen3.8-Flash-Next"
|
||||||
|
DEST="/home/dank/models/ukisai/Swift1.5-Qwen3.8-Flash-Next"
|
||||||
|
EXPECTED_SHARDS=131
|
||||||
|
|
||||||
|
mode="verify"
|
||||||
|
[[ "${1:-}" == "--fetch" ]] && mode="fetch"
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
|
||||||
|
if [[ "$mode" == "fetch" ]]; then
|
||||||
|
# HFGET_JOBS=4 is hfget's default (a few streams saturate the link).
|
||||||
|
bash "$SCRIPT_DIR/hfget.sh" "$REPO" "$DEST" main
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "verifying local payload: $DEST"
|
||||||
|
fail=0
|
||||||
|
|
||||||
|
INDEX="$DEST/model.safetensors.index.json"
|
||||||
|
if [[ ! -f "$INDEX" ]]; then
|
||||||
|
echo " BAD weight-map index missing: $INDEX" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
python3 - "$REPO" "$DEST" <<'PY' || fail=1
|
||||||
|
import hashlib, json, os, sys, urllib.request
|
||||||
|
|
||||||
|
repo, dest = sys.argv[1], sys.argv[2]
|
||||||
|
|
||||||
|
def listing():
|
||||||
|
"""The tree API, following the Link cursor exactly like hfget.sh."""
|
||||||
|
out, api, pages = [], (
|
||||||
|
f"https://huggingface.co/api/models/{repo}/tree/main"
|
||||||
|
"?recursive=true&expand=true"), 0
|
||||||
|
while api:
|
||||||
|
with urllib.request.urlopen(api) as r:
|
||||||
|
out += json.load(r)
|
||||||
|
api = None
|
||||||
|
link = r.headers.get("Link", "")
|
||||||
|
for part in link.split(","):
|
||||||
|
if 'rel="next"' in part:
|
||||||
|
api = part[part.find("<") + 1:part.find(">")]
|
||||||
|
pages += 1
|
||||||
|
if pages > 200:
|
||||||
|
raise SystemExit("listing did not terminate")
|
||||||
|
return out
|
||||||
|
|
||||||
|
files = [e for e in listing() if e.get("type") == "file"]
|
||||||
|
sizes = {e["path"]: e.get("size", 0) for e in files}
|
||||||
|
lfs = {e["path"]: e["lfs"]["oid"] for e in files if e.get("lfs")}
|
||||||
|
|
||||||
|
bad = 0
|
||||||
|
for path, want in sorted(sizes.items()):
|
||||||
|
p = os.path.join(dest, path)
|
||||||
|
if not os.path.isfile(p):
|
||||||
|
print(f" BAD {path}: missing")
|
||||||
|
bad += 1
|
||||||
|
elif os.path.getsize(p) != want:
|
||||||
|
print(f" BAD {path}: {os.path.getsize(p)} bytes, listing says {want}")
|
||||||
|
bad += 1
|
||||||
|
print(f" {'OK' if bad == 0 else 'FAIL'}: {len(sizes) - bad}/{len(sizes)} files present at listed size")
|
||||||
|
if bad:
|
||||||
|
raise SystemExit(1)
|
||||||
|
|
||||||
|
# Shard set against the weight-map index.
|
||||||
|
wm = json.load(open(os.path.join(dest, "model.safetensors.index.json")))["weight_map"]
|
||||||
|
shards = sorted(set(wm.values()))
|
||||||
|
print(f" index maps {len(wm)} tensors across {len(shards)} shards")
|
||||||
|
if len(shards) != 131:
|
||||||
|
print(f" BAD shard count {len(shards)}, want 131")
|
||||||
|
raise SystemExit(1)
|
||||||
|
|
||||||
|
# Content: every LFS-tracked file hashed against the Hub oid. The safetensors
|
||||||
|
# shards dominate the bytes; tokenizer/config files ride the same check when
|
||||||
|
# the Hub tracks them in LFS (they are plain blobs here, already size-checked).
|
||||||
|
todo = [p for p in shards if p in lfs]
|
||||||
|
for i, path in enumerate(todo):
|
||||||
|
h = hashlib.sha256()
|
||||||
|
with open(os.path.join(dest, path), "rb") as f:
|
||||||
|
for chunk in iter(lambda: f.read(1 << 24), b""):
|
||||||
|
h.update(chunk)
|
||||||
|
if h.hexdigest() != lfs[path]:
|
||||||
|
print(f" BAD {path}: sha256 mismatch (want {lfs[path]})")
|
||||||
|
bad += 1
|
||||||
|
if (i + 1) % 16 == 0:
|
||||||
|
print(f" ...hashed {i + 1}/{len(todo)}")
|
||||||
|
print(f" {'OK' if bad == 0 else 'FAIL'}: {len(todo) - bad}/{len(todo)} shards sha256-match the Hub")
|
||||||
|
if bad:
|
||||||
|
raise SystemExit(1)
|
||||||
|
PY
|
||||||
|
|
||||||
|
[[ "$fail" -eq 0 ]] || exit 1
|
||||||
|
echo "payload verified: 131 shards + index + tokenizer + configs, sizes and sha256 checked"
|
||||||
@@ -1,13 +1,13 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# download-frank-primary.sh — payload fetch/verify for frank-primary:
|
# download-frank-archon.sh — payload fetch/verify for frank-archon:
|
||||||
# the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint (131 safetensors
|
# the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint (131 safetensors
|
||||||
# shards + weight-map index + tokenizer + vision preprocessing configs,
|
# shards + weight-map index + tokenizer + vision preprocessing configs,
|
||||||
# ~172.8 GB). 2026-10-05: replaces the orcarouter uncensored payload of the
|
# ~172.8 GB). 2026-10-05: replaces the orcarouter uncensored payload of the
|
||||||
# FreeToken era (that script lived here; its repo was gated and needed
|
# FreeToken era (that script lived here; its repo was gated and needed
|
||||||
# HF_TOKEN — this one is UNGATED and needs no token at all).
|
# HF_TOKEN — this one is UNGATED and needs no token at all).
|
||||||
#
|
#
|
||||||
# bash scripts/download-frank-primary.sh --fetch resume/complete
|
# bash scripts/download-frank-archon.sh --fetch resume/complete
|
||||||
# bash scripts/download-frank-primary.sh verify only
|
# bash scripts/download-frank-archon.sh verify only
|
||||||
#
|
#
|
||||||
# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
|
# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
|
||||||
# listing/resume downloader, curl+jq only): per-file size checks against
|
# listing/resume downloader, curl+jq only): per-file size checks against
|
||||||
+2
-2
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# download-frank-large-flashnext.sh — fetch frank-large's flash-server v2
|
# download-frank-dragoon-flashnext.sh — fetch frank-dragoon's flash-server v2
|
||||||
# payload: the Quat3rnion abliterated Swift1.5 checkpoint of peonist's v2
|
# payload: the Quat3rnion abliterated Swift1.5 checkpoint of peonist's v2
|
||||||
# .hgn, the REQUIRED lookup table and the optional vision tower from
|
# .hgn, the REQUIRED lookup table and the optional vision tower from
|
||||||
# peonist-ai/halogen-qwen3.8-flash-next (both at the revisions the model
|
# peonist-ai/halogen-qwen3.8-flash-next (both at the revisions the model
|
||||||
@@ -29,7 +29,7 @@
|
|||||||
# fetched fresh rather than symlinked, per the no-symlink law.
|
# fetched fresh rather than symlinked, per the no-symlink law.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
DEST="${FRANK_LARGE_MODEL_DIR:-$HOME/models/halogen-flashnext-v2}"
|
DEST="${FRANK_DRAGOON_MODEL_DIR:-$HOME/models/halogen-flashnext-v2}"
|
||||||
ABLIT="https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated/resolve/main"
|
ABLIT="https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated/resolve/main"
|
||||||
PEONIST="https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next/resolve"
|
PEONIST="https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next/resolve"
|
||||||
NGRAM_REV="5cc17cea1a10b502b4a5db41d8d2b1a0f22dba26"
|
NGRAM_REV="5cc17cea1a10b502b4a5db41d8d2b1a0f22dba26"
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# download-frank-large-halogen.sh — fetch frank-large's halogen payload:
|
# download-frank-dragoon-halogen.sh — fetch frank-dragoon's halogen payload:
|
||||||
# the qwen3.8-27b .hgn checkpoint and the export tokenizer, from the
|
# the qwen3.8-27b .hgn checkpoint and the export tokenizer, from the
|
||||||
# PUBLIC peonist-ai/halogen-qwen3.8-27b repo (no token — and per the fleet
|
# PUBLIC peonist-ai/halogen-qwen3.8-27b repo (no token — and per the fleet
|
||||||
# secret law, none would ever touch argv).
|
# secret law, none would ever touch argv).
|
||||||
@@ -10,7 +10,7 @@
|
|||||||
# existing payload without downloading.
|
# existing payload without downloading.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
DEST="${FRANK_LARGE_MODEL_DIR:-$HOME/models/halogen-qwen3.8-27b}"
|
DEST="${FRANK_DRAGOON_MODEL_DIR:-$HOME/models/halogen-qwen3.8-27b}"
|
||||||
BASE="https://huggingface.co/peonist-ai/halogen-qwen3.8-27b/resolve/main"
|
BASE="https://huggingface.co/peonist-ai/halogen-qwen3.8-27b/resolve/main"
|
||||||
|
|
||||||
FILES=(qwen3.8-27b-p1w4d-d2.hgn
|
FILES=(qwen3.8-27b-p1w4d-d2.hgn
|
||||||
+3
-3
@@ -1,6 +1,6 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Download kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF for
|
# Download kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF for
|
||||||
# frank-large (Dankclaw iGPU). Public repo — no auth header, and never a token
|
# frank-dragoon (Dankclaw iGPU). Public repo — no auth header, and never a token
|
||||||
# on the command line (it is world-readable in /proc/<pid>/cmdline).
|
# on the command line (it is world-readable in /proc/<pid>/cmdline).
|
||||||
#
|
#
|
||||||
# The payload is the repo's Q6 AGENT rung:
|
# The payload is the repo's Q6 AGENT rung:
|
||||||
@@ -10,7 +10,7 @@
|
|||||||
# names for MTP serving ("Take Q6 AGENT (114) if you want protected
|
# names for MTP serving ("Take Q6 AGENT (114) if you want protected
|
||||||
# heads"). Like every ROCmFPX file it loads on nothing but the fork.
|
# heads"). Like every ROCmFPX file it loads on nothing but the fork.
|
||||||
#
|
#
|
||||||
# The other two files frank-large needs are NOT re-downloaded:
|
# The other two files frank-dragoon needs are NOT re-downloaded:
|
||||||
# * mtp-Qwen3.8-27B-Q4_0.gguf — the uncensored repo redistributes the
|
# * mtp-Qwen3.8-27B-Q4_0.gguf — the uncensored repo redistributes the
|
||||||
# ALIGNED repo's draft head byte for byte (identical Hub LFS oid
|
# ALIGNED repo's draft head byte for byte (identical Hub LFS oid
|
||||||
# 051a1764…), so it is hardlinked out of the aligned payload dir when that
|
# 051a1764…), so it is hardlinked out of the aligned payload dir when that
|
||||||
@@ -25,7 +25,7 @@
|
|||||||
# byte count and the Hub LFS sha256. Idempotent: complete files are skipped.
|
# byte count and the Hub LFS sha256. Idempotent: complete files are skipped.
|
||||||
#
|
#
|
||||||
# Usage:
|
# Usage:
|
||||||
# scripts/download-frank-large-kingjones-uncensored.sh [DEST_DIR] [--verify-only]
|
# scripts/download-frank-dragoon-kingjones-uncensored.sh [DEST_DIR] [--verify-only]
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
REPO="kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF"
|
REPO="kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF"
|
||||||
+2
-2
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Download kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF for frank-large
|
# Download kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF for frank-dragoon
|
||||||
# (Dankclaw iGPU). Public repo — no auth header, and never a token on the
|
# (Dankclaw iGPU). Public repo — no auth header, and never a token on the
|
||||||
# command line (it is world-readable in /proc/<pid>/cmdline).
|
# command line (it is world-readable in /proc/<pid>/cmdline).
|
||||||
#
|
#
|
||||||
@@ -24,7 +24,7 @@
|
|||||||
# come from the Hub LFS oids (api/models/<repo>/tree/main).
|
# come from the Hub LFS oids (api/models/<repo>/tree/main).
|
||||||
#
|
#
|
||||||
# Usage:
|
# Usage:
|
||||||
# scripts/download-frank-large-kingjones.sh [DEST_DIR] [--verify-only]
|
# scripts/download-frank-dragoon-kingjones.sh [DEST_DIR] [--verify-only]
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
REPO="kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF"
|
REPO="kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF"
|
||||||
@@ -1,9 +1,9 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Download and assemble jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca (the frank-large
|
# Download and assemble jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca (the frank-dragoon
|
||||||
# model) on Dankclaw.
|
# model) on Dankclaw.
|
||||||
#
|
#
|
||||||
# bash scripts/download-frank-large-orca.sh
|
# bash scripts/download-frank-dragoon-orca.sh
|
||||||
# bash scripts/download-frank-large-orca.sh --verify-only
|
# bash scripts/download-frank-dragoon-orca.sh --verify-only
|
||||||
#
|
#
|
||||||
# The package is 136 GB of Hub-sized transport parts (HF caps new uploads at
|
# The package is 136 GB of Hub-sized transport parts (HF caps new uploads at
|
||||||
# 50 GB, so the 79 GB main GGUF, the 52 GB PLE payload and the 4 GB MTP head
|
# 50 GB, so the 79 GB main GGUF, the 52 GB PLE payload and the 4 GB MTP head
|
||||||
@@ -105,4 +105,4 @@ fi
|
|||||||
echo "verifying assembled files against SHA256SUMS"
|
echo "verifying assembled files against SHA256SUMS"
|
||||||
(cd "$MODEL_DIR" && sha256sum -c SHA256SUMS)
|
(cd "$MODEL_DIR" && sha256sum -c SHA256SUMS)
|
||||||
(cd "$MODEL_DIR/vision" && sha256sum -c SHA256SUMS)
|
(cd "$MODEL_DIR/vision" && sha256sum -c SHA256SUMS)
|
||||||
echo "frank-large payload complete: $MODEL_DIR"
|
echo "frank-dragoon payload complete: $MODEL_DIR"
|
||||||
@@ -1,9 +1,9 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Download the Q4_K_XL frank-large payload on Dankclaw:
|
# Download the Q4_K_XL frank-dragoon payload on Dankclaw:
|
||||||
# unsloth/Qwen3.8-27B-GGUF -> Qwen3.8-27B-UD-Q4_K_XL.gguf
|
# unsloth/Qwen3.8-27B-GGUF -> Qwen3.8-27B-UD-Q4_K_XL.gguf
|
||||||
#
|
#
|
||||||
# bash scripts/download-frank-large-q4.sh
|
# bash scripts/download-frank-dragoon-q4.sh
|
||||||
# bash scripts/download-frank-large-q4.sh --verify-only
|
# bash scripts/download-frank-dragoon-q4.sh --verify-only
|
||||||
#
|
#
|
||||||
# Integrity: the file is checked against its declared byte count and sha256
|
# Integrity: the file is checked against its declared byte count and sha256
|
||||||
# (the LFS oid from the Hub tree API, 2026-10-01). NO AUTH HEADER IS USED:
|
# (the LFS oid from the Hub tree API, 2026-10-01). NO AUTH HEADER IS USED:
|
||||||
@@ -21,12 +21,12 @@
|
|||||||
# x 2 (K,V) x 2 B), so 4 sessions x 163840 ctx = 640,000 tokens = 39.06 GiB.
|
# x 2 (K,V) x 2 B), so 4 sessions x 163840 ctx = 640,000 tokens = 39.06 GiB.
|
||||||
#
|
#
|
||||||
# The speculative drafter is UNCHANGED and still fetched by its own script
|
# The speculative drafter is UNCHANGED and still fetched by its own script
|
||||||
# (scripts/download-frank-large.sh, z-lab's Hub repo). That name must not
|
# (scripts/download-frank-dragoon.sh, z-lab's Hub repo). That name must not
|
||||||
# appear here at all — the frank-large seam greps THIS file for it to prove
|
# appear here at all — the frank-dragoon seam greps THIS file for it to prove
|
||||||
# the drafter was not folded into the payload fetch.
|
# the drafter was not folded into the payload fetch.
|
||||||
#
|
#
|
||||||
# NOTE ON MTP: this payload has no baked-in MTP head (the Hub repo ships one
|
# NOTE ON MTP: this payload has no baked-in MTP head (the Hub repo ships one
|
||||||
# separately as MTP/mtp-Qwen3.8-27B-Q4_0.gguf). frank-large does not need it —
|
# separately as MTP/mtp-Qwen3.8-27B-Q4_0.gguf). frank-dragoon does not need it —
|
||||||
# spec decode here is the block-diffusion drafter, not MTP.
|
# spec decode here is the block-diffusion drafter, not MTP.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
@@ -1,10 +1,10 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Download the Swift-1.5 frank-large candidate on Dankclaw:
|
# Download the Swift-1.5 frank-dragoon candidate on Dankclaw:
|
||||||
# ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF
|
# ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF
|
||||||
# Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
|
# Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf
|
||||||
#
|
#
|
||||||
# bash scripts/download-frank-large-swift.sh
|
# bash scripts/download-frank-dragoon-swift.sh
|
||||||
# bash scripts/download-frank-large-swift.sh --verify-only
|
# bash scripts/download-frank-dragoon-swift.sh --verify-only
|
||||||
#
|
#
|
||||||
# Integrity: the file is checked against its declared byte count and sha256
|
# Integrity: the file is checked against its declared byte count and sha256
|
||||||
# (the LFS oid from the Hub, 2026-09-28). NO AUTH HEADER IS USED: the repo is
|
# (the LFS oid from the Hub, 2026-09-28). NO AUTH HEADER IS USED: the repo is
|
||||||
@@ -15,9 +15,9 @@
|
|||||||
# Layout after a successful run (MODEL_DIR):
|
# Layout after a successful run (MODEL_DIR):
|
||||||
# Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf 31457990272
|
# Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-UD-Q8_K_XL.gguf 31457990272
|
||||||
#
|
#
|
||||||
# This is the CANDIDATE payload, not the deployed one. frank-large still
|
# This is the CANDIDATE payload, not the deployed one. frank-dragoon still
|
||||||
# serves scripts/download-frank-large.sh's output; do not repoint
|
# serves scripts/download-frank-dragoon.sh's output; do not repoint
|
||||||
# frank-large.env at MODEL_DIR until the gufo load test passes.
|
# frank-dragoon.env at MODEL_DIR until the gufo load test passes.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
MODEL_DIR="/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL"
|
MODEL_DIR="/home/dank/models/Swift-1.5-Qwen3.8-27B-Uncensored-MTP-UD-Q8_K_XL"
|
||||||
@@ -1,9 +1,9 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Download unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (the
|
# Download unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (the
|
||||||
# frank-large model) on Dankclaw.
|
# frank-dragoon model) on Dankclaw.
|
||||||
#
|
#
|
||||||
# bash scripts/download-frank-large-unsloth.sh
|
# bash scripts/download-frank-dragoon-unsloth.sh
|
||||||
# bash scripts/download-frank-large-unsloth.sh --verify-only
|
# bash scripts/download-frank-dragoon-unsloth.sh --verify-only
|
||||||
#
|
#
|
||||||
# The UD-IQ4_XS GGUF ships as three llama.cpp split parts (HF caps uploads at
|
# The UD-IQ4_XS GGUF ships as three llama.cpp split parts (HF caps uploads at
|
||||||
# 50 GB). The parts ARE the canonical payload: the serve script points -m at
|
# 50 GB). The parts ARE the canonical payload: the serve script points -m at
|
||||||
@@ -112,4 +112,4 @@ if [[ "$mode" == "download" ]]; then
|
|||||||
[[ -e "$MODEL_DIR/vision/mmproj-F16.gguf" ]] || mv "$MODEL_DIR/mmproj-F16.gguf" "$MODEL_DIR/vision/"
|
[[ -e "$MODEL_DIR/vision/mmproj-F16.gguf" ]] || mv "$MODEL_DIR/mmproj-F16.gguf" "$MODEL_DIR/vision/"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
echo "frank-large payload complete: $MODEL_DIR"
|
echo "frank-dragoon payload complete: $MODEL_DIR"
|
||||||
@@ -1,10 +1,10 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# Download the frank-large payload on Dankclaw:
|
# Download the frank-dragoon payload on Dankclaw:
|
||||||
# - unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (the 27B dense main GGUF)
|
# - unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (the 27B dense main GGUF)
|
||||||
# - z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M (the DFlash2 drafter)
|
# - z-lab/Qwen3.8-27B-DFlash2-GGUF:Q4_K_M (the DFlash2 drafter)
|
||||||
#
|
#
|
||||||
# bash scripts/download-frank-large.sh
|
# bash scripts/download-frank-dragoon.sh
|
||||||
# bash scripts/download-frank-large.sh --verify-only
|
# bash scripts/download-frank-dragoon.sh --verify-only
|
||||||
#
|
#
|
||||||
# Integrity: every file is checked against its declared byte count and sha256
|
# Integrity: every file is checked against its declared byte count and sha256
|
||||||
# (the values below are the LFS oids from the Hub, 2026-09-21). An HF token
|
# (the values below are the LFS oids from the Hub, 2026-09-21). An HF token
|
||||||
@@ -86,4 +86,4 @@ if [[ "$mode" == "download" ]]; then
|
|||||||
done
|
done
|
||||||
fi
|
fi
|
||||||
|
|
||||||
echo "frank-large payload complete: $MODEL_DIR"
|
echo "frank-dragoon payload complete: $MODEL_DIR"
|
||||||
@@ -1,88 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
# Download the frank-secondary payload for llama.cpp on the RTX 3090:
|
|
||||||
# ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF:RCO-IQ3_XXS
|
|
||||||
# **with the MTP head in the file** (the -mtp variant), plus the
|
|
||||||
# PUBLISHER's OWN vision projector (mmproj-BF16.gguf, same repo). Public
|
|
||||||
# repo — no auth header, and never a token on the command line (it is
|
|
||||||
# world-readable in /proc/<pid>/cmdline).
|
|
||||||
#
|
|
||||||
# Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized
|
|
||||||
# file on a 31 GiB payload); -f/--fail-with-body so an HTTP error is not a
|
|
||||||
# success, then byte count and sha256. This repo is on Xet storage, so
|
|
||||||
# the Hub tree/blob APIs expose no LFS oid: the sha256 values below were
|
|
||||||
# computed locally after the first verified fetch (byte counts match the
|
|
||||||
# Hub's declared sizes). Idempotent: complete files are skipped. The
|
|
||||||
# projector is stored under a self-documenting dest name; the URL name
|
|
||||||
# and sha stay pinned in the table below.
|
|
||||||
#
|
|
||||||
# 2026-10-06 (Frank, evening): the payload moved to the UNCELERSED
|
|
||||||
# Swift-1.5-Qwen3.8-27B at RCO-IQ3_XXS (1.56 GiB lighter than the
|
|
||||||
# IQ3_S-mtp it replaced; the lighter quant funds the MTP draft ctx plus
|
|
||||||
# the second 131,072 slot). Rollback payload stays on disk:
|
|
||||||
# /home/dank/models/Swift15-IQ3S/ (the IQ3_S-mtp that was live until this
|
|
||||||
# swap); earlier payloads Ornith-Q8/ and MiMo-Q6K/ as well. Their
|
|
||||||
# re-fetch recipes are in git history (predecessors of this script).
|
|
||||||
#
|
|
||||||
# Usage:
|
|
||||||
# scripts/download-frank-secondary.sh [DEST_DIR] [--verify-only]
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
DEST=""
|
|
||||||
VERIFY_ONLY=0
|
|
||||||
for arg in "$@"; do
|
|
||||||
case "$arg" in
|
|
||||||
--verify-only) VERIFY_ONLY=1 ;;
|
|
||||||
*) DEST="$arg" ;;
|
|
||||||
esac
|
|
||||||
done
|
|
||||||
DEST="${DEST:-/home/dank/models/Swift15-UncIQ3XXS}"
|
|
||||||
|
|
||||||
# repo | hub-name | dest-name | bytes | sha256 (computed locally; Xet repo)
|
|
||||||
FILES=(
|
|
||||||
"ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF|RCO/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf|Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf|10442827392|e13d3e7400b6c7f980ebd30c0edd6be43eb1865b50be8562a9db042706c7d803"
|
|
||||||
"ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF|mmproj-BF16.gguf|mmproj-BF16.gguf|931145888|19acd7fcf4ee504328566eddf0f8e603ef853a205b4830209d2a4908213abdea"
|
|
||||||
)
|
|
||||||
|
|
||||||
mkdir -p "$DEST"
|
|
||||||
status=0
|
|
||||||
for entry in "${FILES[@]}"; do
|
|
||||||
IFS='|' read -r repo hubname destname size sha <<<"$entry"
|
|
||||||
url="https://huggingface.co/${repo}/resolve/main/${hubname}"
|
|
||||||
f="$DEST/$destname"
|
|
||||||
have=0
|
|
||||||
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
|
|
||||||
if sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
|
||||||
have=1
|
|
||||||
fi
|
|
||||||
fi
|
|
||||||
if (( have )); then
|
|
||||||
echo "OK $destname"
|
|
||||||
continue
|
|
||||||
fi
|
|
||||||
if (( VERIFY_ONLY )); then
|
|
||||||
echo "MISSING $destname"
|
|
||||||
status=1
|
|
||||||
continue
|
|
||||||
fi
|
|
||||||
echo "FETCHING $destname (from $repo)"
|
|
||||||
tmp="$f.part"
|
|
||||||
curl -fsSL --fail-with-body -o "$tmp" "$url"
|
|
||||||
mv "$tmp" "$f"
|
|
||||||
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
|
|
||||||
echo "SIZE FAIL $destname ($(stat -c%s "$f") != $size)"
|
|
||||||
status=1
|
|
||||||
continue
|
|
||||||
fi
|
|
||||||
if ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
|
||||||
echo "SHA FAIL $destname"
|
|
||||||
status=1
|
|
||||||
continue
|
|
||||||
fi
|
|
||||||
echo "VERIFIED $destname"
|
|
||||||
done
|
|
||||||
if (( status == 0 )); then
|
|
||||||
echo "DOWNLOAD_DONE $DEST"
|
|
||||||
else
|
|
||||||
echo "DOWNLOAD_FAILED $DEST" >&2
|
|
||||||
fi
|
|
||||||
exit "$status"
|
|
||||||
Executable
+90
@@ -0,0 +1,90 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Download the frank-zealot payload for llama.cpp on the RTX 3090:
|
||||||
|
# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at **Q6_K** plus the
|
||||||
|
# PUBLISHER's OWN vision projector (the BF16 mmproj — the model is natively
|
||||||
|
# multimodal, so this file is the endpoint's eyes). Public repo — no auth
|
||||||
|
# header, and never a token on the command line (it is world-readable in
|
||||||
|
# /proc/<pid>/cmdline).
|
||||||
|
#
|
||||||
|
# Fresh downloads, no resume (`curl -C -` once produced a 213 MB oversized
|
||||||
|
# file on a 31 GiB payload); -f/--fail-with-body so an HTTP error is not a
|
||||||
|
# success, then byte count and sha256. The Hub exposes no LFS oid for this
|
||||||
|
# repo (Xet storage), so the sha256 values below were computed locally after
|
||||||
|
# the first verified fetch — byte counts match the Hub's declared sizes.
|
||||||
|
# A sha of "-" means "first fetch: size-verified, sha printed for pinning".
|
||||||
|
# Idempotent: complete files are skipped.
|
||||||
|
#
|
||||||
|
# 2026-10-07 (Frank): the payload moved to Qwen3.5-9B-Uncensored Q6_K at
|
||||||
|
# 2 x 200,704 (qwen35 hybrid: 32 layers, 8 full-attention + 24 Gated
|
||||||
|
# DeltaNet, no MTP tensors in this file). Rollback payloads stay on disk:
|
||||||
|
# /home/dank/models/Gemma4-26B-A4B/ (Q4_K_M, the 2026-10-07 era being
|
||||||
|
# replaced), Swift15-UncIQ3XXS/, MiMo-Q6K/, MiMo-Q5KM/, Ornith-Q8/.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# scripts/download-frank-zealot.sh [DEST_DIR] [--verify-only]
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
DEST=""
|
||||||
|
VERIFY_ONLY=0
|
||||||
|
for arg in "$@"; do
|
||||||
|
case "$arg" in
|
||||||
|
--verify-only) VERIFY_ONLY=1 ;;
|
||||||
|
*) DEST="$arg" ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
DEST="${DEST:-/home/dank/models/Qwen35-9B-HauhauCS-Q6K}"
|
||||||
|
|
||||||
|
# repo | hub-name | dest-name | bytes | sha256 (computed locally; Xet repo)
|
||||||
|
FILES=(
|
||||||
|
"HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive|Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf|Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf|7359259008|c0ba7beb68fd3fe47891bd549486d38dcf62d00817296ea314ad37017f5a4986"
|
||||||
|
"HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive|mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf|mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf|921704448|05f662501f8bd45607b079723a3e238a4e888fd085a10a53f4057a0e250f6934"
|
||||||
|
)
|
||||||
|
|
||||||
|
status=0
|
||||||
|
for entry in "${FILES[@]}"; do
|
||||||
|
IFS='|' read -r repo hubname destname size sha <<<"$entry"
|
||||||
|
url="https://huggingface.co/${repo}/resolve/main/${hubname}"
|
||||||
|
f="$DEST/$destname"
|
||||||
|
have=0
|
||||||
|
if [[ -f "$f" ]] && [[ "$(stat -c%s "$f")" == "$size" ]]; then
|
||||||
|
if [[ "$sha" == "-" ]]; then
|
||||||
|
have=1
|
||||||
|
elif sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||||
|
have=1
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
if (( have )); then
|
||||||
|
echo "OK $destname"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
if (( VERIFY_ONLY )); then
|
||||||
|
echo "MISSING $destname"
|
||||||
|
status=1
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
echo "FETCHING $destname (from $repo)"
|
||||||
|
tmp="$f.part"
|
||||||
|
curl -fsSL --fail-with-body -o "$tmp" "$url"
|
||||||
|
mv "$tmp" "$f"
|
||||||
|
if [[ "$(stat -c%s "$f")" != "$size" ]]; then
|
||||||
|
echo "SIZE FAIL $destname ($(stat -c%s "$f") != $size)"
|
||||||
|
status=1
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
if [[ "$sha" == "-" ]]; then
|
||||||
|
echo "VERIFIED $destname (size only) sha256=$(sha256sum "$f" | cut -d' ' -f1)"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
if ! sha256sum -c --quiet <(printf '%s %s\n' "$sha" "$f"); then
|
||||||
|
echo "SHA FAIL $destname"
|
||||||
|
status=1
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
echo "VERIFIED $destname"
|
||||||
|
done
|
||||||
|
if (( status == 0 )); then
|
||||||
|
echo "DOWNLOAD_DONE $DEST"
|
||||||
|
else
|
||||||
|
echo "DOWNLOAD_FAILED $DEST" >&2
|
||||||
|
fi
|
||||||
|
exit "$status"
|
||||||
Executable
+56
@@ -0,0 +1,56 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# fleet-probes-swift15.sh — RUNS ON DANKSERVER. The cutover contract probes.
|
||||||
|
set -uo pipefail
|
||||||
|
B=http://localhost:7770/v1
|
||||||
|
|
||||||
|
echo "== vision (solid red 512px, expect 'Red')"
|
||||||
|
python3 - <<'PY'
|
||||||
|
import base64, json, urllib.request, struct, zlib
|
||||||
|
def png(color):
|
||||||
|
w = h = 512
|
||||||
|
raw = b"".join(b"\x00" + bytes(color) * w for _ in range(h))
|
||||||
|
def chunk(t, d):
|
||||||
|
c = t + d
|
||||||
|
return struct.pack(">I", len(d)) + c + struct.pack(">I", zlib.crc32(c))
|
||||||
|
return (b"\x89PNG\r\n\x1a\n"
|
||||||
|
+ chunk(b"IHDR", struct.pack(">IIBBBBB", w, h, 8, 2, 0, 0, 0))
|
||||||
|
+ chunk(b"IDAT", zlib.compress(raw)) + chunk(b"IEND", b""))
|
||||||
|
b64 = base64.b64encode(png((255, 0, 0))).decode()
|
||||||
|
body = json.dumps({"model": "frank-archon", "max_tokens": 256, "temperature": 0,
|
||||||
|
"messages": [{"role": "user", "content": [
|
||||||
|
{"type": "image_url", "image_url": {"url": "data:image/png;base64," + b64}},
|
||||||
|
{"type": "text", "text": "What color is this image? Answer with one word."}]}]}).encode()
|
||||||
|
r = urllib.request.Request("http://localhost:7770/v1" + "/chat/completions",
|
||||||
|
data=body, headers={"Content-Type": "application/json"})
|
||||||
|
d = json.load(urllib.request.urlopen(r, timeout=300))
|
||||||
|
m = d["choices"][0]["message"]
|
||||||
|
ans = (m.get("content") or m.get("reasoning_content") or "")
|
||||||
|
print("VISION ANSWER:", ans[:80].replace("\n", " "))
|
||||||
|
PY
|
||||||
|
|
||||||
|
echo "== tools (expect tool_calls)"
|
||||||
|
curl -s -m 120 $B/chat/completions -H 'Content-Type: application/json' -d '{
|
||||||
|
"model":"frank-archon","temperature":0,"max_tokens":128,
|
||||||
|
"messages":[{"role":"user","content":"What is 23 times 47? Use the calculator tool."}],
|
||||||
|
"tools":[{"type":"function","function":{"name":"calculator","description":"evaluate arithmetic",
|
||||||
|
"parameters":{"type":"object","properties":{"expr":{"type":"string"}},"required":["expr"]}}}]}' \
|
||||||
|
| python3 -c 'import json,sys; d=json.load(sys.stdin); m=d["choices"][0]["message"]; print("TOOL_CALLS:", m.get("tool_calls"))'
|
||||||
|
|
||||||
|
echo "== reasoning_effort fold (xhigh/medium/low/high all 200 on froggeric v22.5)"
|
||||||
|
for lvl in xhigh medium low high; do
|
||||||
|
code=$(curl -s -o /dev/null -w "%{http_code}" -m 120 $B/chat/completions \
|
||||||
|
-H 'Content-Type: application/json' \
|
||||||
|
-d "{\"model\":\"frank-archon\",\"max_tokens\":8,\"reasoning_effort\":\"$lvl\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}")
|
||||||
|
echo " reasoning_effort=$lvl -> HTTP $code"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "== reasoning_content channel"
|
||||||
|
curl -s -m 120 $B/chat/completions -H 'Content-Type: application/json' \
|
||||||
|
-d '{"model":"frank-archon","max_tokens":64,"messages":[{"role":"user","content":"Is 97 prime? Think briefly."}]}' \
|
||||||
|
| python3 -c 'import json,sys; m=json.load(sys.stdin)["choices"][0]["message"]; print("has reasoning_content:", bool(m.get("reasoning_content")), "| content:", (m.get("content") or "")[:60].replace(chr(10)," "))'
|
||||||
|
|
||||||
|
echo "== response_format json_object"
|
||||||
|
curl -s -m 120 $B/chat/completions -H 'Content-Type: application/json' \
|
||||||
|
-d '{"model":"frank-archon","max_tokens":64,"response_format":{"type":"json_object"},"messages":[{"role":"user","content":"Return JSON {\"ok\": true}"}]}' \
|
||||||
|
| python3 -c 'import json,sys; d=json.load(sys.stdin); print("JSON MODE:", (d["choices"][0]["message"].get("content") or "")[:60])'
|
||||||
|
echo "PROBES_DONE"
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""Measure per-block / per-expert expert-routing activity on live frank-primary.
|
"""Measure per-block / per-expert expert-routing activity on live frank-archon.
|
||||||
|
|
||||||
Method: AMD IBS op sampling (data addresses, `perf record -e ibs_op// --data`)
|
Method: AMD IBS op sampling (data addresses, `perf record -e ibs_op// --data`)
|
||||||
on the llama-server process while concurrent generations of mixed content run.
|
on the llama-server process while concurrent generations of mixed content run.
|
||||||
@@ -0,0 +1,306 @@
|
|||||||
|
# frank-archon.env — radiance (StillDeadcode, Apache-2.0) serving the
|
||||||
|
# PUBLISHER's 4-bit-codebook container of the official Qwen/Qwen3.8-Flash-Next
|
||||||
|
# (MODEL_FILE below, fetched by scripts/download-frank-archon-iq4r.sh) on
|
||||||
|
# BOTH R9700s (TP=2), dankserver :7770.
|
||||||
|
# 2026-10-15 PAYLOAD SWAP: the model moved from the in-house w4dn8 conversion
|
||||||
|
# of the Swift1.5 finetune to the engine author's published container
|
||||||
|
# qwen3.8-next-flash-fp8-iq4r-moe.rad (116,322 MiB, sha256-verified at fetch —
|
||||||
|
# the w4dn8 measured 116,416 MiB, so this shape carries over with 94 MiB to
|
||||||
|
# spare). NO argument changed: every number below is the 2026-10-13 shape,
|
||||||
|
# kept verbatim. The swap commit is the rollback (the w4dn8 payload stays on
|
||||||
|
# disk).
|
||||||
|
|
||||||
|
#
|
||||||
|
# SHAPE (2026-10-13): THREE decode slots x 200,000 tokens, fp8 (quantized) KV,
|
||||||
|
# with the per-card budget spent on resident experts: 25,600 MiB of slab +
|
||||||
|
# 5,120 MiB of pool = 30,720 MiB, inside the 30,969 MiB ceiling the engine's own
|
||||||
|
# budget table allows (card 32,624 MiB, settle floor 209 MiB). The pool is sized
|
||||||
|
# to the contract and nothing more (3 x 200,000 needs 5,063 MiB at fp8; 5,120
|
||||||
|
# predicts 608,027 tokens, 1.3 % slack), because every MiB taken from the slab
|
||||||
|
# is prefill and TTFT (2026-10-11 A/B: slab -14.8 % cost prefill -29 %, TTFT
|
||||||
|
# +25 % at the full window) while KV beyond the contract buys nothing.
|
||||||
|
# Consequence, deliberate: a FOURTH full-height conversation does not fit this
|
||||||
|
# pair, and the engine refuses it — it never truncates. See the KV block for the
|
||||||
|
# arithmetic and the measured dtype law. The w4dn8 payload is the swap
|
||||||
|
# rollback and keeps its trio under the w4dn8 runbook; the fp8-native payload
|
||||||
|
# that came before it was DELETED from disk 2026-10-15 (Frank's call, to free
|
||||||
|
# the volume for this download — rebuildable via the fetch + build scripts
|
||||||
|
# named in the fp8-native runbook).
|
||||||
|
#
|
||||||
|
# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
|
||||||
|
# pinned host RAM; git tag frank-archon-freetoken-pr587-20261005 is the
|
||||||
|
# pre-cutover state — its unit/env/serve trio and Dockerfile are all in the
|
||||||
|
# tag and rebuildable). Before that: tcclaviger vLLM fork (tag
|
||||||
|
# frank-archon-vllm-shapeG-20261003).
|
||||||
|
#
|
||||||
|
# WHY RADIANCE: the engine is C++/HIP with libr4d kernels built for
|
||||||
|
# gfx1201, drives `podman run` directly from the digest-pinned
|
||||||
|
# stilldeadcode/radiance image (287 MiB; host needs only the amdgpu driver),
|
||||||
|
# and its qwen4exp plugin reads the modelopt FP8 encoding NATIVELY
|
||||||
|
# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram
|
||||||
|
# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring
|
||||||
|
# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram."
|
||||||
|
# THE 4-bit-codebook publisher shape is the DEPLOYED payload since
|
||||||
|
# 2026-10-15. The card's own run line (--host-pool-mib 12288,
|
||||||
|
# --expert-vs-cache-ratio 0.82, seqs 8, spec 3) is the publisher's
|
||||||
|
# suggestion on 32 GB cards; this file's shape is the fleet's measured one.
|
||||||
|
#
|
||||||
|
# PAYLOAD: /home/dank/models/rad/qwen3.8-next-flash-fp8-iq4r-moe.rad — the
|
||||||
|
# official Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's
|
||||||
|
# own author (recipe qwen4exp-w4nl64-i8-hc8m: experts as 4-bit codes into a
|
||||||
|
# 16-level non-uniform codebook after a 128-point Walsh–Hadamard rotation,
|
||||||
|
# GPTQ + 3 coordinate-descent sweeps on a 10M-token calibration, ten
|
||||||
|
# bf16-protected experts per layer, int8 W8A8 trunk + lm_head, fp8 HC
|
||||||
|
# mixtures, PLE n-gram fp8 fixed-scale, MTP head, bf16 vision tower; card KL
|
||||||
|
# 0.0789 all-tokens / top-1 92.85 % vs bf16). Fetched byte-exact + sha256 by
|
||||||
|
# scripts/download-frank-archon-iq4r.sh; the recipe/LICENSE stay in the
|
||||||
|
# staging dir beside it as provenance. ROLLBACK: revert the swap commit — the
|
||||||
|
# in-house qwen4exp-swift15-w4dn8-i8-hc8m.rad (scripts/convert-swift15-w4dn8.sh,
|
||||||
|
# KLD-gated) stays on disk. Behind it, the fp8-native
|
||||||
|
# qwen4exp-fp8-native.rad was DELETED from disk 2026-10-15 (Frank's call,
|
||||||
|
# one-time rm, to free the volume for this payload); rebuild =
|
||||||
|
# scripts/download-frank-archon.sh + scripts/build-frank-archon-rad.sh — a
|
||||||
|
# NO-RECIPE rad-convert copy of the official 131-shard checkpoint under the
|
||||||
|
# descriptor law (no shipped plugin claims (qwen4exp, "fp8"), so the staging
|
||||||
|
# config.json drops quantization_config; per-weight encodings are data, not
|
||||||
|
# the descriptor).
|
||||||
|
#
|
||||||
|
# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml):
|
||||||
|
# - The HTTP listener binds AFTER the full load (main.cpp: engine.start()
|
||||||
|
# then engine.serve()) — during load the port refuses connections, so a
|
||||||
|
# bare curl /health gate is honest. /health == /ping: 200 JSON while
|
||||||
|
# serving, 503 only while stopping (core/server/admin.cpp:385).
|
||||||
|
# - Reasoning: --reasoning-format defaults to auto, emitting thinking in
|
||||||
|
# `reasoning_content` (config.cpp). Tool calls: the qwen3.8 dialect, from
|
||||||
|
# the chat format. Both need no parser flags.
|
||||||
|
# - Sampling: the OpenAI endpoint accepts temperature, top_p, top_k, min_p,
|
||||||
|
# typical_p, presence_penalty, frequency_penalty, repetition_penalty,
|
||||||
|
# repeat_last_n, chat_template_kwargs, reasoning_effort,
|
||||||
|
# enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) —
|
||||||
|
# models.yml's extraBody set is fully inside it. Unknown params are
|
||||||
|
# refused (400), which is why nothing unverified rides the wire.
|
||||||
|
# - MTP: the .rad carries the model's mtp.* head (the publisher's recipe
|
||||||
|
# names the mtp.fc_hidden/fc_embedding/draft_head rules; card default
|
||||||
|
# depth 3). Served at the depth named by FRANK_ARCHON_SPEC_TOKENS below —
|
||||||
|
# the depth this shape's decode ladder was measured at.
|
||||||
|
# - The fused TP all-reduce is declared over max-num-seqs x
|
||||||
|
# (1 + --num-speculative-tokens) rows; 3 x (1+2) = 9 rows, under the 25-row
|
||||||
|
# exactness bound of the wht6 wire.
|
||||||
|
# Engine pin: Codeberg tag v1.0.13 (commit 53282fa), Docker Hub tag 1.0.13,
|
||||||
|
# index digest below. REJECTED UPGRADE: 1.2.0 (sha256:665775ed1d90de6dbb
|
||||||
|
# 25558e07fc60d20328f1346f96f3c6e00845eb763f01fa, tag 1.2.0, commit
|
||||||
|
# e95d397) was benched 2026-10-06 on this exact shape (fp8-native rad,
|
||||||
|
# vram pair, spec 2, seqs 3, 43.6k greedy probe): decode 29.21 t/s and
|
||||||
|
# draft acceptance 0.7123 vs the 1.0.13 control's 30.08 and 0.7549.
|
||||||
|
# Acceptance on the greedy harness is deterministic (the control
|
||||||
|
# reproduced the recorded 0.755 exactly), so the drop is a real numerics
|
||||||
|
# change — the 1.2.0 MoE weight-rotation change (92b8aaf) is the named
|
||||||
|
# suspect; decode loss tracks the acceptance ratio (0.943 vs 0.945).
|
||||||
|
# The vram-pair translation itself is exact (1.2.0 boot ledger took
|
||||||
|
# vram_weights 25.46 GiB / vram_kv 3.46 GiB per card byte-identical to
|
||||||
|
# the 1.0.13 ledger at EVC 0.82). Re-test 1.2.0+ when the rotation change
|
||||||
|
# is untangled; 1.2.0 also REMOVED --expert-vs-cache-ratio in favour of
|
||||||
|
# the --vram-weights-mib / --vram-kv-mib pair (core/config.cpp) — this
|
||||||
|
# file already uses the pair, which BOTH pins accept (rollback symmetry).
|
||||||
|
FRANK_ARCHON_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
|
||||||
|
# The .rad container FILE (mounted under /models) and the state dir for the
|
||||||
|
# prefix-cache disk tier (mounted rw at /data). The model dir holds ONLY
|
||||||
|
# .rad files; the raw HF download and its staging dir live beside it and
|
||||||
|
# are never mounted.
|
||||||
|
FRANK_ARCHON_MODEL_DIR=/home/dank/models/rad
|
||||||
|
FRANK_ARCHON_MODEL_FILE=qwen3.8-next-flash-fp8-iq4r-moe.rad
|
||||||
|
FRANK_ARCHON_STATE_DIR=/home/dank/radiance-state
|
||||||
|
# The state dir's tune/ subdirectory holds rad-tune's kernel-selection
|
||||||
|
# cache (the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune
|
||||||
|
# at boot; $RADIANCE_HOME defaults to /opt/radiance, in the image, so the
|
||||||
|
# cache must ride a host mount — serve script mounts it, see the runbook's
|
||||||
|
# "Tuning levers").
|
||||||
|
# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
|
||||||
|
# name across systemctl and podman ps. The runtime changes; the name does
|
||||||
|
# not. The legacy list retires every predecessor identity that could hold
|
||||||
|
# the GPUs or the port.
|
||||||
|
FRANK_ARCHON_CONTAINER=frank-archon
|
||||||
|
FRANK_ARCHON_LEGACY_CONTAINERS="frank-archon-qwen frank-archon-r9v qwen-fp8 glm-probe"
|
||||||
|
# Served model id — Protoss name (wire rename 2026-10-09). The container,
|
||||||
|
# unit, scripts and this file keep the frank-archon name above; only the
|
||||||
|
# alias on the wire moved. Radiance validates the requested model name, so
|
||||||
|
# every client (harness, bench scripts, probes) MUST send this value.
|
||||||
|
FRANK_ARCHON_SERVED_NAME=frank-archon
|
||||||
|
FRANK_ARCHON_PORT=7770
|
||||||
|
# Frank's chat-template call: froggeric's fixed Qwen3.8 template (the
|
||||||
|
# repo's root chat_template.jinja, vendored here; the checkpoint's own
|
||||||
|
# template is baked into the .rad by rad-convert and this overrides it).
|
||||||
|
# Radiance derives the reasoning/tool-call reply format from it.
|
||||||
|
FRANK_ARCHON_TEMPLATE=/home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja
|
||||||
|
|
||||||
|
# --- engine shape (the boot ledger arbitrates every number below) ---------
|
||||||
|
# Tensor parallel: BOTH R9700s. The board swap (2026-09-20, ProArt B850
|
||||||
|
# x8/x8 on root ports) made collectives legal on this host; radiance's TP
|
||||||
|
# wire is libr4d's own, not RCCL.
|
||||||
|
FRANK_ARCHON_TP=2
|
||||||
|
# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice:
|
||||||
|
# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
|
||||||
|
# engine default.
|
||||||
|
FRANK_ARCHON_TP_WIRE=wht6
|
||||||
|
# Decode slots: THREE, so the served window is a CAPACITY CONTRACT — three
|
||||||
|
# concurrent conversations of --max-model-len fit the KV pool (see the KV
|
||||||
|
# block below). The "seqs 2 cannot boot" law is NOT radiance's: it was the
|
||||||
|
# tcclaviger vLLM fork's torch.compile shape specialization on
|
||||||
|
# query_start_loc (ConstraintViolationError, measured 2026-10-02, retired
|
||||||
|
# with that runtime). Radiance booted seqs 2 at the fp8-native first boot,
|
||||||
|
# and the 2026-10-06 decode sweep measured seqs 2 and 3 statistically tied
|
||||||
|
# (30.9-33.1 t/s, inside the +-3% noise floor) — slot count buys concurrency,
|
||||||
|
# not speed, so a slot costs KV, never decode.
|
||||||
|
# Cost of each extra slot: 110 MiB of per-slot recurrent state inside the KV
|
||||||
|
# pool, and the fused TP all-reduce is declared over seqs x (1 + spec) rows —
|
||||||
|
# 3 x 3 = 9 here, under the 25-row bound at which the wht6 wire stays EXACT
|
||||||
|
# (single-sequence decode and any step of <= 25 rows is bit-exact; only busier
|
||||||
|
# steps are lossy).
|
||||||
|
# The pool is sized to this contract and nothing more (see the KV block):
|
||||||
|
# 3 x 200,000 at fp8 = 5,063 MiB, pool 5,120, and the freed 1,792 MiB went into
|
||||||
|
# the resident expert slab. The trade is explicit and it is Frank's call:
|
||||||
|
# a FOURTH full-height conversation no longer fits (5,120 MiB of pool backs
|
||||||
|
# ~608k tokens; 4 x 200,000 needs 6,601), so a fourth 200k request is refused
|
||||||
|
# as too large to fit — it is NOT squeezed, paged, or truncated. Third-party
|
||||||
|
# traffic that wants four full-height sessions is a --vram-kv-mib change, not
|
||||||
|
# a code change.
|
||||||
|
FRANK_ARCHON_MAX_SEQS=3
|
||||||
|
# Served window. Native is 262,144; 200,000 is the publisher's production
|
||||||
|
# value (compose) and what models.yml's contextWindow must quote — a longer
|
||||||
|
# prompt is refused by the engine, never truncated. The engine treats
|
||||||
|
# `kv group tokens >= max-model-len` as a HARD limit ("ONE request longer
|
||||||
|
# than N tokens is accepted by the API and then refused as too large to ever
|
||||||
|
# fit — this is a hard limit, not a backing ratio"), so the pool is sized
|
||||||
|
# for seqs x maxlen, not for one sequence.
|
||||||
|
FRANK_ARCHON_MAX_MODEL_LEN=200000
|
||||||
|
FRANK_ARCHON_PLACEMENT=expert_tiered
|
||||||
|
# Pinned host pool for the expert rows that do not fit VRAM. vram_weights and
|
||||||
|
# this pool serve the per-rank resident set TOGETHER: at the fp8-KV pair
|
||||||
|
# (26071/3543) the ledger showed the VRAM slab filled to 259 KiB of slack,
|
||||||
|
# i.e. the pair was exactly sized. Cutting vram-weights by 3.8 GiB to pay for
|
||||||
|
# the bf16 KV pool SPILLS ~3.8 GiB/rank here, so the pool moves with it —
|
||||||
|
# under-sizing it is a boot failure (36864 refused the boot in the fp8 era).
|
||||||
|
# The boot ledger's coverage line remains the referee, not this arithmetic.
|
||||||
|
FRANK_ARCHON_HOST_POOL_MIB=44928
|
||||||
|
# VRAM left unclaimed per card (publisher default 96 MiB).
|
||||||
|
FRANK_ARCHON_GPU_HEADROOM_MIB=96
|
||||||
|
# VRAM split of claimed memory per card, MiB (0 = derive from the card).
|
||||||
|
# THE CEILING, read off this engine's own VRAM-budget table (boot 2026-10-13,
|
||||||
|
# pair 24000/5120): card 32,624 MiB; budgets claimed 29,120; `3.42 GiB is
|
||||||
|
# neither budgeted nor accounted — the driver's context, the activation arena
|
||||||
|
# and --gpu-headroom-mib come out of it; anything past those is throughput on
|
||||||
|
# the floor`; and `card 0 settled at 2.01 GiB free against 209.19 MiB (the
|
||||||
|
# 96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)`.
|
||||||
|
# So the reserve above the ENGINE's floor is 2,058 - 209 = 1,849 MiB and the
|
||||||
|
# ceiling is 29,120 + 1,849 = 30,969 MiB. The 2026-10-11 bound of 29,120
|
||||||
|
# ("31,856 - 1,710 outside both budgets - keep 1 GiB free") was MY reserve, not
|
||||||
|
# the engine's; the pair below spends the difference and lands 458 MiB above
|
||||||
|
# the engine's floor. `weights + kv <= 30,969` is the gate.
|
||||||
|
# THE PRICE, and it is paid twice: on this engine idle KV VRAM is ALSO the
|
||||||
|
# expert cache's spillover (`mover: 2275 slots can be lent out of the KV
|
||||||
|
# cache (2.69 GiB)`, `arena: lends 740 MiB while steps are <= 256 tokens`),
|
||||||
|
# so a bigger KV pool steals weight residency from the slab AND from the
|
||||||
|
# lendable cache. Priors on this card/model: stealing 2.14 GiB of weights for
|
||||||
|
# a bigger pool (MAX_MODEL_LEN 262144) cost 2.5% decode; the EVC ladder's
|
||||||
|
# other direction (0.82 -> 0.84, +2.5 GiB weights) cost 8.9%. The 2026-10-11
|
||||||
|
# exact-KV A/B measured the slab's own price (22208 vs the fp8-era 26071):
|
||||||
|
# prefill -29 %, TTFT at the full window +25 %, decode inside the noise band on
|
||||||
|
# 2 of 3 passes, draft acceptance UP 0.661 -> 0.740.
|
||||||
|
# 2026-10-13 (late): THREE slots, so the pool is cut to the contract (5,120)
|
||||||
|
# and every spare MiB goes to the slab: 22208 -> 24000 (the third slot's
|
||||||
|
# surplus KV), then 24000 -> 25600 (the card's settling slack). The direction
|
||||||
|
# the A/B data points: prefill is slab-limited, KV beyond the contract is not.
|
||||||
|
# The margin is thin by design (608,027 predicted tokens against the 600,000
|
||||||
|
# contract), so the boot ledger's `backs N sequences all at 200000 tokens at
|
||||||
|
# once` line is the referee for THIS pair and the concurrent probe is its
|
||||||
|
# regression test.
|
||||||
|
# VERDICT on the 25,600 arm (measured 2026-10-13, scripts/probe-frank-archon-kv.py):
|
||||||
|
# the threshold claim is FALSIFIED — the slab is a SLOPE at this end. A-cold
|
||||||
|
# 43.5k single-stream: 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,584 t/s;
|
||||||
|
# the 3-way 197k arm's aggregate prefill 3,051 -> 3,589 t/s (wall
|
||||||
|
# 193.7 -> 164.5 s). So +1,600 MiB of slab = +8.9 % single-stream and
|
||||||
|
# +17.6 % concurrent, both outside the +-3 % noise floor. The 22,208/24,000
|
||||||
|
# pair landed inside noise of EACH OTHER, which is what made it read as a
|
||||||
|
# plateau; two points are not a curve.
|
||||||
|
# The cliff that remains is between 25,600 (3,584) and 26,071 (4,379, the
|
||||||
|
# 2026-10-11 fp8 control at seqs 3 / pool 3,543) — +22 % for +471 MiB. At THREE
|
||||||
|
# slots that is only reachable by breaking the ceiling (26,112 + 5,120 = 31,232
|
||||||
|
# vs 30,969, ~214 MiB above the engine's floor); at TWO slots it is free:
|
||||||
|
# pool 3,072 + slab 27,897 = exactly the ceiling, slab above the cliff.
|
||||||
|
FRANK_ARCHON_VRAM_WEIGHTS_MIB=25600
|
||||||
|
FRANK_ARCHON_VRAM_KV_MIB=5120
|
||||||
|
# KV cache QUANTIZED (fp8). This engine's flag takes ONLY bf16|fp8 — there is
|
||||||
|
# ONE kv_dtype for the whole cache, so "exact K, quantized V" (llama.cpp's
|
||||||
|
# --cache-type-k/--cache-type-v pair, which is what frank-zealot runs) is
|
||||||
|
# UNSERVABLE here: the rung below bf16 is fp8 on both halves.
|
||||||
|
# Measured per-rank cost, read out of the boot ledger's group table (fp8 boot
|
||||||
|
# 2026-10-08, bf16 boot 2026-10-11):
|
||||||
|
# kv_attn 13 layers, 4-token blocks, page 26.00 KiB (fp8) -> 52.00 KiB
|
||||||
|
# kv_qsa_bkey 13 layers, 4-token blocks, page 3.25 KiB at BOTH dtypes —
|
||||||
|
# the QSA block key is an index, not attention state, and does
|
||||||
|
# NOT follow --kv-cache-dtype
|
||||||
|
# => 7.3125 KiB/token fp8, 13.8125 KiB/token bf16: bf16 costs 1.89x fp8,
|
||||||
|
# not 2x, precisely because the bkey plane is dtype-fixed.
|
||||||
|
# The dtype-INDEPENDENT terms also live in the pool: kv_gdn_state 54 MiB x 2
|
||||||
|
# copies per sequence (+ conv/tail/ple ~= 110 MiB a sequence) and the prefix
|
||||||
|
# checkpoint region 8 x 55.99 = 448 MiB. So:
|
||||||
|
# pool_mib >= tokens x 7.3125 KiB + 110 x seqs + 448
|
||||||
|
# 3 x 200,000 at fp8 needs 4,285 + 330 + 448 = 5,063 MiB; the pool is 5,120,
|
||||||
|
# and it predicts (5120 - 330 - 448) x 1024 / 7.3125 = 608,027 tokens: the
|
||||||
|
# three-slot contract with 1.3 % slack, every MiB above 5,063 having been
|
||||||
|
# handed to the expert slab instead. The ledger line is the referee ("backs N
|
||||||
|
# sequences all at 200000 tokens at once") — a pair this tight is read off the
|
||||||
|
# boot, never assumed.
|
||||||
|
# At bf16 the same three slots cost 8,093 + 778 = 8,871 MiB, which would cut
|
||||||
|
# the slab to 20,249 MiB (-9 %); at FOUR slots bf16 costs 10,791 + 888 =
|
||||||
|
# 11,679 MiB and the slab falls to 17,441 (-33 %). fp8 keeps 22,208 at both
|
||||||
|
# slot counts. The 2026-10-11 bf16 shape (6,912 / seqs 2) backed 462,752
|
||||||
|
# tokens = 2 x 200,000; the historical fp8 pool of 3,543 gave 386,904 (1.93
|
||||||
|
# sessions), and the four-slot fp8 boot of 2026-10-13 19:21 backed 843,276.
|
||||||
|
# Secondary saving: the prefix-cache entry shrinks with the attention plane
|
||||||
|
# (one entry 110.50 KiB at bf16 -> 58.50 KiB at fp8, measured on the two boot
|
||||||
|
# ledgers' `kv tier transfer` lines), so the host tier below parks twice as
|
||||||
|
# many conversations as it did at bf16 — no re-size needed to park three slots.
|
||||||
|
FRANK_ARCHON_KV_DTYPE=fp8
|
||||||
|
# Spec decode: the checkpoint's own MTP head, served at depth 2 — the
|
||||||
|
# publisher's card ships depth 3 as its peak; this fleet's ladder was
|
||||||
|
# measured at 2 and the swap keeps every argument, so 2 stays (rows
|
||||||
|
# 3 x (1+2) = 9, inside the wht6 25-row exactness bound).
|
||||||
|
FRANK_ARCHON_SPEC_TOKENS=2
|
||||||
|
FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
|
||||||
|
# Prefix cache tiers (publisher pattern). The host tier is sized in ENTRIES,
|
||||||
|
# and one entry tracks the cache dtype (58.50 KiB at fp8, 117 KiB at bf16, per
|
||||||
|
# the ledger's `kv tier transfer` line): at fp8, 8192 MiB parks ~5.8 sessions
|
||||||
|
# of 200k (at bf16 the same tier parked ~2.9), so four live slots plus a full
|
||||||
|
# parked set fits with the tier as it stands — at RAM the host has (104 GiB
|
||||||
|
# available with the service loaded).
|
||||||
|
FRANK_ARCHON_PREFIX_HOST_MIB=8192
|
||||||
|
FRANK_ARCHON_PREFIX_DISK_MIB=65536
|
||||||
|
# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
|
||||||
|
# RAM the cards read. `ram` REFUSES the start if the memlock limit is too
|
||||||
|
# small — the unit's LimitMEMLOCK=infinity is the other half of this law.
|
||||||
|
FRANK_ARCHON_NGRAM_PLACEMENT=ram
|
||||||
|
# TTM preflight: the pinned ledger (2 x 44 GiB pools + 47.7 GiB PLE + 8 GiB
|
||||||
|
# prefix host = ~147 GiB) rides TTM's shared pages, capped by ttm's
|
||||||
|
# pages_limit (the current kernel exposes NO pages_max; the cap function
|
||||||
|
# takes the min over the knobs that exist). Below the plan the load dies
|
||||||
|
# mid-pin with an allocator error instead of this named refusal.
|
||||||
|
# THE CAP IS A RUNTIME KNOB — the kernel default is half of RAM, so every
|
||||||
|
# raise dies with a reboot. Measured the hard way: the 2026-10-05 manual raise
|
||||||
|
# to 52,428,800 pages (200 GiB) was gone after the 2026-10-12 23:17 POWER
|
||||||
|
# FAILURE, the cap fell back to 125,988 MiB, and the preflight refused the
|
||||||
|
# start 6,919 times in ~20 h while Restart=always re-ran it. The gate in
|
||||||
|
# scripts/ttm-preflight.sh now RAISES the knob itself before refusing (the
|
||||||
|
# root unit is privileged; the knob is a 0644 root module parameter), so the
|
||||||
|
# recovery is a boot, not a visit to the machine.
|
||||||
|
FRANK_ARCHON_TTM_MIN_MIB=150000
|
||||||
|
FRANK_ARCHON_TTM_TARGET_MIB=204800
|
||||||
|
# Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
|
||||||
|
# multi-flag values as one string; the serve script expands it unquoted on
|
||||||
|
# purpose.
|
||||||
|
FRANK_ARCHON_EXTRA_ARGS=
|
||||||
|
# The .rad's minimum size in MiB (deployer gate; a truncated fetch is a
|
||||||
|
# guaranteed long-timeout boot instead of a fast failure). The deployed
|
||||||
|
# publisher container measures 116,322 MiB (121,969,901,568 B); the
|
||||||
|
# in-house w4dn8 rollback measured 116,416 MiB. Floor sits just under both.
|
||||||
|
FRANK_ARCHON_RAD_MIN_MIB=110000
|
||||||
@@ -1,9 +1,9 @@
|
|||||||
# frank-primary.service — radiance (StillDeadcode, Apache-2.0) endpoint
|
# frank-archon.service — radiance (StillDeadcode, Apache-2.0) endpoint
|
||||||
# (OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 at NATIVE block-FP8 precision, the
|
# (the publisher's 4-bit-codebook container of the official
|
||||||
# rad-convert container qwen4exp-fp8-native.rad) on BOTH R9700s (TP=2,
|
# Qwen/Qwen3.8-Flash-Next, qwen3.8-next-flash-fp8-iq4r-moe.rad, fetched by
|
||||||
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (git tag
|
# scripts/download-frank-archon-iq4r.sh) on BOTH R9700s (TP=2, wht6 wire),
|
||||||
# frank-primary-freetoken-pr587-20261005 is the pre-cutover state; the
|
# dankserver :7770. 2026-10-15 payload swap from the in-house w4dn8 (the
|
||||||
# retired unit/env/serve trio and Dockerfile are in that tag).
|
# swap commit is the revert; its .rad stays on disk).
|
||||||
#
|
#
|
||||||
# Same port, same served alias, same container identity as the unit name;
|
# Same port, same served alias, same container identity as the unit name;
|
||||||
# the harness models.yml id is unchanged. The serve script drives
|
# the harness models.yml id is unchanged. The serve script drives
|
||||||
@@ -19,7 +19,7 @@
|
|||||||
# unit grants LimitMEMLOCK=infinity to the dank process via PID1; the
|
# unit grants LimitMEMLOCK=infinity to the dank process via PID1; the
|
||||||
# engine still runs out of dank's rootless podman storage.
|
# engine still runs out of dank's rootless podman storage.
|
||||||
[Unit]
|
[Unit]
|
||||||
Description=frank-primary — radiance Qwen3.8-Flash-Next-FP8 (native) on 2x R9700 TP=2
|
Description=frank-archon — radiance Qwen3.8-Flash-Next (publisher iq4r) on 2x R9700 TP=2
|
||||||
After=network-online.target
|
After=network-online.target
|
||||||
Wants=network-online.target
|
Wants=network-online.target
|
||||||
# Never let the start-rate limiter give up after a crash loop (a failed LLM
|
# Never let the start-rate limiter give up after a crash loop (a failed LLM
|
||||||
@@ -33,14 +33,21 @@ Group=dank
|
|||||||
# RLIMIT_MEMLOCK smaller than the pinned set fails the load mid-start.
|
# RLIMIT_MEMLOCK smaller than the pinned set fails the load mid-start.
|
||||||
# PID1 grants infinity here — parity with the fleet's other LLM units.
|
# PID1 grants infinity here — parity with the fleet's other LLM units.
|
||||||
LimitMEMLOCK=infinity
|
LimitMEMLOCK=infinity
|
||||||
Environment="FRANK_PRIMARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-primary.env"
|
Environment="FRANK_ARCHON_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-archon.env"
|
||||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-primary.sh
|
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-archon.sh
|
||||||
|
# The `+` prefix runs this as root even though the service is User=dank — the
|
||||||
|
# ttm pages cap is a root-owned knob and the pinned-RAM ledger (~147 GiB) is
|
||||||
|
# unusable below it. After a reboot the knob is back at its default (half of
|
||||||
|
# RAM), so the raise must be part of the start, not a note in a runbook: the
|
||||||
|
# 2026-10-12 power failure reset it and the endpoint refused to boot 6,919
|
||||||
|
# times. The serve script re-verifies the cap after this hook.
|
||||||
|
ExecStartPre=+/home/dank/serverAppConfigs/scripts/ttm-preflight.sh --raise
|
||||||
# systemd stop: wait alone cannot stop the container child; stop it
|
# systemd stop: wait alone cannot stop the container child; stop it
|
||||||
# explicitly. No graceful drain contract is claimed (the OpenAI front end
|
# explicitly. No graceful drain contract is claimed (the OpenAI front end
|
||||||
# holds no queue to drain — in-flight decodes are lost, which a stop
|
# holds no queue to drain — in-flight decodes are lost, which a stop
|
||||||
# during a bench window accepts; radiance takes SIGTERM on the engine
|
# during a bench window accepts; radiance takes SIGTERM on the engine
|
||||||
# process and podman reaps the container).
|
# process and podman reaps the container).
|
||||||
ExecStop=/usr/bin/podman stop -t 120 frank-primary
|
ExecStop=/usr/bin/podman stop -t 120 frank-archon
|
||||||
# Boot cost: .rad load (weights + PLE upload) + expert host-pool pin. The
|
# Boot cost: .rad load (weights + PLE upload) + expert host-pool pin. The
|
||||||
# publisher's compose healthcheck allows a 15-minute start_period; the
|
# publisher's compose healthcheck allows a 15-minute start_period; the
|
||||||
# vLLM-era budget (1800 s) carried forward.
|
# vLLM-era budget (1800 s) carried forward.
|
||||||
@@ -1,11 +1,11 @@
|
|||||||
# frank-large.env — tunables for the frank-large endpoint on Dankclaw's
|
# frank-dragoon.env — tunables for the frank-dragoon endpoint on Dankclaw's
|
||||||
# Strix Halo iGPU (gfx1151), host port 7770.
|
# Strix Halo iGPU (gfx1151), host port 7770.
|
||||||
#
|
#
|
||||||
# RUNTIME (2026-10-06, Frank): halogen-flash-server 0.16.2 — the
|
# RUNTIME (2026-10-06, Frank): halogen-flash-server 0.16.2 — the
|
||||||
# closed-source C++/HIP engine (image ghcr.io/peonist-ai/halogen-flash-server,
|
# closed-source C++/HIP engine (image ghcr.io/peonist-ai/halogen-flash-server,
|
||||||
# digest-pinned below) serving the Quat3rnion ABLITERATED Swift1.5 finetune
|
# digest-pinned below) serving the Quat3rnion ABLITERATED Swift1.5 finetune
|
||||||
# of peonist's Qwen3.8-Flash-Next v2 checkpoint, as a podman container under
|
# of peonist's Qwen3.8-Flash-Next v2 checkpoint, as a podman container under
|
||||||
# a root unit, driven by scripts/serve-frank-large.sh. This CUTS OVER the
|
# a root unit, driven by scripts/serve-frank-dragoon.sh. This CUTS OVER the
|
||||||
# 2026-10-05 halogen 27B-dense shape (censored base, DFlash2, 31.7 t/s):
|
# 2026-10-05 halogen 27B-dense shape (censored base, DFlash2, 31.7 t/s):
|
||||||
# rollback = revert the swap commit's trio; the 27B payload stays on disk at
|
# rollback = revert the swap commit's trio; the 27B payload stays on disk at
|
||||||
# /home/dank/models/halogen-qwen3.8-27b/.
|
# /home/dank/models/halogen-qwen3.8-27b/.
|
||||||
@@ -22,15 +22,25 @@
|
|||||||
# tool calls verified, and the native 262,144 context.
|
# tool calls verified, and the native 262,144 context.
|
||||||
#
|
#
|
||||||
# SHAPE — the knobs that are NOT free, with the arithmetic:
|
# SHAPE — the knobs that are NOT free, with the arithmetic:
|
||||||
# HALOGEN_CTX=200192 is the PER-SLOT window and the per-conversation hard
|
# HALOGEN_CTX=262144 is the PER-SLOT window and the per-conversation hard
|
||||||
# ceiling — a prompt over it is a 400 naming the limit, never a silent
|
# ceiling — a prompt over it is a 400 naming the limit, never a silent
|
||||||
# truncation. Harness compaction fires at 70% = 140,134, and 140,134 +
|
# truncation. RAISED from 200192 on 2026-10-07: the harness clamps
|
||||||
# 57,344 (harness maxTokens) = 197,478 <= 200,192 keeps prompt + budget
|
# max_tokens to contextWindow − est − 4096, its chars/4 est UNDER-counts
|
||||||
# inside the slot. models.yml contextWindow must quote 200192 or the
|
# the rendered prompt by ~3 % on code/agent traffic, and compaction only
|
||||||
# harness discovers the wall as a 400.
|
# fires at 85 % — so with window == engine ceiling, every request over
|
||||||
# HALOGEN_KV_POOL_POSITIONS=400384 = KV_SLOTS(2) x CTX — the explicit
|
# ~146k est (65-85 % of the window) was a guaranteed 400. Measured
|
||||||
|
# 2026-10-07: est 152,938, server 157,388, request 43,158 vs room 42,804
|
||||||
|
# — over by 354, margin 4,096 < the 4,450 under-count.
|
||||||
|
# Harness contract (models.yml): contextWindow 217088, maxTokens 45056.
|
||||||
|
# Compaction fires at 85 % = 184,524 est ≈ 190k measured (Frank's 180k
|
||||||
|
# target). Worst-case request: prompt + budget <= 217088 − 4096 + 3 %
|
||||||
|
# est-error ≈ 219k <= 262144 — the 45k gap between harness window and
|
||||||
|
# engine ceiling absorbs the estimate error with ~7x margin (robust to a
|
||||||
|
# 10 % under-count). Compaction rides on frank-zealot: input ≈
|
||||||
|
# threshold − keepRecent(20k) ≈ 170k measured, inside its 200,704 slot.
|
||||||
|
# HALOGEN_KV_POOL_POSITIONS=524288 = KV_SLOTS(2) x CTX — the explicit
|
||||||
# form of the engine's default, so the seam test can pin the product.
|
# form of the engine's default, so the seam test can pin the product.
|
||||||
# KV costs ~28 KiB/position (FLAGS.md) => 400,384 x 28 KiB = 10.7 GiB.
|
# KV costs ~28 KiB/position (FLAGS.md) => 524,288 x 28 KiB = 14.0 GiB.
|
||||||
# HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
|
# HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
|
||||||
# drafter is NOT slot-locked on this engine (batching and speculation
|
# drafter is NOT slot-locked on this engine (batching and speculation
|
||||||
# coexist — DFlash-era slots>1 law does NOT carry over; the card's own
|
# coexist — DFlash-era slots>1 law does NOT carry over; the card's own
|
||||||
@@ -39,11 +49,11 @@
|
|||||||
# Memory ledger against the box's 121.2 GiB unified (the n-gram table is
|
# Memory ledger against the box's 121.2 GiB unified (the n-gram table is
|
||||||
# mmap'd page cache beside the checkpoint, not a pinned charge — the
|
# mmap'd page cache beside the checkpoint, not a pinned charge — the
|
||||||
# card's own quickstart arithmetic does not bill it):
|
# card's own quickstart arithmetic does not bill it):
|
||||||
# weights ~62 GiB resident + KV 10.7 + arena (HALOGEN_MAX_TOK=8192 =>
|
# weights ~62 GiB resident + KV 14.0 + arena (HALOGEN_MAX_TOK=8192 =>
|
||||||
# ~8 GiB per FLAGS.md arena formula 3.8+8.4*(MT/16384)) + vision ~2
|
# ~8 GiB per FLAGS.md arena formula 3.8+8.4*(MT/16384)) + vision ~2
|
||||||
# = ~82.7 GiB engine-side; HALOGEN_HOST_RESERVE_GIB=20 keeps the host
|
# = ~86 GiB engine-side; HALOGEN_HOST_RESERVE_GIB=20 keeps the host
|
||||||
# honest; baseline co-tenancy (frank-secondary, ComfyUI, Caddy/observ)
|
# honest; baseline co-tenancy (frank-zealot, ComfyUI, Caddy/observ)
|
||||||
# ~10-15. ~18 GiB margin. The boot ledger is the referee — read it
|
# ~10-15. ~15 GiB margin. The boot ledger is the referee — read it
|
||||||
# after deploy and re-size if the engine prints a clamp.
|
# after deploy and re-size if the engine prints a clamp.
|
||||||
# HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy
|
# HALOGEN_SPEC_ADAPT=0 is MANDATORY (card): the default adaptive policy
|
||||||
# turns the MTP head off for whole requests after low acceptance and
|
# turns the MTP head off for whole requests after low acceptance and
|
||||||
@@ -57,8 +67,8 @@
|
|||||||
# an absent temperature is no longer greedy. models.yml extraBody
|
# an absent temperature is no longer greedy. models.yml extraBody
|
||||||
# carries the same values — consistent, not load-bearing.
|
# carries the same values — consistent, not load-bearing.
|
||||||
# * HALOGEN_MAX_TOK=8192 is only the DEFAULT for requests that omit
|
# * HALOGEN_MAX_TOK=8192 is only the DEFAULT for requests that omit
|
||||||
# max_tokens; the harness always sends 57344. Requests over
|
# max_tokens; the harness sends at most its maxTokens 45056. Requests
|
||||||
# HALOGEN_MAX_TOKENS_CAP (65536) are a 400 naming the cap. Cap and
|
# over HALOGEN_MAX_TOKENS_CAP (65536) are a 400 naming the cap. Cap and
|
||||||
# queue timeout are COUPLED (FLAGS.md table): 65536 at ~10 t/s serial
|
# queue timeout are COUPLED (FLAGS.md table): 65536 at ~10 t/s serial
|
||||||
# worst case needs > 6550 s; 7200 honors it. Do not move one alone.
|
# worst case needs > 6550 s; 7200 honors it. Do not move one alone.
|
||||||
# * max_tokens INCLUDES thinking; a reasoning reply over a small budget
|
# * max_tokens INCLUDES thinking; a reasoning reply over a small budget
|
||||||
@@ -81,48 +91,51 @@
|
|||||||
# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
|
# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
|
||||||
# is what was qualified). 0.16.2 = the version the model card names as its
|
# is what was qualified). 0.16.2 = the version the model card names as its
|
||||||
# quickstart pin; RepoDigest captured at pull on 2026-10-06.
|
# quickstart pin; RepoDigest captured at pull on 2026-10-06.
|
||||||
FRANK_LARGE_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
|
FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
|
||||||
# Container identity: kept from the 27B era (unit ExecStop names it).
|
# Container identity: kept from the 27B era (unit ExecStop names it).
|
||||||
FRANK_LARGE_CONTAINER=frank-large-halogen
|
FRANK_DRAGOON_CONTAINER=frank-dragoon-halogen
|
||||||
# Old quadlet-era container identity; the serve script clears stale names.
|
# Old quadlet-era container identity; the serve script clears stale names.
|
||||||
FRANK_LARGE_LEGACY_CONTAINERS=frank-large
|
FRANK_DRAGOON_LEGACY_CONTAINERS=frank-dragoon
|
||||||
# host:container — the frontend listens on 8731 inside (HALOGEN_API_PORT
|
# host:container — the frontend listens on 8731 inside (HALOGEN_API_PORT
|
||||||
# default); the engine port (8730) is never published.
|
# default); the engine port (8730) is never published.
|
||||||
FRANK_LARGE_PUBLISH=7770:8731
|
FRANK_DRAGOON_PUBLISH=7770:8731
|
||||||
|
|
||||||
FRANK_LARGE_MODEL_DIR=/home/dank/models/halogen-flashnext-v2
|
FRANK_DRAGOON_MODEL_DIR=/home/dank/models/halogen-flashnext-v2
|
||||||
# The checkpoint: Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated
|
# The checkpoint: Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated
|
||||||
# qwen38-flash-next-v2-swift15-abliterated.hgn — compacted r2 of the card's
|
# qwen38-flash-next-v2-swift15-abliterated.hgn — compacted r2 of the card's
|
||||||
# append-only original (byte-identical live tensors, padding dropped:
|
# append-only original (byte-identical live tensors, padding dropped:
|
||||||
# 66,687,678,464 B vs 89,775,868,032 B). LFS sha256 verified at fetch.
|
# 66,687,678,464 B vs 89,775,868,032 B). LFS sha256 verified at fetch.
|
||||||
FRANK_LARGE_CHECKPOINT=qwen38-flash-next-v2-swift15-abliterated.hgn
|
FRANK_DRAGOON_CHECKPOINT=qwen38-flash-next-v2-swift15-abliterated.hgn
|
||||||
FRANK_LARGE_CHECKPOINT_BYTES=66687678464
|
FRANK_DRAGOON_CHECKPOINT_BYTES=66687678464
|
||||||
FRANK_LARGE_CHECKPOINT_SHA256=9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
|
FRANK_DRAGOON_CHECKPOINT_SHA256=9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037
|
||||||
# The n-gram lookup table: REQUIRED (the engine reads it beside the
|
# The n-gram lookup table: REQUIRED (the engine reads it beside the
|
||||||
# checkpoint; the card's quickstart names it explicitly). From
|
# checkpoint; the card's quickstart names it explicitly). From
|
||||||
# peonist-ai/halogen-qwen3.8-flash-next @ 5cc17cea.
|
# peonist-ai/halogen-qwen3.8-flash-next @ 5cc17cea.
|
||||||
FRANK_LARGE_NGRAM_TABLE=qwen38-flash-next-ngram.hgn
|
FRANK_DRAGOON_NGRAM_TABLE=qwen38-flash-next-ngram.hgn
|
||||||
FRANK_LARGE_NGRAM_BYTES=51200246144
|
FRANK_DRAGOON_NGRAM_BYTES=51200246144
|
||||||
FRANK_LARGE_NGRAM_SHA256=3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
|
FRANK_DRAGOON_NGRAM_SHA256=3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe
|
||||||
# The vision tower: OPTIONAL on the engine, ENABLED here (image input).
|
# The vision tower: OPTIONAL on the engine, ENABLED here (image input).
|
||||||
# From peonist-ai/halogen-qwen3.8-flash-next @ e053f488.
|
# From peonist-ai/halogen-qwen3.8-flash-next @ e053f488.
|
||||||
FRANK_LARGE_VISION_TOWER=qwen38-flash-next-vision.hgn
|
FRANK_DRAGOON_VISION_TOWER=qwen38-flash-next-vision.hgn
|
||||||
FRANK_LARGE_VISION_BYTES=897916416
|
FRANK_DRAGOON_VISION_BYTES=897916416
|
||||||
FRANK_LARGE_VISION_SHA256=d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
|
FRANK_DRAGOON_VISION_SHA256=d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10
|
||||||
# The tokenizer is a FLAT DIRECTORY, $FRANK_LARGE_MODEL_DIR/tokenizer,
|
# The tokenizer is a FLAT DIRECTORY, $FRANK_DRAGOON_MODEL_DIR/tokenizer,
|
||||||
# mounted whole at /tokenizer. The two repos ship the same one; fetched
|
# mounted whole at /tokenizer. The two repos ship the same one; fetched
|
||||||
# fresh from peonist @ e053f488 (tokenizer.json is LFS, byte-identical to
|
# fresh from peonist @ e053f488 (tokenizer.json is LFS, byte-identical to
|
||||||
# the 27B-era copy; the rest are plain git blobs — size-checked only). The
|
# the 27B-era copy; the rest are plain git blobs — size-checked only). The
|
||||||
# serve gate and the deployer refuse a missing file or ANY symlink: an HF
|
# serve gate and the deployer refuse a missing file or ANY symlink: an HF
|
||||||
# cache snapshot symlinks each file into blobs/, and those links dangle
|
# cache snapshot symlinks each file into blobs/, and those links dangle
|
||||||
# inside the container mount.
|
# inside the container mount.
|
||||||
FRANK_LARGE_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json"
|
FRANK_DRAGOON_TOKENIZER_FILES="tokenizer.json chat_template.jinja tokenizer_config.json merges.txt vocab.json generation_config.json"
|
||||||
|
|
||||||
# --- engine flags (docs/FLAGS.md of the flash-server repo; anything unset
|
# --- engine flags (docs/FLAGS.md of the flash-server repo; anything unset
|
||||||
# stays at the image default, which is what was qualified) ------------------
|
# stays at the image default, which is what was qualified) ------------------
|
||||||
HALOGEN_MODEL_ID=frank-large
|
# Served model id — Protoss name (wire rename 2026-10-09); the service, unit,
|
||||||
HALOGEN_CTX=200192
|
# scripts and this file keep the frank-dragoon name. OpenWebUI api_configs
|
||||||
HALOGEN_KV_POOL_POSITIONS=400384
|
# model_ids for the :7770 backend MUST name this value.
|
||||||
|
HALOGEN_MODEL_ID=frank-dragoon
|
||||||
|
HALOGEN_CTX=262144
|
||||||
|
HALOGEN_KV_POOL_POSITIONS=524288
|
||||||
HALOGEN_KV_SLOTS=2
|
HALOGEN_KV_SLOTS=2
|
||||||
HALOGEN_HOST_RESERVE_GIB=20
|
HALOGEN_HOST_RESERVE_GIB=20
|
||||||
# Spec decode: MTP head is INSIDE the checkpoint (do NOT set HALOGEN_MTP_HEAD);
|
# Spec decode: MTP head is INSIDE the checkpoint (do NOT set HALOGEN_MTP_HEAD);
|
||||||
@@ -1,5 +1,5 @@
|
|||||||
[Unit]
|
[Unit]
|
||||||
Description=halogen-flash-server container (Quat3rnion abliterated Swift1.5 of Qwen3.8-Flash-Next v2, MTP spec decode, 2 slots x 200192, vision ON), root systemd unit (dankclaw, frank-large, Strix Halo iGPU)
|
Description=halogen-flash-server container (Quat3rnion abliterated Swift1.5 of Qwen3.8-Flash-Next v2, MTP spec decode, 2 slots x 200192, vision ON), root systemd unit (dankclaw, frank-dragoon, Strix Halo iGPU)
|
||||||
After=network-online.target
|
After=network-online.target
|
||||||
Wants=network-online.target
|
Wants=network-online.target
|
||||||
|
|
||||||
@@ -11,11 +11,11 @@ Group=dank
|
|||||||
# RLIMIT smaller than the pinned set cannot fail the load. Parity with the
|
# RLIMIT smaller than the pinned set cannot fail the load. Parity with the
|
||||||
# other LLM units on this fleet.
|
# other LLM units on this fleet.
|
||||||
LimitMEMLOCK=infinity
|
LimitMEMLOCK=infinity
|
||||||
Environment="FRANK_LARGE_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-large.env"
|
Environment="FRANK_DRAGOON_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-dragoon.env"
|
||||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-large.sh
|
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-dragoon.sh
|
||||||
# systemd stop alone cannot stop the container child; stop it explicitly
|
# systemd stop alone cannot stop the container child; stop it explicitly
|
||||||
# (frank-primary pattern). The engine drains on SIGTERM.
|
# (frank-archon pattern). The engine drains on SIGTERM.
|
||||||
ExecStop=/usr/bin/podman stop -t 60 frank-large-halogen
|
ExecStop=/usr/bin/podman stop -t 60 frank-dragoon-halogen
|
||||||
TimeoutStartSec=1200
|
TimeoutStartSec=1200
|
||||||
TimeoutStopSec=120
|
TimeoutStopSec=120
|
||||||
# always, not on-failure: an engine that exits 0 after an internal fault
|
# always, not on-failure: an engine that exits 0 after an internal fault
|
||||||
@@ -23,7 +23,7 @@ TimeoutStopSec=120
|
|||||||
Restart=always
|
Restart=always
|
||||||
RestartSec=10
|
RestartSec=10
|
||||||
# The engine pins ~62 GiB of v2 weights and keeps a multi-GiB working set
|
# The engine pins ~62 GiB of v2 weights and keeps a multi-GiB working set
|
||||||
# on a box that also hosts frank-secondary, ComfyUI and the Caddy/observability
|
# on a box that also hosts frank-zealot, ComfyUI and the Caddy/observability
|
||||||
# stack; make it a late OOM candidate, not the first.
|
# stack; make it a late OOM candidate, not the first.
|
||||||
OOMScoreAdjust=-500
|
OOMScoreAdjust=-500
|
||||||
|
|
||||||
@@ -1,142 +0,0 @@
|
|||||||
# frank-primary.env — radiance (StillDeadcode, Apache-2.0) serving the
|
|
||||||
# OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at NATIVE precision
|
|
||||||
# (modelopt block-FP8 throughout, no requantisation) on BOTH R9700s (TP=2),
|
|
||||||
# dankserver :7770.
|
|
||||||
#
|
|
||||||
# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
|
|
||||||
# pinned host RAM; git tag frank-primary-freetoken-pr587-20261005 is the
|
|
||||||
# pre-cutover state — its unit/env/serve trio and Dockerfile are all in the
|
|
||||||
# tag and rebuildable). Before that: tcclaviger vLLM fork (tag
|
|
||||||
# frank-primary-vllm-shapeG-20261003).
|
|
||||||
#
|
|
||||||
# WHY RADIANCE: the engine is C++/HIP with libr4d kernels built for
|
|
||||||
# gfx1201, drives `podman run` directly from the digest-pinned
|
|
||||||
# stilldeadcode/radiance image (287 MiB; host needs only the amdgpu driver),
|
|
||||||
# and its qwen4exp plugin reads the modelopt FP8 encoding NATIVELY
|
|
||||||
# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram
|
|
||||||
# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring
|
|
||||||
# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram."
|
|
||||||
# The 4-bit-codebook publisher shape (qwen3.8-next-flash-fp8-iq4r-moe.rad,
|
|
||||||
# --host-pool-mib 12288) remains the documented fallback if the native
|
|
||||||
# container proves unbootable.
|
|
||||||
#
|
|
||||||
# PAYLOAD: /home/dank/models/rad/qwen4exp-fp8-native.rad — a rad-convert
|
|
||||||
# NO-RECIPE pure copy of the official 131-shard checkpoint (every weight
|
|
||||||
# keeps its block-FP8 encoding; ~161 GiB). The convert runs against a
|
|
||||||
# staging dir whose config.json drops `quantization_config`: the descriptor
|
|
||||||
# string is only the plugin SELECTION key (spec §2.4 — exact match, no
|
|
||||||
# fallback), and no shipped plugin claims (qwen4exp, "fp8"); the shipped
|
|
||||||
# qwen4exp plugin claims (qwen4exp, "") which is what recipe-built
|
|
||||||
# containers carry. Per-weight encodings are data, not the descriptor —
|
|
||||||
# removing the key changes nothing about what the weights hold. Built by
|
|
||||||
# scripts/build-frank-primary-rad.sh (staging + convert + rad-info).
|
|
||||||
#
|
|
||||||
# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml):
|
|
||||||
# - The HTTP listener binds AFTER the full load (main.cpp: engine.start()
|
|
||||||
# then engine.serve()) — during load the port refuses connections, so a
|
|
||||||
# bare curl /health gate is honest. /health == /ping: 200 JSON while
|
|
||||||
# serving, 503 only while stopping (core/server/admin.cpp:385).
|
|
||||||
# - Reasoning: --reasoning-format defaults to auto, emitting thinking in
|
|
||||||
# `reasoning_content` (config.cpp). Tool calls: the qwen3.8 dialect, from
|
|
||||||
# the chat format. Both need no parser flags.
|
|
||||||
# - Sampling: the OpenAI endpoint accepts temperature, top_p, top_k, min_p,
|
|
||||||
# typical_p, presence_penalty, frequency_penalty, repetition_penalty,
|
|
||||||
# repeat_last_n, chat_template_kwargs, reasoning_effort,
|
|
||||||
# enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) —
|
|
||||||
# models.yml's extraBody set is fully inside it. Unknown params are
|
|
||||||
# refused (400), which is why nothing unverified rides the wire.
|
|
||||||
# - MTP: the official checkpoint carries the mtp.* head (3,101 tensors,
|
|
||||||
# mtp_num_hidden_layers=1) and the plugin probes draft depth 3 — the
|
|
||||||
# publisher's throughput peak for this model. Stated explicitly with
|
|
||||||
# --num-speculative-tokens 3.
|
|
||||||
# - The fused TP all-reduce is declared over max-num-seqs x
|
|
||||||
# (1 + --num-speculative-tokens) rows; 2 x 4 = 8 rows fits either wire.
|
|
||||||
FRANK_PRIMARY_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
|
|
||||||
# The .rad container FILE (mounted under /models) and the state dir for the
|
|
||||||
# prefix-cache disk tier (mounted rw at /data). The model dir holds ONLY
|
|
||||||
# .rad files; the raw HF download and its staging dir live beside it and
|
|
||||||
# are never mounted.
|
|
||||||
FRANK_PRIMARY_MODEL_DIR=/home/dank/models/rad
|
|
||||||
FRANK_PRIMARY_MODEL_FILE=qwen4exp-fp8-native.rad
|
|
||||||
FRANK_PRIMARY_STATE_DIR=/home/dank/radiance-state
|
|
||||||
# The state dir's tune/ subdirectory holds rad-tune's kernel-selection
|
|
||||||
# cache (the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune
|
|
||||||
# at boot; $RADIANCE_HOME defaults to /opt/radiance, in the image, so the
|
|
||||||
# cache must ride a host mount — serve script mounts it, see the runbook's
|
|
||||||
# "Tuning levers").
|
|
||||||
# Container identity is the SERVICE name (Frank's call, 2026-10-05): one
|
|
||||||
# name across systemctl and podman ps. The runtime changes; the name does
|
|
||||||
# not. The legacy list retires every predecessor identity that could hold
|
|
||||||
# the GPUs or the port.
|
|
||||||
FRANK_PRIMARY_CONTAINER=frank-primary
|
|
||||||
FRANK_PRIMARY_LEGACY_CONTAINERS="frank-primary-qwen frank-primary-r9v qwen-fp8 glm-probe"
|
|
||||||
FRANK_PRIMARY_SERVED_NAME=frank-primary
|
|
||||||
FRANK_PRIMARY_PORT=7770
|
|
||||||
# Frank's chat-template call: froggeric's fixed Qwen3.8 template (the
|
|
||||||
# repo's root chat_template.jinja, vendored here; the checkpoint's own
|
|
||||||
# template is baked into the .rad by rad-convert and this overrides it).
|
|
||||||
# Radiance derives the reasoning/tool-call reply format from it.
|
|
||||||
FRANK_PRIMARY_TEMPLATE=/home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja
|
|
||||||
|
|
||||||
# --- engine shape (first-boot conservative; the boot ledger arbitrates) --
|
|
||||||
# Tensor parallel: BOTH R9700s. The board swap (2026-09-20, ProArt B850
|
|
||||||
# x8/x8 on root ports) made collectives legal on this host; radiance's TP
|
|
||||||
# wire is libr4d's own, not RCCL.
|
|
||||||
FRANK_PRIMARY_TP=2
|
|
||||||
# Lossy WHT 6-bit wire for large all-reduces (publisher's production choice:
|
|
||||||
# +5.6-6.3% prefill, ~-1.3% 8-stream decode at TP2). min-kb stays the
|
|
||||||
# engine default.
|
|
||||||
FRANK_PRIMARY_TP_WIRE=wht6
|
|
||||||
# Decode slots. 2 is Frank's first-boot value ("use parallelism = 2 to
|
|
||||||
# start"); raise only after a real-traffic look at slot occupancy and the
|
|
||||||
# KV pool's headroom line. The recurrent-state cost scales with it (54
|
|
||||||
# MiB/copy/card, 2 copies with speculation).
|
|
||||||
FRANK_PRIMARY_MAX_SEQS=3
|
|
||||||
# Served window. Native is 262,144; 200,000 is the publisher's production
|
|
||||||
# value (compose) and what models.yml's contextWindow must quote — a longer
|
|
||||||
# prompt is refused by the engine, never truncated.
|
|
||||||
FRANK_PRIMARY_MAX_MODEL_LEN=200000
|
|
||||||
FRANK_PRIMARY_PLACEMENT=expert_tiered
|
|
||||||
# Pinned host pool for the expert rows that do not fit VRAM. Native FP8
|
|
||||||
# experts are 112.8 GiB total (56.4 GiB per rank slice) vs the 4-bit
|
|
||||||
# publisher shape's ~29 GiB — this pool is the cost of native precision.
|
|
||||||
# The boot ledger prints the pool's coverage ("the pool backs N% of the
|
|
||||||
# worst case ... M full-length sessions"); resize from that line, not from
|
|
||||||
# arithmetic.
|
|
||||||
FRANK_PRIMARY_HOST_POOL_MIB=40960
|
|
||||||
# VRAM left unclaimed per card (publisher default 96 MiB).
|
|
||||||
FRANK_PRIMARY_GPU_HEADROOM_MIB=96
|
|
||||||
# Weights share of claimed VRAM (publisher production value; higher = more
|
|
||||||
# prefix-cache hit rate, less KV headroom).
|
|
||||||
FRANK_PRIMARY_EXPERT_VS_CACHE=0.82
|
|
||||||
# KV cache in FP8 (7.32 KiB/token at this model's 12 full-attention layers,
|
|
||||||
# 2 KV heads, 256 head_dim, K+V).
|
|
||||||
FRANK_PRIMARY_KV_DTYPE=fp8
|
|
||||||
# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
|
|
||||||
# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
|
|
||||||
# flag states it so the ledger records the intent).
|
|
||||||
FRANK_PRIMARY_SPEC_TOKENS=2
|
|
||||||
FRANK_PRIMARY_MAX_BATCHED_TOKENS=2048
|
|
||||||
# Prefix cache tiers (publisher pattern; disk tier sized for 2 slots).
|
|
||||||
FRANK_PRIMARY_PREFIX_HOST_MIB=4096
|
|
||||||
FRANK_PRIMARY_PREFIX_DISK_MIB=65536
|
|
||||||
# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
|
|
||||||
# RAM the cards read. `ram` REFUSES the start if the memlock limit is too
|
|
||||||
# small — the unit's LimitMEMLOCK=infinity is the other half of this law.
|
|
||||||
FRANK_PRIMARY_NGRAM_PLACEMENT=ram
|
|
||||||
# TTM preflight: the pinned ledger (2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
|
|
||||||
# prefix host) is ~132 GiB. ttm's pages_limit/pages_max cap the GPU-mappable
|
|
||||||
# pinned pages; the current cap (32,252,917 pages = ~123 GiB) is BELOW the
|
|
||||||
# plan, so the serve script refuses to start until both are raised. This is
|
|
||||||
# the floor it enforces (in MiB), with ~8 GiB of margin.
|
|
||||||
FRANK_PRIMARY_TTM_MIN_MIB=140000
|
|
||||||
# Escape hatch for boot-ladder experiments. Unset -> no extra argv. Keep
|
|
||||||
# multi-flag values as one string; the serve script expands it unquoted on
|
|
||||||
# purpose.
|
|
||||||
FRANK_PRIMARY_EXTRA_ARGS=
|
|
||||||
# The .rad's minimum size in MiB (deployer gate; a truncated convert is a
|
|
||||||
# guaranteed long-timeout boot instead of a fast failure). The converted
|
|
||||||
# container measured 177,109 MiB (1,857,127,997,440 B = 173.0 GiB; the
|
|
||||||
# plan-only estimate 172.76 GiB covers the data planes but not the metadata
|
|
||||||
# tail) - floor sits just under the payload.
|
|
||||||
FRANK_PRIMARY_RAD_MIN_MIB=177000
|
|
||||||
@@ -1,98 +0,0 @@
|
|||||||
# frank-secondary — llama.cpp (master, CUDA sm_86) serving
|
|
||||||
# ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF:RCO-IQ3_XXS
|
|
||||||
# **with the MTP head in the file** on the RTX 3090, :7771.
|
|
||||||
#
|
|
||||||
# 2026-10-06 (Frank, evening): the payload moved to ajgazin's UNCELERSED
|
|
||||||
# Swift-1.5-Qwen3.8-27B at RCO-IQ3_XXS (10,442,827,392 B) — 1.56 GiB
|
|
||||||
# lighter than the IQ3_S-mtp it replaces — with MTP BACK ON (draft-mtp,
|
|
||||||
# depth 3), TWO 131,072 slots, and ubatch 512. The uncensored finetune
|
|
||||||
# changes behavior by design (publisher: 23/100 refusals vs Swift 1.5's
|
|
||||||
# 98/100, KL 0.0884 from Swift 1.5) — Frank's order. The RCO tier is the
|
|
||||||
# same per-tensor ALLOCATION recipe as the IQ3_S file (histogram: iq3_s
|
|
||||||
# 97 / bf16 96 / iq3_xxs 83 / iq2_s 72 / iq4_xs 33 / iq2_xs 32 / iq2_xxs
|
|
||||||
# 30 / q2_k 28 / q4_k 19 / q6_k 8 / iq1_s 4 / iq1_m 4 — all stock types,
|
|
||||||
# stock llama.cpp loads it); publisher KLD: Mean 0.002631, 90 % 0.0421
|
|
||||||
# (kld_results.csv). RCO files are "more experimental" per the README —
|
|
||||||
# the quality floor is the IQ3_S rung, not a promise.
|
|
||||||
#
|
|
||||||
# The architecture is unchanged (same model family): qwen35 hybrid —
|
|
||||||
# block_count 65 (64 real layers + the blk.64 MTP head),
|
|
||||||
# full_attention_interval 4 => SIXTEEN full-attention layers, 24 Q heads
|
|
||||||
# / 4 KV heads, k/v length 256, native window 262144, dense,
|
|
||||||
# nextn_predict_layers=1. The MTP head ships in every main GGUF of this
|
|
||||||
# repo (README); --spec-type draft-mtp loads it against the main model's
|
|
||||||
# own weights.
|
|
||||||
#
|
|
||||||
# THE KV CONTRACT STAYS q8_0 K + q8_0 V (Frank, 2026-10-05): 34 KiB/token
|
|
||||||
# (16 attn layers x 4 heads x 256 x 1.0625 B, K and V).
|
|
||||||
#
|
|
||||||
# Why TWO slots + MTP fits now (it did not under IQ3_S):
|
|
||||||
# card 24,576 MiB. Weights 9,963 MiB (10,442,827,392 B; IQ3_S was
|
|
||||||
# 11,561). Projector ~1,290 MiB (the PUBLISHER's OWN mmproj-BF16.gguf,
|
|
||||||
# 931,145,888 B, at the measured ~1.45x card law — no substitution).
|
|
||||||
# Compute buffers at ubatch 512 ~1.3 GiB. CUDA ctx ~500 MiB. DeltaNet
|
|
||||||
# state ~150 MiB/slot x 2. MTP draft ctx back: ~0.4 GiB. Non-KV ~13.9
|
|
||||||
# GiB. Two q8_0 slots = 8,704 MiB. Predicted free ~2.6-3.2 GiB — back
|
|
||||||
# over the old 2-3 GiB floor that the IQ3_S two-slot boot violated
|
|
||||||
# (measured there: 1,397 MiB free). MEASURE nvidia-smi free at boot
|
|
||||||
# (quote the free field) and let the runbook carry the number.
|
|
||||||
#
|
|
||||||
# Payload history (rollback trail): /home/dank/models/Swift15-IQ3S/ (the
|
|
||||||
# IQ3_S-mtp that was live until this swap), Ornith-Q8/, MiMo-Q6K/ all
|
|
||||||
# stay on disk; one MODEL_FILE + shape edit back each.
|
|
||||||
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
|
||||||
MODEL_FILE=/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf
|
|
||||||
|
|
||||||
# Vision: the publisher SHIPS the projector — mmproj-BF16.gguf
|
|
||||||
# (931,145,888 B) from the same repo, fetched by
|
|
||||||
# scripts/download-frank-secondary.sh. No substitution needed for this
|
|
||||||
# payload (the unsloth base-tower proof of 2026-10-05 remains the
|
|
||||||
# fallback argument for the IQ3_S rollback payload).
|
|
||||||
# Empty string = text-only on purpose.
|
|
||||||
MMPROJ_FILE=/home/dank/models/Swift15-UncIQ3XXS/mmproj-BF16.gguf
|
|
||||||
|
|
||||||
# Fleet endpoint: same port and alias as every frank-secondary before it, so the
|
|
||||||
# harness models.yml id and the homepage probe do not move.
|
|
||||||
ALIAS=frank-secondary
|
|
||||||
BIND_ADDR=0.0.0.0
|
|
||||||
LISTEN_PORT=7771
|
|
||||||
|
|
||||||
# TWO conversations at Frank's 112k cap (2026-10-06 evening): --ctx-size is
|
|
||||||
# the POOL llama.cpp divides across slots; 229376 / 2 = 114,688 per
|
|
||||||
# conversation. Lowering from 128k (131,072) frees ~1 GiB of KV, which
|
|
||||||
# funds the ubatch 512 -> 1024 increase (prefill throughput). The
|
|
||||||
# per-slot window is the product contract, so models.yml follows to
|
|
||||||
# 114,688.
|
|
||||||
CTX_SIZE=229376
|
|
||||||
PARALLEL=2
|
|
||||||
|
|
||||||
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves);
|
|
||||||
# the 1024 rung was the first A/B back once the per-slot context dropped
|
|
||||||
# to 112k (2026-10-06 evening), and 2048 is the second (Frank, same night)
|
|
||||||
# — the rung measured +4.6 % prefill for 616 MiB on the 3090 (2026-10-05
|
|
||||||
# boot pair), affordable inside the 1,751 MiB the 112k cut freed. The pair
|
|
||||||
# is a measurement, not a law (loud failure at graph capture if it does
|
|
||||||
# not fit).
|
|
||||||
BATCH_SIZE=4096
|
|
||||||
UBATCH_SIZE=2048
|
|
||||||
|
|
||||||
# The KV contract for this swap (Frank, 2026-10-05): q8_0 K + q8_0 V —
|
|
||||||
# llama.cpp's FP8 class. The flags stay explicit so the choice is visible in
|
|
||||||
# the argv either way, and --flash-attn stays pinned on in the serve script:
|
|
||||||
# it is required for quantized V (the build carries
|
|
||||||
# GGML_CUDA_FA_ALL_QUANTS=ON since 2026-09-17, so q8_0 KV is a real path,
|
|
||||||
# not a silent f16 fallback).
|
|
||||||
CACHE_TYPE_K=q8_0
|
|
||||||
CACHE_TYPE_V=q8_0
|
|
||||||
|
|
||||||
# Speculative decoding is ON (Frank, 2026-10-06 evening): draft-mtp at
|
|
||||||
# depth 3 against the head IN the file (the -mtp variant ships the
|
|
||||||
# blk.64.nextn.* tensors). ngram-mod stays skipped — on a qwen35 hybrid
|
|
||||||
# it disables recurrent-state rollback, so a rejected draft would
|
|
||||||
# corrupt the DeltaNet state; MTP needs no such bypass (its draft ctx
|
|
||||||
# re-derives state from the target's own weights). The flags move as a
|
|
||||||
# set: the serve script emits BOTH only when SPEC_TYPE names a drafter.
|
|
||||||
SPEC_TYPE=draft-mtp
|
|
||||||
SPEC_DRAFT_N_MAX=3
|
|
||||||
|
|
||||||
NGL=99
|
|
||||||
@@ -1,19 +0,0 @@
|
|||||||
[Unit]
|
|
||||||
Description=llama.cpp Swift-1.5-Qwen3.8-27B Uncensored RCO IQ3_XXS GGUF (qwen35 hybrid, 2x131072 slots, q8_0 K+V, draft-mtp depth 3, mmproj vision) on the RTX 3090 (dankclaw, frank-secondary)
|
|
||||||
After=network-online.target
|
|
||||||
Wants=network-online.target
|
|
||||||
|
|
||||||
[Service]
|
|
||||||
Type=simple
|
|
||||||
User=dank
|
|
||||||
Group=dank
|
|
||||||
# CUDA pins its staging buffers, so the process needs the memlock budget even
|
|
||||||
# though the weights themselves are copied into VRAM.
|
|
||||||
LimitMEMLOCK=infinity
|
|
||||||
Environment="FRANK_SECONDARY_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-secondary.env"
|
|
||||||
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-secondary.sh
|
|
||||||
Restart=on-failure
|
|
||||||
RestartSec=10
|
|
||||||
|
|
||||||
[Install]
|
|
||||||
WantedBy=multi-user.target
|
|
||||||
@@ -0,0 +1,62 @@
|
|||||||
|
# frank-zealot — llama.cpp (master, CUDA sm_86) serving
|
||||||
|
# HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive at Q6_K on the RTX 3090,
|
||||||
|
# :7771.
|
||||||
|
#
|
||||||
|
# 2026-10-07 (Frank): swap from Gemma4-26B-A4B QAT Q4_K_M (one 200,704 slot)
|
||||||
|
# to Qwen3.5-9B Q6_K at TWO 200,704-token conversations, EXACT (f16) KV,
|
||||||
|
# MTP disabled (SPEC_TYPE=none), log verbosity 4.
|
||||||
|
#
|
||||||
|
# Architecture — read from the GGUF's own header (HTTP range read, not the
|
||||||
|
# model card): arch qwen35, 32 layers, dense 9B, full_attention_interval 4
|
||||||
|
# ⇒ 8 full-attention layers (3,7,11,...,31) + 24 Gated DeltaNet linear layers,
|
||||||
|
# 16 attention heads / 4 KV heads × 256 dim, ssm conv_kernel 4 / state_size
|
||||||
|
# 128 / group_count 16 / time_step_rank 32 / inner_size 4096, native 262,144,
|
||||||
|
# vocab 248,320, file_type 18 (Q6_K). Tensor-type histogram: Q6_K×250 +
|
||||||
|
# F32×177 — standard types only, so the stock master build loads it (no
|
||||||
|
# custom-quant lock-in like the ROCmFPX era). NO nextn/mtp tensors exist, so
|
||||||
|
# in-file MTP is impossible here; SPEC_TYPE=none is forced by the payload.
|
||||||
|
#
|
||||||
|
# KV rate law: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per
|
||||||
|
# token per cache. f16 K + f16 V = 32,768 B/token = 32 KiB/token, so the
|
||||||
|
# 401,408-token pool is 12.25 GiB. The 24 DeltaNet layers cost a fixed
|
||||||
|
# recurrent state per SLOT (not per token) — that is why two 200k slots fit
|
||||||
|
# where the 26B MoE fit only one.
|
||||||
|
#
|
||||||
|
# Feasibility comes from the predecessor's MEASURED boots, not a formula:
|
||||||
|
# MiMo-V2.6-Distill-Qwen-9B (identical qwen35 geometry, 7,793,710,624 B) at
|
||||||
|
# 2×200,704 f16 K + f16 V measured 21,276 used / 2,849 free at ubatch 1024
|
||||||
|
# and 21,846 / 2,279 at ubatch 2048. This file is 415 MiB lighter in weights,
|
||||||
|
# its BF16 mmproj is 3 MiB heavier ⇒ projected ~21,430 used / ~2,700 free.
|
||||||
|
# The verbosity-4 boot ledger is the referee.
|
||||||
|
#
|
||||||
|
# Ubatch 2048 is the measured CUDA optimum (measured on the MiMo predecessor
|
||||||
|
# at this exact arch + shape: +4.6 % prefill at 8.9k depth, decode flat, cost
|
||||||
|
# 616 MiB). 1024 is the fallback rung if the boot lands under the 2 GiB floor.
|
||||||
|
LLAMA_BIN=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server
|
||||||
|
MODEL_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf
|
||||||
|
MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf
|
||||||
|
# Served model id — the harness/family name (Protoss rename completed to the
|
||||||
|
# wire 2026-10-09: harness ids + homepage cards 2026-10-08, /v1/models aliases
|
||||||
|
# this day). The SERVICE, unit, scripts and this env file keep the
|
||||||
|
# frank-zealot name; only the alias on the wire changed. Consumers that
|
||||||
|
# key on the served id: OpenWebUI (model row, access_grant, api_configs
|
||||||
|
# model_ids) and the harness models.yml.
|
||||||
|
ALIAS=frank-zealot
|
||||||
|
BIND_ADDR=0.0.0.0
|
||||||
|
LISTEN_PORT=7771
|
||||||
|
# CTX_SIZE is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
|
||||||
|
# per conversation. MUST stay divisible by PARALLEL (the seam enforces it) and
|
||||||
|
# the harness contextWindow quotes the PER-SLOT number — a longer prompt is
|
||||||
|
# answered with a 400, never truncated.
|
||||||
|
CTX_SIZE=401408
|
||||||
|
PARALLEL=2
|
||||||
|
BATCH_SIZE=4096
|
||||||
|
UBATCH_SIZE=2048
|
||||||
|
CACHE_TYPE_K=f16
|
||||||
|
CACHE_TYPE_V=f16
|
||||||
|
SPEC_TYPE=none
|
||||||
|
NGL=99
|
||||||
|
# Log verbosity threshold (llama.cpp -lv). 4 = debug: KV layout, buffer sizes
|
||||||
|
# and graph capture land in the journal, which is what the VRAM-fit forensics
|
||||||
|
# read. Levels above 4 flood the journal.
|
||||||
|
VERBOSITY=4
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
[Unit]
|
||||||
|
Description=llama.cpp Qwen3.5-9B-Uncensored-HauhauCS-Aggressive Q6_K GGUF (qwen35 hybrid, 2x200704 slots, f16 K+V, MTP off, mmproj vision) on the RTX 3090 (dankclaw, frank-zealot)
|
||||||
|
After=network-online.target
|
||||||
|
Wants=network-online.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
User=dank
|
||||||
|
Group=dank
|
||||||
|
# CUDA pins its staging buffers, so the process needs the memlock budget even
|
||||||
|
# though the weights themselves are copied into VRAM.
|
||||||
|
LimitMEMLOCK=infinity
|
||||||
|
Environment="FRANK_ZEALOT_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-zealot.env"
|
||||||
|
ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-zealot.sh
|
||||||
|
Restart=on-failure
|
||||||
|
RestartSec=10
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
Executable
+91
@@ -0,0 +1,91 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# kld-gate-swift15.sh — RUNS ON DANKSERVER. Score the quantized candidate
|
||||||
|
# against the BF16 reference and apply the pre-approved gate:
|
||||||
|
# mean KL <= 0.03 AND top-1 agreement >= 95%.
|
||||||
|
# Same engine (1.0.13) and same boot shape as the reference RECORD
|
||||||
|
# (run-swift15-bf16.sh kld): disk-tier, no speculation, deterministic, fp8 KV.
|
||||||
|
# --kld-ref reads its tokens from the reference (no --kld-corpus here — the
|
||||||
|
# (run-swift15-bf16.sh kld): 1.0.13, tp 2, disk-tier, fp8 KV, no speculation,
|
||||||
|
# deterministic, max-model-len 16384, vram-kv 3072, 4 slots. Two deviations
|
||||||
|
# are forced by the w4 file: MNBT 2048 (the w4 kernels stream from the file
|
||||||
|
# tier ONLY at MNBT<=2048 — at the recording's 32768 the engine demands a
|
||||||
|
# widened form: "DID NOT FIT ... stored form is not the container's"), and
|
||||||
|
# pool 40960 (the recording's 12 GiB pool predates the w4 tables; at 12288 the
|
||||||
|
# gate_up slab cannot be cached). Chunk size is softmax-invariant, so the
|
||||||
|
# MNBT delta is float-order noise riding on the KL floor.
|
||||||
|
set -uo pipefail
|
||||||
|
PIN=sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
|
||||||
|
C=swift15gate
|
||||||
|
LOG=/home/dank/logs/swift15_kldgate.log
|
||||||
|
KLD_REF_DIR=/home/dank/corpora/kld-ref
|
||||||
|
KLD_OUT=/home/dank/radiance-state/kld-gate.json
|
||||||
|
|
||||||
|
[ -d "$KLD_REF_DIR" ] || { echo "missing reference: $KLD_REF_DIR"; exit 2; }
|
||||||
|
: > "$KLD_OUT"
|
||||||
|
cp -f /home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja \
|
||||||
|
/home/dank/models/tpl.jinja 2>/dev/null || true
|
||||||
|
sudo systemctl stop frank-archon >/dev/null 2>&1 || true
|
||||||
|
podman rm -f "$C" >/dev/null 2>&1 || true
|
||||||
|
|
||||||
|
podman run --rm --name "$C" \
|
||||||
|
--network host --init \
|
||||||
|
--security-opt seccomp=unconfined \
|
||||||
|
--ulimit memlock=-1:-1 \
|
||||||
|
--device /dev/kfd --device /dev/dri \
|
||||||
|
-v /home/dank/models:/models:ro \
|
||||||
|
-v /home/dank/radiance-state:/data \
|
||||||
|
-v "$KLD_REF_DIR:/data/kld-ref:ro" \
|
||||||
|
"stilldeadcode/radiance@$PIN" \
|
||||||
|
--model /models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad \
|
||||||
|
--tp 2 --placement expert_tiered --weights-disk-tier \
|
||||||
|
--host-pool-mib 40960 --deterministic --gpu-headroom-mib 768 \
|
||||||
|
--num-speculative-tokens 0 \
|
||||||
|
--kv-cache-dtype fp8 --max-model-len 16384 \
|
||||||
|
--ngram-placement ram \
|
||||||
|
--override-chat-template /models/tpl.jinja \
|
||||||
|
--kld-ref /data/kld-ref --kld-out /data/kld-gate.json \
|
||||||
|
--vram-kv-mib 3072 \
|
||||||
|
--max-num-seqs 4 --max-num-batched-tokens 2048 \
|
||||||
|
2>&1 | tee "$LOG"
|
||||||
|
rc=${PIPESTATUS[0]}
|
||||||
|
[ "$rc" -eq 0 ] || { echo "KLD GATE BOOT FAILED rc=$rc; log: $LOG"; exit "$rc"; }
|
||||||
|
|
||||||
|
python3 - "$KLD_OUT" "$LOG" <<'PY'
|
||||||
|
import json, sys, re
|
||||||
|
out, log = sys.argv[1], sys.argv[2]
|
||||||
|
def num(pat):
|
||||||
|
m = re.findall(pat, open(log, errors="replace").read(), re.I)
|
||||||
|
return float(m[-1]) if m else None
|
||||||
|
kl = top1 = None
|
||||||
|
try:
|
||||||
|
d = json.load(open(out))
|
||||||
|
def find(obj, keys):
|
||||||
|
if isinstance(obj, dict):
|
||||||
|
for k, v in obj.items():
|
||||||
|
if any(s in k.lower() for s in keys) and isinstance(v, (int, float)):
|
||||||
|
return float(v)
|
||||||
|
for v in obj.values():
|
||||||
|
r = find(v, keys)
|
||||||
|
if r is not None: return r
|
||||||
|
return None
|
||||||
|
kl = find(d, ("mean_kl", "mean kl", "kl_mean", "mean"))
|
||||||
|
top1 = find(d, ("top1", "top_1", "top-1", "agreement"))
|
||||||
|
if top1 is not None and top1 <= 1.0:
|
||||||
|
top1 *= 100.0
|
||||||
|
except Exception as e:
|
||||||
|
print("report JSON unusable (%s); falling back to log" % e)
|
||||||
|
if kl is None:
|
||||||
|
kl = num(r"mean[_ ]kl[^0-9]*([0-9]*\.?[0-9]+(?:e[-+][0-9]+)?)")
|
||||||
|
if top1 is None:
|
||||||
|
top1 = num(r"top[-_ ]?1[^0-9]*([0-9]*\.?[0-9]+)")
|
||||||
|
if kl is None or top1 is None:
|
||||||
|
print("KLD GATE PARSE FAILED: mean_kl=%s top1=%s — inspect %s / %s" % (kl, top1, out, log))
|
||||||
|
sys.exit(2)
|
||||||
|
print("KLD GATE: mean KL = %.4f (max 0.03) | top-1 = %.2f%% (min 95.0)" % (kl, top1))
|
||||||
|
ok = (kl <= 0.03) and (top1 >= 95.0)
|
||||||
|
print("KLD GATE " + ("PASS" if ok else "FAIL"))
|
||||||
|
sys.exit(0 if ok else 1)
|
||||||
|
PY
|
||||||
|
gate=$?
|
||||||
|
echo "kld-gate complete rc=$gate; log: $LOG report: $KLD_OUT"
|
||||||
|
exit "$gate"
|
||||||
Executable
+40
@@ -0,0 +1,40 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# kld-then-calib.sh — RUNS ON DANKSERVER. Campaign chain: wait for the KLD
|
||||||
|
# record to finish, verify it, then boot the 1.2.0 calibration server and
|
||||||
|
# push the 10M-token corpus. One GPU, so the phases serialize; prod is
|
||||||
|
# restored by each run script's trap.
|
||||||
|
set -u
|
||||||
|
KLOG=/home/dank/logs/swift15-kld-driver.log
|
||||||
|
CLOG=/home/dank/logs/swift15-calib-driver.log
|
||||||
|
S=/home/dank/serverAppConfigs-swift15/scripts
|
||||||
|
# Phase 1: KLD record (SKIP_KLD=1 when the record is already complete —
|
||||||
|
# kld.json's rows count is the receipt; a mid-run script swap can kill the
|
||||||
|
# wrapper with rc=127 after the engine exited clean).
|
||||||
|
if [ "${SKIP_KLD:-0}" != 1 ]; then
|
||||||
|
while ! grep -q "exited rc=" "$KLOG" 2>/dev/null; do sleep 60; done
|
||||||
|
RC=$(grep -o "exited rc=[0-9]*" "$KLOG" | tail -1)
|
||||||
|
if [ "$RC" != "exited rc=0" ]; then echo "KLD FAILED ($RC); calib NOT started"; exit 1; fi
|
||||||
|
fi
|
||||||
|
N=$(find /home/dank/corpora/kld-ref -type f | wc -l)
|
||||||
|
SZ=$(du -sm /home/dank/corpora/kld-ref | cut -f1)
|
||||||
|
echo "KLD OK files=$N size_mib=$SZ"
|
||||||
|
[ "$N" -ge 5 ] && [ "$SZ" -ge 1000 ] || { echo "kld-ref looks thin; calib NOT started"; exit 1; }
|
||||||
|
grep -q '"rows": 50235' /home/dank/corpora/kld-ref/kld.json || {
|
||||||
|
echo "kld.json rows != 50235; calib NOT started"; exit 1; }
|
||||||
|
|
||||||
|
# Phase 2: calibration serve (1.2.0) + corpus push.
|
||||||
|
nohup bash "$S/run-swift15-bf16.sh" calib > "$CLOG" 2>&1 &
|
||||||
|
CP=$!
|
||||||
|
for _ in $(seq 1 160); do
|
||||||
|
sleep 15
|
||||||
|
curl -sf localhost:7770/health >/dev/null 2>&1 && break
|
||||||
|
done
|
||||||
|
if ! curl -sf localhost:7770/health >/dev/null 2>&1; then
|
||||||
|
echo "CALIB BOOT FAILED; see /home/dank/logs/swift15_bf16.log"; kill "$CP" 2>/dev/null; exit 1
|
||||||
|
fi
|
||||||
|
echo "CALIB UP, pushing corpus"
|
||||||
|
python3 "$S/calib-push.py" --port 7770 --parallel 16 --container swift15ref --max-rounds 6
|
||||||
|
PUSH=$?
|
||||||
|
kill "$CP" 2>/dev/null
|
||||||
|
sleep 10
|
||||||
|
echo "CALIB_CHAIN_DONE push=$PUSH grams=$(find /home/dank/corpora/calib -name 'gram.r*.bin' 2>/dev/null | wc -l)"
|
||||||
Executable
+317
@@ -0,0 +1,317 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Build the two Swift1.5 campaign corpora (calibration + KLD).
|
||||||
|
|
||||||
|
calib.jsonl — 10M tokens of agent-shaped RAW text (45% code, 30% prose,
|
||||||
|
25% tool-call JSON), one {"prompt": text} per line, pushed to
|
||||||
|
/v1/completions with max_tokens=1 during the calibration run (the Hessian
|
||||||
|
tap is of the prompt tokens; docs/MOE-W4.md step 2). Code and prose mix
|
||||||
|
the repo checkout with HF dataset parquets (Frank's call 2026-10-06:
|
||||||
|
local material is only ~1.6M unique tokens; MOE-W4 wants 10M):
|
||||||
|
nampdn-ai/tiny-codes (code) and Salesforce/wikitext wikitext-103-raw
|
||||||
|
(prose). Tool calls are synthesized from commands harvested from the
|
||||||
|
repo's own scripts plus a fleet-ops set, with argument-shaped variants.
|
||||||
|
|
||||||
|
kld.jsonl — 1.8M tokens, one {"prompt", "score_from", "source"} per
|
||||||
|
line; each prompt is a multi-turn conversation rendered through the
|
||||||
|
Froggeric v22.5 template with a >= --kld-min-prompt (8192) token target,
|
||||||
|
and the last 256 tokens (the tail of the assistant answer) are the
|
||||||
|
scored span: score_from = n - 256. At 8k-token prompts the 1.8M corpus
|
||||||
|
is ~220 prompts = ~56k scored positions ~= 25 GB of record, matching
|
||||||
|
the published Flash-Next KLD scale (83k positions, MOE-W4.md). The KLD
|
||||||
|
reference run and every candidate run consume the SAME file (TOOLS.md
|
||||||
|
--kld-record / --kld-ref). The KLD corpus is LOCAL material only, so the
|
||||||
|
file-level disjointness rule (fnv1a64(path) % 5 == 0 -> KLD pool) is
|
||||||
|
untouched by the HF additions.
|
||||||
|
|
||||||
|
Deps: tokenizers, jinja2, pyarrow (venv /home/dank/venvs/radcorpus).
|
||||||
|
Usage: make-calib-corpus.py --repo ROOT --tokenizer-json PATH
|
||||||
|
--template PATH --out-dir DIR [--hf-code-parquet F,F]
|
||||||
|
[--hf-prose-parquet F,F] [--calib-tokens 10000000]
|
||||||
|
[--kld-tokens 1800000]
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import random
|
||||||
|
import re
|
||||||
|
|
||||||
|
from jinja2 import Environment, FileSystemLoader
|
||||||
|
from tokenizers import Tokenizer
|
||||||
|
|
||||||
|
CODE_EXT = (".sh", ".py", ".jinja", ".recipe", ".env", ".example",
|
||||||
|
".yaml", ".yml", ".bats", ".service", ".container", ".toml")
|
||||||
|
JSONL_EXTS = (".jsonl", ".json")
|
||||||
|
DOC_DIR = "serverAppConfigs-docs"
|
||||||
|
|
||||||
|
FLEET_CMDS = [
|
||||||
|
"systemctl status frank-archon --no-pager",
|
||||||
|
"journalctl -u frank-archon -n 100 --no-pager",
|
||||||
|
"podman ps --format '{{.Names}} {{.Status}}'",
|
||||||
|
"podman logs --tail 50 frank-archon",
|
||||||
|
"curl -s http://192.168.0.235:7770/health",
|
||||||
|
"curl -s http://192.168.0.235:7770/metrics | grep radiance_",
|
||||||
|
"df -h /home/dank | tail -1",
|
||||||
|
"free -g",
|
||||||
|
"sudo systemctl restart frank-archon",
|
||||||
|
"cd /home/dank/serverAppConfigs && git pull --ff-only",
|
||||||
|
"./scripts/deploy.sh dankserver --service frank-archon",
|
||||||
|
"./tests/run.sh 2>&1 | tail -5",
|
||||||
|
"jq . /home/dank/models/rad/index.json | head -40",
|
||||||
|
"python3 -c \"import json; print(json.load(open('config.json'))['num_hidden_layers'])\"",
|
||||||
|
"ip neigh show 192.168.0.235",
|
||||||
|
"amdgpu_top --device-info 2>/dev/null | head -20",
|
||||||
|
"grep -E 'MemAvailable' /proc/meminfo",
|
||||||
|
"sha256sum model-00001-of-00131.safetensors",
|
||||||
|
"git log --oneline -10",
|
||||||
|
"awk '{s+=$1} END {print s}' shard_sizes.txt",
|
||||||
|
]
|
||||||
|
TOOL_NAMES = ["bash", "read", "grep", "web_search", "edit", "write"]
|
||||||
|
WORKDIRS = ["/home/dank", "/home/dank/serverAppConfigs", "/home/dank/models",
|
||||||
|
"/etc/systemd/system", "/var/log"]
|
||||||
|
DESCS = ["check service state", "tail the engine ledger", "list containers",
|
||||||
|
"query health", "read metrics", "check disk", "check memory",
|
||||||
|
"restart the endpoint", "sync the checkout", "run deployer",
|
||||||
|
"run the test seams", "inspect the index", "probe the peer"]
|
||||||
|
|
||||||
|
|
||||||
|
def fnv(s: str) -> int:
|
||||||
|
h = 0xcbf29ce484222325
|
||||||
|
for b in s.encode():
|
||||||
|
h = ((h ^ b) * 0x100000001b3) & 0xFFFFFFFFFFFFFFFF
|
||||||
|
return h
|
||||||
|
|
||||||
|
|
||||||
|
def harvest_cmds(repo: str, rng: random.Random) -> list[str]:
|
||||||
|
cmds = list(FLEET_CMDS)
|
||||||
|
pat = re.compile(r"^\s*(?:sudo\s+)?([a-z][\w./-]*(?:\s+\S+){1,6})\s*$")
|
||||||
|
for root, dirs, files in os.walk(repo):
|
||||||
|
dirs[:] = [d for d in dirs if not d.startswith(".")]
|
||||||
|
for f in files:
|
||||||
|
if not f.endswith(".sh"):
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
for line in open(os.path.join(root, f), encoding="utf-8",
|
||||||
|
errors="replace"):
|
||||||
|
m = pat.match(line.rstrip())
|
||||||
|
if m and len(m.group(1)) > 12:
|
||||||
|
cmds.append("ssh dank@192.168.0.235 '"
|
||||||
|
+ m.group(1)[:90] + "'")
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
rng.shuffle(cmds)
|
||||||
|
return cmds[:4000]
|
||||||
|
|
||||||
|
|
||||||
|
def tool_call(rng: random.Random, cmds: list[str]) -> str:
|
||||||
|
"""One agent turn: 2-4 tool calls with argument-shaped JSON."""
|
||||||
|
parts = []
|
||||||
|
for _ in range(rng.randint(2, 4)):
|
||||||
|
args: dict = {"command": rng.choice(cmds)}
|
||||||
|
if rng.random() < 0.4:
|
||||||
|
args["cwd"] = rng.choice(WORKDIRS)
|
||||||
|
if rng.random() < 0.3:
|
||||||
|
args["timeout"] = rng.choice([10, 30, 60, 300, 600])
|
||||||
|
if rng.random() < 0.5:
|
||||||
|
args["description"] = rng.choice(DESCS)
|
||||||
|
parts.append('<tool_call>{"name": "%s", "arguments": %s}<|im_end|>'
|
||||||
|
% (rng.choice(TOOL_NAMES),
|
||||||
|
json.dumps(args, ensure_ascii=False)))
|
||||||
|
return "\n".join(parts)
|
||||||
|
|
||||||
|
|
||||||
|
def load_parquet_texts(path: str, kind: str, rng: random.Random,
|
||||||
|
cap: int) -> list[str]:
|
||||||
|
import pyarrow.parquet as pq
|
||||||
|
d = pq.read_table(path).to_pydict()
|
||||||
|
n = len(next(iter(d.values())))
|
||||||
|
idx = range(n) if n <= cap else rng.sample(range(n), cap)
|
||||||
|
out = []
|
||||||
|
for i in idx:
|
||||||
|
if kind == "code":
|
||||||
|
if "content" in d:
|
||||||
|
frag = d["content"][i]
|
||||||
|
else:
|
||||||
|
frag = ("// %s\n### Instruction:\n%s\n### Response:\n%s"
|
||||||
|
% (d.get("file_name", [""])[i], d["prompt"][i],
|
||||||
|
d["code"][i] or d["response"][i]))
|
||||||
|
else:
|
||||||
|
frag = d["text"][i]
|
||||||
|
if frag and frag.strip():
|
||||||
|
out.append(frag[:8000])
|
||||||
|
rng.shuffle(out)
|
||||||
|
return out[:cap]
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
ap = argparse.ArgumentParser(description=__doc__)
|
||||||
|
ap.add_argument("--repo", required=True)
|
||||||
|
ap.add_argument("--tokenizer-json", required=True)
|
||||||
|
ap.add_argument("--template", required=True)
|
||||||
|
ap.add_argument("--out-dir", required=True)
|
||||||
|
ap.add_argument("--calib-tokens", type=int, default=10_000_000)
|
||||||
|
ap.add_argument("--kld-tokens", type=int, default=1_800_000)
|
||||||
|
ap.add_argument("--chunk-tokens", type=int, default=8192)
|
||||||
|
ap.add_argument("--score-tail", type=int, default=256)
|
||||||
|
ap.add_argument("--kld-min-prompt", type=int, default=8192)
|
||||||
|
ap.add_argument("--hf-code-parquet", default="")
|
||||||
|
ap.add_argument("--hf-prose-parquet", default="")
|
||||||
|
ap.add_argument("--seed", type=int, default=20261006)
|
||||||
|
a = ap.parse_args()
|
||||||
|
|
||||||
|
rng = random.Random(a.seed)
|
||||||
|
tokenizer = Tokenizer.from_file(a.tokenizer_json)
|
||||||
|
env = Environment(loader=FileSystemLoader(os.path.dirname(a.template)),
|
||||||
|
trim_blocks=True, lstrip_blocks=True)
|
||||||
|
env.filters["tojson"] = lambda v: json.dumps(v, ensure_ascii=False)
|
||||||
|
env.filters["raise_exception"] = lambda m: (_ for _ in ()).throw(
|
||||||
|
ValueError(m))
|
||||||
|
tname = os.path.basename(a.template)
|
||||||
|
|
||||||
|
code, prose, jsonl = [], [], []
|
||||||
|
for root, dirs, files in os.walk(a.repo):
|
||||||
|
dirs[:] = [d for d in dirs if not d.startswith(".")]
|
||||||
|
for f in sorted(files):
|
||||||
|
p = os.path.relpath(os.path.join(root, f), a.repo)
|
||||||
|
if DOC_DIR in p and f.endswith(".md"):
|
||||||
|
prose.append(p)
|
||||||
|
elif p.startswith(DOC_DIR):
|
||||||
|
pass
|
||||||
|
elif f.endswith(CODE_EXT):
|
||||||
|
code.append(p)
|
||||||
|
elif f.endswith(JSONL_EXTS) and f != "package-lock.json":
|
||||||
|
jsonl.append(p)
|
||||||
|
# file-level split: bucket 0 -> KLD pool, 1..4 -> calibration pool
|
||||||
|
pools: dict[str, tuple[list, list]] = {}
|
||||||
|
for cls, lst in (("code", code), ("prose", prose), ("tools", jsonl)):
|
||||||
|
pools[cls] = ([p for p in lst if fnv(p) % 5 == 0],
|
||||||
|
[p for p in lst if fnv(p) % 5 != 0])
|
||||||
|
|
||||||
|
def text_of(p: str) -> str:
|
||||||
|
path = os.path.join(a.repo, p)
|
||||||
|
raw = open(path, encoding="utf-8", errors="replace").read()
|
||||||
|
if p.endswith(JSONL_EXTS):
|
||||||
|
lines = [json.loads(l)["content"] for l in raw.splitlines()
|
||||||
|
if l.strip() and "content" in l]
|
||||||
|
raw = "\n".join(lines)
|
||||||
|
toks = tokenizer.encode(raw, add_special_tokens=False).ids
|
||||||
|
return tokenizer.decode(toks[:2400], skip_special_tokens=True)
|
||||||
|
|
||||||
|
cmds = harvest_cmds(a.repo, rng)
|
||||||
|
# fragment pools per class, corpus-split first
|
||||||
|
frags: dict[str, dict[str, list[str]]] = {}
|
||||||
|
for cls in pools:
|
||||||
|
kld_files, cal_files = pools[cls]
|
||||||
|
fr = {}
|
||||||
|
src = {"kld": kld_files, "cal": cal_files}
|
||||||
|
if cls == "tools":
|
||||||
|
fr["kld"] = [tool_call(rng, cmds) for _ in range(600)]
|
||||||
|
fr["cal"] = [tool_call(rng, cmds) for _ in range(40000)]
|
||||||
|
else:
|
||||||
|
fr["kld"] = [text_of(p) for p in src["kld"]]
|
||||||
|
fr["cal"] = [text_of(p) for p in src["cal"]]
|
||||||
|
frags[cls] = fr
|
||||||
|
# per-class fragment caps: prose rows are short (~66 tok) so need many;
|
||||||
|
# codeparrot files are long (~2.6k tok) so a small cap suffices
|
||||||
|
hfcap = {"code": 40000, "prose": 60000, "tools": 0}
|
||||||
|
for cls, spec in (("code", a.hf_code_parquet),
|
||||||
|
("prose", a.hf_prose_parquet)):
|
||||||
|
for path in [p for p in spec.split(",") if p]:
|
||||||
|
frags[cls]["cal"].extend(
|
||||||
|
load_parquet_texts(path, cls, rng, cap=hfcap[cls]))
|
||||||
|
for cls in frags:
|
||||||
|
rng.shuffle(frags[cls]["cal"])
|
||||||
|
|
||||||
|
def toks(s: str) -> int:
|
||||||
|
return len(tokenizer.encode(s, add_special_tokens=False).ids)
|
||||||
|
|
||||||
|
def emit_calib() -> tuple[int, list[dict]]:
|
||||||
|
targets = {"code": .45, "prose": .30, "tools": .25}
|
||||||
|
rows: list[dict] = []
|
||||||
|
for cls, frac in targets.items():
|
||||||
|
want = int(a.calib_tokens * frac)
|
||||||
|
pool = frags[cls]["cal"]
|
||||||
|
if not pool:
|
||||||
|
continue
|
||||||
|
got, i, buf, nbuf = 0, 0, [], 0
|
||||||
|
while got < want:
|
||||||
|
frag = pool[i % len(pool)]
|
||||||
|
i += 1
|
||||||
|
n = toks(frag)
|
||||||
|
if nbuf + n > a.chunk_tokens and buf:
|
||||||
|
rows.append({"prompt": "\n".join(buf), "cls": cls})
|
||||||
|
got += nbuf
|
||||||
|
buf, nbuf = [], 0
|
||||||
|
if got >= want:
|
||||||
|
break
|
||||||
|
buf.append(frag)
|
||||||
|
nbuf += n
|
||||||
|
if i > len(pool) * 40:
|
||||||
|
break # pool exhausted with repeats; stop
|
||||||
|
if buf and got < want:
|
||||||
|
rows.append({"prompt": "\n".join(buf), "cls": cls})
|
||||||
|
got += nbuf
|
||||||
|
print(f"calib {cls}: {got} tokens, {i} fragments")
|
||||||
|
rng.shuffle(rows)
|
||||||
|
return len(rows), [{k: v for k, v in r.items() if k == "prompt"}
|
||||||
|
for r in rows]
|
||||||
|
|
||||||
|
def emit_kld() -> tuple[int, list[dict]]:
|
||||||
|
rows, total = [], 0
|
||||||
|
classes = ["code", "prose", "tools"]
|
||||||
|
i = {c: 0 for c in classes}
|
||||||
|
guard = 0
|
||||||
|
while total < a.kld_tokens and guard < 5000:
|
||||||
|
guard += 1
|
||||||
|
msgs, srcs = [], []
|
||||||
|
for t in range(rng.choice([2, 3, 4])):
|
||||||
|
cls = classes[(len(rows) + t) % 3]
|
||||||
|
pool = frags[cls]["kld"]
|
||||||
|
if not pool:
|
||||||
|
continue
|
||||||
|
frag = pool[i[cls] % len(pool)]
|
||||||
|
i[cls] += 2
|
||||||
|
msgs.append({"role": "user",
|
||||||
|
"content": "Summarize and explain this "
|
||||||
|
f"engineering material:\n{frag}"})
|
||||||
|
msgs.append({"role": "assistant",
|
||||||
|
"content": frag[:4000]})
|
||||||
|
srcs.append(cls)
|
||||||
|
rendered = env.get_template(tname).render(
|
||||||
|
messages=msgs, add_generation_prompt=False)
|
||||||
|
n = toks(rendered)
|
||||||
|
if n < a.kld_min_prompt:
|
||||||
|
continue
|
||||||
|
rows.append({"prompt": rendered,
|
||||||
|
"score_from": n - a.score_tail,
|
||||||
|
"source": "+".join(sorted(set(srcs)))})
|
||||||
|
total += n
|
||||||
|
return len(rows), rows
|
||||||
|
|
||||||
|
os.makedirs(a.out_dir, exist_ok=True)
|
||||||
|
nc, rows = emit_calib()
|
||||||
|
with open(os.path.join(a.out_dir, "calib.jsonl"), "w") as f:
|
||||||
|
for r in rows:
|
||||||
|
f.write(json.dumps(r) + "\n")
|
||||||
|
nk, krows = emit_kld()
|
||||||
|
with open(os.path.join(a.out_dir, "kld.jsonl"), "w") as f:
|
||||||
|
for r in krows:
|
||||||
|
f.write(json.dumps(r) + "\n")
|
||||||
|
scored = sum(len(tokenizer.encode(r["prompt"], add_special_tokens=False).ids)
|
||||||
|
- r["score_from"] for r in krows)
|
||||||
|
manifest = {
|
||||||
|
"seed": a.seed, "template": tname,
|
||||||
|
"tokenizer": os.path.basename(a.tokenizer_json),
|
||||||
|
"calib_prompts": nc, "kld_prompts": nk,
|
||||||
|
"kld_scored_positions": scored,
|
||||||
|
"kld_score_tail": a.score_tail,
|
||||||
|
"disjoint_file_pools": "local KLD material only; HF code feeds calib",
|
||||||
|
"calib_mix": {"code": .45, "prose": .30, "tools": .25},
|
||||||
|
"hf_code": a.hf_code_parquet, "hf_prose": a.hf_prose_parquet,
|
||||||
|
}
|
||||||
|
with open(os.path.join(a.out_dir, "manifest.json"), "w") as f:
|
||||||
|
json.dump(manifest, f, indent=1)
|
||||||
|
print(json.dumps(manifest, indent=1))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -2,7 +2,7 @@
|
|||||||
#
|
#
|
||||||
# STATUS: WIP — first of three patches in the "compile the GLM trunk" changelist. The
|
# STATUS: WIP — first of three patches in the "compile the GLM trunk" changelist. The
|
||||||
# set does NOT yet boot the engine (see 03 for the next blocker), so it is NOT part of
|
# set does NOT yet boot the engine (see 03 for the next blocker), so it is NOT part of
|
||||||
# the deployed frank-primary shape. Nothing in scripts/ mounts these files.
|
# the deployed frank-archon shape. Nothing in scripts/ mounts these files.
|
||||||
#
|
#
|
||||||
# WHY THIS IS THE RIGHT LEVER (measured 2026-09-30, not inferred):
|
# WHY THIS IS THE RIGHT LEVER (measured 2026-09-30, not inferred):
|
||||||
#
|
#
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# gpu_worker-kv-cap.patch — fork-local fix for the expert-offload KV cap.
|
# gpu_worker-kv-cap.patch — fork-local fix for the expert-offload KV cap.
|
||||||
#
|
#
|
||||||
# STATUS: NOT part of the deployed frank-primary shape. It exists because the
|
# STATUS: NOT part of the deployed frank-archon shape. It exists because the
|
||||||
# fork's plan under-models this model's KV, and it is the enabler for a future
|
# fork's plan under-models this model's KV, and it is the enabler for a future
|
||||||
# MTP attempt (the production GLM shape runs without MTP and boots fine on the
|
# MTP attempt (the production GLM shape runs without MTP and boots fine on the
|
||||||
# plan's own reserve).
|
# plan's own reserve).
|
||||||
|
|||||||
Executable
+201
@@ -0,0 +1,201 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""probe-frank-archon-kv.py — verify the seqs x max-model-len KV contract on a
|
||||||
|
frank-archon (radiance) service. Run it ON dankserver: it talks to
|
||||||
|
127.0.0.1:7770 and builds its prompts from the repo checkout there.
|
||||||
|
No scratch container, no mask: the service under test is the service.
|
||||||
|
|
||||||
|
ssh dankserver 'python3 /home/dank/serverAppConfigs/scripts/probe-frank-archon-kv.py'
|
||||||
|
|
||||||
|
Four probes, in increasing cost:
|
||||||
|
A decode at depth — one ~51k-token chat completion, 512 tokens, greedy.
|
||||||
|
Reports the engine's own `timings` (prefill, decode, draft acceptance).
|
||||||
|
The bench law applies: >=128 generated tokens, greedy, real text.
|
||||||
|
B the capacity contract — as many CONCURRENT chats of ~197k prompt tokens as
|
||||||
|
the deployed `FRANK_ARCHON_MAX_SEQS` (read from the repo env file on this
|
||||||
|
host), each carrying a unique marker at its head. All must answer their own
|
||||||
|
marker. This is what the endpoint owes: N full-height conversations
|
||||||
|
resident at once, exact retrieval in every one of them.
|
||||||
|
C prefix-cache restore — probe A's cold prompt again, so the disk tier has
|
||||||
|
to give the blocks back (cache_n > 0, TTFT collapses).
|
||||||
|
D concurrency decode — `seqs` CONCURRENT 512-token generations at ~25 % of
|
||||||
|
the window. This is the cost a slot count imposes on DECODE, which no
|
||||||
|
prefill number shows: the fused-step schedule is unfair, so every stream is
|
||||||
|
printed plus the aggregate. Measured 2026-10-13 at 3 x 200,000 fp8:
|
||||||
|
single-stream 99.0 t/s (43.5k) and 93.3 (197k) — decode is nearly
|
||||||
|
window-free — against 3-stream 113.9/43.3/10.9 t/s = 18.4 t/s aggregate.
|
||||||
|
Concurrency, not the slab, is what decode pays for slots. The 2026-10-13
|
||||||
|
shape hunt missed this for a whole day because leg B generates 24 tokens.
|
||||||
|
PROBE LAW (learned the hard way, twice): radiance routes the thinking block to
|
||||||
|
`reasoning_content`, so a chat request whose budget is spent inside
|
||||||
|
`<think>` returns EMPTY `content` with `finish_reason: stop`. A probe that
|
||||||
|
reads `content` alone scores 0/2 on a healthy server. Every request here sets
|
||||||
|
chat_template_kwargs.enable_thinking=false and reads both fields.
|
||||||
|
|
||||||
|
A/B method for a shape change (no scratch container): write a variant env file
|
||||||
|
on the host, point a transient systemd drop-in at it
|
||||||
|
(`Environment=FRANK_ARCHON_ENV_FILE=...` — drop-ins are read AFTER the unit's
|
||||||
|
own Environment= lines, so they win), restart, probe, delete the drop-in,
|
||||||
|
daemon-reload, restart. `systemctl mask` is not available: the unit file in
|
||||||
|
/etc/systemd/system is a relink symlink into the checkout.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import glob
|
||||||
|
import time
|
||||||
|
import urllib.request
|
||||||
|
import concurrent.futures
|
||||||
|
|
||||||
|
BASE = "http://127.0.0.1:7770"
|
||||||
|
MODEL = "frank-archon"
|
||||||
|
REPO = "/home/dank/serverAppConfigs"
|
||||||
|
GLOBS = [f"{REPO}/scripts/*.sh", f"{REPO}/serverAppConfigs-docs/**/*.md"]
|
||||||
|
|
||||||
|
|
||||||
|
def post(payload, timeout=1800):
|
||||||
|
req = urllib.request.Request(
|
||||||
|
BASE + "/v1/chat/completions",
|
||||||
|
data=json.dumps(payload).encode(),
|
||||||
|
headers={"Content-Type": "application/json"},
|
||||||
|
)
|
||||||
|
t0 = time.time()
|
||||||
|
with urllib.request.urlopen(req, timeout=timeout) as r:
|
||||||
|
body = json.loads(r.read().decode())
|
||||||
|
hdr = dict(r.headers.items())
|
||||||
|
return body, time.time() - t0, hdr
|
||||||
|
|
||||||
|
|
||||||
|
def get(path, timeout=60):
|
||||||
|
with urllib.request.urlopen(BASE + path, timeout=timeout) as r:
|
||||||
|
return json.loads(r.read().decode())
|
||||||
|
|
||||||
|
|
||||||
|
def corpus(min_chars):
|
||||||
|
files = []
|
||||||
|
for g in GLOBS:
|
||||||
|
files.extend(sorted(glob.glob(g, recursive=True)))
|
||||||
|
out, total = [], 0
|
||||||
|
for f in files:
|
||||||
|
try:
|
||||||
|
t = open(f, encoding="utf-8", errors="replace").read()
|
||||||
|
except OSError:
|
||||||
|
continue
|
||||||
|
out.append(t)
|
||||||
|
total += len(t)
|
||||||
|
if total > min_chars:
|
||||||
|
break
|
||||||
|
return "".join(text for text in out)
|
||||||
|
|
||||||
|
|
||||||
|
def prompt(text, chars, salt, fact):
|
||||||
|
"""Real repo text, salted so the prefix cache cannot answer, marker planted
|
||||||
|
near the head, question at the tail."""
|
||||||
|
head = f"\n[SALT {salt}] confidential marker: {fact}\n"
|
||||||
|
body = text + text
|
||||||
|
p = (head + body * max(1, chars // len(body)))[:chars]
|
||||||
|
return p + "\n\nQuestion: what is the confidential marker in the " \
|
||||||
|
"SALT block near the top of this document? Answer with the marker " \
|
||||||
|
"alone, no other words.\n"
|
||||||
|
|
||||||
|
|
||||||
|
def chat(text, chars, salt, fact, max_tokens):
|
||||||
|
return {"model": MODEL,
|
||||||
|
"messages": [{"role": "user",
|
||||||
|
"content": prompt(text, chars, salt, fact)}],
|
||||||
|
"max_tokens": max_tokens, "temperature": 0.0,
|
||||||
|
"chat_template_kwargs": {"enable_thinking": False}}
|
||||||
|
|
||||||
|
|
||||||
|
def show(tag, j, wall, expect=None):
|
||||||
|
tim, u = j.get("timings") or {}, j.get("usage", {})
|
||||||
|
ch = (j.get("choices") or [{}])[0]
|
||||||
|
msg = ch.get("message") or {}
|
||||||
|
txt = (msg.get("content") or msg.get("reasoning_content") or "").strip()
|
||||||
|
acc = (tim["draft_n_accepted"] / tim["draft_n"]) if tim.get("draft_n") else None
|
||||||
|
line = (f"[{tag}] prompt={u.get('prompt_tokens')} gen={u.get('completion_tokens')} "
|
||||||
|
f"wall={wall:.1f}s prefill={tim.get('prompt_per_second', 0):.0f} t/s "
|
||||||
|
f"decode={tim.get('predicted_per_second', 0):.1f} t/s "
|
||||||
|
f"draft={tim.get('draft_n')}/{tim.get('draft_n_accepted')} "
|
||||||
|
f"accept={acc if acc is None else round(acc, 4)} "
|
||||||
|
f"cache_n={tim.get('cache_n')} finish={ch.get('finish_reason')}")
|
||||||
|
if expect:
|
||||||
|
hit = expect.split("-")[0] in txt.lower()
|
||||||
|
line += f" expected={expect} answered={txt[:80]!r} HIT={hit}"
|
||||||
|
print(line)
|
||||||
|
|
||||||
|
|
||||||
|
def deployed_shape():
|
||||||
|
"""The contract is the deployed env, not a constant: slot count and window
|
||||||
|
come from the checkout on this host, so the probe cannot drift from the
|
||||||
|
service it is testing (the 2026-10-13 shape is 4 x 200,000; before that
|
||||||
|
2 x 200,000, and the fp8 era 3 x 200,000)."""
|
||||||
|
shape = {}
|
||||||
|
with open(f"{REPO}/scripts/frank-archon.env") as fh:
|
||||||
|
for line in fh:
|
||||||
|
line = line.strip()
|
||||||
|
if line.startswith("FRANK_ARCHON_") and "=" in line:
|
||||||
|
key, _, val = line.partition("=")
|
||||||
|
shape[key] = val.strip().strip('"')
|
||||||
|
return (int(shape["FRANK_ARCHON_MAX_SEQS"]),
|
||||||
|
int(shape["FRANK_ARCHON_MAX_MODEL_LEN"]),
|
||||||
|
shape["FRANK_ARCHON_KV_DTYPE"])
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
seqs, maxlen, dtype = deployed_shape()
|
||||||
|
print(f"shape: {seqs} x {maxlen:,} tokens, kv {dtype}")
|
||||||
|
print("health:", json.dumps(get("/health")))
|
||||||
|
print("models:", [m.get("id") for m in get("/v1/models").get("data", [])])
|
||||||
|
deep = corpus(160_000)
|
||||||
|
wide = corpus(1_100_000)
|
||||||
|
|
||||||
|
# 3.28 chars/token is the measured ratio of this corpus through the
|
||||||
|
# froggeric template (646,000 chars -> 196,919 tokens at the 2026-10-11
|
||||||
|
# deploy); 98.5 % of the window leaves the response room under the ceiling.
|
||||||
|
chars = int(0.985 * maxlen * 3.28)
|
||||||
|
# Each marker's FIRST hyphen-separated word is unique, because show()
|
||||||
|
# scores the hit on that word — a shared prefix would score every slot HIT
|
||||||
|
# on any answer that says the word.
|
||||||
|
words = ["cobalt", "quartz", "basalt", "ember", "willow", "cobweb", "flint"]
|
||||||
|
facts = {f"B{i+1}": f"{words[i % len(words)]}-flag-{(i + 1) * 137}"
|
||||||
|
for i in range(seqs)}
|
||||||
|
bodies = {s: chat(wide, chars, s, f, 24) for s, f in facts.items()}
|
||||||
|
print(f"[B] {len(bodies['B1']['messages'][0]['content']):,} chars per "
|
||||||
|
f"request, x{seqs} concurrent")
|
||||||
|
|
||||||
|
# A + C first (cheap), B last (the expensive set).
|
||||||
|
for tag, salt in (("A-cold", "A1-9137"), ):
|
||||||
|
j, wall, _ = post(chat(deep, 140_000, salt, "zephyr-lantern-7", 512))
|
||||||
|
show(tag, j, wall, "zephyr-lantern-7")
|
||||||
|
t0 = time.time()
|
||||||
|
with concurrent.futures.ThreadPoolExecutor(max_workers=seqs) as ex:
|
||||||
|
futs = {s: ex.submit(post, b) for s, b in bodies.items()}
|
||||||
|
for s, fut in futs.items():
|
||||||
|
try:
|
||||||
|
j, wall, _ = fut.result()
|
||||||
|
show(f"B-{s}", j, wall, facts[s])
|
||||||
|
except Exception as e: # a preemption/timeout lands here
|
||||||
|
print(f"[B-{s}] FAILED after {time.time() - t0:.1f}s: {e!r}")
|
||||||
|
print(f"[B] total {time.time() - t0:.1f}s")
|
||||||
|
# D — concurrency decode: seqs streams, 512 tokens each, ~25 % depth.
|
||||||
|
dchars = int(0.25 * maxlen * 3.28)
|
||||||
|
dbodies = [chat(deep, dchars, f"D{i + 1}-9137", f"flint-marker-{i}", 512)
|
||||||
|
for i in range(seqs)]
|
||||||
|
print(f"[D] {seqs} concurrent streams x 512 tokens, "
|
||||||
|
f"{dchars:,} chars each")
|
||||||
|
t0 = time.time()
|
||||||
|
with concurrent.futures.ThreadPoolExecutor(max_workers=seqs) as ex:
|
||||||
|
futs = [ex.submit(post, b) for b in dbodies]
|
||||||
|
for i, fut in enumerate(futs, 1):
|
||||||
|
try:
|
||||||
|
j, wall, _ = fut.result()
|
||||||
|
show(f"D-s{i}", j, wall)
|
||||||
|
except Exception as e: # a preemption/timeout lands here
|
||||||
|
print(f"[D-s{i}] FAILED after {time.time() - t0:.1f}s: {e!r}")
|
||||||
|
tot = time.time() - t0
|
||||||
|
print(f"[D] {seqs} x 512 tokens in {tot:.1f}s = "
|
||||||
|
f"{seqs * 512 / tot:.1f} t/s aggregate decode")
|
||||||
|
j, wall, _ = post(chat(deep, 140_000, "A1-9137", "zephyr-lantern-7", 512))
|
||||||
|
show("C-prefix-hit", j, wall, "zephyr-lantern-7")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,105 @@
|
|||||||
|
# qwen4exp-swift15-w4dn8-i8-hc8m.recipe -- the Swift 1.5 finetune of Qwen3.8-Flash-Next
|
||||||
|
# (ukisai/Swift1.5-Qwen3.8-Flash-Next, BF16 source, 335.3 GiB, key set identical to the official
|
||||||
|
# base) as a radiance container for frank-archon: EVERY routed expert -- gate_up AND down --
|
||||||
|
# through GPTQ into the w4nl64a8h table. The plugin's one-table-one-encoding law
|
||||||
|
# ("one table of experts takes one encoding, and only a whole expert kept plain bf16 is served
|
||||||
|
# beside it", qwen4exp_fp8 rad_arch_declare) forbids the original split tier (down block-fp8 +
|
||||||
|
# gate_up w4): the served kernel is w4nl64a8h for the whole expert set, bf16 experts beside it.
|
||||||
|
# The rest is qwen4exp-w5dn64-i8-hc8m.recipe verbatim: int8 trunk linears, block-fp8 hc mixers
|
||||||
|
# and MTP fc, the fixed-scale n-gram table, the 2-bit draft head.
|
||||||
|
#
|
||||||
|
# Why down is the calibrated half: the calibration tap rides the down projection, so its GPTQ
|
||||||
|
# rounding propagates against real down-input Hessians; gate_up shares the same w4 table
|
||||||
|
# uncalibrated. The published w5dn64 comparison (mean KL 0.0789 -> 0.0717, top-1 92.85% ->
|
||||||
|
# 93.34%, decode +4%, prefill -19%) is the 5-bit upper bound; this tier trades part of that
|
||||||
|
# quality back for prefill.
|
||||||
|
#
|
||||||
|
# CALIB=DIR rad-convert <Swift1.5-Qwen3.8-Flash-Next checkpoint> \
|
||||||
|
# --recipe scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe -o qwen4exp-swift15-w4dn8-i8-hc8m.rad
|
||||||
|
#
|
||||||
|
# DIR is the calibration directory the bf16 container wrote (RADIANCE_CALIB_DIR, 10M tokens of
|
||||||
|
# traffic-shaped corpus). Everything not named here is served as the checkpoint holds it (bf16:
|
||||||
|
# embeddings, the vision tower, the QSA indexer); the first rule that matches a weight decides it.
|
||||||
|
# The bf16 container is NOT passed with --reuse: the peak-disk arithmetic does not fit (checkpoint +
|
||||||
|
# container + container + output > purge-freed space), and --reuse only saves a re-read.
|
||||||
|
|
||||||
|
# The protected experts, cast to bf16 ahead of both expert rules. THE NAMES BELOW ARE THE BASE
|
||||||
|
# CHECKPOINT'S (from its 1M-token recording, qwen4exp-w5dn64-i8-hc8m.recipe) -- a warm start only.
|
||||||
|
# The Swift 1.5 finetune's routing distribution is not assumed equal: the derivation step rewrites
|
||||||
|
# this section from Swift's OWN calibration recording before this recipe is ever run, by the base's
|
||||||
|
# law -- an expert whose share of its layer's down-input energy is at least 3% and at least three
|
||||||
|
# times its share of the routing, an even count on each rank's half of the layer (the four-bit GEMM
|
||||||
|
# takes the rest as two equal tables by parity) by adding that half's next most energetic expert.
|
||||||
|
# PROTECTED-BEGIN (rewritten from Swift's OWN grams by derive-protected-experts.py)
|
||||||
|
blk.1.ffn_*_exps.271.weight cast dtype=bf16
|
||||||
|
blk.1.ffn_*_exps.355.weight cast dtype=bf16
|
||||||
|
blk.2.ffn_*_exps.279.weight cast dtype=bf16
|
||||||
|
blk.2.ffn_*_exps.413.weight cast dtype=bf16
|
||||||
|
blk.34.ffn_*_exps.146.weight cast dtype=bf16
|
||||||
|
blk.34.ffn_*_exps.153.weight cast dtype=bf16
|
||||||
|
blk.34.ffn_*_exps.407.weight cast dtype=bf16
|
||||||
|
blk.34.ffn_*_exps.453.weight cast dtype=bf16
|
||||||
|
blk.37.ffn_*_exps.288.weight cast dtype=bf16
|
||||||
|
blk.37.ffn_*_exps.377.weight cast dtype=bf16
|
||||||
|
blk.38.ffn_*_exps.406.weight cast dtype=bf16
|
||||||
|
blk.38.ffn_*_exps.420.weight cast dtype=bf16
|
||||||
|
blk.42.ffn_*_exps.37.weight cast dtype=bf16
|
||||||
|
blk.42.ffn_*_exps.90.weight cast dtype=bf16
|
||||||
|
blk.42.ffn_*_exps.302.weight cast dtype=bf16
|
||||||
|
blk.42.ffn_*_exps.388.weight cast dtype=bf16
|
||||||
|
blk.44.ffn_*_exps.315.weight cast dtype=bf16
|
||||||
|
blk.44.ffn_*_exps.350.weight cast dtype=bf16
|
||||||
|
blk.46.ffn_*_exps.290.weight cast dtype=bf16
|
||||||
|
blk.46.ffn_*_exps.419.weight cast dtype=bf16
|
||||||
|
blk.47.ffn_*_exps.119.weight cast dtype=bf16
|
||||||
|
blk.47.ffn_*_exps.143.weight cast dtype=bf16
|
||||||
|
blk.47.ffn_*_exps.399.weight cast dtype=bf16
|
||||||
|
blk.47.ffn_*_exps.445.weight cast dtype=bf16
|
||||||
|
# PROTECTED-END
|
||||||
|
# (derive-protected-experts.py rewrote this section from Swift's own grams)
|
||||||
|
|
||||||
|
# Every other routed expert, trunk and MTP layer alike: u4 codes into the w4nl table, rotated by
|
||||||
|
# the 128-wide Hadamard, an E4M3 scale a group of 64 under the fixed 2^-13 searched for least
|
||||||
|
# error as stored, chosen by GPTQ against $CALIB and swept three times by coordinate descent.
|
||||||
|
# The calibration Grams are down-input Hessians, so down is the calibrated half of the table;
|
||||||
|
# gate_up has no recording of its own and quantises uncalibrated (said on stderr, as the base
|
||||||
|
# recipe says). The MTP layer has no recording either. Expert mass: gate_up 150.00 GiB -> ~37.6,
|
||||||
|
# down 75.00 GiB -> ~18.8: ~56 GiB of experts, ~28 GiB a rank at TP=2 -- well inside the pool.
|
||||||
|
blk.*.ffn_*_exps.*.weight gptq table=w4nl group=64 scale=fp8_e4m3 scale2=f32 block2=*x* scale2_value=0.0001220703125 transform=fwht128 rule=search cd=3 calib=$CALIB
|
||||||
|
|
||||||
|
# Every connection's two mixing matrices -- the 96 trunk connections, the final mixer and the MTP
|
||||||
|
# head's three -- as E4M3 rows with an f32 scale a group of each row. A 128x128 block cannot cover
|
||||||
|
# them: lowrank is 320, 2.5 blocks. Down is [320, 10240] and takes a scale every 128 columns; up is
|
||||||
|
# [10240, 320] and takes four scales a row, 80 columns each.
|
||||||
|
*_hc_down.weight rtn codes=fp8_e4m3 group=128 scale=f32
|
||||||
|
*_hc_up.weight rtn codes=fp8_e4m3 group=80 scale=f32
|
||||||
|
|
||||||
|
# The MTP head's fc_hidden and fc_embedding, the same rows. They only ever change the drafts.
|
||||||
|
mtp.fc_hidden.weight rtn codes=fp8_e4m3 group=128 scale=f32
|
||||||
|
mtp.fc_embedding.weight rtn codes=fp8_e4m3 group=128 scale=f32
|
||||||
|
|
||||||
|
# The trunk's block linears as int8, a scale per 128 columns of a row searched for least error.
|
||||||
|
# The QSA indexer's projection is not one of them: its top-k is a hard choice between blocks, and
|
||||||
|
# keeping it as the checkpoint holds it costs 20 MB.
|
||||||
|
blk.*.attn_qg.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
|
||||||
|
blk.*.attn_k.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
|
||||||
|
blk.*.attn_v.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
|
||||||
|
blk.*.attn_output.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
|
||||||
|
blk.*.ssm_inz.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
|
||||||
|
blk.*.ssm_out.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
|
||||||
|
blk.*.ffn_gate_up_shexp.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
|
||||||
|
blk.*.ffn_down_shexp.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
|
||||||
|
|
||||||
|
# The lm_head, int8 with a bf16 scale a row per 128 columns (logits_gemm_i8): at E4M3 it costs
|
||||||
|
# ~0.6 points of top-1 on its own.
|
||||||
|
output.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
|
||||||
|
|
||||||
|
# The n-gram table at one fixed E4M3 scale; qwen4exp-w4.recipe says where the value comes from.
|
||||||
|
# The value sits on a three-octave plateau (2^-13..2^-10, rel_l2 0.02662) with 1.7x headroom over
|
||||||
|
# the base table's absmax; the prep phase verifies Swift's PLE absmax lands inside the plateau
|
||||||
|
# before this rule is trusted.
|
||||||
|
blk.*.ple_ngram.weight rtn codes=fp8_e4m3 block=*x* scale=bf16 rule=fixed scale_value=1.99317932128906e-4
|
||||||
|
|
||||||
|
# The MTP head's 2-bit draft head. INERT for this checkpoint: the HF weight map has no
|
||||||
|
# mtp.draft_head.weight (same as the base), kept for shape parity with the family.
|
||||||
|
mtp.draft_head.weight rtn codes=u2 zero=u8 group=128 scale=f16
|
||||||
Executable
+116
@@ -0,0 +1,116 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# run-swift15-bf16.sh — RUNS ON DANKSERVER. Boot the no-recipe bf16 Swift1.5
|
||||||
|
# container for the two offline campaign phases. The engine owns the GPUs, so
|
||||||
|
# the production unit must be stopped first (this script does it and restores
|
||||||
|
# on exit, autoresearch pattern: stop, never mask — relinked unit).
|
||||||
|
#
|
||||||
|
# kld --kld-record the reference logits from the bf16 model
|
||||||
|
# (runs the corpus and EXITS; engine writes DIR and leaves)
|
||||||
|
# calib serve with the calibration tap; a separate driver
|
||||||
|
# (scripts/calib-push.py) pushes the corpus; the engine writes the
|
||||||
|
# Hessians at RADIANCE_CALIB_TOKENS and the tap switches itself off.
|
||||||
|
# Stop this script after calib-push prints TAP_DONE.
|
||||||
|
#
|
||||||
|
# Shape = the vendor's bf16 calibration lines (GUIDE.md §4.3:528 + MOE-W4.md
|
||||||
|
# "calibrate the checkpoint as it ships"): expert_tiered + weights-disk-tier,
|
||||||
|
# no speculation, deterministic. Large prefill steps because a disk-tier step
|
||||||
|
# costs the same at 1 token as at 32K. The container declared --max-ctx 4096;
|
||||||
|
# --max-model-len 16384 exceeds it on purpose (the KLD prompts run to 12,007
|
||||||
|
# tokens, calib to 8,242) — the boot ledger arbitrates the QSA buffers; if
|
||||||
|
# declare refuses, re-convert with a wider --max-ctx.
|
||||||
|
set -uo pipefail
|
||||||
|
|
||||||
|
MODE="${1:?usage: run-swift15-bf16.sh kld|calib}"
|
||||||
|
PIN_1013=sha256:4c3f484564bb2872784e46b0a59cbd71becc9c1e746e301a3596ffae253c322e
|
||||||
|
PIN_120=sha256:665775ed1d90de6dbb25558e07fc60d20328f1346f96f3c6e00845eb763f01fa
|
||||||
|
RAD=/home/dank/models/rad/qwen4exp-swift15-bf16.rad
|
||||||
|
C=swift15ref
|
||||||
|
LOG=/home/dank/logs/swift15_bf16.log
|
||||||
|
KLD_REF_DIR=/home/dank/corpora/kld-ref
|
||||||
|
CALIB_DIR=/home/dank/corpora/calib
|
||||||
|
KLD_CORPUS=/home/dank/corpora/out/kld.jsonl
|
||||||
|
|
||||||
|
restore() { sudo systemctl start frank-archon >/dev/null 2>&1 || true; }
|
||||||
|
trap restore EXIT
|
||||||
|
sudo systemctl stop frank-archon >/dev/null 2>&1 || true
|
||||||
|
podman rm -f "$C" >/dev/null 2>&1 || true
|
||||||
|
|
||||||
|
ARGS=(--model /models/rad/qwen4exp-swift15-bf16.rad
|
||||||
|
--tp 2 --placement expert_tiered --weights-disk-tier
|
||||||
|
--host-pool-mib 12288 --deterministic --gpu-headroom-mib 768
|
||||||
|
--num-speculative-tokens 0
|
||||||
|
--kv-cache-dtype fp8 --max-model-len 16384
|
||||||
|
--ngram-placement ram
|
||||||
|
--override-chat-template /models/tpl.jinja)
|
||||||
|
|
||||||
|
case "$MODE" in
|
||||||
|
kld)
|
||||||
|
# 1.0.13: the engine that will serve the candidate — the reference is
|
||||||
|
# recorded on the same engine that scores it. The weights slab DERIVES as
|
||||||
|
# card - headroom - kv, and the disk tier refuses under 9.57 GiB (4.21 of
|
||||||
|
# layer buffers + 5.36 fixed), so kv is explicit SMALL: recording is
|
||||||
|
# prefill-only, self-batched at MNBT 32768 = two 16k prompts at a time;
|
||||||
|
# 4 slots x 16k fp8 (29 KiB/pos, 1.7 GiB) + headroom fits 3072.
|
||||||
|
IMAGE="stilldeadcode/radiance@$PIN_1013"
|
||||||
|
mkdir -p "$KLD_REF_DIR"
|
||||||
|
ARGS+=(--kld-record /data/kld-ref --kld-corpus /data/kld.jsonl
|
||||||
|
--vram-kv-mib 3072
|
||||||
|
--max-num-seqs 4 --max-num-batched-tokens 32768)
|
||||||
|
;;
|
||||||
|
calib)
|
||||||
|
# 1.2.0: the calibration tap (RADIANCE_CALIB_DIR + RADIANCE_CALIB_TOKENS)
|
||||||
|
# lives in the 1.2.0 qwen4exp plugin; 1.0.13 has only the DIR half with an
|
||||||
|
# unparameterisable threshold. Grams are bf16 activations — engine-invariant.
|
||||||
|
# Same KV ceiling as kld: the disk-tier weights floor (9.57 GiB) plus
|
||||||
|
# ~18 GiB of fixed per-rank claims cap the pool near 3.5 GiB on a 32 GiB
|
||||||
|
# card — pool minus the 2.17 GiB fixed holds ~237k fp8 tokens a rank, so
|
||||||
|
# the vendor's big-step calibration shape is affordable: MOE-W4.md:244
|
||||||
|
# measures 640 tok/s at a 32k step against 930 at 55k, and the doc's own
|
||||||
|
# run drives MNBT 65536. 8 slots x 16k = 131k resident tokens fits the
|
||||||
|
# pool; the step batches 65k. KLD's observed rate confirmed step size,
|
||||||
|
# not slot count, is the throughput lever.
|
||||||
|
IMAGE="stilldeadcode/radiance@$PIN_120"
|
||||||
|
mkdir -p "$CALIB_DIR"
|
||||||
|
ARGS+=(--port 7770 --vram-kv-mib 3072
|
||||||
|
--max-num-seqs 8 --max-num-batched-tokens 65536
|
||||||
|
--prefix-cache-dir /data/calib-kvcache)
|
||||||
|
;;
|
||||||
|
probe)
|
||||||
|
IMAGE="stilldeadcode/radiance@$PIN_1013"
|
||||||
|
ARGS+=(--port 7770 --max-num-seqs 8 --max-num-batched-tokens 4096)
|
||||||
|
;;
|
||||||
|
*) echo "unknown mode: $MODE" >&2; exit 2 ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# The prod template copy, readable from the container: mount the models tree
|
||||||
|
# ro plus the state dir rw (KLD record + prefix tier land there).
|
||||||
|
cp -f /home/dank/serverAppConfigs/scripts/qwen38-froggeric-v22.5.jinja \
|
||||||
|
/home/dank/models/tpl.jinja 2>/dev/null || true
|
||||||
|
|
||||||
|
KLD_MOUNT=()
|
||||||
|
if [ "$MODE" = kld ]; then
|
||||||
|
# Nested under the /data mount, so these must come after it (line order
|
||||||
|
# in podman run below): the corpus read-only, the record dir writable.
|
||||||
|
KLD_MOUNT=(-v "$KLD_CORPUS:/data/kld.jsonl:ro"
|
||||||
|
-v "$KLD_REF_DIR:/data/kld-ref")
|
||||||
|
fi
|
||||||
|
EXTRA_ENV=()
|
||||||
|
if [ "$MODE" = calib ]; then
|
||||||
|
EXTRA_ENV=(-v "$CALIB_DIR:/data/calib"
|
||||||
|
-e RADIANCE_CALIB_DIR=/data/calib
|
||||||
|
-e RADIANCE_CALIB_TOKENS=10000000)
|
||||||
|
fi
|
||||||
|
|
||||||
|
podman run --rm --name "$C" \
|
||||||
|
--network host --init \
|
||||||
|
--security-opt seccomp=unconfined \
|
||||||
|
--ulimit memlock=-1:-1 \
|
||||||
|
--device /dev/kfd --device /dev/dri \
|
||||||
|
-v /home/dank/models:/models:ro \
|
||||||
|
-v /home/dank/radiance-state:/data \
|
||||||
|
"${KLD_MOUNT[@]}" \
|
||||||
|
"${EXTRA_ENV[@]}" \
|
||||||
|
"$IMAGE" "${ARGS[@]}" 2>&1 | tee "$LOG"
|
||||||
|
rc=${PIPESTATUS[0]}
|
||||||
|
echo "swift15-bf16 $MODE exited rc=$rc; log: $LOG"
|
||||||
|
exit "$rc"
|
||||||
Executable
+198
@@ -0,0 +1,198 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# serve-frank-archon.sh — frank-archon endpoint: radiance (StillDeadcode,
|
||||||
|
# Apache-2.0) serving the PUBLISHER's 4-bit-codebook container of the
|
||||||
|
# official Qwen/Qwen3.8-Flash-Next (the engine author's own rad-convert of
|
||||||
|
# the bf16 checkpoint: qwen3.8-next-flash-fp8-iq4r-moe.rad, fetched by
|
||||||
|
# scripts/download-frank-archon-iq4r.sh) on BOTH R9700s (TP=2, wht6 wire),
|
||||||
|
# dankserver :7770. 2026-10-15 payload swap from the in-house w4dn8 of the
|
||||||
|
# Swift1.5 finetune (payload on disk = rollback; the swap commit is the
|
||||||
|
# revert); 2026-10-05 radiance cutover from FreeToken (git tag
|
||||||
|
# frank-archon-freetoken-pr587-20261005 is the pre-radiance state).
|
||||||
|
#
|
||||||
|
# The run line below IS the engine argv: the image's entrypoint is the
|
||||||
|
# engine and its tools sit beside it on PATH. Unlike every predecessor
|
||||||
|
# there is no per-boot VRAM planner to second-guess: the boot ledger in the
|
||||||
|
# log states every budget before anything is allocated (weights vs KV
|
||||||
|
# split, the expert host pool's coverage of the worst case, the PLE pin,
|
||||||
|
# the P2P answer per pair) — read it after every resize.
|
||||||
|
# Deployed shape (2026-10-13): THREE decode slots x 200,000 tokens, fp8
|
||||||
|
# (quantized) KV in a 5,120 MiB pool (contract 3 x 200,000 + 1.3 %) and a
|
||||||
|
# 25,600 MiB resident expert slab — 30,720 MiB of the 30,969 MiB the engine's
|
||||||
|
# own budget table makes claimable, spent on the slab because prefill is
|
||||||
|
# slab-limited. 44 GiB/rank pinned expert pool, MTP depth 2, prefix cache host
|
||||||
|
# 8 GiB + disk 64 GiB, PLE pinned in RAM.
|
||||||
|
#
|
||||||
|
# The root unit (scripts/frank-archon.service, User=dank,
|
||||||
|
# LimitMEMLOCK=infinity) grants the pin budget: 2 x 44 GiB expert pools +
|
||||||
|
# the 47.7 GiB PLE table + 8 GiB prefix host tier — ~147 GiB of pinned
|
||||||
|
# pages, which is ALSO why this script refuses to start when ttm's
|
||||||
|
# pages_limit/pages_max cap is under FRANK_ARCHON_TTM_MIN_MIB (see below).
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
ENV_FILE="${FRANK_ARCHON_ENV_FILE:-$SCRIPT_DIR/frank-archon.env}"
|
||||||
|
|
||||||
|
if [ ! -f "$ENV_FILE" ]; then
|
||||||
|
echo "ERROR: frank-archon env file absent: $ENV_FILE" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
set -a
|
||||||
|
# shellcheck disable=SC1090
|
||||||
|
source "$ENV_FILE"
|
||||||
|
set +a
|
||||||
|
|
||||||
|
: "${FRANK_ARCHON_IMAGE:?FRANK_ARCHON_IMAGE must name the radiance image}"
|
||||||
|
: "${FRANK_ARCHON_MODEL_DIR:?FRANK_ARCHON_MODEL_DIR must name the .rad dir}"
|
||||||
|
: "${FRANK_ARCHON_MODEL_FILE:?FRANK_ARCHON_MODEL_FILE must name the .rad file}"
|
||||||
|
: "${FRANK_ARCHON_STATE_DIR:?FRANK_ARCHON_STATE_DIR must name the /data state dir}"
|
||||||
|
: "${FRANK_ARCHON_CONTAINER:?FRANK_ARCHON_CONTAINER must name the container}"
|
||||||
|
: "${FRANK_ARCHON_SERVED_NAME:?FRANK_ARCHON_SERVED_NAME must name the alias}"
|
||||||
|
: "${FRANK_ARCHON_PORT:?FRANK_ARCHON_PORT must be the port}"
|
||||||
|
: "${FRANK_ARCHON_TEMPLATE:?FRANK_ARCHON_TEMPLATE must name the chat template}"
|
||||||
|
: "${FRANK_ARCHON_TP:?FRANK_ARCHON_TP must be set}"
|
||||||
|
: "${FRANK_ARCHON_TP_WIRE:?FRANK_ARCHON_TP_WIRE must be set}"
|
||||||
|
: "${FRANK_ARCHON_MAX_SEQS:?FRANK_ARCHON_MAX_SEQS must be set}"
|
||||||
|
: "${FRANK_ARCHON_MAX_MODEL_LEN:?FRANK_ARCHON_MAX_MODEL_LEN must be set}"
|
||||||
|
: "${FRANK_ARCHON_PLACEMENT:?FRANK_ARCHON_PLACEMENT must be set}"
|
||||||
|
: "${FRANK_ARCHON_HOST_POOL_MIB:?FRANK_ARCHON_HOST_POOL_MIB must be set}"
|
||||||
|
: "${FRANK_ARCHON_GPU_HEADROOM_MIB:?FRANK_ARCHON_GPU_HEADROOM_MIB must be set}"
|
||||||
|
: "${FRANK_ARCHON_VRAM_WEIGHTS_MIB:?FRANK_ARCHON_VRAM_WEIGHTS_MIB must be set}"
|
||||||
|
: "${FRANK_ARCHON_VRAM_KV_MIB:?FRANK_ARCHON_VRAM_KV_MIB must be set}"
|
||||||
|
: "${FRANK_ARCHON_KV_DTYPE:?FRANK_ARCHON_KV_DTYPE must be set}"
|
||||||
|
: "${FRANK_ARCHON_SPEC_TOKENS:?FRANK_ARCHON_SPEC_TOKENS must be set}"
|
||||||
|
: "${FRANK_ARCHON_MAX_BATCHED_TOKENS:?FRANK_ARCHON_MAX_BATCHED_TOKENS must be set}"
|
||||||
|
: "${FRANK_ARCHON_PREFIX_HOST_MIB:?FRANK_ARCHON_PREFIX_HOST_MIB must be set}"
|
||||||
|
: "${FRANK_ARCHON_PREFIX_DISK_MIB:?FRANK_ARCHON_PREFIX_DISK_MIB must be set}"
|
||||||
|
: "${FRANK_ARCHON_NGRAM_PLACEMENT:?FRANK_ARCHON_NGRAM_PLACEMENT must be set}"
|
||||||
|
: "${FRANK_ARCHON_TTM_MIN_MIB:?FRANK_ARCHON_TTM_MIN_MIB must be set}"
|
||||||
|
: "${FRANK_ARCHON_TTM_TARGET_MIB:?FRANK_ARCHON_TTM_TARGET_MIB must be set}"
|
||||||
|
: "${FRANK_ARCHON_RAD_MIN_MIB:?FRANK_ARCHON_RAD_MIN_MIB must be set}"
|
||||||
|
|
||||||
|
# The image gate is the digest-pinned ref itself: a name@sha256 ref exists
|
||||||
|
# in storage only if THAT manifest was pulled — a retagged look-alike or a
|
||||||
|
# newer latest does not satisfy it. `podman image exists` takes a
|
||||||
|
# name@digest ref in one command — no pipefail/grep race (the 2026-09-30
|
||||||
|
# lesson). Runs as dank: the unit's process uses dank's ROOTLESS podman
|
||||||
|
# storage (root unit, User=dank — the fleet's frank-archon shape).
|
||||||
|
if ! podman image exists "$FRANK_ARCHON_IMAGE"; then
|
||||||
|
echo "ERROR: radiance image absent: $FRANK_ARCHON_IMAGE" >&2
|
||||||
|
echo "frank-archon: podman pull docker://stilldeadcode/radiance:latest (as dank)" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# The payload gate: the .rad container file, sized. A truncated convert
|
||||||
|
# would otherwise buy a guaranteed long-timeout boot instead of this fast
|
||||||
|
# failure. The staging dir / raw checkpoint are NOT this script's concern
|
||||||
|
# (scripts/build-frank-archon-rad.sh built the file; its rad-info output
|
||||||
|
# is the record).
|
||||||
|
RAD="$FRANK_ARCHON_MODEL_DIR/$FRANK_ARCHON_MODEL_FILE"
|
||||||
|
if [ ! -f "$RAD" ]; then
|
||||||
|
echo "ERROR: .rad container missing: $RAD" >&2
|
||||||
|
echo "frank-archon: run scripts/build-frank-archon-rad.sh" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
RAD_MIB=$(( $(stat -c %s "$RAD") / 1048576 ))
|
||||||
|
if [ "$RAD_MIB" -lt "$FRANK_ARCHON_RAD_MIN_MIB" ]; then
|
||||||
|
echo "ERROR: .rad container too small: $RAD is ${RAD_MIB} MiB, want >= $FRANK_ARCHON_RAD_MIN_MIB" >&2
|
||||||
|
echo "frank-archon: the convert was truncated — rebuild it" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [ ! -f "$FRANK_ARCHON_TEMPLATE" ]; then
|
||||||
|
echo "ERROR: chat template missing: $FRANK_ARCHON_TEMPLATE" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
mkdir -p "$FRANK_ARCHON_STATE_DIR/kvcache/frank-archon"
|
||||||
|
# The engine's create is NOT recursive: with only the state dir mounted, its
|
||||||
|
# "cannot create /data/kvcache/frank-archon: No such file or directory"
|
||||||
|
# left the SSD prefix tier off (measured 2026-10-06 boot ledger). Pre-create
|
||||||
|
# the full path on the host so the tier opens on first boot.
|
||||||
|
|
||||||
|
mkdir -p "$FRANK_ARCHON_STATE_DIR/tune"
|
||||||
|
# rad-tune's kernel-selection cache (see the runbook's "Tuning levers"):
|
||||||
|
# the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune at boot;
|
||||||
|
# $RADIANCE_HOME defaults to /opt/radiance, which lives in the image, so
|
||||||
|
# the cache must ride a host mount or it dies with the --rm container
|
||||||
|
# that wrote it. The dir is pre-created for the same reason as kvcache
|
||||||
|
# (the engine's create is not recursive).
|
||||||
|
|
||||||
|
if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
|
||||||
|
echo "ERROR: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
# TTM preflight: the pinned ledger (2 x FRANK_ARCHON_HOST_POOL_MIB + the
|
||||||
|
# 47.7 GiB PLE table + the prefix host tier, ~147 GiB) rides TTM's shared
|
||||||
|
# pages, capped by ttm's pages_limit (the current kernel exposes NO
|
||||||
|
# pages_max; the cap function takes the min over the knobs that exist).
|
||||||
|
# Below FRANK_ARCHON_TTM_MIN_MIB the load dies mid-pin with an allocator
|
||||||
|
# error, so the gate refuses first — and RAISES the knob before refusing:
|
||||||
|
# the cap is a runtime knob whose default is half of RAM, so a power failure
|
||||||
|
# resets it (measured 2026-10-12 23:17 on this host: 200 GiB -> 125,988 MiB,
|
||||||
|
# and Restart=always re-ran the refusal 6,919 times over ~20 h).
|
||||||
|
# FRANK_ARCHON_TTM_TARGET_MIB is the recorded host ceiling, not the floor.
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "$SCRIPT_DIR/ttm-preflight.sh"
|
||||||
|
ttm_ensure_cap || exit 1
|
||||||
|
|
||||||
|
# A crashed/killed unit can leave the container name registered (systemd
|
||||||
|
# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
|
||||||
|
# identities (the FreeToken-era frank-archon-qwen, the retired R9V
|
||||||
|
# container and the ad hoc experiment containers all hold this GPU); all
|
||||||
|
# idempotent.
|
||||||
|
for CTR in $FRANK_ARCHON_LEGACY_CONTAINERS "$FRANK_ARCHON_CONTAINER"; do
|
||||||
|
podman rm -f "$CTR" >/dev/null 2>&1 || true
|
||||||
|
done
|
||||||
|
|
||||||
|
LOG=/home/dank/logs/frank_archon.log
|
||||||
|
mkdir -p "$(dirname "$LOG")"
|
||||||
|
|
||||||
|
# Publisher-vetted container contract (docs/DOCKER.md, compose/common.yaml):
|
||||||
|
# --network host (the server binds the host's addresses directly, no NAT),
|
||||||
|
# --init (the engine releases the cards on SIGTERM), seccomp=unconfined
|
||||||
|
# (Docker/podman's default profile refuses io_uring, which the n-gram
|
||||||
|
# table's reads go through, and the NUMA binding calls the ROCm runtime
|
||||||
|
# makes for host memory), memlock unbounded (the pinned ledger — legal
|
||||||
|
# because the root unit grants LimitMEMLOCK=infinity to this dank process),
|
||||||
|
# /dev/kfd + /dev/dri (render/video groups are moot for a root container).
|
||||||
|
# The template rides a single-file mount; the .rad dir is ro; /data holds
|
||||||
|
# the prefix-cache disk tier.
|
||||||
|
podman run -d --name "$FRANK_ARCHON_CONTAINER" \
|
||||||
|
--network host \
|
||||||
|
--init \
|
||||||
|
--security-opt seccomp=unconfined \
|
||||||
|
--ulimit memlock=-1:-1 \
|
||||||
|
--device /dev/kfd --device /dev/dri \
|
||||||
|
-v "$FRANK_ARCHON_MODEL_DIR":/models:ro \
|
||||||
|
-v "$FRANK_ARCHON_STATE_DIR":/data \
|
||||||
|
-v "$FRANK_ARCHON_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
|
||||||
|
-v "$FRANK_ARCHON_TEMPLATE":/templates/template.jinja:ro \
|
||||||
|
"$FRANK_ARCHON_IMAGE" \
|
||||||
|
--model "/models/$FRANK_ARCHON_MODEL_FILE" \
|
||||||
|
--host :: --port "$FRANK_ARCHON_PORT" \
|
||||||
|
--served-model-name "$FRANK_ARCHON_SERVED_NAME" \
|
||||||
|
--override-chat-template /templates/template.jinja \
|
||||||
|
--tp "$FRANK_ARCHON_TP" --tp-wire "$FRANK_ARCHON_TP_WIRE" \
|
||||||
|
--max-num-seqs "$FRANK_ARCHON_MAX_SEQS" \
|
||||||
|
--max-model-len "$FRANK_ARCHON_MAX_MODEL_LEN" \
|
||||||
|
--placement "$FRANK_ARCHON_PLACEMENT" \
|
||||||
|
--host-pool-mib "$FRANK_ARCHON_HOST_POOL_MIB" \
|
||||||
|
--gpu-headroom-mib "$FRANK_ARCHON_GPU_HEADROOM_MIB" \
|
||||||
|
--vram-weights-mib "$FRANK_ARCHON_VRAM_WEIGHTS_MIB" \
|
||||||
|
--vram-kv-mib "$FRANK_ARCHON_VRAM_KV_MIB" \
|
||||||
|
--kv-cache-dtype "$FRANK_ARCHON_KV_DTYPE" \
|
||||||
|
--prefix-cache-host-mib "$FRANK_ARCHON_PREFIX_HOST_MIB" \
|
||||||
|
--prefix-cache-dir /data/kvcache/frank-archon \
|
||||||
|
--prefix-cache-disk-mib "$FRANK_ARCHON_PREFIX_DISK_MIB" \
|
||||||
|
--ngram-placement "$FRANK_ARCHON_NGRAM_PLACEMENT" \
|
||||||
|
--num-speculative-tokens "$FRANK_ARCHON_SPEC_TOKENS" \
|
||||||
|
--max-num-batched-tokens "$FRANK_ARCHON_MAX_BATCHED_TOKENS" \
|
||||||
|
${FRANK_ARCHON_EXTRA_ARGS:+$FRANK_ARCHON_EXTRA_ARGS} \
|
||||||
|
> "$LOG" 2>&1
|
||||||
|
|
||||||
|
# From here on, this script's lifetime IS the container's: wait returns the
|
||||||
|
# container's exit code, so a crash is a nonzero ExecStart and the unit's
|
||||||
|
# Restart=always is real. (systemd stop goes through ExecStop in the unit —
|
||||||
|
# podman wait alone cannot stop the container child.)
|
||||||
|
exec podman wait "$FRANK_ARCHON_CONTAINER"
|
||||||
@@ -1,13 +1,13 @@
|
|||||||
#!/usr/bin/env bash
|
#!/usr/bin/env bash
|
||||||
# serve-frank-large.sh — frank-large endpoint: halogen-flash-server 0.16.2
|
# serve-frank-dragoon.sh — frank-dragoon endpoint: halogen-flash-server 0.16.2
|
||||||
# (closed-source engine, image ghcr.io/peonist-ai/halogen-flash-server,
|
# (closed-source engine, image ghcr.io/peonist-ai/halogen-flash-server,
|
||||||
# digest-pinned in frank-large.env) serving the Quat3rnion abliterated
|
# digest-pinned in frank-dragoon.env) serving the Quat3rnion abliterated
|
||||||
# Swift1.5 finetune of peonist's Qwen3.8-Flash-Next v2 checkpoint on
|
# Swift1.5 finetune of peonist's Qwen3.8-Flash-Next v2 checkpoint on
|
||||||
# Dankclaw's Strix Halo iGPU, host :7770. 2026-10-06 cutover from the
|
# Dankclaw's Strix Halo iGPU, host :7770. 2026-10-06 cutover from the
|
||||||
# halogen 27B-dense shape — revert the swap commit's trio to roll back
|
# halogen 27B-dense shape — revert the swap commit's trio to roll back
|
||||||
# (the 27B payload stays on disk).
|
# (the 27B payload stays on disk).
|
||||||
#
|
#
|
||||||
# The engine is driven by `podman run` DIRECTLY (the fleet's frank-primary
|
# The engine is driven by `podman run` DIRECTLY (the fleet's frank-archon
|
||||||
# pattern): the image starts engine + OpenAI frontend in one container.
|
# pattern): the image starts engine + OpenAI frontend in one container.
|
||||||
# The frontend listens on 8731 inside (HALOGEN_API_PORT default) and is
|
# The frontend listens on 8731 inside (HALOGEN_API_PORT default) and is
|
||||||
# published as host 7770; the engine's own token port (8730, no auth)
|
# published as host 7770; the engine's own token port (8730, no auth)
|
||||||
@@ -26,13 +26,13 @@
|
|||||||
# Why containers instead of a bare binary: the payload is .hgn and only
|
# Why containers instead of a bare binary: the payload is .hgn and only
|
||||||
# the flash-server line reads it; it ships as a container; the root unit
|
# the flash-server line reads it; it ships as a container; the root unit
|
||||||
# keeps LimitMEMLOCK=infinity for parity and to outlive the login session,
|
# keeps LimitMEMLOCK=infinity for parity and to outlive the login session,
|
||||||
# exactly like frank-primary's.
|
# exactly like frank-archon's.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-large.env}"
|
ENV_FILE="${FRANK_DRAGOON_ENV_FILE:-$SCRIPT_DIR/frank-dragoon.env}"
|
||||||
if [ ! -f "$ENV_FILE" ]; then
|
if [ ! -f "$ENV_FILE" ]; then
|
||||||
echo "frank-large: env file not found: $ENV_FILE" >&2
|
echo "frank-dragoon: env file not found: $ENV_FILE" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
set -a
|
set -a
|
||||||
@@ -40,18 +40,18 @@ set -a
|
|||||||
source "$ENV_FILE"
|
source "$ENV_FILE"
|
||||||
set +a
|
set +a
|
||||||
|
|
||||||
: "${FRANK_LARGE_IMAGE:?FRANK_LARGE_IMAGE must name the digest-pinned image}"
|
: "${FRANK_DRAGOON_IMAGE:?FRANK_DRAGOON_IMAGE must name the digest-pinned image}"
|
||||||
: "${FRANK_LARGE_CONTAINER:?FRANK_LARGE_CONTAINER must name the container}"
|
: "${FRANK_DRAGOON_CONTAINER:?FRANK_DRAGOON_CONTAINER must name the container}"
|
||||||
: "${FRANK_LARGE_PUBLISH:?FRANK_LARGE_PUBLISH must be host:container}"
|
: "${FRANK_DRAGOON_PUBLISH:?FRANK_DRAGOON_PUBLISH must be host:container}"
|
||||||
: "${FRANK_LARGE_MODEL_DIR:?FRANK_LARGE_MODEL_DIR must name the payload dir}"
|
: "${FRANK_DRAGOON_MODEL_DIR:?FRANK_DRAGOON_MODEL_DIR must name the payload dir}"
|
||||||
: "${FRANK_LARGE_CHECKPOINT:?FRANK_LARGE_CHECKPOINT must name the .hgn inside it}"
|
: "${FRANK_DRAGOON_CHECKPOINT:?FRANK_DRAGOON_CHECKPOINT must name the .hgn inside it}"
|
||||||
: "${FRANK_LARGE_CHECKPOINT_BYTES:?FRANK_LARGE_CHECKPOINT_BYTES must pin the size}"
|
: "${FRANK_DRAGOON_CHECKPOINT_BYTES:?FRANK_DRAGOON_CHECKPOINT_BYTES must pin the size}"
|
||||||
: "${FRANK_LARGE_NGRAM_TABLE:?FRANK_LARGE_NGRAM_TABLE must name the lookup table}"
|
: "${FRANK_DRAGOON_NGRAM_TABLE:?FRANK_DRAGOON_NGRAM_TABLE must name the lookup table}"
|
||||||
: "${FRANK_LARGE_NGRAM_BYTES:?FRANK_LARGE_NGRAM_BYTES must pin the table size}"
|
: "${FRANK_DRAGOON_NGRAM_BYTES:?FRANK_DRAGOON_NGRAM_BYTES must pin the table size}"
|
||||||
: "${FRANK_LARGE_VISION_TOWER:?FRANK_LARGE_VISION_TOWER must name the tower}"
|
: "${FRANK_DRAGOON_VISION_TOWER:?FRANK_DRAGOON_VISION_TOWER must name the tower}"
|
||||||
: "${FRANK_LARGE_VISION_BYTES:?FRANK_LARGE_VISION_BYTES must pin the tower size}"
|
: "${FRANK_DRAGOON_VISION_BYTES:?FRANK_DRAGOON_VISION_BYTES must pin the tower size}"
|
||||||
: "${FRANK_LARGE_TOKENIZER_FILES:?FRANK_LARGE_TOKENIZER_FILES must list the tokenizer dir contents}"
|
: "${FRANK_DRAGOON_TOKENIZER_FILES:?FRANK_DRAGOON_TOKENIZER_FILES must list the tokenizer dir contents}"
|
||||||
: "${HALOGEN_MODEL_ID:?HALOGEN_MODEL_ID must be the served alias (frank-large)}"
|
: "${HALOGEN_MODEL_ID:?HALOGEN_MODEL_ID must be the served alias (frank-dragoon)}"
|
||||||
: "${HALOGEN_CTX:?HALOGEN_CTX must be set (per-slot window)}"
|
: "${HALOGEN_CTX:?HALOGEN_CTX must be set (per-slot window)}"
|
||||||
: "${HALOGEN_KV_POOL_POSITIONS:?HALOGEN_KV_POOL_POSITIONS must be set}"
|
: "${HALOGEN_KV_POOL_POSITIONS:?HALOGEN_KV_POOL_POSITIONS must be set}"
|
||||||
: "${HALOGEN_KV_SLOTS:?HALOGEN_KV_SLOTS must be set}"
|
: "${HALOGEN_KV_SLOTS:?HALOGEN_KV_SLOTS must be set}"
|
||||||
@@ -62,9 +62,9 @@ set +a
|
|||||||
# podman dies SIGPIPE 141, pipefail turns the 141 into "no match" — the
|
# podman dies SIGPIPE 141, pipefail turns the 141 into "no match" — the
|
||||||
# 2026-09-30 deploy refusal).
|
# 2026-09-30 deploy refusal).
|
||||||
DIGESTS="$(podman images --format '{{.RepoDigests}}')"
|
DIGESTS="$(podman images --format '{{.RepoDigests}}')"
|
||||||
if ! grep -qF "${FRANK_LARGE_IMAGE#*@}" <<<"$DIGESTS"; then
|
if ! grep -qF "${FRANK_DRAGOON_IMAGE#*@}" <<<"$DIGESTS"; then
|
||||||
echo "frank-large: image $FRANK_LARGE_IMAGE not in podman storage;" >&2
|
echo "frank-dragoon: image $FRANK_DRAGOON_IMAGE not in podman storage;" >&2
|
||||||
echo " pull it: podman pull ${FRANK_LARGE_IMAGE%@*}" >&2
|
echo " pull it: podman pull ${FRANK_DRAGOON_IMAGE%@*}" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|
||||||
@@ -74,45 +74,45 @@ fi
|
|||||||
ck() {
|
ck() {
|
||||||
# ck <env-name> <path> <expected-bytes>
|
# ck <env-name> <path> <expected-bytes>
|
||||||
if [ ! -f "$2" ]; then
|
if [ ! -f "$2" ]; then
|
||||||
echo "frank-large: $1 $2 missing;" >&2
|
echo "frank-dragoon: $1 $2 missing;" >&2
|
||||||
echo " fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
|
echo " fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
local sz
|
local sz
|
||||||
sz="$(stat -c %s "$2")"
|
sz="$(stat -c %s "$2")"
|
||||||
if [ "$sz" != "$3" ]; then
|
if [ "$sz" != "$3" ]; then
|
||||||
echo "frank-large: $1 is $sz bytes, expected $3 (partial download?)" >&2
|
echo "frank-dragoon: $1 is $sz bytes, expected $3 (partial download?)" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
}
|
}
|
||||||
ck checkpoint "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_CHECKPOINT" "$FRANK_LARGE_CHECKPOINT_BYTES"
|
ck checkpoint "$FRANK_DRAGOON_MODEL_DIR/$FRANK_DRAGOON_CHECKPOINT" "$FRANK_DRAGOON_CHECKPOINT_BYTES"
|
||||||
ck n-gram-table "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_NGRAM_TABLE" "$FRANK_LARGE_NGRAM_BYTES"
|
ck n-gram-table "$FRANK_DRAGOON_MODEL_DIR/$FRANK_DRAGOON_NGRAM_TABLE" "$FRANK_DRAGOON_NGRAM_BYTES"
|
||||||
ck vision-tower "$FRANK_LARGE_MODEL_DIR/$FRANK_LARGE_VISION_TOWER" "$FRANK_LARGE_VISION_BYTES"
|
ck vision-tower "$FRANK_DRAGOON_MODEL_DIR/$FRANK_DRAGOON_VISION_TOWER" "$FRANK_DRAGOON_VISION_BYTES"
|
||||||
|
|
||||||
# The payload ships the tokenizer as a FLAT DIRECTORY. A HuggingFace cache
|
# The payload ships the tokenizer as a FLAT DIRECTORY. A HuggingFace cache
|
||||||
# snapshot symlinks each file into a sibling blobs/, and those symlinks
|
# snapshot symlinks each file into a sibling blobs/, and those symlinks
|
||||||
# dangle inside the container mount — so the directory must be real and
|
# dangle inside the container mount — so the directory must be real and
|
||||||
# every named file must be a regular file, not a link.
|
# every named file must be a regular file, not a link.
|
||||||
TOK_DIR="$FRANK_LARGE_MODEL_DIR/tokenizer"
|
TOK_DIR="$FRANK_DRAGOON_MODEL_DIR/tokenizer"
|
||||||
if [ -L "$TOK_DIR" ] || [ ! -d "$TOK_DIR" ]; then
|
if [ -L "$TOK_DIR" ] || [ ! -d "$TOK_DIR" ]; then
|
||||||
echo "frank-large: tokenizer dir $TOK_DIR missing or a symlink (HF cache snapshot?);" >&2
|
echo "frank-dragoon: tokenizer dir $TOK_DIR missing or a symlink (HF cache snapshot?);" >&2
|
||||||
echo " fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
|
echo " fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
for TF in $FRANK_LARGE_TOKENIZER_FILES; do
|
for TF in $FRANK_DRAGOON_TOKENIZER_FILES; do
|
||||||
if [ ! -f "$TOK_DIR/$TF" ] || [ -L "$TOK_DIR/$TF" ]; then
|
if [ ! -f "$TOK_DIR/$TF" ] || [ -L "$TOK_DIR/$TF" ]; then
|
||||||
echo "frank-large: tokenizer file $TOK_DIR/$TF missing or a symlink;" >&2
|
echo "frank-dragoon: tokenizer file $TOK_DIR/$TF missing or a symlink;" >&2
|
||||||
echo " fetch it: bash scripts/download-frank-large-flashnext.sh" >&2
|
echo " fetch it: bash scripts/download-frank-dragoon-flashnext.sh" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
|
|
||||||
# GPU access is mandatory but the node paths are variables so the seam can
|
# GPU access is mandatory but the node paths are variables so the seam can
|
||||||
# drive the gate on a box without an AMD GPU.
|
# drive the gate on a box without an AMD GPU.
|
||||||
DEV_KFD="${FRANK_LARGE_DEV_KFD:-/dev/kfd}"
|
DEV_KFD="${FRANK_DRAGOON_DEV_KFD:-/dev/kfd}"
|
||||||
DEV_DRI="${FRANK_LARGE_DEV_DRI:-/dev/dri}"
|
DEV_DRI="${FRANK_DRAGOON_DEV_DRI:-/dev/dri}"
|
||||||
if [ ! -e "$DEV_KFD" ] || [ ! -d "$DEV_DRI" ]; then
|
if [ ! -e "$DEV_KFD" ] || [ ! -d "$DEV_DRI" ]; then
|
||||||
echo "frank-large: $DEV_KFD or $DEV_DRI missing — no gfx1151 to serve on" >&2
|
echo "frank-dragoon: $DEV_KFD or $DEV_DRI missing — no gfx1151 to serve on" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
# Host-RAM preflight — the engine's pin gate, checked BEFORE podman run.
|
# Host-RAM preflight — the engine's pin gate, checked BEFORE podman run.
|
||||||
@@ -125,15 +125,15 @@ fi
|
|||||||
# self-resolved in ~2 min when the dying 27B engine freed RAM. Check here and
|
# self-resolved in ~2 min when the dying 27B engine freed RAM. Check here and
|
||||||
# WAIT for the RAM to clear, so a transient dip boots once instead of
|
# WAIT for the RAM to clear, so a transient dip boots once instead of
|
||||||
# crash-looping; if it never clears, refuse loudly.
|
# crash-looping; if it never clears, refuse loudly.
|
||||||
MEMINFO="${FRANK_LARGE_MEMINFO:-/proc/meminfo}"
|
MEMINFO="${FRANK_DRAGOON_MEMINFO:-/proc/meminfo}"
|
||||||
PIN_KB=$(( FRANK_LARGE_CHECKPOINT_BYTES / 1024 ))
|
PIN_KB=$(( FRANK_DRAGOON_CHECKPOINT_BYTES / 1024 ))
|
||||||
NEED_KB=$(( PIN_KB + 16 * 1048576 ))
|
NEED_KB=$(( PIN_KB + 16 * 1048576 ))
|
||||||
ram_avail_kb() { awk '/^MemAvailable:/{print $2; exit}' "$MEMINFO"; }
|
ram_avail_kb() { awk '/^MemAvailable:/{print $2; exit}' "$MEMINFO"; }
|
||||||
MEMAVAIL_KB="$(ram_avail_kb)" || MEMAVAIL_KB=""
|
MEMAVAIL_KB="$(ram_avail_kb)" || MEMAVAIL_KB=""
|
||||||
if [ -n "$MEMAVAIL_KB" ] && [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
|
if [ -n "$MEMAVAIL_KB" ] && [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
|
||||||
WAIT_S="${FRANK_LARGE_RAM_WAIT:-600}"
|
WAIT_S="${FRANK_DRAGOON_RAM_WAIT:-600}"
|
||||||
GIB() { awk -v kb="$1" 'BEGIN{printf "%.2f", kb/1048576}'; }
|
GIB() { awk -v kb="$1" 'BEGIN{printf "%.2f", kb/1048576}'; }
|
||||||
echo "frank-large: host RAM below the pin gate — MemAvailable $(GIB "$MEMAVAIL_KB") GiB < needed $(GIB "$NEED_KB") GiB ($(GIB "$PIN_KB") GiB checkpoint + 16.00 GiB floor); waiting up to ${WAIT_S}s for host RAM to clear" >&2
|
echo "frank-dragoon: host RAM below the pin gate — MemAvailable $(GIB "$MEMAVAIL_KB") GiB < needed $(GIB "$NEED_KB") GiB ($(GIB "$PIN_KB") GiB checkpoint + 16.00 GiB floor); waiting up to ${WAIT_S}s for host RAM to clear" >&2
|
||||||
deadline=$(( $(date +%s) + WAIT_S ))
|
deadline=$(( $(date +%s) + WAIT_S ))
|
||||||
while [ "$(date +%s)" -lt "$deadline" ]; do
|
while [ "$(date +%s)" -lt "$deadline" ]; do
|
||||||
sleep 5
|
sleep 5
|
||||||
@@ -143,16 +143,16 @@ if [ -n "$MEMAVAIL_KB" ] && [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
|
|||||||
fi
|
fi
|
||||||
done
|
done
|
||||||
if [ -z "$MEMAVAIL_KB" ] || [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
|
if [ -z "$MEMAVAIL_KB" ] || [ "$MEMAVAIL_KB" -lt "$NEED_KB" ]; then
|
||||||
echo "frank-large: MemAvailable $(GIB "${MEMAVAIL_KB:-0}") GiB after ${WAIT_S}s; refusing to start (the engine would refuse the pin)" >&2
|
echo "frank-dragoon: MemAvailable $(GIB "${MEMAVAIL_KB:-0}") GiB after ${WAIT_S}s; refusing to start (the engine would refuse the pin)" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
echo "frank-large: MemAvailable cleared to $(GIB "$MEMAVAIL_KB") GiB; starting" >&2
|
echo "frank-dragoon: MemAvailable cleared to $(GIB "$MEMAVAIL_KB") GiB; starting" >&2
|
||||||
fi
|
fi
|
||||||
for CTR in ${FRANK_LARGE_LEGACY_CONTAINERS:-} "$FRANK_LARGE_CONTAINER"; do
|
for CTR in ${FRANK_DRAGOON_LEGACY_CONTAINERS:-} "$FRANK_DRAGOON_CONTAINER"; do
|
||||||
podman rm -f "$CTR" >/dev/null 2>&1 || true
|
podman rm -f "$CTR" >/dev/null 2>&1 || true
|
||||||
done
|
done
|
||||||
|
|
||||||
LOG="${FRANK_LARGE_LOG:-/home/dank/logs/frank_large.log}"
|
LOG="${FRANK_DRAGOON_LOG:-/home/dank/logs/frank_dragoon.log}"
|
||||||
mkdir -p "$(dirname "$LOG")"
|
mkdir -p "$(dirname "$LOG")"
|
||||||
# --ipc=host for the udev-granted supplementary group, seccomp unconfined
|
# --ipc=host for the udev-granted supplementary group, seccomp unconfined
|
||||||
# for the ROCm ioctl surface, and unbounded memlock (the unit's
|
# for the ROCm ioctl surface, and unbounded memlock (the unit's
|
||||||
@@ -163,17 +163,17 @@ mkdir -p "$(dirname "$LOG")"
|
|||||||
# and both sibling serve scripts pass no nofile either. No comment lines
|
# and both sibling serve scripts pass no nofile either. No comment lines
|
||||||
# inside the backslash chain — a mid-chain # terminates the continuation
|
# inside the backslash chain — a mid-chain # terminates the continuation
|
||||||
# (the 2026-10-05 exit-127 crash loop on the sibling serve script).
|
# (the 2026-10-05 exit-127 crash loop on the sibling serve script).
|
||||||
podman run --name "$FRANK_LARGE_CONTAINER" \
|
podman run --name "$FRANK_DRAGOON_CONTAINER" \
|
||||||
--publish "$FRANK_LARGE_PUBLISH" \
|
--publish "$FRANK_DRAGOON_PUBLISH" \
|
||||||
--ipc=host \
|
--ipc=host \
|
||||||
--ulimit memlock=-1:-1 \
|
--ulimit memlock=-1:-1 \
|
||||||
--device "$DEV_KFD" --device "$DEV_DRI" \
|
--device "$DEV_KFD" --device "$DEV_DRI" \
|
||||||
--group-add keep-groups \
|
--group-add keep-groups \
|
||||||
--security-opt seccomp=unconfined \
|
--security-opt seccomp=unconfined \
|
||||||
-e HALOGEN_MODEL_ID="$HALOGEN_MODEL_ID" \
|
-e HALOGEN_MODEL_ID="$HALOGEN_MODEL_ID" \
|
||||||
-e HALOGEN_CHECKPOINT="/models/$FRANK_LARGE_CHECKPOINT" \
|
-e HALOGEN_CHECKPOINT="/models/$FRANK_DRAGOON_CHECKPOINT" \
|
||||||
-e HALOGEN_NGRAM_TABLE="/models/$FRANK_LARGE_NGRAM_TABLE" \
|
-e HALOGEN_NGRAM_TABLE="/models/$FRANK_DRAGOON_NGRAM_TABLE" \
|
||||||
-e HALOGEN_VISION_TOWER="/models/$FRANK_LARGE_VISION_TOWER" \
|
-e HALOGEN_VISION_TOWER="/models/$FRANK_DRAGOON_VISION_TOWER" \
|
||||||
-e HALOGEN_TOKENIZER=/tokenizer \
|
-e HALOGEN_TOKENIZER=/tokenizer \
|
||||||
-e HALOGEN_CTX="$HALOGEN_CTX" \
|
-e HALOGEN_CTX="$HALOGEN_CTX" \
|
||||||
-e HALOGEN_KV_POOL_POSITIONS="$HALOGEN_KV_POOL_POSITIONS" \
|
-e HALOGEN_KV_POOL_POSITIONS="$HALOGEN_KV_POOL_POSITIONS" \
|
||||||
@@ -187,9 +187,9 @@ podman run --name "$FRANK_LARGE_CONTAINER" \
|
|||||||
-e HALOGEN_TOP_P="${HALOGEN_TOP_P:-0.95}" \
|
-e HALOGEN_TOP_P="${HALOGEN_TOP_P:-0.95}" \
|
||||||
-e HALOGEN_TOP_K="${HALOGEN_TOP_K:-20}" \
|
-e HALOGEN_TOP_K="${HALOGEN_TOP_K:-20}" \
|
||||||
${HALOGEN_WEIGHTS_LOCK:+-e HALOGEN_WEIGHTS_LOCK="$HALOGEN_WEIGHTS_LOCK"} \
|
${HALOGEN_WEIGHTS_LOCK:+-e HALOGEN_WEIGHTS_LOCK="$HALOGEN_WEIGHTS_LOCK"} \
|
||||||
-v "$FRANK_LARGE_MODEL_DIR:/models:ro" \
|
-v "$FRANK_DRAGOON_MODEL_DIR:/models:ro" \
|
||||||
-v "$FRANK_LARGE_MODEL_DIR/tokenizer:/tokenizer:ro" \
|
-v "$FRANK_DRAGOON_MODEL_DIR/tokenizer:/tokenizer:ro" \
|
||||||
"$FRANK_LARGE_IMAGE" \
|
"$FRANK_DRAGOON_IMAGE" \
|
||||||
> "$LOG" 2>&1
|
> "$LOG" 2>&1
|
||||||
|
|
||||||
# FOREGROUND `podman run`: the engine's stdout/stderr are this script's
|
# FOREGROUND `podman run`: the engine's stdout/stderr are this script's
|
||||||
@@ -1,204 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
# serve-frank-primary.sh — frank-primary endpoint: radiance (StillDeadcode,
|
|
||||||
# Apache-2.0) serving the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at
|
|
||||||
# NATIVE block-FP8 precision from a rad-convert container
|
|
||||||
# (/home/dank/models/rad/qwen4exp-fp8-native.rad), on BOTH R9700s (TP=2,
|
|
||||||
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (PR #587,
|
|
||||||
# one card; git tag frank-primary-freetoken-pr587-20261005 is the
|
|
||||||
# pre-cutover state).
|
|
||||||
#
|
|
||||||
# The run line below IS the engine argv: the image's entrypoint is the
|
|
||||||
# engine and its tools sit beside it on PATH. Unlike every predecessor
|
|
||||||
# there is no per-boot VRAM planner to second-guess: the boot ledger in the
|
|
||||||
# log states every budget before anything is allocated (weights vs KV
|
|
||||||
# split, the expert host pool's coverage of the worst case, the PLE pin,
|
|
||||||
# the P2P answer per pair) — read it after every resize. First-boot shape:
|
|
||||||
# 2 decode slots (Frank's call), 40 GiB/rank pinned expert pool, fp8 KV,
|
|
||||||
# MTP depth 3, prefix cache host 4 GiB + disk 64 GiB, PLE pinned in RAM.
|
|
||||||
#
|
|
||||||
# The root unit (scripts/frank-primary.service, User=dank,
|
|
||||||
# LimitMEMLOCK=infinity) grants the pin budget: 2 x 40 GiB expert pools +
|
|
||||||
# the 47.7 GiB PLE table + 4 GiB prefix host tier — ~132 GiB of pinned
|
|
||||||
# pages, which is ALSO why this script refuses to start when ttm's
|
|
||||||
# pages_limit/pages_max cap is under FRANK_PRIMARY_TTM_MIN_MIB (see below).
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
ENV_FILE="${FRANK_PRIMARY_ENV_FILE:-$SCRIPT_DIR/frank-primary.env}"
|
|
||||||
|
|
||||||
if [ ! -f "$ENV_FILE" ]; then
|
|
||||||
echo "ERROR: frank-primary env file absent: $ENV_FILE" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
set -a
|
|
||||||
# shellcheck disable=SC1090
|
|
||||||
source "$ENV_FILE"
|
|
||||||
set +a
|
|
||||||
|
|
||||||
: "${FRANK_PRIMARY_IMAGE:?FRANK_PRIMARY_IMAGE must name the radiance image}"
|
|
||||||
: "${FRANK_PRIMARY_MODEL_DIR:?FRANK_PRIMARY_MODEL_DIR must name the .rad dir}"
|
|
||||||
: "${FRANK_PRIMARY_MODEL_FILE:?FRANK_PRIMARY_MODEL_FILE must name the .rad file}"
|
|
||||||
: "${FRANK_PRIMARY_STATE_DIR:?FRANK_PRIMARY_STATE_DIR must name the /data state dir}"
|
|
||||||
: "${FRANK_PRIMARY_CONTAINER:?FRANK_PRIMARY_CONTAINER must name the container}"
|
|
||||||
: "${FRANK_PRIMARY_SERVED_NAME:?FRANK_PRIMARY_SERVED_NAME must name the alias}"
|
|
||||||
: "${FRANK_PRIMARY_PORT:?FRANK_PRIMARY_PORT must be the port}"
|
|
||||||
: "${FRANK_PRIMARY_TEMPLATE:?FRANK_PRIMARY_TEMPLATE must name the chat template}"
|
|
||||||
: "${FRANK_PRIMARY_TP:?FRANK_PRIMARY_TP must be set}"
|
|
||||||
: "${FRANK_PRIMARY_TP_WIRE:?FRANK_PRIMARY_TP_WIRE must be set}"
|
|
||||||
: "${FRANK_PRIMARY_MAX_SEQS:?FRANK_PRIMARY_MAX_SEQS must be set}"
|
|
||||||
: "${FRANK_PRIMARY_MAX_MODEL_LEN:?FRANK_PRIMARY_MAX_MODEL_LEN must be set}"
|
|
||||||
: "${FRANK_PRIMARY_PLACEMENT:?FRANK_PRIMARY_PLACEMENT must be set}"
|
|
||||||
: "${FRANK_PRIMARY_HOST_POOL_MIB:?FRANK_PRIMARY_HOST_POOL_MIB must be set}"
|
|
||||||
: "${FRANK_PRIMARY_GPU_HEADROOM_MIB:?FRANK_PRIMARY_GPU_HEADROOM_MIB must be set}"
|
|
||||||
: "${FRANK_PRIMARY_EXPERT_VS_CACHE:?FRANK_PRIMARY_EXPERT_VS_CACHE must be set}"
|
|
||||||
: "${FRANK_PRIMARY_KV_DTYPE:?FRANK_PRIMARY_KV_DTYPE must be set}"
|
|
||||||
: "${FRANK_PRIMARY_SPEC_TOKENS:?FRANK_PRIMARY_SPEC_TOKENS must be set}"
|
|
||||||
: "${FRANK_PRIMARY_MAX_BATCHED_TOKENS:?FRANK_PRIMARY_MAX_BATCHED_TOKENS must be set}"
|
|
||||||
: "${FRANK_PRIMARY_PREFIX_HOST_MIB:?FRANK_PRIMARY_PREFIX_HOST_MIB must be set}"
|
|
||||||
: "${FRANK_PRIMARY_PREFIX_DISK_MIB:?FRANK_PRIMARY_PREFIX_DISK_MIB must be set}"
|
|
||||||
: "${FRANK_PRIMARY_NGRAM_PLACEMENT:?FRANK_PRIMARY_NGRAM_PLACEMENT must be set}"
|
|
||||||
: "${FRANK_PRIMARY_TTM_MIN_MIB:?FRANK_PRIMARY_TTM_MIN_MIB must be set}"
|
|
||||||
: "${FRANK_PRIMARY_RAD_MIN_MIB:?FRANK_PRIMARY_RAD_MIN_MIB must be set}"
|
|
||||||
|
|
||||||
# The image gate is the digest-pinned ref itself: a name@sha256 ref exists
|
|
||||||
# in storage only if THAT manifest was pulled — a retagged look-alike or a
|
|
||||||
# newer latest does not satisfy it. `podman image exists` takes a
|
|
||||||
# name@digest ref in one command — no pipefail/grep race (the 2026-09-30
|
|
||||||
# lesson). Runs as dank: the unit's process uses dank's ROOTLESS podman
|
|
||||||
# storage (root unit, User=dank — the fleet's frank-primary shape).
|
|
||||||
if ! podman image exists "$FRANK_PRIMARY_IMAGE"; then
|
|
||||||
echo "ERROR: radiance image absent: $FRANK_PRIMARY_IMAGE" >&2
|
|
||||||
echo "frank-primary: podman pull docker://stilldeadcode/radiance:latest (as dank)" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# The payload gate: the .rad container file, sized. A truncated convert
|
|
||||||
# would otherwise buy a guaranteed long-timeout boot instead of this fast
|
|
||||||
# failure. The staging dir / raw checkpoint are NOT this script's concern
|
|
||||||
# (scripts/build-frank-primary-rad.sh built the file; its rad-info output
|
|
||||||
# is the record).
|
|
||||||
RAD="$FRANK_PRIMARY_MODEL_DIR/$FRANK_PRIMARY_MODEL_FILE"
|
|
||||||
if [ ! -f "$RAD" ]; then
|
|
||||||
echo "ERROR: .rad container missing: $RAD" >&2
|
|
||||||
echo "frank-primary: run scripts/build-frank-primary-rad.sh" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
RAD_MIB=$(( $(stat -c %s "$RAD") / 1048576 ))
|
|
||||||
if [ "$RAD_MIB" -lt "$FRANK_PRIMARY_RAD_MIN_MIB" ]; then
|
|
||||||
echo "ERROR: .rad container too small: $RAD is ${RAD_MIB} MiB, want >= $FRANK_PRIMARY_RAD_MIN_MIB" >&2
|
|
||||||
echo "frank-primary: the convert was truncated — rebuild it" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [ ! -f "$FRANK_PRIMARY_TEMPLATE" ]; then
|
|
||||||
echo "ERROR: chat template missing: $FRANK_PRIMARY_TEMPLATE" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
mkdir -p "$FRANK_PRIMARY_STATE_DIR/kvcache/frank-primary"
|
|
||||||
# The engine's create is NOT recursive: with only the state dir mounted, its
|
|
||||||
# "cannot create /data/kvcache/frank-primary: No such file or directory"
|
|
||||||
# left the SSD prefix tier off (measured 2026-10-06 boot ledger). Pre-create
|
|
||||||
# the full path on the host so the tier opens on first boot.
|
|
||||||
|
|
||||||
mkdir -p "$FRANK_PRIMARY_STATE_DIR/tune"
|
|
||||||
# rad-tune's kernel-selection cache (see the runbook's "Tuning levers"):
|
|
||||||
# the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune at boot;
|
|
||||||
# $RADIANCE_HOME defaults to /opt/radiance, which lives in the image, so
|
|
||||||
# the cache must ride a host mount or it dies with the --rm container
|
|
||||||
# that wrote it. The dir is pre-created for the same reason as kvcache
|
|
||||||
# (the engine's create is not recursive).
|
|
||||||
|
|
||||||
if [ ! -e /dev/kfd ] || [ ! -d /dev/dri ]; then
|
|
||||||
echo "ERROR: GPU device nodes absent (/dev/kfd, /dev/dri)" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# TTM preflight: the pinned ledger (~2 x 40 GiB pools + 47.7 GiB PLE + 4 GiB
|
|
||||||
# prefix host = ~132 GiB) rides TTM's shared pages, capped by ttm's
|
|
||||||
# pages_limit (the current kernel exposes NO pages_max; the cap function
|
|
||||||
# takes the min over the knobs that exist). Below FRANK_PRIMARY_TTM_MIN_MIB
|
|
||||||
# the load dies mid-pin with an allocator error instead of this named
|
|
||||||
# refusal — raise the knob first (runtime sysfs write; kernel cmdline
|
|
||||||
# fallback + one reboot only if the file is read-only), see the runbook.
|
|
||||||
# RAISED 2026-10-05 pre-cutover to 52,428,800 pages = 200 GiB.
|
|
||||||
ttm_cap_mib() {
|
|
||||||
# min over the knobs THIS kernel exposes — the current kernel has
|
|
||||||
# pages_limit only (no pages_max); a knob that is absent is skipped,
|
|
||||||
# not a zero (a zero would refuse a healthy cap).
|
|
||||||
local v cur=""
|
|
||||||
for p in /sys/module/ttm/parameters/pages_limit /sys/module/ttm/parameters/pages_max; do
|
|
||||||
[ -r "$p" ] || continue
|
|
||||||
v="$(cat "$p" 2>/dev/null)" || continue
|
|
||||||
[ -n "$v" ] || continue
|
|
||||||
v=$(( v * 4 / 1024 ))
|
|
||||||
if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
|
|
||||||
done
|
|
||||||
echo "${cur:-0}"
|
|
||||||
}
|
|
||||||
TTM_CAP="$(ttm_cap_mib)"
|
|
||||||
if [ "$TTM_CAP" -lt "$FRANK_PRIMARY_TTM_MIN_MIB" ]; then
|
|
||||||
echo "ERROR: ttm pages cap is ${TTM_CAP} MiB, the pinned ledger needs >= $FRANK_PRIMARY_TTM_MIN_MIB" >&2
|
|
||||||
echo "frank-primary: raise /sys/module/ttm/parameters/pages_limit (and pages_max if the kernel exposes one) to >= $FRANK_PRIMARY_TTM_MIN_MIB in 4 KiB pages" >&2
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
# A crashed/killed unit can leave the container name registered (systemd
|
|
||||||
# SIGKILLs conmon before cleanup runs). Clear OUR name plus the legacy
|
|
||||||
# identities (the FreeToken-era frank-primary-qwen, the retired R9V
|
|
||||||
# container and the ad hoc experiment containers all hold this GPU); all
|
|
||||||
# idempotent.
|
|
||||||
for CTR in $FRANK_PRIMARY_LEGACY_CONTAINERS "$FRANK_PRIMARY_CONTAINER"; do
|
|
||||||
podman rm -f "$CTR" >/dev/null 2>&1 || true
|
|
||||||
done
|
|
||||||
|
|
||||||
LOG=/home/dank/logs/frank_primary.log
|
|
||||||
mkdir -p "$(dirname "$LOG")"
|
|
||||||
|
|
||||||
# Publisher-vetted container contract (docs/DOCKER.md, compose/common.yaml):
|
|
||||||
# --network host (the server binds the host's addresses directly, no NAT),
|
|
||||||
# --init (the engine releases the cards on SIGTERM), seccomp=unconfined
|
|
||||||
# (Docker/podman's default profile refuses io_uring, which the n-gram
|
|
||||||
# table's reads go through, and the NUMA binding calls the ROCm runtime
|
|
||||||
# makes for host memory), memlock unbounded (the pinned ledger — legal
|
|
||||||
# because the root unit grants LimitMEMLOCK=infinity to this dank process),
|
|
||||||
# /dev/kfd + /dev/dri (render/video groups are moot for a root container).
|
|
||||||
# The template rides a single-file mount; the .rad dir is ro; /data holds
|
|
||||||
# the prefix-cache disk tier.
|
|
||||||
podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
|
||||||
--network host \
|
|
||||||
--init \
|
|
||||||
--security-opt seccomp=unconfined \
|
|
||||||
--ulimit memlock=-1:-1 \
|
|
||||||
--device /dev/kfd --device /dev/dri \
|
|
||||||
-v "$FRANK_PRIMARY_MODEL_DIR":/models:ro \
|
|
||||||
-v "$FRANK_PRIMARY_STATE_DIR":/data \
|
|
||||||
-v "$FRANK_PRIMARY_STATE_DIR/tune":/opt/radiance/share/radiance/tune \
|
|
||||||
-v "$FRANK_PRIMARY_TEMPLATE":/templates/template.jinja:ro \
|
|
||||||
"$FRANK_PRIMARY_IMAGE" \
|
|
||||||
--model "/models/$FRANK_PRIMARY_MODEL_FILE" \
|
|
||||||
--host :: --port "$FRANK_PRIMARY_PORT" \
|
|
||||||
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" \
|
|
||||||
--override-chat-template /templates/template.jinja \
|
|
||||||
--tp "$FRANK_PRIMARY_TP" --tp-wire "$FRANK_PRIMARY_TP_WIRE" \
|
|
||||||
--max-num-seqs "$FRANK_PRIMARY_MAX_SEQS" \
|
|
||||||
--max-model-len "$FRANK_PRIMARY_MAX_MODEL_LEN" \
|
|
||||||
--placement "$FRANK_PRIMARY_PLACEMENT" \
|
|
||||||
--host-pool-mib "$FRANK_PRIMARY_HOST_POOL_MIB" \
|
|
||||||
--gpu-headroom-mib "$FRANK_PRIMARY_GPU_HEADROOM_MIB" \
|
|
||||||
--expert-vs-cache-ratio "$FRANK_PRIMARY_EXPERT_VS_CACHE" \
|
|
||||||
--kv-cache-dtype "$FRANK_PRIMARY_KV_DTYPE" \
|
|
||||||
--prefix-cache-host-mib "$FRANK_PRIMARY_PREFIX_HOST_MIB" \
|
|
||||||
--prefix-cache-dir /data/kvcache/frank-primary \
|
|
||||||
--prefix-cache-disk-mib "$FRANK_PRIMARY_PREFIX_DISK_MIB" \
|
|
||||||
--ngram-placement "$FRANK_PRIMARY_NGRAM_PLACEMENT" \
|
|
||||||
--num-speculative-tokens "$FRANK_PRIMARY_SPEC_TOKENS" \
|
|
||||||
--max-num-batched-tokens "$FRANK_PRIMARY_MAX_BATCHED_TOKENS" \
|
|
||||||
${FRANK_PRIMARY_EXTRA_ARGS:+$FRANK_PRIMARY_EXTRA_ARGS} \
|
|
||||||
> "$LOG" 2>&1
|
|
||||||
|
|
||||||
# From here on, this script's lifetime IS the container's: wait returns the
|
|
||||||
# container's exit code, so a crash is a nonzero ExecStart and the unit's
|
|
||||||
# Restart=always is real. (systemd stop goes through ExecStop in the unit —
|
|
||||||
# podman wait alone cannot stop the container child.)
|
|
||||||
exec podman wait "$FRANK_PRIMARY_CONTAINER"
|
|
||||||
@@ -1,120 +0,0 @@
|
|||||||
#!/usr/bin/env bash
|
|
||||||
# Serve ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF
|
|
||||||
# (RCO-IQ3_XXS, **MTP head in the file**) as frank-secondary on Dankclaw's
|
|
||||||
# RTX 3090 via llama.cpp (master build, CUDA sm_86).
|
|
||||||
#
|
|
||||||
# 2026-10-06 (Frank): TWO 131,072 conversations (262144 / 2) at the 128k
|
|
||||||
# cap, ubatch 512, and MTP BACK ON (draft-mtp, depth 3 — the payload is
|
|
||||||
# 1.56 GiB lighter than the IQ3_S-mtp it replaced, which is what funds
|
|
||||||
# the draft ctx plus the second slot; ngram stays skipped — on a qwen35
|
|
||||||
# hybrid it disables recurrent-state rollback). The q8_0 KV contract
|
|
||||||
# (FP8 class) is unchanged; the per-tensor flags stay explicit either
|
|
||||||
# way.
|
|
||||||
#
|
|
||||||
# Architecture: qwen35 hybrid — 64 real layers, 16 full-attention + 48
|
|
||||||
# Gated DeltaNet (full_attention_interval 4), native 262144 window, plus
|
|
||||||
# the blk.64 MTP head (nextn_predict_layers=1). KV lives only in the 16
|
|
||||||
# attention layers (34 KiB/token at q8_0); the DeltaNet state is per-slot
|
|
||||||
# fixed size.
|
|
||||||
# Vision comes from the projector file (the GGUF carries no vision
|
|
||||||
# tensors; the publisher SHIPS mmproj-BF16.gguf in the same repo), so
|
|
||||||
# --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is the
|
|
||||||
# documented text-only opt-out.
|
|
||||||
#
|
|
||||||
# Sampling: llama.cpp takes it per request (the fleet harness sends its
|
|
||||||
# recipe from models.yml); greedy is the harness default.
|
|
||||||
#
|
|
||||||
# Tunables come from frank-secondary.env unless FRANK_SECONDARY_ENV_FILE overrides.
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
|
||||||
ENV_FILE="${FRANK_SECONDARY_ENV_FILE:-$SCRIPT_DIR/frank-secondary.env}"
|
|
||||||
if [[ -f "$ENV_FILE" ]]; then
|
|
||||||
set -a
|
|
||||||
# shellcheck disable=SC1090
|
|
||||||
source "$ENV_FILE"
|
|
||||||
set +a
|
|
||||||
fi
|
|
||||||
|
|
||||||
# --- defaults (used when the env file does not set them) -------------------
|
|
||||||
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
|
|
||||||
: "${MODEL_FILE:=/home/dank/models/Swift15-UncIQ3XXS/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-RCO-IQ3_XXS.gguf}"
|
|
||||||
: "${ALIAS:=frank-secondary}"
|
|
||||||
: "${BIND_ADDR:=0.0.0.0}"
|
|
||||||
: "${LISTEN_PORT:=7771}"
|
|
||||||
# ctx-size is the POOL llama.cpp divides across slots: 262144 / 2 gives
|
|
||||||
# TWO 131,072 conversations — Frank's 128k cap each. 2026-10-06 (Frank):
|
|
||||||
# drop MTP, drop ubatch to 512, take two slots; the fit is tight by his
|
|
||||||
# explicit order (frank-secondary.env carries the arithmetic and the floor
|
|
||||||
# violation).
|
|
||||||
: "${CTX_SIZE:=262144}"
|
|
||||||
: "${PARALLEL:=2}"
|
|
||||||
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
|
|
||||||
# Frank's 2026-10-06 tight-fit order ships 4096/512 (claws back ~0.3 GiB
|
|
||||||
# toward the second slot); ubatch 1024 is the first A/B back if a future
|
|
||||||
# boot lands with headroom (a miss fails loud at graph capture). The
|
|
||||||
# logical batch is VRAM- and throughput-neutral at a fixed ubatch.
|
|
||||||
: "${BATCH_SIZE:=4096}"
|
|
||||||
: "${UBATCH_SIZE:=512}"
|
|
||||||
# KV at FP8 class = q8_0 on llama.cpp (1.0625 B/elem => 8.5 effective bits;
|
|
||||||
# there is no true fp8 KV dtype). Explicit either way so the contract is
|
|
||||||
# visible in the argv.
|
|
||||||
: "${CACHE_TYPE_K:=q8_0}"
|
|
||||||
: "${CACHE_TYPE_V:=q8_0}"
|
|
||||||
# Speculative decoding is ON by default (Frank, 2026-10-06 evening):
|
|
||||||
# draft-mtp against the head that is IN the file. SPEC_TYPE none strips
|
|
||||||
# BOTH speculative flags (flags move as a set) — a no-spec baseline must
|
|
||||||
# not carry half a spec pair.
|
|
||||||
: "${SPEC_TYPE:=draft-mtp}"
|
|
||||||
: "${SPEC_DRAFT_N_MAX:=3}"
|
|
||||||
: "${NGL:=99}"
|
|
||||||
# Vision projector. No colon on purpose: an explicitly empty value means
|
|
||||||
# "text-only, on purpose" and must NOT be replaced by the default path.
|
|
||||||
: "${MMPROJ_FILE=/home/dank/models/Swift15-UncIQ3XXS/mmproj-BF16.gguf}"
|
|
||||||
|
|
||||||
# Fail loudly rather than half-loading: a missing file is a download that
|
|
||||||
# has not finished, and Restart=always would turn it into a crash loop that
|
|
||||||
# reads like a runtime fault.
|
|
||||||
for path in "$LLAMA_BIN" "$MODEL_FILE"; do
|
|
||||||
[[ -e "$path" ]] || { echo "frank-secondary: missing $path" >&2; exit 1; }
|
|
||||||
done
|
|
||||||
EXTRA_ARGS=()
|
|
||||||
if [[ -n "$MMPROJ_FILE" ]]; then
|
|
||||||
[[ -e "$MMPROJ_FILE" ]] || { echo "frank-secondary: missing $MMPROJ_FILE" >&2; exit 1; }
|
|
||||||
EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Speculative flags move as a set: they reach the argv ONLY when a drafter
|
|
||||||
# is named. SPEC_TYPE=none (the shipped contract) strips both — a no-spec
|
|
||||||
# baseline must not carry half a spec pair.
|
|
||||||
if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
|
|
||||||
EXTRA_ARGS+=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
|
|
||||||
fi
|
|
||||||
|
|
||||||
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
|
|
||||||
# open after boot; make sure it exists before the server needs it.
|
|
||||||
nvidia-modprobe -u -c=0 2>/dev/null || true
|
|
||||||
|
|
||||||
# Flash attention stays pinned on: it is REQUIRED for quantized V (q8_0),
|
|
||||||
# and the build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17), so
|
|
||||||
# the q8_0 KV contract in frank-secondary.env is a real path — without this
|
|
||||||
# flag or that build option the engine silently falls back and the
|
|
||||||
# contract becomes a lie.
|
|
||||||
ARGS=(
|
|
||||||
--model "$MODEL_FILE"
|
|
||||||
--alias "$ALIAS"
|
|
||||||
--host "$BIND_ADDR"
|
|
||||||
--port "$LISTEN_PORT"
|
|
||||||
--ctx-size "$CTX_SIZE"
|
|
||||||
--parallel "$PARALLEL"
|
|
||||||
--batch-size "$BATCH_SIZE"
|
|
||||||
--ubatch-size "$UBATCH_SIZE"
|
|
||||||
--n-gpu-layers "$NGL"
|
|
||||||
--flash-attn on
|
|
||||||
--cache-type-k "$CACHE_TYPE_K"
|
|
||||||
--cache-type-v "$CACHE_TYPE_V"
|
|
||||||
--jinja
|
|
||||||
--reasoning auto
|
|
||||||
--reasoning-format deepseek
|
|
||||||
)
|
|
||||||
exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}
|
|
||||||
Executable
+132
@@ -0,0 +1,132 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# Serve HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive (Q6_K) as
|
||||||
|
# frank-zealot on Dankclaw's RTX 3090 via llama.cpp (master build, CUDA sm_86).
|
||||||
|
#
|
||||||
|
# 2026-10-07 (Frank): Qwen3.5-9B swap at TWO 200,704-token conversations
|
||||||
|
# (--ctx-size 401408 --parallel 2) with EXACT f16 KV — the 9B rung frees the
|
||||||
|
# room the 26B MoE Gemma4 shape lacked, so quantized V retires. MTP stays off:
|
||||||
|
# this GGUF carries no MTP tensors at all (verified from its tensor-name
|
||||||
|
# list), so there is no in-file head to draft with. Verbosity 4 ships so the
|
||||||
|
# VRAM-fit forensics (KV layout, buffer sizes, graph capture) land in the journal.
|
||||||
|
#
|
||||||
|
# Architecture: qwen35 — dense 9B, 32 layers with full_attention_interval 4
|
||||||
|
# (8 full-attention layers at 3,7,...,31 + 24 Gated DeltaNet linear layers),
|
||||||
|
# 16 heads / 4 KV heads × 256 dim, native 262,144, vocab 248,320. The KV pool
|
||||||
|
# scales with the 8 full layers: 32 KiB/token at f16 K + f16 V (12.25 GiB at
|
||||||
|
# the 401,408 pool); the DeltaNet layers cost a fixed recurrent state per
|
||||||
|
# slot. Feasibility is priced from the MiMo-V2.6-Distill-Qwen-9B boots (same
|
||||||
|
# geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279
|
||||||
|
# at ubatch 2048 — the boot ledger, not arithmetic, is the referee.
|
||||||
|
# Vision comes from the publisher's own BF16 projector (the model is natively
|
||||||
|
# multimodal), so --mmproj is what gives this endpoint eyes; MMPROJ_FILE="" is
|
||||||
|
# the documented text-only opt-out.
|
||||||
|
#
|
||||||
|
# Sampling: llama.cpp takes it per request (the fleet harness sends the
|
||||||
|
# publisher recipe from models.yml: thinking temp 0.6 / top_p 0.95 / top_k 20).
|
||||||
|
#
|
||||||
|
# Tunables come from frank-zealot.env unless FRANK_ZEALOT_ENV_FILE overrides.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
ENV_FILE="${FRANK_ZEALOT_ENV_FILE:-$SCRIPT_DIR/frank-zealot.env}"
|
||||||
|
if [[ -f "$ENV_FILE" ]]; then
|
||||||
|
set -a
|
||||||
|
# shellcheck disable=SC1090
|
||||||
|
source "$ENV_FILE"
|
||||||
|
set +a
|
||||||
|
fi
|
||||||
|
|
||||||
|
# --- defaults (used when the env file does not set them) -------------------
|
||||||
|
: "${LLAMA_BIN:=/home/dank/GitPackages/llama-master/build-cuda/bin/llama-server}"
|
||||||
|
: "${MODEL_FILE:=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-Q6_K.gguf}"
|
||||||
|
: "${ALIAS:=frank-zealot}"
|
||||||
|
: "${BIND_ADDR:=0.0.0.0}"
|
||||||
|
: "${LISTEN_PORT:=7771}"
|
||||||
|
# ctx-size is the POOL llama.cpp divides across slots: 401,408 / 2 = 200,704
|
||||||
|
# per conversation, inside the model's trained 262,144. The harness
|
||||||
|
# contextWindow MUST quote the PER-SLOT number — a longer prompt is answered
|
||||||
|
# with a 400, never truncated.
|
||||||
|
: "${CTX_SIZE:=401408}"
|
||||||
|
: "${PARALLEL:=2}"
|
||||||
|
# Compute-buffer geometry. Ubatch is a prefill knob (decode never moves).
|
||||||
|
# 2048 is the measured CUDA optimum (MiMo era, same arch/shape: +4.6 % prefill
|
||||||
|
# at 8.9k depth, decode flat, 616 MiB). The logical batch is VRAM- and
|
||||||
|
# throughput-neutral at a fixed ubatch.
|
||||||
|
: "${BATCH_SIZE:=4096}"
|
||||||
|
: "${UBATCH_SIZE:=2048}"
|
||||||
|
# Exact (unquantized) KV: f16 K + f16 V, restoring the MiMo-era contract that
|
||||||
|
# the 26B Gemma4 shape could not afford. 32 KiB/token at this geometry
|
||||||
|
# (8 full-attention layers × 4 KV heads × 256 dim); 12.25 GiB at the 401,408
|
||||||
|
# pool. Explicit either way so the contract is visible in the argv, and
|
||||||
|
# --flash-attn stays required for any quantized-V fallback (q8_0 = llama.cpp's
|
||||||
|
# FP8 class; no true fp8 KV dtype exists in this build).
|
||||||
|
: "${CACHE_TYPE_K:=f16}"
|
||||||
|
: "${CACHE_TYPE_V:=f16}"
|
||||||
|
# Speculative decoding is OFF: this Q6_K file carries NO nextn/mtp tensors, so
|
||||||
|
# there is no in-file head (the law: MTP is a property of the file, not of the
|
||||||
|
# flags). SPEC_TYPE=none strips BOTH speculative flags (flags move as a set);
|
||||||
|
# naming a drafter requires SPEC_DRAFT_FILE alongside SPEC_TYPE — the Gemma4
|
||||||
|
# assistant on disk is arch gemma4-assistant and cannot drafter-match qwen35.
|
||||||
|
: "${SPEC_TYPE:=none}"
|
||||||
|
: "${SPEC_DRAFT_N_MAX:=3}"
|
||||||
|
: "${SPEC_DRAFT_FILE:=}"
|
||||||
|
: "${NGL:=99}"
|
||||||
|
# Log verbosity threshold (llama.cpp -lv). 3 is the engine default (info);
|
||||||
|
# 4 adds debug detail (KV layout, buffer sizes, graph capture) which is what
|
||||||
|
# the VRAM-fit forensics need. Levels above 4 flood the journal.
|
||||||
|
: "${VERBOSITY:=3}"
|
||||||
|
# Vision projector. No colon on purpose: an explicitly empty value means
|
||||||
|
# "text-only, on purpose" and must NOT be replaced by the default path.
|
||||||
|
: "${MMPROJ_FILE=/home/dank/models/Qwen35-9B-HauhauCS-Q6K/mmproj-Qwen3.5-9B-Uncensored-HauhauCS-Aggressive-BF16.gguf}"
|
||||||
|
|
||||||
|
# Fail loudly rather than half-loading: a missing file is a download that
|
||||||
|
# has not finished, and Restart=always would turn it into a crash loop that
|
||||||
|
# reads like a runtime fault.
|
||||||
|
for path in "$LLAMA_BIN" "$MODEL_FILE"; do
|
||||||
|
[[ -e "$path" ]] || { echo "frank-zealot: missing $path" >&2; exit 1; }
|
||||||
|
done
|
||||||
|
EXTRA_ARGS=()
|
||||||
|
if [[ -n "$MMPROJ_FILE" ]]; then
|
||||||
|
[[ -e "$MMPROJ_FILE" ]] || { echo "frank-zealot: missing $MMPROJ_FILE" >&2; exit 1; }
|
||||||
|
EXTRA_ARGS+=(--mmproj "$MMPROJ_FILE")
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Speculative flags move as a set: they reach the argv ONLY when a drafter
|
||||||
|
# is named. SPEC_TYPE=none (the shipped contract) strips both — a no-spec
|
||||||
|
# baseline must not carry half a spec pair.
|
||||||
|
if [[ -n "$SPEC_TYPE" && "$SPEC_TYPE" != "none" ]]; then
|
||||||
|
EXTRA_ARGS+=(--spec-type "$SPEC_TYPE" --spec-draft-n-max "$SPEC_DRAFT_N_MAX")
|
||||||
|
if [[ -n "$SPEC_DRAFT_FILE" ]]; then
|
||||||
|
[[ -e "$SPEC_DRAFT_FILE" ]] || { echo "frank-zealot: missing $SPEC_DRAFT_FILE" >&2; exit 1; }
|
||||||
|
EXTRA_ARGS+=(--spec-draft-model "$SPEC_DRAFT_FILE")
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# The UVM device node is created lazily by nvidia-modprobe on the first CUDA
|
||||||
|
# open after boot; make sure it exists before the server needs it.
|
||||||
|
nvidia-modprobe -u -c=0 2>/dev/null || true
|
||||||
|
|
||||||
|
# Flash attention stays pinned on: it is REQUIRED for quantized V (q8_0),
|
||||||
|
# and the build carries GGML_CUDA_FA_ALL_QUANTS=ON (since 2026-09-17), so
|
||||||
|
# the q8_0 KV contract in frank-zealot.env is a real path — without this
|
||||||
|
# flag or that build option the engine silently falls back and the
|
||||||
|
# contract becomes a lie.
|
||||||
|
ARGS=(
|
||||||
|
--model "$MODEL_FILE"
|
||||||
|
--alias "$ALIAS"
|
||||||
|
--host "$BIND_ADDR"
|
||||||
|
--port "$LISTEN_PORT"
|
||||||
|
--ctx-size "$CTX_SIZE"
|
||||||
|
--parallel "$PARALLEL"
|
||||||
|
--batch-size "$BATCH_SIZE"
|
||||||
|
--ubatch-size "$UBATCH_SIZE"
|
||||||
|
--n-gpu-layers "$NGL"
|
||||||
|
--flash-attn on
|
||||||
|
--cache-type-k "$CACHE_TYPE_K"
|
||||||
|
--cache-type-v "$CACHE_TYPE_V"
|
||||||
|
--log-verbosity "$VERBOSITY"
|
||||||
|
--jinja
|
||||||
|
--reasoning auto
|
||||||
|
--reasoning-format deepseek
|
||||||
|
)
|
||||||
|
exec "$LLAMA_BIN" "${ARGS[@]}" ${EXTRA_ARGS[@]+"${EXTRA_ARGS[@]}"}
|
||||||
+11
-11
@@ -145,12 +145,12 @@ for path in "${CHANGED[@]}"; do
|
|||||||
add_test tests/homepage_caddy.bats
|
add_test tests/homepage_caddy.bats
|
||||||
add_test tests/deployment_scoping.bats
|
add_test tests/deployment_scoping.bats
|
||||||
;;
|
;;
|
||||||
# frank-large is a rootless quadlet again since 2026-09-16 (the
|
# frank-dragoon is a rootless quadlet again since 2026-09-16 (the
|
||||||
# halogen-flash-server restore). Its unit carries the fleet's GPU and
|
# halogen-flash-server restore). Its unit carries the fleet's GPU and
|
||||||
# podman posture, so it gets its own seam rather than the generic
|
# podman posture, so it gets its own seam rather than the generic
|
||||||
# fallback below.
|
# fallback below.
|
||||||
dankclaw/frank-large/*)
|
dankclaw/frank-dragoon/*)
|
||||||
add_test tests/frank_large_unit.bats
|
add_test tests/frank_dragoon_unit.bats
|
||||||
add_test tests/deployment_scoping.bats
|
add_test tests/deployment_scoping.bats
|
||||||
add_test tests/docs_completeness.bats
|
add_test tests/docs_completeness.bats
|
||||||
;;
|
;;
|
||||||
@@ -187,29 +187,29 @@ for path in "${CHANGED[@]}"; do
|
|||||||
# as additions, with "no focused-test mapping".
|
# as additions, with "no focused-test mapping".
|
||||||
.AGENTTRASH/*|*/.AGENTTRASH/*)
|
.AGENTTRASH/*|*/.AGENTTRASH/*)
|
||||||
;;
|
;;
|
||||||
# frank-primary is llama.cpp (the unsloth b11030-mix prebuilt) on the
|
# frank-archon is llama.cpp (the unsloth b11030-mix prebuilt) on the
|
||||||
# two R9700s as a ROOT unit — the 2026-09-26 swap that replaced the
|
# two R9700s as a ROOT unit — the 2026-09-26 swap that replaced the
|
||||||
# Lucebox DS4.1 quadlet on :7770. Its env, unit and serve script all
|
# Lucebox DS4.1 quadlet on :7770. Its env, unit and serve script all
|
||||||
# live in scripts/, which is why this case has to precede the generic
|
# live in scripts/, which is why this case has to precede the generic
|
||||||
# scripts/*.service and scripts/* fallbacks below.
|
# scripts/*.service and scripts/* fallbacks below.
|
||||||
scripts/frank-primary.*|scripts/serve-frank-primary.sh)
|
scripts/frank-archon.*|scripts/serve-frank-archon.sh)
|
||||||
add_test tests/frank_primary_unit.bats
|
add_test tests/frank_archon_unit.bats
|
||||||
add_test tests/llm_units.bats
|
add_test tests/llm_units.bats
|
||||||
add_test tests/deployment_scoping.bats
|
add_test tests/deployment_scoping.bats
|
||||||
add_test tests/scripts_syntax.bats
|
add_test tests/scripts_syntax.bats
|
||||||
;;
|
;;
|
||||||
# frank-large's chat template lives in scripts/ (moved there on
|
# frank-dragoon's chat template lives in scripts/ (moved there on
|
||||||
# 2026-09-26 when its old home was retired): a template edit changes
|
# 2026-09-26 when its old home was retired): a template edit changes
|
||||||
# what the live endpoint renders, so it selects the large unit seam.
|
# what the live endpoint renders, so it selects the large unit seam.
|
||||||
scripts/qwen-sharp-v22.4.0.jinja)
|
scripts/qwen-sharp-v22.4.0.jinja)
|
||||||
add_test tests/frank_large_unit.bats
|
add_test tests/frank_dragoon_unit.bats
|
||||||
;;
|
;;
|
||||||
# frank-secondary is llama.cpp on the 3090 as a ROOT unit: a native CUDA
|
# frank-zealot is llama.cpp on the 3090 as a ROOT unit: a native CUDA
|
||||||
# binary, so no generated quadlet is involved. Its env, unit and serve
|
# binary, so no generated quadlet is involved. Its env, unit and serve
|
||||||
# script all live in scripts/, which is why this case has to precede
|
# script all live in scripts/, which is why this case has to precede
|
||||||
# the generic scripts/*.service and scripts/* fallbacks below.
|
# the generic scripts/*.service and scripts/* fallbacks below.
|
||||||
scripts/frank-secondary.*|scripts/serve-frank-secondary.sh)
|
scripts/frank-zealot.*|scripts/serve-frank-zealot.sh)
|
||||||
add_test tests/frank_secondary_unit.bats
|
add_test tests/frank_zealot_unit.bats
|
||||||
add_test tests/llm_units.bats
|
add_test tests/llm_units.bats
|
||||||
add_test tests/deployment_scoping.bats
|
add_test tests/deployment_scoping.bats
|
||||||
add_test tests/scripts_syntax.bats
|
add_test tests/scripts_syntax.bats
|
||||||
|
|||||||
Executable
+104
@@ -0,0 +1,104 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# ttm-preflight.sh — the TTM pages-cap gate for engines that PIN host RAM.
|
||||||
|
#
|
||||||
|
# Sourced, never executed. An engine that pins tens of GiB of host RAM (the
|
||||||
|
# radiance expert pools + the PLE n-gram table) has its pinned pages charged
|
||||||
|
# against the kernel's ttm page cap: /sys/module/ttm/parameters/pages_limit
|
||||||
|
# (and pages_max on kernels that expose one). The cap function below takes the
|
||||||
|
# MIN over the knobs the running kernel actually exposes — an absent knob is
|
||||||
|
# skipped, never read as zero, because a zero would refuse a healthy cap.
|
||||||
|
#
|
||||||
|
# THE LAW THIS FILE EXISTS FOR: the cap is a RUNTIME knob. The kernel default is
|
||||||
|
# half of RAM, so every raise is lost on reboot. On 2026-10-12 23:17 a power
|
||||||
|
# failure on dankserver reset it 200 GiB -> 125,988 MiB, and frank-archon's
|
||||||
|
# preflight refused the start — 6,919 crash-restarts over ~20 hours, with
|
||||||
|
# `Restart=always` cheerfully re-running a refusal. A manual sysfs write is
|
||||||
|
# therefore not an acceptable repair procedure: the gate raises the knob itself
|
||||||
|
# (the units that need it run as root, and the knob is a 0644 root-owned module
|
||||||
|
# parameter), re-reads, and only then refuses.
|
||||||
|
#
|
||||||
|
# Contract (set by the sourcing script's env file):
|
||||||
|
# FRANK_ARCHON_TTM_MIN_MIB floor the pinned ledger needs; refuse below it
|
||||||
|
# FRANK_ARCHON_TTM_TARGET_MIB value to raise the cap to when it is below the
|
||||||
|
# floor (the recorded host ceiling — writing the
|
||||||
|
# floor itself would leave the ledger no slack)
|
||||||
|
# FRANK_ARCHON_TTM_SYS_DIR knob directory; defaults to the real sysfs
|
||||||
|
# path, a test points it at a temp dir.
|
||||||
|
# Pages are 4 KiB: pages = MiB x 256.
|
||||||
|
|
||||||
|
: "${FRANK_ARCHON_TTM_SYS_DIR:=/sys/module/ttm/parameters}"
|
||||||
|
|
||||||
|
# Prints the effective cap in MiB, 0 when no knob is readable.
|
||||||
|
ttm_cap_mib() {
|
||||||
|
local p v cur=""
|
||||||
|
for p in "$FRANK_ARCHON_TTM_SYS_DIR/pages_limit" \
|
||||||
|
"$FRANK_ARCHON_TTM_SYS_DIR/pages_max"; do
|
||||||
|
[ -r "$p" ] || continue
|
||||||
|
v="$(cat "$p" 2>/dev/null)" || continue
|
||||||
|
[ -n "$v" ] || continue
|
||||||
|
v=$(( v * 4 / 1024 ))
|
||||||
|
if [ "$v" -lt "${cur:-999999999}" ]; then cur=$v; fi
|
||||||
|
done
|
||||||
|
echo "${cur:-0}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Writes $1 MiB (as 4 KiB pages) into every knob present. Failures are
|
||||||
|
# swallowed: the caller re-reads the cap, and a write the kernel refused shows
|
||||||
|
# up there (ttm refuses a limit below the pages it has already pinned).
|
||||||
|
#
|
||||||
|
# PRIVILEGE, measured the hard way: the knob is root-owned 0644 and the service
|
||||||
|
# process runs as User=dank, so `-w` is FALSE there and a plain echo fails. The
|
||||||
|
# canonical privileged path is the unit's ExecStartPre with the `+` prefix
|
||||||
|
# (systemd runs it as root even under User=); `sudo -n` is the fallback for a
|
||||||
|
# human running the gate from a shell (passwordless sudo on both hosts).
|
||||||
|
ttm_raise_cap() {
|
||||||
|
local pages=$(( $1 * 256 )) p
|
||||||
|
for p in "$FRANK_ARCHON_TTM_SYS_DIR/pages_limit" \
|
||||||
|
"$FRANK_ARCHON_TTM_SYS_DIR/pages_max"; do
|
||||||
|
[ -e "$p" ] || continue
|
||||||
|
if [ -w "$p" ]; then
|
||||||
|
echo "$pages" > "$p" 2>/dev/null || true
|
||||||
|
elif command -v sudo >/dev/null 2>&1; then
|
||||||
|
sudo -n sh -c "echo $pages > '$p'" >/dev/null 2>&1 || true
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# Returns 0 when the cap clears FRANK_ARCHON_TTM_MIN_MIB, raising it first if
|
||||||
|
# needed; returns 1 with a named refusal when it cannot be made to.
|
||||||
|
ttm_ensure_cap() {
|
||||||
|
local cap
|
||||||
|
cap="$(ttm_cap_mib)"
|
||||||
|
if [ "$cap" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
|
||||||
|
echo "frank-archon: ttm pages cap is ${cap} MiB, below the pinned-ledger" \
|
||||||
|
"floor $FRANK_ARCHON_TTM_MIN_MIB — raising the knob to" \
|
||||||
|
"$FRANK_ARCHON_TTM_TARGET_MIB MiB ($(( FRANK_ARCHON_TTM_TARGET_MIB * 256 )) pages)" >&2
|
||||||
|
ttm_raise_cap "$FRANK_ARCHON_TTM_TARGET_MIB"
|
||||||
|
cap="$(ttm_cap_mib)"
|
||||||
|
fi
|
||||||
|
if [ "$cap" -lt "$FRANK_ARCHON_TTM_MIN_MIB" ]; then
|
||||||
|
echo "ERROR: ttm pages cap is ${cap} MiB, the pinned ledger needs >= $FRANK_ARCHON_TTM_MIN_MIB" >&2
|
||||||
|
echo "frank-archon: raise $FRANK_ARCHON_TTM_SYS_DIR/pages_limit (and pages_max if the kernel exposes one) to >= $(( FRANK_ARCHON_TTM_MIN_MIB * 256 )) in 4 KiB pages; a read-only knob needs the module option (options ttm pages_limit=N) and a reboot" >&2
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
echo "frank-archon: ttm pages cap ${cap} MiB clears the pinned-ledger floor $FRANK_ARCHON_TTM_MIN_MIB" >&2
|
||||||
|
}
|
||||||
|
|
||||||
|
# Executed (not sourced) it is the privileged pre-start hook: systemd runs this
|
||||||
|
# as root from the unit's `ExecStartPre=+…`, where the knob write is legal and
|
||||||
|
# the `Restart=always` loop can never outrun it.
|
||||||
|
if [ "${BASH_SOURCE[0]}" = "${0}" ]; then
|
||||||
|
set -euo pipefail
|
||||||
|
case "${1:-}" in
|
||||||
|
--raise)
|
||||||
|
TTM_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
# shellcheck source=/dev/null
|
||||||
|
source "${FRANK_ARCHON_ENV_FILE:-$TTM_DIR/frank-archon.env}"
|
||||||
|
ttm_ensure_cap
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "usage: ${0##*/} --raise (raise the ttm pages cap, then verify)" >&2
|
||||||
|
exit 2
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
fi
|
||||||
Executable
+87
@@ -0,0 +1,87 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Streaming absmax audit of the Swift1.5 PLE n-gram table (BF16).
|
||||||
|
|
||||||
|
The PLE fp8 encoding uses ONE fixed scale (1.9931793212890625e-04, the
|
||||||
|
official checkpoint's plateau value). fp8_e4m3 saturates at 448, so the
|
||||||
|
fixed scale is legal iff absmax <= 448 * SCALE. This walks every
|
||||||
|
ngram_embedding.shard_*.weight tensor in the checkpoint (mmap, no copy),
|
||||||
|
computes the absmax from the BF16 bit pattern (clear the sign bit, max
|
||||||
|
the remaining uint16 — BF16 magnitudes are monotone in their bits), and
|
||||||
|
reports the headroom. Exit 0 only when headroom >= 1.5x.
|
||||||
|
|
||||||
|
Run from any cwd EXCEPT /tmp on dankserver (/tmp/inspect.py shadows the
|
||||||
|
stdlib). Usage: verify-swift15-ple.py <ckpt-dir>
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
import mmap
|
||||||
|
import os
|
||||||
|
import struct
|
||||||
|
import sys
|
||||||
|
|
||||||
|
SCALE = 1.9931793212890625e-04 # bf16 0x3951, the official plateau scale
|
||||||
|
FP8_E4M3_MAX = 448.0
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
gmax = 0.0
|
||||||
|
gfile = ""
|
||||||
|
over_cap = 0
|
||||||
|
total = 0
|
||||||
|
|
||||||
|
ckpt = sys.argv[1]
|
||||||
|
index = json.load(open(os.path.join(ckpt, "model.safetensors.index.json")))
|
||||||
|
shards: dict[str, list[str]] = {}
|
||||||
|
for k, f in index["weight_map"].items():
|
||||||
|
if "ngram_embedding.shard_" in k and k.endswith(".weight"):
|
||||||
|
shards.setdefault(f, []).append(k)
|
||||||
|
|
||||||
|
files = len(shards)
|
||||||
|
tensors = sum(len(v) for v in shards.values())
|
||||||
|
print(f"n-gram shards: {files} files, {tensors} tensors")
|
||||||
|
|
||||||
|
import math
|
||||||
|
|
||||||
|
cap = FP8_E4M3_MAX * SCALE
|
||||||
|
# largest bf16 <= cap: truncate the float32 bits (cap > 0); bf16
|
||||||
|
# magnitudes are monotone in their bits, so |w| > cap <=> bits > cap_bits
|
||||||
|
cap_bits = int((np.float32(cap).view(np.uint32) >> 16) & 0xFFFF)
|
||||||
|
for fname, keys in sorted(shards.items()):
|
||||||
|
path = os.path.join(ckpt, fname)
|
||||||
|
with open(path, "rb") as fh:
|
||||||
|
hlen = struct.unpack("<Q", fh.read(8))[0]
|
||||||
|
header = json.loads(fh.read(hlen))
|
||||||
|
base = 8 + hlen
|
||||||
|
mm = mmap.mmap(fh.fileno(), 0, prot=mmap.PROT_READ)
|
||||||
|
for k in keys:
|
||||||
|
meta = header[k]
|
||||||
|
off, end = meta["data_offsets"]
|
||||||
|
a = np.frombuffer(mm, dtype=np.uint16, count=(end - off) // 2,
|
||||||
|
offset=base + off)
|
||||||
|
v = a & 0x7FFF
|
||||||
|
m = int(v.max())
|
||||||
|
over_cap += int((v > np.uint16(cap_bits)).sum())
|
||||||
|
total += v.size
|
||||||
|
del a
|
||||||
|
if m > gmax:
|
||||||
|
gmax = m
|
||||||
|
gfile = k
|
||||||
|
mm.close()
|
||||||
|
|
||||||
|
# rebuild the BF16 value: the 16 bits are the HIGH half of the float32
|
||||||
|
absmax = np.frombuffer(b"\x00\x00" + np.uint16(gmax).tobytes(),
|
||||||
|
dtype=np.float32)[0]
|
||||||
|
|
||||||
|
print(f"absmax = {absmax!r} (tensor: {gfile})")
|
||||||
|
print(f"2^log2 = 2^{math.log2(absmax):.3f}")
|
||||||
|
print(f"fp8 cap = 448 * {SCALE:.8g} = {cap:.8g}")
|
||||||
|
print(f"headroom = {cap / absmax:.3f}x")
|
||||||
|
ok = cap / absmax >= 1.5
|
||||||
|
print("VERDICT:", "PASS (>= 1.5x headroom over the fixed scale)" if ok
|
||||||
|
else "FAIL — fixed scale does not cover the Swift PLE")
|
||||||
|
print(f"values > cap = {over_cap} of {total} ({over_cap / total:.3e})")
|
||||||
|
nxt_bits = int((np.float32(absmax / FP8_E4M3_MAX).view(np.uint32)
|
||||||
|
>> 16) & 0xFFFF) + 1
|
||||||
|
nxt_scale = np.frombuffer(b"\x00\x00" + np.uint16(nxt_bits).tobytes(),
|
||||||
|
dtype=np.float32)[0]
|
||||||
|
print(f"next-bf16 scale covering absmax: {nxt_scale:.8g} "
|
||||||
|
f"(0x{nxt_bits:04x}, headroom {FP8_E4M3_MAX * nxt_scale / absmax:.4f}x)")
|
||||||
|
sys.exit(0 if ok else 1)
|
||||||
@@ -13,6 +13,18 @@ registry; single source of truth — there is no copy on the servers) and
|
|||||||
`~/.omp/agent/config.yml` (roles, compaction, global samplers). Nothing in
|
`~/.omp/agent/config.yml` (roles, compaction, global samplers). Nothing in
|
||||||
this file is repo-managed; edits are harness-side only.
|
this file is repo-managed; edits are harness-side only.
|
||||||
|
|
||||||
|
**Model ids are the StarCraft unit names (official 2026-10-08, on the wire
|
||||||
|
since 2026-10-09):** `frank-archon` (dankserver :7770, radiance),
|
||||||
|
`frank-dragoon` (dankclaw :7770, flash-server), `frank-zealot` (dankclaw
|
||||||
|
:7771, the 3090 compaction hub) — formerly `frank-archon`, `frank-dragoon`,
|
||||||
|
`frank-zealot`. The SERVICES keep the old names (units, scripts, env
|
||||||
|
files, runbook pages); pages written in the old era name the service,
|
||||||
|
the harness names the model. The wire now matches: each endpoint's
|
||||||
|
`/v1/models` answers with the Protoss name, so `models.yml`, the OpenWebUI
|
||||||
|
picker/grants, the homepage docs and the raw API all use one id per model
|
||||||
|
(see [[OpenWebUI]] for the grant re-key and the [[frank-archon-radiance-swift15-w4dn8]]
|
||||||
|
note on radiance echoing the requested name).
|
||||||
|
|
||||||
## Compaction chain arithmetic (verify before ANY window change)
|
## Compaction chain arithmetic (verify before ANY window change)
|
||||||
|
|
||||||
When a fleet endpoint's context grows past its harness-declared window, OMP
|
When a fleet endpoint's context grows past its harness-declared window, OMP
|
||||||
@@ -33,14 +45,14 @@ The arithmetic (the coupled contract every endpoint's maxTokens comment
|
|||||||
encodes):
|
encodes):
|
||||||
|
|
||||||
1. **Source trigger** = `thresholdPercent × source contextWindow`
|
1. **Source trigger** = `thresholdPercent × source contextWindow`
|
||||||
(0.70 × 200,704 = 140,492 for frank-primary on the current shape)
|
(0.70 × 200,704 = 140,492 for frank-archon on the current shape)
|
||||||
2. **Output budget** = capped at the target model's `maxTokens`
|
2. **Output budget** = capped at the target model's `maxTokens`
|
||||||
(32,768 for frank-secondary; 36,864 frank-primary; 57,344 frank-large — the
|
(32,768 for frank-zealot; 36,864 frank-archon; 57,344 frank-dragoon — the
|
||||||
largest, because halogen's `HALOGEN_MAX_TOKENS_CAP` must stay ≥ it)
|
largest, because halogen's `HALOGEN_MAX_TOKENS_CAP` must stay ≥ it)
|
||||||
3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
|
3. **REQUIREMENT**: prompt (≤ trigger) + budget ≤ the target's per-slot
|
||||||
window. Worked pairs from the live registry: 140,492 + 36,864 = 177,356 ≤
|
window. Worked pairs from the live registry: 140,492 + 36,864 = 177,356 ≤
|
||||||
200,704 (frank-primary); 140,490 + 57,344 = 197,834 ≤ 200,700
|
200,704 (frank-archon); 140,490 + 57,344 = 197,834 ≤ 200,700
|
||||||
(frank-large); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-secondary).
|
(frank-dragoon); 140,492 + 32,768 = 173,260 ≤ 200,704 (frank-zealot).
|
||||||
|
|
||||||
Procedure (recompute from the LIVE files every time — windows rotate with
|
Procedure (recompute from the LIVE files every time — windows rotate with
|
||||||
every swap; the numbers above are the 2026-10-05 state):
|
every swap; the numbers above are the 2026-10-05 state):
|
||||||
@@ -100,19 +112,19 @@ Use **wire names**, not OMP camelCase. Two traps:
|
|||||||
`qwenPreserveThinking` is auto-derived in `buildOpenAICompat` from the
|
`qwenPreserveThinking` is auto-derived in `buildOpenAICompat` from the
|
||||||
*auto-detected* `thinkingFormat` + local baseUrl, **before**
|
*auto-detected* `thinkingFormat` + local baseUrl, **before**
|
||||||
`applyCompatOverrides`. Setting `compat.thinkingFormat: qwen` in
|
`applyCompatOverrides`. Setting `compat.thinkingFormat: qwen` in
|
||||||
models.yml does **not** enable it. Custom-named models (`frank-large`)
|
models.yml does **not** enable it. Custom-named models (`frank-dragoon`)
|
||||||
match no Qwen id pattern → use `extraBody: {preserve_thinking: true}`.
|
match no Qwen id pattern → use `extraBody: {preserve_thinking: true}`.
|
||||||
2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy
|
2. **Never put `chat_template_kwargs` in `extraBody`** where a compat policy
|
||||||
also sets it — `Object.assign` is shallow and replaces the whole object,
|
also sets it — `Object.assign` is shallow and replaces the whole object,
|
||||||
silently dropping the `reasoning_effort` kwarg the compat policy set.
|
silently dropping the `reasoning_effort` kwarg the compat policy set.
|
||||||
Top-level `preserve_thinking` is honoured by halogen. (frank-primary's
|
Top-level `preserve_thinking` is honoured by halogen. (frank-archon's
|
||||||
FreeToken entry currently carries a `chat_template_kwargs` extraBody with
|
FreeToken entry currently carries a `chat_template_kwargs` extraBody with
|
||||||
a comment asking to verify-and-prune — the exact trap to check at that
|
a comment asking to verify-and-prune — the exact trap to check at that
|
||||||
endpoint's next swap.)
|
endpoint's next swap.)
|
||||||
|
|
||||||
**Gotcha: silent greedy decode.** halogen has **no server-side sampler
|
**Gotcha: silent greedy decode.** halogen has **no server-side sampler
|
||||||
defaults** — `temperature` absent or `0` = greedy. If OMP sends nothing, the
|
defaults** — `temperature` absent or `0` = greedy. If OMP sends nothing, the
|
||||||
endpoint decodes greedy while looking configured. This is why frank-large's
|
endpoint decodes greedy while looking configured. This is why frank-dragoon's
|
||||||
entry carries the recipe on the wire and a comment saying "do not drop it".
|
entry carries the recipe on the wire and a comment saying "do not drop it".
|
||||||
|
|
||||||
**Reasoning-effort folding.** Qwen3.8's template accepts only
|
**Reasoning-effort folding.** Qwen3.8's template accepts only
|
||||||
|
|||||||
@@ -23,7 +23,7 @@ marker. So `marker != HEAD` means "not dispatched via deploy.sh", NOT
|
|||||||
|
|
||||||
**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
|
**Why it matters:** a plain `deploy.sh` run re-selects and restarts a service
|
||||||
whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of
|
whose marker lags. On llama.cpp/R9V endpoints that is a ~16–19 min reload of
|
||||||
a 100+ GiB payload — and frank-primary (dankserver:7770) is also the
|
a 100+ GiB payload — and frank-archon (dankserver:7770) is also the
|
||||||
harness's own model backend, so a blind deploy is a self-inflicted outage for
|
harness's own model backend, so a blind deploy is a self-inflicted outage for
|
||||||
possibly zero behavioral change.
|
possibly zero behavioral change.
|
||||||
|
|
||||||
|
|||||||
@@ -62,8 +62,8 @@ service on the other host.
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
|
| Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
|
||||||
| SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. |
|
| SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. |
|
||||||
| `frank-large` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
|
| `frank-dragoon` | systemd + `systemctl` (root unit) | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
|
||||||
| `frank-secondary` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
|
| `frank-zealot` | systemd + `systemctl` (root unit) | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
|
||||||
| `frank-medium` | rootless Quadlet + `systemctl --user` | Installs `dankserver/frank-medium/quadlet/frank-medium.container`; on first deploy retires the llama.cpp root unit and the vLLM replica quadlets into the home trash, then restarts the generated unit. Waits up to 20 minutes for `/health`. |
|
| `frank-medium` | rootless Quadlet + `systemctl --user` | Installs `dankserver/frank-medium/quadlet/frank-medium.container`; on first deploy retires the llama.cpp root unit and the vLLM replica quadlets into the home trash, then restarts the generated unit. Waits up to 20 minutes for `/health`. |
|
||||||
| Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
|
| Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver/<svc>/quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/<svc>` to `/home/dank/Containers/<svc>` (mapped-uid roots move without chown). Steady state is restart-only. |
|
||||||
|
|
||||||
|
|||||||
@@ -95,7 +95,7 @@ host-side `workflows/` dir and the repo copy.
|
|||||||
## Recommended additions
|
## Recommended additions
|
||||||
|
|
||||||
Quantized preferred, sized for the 8060S iGPU (GTT-backed; the RTX 3090 is
|
Quantized preferred, sized for the 8060S iGPU (GTT-backed; the RTX 3090 is
|
||||||
frank-secondary's). All repos verified 2026-10-05. None are on disk yet.
|
frank-zealot's). All repos verified 2026-10-05. None are on disk yet.
|
||||||
|
|
||||||
| # | What | Repo | Grab | Size | Why |
|
| # | What | Repo | Grab | Size | Why |
|
||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
|
|||||||
@@ -13,7 +13,7 @@ timestamp: 2026-08-26
|
|||||||
# Dankclaw Stack
|
# Dankclaw Stack
|
||||||
|
|
||||||
Dankclaw (192.168.0.143) runs the AI, search, development, and observability
|
Dankclaw (192.168.0.143) runs the AI, search, development, and observability
|
||||||
stack. Its LLM inference endpoint `frank-large` (`:7770`) is a ROCmFPX
|
stack. Its LLM inference endpoint `frank-dragoon` (`:7770`) is a ROCmFPX
|
||||||
llama.cpp fork serving the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF on
|
llama.cpp fork serving the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF on
|
||||||
the card's pinned runtime; an NPU endpoint (`frank-npu`, `:7771`,
|
the card's pinned runtime; an NPU endpoint (`frank-npu`, `:7771`,
|
||||||
FLM v1.0.1) serves sight+sound models. The completion endpoint
|
FLM v1.0.1) serves sight+sound models. The completion endpoint
|
||||||
@@ -30,7 +30,7 @@ graph LR
|
|||||||
S --> V[Gluetun VPN]
|
S --> V[Gluetun VPN]
|
||||||
O --> L[llama.cpp dankserver :7770]
|
O --> L[llama.cpp dankserver :7770]
|
||||||
O --> H[Hermes]
|
O --> H[Hermes]
|
||||||
O --> E[frank-large :7770]
|
O --> E[frank-dragoon :7770]
|
||||||
```
|
```
|
||||||
|
|
||||||
## Services
|
## Services
|
||||||
@@ -48,14 +48,14 @@ graph LR
|
|||||||
| Seerr | `seerr.franksdank.cloud` (guarded) | Caddy routes post-auth to `seerr:5055` | [[Seerr]] |
|
| Seerr | `seerr.franksdank.cloud` (guarded) | Caddy routes post-auth to `seerr:5055` | [[Seerr]] |
|
||||||
| Homepage | `franksdank.cloud` + `www.` (root site; guarded for admin actions) | served by Caddy itself from `/srv/homepage` | — |
|
| Homepage | `franksdank.cloud` + `www.` (root site; guarded for admin actions) | served by Caddy itself from `/srv/homepage` | — |
|
||||||
| Pi-hole | `pihole.franksdank.cloud` (basicauth edge, not the guard chain) | admin UI `8083`; DNS on `:53` LAN-wide | [[Pi-hole]] |
|
| Pi-hole | `pihole.franksdank.cloud` (basicauth edge, not the guard chain) | admin UI `8083`; DNS on `:53` LAN-wide | [[Pi-hole]] |
|
||||||
| frank-large (host service) | — | `7770` | [[deployment]] |
|
| frank-dragoon (host service) | — | `7770` | [[deployment]] |
|
||||||
| frank-npu (host service) | — | `7771` (`/v1/version` health) | [[frank-npu]] |
|
| frank-npu (host service) | — | `7771` (`/v1/version` health) | [[frank-npu]] |
|
||||||
| frank-medium (host service, Dankserver) | — | `7770` | [[frank-medium-rocmfpx]] |
|
| frank-medium (host service, Dankserver) | — | `7770` | [[frank-medium-rocmfpx]] |
|
||||||
|
|
||||||
The LLM rows are bare-metal systemd services, not containers. This
|
The LLM rows are bare-metal systemd services, not containers. This
|
||||||
repo owns their scripts, units, and knowledge base (see
|
repo owns their scripts, units, and knowledge base (see
|
||||||
[LLM fleet knowledge base](../LLM/index.md)). Current mapping (2026-09-03):
|
[LLM fleet knowledge base](../LLM/index.md)). Current mapping (2026-09-03):
|
||||||
`frank-large` runs the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF
|
`frank-dragoon` runs the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF
|
||||||
on the card's pinned `kingjones30/ROCmFPX` llama.cpp build;
|
on the card's pinned `kingjones30/ROCmFPX` llama.cpp build;
|
||||||
`frank-npu` runs FLM v1.0.1 on the XDNA2 NPU;
|
`frank-npu` runs FLM v1.0.1 on the XDNA2 NPU;
|
||||||
`frank-medium` runs charlie12345/ROCmFPX llama.cpp serving
|
`frank-medium` runs charlie12345/ROCmFPX llama.cpp serving
|
||||||
|
|||||||
@@ -47,7 +47,7 @@ is not durable. The tracked, sanitized configuration baseline is
|
|||||||
`0.0.0.0:8080`, so the Signal HTTP API is reachable from the whole LAN.
|
`0.0.0.0:8080`, so the Signal HTTP API is reachable from the whole LAN.
|
||||||
- The data volume persists skills, memory, and session history.
|
- The data volume persists skills, memory, and session history.
|
||||||
- Hermes has two named local providers: `dankclaw-large` reaches
|
- Hermes has two named local providers: `dankclaw-large` reaches
|
||||||
`http://192.168.0.143:7770/v1` and exposes `frank-large`;
|
`http://192.168.0.143:7770/v1` and exposes `frank-dragoon`;
|
||||||
`dankserver-small` reaches `http://192.168.0.235:7770/v1` and exposes
|
`dankserver-small` reaches `http://192.168.0.235:7770/v1` and exposes
|
||||||
`frank-medium` (renamed from `frank-small` on 2026-09-15).
|
`frank-medium` (renamed from `frank-small` on 2026-09-15).
|
||||||
- The live `config.yaml` contains secrets and user state and is not committed;
|
- The live `config.yaml` contains secrets and user state and is not committed;
|
||||||
@@ -64,7 +64,7 @@ is not durable. The tracked, sanitized configuration baseline is
|
|||||||
## Gotchas
|
## Gotchas
|
||||||
|
|
||||||
- Hermes is under active development; pin the image tag for reproducibility.
|
- Hermes is under active development; pin the image tag for reproducibility.
|
||||||
- Model choice drives quality; the local `frank-large` (Dankclaw, 7770) is the intended backend.
|
- Model choice drives quality; the local `frank-dragoon` (Dankclaw, 7770) is the intended backend.
|
||||||
- Do not expose the gateway without auth — put it behind [[Caddy]] with basic auth.
|
- Do not expose the gateway without auth — put it behind [[Caddy]] with basic auth.
|
||||||
|
|
||||||
## Browser automation (Chrome DevTools Protocol daemon)
|
## Browser automation (Chrome DevTools Protocol daemon)
|
||||||
@@ -189,7 +189,7 @@ What actually breaks, and the fix:
|
|||||||
auxiliary:
|
auxiliary:
|
||||||
title_generation:
|
title_generation:
|
||||||
provider: dankserver-small # names track the live config; diff first
|
provider: dankserver-small # names track the live config; diff first
|
||||||
model: frank-secondary
|
model: frank-zealot
|
||||||
api_mode: codex_responses # real per-task key
|
api_mode: codex_responses # real per-task key
|
||||||
extra_body:
|
extra_body:
|
||||||
reasoning:
|
reasoning:
|
||||||
|
|||||||
Loaded 100 of 280 files, more files were not shown because too many files have changed in this diff.
Show more
Reference in new issue
Block a user