Compare commits
5
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f96898d4c5 | ||
|
|
3e140cd077 | ||
|
|
c5ac0725f7 | ||
|
|
f44c39812f | ||
|
|
6dcf4a5913 |
No files matched your search
@@ -25,7 +25,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | ukisai/Swift1.5-Qwen3.8-Flash-Next BF16 (335.3 GiB, 397/397 sha256-verified) converted by `scripts/convert-swift15-w4dn8.sh` to `qwen4exp-swift15-w4dn8-i8-hc8m.rad` (**116,416 MiB**): **all 49,152 experts GPTQ w4** (10M-token calibration on engine 1.2.0) + **24 whole bf16-protected experts/layer** (energy-share law), int8 trunk/lm_head, fp8 HC, PLE n-gram fp8 fixed-scale, on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **44 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the **30,969 MiB** the engine's own budget table makes claimable: card 32,624 MiB, settle floor 209 MiB), **47.7 GiB n-gram copy pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV** (pool 5,120 MiB = the 3-slot contract + 1.3 % — the only dtype that funds 3 × 200,000 without gutting the slab, since bf16 bills 13.8 KiB/token), **MTP spec decode depth 2**, `--max-num-seqs 3` (the capacity contract is 3 × 200,000; 3 × (1+2) = 9 all-reduce rows, inside the 25-row wht6 exactness bound), `--max-model-len 200000`, MNBT 2048, Froggeric v22.5 template. KLD-gated (mean KL 0.0131, top-1 97.92 % vs BF16); decode ~190 t/s live (gate bar 30.8); prefill is **slab-limited and pays for it** — 22,208 → 3,397, 24,000 → 3,290, 25,600 → **3,584** t/s at 43.5k single-stream, 3-way 197k aggregate 3,051 → **3,589** t/s, with a cliff at ~26 GiB (26,071 → 4,379). Cutover 2026-10-08 from fp8-native (payload on disk = rollback); 3-slot fp8-KV shape 2026-10-13, re-cut twice in the slab's favour (22208/6912 at 4 slots → 24000/5120 → 25600/5120); the 4-slot shape (22208/6912, `--max-num-seqs 4`) is the documented rollback. | 7770 |
|
||||
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | **Publisher payload** (swap 2026-10-15, MODEL_FILE-only): `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` → **`qwen3.8-next-flash-fp8-iq4r-moe.rad`** (121,969,901,568 B = **116,322 MiB**, Hub-LFS sha256 `0af5e962…` verified at fetch by `scripts/download-frank-archon-iq4r.sh`) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as **4-bit codes into a 16-level non-uniform codebook** (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), **10 whole bf16-protected experts/layer**, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), **MTP head + bf16 vision tower in the container**; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. **Base-alignment (censored) model** — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: **3 × 200,000 fp8-KV** (`--max-num-seqs 3`), **25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable**, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the **w4dn8 record** (the bar to beat; new-payload probes pending). Rollback: revert the swap commit (w4dn8 `.rad` on disk); the fp8-native `.rad` + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts). | 7770 |
|
||||
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 262,144 ctx** (KV pool 524,288 positions ≈ 14.0 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), harness window 217,088 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold)|7770|
|
||||
| Dankclaw | `frank-zealot` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive **Q6_K** (7,359,259,008 B; dense `qwen35` hybrid: 32 layers, 8 full-attention + 24 Gated DeltaNet, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP tensors in the file**) + the publisher's own **BF16 mmproj** (921,704,448 B) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **exact f16 K + f16 V** under flash attention (32 KiB/token over the 8 full layers = 12.25 GiB pool), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (forced: no `nextn` tensors exist). Priced from the MiMo Q6_K boots (same geometry, same card, same shape): 21,276/2,849 at ubatch 1024, 21,846/2,279 at 2048. Cutover 2026-10-07 (late) from Gemma4-26B-A4B Q4_K_M 1×200,704. All three endpoints route harness **compaction** here. | 7771 |
|
||||
|
||||
@@ -498,32 +498,33 @@ Rollback: revert the swap commit — the Gemma4 payload
|
||||
(`/home/dank/models/Ornith-Q8/`) all remain on disk.
|
||||
|
||||
|
||||
### frank-archon — Radiance (StillDeadcode), Swift1.5-Qwen3.8-Flash-Next GPTQ w4, TP=2 (Dankserver, ACTIVE)
|
||||
### frank-archon — Radiance (StillDeadcode), publisher Qwen3.8-Flash-Next iq4r, TP=2 (Dankserver, ACTIVE)
|
||||
|
||||
The **radiance engine** (StillDeadcode, Apache-2.0; image
|
||||
`stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned, **pin 1.0.13**
|
||||
for serve/convert/KLD; `libr4d` gfx1201 kernels, plain `podman run`, root
|
||||
systemd unit, container name = unit name) serving
|
||||
**ukisai/Swift1.5-Qwen3.8-Flash-Next (BF16, 335.3 GiB, 131 shards,
|
||||
397/397 sha256-verified)** converted by `scripts/convert-swift15-w4dn8.sh`
|
||||
with recipe `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe` to
|
||||
`qwen4exp-swift15-w4dn8-i8-hc8m.rad` (~114 GiB): **all 49,152 experts GPTQ
|
||||
w4** (act-order, damp 0.05, 10,000,000-token calibration captured on engine
|
||||
**1.2.0** via `RADIANCE_CALIB_DIR`/`RADIANCE_CALIB_TOKENS` — the only pin
|
||||
with the token knob; the RADGRAM2 grams are format-identical across pins and
|
||||
cross the boundary by design), **24 whole experts per layer cast bf16** (the
|
||||
PROTECTED set, derived from the calibration grams' energy share),
|
||||
**int8 trunk/lm_head**, **fp8 HC mixtures** (`--hc-mix-dtype`), PLE n-gram
|
||||
table at fp8 fixed scale (47.7 GiB in the container, from 95.4 GiB bf16).
|
||||
The plugin's **one-table-one-encoding law** (measured at rad-convert
|
||||
declare): one expert table takes ONE encoding; only whole bf16 experts are
|
||||
legal beside it — no w4/fp8 split of the down/gate_up pair (24-value
|
||||
`rad_arch_quant_desc` array, one per layer). KLD gate vs the BF16
|
||||
reference (recorded on the same 1.0.13 pin) is the precision referee;
|
||||
the throughput gate (decode ≥ 30.8 t/s) was passed at 4.5× the bar
|
||||
(122–173 t/s at 44.5k depth). Measured decode/prefill in the runbook.
|
||||
Rollback: revert the swap commit — the fp8-native `.rad`
|
||||
(`qwen4exp-fp8-native.rad`) and every prior payload remain on disk.
|
||||
for serve; `libr4d` gfx1201 kernels, plain `podman run`, root
|
||||
systemd unit, container name = unit name) serving the
|
||||
**publisher's container of the official Qwen/Qwen3.8-Flash-Next**:
|
||||
`StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` →
|
||||
`qwen3.8-next-flash-fp8-iq4r-moe.rad` (116,322 MiB, Hub-LFS sha256
|
||||
verified at fetch by `scripts/download-frank-archon-iq4r.sh`; recipe
|
||||
`qwen4exp-w4nl64-i8-hc8m`): **routed experts as 4-bit codes into a
|
||||
16-level non-uniform codebook** after a 128-point Walsh–Hadamard rotation
|
||||
(E4M3 scale per 64 weights; codes by GPTQ on a 10M-token calibration with
|
||||
per-expert down-proj Hessians, refined by 3 coordinate-descent sweeps),
|
||||
**10 whole bf16-protected experts/layer**, **int8 W8A8 trunk/lm_head**
|
||||
(scale per 128 columns searched), **fp8 HC mixtures**, PLE n-gram table at
|
||||
fp8 fixed scale (47.7 GiB), **MTP head and bf16 vision tower in the
|
||||
container**. Card quality (full-vocab teacher-forced KL vs bf16, ~75K
|
||||
positions): mean 0.0789 all-tokens / 0.0239 assistant turns, top-1 92.85 %.
|
||||
**MODEL_FILE is the only serving argument that moved** (2026-10-15); every
|
||||
number of the 2026-10-13 shape below carries verbatim. The plugin's
|
||||
**one-table-one-encoding law** (fleet-measured at rad-convert declare):
|
||||
one expert table takes ONE encoding; only whole bf16 experts are legal
|
||||
beside it. Rollback: revert the swap commit — the in-house w4dn8 `.rad`
|
||||
(KLD-gated Swift1.5 conversion) stays on disk; the fp8-native `.rad` was
|
||||
deleted 2026-10-15 (rebuild: `scripts/download-frank-archon.sh` +
|
||||
`scripts/build-frank-archon-rad.sh`).
|
||||
|
||||
**Shape (both R9700s, TP=2)**: `--tp-wire wht6`, `--max-num-seqs 3`,
|
||||
`--max-model-len 200000` (the harness contextWindow must quote this),
|
||||
@@ -630,8 +631,9 @@ chat format — no parser flags; vision native (the 27-layer Qwen-VL tower,
|
||||
`--mm-max-patches`); `/metrics` carries vLLM-compatible names +
|
||||
`radiance:*` families — the Dankclaw Prometheus scrape is re-added.
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md`.
|
||||
Retired fp8-native era: `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md`.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md`.
|
||||
Retired Swift1.5 w4dn8 era (payload on disk = rollback): `serverAppConfigs-docs/LLM/frank-archon-radiance-swift15-w4dn8.md`.
|
||||
Retired fp8-native era (payload deleted 2026-10-15, rebuildable): `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md`.
|
||||
Retired FreeToken era: `serverAppConfigs-docs/LLM/frank-archon-freetoken-pr587.md`.
|
||||
Retired vLLM era: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
|
||||
Retired GLM experiment: `serverAppConfigs-docs/LLM/frank-archon-glm53-flash-w4a16.md`.
|
||||
@@ -1091,7 +1093,7 @@ Active:
|
||||
|---------|---------|
|
||||
| frank-dragoon (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×262,144, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md` |
|
||||
| frank-zealot (llama.cpp CUDA, Qwen3.5-9B-Uncensored Q6_K, 2×200,704 exact f16 KV MTP-off, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-zealot-llama-qwen35-9b-q6k-2x200704.md` |
|
||||
| frank-archon (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-native-fp8.md` |
|
||||
| frank-archon (radiance StillDeadcode, publisher Qwen3.8-Flash-Next iq4r .rad, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-archon-radiance-qwen38-iq4r.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
| Deployment flow + incidents | `serverAppConfigs-docs/LLM/deployment.md` |
|
||||
|
||||
@@ -1099,6 +1101,7 @@ Retired (rollback targets and history):
|
||||
|
||||
| Runtime | Host | Window | Runbook |
|
||||
|---------|------|--------|---------|
|
||||
| radiance Swift1.5-Qwen3.8-Flash-Next GPTQ w4dn8, 3×200,000 fp8-KV slab 25600 (`frank-archon`) | Dankserver | 2026-10-08 → 10-15 (rollback target — `.rad` on disk; owns the slab-slope ladder, KV dtype law, KLD campaign, and every shape law the iq4r swap carried verbatim) | `frank-archon-radiance-swift15-w4dn8.md` |
|
||||
| llama.cpp CUDA Gemma4-26B-A4B QAT Q4_K_M, 1×200,704 f16-K/q8_0-V MTP-off (`frank-zealot`) | Dankclaw | 2026-10-07 → 10-07 (same-day swap; rollback target — payload on disk; owns the `attention_k_eq_v` pair-billing KV law and the gemma4-assistant drafter-cost boots) | `frank-zealot-llama-gemma4-q4km-200704.md` |
|
||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-dragoon`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-dragoon-rocmfpx-q6agent-uncensored-mtp.md` |
|
||||
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-archon`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-archon-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
||||
|
||||
@@ -27,8 +27,7 @@ CALIB=/home/dank/corpora/calib
|
||||
OUT=/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad
|
||||
LOG=/home/dank/logs/swift15_convert.log
|
||||
# gate_up ~37.6 + down ~18.8 + PLE 47.7 + trunk/embeddings/head ~10 GiB (all-w4 expert table;
|
||||
# the format assigns ONE encoding per expert table, so a per-tensor block-fp8
|
||||
# down tier cannot be served beside a w4 gate_up tier -- see the recipe header).
|
||||
# the plugin's one-encoding law dropped the block-fp8 down tier -- see the recipe header).
|
||||
MIN_MIB=${FRANK_SWIFT15_RAD_MIN_MIB:-108000}
|
||||
|
||||
# Gate 1: inputs. A missing Gram directory converts UNCALIBRATED with a
|
||||
|
||||
+12
-10
@@ -947,7 +947,7 @@ deploy_plex() {
|
||||
# accepted and then answered with nothing (curl 52, Empty reply) against a
|
||||
# perfectly healthy server. Measured 2026-10-02 — the gate burned its full
|
||||
# 600s and failed a plex whose https://127.0.0.1:32400/identity answered
|
||||
# 200 in 44 ms the whole time. The same requirement is why the Caddyfile dials the
|
||||
# 200 in 44 ms the whole time. The same law is why the Caddyfile dials the
|
||||
# plex upstream over TLS (tests/caddyfile.bats pins it).
|
||||
# -k: PMS presents its own self-signed cert on 32400.
|
||||
#
|
||||
@@ -1207,9 +1207,8 @@ deploy_llm_dragoon() {
|
||||
# on the FULL container name, which appears in the argv of a manual
|
||||
# `podman run --name frank-dragoon-halogen` / `podman wait` but NOT in
|
||||
# this deployer's own command line (`bash deploy-remote.sh
|
||||
# frank-dragoon`) — so the sweep can never match the deployer's own argv
|
||||
# (`pkill -f` kills the invoker when the pattern appears in the invoker's
|
||||
# own command line), and it can never match frank-zealot's llama-server.
|
||||
# frank-dragoon`) — so it cannot shoot itself in the foot (the fleet
|
||||
# pkill law), and it can never match frank-zealot's llama-server.
|
||||
run_privileged systemctl stop frank-dragoon.service 2>/dev/null || true
|
||||
pkill -f 'frank-dragoon-halogen' 2>/dev/null || true
|
||||
podman rm -f frank-dragoon frank-dragoon-halogen >/dev/null 2>&1 || true
|
||||
@@ -1278,11 +1277,14 @@ deploy_llm_zealot() {
|
||||
}
|
||||
|
||||
# deploy_llm_archon — frank-archon is the radiance engine (StillDeadcode,
|
||||
# digest-pinned stilldeadcode/radiance image) serving the OFFICIAL
|
||||
# Qwen/Qwen3.8-Flash-Next-FP8 at native block-FP8 precision from the
|
||||
# rad-convert container qwen4exp-fp8-native.rad, on BOTH R9700s (TP=2) as a
|
||||
# digest-pinned stilldeadcode/radiance image) serving the PUBLISHER's
|
||||
# 4-bit-codebook container of the official Qwen/Qwen3.8-Flash-Next
|
||||
# (qwen3.8-next-flash-fp8-iq4r-moe.rad, fetched by
|
||||
# scripts/download-frank-archon-iq4r.sh) on BOTH R9700s (TP=2) as a
|
||||
# ROOT unit whose serve script drives `podman run` directly (the engine is
|
||||
# the image entrypoint; the image lives in dank's rootless podman storage).
|
||||
# 2026-10-15 payload swap from the in-house w4dn8 (payload on disk =
|
||||
# rollback).
|
||||
# 2026-10-05 cutover from FreeToken (pre-cutover tag
|
||||
# frank-archon-freetoken-pr587-20261005); before that, the tcclaviger vLLM
|
||||
# fork (tag frank-archon-vllm-shapeG-20261003). Same port and alias; the
|
||||
@@ -1291,7 +1293,7 @@ deploy_llm_zealot() {
|
||||
# Gates: the image must exist under its digest ref in dank's rootless
|
||||
# storage (FRANK_ARCHON_IMAGE), the .rad container file must exist at the
|
||||
# env's size floor (FRANK_ARCHON_MODEL_FILE >= FRANK_ARCHON_RAD_MIN_MIB —
|
||||
# a truncated convert is a guaranteed long-timeout boot), and the unit env
|
||||
# a truncated fetch is a guaranteed long-timeout boot), and the unit env
|
||||
# must be complete. Per-file sha256 verification is the fetch script's job
|
||||
# at download time and rad-convert/rad-info's job at build time, never
|
||||
# re-hashed here.
|
||||
@@ -1326,13 +1328,13 @@ deploy_llm_archon() {
|
||||
fi
|
||||
if [[ ! -f "$model_dir/$model_file" ]]; then
|
||||
echo "ERROR: .rad container missing: $model_dir/$model_file" >&2
|
||||
echo "frank-archon: run scripts/build-frank-archon-rad.sh" >&2
|
||||
echo "frank-archon: run scripts/download-frank-archon-iq4r.sh --fetch" >&2
|
||||
return 1
|
||||
fi
|
||||
mib=$(( $(stat -c %s "$model_dir/$model_file") / 1048576 ))
|
||||
if [[ "$mib" -lt "$rad_min" ]]; then
|
||||
echo "ERROR: .rad container too small: $model_dir/$model_file is ${mib} MiB, want >= $rad_min" >&2
|
||||
echo "frank-archon: rebuild it (scripts/build-frank-archon-rad.sh --force)" >&2
|
||||
echo "frank-archon: re-fetch it (scripts/download-frank-archon-iq4r.sh --fetch; delete the staged file first if its sha256 failed)" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
"""derive-protected-experts.py — rewrite the recipe's protected-expert section
|
||||
from a calibration recording. RUNS ON DANKSERVER (venv: numpy).
|
||||
|
||||
The protection rule (docs/MOE-W4.md "Protected experts", restated in the recipe):
|
||||
The law (docs/MOE-W4.md "Protected experts", restated in the recipe):
|
||||
an expert whose share of its layer's down-input energy is at least 3% AND
|
||||
at least three times its share of the routing is kept bf16; the count must
|
||||
be EVEN on each rank's half of the layer (the four-bit GEMM takes the rest
|
||||
|
||||
Executable
+83
@@ -0,0 +1,83 @@
|
||||
#!/usr/bin/env bash
|
||||
# download-frank-archon-iq4r.sh — payload fetch/verify for the frank-archon
|
||||
# swap to the PUBLISHER's 4-bit-codebook container:
|
||||
# StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe — the official
|
||||
# Qwen/Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's own
|
||||
# author into ONE .rad: routed experts as 4-bit codes into a 16-level
|
||||
# non-uniform codebook (Walsh–Hadamard rotation, GPTQ + coordinate-descent,
|
||||
# 10M-token calibration), ten bf16-protected experts per layer, int8 trunk +
|
||||
# lm_head, fp8 HC mixtures, the PLE n-gram table at fp8 fixed scale, the MTP
|
||||
# head, and the bf16 vision tower. File:
|
||||
# qwen3.8-next-flash-fp8-iq4r-moe.rad — 121,969,901,568 B (113.6 GiB,
|
||||
# 116,322 MiB), card KL 0.0789 all-tokens / top-1 92.85 % vs bf16.
|
||||
# Repo is UNGATED — no token needed (never put one on the command line).
|
||||
#
|
||||
# bash scripts/download-frank-archon-iq4r.sh --fetch resume/complete
|
||||
# bash scripts/download-frank-archon-iq4r.sh verify only
|
||||
#
|
||||
# Fetch uses the vendored scripts/hfget.sh (curl+jq only): per-file size
|
||||
# checks against the Hub listing, `-C -` resume, disk-fit gate. The side
|
||||
# files (LICENSE, README, the w4nl recipe) stay in the staging dir as
|
||||
# provenance; the serve gate mounts ONLY /home/dank/models/rad, so the
|
||||
# verified .rad alone moves into it.
|
||||
#
|
||||
# Verify (default) is the AGENTS.md law in two parts: the file lands at the
|
||||
# final path with the Hub's EXACT byte count, and its sha256 equals the
|
||||
# Hub's LFS oid. A truncated transfer is a guaranteed long-timeout boot
|
||||
# without this gate.
|
||||
set -euo pipefail
|
||||
|
||||
REPO="StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe"
|
||||
RAD_NAME="qwen3.8-next-flash-fp8-iq4r-moe.rad"
|
||||
STAGE="/home/dank/models/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe"
|
||||
DEST="/home/dank/models/rad/$RAD_NAME"
|
||||
SIZE=121969901568
|
||||
SHA256="0af5e96244e80c21ac8edca1719dae49b8a201b94a2f3994c3e24026ceaa4d20"
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
|
||||
mode="verify"
|
||||
[[ "${1:-}" == "--fetch" ]] && mode="fetch"
|
||||
|
||||
if [[ "$mode" == "fetch" ]]; then
|
||||
# HFGET_JOBS=4 is hfget's default (a few streams saturate the link).
|
||||
bash "$SCRIPT_DIR/hfget.sh" "$REPO" "$STAGE" main
|
||||
|
||||
staged="$STAGE/$RAD_NAME"
|
||||
if [[ ! -f "$staged" ]]; then
|
||||
echo "FETCH FAILED: $staged absent after hfget" >&2
|
||||
exit 1
|
||||
fi
|
||||
got=$(stat -c %s "$staged")
|
||||
if [[ "$got" != "$SIZE" ]]; then
|
||||
echo "FETCH FAILED: staged .rad is $got B, want $SIZE B" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "staged .rad byte-exact ($SIZE B); hashing (one full read, minutes)"
|
||||
got_sha=$(sha256sum "$staged" | cut -d' ' -f1)
|
||||
if [[ "$got_sha" != "$SHA256" ]]; then
|
||||
echo "FETCH FAILED: sha256 $got_sha, want $SHA256 — DELETE the staged" \
|
||||
"file before re-fetching (curl -C - cannot repair a corrupt body)" >&2
|
||||
exit 1
|
||||
fi
|
||||
mkdir -p "$(dirname "$DEST")"
|
||||
mv "$staged" "$DEST"
|
||||
echo "payload placed: $DEST"
|
||||
fi
|
||||
|
||||
echo "verifying deployed payload: $DEST"
|
||||
if [[ ! -f "$DEST" ]]; then
|
||||
echo " BAD payload absent: $DEST" >&2
|
||||
exit 1
|
||||
fi
|
||||
got=$(stat -c %s "$DEST")
|
||||
if [[ "$got" != "$SIZE" ]]; then
|
||||
echo " BAD payload is $got B, want $SIZE B" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "byte-exact ($SIZE B); hashing (one full read, minutes)"
|
||||
got_sha=$(sha256sum "$DEST" | cut -d' ' -f1)
|
||||
if [[ "$got_sha" != "$SHA256" ]]; then
|
||||
echo " BAD sha256 $got_sha, want $SHA256" >&2
|
||||
exit 1
|
||||
fi
|
||||
echo "payload verified: $RAD_NAME, size and sha256 match the Hub LFS oid"
|
||||
@@ -14,7 +14,7 @@
|
||||
# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
|
||||
# listing/resume downloader, curl+jq only): per-file size checks against
|
||||
# the Hub listing, `-C -` resume, pickle-format duplicates skipped.
|
||||
# Verify (default) enforces the download contract in three parts: every listed file
|
||||
# Verify (default) is the AGENTS.md law in three parts: every listed file
|
||||
# present with the listed size, the shard count matches the weight-map
|
||||
# index, and every shard's sha256 equals the Hub's LFS oid. Verify needs
|
||||
# only the public API.
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
# Fetch uses the vendored scripts/hfget.sh (StillDeadcode's Apache-2.0
|
||||
# listing/resume downloader, curl+jq only): per-file size checks against
|
||||
# the Hub listing, `-C -` resume, pickle-format duplicates skipped.
|
||||
# Verify (default) enforces the download contract in three parts: every listed file
|
||||
# Verify (default) is the AGENTS.md law in three parts: every listed file
|
||||
# present with the listed size, the shard count matches the weight-map
|
||||
# index, and every shard's sha256 equals the Hub's LFS oid. Verify needs
|
||||
# only the public API.
|
||||
|
||||
@@ -4,10 +4,9 @@
|
||||
# .hgn, the REQUIRED lookup table and the optional vision tower from
|
||||
# peonist-ai/halogen-qwen3.8-flash-next (both at the revisions the model
|
||||
# card pins), and the flat tokenizer the two repos share. All public, no
|
||||
# token — and a token never rides a command line: argv is readable by every
|
||||
# process on the host through /proc, so a key placed there is a key exposed.
|
||||
# token — and per the fleet secret law, none would ever touch argv.
|
||||
#
|
||||
# The fetch discipline, same as every sibling script: curl -f (an access error
|
||||
# The fetch law, same as every sibling script: curl -f (an access error
|
||||
# must not become a "successful" 153-byte artifact), exact byte count from
|
||||
# the Hub tree API, LFS sha256 verification, --verify-only to re-check an
|
||||
# existing payload without downloading.
|
||||
@@ -27,8 +26,7 @@
|
||||
# tokenizer/* @e053f488 — byte-identical to the 27B-era tokenizer
|
||||
# already on disk (tokenizer.json LFS
|
||||
# 0997f410c57a1f4e53b09e4be8f4a172d90edd9564368fb0847030937229b9f3);
|
||||
# fetched fresh rather than symlinked, so the payload dir is
|
||||
# self-contained — every gate and every mount sees real files inside it.
|
||||
# fetched fresh rather than symlinked, per the no-symlink law.
|
||||
set -euo pipefail
|
||||
|
||||
DEST="${FRANK_DRAGOON_MODEL_DIR:-$HOME/models/halogen-flashnext-v2}"
|
||||
|
||||
@@ -1,11 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
# download-frank-dragoon-halogen.sh — fetch frank-dragoon's halogen payload:
|
||||
# the qwen3.8-27b .hgn checkpoint and the export tokenizer, from the
|
||||
# PUBLIC peonist-ai/halogen-qwen3.8-27b repo (no token — and a token never
|
||||
# rides a command line: argv is readable by every process on the host
|
||||
# through /proc, so a key placed there is a key exposed).
|
||||
#
|
||||
# The fetch discipline, same as every sibling script: curl -f (an access error
|
||||
# PUBLIC peonist-ai/halogen-qwen3.8-27b repo (no token — and per the fleet
|
||||
# secret law, none would ever touch argv).
|
||||
#
|
||||
# The fetch law, same as every sibling script: curl -f (an access error
|
||||
# must not become a "successful" 153-byte artifact), exact byte count from
|
||||
# the model card, LFS sha256 verification, --verify-only to re-check an
|
||||
# existing payload without downloading.
|
||||
|
||||
@@ -11,8 +11,8 @@ Tensor extents are DERIVED from consecutive GGUF header offsets — no quant
|
||||
type-size table (those are easy to get subtly wrong; a wrong per-element size
|
||||
silently under-covers each tensor and dumps tail samples into a junk bucket).
|
||||
|
||||
What this instrument shows (75 s window, 4 mixed streams, 2.7M
|
||||
expert-tensor samples): per-BLOCK read volume is FLAT
|
||||
Fleet law this instrument established (2026-09-26, 75 s window, 4 mixed
|
||||
streams, 2.7M expert-tensor samples): per-BLOCK read volume is FLAT
|
||||
(0.93x-1.08x of mean across all 40 CPU-resident blocks) because MoE top-k
|
||||
routing fires every block every token — there are no "most active layers" at
|
||||
`-ot` granularity. The real skew is per-EXPERT (top 10% of (block,expert)
|
||||
|
||||
+64
-44
@@ -1,7 +1,15 @@
|
||||
# frank-archon.env — radiance (StillDeadcode, Apache-2.0) serving
|
||||
# ukisai/Swift1.5-Qwen3.8-Flash-Next converted to GPTQ w4 (MODEL_FILE below,
|
||||
# built by scripts/convert-swift15-w4dn8.sh) on BOTH R9700s (TP=2),
|
||||
# dankserver :7770.
|
||||
# frank-archon.env — radiance (StillDeadcode, Apache-2.0) serving the
|
||||
# PUBLISHER's 4-bit-codebook container of the official Qwen/Qwen3.8-Flash-Next
|
||||
# (MODEL_FILE below, fetched by scripts/download-frank-archon-iq4r.sh) on
|
||||
# BOTH R9700s (TP=2), dankserver :7770.
|
||||
# 2026-10-15 PAYLOAD SWAP: the model moved from the in-house w4dn8 conversion
|
||||
# of the Swift1.5 finetune to the engine author's published container
|
||||
# qwen3.8-next-flash-fp8-iq4r-moe.rad (116,322 MiB, sha256-verified at fetch —
|
||||
# the w4dn8 measured 116,416 MiB, so this shape carries over with 94 MiB to
|
||||
# spare). NO argument changed: every number below is the 2026-10-13 shape,
|
||||
# kept verbatim. The swap commit is the rollback (the w4dn8 payload stays on
|
||||
# disk).
|
||||
|
||||
#
|
||||
# SHAPE (2026-10-13): THREE decode slots x 200,000 tokens, fp8 (quantized) KV,
|
||||
# with the per-card budget spent on resident experts: 25,600 MiB of slab +
|
||||
@@ -13,9 +21,11 @@
|
||||
# +25 % at the full window) while KV beyond the contract buys nothing.
|
||||
# Consequence, deliberate: a FOURTH full-height conversation does not fit this
|
||||
# pair, and the engine refuses it — it never truncates. See the KV block for the
|
||||
# arithmetic and the measured per-dtype cost. The fp8-native payload this file was
|
||||
# written for is the rollback target and keeps its own trio under the
|
||||
# fp8-native runbook.
|
||||
# arithmetic and the measured dtype law. The w4dn8 payload is the swap
|
||||
# rollback and keeps its trio under the w4dn8 runbook; the fp8-native payload
|
||||
# that came before it was DELETED from disk 2026-10-15 (Frank's call, to free
|
||||
# the volume for this download — rebuildable via the fetch + build scripts
|
||||
# named in the fp8-native runbook).
|
||||
#
|
||||
# 2026-10-05 cutover from FreeToken (PR #587 ROCm, one card, experts in
|
||||
# pinned host RAM; git tag frank-archon-freetoken-pr587-20261005 is the
|
||||
@@ -30,20 +40,30 @@
|
||||
# (`X.weight_scale_inv` is the scale plane — checkpoint.h). The PLE n-gram
|
||||
# table (51B params, 47.7 GiB) is served from pinned host RAM via io_uring
|
||||
# reads — Frank's call: "Native FP8. Experts from ram. Pin the PLE to ram."
|
||||
# The 4-bit-codebook publisher shape (qwen3.8-next-flash-fp8-iq4r-moe.rad,
|
||||
# --host-pool-mib 12288) remains the documented fallback if the native
|
||||
# container proves unbootable.
|
||||
# THE 4-bit-codebook publisher shape is the DEPLOYED payload since
|
||||
# 2026-10-15. The card's own run line (--host-pool-mib 12288,
|
||||
# --expert-vs-cache-ratio 0.82, seqs 8, spec 3) is the publisher's
|
||||
# suggestion on 32 GB cards; this file's shape is the fleet's measured one.
|
||||
#
|
||||
# PAYLOAD: /home/dank/models/rad/qwen4exp-fp8-native.rad — a rad-convert
|
||||
# NO-RECIPE pure copy of the official 131-shard checkpoint (every weight
|
||||
# keeps its block-FP8 encoding; ~161 GiB). The convert runs against a
|
||||
# staging dir whose config.json drops `quantization_config`: the descriptor
|
||||
# string is only the plugin SELECTION key (spec §2.4 — exact match, no
|
||||
# fallback), and no shipped plugin claims (qwen4exp, "fp8"); the shipped
|
||||
# qwen4exp plugin claims (qwen4exp, "") which is what recipe-built
|
||||
# containers carry. Per-weight encodings are data, not the descriptor —
|
||||
# removing the key changes nothing about what the weights hold. Built by
|
||||
# scripts/build-frank-archon-rad.sh (staging + convert + rad-info).
|
||||
# PAYLOAD: /home/dank/models/rad/qwen3.8-next-flash-fp8-iq4r-moe.rad — the
|
||||
# official Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's
|
||||
# own author (recipe qwen4exp-w4nl64-i8-hc8m: experts as 4-bit codes into a
|
||||
# 16-level non-uniform codebook after a 128-point Walsh–Hadamard rotation,
|
||||
# GPTQ + 3 coordinate-descent sweeps on a 10M-token calibration, ten
|
||||
# bf16-protected experts per layer, int8 W8A8 trunk + lm_head, fp8 HC
|
||||
# mixtures, PLE n-gram fp8 fixed-scale, MTP head, bf16 vision tower; card KL
|
||||
# 0.0789 all-tokens / top-1 92.85 % vs bf16). Fetched byte-exact + sha256 by
|
||||
# scripts/download-frank-archon-iq4r.sh; the recipe/LICENSE stay in the
|
||||
# staging dir beside it as provenance. ROLLBACK: revert the swap commit — the
|
||||
# in-house qwen4exp-swift15-w4dn8-i8-hc8m.rad (scripts/convert-swift15-w4dn8.sh,
|
||||
# KLD-gated) stays on disk. Behind it, the fp8-native
|
||||
# qwen4exp-fp8-native.rad was DELETED from disk 2026-10-15 (Frank's call,
|
||||
# one-time rm, to free the volume for this payload); rebuild =
|
||||
# scripts/download-frank-archon.sh + scripts/build-frank-archon-rad.sh — a
|
||||
# NO-RECIPE rad-convert copy of the official 131-shard checkpoint under the
|
||||
# descriptor law (no shipped plugin claims (qwen4exp, "fp8"), so the staging
|
||||
# config.json drops quantization_config; per-weight encodings are data, not
|
||||
# the descriptor).
|
||||
#
|
||||
# SERVING FACTS (docs/GUIDE.md, docs/DOCKER.md, compose/flashnext.yaml):
|
||||
# - The HTTP listener binds AFTER the full load (main.cpp: engine.start()
|
||||
@@ -59,12 +79,12 @@
|
||||
# enable_thinking, preserve_thinking (core/server/oai.cpp:194-205) —
|
||||
# models.yml's extraBody set is fully inside it. Unknown params are
|
||||
# refused (400), which is why nothing unverified rides the wire.
|
||||
# - MTP: the official checkpoint carries the mtp.* head (3,101 tensors,
|
||||
# mtp_num_hidden_layers=1) and the plugin probes draft depth 3 — the
|
||||
# publisher's throughput peak for this model. Stated explicitly with
|
||||
# --num-speculative-tokens 3.
|
||||
# - MTP: the .rad carries the model's mtp.* head (the publisher's recipe
|
||||
# names the mtp.fc_hidden/fc_embedding/draft_head rules; card default
|
||||
# depth 3). Served at the depth named by FRANK_ARCHON_SPEC_TOKENS below —
|
||||
# the depth this shape's decode ladder was measured at.
|
||||
# - The fused TP all-reduce is declared over max-num-seqs x
|
||||
# (1 + --num-speculative-tokens) rows; 2 x 3 = 6 rows, under the 25-row
|
||||
# (1 + --num-speculative-tokens) rows; 3 x (1+2) = 9 rows, under the 25-row
|
||||
# exactness bound of the wht6 wire.
|
||||
# Engine pin: Codeberg tag v1.0.13 (commit 53282fa), Docker Hub tag 1.0.13,
|
||||
# index digest below. REJECTED UPGRADE: 1.2.0 (sha256:665775ed1d90de6dbb
|
||||
@@ -88,7 +108,7 @@ FRANK_ARCHON_IMAGE=stilldeadcode/radiance@sha256:4c3f484564bb2872784e46b0a59cbd7
|
||||
# .rad files; the raw HF download and its staging dir live beside it and
|
||||
# are never mounted.
|
||||
FRANK_ARCHON_MODEL_DIR=/home/dank/models/rad
|
||||
FRANK_ARCHON_MODEL_FILE=qwen4exp-swift15-w4dn8-i8-hc8m.rad
|
||||
FRANK_ARCHON_MODEL_FILE=qwen3.8-next-flash-fp8-iq4r-moe.rad
|
||||
FRANK_ARCHON_STATE_DIR=/home/dank/radiance-state
|
||||
# The state dir's tune/ subdirectory holds rad-tune's kernel-selection
|
||||
# cache (the engine's selector reads $RADIANCE_HOME/tune/<machine>.tune
|
||||
@@ -124,12 +144,13 @@ FRANK_ARCHON_TP=2
|
||||
FRANK_ARCHON_TP_WIRE=wht6
|
||||
# Decode slots: THREE, so the served window is a CAPACITY CONTRACT — three
|
||||
# concurrent conversations of --max-model-len fit the KV pool (see the KV
|
||||
# block below). A runtime that compiles its kernels specialized to the exact
|
||||
# batch shape can refuse a slot count it has never compiled for; a
|
||||
# shape-agnostic engine accepts any count that fits memory. Radiance is the
|
||||
# latter: seqs 2 boots, and measured decode at seqs 2 and 3 is statistically
|
||||
# tied (30.9-33.1 t/s, inside the +-3% noise floor) — slot count buys
|
||||
# concurrency, not speed, so a slot costs KV, never decode.
|
||||
# block below). The "seqs 2 cannot boot" law is NOT radiance's: it was the
|
||||
# tcclaviger vLLM fork's torch.compile shape specialization on
|
||||
# query_start_loc (ConstraintViolationError, measured 2026-10-02, retired
|
||||
# with that runtime). Radiance booted seqs 2 at the fp8-native first boot,
|
||||
# and the 2026-10-06 decode sweep measured seqs 2 and 3 statistically tied
|
||||
# (30.9-33.1 t/s, inside the +-3% noise floor) — slot count buys concurrency,
|
||||
# not speed, so a slot costs KV, never decode.
|
||||
# Cost of each extra slot: 110 MiB of per-slot recurrent state inside the KV
|
||||
# pool, and the fused TP all-reduce is declared over seqs x (1 + spec) rows —
|
||||
# 3 x 3 = 9 here, under the 25-row bound at which the wht6 wire stays EXACT
|
||||
@@ -241,9 +262,10 @@ FRANK_ARCHON_VRAM_KV_MIB=5120
|
||||
# ledgers' `kv tier transfer` lines), so the host tier below parks twice as
|
||||
# many conversations as it did at bf16 — no re-size needed to park three slots.
|
||||
FRANK_ARCHON_KV_DTYPE=fp8
|
||||
# Spec decode: the checkpoint's own MTP head at the publisher's depth-3
|
||||
# operating point (probe() returns 3 when mtp_num_hidden_layers > 0; the
|
||||
# flag states it so the ledger records the intent).
|
||||
# Spec decode: the checkpoint's own MTP head, served at depth 2 — the
|
||||
# publisher's card ships depth 3 as its peak; this fleet's ladder was
|
||||
# measured at 2 and the swap keeps every argument, so 2 stays (rows
|
||||
# 3 x (1+2) = 9, inside the wht6 25-row exactness bound).
|
||||
FRANK_ARCHON_SPEC_TOKENS=2
|
||||
FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
|
||||
# Prefix cache tiers (publisher pattern). The host tier is sized in ENTRIES,
|
||||
@@ -254,10 +276,9 @@ FRANK_ARCHON_MAX_BATCHED_TOKENS=2048
|
||||
# available with the service loaded).
|
||||
FRANK_ARCHON_PREFIX_HOST_MIB=8192
|
||||
FRANK_ARCHON_PREFIX_DISK_MIB=65536
|
||||
# The PLE n-gram table (47.7 GiB) serves from pinned host RAM the cards
|
||||
# read directly, so it cannot be evicted or page-faulted mid-token. `ram`
|
||||
# placement REFUSES the start if the memlock limit is too small, and the
|
||||
# unit's LimitMEMLOCK=infinity is the matching half of that pairing.
|
||||
# Frank's pin law: the PLE n-gram table (47.7 GiB) serves from pinned host
|
||||
# RAM the cards read. `ram` REFUSES the start if the memlock limit is too
|
||||
# small — the unit's LimitMEMLOCK=infinity is the other half of this law.
|
||||
FRANK_ARCHON_NGRAM_PLACEMENT=ram
|
||||
# TTM preflight: the pinned ledger (2 x 44 GiB pools + 47.7 GiB PLE + 8 GiB
|
||||
# prefix host = ~147 GiB) rides TTM's shared pages, capped by ttm's
|
||||
@@ -278,9 +299,8 @@ FRANK_ARCHON_TTM_TARGET_MIB=204800
|
||||
# multi-flag values as one string; the serve script expands it unquoted on
|
||||
# purpose.
|
||||
FRANK_ARCHON_EXTRA_ARGS=
|
||||
# The .rad's minimum size in MiB (deployer gate; a truncated convert is a
|
||||
# guaranteed long-timeout boot instead of a fast failure). The converted
|
||||
# container measured 177,109 MiB (1,857,127,997,440 B = 173.0 GiB; the
|
||||
# plan-only estimate 172.76 GiB covers the data planes but not the metadata
|
||||
# tail) - floor sits just under the payload.
|
||||
# The .rad's minimum size in MiB (deployer gate; a truncated fetch is a
|
||||
# guaranteed long-timeout boot instead of a fast failure). The deployed
|
||||
# publisher container measures 116,322 MiB (121,969,901,568 B); the
|
||||
# in-house w4dn8 rollback measured 116,416 MiB. Floor sits just under both.
|
||||
FRANK_ARCHON_RAD_MIN_MIB=110000
|
||||
@@ -1,9 +1,9 @@
|
||||
# frank-archon.service — radiance (StillDeadcode, Apache-2.0) endpoint
|
||||
# (OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 at NATIVE block-FP8 precision, the
|
||||
# rad-convert container qwen4exp-fp8-native.rad) on BOTH R9700s (TP=2,
|
||||
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (git tag
|
||||
# frank-archon-freetoken-pr587-20261005 is the pre-cutover state; the
|
||||
# retired unit/env/serve trio and Dockerfile are in that tag).
|
||||
# (the publisher's 4-bit-codebook container of the official
|
||||
# Qwen/Qwen3.8-Flash-Next, qwen3.8-next-flash-fp8-iq4r-moe.rad, fetched by
|
||||
# scripts/download-frank-archon-iq4r.sh) on BOTH R9700s (TP=2, wht6 wire),
|
||||
# dankserver :7770. 2026-10-15 payload swap from the in-house w4dn8 (the
|
||||
# swap commit is the revert; its .rad stays on disk).
|
||||
#
|
||||
# Same port, same served alias, same container identity as the unit name;
|
||||
# the harness models.yml id is unchanged. The serve script drives
|
||||
@@ -19,7 +19,7 @@
|
||||
# unit grants LimitMEMLOCK=infinity to the dank process via PID1; the
|
||||
# engine still runs out of dank's rootless podman storage.
|
||||
[Unit]
|
||||
Description=frank-archon — radiance Qwen3.8-Flash-Next-FP8 (native) on 2x R9700 TP=2
|
||||
Description=frank-archon — radiance Qwen3.8-Flash-Next (publisher iq4r) on 2x R9700 TP=2
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
# Never let the start-rate limiter give up after a crash loop (a failed LLM
|
||||
|
||||
@@ -42,10 +42,10 @@
|
||||
# form of the engine's default, so the seam test can pin the product.
|
||||
# KV costs ~28 KiB/position (FLAGS.md) => 524,288 x 28 KiB = 14.0 GiB.
|
||||
# HALOGEN_KV_SLOTS=2 restores the w4b-era shape on this box: the MTP
|
||||
# drafter is NOT slot-locked on this engine: some engines silently
|
||||
# disable drafting the moment a second slot is enabled, this one does not
|
||||
# (the card's own quickstart ships 2 slots WITH the head), so the second
|
||||
# slot is concurrent capacity, not a speculation tax.
|
||||
# drafter is NOT slot-locked on this engine (batching and speculation
|
||||
# coexist — DFlash-era slots>1 law does NOT carry over; the card's own
|
||||
# quickstart ships 2 slots WITH the head), so the second slot is
|
||||
# concurrent capacity, not a speculation tax.
|
||||
# Memory ledger against the box's 121.2 GiB unified (the n-gram table is
|
||||
# mmap'd page cache beside the checkpoint, not a pinned charge — the
|
||||
# card's own quickstart arithmetic does not bill it):
|
||||
@@ -88,9 +88,8 @@
|
||||
# before. Kernel needs CONFIG_HSA_AMD_SVM (CachyOS ships it) and
|
||||
# amdgpu.noretry=0 (check at deploy).
|
||||
|
||||
# Image: digest-pinned (a tag is a mutable pointer — the same tag can
|
||||
# silently deliver different bits — so the digest pins exactly the bits
|
||||
# that were qualified). 0.16.2 = the version the model card names as its
|
||||
# Image: digest-pinned (fetch law: tag pulls silently re-point; the digest
|
||||
# is what was qualified). 0.16.2 = the version the model card names as its
|
||||
# quickstart pin; RepoDigest captured at pull on 2026-10-06.
|
||||
FRANK_DRAGOON_IMAGE=ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
|
||||
# Container identity: kept from the 27B era (unit ExecStop names it).
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
# custom-quant lock-in like the ROCmFPX era). NO nextn/mtp tensors exist, so
|
||||
# in-file MTP is impossible here; SPEC_TYPE=none is forced by the payload.
|
||||
#
|
||||
# KV rate arithmetic: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per
|
||||
# KV rate law: 8 full layers × 4 KV heads × 256 dim = 8,192 elements per
|
||||
# token per cache. f16 K + f16 V = 32,768 B/token = 32 KiB/token, so the
|
||||
# 401,408-token pool is 12.25 GiB. The 24 DeltaNet layers cost a fixed
|
||||
# recurrent state per SLOT (not per token) — that is why two 200k slots fit
|
||||
|
||||
@@ -29,8 +29,7 @@
|
||||
# its base SupportsEagleBase (has_own_lm_head, has_own_embed_tokens) -- and
|
||||
# getattr_static()s each one off the instance, breaking on the first miss.
|
||||
# So BOTH wrappers need all three attributes.
|
||||
# * Method-resolution order forbids the obvious inheritance: adding
|
||||
# SupportsEagle3 as a base dies at class creation with
|
||||
# * MRO law: INHERITING SupportsEagle3 as a base dies at class creation with
|
||||
# "Cannot create a consistent method resolution order (MRO) for bases ..."
|
||||
# (GLM's wrappers already carry four/five Protocol-derived bases). The
|
||||
# attribute route avoids the MRO entirely. Also proven by simulation:
|
||||
|
||||
@@ -9,8 +9,7 @@ No scratch container, no mask: the service under test is the service.
|
||||
Four probes, in increasing cost:
|
||||
A decode at depth — one ~51k-token chat completion, 512 tokens, greedy.
|
||||
Reports the engine's own `timings` (prefill, decode, draft acceptance).
|
||||
Measurement discipline: generate at least 128 tokens (a short run
|
||||
measures first-token noise, not the server), greedy, real text.
|
||||
The bench law applies: >=128 generated tokens, greedy, real text.
|
||||
B the capacity contract — as many CONCURRENT chats of ~197k prompt tokens as
|
||||
the deployed `FRANK_ARCHON_MAX_SEQS` (read from the repo env file on this
|
||||
host), each carrying a unique marker at its head. All must answer their own
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# qwen4exp-swift15-w4dn8-i8-hc8m.recipe -- the Swift 1.5 finetune of Qwen3.8-Flash-Next
|
||||
# (ukisai/Swift1.5-Qwen3.8-Flash-Next, BF16 source, 335.3 GiB, key set identical to the official
|
||||
# base) as a radiance container for frank-archon: EVERY routed expert -- gate_up AND down --
|
||||
# through GPTQ into the w4nl64a8h table. The format assigns exactly ONE encoding per table
|
||||
# through GPTQ into the w4nl64a8h table. The plugin's one-table-one-encoding law
|
||||
# ("one table of experts takes one encoding, and only a whole expert kept plain bf16 is served
|
||||
# beside it", qwen4exp_fp8 rad_arch_declare — so the original split tier (down block-fp8 +
|
||||
# beside it", qwen4exp_fp8 rad_arch_declare) forbids the original split tier (down block-fp8 +
|
||||
# gate_up w4): the served kernel is w4nl64a8h for the whole expert set, bf16 experts beside it.
|
||||
# The rest is qwen4exp-w5dn64-i8-hc8m.recipe verbatim: int8 trunk linears, block-fp8 hc mixers
|
||||
# and MTP fc, the fixed-scale n-gram table, the 2-bit draft head.
|
||||
@@ -26,8 +26,8 @@
|
||||
# The protected experts, cast to bf16 ahead of both expert rules. THE NAMES BELOW ARE THE BASE
|
||||
# CHECKPOINT'S (from its 1M-token recording, qwen4exp-w5dn64-i8-hc8m.recipe) -- a warm start only.
|
||||
# The Swift 1.5 finetune's routing distribution is not assumed equal: the derivation step rewrites
|
||||
# this section from Swift's OWN calibration recording before this recipe is ever run, by the
|
||||
# protection rule -- an expert whose share of its layer's down-input energy is at least 3% and at least three
|
||||
# this section from Swift's OWN calibration recording before this recipe is ever run, by the base's
|
||||
# law -- an expert whose share of its layer's down-input energy is at least 3% and at least three
|
||||
# times its share of the routing, an even count on each rank's half of the layer (the four-bit GEMM
|
||||
# takes the rest as two equal tables by parity) by adding that half's next most energetic expert.
|
||||
# PROTECTED-BEGIN (rewritten from Swift's OWN grams by derive-protected-experts.py)
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
#!/usr/bin/env bash
|
||||
# serve-frank-archon.sh — frank-archon endpoint: radiance (StillDeadcode,
|
||||
# Apache-2.0) serving the OFFICIAL Qwen/Qwen3.8-Flash-Next-FP8 checkpoint at
|
||||
# NATIVE block-FP8 precision from a rad-convert container
|
||||
# (/home/dank/models/rad/qwen4exp-fp8-native.rad), on BOTH R9700s (TP=2,
|
||||
# wht6 wire), dankserver :7770. 2026-10-05 cutover from FreeToken (PR #587,
|
||||
# one card; git tag frank-archon-freetoken-pr587-20261005 is the
|
||||
# pre-cutover state).
|
||||
# Apache-2.0) serving the PUBLISHER's 4-bit-codebook container of the
|
||||
# official Qwen/Qwen3.8-Flash-Next (the engine author's own rad-convert of
|
||||
# the bf16 checkpoint: qwen3.8-next-flash-fp8-iq4r-moe.rad, fetched by
|
||||
# scripts/download-frank-archon-iq4r.sh) on BOTH R9700s (TP=2, wht6 wire),
|
||||
# dankserver :7770. 2026-10-15 payload swap from the in-house w4dn8 of the
|
||||
# Swift1.5 finetune (payload on disk = rollback; the swap commit is the
|
||||
# revert); 2026-10-05 radiance cutover from FreeToken (git tag
|
||||
# frank-archon-freetoken-pr587-20261005 is the pre-radiance state).
|
||||
#
|
||||
# The run line below IS the engine argv: the image's entrypoint is the
|
||||
# engine and its tools sit beside it on PATH. Unlike every predecessor
|
||||
|
||||
@@ -70,9 +70,7 @@ fi
|
||||
|
||||
# Size-exact gates on all three payload files: a truncated curl exits 0 on
|
||||
# some failures and a partial .hgn loads far enough to look alive, so the
|
||||
# byte counts from the Hub tree API are the contract: curl exits 0 on an
|
||||
# HTTP error and writes the error body as the file, so a truncated download
|
||||
# is caught by size here and never boots.
|
||||
# byte counts from the Hub tree API are the contract (fleet fetch law).
|
||||
ck() {
|
||||
# ck <env-name> <path> <expected-bytes>
|
||||
if [ ! -f "$2" ]; then
|
||||
|
||||
@@ -63,9 +63,8 @@ fi
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
# Speculative decoding is OFF: this Q6_K file carries NO nextn/mtp tensors, so
|
||||
# there is no in-file head — this kind of drafter exists only as tensors
|
||||
# inside the checkpoint file, and no flag can supply one the file lacks.
|
||||
# SPEC_TYPE=none strips BOTH speculative flags (they move as a set);
|
||||
# there is no in-file head (the law: MTP is a property of the file, not of the
|
||||
# flags). SPEC_TYPE=none strips BOTH speculative flags (flags move as a set);
|
||||
# naming a drafter requires SPEC_DRAFT_FILE alongside SPEC_TYPE — the Gemma4
|
||||
# assistant on disk is arch gemma4-assistant and cannot drafter-match qwen35.
|
||||
: "${SPEC_TYPE:=none}"
|
||||
|
||||
@@ -298,7 +298,7 @@ def main():
|
||||
and list(dssh) == [(dsh[0] + 127) // 128, (dsh[1] + 127) // 128])
|
||||
if not scale_law:
|
||||
log(f"ERROR: unexpected expert shapes: gate {gdt}{gsh} scale {gsd}{gssh} "
|
||||
f"down {ddt}{dsh} scale {dsd}{dssh} (128x128 block scales)")
|
||||
f"down {ddt}{dsh} scale {dsd}{dssh} (scale law 128x128)")
|
||||
return 2
|
||||
shapes = {"ffn2_h": (gsh[0] * 2, gsh[1]), "ffn": (dsh[1],),
|
||||
"gs": tuple(gssh), "ds": tuple(dssh)}
|
||||
|
||||
@@ -452,7 +452,8 @@ access is decided entirely by GROUP MEMBERSHIP consumed downstream —
|
||||
- `Non-Admin Users` = the plain-user marker group (precedents: qa-e2e, and
|
||||
every human account).
|
||||
- **The human-user template** (ridley, kassie, alex, artem, kirby, ry, nic,
|
||||
debo, freya — kirby/ry/nic/debo/freya (pk 17–21) created 2026-10-09, kassie
|
||||
debo, freya, mitch, eric — mitch (pk 22) and eric (pk 23) created
|
||||
2026-10-15, kirby/ry/nic/debo/freya (pk 17–21) created 2026-10-09, kassie
|
||||
and alex created 2026-10-08, artem (pk 16) created 2026-10-07): `username` +
|
||||
`<user>@franksdank.cloud` + display name +
|
||||
`is_active=True` + the password set via `set_password`, membership in
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
# frank-archon — Radiance (StillDeadcode), publisher Qwen3.8-Flash-Next iq4r, 2× R9700 TP=2
|
||||
|
||||
**Status: LIVE — deployed 2026-10-15.** · **Host:** Dankserver (192.168.0.235) · **Port:** 7770 · **GPU:** 2× Radeon AI PRO R9700 (gfx1201), TP=2
|
||||
|
||||
The in-house w4dn8 conversion of the Swift1.5 finetune
|
||||
([[frank-archon-radiance-swift15-w4dn8]]) is replaced by the **engine
|
||||
author's published container**:
|
||||
[`StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe`](https://huggingface.co/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe)
|
||||
— the official `Qwen/Qwen3.8-Flash-Next` bf16 checkpoint rad-converted by
|
||||
StillDeadcode. Same port, alias, container name, systemd unit, digest-pinned
|
||||
engine pin **1.0.13** (`stilldeadcode/radiance@sha256:4c3f484564…`), and
|
||||
**every serving argument carried verbatim** from the 2026-10-13 shape.
|
||||
Rollback: revert the swap commit — the w4dn8 `.rad` stays on disk.
|
||||
|
||||
## The payload
|
||||
|
||||
`qwen3.8-next-flash-fp8-iq4r-moe.rad` — **121,969,901,568 B = 116,322 MiB**,
|
||||
sha256 `0af5e96244e80c21ac8edca1719dae49b8a201b94a2f3994c3e24026ceaa4d20`
|
||||
(Hub LFS oid), fetched byte-exact by `scripts/download-frank-archon-iq4r.sh`
|
||||
(hfget resume + size + sha256; staging dir
|
||||
`/home/dank/models/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe/` keeps the
|
||||
card, LICENSE, and recipe `qwen4exp-w4nl64-i8-hc8m` as provenance).
|
||||
|
||||
| Plane | Encoding (publisher recipe `qwen4exp-w4nl64-i8-hc8m`) |
|
||||
| --- | --- |
|
||||
| Routed experts | 4-bit codes into a 16-level **non-uniform** codebook after a 128-point Walsh–Hadamard rotation, one E4M3 scale per 64 weights; codes by GPTQ (10M-token calibration, per-expert down-proj Hessians) + 3 coordinate-descent sweeps |
|
||||
| Protected experts | **ten** per layer, whole-expert bf16 (down-projection energy) |
|
||||
| Trunk + lm_head | int8 W8A8, scale per 128 columns searched; HC mixtures E4M3 |
|
||||
| PLE n-gram | fp8 fixed-scale, in-container (served `--ngram-placement ram`, same 47.7 GiB pin law) |
|
||||
| Speculator | the model's own MTP head (card default depth 3; served at 2) |
|
||||
| Vision | bf16 vision tower, in-container |
|
||||
|
||||
**Base model: the OFFICIAL (censored) Qwen3.8-Flash-Next** — the fleet's
|
||||
only frontier model at base alignment; the abliterated family lives on
|
||||
frank-dragoon. Card quality: full-vocab teacher-forced KL vs bf16 over
|
||||
~75K positions — mean 0.0789 all-tokens / 0.0239 assistant turns, top-1
|
||||
agreement 92.85 %.
|
||||
|
||||
## Why this swap
|
||||
|
||||
- **Publisher-graded precision**: the w4nl codebook + coordinate-descent
|
||||
refinement is the engine author's own quality ceiling for this
|
||||
checkpoint; the card's KL ladder (p99 1.65, p99.9 5.33) is published and
|
||||
reproducible, where our w4dn8 gate measured against our own reference.
|
||||
- **Formal parity**: the served file is the one the engine author supports
|
||||
— decode/prefill behavior on 1.0.13 is their tested path (the card's run
|
||||
line speaks the 1.0.13-era flags; the `--vram-weights-mib`/`--vram-kv-mib`
|
||||
pair we ship is accepted by pin 1.0.13 per the 2026-10-06 A/B, so the
|
||||
shape needed no translation).
|
||||
- **Cost**: the Swift1.5 finetune's refusal-bending behavior leaves this
|
||||
endpoint (it returns to the official base alignment); the 4-slot/200k
|
||||
economics and all shape laws carry over unchanged.
|
||||
|
||||
## Shape (unchanged from 2026-10-13)
|
||||
|
||||
All of it, verbatim — the payload moved 116,416 → 116,322 MiB, inside the
|
||||
same budget:
|
||||
|
||||
- `--tp 2 --tp-wire wht6`, `--max-num-seqs 3`, `--max-model-len 200000`
|
||||
(capacity contract 3 × 200,000), `--kv-cache-dtype fp8`
|
||||
- `--placement expert_tiered --host-pool-mib 44928` + explicit
|
||||
`--vram-weights-mib 25600 --vram-kv-mib 5120` (30,720 of the engine's
|
||||
30,969 MiB ceiling; pool = contract + 1.3 %)
|
||||
- `--num-speculative-tokens 2`, `--max-num-batched-tokens 2048`
|
||||
(MNBT 2048 law), prefix cache 8 GiB host + 64 GiB disk,
|
||||
`--ngram-placement ram`, froggeric v22.5 `--override-chat-template`
|
||||
- TTM preflight floor 150,000 / target 204,800 MiB; pinned ledger
|
||||
2 × 44 GiB pool + 47.7 GiB PLE + 8 GiB prefix ≈ 147 GiB (see
|
||||
[[frank-archon-radiance-swift15-w4dn8]] for the ledger arithmetic and
|
||||
the KV dtype law — both are engine properties, payload-independent)
|
||||
|
||||
The publisher's card run line (`--host-pool-mib 12288`,
|
||||
`--expert-vs-cache-ratio 0.82`, seqs 8, spec 3, ~32 GB host RAM) is their
|
||||
shape on stock 32 GB cards; the fleet ships its own measured shape against
|
||||
the same file.
|
||||
|
||||
## Cutover record — 2026-10-15
|
||||
|
||||
1. `rm` (Frank's one-time authorization): the raw
|
||||
`ukisai/Swift1.5-Qwen3.8-Flash-Next` BF16 checkpoint (336 GiB, spent
|
||||
conversion source) and `qwen4exp-fp8-native.rad` (173 GiB, two eras
|
||||
back) left `/home/dank/models` — 150 GB free → 658 GB free. The fp8-native
|
||||
payload is **no longer on disk**: rebuild =
|
||||
`scripts/download-frank-archon.sh` + `scripts/build-frank-archon-rad.sh`.
|
||||
2. Fetch: `scripts/download-frank-archon-iq4r.sh --fetch` as transient unit
|
||||
`iq4r-fetch` (detached-job protocol; log
|
||||
`/home/dank/logs/iq4r-fetch.log`, rc sentinel `…rc`).
|
||||
3. Deploy: `scripts/deploy.sh dankserver --service frank-archon` — launched
|
||||
detached (the swap restarts the very endpoint the operator session runs
|
||||
on; the harness timeout catches the session on return).
|
||||
4. **Boot ledger (2026-10-15, first boot, pin 1.0.13 — the w4nl codebook
|
||||
loaded, no pin bump)**: `kv_attn` fp8 page **26.00 KiB** at **607,740
|
||||
tokens** (151,935 blocks = 3.77 GiB) — byte-identical to the w4dn8 boot,
|
||||
the sizing law carries; `kv_qsa_bkey` 3.25 KiB at the same token count;
|
||||
`kv_gdn_state` 54 MiB × 6 slots, `kv_gdn_conv` 1.76 MiB × 3,
|
||||
`kv_qsa_tail` 19.5 KiB × 6; pool **5.00 GiB**, prefix checkpoints 8 ×
|
||||
56 MiB; the sizing line reads `kv cache 5.00 GiB … backs 3 sequences all
|
||||
at 200000 tokens at once, which is the worst case`; samplers print **9
|
||||
rows** per rank (3 × (1+2), inside the wht6 25-row bound);
|
||||
`speculation: 'mtp' drafts 2 token(s) a step in 2 serial pass(es)` +
|
||||
prompt-lookup fallback; slab **17,610 (rank0) / 17,658 (rank1) slots**
|
||||
(64 shadow) — above the w4dn8 record of 17,548 (ten bf16-protected
|
||||
experts per layer replace twenty-four, so more of the 4-bit table
|
||||
resides), pinned pool 7,532/7,484 slots; KV lending **4.22 GiB**
|
||||
(3,571/3,568 mover slots); arenas 896.02/988.17 MiB charged before the
|
||||
pair split; prefix cache RECOVERED 900,176 blocks + 126 checkpoint slots
|
||||
from the 64 GiB disk tier (same arch + tokenizer as w4dn8 — the tier
|
||||
survived the payload swap); froggeric template 28,234 B; vision encoder
|
||||
live (≤ 4,194,304 px, 16,384 patches/pass). Health gate passed in ~100 s
|
||||
(radiance binds the port after the full load).
|
||||
5. **Vision probe (2026-10-15)**: 300 px solid-red PNG via `image_url` —
|
||||
reasoning stream: "a solid fill of a deep, saturated red hue". Content
|
||||
came back EMPTY at `max_tokens 64`: the thinking budget consumed all 64
|
||||
(`reasoning_tokens: 64`, finish `length`) — fleet law, probe thinking
|
||||
models with ≥ a few hundred tokens.
|
||||
6. **e2e gate**: deploy exited DEPLOY INCOMPLETE — 30/33 Playwright green,
|
||||
3 red (`12-crawl`, `13-sso` search SSO, `16-marreta`) on DANKCLAW
|
||||
surfaces (vhosts intermittently returned `ERR_CONNECTION_RESET` from the
|
||||
workstation); frank-archon's own health gate passed. The swap touched no
|
||||
dankclaw runtime file (only homepage services.json text). Census of the
|
||||
Caddy/auth stack pending; see `log.md`.
|
||||
7. **Post-boot measurements (pending)**: decode ladder (99.0 t/s single
|
||||
@ 43.5k, 93.3 @ 197k, 3-way 113.9/43.3/10.9 = 18.4 t/s aggregate —
|
||||
these are the **w4dn8** numbers, the record to beat, not this file's),
|
||||
slab-limited prefill (22208/3,397 → 25600/3,584 t/s A-cold 43.5k), draft
|
||||
acceptance, and `scripts/probe-frank-archon-kv.py` leg D.
|
||||
|
||||
## Rollback
|
||||
|
||||
Revert the swap commit (`scripts/frank-archon.env` MODEL_FILE line) +
|
||||
redeploy: `/home/dank/models/rad/qwen4exp-swift15-w4dn8-i8-hc8m.rad`
|
||||
(116,416 MiB) stays on disk, KLD-gated, same shape. The fp8-native era is
|
||||
rebuild-only (see the cutover record). The w4dn8 runbook keeps every shape
|
||||
law, the KV dtype table, the slab ladder, and the tuning levers:
|
||||
[[frank-archon-radiance-swift15-w4dn8]].
|
||||
|
||||
## Related
|
||||
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — the replaced payload and the
|
||||
full shape-law library (slab slope, KV dtype arithmetic, TTM law)
|
||||
- [[frank-archon-radiance-qwen38-native-fp8]] — the fp8-native era
|
||||
(rebuild-only now; owns the descriptor-selection law)
|
||||
- [[Dual-R9700-Cooperation]] — the board swap that made TP=2 legal here
|
||||
- [[deployment]] — the service-scoped deploy this cutover rides
|
||||
@@ -3,14 +3,17 @@
|
||||
**Dankserver runs the radiance engine (StillDeadcode) on frank-archon**
|
||||
(digest-pinned `stilldeadcode/radiance` image, engine as entrypoint,
|
||||
driving `podman run` directly, root systemd unit, :7770):
|
||||
`ukisai/Swift1.5-Qwen3.8-Flash-Next` converted to **GPTQ w4**
|
||||
(`qwen4exp-swift15-w4dn8-i8-hc8m.rad`, 116,416 MiB, KLD-gated), on BOTH R9700s
|
||||
the **publisher's** 4-bit-codebook container of the official
|
||||
`Qwen/Qwen3.8-Flash-Next` (`qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
|
||||
sha256-verified at fetch — w4nl codebook experts + int8 trunk, PLE n-gram,
|
||||
MTP head and vision tower in-container), on BOTH R9700s
|
||||
TP=2 (`--tp-wire wht6`), experts in a 44 GiB/rank pinned host pool + VRAM cache
|
||||
(explicit `--vram-weights-mib 25600 --vram-kv-mib 5120` = 30,720 of the
|
||||
30,969 MiB the engine's own budget table makes claimable), the 47.7 GiB PLE
|
||||
table pinned to host RAM, **fp8 KV at 3 × 200,000**, MTP spec decode depth 2,
|
||||
froggeric v22.5 template. Rollback tags frank-archon-freetoken-pr587-20261005
|
||||
and frank-archon-vllm-shapeG-20261003),
|
||||
froggeric v22.5 template. Rollback: the in-house Swift1.5 w4dn8 `.rad` (on
|
||||
disk); tags frank-archon-freetoken-pr587-20261005 and
|
||||
frank-archon-vllm-shapeG-20261003),
|
||||
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
|
||||
(frank-dragoon, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
|
||||
digest serving the Quat3rnion ABLITERATED Swift1.5 finetune of
|
||||
@@ -37,10 +40,10 @@ graph LR
|
||||
gfx1201, 32 GiB each]
|
||||
B2[radiance StillDeadcode, digest-pinned image
|
||||
podman run, frank-archon, port 7770]
|
||||
C[Swift1.5-Qwen3.8-Flash-Next GPTQ w4 .rad
|
||||
C[Publisher qwen3.8-next-flash-fp8-iq4r-moe .rad
|
||||
2x R9700 TP=2 wht6, experts 44 GiB/rank pinned pool + VRAM,
|
||||
PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 2,
|
||||
4 x 200000, froggeric v22.5]
|
||||
3 x 200000, froggeric v22.5]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
@@ -153,7 +156,22 @@ frank-zealot, port 7771]
|
||||
target; payload on disk; owns the measured boot-pair table)**: the same
|
||||
runtime and the same 2 x 200,704 + f16 KV shape one quant rung lower —
|
||||
104.8 tok/s, 3,571 MiB true free
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — **ACTIVE since 2026-10-08**:
|
||||
- [[frank-archon-radiance-qwen38-iq4r]] — **ACTIVE since 2026-10-15**:
|
||||
the engine author's published container of the OFFICIAL (base-alignment)
|
||||
Qwen3.8-Flash-Next — `qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
|
||||
Hub-sha256-verified at fetch: 4-bit codes into a 16-level non-uniform
|
||||
codebook after a Walsh–Hadamard rotation (GPTQ + 3 coordinate-descent
|
||||
sweeps, 10M-token calibration), ten bf16-protected experts per layer,
|
||||
int8 trunk/lm_head, fp8 HC + PLE, MTP head, bf16 vision tower; card KL
|
||||
0.0789 all-tokens / top-1 92.85 % vs bf16. **MODEL_FILE is the only line
|
||||
that moved** — the 2026-10-13 shape (25600/5120 pair, 44928 pool, fp8 KV
|
||||
3 × 200,000, MTP 2, wht6, MNBT 2048, froggeric v22.5) carries verbatim.
|
||||
Rollback: revert the swap commit (w4dn8 `.rad` stays on disk). The
|
||||
fp8-native `.rad` and the raw Swift1.5 BF16 source were DELETED from
|
||||
dankserver 2026-10-15 (Frank's one-time rm, volume for this payload;
|
||||
rebuild via the fetch + build scripts named in the fp8-native runbook)
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — **RETIRED 2026-10-08 → 10-15
|
||||
(superseded by the publisher iq4r swap; payload on disk as the rollback)**:
|
||||
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` on radiance
|
||||
1.0.13 (recipe + 10M-token calibration + KLD gate vs the BF16 container);
|
||||
the FP8-native payload remains the rollback. **2026-10-13: fp8 KV, and the
|
||||
|
||||
@@ -1,199 +0,0 @@
|
||||
---
|
||||
type: reference
|
||||
title: LLM Usage Attribution
|
||||
description: Research catalog of every viable way to answer "who is using the fleet LLMs" — engine-native identity, gateways, auth proxies, tracing platforms, network/host attribution — with the measured fleet baseline (2026-10-13) and a phased adoption plan.
|
||||
tags: [llm, observability, auth, gateway, litellm, tracking, identity]
|
||||
timestamp: 2026-10-14T00:00:00-07:00
|
||||
---
|
||||
|
||||
# LLM Usage Attribution
|
||||
|
||||
Deep-research catalog (2026-10-13/14): solutions for tracking WHO uses the
|
||||
fleet endpoints (`frank-archon`, `frank-dragoon`, `frank-zealot`). Full web
|
||||
citations; live-fleet measurements marked **(measured)**.
|
||||
|
||||
## Baseline: what the fleet answers today
|
||||
|
||||
- All three endpoints are `auth: none`, plain HTTP, LAN-only
|
||||
(`models.yml` providers; no vhost on 7770/7771).
|
||||
- **(measured)** Live `/metrics` label census: `frank-archon` exports
|
||||
`vllm:*`+`radiance:*` with labels only `{model_name, engine, le,
|
||||
finished_reason, position, class}` — zero identity labels.
|
||||
`frank-dragoon` exports `halogen:*`+`llamacpp:*`.
|
||||
`frank-zealot` answers **501** — `serve-frank-zealot.sh` never passes
|
||||
`--metrics`, so the Prometheus job is structurally never-green
|
||||
(`LlmEndpointDown` fires). Fix: add `--metrics` to the ARGS array.
|
||||
- **(measured)** frank-zealot verbosity-4 journal lines carry slot/task/timing
|
||||
only — no client IP, no user. Alloy ships journal + podman ctr.log to Loki,
|
||||
so what lands there has no identity either.
|
||||
- Caddy has no `log` directive anywhere → no per-request access logs on the
|
||||
OpenWebUI vhost.
|
||||
- Net: fleet-level token telemetry exists per ENDPOINT; per-CONSUMER
|
||||
identity exists nowhere.
|
||||
|
||||
## The identity layers (pick the layer, get its resolution)
|
||||
|
||||
| # | Layer | Identity | Answers |
|
||||
|---|---|---|---|
|
||||
| L0 | DHCP/ARP | MAC, hostname, lease | which device was on the LAN |
|
||||
| L1 | DNS (Pi-hole v6) | device → queries | which device resolved/used a name |
|
||||
| L2 | L3/L4 (nftables/conntrack, JA3/JA4) | IP, TLS client lib | bytes per client, client software |
|
||||
| L3 | Transport (API keys, mTLS, SPIFFE, Tailscale) | key/cert/node | which credential connected |
|
||||
| L4 | App gateway (virtual keys → user/team/end_user) | named consumer | tokens/spend/cost per who |
|
||||
| L5 | Trace/content (Langfuse, Helicone, halogen records) | user_id, session, prompt text | what they asked, per session |
|
||||
| L6 | Client host (auditd, eBPF, netdata) | pid/uid/comm per connect | which process on which host |
|
||||
|
||||
Keys/headers/`user` fields are CLIENT-DECLARED — they group traffic, they
|
||||
prove nothing about a human (halogen's own docs warn this verbatim:
|
||||
"they group requests, and they prove nothing about who sent them").
|
||||
|
||||
## Catalog
|
||||
|
||||
### A. Engine-native (per-consumer identity inside the inference server)
|
||||
|
||||
| Engine | AuthN | Identity in logs | Per-identity metrics | OTel | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| llama.cpp | `--api-key` (multi, comma) / `--api-key-file` **(measured in the deployed build's `--help`)** | request `user` field drives slot assignment/eviction; `--log-requests` dumps bodies | `--metrics` exports `llamacpp:*`, NO key/user label | none upstream | [server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) |
|
||||
| vLLM | `--api-key` (multi) | request log; no user persistence | `vllm:*`; NO `api_key` label (cardinality, issue #53926) | native (`--otlp-traces-endpoint`) | [metrics design](https://github.com/vllm-project/vllm/blob/main/docs/design/metrics.md) |
|
||||
| radiance | no public auth docs **[unverified]** | — | **(measured)** no identity labels on live /metrics | none found | closed deployment surface; treat authN as absent until probed |
|
||||
| halogen-flash-server | `HALOGEN_API_KEY` / `HALOGEN_API_KEY_FILE` (multi, comma) | **`HALOGEN_RECORD_DIR` request records**: `turns/*.jsonl` per request — `client_family` (User-Agent), tokens, timings, finish, tool calls, plus HASHES of `user` / `prompt_cache_key` / `safety_identifier` / `metadata.user_id` (`.salt`-keyed); `units/` content-addressed text; caps `HALOGEN_RECORD_GIB`=2, `HALOGEN_RECORD_DAYS`=90; `HALOGEN_RECORD_TEXT=none` for metadata-only; `stats`/`prune` tools in-container | `halogen:*`+`llamacpp:*`, no per-key label | none | [configuration.md](https://github.com/peonist-ai/halogen-flash-server/blob/main/docs/configuration.md) — verbatim-verified 2026-10-13. Best native ledger of the fleet. |
|
||||
| SGLang | `--api-key` (single) | `--log-requests` | `sglang:*`, no key label | native (`--enable-trace`) | for any future swap |
|
||||
| Ollama | single key | request log | `ollama:` no key label | none | not in fleet |
|
||||
|
||||
Wire fields usable from clients today (all client-declared): OpenAI
|
||||
`user`, `safety_identifier`, `prompt_cache_key`, Messages
|
||||
`metadata.user_id`; LiteLLM `end_user_id`; gateway header conventions
|
||||
(`X-Forwarded-User`, `Helicone-User-Id`, `cf-aig-metadata`).
|
||||
|
||||
### B. LLM gateways (front door with virtual keys — the only layer with
|
||||
per-consumer TOKEN counts)
|
||||
|
||||
| Solution | Identity | Usage lands | License | Footprint |
|
||||
|---|---|---|---|---|
|
||||
| **LiteLLM Proxy** | virtual keys → `user_id`/`team_id`/`end_user`; budgets, rate limits | Postgres spend ledger; `/metrics` labels `api_key_alias,user,team,end_user,model,client_ip,user_agent` (`litellm_*_tokens_metric`, spend, budget gauges); OTel/Langfuse callbacks | MIT core (SSO/RBAC/audit-log enterprise) | ~150–300 MiB + Postgres |
|
||||
| Kong AI Gateway | Consumer + key-auth/JWT/OIDC/mTLS; AI consumer groups | Postgres; `kong_*` per-consumer; file-log/zipkin/otel | OSS core; token-aware AI plugins enterprise | 0.5–1 GiB |
|
||||
| Apache APISIX | Consumer + key-auth/JWT | `apisix_llm_prompt_tokens`/`completion_tokens` with `consumer` label; http-logger/kafka | Apache-2.0 | 200–400 MiB |
|
||||
| Bifrost (maximhq) | virtual keys `sk-bf-*`/`x-bf-vk`, budgets | Prometheus/OTel, config store (SQLite/Postgres) | Apache-2.0, Go | ~150–300 MiB |
|
||||
| Portkey OSS gateway | virtual keys | logs/OTel | MIT | 200–400 MiB |
|
||||
| Higress AI GW | keys (+OIDC ent) | `higress_*`, ai-logging | Apache-2.0 CE | Istio-based, 0.5 GiB+ |
|
||||
| Envoy AI GW / Gateway API Inference Extension / LLM Gatekeeper | delegated auth | Envoy metrics/access logs | Apache-2.0 | K8s-native |
|
||||
| Gloo / KrakenD / Gravitee AI gateways | keys, OIDC | metrics/logs | OSS core, AI features enterprise | heavy |
|
||||
| Cloudflare AI Gateway | Access JWT `cf.user_id`, `cf-aig-metadata` | Cloudflare-managed analytics/logs | SaaS (exotic for LAN; Tunnel possible) | edge |
|
||||
|
||||
([LiteLLM virtual keys](https://docs.litellm.ai/docs/proxy/virtual_keys),
|
||||
[prometheus](https://docs.litellm.ai/docs/proxy/prometheus),
|
||||
[APISIX ai-proxy](https://apisix.apache.org/docs/apisix/plugins/ai-proxy/),
|
||||
[Bifrost](https://github.com/maximhq/bifrost),
|
||||
[Kong AI](https://developer.konghq.com/index/ai-gateway/))
|
||||
|
||||
### C. Auth proxies / identity infrastructure (identity without token counts)
|
||||
|
||||
- **Ory Oathkeeper** (MIT, ~50–100 MiB): per-consumer api-key/JWT/
|
||||
client-credentials → header mutation (`X-User: {{ .Subject }}`) +
|
||||
decision logs.
|
||||
- **oauth2-proxy** (Apache-2.0): OIDC (Authentik) incl. client_credentials,
|
||||
`--pass-user-headers` → `X-Forwarded-User`; ~100 MiB.
|
||||
- **Authentik itself**: proxy-provider custom headers (already injects
|
||||
`X-Authentik-Username`/`-Groups`, spoof-stripped — the /whoami seam proves
|
||||
the pattern on this host); event log + webhook forwarding.
|
||||
- **Caddy `tls client_auth`** (mTLS per client cert) + access-log fields —
|
||||
Caddy already fronts both hosts.
|
||||
- **Smallstep step-ca**: short-lived per-device certs, ACME-style.
|
||||
- **SPIFFE/SPIRE**: per-workload SVIDs (k8s-flavored; heavy for quadlets).
|
||||
- **Tailscale / NetBird**: per-node identity, ACL + network-flow logs,
|
||||
headscale for self-hosted control. Best for off-LAN laptops.
|
||||
- **ApiAuth** (nginx auth_request): the minimal per-consumer apikey sidecar.
|
||||
|
||||
### D. LLM observability / tracing (prompt-level per-user audit)
|
||||
|
||||
| Platform | Identity | Content | Self-host | Shape |
|
||||
|---|---|---|---|---|
|
||||
| **Langfuse** | `user_id`,`session_id`, tags | full input/output, masking hook | ✅ ClickHouse, Apache-2.0 | fed via LiteLLM callback / OTLP / SDK |
|
||||
| **Helicone** | api-key→user, `Helicone-User-Id` | full, tiered redaction | ✅ compose (ClickHouse+PG+Redis+Jawn), Apache-2.0 | PROXY — can be the front door itself |
|
||||
| Arize Phoenix | `user.id`,`session.id` | opt-in `gen_ai.*` | ✅, Apache-2.0 | OTLP collector |
|
||||
| OpenLIT | `user.id`,`session.id` | 3-tier capture | ✅, Apache-2.0 | OTLP+ClickHouse |
|
||||
| SigNoz LLM | OTel attrs | opt-in | ✅, Apache-2.0 | OTLP |
|
||||
| OpenLLMetry/Traceloop | assoc props | opt-in | ✅ Helm | OTel |
|
||||
| Lunary / W&B Weave / LangSmith | user/session | full | enterprise-gated | — |
|
||||
|
||||
OTel GenAI semconv (Development-stability): `gen_ai.operation.name`,
|
||||
`gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`/
|
||||
`output_tokens`, `gen_ai.conversation.id`, content attrs opt-in; `user.id`
|
||||
from the registry. vLLM and SGLang emit OTLP natively; llama.cpp only via
|
||||
external SDK — **the fleet's engines are OTel-silent at request level**,
|
||||
which is why the gateway callback (LiteLLM→Langfuse) is the practical feed.
|
||||
|
||||
### E. Network/host passive attribution (no client cooperation)
|
||||
|
||||
- **Pi-hole v6** (already on Dankclaw): query log API per device,
|
||||
`pihole6_exporter` + query-logger→Loki (GPLv3 exporter).
|
||||
- **Caddy access logs**: add a `log` directive (JSON) → Alloy→Loki; with the
|
||||
auth chain it carries `X-Authentik-Username` for browser users.
|
||||
- **OpenWebUI admin**: per-user daily token stats, events/logs, analytics API
|
||||
(`/api/v1/analytics/*`) — per-USER identity exists here already (MIT).
|
||||
- **auditd** connect-syscall rules (uid/pid/comm per connect to 7770/7771),
|
||||
ship via audisp→Loki.
|
||||
- **eBPF**: Tetragon (process+socket events, policies), Coroot, DeepFlow,
|
||||
bpftrace `tracepoint:syscalls:sys_enter_connect` one-liners. K8s-shaped
|
||||
for single hosts, but agents.plugin-style tools work.
|
||||
- **nftables named counters** per consumer IP (bytes; reload resets).
|
||||
- **netdata apps.plugin**: per-process network bytes, auto-discovered.
|
||||
- **JA3/JA4 TLS fingerprinting**: client library only, TLS-only; the LAN is
|
||||
plaintext HTTP today → use the HTTP **User-Agent census** instead (free
|
||||
from any access log).
|
||||
- **DHCP/ARP census** (leases + `ip neigh`): device roster.
|
||||
|
||||
## Phased plan for this fleet
|
||||
|
||||
**Phase 0 — repo-only, no new services**
|
||||
1. frank-zealot: add `--metrics` (fixes the measured 501 + the
|
||||
structurally-red Prometheus job + the firing `LlmEndpointDown`).
|
||||
2. frank-dragoon: set `HALOGEN_API_KEY_FILE` (multi-key per consumer) +
|
||||
mount `HALOGEN_RECORD_DIR` (start `HALOGEN_RECORD_TEXT=none`; flip to
|
||||
`all` if prompt audit is wanted) + Alloy ship of `turns/`.
|
||||
3. frank-zealot: `--api-key-file` (multi-key, verified in the deployed
|
||||
build); radiance: probe for `--api-key` support (contract facts are
|
||||
silent — add to `fleet-probes`).
|
||||
4. models.yml: per-consumer bearer keys (`auth: none` → key) — the provider
|
||||
schema supports apiKey (pi-era conversion note).
|
||||
5. Caddy: `log` on `ai.franksdank.cloud` (OpenWebUI browser census, JSON →
|
||||
Loki).
|
||||
Limits: keys gate access, and only the engines' request records (halogen)
|
||||
see them per-request; llama.cpp/radiance/vLLM metrics stay identity-blind.
|
||||
|
||||
**Phase 1 — LiteLLM Proxy quadlet on Dankclaw = the front door.**
|
||||
One virtual key per consumer (omp-<workstation> ×N, openwebui, hermes,
|
||||
agents, bench). Per-key/user/team Prometheus labels into the existing
|
||||
Grafana+Alertmanager; Postgres spend ledger; budgets + rate limits per
|
||||
key. This is the ONLY layer that yields tokens-per-who without engine
|
||||
cooperation. (Bifrost is the lighter Apache-2.0 Go alternative; Helicone
|
||||
the proxy-native-observability alternative.)
|
||||
|
||||
**Phase 2 — prompt-level audit.** Langfuse self-hosted fed by the LiteLLM
|
||||
callback: per-user traces, sessions, cost; masking + retention knobs.
|
||||
|
||||
**Phase 3 — ambitious identity infra.** step-ca + Caddy mTLS per device;
|
||||
Tailscale (+flow logs) for off-LAN; auditd/eBPF process attribution on the
|
||||
agent hosts; Pi-hole query logger; nft counters per consumer IP. Browser
|
||||
users stay owned by OpenWebUI analytics + its per-user API keys; a pipeline
|
||||
can forward `X-Forwarded-User` so the gateway sees them too.
|
||||
|
||||
## Cautions
|
||||
|
||||
- Client-declared identity (keys, `user`, headers) groups traffic; it does
|
||||
not authenticate humans. Authenticated identity = Authentik (browser) /
|
||||
mTLS (device) / Tailscale (node).
|
||||
- Plain-HTTP LAN = bearer keys sniffable by any tenant; treat keys as
|
||||
identity labels on the LAN, not secrets, until TLS/mTLS at the door.
|
||||
- Prompt-content logging stores a plaintext copy of everything — set
|
||||
retention caps (halogen GIB/days; Langfuse masking; Loki retention) and
|
||||
gitignore the record dirs.
|
||||
- Household-scale cardinality is safe for per-key Prometheus labels;
|
||||
the engines' refusal of key labels is a multi-tenant concern, not ours.
|
||||
|
||||
## Related
|
||||
- [[Fleet Audit and Posture]] — monitoring gaps census
|
||||
- [Observability](../Infrastructure/Observability.md) — the Loki/Prometheus
|
||||
pipes every sink above uses
|
||||
- [LLM index](../LLM/index.md) — endpoint runbooks (env trios the Phase-0
|
||||
edits touch)
|
||||
@@ -23,9 +23,6 @@ Procedures for operating the fleet. The LLM-specific pages live under
|
||||
posture, whole-fleet audit, bulk redaction, stash audits
|
||||
- [[SSH and Remote Access]] — passwordless bootstrap without sshpass, fish
|
||||
login shells, GUI-hang diagnosis over ssh
|
||||
- [[LLM Usage Attribution]] — who-is-using-the-fleet-LLMs research catalog:
|
||||
engine-native identity, gateways, tracing, network/host attribution, and
|
||||
the phased plan
|
||||
|
||||
## Related
|
||||
- [Observability Provisioning](../Infrastructure/Observability%20Provisioning.md)
|
||||
|
||||
@@ -134,9 +134,6 @@ skill-cluster. See [log.md](log.md) for the wave plan and results.
|
||||
posture, whole-fleet audit, bulk redaction
|
||||
- [[SSH and Remote Access]] — passwordless bootstrap, fish login shells,
|
||||
GUI-hang diagnosis over ssh
|
||||
- [[LLM Usage Attribution]] — who-is-using-the-fleet-LLMs research catalog:
|
||||
engine-native identity, gateways, tracing, network/host attribution,
|
||||
phased plan
|
||||
|
||||
**Architecture:**
|
||||
|
||||
@@ -185,12 +182,20 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
### Deployment and Topology
|
||||
|
||||
- [[deployment]] — unified deploy and update process for both servers
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — STAGED frank-archon runbook:
|
||||
- [[frank-archon-radiance-qwen38-iq4r]] — **ACTIVE frank-archon runbook
|
||||
(2026-10-15)**: the PUBLISHER's container of the official
|
||||
Qwen3.8-Flash-Next (`qwen3.8-next-flash-fp8-iq4r-moe.rad`, 116,322 MiB,
|
||||
w4nl 4-bit-codebook experts + int8 trunk, Hub-sha256-verified at fetch)
|
||||
swapped in as a MODEL_FILE-only change — the 2026-10-13 shape (slab
|
||||
25600 / fp8-KV 5120, 3 × 200,000, host pool 44,928, MTP depth 2) carries
|
||||
verbatim; rollback = revert the swap commit (the w4dn8 `.rad` stays on
|
||||
disk; the fp8-native `.rad` was deleted 2026-10-15, rebuildable)
|
||||
- [[frank-archon-radiance-swift15-w4dn8]] — RETIRED 2026-10-08 → 10-15 frank-archon runbook:
|
||||
Swift1.5 Qwen3.8-Flash-Next BF16 → GPTQ-calibrated W4Dn8 `.rad` (radiance
|
||||
1.0.13), gated on KLD ≤ 0.03 vs the BF16 container and decode ≥ 30.8 t/s;
|
||||
activates by flipping `MODEL_FILE` in `scripts/frank-archon.env`, rollback
|
||||
= revert that line (the FP8-native `.rad` stays on disk)
|
||||
- [[frank-archon-radiance-qwen38-native-fp8]] — the ACTIVE frank-archon
|
||||
- [[frank-archon-radiance-qwen38-native-fp8]] — the RETIRED 2026-10-05 → 10-08 frank-archon
|
||||
runbook (2026-10-05): the StillDeadcode radiance engine (digest-pinned
|
||||
287 MiB image, libr4d gfx1201 kernels) serving the OFFICIAL
|
||||
`Qwen/Qwen3.8-Flash-Next-FP8` at native block-FP8 (no-recipe convert
|
||||
|
||||
@@ -1745,3 +1745,57 @@ every restart since the cutover.
|
||||
[[frank-archon-radiance-swift15-w4dn8]], and the LLM index entry corrected
|
||||
(the w4dn8 entry said STAGED, the fp8-native one said ACTIVE).
|
||||
|
||||
## 2026-10-15 — frank-archon: publisher iq4r payload swap (MODEL_FILE-only)
|
||||
|
||||
- Ask: replace the served model with
|
||||
`StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` — the engine author's
|
||||
published rad-convert of the OFFICIAL Qwen3.8-Flash-Next (w4nl codebook
|
||||
experts, ten bf16-protected/layer, int8 trunk, PLE/MTP/vision in-container,
|
||||
card KL 0.0789 / top-1 92.85 %) — **no other argument changes**, mindful
|
||||
the operator session runs on the endpoint being swapped.
|
||||
- Disk: Frank authorized one `rm` — the spent Swift1.5 BF16 source (336 GiB)
|
||||
and the two-era-old `qwen4exp-fp8-native.rad` (173 GiB) left
|
||||
`/home/dank/models`; 150 GB → 658 GB free. fp8-native is now rebuild-only
|
||||
(fetch + build scripts); rollback for THIS swap is the w4dn8 `.rad`
|
||||
(on disk) by reverting the commit.
|
||||
- Fetch: new `scripts/download-frank-archon-iq4r.sh` (hfget, exact bytes
|
||||
121,969,901,568 + Hub LFS sha256, `--verify-only` default) as transient
|
||||
unit `iq4r-fetch`; verified, staged sidecar files kept as provenance.
|
||||
- Shape carried VERBATIM (25600/5120 pair, 44928 pool, fp8 KV 3 × 200,000,
|
||||
MTP 2, wht6, MNBT 2048, froggeric v22.5). Tests re-keyed
|
||||
(`frank_archon_unit.bats` pins the new payload id + sha256, keeps the
|
||||
w4dn8 rollback provenance), homepage card + trio lore + both indexes
|
||||
updated; new runbook [[frank-archon-radiance-qwen38-iq4r]]. Commits
|
||||
`6dcf4a5` (swap) + `f44c398` + `c5ac072` (docs), full bats green.
|
||||
- Cutover self-severed by design (detached `deploy.sh` survived the session
|
||||
death when frank-archon restarted); engine healthy in ~100 s. First-boot
|
||||
ledger: w4nl loads on pin 1.0.13 (no pin bump); kv_attn 26.00 KiB fp8 @
|
||||
607,740 tokens (identical geometry); slab 17,610/17,658 slots per rank
|
||||
(> w4dn8's 17,548 — fewer bf16-protected experts pack more 4-bit table);
|
||||
sampler 9 rows; MTP 2/step; prefix tier RECOVERED 900,176 blocks across
|
||||
the swap (arch+tokenizer unchanged). Vision probe green (reasoning read
|
||||
the solid-red PNG; the 64-token budget went all to thinking — fleet law).
|
||||
- e2e gate went red on three dankclaw specs — NOT the swap: dankclaw rebooted
|
||||
01:41 and the gluetun boot-race dependency-failed `searxng`, `marreta`,
|
||||
and `marreta-browser` (jobs consumed while gluetun's start job retried;
|
||||
`Restart=always` never armed because the services never ran). Cascade
|
||||
restarted; skill codified (`gluetun-dependent-boot-race-recovery`).
|
||||
Gate rerunning against the repaired fleet; frank-archon's own health gate
|
||||
passed every deploy. Open: decode/prefill ladders vs the w4dn8 record,
|
||||
probe leg D — scheduled probe window.
|
||||
|
||||
## 2026-10-15 — authentik: two more human accounts (Dankclaw)
|
||||
- **Live — mitch (pk 22) and eric (pk 23) provisioned via `ak shell` ORM, no
|
||||
repo objects.** Each `<user>@franksdank.cloud` in the human-user template
|
||||
shape (`is_active=True`, display name, password via `set_password` on stdin
|
||||
only — never argv, so nothing lands in `ps`), groups `Non-Admin Users` +
|
||||
`jellyfin_users` + `openwebui_users`; no admin group granted. Verified
|
||||
through the exact PasswordStage code path (`authenticate()` accepted each
|
||||
password and rejected a wrong one), `app_entitlements(open-webui)` →
|
||||
`Open WebUI Users` for both (identical to the ridley precedent), and
|
||||
`is_superuser=False` / `path=users`. Zero `proxysession` rows for either —
|
||||
groups were set before any login, so no claim purge was due. Downstream:
|
||||
Jellyfin visibility rides the LDAP outpost cache (expect lag); OpenWebUI
|
||||
mints their accounts as role `pending` on first login.
|
||||
- `/serverAppConfigs-docs/Infrastructure/Authentik.md` — human-user template
|
||||
census extended with pk 22–23.
|
||||
@@ -366,13 +366,13 @@
|
||||
"model": {
|
||||
"unit": "archon",
|
||||
"role": "Frank's strongest and fastest model runtime",
|
||||
"hf": "https://huggingface.co/frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700",
|
||||
"hf": "https://huggingface.co/StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe",
|
||||
"tagline": "...Power Overwhelming...",
|
||||
"sfx": [
|
||||
"archon-power.ogg"
|
||||
],
|
||||
"equiv": "Equivalent to Claude Opus 4.8 / OpenAI-Terra (Summer 2026 frontier intelligence).",
|
||||
"spec": "Our own build: a GPTQ-w4 quant of Swift1.5-Qwen3.8-Flash-Next, quantized and tuned in-house (radiance, ~114 GiB) on twin AMD R9700s with more than 200 GB of system RAM — 200k-token context, three submissions at a time; sees images.",
|
||||
"spec": "The engine author's published quant: 4-bit codebook experts + int8 trunk of Qwen3.8-Flash-Next (radiance, ~114 GiB) on twin AMD R9700s with more than 200 GB of system RAM — 200k-token context, three submissions at a time; sees images.",
|
||||
"inputs": [
|
||||
"text",
|
||||
"image"
|
||||
|
||||
@@ -3,8 +3,10 @@
|
||||
#
|
||||
# This file asserts the RADIANCE shape (StillDeadcode engine, digest-pinned
|
||||
# stilldeadcode/radiance image, BOTH R9700s TP=2 on the wht6 wire, serving
|
||||
# the Swift1.5 Qwen3.8-Flash-Next GPTQ-w4 .rad built by
|
||||
# scripts/convert-swift15-w4dn8.sh). It replaces the FreeToken seam of
|
||||
# the publisher's 4-bit-codebook .rad of the official Qwen3.8-Flash-Next
|
||||
# (fetched by scripts/download-frank-archon-iq4r.sh; the in-house
|
||||
# Swift1.5 w4dn8 .rad of scripts/convert-swift15-w4dn8.sh is the on-disk
|
||||
# rollback). It replaces the FreeToken seam of
|
||||
# 2026-10-05 (git tag frank-archon-freetoken-pr587-20261005) and pins what
|
||||
# flags alone do not say: the served window clears the harness worst case,
|
||||
# the KV pool backs seqs x max-model-len at the declared cache dtype (an
|
||||
@@ -14,7 +16,7 @@
|
||||
# half of RAM), the served window is what models.yml must
|
||||
# quote, the host-networking contract (no --publish), the image gate on a
|
||||
# name@sha256 ref, the .rad payload gate with its size floor, the
|
||||
# chat-template override mount, the self-healing requirement (the run returns before
|
||||
# chat-template override mount, the self-healing law (the run returns before
|
||||
# the engine is up, so the serve script must wait on the container for
|
||||
# Restart=always to mean anything), the deployer's false-green MainPID check,
|
||||
# the vendored hfget fetch path (no token for this ungated repo), and the
|
||||
@@ -28,7 +30,7 @@ SERVE="$REPO_ROOT/scripts/serve-frank-archon.sh"
|
||||
TTM_LIB="$REPO_ROOT/scripts/ttm-preflight.sh"
|
||||
ENV_FILE="$REPO_ROOT/scripts/frank-archon.env"
|
||||
REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
|
||||
DOWNLOAD="$REPO_ROOT/scripts/download-frank-archon-swift15.sh"
|
||||
DOWNLOAD="$REPO_ROOT/scripts/download-frank-archon-iq4r.sh"
|
||||
BUILD="$REPO_ROOT/scripts/convert-swift15-w4dn8.sh"
|
||||
HFGET="$REPO_ROOT/scripts/hfget.sh"
|
||||
TEMPLATE="$REPO_ROOT/scripts/qwen38-froggeric-v22.5.jinja"
|
||||
@@ -136,11 +138,10 @@ nosudo_bin() {
|
||||
# Concurrency: 3 slots x the served window is the capacity contract, and
|
||||
# the fused TP all-reduce is declared over seqs x (1 + spec) = 9 rows,
|
||||
# under the 25-row bound at which the wht6 wire stays bit-exact. The
|
||||
# "seqs 2 cannot boot" is the failure mode of a runtime that compiles
|
||||
# kernels specialized to the batch shape — it refuses a slot count it has
|
||||
# never compiled for. This engine is shape-agnostic: seqs 2 boots, and
|
||||
# measured decode ties seqs 2 and 3 inside the +-3% noise floor — slots
|
||||
# buy concurrency, not speed.
|
||||
# "seqs 2 cannot boot" law belongs to the retired tcclaviger vLLM fork
|
||||
# (torch.compile query_start_loc constraint, 2026-10-02); radiance booted
|
||||
# seqs 2 in the fp8-native era and the 2026-10-06 sweep had seqs 2 and 3
|
||||
# tied inside the +-3% noise floor — slots buy concurrency, not speed.
|
||||
grep -q '^FRANK_ARCHON_MAX_SEQS=3$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_TP=2$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_TP_WIRE=wht6$' "$ENV_FILE"
|
||||
@@ -148,8 +149,8 @@ nosudo_bin() {
|
||||
# expert_tiered placement: vram_weights + the pinned host pool serve the
|
||||
# per-rank resident set TOGETHER, so cutting the VRAM slab to pay for the
|
||||
# KV pool moves the pool with it. KV is QUANTIZED fp8 (the engine takes
|
||||
# only bf16|fp8 — no per-tensor K/V split), and the PLE n-gram table is
|
||||
# pinned in host RAM so the cards read it without page-faulting mid-token.
|
||||
# only bf16|fp8 — no per-tensor K/V split), the PLE n-gram table is pinned
|
||||
# in host RAM per Frank's law.
|
||||
grep -q '^FRANK_ARCHON_PLACEMENT=expert_tiered$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_HOST_POOL_MIB=44928$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_VRAM_WEIGHTS_MIB=[1-9]' "$ENV_FILE"
|
||||
@@ -382,18 +383,23 @@ nosudo_bin() {
|
||||
grep -q 'exec podman wait "$FRANK_ARCHON_CONTAINER"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "the model id matches the payload's build and fetch scripts" {
|
||||
# The payload dir/file are the convert script's output layout; the fetch
|
||||
# script must name the repo the convert stages from. (FP8-native era,
|
||||
# the rollback payload, keeps its own trio: build-frank-archon-rad.sh +
|
||||
# download-frank-archon.sh.)
|
||||
@test "the model id matches the payload's fetch script (and the rollback's build script)" {
|
||||
# The deployed payload is the PUBLISHER's .rad: the fetch script names
|
||||
# the repo, the file, and the Hub's LFS sha256; nothing is converted on
|
||||
# the host for it. The in-house w4dn8 rollback keeps its own trio
|
||||
# (convert-swift15-w4dn8.sh + download-frank-archon-swift15.sh, and the
|
||||
# fp8-native era keeps build-frank-archon-rad.sh +
|
||||
# download-frank-archon.sh).
|
||||
grep -q '^FRANK_ARCHON_MODEL_DIR=/home/dank/models/rad$' "$ENV_FILE"
|
||||
grep -q '^FRANK_ARCHON_MODEL_FILE=qwen4exp-swift15-w4dn8-i8-hc8m.rad$' "$ENV_FILE"
|
||||
grep -q 'qwen4exp-swift15-w4dn8-i8-hc8m.rad' "$BUILD"
|
||||
grep -q 'ukisai/Swift1.5-Qwen3.8-Flash-Next' "$DOWNLOAD"
|
||||
grep -q '^FRANK_ARCHON_MODEL_FILE=qwen3.8-next-flash-fp8-iq4r-moe.rad$' "$ENV_FILE"
|
||||
grep -q 'StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe' "$DOWNLOAD"
|
||||
grep -q 'qwen3.8-next-flash-fp8-iq4r-moe.rad' "$DOWNLOAD"
|
||||
grep -q '0af5e96244e80c21ac8edca1719dae49b8a201b94a2f3994c3e24026ceaa4d20' "$DOWNLOAD"
|
||||
grep -q 'hfget.sh' "$DOWNLOAD"
|
||||
# The repo is UNGATED — no token may be required on the fetch path.
|
||||
! grep -q 'HF_TOKEN' "$DOWNLOAD"
|
||||
# Rollback provenance: the w4dn8 build script names its payload.
|
||||
grep -q 'qwen4exp-swift15-w4dn8-i8-hc8m.rad' "$BUILD"
|
||||
}
|
||||
|
||||
@test "no bare HIP-reserved env var is defined" {
|
||||
|
||||
@@ -11,9 +11,8 @@
|
||||
# vision file names AND byte counts inside the /models mount, and the
|
||||
# tokenizer's flat-file shape (a HF cache snapshot's symlinks dangle inside
|
||||
# the container). A serve gate and a deployer gate that disagree is the
|
||||
# crash-loop class this file exists to kill: both sides are pinned here,
|
||||
# plus the one engine rule the card is emphatic about: HALOGEN_SPEC_ADAPT=0 —
|
||||
# the default
|
||||
# 2026-10-01 crash-loop; both sides are pinned here, plus the one engine
|
||||
# law the card is emphatic about: HALOGEN_SPEC_ADAPT=0 — the default
|
||||
# adaptive policy silently disables the MTP head per request and HALVES
|
||||
# decode.
|
||||
#
|
||||
@@ -112,10 +111,9 @@ good_env() {
|
||||
@test "the head stays on: adaptive spec must stay 0 and the window shape pinned" {
|
||||
# The card is emphatic: the default adaptive policy turns the MTP head
|
||||
# off for whole requests after low acceptance and HALVES decode. The
|
||||
# two-slot coupling of the 27B engine — which disabled drafting whenever a
|
||||
# second slot existed — does NOT carry over: on this line batching and
|
||||
# speculation coexist (the card's quickstart ships 2 slots WITH the head),
|
||||
# so 2 is concurrent capacity, not a tax.
|
||||
# two-slot law does NOT carry over from the 27B engine — on this line
|
||||
# batching and speculation coexist (the card's quickstart ships 2
|
||||
# slots WITH the head), so 2 is concurrent capacity, not a tax.
|
||||
grep -qE '^HALOGEN_SPEC_ADAPT=0$' "$ENV_FILE"
|
||||
grep -qE '^HALOGEN_KV_SLOTS=2$' "$ENV_FILE"
|
||||
# The cap/timeout pair is coupled (FLAGS.md table): a request at the
|
||||
@@ -180,9 +178,8 @@ good_env() {
|
||||
}
|
||||
|
||||
@test "the serve script refuses a checkpoint of the wrong byte count" {
|
||||
# A truncated curl exit-0 artifact must not boot: curl exits 0 on an HTTP
|
||||
# error and writes the error body as the file, so the Hub's declared byte
|
||||
# count is the contract that catches it.
|
||||
# A truncated curl exit-0 artifact must not boot; the Hub's byte
|
||||
# count is the contract (fleet fetch law).
|
||||
podman_stub
|
||||
good_env
|
||||
run env PATH="$TEST_TMP/bin:$PATH" "${GOOD_ENV[@]}" \
|
||||
@@ -283,7 +280,7 @@ $(cat "$TEST_TMP/serve.log")"
|
||||
! grep -q 'PODMAN wait' "$TEST_TMP/serve.log"
|
||||
}
|
||||
|
||||
@test "the shipped env keeps the harness fit budget inside the engine ceiling" {
|
||||
@test "the shipped env keeps the harness fit law inside the ceiling" {
|
||||
# 262144 (the checkpoint's native window) is the per-slot hard ceiling
|
||||
# (a longer prompt is a 400 naming the limit, never a silent
|
||||
# truncation). The harness clamps max_tokens = min(45056, 217088 − est
|
||||
|
||||
Reference in new issue
Block a user