Frank
a9d45e7012
Measured, 512 tokens, greedy, cold, at the deployed 3 x 200,000 fp8 shape: prefill 22,208 -> 3,397, 24,000 -> 3,290, 25,600 -> 3,290 t/s at 43.5k — the third slab step bought nothing, so the slab is a threshold whose cliff is the 2026-10-11 arm at 26,071 (4,379), not a slope. Decode ladder: 99.0 t/s single-stream at 43.5k, 93.3 at 197k (nearly window-free), and 3 concurrent streams 113.9/43.3/10.9 = 18.4 t/s aggregate — the slot count is paid in decode concurrency, the axis the shape hunt never measured because probe leg B generates 24 tokens. probe-frank-archon-kv.py gains leg D (N-stream decode, 512 tokens, per-stream + aggregate) so no resize is priced on prefill alone.
1748 lines
140 KiB
Markdown
1748 lines
140 KiB
Markdown
# Bundle Change Log
|
||
|
||
## 2026-10-13
|
||
- **frank-archon outage: a kernel knob, not the model.** The 2026-10-12 23:17
|
||
power failure on dankserver reset `/sys/module/ttm/parameters/pages_limit`
|
||
from the manual 200 GiB raise back to its default (half of RAM = 125,988 MiB),
|
||
below the ~147 GiB pinned-RAM ledger the engine needs; the serve script's
|
||
preflight named the refusal and exited 1, and `Restart=always` re-ran it
|
||
**6,919 times in ~20 h** — `activating (auto-restart)`, exit 1 in 50 ms, no
|
||
container, no engine log. Law adopted: **a sysfs raise is a countdown to the
|
||
next reboot**, so the gate raises the knob itself. New
|
||
`scripts/ttm-preflight.sh` is sourced by the serve script (verify, raise,
|
||
re-verify, then refuse by name) and is also the unit's
|
||
`ExecStartPre=+…--raise` root hook — `+` because the service is `User=dank`
|
||
and the knob is root-owned 0644, which is what the first attempt (a plain
|
||
write inside the serve script) got wrong. `FRANK_ARCHON_TTM_TARGET_MIB=204800`
|
||
records the host ceiling.
|
||
- **Shape: 4 × 200,000 with fp8 KV (Frank's call).** `--max-num-seqs 4`,
|
||
`--kv-cache-dtype fp8`, pool and slab untouched (6,912 / 22,208 MiB = the
|
||
measured 29,120/card ceiling): the pool must back `seqs × max-model-len`, and
|
||
at bf16 four slots cost 11,679 MiB, which would cut the resident expert slab
|
||
to 17,441 (−33 %) — fp8 costs 6,601 and leaves it alone. 4 × (1 + spec 2) = 12
|
||
fused-all-reduce rows, inside the 25-row bound where `wht6` is bit-exact.
|
||
`models.yml` needed no change: the per-request window is unchanged, only the
|
||
concurrency.
|
||
- **Deployed at `bae78e7`, gates green.** Boot ledger: `kv cache 6.75 GiB …
|
||
backs 4 sequences all at 200000 tokens at once` — 843,276 tokens against the
|
||
sizing law's 843,566 (0.03 %), sampler at 12 rows, weights filled to 147 KiB
|
||
of slack, cards ~2 GiB free, KV lending 5.87 GiB to the slab. Live probe
|
||
(now reading the shape from the deployed env): four concurrent 196,960-token
|
||
chats, **4/4 exact marker retrieval**, 787,842 tokens in 238.4 s = 3,305 t/s
|
||
aggregate prefill, no preemption; prefix cache verified (repeat answered in
|
||
1.0 s, `cache_n=43008`). Playwright e2e 32 passed + 1 flaky (the known Plex
|
||
homepage retry); archon bats seam 21/21 including three new ttm-gate tests.
|
||
- **Re-cut twice the same evening (Frank's calls), each read off the ledger.**
|
||
4 → 3 slots, pool 6,912 → 5,120: three slots need 5,063 MiB and the 6,912 pool
|
||
was funding 858,969 tokens — 43 % over the contract. Then slab
|
||
24,000 → 25,600, claiming 1,600 of the 1,849 MiB the engine leaves above its
|
||
OWN settle floor (`card 0 settled at 2.01 GiB free against 209.19 MiB (the
|
||
96.00 MiB --gpu-headroom-mib and 113.19 MiB kept for the static region)`): the
|
||
real ceiling is 30,969 MiB, not the 29,120 my 1 GiB "keep free" reserve
|
||
invented on 2026-10-11. The pair now spends 30,720 and settles
|
||
`unclaimed 477.30 MiB` (cards ~336 MiB free).
|
||
- **The prediction shipped with that arm was falsified by its own measurement,
|
||
which is the whole point of banking the A/B.** Claim: the slab is a THRESHOLD
|
||
below ~26 GiB (22,208 → 3,397 and 24,000 → 3,290 t/s are inside noise of each
|
||
other), so 1.6 GiB buys ~0. Measured at 25,600: **3,584 t/s** single-stream
|
||
cold at 43.5k (+8.9 %) and **3,589 t/s** 3-way aggregate at 197k (wall
|
||
193.7 → 164.5 s, +17.6 %) — slab 16,235 → 17,548 slots while the pinned host
|
||
pool hands the same experts back (8,893 → 7,580). The slab is a SLOPE; two
|
||
points inside noise of each other are not a plateau. The cliff is above
|
||
25,600: 26,071 measured 4,379 t/s (+22 % for +471 MiB), unreachable at three
|
||
slots (26,112 + 5,120 > the ceiling) and free at two.
|
||
- **Capacity verified live at 3 slots:** three concurrent 199,984-token prompts
|
||
accepted and completed (599,952 tokens resident), ledger
|
||
`backs 3 sequences all at 200000 tokens at once` at 607,740 tokens against the
|
||
law's 608,027. A fourth full-height request is refused as too large to ever
|
||
fit — radiance never truncates. Probe law: the oversized-prompt counter is
|
||
FREE (the refusal fires when PROMPT tokens alone exceed the window —
|
||
`prompt: 202,777 prompt tokens does not fit the 200000 token context`) and the
|
||
window is ~514,286 characters at the measured 2.5714 chars/token, so a
|
||
calibration probe below that pays a real prefill instead of reporting.
|
||
- **Open call for Frank:** two slots (pool 3,072 + slab 27,897 = the ceiling
|
||
exactly) crosses the cliff for the +22 %, and costs the third concurrent 200k
|
||
conversation. Untested arm; capacity decision, not tuning.
|
||
- Seam: the pair ceiling is derived from the engine's budget table (30,969) and
|
||
owns that arithmetic alone; the KV-pool contract (seqs × window at the dtype's
|
||
measured rate + 110 MiB a sequence + 448 MiB of prefix checkpoints) keeps its
|
||
own test. 371/371 tests and e2e 33/33 on both deploys.
|
||
- **Power-failure casualties, NOT fixed (outside the archon window):**
|
||
`mnt-md0.mount` failed at boot, so `/mnt/md0` is an empty mountpoint (the
|
||
raid6 array itself is healthy, 7/7; fstab has `nofail` so boot continued) —
|
||
media libraries on it read empty. `frank-job-digest.service` (frankbook
|
||
job-hunt digest) is in `list-units --state=failed`.
|
||
- **The slab is a threshold, not a slope** — measured, same method (probe A
|
||
leg: 43,512 tokens, greedy, cold, disk prefix cache clean on every boot):
|
||
22,208 → 3,397 t/s, 24,000 → 3,290, **25,600 → 3,290**. The third step
|
||
bought nothing; the +22 % belongs to the 2026-10-11 arm at slab 26,071
|
||
(4,379 t/s), so the cliff is real and sits at ~26 GiB. The 2-slot arm that
|
||
would clear it (pool 3,072 + slab 27,897) remains untested — and the
|
||
paragraph above is amended: that arm's payoff is +22 % prefill at 2 slots,
|
||
not the +8.9 % the deployed pair ever showed (24,000 → 25,600 = 0 %).
|
||
- **Frank's decode challenge, answered with numbers**: 512 tokens, greedy,
|
||
cold — **decode 99.0 t/s single-stream at 43.5k and 93.3 at 197k** (decode
|
||
is nearly window-free: −6 % for 4.5× depth), while **3 concurrent streams
|
||
give 113.9/43.3/10.9 t/s = 18.4 t/s aggregate** at 43.5k and 12.5 at 197k.
|
||
The slot count is paid in decode concurrency — a fused-step lottery, not a
|
||
fair share — which is exactly the axis the shape hunt was blind to (probe
|
||
leg B generates 24 tokens). Corpus caveat at the claim: the ladder's filler
|
||
is repetitive, acceptance 0.93–0.97 vs the 0.740 of the 2026-10-11 decode
|
||
table, so these rows are the baseline for this corpus, not a replacement.
|
||
- **Seam:** `probe-frank-archon-kv.py` gains **leg D** (N-stream decode, 512
|
||
tokens, per-stream + aggregate) so no future shape change can be priced on
|
||
prefill alone.
|
||
|
||
|
||
## 2026-10-10
|
||
- **Fleet service rename: the StarCraft names became the SERVICE names.**
|
||
The 2026-10-09 wire alias moved onto every repo artifact: units, serve
|
||
scripts, env files and their `FRANK_*` variables, deployer service keys,
|
||
container names, Prometheus/Grafana job names, bats seams, runbook
|
||
filenames and all historical prose. `frank-archon` (dankserver :7770),
|
||
`frank-dragoon` (dankclaw :7770), `frank-zealot` (dankclaw :7771). The
|
||
three former service names are fully retired: `git grep` over the tree
|
||
returns zero, rollback tags were renamed in lockstep (e.g. tag
|
||
`frank-archon-freetoken-pr587-20261005`), and the harness `models.yml`
|
||
lore matches. Host cutover = stop old unit + remove old container,
|
||
deploy the renamed service (the deployer relinks the unit under the new
|
||
name; `add_pending_services` auto-trashes the retired markers). The
|
||
retired `frank-medium`/`frank-small`/`frank-npu` names are historical
|
||
and stay as they are.
|
||
- **Known casualty:** the two vendored vLLM-fork patches under
|
||
`scripts/patches/` carry the old service names in context lines; the
|
||
sweep rewrote them, so those patches (retired eras, rebuildable from
|
||
the renamed tags) no longer apply byte-exact.
|
||
- **Executed live at `55f87fc`, all three gates green.** Per host: stop
|
||
old unit → trash its deploy-state marker + remove the unit symlink and
|
||
old container → `deploy.sh <host> --service <new-name>`. Result:
|
||
`frank-archon` active/enabled (container `frank-archon`, :7770),
|
||
`frank-dragoon` active/enabled (container `frank-dragoon-halogen`,
|
||
:7770), `frank-zealot` active/enabled (journal shows 87–93 t/s serving
|
||
compaction on :7771). `/health` 200 ×3; `/v1/models` reports the three
|
||
ids; host checkouts grep zero residue; e2e passed 32+1flaky, 32+1flaky,
|
||
33; Prometheus `frank-archon`/`frank-dragoon` up.
|
||
- **Pre-existing gap found by the verification, NOT a rename regression:**
|
||
`frank-zealot` `/metrics` answers 501 — llama.cpp gates the endpoint
|
||
behind `--metrics` (default disabled) and the serve script never passed
|
||
it, so `up{job="frank-secondary"}` had NO series at all in the last 24 h
|
||
(the zealot metrics + the queue-depth alerts on it have been dead since
|
||
the Qwen3.5 swap). Fix = one flag in `serve-frank-zealot.sh` + a ~1 min
|
||
zealot restart; not applied in the rename window.
|
||
|
||
## 2026-10-09
|
||
- **Live — five authentik human accounts provisioned (Dankclaw, `ak shell`
|
||
ORM, no repo objects):** kirby (pk 17), ry (18), nic (19), debo (20),
|
||
freya (21), each `<user>@franksdank.cloud` in the human-user template shape
|
||
(`is_active=True`, password via `set_password` on stdin only, groups
|
||
`Non-Admin Users` + `jellyfin_users` + `openwebui_users`). Verified through
|
||
the exact PasswordStage code path (`authenticate()` accepts each password and
|
||
rejects a wrong one) and `app_entitlements(open-webui)` → `Open WebUI Users`
|
||
for all five (identical to the ridley precedent). No `proxysession` rows
|
||
existed for them — groups were set before any login, so no claim purge was
|
||
due. Downstream: Jellyfin visibility rides the LDAP outpost cache (expect
|
||
lag); OpenWebUI mints their accounts as role `pending` on first login.
|
||
- `/serverAppConfigs-docs/Infrastructure/Authentik.md` — human-user template
|
||
census extended with pk 17–21.
|
||
- `/serverAppConfigs-docs/Dankclaw/OpenWebUI.md` — new SSO bullet: first login
|
||
lands as role `pending` (`ui.default_user_role=pending` in the config DB),
|
||
the role-flip SQL, and the global `access_grant` (`principal_id='*'`) fact
|
||
that makes the picker model visible to every `role='user'` account.
|
||
|
||
- **Live — the Protoss rename finished on the wire (all three endpoints).**
|
||
Served model ids moved `frank-archon` → **frank-archon**
|
||
(`FRANK_ARCHON_SERVED_NAME`), `frank-dragoon` → **frank-dragoon**
|
||
(`HALOGEN_MODEL_ID`), `frank-zealot` → **frank-zealot** (`ALIAS`). The
|
||
services, units, scripts, env files and runbook pages keep the old names —
|
||
container/unit identity stays stable, only the alias moved. Deploys in order:
|
||
`dankclaw --service frank-zealot` (e2e gate 33/33 green), `dankclaw
|
||
--service frank-dragoon` (healthy 30 s), `dankserver --service frank-archon`
|
||
(healthy 102 s), `dankclaw --service homepage`. Verified on `/v1/models`,
|
||
not on chat echoes: `frank-zealot` (`n_ctx 200704`, Q6_K), `frank-dragoon`
|
||
(`context_length 262144`), `frank-archon`.
|
||
- **Live — OpenWebUI re-keyed in the same window** (snapshot
|
||
`webui.db.bak-20261009`): `model` row (`frank-zealot` → `frank-zealot`,
|
||
params + `meta.description` preserved), the `access_grant` row
|
||
(`model`/`frank-zealot`/`user`/`*`/`read`), and `openai.api_configs`
|
||
`model_ids` on all three backends; `systemctl --user restart
|
||
open-webui.service`. Family-account proof, in a real browser as `qa-e2e`:
|
||
the welcome screen selects `frank-zealot` and
|
||
`POST /api/v1/chat/completions {"model":"frank-zealot"}` answers
|
||
`zealous` with `"model":"frank-zealot"`. A `GET /api/v1/models/list`
|
||
probe returns `{"items":[],"total":0}` for that account while the picker
|
||
is plainly populated — the API census is not the acceptance test, the UI
|
||
snapshot is.
|
||
- **Live — Hermes re-keyed** (`config.yaml` `frank-dragoon` → `frank-dragoon`
|
||
in the provider block, `model.default` and the three `dankclaw-large`
|
||
auxiliary routes; backup `config.yaml.bak-20261009`; `hermes.service` +
|
||
`hermes-webui.service` restarted, gateway log clean, `/health` ok) and the
|
||
repo baseline `dankclaw/hermes/config.yaml.example` synced to the live
|
||
shape (context 262144). **Flagged, not fixed:** Hermes still routes its
|
||
`dankserver-small` auxiliary tasks at the id `frank-medium` on
|
||
`192.168.0.235:7770`, an endpoint that has served `frank-archon` since
|
||
2026-09-20 — those routes were dead before this rename and remain dead;
|
||
they need a routing decision (archon capacity), not a rename.
|
||
- **Deployer blind spot found by accident:** every
|
||
`deploy.sh dankserver --service frank-archon` since 2026-10-08 04:09
|
||
restarted the **staging worktree**, not the deploy checkout — a campaign
|
||
drop-in `frank-archon.service.d/bench.conf` overrode `Environment=` and
|
||
`ExecStart`. The health gate went green each time. Verified parity before
|
||
removing it (serve script + unit byte-identical, env identical except the
|
||
alias, staging HEAD an ancestor of `main`), moved the drop-in to
|
||
`/home/dank/.AGENTTRASH/20261009-frank-archon-bench-dropin/`, and
|
||
restarted: `/v1/models` then answered `frank-archon`. Post-deploy
|
||
behavioural proof is now part of the runbook.
|
||
- Repo: seams re-pinned (`frank_zealot`/`frank_dragoon`/`frank_archon`
|
||
unit bats), fleet harnesses retargeted (`bench-probe`, `deep-probe`,
|
||
`calib-push`, `fleet-probes-swift15`, `autoresearch*`,
|
||
`bench-threads-batch-http`, `bench-decode`), `shared/homepage/docs.html`
|
||
now teaches the ids it ships (curl/opencode/Pi send `frank-zealot`),
|
||
`AGENTS.md`, [[OMP Harness]], [[Homepage]], [[OpenWebUI]] and the three
|
||
endpoint runbooks updated. Suites: `./tests/run.sh` 366 green.
|
||
- **Transcript secret incident:** dumping `podman exec hermes-webui env`
|
||
printed the Porkbun API key + secret key into this session's transcript —
|
||
same class as the 2026-09-16 HF-token-in-`ps` leak. Not committed
|
||
anywhere; rotation is Frank's call.
|
||
|
||
## 2026-10-08
|
||
- **frank-archon (Dankserver): Swift1.5-Qwen3.8-Flash-Next GPTQ w4 —
|
||
the radiance quantization campaign.** ukisai's BF16 checkpoint (335.3
|
||
GiB, 397/397 sha256-verified) → `qwen4exp-swift15-w4dn8-i8-hc8m.rad`
|
||
(~114 GiB): all 49,152 experts GPTQ w4 (act-order, damp 0.05) calibrated
|
||
on 10,000,000 live-model tokens, 24 whole bf16-protected experts per
|
||
layer derived from the calibration grams' energy share, int8 KV, fp8 HC
|
||
mixtures. Engine pins: **1.0.13** for serve/convert/KLD, **1.2.0** only
|
||
for calibration (the sole pin exposing `RADIANCE_CALIB_TOKENS`; the
|
||
RADGRAM2 gram format is byte-identical across pins, so grams cross the
|
||
boundary by design). Chain scripts in `scripts/`:
|
||
`make-calib-corpus.py`, `run-swift15-bf16.sh` (bf16 reference: KLD record
|
||
+ calib tap), `calib-push.py`, `derive-protected-experts.py`,
|
||
`apply-protected-cast.sh`, `convert-swift15-w4dn8.sh`,
|
||
`kld-gate-swift15.sh`, `fleet-probes-swift15.sh`.
|
||
- **Engine laws learned (measured, not inferred).** (1) The qwen4exp_fp8
|
||
plugin declares **one encoding per expert table**: a recipe may NOT split
|
||
`ffn_gate_up_exps` (w4) from `ffn_down_exps` (block-fp8) — the serve
|
||
kernel is a single w4nl64a8h table for the whole expert set; the only
|
||
legal beside-table content is whole bf16 experts (the PROTECTED section).
|
||
(2) w4 expert tables stream from the file tier **in place only at
|
||
MNBT ≤ 2048**; at MNBT 32768 the engine demands a widened form and the
|
||
disk tier refuses (`DID NOT FIT … stored form is not the container's`).
|
||
(3) The calibration tap lives in the architecture plugin
|
||
(`/opt/radiance/share/radiance/architectures/qwen4exp_fp8.so`), not the
|
||
binary — `grep -ac` the `.so` before declaring a flag missing.
|
||
(4) Disk-tier boots need ~9.6 GiB of weight buffers + arena charged to
|
||
the VRAM weights slab; explicit `--vram-weights-mib`/`--vram-kv-mib`
|
||
pairs replaced the fp8-era EVC ratio.
|
||
- **Throughput gate PASSED with 4.5× margin** (campaign boot, TP=2, int8
|
||
KV, MTP 3): decode **136.0 t/s median** (122–173 across probes) at
|
||
44.5k-token depth vs the 30.8 t/s bar; MTP acceptance 0.725; warm TTFT
|
||
9.45 s; prefill 6.4k t/s. The fp8-native control on 1.0.13 measured
|
||
30.08 t/s decode, and the 1.2.0 engine re-baseline was REJECTED
|
||
(29.21 t/s, acceptance 0.7123 < 0.7549) — serve stays on 1.0.13.
|
||
- **Deep-output audit:** a 44.5k-token verbatim-recall probe enumerated all
|
||
48 planted documents in order — the DDR5-corruption signature is absent.
|
||
- Fleet probes (`scripts/fleet-probes-swift15.sh`): tools, reasoning
|
||
levels x4, reasoning_content, response_format, vision — all PASS.
|
||
KLD gate vs the BF16 reference: recorded on 1.0.13 (50,235 positions,
|
||
197 prompts), scored on the same pin at the w4-legal shape.
|
||
Verdict: see runbook [[frank-archon-radiance-swift15-w4dn8]].
|
||
|
||
## 2026-10-06
|
||
- **frank-zealot (Dankclaw): ubatch 2048 — the second rung (llama.cpp,
|
||
RTX 3090).** "set ubatch to 2k" → commit d8da7a9: `UBATCH_SIZE`
|
||
1024 → 2048 at the 2×114,688 shape. **Cost 746 MiB**, not the 616 MiB
|
||
the same rung measured on the MiMo shape on 2026-10-05 (a rung's price
|
||
is shape-dependent; the 616 figure was not transferable). Boot:
|
||
**23,120 used / 1,005 MiB free**; a 512×512 vision probe then took
|
||
~240 MiB **and kept it** — settled **763 MiB free**, the encoder's
|
||
buffers staying resident across requests on this build, unlike the
|
||
evening boot where the 34 MiB transient returned. Decode 52.0 t/s on
|
||
the shared prompt (58.2 at the 1024 rung) with acceptance 34.2 % vs
|
||
42.8 %: ubatch does not move decode, so the gap is temp-1.0
|
||
acceptance noise, not a regression. **Prefill 1,209 t/s @ 3,760
|
||
tokens** — the first prefill number this endpoint has recorded, and
|
||
now the baseline future rungs compare against. `BATCH_SIZE` stays
|
||
4096, so the seam's batch ≥ ubatch guard holds; **2048 is the ceiling
|
||
for this shape** (4096 = no micro-split doubles the buffer growth
|
||
again against 0.7–1.0 GiB of headroom). e2e gate 33/33 green.
|
||
- **frank-zealot (Dankclaw): RENAMED from frank-small, then reshaped
|
||
to 2×114,688 / ubatch 1024 (llama.cpp, RTX 3090).** The rename (commit
|
||
87721ab, 87 files: serve trio, unit, bats seam, 9 runbooks, identifiers
|
||
`FRANK_ZEALOT_*`/`llm_zealot`/`deploy_llm_zealot`; historical
|
||
refs to the OLD FreeToken/CUDA frank-small and to the Dankserver
|
||
service that became frank-medium stay frank-small, as does
|
||
`freetoken-frank-small.md`). Live unit transition by hand before the
|
||
deploy: `systemctl stop frank-small` + `systemctl disable frank-small`
|
||
(a relinked unit — disable removes only its stale symlink) →
|
||
`deploy.sh dankclaw --service frank-zealot`. The reshape (commit
|
||
f2575a1, Frank's order "lower max context to 112k so we can increase
|
||
ubatch to at least 1k"): `CTX_SIZE 262144→229376` (per-slot
|
||
131,072→114,688), `UBATCH_SIZE 512→1024` → **22,374 used / 1,751 MiB
|
||
free** (the per-slot cut frees ~1.05 GiB of KV; ubatch 1024 spends
|
||
part, netting ~1.25 GiB back over the 489 MiB fit), decode 58.2 t/s
|
||
recipe / 66.9 temp 0 (acceptance 42.8–54.4 % — the drop vs the prior
|
||
boot's 63.3/72.9 is confounded by prompt text, not a shape verdict).
|
||
Harness retargeted: models.yml (id/name/compactionModels,
|
||
`contextWindow: 114688`) + config.yml modelRoles ×14. Runbook renamed:
|
||
`serverAppConfigs-docs/LLM/frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md`.
|
||
- **frank-zealot (Dankclaw): two Swift-1.5-27B reshapes (llama.cpp, RTX 3090).**
|
||
Morning: Frank's order "drop mtp, skip ngram, ubatch 512, two 128k
|
||
contexts" → 2×131,072 no-spec on the IQ3_S-mtp (commit 812d9eb):
|
||
22,728 used / 1,397 MiB free (the tight fit he ordered; the drafter's
|
||
~0.4 GiB + ubatch ~0.3 GiB bought the second 4,352 MiB KV slot), decode
|
||
48.0 t/s, vision `Red`. Evening: swap to
|
||
`ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF`
|
||
RCO-IQ3_XXS-mtp (10,442,827,392 B, sha256 `e13d3e74…`; 1.56 GiB lighter)
|
||
+ the publisher's own `mmproj-BF16.gguf`, MTP back on (draft-mtp depth
|
||
3) → commit da36fa3: **23,636 used / 489 MiB free** (the tightest fit in
|
||
the endpoint's history — the draft ctx measured ~2.4 GiB, built against
|
||
the target's full 262,144 context, not the 3-token proposal depth; the
|
||
lighter quant funds it), decode 63.3 t/s recipe / 72.9 temp 0 (acceptance
|
||
49.8–62.4 % vs 48.0 no-spec), vision `Red` at a 128-token budget (the
|
||
uncensored model thinks more — 64 tokens is all `reasoning_content`).
|
||
ngram-mod still skipped (recurrent-state rollback law on the qwen35
|
||
hybrid). Runbook renamed + rewritten:
|
||
`serverAppConfigs-docs/LLM/frank-zealot-llama-swift15-unc-iq3xxs-mtp-114688.md`.
|
||
- **frank-dragoon (Dankclaw): halogen 27B-dense → halogen-flash-server
|
||
0.16.2 v2 Swift1.5 ABLITERATED (Strix Halo iGPU, port 7770).** Cutover
|
||
commit 31b1ec3 (03:07:40): the v2 `.hgn` payload loads ONLY in the
|
||
flash-server line, so the engine moved with the model. Abliterated base
|
||
(refusals gone), vision back (the v2 tower loads beside the checkpoint),
|
||
MTP head inside the checkpoint, 2×200,192 slots, `HALOGEN_SPEC_ADAPT=0`.
|
||
Runbook: `serverAppConfigs-docs/LLM/frank-dragoon-flash-server-v2-swift15.md`.
|
||
**Incident (03:09:39–03:11:42):** the deploy started the unit while host
|
||
`MemAvailable` was ~74.7 GiB, 3.34 GiB short of the engine's pin-gate
|
||
requirement (62.11 GiB checkpoint + 16 GiB floor = 78.11 GiB); it
|
||
crash-looped 8 boots, each dying in ~5 s at the pin gate, and
|
||
self-resolved in ~2 min when the dying 27B engine freed RAM (boot #9
|
||
pinned clean at 88.4 GiB available). **Latent risk + fix:** the host was
|
||
left with only ~1.8 GiB `MemAvailable`, so a restart would crash-loop
|
||
again; `serve-frank-dragoon.sh` now has a host-RAM preflight that checks
|
||
`MemAvailable` against the pin gate BEFORE `podman run` and WAITS up to
|
||
`FRANK_DRAGOON_RAM_WAIT` (600 s) for the RAM to clear, refusing loudly if
|
||
it never does (the 16 GiB floor is the engine's, not
|
||
`HALOGEN_HOST_RESERVE_GIB`).
|
||
|
||
## 2026-10-05
|
||
- **frank-archon: tcclaviger vLLM fork → FreeToken (PR #587 ROCm) on
|
||
dankserver.** Built FreeToken at PR head `77a7201` on the host into the
|
||
local image `frank-archon-freetoken:pr587-77a7201`
|
||
(digest-pinned rocm/pytorch rocm7.14 torch-2.11.0 base; two build laws
|
||
bought with failures: the pip-ROCm wheel needs a `libhiprtc.so` dev
|
||
symlink for the PR's `_cpu_moe` link, and a stale `docker login` at
|
||
`~/.docker/config.json` poisons podman's ANONYMOUS pulls). Trio rewritten
|
||
through git (env/serve/unit + seam + deployer): ONE R9700, experts in
|
||
pinned host RAM behind a 0.10 GPU LRU, PLE 47.7 GiB pinned, KV pool
|
||
299,968 tokens (multiple-of-64 — the `qsa_sparse` auto backend overrides
|
||
page size to 64; 300,000 was refused), 2 slots, parsers AUTO (thinking →
|
||
`reasoning_content`; models.yml flipped accordingly). Container renamed
|
||
to the unit name (Frank's call; `frank-archon-qwen` → legacy list);
|
||
deployer image gate → `podman image exists` (local builds carry no
|
||
RepoDigests) and health gate reads the BODY — FreeToken 200s with
|
||
`{"status":"loading"}` mid-load, which false-greened a deploy at 6 s
|
||
(vLLM 503'd; the bare curl was honest there). Verified: FLEET-OK
|
||
completion, reasoning split (17×24 → "408"), native `tool_calls`, vision
|
||
"Red" probe, boot 2.3 min cold / 92 s warm, decode 17.1 tok/s
|
||
single-stream (vLLM was ~30 on two cards; the KV contract caps the
|
||
single-card LRU; TP=2 exists but the pynccl JIT can't compile for
|
||
gfx1201 — probed, signature in the runbook). `/metrics` is a 404 — the
|
||
Dankclaw Prometheus llm-fleet scrape of :7770 reads down until
|
||
repointed. Rollback: tag `frank-archon-vllm-shapeG-20261003`.
|
||
Runbook: `serverAppConfigs-docs/LLM/frank-archon-freetoken-pr587.md`.
|
||
- **Skill library distilled into the vault (waves 1–7).** All ~173 managed
|
||
skills read in full and merged, one polished reference page per
|
||
skill-cluster:
|
||
- **LLM cross-cutting (16 pages, `LLM/`):** GGUF Compatibility,
|
||
Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and
|
||
Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget,
|
||
vLLM Boot Forensics and Config Interrogation, vLLM Porting and Compile
|
||
Triage, LLM Bench Windows and Measurement, LLM Performance Diagnosis,
|
||
Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split
|
||
Triage, FreeToken Builds, plus deltas folded into the retired runbooks
|
||
(GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline,
|
||
FLM pack registration, ik hybrid envelope, frank-medium split
|
||
rebalance, frank-archon llama.cpp GGUF preflight, batch-ubatch split
|
||
section, bench-window D-state/stale-read notes, second-endpoint
|
||
bring-up in Model Swap).
|
||
- **Architecture (3):** Stale Signal Audit, Quadlet Lifecycle, Deploy
|
||
State Triage.
|
||
- **Operations (6 + index):** Remote Job Runner, Payload and Disk
|
||
Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit
|
||
and Posture, SSH and Remote Access.
|
||
- **Infrastructure (2 + updates):** Observability Provisioning, Homepage
|
||
Status Probes; updates to Authentik (guard-chain flip recipe, `ak
|
||
shell` mechanics, user-provisioning playbook, outpost netns probe),
|
||
Anubis (guard-chain layer diagnosis), LAN-DNS (per-client resolver
|
||
drop-in + browser DoH bypass, Porkbun zone ops), Swap Configuration
|
||
(filesystem/rename caveats), OpenWebUI (introspection, verification
|
||
battery, ComfyUI imagegen), Hermes (config-home drift, live config
|
||
edits, auxiliary routing, password rotation — the pre-existing page was
|
||
preserved and extended).
|
||
- **Host pages:** FrankBook-Pro ARR-Stack filled with the API-version
|
||
split, schema-first probing, auth-posture method, auth-config trap, and
|
||
the qBittorrent E2E verification procedure added to [[FrankBook-Pro/qbittorrent]].
|
||
- **Desktop/Agent/Research:** Noctalia and Niri (config layers, remote
|
||
IPC, palette recolor, offline template render, rotation pinning, iCX3
|
||
RGB), Agent/ (OMP Harness, Agent Docs and Edits, Command Center
|
||
Dotfiles, index), PICO-8 Asset Standards.
|
||
- Index updated: `LLM/index.md` gained a Cross-Cutting References
|
||
section; Operations/ and Agent/ got index pages. AGENTS.md now points
|
||
at the new sections. Link checker + full test suite green.
|
||
- **e2e: the FrankBook-Pro leg scoped to its own trigger-gated Playwright
|
||
project.** The `11-admin` walk included the six `*arr`/qBittorrent admin
|
||
links that terminate on 192.168.0.112, so every deploy gate died there when
|
||
the Mac mini slept (Caddy 502 on the five vhosts, connection timeout on the
|
||
raw Sonarr link). Split: those links moved to
|
||
`tests/e2e/specs/19-frankbook.spec.ts` under a `frankbook` project;
|
||
`tests/e2e/run.sh` excludes it from the default project set, and
|
||
`scripts/deploy.sh` computes a trigger from the same BASE..HEAD range the
|
||
plan uses (the spec, `frankbook-pro/**`, `shared/homepage/services.json`,
|
||
the `dankclaw/Caddyfile` arr vhosts) and sets `E2E_INCLUDE_FRANKBOOK=1`.
|
||
Verified live with the host asleep: full suite 33/33 in 1.1 min, the admin
|
||
walk back to 5.4 s; `--project=frankbook --list` proves the selection.
|
||
Live pass of the frankbook walk itself is deferred to the next deploy with
|
||
an awake host.
|
||
- **Catch while verifying: the 17:47 Ornith-1.5-9B swap (`77decb5`) had
|
||
left `frank_zealot_unit.bats` and the deployer's payload gate stale** —
|
||
the seam still expected MiMo-Q6K names and `deploy-remote.sh` gated
|
||
`LLM_SMALL_MODEL_FILE`/`LLM_SMALL_MMPROJ` on files the unit no longer
|
||
loads (the anti-drift seam doing its job). Both repointed to the
|
||
Ornith-Q8 paths; full `./tests/run.sh` back to green (366 tests).
|
||
- **frank-zealot: compute-buffer pair doubled to 4096/2048, and the prefill win
|
||
measured rather than assumed.** Frank asked for the batch and ubatch doubled.
|
||
The blocker was on record: the Q5_K_M predecessor documented **"ubatch 2048
|
||
failed the scratch boot"**, and this shape carries 616 MiB less free space
|
||
than that predecessor did.
|
||
- The A/B used the service itself as the bench (one card, one server — no
|
||
scratch port): the old rung was re-booted through a transient
|
||
`FRANK_SMALL_ENV_FILE` systemd drop-in, and it reproduced the 2026-10-03
|
||
boot **byte-identically** (21,276 used / 2,849 free), which is what makes
|
||
ubatch the only variable. Repo never saw a fake revert commit; scaffolding
|
||
staged to `~/.AGENTTRASH/20261005-frank-zealot-ubatch-ab/`.
|
||
- **At 8,928 prompt tokens: prefill 3,972/4,077 -> 4,207/4,212 t/s, +4.6 %,
|
||
for 616 MiB (free 2,849 -> 2,279).** Decode flat (91.9–93.0 vs 93.8–94.3 —
|
||
noise). The control that proves it is geometry and not drift: a 279-token
|
||
prompt sits below *both* micro-batches, so it cannot see the knob, and it
|
||
did not move (2,676 vs 2,658–2,811).
|
||
- **The predecessor's boot failure did not reproduce** — that was a shape
|
||
observation, not a law; this shape boots 2048 clean. And the fleet lesson
|
||
"**`--ubatch-size` has an optimum**" is now scoped to what it actually is:
|
||
true on the 2x R9700 layer split (peak 512, −10 % by 2048), false on the
|
||
CUDA 3090 where the curve climbs monotonically to 2048. AGENTS.md corrected
|
||
— the curve is backend-specific and must never be carried across backends.
|
||
- The seam stopped hard-coding a rung: `serve_argv` now carries `BATCH_SIZE`
|
||
and `UBATCH_SIZE` the way it already carried `CTX_SIZE`, and the pair test
|
||
asserts the argv follows the shipped env plus the `ubatch <= batch`
|
||
invariant. Proven live both ways — retuning the env to 8192/4096 keeps it
|
||
green, inverting to 2048/4096 turns it red. (The old assertions only ever
|
||
exercised the serve-script *defaults*, because the test env never carried
|
||
these two keys.)
|
||
- `4096/1536` is the documented middle rung; 2,279 MiB free sits at the bottom
|
||
edge of the 2–3 GiB floor. Deployed `e5893dc`, e2e gate PASSED (2 mini specs
|
||
flaky-then-passed once the sleeping FrankBook-Pro woke mid-run — WoL took a
|
||
second burst and ~90 s). Docs:
|
||
[[frank-zealot-llama-mimo-q6k-2x200704]] (*Compute-buffer geometry*).
|
||
|
||
## 2026-10-03
|
||
- **frank-archon: shape F then shape G — MNBT 2048, the CUDA-graph plan term
|
||
pinned to the measured capture, KV forced to 400,700 tokens, and UTIL 1.0
|
||
proven UNBOOTABLE.** Frank asked what the batch was with GPUs showing open
|
||
space; the audit found the arena is `budget - non-expert - KV reserve -
|
||
graphs - overhead` where the only deploy-reachable terms are the budget and
|
||
the graph estimate (registry KV price 9,190 B/token and the 5.25 GiB
|
||
overhead are image constants; the ~3.5 GiB of driver-free VRAM was
|
||
estimator slack, not allocatable).
|
||
- Shape F (c682c63): MNBT 1024 -> 2048 (arena-free — the registry prices
|
||
overhead flat up to its nbt 4096 anchor and the fitted 5.25 was fitted AT
|
||
4096), util 0.98 -> 0.982, graph term 0.55 -> 0.32 GiB (256 base +
|
||
12/graph vs the fork's 256 + 52/graph). Measured: arena 17.52 GiB /
|
||
7,653 slots, budget 31.29.
|
||
- Shape G v1 (0a4ff30) tried util 1.0: **both workers refused init before a
|
||
single weight loaded** — `ValueError: Free memory on device cuda:N
|
||
(31.29/31.86 GiB ...)`, NRestarts=2 crash loop, zero payload progress.
|
||
The CLI field is `Field(gt=0, le=1)`, but the worker demands free >
|
||
budget, so budget == total is a trap, not a knife-edge. Frank: ".982 is
|
||
the law, never 1.00" — now with a second proof.
|
||
- Shape G v2 (c67a2ae, deployed green incl. e2e 33/33): util back to
|
||
0.982, graph base to 0 (the shape F boot reported a NEGATIVE -0.06 GiB
|
||
capture delta, so the 256 MiB base was fiction; 0 + 12 MiB/graph still
|
||
prices a future 18-graph spec-decode shape at 0.21 GiB), KV_BYTES
|
||
3.15e9 -> 3.1665e9 = Frank's 400,700-token ask.
|
||
- MEASURED live: arena **17.77 GiB / 7,762 slots** (+240 over shape E),
|
||
KV **401,408 tokens / 2.00x concurrency** — the first shape holding two
|
||
full-length conversations with real spare; capture measured 0.31/0.10
|
||
GiB against the 0.07 charge (under-charged, boots and holds); driver
|
||
used 30.97 of 31.86 GiB -> **0.89 GiB real free per card**, held through
|
||
a live 4,096-token generation (~38-39 tok/s single-request wall-incl.,
|
||
+13% vs the 33.99 shape-E baseline; single sample, not a bench).
|
||
- Verification seam rewritten to assert the shape by its OUTPUTS (util at
|
||
the 0.982 law, graph charge below the analytic default and per-graph
|
||
positive, KV pool >= 400,700 tokens at the measured 7,892 B/token).
|
||
Runbook: `serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`.
|
||
|
||
## 2026-10-01
|
||
- **frank-archon rolled BACK to Qwen3.8-Flash-Next-Uncensored-FP8 + MTP4;
|
||
GLM/DFlash2 declared failed and purged.** The DFlash2 drafter booted and
|
||
served (window 6j, 65,536 ctx, `glm53-flash` on :7773, both TP ranks
|
||
captured FULL dflash2 cuda graphs — five patches deep: EAGLE3 wrapper
|
||
interface v5, KV unifier MLA-multiple, kpool tailpin, glm5 KV pool with
|
||
draft+hidden groups, registry method-signal), but **acceptance pinned at
|
||
1.00-1.38 on every text class INCLUDING verbatim repetition** — a mis-wired
|
||
aux-hidden-state capture, not a weak drafter. At ~1.2 acceptance the
|
||
8-token verify tax put decode at 1.6-1.8 tok/s vs the 9.9-10.0 baseline
|
||
(5x slower). Frank's independently reported 18 tok/s / ~370 prefill did
|
||
not originate from this engine (APIServer access log: every completion
|
||
POST was host-local; his workstation appears once, a /v1/models GET).
|
||
Restored the verified 37.4-40.6 tok/s arm from the `2339fae` trio
|
||
(200k x 3 seqs, fp8 KV, prefetch off); GLM payload (170 GiB) + drafter
|
||
(2.2 GiB) + scratch staged to `/home/dank/.AGENTTRASH/20261001-glm-purge/`.
|
||
KV-cache patches 02/03/04/04b survive in `scripts/patches/` (generic
|
||
unifier/tailpin/grouping/registry fixes, apply-verified).
|
||
|
||
## 2026-09-30
|
||
- **frank-archon cut over to GLM-5.3-Flash W4A16 int4 — it serves, but NOT at
|
||
the 35 tok/s target, and the shortfall is measured rather than inferred.**
|
||
- Payload `Intel/GLM-5.3-Flash-W4A16-AutoRound` (AutoRound symmetric int4
|
||
group 128, `auto_round:auto_gptq` packing, ~170 GiB, 34 shards + 2 extras,
|
||
113,074 tensors). It is the only W4A16 checkpoint this fork can serve WITH
|
||
expert offload: its `quant_method: auto-round` is overridden to `inc`, which
|
||
routes the MoE to libr4d's grouped W4A16 GEMM (`R4dW4a16MoEMethod`). The
|
||
compressed-tensors W4A16 build of the same model (`dealignai/…-CYBERSECURITY-W4A16`,
|
||
182 GiB, fetched earlier) resolves to `TritonWNA16Experts`, which has NO
|
||
offload recipe — and 170 GiB of experts cannot fit 64 GiB of VRAM, so it
|
||
refused to serve at all. That checkpoint was purged with the other retired
|
||
payloads.
|
||
- Shape: tcclaviger RDNA4 vLLM fork (same digest-pinned image as the qwen
|
||
cutover), TP=2, 131,072 ctx x 2 seqs, bf16 KV, util 0.98, NBATCH 2048,
|
||
`--enable-expert-offload --expert-precise-memory`, prefix caching, vision +
|
||
`glm47` tool and reasoning parsers, MTP OFF, prefetch OFF.
|
||
- Measured on the deployed engine: **decode 9.9-10.0 tok/s, prefill 918 tok/s**
|
||
(warm, 2.6k prompt), KV 138,561 tokens; arena 14.51 GiB = 2,383 slots per
|
||
rank over 12,096 host-backed experts (73.64 GiB pinned per rank).
|
||
- **The wall — and a corrected one.** The first account of the shortfall
|
||
bounded the step with throughput arithmetic (a step moves 0.82 GiB of misses
|
||
in ~100 ms = "8.2 GB/s effective against a 22.74 GB/s link", needing a 0.89
|
||
hit rate for 35 tok/s). **A torch-profiler kernel-time window falsified it**
|
||
(496 decode tokens, rank0 trace parsed for device-activity time): 46.18 s of
|
||
kernel time = ~93 ms/token, of which **44 % is pointwise glue** —
|
||
`vectorized_elementwise_kernel` alone 16.07 s over **708,500 launches (1,428
|
||
per token)**, plus 3.11 s of `elementwise_kernel_manual_unroll` — against
|
||
**1.05 % for `r4d_moe_lru_gather_kernel`**. The MoE GEMM is 10 %, the split-K
|
||
GEMM 6.4 %, the router/top-k 8.9 %, the mHC kernels 6.5 %, and the inductor
|
||
fusions that DO happen 6 %. The real cause is that the fork compiles its own
|
||
Qwen trunk (`@support_torch_compile` in `models/qwen4_exp/amd/model.py`) and
|
||
**not** its GLM trunk, so every pointwise op is a separate eager kernel; the
|
||
engine says so at boot ("...provides neither a compiled submodule nor
|
||
breakable CUDA graphs. Overriding to FULL_DECODE_ONLY."). Growing the arena
|
||
13.08 -> 14.51 GiB still changed nothing, consistent with a 1 % gather.
|
||
- **Fork work staged (compile changelist, WIP — does not boot yet).**
|
||
`scripts/patches/glm53-compile-01-model-decorator.patch`,
|
||
`-02-trace-safe-probes.patch`, `-03-fullgraph-optout.patch`, each with its own
|
||
evidence header and base/patched sha256. Boots established, in order: (1)
|
||
`functools.lru_cache` does NOT make a probe trace-safe — dynamo re-traces the
|
||
wrapped body even when warm, so `r4d_mhc_supported -> r4d_mhc_enabled ->
|
||
available -> os.path.exists` breaks the graph (a plain module-global read
|
||
compiles; verified in-image); (2) `suppress_errors` does NOT cover graph
|
||
breaks, only `fullgraph=False` does; (3) the GLM trunk can never satisfy
|
||
`fullgraph` because its indexer metadata syncs to host
|
||
(`glm5next/amd/sparse_indexer.py:263,419`), and AOT compile refuses breaks
|
||
outright. With all three the trunk **traces** (`Dynamo bytecode transform
|
||
time: 1.08 s`) and then dies on `KeyError: 'in_proj_qkvbfg_a'` in the
|
||
mHC/KDA path — a latent bug the compile exposes. That is the remaining
|
||
blocker.
|
||
- **No new HIP kernels are possible** (the image ships `r4d.so` with no
|
||
sources), which binary-blocks the second-largest lever: the fused router is
|
||
gated `num_experts in {128, 256, 512}` and only
|
||
`r4d_moe_route_kernel<128|256|512>` instantiations exist, while GLM has 288
|
||
experts. `--kv-cache-dtype fp8` buys no capacity either (the plan's KV
|
||
reserve is dtype-blind), only removing gfx12's only MLA decode path.
|
||
- **DFlash2 for GLM is a target-side port, not a drafter port.**
|
||
`incoai/GLM-5.3-Flash-DFlash2` declares `architectures: ["DFlash2DraftModel"]`
|
||
and `model_type: "qwen3"` (5 dense layers, sliding attention, block 8,
|
||
`target_layer_ids [5, 14, 24, 33, 42]` of 45) and the fork **already maps it**
|
||
(`registry.py:636` -> `qwen3_dflash2.DFlash2Qwen3ForCausalLM`; spec method
|
||
`"dflash"`). What `glm5next` lacks is the target side: aux-hidden-state
|
||
capture (`set_aux_hidden_state_layers` / per-layer capture, as `deepseek_v2.py`
|
||
demonstrates). No speedup is budgeted: a verification pass pays pointwise cost
|
||
proportional to its token count.
|
||
- **DFlash2 tried, and the gap was exactly that one interface — now closed.**
|
||
Payload fetched and hash-verified (`/home/dank/models/GLM-5.3-Flash-DFlash2`,
|
||
2,342,169,800 B, sha256 `b038e1d9…` = the Hub's LFS oid). Booting with
|
||
`--speculative-config '{"method":"dflash","model":"/draft","num_speculative_tokens":8}'`
|
||
accepted the config and the drafter and died at worker init with
|
||
`RuntimeError: Model does not support EAGLE3 interface`. The port
|
||
(`scripts/patches/glm53-dflash-01-aux-hidden-states.patch`) took FIVE versions —
|
||
hand-written methods -> Protocol base (MRO kill at class creation) -> attrs on the
|
||
text model -> attrs + methods on the text model -> **v5: attrs + methods on BOTH
|
||
wrappers** (the runner checks the LOADED instance, which for GLM is the multimodal
|
||
`Glm5NextForConditionalGeneration` holding the text model as `self.language_model`;
|
||
`pixtral.py:498` is the reference delegation pattern). v5 CLEARS the gate: both TP
|
||
workers log `Using Eagle3 auxiliary layers from config: (6, 15, 25, 34, …)` and the
|
||
DFlash2 drafter resolves + loads. Getting past that point was then interrupted twice
|
||
by experiment-window RAM contention (see the window discipline below), so the
|
||
drafter-serving end state was still unproven when this entry was written.
|
||
- **Compile work: one confirmed blocker, and a launcher bug that misled four boots.**
|
||
With breaks genuinely tolerated (`fullgraph=False`, AOT off) the trunk traces and
|
||
dies on the dynamo name-map `KeyError: 'in_proj_qkvbfg_a'`. Four later boots
|
||
reported *different* fatal breaks — `Unsupported: Read uninitialized cell` at
|
||
`mla.py:208` (the fused-norm JIT kernel), then `Unsupported method call` in the
|
||
indexer's in-forward workspace acquisition
|
||
(`sparse_indexer.py:132 -> workspace.py:135 get_simultaneous / :155
|
||
_ensure_workspace_size`), then `Skip calling torch.compiler.disable()'d function`
|
||
— and the cause was **my own scratch launcher**: its compile branches were
|
||
restructured and the `-e VLLM_COMPILE_FULLGRAPH=0` line was dropped, so those
|
||
boots ran `fullgraph=True`, where *every* break is an error and the first one
|
||
wins. I first blamed `TORCH_LOGS=graph_breaks` for the difference; that was wrong
|
||
(the flag only logs). The moving reports are still useful — they are the break
|
||
list — and patch 04 demonstrably removes the first of them (the fused-norm cell),
|
||
patch 05 makes the workspace acquisition compiler-opaque.
|
||
`scripts/patches/glm53-compile-04-fused-qkv-rmsnorm-optout.patch` takes GLM off
|
||
the fused-norm kernel (`R4D_GLM_FUSE_QKV_RMSNORM=0`, a constructor flag glm5next
|
||
sets at `attention.py:583`).
|
||
- **Diagnostics built for this (reusable).** `TORCH_LOGS=graph_breaks` +
|
||
`VLLM_USE_AOT_COMPILE=0` names each break with its user stack (AOT compile
|
||
hard-refuses breaks outright); `/home/dank/dynamo-probe` on `PYTHONPATH` with
|
||
`VLLM_DYNAMO_DUMP_ATTRS=1` prints the dynamo name-map miss detail (node target,
|
||
near-miss keys, whether another live `OutputGraph` holds the key). Two gotchas
|
||
recorded: a `.pth` on `PYTHONPATH` is NOT executed (use `sitecustomize.py`,
|
||
chaining the image's apport hook), and the scratch launcher used to reset
|
||
`ENVARGS` (now `${ENVARGS:-}`), which silently dropped env vars and wasted a boot.
|
||
All four patch artifacts are now genuine `diff -u` patches, verified to apply to
|
||
the pinned image and to reproduce their recorded sha256.
|
||
- Measured and rejected: **MTP4** (the head loads, arch resolves to
|
||
`Glm5NextMTPModel`, acceptance is excellent at mean length 3.76-4.90 — but
|
||
decode was unchanged (9.6-11.2 vs 10.0) while prefill COLLAPSED 918 -> 255.
|
||
The profiler window above explains it: the per-token cost is dominated by
|
||
pointwise glue that scales with the token count, so verifying a draft batch
|
||
pays ~N x that glue for the same N tokens of output);
|
||
**fp8 KV** (disables the sparse MLA backend's rope-free Triton path, and
|
||
aiter's MLA decode op is gated on CDNA>2, so the MTP draft step demands an op
|
||
that cannot exist on gfx12); **NBATCH 8192** (boots, identical prefill).
|
||
- Fork bug found: the plan's KV reserve under-models this model's
|
||
per-sequence KDA state and the MTP draft's KV, and `gpu_worker` caps KV at
|
||
that reserve, making a legal `--max-model-len` unbootable. Shipped
|
||
`scripts/patches/gpu_worker-kv-cap.patch` (lets an explicit operator KV
|
||
request raise the cap) — NOT needed by the deployed shape.
|
||
- Rollback: commit `2339fae` (qwen FP8 + MTP4); the qwen payload and the R9V
|
||
state dirs are intact on disk.
|
||
|
||
|
||
## 2026-09-29
|
||
- **frank-dragoon tuning investigation: a measured NEGATIVE result, nothing
|
||
changed.** Frank asked for iterative decode/prefill improvement at a 32k
|
||
working point without touching the model or quantization. **No win exists
|
||
at gufo's served settings**, so no config change was shipped. frank-dragoon
|
||
was stopped for the whole investigation (~4h) per the benchmarking policy
|
||
and restored healthy in 9s afterwards.
|
||
- Measured on the repo's own prose and shell sources at **32,762 prompt
|
||
tokens**, salted per run so a warm prefix cache could not flatter a
|
||
later run, 3–5 repetitions per configuration.
|
||
- **`--prefill-chunk` is dead**: 512 → 8192 moves prefill 410.7–412.4 t/s.
|
||
A 16x change in chunk size, 0.4% out.
|
||
- **`--draft-tokens` is flat**: 5 → 10 gives 13.79–15.86 t/s decode, all
|
||
inside the run-to-run noise band. The 5-rep confirmation round
|
||
**reversed** the 3-rep leaders — `dt9 + chunk4096` looked best and came
|
||
**last** (12.86), while the **deployed config came first** (14.71).
|
||
- `--draft-policy` fixed vs adaptive: no effect. `--min-draft-tokens`:
|
||
**rejected by the engine** ("DFlash2 requires ... 1"). MTP: **unsupported
|
||
for Q8 targets** (`Unsupported Qwen MTP tensor type for
|
||
token_embd.weight: Q8_0`).
|
||
- `--context 262144 → 40960` was tested as a diagnostic (the deployed value
|
||
preallocates KV ~8x larger than a 32k conversation needs): **no gain**
|
||
(13.93 vs 14.71 t/s), and it would break `models.yml`'s declared
|
||
`contextWindow: 262144`. Not shipped.
|
||
- **What actually moves the number is speculation itself: 6.65 t/s off →
|
||
~15 t/s on, 5.3x** — already enabled. Prefill is insensitive to
|
||
everything tried, consistent with gufo's own `EXPERIMENTS.md` where
|
||
dozens of kernel variants are recorded rejected for "no release
|
||
throughput improvement".
|
||
- **Two traps are now in AGENTS.md's benchmarking policy**, because they
|
||
cost most of the wall time and both produced *false* results:
|
||
(1) **`gufo bench` cannot rank speculative settings and inverted the
|
||
ranking** — its corpus reports `acceptance=0.933 +/- 0.000` with an
|
||
identical trace sha256 across draft widths (the same highly-predictable
|
||
text every run, 93% of proposals accepted). It reported 29–37 t/s where
|
||
real traffic gives ~15, and made dt16 look 6% faster than dt7 when real
|
||
text made it 14% slower. (2) **A live A/B needs >=128 generated tokens**;
|
||
at 8, `inter_token` is first-token noise and the first pass of the sweep
|
||
produced a nonsense ordering that the 128-token pass reversed.
|
||
- Also worth recording: the `Server-Timing` header is
|
||
`ttft;dur=402.485` — the unit is in the **name** half, so splitting on
|
||
`=` keys the dict `ttft;dur` and every lookup silently returns nothing.
|
||
Four configurations reported "no Server-Timing" while the header was
|
||
present on all of them.
|
||
- **dankserver FP8 decode-tuning campaign: flags exhausted at 26.8 tok/s,
|
||
mechanism measured.** The ad hoc qwen-fp8 container (tcclaviger RDNA4
|
||
fork, orcarouter Qwen3.8-Flash-Next-Uncensored-FP8, TP=2) ran a full
|
||
flag-level A/B battery at the 32k working point; every number is from the
|
||
server's usage counters (SSE chunks are invalid under spec decode). New
|
||
runbook: [[dankserver-qwen38-fp8-expert-offload-tuning]].
|
||
- Incumbent stands: MTP depth 4, stock gather 8x16, prefetch depth 1,
|
||
TUNABLE off — **26.80/25.15/26.81 tok/s** at 166/1087/4064 ctx,
|
||
acceptance 60.7%; the re-check reproduced it (25.80/24.38/26.63).
|
||
- **Ruled out by measurement:** gather geometry 16x32 (−9%) and 32x64
|
||
(parity) — the fork's "tuned at the Gen4 ceiling" comment is stale, the
|
||
stock constants are already right on Gen5 x8; MTP depth 3 (−9%, flat in
|
||
ctx) and depth 6 (collapses to 13.5–19, acceptance 17.4% — and proves
|
||
the UVA read-through path is SLOWER than copy-in); MNBT 8192 (neutral);
|
||
prefetch depth 2 (neutral — the tier is 16 slots/layer against a
|
||
~2,400-slab working set at 73% misses); 2-way concurrency (summed
|
||
25.49 < single 26.8 — under TP both streams' misses gather on each
|
||
rank, the links do not add capacity, and the shared arena thrashes).
|
||
- **TunableOp rejected:** it silently skips under expert offload unless
|
||
`CLAV_TUNABLEOP_WITH_OFFLOAD=1`; once forced on it corrupts numerics —
|
||
repetition loops, the server's repetition-halt firing at ~140 of 256
|
||
tokens, "79.8 tok/s" measured on halted degenerate completions, and a
|
||
fresh-prompt check at 20.01 (below baseline). +6.7 min warmup, no
|
||
persistent cache in the container lifecycle.
|
||
- **Infeasible:** `UTIL=0.99` (free-memory wall: 31.29 of 31.86 GiB free
|
||
at worker init); `--max-num-seqs 2` (compile constraint violation at
|
||
CUDA-graph capture); ngram-only drafter (fork bug: "PLE inputs were not
|
||
prepared" — PLE prep is wired only for the MTP spec path).
|
||
- **The wall, measured:** the link does Gen5 x8 = 31.5 GB/s/direction;
|
||
the HIP probe reads 22.74 GB/s at the exact 2.345 MiB expert slab (91%
|
||
of ceiling at bulk); the engine's effective gather rate is ~9.5 GB/s =
|
||
42% of that. A decode step is ~100 ms compute + ~400 ms PCIe wait
|
||
(peak util 100%, smooth ~29%). The gap to the 35 tok/s target is the
|
||
fork's gather read path — fork-patch territory, not flags.
|
||
- **Telemetry fork bugs:** the torch-profiler worker stop hangs on ROCm
|
||
(>10 min, two attempts, while the AsyncLLM wrapper logs success in
|
||
1 s), and the default `torch_profiler_with_stack=true` grew a 6.5-min
|
||
profile window's trace buffer ~40 GB → the kernel OOM-killed the engine
|
||
core (anon-rss 95.2 GB, `oom-kill` in `journalctl -k`). The per-layer
|
||
expert-cache hit rates (`R4D_LRU_TELEMETRY=1`) are reachable ONLY
|
||
through that stop path, so the telemetry JSON is unreachable without a
|
||
fork patch.
|
||
- **256k ctx boots** (the last original-goal dimension): a 262,144-token
|
||
window needs **4.0 GiB of KV** (hybrid attention + mamba pages ≈
|
||
16 KiB/token, not the 32k plan's 9,190 B/token) — 3 GiB boot-refused
|
||
(`4.0 GiB needed > 2.98 available`; the engine's own max-len estimate
|
||
at that pool: 184,224). At KV 4.5 GiB it serves **36.65/34.21/34.77**
|
||
(arena 6,574→5,648 slots, −14%, ~10% decode cost; 2 of 3 depths just
|
||
under 35). KV-exact-4.0 arm in flight for the last +3.5% slots.
|
||
Correction recorded: the planner sizes the arena AFTER the KV override
|
||
(the earlier "before" reading was wrong) — the arena is what remains.
|
||
- **Final 256k probe: the KV trim is inert.** KV-exact 4.0 GiB
|
||
boot-refused by 20 MB of accounting (3.98 available < 4.0 needed;
|
||
engine estimate at 3.98: max len 260,176); +32 MiB (`KV_BYTES=
|
||
4328521728`) boots — but the arena is **identical to the 4.5 GiB arm
|
||
(5,648 slots)**: the planner buckets, the 0.47 GiB went to margin.
|
||
Decode band **36.48/34.13/35.02** (acceptance 39.4%) — same 34.1-36.5
|
||
band, straddling the 35 target within noise. Final 256k config:
|
||
`MAXLEN=262144 KV_BYTES=4328521728`, prefetch off. The full-speed
|
||
alternative stays `MAXLEN=184224` at 3 GiB (37.4-40.6 band).
|
||
- **GLM-5.3-Flash readiness REVISED by source reading (Frank's question
|
||
about int4 vs mxfp4):** the tcclaviger fork in the qwen-fp8 container
|
||
registers `Glm5NextForCausalLM`/`Glm5NextMTP` with a backend-split
|
||
module (`glm5next/amd|common|nvidia`, `sparse_indexer.py` for DSA);
|
||
`offloader/expert.py` accepts `R4dDsfp4MoEExperts, R4dW4a16MoEExperts,
|
||
Fp8HipMoEExperts` — and **`R4dDsfp4MoEExperts` is MXFP4 weights with
|
||
the DeepSeek activation law** (docstring; SwiGLU `gemm1_clamp_limit`
|
||
default 10.0), inheriting the mxfp4 repack/GEMM/emulation-fallback.
|
||
**`r4d_lib.has_dsfp4()` is True** in this build — native `moe_dsfp4` /
|
||
`moe_dsfp4_mt1` entries exist, so the old "MXFP4 = emulation, missing
|
||
kernels" GLM verdict is dead for this fork. WNA16 (int4) backend
|
||
oracle on ROCm reduces to HUMMING/TRITON (Marlin/TRTLLM/XPU are
|
||
CUDA/NVIDIA/Intel-only), payload must be compressed-tensors W4A16
|
||
group-wise symmetric. Remaining GLM gate: a compressed-tensors MXFP4
|
||
(~165 GiB) or W4A16 payload + a glm5next-on-HIP boot probe; community
|
||
GLM-5.3 quants are GGUF (llama.cpp path only), native FP8 is 328 GB
|
||
(> 246 GiB RAM). Feasibility doc rows updated.
|
||
- **Revert pins installed (Frank's request):** (1) repo tag
|
||
`qwen-fp8-tuned-20260929` at f532e03 (the campaign's docs + measured
|
||
config; `git checkout qwen-fp8-tuned-20260929` restores the state);
|
||
(2) image pinned by digest — serve script now launches
|
||
`docker.io/tcclaviger/vllm@sha256:da402966...` (ID 3322dbbbc96d) with a
|
||
local alias `qwen-fp8-runtime:tuned-20260929`; the floating `:latest`
|
||
is no longer referenced anywhere (the other image on the host,
|
||
`28.06.1-Code000` = fffca39f, is a DIFFERENT build — do not confuse).
|
||
Runbook deployment-shape section updated.
|
||
- **Endpoint published:** `PORTMAP=7773:7770` hook added to the serve
|
||
script; the 256k config relaunched with it. Host-side checks: health
|
||
200 on 127.0.0.1:7773 AND 192.168.0.235:7773, /v1/models lists
|
||
qwen3.8-flash-next, decode through the published path
|
||
36.49/34.55/35.98 (same band). 7773 chosen over 7770 — frank-archon
|
||
owns 7770 when it ever returns.
|
||
|
||
## 2026-09-28
|
||
- **Dankclaw model cleanup** (disk was at 93%, 137 G free). Frank asked to
|
||
clean up old models. **510 G staged** to
|
||
`~/.AGENTTRASH/20260928-retired-models/` — moved, NOT deleted, with a
|
||
`MANIFEST.md` giving a restore command per item. **Nothing is freed until
|
||
the purge is approved**; that is the point of the trash policy, and the
|
||
number to know before expecting the disk to move.
|
||
- Staged: CIRU Orca 128 G, halogen 123 G, unsloth Flash-Next IQ4_XS 91 G,
|
||
mradermacher IQ4_XS 91 G, unsloth 27B 29.3 G, Kairic Edge 27 G,
|
||
ISTA-DASLab GSQ-RCO 13 G, vLLM Gemma4 w4a16 9.6 G, vLLM Gemma4 drafter
|
||
838 M. Every one maps to a runtime already retired in the runbook index.
|
||
- **The 29.3 G rollback target went with them**, by Frank's decision. The
|
||
payload-only rollback is now a re-fetch first; both the runbook and
|
||
AGENTS.md say so. **The DFlash2 drafter was NOT reclaimed** — only the
|
||
`.gguf` was taken out of that directory, because the live frank-dragoon
|
||
drafter lives in it. The manifest warns against restoring the directory
|
||
wholesale.
|
||
- **Kept deliberately:** `checkpoints/` 23 G. It is unreferenced by the
|
||
repo, but ComfyUI is live and its workflows were not searched — not
|
||
confirmed dead, so not taken. `logs/x4e-run.log` also stays: it is a
|
||
**sparse** file, 3.3 G apparent but 4 K real, so removing it frees
|
||
nothing.
|
||
- **A live reference check came first, and it earned its keep.**
|
||
`serve-frank-zealot.sh`'s *fallback* `MODEL_DIR` still pointed at the
|
||
retired GSQ-RCO payload frank-zealot served on 2026-09-16/17. The env
|
||
file has always overridden it, so the drift was invisible — but a
|
||
missing or unreadable env file would have turned a config problem into a
|
||
"missing asset" naming a model that no longer exists. Repointed at
|
||
`Gemma4-12B-QAT/`, which is what the env file already sets. Every other
|
||
hit was a comment, a retired fetch script's destination, or
|
||
`retire_halogen_dragoon_quadlet` (which moves a *quadlet*, not a model).
|
||
- **Not taken, by Frank's decision — scope was models only:** podman
|
||
images (**279 G reclaimable, 95%**; 170 images, 23 active) and the
|
||
uv+pip caches (**73 G**). Together that is 352 G of regenerable garbage
|
||
larger than the models reclaimed, and it is the obvious next move.
|
||
- Pre-existing: **118 G of trash from 2026-09-21** was already staged
|
||
(`20260921-frank-dragoon-assembled` 88 G, `-redownload` 30 G) and never
|
||
purged. Total now staged on this host: **627 G**.
|
||
- Verified after the moves: frank-dragoon and frank-zealot both `active`,
|
||
both `/health` 200, and all three live assets (Swift target, DFlash2
|
||
drafter, Gemma4 target) still present.
|
||
- **frank-dragoon payload swapped to Swift-1.5 (uncensored)** (production
|
||
change, Dankclaw). Frank asked whether the UD Q8 of
|
||
`ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF` would load on
|
||
gufo, then said to do it. Port 7770, alias `frank-dragoon`, one 262,144
|
||
session, the DFlash2 drafter and the client contract are unchanged; the
|
||
27B dense weights are an **abliterated (uncensored)** fine-tune of the
|
||
same base, under the **non-OSI SWIFT OPEN LICENSE 1.0**.
|
||
- **Compatibility was established from the GGUF headers, not from the
|
||
filename.** Both files were range-fetched and parsed: same
|
||
`general.architecture` (`qwen35`), same 866 tensors with **identical
|
||
names in identical order**, **zero** per-tensor `ggml_type` or dim
|
||
mismatches (Q8_0×453, F32×360, BF16×53), same MTP head at `blk.64`, same
|
||
geometry. The two scary metadata differences are both no-ops, confirmed
|
||
against gufo's source: `general.file_type` (15 vs 7) is *"deliberately
|
||
not consulted"* by `gguf_reader.cpp` — the file gufo already served
|
||
records 15 (Q4_K_M) while containing no Q4_K tensor — and Swift's extra
|
||
`qwen35.attention.recurrent_layers` key is **never read** by gufo
|
||
anywhere in `src/`; layer kinds come from `full_attention_interval`,
|
||
which both files carry as `4` and which agrees with it.
|
||
- **Fetched and verified:** 31,457,990,272 bytes, sha256 `9076f99a…`
|
||
matching the Hub LFS oid, via the new
|
||
`/scripts/download-frank-dragoon-swift.sh`. It sends **no auth header** —
|
||
the repo is public and ungated, and a token on the curl command line is
|
||
world-readable via `/proc/<pid>/cmdline`. The anonymous transfer ran at
|
||
~1.8 MiB/s; resuming with the host's token through a **mode-600
|
||
`curl -K` config** took it to ~7 MiB/s. The download survived an ssh
|
||
drop by running `setsid nohup`, not under the session.
|
||
- **Load test, service stopped first** (the GPU policy): **9.2 s** from
|
||
`load_started` to `load_completed`, 66.5 GiB of the 116.7 GiB iGPU at
|
||
context capacity 262,144, no loader warnings. Contract battery green:
|
||
`context_length 262144`, `content` + `reasoning_content`,
|
||
`finish_reason: tool_calls`, SSE, and a **two-turn tool loop** with prior
|
||
reasoning preserved. Spot-checks: "17*23" → `391`; asked for a
|
||
lockpicking scene it wrote one without hedging, confirming the
|
||
abliteration.
|
||
- **THE TEMPLATE LAW IS GONE, by decision.** Frank asked to "use our chat
|
||
template"; gufo **cannot** take it. This is not a missing flag —
|
||
`src/models/qwen/chat_template.cpp` is 587 lines of hand-written C++
|
||
that appends strings, with no Jinja engine, no `--chat-template-file`,
|
||
no env override and no template body in the binary. Qwen-Sharp
|
||
v22.4.0 is 453 lines of Jinja2 with `namespace()`, macros and
|
||
`raise_exception`, so supplying it is a **port, not a config line**.
|
||
Frank chose to ship on gufo's stock renderer and accept the loss: the
|
||
`terse` system-prompt injection, in-band `<|think_*|>` control, the
|
||
`json|xml` `tool_call_format` switch, tool-argument truncation and
|
||
`add_vision_id` prefixes are all gone. gufo's renderer does keep the
|
||
effort instruction strings and `preserve_thinking`. **Worth watching:
|
||
an abliterated model now runs without the prompt that tells it to stop
|
||
waffling** — the cheap fix is to send the `terse` text as an ordinary
|
||
system message from `models.yml`, not to touch a template.
|
||
- **The payload is now two files in two directories.** The target moved
|
||
repo; the DFlash2 drafter did not move, so `DRAFTER_DIR` is its own
|
||
tunable. That broke the deployer's stray-run sweep, which keyed on the
|
||
*target* filename and would have silently stopped matching — a stray
|
||
run would then have held 7770 past the port gate. It now keys on the
|
||
drafter, which every frank-dragoon command line carries via
|
||
`--dflash-model` and which did not move.
|
||
- **`/tests/frank_dragoon_unit.bats`** — 20 assertions. The mutation pass
|
||
found **four real gaps** on the first run, all now fixed: two fetch
|
||
assertions were satisfied by a **header comment** rather than the URL
|
||
assignment, so repointing a repo left them green; and nothing bound the
|
||
env's `MODEL_FILE`/`DRAFTER_FILE` to the filenames the fetch scripts
|
||
actually download, so a re-upload or an env edit would break the
|
||
endpoint with a "missing asset" no test predicted. There is now an
|
||
explicit cross-file invariant. A second harness bug worth recording:
|
||
a `bats --filter` that matches no test reports zero `not ok`, which
|
||
looks exactly like a mutation that survived — the harness now asserts
|
||
the `1..N` count before believing a result. 16/16 mutations caught.
|
||
- Docs: runbook gained a **Payload** section (the header evidence),
|
||
**The template law** section and a two-way **Rollback**; AGENTS.md
|
||
fleet table, frank-dragoon section and rollback.
|
||
- **frank-dragoon runtime swapped to gufo** (production change, Dankclaw).
|
||
Frank asked for `github.com/gufo-org/gufo` — a vertical inference engine
|
||
built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S,
|
||
`gfx1151`) — in place of the stock llama.cpp prebuilt. Port 7770, alias
|
||
`frank-dragoon`, one 262,144-token session and the client contract were all
|
||
unchanged; only the engine was new, and the payload was the unsloth
|
||
Qwen3.8-27B UD-Q8_K_XL **at that point** — it was swapped for the
|
||
abliterated Swift-1.5 later the same day, see the entry above.
|
||
- **The runtime is built from source on the host.** gufo ships no prebuilt
|
||
Linux binary (the v0.1.0/v0.1.1 release assets are empty) and the host
|
||
has no Nix. Pinned at `v0.1.1` / `b0f8673`, installed to
|
||
`/home/dank/GitPackages/gufo-v0.1.1/` by the new
|
||
`/scripts/build-gufo.sh`, which is reproducible and fails closed on a
|
||
moved tag. Unlike the unsloth prebuilt it replaces, gufo **links the host
|
||
ROCm 7.2.4** through its RPATH, so the HIP dev packages must stay.
|
||
- **Two host workarounds, both encoded in the build script.** Ubuntu's
|
||
`librocwmma-dev` ships a truncated rocWMMA header set into
|
||
`/usr/include` that shadows AMD's complete one; the build died on
|
||
`'internal/accessors.hpp' file not found`, and a `-I` cannot fix it
|
||
because in HIP mode the host compilation pass searches the system
|
||
directories first. The package has no reverse dependencies; it is
|
||
removed. Second, Ubuntu 26.04 ships `/usr/include/c++/16` and the ROCm
|
||
clang prefers the newest tree, whose `<atomic_wait.h>` needs a
|
||
`std::__detail::__notify_impl` symbol version the installed
|
||
`libstdc++.so.6` (GLIBCXX_3.4.35) does not export — the link failed until
|
||
`--gcc-install-dir` was pinned to GCC 15.
|
||
- `gufo diagnose` reports `[PASS]` on `gfx1151` (ROCm 7.2.3, amdgpu
|
||
loaded, 20 CUs). It self-caps compute queues at 2 resident for
|
||
`serve llm` after reading `/sys/class/kfd/kfd/proc/*/queues/*/type`,
|
||
which matters because ComfyUI holds queues on the same device.
|
||
- **Shape:** one session, `--context 262144`, DFlash2 with the **adaptive**
|
||
block policy capped at `--draft-tokens 7` (llama.cpp's fixed depth 4 is
|
||
gone), greedy, `--think auto --preserve-thinking on`, text-only.
|
||
- **Client-contract deltas, all in the runbook:** the chat template is
|
||
compiled into the binary (no `--chat-template-file`, so the Qwen-Sharp
|
||
73k-token-prompt failure mode cannot recur); `reasoning_effort` now
|
||
accepts the full OpenAI set, so `models.yml`'s `reasoningEffortMap` fold
|
||
is no longer required — **kept anyway** so this commit does not also move
|
||
every `high` role from `xhigh` to `high` effort; and there is no
|
||
KV-quantization or prompt-cache-RAM flag.
|
||
- **Measured at cutover:** 13 s from `load_started` to listening (target
|
||
8 s, drafter 2 s, session 3 s) against a 20-minute `LLM_WAIT` budget;
|
||
RSS 30.6 GiB; a live short-prompt request returned
|
||
`Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (~41 tok/s).
|
||
Verified live: `/health` 200, `/ready` 200, `/v1/models` →
|
||
`frank-dragoon` at `context_length 262144`, a chat completion carrying
|
||
`reasoning_content`, a `tool_calls` response with
|
||
`finish_reason: tool_calls`, and SSE streaming.
|
||
- **The trade, stated plainly.** gufo's own Q8_K_XL tables against
|
||
llama.cpp b11069: prefill +37 % to +80 %, cold load 4.5× faster, decode
|
||
+7 % at zero depth — but **−14 % at 65,536 tokens of depth and −24 % at
|
||
131,072**. On the 8-bit target the crossover is ~32k and frank-dragoon runs
|
||
one 262,144-token conversation, so long agent sessions sit on the losing
|
||
side of it. The Q4_K_XL target shows no such regression; revisiting the
|
||
quantization is a separate decision.
|
||
- `/tests/frank_dragoon_unit.bats` — Rewritten around behaviour: the seam
|
||
runs the serve script against a stub engine and asserts on the argv it
|
||
actually execs, instead of grepping the script's text. A caveat found
|
||
while writing it: **bats 1.13 ignores a failing `A && B` in the middle of
|
||
a test** (bash exempts the non-final element of an AND-list from
|
||
`errexit`), so every such assertion was split into standalone `[ ]` lines.
|
||
Mutations confirmed the seam bites: dead runtime gate, `DRAFT_TOKENS=0`,
|
||
a wrong context/sessions/port, an alias swap, a moved drafter, a removed
|
||
missing-asset message and a neutered commit pin each fail it.
|
||
- `/scripts/deploy-remote.sh` — `deploy_llm_dragoon` gates on the runtime
|
||
binary as well as the payload and returns **before** it stops the serving
|
||
unit, so a host whose gufo was never built reports the failure instead of
|
||
taking the endpoint down on the way there.
|
||
- Docs: new `/serverAppConfigs-docs/LLM/frank-dragoon-gufo-qwen38-27b.md`;
|
||
`frank-dragoon-unsloth-qwen38-27b-q8xl.md` marked RETIRED (still the
|
||
rollback target — the prebuilt and the payload are both intact on disk);
|
||
AGENTS.md fleet table, frank-dragoon section, bench policy and runbook
|
||
index; both vault indexes; `Dankclaw System Specs.md`; `README.md`.
|
||
- **e2e gate: 32/33.** The one failure is `192.168.0.112` (the FrankBook-Pro
|
||
Mac mini) at 100 % packet loss — offline, and the same failure recorded
|
||
in `564a781`. Not related to this change; no frank-dragoon test failed.
|
||
- **Frontier-MoE feasibility study on Dankserver** (research + build, no
|
||
production change). Frank asked which of DeepSeek-V4.1-Flash,
|
||
MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash would run best on the 2x R9700 box,
|
||
and whether Affinity / vllm-radiance / FreeToken-AMD could be modified to run
|
||
them fast. Findings, all evidenced:
|
||
- **V4.1 does not fit.** 475.27 GiB checkpoint (measured from the HF API),
|
||
of which **189.13 GiB is Engram** in two 94.56 GiB shards. A token selects
|
||
240 routed experts at ~11 MB; at the 37-of-384-per-layer residency the hot
|
||
stack can hold, ~2.4 GB crosses PCIe per token. At 55 GB/s real that is a
|
||
**~23 tok/s ceiling** — below the 35 tok/s goal before any attention or
|
||
dense work. The 1000 t/s prefill target is ~20x away: the backend refuses
|
||
layer-major prefill for V4.1, so the only legal prefill is token-by-token.
|
||
- **Affinity** (Codeberg, StillDeadcode) is a *DeepSeek-V4-Flash* engine
|
||
claiming 110 tok/s on two R9700s; it **builds clean here** (hipcc 23,
|
||
gfx1201, zero errors). The 110 works because the 101 GiB container leaves
|
||
the cold expert set in a pinned host pool in RAM. A V4.1 port reuses the
|
||
quantiser, container, slab, host pool and placement engine (expert tensor
|
||
naming is identical, 15,360 tensors at 40x384) but must write CED, CSA2's
|
||
new indexer, the DSpark draft geometry and Engram — and Engram fails
|
||
*silently*, because V4.1 has no `num_hash_layers` key so Affinity's
|
||
`model.h:36` default of 3 survives. Weeks of work, same capacity wall.
|
||
- **Lucebox** is the only engine that runs V4.1 on gfx1201, on the unmerged
|
||
`feat/ds41-engine` branch. Its own DS41.md measures 0.8-1.9 tok/s on an
|
||
R9700 (best tuned three-tier case 9.1) with **61 GB of RAM**; Dankserver's
|
||
246 GiB is the one untested lever, so the 377.5 GiB ROCMFP23 payload is
|
||
downloading for a three-arm bench.
|
||
- `/serverAppConfigs-docs/LLM/Dankserver System Specs.md` — **RAM corrected
|
||
60 GiB -> 246 GiB** (stale since the 2026-09-20 board swap; the Lucebox
|
||
runbook had it right) and storage refreshed; `/mnt/entropy` is at 100%
|
||
(244 G) and must not be planned for payload.
|
||
- `/serverAppConfigs-docs/LLM/dankserver-frontier-moe-feasibility.md` — new
|
||
runbook; indexed in `LLM/index.md`.
|
||
- **LANDMINE, not changed here:** `/home/dank/models` was cleared by Frank
|
||
mid-session to free space. frank-archon is still answering `:7770` from
|
||
mmapped-but-unlinked inodes (`/models` is empty inside the running
|
||
container) — **the endpoint cannot be restarted until the 175 GiB payload
|
||
is re-fetched.** A reboot or a container restart takes frank-archon down.
|
||
- Deploy-state reconciliation (Frank's call: no restart) — the autoresearch trio
|
||
(`f187165` PLE 1800, `41e9189` prefix caching, `ba2100e` docs) was found already
|
||
MERGED to main + pushed AND already live on frank-archon: the running
|
||
`frank-archon-r9v` container (booted 12:13 from the `ba2100e` checkout) carries
|
||
`--enable-prefix-caching` in argv and `VLLM_PLE_OFFLOAD_READY_TIMEOUT=1800` in env,
|
||
`/health` 200 — but `deploy-state/frank-archon` was still stamped `f187165`. A
|
||
`deploy.sh` run would have re-selected frank-archon and reloaded the 175 GiB
|
||
payload (~16–19 min) for zero behavioral change, on the endpoint this harness itself
|
||
runs on. Stamped the marker `f187165 → ba2100e` (status `deployed`, atomic
|
||
mktemp+mv per the deployer idiom); service untouched. e2e gate 32/33 — the lone
|
||
failure `11-admin` is the FrankBook-Pro Mac mini (`192.168.0.112:8989`) offline
|
||
(100 % ICMP loss + TCP timeout), the documented asleep-host condition, not a
|
||
regression.
|
||
- `scripts/frank-archon.env`, runbook — **prefix caching DEPLOYED on
|
||
frank-archon** (Frank's go-ahead; commit `41e9189`): E8-qualified, zero
|
||
VRAM cost, 9x/63x reuse TTFT. Verified on :7770 (argv flag, health 200,
|
||
completion "PREFIX-CACHE-LIVE"). Gotcha re-confirmed: the container is
|
||
in dank's ROOTLESS podman store (`sudo podman` inspects an empty store
|
||
and reports nothing); probe via the shim's `docker` or the unit MainPID.
|
||
- `/scripts/frank-archon.env`, `/serverAppConfigs-docs/LLM/frank-archon-r9v-radiance-qwen38-q8.md`, `/serverAppConfigs-docs/index.md` — **frank-archon prefill/decode bench matrix** (scratch :7773, prod stopped 00:06 for the window per fleet bench law): baseline E0 (chunk 4096 / 80-slot sync / MTP4) = PP8K 954.6, PP131K 883.0 tok/s, decode 141.9 ms/step ≈ 23.8 t/s effective. Chunk 8192 (E1R): PP8K +25.9% (1201.4) but decode −8.5% and PP131K −4.4% — short-prompt win, decode tax, not the fleet's dominant shape. Chunk 16384 boots but OOMs at request time (0.93 GiB free) — 4096 is the ceiling (engine spec-config cap + memory). Expert cache 80 slots is the policy max (doctor: capacities 48/64/80, synchronous LRU only — async unsupported, 96/112 rejected); MTP depth 6 is 3.7x SLOWER than depth 4. Prefix caching (E8/E9): zero VRAM cost, cold path unchanged, reuse hits 32k 44.2→3.6–4.3 s (9x), 131k 439→6.9 s (63x); hit pattern rounds 1–2 miss / 3+ hit — RECOMMENDED, pending Frank's go-ahead. Two failure laws recorded: unqualified shapes fail mid-request (health stays 200), and PLE registration can blow the engine's 600 s `VLLM_PLE_OFFLOAD_READY_TIMEOUT` under page-cache churn (measured 740 s) — launch.sh gained `R9V_PLE_READY_TIMEOUT` forwarding (fork `2de0357`, local-only) and prod now sets 1800.
|
||
- **Incident — frank-archon down 00:06 → ~11:00 (11h)**: the bench window's required prod stop was never restored — the matrix + follow-ups ran 00:44 → 09:02 and the session let the endpoint sit dead (the 2026-09-18 "a bench session owes the fleet a health 200" lesson, re-offended). The first restore boot (09:46) then failed on the PLE 600 s timeout above (unit "Deactivated successfully", no journal error — the serve script exits 0 on the wait). Restored by the `R9V_PLE_READY_TIMEOUT=1800` fix deploy (f187165, from main — the deploy gate refuses non-main): health 200 + MainPID = the serve script's own `podman wait` + live completion verified ~11:00. Deploy e2e gate 32/33 — the one failure (11-admin) needs the FrankBook-Pro Mac mini, which was asleep and ignored two WoL bursts (240 packets); recurring pattern per the wol-wake skill. Lesson: a multi-hour bench window needs an explicit restore task, not a "watch for DONE" — DONE ≠ prod up.
|
||
|
||
## 2026-09-27
|
||
- `/scripts/frank-archon.env`, `/scripts/frank-archon.service`, `/scripts/serve-frank-archon.sh`, `/scripts/deploy-remote.sh`, `/tests/frank_archon_unit.bats`, `/serverAppConfigs-docs/LLM/frank-archon-r9v-radiance-qwen38-q8.md` (new), AGENTS.md + LLM index — **frank-archon cut over to the Q8_0 payload** (uncensored IQ4_XS → unsloth Qwen3.8-Flash-Next Q8_0, 175.30 GiB / 6 shards) at Frank's direction. Launcher gained the opt-in `R9V_CHAT_TEMPLATE_FILE`/`R9V_CHAT_TEMPLATE_KWARGS` passthrough (fork commit `30d4a32`, local-only) — the Qwen-Sharp v22.4.0 template now serves, fleet parity with frank-dragoon. Env points at `/home/dank/r9v-state/q8xl/setup.json` + the quantized-inject dev-overlay adapter (fork `363d606`) + `R9V_FULL_MUTABLE_CACHE=0`; shard gates extended to SHARD4/5/6 (unset-safe); seams recalibrated; ctx window 131072 (models.yml maxTokens 36864). Spec decode is MTP4 only — the fork's `speculative_config` is a single dict, no chaining, so llama.cpp's `draft-mtp,ngram-mod` pattern is NOT expressible (PLE n-gram rides along architecturally). Uncensored state + payload remain on disk as the rollback target. 346 tests green.
|
||
- Dankserver frank-archon slot (experiment window) — the `radiance-vllm-qwen4exp` fork qualified the unsloth `Qwen3.8-Flash-Next-GGUF` **Q8_0** payload (175.30 GiB, 6 split parts) on the dual R9700. Two launch blockers fixed loader-side in the fork: `runtimes/dev-overrides/qwen4_exp.py` dequantizes quantized `block_inject` shards in-stream (the Q8_0 payload quantizes BOTH fused-HC shards; IQ4_XS ships inject as F32 — the uniformity check rejected the mixed projection) and `R9V_FULL_MUTABLE_CACHE=0` passthrough for the 80-slot tiered cache (fork commits 363d606, 2fe6dbb, 50a79ec — GitHub remote, not pushed). Battery on :7773: text + reasoning channel, vision ("Red" on a red PNG), prose ≈11.0 tok/s wall, MTP4 mean acceptance length 3.19–5.00; PLE pinned residency proven (`mapping_bytes=54400261120`, `mapping_rss_bytes` == mapping size, both ranks required 108.8 GiB vs 140.9 available). Relaunch recipe saved at `dankserver:/home/dank/r9v-state/q8xl/START-COMMAND.txt`; record at fork `docs/qualification/qwen38-q8-0-dual-r9700-2026-09-27.md`.
|
||
- Dankserver frank-archon restore — uncensored UD-IQ4_XS back on `0.0.0.0:7770` as `frank-archon-r9v` from its frozen config (health 200 in ~60 s; LAN-path completion returns content + reasoning); `frank-archon.service` (llama.cpp) **runtime-masked** for the window (`/run/systemd/system/frank-archon.service → /dev/null` — the `/etc` relink symlink blocks a plain mask; self-clears at reboot, unit file untouched) and the container runs `restart=no`, so a reboot returns the repo-configured llama.cpp with no contention. Restore config preserved at `dankserver:/home/dank/r9v-state/uncensored/restore-config-20260927.env`; scratch in `dankserver:~/.AGENTTRASH/20260927-q8xl-bringup/`.
|
||
|
||
## 2026-09-25
|
||
- `/serverAppConfigs-docs/Dankserver/Jellyfin.md` — Jellyfin LDAP 401 resolved: the auth works via the modern `/Users/AuthenticateByName` endpoint (HTTP 200, alex authenticated, full plugin flow logged: service bind → search → ldapUid → user bind → token). The 401 was a test artifact — I had been probing the legacy `/Users/Auth/Password` route, which Jellyfin 12.1 removed (the `LegacyEmbyRouteRewriteMiddleware` is gone). The web UI uses the modern endpoint, so user logins are fine. No config change needed; the LDAP-Auth 23.0.0.0 plugin + authentik 2026.8.3 outpost are healthy.
|
||
- `/shared/homepage/services.json`, `/shared/homepage/docs.html` — Suggested uses added to the three LLM model descriptions (status cards + the `/docs` model guide + the "which one to pick" foot-note): frank-dragoon → cybersecurity + debugging (use requires Frank's permission); frank-medium → the primary intelligent model (surpasses Claude Sonnet 5 per Artificial Analysis) for software engineering, sysadmin, instruction tasks; frank-zealot → the fast/cheap utility model (agentic search, RAG, research subagents, translation, roleplay, summarization). Deployed to `/srv/homepage`; bats 19/19 + non-Mac-mini e2e green (d3c8197).
|
||
- `/shared/homepage/{services.json,app.js,style.css,docs.html}` — Model cards reworked to a structured three-line shape: each LLM card now carries a `details` object (role / spec / use) rendered as a bold pale-green role, a bright-green spec line, and a dim "Use:" line (`app.js` `descHTML()` falls back to the plain `description` for non-LLM cards; new `.desc-role`/`.desc-spec`/`.desc-use` CSS + bare-mode reset). Roles made neutral (no nicknames): frank-dragoon = deep reasoning, frank-medium = primary model, frank-zealot = fast utility model. The `/docs` model guide was also fixed — it had drifted to retired model info (frank-dragoon listed as Qwen3.8-Flash-Next 87.3 GiB / 200k, frank-medium as FP8 ~31 GiB) — now Qwen3.8-27B UD-Q8_K_XL 31.5 GiB / 262k and MXFP4/W4A8 19.4 GiB, reordered large→medium→small. bats 30/30 + live browser render verified.
|
||
- Fleet-wide container update (dankclaw + dankserver) — every pullable image brought current. Pinned bumps: authentik `2026.8.0`→`2026.8.3` (server/worker/outpost), tautulli `v2.18.1-ls242`→`ls245`, calibre-web `0.6.27-ls397`→`ls402`, plex `1.43.3.10896`→`1.43.4.10903`; seerr already at `v3.4.1`. Local-build rebuilds (forced `--no-cache`, the base images had moved): caddy (xcaddy porkbun), alloy, loki, comfyui-rocm (ComfyUI re-cloned), hermes ×3 (fresh `:latest`, `23986f31`→`e1e87aec`). anubis already at latest stable `v1.27.0` (only pre-releases exist beyond). Verified: all 16 floating-tag services CURRENT (zero stale), every real container active, caddy edge serving (auth 302 / ai 200), 340 bats green. Excluded: frank-medium/radiance-vllm (custom LLM build).
|
||
- `/dankserver/{tautulli,calibre-web,plex}/quadlet/*.container`, `/dankclaw/authentik/quadlet/*.container` — version pins bumped; docs (Tautulli.md, Calibre-Web.md, Plex.md, Authentik.md + server/worker, index.md) and the dead `dankserver/compose.yaml` plex entry synced (0b1ea2e).
|
||
|
||
## 2026-09-22
|
||
- `/serverAppConfigs-docs/**` — OKF v0.1 frontmatter migration: `type`/`description`/`tags`/`timestamp` added to the 65 non-reserved pages lacking them (timestamps from each file's last git commit); `timestamp` added to `LLM/frank-medium-batch-ubatch.md` and `LLM/frank-zealot-benchmark.md`; unquoted YAML scalars (colons, `*arr`) fixed in 13 pre-existing frontmatter blocks; frontmatter stripped from the reserved `LLM/Pipeline Parallelism/index.md`; `log.md` restructured to the OKF format (date-grouped, newest first, one line per change) from 40 narrative entries.
|
||
- `/serverAppConfigs-docs/**` — Fact-check pass against scripts/AGENTS.md ground truth: 6 false "Active" banners fixed on retired runbooks, 96→114 GiB GTT in `Shared Memory Architecture Dankclaw.md`, `deployment.md` + `Architecture/Deployment Flow.md` rewritten to the Quadlet/deploy-plan.py truth, `Calibre.md` 9.9.0→9.15.0 + CVE-2025-7404 conflation (it is the Calibre-Web CVE), `Copyparty.md` v1.20.24, `RomM.md` 5.3.0 breaking change; `*arr` docs moved to a new `FrankBook-Pro/` section (git mv, link-safe); active runbook bodies written for frank-dragoon (unsloth UD-Q8_K_XL) and frank-zealot (Gemma4-12B).
|
||
- `/serverAppConfigs-docs/Research/` — Three new research briefs (OKF): `Local TTS and Voice Cloning.md`, `AI Job Search and Resume Tools.md`, `Most Popular Self-Hosted AI Apps.md` (the 15 most popular self-hosted AI apps of Jun–Sep 2026, ranked by GitHub stars) + a self-contained dark-theme HTML report `popular-self-hosted-ai-apps.html`; upstream-version rows added to the ComfyUI/OpenWebUI/Hermes app docs (ComfyUI 0.37.0 native Qwen-Image 2.1, OpenWebUI 0.11.4 + `/api/v1/images/generations`, Hermes v0.21.4/CalVer v2026.9.21). All 336 tests pass; 136 pages, 0 unresolved wikilinks; docs-only deploy no-op.
|
||
|
||
## 2026-09-21
|
||
- `/dankserver/frank-medium/quadlet/frank-medium.container` — frank-medium cut back to the MXFP4 checkpoint (`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq`); `KV_DTYPE` became a quadlet tunable, set `auto` (unquantized bf16 KV) to test whether fp8 KV had been degrading W4A8 output quality; pool auto-sized to 486,464 tokens; vision re-verified; e2e 32/32 (a45f396).
|
||
- `/dankserver/calibre-server/quadlet/calibre-server.container` — New boot-persistent Quadlet: Calibre Content Server on 192.168.0.235:8084 (OPDS + web UI), HTTP Basic auth, LAN-only; desktop image 9.15.0, `--auth-mode=basic` pinned, internal port 8090.
|
||
- `/serverAppConfigs-docs/Dankserver/Calibre.md` — Content-server section rewritten from "not running" to the deployed shape; Calibre-Web → authentik feasibility answered (guard chain, not in-app OIDC).
|
||
- `/scripts/serve-frank-dragoon.sh`, `/scripts/frank-dragoon.env`, `/scripts/frank-dragoon.service`, `/scripts/download-frank-dragoon-unsloth.sh` — frank-dragoon (dankclaw :7770) moved CIRU Orca → unsloth llama.cpp b11030 prebuilt serving `unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS` (87.3 GiB), 200k ctx, MTP depth 2 via explicit `-md`; the split parts ARE the payload — concatenating them trips `invalid split file name` (part 1's header advertises `file_count=3`); draft acceptance verified live 0.83–0.92.
|
||
- `/serverAppConfigs-docs/LLM/frank-dragoon-unsloth-qwen38-flash-next.md` — New runbook; `frank-dragoon-ciru-orca.md` marked RETIRED (rollback target); AGENTS.md + LLM index updated.
|
||
- `/shared/prometheus/prometheus.yml` — `node-dankclaw` target fixed to `host.containers.internal:9100` (netavark DNATs published ports only; the node-exporter's host bind was unreachable by construction — a fourth down job the 09-16 audit had not listed); `Infrastructure/Prometheus.md` targets table + `tests/observability_quadlet.bats` contract updated.
|
||
|
||
## 2026-09-20
|
||
- `/dankserver/frank-medium/quadlet/frank-medium.container` — Checkpoint swapped MXFP4 → FP8 (`orcarouter/Qwen3.8-27B-Uncensored-FP8`, MTP self-speculation depth 8, acceptance 5.66/8); the swap is now one command via `scripts/hf-manifest.py` + `scripts/hf-fetch.sh` + `scripts/swap-frank-medium.sh`; A/B on the FP8 target: DFlash2 100.0 t/s decode beats MTP 85.9 (accept length is a proxy, not the metric) — DFlash2 is the deployed shape.
|
||
- `/dankserver/frank-medium/quadlet/frank-medium.container` — Cutover to vllm-radiance TP=2 (95.8 t/s / 0.21 s TTFT / 259.4 burst-4 vs llama layer split 45.7 / 0.72 / 71.7; the winning arm is also 4-bit, part of the margin is quant); the llama root unit moved to `scripts/.AGENTTRASH/20260920-llama-layersplit/` (never `systemctl disable`); KV pool auto-sized to 943,581 tokens (3.59× the window); tools, reasoning and vision verified live against the endpoint.
|
||
- `/scripts/build-llama-rdna4.sh` — Now takes `GGML_HIP_RCCL`; the RCCL build (`build-rdna4-rocm714-rccl`) made the tensor-mode bench arm fair (Ubuntu RCCL 2.27 carries gfx1201 device code; TheRock's does not — `invalid device function`).
|
||
- `/shared/prometheus/rules/llm-fleet.yml`, `/shared/prometheus/prometheus.yml`, `/shared/grafana/provisioning/` — Monitoring spans the `vllm:` + `llamacpp:` namespaces (6 alerts + 3 recording rules); frank-zealot/frank-dragoon targets fixed to `host.containers.internal`; the prefix-cache reuse rule divides by `cached + prompt` (llama.cpp's `prompt_tokens_total` EXCLUDES cached tokens); prefix caching measured at 1,648-token block granularity (13,021-token prompt 2.30 s → 0.68 s, 76 % reused).
|
||
- `/scripts/dankserver-route-primary.sh` — `eno1` pinned primary (metric 100), `eno2` retained failover (metric 200); applied with `nmcli device reapply` (no `connection reapply` in NM 1.54).
|
||
- Hardware — dankserver board swapped to the ASUS ProArt B850-Creator WiFi NEO: both R9700s on CPU root ports at Gen5 x8, every bridge `AtomicOpsCap: Routing+`, both endpoints `ReqEn+` — the PCIe-atomics wall is gone; GPU BDFs moved, kernel 7.0.0-31, SMT back on (16 CPUs).
|
||
- `/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md` — Now the ACTIVE runbook (deployed flags, battery, capabilities, rollback); `frank-medium-llamacpp-dflash2.md` carries the SUPERSEDED banner; `Dual-R9700-Cooperation.md` states the measured verdict; `Dankserver System Specs.md` gained a Motherboard section; `frank-medium-model-swap.md` is the new swap runbook; AGENTS.md topology/sections/lessons flipped.
|
||
|
||
## 2026-09-19
|
||
- `/scripts/frank-dragoon.service`, `/scripts/serve-frank-dragoon.sh` — frank-dragoon: halogen → CIRU v4.4.1 Orca (jcbtc Qwen3.8-Flash-CIRU-STRIX-Orca w4b, 262144 ctx, MTP depth 4, 4096 MiB PLE cache, text only); halogen is the rollback target (b6ce3af, f68bda5).
|
||
- `/serverAppConfigs-docs/LLM/frank-dragoon-ciru-orca.md` — New runbook.
|
||
- `/scripts/frank-medium.env`, `/scripts/serve-frank-medium.sh` — frank-medium: ajgazin UD-Q8_K_XL → orcarouter Q8_0 + z-lab DFlash2 Q4_K_M drafter; split re-balanced 35:30 → 32:33; pool 393216 → 327680 (d374ab9).
|
||
- `/scripts/frank-zealot.env`, `/scripts/serve-frank-zealot.sh` — frank-zealot: same-day vLLM detour reverted to mainline llama.cpp Gemma4-12B Q4_K_M + MTP sidecar + ngram-mod, q8_0 KV, 4 slots of 262144 (374041c).
|
||
- Hardware — 3090 EVGA FTW3 iCX3 RGB set MODE_OFF and saved to EEPROM (persists across reboots, headless).
|
||
- AGENTS.md, both indexes, halogen runbook retirement banner (764cf41, 88331e2); 331 bats + 32 e2e green.
|
||
|
||
## 2026-09-18
|
||
- `/scripts/frank-medium.env` — `CACHE_RAM` 8192 → 16384: the old cap was binding (137 journal refusals; a full 196608-token slot state ≈ 12.8 GiB cannot fit 8 GiB); the new cap clears one full slot and stays under half the host RAM; `tests/frank_medium_unit.bats` seam added; `--ctx-checkpoints 0` confirmed a deliberate deviation (build default is 32) (1acca43).
|
||
- Incident — frank-medium found stopped twice: (morning) a bench round ended without the protocol's restore half; (midday) an orphaned `llama-server` from `third_party/llamacpp/build-merge` (PPID 1, loopback :17770) held ~16 GiB on each R9700 and would have starved the 35:30 allocation — proven abandoned (frozen CPU ticks, zero connections) then killed; health 200 in ~20 s. Lesson: "GPUs idle" is a check, not an assumption.
|
||
|
||
## 2026-09-17
|
||
- `/scripts/frank-zealot.env`, `/scripts/serve-frank-zealot.sh` — frank-zealot: ISTA-DASLab GSQ-RCO IQ3_S → `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` Q4_K_M + separate MTP assistant sidecar (`--spec-type draft-mtp,ngram-mod`); q8_0 KV (`build-cuda` rebuilt with `-DGGML_CUDA_FA_ALL_QUANTS=ON`, else ~12× prefill collapse); `--fit` chose the full 262144 window; audio transcribes verbatim; Gemma4 accepts every `reasoning_effort` level.
|
||
- `/scripts/frank-zealot.env` — `CTX_SIZE=1048576` / `PARALLEL=4` (four FULL 262144-token slots, the measured card ceiling: ~3,450 MiB/slot + ~8,546 fixed); ubatch 1024 deployed (pp ~965 / tg 226–302 at 253k; ubatch 2048 OOMs the MTP draft's pp buffer); draft n-max 4 + poll 100 measured WORSE at long ctx.
|
||
- `/serverAppConfigs-docs/LLM/frank-zealot-gemma4-12b.md` — New ACTIVE runbook; `frank-zealot-qwen38-gsq-rco.md` retired as rollback target.
|
||
- `/shared/homepage/` — The three LLM cards moved to the user page with live descriptions; new static `/docs` user page (access chain, model guide, how-tos) routed in the post-auth listener (32751c0).
|
||
- `/scripts/deploy-remote.sh` — The pending pass now prunes stale `pending` state entries for retired services into `~/.AGENTTRASH` and keeps going (a stale `frank-default-sightsound` entry made every `--pending` run exit 2) (02a7876).
|
||
- Fleet audit — Every vhost classified against the guard policy (no new exposure); `systemd-hermes` logged as a boot-persistence gap (running container, no quadlet); stray stopped `test-vllm` container on dankserver held for approval.
|
||
|
||
## 2026-09-16
|
||
- `/scripts/bench-batch-ubatch.sh` — New harness: one `llama-bench` invocation per cell (a test that cannot allocate `exit(1)`s, taking a whole sweep with it), per-cell VRAM sampler, service stopped/started from a trap; must run from a transient unit (`LimitMEMLOCK=infinity` is load-bearing) (87845fc, 77a07ff).
|
||
- `/scripts/frank-medium.env` — `UBATCH_SIZE` 1536 → 512 (prefill peaks at 512: 1193 vs 1078 t/s at 2048; 3.71 GiB cheaper) and `TENSOR_SPLIT` 36,29 → 35,30 (ubatch buffers cost ROCm1 ~2× what they cost ROCm0) (a11fae0); `tests/frank_medium_unit.bats` balance seam re-fitted as a shape assertion.
|
||
- `/serverAppConfigs-docs/LLM/frank-medium-batch-ubatch.md` — New page: the sweep, the logical-batch irrelevance, the amdgpu settle-after-kill trap.
|
||
- `/dankclaw/frank-dragoon/quadlet/frank-dragoon.container` — halogen restored as frank-dragoon, retuned: ctx 262144 → 200192, pool 401408 → 400384, slots 3 → 2, max-tok 32768 → 24576, host-reserve 15 → 20 GiB, vision tower unset (moved to frank-zealot); ledger 95.4 GiB held / 14.7 GiB spare, no clamp.
|
||
- `/scripts/frank-zealot.env`, `/scripts/frank-zealot.service`, `/scripts/serve-frank-zealot.sh` — New frank-zealot: mainline llama.cpp 37b53fd on the RTX 3090 serving ISTA-DASLab GSQ-RCO IQ3_S with the in-file MTP head (`draft-mtp`), BF16 projector, 131072 ctx, f16 KV, port 7771.
|
||
- `/scripts/deploy-remote.sh` — `deploy_llm_dragoon` flipped back to halogen with a `wait_port_free 7770` gate (the first cutover died on `Address already in use` two seconds after the old server exited); `deploy_llm_small` added.
|
||
- `/serverAppConfigs-docs/LLM/frank-zealot-qwen38-gsq-rco.md`, `/serverAppConfigs-docs/LLM/halogen-flash-server.md` — New / re-activated runbooks; `frank-dragoon-ik-uncensored.md` marked retired.
|
||
- `/serverAppConfigs-docs/LLM/x4-pair-experiment.md` — New page: 3090 + Strix Halo over the x4 link. iGPU-only Vulkan matches halogen's ~25 tok/s with the 3090 idle but commits 116 of 121 GiB (throughput parity, memory bankruptcy); per-tensor cross-device placement is killed by `-ot` itself (`has_tensor_overrides()` → `pipeline_parallel` → `n_copies == 1`); the first "CUDA + Vulkan cannot serve one model" conclusion was a llama-bench parsing artifact (`-dev A,B` is a SWEEP, not a list — correct `-dev A/B -ts x/y` gives 27.55 / 27.23 tok/s, +7 % decode / +16 % prefill vs iGPU-only).
|
||
- Incident — The rootless podman pause process died (trigger NOT established; no OOM kill in the kernel log): every `podman` command failed, all 28 quadlets looped, `podman system migrate` panics in 5.7.0; a reboot repaired it. The reboot exposed `hermes.franksdank.cloud` 502: the Caddyfile dialed `192.168.0.143:8787` (a host bind, not a published port) — fixed to `host.containers.internal:8787`, and the seam that had accepted any numeric upstream was strengthened.
|
||
- Host — Model cache cleared on operator instruction (298 → 895 GB free) and re-fetched: halogen's sidecars came back LARGER (in-place versioning — a stale copy is invisible from the file name); Q4_K_S vs Q4_K_M measured +2–3 % with ~5 GiB more host RAM; IQ4_XS fetched as the next comparison payload; `halogen-flash-server.md` gained a "Refreshing the weights" note.
|
||
|
||
## 2026-09-15
|
||
- `/scripts/frank-medium.env`, `/scripts/serve-frank-medium.sh`, `/scripts/frank-medium.service` — frank-medium: the 3090 left Dankserver, two R9700s (gfx1201) in; runtime charlie12345/ROCmFPX c49ebdb with its self-contained ROCm 7.14 toolchain (bypassing the host's broken mixed ROCm); kingjones Q8_0 AGENT + Q4_0 MTP head; layer split chosen by measurement (1991/18.8 t/s vs tensor 271.8/26.3 — tensor's decode win costs a 7× prefill collapse with no RCCL); draft pinned to ROCm1; frank-small retired (`retire_root_llm_small()` archived the CUDA-era unit).
|
||
- `/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md` — New runbook; `freetoken-frank-small.md` and `kingjones-q8agent-rocmfpx.md` marked retired/superseded; AGENTS.md, indexes, System Specs, deployment/Hermes/OpenWebUI/Plex/Stack pages updated.
|
||
- Host — Purged 167 stale AMD-repo ROCm 6.3.3 packages, removed the radeon apt pin and the `ld.so.conf` hijack (staged in `~/.AGENTTRASH/20260915-rocm633-cleanup/`).
|
||
- `/scripts/frank-dragoon.service`, `/scripts/serve-frank-dragoon.sh` — frank-dragoon cut to ik_llama.cpp (halogen quadlet retired to `~/.AGENTTRASH/20260915-ik-swap/`); overnight decode campaign: 3.8–4.3 t/s across every safe configuration — the wall is ~100 CPU↔GPU section launches per token, not compute; mainline 37b53fd runs 1.0 t/s; the GGUF has no MTP head; 84 GiB of experts cannot fit 24 GiB VRAM (0c1e10a, 795b0b3).
|
||
- `/serverAppConfigs-docs/LLM/frank-dragoon-ik-uncensored.md` — New runbook with the full evidence chain.
|
||
|
||
## 2026-09-14
|
||
- `/dankclaw/compose.yaml`, `/dankclaw/.env.example` — New `hermes-webui` service (Hermex): `bootstrap.py --foreground` in the agent image, shares `/home/dank/hermes-data`, loopback-only publish 127.0.0.1:8787, password-gated via `HERMES_WEBUI_PASSWORD`.
|
||
- `/scripts/deploy-plan.py`, `/scripts/deploy.sh`, `/scripts/deploy-remote.sh` — `hermes-webui` registered as an isolated Dankclaw service in all three inventories.
|
||
- `/tests/hermes_webui.bats` — New seam (entrypoint, loopback publish + auth, shared volume, healthcheck, planner narrowing, deployer dispatch).
|
||
- `/serverAppConfigs-docs/Dankclaw/Hermes-WebUI.md` — New page; remote access = Termius local forward over the existing router :22 port forward.
|
||
|
||
## 2026-09-12
|
||
- `/dankclaw/marreta/quadlet/` — New Marreta content-cleaning proxy (Laravel) + Lightpanda browser quadlet pair, both joining gluetun's netns so browser fetch egress exits the ProtonVPN tunnel; `NGINX_HTTP_PORT=8085` (SearXNG holds 8080), gluetun publishes 8082:8085; guard chain marreta.franksdank.cloud → Anubis → authentik (91e9410).
|
||
- `/tests/marreta_quadlet.bats` — New seam (17 contracts); deploy-plan.py: marreta joins the gluetun netns family.
|
||
- `/serverAppConfigs-docs/Dankclaw/Marreta.md` — New page (incl. the `base64:`-prefixed APP_KEY gotcha).
|
||
- Live — authentik ProxyProvider `marreta` (pk 13) + Application link: the outpost's config API filters `application__isnull=False`, so a provider with no linked Application is never loaded (the login-wall root cause); a null `invalidation_flow` crashed the Rust outpost's serde and took down EVERY provider on the outpost; regression spec `16-marreta.spec.ts`.
|
||
|
||
## 2026-09-04
|
||
- `/frankbook-pro/` — *arr fleet consolidated on FrankBook-Pro as the sole deployment (dankserver compose copy removed, `/opt` config roots trashed); all six containers share gluetun's netns — all egress exits the ProtonVPN tunnel by construction; the arr UI ports moved onto the gluetun unit; qB download clients repointed `10.88.0.1:8080` → `127.0.0.1:8080`.
|
||
- `/tests/frankbook_quadlet.bats` — New seam: every arr+qB unit joins gluetun's netns and publishes no port; gluetun publishes the fleet's whole port set.
|
||
- `/dankserver/` — *arr stack reactivated behind six own-auth TLS vhosts (the root-owned `/opt/<svc>/config` dirs that crashed every app at logger init are gone); documented the verified rootless identity model: container root (PUID=0) maps to host uid 1000 by default, and `userns_mode` must NEVER be added (podman-compose runs services in pods; `--userns` + `--pod` is rejected).
|
||
- `/dankclaw/Caddyfile`, `/dankclaw/pihole/addn-hosts` — Sonarr moved to the admin side: the unguarded public vhost + post-auth handler retired, Pi-hole A record dropped — no WAN path at all; homepage card repointed at 192.168.0.112:8989.
|
||
- `/tests/e2e/` — Specs 11-admin/12-crawl de-hardcoded (expected open-link counts derive from the live services.json registry).
|
||
|
||
## 2026-09-03
|
||
- `/serverAppConfigs-docs/` — Full fact-check sweep of every page against the live hosts (read-only SSH), quadlet units, and the Caddyfile; surgical edits only (docs-only change = deploy no-op): Dankserver core rewritten to live quadlet facts, the Caddy guard chain documented end-to-end, Gitea data roots + UID gotcha corrected, `Prowlarr.md`/`prowlarr.md` deduped into one canonical page, archive LLM pages reworded from current-tense to dated historical framing.
|
||
|
||
## 2026-09-01
|
||
- `/frankbook-pro/gluetun/`, `/frankbook-pro/qbittorrent/` — WireGuard credentials provisioned (second Proton config, PA-12; one config = one keypair — two hosts on one config bounce the server-side endpoint and kill SearXNG); egress verified on the Proton exit; qB first-seed bug fixed (`WebUI\Password_PBKDF2`, not the legacy `WebUI\Password` key qB ≥ 4.2 ignores); `DefaultSavePath` → `/Volumes/COMPLETE/Finished Torrents`; download clients created in all four arrs; Sintel e2e through the tunnel at ~19 MB/s.
|
||
- `/serverAppConfigs-docs/Infrastructure/FrankBook-Pro.md` — Rewritten to the active-tunnel reality (VM placement, WG gate, socat LaunchAgents, PBKDF2 seed); RDAP proved the "custom WG server" is a ProtonVPN Seattle server (full 65535-port scan: no admin surface anywhere).
|
||
- `/dankclaw/authentik/quadlet/authentik-outpost-ldap.container` — New LDAP outpost for Jellyfin (amd64 single-platform digest-pinned — podman 5.7.0 refuses the tag's nested manifest list); authentik objects provisioned via `ak shell` ORM; live bind tests proved two rules: the token intent must be `verification`, and the provider needs the `search_full_directory` object permission.
|
||
- `/serverAppConfigs-docs/Infrastructure/Authentik.md`, `/serverAppConfigs-docs/Dankclaw/Jellyfin.md` — LDAP-outpost section (objects table + rules) and the LDAP-Auth 23.0.0.0 plugin section (`LdapSearchAttributes` is comma-separated; username attribute `cn`).
|
||
- `/scripts/frank-zealot.env`, `/scripts/serve-frank-zealot.sh` — Two compressed-tensors FP8 swaps attempted on FreeToken and both failed to load (ornith-1.5: expert naming; HauhauCS aggressive: block scales do not broadcast in `_dequant_fp8_weight`); reverted to Huihui FP8 within the hour; `freetoken-frank-small.md` quant matrix updated (modelopt block-fp8 is the only FP8 path that loads); the deploy health gate passed for BOTH failed loads — a swap is only proven by the journal + a real completion.
|
||
- `/dankserver/{romm,jellyfin,calibre-web,copyparty}/quadlet/` — After a hard crash left every bare compose container dead at boot, the media stack standardized to quadlet: six units shipped + cutover handlers (retire bare containers, rescue RomM's anonymous volume, merge-move `/opt/<svc>` → `/home/dank/Containers/<svc>`); v1 generator quirks fixed (`Command=`→`Exec=`, `Device=`→`AddDevice=`, `HealthCmd=` space-splitting, the `HOME=/root` EnvironmentFile trap).
|
||
- `/dankclaw/` — Data roots of gitea, authentik, and the hermes trios moved to `/home/dank/Containers/<svc>/` while running (rename is inode-bound — zero downtime); `deploy_hermes` ordering fixed (generated units reload AFTER the unit files are copied).
|
||
- `/dankclaw/seerr/quadlet/seerr.container` — New Seerr v3.4.1 request manager: loopback-only 127.0.0.1:5055, guarded vhost + outpost provider `seerr-proxy`; `/tests/seerr_quadlet.bats` new seam; `/serverAppConfigs-docs/Dankclaw/Seerr.md` new page.
|
||
- `/scripts/deploy-remote.sh` — frankbook handler: the credential-gate SKIP state now propagates Mac-side before restart+verify; qB first-boot seed writes a PBKDF2 password + `HostHeaderValidation=false` + the four categories; full 7-service deploy run.
|
||
- `/frankbook-pro/qbittorrent/quadlet/qbittorrent.container` — qB pinned `4.6.7` (5.x's `204` login breaks Readarr's retired client; 5.x's conf rewrite leaves a PBKDF2 line 4.6.7 rejects).
|
||
- `/serverAppConfigs-docs/Dankserver/{RomM,Jellyfin,Calibre-Web,Copyparty}.md`, `/serverAppConfigs-docs/Architecture/Storage Policy.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — Updated to the quadlet reality, the Containers layout, and the two ownership regimes.
|
||
|
||
## 2026-08-30
|
||
- `/scripts/frank-npu.env`, `/scripts/serve-frank-npu.sh`, `/scripts/frank-npu.service` — New `frank-npu` on Dankclaw: FLM v1.0.1 serving the whole local registry (36 tags) on 52626; first deploy crash-looped (`/dev/accel/accel0` err=-19) — root cause: the XDNA2 SVA bind fails whenever the AMD IOMMU is off; fixed with `iommu=pt` + reboot; vision + audio towers proven end-to-end; health probe is `/v1/version` (FLM has no `/health` route) (c3ed3a6, cf0c541).
|
||
- `/serverAppConfigs-docs/LLM/frank-npu.md` — New runbook; AGENTS.md fleet table + the IOMMU/SVA hard requirement.
|
||
- `/dankclaw/frank-dragoon/quadlet/frank-dragoon.container` — frank-dragoon: swapped to `kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` on the pinned `kingjones30/ROCmFPX` fork (fa-on works UNPATCHED — the QSA F32 assert was agentionai-specific); ~225 s load, real 111k-token prefill at 213–232 t/s.
|
||
- `/scripts/frank-zealot.env` — frank-zealot (FreeToken): model swapped to `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (native block-fp8); ~40 tok/s verified.
|
||
- `/serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md` — Reactivated as the large runbook; `agentionai-fast-imatrix.md` marked RETIRED; `freetoken-frank-small.md` model sections updated; LLM index + AGENTS fleet tables refreshed.
|
||
- `/serverAppConfigs-docs/Dankserver/prowlarr.md`, `/serverAppConfigs-docs/Dankserver/qbittorrent.md` — Rewritten: the arr↔Prowlarr feed contract (`/{indexerId}/api`, the global key — the hand-rolled `/torznab/` paths were Forms-auth catch-alls returning 302), TV + music chains proven live, first real grab (The Dark Knight 2160p, imported), Prowlarr auto-sync broken by version skew, qB persistent credentials (`WebUI\Password_PBKDF2`, host-local 0600), Readarr parked (abandoned upstream; its client cannot authenticate against qB 5).
|
||
|
||
## 2026-08-29
|
||
- `/dankclaw/Caddyfile` — `:8016` is now a uniform whole-request `reverse_proxy authentik-server:9000` preserving `Host`; the per-vhost redirect/flow design and stale comments deleted (29f9a86).
|
||
- `/dankclaw/anubis/policy.json` — ALLOW `^/outpost\.goauthentik\.io/` so login/transport paths are reachable before the challenge dance (2a72efa); the policy bind-mount pins the file inode — a git checkout does NOT reach the running distroless container, so a `--service anubis` deploy restart is mandatory after policy edits.
|
||
- `/tests/authentik.bats` — Meta-header test replaced by the upheld contract (Host preserved; no absolute cross-host `next=` anywhere in the Caddyfile).
|
||
- `/serverAppConfigs-docs/Infrastructure/Authentik.md`, `/serverAppConfigs-docs/Infrastructure/Anubis.md`, `/serverAppConfigs-docs/Architecture/Network Architecture.md` — Rewritten to the live-verified mechanism: the 2026.8 embedded outpost routes solely on the original `Host` vs each provider's `external_host` (meta-header routing disproven live); proxy providers + applications provisioned for search/hermes/prometheus/romm/files; null `authorization_flow`/`invalidation_flow` starve the outpost's config fetch; the public zone gained the `auth` A record via the Porkbun API (ee1e038).
|
||
|
||
## 2026-08-26
|
||
- `/dankclaw/Caddyfile` — Both Plex vhosts proxy to PMS over a TLS upstream (`insecure_skip_verify`); PMS drops plain-HTTP connections from non-loopback sources (empty-body 502s).
|
||
- `/dankclaw/gitea/quadlet/gitea-caddy.container` — Standalone Gitea Caddy quadlet retired; web proxy ownership moved to canonical Caddy; deploy-plan/deploy.sh/deploy-remote.sh + tests updated for the migration.
|
||
- Incident — Post-reboot: Caddy (a bare compose container — `restart: unless-stopped` does not survive shutdown's explicit stop) and Pi-hole (quadlet crash-loop, restart counter 1645) were both down. Root cause: `net.ipv4.ip_unprivileged_port_start` reset to 80 by a later-sorted sysctl drop-in AND a stale line in `/etc/sysctl.conf`, which systemd-sysctl applies AFTER all of `sysctl.d/`.
|
||
- `/AGENTS.md` — New MANDATORY "Boot Persistence & Self-Healing" section.
|
||
- `/dankclaw/caddy/quadlet/caddy.container`, `/dankclaw/caddy/env.example` — Caddy is now a boot-persistent quadlet (joins `dankclaw_default`, `Pull=never`, env from `/home/dank/Containers/caddy/env`); removed from compose.yaml.
|
||
- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path).
|
||
- `/tests/caddy_quadlet.bats` (new) + caddyfile/deployment_flow/pihole seams — quadlet boot persistence, self-healing, network, env, and sysctl coverage.
|
||
- `/serverAppConfigs-docs/Infrastructure/Caddy.md`, `/serverAppConfigs-docs/Infrastructure/Pi-hole.md`, `/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md`, `/serverAppConfigs-docs/Dankclaw/Gitea.md`, `/serverAppConfigs-docs/Architecture/Deployment Flow.md` — quadlet runtime facts, the 2026-08-26 incident, the sysctl-conflict failure mode, canonical Caddy ownership.
|
||
|
||
## 2026-10-01 — frank-archon RESTORED to fp8 3 x 262144 (bf16 boot fails)
|
||
|
||
bf16 (--kv-cache-dtype auto) at 2 x 262144 did not stay up, with OR without
|
||
--expert-precise-memory:
|
||
|
||
- with precise: NRestarts 1->3, each attempt ~6.7 min then
|
||
`Deactivated successfully` (API server drains after an EngineCore fatal
|
||
error) — cause not captured, the serve script rm'd the container.
|
||
- without precise (default 7.60 GiB overhead, arena 14.3 GiB): NRestarts
|
||
climbing, container `Exited (1)`, and the journal finally showed the real
|
||
error:
|
||
`RuntimeError: Worker failed with error 'Constraints violated
|
||
(L['query_start_loc'].size()[0])! You marked L['query_start_loc'].size()[0]
|
||
as dynamic but your code specialized it to be a constant (3)'`
|
||
— a torch.compile/dynamo dynamic-shape constraint violation, NOT an OOM.
|
||
`~/.cache/vllm` holds only `modelinfos` (16K) and each `podman run` gets a
|
||
fresh container filesystem, so a stale compile cache is ruled out: the
|
||
conflict is produced by this configuration's own compile path.
|
||
|
||
Differences from the last shape measured serving (fp8 3 x 262144, MNBT 2048,
|
||
KV 6.1e9 -> 792,069 tokens / 3.02x / arena 6,248 / 3.59 GiB free): KV dtype
|
||
(fp8 -> bf16) and SEQS (3 -> 2). Both are candidates for the dynamo trigger;
|
||
`--enforce-eager` is the bypass that would sidestep the constraint path.
|
||
|
||
Restored: fp8 3 x 262144, MNBT 2048, KV_BYTES 6,100,000,000, no precise flag.
|
||
Next: bisect bf16 vs SEQS in a window (bf16 + SEQS=3 isolates the dtype).
|
||
|
||
## 2026-10-01 — frank-archon: --expert-precise-memory + bf16 = boot crash loop
|
||
|
||
**Reverted.** The two changes Frank asked for were applied together and the
|
||
result crash-looped: `NRestarts` climbed 1 -> 2 -> 3, each attempt running
|
||
~6.7 minutes then logging `Deactivated successfully` (vLLM's API server drains
|
||
and exits 0 after an EngineCore fatal error, so `podman wait` returns 0; the
|
||
new `Restart=always` turned a silent death into a visible loop).
|
||
|
||
The budget arithmetic explains it without needing the error text, because the
|
||
arena is DEVICE-resident (the fp8 3-window shape's plan predicts 28.27 GiB and
|
||
that is exactly what `rocm-smi` measured):
|
||
|
||
| line | bf16 2 x 262144 + precise | fp8 3 x 262144 |
|
||
|---|---|---|
|
||
| non-expert weights | 6.28 | 6.28 |
|
||
| pinned pool | 6.80 | 5.68 |
|
||
| CUDA graphs | 0.55 | 0.55 |
|
||
| arena | 16.65 (precise: overhead 5.25) | 14.31 (default: overhead 7.60) |
|
||
| non-torch | ~1.45 | ~1.45 |
|
||
| **total** | **31.73 of 31.86 GiB** | 28.27 (measured) |
|
||
| free at boot | **~0.13 GiB** | 3.59 |
|
||
|
||
`--expert-precise-memory` hands the 2.35 GiB compile/graph-capture headroom to
|
||
the arena, and bf16 simultaneously adds 1.08 GiB of pinned pool; together they
|
||
put graph capture ~0.13 GiB from the ceiling, where every transient allocation
|
||
dies. So the flag is dropped and bf16 2 x 262144 ships with the default
|
||
overhead: arena 14.31 GiB, ~2.4 GiB free at boot.
|
||
|
||
**Lesson:** the registry's "compile headroom" is not slack to be reclaimed by
|
||
default — it is the budget for the phase that allocates last. Reclaim it only
|
||
by paying for it elsewhere (smaller pool, fewer sessions, or a shorter context).
|
||
|
||
## 2026-10-01 — frank-archon: 2 bf16 windows + --expert-precise-memory
|
||
|
||
Frank's final shape for the night. Two changes at once, both measured in
|
||
principle before the boot:
|
||
|
||
- **KV is bf16 (`--kv-cache-dtype auto`), two full-size windows (2 × 262144).**
|
||
bf16 measures **13,839 B/token** (from the bf16 3×241664 boot:
|
||
11,047,010,304 B -> 798,280 tokens), decomposing as **12,276 attention +
|
||
1,563 mamba/side caches** against fp8's **6,138 + 1,563 = 7,701**. So
|
||
524,288 tokens needs 7.256e9 B; pinned at 7,300,000,000 (~527,500 tokens,
|
||
3,206 margin). Cost: the pool goes 5.68 -> 6.76 GiB per rank.
|
||
- **`--expert-precise-memory`.** The registry's default overhead for this
|
||
architecture is 7.60 GiB = the fitted 5.25 GiB (gfx1201 TP2
|
||
nbt4096/seqs16/len262144) + 2.35 GiB of compile/graph-capture headroom.
|
||
The flag selects the fitted figure, which moves only the plan's `room` term
|
||
— arena 6,248 -> ~7,278 slots (~1,025 more, at 2.344 MiB/slot) — at the
|
||
cost of boot-time capture slack. The overhead formula adds 0.22 GiB per
|
||
1024 nbt above 4096, so MNBT 2048 pays nothing.
|
||
|
||
Budget note worth keeping: the arena is `(budget - non_expert_weights -
|
||
kv_reserve - graphs - overhead) // expert_bytes`, where `kv_reserve` is the
|
||
planner's OWN one-max-model-len reserve (2.48 GiB at TP2) and does NOT scale
|
||
with SEQS. The pinned `--kv-cache-memory-bytes` therefore does not shrink the
|
||
planned arena; it comes out of free VRAM instead.
|
||
|
||
## 2026-10-01 — frank-dragoon: Q4_K_XL payload + 4 x 163840 sessions
|
||
|
||
Frank's call. Payload swapped from ajgazin's **abliterated** Swift-1.5
|
||
UD-Q8_K_XL (29.3 GiB) to **unsloth/Qwen3.8-27B-GGUF UD-Q4_K_XL** (16.35 GiB,
|
||
sha256 3f227079…b01e, vanilla/Apache-2.0 — the endpoint is no longer
|
||
uncensored). New fetcher `scripts/download-frank-dragoon-q4.sh`; the DFlash2
|
||
drafter and its fetcher are unchanged.
|
||
|
||
Shape: **`--sessions 4 --context 163840`** with unquantized KV. gufo has no
|
||
KV-quantization lever at all (`execution_policy.hpp` pins `kv_cache_storage`
|
||
to `kFp16`), so "unquantized" is inherent.
|
||
|
||
KV arithmetic (why this fits where one 262144 session used to sit):
|
||
f16 KV costs **64 KiB/token** — 16 full-attention layers (the hybrid is
|
||
`full_attention_interval: 4` over 64 layers) x 4 KV heads x 256 head_dim x
|
||
2 (K,V) x 2 bytes; the other 48 layers are linear-attention with a
|
||
per-SEQUENCE fp32 state (~144 MiB), not a per-token cache. 4 x 163840 =
|
||
655,360 tokens = 40.0 GiB of KV, and each session is exactly 10 GiB. Budget:
|
||
weights 16.35 + KV 40.0 + drafter 1.07 + linear state 0.56 + runtime ~4 =
|
||
~62 GiB against the 114 GiB GTT.
|
||
|
||
## 2026-10-01 — e2e gate flakiness (12-crawl) — do not panic at the banner
|
||
|
||
`specs/12-crawl.spec.ts:23` ("every user open link reaches its app") failed the
|
||
deploy gate twice and then passed cleanly on a manual re-run:
|
||
- run 1 (deploy): `Expected "plex.franksdank.cloud"` / `Received "app.plex.tv"`
|
||
— Plex redirects its web UI to the hosted app; the same failure appeared on
|
||
the previous deploy too.
|
||
- run 2 (manual): `page.goto: net::ERR_ABORTED at https://www.franksdank.cloud/`
|
||
— the homepage answered `http=200` in 40 ms from the dev box at the same
|
||
moment, so this is a browser-level flake.
|
||
- run 3 (manual): **33 passed**, exit 0.
|
||
|
||
So a red gate on this spec alone is NOT proof the fleet is broken, and the
|
||
deployer's `DEPLOY INCOMPLETE` banner can be raised by a flake. Re-run
|
||
`tests/e2e/run.sh` before treating it as real; the Plex redirect is the one
|
||
stable underlying issue and it lives in Plex's web-client/image config, not in
|
||
any repo service.
|
||
|
||
## 2026-10-01 — frank-archon bf16 no-spec cutover
|
||
|
||
- `/scripts/frank-archon.env` — bf16 no-spec cutover, FINAL shape after the
|
||
night ladder: MAXLEN **241664** x SEQS 3, KV_BYTES **10,184,000,000**/rank,
|
||
KV_DTYPE auto (bf16), MNBT 1024, GATHER CHUNKS=32/LANES=64, UTIL 0.98;
|
||
FRANK_ARCHON_SPEC deleted (no spec). Engine measured at the winner:
|
||
"KV cache size: 732,359 tokens, Maximum concurrency for 241,664 tokens per
|
||
request: 3.03x".
|
||
- **The night ladder (Frank's method).** 262144 OOMed at CUDA graph capture
|
||
(`Tried to allocate 76.00 MiB ... 0 bytes free`, 30.43 GiB of 31.86 GiB
|
||
allocated by PyTorch) and 245760 OOMed identically; **237568 PASSED**
|
||
(KV 718,606 tokens) and the +4096 bonus step **241664 PASSED and is
|
||
PRODUCTION**. 241,664 is max possible on this box at util 0.98 — the
|
||
model's native 262,144 does not fit with a 3-seq contract. Every rung was
|
||
booted under a transient systemd unit at uid dank with
|
||
`LimitMEMLOCK=infinity` (the rootless bench probe masked the production
|
||
memory environment) with full engine-log capture.
|
||
- **Util stays 0.98.** The earlier 0.90 edit was wrong — lowering
|
||
`--gpu-memory-utilization` shrinks the engine's own budget rather than
|
||
buying slack; Frank forbade it and the winning ladder ran at 0.98.
|
||
- `/scripts/serve-frank-archon.sh` — `--speculative-config` line removed;
|
||
`--max-num-batched-tokens` and the `-e R4D_LRU_GATHER_CHUNKS/LANES`
|
||
container-env passthrough added; shape header rewritten.
|
||
- `/tests/frank_archon_unit.bats` — seam re-pinned: bf16-KV-present,
|
||
spec-absent-by-measurement, gather-32x64-present, MNBT-1024 tests; the
|
||
qualified-shape and flags tests flipped to the new env keys.
|
||
- Overnight redo campaign (prefetch OFF everywhere) tabled in
|
||
`/serverAppConfigs-docs/LLM/dankserver-qwen38-fp8-expert-offload-tuning.md`
|
||
with the 2026-10-01 cutover section: gather 32x64 +18.7%, MNBT flat
|
||
1024-4096, KV pool law (7,926 fp8 / 14,047 bf16 B/token), arena flat zone,
|
||
six-slot refusal, bf16 3x262144 boot failure root cause (CUDA OOM at
|
||
KV_BYTES 11.3e9; boots clean at 11.047e9 — captured from journald `-o json`).
|
||
- `/AGENTS.md` — fleet table frank-archon row rewritten to the bf16 shape.
|
||
- `/serverAppConfigs-docs/LLM/index.md` — fleet intro + mermaid node
|
||
updated out of the GLM era.
|
||
- **Client contract fixed (thinking + tools).** The live endpoint returned a
|
||
null `reasoning` field and dumped the thinking text INLINE in `content`
|
||
with a dangling `</think>` (measured: `'We need answer…Brief.\n</think>\n\n391'`)
|
||
— the malformed thinking block clients choke on — because the deployed argv
|
||
had no `--reasoning-parser`. The GLM rollback (`d91379d`) had dropped all
|
||
three contract flags and the qwen rollback inherited the gap. Restored from
|
||
`ecaad36`'s values, confirmed against this fork's registries:
|
||
`--reasoning-parser qwen3`, `--enable-auto-tool-choice
|
||
--tool-call-parser qwen3_xml`, `--default-chat-template-kwargs
|
||
'{"enable_thinking": true, "preserve_thinking": true}'`. The seam's
|
||
"NO tool dialect" test is flipped to a PRESENT test.
|
||
- **Final shape (Frank's call): 5 seqs x the full native 262144 window at
|
||
fp8 KV.** 6 x 241664 at fp8 died (`Tried to allocate 10.70 GiB`, 9.18 GiB
|
||
free) — six full-length contexts do not fit at any KV dtype — so the
|
||
contract moved to five FULL windows: 5 x 262144 fp8 PASSED on the first
|
||
rung, KV_BYTES 10,388,766,720/rank, measured **1,350,182 tokens /
|
||
5.15x concurrency** (39,462 tokens of margin). Seam re-pinned; the bf16
|
||
three-context shape (241,664, KV 732,359 / 3.03x, measured earlier the
|
||
same night) is the documented max at 3 contexts.
|
||
- Harness `/home/blank/.omp/agent/models.yml` — contextWindow back to 262144
|
||
with the five-window fp8 shape (compaction gap 183,501 + 36,864 = 220,365
|
||
<= 262,144) and
|
||
`preserve_thinking` moved from a bare top-level body field to
|
||
`chat_template_kwargs` (the form vLLM actually reads).
|
||
- **Runtime-survivability correction: five slots -> FOUR.** 5 x 262144 fp8
|
||
passed every boot gate (KV 1,350,182 tokens / 5.15x) but left 0 bytes free
|
||
at runtime; a real client request (995-token prompt, `max_tokens=261149`,
|
||
sampling params straight from models.yml) killed Worker_TP1 with
|
||
`torch.OutOfMemoryError: Tried to allocate 50.00 MiB ... 0 bytes free` and
|
||
the engine shut down. Boot is not proof of runtime survivability — after a
|
||
knife-edge shape change, send a large-`max_tokens` request before calling it
|
||
done. FOUR slots x 262144 fp8 (KV_BYTES 8,311,013,376/rank) returns ~1.93
|
||
GiB per rank for activations/graph capture. Frank's call 2026-10-01.
|
||
- `/scripts/frank-archon.service` — **Restart=always, not on-failure** (plus
|
||
`StartLimitIntervalSec=0` and `OOMScoreAdjust=-500`). A runtime worker fault
|
||
makes vLLM's API server exit CLEAN, so ExecStart returned 0, systemd logged
|
||
"Deactivated successfully", and on-failure NEVER fired: the endpoint stayed
|
||
down until a human noticed. An explicit `systemctl stop` still wins, so
|
||
bench windows and deploys are unaffected.
|
||
- **Final shape (Frank's call, 2026-10-01): 3 x 262144, pool at the FLOOR,
|
||
MNBT 2048, rest to the arena.** MAXLEN 262144 x SEQS 3 = 786,432 tokens of
|
||
contract; measured fp8 costs 7,698 B/token, so 6.054e9 B is the pool floor
|
||
and the deployed 6,100,000,000 B leaves a ~5.9k-token margin while handing
|
||
the remaining ~2.2 GiB (versus the 4-slot shape) to the expert arena — the
|
||
decode-critical cache the fork sizes from the remainder. MNBT 2048 costs the
|
||
arena nothing (the sweep measured 6,499 slots at every rung <= 4096).
|
||
- **DCP investigated and closed** (see the parallelism survey in the runbook):
|
||
`--decode-context-parallel-size 2` is rejected at boot — DCP for GQA requires
|
||
TP > the model's KV heads (2), i.e. TP >= 4, i.e. four GPUs. Not a tuning
|
||
knob on this box.
|
||
- Production bring-up: the shape boots and serves on :7770 — engine log
|
||
verified (KV cache size, concurrency, MNBT 1024, gather 32x64) plus live
|
||
contract probes (reasoning split, tool_calls). Rollback target for the
|
||
2026-09-29 fp8+MTP4 shape: git tag `frank-archon-fp8-mtp-20261001`.
|
||
|
||
## 2026-10-01 — frank-dragoon cutover: gufo -> vLLM TheRock, MiMo-V2.6 BF16 + cross-model MTP
|
||
|
||
- **Question that started it:** can a Qwen3.5 MTP drafter attach to
|
||
frank-zealot, and how does vLLM run the FULL BF16 MiMo checkpoint? Answer 1:
|
||
not on frank-zealot — SGLang reads the MTP head from the TARGET checkpoint
|
||
and MiMo ships ZERO `mtp.*` tensors (base and INT8 alike). Answer 2: the
|
||
digest-pinned `kyuz0/vllm-therock-gfx1151` image (vLLM 0.30 / torch 2.11 /
|
||
TheRock ROCm 10) boots BF16 MiMo on the gfx1151 iGPU in ~200-240 s, and
|
||
vLLM reads the draft from a SEPARATE path — so cross-model MTP from
|
||
`Qwen/Qwen3.5-9B` (the distill's source, identical 4096/248320/32-layer
|
||
geometry) works. Frank's call: ship that as frank-dragoon.
|
||
- **Bench window (frank-dragoon stopped, one session, temp 0, 28,647-token
|
||
prefill / 512 decode):** base 17,876 t/s prefill / 13.1 t/s decode (13.1
|
||
is ~94% of the LPDDR5x BF16 roof - decode here is bandwidth, not a bug);
|
||
+DFlash-8 7,976 / 17.4 (acceptance 59%/26% per position); +MTP-2 7,380 /
|
||
19.8 (71%/44%, token rate 57.4%). Deployed MTP-2: +51% decode. Both spec
|
||
arms pay ~55-59% of prefill. `num_speculative_tokens` beyond 2 unmeasured.
|
||
- **Measured numerics law:** spec arms do NOT reproduce the no-spec greedy
|
||
text (bf16 batched verification reorders reductions; near-tie argmaxes
|
||
cascade). vLLM spec decode is distribution-preserving, not
|
||
bitwise-greedy-preserving.
|
||
- **Client-contract delta wired:** vLLM emits `reasoning`, not
|
||
`reasoning_content` - `models.yml` frank-dragoon block now sets
|
||
`reasoningContentField: "reasoning"`, and the extraBody moved to the
|
||
Qwen3.5-family recipe (0.7/0.8) with `chat_template_kwargs`
|
||
(a top-level `preserve_thinking` is ignored by vLLM).
|
||
- Deployer gap found pre-deploy: the spec-draft gate demanded a single
|
||
`model.safetensors` (the dflash layout) and would have rejected the
|
||
deployed MTP arm (4 shards behind an index); fixed to accept both
|
||
layouts, gated by bats. `z-lab/Qwen3.5-9B-DFlash`'s "vLLM PR #40898
|
||
unmerged" card claim is FALSE - `vllm/v1/spec_decode/dflash.py` ships in
|
||
0.30 (verified in-image); the drafter stays on disk as the alternative.
|
||
- Host state: +38 GiB fetched (MiMo BF16 17.53 GiB, Qwen3.5-9B 18 GiB,
|
||
DFlash 2.58 GiB), disk down to ~26 GiB free - reclaim pass due. gufo
|
||
rollback pinned at git tag `frank-dragoon-gufo-q4kxl-20261001`; engine,
|
||
Q4_K_XL payload and DFlash2 drafter all remain on the host.
|
||
- First deploy attempt #2 crash-looped the container 51 times: the serve
|
||
script handed vLLM the HOST model path while the tree is mounted at
|
||
`/models` inside the container — transformers rejected it as a malformed
|
||
repo id. Fixed by converting to the container path (the spec draft was
|
||
already converted; the target was not), pinned by bats. Also fixed the
|
||
deployer's own draft gate (same single-file bug as the serve script's).
|
||
- Reclaim (Frank approved): Swift-1.5 Q8_K_XL payload (30 G, refetchable via
|
||
`scripts/download-frank-dragoon-swift.sh`) and the orphaned flux-2-klein
|
||
tree under `models/checkpoints` (23 G; ComfyUI mounts
|
||
`comfyui-data/models`, not this) purged via
|
||
`~/.AGENTTRASH/20261001-frank-dragoon-reclaim/` — 52 G freed. KEPT the
|
||
1.1 G `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` (it is the gufo rollback's
|
||
drafter, not a remnant).
|
||
|
||
## 2026-10-02 — frank-dragoon cutover: vLLM MiMo -> ROCmFPX Q8_0_ROCMFPX_AGENT + MTP
|
||
|
||
Frank's order was the highest quality of
|
||
`kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF` on one full context, "whatever
|
||
runtime it requires". The answer is forced by the file: `Q8_0_ROCMFPX_AGENT.gguf`
|
||
carries tensor type 103 (`GGML_TYPE_Q8_0_ROCMFPX`, ftype 115), which stock
|
||
llama.cpp, gufo and every other loader on the fleet cannot read — only the
|
||
kingjones30/ROCmFPX fork can, so the runtime is pinned to commit
|
||
`36e9acd40e10a87cd3c3ef8ec734668757dc8520` and built by
|
||
`scripts/build-rocmfpx.sh`. Shape: 1 session x 262,144 (native), f16 KV, the
|
||
bundled BF16 mmproj, and MTP from the repo's own `mtp-Qwen3.8-27B-Q4_0.gguf`
|
||
(n-max 4, `--spec-draft-device ROCm0`).
|
||
|
||
- **MTP is the whole performance story: 17.0-17.1 t/s with, 7.60-7.66 without**
|
||
(+123 %), one session, temp 0, production stopped per arm, acceptance 0.462 /
|
||
mean length ~1.86, at an 8-15 % prefill tax (325-351 vs 381 t/s). The AGENT
|
||
routing variant was chosen over the card's plain Q8_0 because it keeps 187
|
||
output-head tensors at plain Q8_0 instead of 2, which measured MTP acceptance
|
||
0.953 vs 0.911.
|
||
- Steady GTT occupancy 63.1 GiB of the ~114 GiB aperture; boot to healthy 16 s.
|
||
- Rollback: git tag `frank-dragoon-gufo-q4kxl-20261001`. The retired vLLM-era
|
||
fetchers moved to `old/`. Runbook
|
||
[[frank-dragoon-rocmfpx-q8agent-mtp]].
|
||
|
||
## 2026-10-02 — frank-zealot: Q5_K_M, then three contexts + vision, sized by two boots
|
||
|
||
Three orders in one day on the same endpoint: `bartowski/…Q5_K_M` on llama.cpp
|
||
with two contexts ("whatever context size fits the 3090"), then **three
|
||
contexts** "as much as can fit while keeping overhead open space to 2-3gb on
|
||
the 3090", then **"frank-zealot should have vision"**.
|
||
|
||
- **Final shape: `CTX_SIZE=540672 PARALLEL=3` = 3 x 180,224**, f16 K + q8_0 V
|
||
under flash attention, `--mmproj` vision, all layers on the 3090. Measured
|
||
**21,684 MiB used / 2,441 MiB free** of 24,576 — inside the 2-3 GiB band.
|
||
- **The pool came from two boots, not a formula.** 2 x 262,144 with no
|
||
projector: 20,190 MiB used, KV 12,800 => non-KV **7,390 MiB**. 3 x 196,608
|
||
with the projector: 22,972 used, KV 14,113 => non-KV **8,859 MiB**. The
|
||
difference prices the vision tower at **~1.4 GiB for a 918 MB file** — the
|
||
encoder's own buffers ride with it. The first 3-slot deploy (3 x 196,608)
|
||
booted healthy but left only 1,153 MiB free, which is how the band got
|
||
violated and why the seam now prices non-KV at the measured 9 GiB and
|
||
REJECTS that shape.
|
||
- **Three contexts cost window, and that is the tradeoff Frank was told:**
|
||
180,224 tokens per conversation, not the native 262,144. A longer prompt is
|
||
a hard `400 exceed_context_size_error`, so `models.yml` sets
|
||
`contextWindow: 180224`. Consequence at the time: frank-archon's then
|
||
183,501-token compaction request did not fit here with summary room —
|
||
resolved later the same day when frank-archon's window moved to 200,704
|
||
and its trigger fell to 140,492 (see the shape-D entry below).
|
||
- Measured (2026-10-02, this shape): decode **106.8 t/s**, prefill **3,814
|
||
t/s** (~5 % faster decode than 2 x 262,144 — shorter slot, shorter attention
|
||
scan), three concurrent requests answered in parallel in 1.7 s with a fourth
|
||
queued at 3.0 s, native `tool_calls`, `reasoning_content` populated,
|
||
`reasoning_effort: low` accepted, solid-red PNG answered **"Red."**
|
||
- **Vision contract gotcha:** `/v1/chat/completions` ignores the legacy
|
||
top-level `images` field — that request answered "Brown." for the same red
|
||
image, i.e. it generated from text alone. Only the `image_url` content part
|
||
reaches the model. `/completion` accepts `images` but this build still
|
||
requires `messages`.
|
||
- Deployer gates on the projector too (`LLM_SMALL_MMPROJ`), so a missing
|
||
mmproj cannot silently ship a blind endpoint; `MMPROJ_FILE=` (explicitly
|
||
empty) in the env file is the documented text-only opt-out and returns ~1.4
|
||
GiB to KV.
|
||
- Reclaim staged, **not purged, awaiting approval**:
|
||
`~/.AGENTTRASH/20261002-retired-llm-payloads/` (~59 GiB of superseded
|
||
payloads). Kept in place for rollback: `MiMo-Q6K/`, `MiMo-Q5KM/`,
|
||
`Qwen3.8-27B-UD-Q4_K_XL/` and `Qwen3.8-27B-UD-Q8_K_XL/` (the latter holds the
|
||
gufo DFlash2 drafter). Runbook [[frank-zealot-llama-mimo-q5km]].
|
||
|
||
## 2026-10-02 — frank-archon shape D: 3 x 200704, fp8, pool sized for 2 contexts
|
||
|
||
Frank's order: shrink frank-archon to a 200704-token window with about two
|
||
resident full-length contexts, cut the KV pool, and drop MNBT to 1024. The
|
||
first attempt (`30904ba`, 2 x 200704 + SEQS=2) **crash-looped**; the revert
|
||
went live as `e460bbb`, and the shape was then re-applied one law at a time on
|
||
top of the proven-good base, root cause first.
|
||
|
||
- **ROOT CAUSE: `--max-num-seqs 2` cannot compile this model. It was never OOM,
|
||
never MNBT, never the KV dtype.** Four single-variable boot probes against
|
||
the then-deployed shape (production stopped for each, candidate booted AS the
|
||
service):
|
||
|
||
| MAXLEN | SEQS | MNBT | result |
|
||
|---|---|---|---|
|
||
| 262144 | 2 | 1024 | **Engine core initialization failed** |
|
||
| 262144 | 2 | 2048 | **Engine core initialization failed** |
|
||
| 262144 | 3 | 1024 | healthy - 792,069 tokens, 3.02x, arena 6,248 |
|
||
| 262144 | 3 | 2048 | healthy - the then-deployed incumbent |
|
||
|
||
`torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints
|
||
violated (L['query_start_loc'].size()[0]) ... specialized it to be a constant
|
||
(3)`, raised from AOT `build_guards(strict_error=True)` during
|
||
`determine_available_memory -> profile_run -> _dummy_run`, ~7 min in.
|
||
`model_runner.py` sizes that input buffer `max_num_reqs + 1` and `_dummy_run`
|
||
sets `num_reqs = min(max_num_batched_tokens, max_num_reqs)`, so its width is
|
||
**SEQS + 1** for ANY `MNBT >= SEQS` - 3 at SEQS 2, 4 at SEQS 3, exactly the
|
||
constant in the message. MNBT only moves the compile RANGE ((1,1024) vs
|
||
(1,2048)), which is why the identical error appears at both rungs.
|
||
`--enforce-eager` bypasses the path but drops CUDA graphs. The 2026-10-01
|
||
bf16 2 x 262144 failure was this same error class; its message named "KV
|
||
dtype and SEQS" together and nobody bisected, so the dtype was blamed - it
|
||
was a red herring.
|
||
- **THE ARENA LEVER IS `max_model_len`, NOT `--kv-cache-memory-bytes`.** The
|
||
planner sizes expert cache as `budget - non-expert - KV reserve - graphs -
|
||
overhead`, and the KV reserve is registry-priced at **9190 B/token x
|
||
max_model_len**, independent of the pool override:
|
||
|
||
| MAXLEN | KV reserve | arena |
|
||
|---|---|---|
|
||
| 262144 | 2.48 GiB | 14.30 GiB / **6,248 slots** |
|
||
| 200704 | 1.91 GiB | 14.87 GiB / **6,496 slots** |
|
||
|
||
d(reserve) 0.57 GiB = d(arena) 0.57 GiB = +248 slots. The 6.1e9 -> 3.15e9
|
||
pool cut (2.75 GiB/rank) bought **zero** arena slots. The previous session's
|
||
premise - "the pool cut becomes arena, ~7,500 slots" - was wrong; the pool
|
||
cut is a runtime-HEADROOM lever (the shape it replaced died with `0 bytes
|
||
free` on a real 995-token request).
|
||
- **Final shape (option D):** `--max-model-len 200704` x 3 slots, fp8 KV,
|
||
`--kv-cache-memory-bytes 3,150,000,000`/rank, MNBT 1024, gather 32x64, no
|
||
speculative decode, prefetch off, no `--expert-precise-memory`. The argv
|
||
keeps the only slot count that compiles; the POOL caps resident
|
||
conversations. Deployed `c769ff2`, healthy in 558 s, e2e 33/33.
|
||
- **Measured live, and one prediction corrected.** `expert cache space 14.87
|
||
GiB / 6,496 slots` - the arena prediction landed exactly. But `GPU KV cache
|
||
size: 398,658 tokens, Maximum concurrency for 200,704 tokens per request:
|
||
1.99x` (3,146,129,408 B reserved), not the 2.04x the arithmetic predicted:
|
||
**the per-token KV price is shape-dependent** - 7,701 B/token at MAXLEN
|
||
262144 (6,100,000,000 B -> 792,069 tokens) vs **7,892** at 200704, ~2.5 %
|
||
dearer. Never extrapolate a concurrency figure across geometries.
|
||
- **Smoke-verified on the live endpoint:** three concurrent 67-token requests
|
||
answered in parallel (~4.1 s each) - the 3 slots are real; native `tool_calls`
|
||
(`get_weather {"city":"Seattle"}`); thinking in `reasoning` with clean
|
||
`content`; `reasoning_effort` accepts only {xhigh, medium, low} and answers
|
||
400 for anything else (re-measured; the fork's message names the set).
|
||
- **Harness contract followed the window:** `models.yml` `contextWindow`
|
||
262144 -> **200704** (compaction trigger 140,492 + 36,864 = 177,356 <=
|
||
200,704). Side effect: frank-archon's compaction request now fits inside a
|
||
frank-zealot slot again, which it had stopped doing when frank-zealot went to
|
||
3 x 180,224.
|
||
- **Comment-only follow-up** `295d578` recorded the measured numbers and was
|
||
deployed so the per-service revision marker sits at HEAD; e2e 33/33 again.
|
||
- Rollback for the pre-shape-D fp8+MTP4 era stays git tag
|
||
`frank-archon-fp8-mtp-20261001`. Runbook
|
||
[[dankserver-qwen38-fp8-expert-offload-tuning]].
|
||
|
||
## 2026-10-02 - plex: the readiness gate probed HTTP against a TLS-only service
|
||
|
||
Found by accident, and it had been latent since the secure-connections posture
|
||
was set: `deploy_plex` gated on `curl -fsS http://127.0.0.1:32400/identity`,
|
||
which **cannot** succeed. PMS drops plain HTTP off-loopback - the probe's
|
||
connection arrives on the container's eth0 - so Plex accepts the TCP connection
|
||
and then answers with nothing, and curl reports "Empty reply from server" (52).
|
||
|
||
- Trigger: a bare `./scripts/deploy.sh dankserver` (no `--service`) drains the
|
||
whole pending backlog, not just the service you were thinking about. plex's
|
||
per-service revision had been behind for long enough that this gate had never
|
||
run against the current posture. It burned its full 600 s and failed the run;
|
||
every service after plex in the order was left undeployed.
|
||
- **Plex was healthy the entire time.** `https://127.0.0.1:32400/identity`
|
||
returned 200 with the MediaContainer XML in 44 ms while the gate was still
|
||
timing out. The in-container `http://127.0.0.1:32400/identity` also answers
|
||
200 - loopback is the one path PMS serves in the clear, which is exactly why
|
||
the host-side probe could not see the difference.
|
||
- Evidence chain that settled it: `podman exec plex curl http://127.0.0.1:32400`
|
||
= 200; `curl http://127.0.0.1:32400` from the host = empty reply;
|
||
`/proc/net/tcp` inside the container showed `00000000:7E90` LISTEN plus stuck
|
||
ESTABLISHED pairs on `1703590A:7E90` (10.89.3.23, the container's own eth0
|
||
address) that never produced a response.
|
||
- **The repo already knew this law, in a different file:**
|
||
`tests/caddyfile.bats` pins the plex vhosts to a TLS upstream with the comment
|
||
"PMS drops plain HTTP off-loopback". The deployer's gate never got the same
|
||
treatment. Fixed to `curl -fsSk --max-time 3 https://127.0.0.1:32400/identity`
|
||
(`-k`: PMS serves its own self-signed cert on 32400).
|
||
- Re-deployed `39f0e0d`: plex recorded deployed, the *arr services recorded
|
||
(compose-recorded on this host, no restart attempted), tautulli deployed,
|
||
e2e 33/33. Verified after: `plex-https=200`, plex container up, tautulli
|
||
healthy, frank-archon untouched (`health=200`, NRestarts=0).
|
||
- **Operating lesson: `deploy.sh <target>` with no `--service` is a fleet-wide
|
||
drain.** It restarts every service whose recorded revision is behind HEAD and
|
||
whose paths changed since. To touch one service, name it.
|
||
|
||
## 2026-10-02 — frank-archon shape E: `--expert-precise-memory` (+1,026 arena slots), a mounted compile cache, and the P2P answer
|
||
|
||
Shape D left one question open — the plan's overhead term was reserving 2.35 GiB
|
||
of compile headroom the boot never used — and the answer to Frank's free-VRAM
|
||
question arrived together with a boot-cost bug that had been costing 181 s on
|
||
every restart since the cutover.
|
||
|
||
- **`--expert-precise-memory` is real and it fits (option A).** Boot-probed on
|
||
the shape-D base with production stopped, the candidate booted AS the service
|
||
(`FRANK_ARCHON_ENV_FILE=/tmp/probe-b.env bash scripts/serve-frank-archon.sh`,
|
||
trap-guaranteed restore), the numbers read off the engine's own plan lines:
|
||
|
||
| plan line | shape D | shape E |
|
||
|---|---|---|
|
||
| `runtime overhead (non-torch + buffers + activation)` | 7.60 GiB | **5.25 GiB** (`precise (gfx1201 TP2 nbt4096/seqs16/len262144)`) |
|
||
| `expert cache space` | 14.87 GiB | **17.22 GiB** |
|
||
| slots | 6,496 | **7,522** (+1,026, +16 %) |
|
||
| `swap region` | — | 13.7 GiB / 5,986 slots, 133-134 per layer |
|
||
| boot | — | **healthy in 420 s**, no OOM |
|
||
|
||
The flag also prints `'expert_precise_memory': True` in `non-default args`, so
|
||
a silent argparse drop cannot hide. It was refused on 2026-10-01 only because
|
||
the 2 x 262144 bf16 shape was already 0.13 GiB under the ceiling — there the
|
||
flag was the last stack over the line, not the cause.
|
||
- **The pool was NOT cut (option A) because the pool is not an arena lever.**
|
||
A 2.2e9 arm was probed in the same window and reports `277,686 tokens,
|
||
1.38x` — measured, rejected as the shipped value, kept as the conservative
|
||
fallback. With 3.15e9/rank the endpoint keeps shape D's 1.99x concurrency and
|
||
gains the arena, which is the shape D follow-up had actually been looking for.
|
||
- **THE COMPILE CACHE WAS BEING THROWN AWAY ON EVERY RESTART — 181 s each.**
|
||
Measured: a fresh container has **no `/root/.cache` at all** (`No such file or
|
||
directory`); after our boot it holds `/root/.cache/vllm` at
|
||
**1,154,714,333 B in 4,665 files** plus `/root/.cache/comgr` at 229 MB, inside
|
||
the ephemeral layer, and `podman rm -f` (every deploy, restart, reboot)
|
||
discards them. The engine names the loss itself: `Compiling a graph for compile
|
||
range (1, 1024) takes 173.19 s` ... `torch.compile took 181.20 s in total` ...
|
||
`init engine (profile, create kv cache, warmup model) took 217.26 s
|
||
(compilation: 181.20 s)`. The serve script now mounts
|
||
`FRANK_ARCHON_CACHE_DIR` at `/root/.cache` — the **parent**, so the vLLM
|
||
AOT/torch_compile tree (`VLLM_CACHE_ROOT`, default `/root/.cache/vllm`) and
|
||
the comgr JIT cache both persist, and nothing is hidden because the image has
|
||
no such directory. Same class of fix as the recipe's
|
||
`/cache/{triton,vllm,inductor}` volumes; the compile RANGE is part of the
|
||
cache key, so an MNBT change costs one recompilation, not a permanent tax.
|
||
- **`GPU_MAX_HW_QUEUES=2`** (HSA hardware queues per GPU) added at Frank's
|
||
direction. Verified a real knob rather than a no-op — the literal is read from
|
||
`/opt/rocm/core-10.0/lib/libhsa-runtime64.so` — and shipped deliberately
|
||
alongside the 32x64 gather geometry. **Not A/B measured.**
|
||
- **P2P: the hardware supports it, and the engine is already using it.** The
|
||
live log shows `r4d all-reduce INSTALLED (rank=0 max=49152KB finegrained=T…)`
|
||
on BOTH ranks and `Using ['R4D_AR', 'PYNCCL'] all-reduce backends` — R4D's
|
||
P2P path, with RCCL as fallback, measured at **45.274 GB/s bidirectional**.
|
||
Root ports `00:01.1`/`00:01.3` both report ACS `ReqRedir+`, so P2P hairpins at
|
||
the root complex; there is no XGMI. The old FAIL predates the 2026-09-20 board
|
||
swap. Kernel support confirmed: `CONFIG_HSA_AMD_P2P=y`, `CONFIG_PCI_P2PDMA=y`,
|
||
`CONFIG_HSA_AMD=y`, `CONFIG_ZONE_DEVICE=y`, `iommu=pt` on the cmdline.
|
||
- **Recipe diff (Frank's 3 x R9700 compose vs this podman run), ranked:**
|
||
- *Acted on:* persistent compile caches (above — the one real operational
|
||
gap), and `GPU_MAX_HW_QUEUES`.
|
||
- *Retracted:* pinned `--compilation-config cudagraph_capture_sizes` was on my
|
||
list; measured, it is **not** a gap here — capture is 3 PIECEWISE + 2 FULL
|
||
graphs, `Graph capturing finished in 3 secs, took 0.31 GiB` (0.11 GiB on the
|
||
lighter rank). No 15-52-size ladder to cut.
|
||
- *Not acted on — measured candidates, none shipped (no bench window; these
|
||
are CPU-side and our decode is gather-bound, so a spin-wait stealing cores
|
||
from the gather threads is plausible but unproven):* `OMP_NUM_THREADS=8`
|
||
(defaults to all 16 logical CPUs on this 7700X), `HSA_ENABLE_INTERRUPT=1`,
|
||
`HSA_ENABLE_MWAITX=1`, `HSA_ENABLE_SDMA`, `HSA_NO_SCRATCH_RECLAIM`,
|
||
`HSA_ENABLE_IPC_MODE_LEGACY`. All six HSA names verified real — the literals
|
||
are read out of `/opt/rocm/core-10.0/lib/libhsa-runtime64.so` — so they are
|
||
knobs, not no-ops; any of them needs its own A/B before shipping.
|
||
- *Deliberate divergence, keep ours:* no MTP (the recipe has it; measured
|
||
fastest here), `qwen3_xml` vs `qwen3_coder` (ours verified live), no
|
||
`--override-generation-config` (server-side sampling defaults), published
|
||
port + Caddy rather than `network_mode: host`.
|
||
- *Non-gaps:* `shm_size 32gb` (ours is `--ipc=host`), `memlock -1` (same),
|
||
`SYS_PTRACE` (same), `/dev/kfd` + `/dev/dri` (same), `--enable-chunked-prefill`
|
||
(V1 default), `plecache` volume (the flag is not even passed), and
|
||
`ROCR_VISIBLE_DEVICES=1,2,3` vs our `HIP_VISIBLE_DEVICES=0,1` — verified
|
||
equivalent here: `rocm-smi --showid` puts the two R9700s at GPU[0]/GPU[1] and
|
||
the 7700X iGPU at GPU[2], so our indices are the right two cards.
|
||
- **Harness note, because it cost two probe attempts:** a probe that reported an
|
||
instantaneous candidate death was a logging artifact of the inline SSH script,
|
||
not a crash — the container and its serve script must be polled by recorded id
|
||
and their logs dumped on the exit path, or a failure looks like a bad flag.
|
||
The flag was fine both times.
|
||
- Live: `health=200`, `NRestarts=0`, arena 17.22 GiB / 7,522 slots, KV 398,658
|
||
tokens / 1.99x. Docs: [[dankserver-qwen38-fp8-expert-offload-tuning]].
|
||
- **Deployed `1ac4d48`, e2e 33/33, and the live plan matched the probe exactly:**
|
||
`'expert_precise_memory': True`, `runtime overhead … 5.25 GiB`, `expert cache
|
||
space 17.22 GiB / 7522 slots`, `swap region 13.7 GiB / 5986 slots`, `GPU KV
|
||
cache size: 398,658 tokens … 1.99x`, pool 3,150,000,000, MNBT 1024. The
|
||
precise overhead and the arena growth are now production numbers, not a
|
||
probe's projection.
|
||
- **The compile-cache fix was then proven on the same endpoint, two consecutive
|
||
boots of the deployed shape:** cold `Compiling a graph for compile range
|
||
(1, 1024) takes 179.24 s` → `torch.compile took 188.26 s`; warm `Directly load
|
||
the compiled graph(s) for compile range (1, 1024) from the cache, took
|
||
2.143 s` → `torch.compile took 3.97 s in total` → `init engine (profile,
|
||
create kv cache, warmup model) took 41.13 s (compilation: 3.98 s)`. The host
|
||
dir held 1.4 GB in 4,707 files after the cold boot. **Time to `/health` is
|
||
dominated by the 172.8 GiB payload load** (the warm boot still needed 490 s),
|
||
so the honest claim is ~184 s of inductor/AOT build recovered per restart —
|
||
not a proportional boot cut.
|
||
|
||
## 2026-10-11 — frank-archon: unquantized (bf16) KV at 2 × 200,000
|
||
|
||
- Ask: two concurrent 200,000-token conversations on exact attention state.
|
||
Sized from the live fp8 boot ledger, not arithmetic: the paged groups cost
|
||
7.3125 KiB/token per rank (`kv_attn` 26.00 KiB per 4-token block +
|
||
`kv_qsa_bkey` 3.25), so the 3,543 MiB pool bought 386,904 tokens = **1.93
|
||
sessions of the served window**.
|
||
- Engine door: `--kv-cache-dtype` takes ONLY `bf16|fp8` — one dtype for the
|
||
whole cache, so llama.cpp's exact-K / quantized-V pair (what frank-zealot
|
||
runs) has **no radiance equivalent**. Measured bf16 rate is **13.8125
|
||
KiB/token = 1.89× fp8, not 2×**: the QSA block key (13 layers, 3.25 KiB per
|
||
4 tokens) does not follow the dtype. Pool formula
|
||
`tokens × rate + 110 MiB × seqs (recurrent state) + 448 MiB (prefix
|
||
checkpoints)` reproduces both boots within 0.1 %, and is now a test seam.
|
||
- Deployed (`0bf26b0` + docs): pool 3543 → **6912** MiB (462,752 tokens; ledger
|
||
"backs 2 sequences all at 200000 tokens at once"), weights slab 26071 →
|
||
22208, host pool 40960 → 44928 (the evicted experts land there; pinned pool
|
||
slots 7,190 → 10,375), prefix host 4096 → 8192 (the entry grows 58.50 →
|
||
110.50 KiB), TTM floor 140000 → 150000 (pinned ledger ~147 GiB, cap 204,800).
|
||
`weights + kv` pinned to the measured ceiling **29,120 MiB/card**; cards
|
||
settled 1.85 GiB free — more than the fp8 boot's 1.39, because the total
|
||
claim dropped. Half the cost is refunded by the engine: idle KV VRAM is the
|
||
expert cache's spillover, and the bigger pool *lends* 6.08 GiB where the fp8
|
||
pool lent 2.69.
|
||
- `--max-num-seqs 3 → 2`: the fleet's "seqs 2 cannot boot" law is the retired
|
||
tcclaviger vLLM fork's torch.compile `query_start_loc` constraint, not
|
||
radiance's — radiance booted seqs 2 in the fp8-native era and the 2026-10-06
|
||
sweep had 2 and 3 tied inside ±3 %. Bonus: the fused all-reduce is declared
|
||
over 2 × (1+2) = 6 rows, inside the 25-row bound where the wht6 wire is
|
||
bit-exact.
|
||
- Verified live (no scratch container, no bench): `/health` 200, served id
|
||
`frank-archon`; 2 × 196,919 concurrent conversations each answered the
|
||
marker buried at the head of its own prompt (393,837 tokens resident, above
|
||
the fp8 pool's 386,904); decode at 51.5k depth 94.1 cold / 102.3 warm / 117.6
|
||
prefix-hit t/s, draft acceptance 0.7284 / 0.7525.
|
||
- fp8 control arm through a transient `FRANK_ARCHON_ENV_FILE` drop-in (the
|
||
frank-zealot ubatch method, argv + ledger page size verified): prefill
|
||
**−29 %** (4,379 → 3,126 cold, 4,864 → 3,333 warm), decode inside the noise
|
||
band on two of three passes, TTFT at the full window **+25 %** (the pair:
|
||
110.5 s → 138.1 s), draft acceptance **up 0.661 → 0.740**. The price is
|
||
prefill — a 14.8 % smaller resident slab, and prefill is the pass that
|
||
touches every expert — the gain is exact state plus acceptance. **Open call
|
||
for Frank:** keep bf16, or take the cheap fp8 2 × 200k fit (pool 3,648 MiB,
|
||
≈ free) and leave KV quantized.
|
||
- Harness law re-learned: radiance puts the thinking block in
|
||
`reasoning_content`, so a probe that reads `content` on a small budget sees
|
||
empty text with `finish_reason: stop` — a healthy server scored 0/2 until the
|
||
probe set `chat_template_kwargs.enable_thinking=false`. Tool committed as
|
||
`scripts/probe-frank-archon-kv.py`.
|
||
- Tests 368/368, e2e 33/33 (one unrelated homepage retry). Docs:
|
||
[[frank-archon-radiance-swift15-w4dn8]], and the LLM index entry corrected
|
||
(the w4dn8 entry said STAGED, the fp8-native one said ACTIVE).
|
||
|