docs: frank-large flash-server v2 cutover — runbook, index, AGENTS.md
- New runbook: frank-large-flash-server-v2-swift15.md (engine, shape, memory model, engine laws, boot ledger, probes, rollback) - LLM/index.md: active entry + retired halogen 27B entry - Vault index: link the new runbook from the LLM Fleet section - AGENTS.md: fleet topology table, Model Config section, Runbook Index, retired table (halogen 27B → 10-06)
This commit is contained in:
@@ -10,11 +10,12 @@ experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the
|
||||
47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3,
|
||||
2 × 200,000 first boot. Rollback tags frank-primary-freetoken-pr587-20261005
|
||||
and frank-primary-vllm-shapeG-20261003),
|
||||
and Dankclaw runs two endpoints — **halogen container (frank-large, port 7770:
|
||||
ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai
|
||||
/halogen-qwen3.8-27b, a 33.4 GiB .hgn checkpoint that loads in no other
|
||||
engine, DFlash2 spec decode, ONE slot at 200,700 ctx, text-only — vision
|
||||
lives on frank-small)** and
|
||||
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
|
||||
(frank-large, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
|
||||
digest serving the Quat3rnion ABLITERATED Swift1.5 finetune of
|
||||
Qwen3.8-Flash-Next v2, a ~62 GiB v2 .hgn that loads in no other engine,
|
||||
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots at 200,192
|
||||
ctx, vision ON)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: ajgazin Swift-1.5-Qwen3.8-27B
|
||||
Uncensored Dynamic-MTP RCO-IQ3_XXS GGUF — 27B (the MTP head in the file
|
||||
IS drafted, draft-mtp depth 3), TWO 131,072 conversations at Frank's
|
||||
@@ -45,10 +46,10 @@ PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[RTX 3090 (PCIe x4 dock) +
|
||||
gfx1151 iGPU 114 GiB GTT]
|
||||
E[halogen container gfx1151, digest-pinned image
|
||||
E[flash-server container gfx1151, digest-pinned image
|
||||
frank-large, port 7770]
|
||||
F[halogen-qwen3.8-27b .hgn p1w4d-d2 + DFlash2 drafter
|
||||
1 x 200700 ctx, d2 KV, text-only]
|
||||
F[flash-server v2 abliterated Swift1.5 .hgn + ngram table + vision tower
|
||||
2 x 200192 ctx, MTP always-on, image input]
|
||||
G[llama.cpp master CUDA sm_86
|
||||
frank-small, port 7771]
|
||||
H[Swift-1.5-27B Uncensored RCO-IQ3_XXS-mtp GGUF
|
||||
@@ -71,30 +72,37 @@ frank-small, port 7771]
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-large-halogen27b-dflash2]] — **ACTIVE since 2026-10-05**
|
||||
- [[frank-large-flash-server-v2-swift15]] — **ACTIVE since 2026-10-06**
|
||||
(root systemd unit + podman, port 7770): the digest-pinned
|
||||
`ghcr.io/peonist-ai/halogen` image serving `peonist-ai/halogen-qwen3.8-27b`
|
||||
(`qwen3.8-27b-p1w4d-d2.hgn`, 33.4 GiB) with the DFlash2 drafter on ONE slot
|
||||
of 200,700 ctx (slots>1 disables speculation — engine law). Publisher card
|
||||
31.7 t/s mean across eight text workloads. Accepted tradeoffs: TEXT-ONLY
|
||||
(the engine ignores the vision encoder; frank-small owns images) and the
|
||||
CENSORED base model (the ROCmFPX AGENT file was Uncensored by construction).
|
||||
No server-side sampler — the thinking recipe rides the wire in models.yml.
|
||||
Rollback: revert the swap commit; the ROCmFPX payload + fork build +
|
||||
`build-rocmfpx.sh` pin all remain on disk
|
||||
Tuned 2026-10-06: `HALOGEN_CACHE_MB=16384` pins the prompt cache —
|
||||
auto was OFF at the 36.9 GB two-entry floor, so every deep request
|
||||
re-prefilled its 79-81k context (85-180 s each); a 17.3k cold prefill
|
||||
of 27.0 s now resumes in 1.7 s on the warm turn (cache_n 16,384). The
|
||||
serve script captures the engine ledger on disk (foreground `podman
|
||||
run`; the `-d` era wrote only the container ID). Decode-at-depth is
|
||||
structural: 3.3-3.8 t/s at 80k vs 30-50 shallow; no supported flag
|
||||
touches it.
|
||||
The FA tile lever from upstream issue #12 (`HALOGEN_FA_TQ=256`) was
|
||||
A/B'd at 80k on 2026-10-06: decode ±0.3 % (rejected), prefill −9.5 %
|
||||
on this topology — the default tile stands. The deep harness roles
|
||||
(`slow`, `advisor`) rerouted to frank-primary; frank-large keeps
|
||||
OpenWebUI and shallow callers.
|
||||
`ghcr.io/peonist-ai/halogen-flash-server` 0.16.2 engine serving the
|
||||
Quat3rnion ABLITERATED Swift1.5 finetune of peonist's
|
||||
Qwen3.8-Flash-Next v2 checkpoint (`qwen38-flash-next-v2-swift15-abliterated.hgn`,
|
||||
66.7 GB compacted r2 of the card's 89.8 GB append-only original,
|
||||
sha256-verified) with the REQUIRED n-gram lookup table (51.2 GB,
|
||||
@peonist 5cc17cea) and the vision tower (897.9 MB, @e053f488) beside
|
||||
it. TWO slots of 200,192 ctx, MTP head ALWAYS ON (`HALOGEN_SPEC_ADAPT=0`
|
||||
— the adaptive default turns the head off per request and halves
|
||||
decode), server-side sampler at the card's 1.0/0.95/20, coupled
|
||||
cap/timeout 65536/7200, prompt cache ON by default (the 27B engine's
|
||||
manual `HALOGEN_CACHE_MB` pin era is over — warm 100k turn ≈ 2 s vs
|
||||
≈ 88 s cold). Vision ON (image_url parts only). Card: 53-57 t/s
|
||||
greedy, ~1,640 t/s prefill at 34k, ~72 % fewer thinking tokens.
|
||||
Rollback: revert the swap commit; the 27B payload + fetch script stay
|
||||
on disk
|
||||
- [[frank-large-halogen27b-dflash2]] — **RETIRED 2026-10-05 → 10-06
|
||||
(rollback target; payload + fetch script + image on disk)**: the
|
||||
digest-pinned `ghcr.io/peonist-ai/halogen` 0.1.4 engine serving
|
||||
`peonist-ai/halogen-qwen3.8-27b` (`qwen3.8-27b-p1w4d-d2.hgn`, 33.4 GiB)
|
||||
— DFlash2 drafter, ONE 200,700 slot, text-only (the engine ignored the
|
||||
vision encoder), censored base. Owns the measured lessons: the prompt
|
||||
cache pin (`HALOGEN_CACHE_MB=16384` — auto was OFF at the 36.9 GB
|
||||
two-entry floor; a 17.3k cold prefill of 27.0 s resumed in 1.7 s warm),
|
||||
the FA tile rejection (`HALOGEN_FA_TQ=256`: prefill −9.5 %, decode
|
||||
±0.3 %), and deep decode at 3.3-3.8 t/s@80k being structural. The
|
||||
foreground-`podman run` ledger capture and the deep-role rerouting
|
||||
(`slow`/`advisor` → frank-primary) carried forward. Superseded by the
|
||||
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
|
||||
engine moved with the model
|
||||
- [[frank-small-llama-swift15-unc-iq3xxs-mtp-131072]] — **ACTIVE since
|
||||
2026-10-05 (evening); UNCECERSED RCO-IQ3_XXS with MTP back on since
|
||||
2026-10-06 evening** (root systemd unit, port 7771): llama.cpp master
|
||||
@@ -400,7 +408,7 @@ frank-small, port 7771]
|
||||
(deployed and reverted 2026-08-22)
|
||||
- [[Dankserver System Specs]] — the 2x R9700 box (frank-primary)
|
||||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on
|
||||
the unsloth b11030 runtime since 2026-09-21, frank-small on the RTX 3090)
|
||||
the flash-server v2 runtime since 2026-10-06, frank-small on the RTX 3090)
|
||||
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
|
||||
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
|
||||
the retirement record — the IOMMU/SVA err=-19 root cause and where
|
||||
|
||||
Reference in New Issue
Block a user