e2e: crawl reset survives app navigation; dragoon docs carry the cut ledger
Two faces of the same verification round. 12-crawl returned to the homepage between links with a bare page.goto, and Plex Web (signed out, so it renders its own sign-in iframe for its Plex-account flow) keeps redirecting the top frame: Chromium cancelled our goto with net::ERR_ABORTED, outside the try, so the whole crawl died — once as a 420 s timeout in the deploy gate, once at 5.4 s on the same link. backHome() tolerates that one error class and re-issues; any other error still propagates, and the per-app verdict (guard settled, hostname, no authentik uidField, status < 500) is unchanged. Plex was healthy throughout (container up 2 days, 32400 answering HTTPS 401 / plain-HTTP empty reply), so this was app behaviour, not a fault — recorded as gotcha 6 on the E2E page. The 200,704 boot printed 5.5 GiB reserved and 74.1 GiB in all (not the ~5.4/~74 my arithmetic predicted), so every prose figure is re-keyed to the ledger, and the runbook gains the second production-boot section: marker revision, pool reservation, shape line, prompt-cache entry size, and the refusal contract at the new ceiling (400 in 0.86 s naming 200704 and the 96,405-token overage, before any prefill). Cost stated (two maximum-length conversations never coexist; the 180k compaction target is unreachable inside a 155,648 window) and purchase stated (1.7 GiB of page cache, 9.1 GiB against the 2026-10-06 shape).
This commit is contained in:
@@ -26,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-archon` | **radiance** (StillDeadcode, **pin 1.0.13** digest-pinned `stilldeadcode/radiance@sha256:4c3f484564…`, plain `podman run`, root systemd unit) | **Publisher payload** (swap 2026-10-15, MODEL_FILE-only): `StillDeadcode/qwen3.8-next-flash-fp8-iq4r-moe` → **`qwen3.8-next-flash-fp8-iq4r-moe.rad`** (121,969,901,568 B = **116,322 MiB**, Hub-LFS sha256 `0af5e962…` verified at fetch by `scripts/download-frank-archon-iq4r.sh`) — the OFFICIAL Qwen3.8-Flash-Next bf16 checkpoint rad-converted by the engine's author: routed experts as **4-bit codes into a 16-level non-uniform codebook** (128-pt Walsh–Hadamard rotation, GPTQ on a 10M-token calibration + 3 coordinate-descent sweeps, E4M3 scales per 64 weights), **10 whole bf16-protected experts/layer**, int8 W8A8 trunk/lm_head, fp8 HC mixtures, fp8 fixed-scale PLE n-gram (47.7 GiB), **MTP head + bf16 vision tower in the container**; card KL 0.0789 mean all-tokens / top-1 92.85 % vs BF16. **Base-alignment (censored) model** — the abliterated family lives on frank-dragoon. Shape carried VERBATIM from the 2026-10-13 record: **3 × 200,000 fp8-KV** (`--max-num-seqs 3`), **25,600 MiB slab + 5,120 MiB pool = 30,720 of the 30,969 MiB claimable**, 44 GiB/rank pinned host pool, PLE pinned to RAM, MTP depth 2, MNBT 2048, Froggeric v22.5 template. Decode/prefill ladders in the runbook are the **w4dn8 record** (the bar to beat; new-payload probes pending). Rollback: re-download the w4dn8 `.rad` from the HF mirror `frank8465/Swift-1.5-Qwen3.8-Flash-Next-GPTQ-W4A8-Radiance-AMD-R9700` (LFS sha256 `09dd9306…`) + revert the swap commit — the local copy was **purged 2026-10-17** (it was the last non-iq4r model on the host; the GLM-4.7 GGUF, the radiance kvcache tiers, and every stale compile cache went in the same purge, 763 GiB free). The fp8-native `.rad` + the raw Swift1.5 BF16 source were DELETED from disk 2026-10-15 (Frank's one-time rm, volume for this payload; rebuildable via the fetch/build scripts). | 7770 |
|
||||
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=200704` pool at `HALOGEN_CTX=200704`** (lowered from the card's 262,144 the same evening — ~5.4 GiB where 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; pool == ctx so ONE maximum-length request fills it, and the freed RAM is page cache for the n-gram table), harness window 155,648 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770|
|
||||
| Dankclaw | `frank-dragoon` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-dragoon.sh`, root systemd unit) | Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-abliterated.hgn`** (66,687,678,432 B ≈ 62.1 GiB resident, Hub LFS sha256 `b5bd925f…` verified at fetch — the **NATIVE 2026-10-11 v2 conversion** of orcarouter's pinned BF16, abliterated, NOT the writer transplant that carried this filename; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea) **+ vision tower** (897,916,416 B, @e053f488) — both **hardlinked** from the swift15 payload dir (the release publishes only the main file; the engine's own `niah` 18/18 + `ppl` 3.968 with mean KL 0.155 vs the swift15 logits certified the stock companions), **vision ON** (`image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the drafter per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset), **TWO slots over ONE `HALOGEN_KV_POOL_POSITIONS=200704` pool at `HALOGEN_CTX=200704`** (lowered from the card's 262,144 the same evening — 5.5 GiB reserved where 262,144 cost 7.2 and the 2026-10-06 pool measured 14.6; pool == ctx so ONE maximum-length request fills it, and the freed RAM is page cache for the n-gram table), harness window 155,648 = ceiling − 45k estimate headroom, **server-side sampler 1.0/0.95/20** (peonist engine recipe — Quat3rnion publishes none; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold), **n-gram table disk-backed** (mmap page cache, `Pin::None`); measured vs the swift15 boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18, PPL +2.3 %, thinking tokens ×1.9|7770|
|
||||
| Dankclaw | `frank-zealot` | **llama.cpp PrismML fork build** (vendor release `0.2.0-dev` build **10754**, commit `2459f68b5`; root systemd unit, relinked — the ONLY loader for this payload) | `prism-ml/Ternary-Bonsai-2-27B-gguf` **PQ2_0** (7,206,168,928 B, sha256 `3907dc16…`; `qwen35` hybrid: **64 blocks, 16 full-attention + 48 Gated DeltaNet**, 4 KV heads × 256 dim, vocab 248,320, native 262,144, **no MTP head in the file**; `general.file_type 141` = custom tensor type **142** + `prism.hadamard.*` dequant metadata, which is what locks the runtime) + the publisher's own **Q8_0 mmproj** (629,246,976 B; Qwen3-VL tower, 27 blocks, 768 px, `qwen3vl_merger`) for **vision**, on the RTX 3090, **TWO 200,704-token slots** (`--ctx-size 401408 --parallel 2`), **q8_0 K + q8_0 V** under flash attention (34 KiB/token over the 16 full layers = 13,328 MiB pool; **f16 KV needs 25,088 MiB and cannot boot this card**), batch 4096 / **ubatch 2048** (measured CUDA optimum), `--log-verbosity 4`, **MTP OFF** (no head in the file). Boot 22,664 used / 1,461 free; 76.6 t/s single-stream decode, 691.7 t/s prefill at 166k. Cutover 2026-10-17 from Qwen3.5-9B-Uncensored Q6_K (same shape; payload + master CUDA build on disk as rollback). |
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
@@ -399,8 +399,8 @@ evening** (Frank's call: every position costs KV RAM and nothing in the fleet
|
||||
asks for the biggest window) — is **ONE pool behind both slots**, ledger form
|
||||
`2 slots over ONE <pool>-position KV pool, each request up to HALOGEN_CTX`, and
|
||||
at pool == ctx a single MAXIMUM-length request fills it: two maximum-length
|
||||
conversations never coexist (two of ≤100k do). ~28 KiB/position ⇒ ~5.4 GiB,
|
||||
where 262,144 cost 7.2 and the 2026-10-06 shape's 524,288 measured 14.6; the
|
||||
conversations never coexist (two of ≤100k do). ~28 KiB/position ⇒ the ledger
|
||||
reserves "about 5.5 GiB", where 262,144 cost 7.2 and the 2026-10-06 shape's 524,288 measured 14.6; the
|
||||
engine's preflight FORMULA bills slightly more than the reservation (block
|
||||
scratch + ~115 MiB/slot of O(1) state) — quote the reservation.
|
||||
`HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the
|
||||
@@ -419,10 +419,11 @@ stays mandatory. Thinking tokens COUNT against the budget (empty content +
|
||||
`image_url` content parts only; ≥256 px probe images or the model answers
|
||||
"video".
|
||||
|
||||
**Memory ledger** (121.2 GiB unified; the referee is the boot ledger):
|
||||
weights ~62.9 (trunk 62.11 + vision 0.84) + KV ~5.4 + working ~5.7
|
||||
(`3.8 + 8.4 × MAX_TOK/16384`) ≈ 74 GiB engine-side, the ~1.0 GiB prompt cache
|
||||
booked separately by the ledger, + `HALOGEN_HOST_RESERVE_GIB=20` → ~30 GiB
|
||||
**Memory ledger** (121.2 GiB unified; the referee is the boot ledger — the
|
||||
200,704 boot printed exactly): weights 62.9 (trunk 62.11 + vision 0.84) + KV
|
||||
5.5 + working 5.7 (`3.8 + 8.4 × MAX_TOK/16384`) = **74.1 GiB in all**, the
|
||||
~1.0 GiB prompt cache booked separately by the ledger, +
|
||||
`HALOGEN_HOST_RESERVE_GIB=20` → ~30 GiB
|
||||
margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd page
|
||||
cache beside the checkpoint (ledger: `mapped 51.2 GB, NOT pinned
|
||||
(Pin::None)`) — which is why every pool cut is a double saving: **RAM the pool
|
||||
|
||||
@@ -82,3 +82,10 @@ real IdP credentials.
|
||||
5. **Solve-once cookie.** Since Anubis v1.27.0 + `COOKIE_DOMAIN` (see
|
||||
[[Anubis]]), one PoW per fresh context covers every guarded vhost — crawl
|
||||
and SSO specs complete in ~0.5 s per vhost instead of a PoW each.
|
||||
6. **Apps navigate too.** `12-crawl` returns to the homepage between links, and
|
||||
Plex Web (signed out, so it shows its own *"would like to sign in to your
|
||||
Plex account"* iframe) keeps redirecting the top frame — Chromium cancels our
|
||||
`goto("/")` with `net::ERR_ABORTED`, which used to abort the whole crawl
|
||||
(once as a 420 s timeout, once at 5 s on the same link). `backHome()` re-issues
|
||||
the navigation; the app itself was already judged, so the abort is the app's
|
||||
behaviour, not a guard-chain failure.
|
||||
|
||||
@@ -99,8 +99,10 @@ printed `pinned 62.11 GiB in 1 range(s) in 1.6 s (41.0 GB/s)` trunk,
|
||||
`0.84 GiB` vision tower, `reserving the KV pool, 262144 positions (about
|
||||
7.2 GiB)`, and the engine's own sum: `memory: 62.9 GiB of weights locked in
|
||||
RAM (0.8 of them the vision tower's), 7.2 GiB of KV pool, …, 75.8 GiB in
|
||||
all`. After the same-evening ctx/pool cut to 200,704 the reservation is ~5.4
|
||||
GiB and the total ~74. The n-gram table rides as page cache — `mapped 51.2 GB,
|
||||
all`. The cut boot printed the same shape at the new size: `reserving the KV
|
||||
pool, 200704 positions (about 5.5 GiB)` and `… 5.5 GiB of KV pool, 5.7 GiB of
|
||||
working memory, 74.1 GiB in all`.
|
||||
The n-gram table rides as page cache — `mapped 51.2 GB,
|
||||
NOT pinned (Pin::None)` — which is exactly why every pool cut is a *double*
|
||||
saving. FLAGS.md law: **RAM the pool takes is RAM that table loses**; the ~9
|
||||
GiB the two cuts together handed back is page cache for the 47.68 GiB table
|
||||
@@ -225,3 +227,45 @@ on the FIRST request after boot, i.e. the warmup artifact; the trial's settled
|
||||
figure is 50.0. OpenWebUI's `frank-dragoon` row description re-keyed to the
|
||||
native payload (backup `webui.db.bak-20261017b`; no container restart forced —
|
||||
forcing it logs every family session out for one cosmetic string).
|
||||
|
||||
## Ceiling lowered to 200,704 — the second production boot (`7c4dbeb`)
|
||||
|
||||
The same evening, after the ledger above was read: `HALOGEN_CTX` and
|
||||
`HALOGEN_KV_POOL_POSITIONS` 262144 → **200704** (49 × 4096, the per-slot rung
|
||||
the llama.cpp endpoint frank-zealot already runs). The checkpoint's native
|
||||
window is 262,144 and the engine serves anything below it, so this is a RAM
|
||||
decision, not a capability limit. Nothing in the fleet asks for the bigger
|
||||
window; the harness moved WITH the ceiling (400-zone law), `models.yml`
|
||||
`contextWindow: 155648`.
|
||||
|
||||
|Line|Figure|
|
||||
|---|---|
|
||||
|marker|`revision=7c4dbeb…`, `status=deployed`; unit `active`, `/health` 200|
|
||||
|KV pool|`reserving the KV pool, 200704 positions (about 5.5 GiB)`|
|
||||
|shape|`2 slots over ONE 200704-position KV pool, each request up to 200704 (it reserves prompt + max_tokens), max_tok 8192 … MTP depth-2 selectable (drafter 1…)`|
|
||||
|working|5.7 GiB (unchanged — it scales with `HALOGEN_MAX_TOK`, not the window)|
|
||||
|total|`62.9 GiB of weights locked … 5.5 GiB of KV pool, 5.7 GiB of working memory, 74.1 GiB in all`|
|
||||
|prompt cache|ON, resume-anywhere, 12 entries, chunk 32768, **111 MiB each at ctx 200704**|
|
||||
|clamps|none — requested pool == delivered pool|
|
||||
|
||||
The refusal contract at the new ceiling, asked for explicitly (the 1.29 MB
|
||||
wikitext corpus through `/v1/chat/completions`):
|
||||
|
||||
```
|
||||
HTTP 400 {"error":{"message":"the prompt does not fit: it is 297109 tokens and
|
||||
the context is 200704, 96405 tokens over. Shorten the prompt, or raise the
|
||||
context (HALOGEN_CTX, and HALOGEN_ROPE_YARN for more than the native 262144).",
|
||||
"type":"invalid_request_error"}}
|
||||
```
|
||||
|
||||
Refused in 0.86 s — the preflight rejects before any prefill, so probing the
|
||||
ceiling costs the endpoint nothing. OpenWebUI's `frank-dragoon` row description
|
||||
re-keyed to "ONE 200,704-position KV pool at ctx 200,704" (still no container
|
||||
restart; `webui.db.bak-20261017b` holds the pre-cutover row).
|
||||
|
||||
**What the cut cost:** with pool == ctx one maximum-length request fills the
|
||||
pool, so two 200k conversations never coexist (two of ≤100 k do), and the
|
||||
harness window is now 155,648 — the 180 k compaction target of the 262,144 era
|
||||
is unreachable because the window itself is smaller. **What it bought:** 1.7
|
||||
GiB back as page cache for the 47.68 GiB n-gram table (9.1 GiB against the
|
||||
2026-10-06 shape's 524,288 pool), and a smaller prompt-cache entry.
|
||||
|
||||
@@ -89,8 +89,8 @@ frank-zealot, port 7771]
|
||||
3.968 with mean KL 0.155 against the Swift1.5 logits on the same corpus).
|
||||
`HALOGEN_CTX=200704` over `HALOGEN_KV_POOL_POSITIONS=200704`: **one pool
|
||||
behind two slots** (the card ships 262,144/262,144 = 7.2 GiB; Frank took the
|
||||
RAM the same evening — ~5.4 GiB, one maximum-length request filling the
|
||||
pool), and the freed RAM becomes page cache for the n-gram
|
||||
RAM the same evening — 5.5 GiB reserved, one maximum-length request filling
|
||||
the pool), and the freed RAM becomes page cache for the n-gram
|
||||
table. MTP head always on, sampler 1.0/0.95/20, coupled cap/timeout
|
||||
65536/7200, prompt cache ON, vision ON. Measured against the Swift1.5
|
||||
boot: cold prefill +12.6 %, decode 50.0 vs 46.8 t/s, NIAH 18/18 vs 17/18,
|
||||
|
||||
@@ -20,6 +20,25 @@ async function settleThroughGuard(page: Page, targetHost: string): Promise<void>
|
||||
expect(await page.locator('input[name="uidField"]').count()).toBe(0);
|
||||
}
|
||||
|
||||
async function backHome(page: Page): Promise<void> {
|
||||
// The homepage is the loop's only anchor, and some apps keep issuing their
|
||||
// OWN navigation after load — Plex Web's "would like to sign in to your
|
||||
// Plex account" iframe redirects the top frame, which cancels our goto with
|
||||
// net::ERR_ABORTED. That abort is the app's behaviour, not a guard-chain
|
||||
// failure (the app itself was already judged above), so tolerate it and
|
||||
// re-issue rather than letting one host abort the whole crawl.
|
||||
for (let attempt = 0; attempt < 3; attempt++) {
|
||||
try {
|
||||
await page.goto("/", { waitUntil: "domcontentloaded" });
|
||||
return;
|
||||
} catch (err) {
|
||||
if (!String(err).includes("ERR_ABORTED")) throw err;
|
||||
await page.waitForTimeout(1_000);
|
||||
}
|
||||
}
|
||||
await page.goto("/");
|
||||
}
|
||||
|
||||
test("every user open link reaches its app", async ({ page }) => {
|
||||
test.setTimeout(420_000);
|
||||
// Expected link count comes from the live registry, mirroring the
|
||||
@@ -47,7 +66,7 @@ test("every user open link reaches its app", async ({ page }) => {
|
||||
} catch (err) {
|
||||
failures.push(`${link.label}: ${err instanceof Error ? err.message : String(err)}`);
|
||||
}
|
||||
await page.goto("/");
|
||||
await backHome(page);
|
||||
await expect(page.locator("#grid a.open")).toHaveCount(expectedUserLinks);
|
||||
}
|
||||
expect(failures, failures.join("\n")).toEqual([]);
|
||||
|
||||
Reference in New Issue
Block a user