Frank
7c4dbebe24
Frank's call the same evening as the native-payload cutover: HALOGEN_CTX and HALOGEN_KV_POOL_POSITIONS both 262144 -> 200704 (a multiple of 4096, the rung frank-zealot already runs). The checkpoint's native window is 262,144 and the engine serves anything below it, so this is a RAM decision, not a capability limit: the KV reservation drops 7.2 GiB -> ~5.4 (the 2026-10-06 pool measured 14.6), and by the FLAGS.md law -- RAM the pool takes is RAM that n-gram table loses -- the freed positions become page cache for the 47.68 GiB table that long prompts read. Engine-side total ~74 GiB instead of 75.8. The 400-zone law is invariant to the value, so the harness moves WITH the ceiling: models.yml contextWindow 217088 -> 155648 = ceiling - 45056, keeping the gap that absorbs the chars/4 under-count (worst-case request ~156.2k vs the 200,704 ceiling, ~44k margin). Compaction now fires at 85% = 132,300 est (~136k measured); the 180k target of the 262,144 era is unreachable because the window itself is 155,648. maxTokens 45056 and the coupled cap/timeout 65536/7200 are untouched. Cost stated plainly: with pool == ctx, ONE maximum-length request fills the pool, so two long conversations never coexist (two of <=100k do). Acceptable because frank-dragoon carries only the slow role and the security-reviewer subagent, not the interactive workhorse. Seam re-keyed: the fit-law test now proves the invariant at the new pair, and the pool law (ctx <= pool <= slots * ctx) still holds with pool == ctx.
228 lines
13 KiB
Markdown
228 lines
13 KiB
Markdown
# frank-dragoon — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 NATIVE abliterated (Dankclaw, Strix Halo iGPU)
|
||
|
||
**ACTIVE since 2026-10-17.** Rollback = revert the swap commit and redeploy
|
||
(`scripts/frank-dragoon.env`, `scripts/serve-frank-dragoon.sh`,
|
||
`scripts/frank-dragoon.service`); the Swift1.5 payload stays on disk at
|
||
`/home/dank/models/halogen-flashnext-v2/` (its fetch script
|
||
`scripts/download-frank-dragoon-flashnext.sh` stays the door back), and the
|
||
27B-era payload beside that at `/home/dank/models/halogen-qwen3.8-27b/`.
|
||
Era write-up: [[frank-dragoon-flash-server-v2-swift15]].
|
||
|
||
## Why this payload
|
||
|
||
Frank's ask: serve
|
||
`https://huggingface.co/Quat3rnion/halogen-qwen3.8-flash-next-v2-abliterated`
|
||
with the card's recommended settings, and **save memory**.
|
||
|
||
This is the **native 2026-10-11 v2 conversion** of orcarouter's pinned BF16
|
||
Qwen3.8-Flash-Next, abliterated — the base itself, converted straight into
|
||
the v2 `.hgn` format. It is **not** the writer transplant that carried this
|
||
filename before: the artifact measures **66,687,678,432 B** where the
|
||
Swift1.5 r2 measured 66,687,678,464 B — 32 bytes apart, same filename
|
||
family, different model. Byte counts and the Hub LFS sha256 are the only way
|
||
to tell them apart, which is why the fetch script and both gates check them.
|
||
|
||
What the swap buys, measured (table below): retrieval at depth, decode,
|
||
prefill, and **7.4 GiB of RAM handed back**. What it costs: the Swift1.5
|
||
finetune's compressed reasoning and its small perplexity edge.
|
||
|
||
## The companion law (the reason this swap needed a referee)
|
||
|
||
The release publishes **only the main file**. The n-gram lookup table
|
||
(REQUIRED — the engine reads it beside the checkpoint), the vision tower,
|
||
and the tokenizer are peonist's, and the card does **not** establish them as
|
||
interchangeable with this native conversion. No source-matched pair is
|
||
published anywhere, so the only available companions are the stock ones.
|
||
|
||
They are **hardlinked** out of the verified Swift1.5 payload directory
|
||
(`scripts/download-frank-dragoon-v2abl.sh`): a hardlink is a regular file, so
|
||
it satisfies the serve gate's no-symlink law, and it costs no disk and no
|
||
copy next to a 62 GB payload. The engine's own checkpoint tools are the
|
||
referee, and they answered: **NIAH 18/18** (vs 17/18 for the payload those
|
||
files were published with), needle retrieval exact at both depths, the red-PNG
|
||
vision probe exact, perplexity in the normal band. A wrong PLE table does not
|
||
whisper — see Verification.
|
||
|
||
## The payload — one new file, three inherited ones
|
||
|
||
`/home/dank/models/halogen-v2-abliterated/`, fetched by
|
||
`scripts/download-frank-dragoon-v2abl.sh` (`-f` + exact byte counts + LFS
|
||
sha256 + `--verify-only`; no token on the command line):
|
||
|
||
|File|Bytes|sha256|Source|
|
||
|---|---|---|---|
|
||
|`qwen38-flash-next-v2-abliterated.hgn`|66,687,678,432|`b5bd925f…`|Quat3rnion `@main`, fetched|
|
||
|`qwen38-flash-next-ngram.hgn`|51,200,246,144|`3450acada…`|peonist `@5cc17cea`, **hardlinked**, REQUIRED|
|
||
|`qwen38-flash-next-vision.hgn`|897,916,416|`d62e0ae5…`|peonist `@e053f488`, hardlinked, vision ON|
|
||
|`tokenizer/` (6 files)| — |`tokenizer.json` LFS|peonist `@e053f488`, hardlinked|
|
||
|
||
The engine stays **halogen-flash-server 0.16.2**, digest-pinned (the card
|
||
tested 0.15.0; 0.16.2 is newer in the same line and is what the v2 format was
|
||
qualified on in this fleet):
|
||
|
||
```
|
||
ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
|
||
```
|
||
|
||
## Shape — the card's settings, and the three deliberate deviations
|
||
|
||
`HALOGEN_CTX=200704` (per-request ceiling — the card's 262,144 lowered the
|
||
same evening, see below), `HALOGEN_KV_SLOTS=2`,
|
||
**`HALOGEN_KV_POOL_POSITIONS=200704`** (one pool behind both slots; the card
|
||
recommends it equal to ctx, and that equality is where the memory is),
|
||
`HALOGEN_SPEC_ADAPT=0`, `HALOGEN_MAX_TOK=8192`,
|
||
`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` (coupled pair),
|
||
sampler `1.0 / 0.95 / 20`, `HALOGEN_WEIGHTS_LOCK=1`,
|
||
`HALOGEN_HOST_RESERVE_GIB=20`.
|
||
|
||
Deviations from the card, each on purpose:
|
||
|
||
- `HALOGEN_MODEL_ID=frank-dragoon`, not the card's `halogen-v2-abliterated` —
|
||
the fleet naming law (service = unit = served id).
|
||
- `HALOGEN_SPEC_ADAPT=0` — the card is silent; this is a measured 0.16.2 law
|
||
(the adaptive default disables the drafter per request and halves decode).
|
||
- the sampler values are peonist's engine recipe; Quat3rnion publishes none.
|
||
|
||
**The pool is one budget, not a partition.** Boot ledger form: `listening on
|
||
127.0.0.1:8730, 2 slots over ONE <pool>-position KV pool, each request up to
|
||
<HALOGEN_CTX>`. Both numbers were lowered to **200,704** the same evening
|
||
(the card ships 262,144; Frank took the RAM — nothing else in the fleet asks
|
||
for the bigger window), so `models.yml` `contextWindow` moved to **155,648**
|
||
= ceiling − 45k with it. With pool == ctx, one maximum-length request fills
|
||
the pool: **two long conversations can no longer share the endpoint** (two of
|
||
≤100k can). That is acceptable here because frank-dragoon carries the `slow`
|
||
(medium-thinking) role and the `security-reviewer` subagent, not the
|
||
interactive workhorse (that is frank-archon).
|
||
|
||
**Memory ledger** (121.2 GiB unified), from the boot ledgers. The 22:20 boot
|
||
printed `pinned 62.11 GiB in 1 range(s) in 1.6 s (41.0 GB/s)` trunk,
|
||
`0.84 GiB` vision tower, `reserving the KV pool, 262144 positions (about
|
||
7.2 GiB)`, and the engine's own sum: `memory: 62.9 GiB of weights locked in
|
||
RAM (0.8 of them the vision tower's), 7.2 GiB of KV pool, …, 75.8 GiB in
|
||
all`. After the same-evening ctx/pool cut to 200,704 the reservation is ~5.4
|
||
GiB and the total ~74. The n-gram table rides as page cache — `mapped 51.2 GB,
|
||
NOT pinned (Pin::None)` — which is exactly why every pool cut is a *double*
|
||
saving. FLAGS.md law: **RAM the pool takes is RAM that table loses**; the ~9
|
||
GiB the two cuts together handed back is page cache for the 47.68 GiB table
|
||
that long prompts read.
|
||
|
||
`HALOGEN_FLASH_PIN_TRUNK=0` is **rejected by the v2 loader** — the trunk
|
||
stays pinned. Disk-backing the weights is not a lever on this format.
|
||
|
||
## Measured A/B — same battery, both boots (2026-10-17)
|
||
|
||
Behavior probes ran against the live service (tagged `swift15` / `v2abl`);
|
||
the engine tools ran with the service **stopped**, per the fleet benchmarking
|
||
law, under one identical knob set (`scripts/census-frank-dragoon.sh`).
|
||
|
||
|Probe|Swift1.5 r2 (out)|native v2 (in)|
|
||
|---|---|---|
|
||
|`verify` on the file|—|OK|
|
||
|NIAH 18 cases (4k/32k × 3 depths × 3 positions)|17/18 (94.4 %)|**18/18 (100 %)**|
|
||
|wikitext-2 PPL (297,053 ids, 2 × 262,144 seqs)|**3.8772**|3.9679 (+2.3 %)|
|
||
|KL of the new model against the old model's logits|—|mean 0.155 / median 0.057, top-1 85.1 %|
|
||
|refusal census (5 hostile prompts)|0/5 refusing|0/5 refusing|
|
||
|thinking tokens, mean of 6 questions (temp 1.0)|**784**|1,476 (×1.9)|
|
||
|tool call, 1-turn / 2-turn|`{"city": "Seattle"}` / answer|`{"city": "Seattle"}` / answer|
|
||
|vision (512 px solid red, `image_url` part)|Red|Red|
|
||
|needle @21,568 tokens, planted 0.85 / 0.10|hit / hit|hit / hit|
|
||
|decode, greedy, 512 tokens|46.8 t/s (spread 25.6–46.8)|50.0 t/s (steady 50.0/50.2)|
|
||
|prefill, cold, 59,024 tokens|1,424.8 t/s|**1,604.3 t/s (+12.6 %)**|
|
||
|prompt-cache ratio (same prompt, second pass)|136.7×|131.6×|
|
||
|KV pool|14.6 GiB (524,288 positions)|**7.2 GiB (262,144)**|
|
||
|
||
Read the KL honestly: it is a **model-vs-model** divergence, not the
|
||
quant-vs-source comparison the tool was built for. Two different models on
|
||
the same text *should* differ; median 0.057 with 85 % top-1 agreement is the
|
||
signature of two sound models that disagree on content, and the worst rows
|
||
are date/name completions. A n-gram table mismatched to the checkpoint
|
||
corrupts the forward pass and shows up as elevated KL **everywhere** plus
|
||
degraded retrieval — which NIAH (18/18) and the needle probes rule out.
|
||
|
||
The decode line needs care: the baseline's own spread was 25.6–46.8 t/s
|
||
because its samples were taken while the 62 GB fetch was still streaming into
|
||
page cache, and the candidate's **first** sample after the battery's long
|
||
prompts read 32.8 before settling at 50.0/50.2. Honest claim: **no decode
|
||
regression, plausibly faster**; never quote either single number.
|
||
|
||
## Verification — how to reproduce
|
||
|
||
```
|
||
# 1. payload: bytes + LFS sha256 + the hardlinked trio
|
||
bash scripts/download-frank-dragoon-v2abl.sh --verify-only
|
||
# 2. engine tools on a STOPPED service (fleet law: no GPU experiment beside
|
||
# a live endpoint; stop the unit first — mask fails on a relinked unit)
|
||
bash scripts/census-frank-dragoon.sh verify # no GPU, safe while serving
|
||
sudo systemctl stop frank-dragoon
|
||
bash scripts/census-frank-dragoon.sh ref # reference dump, old payload
|
||
bash scripts/census-frank-dragoon.sh cand # PPL + KL/top-1 vs that dump
|
||
bash scripts/census-frank-dragoon.sh niah-cur # baseline retrieval
|
||
bash scripts/census-frank-dragoon.sh niah-new # candidate retrieval
|
||
# 3. boot the candidate through a transient drop-in, then the battery
|
||
bash scripts/trial-frank-dragoon.sh candidate
|
||
python3 scripts/battery-frank-dragoon.py v2abl all
|
||
bash scripts/trial-frank-dragoon.sh restore # back onto the repo env
|
||
```
|
||
|
||
The battery needs two staged assets in `/tmp/halogen-eval/` (a 512 px solid
|
||
red PNG as base64, `red512.b64`, and the wikitext-2 raw test split,
|
||
`wikitext-2-raw-test.txt`). Laws it encodes:
|
||
|
||
- **the engine bills wikitext at ~4.33 characters/token** — the harness
|
||
chars/4 estimate under-bills by ~8 %, and a prompt over `HALOGEN_CTX` is
|
||
refused with a 400 that names the exact overage (never truncated). Size
|
||
haystacks in characters against a **token** budget.
|
||
- the wikitext-2 raw test split as staged joins rows with a **single**
|
||
newline (2,891 lines, zero blank lines), so a blank-line split returns one
|
||
1.29 MB "paragraph" and every budget silently becomes "everything".
|
||
- the first request after a boot is a warmup artifact; the battery discards
|
||
it (`warmup` row) and never measures it.
|
||
- `ppl`/`niah` **load the model** → stopped service. `verify` reads the file
|
||
back with no GPU and can run while serving.
|
||
|
||
## Contract (unchanged from the Swift1.5 era)
|
||
|
||
`reasoning_content` carries the thinking and counts against `max_tokens`
|
||
(≥512-token budget on any vision probe or the endpoint looks blind while it
|
||
is mid-thought); vision via OpenAI `image_url` content parts only, ≥256 px
|
||
probes; `response_format` unsupported; the chat template rejects
|
||
`reasoning_effort` outside `xhigh|medium|low` (the `models.yml` fold stays
|
||
mandatory); unknown sampling params are a 400; the prompt cache is on by
|
||
default, so a warm 100k follow-up costs seconds. See
|
||
[[frank-dragoon-flash-server-v2-swift15]] for the engine-flag detail that
|
||
carries verbatim, and [[deployment]] for the deploy/health gate.
|
||
|
||
## Production boot — the deploy ledger (2026-10-17 22:20, `fcd7630`)
|
||
|
||
`./scripts/deploy.sh dankclaw --service frank-dragoon`: healthy after ~12 s,
|
||
deploy marker `fcd7630`, e2e gate 33/33. The trial drop-in went to
|
||
`~/.AGENTTRASH/20261017-v2abl-trial/` FIRST, so this boot is governed by the
|
||
repo env — `systemctl show -p DropInPaths` is empty.
|
||
|
||
Ledger lines that settle the shape (engine stdout, not the journal):
|
||
|
||
|Line|Figure|
|
||
|---|---|
|
||
|preflight|`MemAvailable now 105.4 GiB` against the engine's own estimate of 62.1 weights + 6.0 working + 7.4 KV + 1.3 prompt cache|
|
||
|weights|`62.9 GiB of weights locked in RAM (0.8 of them the vision tower's)`|
|
||
|KV pool|`reserving the KV pool, 262144 positions (about 7.2 GiB)`; `the KV is ONE pool of 262144 positions (6656.0 MiB) carved into per-request regions`; slots `1 -> 2 POOLED, 111.0 MiB each of O(1) state`|
|
||
|working memory|5.7 GiB at `HALOGEN_MAX_TOK 8192`|
|
||
|total|`75.8 GiB in all` — ~25 GiB of the 121.2 GiB box left for co-tenants|
|
||
|n-gram|`47.68 GiB from /models/qwen38-flash-next-ngram.hgn`, `checkpoint: mapped 51.2 GB, NOT pinned (Pin::None)` — disk-backed exactly as designed|
|
||
|prompt cache|ON, resume-anywhere, 12 entries, chunk 32768, `HALOGEN_CACHE_INPLACE=1` (the KV rows are never copied)|
|
||
|spec decode|`MTP depth-2 selectable (drafter 1, speculates while alone)`|
|
||
|clamps|none — the requested pool is the delivered pool|
|
||
|
||
Read the two KV figures together: the engine's preflight FORMULA bills 7.4 GiB
|
||
(~28 KiB/position **including** block scratch and ~115 MiB/slot of O(1) state)
|
||
while the RESERVATION is 7.2 GiB / 6,656 MiB. Quote the reservation for the
|
||
RAM ledger and the formula when comparing pool sizes.
|
||
|
||
Live probes on the deployed endpoint: `27*59` → `1593` with a warm thinking
|
||
channel, tool call `{"city": "Seattle"}`, 512 px solid-red PNG → **Red** (320
|
||
prompt tokens — the image reached the model), greedy 512-token decode 46.9 t/s
|
||
on the FIRST request after boot, i.e. the warmup artifact; the trial's settled
|
||
figure is 50.0. OpenWebUI's `frank-dragoon` row description re-keyed to the
|
||
native payload (backup `webui.db.bak-20261017b`; no container restart forced —
|
||
forcing it logs every family session out for one cosmetic string).
|