docs: frank-large flash-server v2 cutover — runbook, index, AGENTS.md
- New runbook: frank-large-flash-server-v2-swift15.md (engine, shape, memory model, engine laws, boot ledger, probes, rollback) - LLM/index.md: active entry + retired halogen 27B entry - Vault index: link the new runbook from the LLM Fleet section - AGENTS.md: fleet topology table, Model Config section, Runbook Index, retired table (halogen 27B → 10-06)
This commit is contained in:
Frank committed
1 parent
20d7182f4b
commit
2d89bf97cc
3 files changed
+312
-109
No files matched your search
@@ -0,0 +1,178 @@
|
||||
# frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, Strix Halo iGPU)
|
||||
|
||||
**ACTIVE since 2026-10-06.** Rollback = revert the swap commit's trio
|
||||
(`scripts/frank-large.env`, `scripts/serve-frank-large.sh`,
|
||||
`scripts/frank-large.service`); the retired 27B payload stays on disk at
|
||||
`/home/dank/models/halogen-qwen3.8-27b/` and its fetch script
|
||||
(`scripts/download-frank-large-halogen.sh`) stays in the repo.
|
||||
|
||||
## Why this shape exists
|
||||
|
||||
Frank's ask: replace the frank-large model with
|
||||
`https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`.
|
||||
That payload is an **abliterated Swift1.5 finetune of peonist's
|
||||
Qwen3.8-Flash-Next v2 checkpoint in the v2 `.hgn` format** — a file that
|
||||
loads ONLY in the **halogen-flash-server** engine line, NOT in the halogen
|
||||
0.1.4 container that served the 27B dense checkpoint. The model move is
|
||||
therefore a **full engine migration**, and the accepted-product facts flip
|
||||
twice in the fleet's favour:
|
||||
|
||||
- the base is **abliterated** (the 27B halogen endpoint was a finetune of
|
||||
the censored `Qwen/Qwen3.8-27B`; refusals were an accepted regression there),
|
||||
- **vision is back** — the v2 vision tower loads beside the checkpoint and
|
||||
the engine runs it (the 27B halogen engine silently ignored its encoder),
|
||||
- **MTP speculative decoding returns** with the head inside the checkpoint
|
||||
itself (the 27B shipped a DFlash2 drafter instead; comparable purpose,
|
||||
different file plumbing),
|
||||
- the card claims **53–57 t/s greedy decode, ~1,640 t/s prefill at 34k, ~72 %
|
||||
fewer thinking tokens** than the censored 27B (905 mean), and **262,144
|
||||
native context**. Publisher-card numbers — verify against live traffic
|
||||
before quoting further (see Verification).
|
||||
|
||||
The engine is **closed-source**, ships as `ghcr.io/peonist-ai/
|
||||
halogen-flash-server:0.16.2` (the version the model card's quickstart pins),
|
||||
digest-pinned:
|
||||
|
||||
```
|
||||
ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
|
||||
```
|
||||
|
||||
(3.6 GB image; RepoDigest captured at pull 2026-10-06. Card-tested fallback
|
||||
if a future pin misbehaves: 0.15.1.)
|
||||
|
||||
## The payload — a triangle of files, all sha256-verified at fetch
|
||||
|
||||
| File | Bytes | Source repo @ revision | sha256 (LFS) |
|
||||
|---|---|---|---|
|
||||
| `qwen38-flash-next-v2-swift15-abliterated.hgn` | 66,687,678,464 | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated @ main (`69a85db`) | `9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037` |
|
||||
| `qwen38-flash-next-ngram.hgn` (REQUIRED) | 51,200,246,144 | peonist-ai/halogen-qwen3.8-flash-next @ `5cc17cea` | `3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe` |
|
||||
| `qwen38-flash-next-vision.hgn` (enabled) | 897,916,416 | peonist-ai/halogen-qwen3.8-flash-next @ `e053f488` | `d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10` |
|
||||
| `tokenizer/` (6 files, flat dir) | 22.4 MB total | peonist @ `e053f488` | `tokenizer.json` is LFS `0997f410…` (byte-identical to the 27B-era copy); the rest are plain git blobs, size-checked |
|
||||
|
||||
Payload notes worth keeping:
|
||||
|
||||
- The deployed checkpoint is a **compacted r2** of the card's append-only
|
||||
original (89,775,868,032 B → 66,687,678,464 B): the abliteration workflow
|
||||
appends and the repo's `tooling/` compact drops dead padding; live tensors
|
||||
are byte-identical, the LFS sha256 is the contract, and the 24 GiB saving
|
||||
is why it fits.
|
||||
- **`HALOGEN_MTP_HEAD` stays UNSET**: the MTP head is INSIDE the v2
|
||||
checkpoint (the separate `qwen38-flash-next-mtp.hgn` sidecar is for a
|
||||
different assembly and is deliberately not fetched).
|
||||
- The **n-gram lookup table is REQUIRED**. The engine reads it beside the
|
||||
checkpoint (documented fallback); the card's quickstart also names it
|
||||
explicitly via `HALOGEN_NGRAM_TABLE`, and the serve script passes both —
|
||||
the env var is belt-and-braces over the documented beside-the-checkpoint
|
||||
default, and both mechanisms resolve to the same file.
|
||||
- The tokenizer is fetched fresh from peonist @ `e053f488` into
|
||||
`/home/dank/models/halogen-flashnext-v2/tokenizer/` — same flat-directory,
|
||||
no-symlink law as every other payload (an HF cache snapshot's symlinks
|
||||
dangle inside the container mount; both the serve gate and the deployer
|
||||
refuse them).
|
||||
|
||||
Fetch: `scripts/download-frank-large-flashnext.sh` (`--verify-only` to
|
||||
re-check; `-f` + exact byte counts + LFS sha256 + resolve-the-CDN-then-`-C -`
|
||||
resume law throughout; no token — the repos are public).
|
||||
|
||||
## Shape — the knobs that are not free
|
||||
|
||||
`scripts/frank-large.env` carries the full rationale inline; the table:
|
||||
|
||||
| Env | Value | Why |
|
||||
|---|---|---|
|
||||
| `HALOGEN_CTX` | 200192 | per-slot HARD ceiling (a longer prompt is a 400 naming the limit, never truncation). Harness compaction: 70 % = 140,134; 140,134 + 57,344 = 197,478 ≤ 200,192 |
|
||||
| `HALOGEN_KV_SLOTS` | 2 | the 27B engine's "slots>1 disables the drafter" law does NOT carry over — on this engine batching and speculation coexist (the card's quickstart ships 2 slots WITH the head). 2 slots = concurrent capacity |
|
||||
| `HALOGEN_KV_POOL_POSITIONS` | 400384 | explicit form of the default (2 × ctx), so the seam test pins the product; ~28 KiB/position ⇒ ~10.7 GiB KV |
|
||||
| `HALOGEN_HOST_RESERVE_GIB` | 20 | keeps the host honest (the w4b-era lesson) |
|
||||
| `HALOGEN_SPEC_ADAPT` | **0** | MANDATORY (card): the default adaptive policy turns the MTP head off for whole requests after low acceptance and **halves decode**. The head stays on; temp-1.0 traffic pays some acceptance for it — accepted trade |
|
||||
| `HALOGEN_MAX_TOK` / `HALOGEN_MAX_TOKENS_CAP` / `HALOGEN_QUEUE_TIMEOUT` | 8192 / 65536 / 7200 | 8192 is only the per-request DEFAULT (the harness always sends 57,344); the cap/timeout pair is COUPLED per FLAGS.md (65536 at ~10 t/s serial needs > 6550 s) |
|
||||
| `HALOGEN_TEMPERATURE` / `TOP_P` / `TOP_K` | 1.0 / 0.95 / 20 | the card's thinking recipe as SERVER-SIDE defaults — the engine HAS a sampler now, so an absent temperature is no longer greedy; models.yml extraBody matches it (consistent, not load-bearing) |
|
||||
| `HALOGEN_WEIGHTS_LOCK` | 1 | card's belt-and-braces; the engine pins weights by default on v2 (`HALOGEN_FLASH_PIN_TRUNK=0` is a w4b-only knob and is REFUSED by v2) |
|
||||
| `HALOGEN_MODEL_ID` / publish | frank-large / 7770:8731 | unchanged aliases; engine port 8730 never published |
|
||||
| MTP depth | default 2 | FLAGS.md: 2 is best all-round, 3 for code — Frank's traffic is agent/prose |
|
||||
|
||||
**Memory ledger** (121.2 GiB unified, the referee is the boot ledger):
|
||||
weights ~62 GiB resident + KV ~10.7 + arena ~8 (`3.8 + 8.4 × 8192/16384`)
|
||||
+ vision ~2 ≈ **82.7 GiB engine-side**, + 20 host reserve = ~103 with
|
||||
~18 GiB margin over the box's baseline co-tenancy. The 51.2 GB n-gram
|
||||
table rides as mmap'd page cache beside the checkpoint — the card's own
|
||||
quickstart arithmetic does not bill it as resident, but WATCH the ledger
|
||||
at boot and re-size if the engine prints a clamp. Prompt cache is ON by
|
||||
default (`PROMPT_CACHE=2`): a follow-up turn at 100k ctx costs ~2 s
|
||||
instead of ~88 s cold — the big-TTFT win the 27B engine needed a manual
|
||||
`HALOGEN_CACHE_MB` pin for is built in here.
|
||||
|
||||
## Engine contract facts (FLAGS.md of the flash-server repo)
|
||||
|
||||
- OpenAI chat **and** `/v1/responses` **and** `/v1/messages` **and**
|
||||
`/v1/completions`; tool calls verified on the card; `response_format`
|
||||
still unsupported (no constrained decoding).
|
||||
- Thinking → **`reasoning_content`** (same field as every engine this
|
||||
endpoint has run); the v2 chat template rejects `reasoning_effort`
|
||||
outside `xhigh|medium|low` — the models.yml fold (high→xhigh) stays
|
||||
mandatory.
|
||||
- Vision: OpenAI `image_url` content parts only (the fleet's standard
|
||||
contract; the legacy top-level `images` field is the frank-small trap).
|
||||
≥256 px probe images or the model answers "video".
|
||||
- `HALOGEN_MAX_TOKENS_CAP` over-cap is a 400 naming the limit — never a
|
||||
truncation. `max_tokens` INCLUDES thinking.
|
||||
- All flags read once at startup; every flag is BITWISE except
|
||||
`HALOGEN_CK_OVERLAY`, `HALOGEN_MATMUL_TUNING_FILE`,
|
||||
`HALOGEN_INDEXER_BUDGET`, `HALOGEN_COMPOSABLE_CONTEXT`. Unset
|
||||
`HALOGEN_CHECKPOINT` defaults to `/models/qwen38-flash-next-v2.hgn`
|
||||
with the lookup table beside it.
|
||||
- Prompt-lookup drafting (3,3 window) rides beside the MTP head by
|
||||
default and is byte-identical to greedy verification where it fires.
|
||||
- Watchdog: `HALOGEN_ENGINE_WATCHDOG_S=180` with defer-logic for memory
|
||||
stalls — on a shared host the defaults are the safe choice; the
|
||||
restart policy stays real through the root unit's `Restart=always`.
|
||||
|
||||
Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it); `amdgpu.noretry=-1`
|
||||
(auto) — for this IP auto resolves to retry-on, the same effective state
|
||||
the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
|
||||
|
||||
## Mechanics
|
||||
|
||||
- **Root systemd unit** (`scripts/frank-large.service`, `User=dank`,
|
||||
`LimitMEMLOCK=infinity`, `Restart=always`, `ExecStop=podman stop -t 60
|
||||
frank-large-halogen`), driven by `scripts/serve-frank-large.sh` — the
|
||||
fleet's `podman run`-direct pattern. The script runs the container
|
||||
FOREGROUND so the boot ledger (weights resident, pool fit, MTP head,
|
||||
prompt cache, any clamp) lands in `/home/dank/logs/frank_large.log`.
|
||||
- Serve gates mirror the deployer EXACTLY (2026-10-01 crash-loop law):
|
||||
image digest in podman storage, byte-exact checkpoint + n-gram table +
|
||||
vision tower, flat tokenizer with all six files as regular files,
|
||||
GPU nodes present.
|
||||
- Engine stdout goes to the log, NOT the journal. The FIRST request after
|
||||
a (re)start can sit silent ~45–60 s on this engine class (health
|
||||
`engine_unresponsive`, 503s) and self-recovers — do not restart over it.
|
||||
|
||||
## Verification checklist (run after any redeploy)
|
||||
|
||||
1. Boot ledger greps in `/home/dank/logs/frank_large.log`: resident
|
||||
weights (~62 GiB), KV pool line, MTP/drafter line, prompt-cache line,
|
||||
**no clamp named**.
|
||||
2. `curl http://192.168.0.143:7770/health` → 200.
|
||||
3. Text completion probe: reasoning lands in `reasoning_content`,
|
||||
`timings` honest (generate ≥128 tokens; at 8 the inter_token number is
|
||||
first-token noise).
|
||||
4. Vision probe: a 512×512 solid-red PNG through an `image_url` part →
|
||||
"Red" (≥256 px, one word expected).
|
||||
5. Spec decode: `draft_n`/`draft_n_accepted` present in timings on
|
||||
real-shaped traffic (SPEC_ADAPT=0 means the head is always engaged).
|
||||
|
||||
## Client contract (harness)
|
||||
|
||||
`~/.omp/agent/models.yml` — provider `llamacpp (dankclaw)` @
|
||||
`http://192.168.0.143:7770/v1`: `contextWindow: 200192`,
|
||||
`maxTokens: 57344`, `input: [text, image]`,
|
||||
`reasoningContentField: "reasoning_content"`, the `high→xhigh` effort
|
||||
fold, `extraBody` temperature 1.0 / top_p 0.95. Compaction still rides
|
||||
frank-small (the `compactionModel` steering is unchanged).
|
||||
|
||||
## Rollback
|
||||
|
||||
Revert the swap commit and redeploy — the 27B trio is intact on disk
|
||||
(payload at `~/models/halogen-qwen3.8-27b/`, fetch script, and the halogen
|
||||
0.1.4 image digest still in podman storage). The older ROCmFPX rollback
|
||||
door (`build-rocmfpx.sh` pin) is untouched.
|
||||
Reference in new issue
Block a user