Frank
87721ab0fe
Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:
- Service trio: scripts/{frank-small.env,frank-small.service,
serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
frank-secondary-*.md (9 files); the FreeToken runbook stays
freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
3090, and the Dankserver service that became frank-medium) are
preserved verbatim — they describe what the service was called
at the time.
192 lines
12 KiB
Markdown
192 lines
12 KiB
Markdown
# frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, Strix Halo iGPU)
|
||
|
||
**ACTIVE since 2026-10-06.** Rollback = revert the swap commit's trio
|
||
(`scripts/frank-large.env`, `scripts/serve-frank-large.sh`,
|
||
`scripts/frank-large.service`); the retired 27B payload stays on disk at
|
||
`/home/dank/models/halogen-qwen3.8-27b/` and its fetch script
|
||
(`scripts/download-frank-large-halogen.sh`) stays in the repo.
|
||
|
||
## Why this shape exists
|
||
|
||
Frank's ask: replace the frank-large model with
|
||
`https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`.
|
||
That payload is an **abliterated Swift1.5 finetune of peonist's
|
||
Qwen3.8-Flash-Next v2 checkpoint in the v2 `.hgn` format** — a file that
|
||
loads ONLY in the **halogen-flash-server** engine line, NOT in the halogen
|
||
0.1.4 container that served the 27B dense checkpoint. The model move is
|
||
therefore a **full engine migration**, and the accepted-product facts flip
|
||
twice in the fleet's favour:
|
||
|
||
- the base is **abliterated** (the 27B halogen endpoint was a finetune of
|
||
the censored `Qwen/Qwen3.8-27B`; refusals were an accepted regression there),
|
||
- **vision is back** — the v2 vision tower loads beside the checkpoint and
|
||
the engine runs it (the 27B halogen engine silently ignored its encoder),
|
||
- **MTP speculative decoding returns** with the head inside the checkpoint
|
||
itself (the 27B shipped a DFlash2 drafter instead; comparable purpose,
|
||
different file plumbing),
|
||
- the card claims **53–57 t/s greedy decode, ~1,640 t/s prefill at 34k, ~72 %
|
||
fewer thinking tokens** than the censored 27B (905 mean), and **262,144
|
||
native context**. Publisher-card numbers — verify against live traffic
|
||
before quoting further (see Verification).
|
||
|
||
The engine is **closed-source**, ships as `ghcr.io/peonist-ai/
|
||
halogen-flash-server:0.16.2` (the version the model card's quickstart pins),
|
||
digest-pinned:
|
||
|
||
```
|
||
ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
|
||
```
|
||
|
||
(3.6 GB image; RepoDigest captured at pull 2026-10-06. Card-tested fallback
|
||
if a future pin misbehaves: 0.15.1.)
|
||
|
||
## The payload — a triangle of files, all sha256-verified at fetch
|
||
|
||
| File | Bytes | Source repo @ revision | sha256 (LFS) |
|
||
|---|---|---|---|
|
||
| `qwen38-flash-next-v2-swift15-abliterated.hgn` | 66,687,678,464 | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated @ main (`69a85db`) | `9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037` |
|
||
| `qwen38-flash-next-ngram.hgn` (REQUIRED) | 51,200,246,144 | peonist-ai/halogen-qwen3.8-flash-next @ `5cc17cea` | `3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe` |
|
||
| `qwen38-flash-next-vision.hgn` (enabled) | 897,916,416 | peonist-ai/halogen-qwen3.8-flash-next @ `e053f488` | `d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10` |
|
||
| `tokenizer/` (6 files, flat dir) | 22.4 MB total | peonist @ `e053f488` | `tokenizer.json` is LFS `0997f410…` (byte-identical to the 27B-era copy); the rest are plain git blobs, size-checked |
|
||
|
||
Payload notes worth keeping:
|
||
|
||
- The deployed checkpoint is a **compacted r2** of the card's append-only
|
||
original (89,775,868,032 B → 66,687,678,464 B): the abliteration workflow
|
||
appends and the repo's `tooling/` compact drops dead padding; live tensors
|
||
are byte-identical, the LFS sha256 is the contract, and the 24 GiB saving
|
||
is why it fits.
|
||
- **`HALOGEN_MTP_HEAD` stays UNSET**: the MTP head is INSIDE the v2
|
||
checkpoint (the separate `qwen38-flash-next-mtp.hgn` sidecar is for a
|
||
different assembly and is deliberately not fetched).
|
||
- The **n-gram lookup table is REQUIRED**. The engine reads it beside the
|
||
checkpoint (documented fallback); the card's quickstart also names it
|
||
explicitly via `HALOGEN_NGRAM_TABLE`, and the serve script passes both —
|
||
the env var is belt-and-braces over the documented beside-the-checkpoint
|
||
default, and both mechanisms resolve to the same file.
|
||
- The tokenizer is fetched fresh from peonist @ `e053f488` into
|
||
`/home/dank/models/halogen-flashnext-v2/tokenizer/` — same flat-directory,
|
||
no-symlink law as every other payload (an HF cache snapshot's symlinks
|
||
dangle inside the container mount; both the serve gate and the deployer
|
||
refuse them).
|
||
|
||
Fetch: `scripts/download-frank-large-flashnext.sh` (`--verify-only` to
|
||
re-check; `-f` + exact byte counts + LFS sha256 + resolve-the-CDN-then-`-C -`
|
||
resume law throughout; no token — the repos are public).
|
||
|
||
## Shape — the knobs that are not free
|
||
|
||
`scripts/frank-large.env` carries the full rationale inline; the table:
|
||
|
||
| Env | Value | Why |
|
||
|---|---|---|
|
||
| `HALOGEN_CTX` | 200192 | per-slot HARD ceiling (a longer prompt is a 400 naming the limit, never truncation). Harness compaction: 70 % = 140,134; 140,134 + 57,344 = 197,478 ≤ 200,192 |
|
||
| `HALOGEN_KV_SLOTS` | 2 | the 27B engine's "slots>1 disables the drafter" law does NOT carry over — on this engine batching and speculation coexist (the card's quickstart ships 2 slots WITH the head). 2 slots = concurrent capacity |
|
||
| `HALOGEN_KV_POOL_POSITIONS` | 400384 | explicit form of the default (2 × ctx), so the seam test pins the product; ~28 KiB/position ⇒ ~10.7 GiB KV |
|
||
| `HALOGEN_HOST_RESERVE_GIB` | 20 | keeps the host honest (the w4b-era lesson) |
|
||
| `HALOGEN_SPEC_ADAPT` | **0** | MANDATORY (card): the default adaptive policy turns the MTP head off for whole requests after low acceptance and **halves decode**. The head stays on; temp-1.0 traffic pays some acceptance for it — accepted trade |
|
||
| `HALOGEN_MAX_TOK` / `HALOGEN_MAX_TOKENS_CAP` / `HALOGEN_QUEUE_TIMEOUT` | 8192 / 65536 / 7200 | 8192 is only the per-request DEFAULT (the harness always sends 57,344); the cap/timeout pair is COUPLED per FLAGS.md (65536 at ~10 t/s serial needs > 6550 s) |
|
||
| `HALOGEN_TEMPERATURE` / `TOP_P` / `TOP_K` | 1.0 / 0.95 / 20 | the card's thinking recipe as SERVER-SIDE defaults — the engine HAS a sampler now, so an absent temperature is no longer greedy; models.yml extraBody matches it (consistent, not load-bearing) |
|
||
| `HALOGEN_WEIGHTS_LOCK` | 1 | card's belt-and-braces; the engine pins weights by default on v2 (`HALOGEN_FLASH_PIN_TRUNK=0` is a w4b-only knob and is REFUSED by v2) |
|
||
| `HALOGEN_MODEL_ID` / publish | frank-large / 7770:8731 | unchanged aliases; engine port 8730 never published |
|
||
| MTP depth | default 2 | FLAGS.md: 2 is best all-round, 3 for code — Frank's traffic is agent/prose |
|
||
|
||
**Memory ledger** (121.2 GiB unified, the referee is the boot ledger):
|
||
weights ~62 GiB resident + KV ~10.7 + arena ~8 (`3.8 + 8.4 × 8192/16384`)
|
||
+ vision ~2 ≈ **82.7 GiB engine-side**, + 20 host reserve = ~103 with
|
||
~18 GiB margin over the box's baseline co-tenancy. The 51.2 GB n-gram
|
||
table rides as mmap'd page cache beside the checkpoint — the card's own
|
||
quickstart arithmetic does not bill it as resident, but WATCH the ledger
|
||
at boot and re-size if the engine prints a clamp. Prompt cache is ON by
|
||
default (`PROMPT_CACHE=2`): a follow-up turn at 100k ctx costs ~2 s
|
||
instead of ~88 s cold — the big-TTFT win the 27B engine needed a manual
|
||
`HALOGEN_CACHE_MB` pin for is built in here.
|
||
|
||
## Engine contract facts (FLAGS.md of the flash-server repo)
|
||
|
||
- OpenAI chat **and** `/v1/responses` **and** `/v1/messages` **and**
|
||
`/v1/completions`; tool calls verified on the card; `response_format`
|
||
still unsupported (no constrained decoding).
|
||
- Thinking → **`reasoning_content`** (same field as every engine this
|
||
endpoint has run); the v2 chat template rejects `reasoning_effort`
|
||
outside `xhigh|medium|low` — the models.yml fold (high→xhigh) stays
|
||
mandatory.
|
||
- Vision: OpenAI `image_url` content parts only (the fleet's standard
|
||
contract; the legacy top-level `images` field is the frank-secondary trap).
|
||
≥256 px probe images or the model answers "video".
|
||
- `HALOGEN_MAX_TOKENS_CAP` over-cap is a 400 naming the limit — never a
|
||
truncation. `max_tokens` INCLUDES thinking.
|
||
- All flags read once at startup; every flag is BITWISE except
|
||
`HALOGEN_CK_OVERLAY`, `HALOGEN_MATMUL_TUNING_FILE`,
|
||
`HALOGEN_INDEXER_BUDGET`, `HALOGEN_COMPOSABLE_CONTEXT`. Unset
|
||
`HALOGEN_CHECKPOINT` defaults to `/models/qwen38-flash-next-v2.hgn`
|
||
with the lookup table beside it.
|
||
- Prompt-lookup drafting (3,3 window) rides beside the MTP head by
|
||
default and is byte-identical to greedy verification where it fires.
|
||
- Watchdog: `HALOGEN_ENGINE_WATCHDOG_S=180` with defer-logic for memory
|
||
stalls — on a shared host the defaults are the safe choice; the
|
||
restart policy stays real through the root unit's `Restart=always`.
|
||
|
||
Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it); `amdgpu.noretry=-1`
|
||
(auto) — for this IP auto resolves to retry-on, the same effective state
|
||
the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
|
||
|
||
## Mechanics
|
||
|
||
- **Root systemd unit** (`scripts/frank-large.service`, `User=dank`,
|
||
`LimitMEMLOCK=infinity`, `Restart=always`, `ExecStop=podman stop -t 60
|
||
frank-large-halogen`), driven by `scripts/serve-frank-large.sh` — the
|
||
fleet's `podman run`-direct pattern. The script runs the container
|
||
FOREGROUND so the boot ledger (weights resident, pool fit, MTP head,
|
||
prompt cache, any clamp) lands in `/home/dank/logs/frank_large.log`.
|
||
- Serve gates mirror the deployer EXACTLY (2026-10-01 crash-loop law):
|
||
image digest in podman storage, byte-exact checkpoint + n-gram table +
|
||
vision tower, flat tokenizer with all six files as regular files,
|
||
GPU nodes present.
|
||
- **Host-RAM pin-gate preflight** (added 2026-10-06): the engine
|
||
refuses to pin when `MemAvailable` is below the checkpoint (62.11 GiB)
|
||
plus a **16 GiB floor** — pinned pages cannot be swapped, and under the
|
||
floor the n-gram lookup pages in from disk on every long prompt. A
|
||
RAM-short host crash-loops the unit (each boot dies in ~5 s at the pin
|
||
gate) until the RAM clears; the 2026-10-06 cutover hit this (8 boots,
|
||
MemAvailable 74.77 GiB vs the 78.11 GiB needed, self-resolved in ~2 min
|
||
when the dying 27B engine freed RAM). The serve script now checks
|
||
`MemAvailable` BEFORE `podman run` and WAITS up to `FRANK_LARGE_RAM_WAIT`
|
||
(default 600 s) for the RAM to clear, so a transient dip boots once
|
||
instead of crash-looping; if it never clears it refuses loudly. The 16
|
||
GiB floor is the engine's (not `HALOGEN_HOST_RESERVE_GIB=20` — a
|
||
separate, higher, overall-budget knob).
|
||
- Engine stdout goes to the log, NOT the journal. The FIRST request after
|
||
a (re)start can sit silent ~45–60 s on this engine class (health
|
||
`engine_unresponsive`, 503s) and self-recovers — do not restart over it.
|
||
|
||
## Verification checklist (run after any redeploy)
|
||
|
||
1. Boot ledger greps in `/home/dank/logs/frank_large.log`: resident
|
||
weights (~62 GiB), KV pool line, MTP/drafter line, prompt-cache line,
|
||
**no clamp named**.
|
||
2. `curl http://192.168.0.143:7770/health` → 200.
|
||
3. Text completion probe: reasoning lands in `reasoning_content`,
|
||
`timings` honest (generate ≥128 tokens; at 8 the inter_token number is
|
||
first-token noise).
|
||
4. Vision probe: a 512×512 solid-red PNG through an `image_url` part →
|
||
"Red" (≥256 px, one word expected).
|
||
5. Spec decode: `draft_n`/`draft_n_accepted` present in timings on
|
||
real-shaped traffic (SPEC_ADAPT=0 means the head is always engaged).
|
||
|
||
## Client contract (harness)
|
||
|
||
`~/.omp/agent/models.yml` — provider `llamacpp (dankclaw)` @
|
||
`http://192.168.0.143:7770/v1`: `contextWindow: 200192`,
|
||
`maxTokens: 57344`, `input: [text, image]`,
|
||
`reasoningContentField: "reasoning_content"`, the `high→xhigh` effort
|
||
fold, `extraBody` temperature 1.0 / top_p 0.95. Compaction still rides
|
||
frank-secondary (the `compactionModel` steering is unchanged).
|
||
|
||
## Rollback
|
||
|
||
Revert the swap commit and redeploy — the 27B trio is intact on disk
|
||
(payload at `~/models/halogen-qwen3.8-27b/`, fetch script, and the halogen
|
||
0.1.4 image digest still in podman storage). The older ROCmFPX rollback
|
||
door (`build-rocmfpx.sh` pin) is untouched.
|