--- type: reference title: halogen-flash-server on Dankclaw (frank-large) description: The active frank-large runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing. tags: - llm - halogen - dankclaw - frank-large - strix-halo - gfx1151 - qwen4exp - rocm timestamp: 2026-09-12 --- # halogen-flash-server on Dankclaw (frank-large) > **ACTIVE since 2026-09-11.** Replaced the llama.cpp / kingjones30-ROCmFPX > STRIX deployment (see [[kingjones-strix-rocmfp4]], now the rollback target). > Deployed as a rootless Quadlet container, > `dankclaw/frank-large/quadlet/frank-large.container`, port 7770. The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs. Upstream's own comparison table measures it at **roughly 4x the prefill throughput** of the llama.cpp-family runtimes it ships against (EngramHalo.cpp, ROCmFP4, CIRU-IU4) on a 32K prompt, and prefill is most of the wall clock on any prompt with real context. That, not decode, is the reason for the swap. ## Shape One container runs both halves of the stack: | half | port | reachable | |---|---|---| | engine | 8730 | container loopback only — **unauthenticated, never publish it** | | OpenAI front-end | 8731 | published as host **7770** | The `all` mode is upstream's default and the shape a single-container host wants: the front-end talks to the engine over loopback inside the container, so there is one process tree, one log, and one published port. `HALOGEN_MODEL_ID=frank-large` sets the id reported at `/v1/models`, in `/health`, and in every response body. Requests are **never rejected for naming a different model** — the server serves whatever asks. Keeping the fleet alias is therefore free, and it means Hermes, the harness config, the OpenWebUI discovery, the `services.json` probe and the docs needed no rename. ## Weights `peonist-ai/halogen-qwen3.8-flash-next` (HuggingFace, public and ungated): | file | size | role | |---|---|---| | `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint | | `qwen38-flash-next-w4b.overlay.hgn` | 2.31 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing | | `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) | | `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount | `HALOGEN_DOWNLOAD` fetches the whole repo into the bind mount on **first start only** — the entrypoint acts only when the checkpoint is absent, so a restart never re-downloads, and `huggingface_hub` resumes an interrupted pull. The mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in `/home/dank/models/halogen`. `overlay-speed.hgn` (2.22 GiB) comes along with the repo but is unused: it trades the `o_proj` calibration for about 2 % of decode, and the quality sidecar is the default for a reason. ## Deployed flags Everything else is configured by environment variable — there is no config file, and the flags are read once at startup. These are the departures from upstream's defaults: | flag | value | why | |---|---|---| | `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename | | `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare | | `HALOGEN_KV_POOL_POSITIONS` | `524288` | **in force** — two full-length conversations resident, or four at 131k, 13,312 MiB of KV. Was silently clamped to 262,144 until the reserve below dropped; `HALOGEN_KV_POOL_FIT` still guards the boot, so read the ledger rather than trusting this column | | `HALOGEN_KV_SLOTS` | `3` | Concurrency cap — **lowered from 4 on 2026-09-12** to the real fleet size. Each slot is 111.0 MiB of O(1) state, so this freed ~111 MiB (held 103.5 → 103.4 GiB). Slots share the ONE pool and **oversubscribe** it (4 × 262144 > 524288), so admission was always pool-bound: at most two full-length conversations, or four at 131k. This trims simultaneous requests, not how much context fits | | `HALOGEN_MAX_TOK` | `32768` | upstream's default, kept. The arena is 20.6 GiB here against 12.2 at 16384; it only fits alongside the doubled pool because the reserve dropped to 15. At 65536 the arena alone exceeds the budget and the server **will not boot** | | `HALOGEN_HOST_RESERVE_GIB` | `15` | **lowered from 24 (vendor default 20) on 2026-09-12** to buy both features. The only term you own in the device budget (24 → 29.5 GiB, 15 → 38.5 GiB), and **the ceiling for the doubled pool at `max_tok=32768` is 16** — the engine needs budget ≥ 36.5, and 18 gives 35.5, which silently re-clamps the pool. Matched pair with the pool. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away | | `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired. Charges 1.97 GiB against the budget — the term missing from my first estimate | | `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch | | `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug | Everything else is upstream's default, deliberately: prompt cache mode `2` (two entries per conversation — end-of-system-prompt and end-of-history — which already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at temperature 0, and it stands down whenever a second conversation is generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value gives up the byte-identity property), and the default `HALOGEN_MAX_TOKENS_CAP=65536`. ## Memory model Three numbers explain the whole box: - **~68 GiB pinned weights.** The checkpoint is mapped and registered with the GPU in place, never copied. The kernel counts the locked pages as reclaimable file cache, so `free`, `MemAvailable` and every dashboard built on them **overstate free memory by the size of the model**. The server prints the corrected figure at startup; believe that line, not Grafana. - **~28 KiB per KV position** (measured: 262,144 positions = 6,656 MiB of KV, ~7.2 GiB with block scratch), plus the prefill arena (16.7 GiB at `HALOGEN_MAX_TOK=32768`, 8.4 GiB at 16384). The engine's device-side budget stops at roughly 47 GiB on a 128 GB machine *with the weights pinned* — and lower on a machine carrying more besides this server, which Dankclaw is. - **`HALOGEN_HOST_RESERVE_GIB`** is what is left for the page cache the lookup table is read through. Too little and long prompts crawl with the disk busy and the process in uninterruptible sleep — that is a cache problem, not a memory problem, and the fix is a *smaller* pool, not a bigger one. `HALOGEN_KV_POOL_FIT=1` (default) measures that budget at startup and lowers the pool itself if the configured one will not fit, printing what it chose. Read those lines on every restart; they are the ground truth for sizing. **Measured ledger, seven restarts, read live 2026-09-12.** The engine's own startup account, including three deliberate experiments: | pool | max_tok | reserve | KV | arena | need | budget | verdict | |---|---|---|---|---|---|---|---| | 262144 (baseline) | 32768 | 24 | 7.2 | 20.6 | 29.8 | 29.5 | boots 0.3 over — prints **no** sizing lines | | 524288 | 16384 | 24 | 14.4 | 12.2 | 28.6 | 29.5 | fits, 0.9 spare | | 524288 | 32768 | 24 | 14.4 | 20.6 | 37.0 | 29.5 | clamped to 262144, deploy still green | | 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** | | 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** | | 524288 | 32768 | 16 | 14.4 | 20.6 | 37.0 | 37.5 | fits, footprint unchanged — see the ceiling below | ### The sizing rule One budget covers the pool, the arena **and** the vision tower together: ``` device budget = host RAM - resident weights - HALOGEN_HOST_RESERVE_GIB = 121.2 - 67.7 - reserve reserve 24 -> 29.5 GiB reserve 15 -> 38.5 GiB need = KV (28 KiB/position) + arena + vision (1.97 measured) arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384) ``` The `121` the engine prints is rounded **down** from `MemTotal 127104060 kB = 121.2159 GiB`, so subtracting the displayed figures gives 29.3 rather than 29.5. Use the unrounded value. Two of the three terms are fixed hardware and checkpoint facts; `HALOGEN_HOST_RESERVE_GIB` is the only one you own — which buys the *fit margin*, not the host's memory. It is a term in this inequality, not an allocation, and the section at the end of this page shows the live test that proves it. Trust the engine's printed requirement over this model wherever they disagree: a boot clamped at reserve 24 stated `524288 positions need ~35.0 GiB (plus 1.5 of margin)`, i.e. **budget >= 36.5**, against a modelled 37.0 here. Close enough to plan with, and the authoritative number to cut the other way. The arena term is **linear in `HALOGEN_MAX_TOK`** and predicts every measured point to 0.1 GiB — including upstream's three published "on the device" figures (524,288 → ~35, 786,432 → ~42, 1,048,576 at `max_tok=16384` → ~41), which are **pool + arena totals at the default `max_tok`**, not pool alone. That is why their largest pool "needs `HALOGEN_MAX_TOK=16384`": at 32768 a 1M pool would be 49.4. Reading those figures as pool-only is the trap that makes a doubling look affordable when it is not. At the baseline reserve of 24 the doubled pool leaves `29.5 - 1.97 - 14.4 = 13.1` for the arena, so `max_tok <= 18,130` and **16384 is that reserve's ceiling, not a conservative setting.** The corner check agrees: at 32768 only 7.0 GiB is left for the KV, ~254k positions, and the pool drops back toward 262144. Cut the reserve to 15 and the same arithmetic gives `38.5 - 1.97 - 14.4 = 22.1` for the arena — `max_tok <= 35,700`, which is exactly why 32768 now fits unmodified. ### What the reserve actually costs The device budget is not the whole story. Freeing the doubled pool by cutting the reserve moves the machine's memory position considerably: | config | held | host left | contiguous 2 MiB blocks | |---|---|---|---| | 262144 / 32768, reserve 24 | 96.3 | 12.1 GiB | 2081 | | 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 | | **524288 / 32768, reserve 15** | **103.5** | **7.1 GiB** | **310** | | 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 | At the deployed setting the server holds 103.5 of 121 GiB and the engine now volunteers a note it did not print before: ``` halogen: NOTE this server now holds most of the machine, and 310 contiguous 2 MiB blocks remain. It runs best on a host of its own. If other large processes share this machine they will compete for what is left, and when it runs out BOTH they and this server can stall for minutes at a time at full CPU with no disk activity and no output, which is not a crash and needs no restart. ``` That is the failure mode to watch, and it is the least diagnosable of the three: not a silent clamp, not a boot refusal, but a minutes-long freeze that looks like a hang, for the neighbours as much as for frank-large. `free` is useless for spotting it — `MemAvailable` still reads ~74 GiB because the page cache counts as reclaimable. Watch the `host memory left for everything else` line and the contiguous-block count at boot instead. 28 co-tenant containers (~4.3 GiB working set) stayed up through the change, so it is survivable. ### The reserve cannot buy air — ceiling 16 The last row above is the experiment that proves it. Raising the reserve from 15 to 16 left `held` and `host left` **byte-for-byte identical** (103.5 / 7.1 GiB). `HALOGEN_HOST_RESERVE_GIB` feeds the fit-check budget and is *not* an allocation; the actual footprint is `weights + KV + arena + vision`, none of which the reserve touches. So the knob has exactly two regimes: | reserve | budget | vs required 36.5 | effect | |---|---|---|---| | 15 | 38.5 | +2.0 | deployed | | 16 | 37.5 | +1.0 | no change to footprint, thinner margin — pointless | | 17 | 36.5 | +0.02 | knife edge, inside the noise of the 67.7 term | | **18** | **35.5** | **−1.0** | **clamps the pool to 262144** | | 24 | 29.5 | −7.0 | clamps the pool to 262144 | The 36.5 requirement is the engine's own, read from a clamped boot: `524288 positions need ~35.0 GiB (plus 1.5 of margin)`. Above 16 the only thing a bigger reserve can do is take the doubling away. **To actually free host memory, change the pool or the arena**, and the reserve then becomes free to rise: `HALOGEN_KV_POOL_POSITIONS=262144` hands back 7.2 GiB, `HALOGEN_MAX_TOK=16384` hands back 8.4. Both are one-line edits that the fit check will confirm on the next boot. ### The `WARNING` line only appears when the fit check had to move This one cost me a wrong conclusion, so it is worth stating plainly. When the configured pool already fits, the engine reserves it **and prints nothing** — the 09-11 baseline boot goes straight from the pre-flight line to `checkpoint: mapped` with no `kv pool:` sizing block at all. The sizing block and its `WARNING even 262144 positions ... host RAM cannot spare it` appear **only after a clamp**. So the absence of a warning in old logs is not evidence of headroom. Both `262144/32768` rows above need 29.8 against a 29.5 budget — 0.27 over — and the baseline has lived in that state since it was deployed without incident, because the 24 GiB reserve is itself generous (the host reports 12–14 GiB free at that point). The engine's budget is a conservative floor, not a hard wall. ### Two failure modes, and only one is dangerous Raising `max_tok` past the ceiling does not shrink the pool — the arena alone outgrows the budget, and the server cannot allocate at all: ``` kv pool: 524288 positions need ~51.7 GiB ... LOWERING THE POOL TO 262144 (~44.5 GiB). kv pool: WARNING even 262144 positions (the context, the smallest pool that can serve one request) needs ~44.5 GiB and host RAM cannot spare it. dmalloc: FAILED requesting 0.250 GiB after 39.970 GiB in 566 allocations (out of memory) ``` This is the quiet one: the unit stays `active`, the container reports `Up (starting)`, and `deploy.sh` sits in its health wait for the whole `LLM_WAIT` rather than failing fast. It confirms upstream's warning that a large `HALOGEN_MAX_TOK` means "the server will not start", not "it will be slower". Under-running the ceiling is far more benign and far easier to miss. At 524288 with the arena back at 32768, `HALOGEN_KV_POOL_FIT` lowered the pool and `deploy.sh` still reported healthy in 12 seconds: ``` kv pool: 524288 positions need ~35.0 GiB (plus 1.5 of margin) and host RAM cannot spare it. LOWERING THE POOL TO 262144 (~27.8 GiB). ``` A green health check does not mean the pool you asked for is the pool you got — read the `kv pool:` lines after every restart. The 47.7 GiB n-gram lookup table is never held in RAM; it is read through the file cache. The 2.0 GiB the firmware carves for the iGPU appears in no `/proc/meminfo` figure at all. **Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published decode numbers were measured with, and it means this container holds most of the host. If a co-tenant then takes the remainder, both it and the engine can stall for minutes at a time at full CPU with no disk activity and no output. That is documented behaviour, not a crash, and it needs no restart. `HALOGEN_FLASH_PIN_TRUNK=0` gives the memory back and costs several times the decode speed — a last resort, not a tuning option. ## Client contract The differences from llama.cpp that bite are all in the request, not the deployment: - **No server-side sampler defaults.** `temperature` absent or `0` is **greedy** decode; above 0 the request samples on the same drafter. Supported fields: `temperature`, `top_p`, `top_k`, `min_p`, `seed`, `presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`. Refused with a 400: `top_logprobs`, `logprobs` with `stream: true`, `n > 1`, and any value outside its defined range (values are refused, never clamped). The fleet runs greedy for clients that stay silent — deterministic, and byte-identical with the drafter. - **The token budget covers thinking.** The default is **8192**, and a budget that runs out mid-thought returns `finish_reason: "length"` with an **empty `content`** and the partial reasoning in `reasoning_content` — which most clients do not display. Above the cap (`HALOGEN_MAX_TOKENS_CAP`, 65536) a request is a 400 rather than a silent truncation. `max_tokens`, `max_completion_tokens` and `max_output_tokens` all work and must agree if more than one is sent. - **`reasoning_effort`** is accepted at the top level and inside `chat_template_kwargs`: `minimal`, `low`, `medium`, `high`, `xhigh`. The model's default is `xhigh`. `minimal` aliases the template's `low` and still reasons; `enable_thinking: false` turns thinking off; `preserve_thinking` is honoured. `preserve_reasoning` is a **400** — it is the llama.cpp-era key. - `/v1/models`, `/health` and completions all report the served id, and `reasoning_content` carries the reasoning trace. **Pool admission is the one arithmetic trap.** A request reserves its prompt **plus** `max_tokens` positions when it is admitted; if the pool cannot hold it the request waits in arrival order, and after the queue timeout (3600 s) it gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds the pool can therefore never be admitted at all. With the pool at 524,288 the per-conversation ceiling is `HALOGEN_CTX` (262,144) — the pool binds **how many** conversations stay resident, not how long any one of them can be. Under the previous 262,144 pool the two numbers were identical, which is why four concurrent 65k chats were contending for one conversation's worth of positions. ## Vision With the tower loaded, `/v1/chat/completions` accepts the usual OpenAI image content part carrying a `data:` URL or bare base64 — **`http(s)` URLs are refused deliberately**, because fetching one would make the server issue outbound requests to wherever a client pointed it. One image costs about 2,040 tokens of context and 5.5 / 11.8 / 25.3 s at 1280x800 / 1920x1080 / 2560x1440; text at 12 pt and above is read exactly, and a bigger frame past that point adds empty area rather than detail. Audio and video have no path here at all — that capability left with [[frank-npu]]. ## Deploy and operate ```bash ./scripts/deploy.sh dankclaw --service frank-large ``` The handler retires the llama.cpp-era **root** unit first (stops it, drops its enable symlinks, moves `/etc/systemd/system/frank-large.service` into the home trash — never `rm`), then pulls `ghcr.io/peonist-ai/halogen-flash-server:0.5.8`, re-tags it to the `localhost/` alias the unit references, installs the Quadlet, and restarts the generated unit. Boot persistence comes from `[Install] WantedBy=default.target` in the unit file plus linger on the deploy user — generated units **cannot** be `systemctl --user enable`d on this host. The health gate is bootstrap-aware: while the checkpoint is absent the container is downloading and `/health` cannot answer, so the handler waits for the file instead of burning its budget and reporting a failure that is not one. ```bash systemctl --user status frank-large.service # the generated unit podman logs frank-large | head -40 # version banner + memory ledger curl -s http://127.0.0.1:7770/health # 200 only after the engine PONGs curl -s http://127.0.0.1:7770/v1/models ``` `/health` asks the **engine** for a PONG before answering, so a 200 means "ready to serve", not merely "process alive". During the first-start download it stays unhealthy by design. ## Troubleshooting | symptom | cause | |---|---| | `dmalloc: FAILED requesting … (out of memory)` then `HIP … out of memory` | the KV pool did not fit. `HALOGEN_KV_SLOTS` will not fix it (slots share the pool); lower `HALOGEN_KV_POOL_POSITIONS` to 262144, then `HALOGEN_MAX_TOK` to 16384 (~9 % prefill) | | starts, answers short prompts, collapses to a few tok/s on a long one, disk busy, process in uninterruptible sleep | short of **file cache**, not memory — lower the pool and/or raise `HALOGEN_HOST_RESERVE_GIB` | | long replies that look cut off | read `finish_reason`: `stop` means you have the whole answer, `length` means you ran out of budget (thinking counts) | | a request that never returns, then `503 engine_busy` | the pool could not reserve prompt + `max_tokens`; see the admission trap above | | both the engine and a co-tenant stall for minutes at full CPU | the pinned-trunk trade; see Memory model | | `free`, `MemAvailable`, Grafana all wrong by ~68 GiB | expected — locked weights are counted as reclaimable | `HALOGEN_VERBOSE=1` ships in the unit, so the fullest startup account is already in the log when something goes wrong. `HALOGEN_DMALLOC_LOG` is deliberately **not** set: it logs every allocation over 64 MB, which is steady-state noise on a shared host — raise it only for a live OOM report. The lines worth grepping first: ```bash podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):' ``` Retrieval: `podman logs frank-large` covers the **current** instance only — the Quadlet runs `--replace --rm`, so everything before the last recreate is gone from the container log. `journalctl --user -u frank-large.service` is the durable record. Both are rootless: `sudo podman logs frank-large` looks in root's storage and answers "no such container". ## Rollback The llama.cpp era is intact: the STRIX payload stays at `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX`, the `kingjones30/ROCmFPX` build stays at `/home/dank/GitPackages/kingjones-ROCmFPX/build`, and the retired unit, env and serve script are in `serverAppConfigs/.AGENTTRASH/` (suffixed `.llama.cpp`). Rolling back is a `git revert` of this change set plus a deploy — **but the Quadlet unit must be removed from `~/.config/containers/systemd/` first** (otherwise the restarted root unit and the still-installed container both claim port 7770). See [[kingjones-strix-rocmfp4]]. The `amd_iommu=off` kernel argument that came with this change is separate and reverts on its own — see [[dankclaw-iommu-translation-off]]. ## Citations - [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README, `docs/FLAGS.md`, `deploy/entrypoint.sh`, `docker-compose.yml`, CHANGELOG (v0.5.8) - [peonist-ai/halogen-qwen3.8-flash-next](https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next) — weights and file inventory - [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) — base model