serverAppConfigs/serverAppConfigs-docs/LLM/halogen-flash-server.md

---
type: reference
title: halogen-flash-server on Dankclaw (frank-large)
description: The active frank-large runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing.
tags:
  - llm
  - halogen
  - dankclaw
  - frank-large
  - strix-halo
  - gfx1151
  - qwen4exp
  - rocm
timestamp: 2026-09-12
---

# halogen-flash-server on Dankclaw (frank-large)

> **ACTIVE since 2026-09-11.** Replaced the llama.cpp / kingjones30-ROCmFPX
> STRIX deployment (see [[kingjones-strix-rocmfp4]], now the rollback target).
> Deployed as a rootless Quadlet container,
> `dankclaw/frank-large/quadlet/frank-large.container`, port 7770.

The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next
architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs.
Upstream's own comparison table measures it at **roughly 4x the prefill
throughput** of the llama.cpp-family runtimes it ships against (EngramHalo.cpp,
ROCmFP4, CIRU-IU4) on a 32K prompt, and prefill is most of the wall clock on any
prompt with real context. That, not decode, is the reason for the swap.

## Shape

One container runs both halves of the stack:

| half | port | reachable |
|---|---|---|
| engine | 8730 | container loopback only — **unauthenticated, never publish it** |
| OpenAI front-end | 8731 | published as host **7770** |

The `all` mode is upstream's default and the shape a single-container host
wants: the front-end talks to the engine over loopback inside the container, so
there is one process tree, one log, and one published port.

`HALOGEN_MODEL_ID=frank-large` sets the id reported at `/v1/models`, in
`/health`, and in every response body. Requests are **never rejected for naming
a different model** — the server serves whatever asks. Keeping the fleet alias
is therefore free, and it means Hermes, the harness config, the OpenWebUI
discovery, the `services.json` probe and the docs needed no rename.

## Weights

`peonist-ai/halogen-qwen3.8-flash-next` (HuggingFace, public and ungated):

| file | size | role |
|---|---|---|
| `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint |
| `qwen38-flash-next-w4b.overlay.hgn` | 2.31 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing |
| `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) |
| `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount |

`HALOGEN_DOWNLOAD` fetches the whole repo into the bind mount on **first start
only** — the entrypoint acts only when the checkpoint is absent, so a restart
never re-downloads, and `huggingface_hub` resumes an interrupted pull. The
mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in
`/home/dank/models/halogen`.

`overlay-speed.hgn` (2.22 GiB) comes along with the repo but is unused: it
trades the `o_proj` calibration for about 2 % of decode, and the quality
sidecar is the default for a reason.

## Deployed flags

Everything else is configured by environment variable — there is no config
file, and the flags are read once at startup. These are the departures from
upstream's defaults:

| flag | value | why |
|---|---|---|
| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
| `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare |
| `HALOGEN_KV_POOL_POSITIONS` | `524288` | **doubled 2026-09-12** to upstream's own default (2 × `HALOGEN_CTX`): two full-length conversations resident, or four at 131k, at ~14.4 GiB |
| `HALOGEN_KV_SLOTS` | `4` | four concurrent generations; slots share the pool and cost ~115 MiB each |
| `HALOGEN_HOST_RESERVE_GIB` | `24` | RAM kept for the n-gram table the engine reads through the page cache — above upstream's 20 because a dozen containers share this host |
| `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired |
| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |

Everything else is upstream's default, deliberately: prompt cache mode `2`
(two entries per conversation — end-of-system-prompt and end-of-history — which
already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot
stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at
temperature 0, and it stands down whenever a second conversation is
generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value
gives up the byte-identity property), the baked `HALOGEN_MAX_TOK=32768`
(16.7 GiB prefill arena), and the default `HALOGEN_MAX_TOKENS_CAP=65536`.

## Memory model

Three numbers explain the whole box:

- **~68 GiB pinned weights.** The checkpoint is mapped and registered with the
  GPU in place, never copied. The kernel counts the locked pages as reclaimable
  file cache, so `free`, `MemAvailable` and every dashboard built on them
  **overstate free memory by the size of the model**. The server prints the
  corrected figure at startup; believe that line, not Grafana.
- **~28 KiB per KV position** (measured: 262,144 positions = 6,656 MiB of KV,
  ~7.2 GiB with block scratch), plus the prefill arena (16.7 GiB at
  `HALOGEN_MAX_TOK=32768`, 8.4 GiB at 16384). The engine's device-side budget
  stops at roughly 47 GiB on a 128 GB machine *with the weights pinned* — and
  lower on a machine carrying more besides this server, which Dankclaw is.
- **`HALOGEN_HOST_RESERVE_GIB`** is what is left for the page cache the lookup
  table is read through. Too little and long prompts crawl with the disk busy
  and the process in uninterruptible sleep — that is a cache problem, not a
  memory problem, and the fix is a *smaller* pool, not a bigger one.

`HALOGEN_KV_POOL_FIT=1` (default) measures that budget at startup and lowers
the pool itself if the configured one will not fit, printing what it chose.
Read those lines on every restart; they are the ground truth for sizing.

**Measured ledger (prior 262,144 config, read live 2026-09-12).** The
engine's own startup account: 68.0 GiB of weights locked + 7.2 GiB KV pool +
21.1 GiB working memory = **96.3 GiB held**, with **12.1 GiB left on the
host**, only ~7.3 GiB of that in contiguous 2 MiB pieces. The 47.7 GiB n-gram
lookup table is never held in RAM — it is read through the file cache — and
the 2.0 GiB firmware carves for the iGPU appears in no `/proc/meminfo`
figure.

Doubling to 524,288 puts the held total near **103.5 GiB** and the host
remainder at roughly **5 GiB**, against a measured ~4.3 GiB working set for
the 25 co-tenant containers on this box. The extra 7.2 GiB comes out of the
page cache, which falls from ~76 GiB to ~69 GiB — still above the 47.7 GiB
lookup table, so the table can stay hot. Tight, not an overcommit. If long
prompts start crawling with the disk busy, `HALOGEN_MAX_TOK=16384` shrinks
the prefill arena 16.7 → 8.4 GiB and pays for the doubling outright at about
8 % of prefill throughput.

**Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published
decode numbers were measured with, and it means this container holds most of
the host. If a co-tenant then takes the remainder, both it and the engine can
stall for minutes at a time at full CPU with no disk activity and no output.
That is documented behaviour, not a crash, and it needs no restart.
`HALOGEN_FLASH_PIN_TRUNK=0` gives the memory back and costs several times the
decode speed — a last resort, not a tuning option.

## Client contract

The differences from llama.cpp that bite are all in the request, not the
deployment:

- **No server-side sampler defaults.** `temperature` absent or `0` is
  **greedy** decode; above 0 the request samples on the same drafter. Supported
  fields: `temperature`, `top_p`, `top_k`, `min_p`, `seed`,
  `presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`. Refused
  with a 400: `top_logprobs`, `logprobs` with `stream: true`, `n > 1`, and any
  value outside its defined range (values are refused, never clamped).
  The fleet runs greedy for clients that stay silent — deterministic, and
  byte-identical with the drafter.
- **The token budget covers thinking.** The default is **8192**, and a budget
  that runs out mid-thought returns `finish_reason: "length"` with an **empty
  `content`** and the partial reasoning in `reasoning_content` — which most
  clients do not display. Above the cap (`HALOGEN_MAX_TOKENS_CAP`, 65536) a
  request is a 400 rather than a silent truncation. `max_tokens`,
  `max_completion_tokens` and `max_output_tokens` all work and must agree if
  more than one is sent.
- **`reasoning_effort`** is accepted at the top level and inside
  `chat_template_kwargs`: `minimal`, `low`, `medium`, `high`, `xhigh`. The
  model's default is `xhigh`. `minimal` aliases the template's `low` and still
  reasons; `enable_thinking: false` turns thinking off; `preserve_thinking` is
  honoured. `preserve_reasoning` is a **400** — it is the llama.cpp-era key.
- `/v1/models`, `/health` and completions all report the served id, and
  `reasoning_content` carries the reasoning trace.

**Pool admission is the one arithmetic trap.** A request reserves its prompt
**plus** `max_tokens` positions when it is admitted; if the pool cannot hold it
the request waits in arrival order, and after the queue timeout (3600 s) it
gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds
the pool can therefore never be admitted at all. With the pool at 524,288 the
per-conversation ceiling is `HALOGEN_CTX` (262,144) — the pool binds **how
many** conversations stay resident, not how long any one of them can be. Under
the previous 262,144 pool the two numbers were identical, which is why four
concurrent 65k chats were contending for one conversation's worth of
positions.

## Vision

With the tower loaded, `/v1/chat/completions` accepts the usual OpenAI image
content part carrying a `data:` URL or bare base64 — **`http(s)` URLs are
refused deliberately**, because fetching one would make the server issue
outbound requests to wherever a client pointed it. One image costs about 2,040
tokens of context and 5.5 / 11.8 / 25.3 s at 1280x800 / 1920x1080 / 2560x1440;
text at 12 pt and above is read exactly, and a bigger frame past that point
adds empty area rather than detail. Audio and video have no path here at all —
that capability left with [[frank-npu]].

## Deploy and operate

```bash
./scripts/deploy.sh dankclaw --service frank-large
```

The handler retires the llama.cpp-era **root** unit first (stops it, drops its
enable symlinks, moves `/etc/systemd/system/frank-large.service` into the home
trash — never `rm`), then pulls `ghcr.io/peonist-ai/halogen-flash-server:0.5.8`,
re-tags it to the `localhost/` alias the unit references, installs the Quadlet,
and restarts the generated unit. Boot persistence comes from
`[Install] WantedBy=default.target` in the unit file plus linger on the deploy
user — generated units **cannot** be `systemctl --user enable`d on this host.

The health gate is bootstrap-aware: while the checkpoint is absent the
container is downloading and `/health` cannot answer, so the handler waits for
the file instead of burning its budget and reporting a failure that is not one.

```bash
systemctl --user status frank-large.service     # the generated unit
podman logs frank-large | head -40              # version banner + memory ledger
curl -s http://127.0.0.1:7770/health            # 200 only after the engine PONGs
curl -s http://127.0.0.1:7770/v1/models
```

`/health` asks the **engine** for a PONG before answering, so a 200 means
"ready to serve", not merely "process alive". During the first-start download
it stays unhealthy by design.

## Troubleshooting

| symptom | cause |
|---|---|
| `dmalloc: FAILED requesting … (out of memory)` then `HIP … out of memory` | the KV pool did not fit. `HALOGEN_KV_SLOTS` will not fix it (slots share the pool); lower `HALOGEN_KV_POOL_POSITIONS` to 262144, then `HALOGEN_MAX_TOK` to 16384 (~9 % prefill) |
| starts, answers short prompts, collapses to a few tok/s on a long one, disk busy, process in uninterruptible sleep | short of **file cache**, not memory — lower the pool and/or raise `HALOGEN_HOST_RESERVE_GIB` |
| long replies that look cut off | read `finish_reason`: `stop` means you have the whole answer, `length` means you ran out of budget (thinking counts) |
| a request that never returns, then `503 engine_busy` | the pool could not reserve prompt + `max_tokens`; see the admission trap above |
| both the engine and a co-tenant stall for minutes at full CPU | the pinned-trunk trade; see Memory model |
| `free`, `MemAvailable`, Grafana all wrong by ~68 GiB | expected — locked weights are counted as reclaimable |

`HALOGEN_VERBOSE=1` ships in the unit, so the fullest startup account is
already in the log when something goes wrong. `HALOGEN_DMALLOC_LOG` is
deliberately **not** set: it logs every allocation over 64 MB, which is
steady-state noise on a shared host — raise it only for a live OOM report.
The lines worth grepping first:

```bash
podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'
```

Retrieval: `podman logs frank-large` covers the **current** instance only —
the Quadlet runs `--replace --rm`, so everything before the last recreate is
gone from the container log. `journalctl --user -u frank-large.service` is
the durable record. Both are rootless: `sudo podman logs frank-large` looks
in root's storage and answers "no such container".

## Rollback

The llama.cpp era is intact: the STRIX payload stays at
`/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX`, the
`kingjones30/ROCmFPX` build stays at
`/home/dank/GitPackages/kingjones-ROCmFPX/build`, and the retired unit, env and
serve script are in `serverAppConfigs/.AGENTTRASH/` (suffixed `.llama.cpp`).
Rolling back is a `git revert` of this change set plus a deploy — **but the
Quadlet unit must be removed from `~/.config/containers/systemd/` first**
(otherwise the restarted root unit and the still-installed container both
claim port 7770). See [[kingjones-strix-rocmfp4]].

The `amd_iommu=off` kernel argument that came with this change is separate and
reverts on its own — see [[dankclaw-iommu-translation-off]].

## Citations

- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README, `docs/FLAGS.md`, `deploy/entrypoint.sh`, `docker-compose.yml`, CHANGELOG (v0.5.8)
- [peonist-ai/halogen-qwen3.8-flash-next](https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next) — weights and file inventory
- [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) — base model