serverAppConfigs/serverAppConfigs-docs/LLM/halogen-flash-server.md

---
type: reference
title: halogen-flash-server on Dankclaw (frank-large)
description: The active frank-large runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing.
tags:
  - llm
  - halogen
  - dankclaw
  - frank-large
  - strix-halo
  - gfx1151
  - qwen4exp
  - rocm
timestamp: 2026-09-11
---

# halogen-flash-server on Dankclaw (frank-large)

> **ACTIVE since 2026-09-11.** Replaced the llama.cpp / kingjones30-ROCmFPX
> STRIX deployment (see [[kingjones-strix-rocmfp4]], now the rollback target).
> Deployed as a rootless Quadlet container,
> `dankclaw/frank-large/quadlet/frank-large.container`, port 7770.

The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next
architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs.
Upstream's own comparison table measures it at **roughly 4x the prefill
throughput** of the llama.cpp-family runtimes it ships against (EngramHalo.cpp,
ROCmFP4, CIRU-IU4) on a 32K prompt, and prefill is most of the wall clock on any
prompt with real context. That, not decode, is the reason for the swap.

## Shape

One container runs both halves of the stack:

| half | port | reachable |
|---|---|---|
| engine | 8730 | container loopback only — **unauthenticated, never publish it** |
| OpenAI front-end | 8731 | published as host **7770** |

The `all` mode is upstream's default and the shape a single-container host
wants: the front-end talks to the engine over loopback inside the container, so
there is one process tree, one log, and one published port.

`HALOGEN_MODEL_ID=frank-large` sets the id reported at `/v1/models`, in
`/health`, and in every response body. Requests are **never rejected for naming
a different model** — the server serves whatever asks. Keeping the fleet alias
is therefore free, and it means Hermes, the harness config, the OpenWebUI
discovery, the `services.json` probe and the docs needed no rename.

## Weights

`peonist-ai/halogen-qwen3.8-flash-next` (HuggingFace, public and ungated):

| file | size | role |
|---|---|---|
| `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint |
| `qwen38-flash-next-w4b.overlay.hgn` | 2.31 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing |
| `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) |
| `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount |

`HALOGEN_DOWNLOAD` fetches the whole repo into the bind mount on **first start
only** — the entrypoint acts only when the checkpoint is absent, so a restart
never re-downloads, and `huggingface_hub` resumes an interrupted pull. The
mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in
`/home/dank/models/halogen`.

`overlay-speed.hgn` (2.22 GiB) comes along with the repo but is unused: it
trades the `o_proj` calibration for about 2 % of decode, and the quality
sidecar is the default for a reason.

## Deployed flags

Everything else is configured by environment variable — there is no config
file, and the flags are read once at startup. These are the departures from
upstream's defaults:

| flag | value | why |
|---|---|---|
| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
| `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare |
| `HALOGEN_KV_POOL_POSITIONS` | `262144` | conservative for a **shared** 121 GiB host: one full-length conversation, or four at 65k, at ~7.2 GiB |
| `HALOGEN_KV_SLOTS` | `4` | four concurrent generations; slots share the pool and cost ~115 MiB each |
| `HALOGEN_HOST_RESERVE_GIB` | `24` | RAM kept for the n-gram table the engine reads through the page cache — above upstream's 20 because a dozen containers share this host |
| `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired |
| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |

Everything else is upstream's default, deliberately: prompt cache mode `2`
(two entries per conversation — end-of-system-prompt and end-of-history — which
already covers the shared-prefix case), `HALOGEN_CACHE_INPLACE=1` (the snapshot
stays ~115 MiB at any depth), the speculative drafter `1` (byte-identical at
temperature 0, and it stands down whenever a second conversation is
generating), the quality sidecar, `HALOGEN_ADMIT_CHUNK=0` (a non-zero value
gives up the byte-identity property), the baked `HALOGEN_MAX_TOK=32768`
(16.7 GiB prefill arena), and the default `HALOGEN_MAX_TOKENS_CAP=65536`.

## Memory model

Three numbers explain the whole box:

- **~68 GiB pinned weights.** The checkpoint is mapped and registered with the
  GPU in place, never copied. The kernel counts the locked pages as reclaimable
  file cache, so `free`, `MemAvailable` and every dashboard built on them
  **overstate free memory by the size of the model**. The server prints the
  corrected figure at startup; believe that line, not Grafana.
- **~7.2 GiB per 262,144 KV positions**, plus the prefill arena (16.7 GiB at
  `HALOGEN_MAX_TOK=32768`, 8.4 GiB at 16384). The engine's device-side budget
  stops at roughly 47 GiB on a 128 GB machine *with the weights pinned* — and
  lower on a machine carrying more besides this server, which Dankclaw is.
- **`HALOGEN_HOST_RESERVE_GIB`** is what is left for the page cache the lookup
  table is read through. Too little and long prompts crawl with the disk busy
  and the process in uninterruptible sleep — that is a cache problem, not a
  memory problem, and the fix is a *smaller* pool, not a bigger one.

`HALOGEN_KV_POOL_FIT=1` (default) measures that budget at startup and lowers
the pool itself if the configured one will not fit, printing what it chose.
Read those lines on every restart; they are the ground truth for sizing.

**Co-tenancy.** `HALOGEN_FLASH_PIN_TRUNK=1` (default) is what the published
decode numbers were measured with, and it means this container holds most of
the host. If a co-tenant then takes the remainder, both it and the engine can
stall for minutes at a time at full CPU with no disk activity and no output.
That is documented behaviour, not a crash, and it needs no restart.
`HALOGEN_FLASH_PIN_TRUNK=0` gives the memory back and costs several times the
decode speed — a last resort, not a tuning option.

## Client contract

The differences from llama.cpp that bite are all in the request, not the
deployment:

- **No server-side sampler defaults.** `temperature` absent or `0` is
  **greedy** decode; above 0 the request samples on the same drafter. Supported
  fields: `temperature`, `top_p`, `top_k`, `min_p`, `seed`,
  `presence_penalty`, `frequency_penalty`, `logit_bias`, `logprobs`. Refused
  with a 400: `top_logprobs`, `logprobs` with `stream: true`, `n > 1`, and any
  value outside its defined range (values are refused, never clamped).
  The fleet runs greedy for clients that stay silent — deterministic, and
  byte-identical with the drafter.
- **The token budget covers thinking.** The default is **8192**, and a budget
  that runs out mid-thought returns `finish_reason: "length"` with an **empty
  `content`** and the partial reasoning in `reasoning_content` — which most
  clients do not display. Above the cap (`HALOGEN_MAX_TOKENS_CAP`, 65536) a
  request is a 400 rather than a silent truncation. `max_tokens`,
  `max_completion_tokens` and `max_output_tokens` all work and must agree if
  more than one is sent.
- **`reasoning_effort`** is accepted at the top level and inside
  `chat_template_kwargs`: `minimal`, `low`, `medium`, `high`, `xhigh`. The
  model's default is `xhigh`. `minimal` aliases the template's `low` and still
  reasons; `enable_thinking: false` turns thinking off; `preserve_thinking` is
  honoured. `preserve_reasoning` is a **400** — it is the llama.cpp-era key.
- `/v1/models`, `/health` and completions all report the served id, and
  `reasoning_content` carries the reasoning trace.

**Pool admission is the one arithmetic trap.** A request reserves its prompt
**plus** `max_tokens` positions when it is admitted; if the pool cannot hold it
the request waits in arrival order, and after the queue timeout (3600 s) it
gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds
the pool can therefore never be admitted at all. The 262,144-position pool is
the binding constraint on how large a conversation can get, not `HALOGEN_CTX`
and not the client's declared `context_length`.

## Vision

With the tower loaded, `/v1/chat/completions` accepts the usual OpenAI image
content part carrying a `data:` URL or bare base64 — **`http(s)` URLs are
refused deliberately**, because fetching one would make the server issue
outbound requests to wherever a client pointed it. One image costs about 2,040
tokens of context and 5.5 / 11.8 / 25.3 s at 1280x800 / 1920x1080 / 2560x1440;
text at 12 pt and above is read exactly, and a bigger frame past that point
adds empty area rather than detail. Audio and video have no path here at all —
that capability left with [[frank-npu]].

## Deploy and operate

```bash
./scripts/deploy.sh dankclaw --service frank-large
```

The handler retires the llama.cpp-era **root** unit first (stops it, drops its
enable symlinks, moves `/etc/systemd/system/frank-large.service` into the home
trash — never `rm`), then pulls `ghcr.io/peonist-ai/halogen-flash-server:0.5.8`,
re-tags it to the `localhost/` alias the unit references, installs the Quadlet,
and restarts the generated unit. Boot persistence comes from
`[Install] WantedBy=default.target` in the unit file plus linger on the deploy
user — generated units **cannot** be `systemctl --user enable`d on this host.

The health gate is bootstrap-aware: while the checkpoint is absent the
container is downloading and `/health` cannot answer, so the handler waits for
the file instead of burning its budget and reporting a failure that is not one.

```bash
systemctl --user status frank-large.service     # the generated unit
podman logs frank-large | head -40              # version banner + memory ledger
curl -s http://127.0.0.1:7770/health            # 200 only after the engine PONGs
curl -s http://127.0.0.1:7770/v1/models
```

`/health` asks the **engine** for a PONG before answering, so a 200 means
"ready to serve", not merely "process alive". During the first-start download
it stays unhealthy by design.

## Troubleshooting

| symptom | cause |
|---|---|
| `dmalloc: FAILED requesting … (out of memory)` then `HIP … out of memory` | the KV pool did not fit. `HALOGEN_KV_SLOTS` will not fix it (slots share the pool); lower `HALOGEN_KV_POOL_POSITIONS` to 262144, then `HALOGEN_MAX_TOK` to 16384 (~9 % prefill) |
| starts, answers short prompts, collapses to a few tok/s on a long one, disk busy, process in uninterruptible sleep | short of **file cache**, not memory — lower the pool and/or raise `HALOGEN_HOST_RESERVE_GIB` |
| long replies that look cut off | read `finish_reason`: `stop` means you have the whole answer, `length` means you ran out of budget (thinking counts) |
| a request that never returns, then `503 engine_busy` | the pool could not reserve prompt + `max_tokens`; see the admission trap above |
| both the engine and a co-tenant stall for minutes at full CPU | the pinned-trunk trade; see Memory model |
| `free`, `MemAvailable`, Grafana all wrong by ~68 GiB | expected — locked weights are counted as reclaimable |

`HALOGEN_VERBOSE=1` and `HALOGEN_DMALLOC_LOG=1` print the fullest startup
account and every allocation over 64 MB; both are the vendor's suggested
evidence to attach to a report. The two lines worth grepping first:

```bash
podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'
```

## Rollback

The llama.cpp era is intact: the STRIX payload stays at
`/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX`, the
`kingjones30/ROCmFPX` build stays at
`/home/dank/GitPackages/kingjones-ROCmFPX/build`, and the retired unit, env and
serve script are in `serverAppConfigs/.AGENTTRASH/` (suffixed `.llama.cpp`).
Rolling back is a `git revert` of this change set plus a deploy — **but the
Quadlet unit must be removed from `~/.config/containers/systemd/` first**
(otherwise the restarted root unit and the still-installed container both
claim port 7770). See [[kingjones-strix-rocmfp4]].

The `amd_iommu=off` kernel argument that came with this change is separate and
reverts on its own — see [[dankclaw-iommu-translation-off]].

## Citations

- [peonist-ai/halogen-flash-server](https://github.com/peonist-ai/halogen-flash-server) — README, `docs/FLAGS.md`, `deploy/entrypoint.sh`, `docker-compose.yml`, CHANGELOG (v0.5.8)
- [peonist-ai/halogen-qwen3.8-flash-next](https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next) — weights and file inventory
- [Qwen/Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) — base model