serverAppConfigs/serverAppConfigs-docs/LLM/halogen-flash-server.md

---

type: reference title: halogen-flash-server on Dankclaw (frank-large) description: The active frank-large runbook — peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b) on the Dankclaw gfx1151 box: container shape, deployed flags, the memory model, the client contract, and the failure modes worth knowing. tags:

• llm
• halogen
• dankclaw
• frank-large
• strix-halo
• gfx1151
• qwen4exp
• rocm timestamp: 2026-09-12

---

halogen-flash-server on Dankclaw (frank-large)

ACTIVE since 2026-09-11. Replaced the llama.cpp / kingjones30-ROCmFPX STRIX deployment (see kingjones-strix-rocmfp4, now the rollback target). Deployed as a rootless Quadlet container, dankclaw/frank-large/quadlet/frank-large.container, port 7770.

The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs. Upstream's own comparison table measures it at roughly 4x the prefill throughput of the llama.cpp-family runtimes it ships against (EngramHalo.cpp, ROCmFP4, CIRU-IU4) on a 32K prompt, and prefill is most of the wall clock on any prompt with real context. That, not decode, is the reason for the swap.

Shape

One container runs both halves of the stack:

half	port	reachable
engine	8730	container loopback only — unauthenticated, never publish it
OpenAI front-end	8731	published as host 7770

The all mode is upstream's default and the shape a single-container host wants: the front-end talks to the engine over loopback inside the container, so there is one process tree, one log, and one published port.

HALOGEN_MODEL_ID=frank-large sets the id reported at /v1/models, in /health, and in every response body. Requests are never rejected for naming a different model — the server serves whatever asks. Keeping the fleet alias is therefore free, and it means Hermes, the harness config, the OpenWebUI discovery, the services.json probe and the docs needed no rename.

Weights

peonist-ai/halogen-qwen3.8-flash-next (HuggingFace, public and ungated):

file	size	role
qwen38-flash-next-w4b.hgn	115.55 GiB	the checkpoint
qwen38-flash-next-w4b.overlay.hgn	2.31 GiB	quality sidecar — picked up automatically; the engine warns at startup if it is missing
qwen38-flash-next-vision.hgn	0.84 GiB	vision tower (see below)
tokenizer/	~21 MB	flat tokenizer dir; found beside the checkpoint, so no extra mount

HALOGEN_DOWNLOAD fetches the whole repo into the bind mount on first start only — the entrypoint acts only when the checkpoint is absent, so a restart never re-downloads, and huggingface_hub resumes an interrupted pull. The mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in /home/dank/models/halogen.

overlay-speed.hgn (2.22 GiB) comes along with the repo but is unused: it trades the o_proj calibration for about 2 % of decode, and the quality sidecar is the default for a reason.

Deployed flags

Everything else is configured by environment variable — there is no config file, and the flags are read once at startup. These are the departures from upstream's defaults:

flag	value	why
HALOGEN_MODEL_ID	frank-large	fleet identity; no client rename
HALOGEN_CTX	262144	the model's native context, and what both clients declare
HALOGEN_KV_POOL_POSITIONS	524288	doubled 2026-09-12 to upstream's own default (2 × HALOGEN_CTX): two full-length conversations resident, or four at 131k — 13,312 MiB of KV
HALOGEN_KV_SLOTS	4	four concurrent generations; slots share the pool and cost ~115 MiB each
HALOGEN_MAX_TOK	16384	load-bearing for the doubling. Halves the prefill arena so the pool fits the 29.5 GiB device budget, at ~8–9 % of prefill throughput. Leave it at upstream's 32768 and HALOGEN_KV_POOL_FIT silently reverts the pool to 262144 while the deploy still reports healthy
HALOGEN_VISION_TOWER	1	image input; the fleet's only image path since frank-npu was retired
HALOGEN_DOWNLOAD	peonist-ai/halogen-qwen3.8-flash-next	first-start fetch
HALOGEN_VERBOSE	1	verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug

Everything else is upstream's default, deliberately: prompt cache mode 2 (two entries per conversation — end-of-system-prompt and end-of-history — which already covers the shared-prefix case), HALOGEN_CACHE_INPLACE=1 (the snapshot stays ~115 MiB at any depth), the speculative drafter 1 (byte-identical at temperature 0, and it stands down whenever a second conversation is generating), the quality sidecar, HALOGEN_ADMIT_CHUNK=0 (a non-zero value gives up the byte-identity property), and the default HALOGEN_MAX_TOKENS_CAP=65536.

Memory model

Three numbers explain the whole box:

• ~68 GiB pinned weights. The checkpoint is mapped and registered with the GPU in place, never copied. The kernel counts the locked pages as reclaimable file cache, so free, MemAvailable and every dashboard built on them overstate free memory by the size of the model. The server prints the corrected figure at startup; believe that line, not Grafana.
• ~28 KiB per KV position (measured: 262,144 positions = 6,656 MiB of KV, ~7.2 GiB with block scratch), plus the prefill arena (16.7 GiB at HALOGEN_MAX_TOK=32768, 8.4 GiB at 16384). The engine's device-side budget stops at roughly 47 GiB on a 128 GB machine with the weights pinned — and lower on a machine carrying more besides this server, which Dankclaw is.
• HALOGEN_HOST_RESERVE_GIB is what is left for the page cache the lookup table is read through. Too little and long prompts crawl with the disk busy and the process in uninterruptible sleep — that is a cache problem, not a memory problem, and the fix is a smaller pool, not a bigger one.

HALOGEN_KV_POOL_FIT=1 (default) measures that budget at startup and lowers the pool itself if the configured one will not fit, printing what it chose. Read those lines on every restart; they are the ground truth for sizing.

Measured ledger, both configs, read live 2026-09-12. The engine's own startup account:

config	weights	KV pool	working mem	held	host left
pool 262144 / max_tok 32768 (prior)	68.0	7.2	21.1	96.3	12.1
pool 524288 / max_tok 16384 (now)	68.0	14.4	12.3	94.6	13.8

The doubled pool is 1.7 GiB cheaper in total than the config it replaced. Halving the prefill arena gives back more than the extra pool takes: the arena is ~20.6 GiB regardless of context, while the KV costs only ~28 KiB a position.

The first attempt — 524288 with the arena left at 32768 — did not fit, and the way it failed is the trap on this host:

kv pool: host RAM 121 GiB, less 67.7 resident weights and 24.0 reserved
         for the n-gram page cache and the OS = 29.5 GiB for the device
kv pool: 524288 positions need ~35.0 GiB (plus 1.5 of margin) and host
         RAM cannot spare it. LOWERING THE POOL TO 262144 (~27.8 GiB).

One budget covers the pool and the arena together. HALOGEN_KV_POOL_FIT lowered the pool, and deploy.sh still reported healthy in 12 seconds. A green health check does not mean the pool you asked for is the pool you got — always read the kv pool: lines after a restart.

The 47.7 GiB n-gram lookup table is never held in RAM; it is read through the file cache. The 2.0 GiB the firmware carves for the iGPU appears in no /proc/meminfo figure at all.

Co-tenancy. HALOGEN_FLASH_PIN_TRUNK=1 (default) is what the published decode numbers were measured with, and it means this container holds most of the host. If a co-tenant then takes the remainder, both it and the engine can stall for minutes at a time at full CPU with no disk activity and no output. That is documented behaviour, not a crash, and it needs no restart. HALOGEN_FLASH_PIN_TRUNK=0 gives the memory back and costs several times the decode speed — a last resort, not a tuning option.

Client contract

The differences from llama.cpp that bite are all in the request, not the deployment:

• No server-side sampler defaults. temperature absent or 0 is greedy decode; above 0 the request samples on the same drafter. Supported fields: temperature, top_p, top_k, min_p, seed, presence_penalty, frequency_penalty, logit_bias, logprobs. Refused with a 400: top_logprobs, logprobs with stream: true, n > 1, and any value outside its defined range (values are refused, never clamped). The fleet runs greedy for clients that stay silent — deterministic, and byte-identical with the drafter.
• The token budget covers thinking. The default is 8192, and a budget that runs out mid-thought returns finish_reason: "length" with an empty content and the partial reasoning in reasoning_content — which most clients do not display. Above the cap (HALOGEN_MAX_TOKENS_CAP, 65536) a request is a 400 rather than a silent truncation. max_tokens, max_completion_tokens and max_output_tokens all work and must agree if more than one is sent.
• reasoning_effort is accepted at the top level and inside chat_template_kwargs: minimal, low, medium, high, xhigh. The model's default is xhigh. minimal aliases the template's low and still reasons; enable_thinking: false turns thinking off; preserve_thinking is honoured. preserve_reasoning is a 400 — it is the llama.cpp-era key.
• /v1/models, /health and completions all report the served id, and reasoning_content carries the reasoning trace.

Pool admission is the one arithmetic trap. A request reserves its prompt plus max_tokens positions when it is admitted; if the pool cannot hold it the request waits in arrival order, and after the queue timeout (3600 s) it gets a 503 engine_busy. A prompt large enough that prompt + budget exceeds the pool can therefore never be admitted at all. With the pool at 524,288 the per-conversation ceiling is HALOGEN_CTX (262,144) — the pool binds how many conversations stay resident, not how long any one of them can be. Under the previous 262,144 pool the two numbers were identical, which is why four concurrent 65k chats were contending for one conversation's worth of positions.

Vision

With the tower loaded, /v1/chat/completions accepts the usual OpenAI image content part carrying a data: URL or bare base64 — http(s) URLs are refused deliberately, because fetching one would make the server issue outbound requests to wherever a client pointed it. One image costs about 2,040 tokens of context and 5.5 / 11.8 / 25.3 s at 1280x800 / 1920x1080 / 2560x1440; text at 12 pt and above is read exactly, and a bigger frame past that point adds empty area rather than detail. Audio and video have no path here at all — that capability left with frank-npu.

Deploy and operate

./scripts/deploy.sh dankclaw --service frank-large

The handler retires the llama.cpp-era root unit first (stops it, drops its enable symlinks, moves /etc/systemd/system/frank-large.service into the home trash — never rm), then pulls ghcr.io/peonist-ai/halogen-flash-server:0.5.8, re-tags it to the localhost/ alias the unit references, installs the Quadlet, and restarts the generated unit. Boot persistence comes from [Install] WantedBy=default.target in the unit file plus linger on the deploy user — generated units cannot be systemctl --user enabled on this host.

The health gate is bootstrap-aware: while the checkpoint is absent the container is downloading and /health cannot answer, so the handler waits for the file instead of burning its budget and reporting a failure that is not one.

systemctl --user status frank-large.service     # the generated unit
podman logs frank-large | head -40              # version banner + memory ledger
curl -s http://127.0.0.1:7770/health            # 200 only after the engine PONGs
curl -s http://127.0.0.1:7770/v1/models

/health asks the engine for a PONG before answering, so a 200 means "ready to serve", not merely "process alive". During the first-start download it stays unhealthy by design.

Troubleshooting

symptom	cause
dmalloc: FAILED requesting … (out of memory) then HIP … out of memory	the KV pool did not fit. HALOGEN_KV_SLOTS will not fix it (slots share the pool); lower HALOGEN_KV_POOL_POSITIONS to 262144, then HALOGEN_MAX_TOK to 16384 (~9 % prefill)
starts, answers short prompts, collapses to a few tok/s on a long one, disk busy, process in uninterruptible sleep	short of file cache, not memory — lower the pool and/or raise HALOGEN_HOST_RESERVE_GIB
long replies that look cut off	read finish_reason: stop means you have the whole answer, length means you ran out of budget (thinking counts)
a request that never returns, then 503 engine_busy	the pool could not reserve prompt + max_tokens; see the admission trap above
both the engine and a co-tenant stall for minutes at full CPU	the pinned-trunk trade; see Memory model
free, MemAvailable, Grafana all wrong by ~68 GiB	expected — locked weights are counted as reclaimable

HALOGEN_VERBOSE=1 ships in the unit, so the fullest startup account is already in the log when something goes wrong. HALOGEN_DMALLOC_LOG is deliberately not set: it logs every allocation over 64 MB, which is steady-state noise on a shared host — raise it only for a live OOM report. The lines worth grepping first:

podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'

Retrieval: podman logs frank-large covers the current instance only — the Quadlet runs --replace --rm, so everything before the last recreate is gone from the container log. journalctl --user -u frank-large.service is the durable record. Both are rootless: sudo podman logs frank-large looks in root's storage and answers "no such container".

Rollback

The llama.cpp era is intact: the STRIX payload stays at /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX, the kingjones30/ROCmFPX build stays at /home/dank/GitPackages/kingjones-ROCmFPX/build, and the retired unit, env and serve script are in serverAppConfigs/.AGENTTRASH/ (suffixed .llama.cpp). Rolling back is a git revert of this change set plus a deploy — but the Quadlet unit must be removed from ~/.config/containers/systemd/ first (otherwise the restarted root unit and the still-installed container both claim port 7770). See kingjones-strix-rocmfp4.

The amd_iommu=off kernel argument that came with this change is separate and reverts on its own — see dankclaw-iommu-translation-off.

Citations

• peonist-ai/halogen-flash-server — README, docs/FLAGS.md, deploy/entrypoint.sh, docker-compose.yml, CHANGELOG (v0.5.8)
• peonist-ai/halogen-qwen3.8-flash-next — weights and file inventory
• Qwen/Qwen3.8-Flash-Next — base model