dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: the vault and AGENTS.md describe two Dankclaw endpoints

Browse Source 
halogen-flash-server is ACTIVE again with its 2026-09-16 measured ledger (budget
33.5 GiB against a need of 27.2 — no clamp — and 14.7 GiB left for everything
else), ik_llama.cpp is retired with its measurements kept as the record of why,
and the new frank-small runbook covers the -mtp file, the 64 KiB/token KV
arithmetic, the MTP contract, and the reasoning_effort trap that cost me every
subagent on the endpoint.

AGENTS.md: topology, Active Services, runbook index, and three new llama.cpp
lessons (MTP is a property of the file; --jinja is what `tools` needs; a model's
template can reject the harness's thinking level). Three pre-existing defects
found while editing it: a truncated sentence mid-file, a harness path that does
not exist on either server, and the benchmarking stop/mask recipe still calling
frank-large a root unit.

Truthfulness notes worth keeping: the co-tenant stall NOTE prints again, so the
claim that it "no longer prints" is gone and the three observed block counts are
recorded instead of a guessed threshold; and the Observability page says out
loud why frank-large has no scrape job rather than leaving a silent absence.
This commit is contained in:
Frank committed 2026-09-16 14:52:51 -07:00
1 parent 40850bbc8f
commit 49e8dfff66
8 files changed
+565 -153
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/LLM/halogen-flash-server.md
+82 -52
View File
@@ -16,10 +16,14 @@ timestamp: 2026-09-13
# halogen-flash-server on Dankclaw (frank-large)
> **ACTIVE since 2026-09-11.** Replaced the llama.cpp / kingjones30-ROCmFPX
> STRIX deployment (see [[kingjones-strix-rocmfp4]], now the rollback target).
> Deployed as a rootless Quadlet container,
> `dankclaw/frank-large/quadlet/frank-large.container`, port 7770.
> **ACTIVE again since 2026-09-16, and retuned.** It replaced the llama.cpp /
> kingjones30-ROCmFPX STRIX deployment on 2026-09-11 (see
> [[kingjones-strix-rocmfp4]]), was displaced for two days by ik_llama.cpp (see
> [[frank-large-ik-uncensored]], the current rollback), and came back as the
> shape the fleet asked for: two 200k-token conversations rather than the
> native-context pair. Deployed as a rootless Quadlet container,
> `dankclaw/frank-large/quadlet/frank-large.container`, port 7770. The RTX 3090
> it does not touch now belongs to [[frank-small-qwen38-gsq-rco]].
The engine is a from-scratch serving runtime for the Qwen3.8-Flash-Next
architecture on AMD's gfx1151 (Strix Halo) — the same silicon this host runs.
@@ -77,12 +81,12 @@ upstream's defaults:
| flag | value | why |
|---|---|---|
| `HALOGEN_MODEL_ID` | `frank-large` | fleet identity; no client rename |
| `HALOGEN_CTX` | `262144` | the model's native context, and what both clients declare |
| `HALOGEN_KV_POOL_POSITIONS` | `401408` | **trimmed 2026-09-13** from the 524,288 the doubling deployed: one full 262k conversation resident beside a ~139k one, or three at 131k, ~11.0 GiB of KV as reserved. Measured against the doubled pool the host went from 7.1 GiB / 310 contiguous 2 MiB blocks to 9.1 / 1396, and the co-tenant stall NOTE no longer prints. `HALOGEN_KV_POOL_FIT` still guards the boot, so read the ledger rather than trusting this column |
| `HALOGEN_KV_SLOTS` | `3` | Concurrency cap — **lowered from 4 on 2026-09-12** to the real fleet size. Each slot is 111.0 MiB of O(1) state, so this freed ~111 MiB (held 103.5 → 103.4 GiB). Slots share the ONE pool and **oversubscribe** it (3 × 262144 > 401408), so admission is pool-bound: one full-length conversation plus a ~139k second, or three at 131k. This trims simultaneous requests, not how much context fits |
| `HALOGEN_MAX_TOK` | `32768` | upstream's default, kept. The arena is 20.6 GiB here against 12.2 at 16384; it fit alongside the doubled pool only because the reserve dropped to 15, and it fits the trimmed 401,408 pool with room to spare. At 65536 the arena alone exceeds the budget and the server **will not boot** |
| `HALOGEN_HOST_RESERVE_GIB` | `15` | **lowered from 24 (vendor default 20) on 2026-09-12** to buy the doubled pool. The only term you own in the device budget (24 → 29.5 GiB, 15 → 38.5 GiB). Against the doubled pool the ceiling was 16 — the engine needed budget ≥ 36.5, and 18 gave 35.5, silently re-clamping; at 401,408 the need is ~33.4 so the ceiling loosens, but a bigger reserve still buys nothing. Matched pair with the pool. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away |
| `HALOGEN_VISION_TOWER` | `1` | image input; the fleet's only image path since [[frank-npu]] was retired. Charges 1.97 GiB against the budget — the term missing from my first estimate |
| `HALOGEN_CTX` | `200192` | **set 2026-09-16**: exactly half the pool, so two conversations of the full window are always resident. Not a round 200,000 — pool/2 is. The model's native window is 262,144, so this is a real reduction in the per-conversation ceiling, and a client asking for more gets a 400 (never a silent truncation). The harness declares 200192 for `frank-large` |
| `HALOGEN_KV_POOL_POSITIONS` | `400384` | **retuned 2026-09-16** (from 401,408): 2 x 200,192. Reserves ~11.0 GiB of KV, confirmed by the boot ledger as `400384 positions, ~27.2 GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` against a 33.5 GiB device budget — **no clamp**. Slots still oversubscribe the pool for any real request (admission reserves prompt + `max_tokens`), so a near-full pair that is both generating is pool-bound. `HALOGEN_KV_POOL_FIT` guards the boot: read the ledger, not this column |
| `HALOGEN_KV_SLOTS` | `2` | **lowered from 3 on 2026-09-16** — one slot per half of the pool, and two is the fleet's shape now that Dankclaw serves two endpoints. Each slot is 111.0 MiB of O(1) state, so this hands ~111 MiB back as well |
| `HALOGEN_MAX_TOK` | `24576` | **halved from the 32768 default on 2026-09-16** for frank-small, its co-tenant on the same host. The arena's variable term goes `8.40 x 2 → 8.40 x 1.5`, handing ~4.2 GiB back at roughly a 4-5 % prefill cost — the same lever at 16,384 costs 8-9 %. The vendor's own notes call 24,576 the largest arena that allocated with headroom to spare, so it stays inside their tested range. 65536 still cannot boot. It sizes a per-call PREFILL CHUNK, not the prompt ceiling: a 200k conversation is prefilled in arena-sized pieces |
| `HALOGEN_HOST_RESERVE_GIB` | `20` | **raised from 15 with the same retune**, because the arena shrank: `budget = 121.2 − 67.7 − 20 = 33.5 GiB` against a measured need of ~27.2, so the fit check kept ~6 GiB of margin and did not clamp. It is **not** an allocation: raising it cannot return memory to the host, only take the pool away (see the ceiling section below) |
| `HALOGEN_VISION_TOWER` | *unset* | **removed 2026-09-16.** The tower charges 1.97 GiB against the same device budget as the pool and the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]] (llama.cpp with `--mmproj`). This server refuses image content parts, and `models.yml` declares `frank-large` text-only to match |
| `HALOGEN_DOWNLOAD` | `peonist-ai/halogen-qwen3.8-flash-next` | first-start fetch |
| `HALOGEN_VERBOSE` | `1` | verbose engine logging — the fullest startup account, on permanently so an incident report needs no restart-into-debug |
@@ -104,10 +108,13 @@ Three numbers explain the whole box:
**overstate free memory by the size of the model**. The server prints the
corrected figure at startup; believe that line, not Grafana.
- **~28 KiB per KV position** (measured: 262,144 positions = 6,656 MiB of KV,
~7.2 GiB with block scratch), plus the prefill arena (16.7 GiB at
`HALOGEN_MAX_TOK=32768`, 8.4 GiB at 16384). The engine's device-side budget
stops at roughly 47 GiB on a 128 GB machine *with the weights pinned* — and
lower on a machine carrying more besides this server, which Dankclaw is.
~7.2 GiB with block scratch), plus the prefill arena. The 2026-09-16 ledger
reads the two together in the engine's own words: `400384 positions, ~27.2
GiB (16.2 GiB of it for everything but the pool, at max_tok 24576)` — so
~11.0 GiB of pool and 16.2 GiB of arena, working slack and (no longer) a
vision tower. The engine's device-side budget stops at roughly 47 GiB on a
128 GB machine *with the weights pinned* — and lower on a machine carrying
more besides this server, which Dankclaw is.
- **`HALOGEN_HOST_RESERVE_GIB`** is what is left for the page cache the lookup
table is read through. Too little and long prompts crawl with the disk busy
and the process in uninterruptible sleep — that is a cache problem, not a
@@ -128,6 +135,8 @@ startup account, including three deliberate experiments:
| 524288 | 65536 | 24 | — | 37.3 | 46.5 | 29.5 | **OOM — will not boot** |
| 524288 | 32768 | **15** | 14.4 | 20.6 | 37.0 | **38.5** | **deployed — both features** |
| 524288 | 32768 | 16 | 14.4 | 20.6 | 37.0 | 37.5 | fits, footprint unchanged — see the ceiling below |
| **400384** | **24576** | **20** | **11.0** | **16.4** | **27.2** | **33.5** | **deployed 2026-09-16 — ledger need 27.2, no clamp** |
| 401408 | 32768 | 15 | 11.0 | 20.6 | 33.4 | 38.5 | 2026-09-13 → 09-16, superseded by the retune |
### The sizing rule
@@ -183,11 +192,14 @@ The device budget is not the whole story. Memory position by configuration
| 524288 / 16384, reserve 24 | 94.6 | 13.8 GiB | 3742 |
| 524288 / 32768, reserve **16** | **103.5** | **7.1 GiB** | 293 |
| **401408 / 32768, reserve 15** | **100.0** | **9.1 GiB** | **1396** |
| **400384 / 24576, reserve 20** | **95.4** | **14.7 GiB** | **475** |
At the doubled setting (2026-09-12) the server held 103.5 of 121 GiB and the
engine volunteered a note it had not printed before (it no longer prints at
the 401,408 trim — 9.1 GiB / 1396 blocks sits below whatever threshold
triggers it):
engine volunteered the stall note. It did not print it at the 401,408 trim, and
it prints it again after the 2026-09-16 retune. Three data points, and they do
not line up with a "free memory below X" reading: 310 blocks printed, 1396 did
not, 475 did. So treat the note as "the machine is tight" and read the
`host memory left` line rather than waiting to be told:
```
halogen: NOTE this server now holds most of the machine, and 310 contiguous
@@ -326,25 +338,33 @@ deployment:
**plus** `max_tokens` positions when it is admitted; if the pool cannot hold it
the request waits in arrival order, and after the queue timeout (3600 s) it
gets a `503 engine_busy`. A prompt large enough that prompt + budget exceeds
the pool can therefore never be admitted at all. With the pool at 401,408
(since 2026-09-13) the per-conversation ceiling is still `HALOGEN_CTX`
(262,144) — even a full-context prompt plus the 65,536 cap is 327,680 ≤
401,408. The pool binds **how many** conversations stay resident: at 524,288
two full-length ones fit; at 401,408 it is one full-length beside a ~139k
second, or three at 131k. Under the pre-09-12 262,144 pool the two numbers
were identical, which is why four concurrent 65k chats were contending for
one conversation's worth of positions.
the pool can therefore never be admitted at all. The 2026-09-16 shape is a
400,384-position pool with `HALOGEN_CTX` 200,192 and two slots: two
conversations at the ceiling fit the pool exactly, so a pair that is both
near-full and both generating is pool-bound — the second waits for the first to
release its region. The pool binds **how many** conversations stay resident,
not how long one may be; the older 524,288 / 401,408 numbers live in the ledger
tables above.
## Vision
With the tower loaded, `/v1/chat/completions` accepts the usual OpenAI image
content part carrying a `data:` URL or bare base64 — **`http(s)` URLs are
refused deliberately**, because fetching one would make the server issue
outbound requests to wherever a client pointed it. One image costs about 2,040
tokens of context and 5.5 / 11.8 / 25.3 s at 1280x800 / 1920x1080 / 2560x1440;
text at 12 pt and above is read exactly, and a bigger frame past that point
adds empty area rather than detail. Audio and video have no path here at all —
that capability left with [[frank-npu]].
**None here since 2026-09-16.** `HALOGEN_VISION_TOWER` is deliberately unset:
the tower is a 1.97 GiB charge against the same device budget as the pool and
the arena, and the fleet's image path moved to [[frank-small-qwen38-gsq-rco]],
which loads the BF16 projector alongside its own weights. This server refuses
image content parts and says why, rather than ignoring them:
```json
{"error": {"message": "this server does not accept images: the engine was
started without a vision tower (HALOGEN_VISION_TOWER)", "type":
"invalid_request_error"}}
```
The llama.cpp side handles images the same way this engine used to: a `data:`
URL or bare base64, about 2,040 tokens of context per image, and `http(s)` URLs
refused deliberately (fetching one would make the server issue outbound
requests to wherever a client pointed it). Audio and video have no path
anywhere in the fleet — that left with [[frank-npu]].
## Deploy and operate
@@ -352,17 +372,27 @@ that capability left with [[frank-npu]].
./scripts/deploy.sh dankclaw --service frank-large
```
The handler retires the llama.cpp-era **root** unit first (stops it, drops its
enable symlinks, moves `/etc/systemd/system/frank-large.service` into the home
trash — never `rm`), then pulls `ghcr.io/peonist-ai/halogen-flash-server:0.5.8`,
re-tags it to the `localhost/` alias the unit references, installs the Quadlet,
and restarts the generated unit. Boot persistence comes from
`[Install] WantedBy=default.target` in the unit file plus linger on the deploy
user — generated units **cannot** be `systemctl --user enable`d on this host.
The handler retires the ik-era **root** unit first (stop, disable, move
`/etc/systemd/system/frank-large.service` into
`~/.AGENTTRASH/20260916-halogen-restore/host-units` — never `rm`; this host's
`disable` may delete the file outright, which the handler now reports), stops
the Quadlet's own unit if it is running, then pulls
`ghcr.io/peonist-ai/halogen-flash-server:0.5.8`, re-tags it to the
`localhost/` alias the unit references, and **waits for port 7770 to be free**
before installing and starting anything.
The health gate is bootstrap-aware: while the checkpoint is absent the
container is downloading and `/health` cannot answer, so the handler waits for
the file instead of burning its budget and reporting a failure that is not one.
That last gate is not paranoia. On the first cutover attempt the ik server
logged "Deactivated successfully", and pasta — rootless podman's userspace
network stack, which binds the published port at container CREATE — still
reported `Address already in use` a second later, exiting the unit with code
126. Only `Restart=on-failure` recovered it, minutes later. A deploy that
starts a doomed container has not deployed anything, so `wait_port_free` polls
every socket on the port (listener, accepted, TIME_WAIT) and fails loudly with
the socket table if it never clears.
Boot persistence comes from `[Install] WantedBy=default.target` in the unit
file plus linger on the deploy user — generated units **cannot** be
`systemctl --user enable`d on this host.
```bash
systemctl --user status frank-large.service # the generated unit
@@ -404,15 +434,15 @@ in root's storage and answers "no such container".
## Rollback
The llama.cpp era is intact: the STRIX payload stays at
`/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX`, the
`kingjones30/ROCmFPX` build stays at
`/home/dank/GitPackages/kingjones-ROCmFPX/build`, and the retired unit, env and
serve script are in `serverAppConfigs/.AGENTTRASH/` (suffixed `.llama.cpp`).
Rolling back is a `git revert` of this change set plus a deploy — **but the
Quadlet unit must be removed from `~/.config/containers/systemd/` first**
(otherwise the restarted root unit and the still-installed container both
claim port 7770). See [[kingjones-strix-rocmfp4]].
The predecessor is intact: the ik payload stays at
`/home/dank/models/mradermacher-Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M`, the
ik build at `/home/dank/GitPackages/ik_llama.cpp/build`, and its unit, env,
serve script and the Qwen-Sharp template are archived in the repo under
`scripts/.AGENTTRASH/*.ik-llama.cpp` and `*.ik-llama-qwen-sharp`. Rolling back
is a `git revert` of this change set plus a deploy — **but the Quadlet must be
removed from `~/.config/containers/systemd/` first**, or the restarted root
unit and the still-installed container both claim port 7770. The port gate in
the handler now covers exactly that race. See [[frank-large-ik-uncensored]].
The `amd_iommu=off` kernel argument that came with this change is separate and
reverts on its own — see [[dankclaw-iommu-translation-off]].