The model cards become the StarCraft trio: frank-large -> frank-dragoon,
frank-primary -> frank-archon, frank-secondary -> frank-zealot (family
names only; endpoint/API ids unchanged). The three cards fuse into one
Model Information component: per-unit Protoss accent colours and hex-
clipped unit art, 15px type, clickable StarCraft unit-voice quotations
(original .ogg files from the fandom wiki, served from /sfx/), supervision
notes (dragoon/archon safe for autonomous use, zealot tight supervision),
compaction-model recommendation for frank-zealot, and uncensored-model
legal notes rendered as hover tooltips instead of wall text. docs.html
carries the same renames plus api-id mapping.
App-card updates: Seerr, Calibre-Web, RomM, Jellyfin set to WIP: DOWN
with Open links removed (new wip "down" badge state); Copyparty notes it
serves all other media apps content including WIP apps; Gitea and Plex
noted as needing logins separate from SSO; SearXNG noted as obfuscated,
not anonymous (upstream API keys tie to Frank).
Plumbing: Caddy @data allowlist grew /sfx/*; deploy_homepage copies sfx/;
homepage_config.bats pins the new probe names, the component fields,
art/voice-file presence, and the wip-down schema.
The @data path allowlist on the post-auth listener did not include /img/*,
so docs screenshots fell through to the index.html catch-all and the
browser got 200 text/html for every PNG (docs images broken live
2026-10-08). Pinned by the homepage_caddy seam.
The parallel session's uncommitted swap, committed at Frank's direction
(commit whatever's in the working tree):
- llama.cpp master on the RTX 3090 now serves ukisai's
Swift-1.5-Qwen3.8-27B GSQ-RCO IQ3_S WITH the in-file MTP head, ONE
131,072-token slot at Frank's 128k cap, q8_0 K + q8_0 V, vision through
the base-model projector; MiMo-V2.6 Q6_K (2x200704 f16 KV) retired to
rollback target.
- env/serve/download + seam twins updated together (deployer payload gates
rode the previous commit's deploy-remote.sh).
- docs: new swift15 runbook; MiMo Q6K runbook retired; AGENTS.md topology
and both vault indexes follow (root index ACTIVE/RETIRED flip, test 135
seam green).
- comfyui: two qwen-image t2i workflow drops under dankclaw/comfyui/
workflows/ (12 KiB).
The stock SearXNG image ships a native kagi engine; enabling it needs an
API key that must never enter git, and SearXNG does not expand env vars
in settings.yml (the live secret_key literally was '${SEARXNG_SECRET}'
until now — the app tolerated the placeholder string silently).
- settings.yml: kagi engine enabled (weight 2, kg shortcut, paging off so
exactly one API credit per aggregated query, timeout 8.0 vs 5.0 scrapers
so a slow paid query is not a wasted credit); api_key stays ${KAGI_API_KEY}.
- new scripts/searxng-render-settings.py: deploy-time renderer, allowlisted
to KAGI_API_KEY/SEARXNG_SECRET/SEARXNG_BASE_URL, fails the deploy loudly
on an unresolvable placeholder, idempotent on redeploys.
- deploy-remote.sh copy_searxng_config runs the renderer against
/opt/searxng/.env after copying settings.yml into /opt/searxng/config.
- KAGI_API_KEY placeholder added to dankclaw/.env.example (real key staged
into the host-side /opt/searxng/.env, mode 600, out of band).
- bats: engine wiring, renderer substitution, loud-failure, deployer seam.
- docs: SearXNG.md Kagi section + renderer law; Providers.md carve-out on
the no-credentials policy (paid engine acceptable when bounded + ordered).
Verified live on dankclaw: direct API probe via the gluetun netns (200,
data.search[] shape), then an aggregated query returned 29/29 results
tagged kagi, zero unresponsive engines.
The authentik OIDC round-trip is verified, so retire the local
username/password path. ENABLE_LOGIN_FORM hides the form and
ENABLE_PASSWORD_AUTH refuses password logins; the only way in is now
the OIDC flow, which folds into the local account by email. The
ui.enable_login_form config-DB override is flipped to false in the
same change (the DB wins over env, so both must agree).
systemd splits unquoted Environment= values on whitespace, so
'OAUTH_SCOPES=openid email profile roles' arrived as 'openid' and the
authorize request carried a single scope — the id_token then lacked the
email claim and the callback failed. Quoted, all four scopes flow.
Also: assign the default openid/email/profile ScopeMappings to the
provider (2026.x no longer auto-creates them; without them the token
grant is the intersection with the provider's mappings = roles only,
and userinfo 403s).
The Qwen-Image 2.1 VAE is a video VAE (5D convs, time_conv, depth-3).
ComfyUI detects it as WanVAE (latent_dim=3), but EmptyLatentImage produces
a 4D latent. The decode function's memory_used_decode lambda accesses
shape[4] which is out of range for 4D.
Fix: expand 4D latents to 5D in the decode function when latent_dim==3.
Added to the patch script for persistence across container recreations.
The qwen_image21 detection check and GGUF node arch list patch are
applied by a startup wrapper (comfyui-start.sh) that runs before
ComfyUI starts. The patch script and wrapper are baked into the image
so they survive container recreations. The quadlet's Exec now points
at the wrapper instead of main.py directly.
The per-key config DB already pointed web search at SearXNG (engine=searxng,
query URL http://192.168.0.143:8081/search), but only inside the data volume.
The quadlet now carries the same values as seeding env vars, so a fresh data
dir comes up wired, and the doc records the access path: the internal
host-published port, not the search. vhost — the Anubis->Authentik guard
chain is browser-only, and the bridge egress sits in SearXNG's pass_ip, so
the 4-req/IP/hour JSON cap never applies.
The board rendered vertically for a user whose browser had cached the
pre-sectioning style.css: deploys overwrite /srv/homepage in place, and
nothing told the browser to revalidate. The @home handler on the post-auth
listener now sends Cache-Control: no-cache, so every load is an ETag
round-trip (a cheap 304 when unchanged) and a deploy is always visible on
the next navigation.
Operator call 2026-09-19: switch the iGPU endpoint from
halogen-flash-server to jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca on the CIRU
v4.4.1 runtime with the QSA window hotfix (host build against the private
ROCm 10 SDK), one stream of the full 262144 trained window. The service
leaves the quadlet shape for a root unit (the same shape as frank-small and
frank-medium: a private-SDK host binary has no image to install and no
generated unit to dodge); the halogen quadlet is archived, the deploy
handler retires it on the host, gates on 7770, and its stray-run sweep
matches the model path so it cannot take frank-small down.
The vLLM detour (a2b39b1, same day) is retired: the Q4_K_M llama.cpp setup
is the requested shape and the vLLM KV-quant options on sm_86 all measured
dead (fp8 hardware-impossible, int8 page-size unification, turboquant
mm-prefix). Root unit restored, vLLM quadlet/Dockerfile/fetch script moved
to .AGENTTRASH, runbook listed as same-day-reverted. Rollback target:
frank-small-vllm-gemma4.
podman 5.7.0's v1 quadlet generator rejects both ContainerEntrypoint
and Entry; the key is Entrypoint= (pkg/systemd/quadlet/quadlet.go
KeyEntrypoint, mapped to --entrypoint).
The podman 5.x v1 quadlet generator on dankclaw rejects
ContainerEntrypoint with 'unsupported key in group Container', so the
unit was never generated and the deploy died at restart with 'Unit
frank-small.service not found'. Entry= is the v1 spelling.
Runtime swap on the RTX 3090 (port 7771):
- vLLM gemma-x86_64-cu129 dev build, digest-pinned, audio extra pip-installed
into localhost/vllm/vllm-openai:frank-small (Dockerfile, no COPY lines)
- google/gemma-4-12B-it-qat-w4a16-ct (10.26 GiB int4) + Google's bf16 MTP
assistant (845 MB) — the w4a16-ct assistant cannot draft in vLLM (drafter
path skips Marlin; acceptance 1.00), bf16 measures 2.80-3.56
- root systemd unit -> rootless Quadlet (dankclaw/frank-small/quadlet/);
manual CUDA device+lib mounts (no nvidia-container-toolkit on host)
- bf16 KV (fp8 hardware-dead on sm_86), 289,647-token pool, 2 concurrent
full-context sessions, CUDA graphs on
- native text+image+audio: the fleet's multimodal endpoint
- measured: pp@67k 1293 t/s, pp@213k 608 t/s, tg@213k ~13.6 t/s,
tg@85k ~17.6 t/s
- host driver 610.57.04 (Ubuntu 26.04); sm_86 is correct, no arch override
- planner/scoping: frank-small now selected by dankclaw/frank-small/ +
env/serve/download trio; old unit retired to scripts/.AGENTTRASH/
- docs: new runbook frank-small-vllm-gemma4.md; AGENTS.md, LLM index,
top docs index updated; old runbook moved to retired
- services.json: frank-large/medium/small move from admin-only to the user
page with current descriptions (halogen w4b iGPU; 35:30 layer-split MTP
27B on the R9700s; Gemma4-12B QAT vision path on the 3090) and docs
links to /docs#models
- docs.html: static user documentation (access chain, model guide, media
how-tos, troubleshooting); served at /docs via the post-auth homepage
listener; shipped by deploy_homepage; footer + index link to it
- tests: pages seam asserts the docs contract (page, link, Caddy route,
deployer copy)
- vault: Homepage.md synced (docs landing shipped, cards reclassified)
169.254.1.2 is not a local address on this host (netavark runs its
bridges in aardvark-dns' netns), so the relay cannot bind it — and a
loopback bind refuses container traffic. Probe-socket proof: a host
socket on 192.168.0.143:18094 answers the container's 169.254.1.2:18094
dial while the LAN IP refuses from the container. The relay binds
192.168.0.143:18093 and the deployer installs an iptables INPUT pair
(ACCEPT 10.89.0.0/16, DROP rest) so the unauthenticated endpoint stays
off the LAN.
A socket bound to 127.0.0.1 refuses container traffic that arrives via
the podman bridge: the first live webhook failed with 'dial tcp
169.254.1.2:18093: connection refused'. The relay now binds the
host-gateway endpoint — link-local to the podman bridge, off the LAN —
and the deployer's health probe follows it there.
hermes-webui is Network=host, so 8787 is a bare host-netns bind, and netavark
DNATs only *published* ports. From Caddy's bridge netns 192.168.0.143:8787 and
10.89.1.1:8787 are both refused, while host.containers.internal:8787
(169.254.1.2) answers 200 — measured live 2026-09-16 from inside the Caddy
container. The dashboard's published 9119 answers on that same LAN IP, which is
what made the old upstream look sound.
The vhost comment asserted "192.168.0.143:8787 is open" from that bridge; it is
not, so the claim is replaced with the measurement, and the hermes-webui quadlet
comment carries the same correction. The seam guarding this invariant accepted
any numeric upstream, so the LAN IP passed it; it now pins the measured-working
form and fails on the LAN IP (teeth verified by substitution).
Dankclaw's endpoints are re-split. frank-large is the halogen-flash-server
quadlet again -- the 2026-09-13 unit file was never touched by the two-day
ik_llama.cpp interlude, so the runtime was still in the repo and its image is
still in local storage -- retuned to the shape asked for: HALOGEN_CTX 200192
with a 400,384-position pool and two slots, exactly two 200k conversations.
frank-small is new: mainline llama.cpp on the RTX 3090 serving
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF at IQ3_S, using the -mtp build (it carries
blk.64.nextn.*, which is what lets --spec-type draft-mtp run on the main
model's own weights instead of a second 12 GiB copy) plus the BF16 vision
projector, as one 131,072-token slot with f16 KV on port 7771.
The host is shared now, and that is what the halogen retune pays for:
HALOGEN_MAX_TOK 24576 (arena 20.6 -> 16.4 GiB, ~4-5% prefill) and the vision
tower dropped (1.97 GiB) hand ~6 GiB back to the host, and the reserve rises to
20 to keep the fit check honest. Vision moved wholesale to frank-small.
Retired to scripts/.AGENTTRASH/: the ik trio (env, service, serve script), the
Qwen-Sharp template only that runtime read, and four benchmark harnesses bound
to retired runtimes. Because retiring a file makes any test that greps it pass
by absence, the seams assert the archive exists AND the live path does not.
Retiring dankclaw/compose.yaml and adding alert rules invalidated claims across
the repo, including in the rule file agents read first. Clean cutover includes
the prose.
**Tests that had gone hollow.** Four assertions grepped dankclaw/compose.yaml
for a service name. With the file gone they passed by *absence* — a gate that
can no longer fail. Each now asserts the real invariant ("no dankclaw compose
file competes with the quadlet units") and keeps a live check for a re-added
entry, so it still fails when one appears: verified by recreating a compose file
with a comfyui service and watching three tests go red, then removing it.
**AGENTS.md, which was wrong about ComfyUI.** It said ComfyUI runs CPU-only
("no ROCm wheels for gfx1151, CUDA images wrong"). That was true in the
compose era and is false now: the restored unit runs with /dev/kfd + /dev/dri and
must NOT pass --cpu. An agent trusting the rule file would have re-broken it.
Also corrected: the observability paragraph ("RUNNING BUT UNCONFIGURED, no
datasource, no rules" — all three have been configured since 2026-09-16), the
Active runbook row that still named halogen for frank-large, halogen's absence
from the Retired table, the benchmark policy naming halogen's harness, and the
seam list.
**Unit and page claims.** prometheus.container said "no alerting rules anywhere
in this stack"; node-exporter.container said Dankserver runs from its compose
stack; Prometheus.md documented three jobs as working (they scrape HTML: they
can never report up) and said no dashboard existed; index.md and the three
authentik service pages pointed at the retired file; Authentik.md's upgrade
instruction told you to bump an image tag in it.
One bad edit of my own on the way (a stale line number clobbered a See Also
entry in Prometheus.md) — caught by reading back, fixed, and the vault link
checker now proves the cross-references hold.
I tightened HERMES_WEBUI_HOST to 127.0.0.1 on the reasoning that the compose
service had published loopback and the unit had widened it. That reasoning
missed the consumer: this unit is host-networked (it shares the CDP daemon's
loopback), so Caddy cannot reach it by container name and fronts the guarded
vhost hermes.franksdank.cloud with `reverse_proxy 192.168.0.143:8787` — recorded
in dankclaw/Caddyfile with the same explanation. With the app on loopback that
upstream is dead, so the vhost broke.
The wildcard bind is what serves BOTH consumers: Caddy on the LAN address and
the phone's SSH tunnel on the host's localhost. Controls are the auth chain in
front of the vhost plus the webui password. The unit comment now states this
instead of the incorrect 'as before' claim, and the near-miss is worth the
paragraph: the e2e gate stayed green because no spec covers that vhost.
Two things, found while checking whether the hermes trio still needed migrating.
1. **The phone-facing webui was exposed to the LAN.** Its unit runs with
Network=host, so there is no Podman port forwarder and the app's own bind is
the exposure. The compose service it replaced published
`127.0.0.1:8787:8787` (loopback); the unit bound `HERMES_WEBUI_HOST=0.0.0.0`,
and the LAN address answered 200 — password-gated, but on every interface,
which is not the documented "SSH tunnel is the sole ingress" posture. It now
binds 127.0.0.1, which serves the tunnel just as well (sshd dials
localhost:8787 from the host). The unit comment claiming "LAN-facing as
before" was wrong about "before", and is corrected.
2. **Dankclaw's compose.yaml is retired.** Auditing ownership showed every
running container on that host reports a PODMAN_SYSTEMD_UNIT label — the
hermes trio's units have owned their containers since 2026-09-12 — so the
three service blocks left in compose.yaml were dead weight that could only
create a duplicate container fighting a unit for a name. deploy_hermes was
already fully unit-based. The file is staged in .AGENTTRASH/; the planner
plans nothing for it, an unknown service on that target now errors instead of
silently no-oping through `podman-compose`, and DANKCLAW_COMPOSE_SERVICES is
empty.
The hermes-webui seam now reads the unit — it read compose, which is exactly how
the bind drifted while the test kept passing against a file nobody ran.
ComfyUI had failed 9,146 times and had been down ~7 hours when this was found:
its container was created 2026-09-15 14:12, BEFORE the GPU declarations were
added to the compose service, and `restart: unless-stopped` kept restarting
that stale container — no /dev/kfd, no /dev/dri, no keep-groups — so it died
every time on `RuntimeError: No HIP GPUs are available`. Its "unit" was a
hand-written ~/.config/systemd/user/comfyui.service that shelled out to
`podman compose up -d`, which is not repo-tracked state and collides by name
with the Quadlet the generator produces.
- dankclaw/comfyui/quadlet/comfyui.container: image comfyui-rocm:gfx1151
(built from Containerfile.comfyui), /dev/kfd + /dev/dri, keep-groups,
the four comfyui-data volumes, loopback-only publish (an unguarded service
is internal-only here; the homepage card probes 127.0.0.1:8188), Restart=
always, WantedBy=default.target.
- the compose service is deleted — one owner per container name.
- deploy_comfyui builds the image, installs the unit, retires the compose-era
unit into ~/.AGENTTRASH, removes the stale container so the unit's own can
take the name, and waits on :8188.
- the healthcheck is a file, because the podman 5.x v1 generator splits
HealthCmd on spaces and an inline `python3 -c "..."` becomes a quoting
puzzle that fails silently.
The comfyui seam now reads the unit instead of compose for devices, command and
groups (and resolves the Containerfile from the deployer's declaration, not
from a directory listing).
dankclaw/comfyui/Containerfile is referenced by nothing — compose builds from
dankclaw/Containerfile.comfyui, deploy-remote.sh copies that one, and the
ComfyUI doc names it — and its content is an older iteration (torch
2.9.1+rocm7.12.0rc1 against the deployed gfx1151 base). It also caused real
damage: the comfyui test's directory-listing selector picked non-comfyui files
and reported a red suite for sessions.
Staged in .AGENTTRASH/ rather than deleted, so it stays recoverable and visible
until Frank decides it can go for good. This is also why the earlier test fix
had to resolve the Containerfile from compose rather than from the filesystem:
the directory is not the contract.
The radiance MXFP4 stack cannot run TP=2 on this host. RCCL's collective
kernels carry a hidden hostcall-buffer parameter, and ROCr refuses to
dispatch them on a device without PCIe atomics: the AQL dispatch fails with
hipErrorIllegalState ("the operation cannot be performed in the present
state"), which is what every two-rank collective hit. The second R9700 sits
behind the chipset bridges 05:00.0/06:09.0, which advertise
`AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
atomic routing on that path and the driver's
pci_enable_atomic_ops_to_root() fails for that card. The first card, on CPU
root port 00:01.1, has atomics enabled.
Ruled out: kernel version (7.0.0-28/-30/-31 identical, after a full upgrade),
iommu=pt, P2P hardware (45.3 GB/s between the cards), the shipped librccl (a
from-source rebuild reproduces the failure at the same call path), vLLM and
pynccl (a bare two-rank torch.distributed allreduce fails the same way), all
NCCL transport knobs, and single-rank tests (NCCL short-circuits at
world_size=1, so one-card success proves nothing).
Deployment: three rootless Quadlets — frank-medium-a (:7771, card 1),
frank-medium-b (:7772, card 2), both TP=1 at 114688 context (262144 and
131072 do not fit one card's KV) with vLLM-profiled KV — and
frank-medium-proxy (:7770), a round-robin that preserves the fleet contract
of one endpoint and one served model name. frank-medium.container stays as
the TP=2 target for a host whose second card is atomic-capable; the deploy
handler does not install it and trashes any stale copy.
Measured through the proxy (vllm bench serve, random dataset, --ignore-eos):
conc 1 42.8 tok/s, conc 4 147.6 tok/s, conc 8 230.8 tok/s, 32k prefill at
~2.8k tok/s per card, TPOT p50 23.6-28.8 ms.
Also: healthcheck reads PORT so one probe serves all three units, the hermes
example and the harness models.yml drop to the real 114688 context, and the
runbook records the full evidence chain.
Predicted ~10.8 GiB KV / ~33.4 need / ~10.5 host-left; the boot ledger
printed ~11.0 / ~31.5 / 100.0 GiB held with 9.1 GiB + 1396 contiguous
2 MiB blocks left. Stall NOTE no longer prints at this footprint. Also
documents the ~45-60 s first-request PING silence after a restart (the
healthcheck's HealthStartPeriod covers it; do not restart the unit).
The 09-12 doubled pool fit only at reserve 15, leaving 7.1 GiB / 310
contiguous 2 MiB blocks for the ~28 co-tenant containers and making the
engine print its minutes-long-stall warning on every boot. 401,408
positions (~10.8 GiB KV) hands ~3.4 GiB back while keeping MAX_TOK=32768
and reserve 15: modelled need ~33.4 GiB vs budget 38.5, so the fit check
should not clamp. Capacity becomes one full 262k conversation resident
beside a ~139k one, or three concurrent 131k chats (three slots).
Docs: AGENTS.md fleet table + halogen section, runbook flags table /
co-tenancy tense / admission-arithmetic paragraph updated for the trim.
Concurrency cap trimmed to the real fleet size. Each slot is 111.0 MiB of
O(1) state, so this hands ~111 MiB back on a host sitting at 7.1 GiB left
for everything else.
Note the slots already oversubscribed the pool (4 x 262144 > 524288), so
admission was pool-bound regardless: at most two full-length conversations,
or four at 131k. This does not change how much context fits -- it caps
simultaneous requests at three, and a fourth now queues.
Tested 16 live. Pool stayed 524288 (no clamp), server held 103.5 GiB and
reported "7.1 GiB left for everything else" at BOTH 15 and 16 — identical
footprint. HALOGEN_HOST_RESERVE_GIB feeds the fit-check budget only; it is not
an allocation, so raising it cannot give RAM back to the co-tenants. Its sole
observable effect above 16 is to take the KV pool away.
16 therefore cost half the fit margin (1.0 vs 2.0 against the engine's 36.5 GiB
requirement) and bought nothing. 15 is strictly better at the same cost.
Recorded the ceiling (16) and the real air levers in the unit comment: pool
262144 frees 7.2 GiB, max_tok 16384 frees 8.4.
Operator asked for 18. 18 is not available: the engine's own fit-check line,
read from a clamped boot rather than modelled, requires budget >= 36.5 GiB to
keep 524288 at max_tok 32768 ("524288 positions need ~35.0 GiB (plus 1.5 of
margin)"). budget(18) = 35.5, so 18 silently un-doubles the pool we just paid
the reserve for. 17 fits by 0.02, inside the noise of the 67.7 resident-weights
term. 16 is the largest reserve that survives the check with real margin:
37.5 budget vs 36.5 required.
Recorded the ceiling and both remedies in the unit comment. The reserve is now
spent, so a future stall is answered by the pool or the arena, not here.
Ledger confirms the doubled pool is actually reserved (524288, 13312.0 MiB)
at max_tok 32768 with no clamp and no WARNING: 38.5 budget vs 37.0 needed.
Recorded the cost honestly. The server now holds 103.5 of 121.2 GiB and the
engine reports only 310 contiguous 2 MiB blocks / 7.1 GiB left for everything
else (24-reserve baseline: 2081 blocks / 12.1 GiB). All 28 co-tenants stayed
up, but the engine volunteers a new note: when memory runs out, frank-large
AND its neighbours "can stall for minutes at a time at full CPU with no disk
activity and no output, which is not a crash and needs no restart". That is
the least diagnosable of the three failure modes seen today, and `free` hides
it — MemAvailable still reads ~74 GiB because the pinned weights count as
reclaimable page cache.
Sizing rule now expresses the budget as a function of reserve, shows both
endpoints (24 -> 29.5, 15 -> 38.5), and the max_tok ceiling is labelled as
reserve-conditional (<= 18,130 at 24; <= 35,700 at 15).
Neither lever alone could give both features. At reserve 24 the device budget
is 29.5 GiB and max_tok 32768's 37.0 GiB need always lost the doubling to the
fit-check clamp; at max_tok 16384 the doubling survived but prefill paid 8-9%.
Dropping the reserve to 15 moves the budget to 38.5:
121.2 host - 67.7 resident weights - 15 reserve = 38.5
need = 14.4 pool + 20.6 arena + 1.97 vision = 37.0 -> 1.5 GiB spare
So the payment is page cache for the n-gram lookup table rather than either
feature. 15 GiB, against a table of 47.7 GiB read through the file cache.
Two reasons this is a survivable risk rather than a cliff:
- The engine's own ledger reported only 12-17 GiB left at the previous 24
setting, so the declared reserve was never held literally; 15 may cost
less real cache than it reads as.
- The failure mode is legible: long prompts fault heavily against a busy
disk, and raising the reserve is the documented remedy. Not the silent
clamp, and not the 65536 no-boot mode.
1.5 GiB of headroom is thin and deliberate. Documented in the unit as a
matched pair with the pool: raise the reserve back to 24 and the doubling
quietly goes away again.
Deployed HALOGEN_MAX_TOK=32768 by operator call (upstream default, full prefill
speed) and confirmed the prediction from the ledger:
kv pool: 524288 positions need ~35.0 GiB ... LOWERING THE POOL TO 262144 (~27.8 GiB)
memory: 68.0 weights + 7.2 KV pool + 21.1 working = 96.3 GiB in all
so the effective pool is 262144 again and the doubling this task set out to do
is not in force. Healthy, completion verified. Left the pool requesting 524288
deliberately: with KV_POOL_FIT on that is a grow-if-RAM-frees-up declaration,
not a mistake.
Two corrections to what I wrote earlier today:
1. The engine's sizing block, and its "WARNING even 262144 ... host RAM cannot
spare it" line, print ONLY when the fit check had to clamp something. The
09-11 baseline boot prints no kv pool: sizing lines at all. So the absence
of a warning in old logs was never evidence of headroom.
2. The budget needs a vision term. Actual need is pool + arena + 1.97 GiB of
vision tower, not pool + a 1.5 GiB nominal margin. That makes the ceiling
for a doubled pool max_tok <= 18,130 rather than 19,115, and it means the
32768/262144 corner this box has run in since 09-11 is ~0.3 GiB OVER its own
stated budget -- and has been the whole time. It boots and serves with
12-14 GiB genuinely free because the 24 GiB reserve is generous. A soft
floor, not a wall.
The arena model also reproduces upstream's three published "on the device"
figures exactly (35 / 42 / 41 GiB), confirming they are pool+arena totals at
the default max_tok and not pool-only -- which is the misreading that made the
doubling look affordable in my first estimate.
Operator call. Prediction from the measured model: 524288 positions cost
35.0 GiB against a 29.5 budget, so HALOGEN_KV_POOL_FIT clamps the pool to
262144 (27.8). Rollback is a single revert; 16384 was the config that made
the doubling real.
Raising HALOGEN_MAX_TOK to 65536 did not shrink the pool, it OOMed the boot:
the arena alone (37.3 GiB) exceeds the 29.5 GiB device budget, so even the
minimum pool faulted and dmalloc failed at 39.97 GiB. Rolled back to 16384,
which restored the 524288 pool and a clean completion.
The three measured arenas are exactly linear in the flag — 3.80 GiB base plus
8.40 GiB per 16384 tokens, predicting all three to 0.1 GiB. That gives a real
sizing rule instead of guesswork, and it says the deployed 16384 is the
CEILING, not a conservative choice: 29.5 - 1.5 margin - 14.4 pool leaves 13.6
for the arena, i.e. max_tok <= 19,115.
The corner check falls out of the same model: set max_tok back to 32768 and
only 7.4 GiB remains for the pool, ~269k positions, which is precisely the
262144 config this host ran before. The two corners of the feasible box are
the only two points it has ever served.
No functional change — the quadlet edit is comments only; the running service
is already at 16384 from the rollback deploy (258cccb).
Operator call, measuring what the prefill arena actually costs against
upstream's do-not-raise warning. Rollback is one revert: the settled
config was 16384, which made the 524288 pool fit at 26.6 GiB.
The previous commit's doubling was a silent no-op. HALOGEN_KV_POOL_FIT
measured the device budget at startup and lowered the pool straight back
to 262144, while the deploy still reported healthy in 12s.
The engine's arithmetic: 121 GiB host - 67.7 resident weights - 24.0
reserved for the n-gram page cache and the OS = 29.5 GiB of device
budget, and that single budget carries the KV pool AND the prefill arena.
At HALOGEN_MAX_TOK=32768 the arena is ~20.6 GiB, so 524288 positions
need 35.0 GiB against 29.5 available.
HALOGEN_MAX_TOK=16384 halves the arena to ~10.5 GiB, bringing the
524288 need to ~24.9 GiB, which fits with ~3 GiB of margin. Cost is
about 8-9% of prefill throughput, per upstream. This is the engine's own
prescribed fix, quoted verbatim in its refusal message.
Lowering HALOGEN_CTX was the other option the engine offered; rejected
because both clients declare 262144 as the context window, so it would
break the client contract rather than pay for the pool.
HALOGEN_KV_POOL_POSITIONS 262144 -> 524288, which is upstream's own
default (2 x HALOGEN_CTX). The old pool was the binding constraint on
concurrency, not context: a request reserves prompt + max_tokens against
the pool, so with pool == CTX four concurrent 65k chats were contending
for one full-length conversation's worth of positions. Two full-length
conversations, or four at 131k, now stay resident.
Cost is ~7.2 GiB more device RAM (measured 28 KiB/position), taken from
the page cache the 47.7 GiB n-gram lookup table reads through. Live
startup ledger for the prior config: 68.0 weights + 7.2 pool + 21.1
working = 96.3 GiB held, 12.1 GiB left on the host. The doubled pool
puts the remainder near 5 GiB against a ~4.3 GiB co-tenant working set,
and cache at ~69 GiB -- still above the lookup table, so the table can
stay hot. Compensating lever if long prompts start crawling:
HALOGEN_MAX_TOK=16384, which frees 8.3 GiB of prefill arena for ~8%
of prefill throughput.
Docs updated to match, including the admission-trap paragraph, which was
only true while pool == CTX.
Frank