Frank
43ec77fee6
Dankclaw's endpoints are re-split. frank-large is the halogen-flash-server quadlet again -- the 2026-09-13 unit file was never touched by the two-day ik_llama.cpp interlude, so the runtime was still in the repo and its image is still in local storage -- retuned to the shape asked for: HALOGEN_CTX 200192 with a 400,384-position pool and two slots, exactly two 200k conversations. frank-small is new: mainline llama.cpp on the RTX 3090 serving ISTA-DASLab/Qwen3.8-27B-GSQ-RCO-GGUF at IQ3_S, using the -mtp build (it carries blk.64.nextn.*, which is what lets --spec-type draft-mtp run on the main model's own weights instead of a second 12 GiB copy) plus the BF16 vision projector, as one 131,072-token slot with f16 KV on port 7771. The host is shared now, and that is what the halogen retune pays for: HALOGEN_MAX_TOK 24576 (arena 20.6 -> 16.4 GiB, ~4-5% prefill) and the vision tower dropped (1.97 GiB) hand ~6 GiB back to the host, and the reserve rises to 20 to keep the fit check honest. Vision moved wholesale to frank-small. Retired to scripts/.AGENTTRASH/: the ik trio (env, service, serve script), the Qwen-Sharp template only that runtime read, and four benchmark harnesses bound to retired runtimes. Because retiring a file makes any test that greps it pass by absence, the seams assert the archive exists AND the live path does not.
220 lines
12 KiB
INI
220 lines
12 KiB
INI
# Quadlet unit: frank-large (rootless, boot-persistent).
|
|
#
|
|
# Install path on dankclaw: ~/.config/containers/systemd/frank-large.container
|
|
# Unit name: frank-large.service (container name: frank-large)
|
|
#
|
|
# The large inference service. The halogen-flash-server runtime (since
|
|
# 2026-09-11, restored 2026-09-16 after a two-day ik_llama.cpp hybrid
|
|
# interlude) serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next,
|
|
# w4b .hgn checkpoint + quality overlay sidecar). It replaced the llama.cpp /
|
|
# kingjones30-ROCmFPX STRIX deployment because the engine is ~4x faster on
|
|
# prefill (the dominant cost of any real prompt) on this exact SKU. The RTX
|
|
# 3090 it briefly shared with frank-large is now frank-small's card.
|
|
#
|
|
# SHAPE. The image runs both halves in one container: the engine on container
|
|
# loopback (:8730, UNAUTHENTICATED — never publish it) and the OpenAI
|
|
# front-end on :8731, published here as host 7770 so the fleet's clients,
|
|
# services.json probe and docs keep the port they have always used. Same for
|
|
# HALOGEN_MODEL_ID=frank-large: it is the id at /v1/models, in /health and in
|
|
# every response, so no client had to be renamed. Halogen never rejects a
|
|
# request for naming a different model, it just serves it.
|
|
#
|
|
# DEVICES. gfx1151 through the amdgpu/KFD stack: /dev/kfd plus the DRM render
|
|
# node. Both are 0666 on this host, so the rootless user namespace can open
|
|
# them (id dank is also in render/video). seccomp=unconfined and --ipc=host are
|
|
# the image's documented requirements; memlock must be unlimited.
|
|
#
|
|
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
|
|
# runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
|
#
|
|
# MEMORY. Weights are pinned (HALOGEN_FLASH_PIN_TRUNK default 1) — that is the
|
|
# configuration the published decode numbers were measured with, and `free`,
|
|
# MemAvailable and every monitor built on them will overstate free memory by
|
|
# the size of the model (~68 GiB) because the kernel counts the locked weights
|
|
# as reclaimable page cache. Halogen prints the corrected figure at startup.
|
|
# The pinned trunk means this container holds most of the host; a co-tenant
|
|
# that then takes the remainder can stall both for minutes at full CPU with no
|
|
# disk activity and no output. That is documented behaviour, not a crash, and
|
|
# the escape hatch is HALOGEN_FLASH_PIN_TRUNK=0 (at several times the decode
|
|
# cost). Sizing is chosen for a SHARED 121 GiB host, and HALOGEN_KV_POOL_FIT
|
|
# stays on as the safety net: if the machine refuses the configured pool the
|
|
# engine lowers it itself and prints what it settled on. Read that ledger on
|
|
# every restart — it is the ground truth for sizing, not this comment.
|
|
#
|
|
# POOL RETUNED 2026-09-16: 400,384 positions -- exactly two 200,192-token
|
|
# conversations -- beside a per-conversation ceiling of 200,192 and two slots.
|
|
# It replaces the 401,408 the 2026-09-13 trim left behind, which held one full
|
|
# native-262k conversation beside a ~139k one, or three at 131k.
|
|
#
|
|
# MEASURED AT 401,408 (2026-09-13 deploy ledger): 68.0 weights + 11.0 KV +
|
|
# 21.0 working = 100.0 GiB held, 9.1 GiB / 1396 contiguous 2 MiB blocks left,
|
|
# engine need printed ~31.5 against the modelled ~33.4, and NO stall-warning
|
|
# NOTE. Fit-check budget here: 121.2 - 67.7 - 20 = 33.5. That boot is the
|
|
# baseline the 09-16 retune is measured against: the vision tower (1.97 GiB)
|
|
# and roughly 4.2 GiB of arena come back to the host, which is what pays for
|
|
# the co-tenant below.
|
|
#
|
|
# CO-TENANT since 2026-09-16: this host also serves frank-small on the RTX
|
|
# 3090 (llama.cpp, ~12.2 GiB of weights that load through the page cache and
|
|
# then live in VRAM, plus a CUDA context). It is the first real co-tenant this
|
|
# service has had, which is why the arena was halved and the reserve raised.
|
|
#
|
|
# BOOT NOTE: on the first request after a (re)start the engine can go silent
|
|
# for ~45-60 s ("has not answered PING") while it first-touches the freshly
|
|
# reserved pool; /health reads engine_unresponsive during it and recovers by
|
|
# itself. The quadlet health check's HealthStartPeriod=300s covers it; do not
|
|
# restart the unit over it.
|
|
#
|
|
# WEIGHTS. HALOGEN_DOWNLOAD fetches the whole weights repo on FIRST START ONLY
|
|
# (the entrypoint checks whether the checkpoint is absent) into the bind mount
|
|
# below, which must therefore be read-WRITE. huggingface_hub resumes partial
|
|
# files natively, and the quadlet runs the container with --replace --rm, so a
|
|
# restart mid-fetch continues rather than starting over. Roughly 127 GB: the
|
|
# 115.55 GiB checkpoint, the 2.31 GiB quality overlay, the 0.84 GiB vision
|
|
# tower and the tokenizer. The repo is public and ungated, so no token is
|
|
# needed; later starts open no outbound connections at all.
|
|
#
|
|
# VISION. Deliberately NOT set since 2026-09-16. The tower is a 1.97 GiB
|
|
# charge against the same device budget as the pool and the arena, and the
|
|
# fleet's image path moved to frank-small (llama.cpp on the RTX 3090, loaded
|
|
# with --mmproj mmproj-Qwen3.8-27B-BF16). This server refuses image content
|
|
# parts, and models.yml declares frank-large text-only to match.
|
|
#
|
|
# CLIENT CONTRACT (see serverAppConfigs-docs/LLM/halogen-flash-server.md).
|
|
# There are no server-side sampler defaults: a request that sends no
|
|
# temperature decodes GREEDY. The per-request budget defaults to 8,192 tokens
|
|
# and REASONING COUNTS AGAINST IT, so a budget that runs out mid-thought
|
|
# returns finish_reason "length" with empty content and the partial reasoning
|
|
# in reasoning_content. The cap is 65,536; above it a request is a 400, never a
|
|
# silent truncation.
|
|
#
|
|
# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet
|
|
# path cannot resolve registry-prefixed refs from local storage at create
|
|
# time. deploy-remote.sh pulls the ghcr.io ref and re-tags it before the
|
|
# restart. Pull=never keeps boots off the registry.
|
|
[Unit]
|
|
Description=frank-large — halogen-flash-server, Qwen3.8-Flash-Next (w4b) on gfx1151
|
|
After=network-online.target
|
|
|
|
[Container]
|
|
ContainerName=frank-large
|
|
Image=localhost/peonist-ai/halogen-flash-server:0.5.8
|
|
# Published port stays 7770: Hermes, open-webui, the harness and the homepage
|
|
# probe all point here. 8730 (the engine) is deliberately not published.
|
|
PublishPort=7770:8731
|
|
AddDevice=/dev/kfd
|
|
AddDevice=/dev/dri
|
|
# Read-write: the first start downloads into it. Contains only weights.
|
|
Volume=/home/dank/models/halogen:/models
|
|
# Fetch the weights on first start only (absent checkpoint gates it).
|
|
Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
|
|
# Fleet identity: the id at /v1/models, /health and every response body.
|
|
Environment=HALOGEN_MODEL_ID=frank-large
|
|
# Per-conversation ceiling: exactly half the pool, so two conversations of the
|
|
# full window are always resident. 200,192 rather than a round 200,000 is
|
|
# pool/2. The model's native window is 262,144; a conversation past this
|
|
# ceiling is refused, never silently truncated. Clients declare 200192 (see
|
|
# the frank-large contextWindow in models.yml).
|
|
Environment=HALOGEN_CTX=200192
|
|
# The KV pool: 400,384 positions = 2 x 200,192 (the 2026-09-16 retune, down
|
|
# from 401,408). ~28 KiB per position including block scratch, so ~11.0 GiB
|
|
# held -- the 401,408 ledger entry measured exactly 11.0. The two slots still
|
|
# OVERSUBSCRIBE the pool for any real request, because admission reserves
|
|
# prompt + max_tokens: two full-window conversations each holding an output
|
|
# budget are pool-bound, and the second waits in arrival order for the first
|
|
# to release its region. Read the `kv pool:` lines on every boot;
|
|
# HALOGEN_KV_POOL_FIT lowers the pool itself if the budget refuses the
|
|
# configured one, and a green health check does not mean the pool you asked
|
|
# for is the pool you got.
|
|
Environment=HALOGEN_KV_POOL_POSITIONS=400384
|
|
# Concurrency cap: two, one per half of the pool. Each slot is 111.0 MiB of
|
|
# O(1) state, so this hands ~111 MiB back against the three set on 2026-09-12
|
|
# for the fleet size of the time.
|
|
Environment=HALOGEN_KV_SLOTS=2
|
|
# Sizing is budget arithmetic, all of it. ONE budget covers the KV pool, the
|
|
# per-call prefill arena and the vision tower together:
|
|
#
|
|
# budget = 121.2 host - 67.7 resident weights - reserve
|
|
# need = KV (~28 KiB/position) + arena + vision
|
|
# arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
|
|
#
|
|
# The arena term is linear in max_tok, and it reproduces five live restarts
|
|
# AND upstream's three published "on the device" figures to within 0.1 GiB —
|
|
# note those published figures are pool+arena totals, not pool alone, which is
|
|
# the misreading that first made a doubling look affordable here. Corners this
|
|
# box offered at a reserve of 24, at the old 262,144-position pool:
|
|
#
|
|
# 16384 -> need 28.6 vs 29.5 FITS, 0.9 spare (doubling, 8-9% prefill cost)
|
|
# 32768 -> need 37.0 vs 29.5 POOL CLAMPED to 262144, deploy still green
|
|
# 65536 -> need 46.5 vs 29.5 OOM, will not boot; unit still reads ACTIVE
|
|
#
|
|
# LOWERED 32768 -> 24576 on 2026-09-16 for the frank-small co-tenant on this
|
|
# same host. The arena's variable term goes 8.40 x 2 -> 8.40 x 1.5, handing
|
|
# ~4.2 GiB back to the host at roughly a 4-5 % prefill cost -- the same lever
|
|
# at 16,384 costs 8-9 %. The vendor's own notes name 24,576 as the largest
|
|
# arena that allocated with headroom to spare, so this stays inside the range
|
|
# it has tested.
|
|
#
|
|
# The arena is a PER-CALL PREFILL CHUNK, not a prompt ceiling: the
|
|
# 200,192-token conversation is prefilled in arena-sized pieces, so a smaller
|
|
# arena means more pieces and a slower prefill, not a smaller conversation.
|
|
#
|
|
# THE POOL, ARENA AND RESERVE ARE A MATCHED SET, NOT independent tunables --
|
|
# the fit check's printed ledger on each boot is the arbiter, not these
|
|
# comments.
|
|
Environment=HALOGEN_MAX_TOK=24576
|
|
# RAM the fit check refuses to count on, watched over by the engine's own
|
|
# n-gram lookup table (47.7 GiB, read through the page cache and never held)
|
|
# and, since 2026-09-16, by frank-small on the same host.
|
|
#
|
|
# RAISED 15 -> 20 with the 2026-09-16 retune. This knob feeds the fit-check
|
|
# budget and is NOT an allocation: raising it cannot hand memory back to the
|
|
# host, because the actual footprint is weights + KV + arena + vision, none of
|
|
# which it touches — it can only take the pool away. What it does is keep the
|
|
# fit check honest at the smaller arena. With the arena at 24,576 and no
|
|
# vision tower, the engine's printed need falls to roughly 25 GiB against a
|
|
# budget of 121.2 - 67.7 - 20 = 33.5, leaving ~8 GiB of margin, so the pool is
|
|
# not clamped.
|
|
#
|
|
# The 2026-09-13 profile for comparison: 103.5 GiB held and "7.1 GiB left for
|
|
# everything else" (310 contiguous 2 MiB blocks) at the doubled 524,288 pool,
|
|
# which is why the pool came down; at 401,408 the same ledger read 9.1 GiB /
|
|
# 1396 blocks. The 09-16 retune gives back the vision tower (1.97) and ~4.2 of
|
|
# arena on top of that for the co-tenant, so the expected figure at this boot
|
|
# is ~15 GiB left. Read the ledger; do not trust this arithmetic.
|
|
#
|
|
# To free host memory, change the POOL or the ARENA, and the reserve then
|
|
# becomes free to rise: HALOGEN_KV_POOL_POSITIONS=262144 hands back 3.6 GiB,
|
|
# HALOGEN_MAX_TOK=16384 hands back another 4.2. Both are one-line edits the
|
|
# fit check confirms on the next boot.
|
|
Environment=HALOGEN_HOST_RESERVE_GIB=20
|
|
# Verbose engine logging: the vendor's first-line evidence for load,
|
|
# allocation and admission questions. Shipping it on means an incident report
|
|
# already has the full startup account instead of needing a restart-into-debug.
|
|
# HALOGEN_DMALLOC_LOG is deliberately NOT set — it logs every allocation
|
|
# over 64 MB, which is steady-state noise here; raise it only for a live
|
|
# OOM hunt.
|
|
Environment=HALOGEN_VERBOSE=1
|
|
# Required by the image: the engine maps the checkpoint and registers it with
|
|
# the GPU in place, which needs host IPC and an unconfined seccomp profile.
|
|
PodmanArgs=--ipc=host --security-opt seccomp=unconfined
|
|
Ulimit=memlock=-1:-1
|
|
# The image ships its own HEALTHCHECK; this pins it. /health asks the ENGINE
|
|
# for a PONG before answering, so 200 means "ready to serve", not "process
|
|
# alive". During the first-start download it stays unhealthy by design.
|
|
HealthCmd=halogen-healthcheck api
|
|
HealthInterval=30s
|
|
HealthTimeout=10s
|
|
HealthRetries=3
|
|
HealthStartPeriod=300s
|
|
Pull=never
|
|
|
|
[Service]
|
|
# The first start reads ~68 GiB off disk and pins it before serving.
|
|
TimeoutStartSec=900
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=default.target
|