# Quadlet unit: frank-large (rootless, boot-persistent).
#
# Install path on dankclaw: ~/.config/containers/systemd/frank-large.container
# Unit name: frank-large.service (container name: frank-large)
#
# The large inference service. The halogen-flash-server runtime (since
# 2026-09-11, restored 2026-09-16 after a two-day ik_llama.cpp hybrid
# interlude) serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next,
# w4b .hgn checkpoint + quality overlay sidecar). It replaced the llama.cpp /
# kingjones30-ROCmFPX STRIX deployment because the engine is ~4x faster on
# prefill (the dominant cost of any real prompt) on this exact SKU. The RTX
# 3090 it briefly shared with frank-large is now frank-small's card.
#
# SHAPE. The image runs both halves in one container: the engine on container
# loopback (:8730, UNAUTHENTICATED — never publish it) and the OpenAI
# front-end on :8731, published here as host 7770 so the fleet's clients,
# services.json probe and docs keep the port they have always used. Same for
# HALOGEN_MODEL_ID=frank-large: it is the id at /v1/models, in /health and in
# every response, so no client had to be renamed. Halogen never rejects a
# request for naming a different model, it just serves it.
#
# DEVICES. gfx1151 through the amdgpu/KFD stack: /dev/kfd plus the DRM render
# node. Both are 0666 on this host, so the rootless user namespace can open
# them (id dank is also in render/video). seccomp=unconfined and --ipc=host are
# the image's documented requirements; memlock must be unlimited.
#
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
# runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
#
# MEMORY. Weights are pinned (HALOGEN_FLASH_PIN_TRUNK default 1) — that is the
# configuration the published decode numbers were measured with, and `free`,
# MemAvailable and every monitor built on them will overstate free memory by
# the size of the model (~68 GiB) because the kernel counts the locked weights
# as reclaimable page cache. Halogen prints the corrected figure at startup.
# The pinned trunk means this container holds most of the host; a co-tenant
# that then takes the remainder can stall both for minutes at full CPU with no
# disk activity and no output. That is documented behaviour, not a crash, and
# the escape hatch is HALOGEN_FLASH_PIN_TRUNK=0 (at several times the decode
# cost). Sizing is chosen for a SHARED 121 GiB host, and HALOGEN_KV_POOL_FIT
# stays on as the safety net: if the machine refuses the configured pool the
# engine lowers it itself and prints what it settled on. Read that ledger on
# every restart — it is the ground truth for sizing, not this comment.
#
# POOL RETUNED 2026-09-16: 400,384 positions -- exactly two 200,192-token
# conversations -- beside a per-conversation ceiling of 200,192 and two slots.
# It replaces the 401,408 the 2026-09-13 trim left behind, which held one full
# native-262k conversation beside a ~139k one, or three at 131k.
#
# MEASURED AT 401,408 (2026-09-13 deploy ledger): 68.0 weights + 11.0 KV +
# 21.0 working = 100.0 GiB held, 9.1 GiB / 1396 contiguous 2 MiB blocks left,
# engine need printed ~31.5 against the modelled ~33.4, and NO stall-warning
# NOTE. Fit-check budget here: 121.2 - 67.7 - 20 = 33.5. That boot is the
# baseline the 09-16 retune is measured against: the vision tower (1.97 GiB)
# and roughly 4.2 GiB of arena come back to the host, which is what pays for
# the co-tenant below.
#
# CO-TENANT since 2026-09-16: this host also serves frank-small on the RTX
# 3090 (llama.cpp, ~12.2 GiB of weights that load through the page cache and
# then live in VRAM, plus a CUDA context). It is the first real co-tenant this
# service has had, which is why the arena was halved and the reserve raised.
#
# BOOT NOTE: on the first request after a (re)start the engine can go silent
# for ~45-60 s ("has not answered PING") while it first-touches the freshly
# reserved pool; /health reads engine_unresponsive during it and recovers by
# itself. The quadlet health check's HealthStartPeriod=300s covers it; do not
# restart the unit over it.
#
# WEIGHTS. HALOGEN_DOWNLOAD fetches the whole weights repo on FIRST START ONLY
# (the entrypoint checks whether the checkpoint is absent) into the bind mount
# below, which must therefore be read-WRITE. huggingface_hub resumes partial
# files natively, and the quadlet runs the container with --replace --rm, so a
# restart mid-fetch continues rather than starting over. Roughly 127 GB: the
# 115.55 GiB checkpoint, the 2.31 GiB quality overlay, the 0.84 GiB vision
# tower and the tokenizer. The repo is public and ungated, so no token is
# needed; later starts open no outbound connections at all.
#
# VISION. Deliberately NOT set since 2026-09-16. The tower is a 1.97 GiB
# charge against the same device budget as the pool and the arena, and the
# fleet's image path moved to frank-small (llama.cpp on the RTX 3090, loaded
# with --mmproj mmproj-Qwen3.8-27B-BF16). This server refuses image content
# parts, and models.yml declares frank-large text-only to match.
#
# CLIENT CONTRACT (see serverAppConfigs-docs/LLM/halogen-flash-server.md).
# There are no server-side sampler defaults: a request that sends no
# temperature decodes GREEDY. The per-request budget defaults to 8,192 tokens
# and REASONING COUNTS AGAINST IT, so a budget that runs out mid-thought
# returns finish_reason "length" with empty content and the partial reasoning
# in reasoning_content. The cap is 65,536; above it a request is a 400, never a
# silent truncation.
#
# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet
# path cannot resolve registry-prefixed refs from local storage at create
# time. deploy-remote.sh pulls the ghcr.io ref and re-tags it before the
# restart. Pull=never keeps boots off the registry.
[Unit]
Description=frank-large — halogen-flash-server, Qwen3.8-Flash-Next (w4b) on gfx1151
After=network-online.target

[Container]
ContainerName=frank-large
Image=localhost/peonist-ai/halogen-flash-server:0.5.8
# Published port stays 7770: Hermes, open-webui, the harness and the homepage
# probe all point here. 8730 (the engine) is deliberately not published.
PublishPort=7770:8731
AddDevice=/dev/kfd
AddDevice=/dev/dri
# Read-write: the first start downloads into it. Contains only weights.
Volume=/home/dank/models/halogen:/models
# Fetch the weights on first start only (absent checkpoint gates it).
Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
# Fleet identity: the id at /v1/models, /health and every response body.
Environment=HALOGEN_MODEL_ID=frank-large
# Per-conversation ceiling: exactly half the pool, so two conversations of the
# full window are always resident. 200,192 rather than a round 200,000 is
# pool/2. The model's native window is 262,144; a conversation past this
# ceiling is refused, never silently truncated. Clients declare 200192 (see
# the frank-large contextWindow in models.yml).
Environment=HALOGEN_CTX=200192
# The KV pool: 400,384 positions = 2 x 200,192 (the 2026-09-16 retune, down
# from 401,408). ~28 KiB per position including block scratch, so ~11.0 GiB
# held -- the 401,408 ledger entry measured exactly 11.0. The two slots still
# OVERSUBSCRIBE the pool for any real request, because admission reserves
# prompt + max_tokens: two full-window conversations each holding an output
# budget are pool-bound, and the second waits in arrival order for the first
# to release its region. Read the `kv pool:` lines on every boot;
# HALOGEN_KV_POOL_FIT lowers the pool itself if the budget refuses the
# configured one, and a green health check does not mean the pool you asked
# for is the pool you got.
Environment=HALOGEN_KV_POOL_POSITIONS=400384
# Concurrency cap: two, one per half of the pool. Each slot is 111.0 MiB of
# O(1) state, so this hands ~111 MiB back against the three set on 2026-09-12
# for the fleet size of the time.
Environment=HALOGEN_KV_SLOTS=2
# Sizing is budget arithmetic, all of it. ONE budget covers the KV pool, the
# per-call prefill arena and the vision tower together:
#
#   budget = 121.2 host - 67.7 resident weights - reserve
#   need   = KV (~28 KiB/position) + arena + vision
#   arena  = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
#
# The arena term is linear in max_tok, and it reproduces five live restarts
# AND upstream's three published "on the device" figures to within 0.1 GiB —
# note those published figures are pool+arena totals, not pool alone, which is
# the misreading that first made a doubling look affordable here. Corners this
# box offered at a reserve of 24, at the old 262,144-position pool:
#
#   16384 -> need 28.6 vs 29.5  FITS, 0.9 spare   (doubling, 8-9% prefill cost)
#   32768 -> need 37.0 vs 29.5  POOL CLAMPED to 262144, deploy still green
#   65536 -> need 46.5 vs 29.5  OOM, will not boot; unit still reads ACTIVE
#
# LOWERED 32768 -> 24576 on 2026-09-16 for the frank-small co-tenant on this
# same host. The arena's variable term goes 8.40 x 2 -> 8.40 x 1.5, handing
# ~4.2 GiB back to the host at roughly a 4-5 % prefill cost -- the same lever
# at 16,384 costs 8-9 %. The vendor's own notes name 24,576 as the largest
# arena that allocated with headroom to spare, so this stays inside the range
# it has tested.
#
# The arena is a PER-CALL PREFILL CHUNK, not a prompt ceiling: the
# 200,192-token conversation is prefilled in arena-sized pieces, so a smaller
# arena means more pieces and a slower prefill, not a smaller conversation.
#
# THE POOL, ARENA AND RESERVE ARE A MATCHED SET, NOT independent tunables --
# the fit check's printed ledger on each boot is the arbiter, not these
# comments.
Environment=HALOGEN_MAX_TOK=24576
# RAM the fit check refuses to count on, watched over by the engine's own
# n-gram lookup table (47.7 GiB, read through the page cache and never held)
# and, since 2026-09-16, by frank-small on the same host.
#
# RAISED 15 -> 20 with the 2026-09-16 retune. This knob feeds the fit-check
# budget and is NOT an allocation: raising it cannot hand memory back to the
# host, because the actual footprint is weights + KV + arena + vision, none of
# which it touches — it can only take the pool away. What it does is keep the
# fit check honest at the smaller arena. With the arena at 24,576 and no
# vision tower, the engine's printed need falls to roughly 25 GiB against a
# budget of 121.2 - 67.7 - 20 = 33.5, leaving ~8 GiB of margin, so the pool is
# not clamped.
#
# The 2026-09-13 profile for comparison: 103.5 GiB held and "7.1 GiB left for
# everything else" (310 contiguous 2 MiB blocks) at the doubled 524,288 pool,
# which is why the pool came down; at 401,408 the same ledger read 9.1 GiB /
# 1396 blocks. The 09-16 retune gives back the vision tower (1.97) and ~4.2 of
# arena on top of that for the co-tenant, so the expected figure at this boot
# is ~15 GiB left. Read the ledger; do not trust this arithmetic.
#
# To free host memory, change the POOL or the ARENA, and the reserve then
# becomes free to rise: HALOGEN_KV_POOL_POSITIONS=262144 hands back 3.6 GiB,
# HALOGEN_MAX_TOK=16384 hands back another 4.2. Both are one-line edits the
# fit check confirms on the next boot.
Environment=HALOGEN_HOST_RESERVE_GIB=20
# Verbose engine logging: the vendor's first-line evidence for load,
# allocation and admission questions. Shipping it on means an incident report
# already has the full startup account instead of needing a restart-into-debug.
# HALOGEN_DMALLOC_LOG is deliberately NOT set — it logs every allocation
# over 64 MB, which is steady-state noise here; raise it only for a live
# OOM hunt.
Environment=HALOGEN_VERBOSE=1
# Required by the image: the engine maps the checkpoint and registers it with
# the GPU in place, which needs host IPC and an unconfined seccomp profile.
PodmanArgs=--ipc=host --security-opt seccomp=unconfined
Ulimit=memlock=-1:-1
# The image ships its own HEALTHCHECK; this pins it. /health asks the ENGINE
# for a PONG before answering, so 200 means "ready to serve", not "process
# alive". During the first-start download it stays unhealthy by design.
HealthCmd=halogen-healthcheck api
HealthInterval=30s
HealthTimeout=10s
HealthRetries=3
HealthStartPeriod=300s
Pull=never

[Service]
# The first start reads ~68 GiB off disk and pins it before serving.
TimeoutStartSec=900
Restart=on-failure
RestartSec=10

[Install]
WantedBy=default.target