Frank
65d1be06ac
Deployed HALOGEN_MAX_TOK=32768 by operator call (upstream default, full prefill speed) and confirmed the prediction from the ledger: kv pool: 524288 positions need ~35.0 GiB ... LOWERING THE POOL TO 262144 (~27.8 GiB) memory: 68.0 weights + 7.2 KV pool + 21.1 working = 96.3 GiB in all so the effective pool is 262144 again and the doubling this task set out to do is not in force. Healthy, completion verified. Left the pool requesting 524288 deliberately: with KV_POOL_FIT on that is a grow-if-RAM-frees-up declaration, not a mistake. Two corrections to what I wrote earlier today: 1. The engine's sizing block, and its "WARNING even 262144 ... host RAM cannot spare it" line, print ONLY when the fit check had to clamp something. The 09-11 baseline boot prints no kv pool: sizing lines at all. So the absence of a warning in old logs was never evidence of headroom. 2. The budget needs a vision term. Actual need is pool + arena + 1.97 GiB of vision tower, not pool + a 1.5 GiB nominal margin. That makes the ceiling for a doubled pool max_tok <= 18,130 rather than 19,115, and it means the 32768/262144 corner this box has run in since 09-11 is ~0.3 GiB OVER its own stated budget -- and has been the whole time. It boots and serves with 12-14 GiB genuinely free because the 24 GiB reserve is generous. A soft floor, not a wall. The arena model also reproduces upstream's three published "on the device" figures exactly (35 / 42 / 41 GiB), confirming they are pool+arena totals at the default max_tok and not pool-only -- which is the misreading that made the doubling look affordable in my first estimate.
169 lines
9.2 KiB
INI
169 lines
9.2 KiB
INI
# Quadlet unit: frank-large (rootless, boot-persistent).
|
|
#
|
|
# Install path on dankclaw: ~/.config/containers/systemd/frank-large.container
|
|
# Unit name: frank-large.service (container name: frank-large)
|
|
#
|
|
# The large inference service, since 2026-09-11 the halogen-flash-server
|
|
# runtime serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next,
|
|
# w4b .hgn checkpoint + quality overlay sidecar). It replaced the llama.cpp /
|
|
# kingjones30-ROCmFPX STRIX deployment because the engine is ~4x faster on
|
|
# prefill (the dominant cost of any real prompt) on this exact SKU.
|
|
#
|
|
# SHAPE. The image runs both halves in one container: the engine on container
|
|
# loopback (:8730, UNAUTHENTICATED — never publish it) and the OpenAI
|
|
# front-end on :8731, published here as host 7770 so the fleet's clients,
|
|
# services.json probe and docs keep the port they have always used. Same for
|
|
# HALOGEN_MODEL_ID=frank-large: it is the id at /v1/models, in /health and in
|
|
# every response, so no client had to be renamed. Halogen never rejects a
|
|
# request for naming a different model, it just serves it.
|
|
#
|
|
# DEVICES. gfx1151 through the amdgpu/KFD stack: /dev/kfd plus the DRM render
|
|
# node. Both are 0666 on this host, so the rootless user namespace can open
|
|
# them (id dank is also in render/video). seccomp=unconfined and --ipc=host are
|
|
# the image's documented requirements; memlock must be unlimited.
|
|
#
|
|
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
|
|
# runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
|
#
|
|
# MEMORY. Weights are pinned (HALOGEN_FLASH_PIN_TRUNK default 1) — that is the
|
|
# configuration the published decode numbers were measured with, and `free`,
|
|
# MemAvailable and every monitor built on them will overstate free memory by
|
|
# the size of the model (~68 GiB) because the kernel counts the locked weights
|
|
# as reclaimable page cache. Halogen prints the corrected figure at startup.
|
|
# The pinned trunk means this container holds most of the host; a co-tenant
|
|
# that then takes the remainder can stall both for minutes at full CPU with no
|
|
# disk activity and no output. That is documented behaviour, not a crash, and
|
|
# the escape hatch is HALOGEN_FLASH_PIN_TRUNK=0 (at several times the decode
|
|
# cost). Sizing is chosen for a SHARED 121 GiB host, and HALOGEN_KV_POOL_FIT
|
|
# stays on as the safety net: if the machine refuses the configured pool the
|
|
# engine lowers it itself and prints what it settled on. Read that ledger on
|
|
# every restart — it is the ground truth for sizing, not this comment.
|
|
#
|
|
# DOUBLED 2026-09-12: pool 524,288 positions (~14.4 GiB of KV at the
|
|
# measured 28 KiB/position). This did NOT fit on the first attempt. The
|
|
# engine's own ledger for the prior 262,144 config was 68.0 weights + 7.2
|
|
# pool + 21.1 working = 96.3 GiB held, 12.1 GiB left on the host, and its
|
|
# pre-flight arithmetic is: 121 GiB - 67.7 resident weights - 24.0 reserve
|
|
# = 29.5 GiB of device budget, which has to cover the pool AND the prefill
|
|
# arena together. 524,288 with the 32,768-token arena needs 35.0 GiB, so
|
|
# HALOGEN_KV_POOL_FIT lowered the pool straight back to 262,144 and the
|
|
# deploy still reported healthy. The doubling only becomes real together
|
|
# with HALOGEN_MAX_TOK=16384 further down, which halves the arena and brings
|
|
# the need to ~24.9 GiB. Read the `kv pool:` lines after every restart —
|
|
# a green health check does NOT mean the pool you asked for is the pool you got.
|
|
#
|
|
# WEIGHTS. HALOGEN_DOWNLOAD fetches the whole weights repo on FIRST START ONLY
|
|
# (the entrypoint checks whether the checkpoint is absent) into the bind mount
|
|
# below, which must therefore be read-WRITE. huggingface_hub resumes partial
|
|
# files natively, and the quadlet runs the container with --replace --rm, so a
|
|
# restart mid-fetch continues rather than starting over. Roughly 127 GB: the
|
|
# 115.55 GiB checkpoint, the 2.31 GiB quality overlay, the 0.84 GiB vision
|
|
# tower and the tokenizer. The repo is public and ungated, so no token is
|
|
# needed; later starts open no outbound connections at all.
|
|
#
|
|
# VISION. HALOGEN_VISION_TOWER=1 finds the sidecar beside the checkpoint. This
|
|
# is the fleet's only image input since the FLM NPU endpoint was retired on
|
|
# 2026-09-11; audio input has no replacement.
|
|
#
|
|
# CLIENT CONTRACT (see serverAppConfigs-docs/LLM/halogen-flash-server.md).
|
|
# There are no server-side sampler defaults: a request that sends no
|
|
# temperature decodes GREEDY. The per-request budget defaults to 8,192 tokens
|
|
# and REASONING COUNTS AGAINST IT, so a budget that runs out mid-thought
|
|
# returns finish_reason "length" with empty content and the partial reasoning
|
|
# in reasoning_content. The cap is 65,536; above it a request is a 400, never a
|
|
# silent truncation.
|
|
#
|
|
# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet
|
|
# path cannot resolve registry-prefixed refs from local storage at create
|
|
# time. deploy-remote.sh pulls the ghcr.io ref and re-tags it before the
|
|
# restart. Pull=never keeps boots off the registry.
|
|
[Unit]
|
|
Description=frank-large — halogen-flash-server, Qwen3.8-Flash-Next (w4b) on gfx1151
|
|
After=network-online.target
|
|
|
|
[Container]
|
|
ContainerName=frank-large
|
|
Image=localhost/peonist-ai/halogen-flash-server:0.5.8
|
|
# Published port stays 7770: Hermes, open-webui, the harness and the homepage
|
|
# probe all point here. 8730 (the engine) is deliberately not published.
|
|
PublishPort=7770:8731
|
|
AddDevice=/dev/kfd
|
|
AddDevice=/dev/dri
|
|
# Read-write: the first start downloads into it. Contains only weights.
|
|
Volume=/home/dank/models/halogen:/models
|
|
# Fetch the weights on first start only (absent checkpoint gates it).
|
|
Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
|
|
# Fleet identity: the id at /v1/models, /health and every response body.
|
|
Environment=HALOGEN_MODEL_ID=frank-large
|
|
# Native context of the model. Matches what both clients declare.
|
|
Environment=HALOGEN_CTX=262144
|
|
# Requested 2026-09-12 as double the 262144 this had run since 09-11, to get
|
|
# two full-length conversations resident instead of one. NOTE THE ASYMMETRY
|
|
# BELOW: with HALOGEN_MAX_TOK at 32768 (current) the fit check clamps this to
|
|
# 262144 at every boot, so the EFFECTIVE pool is 262144 and only one
|
|
# full-length conversation stays resident. It is left at 524288 deliberately:
|
|
# it is a "grow if the RAM frees up" declaration, and HALOGEN_KV_POOL_FIT
|
|
# raises nothing but honours it the moment the arena stops eating the budget.
|
|
# Set HALOGEN_MAX_TOK=16384 to actually get the doubled pool today.
|
|
Environment=HALOGEN_KV_POOL_POSITIONS=524288
|
|
Environment=HALOGEN_KV_SLOTS=4
|
|
# The device budget is ONE budget, covering the KV pool, the prefill arena and
|
|
# the vision tower:
|
|
#
|
|
# budget = 121 host - 67.7 resident weights - 24.0 reserve = 29.5 GiB
|
|
# need = KV(~28 KiB/position) + arena + vision(1.97 measured)
|
|
# arena = 3.80 GiB + 8.40 GiB x (HALOGEN_MAX_TOK / 16384)
|
|
#
|
|
# That arena term is linear in max_tok and was checked against five restarts
|
|
# and against upstream's three published "on the device" figures, all to
|
|
# 0.1 GiB. Measured on this host:
|
|
# max_tok 16384 -> arena 12.2, pool 524288 total 28.6 -> FITS, 0.9 spare
|
|
# max_tok 32768 -> arena 20.6, pool 524288 total 37.0 -> clamps to 262144
|
|
# max_tok 32768 -> pool floor 262144 total 29.8 -> 0.3 OVER budget, boots
|
|
# max_tok 65536 -> arena 37.3, even the floor needs 46.5 -> OOM, no boot
|
|
# So 16384 is the ceiling for a doubled pool, not a conservative setting. The
|
|
# 0.3 over-budget at the 32768/262144 corner is where this box has been ever
|
|
# since it was deployed on 09-11; the 24 GiB reserve is generous, so it boots
|
|
# and serves with 12-14 GiB genuinely left. It is a soft floor, not a wall.
|
|
#
|
|
# Do NOT raise max_tok past 16384 while asking for the doubled pool: 65536 did
|
|
# not shrink the pool, it stopped the server booting — and the unit still reads
|
|
# active with the container "Up (starting)", so the health check waits out
|
|
# LLM_WAIT rather than failing fast.
|
|
Environment=HALOGEN_MAX_TOK=32768
|
|
# RAM left for the n-gram table the engine reads through the page cache.
|
|
# Above the vendor's 20 because a dozen containers share this host; too low
|
|
# and long prompts crawl with the disk busy.
|
|
Environment=HALOGEN_HOST_RESERVE_GIB=24
|
|
# Image input (the sidecar ships inside the weights repo).
|
|
Environment=HALOGEN_VISION_TOWER=1
|
|
# Verbose engine logging: the vendor's first-line evidence for load,
|
|
# allocation and admission questions. Shipping it on means an incident report
|
|
# already has the full startup account instead of needing a restart-into-debug.
|
|
# HALOGEN_DMALLOC_LOG is deliberately NOT set — it logs every allocation
|
|
# over 64 MB, which is steady-state noise here; raise it only for a live
|
|
# OOM hunt.
|
|
Environment=HALOGEN_VERBOSE=1
|
|
# Required by the image: the engine maps the checkpoint and registers it with
|
|
# the GPU in place, which needs host IPC and an unconfined seccomp profile.
|
|
PodmanArgs=--ipc=host --security-opt seccomp=unconfined
|
|
Ulimit=memlock=-1:-1
|
|
# The image ships its own HEALTHCHECK; this pins it. /health asks the ENGINE
|
|
# for a PONG before answering, so 200 means "ready to serve", not "process
|
|
# alive". During the first-start download it stays unhealthy by design.
|
|
HealthCmd=halogen-healthcheck api
|
|
HealthInterval=30s
|
|
HealthTimeout=10s
|
|
HealthRetries=3
|
|
HealthStartPeriod=300s
|
|
Pull=never
|
|
|
|
[Service]
|
|
# The first start reads ~68 GiB off disk and pins it before serving.
|
|
TimeoutStartSec=900
|
|
Restart=on-failure
|
|
RestartSec=10
|
|
|
|
[Install]
|
|
WantedBy=default.target
|