feat(llm): swap frank-large to halogen-flash-server, retire frank-npu

frank-large moves from the kingjones30/ROCmFPX llama.cpp build to peonist-ai/halogen-flash-server 0.5.8 serving Qwen3.8-Flash-Next (w4b .hgn 115.55 GiB + quality overlay). Upstream measures this engine at roughly 4x the prefill throughput of the llama.cpp family on the same gfx1151 silicon, and prefill dominates any prompt with real context. Shape: a rootless Quadlet container (dankclaw/frank-large/quadlet/), engine on container loopback :8730 (never published), OpenAI front-end :8731 published as host 7770, HALOGEN_MODEL_ID=frank-large so no client renamed anything. Context 262144, KV pool 262144 positions (~7.2 GiB, conservative for a shared 121 GiB host), 4 slots, host reserve 24, vision tower on, weights pinned, all other flags at vendor defaults. Weights are fetched by HALOGEN_DOWNLOAD on first start into a writable /home/dank/models/halogen. The llama.cpp era (unit, env, serve script) is staged in .AGENTTRASH/ rather than deleted; the STRIX payload and the ROCmFPX build stay on disk as the rollback target. deploy_llm_large becomes a user-quadlet handler: it retires the old ROOT unit (moving its checkout symlink into the home trash, never rm) and gates the first start on the download rather than reporting a failure for a fetch that is working exactly as designed. frank-npu is retired in the same change. The kernel cmdline moves from iommu=pt to amd_iommu=off for the 13-16% of prefill upstream measured, and that takes the XDNA2 NPU with it: without PASID/SVA the driver cannot open /dev/accel/accel0 (err=-19). Its five dependent routes are repointed (Hermes vision -> frank-large, Hermes title generation -> frank-small, harness vision -> frank-large, tiny -> frank-small, task -> frank-large:medium) and its provider blocks are removed from both the harness config and Hermes's live config. Audio input leaves the fleet with no replacement; image input survives on halogen's vision tower. Docs: new halogen-flash-server runbook, new IOMMU decision record, and frank-npu rewritten as a retirement record. AGENTS.md topology, runtime sections, benchmarking policy (frank-large is a user unit now) and the harness model-config section are corrected.
2026-09-11 14:48:28 -07:00
1 parent 184df9aa0d
commit 301914b1c0
@@ -0,0 +1,121 @@
+# Quadlet unit: frank-large (rootless, boot-persistent).
+#
+# Install path on dankclaw: ~/.config/containers/systemd/frank-large.container
+# Unit name: frank-large.service (container name: frank-large)
+#
+# The large inference service, since 2026-09-11 the halogen-flash-server
+# runtime serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next,
+# w4b .hgn checkpoint + quality overlay sidecar). It replaced the llama.cpp /
+# kingjones30-ROCmFPX STRIX deployment because the engine is ~4x faster on
+# prefill (the dominant cost of any real prompt) on this exact SKU.
+#
+# SHAPE. The image runs both halves in one container: the engine on container
+# loopback (:8730, UNAUTHENTICATED — never publish it) and the OpenAI
+# front-end on :8731, published here as host 7770 so the fleet's clients,
+# services.json probe and docs keep the port they have always used. Same for
+# HALOGEN_MODEL_ID=frank-large: it is the id at /v1/models, in /health and in
+# every response, so no client had to be renamed. Halogen never rejects a
+# request for naming a different model, it just serves it.
+#
+# DEVICES. gfx1151 through the amdgpu/KFD stack: /dev/kfd plus the DRM render
+# node. Both are 0666 on this host, so the rootless user namespace can open
+# them (id dank is also in render/video). seccomp=unconfined and --ipc=host are
+# the image's documented requirements; memlock must be unlimited.
+#
+# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the HIP
+# runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
+#
+# MEMORY. Weights are pinned (HALOGEN_FLASH_PIN_TRUNK default 1) — that is the
+# configuration the published decode numbers were measured with, and `free`,
+# MemAvailable and every monitor built on them will overstate free memory by
+# the size of the model (~68 GiB) because the kernel counts the locked weights
+# as reclaimable page cache. Halogen prints the corrected figure at startup.
+# The pinned trunk means this container holds most of the host; a co-tenant
+# that then takes the remainder can stall both for minutes at full CPU with no
+# disk activity and no output. That is documented behaviour, not a crash, and
+# the escape hatch is HALOGEN_FLASH_PIN_TRUNK=0 (at several times the decode
+# cost). Sizing chosen 2026-09-11 for a SHARED 121 GiB host: KV pool 262,144
+# positions (~7.2 GiB) + the baked 32,768-token prefill arena (16.7 GiB) sits
+# well inside the device budget once the ~68 GiB of weights and the ~5 GiB of
+# co-tenant containers are accounted for. HALOGEN_KV_POOL_FIT stays on as the
+# safety net: if the machine refuses, the engine lowers the pool itself and
+# prints what it chose.
+#
+# WEIGHTS. HALOGEN_DOWNLOAD fetches the whole weights repo on FIRST START ONLY
+# (the entrypoint checks whether the checkpoint is absent) into the bind mount
+# below, which must therefore be read-WRITE. huggingface_hub resumes partial
+# files natively, and the quadlet runs the container with --replace --rm, so a
+# restart mid-fetch continues rather than starting over. Roughly 127 GB: the
+# 115.55 GiB checkpoint, the 2.31 GiB quality overlay, the 0.84 GiB vision
+# tower and the tokenizer. The repo is public and ungated, so no token is
+# needed; later starts open no outbound connections at all.
+#
+# VISION. HALOGEN_VISION_TOWER=1 finds the sidecar beside the checkpoint. This
+# is the fleet's only image input since the FLM NPU endpoint was retired on
+# 2026-09-11; audio input has no replacement.
+#
+# CLIENT CONTRACT (see serverAppConfigs-docs/LLM/halogen-flash-server.md).
+# There are no server-side sampler defaults: a request that sends no
+# temperature decodes GREEDY. The per-request budget defaults to 8,192 tokens
+# and REASONING COUNTS AGAINST IT, so a budget that runs out mid-thought
+# returns finish_reason "length" with empty content and the partial reasoning
+# in reasoning_content. The cap is 65,536; above it a request is a 400, never a
+# silent truncation.
+#
+# IMAGE. The unit runs the localhost/-aliased tag: this host's podman quadlet
+# path cannot resolve registry-prefixed refs from local storage at create
+# time. deploy-remote.sh pulls the ghcr.io ref and re-tags it before the
+# restart. Pull=never keeps boots off the registry.
+[Unit]
+Description=frank-large — halogen-flash-server, Qwen3.8-Flash-Next (w4b) on gfx1151
+After=network-online.target
+
+[Container]
+ContainerName=frank-large
+Image=localhost/peonist-ai/halogen-flash-server:0.5.8
+# Published port stays 7770: Hermes, open-webui, the harness and the homepage
+# probe all point here. 8730 (the engine) is deliberately not published.
+PublishPort=7770:8731
+AddDevice=/dev/kfd
+AddDevice=/dev/dri
+# Read-write: the first start downloads into it. Contains only weights.
+Volume=/home/dank/models/halogen:/models
+# Fetch the weights on first start only (absent checkpoint gates it).
+Environment=HALOGEN_DOWNLOAD=peonist-ai/halogen-qwen3.8-flash-next
+# Fleet identity: the id at /v1/models, /health and every response body.
+Environment=HALOGEN_MODEL_ID=frank-large
+# Native context of the model. Matches what both clients declare.
+Environment=HALOGEN_CTX=262144
+# Conservative pool for a shared host: one full-length conversation, or four
+# at 65k. Raising it to 524288 doubles the resident conversations for ~7 GiB
+# more device memory — do that only after reading the startup ledger.
+Environment=HALOGEN_KV_POOL_POSITIONS=262144
+Environment=HALOGEN_KV_SLOTS=4
+# RAM left for the n-gram table the engine reads through the page cache.
+# Above the vendor's 20 because a dozen containers share this host; too low
+# and long prompts crawl with the disk busy.
+Environment=HALOGEN_HOST_RESERVE_GIB=24
+# Image input (the sidecar ships inside the weights repo).
+Environment=HALOGEN_VISION_TOWER=1
+# Required by the image: the engine maps the checkpoint and registers it with
+# the GPU in place, which needs host IPC and an unconfined seccomp profile.
+PodmanArgs=--ipc=host --security-opt seccomp=unconfined
+Ulimit=memlock=-1:-1
+# The image ships its own HEALTHCHECK; this pins it. /health asks the ENGINE
+# for a PONG before answering, so 200 means "ready to serve", not "process
+# alive". During the first-start download it stays unhealthy by design.
+HealthCmd=halogen-healthcheck api
+HealthInterval=30s
+HealthTimeout=10s
+HealthRetries=3
+HealthStartPeriod=300s
+Pull=never
+
+[Service]
+# The first start reads ~68 GiB off disk and pins it before serving.
+TimeoutStartSec=900
+Restart=on-failure
+RestartSec=10
+
+[Install]
+WantedBy=default.target
@@ -25,23 +25,17 @@ providers:
    models:
      frank-small:
        context_length: 163840
-  dankclaw-npu:
-    api: http://192.168.0.143:7771/v1
-    discover_models: false
-    models:
-      gemma4-it:e4b:
-        context_length: 131072

 # Auxiliary routing from the live configuration. Named providers route via
 # their `api:` URL; `custom` entries route by their explicit `base_url`.
 auxiliary:
-  vision: {provider: dankclaw-npu, model: gemma4-it:e4b}
+  vision: {provider: dankclaw-large, model: frank-large}
  web_extract: {provider: dankserver-small, model: frank-small}
  compression: {provider: dankserver-small, model: frank-small}
  skills_hub: {provider: dankserver-small, model: frank-small}
  approval: {provider: dankserver-small, model: frank-small}
  mcp: {provider: dankserver-small, model: frank-small}
-  title_generation: {provider: dankclaw-npu, model: gemma4-it:e4b}
+  title_generation: {provider: dankserver-small, model: frank-small}
  memory_query_rewrite: {provider: custom, model: frank-small, base_url: http://192.168.0.235:7770/v1}
  tts_audio_tags: {provider: custom, model: frank-small, base_url: http://192.168.0.235:7770/v1}
  triage_specifier: {provider: dankclaw-large, model: frank-large}