dank/serverAppConfigs
1
0
Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-large: pin the prompt cache (16 GB) and capture the engine ledger on disk

Browse Source 
Auto cache-sizing is OFF on this box: the boot ledger refused it at
36.9 GB (2 x full-entry) floor vs 35.7 GB free, so every deep agent
request re-prefills its whole 79-81k context (84.9-179.8 s measured on
serve_api lines, 2026-10-06) for ~500-650 new tokens per turn. Pin
HALOGEN_CACHE_MB=16384 (>= one full-slot entry at ~70 KB/token) — the
documented override; the per-store reserve re-check keeps the host safe.

Also: the serve script's 'podman run -d > LOG' captured only the
container ID — the runbook's boot-ledger grep read a one-line hex file
and the ledger existed nowhere on disk. Foreground 'podman run' makes
the ledger land in the log file as documented; exit-code and ExecStop
semantics are unchanged. Test seam updated for both.
This commit is contained in:
Frank
2026-10-06 00:26:28 -07:00
parent 8343397584
commit ba67ca5c9e
3 changed files with 47 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/frank_large_unit.bats
+17 -10
View File
@@ -74,6 +74,7 @@ good_env() {
"HALOGEN_SLOT_CTX=200700"
"HALOGEN_KV_SLOTS=1"
"HALOGEN_DRAFTER=2"
"HALOGEN_CACHE_MB=16384"
"STUB_DIGESTS=localhost/x ghcr.io/peonist-ai/halogen@sha256:cafe"
)
}
@@ -115,17 +116,20 @@ good_env() {
[[ "$output" == *"no gfx1151 to serve on"* ]]
}
@test "the KV budget fits unified memory with margin" {
@test "the KV pool plus pinned cache fit unified memory with margin" {
# Pool arithmetic is the engine's documented product: slots x
# slot_ctx x 64 KiB. Against it: 33.4 GiB of mapped weights, the
# engine working set + auto prompt cache (~25 GiB of headroom
# allowed), host baseline and frank-small co-tenancy. The box has
# 121 GiB unified; the 2026-09-16 halogen era ran the same engine
# class here on a LARGER pool with less co-tenancy slack.
# slot_ctx x 64 KiB. The prompt cache is PINNED (HALOGEN_CACHE_MB,
# ~70 KB/token per entry — 16384 MB >= one full-slot entry). Against
# both: 33.4 GiB of mapped weights, ~20 GiB engine working set, host
# baseline and frank-small co-tenancy. The box has 121 GiB unified;
# the 2026-09-16 halogen era ran the same engine class here on a
# LARGER pool with less co-tenancy slack.
ctx="$(sed -nE 's/^HALOGEN_SLOT_CTX=([0-9]+)$/\1/p' "$ENV_FILE")"
slots="$(sed -nE 's/^HALOGEN_KV_SLOTS=([0-9]+)$/\1/p' "$ENV_FILE")"
cache_mb="$(sed -nE 's/^HALOGEN_CACHE_MB=([0-9]+)$/\1/p' "$ENV_FILE")"
pool_gib=$(( slots * ctx * 65536 / 1073741824 ))
total_gib=$(( 34 + pool_gib + 25 + 15 ))
cache_gib=$(( ${cache_mb:-0} / 1024 ))
total_gib=$(( 34 + pool_gib + cache_gib + 20 + 15 ))
[ "$total_gib" -le 121 ]
}
@@ -192,9 +196,12 @@ $(cat "$TEST_TMP/serve.log")"
[[ "$output" == *"--device $TEST_TMP/qwen.hgn"* ]]
[[ "$output" == *"$TEST_TMP/tokenizer:/tokenizer:ro"* ]]
[[ "$output" == *"ghcr.io/peonist-ai/halogen@sha256:cafe"* ]]
# The unit's restart policy is real because the script becomes
# `podman wait`: the container's exit code is the service's exit.
[[ "$output" == *"PODMAN wait frank-large-halogen"* ]]
# Foreground run, never -d: the engine ledger must land in the log
# file (`podman run -d` captured only the container ID and the
# runbook's ledger grep read a one-line hex file, found 2026-10-06).
! grep -qE '(^|[[:space:]])-d([[:space:]]|$)' "$TEST_TMP/serve.log"
! grep -q 'PODMAN wait' "$TEST_TMP/serve.log"
[[ "$output" == *"-e HALOGEN_CACHE_MB=16384"* ]]
}
@test "the shipped env pins the fleet's window and shape" {