frank-primary: shape E — --expert-precise-memory (arena 6,496->7,522 slots), GPU_MAX_HW_QUEUES=2, and a mounted /root/.cache
Measured by boot probe on the shape-D base (production stopped, candidate booted AS the service), plan numbers read off the engine's own log: runtime overhead 7.60 -> 5.25 GiB (precise gfx1201 TP2 registry) expert cache 14.87 -> 17.22 GiB slots 6,496 -> 7,522 (+1,026, +16%) swap region 13.7 GiB / 5,986 slots boot healthy in 420 s, no OOM '--expert-precise-memory': True also appears in non-default args, so a silent argparse drop is visible. The pool is NOT cut (option A) because the pool is not an arena lever: 3.15e9/rank keeps shape D's measured 1.99x concurrency, and a 2.2e9 arm probed in the same window (277,686 tokens, 1.38x) is recorded as the conservative fallback. Boot-cost bug found while diffing Frank's 3xR9700 recipe: the image ships no /root/.cache at all, our boot writes 1.1 GB of vLLM AOT/torch_compile artifacts (4,665 files) plus 229 MB of comgr JIT kernels into the ephemeral layer, and podman rm -f discards them — so every deploy/restart/reboot re-pays 'torch.compile took 181.20 s in total' (173.19 s of it one graph). Mounting FRANK_PRIMARY_CACHE_DIR at /root/.cache persists both and hides nothing. GPU_MAX_HW_QUEUES=2 added at Frank's direction (real HSA knob); not A/B measured. Retracted my own recipe-diff claim that cudagraph_capture_sizes needed pinning: capture is 3 PIECEWISE + 2 FULL graphs, 3 s, 0.31 GiB. Tests: 367/367.
This commit is contained in:
1 parent
812193aa63
commit
1ac4d48371
7 files changed
+314
-51
No files matched your search
@@ -119,10 +119,25 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
grep -q '^FRANK_PRIMARY_KV_BYTES=3150000000$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_MNBT=1024$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_KV_DTYPE=fp8$' "$ENV_FILE"
|
||||
# --expert-precise-memory is NOT set: it put the bf16 2 x 262144 boot
|
||||
# 0.13 GiB from the 31.86 GiB ceiling (arena 16.65 device-resident) and
|
||||
# crash-looped at graph capture. The serve script keeps the opt-in flag.
|
||||
! grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=' "$ENV_FILE"
|
||||
# --expert-precise-memory IS set (shape E, 2026-10-02). It was refused
|
||||
# while the 2 x 262144 bf16 boot sat 0.13 GiB under the 31.86 GiB ceiling
|
||||
# (arena 16.65 device-resident, NRestarts=3 at ~6.7 min each) — there it was
|
||||
# the last 2.35 GiB over the line, not the cause. Probed on the shape-D base
|
||||
# with production stopped: boots healthy in 420 s, plan reports runtime
|
||||
# overhead 5.25 GiB (not 7.60), expert cache 17.22 GiB / 7,522 slots
|
||||
# (+1,026 / +16% over 6,496). The pool is NOT cut with it (option A): the
|
||||
# pool is not an arena lever, so 3.15e9 keeps the measured 1.99x.
|
||||
grep -q '^FRANK_PRIMARY_PRECISE_MEMORY=1$' "$ENV_FILE"
|
||||
# GPU_MAX_HW_QUEUES caps HSA hw queues per GPU; real knob (the literal is in
|
||||
# libhsa-runtime64.so), passed inside the container as a bare name — the env
|
||||
# file keeps only the prefixed form.
|
||||
grep -q '^FRANK_PRIMARY_GPU_MAX_HW_QUEUES=2$' "$ENV_FILE"
|
||||
# The compile cache is a BOOT-COST fix: the image ships no /root/.cache at
|
||||
# all, our boot writes 1.1 GB of vLLM AOT artifacts (4,665 files) plus
|
||||
# 229 MB of comgr JIT kernels into the ephemeral layer, and every
|
||||
# `podman rm -f` throws them away — re-paying `torch.compile took 181.20 s`
|
||||
# (173.19 s of it one graph) on every deploy, restart and reboot.
|
||||
grep -q '^FRANK_PRIMARY_CACHE_DIR=/home/dank/.cache/frank-primary$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_CHUNKS=32$' "$ENV_FILE"
|
||||
grep -q '^FRANK_PRIMARY_GATHER_LANES=64$' "$ENV_FILE"
|
||||
! grep -q '^FRANK_PRIMARY_SPEC=' "$ENV_FILE"
|
||||
@@ -150,6 +165,13 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
||||
# moves the plan's `room` term and therefore the expert arena: ~1,025
|
||||
# more slots at 2.344 MiB each.
|
||||
grep -q -- '${FRANK_PRIMARY_PRECISE_MEMORY:+--expert-precise-memory}' "$SERVE"
|
||||
# Shape E host/CPU wiring: the cache dir is created before it is mounted
|
||||
# (a cold boot must have a target) and the mount lands on /root/.cache,
|
||||
# whose vllm/ child is the path VLLM_CACHE_ROOT defaults to — the engine log
|
||||
# names it as `Using cache directory: /root/.cache/vllm/torch_compile_cache/…`.
|
||||
grep -q -- 'mkdir -p "$CACHE_DIR"' "$SERVE"
|
||||
grep -q -- '${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}' "$SERVE"
|
||||
grep -q -- '-e GPU_MAX_HW_QUEUES="$FRANK_PRIMARY_GPU_MAX_HW_QUEUES"' "$SERVE"
|
||||
}
|
||||
|
||||
@test "tools and reasoning are wired — the client contract is served" {
|
||||
|
||||
Reference in new issue
Block a user