dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

frank-primary: size the hot expert stack explicitly (LUCE_EXPERT_BUDGET_MB)

Browse Source 
The runtime's auto budget is gpu_free - kv - warm - safety (22.35 GiB), which
leaves only the 0.50 GiB safety margin free while the fit check demands the
1.50 GiB discrete headroom, so auto-sizing refuses on any 32 GiB target:
"device 0 memory: 30.97 of 31.86 GiB used, 0.89 GiB free (headroom 1.50
GiB): does not fit". 18000 MB is the value DS41.md measured its R9700 rows
at: ~1500 experts resident, ~5 GiB free on device 0.
This commit is contained in:
Frank
2026-09-26 17:13:05 -07:00
parent dcdf044aa1
commit 32d264ccda
1 changed files with 9 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
dankserver/frank-primary/frank-primary-entrypoint.sh
+9
View File
@@ -56,6 +56,15 @@ BIN="${FRANK_PRIMARY_BIN:-/opt/lucebox-hub/server/build/luce_server}"
# The second R9700 holds the streamed-expert slot cache and computes them.
export LUCE_EXPERT_STREAM_DEVICE="${LUCE_EXPERT_STREAM_DEVICE:-1}"
# The hot expert stack must be sized EXPLICITLY. Left alone the runtime
# budgets gpu_free - kv - warm - safety (22.35 GiB here), which leaves only
# its 0.50 GiB safety margin free while the fit check then demands the 1.50
# GiB discrete headroom — its own auto-sizing can never pass on a 32 GiB
# target: "device 0 memory: 30.97 of 31.86 GiB used, 0.89 GiB free (headroom
# 1.50 GiB): does not fit". 18000 MB is the value DS41.md measured its R9700
# rows at; it puts ~1500 experts (37/layer) resident and leaves device 0
# ~5 GiB free, so the fit check passes with room for transient buffers.
export LUCE_EXPERT_BUDGET_MB="${LUCE_EXPERT_BUDGET_MB:-18000}"
# rocBLAS/BLAS dispatch and queue depth, as the fleet's other ROCm servers.
export ROCBLAS_USE_HIPBLASLT="${ROCBLAS_USE_HIPBLASLT:-0}"
export GPU_MAX_HW_QUEUES="${GPU_MAX_HW_QUEUES:-8}"