dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: 5th expert block onto ROCm0 (operator-directed fill)

Browse Source 
Blocks 0-4 on ROCm0 + 45-47 on ROCm1. This exact shape was probed and
battery-verified earlier the same session: card0 2.54 GiB free at boot,
2.11 GiB after the vision graph (steady under load), card1 unchanged
at ~3.3 GiB. Card1 cannot take another block (0.82 GiB would OOM the
vision graph + boot drift), so 5/3 is the physical limit at the
4x262144 / V-q8_0 shape. OT assertion pins the new value.
This commit is contained in:
Frank
2026-09-26 21:38:05 -07:00
parent e330d56a06
commit d317d6b246
2 changed files with 8 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-primary.env
+7 -5
View File
@@ -55,10 +55,12 @@ SPEC_DRAFT_N_MAX=2
# whole per-block expert tensor (512 experts share one tensor), so "critical
# experts" = blocks co-located on the card that holds their layers.
# Measured cost: 2.42 GiB/block on ROCm0, 2.49 GiB/block on ROCm1.
# Blocks 0-3 on ROCm0 + 45-47 on ROCm1 leave ~4.9 GB / ~3.4 GB free under
# load (vision graph included). A 5th block on ROCm0 lands at 2.27 GB —
# under the 3 GB checkpoint floor — so it stays off; blocks are atomic,
# nothing lands card0 inside the 3-4 GB band.
# Blocks 0-4 on ROCm0 + 45-47 on ROCm1: card0 runs at ~2.5 GiB free at
# boot / ~2.1 GB under load (vision graph included) — the operator
# directed the fill PAST the 3 GB checkpoint floor on 2026-09-26
# ("shift a few more experts"); measured, battery-verified. Card1 keeps
# ~3.4 GB. Card1 CANNOT take another block: 3.31 GiB free − 2.49 GiB =
# 0.82 GiB, and the vision graph + boot drift alone exceed that.
# * exps=CPU — the remaining routed experts (blk.*.ffn_*_exps.weight,
# ~100 GiB at Q8_0 even with 7 blocks GPU-side), read once per token;
# the CPU backend computes them from page cache.
@@ -72,7 +74,7 @@ SPEC_DRAFT_N_MAX=2
# (the table must stay page-cache-backed; anonymous memory
# cgroup-kills silently).
NGL=999
OT_OVERRIDES='blk\.[0-3]\.ffn_(up|gate|down)_exps=ROCm0,blk\.4[5-7]\.ffn_(up|gate|down)_exps=ROCm1,exps=CPU,per_layer_token_embd.weight=CPU'
OT_OVERRIDES='blk\.[0-4]\.ffn_(up|gate|down)_exps=ROCm0,blk\.4[5-7]\.ffn_(up|gate|down)_exps=ROCm1,exps=CPU,per_layer_token_embd.weight=CPU'
TENSOR_SPLIT=1,1
# The drafter MUST sit on the SECOND R9700 (ROCm1): with -ts 1,1 the main
tests/frank_primary_unit.bats
+1 -1
View File
@@ -66,7 +66,7 @@ REMOTE_DEPLOY="$REPO_ROOT/scripts/deploy-remote.sh"
! grep -q -- '--no-mmap' "$SERVE"
! grep -q '^MMPROJ_ON_GPU' "$ENV_FILE"
ot="$(sed -nE "s/^OT_OVERRIDES='(.*)'$/\1/p" "$ENV_FILE")"
[ "$ot" = 'blk\.[0-3]\.ffn_(up|gate|down)_exps=ROCm0,blk\.4[5-7]\.ffn_(up|gate|down)_exps=ROCm1,exps=CPU,per_layer_token_embd.weight=CPU' ]
[ "$ot" = 'blk\.[0-4]\.ffn_(up|gate|down)_exps=ROCm0,blk\.4[5-7]\.ffn_(up|gate|down)_exps=ROCm1,exps=CPU,per_layer_token_embd.weight=CPU' ]
}
@test "the KV pool is V-quantized, K held at f16" {