frank-primary: four full-window slots, V q8_0 KV, hot expert blocks on-GPU
- CONTEXT_SIZE=1048576 / PARALLEL_SLOTS=4: four conversations, each the full native 262144 window (per-slot window unchanged; slots divide the pool, they never multiply it). - CACHE_TYPE_V=q8_0 / CACHE_TYPE_K=f16: main KV drops 24,576 -> 18,816 B/token (-23.4%); the shared MTP drafter's KV ignores the V type (measured: 2,048 MiB draft pool = the f16 law at 1,048,576 cells). - OT_OVERRIDES: blocks 0-3 on ROCm0 and 45-47 on ROCm1 stay GPU-resident (2.42/2.49 GiB per block, measured); the rest stay CPU-computed. Measured free under load: ~4.9 GB card0 / ~3.4 GB card1; a 5th block on card0 lands at 2.27 GB, under the 3 GB checkpoint floor, so blocks are atomic and 4 is the safe fill. - serve script: --cache-type-k/v env-wired tunables; ONE joined --override-tensor flag (b11030 deprecates repeated flags, and the loader is first-match-wins, so the specific patterns must precede the broad exps=CPU sweep). - seams: the window test asserts per-slot = 262144 at 4 slots; new KV-type test; OT exact-value assertion pins the ordering law.
This commit is contained in:
1 parent
de4d22785d
commit
de85654b76
3 files changed
+76
-34
No files matched your search
@@ -55,7 +55,8 @@ fi
|
||||
: "${SPEC_DRAFT_DEVICE:=ROCm0}"
|
||||
: "${BATCH_SIZE:=8192}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
: "${CACHE_RAM:=16384}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${VERBOSITY:=4}"
|
||||
|
||||
SERVER_BIN="$RUNTIME_DIR/llama-server"
|
||||
@@ -108,7 +109,8 @@ ARGS=(
|
||||
--reasoning auto
|
||||
--reasoning-format deepseek
|
||||
--reasoning-budget -1
|
||||
--flash-attn on
|
||||
--cache-type-k "$CACHE_TYPE_K"
|
||||
--cache-type-v "$CACHE_TYPE_V"
|
||||
--batch-size "$BATCH_SIZE"
|
||||
--ubatch-size "$UBATCH_SIZE"
|
||||
--cache-ram "$CACHE_RAM"
|
||||
@@ -128,10 +130,11 @@ ARGS=(
|
||||
--min-p 0.0
|
||||
)
|
||||
|
||||
# Comma-separated <pattern>=<backend> pairs -> repeated --override-tensor.
|
||||
IFS=',' read -ra _ots <<< "$OT_OVERRIDES"
|
||||
for ot in "${_ots[@]}"; do
|
||||
ARGS+=(--override-tensor "$ot")
|
||||
done
|
||||
# Comma-separated <pattern>=<backend> pairs, ONE flag: b11030 logs
|
||||
# DEPRECATED for repeated --override-tensor and wants the joined list.
|
||||
# The loader applies the FIRST matching pattern per tensor, so the
|
||||
# specific block patterns must precede the broad exps=CPU sweep in
|
||||
# frank-primary.env.
|
||||
ARGS+=(--override-tensor "$OT_OVERRIDES")
|
||||
|
||||
exec "$SERVER_BIN" "${ARGS[@]}"
|
||||
Reference in new issue
Block a user