perf(frank-small): cap the CPU RAM cache at 2048 MiB
The build default is 8192 MiB. dankclaw is memory-tight (frank-large's 70 GiB GTT plus the rest of the stack) and Gemma4 runs fully on the 3090 (-ngl 999), so the CPU-side cache is not the hot path. Cap it at 2048 to give back ~6 GiB of LPDDR. Gemma4 is not a DeltaNet model, so the >12.8 GiB CACHE_RAM floor does not apply. Wired through frank-small.env (CACHE_RAM) -> serve-frank-small.sh (--cache-ram), with a wiring seam so the cap cannot be silently unwired.
This commit is contained in:
1 parent
e22b43d589
commit
e23df8b229
3 files changed
+14
No files matched your search
@@ -58,6 +58,7 @@ fi
|
||||
: "${PARALLEL:=1}"
|
||||
: "${CACHE_TYPE_K:=f16}"
|
||||
: "${CACHE_TYPE_V:=f16}"
|
||||
: "${CACHE_RAM:=8192}"
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=512}"
|
||||
# Decode-shape knobs: the MTP drafter's per-round proposal length (build
|
||||
@@ -130,6 +131,7 @@ exec "$LLAMA_BIN" \
|
||||
--poll "$POLL_LEVEL" \
|
||||
--cache-type-k "$CACHE_TYPE_K" \
|
||||
--cache-type-v "$CACHE_TYPE_V" \
|
||||
--cache-ram "$CACHE_RAM" \
|
||||
"${SPEC_ARGS[@]}" \
|
||||
--jinja \
|
||||
--metrics \
|
||||
|
||||
Reference in new issue
Block a user