perf(frank-small): cap the CPU RAM cache at 2048 MiB

The build default is 8192 MiB. dankclaw is memory-tight (frank-large's
70 GiB GTT plus the rest of the stack) and Gemma4 runs fully on the 3090
(-ngl 999), so the CPU-side cache is not the hot path. Cap it at 2048 to
give back ~6 GiB of LPDDR. Gemma4 is not a DeltaNet model, so the
>12.8 GiB CACHE_RAM floor does not apply.

Wired through frank-small.env (CACHE_RAM) -> serve-frank-small.sh
(--cache-ram), with a wiring seam so the cap cannot be silently unwired.
This commit is contained in:
Frank committed 2026-09-21 14:57:32 -07:00
1 parent e22b43d589
commit e23df8b229
3 files changed
+14

No files matched your search

+2
View File
@@ -58,6 +58,7 @@ fi
: "${PARALLEL:=1}"
: "${CACHE_TYPE_K:=f16}"
: "${CACHE_TYPE_V:=f16}"
: "${CACHE_RAM:=8192}"
: "${BATCH_SIZE:=2048}"
: "${UBATCH_SIZE:=512}"
# Decode-shape knobs: the MTP drafter's per-round proposal length (build
@@ -130,6 +131,7 @@ exec "$LLAMA_BIN" \
--poll "$POLL_LEVEL" \
--cache-type-k "$CACHE_TYPE_K" \
--cache-type-v "$CACHE_TYPE_V" \
--cache-ram "$CACHE_RAM" \
"${SPEC_ARGS[@]}" \
--jinja \
--metrics \