dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

perf(frank-small): cap the CPU RAM cache at 2048 MiB

Browse Source 
The build default is 8192 MiB. dankclaw is memory-tight (frank-large's
70 GiB GTT plus the rest of the stack) and Gemma4 runs fully on the 3090
(-ngl 999), so the CPU-side cache is not the hot path. Cap it at 2048 to
give back ~6 GiB of LPDDR. Gemma4 is not a DeltaNet model, so the
>12.8 GiB CACHE_RAM floor does not apply.

Wired through frank-small.env (CACHE_RAM) -> serve-frank-small.sh
(--cache-ram), with a wiring seam so the cap cannot be silently unwired.
This commit is contained in:
Frank committed 2026-09-21 14:57:32 -07:00
1 parent e22b43d589
commit e23df8b229
3 files changed
+14
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/frank-small.env
+6
View File
@@ -74,6 +74,12 @@ CTX_SIZE=1048576
PARALLEL=4
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=q8_0
# CPU RAM cache cap in MiB (build default 8192). This host is memory-tight
# (frank-large's 70 GiB GTT plus the rest of the stack), and Gemma4 runs
# fully on the 3090 (-ngl 999), so the CPU-side cache is not the hot path;
# cap it at 2048 to give back ~6 GiB of LPDDR. Gemma4 is not a DeltaNet
# model, so the >12.8 GiB CACHE_RAM floor does not apply.
CACHE_RAM=2048
# Batch geometry: 8192 logical / 1024 physical. Measured on this card at
# max length (254k-token prefill / 253k-ctx decode, 4 slots, q8 KV):