dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: exact prefill, the only mode the streamed tier runs

Browse Source 
--ds4-prefill dense made requires_monolithic_model() true, and a monolithic
load asks ggml for the entire 187 GiB expert set as one device-0 buffer:
"allocating 191932.25 MiB on device 0: cudaMalloc failed: out of memory",
followed by "monolithic HIP load required for dense prefill". Approximate
prefill (dense, sparse) is expressible only where the experts are resident
(in-process owners). exact feeds the prompt token by token, which is the
path the hybrid tier streams experts for. Chunk follows DS41.md's
cold-expert recipe (1024).
This commit is contained in:
Frank
2026-09-26 17:11:31 -07:00
parent de513ab080
commit dcdf044aa1
1 changed files with 16 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
dankserver/frank-primary/frank-primary-entrypoint.sh
+16 -9
View File
@@ -15,13 +15,20 @@
# that branch — see serverAppConfigs-docs/LLM/frank-primary-lucebox-ds41.md. # that branch — see serverAppConfigs-docs/LLM/frank-primary-lucebox-ds41.md.
# #
# LAUNCH SHAPE (server/docs/DS41.md — the V4.1 page, not the V4 DS4.md one): # LAUNCH SHAPE (server/docs/DS41.md — the V4.1 page, not the V4 DS4.md one):
# --ds4-prefill dense the V4.1 batched prefill. The default `exact` walks # --ds4-prefill exact THE ONLY prefill this placement can run. Any
# the prompt token by token, and the V4-only modes are # approximate mode (dense, sparse) makes
# REFUSED for V4.1: `sparse`, --ds4-fused-decode, # DeepSeek4Backend::requires_monolithic_model() true,
# --ds4-fused-verify-f16-kv, layer split and --mmproj # and a monolithic load asks ggml for the whole 187 GiB
# all fail validate_model_features at load. # expert set as ONE device-0 buffer — measured
# --chunk 4096 the chunk the V4.1 dense-prefill numbers were taken # 2026-09-26: "allocating 191932.25 MiB on device 0:
# at; the runtime caps it to what the device holds. # cudaMalloc failed: out of memory", then
# "monolithic HIP load required for dense prefill".
# DS41.md's dense-prefill section measures that mode
# where the experts fit (in-process owners); on a
# 32 GiB target it cannot. exact feeds the prompt token
# by token, which is what the hybrid tier streams for.
# --chunk 1024 DS41.md's cold-expert chunk. The runtime caps the
# chunk to what fits after headroom regardless.
# --max-ctx 131072 the V4.1 default window. The MLA compressed caches # --max-ctx 131072 the V4.1 default window. The MLA compressed caches
# are F16 by design and stay small; --cache-type-k/v # are F16 by design and stay small; --cache-type-k/v
# are ignored by the DeepSeek4 backend, so they are # are ignored by the DeepSeek4 backend, so they are
@@ -68,7 +75,7 @@ exec "$BIN" "$MODEL" \
--model-name frank-primary \ --model-name frank-primary \
--target-device hip:0 \ --target-device hip:0 \
--max-ctx 131072 \ --max-ctx 131072 \
--chunk 4096 \ --chunk 1024 \
--ds4-prefill dense \ --ds4-prefill exact \
--prefix-cache-slots 0 \ --prefix-cache-slots 0 \
--prefill-cache-slots 0 --prefill-cache-slots 0