frank-default-large: full native 262144 context per slot (524288 total)
llama.cpp splits -c across -np slots; to give each of the 2 slots the full 262144 native context, set CTX_SIZE=524288. KV ~32 GiB; fits the 96 GiB unified pool. Update env default, serve-script fallback, and the deployment doc line.
This commit is contained in:
+3
-2
@@ -120,8 +120,9 @@ The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
|
||||
companions) under the alias `frank-default-large` on port 52625. It runs
|
||||
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
|
||||
7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP
|
||||
(`--spec-type draft-mtp`), f16 KV for both K and V, context 262144 across
|
||||
2 parallel slots, batch 2048 / ubatch 2048, 32 context checkpoints, 8 GiB
|
||||
(`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context per
|
||||
slot (524288 total across 2 parallel slots), batch 2048 / ubatch 2048, 32
|
||||
context checkpoints, 8 GiB
|
||||
prompt cache, and reasoning in deepseek format. The unit file is
|
||||
`scripts/frank-default-large.service`. It launches through
|
||||
`scripts/serve-frank-large.sh`, which reads tunables from
|
||||
|
||||
@@ -39,12 +39,14 @@ ALIAS=frank-default-large
|
||||
BIND_ADDR=0.0.0.0
|
||||
LISTEN_PORT=52625
|
||||
|
||||
# Serving: 256K context (native max, n_ctx_train = 262144), 2 slots, f16 KV,
|
||||
# context checkpoints, prompt cache. Native MTP depth-4. The Strix Halo is
|
||||
# unified memory; the addressable pool is 96 GiB (VRAM is a pointer). KV at
|
||||
# 262144 x 2 slots is ~32 GiB (64 KiB/cell, 16 full-attn layers); total
|
||||
# sits well inside the 96 GiB pool.
|
||||
CTX_SIZE=262144
|
||||
# Serving: full native 256K context PER SLOT (n_ctx_train = 262144), 2
|
||||
# slots. -c is the TOTAL context; llama.cpp splits it across the -np slots,
|
||||
# so to give each slot the full 262144 we set CTX_SIZE = 2 x 262144 =
|
||||
# 524288. f16 KV, context checkpoints, prompt cache, native MTP depth-4.
|
||||
# The Strix Halo is unified memory; the addressable pool is 96 GiB (VRAM is
|
||||
# a pointer). KV at 524288 x 64 KiB/cell is ~32 GiB; total sits inside the
|
||||
# 96 GiB pool.
|
||||
CTX_SIZE=524288
|
||||
PARALLEL=2
|
||||
BATCH_SIZE=2048
|
||||
UBATCH_SIZE=2048
|
||||
|
||||
@@ -37,7 +37,7 @@ fi
|
||||
: "${ALIAS:=frank-default-large}"
|
||||
: "${BIND_ADDR:=0.0.0.0}"
|
||||
: "${LISTEN_PORT:=52625}"
|
||||
: "${CTX_SIZE:=262144}"
|
||||
: "${CTX_SIZE:=524288}"
|
||||
: "${PARALLEL:=2}"
|
||||
: "${BATCH_SIZE:=2048}"
|
||||
: "${UBATCH_SIZE:=2048}"
|
||||
|
||||
Reference in New Issue
Block a user