dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

Reduce frank-default-large context for KV headroom

Browse Source
This commit is contained in:
Frank committed 2026-08-25 04:05:33 -07:00
1 parent 7ca9dc248e
commit ee3c3de750
5 files changed
+18 -11
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/frank-default-large.env
+5 -5
View File
@@ -36,11 +36,11 @@ ALIAS=frank-default-large
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
# Serving: full 262144 context, 1 slot. f16 KV, context checkpoints, prompt
# cache, strict-MTP embedded draft on ROCm0. Unified memory; rocm-smi reports
# the 64 GiB budget. Concurrent requests queue; the server never preempts a
# running generation.
CTX_SIZE=262144
# Serving: 196608 context (64K below the model's 262144 maximum), 1 slot. f16
# KV, context checkpoints, prompt cache, strict-MTP embedded draft on ROCm0.
# This leaves sufficient unified-memory headroom for the model plus KV cache.
# Concurrent requests queue; the server never preempts a running generation.
CTX_SIZE=196608
PARALLEL=1
# Batch/ubatch follow the qualified ROCmI4 W4A4 profile (see the bench in
# scripts/bench-ubatch-rocmi4.sh for the ubatch sweep on 8k contexts).
scripts/frank-default-large.service
+1 -1
View File
@@ -1,5 +1,5 @@
[Unit]
Description=CIRU llama.cpp Qwen3.8-27B ROCmFP4 STRIX MTP + Sharp template (dankclaw, frank-default-large)
Description=ROCmFPX llama.cpp Qwen3.8-27B ROCmI4 embedded MTP + Sharp template (dankclaw, frank-default-large)
After=network-online.target
Wants=network-online.target