dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-large: full 262144 context, 1 slot

Browse Source 
User request 2026-08-19 (second pass): parallel slots degraded
performance. PARALLEL=1, CTX_SIZE=262144 (native max, -c equals the
per-slot context with one slot). Concurrent requests queue on the
single slot; the server never preempts a running generation.
pi models.json contextWindow/maxTokens back to 262144 (outside repo).
Docs updated to match.
This commit is contained in:
Frank
2026-08-19 11:17:31 -07:00
parent 8e4d63b68d
commit 048e7eaaba
6 changed files with 22 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+3 -3
View File
@@ -8,7 +8,7 @@ local machines. Topology (2026-08-18):
| Target | Host | Runtime | Service | Model | Port |
|--------|------|---------|---------|-------|------|
| Dankserver | dank@dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP, PromptForge DualView, 200K context / 2 slots | 52625 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP, PromptForge DualView, 262K context / 1 slot | 52625 |
The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw; the
@@ -16,7 +16,7 @@ CIRU ROCmFPX llama.cpp runtime now runs on the Strix Halo iGPU (gfx1151).
A second 2026-08-18 swap renamed the services and restored the Qwen3.6 35B
A3B Q6_K_P model on Dankserver: Dankserver is now `frank-default-small`
(Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE experts on CPU) and Dankclaw is
now `frank-default-large` (CIRU Qwen3.8-27B, 200K context / 2 slots). The
now `frank-default-large` (CIRU Qwen3.8-27B, 262K context / 1 slot). The
retired Qwen3.8-27B IQ4_XS FastMTP files live in `.AGENTTRASH/`. See
docs/ciru-activefpx-promptforge.md. The old scripts live in `old/scripts/`
for reference; they describe the retired PP topology.
@@ -105,7 +105,7 @@ When you rename a service or move a model to a new host, update
`models.json`. Update the provider key, the model `id`, `contextWindow`, and
`input`. If the default model moved hosts, also update `defaultProvider` in
`settings.json`. Set `contextWindow` to the per-slot context (dankserver
`175104`, dankclaw `200000`). The full schema is in the agent knowledge base
`175104`, dankclaw `262144`). The full schema is in the agent knowledge base
page `~/.pi/agent/docs/pi/models.md`.
## Runtime & Troubleshooting
README.md
+1 -1
View File
@@ -15,7 +15,7 @@ upstream llama.cpp on Dankserver and CIRU ROCmFPX llama.cpp on Dankclaw.
| Service | Host | Runtime | Model | Port |
|---------|------|---------|-------|------|
| frank-default-small | dankserver | llama.cpp (upstream b10121) | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU, 4 slots x ~175K ctx | 7770 |
| frank-default-large | dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK) | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView, 200K ctx / 2 slots | 52625 |
| frank-default-large | dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK) | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView, 262K ctx / 1 slot | 52625 |
The 2026-08-17 redo retired the pipeline-parallel split (PP head + RPC
worker) and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the
docs/ciru-activefpx-promptforge.md
+4 -3
View File
@@ -131,9 +131,10 @@ Downloaded with the HF token to
### Launch profile
- 200K context per slot (user request 2026-08-19; native max 262144),
2 slots, f16 KV (~24 GiB total for both slots), context checkpoints
(16), prompt cache
- 262144 context (native max), 1 slot (user request 2026-08-19; parallel
slots degraded performance), f16 KV (~16 GiB), context checkpoints (16),
prompt cache. Concurrent requests queue on the single slot; the server
never preempts a running generation
- Native MTP draft, depth 4, device ROCm0
- `--temp 0.7 --top-p 0.95`, reasoning on (`--reasoning-format deepseek`
puts `<think>` reasoning in `message.reasoning_content`)
docs/deployment.md
+4 -3
View File
@@ -9,7 +9,7 @@ Two servers run LLM inference services. Each server pulls its scripts from this
| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView, 200K ctx / 2 slots) | frank-default-large | 52625 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 52625 |
## Workflow
@@ -120,8 +120,9 @@ The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
companions) under the alias `frank-default-large` on port 52625. It runs
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP
(`--spec-type draft-mtp`), f16 KV for both K and V, 200000 context per
slot (400000 total across 2 parallel slots), batch 2048 / ubatch 2048, 16
(`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context
(1 slot; concurrent requests queue, the server never preempts a running
generation), batch 2048 / ubatch 2048, 16
context checkpoints, 8 GiB
prompt cache, and reasoning in deepseek format. The unit file is
`scripts/frank-default-large.service`. It launches through
docs/index.md
+1 -1
View File
@@ -22,7 +22,7 @@ ngram-mod, 36 MoE experts on CPU, 700K ctx]
E[CIRU ROCmFPX llama.cpp
frank-default-large, port 52625]
F[jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge
200K ctx / 2 slots, native MTP, PromptForge DualView]
262K ctx / 1 slot, native MTP, PromptForge DualView]
D --> E --> F
end
scripts/frank-default-large.env
+9 -9
View File
@@ -39,15 +39,15 @@ ALIAS=frank-default-large
BIND_ADDR=0.0.0.0
LISTEN_PORT=52625
# Serving: 200000 context PER SLOT (user request 2026-08-19; native max
# n_ctx_train = 262144), 2 slots. -c is the TOTAL context; llama.cpp
# splits it across the -np slots, so CTX_SIZE = 2 x 200000 = 400000.
# f16 KV, context checkpoints, prompt cache, native MTP depth-4.
# The Strix Halo is unified memory; rocm-smi reports the 64 GiB budget.
# KV at 400000 x 64 KiB/cell is ~24 GiB. The ~8 GiB saved versus 524288
# adds OOM headroom (see the 2026-08-18 CTX_CHECKPOINTS 32 -> 16 change).
CTX_SIZE=400000
PARALLEL=2
# Serving: full native 262144 context (n_ctx_train), 1 slot (user request
# 2026-08-19: parallel slots degraded performance). -c is the TOTAL
# context; with PARALLEL=1 it equals the per-slot context. Concurrent
# requests queue on the single slot; the server never preempts a running
# generation. f16 KV, context checkpoints, prompt cache, native MTP
# depth-4. The Strix Halo is unified memory; rocm-smi reports the 64 GiB
# budget. KV at 262144 x 64 KiB/cell is ~16 GiB.
CTX_SIZE=262144
PARALLEL=1
BATCH_SIZE=2048
UBATCH_SIZE=2048
CACHE_TYPE_K=f16