From 8e4d63b68dce05ab7e16cecaa9cf3cffd150cf6a Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 19 Aug 2026 10:38:53 -0700 Subject: [PATCH] frank-default-large: 200000 context per slot (400000 total, 2 slots) User request 2026-08-19: lower per-stream context from 262144 to 200000. CTX_SIZE is the total; llama.cpp splits it across the 2 parallel slots. f16 KV drops from ~32 GiB to ~24 GiB, adding OOM headroom. Docs: sync the 2026-08-18 service swap (frank-default-large on dankclaw, frank-default-small on dankserver) across AGENTS.md, README.md, deployment.md, ciru-activefpx-promptforge.md, and index.md. The pi models.json contextWindow/maxTokens for frank-default-large move 262144 -> 200000 (separate file, outside the repo). --- AGENTS.md | 6 +++--- README.md | 15 ++++++++------- docs/ciru-activefpx-promptforge.md | 18 +++++++++--------- docs/deployment.md | 6 +++--- docs/index.md | 2 +- scripts/frank-default-large.env | 16 ++++++++-------- 6 files changed, 32 insertions(+), 31 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 826530d..c384857 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,7 +8,7 @@ local machines. Topology (2026-08-18): | Target | Host | Runtime | Service | Model | Port | |--------|------|---------|---------|-------|------| | Dankserver | dank@dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 | -| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP, PromptForge DualView, 256K context / 2 slots | 52625 | +| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP, PromptForge DualView, 200K context / 2 slots | 52625 | The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw; the @@ -16,7 +16,7 @@ CIRU ROCmFPX llama.cpp runtime now runs on the Strix Halo iGPU (gfx1151). A second 2026-08-18 swap renamed the services and restored the Qwen3.6 35B A3B Q6_K_P model on Dankserver: Dankserver is now `frank-default-small` (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE experts on CPU) and Dankclaw is -now `frank-default-large` (CIRU Qwen3.8-27B, 256K context / 2 slots). The +now `frank-default-large` (CIRU Qwen3.8-27B, 200K context / 2 slots). The retired Qwen3.8-27B IQ4_XS FastMTP files live in `.AGENTTRASH/`. See docs/ciru-activefpx-promptforge.md. The old scripts live in `old/scripts/` for reference; they describe the retired PP topology. @@ -105,7 +105,7 @@ When you rename a service or move a model to a new host, update `models.json`. Update the provider key, the model `id`, `contextWindow`, and `input`. If the default model moved hosts, also update `defaultProvider` in `settings.json`. Set `contextWindow` to the per-slot context (dankserver -`175104`, dankclaw `262144`). The full schema is in the agent knowledge base +`175104`, dankclaw `200000`). The full schema is in the agent knowledge base page `~/.pi/agent/docs/pi/models.md`. ## Runtime & Troubleshooting diff --git a/README.md b/README.md index ad74da6..1bfe426 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # vLLM Scripts -Deployment scripts for the LLM inference fleet (2026-08-17 topology): -llama.cpp FastMTP on Dankserver and vLLM ROCm on Dankclaw. +Deployment scripts for the LLM inference fleet (2026-08-18 topology): +upstream llama.cpp on Dankserver and CIRU ROCmFPX llama.cpp on Dankclaw. ## Targets @@ -14,12 +14,13 @@ llama.cpp FastMTP on Dankserver and vLLM ROCm on Dankclaw. | Service | Host | Runtime | Model | Port | |---------|------|---------|-------|------| -| frank-default-large | dankserver | llama.cpp (patched FastMTP build) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS, FastMTP 32K draft + ngram-mod, K q8_0 / V f16 KV, 128K ctx | 7770 | -| frank-default-small | dankclaw | vLLM ROCm (vllm-venv) | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, full 262144 ctx, no KV quantization, 4 seqs, batch 8192, ngram spec + tool calling | 52625 | +| frank-default-small | dankserver | llama.cpp (upstream b10121) | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU, 4 slots x ~175K ctx | 7770 | +| frank-default-large | dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK) | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView, 200K ctx / 2 slots | 52625 | The 2026-08-17 redo retired the pipeline-parallel split (PP head + RPC -worker) and the FastFlowLM NPU endpoint. Their scripts are preserved in -`old/scripts/` for reference. +worker) and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the +vLLM service on Dankclaw and renamed the services (see AGENTS.md). Their +scripts are preserved in `old/scripts/` for reference. ## Structure @@ -31,7 +32,7 @@ worker) and the FastFlowLM NPU endpoint. Their scripts are preserved in See [docs/deployment.md](docs/deployment.md). Run `FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver` for the llama.cpp box or -`FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw` for the vLLM box. +`FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw` for the CIRU box. ## Knowledge Base diff --git a/docs/ciru-activefpx-promptforge.md b/docs/ciru-activefpx-promptforge.md index d9cf285..192d2b3 100644 --- a/docs/ciru-activefpx-promptforge.md +++ b/docs/ciru-activefpx-promptforge.md @@ -1,7 +1,7 @@ --- type: reference title: CIRU ActiveFPX PromptForge on Dankclaw -description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-default-small service. +description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-default-large service. tags: - llama.cpp - rocm @@ -9,12 +9,12 @@ tags: - gfx1151 - ciru - promptforge -timestamp: 2026-08-18 +timestamp: 2026-08-19 --- # CIRU ActiveFPX PromptForge on Dankclaw -`frank-default-small` (port 52625) now serves +`frank-default-large` (port 52625) serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` through the CIRU ROCmFPX llama.cpp runtime. This replaced the retired vLLM service that previously occupied the slot. @@ -124,16 +124,16 @@ Downloaded with the HF token to ## Service -- Unit: `frank-default-small.service`, port 52625 -- Script: `scripts/serve-frank-small.sh`, env `scripts/frank-default-small.env` +- Unit: `frank-default-large.service`, port 52625 +- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-default-large.env` - The script exports the ROCm 7.15 SDK libraries in `LD_LIBRARY_PATH` before the system ROCm. `ROCM_PATH` points at the devel SDK. -### Launch profile (from the model card) +### Launch profile -- 128K context (user request 2026-08-18; model card default is 64K), one - slot, f16 KV (8 GiB at 128K, ~51.5 GiB total used of 64 GiB), context - checkpoints (32), prompt cache +- 200K context per slot (user request 2026-08-19; native max 262144), + 2 slots, f16 KV (~24 GiB total for both slots), context checkpoints + (16), prompt cache - Native MTP draft, depth 4, device ROCm0 - `--temp 0.7 --top-p 0.95`, reasoning on (`--reasoning-format deepseek` puts `` reasoning in `message.reasoning_content`) diff --git a/docs/deployment.md b/docs/deployment.md index 11986d8..b2fe645 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -9,7 +9,7 @@ Two servers run LLM inference services. Each server pulls its scripts from this | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| | Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 | -| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView, 256K ctx / 2 slots) | frank-default-large | 52625 | +| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView, 200K ctx / 2 slots) | frank-default-large | 52625 | ## Workflow @@ -120,8 +120,8 @@ The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving companions) under the alias `frank-default-large` on port 52625. It runs `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm 7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP -(`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context per -slot (524288 total across 2 parallel slots), batch 2048 / ubatch 2048, 32 +(`--spec-type draft-mtp`), f16 KV for both K and V, 200000 context per +slot (400000 total across 2 parallel slots), batch 2048 / ubatch 2048, 16 context checkpoints, 8 GiB prompt cache, and reasoning in deepseek format. The unit file is `scripts/frank-default-large.service`. It launches through diff --git a/docs/index.md b/docs/index.md index 6cc1842..8a794b2 100644 --- a/docs/index.md +++ b/docs/index.md @@ -22,7 +22,7 @@ ngram-mod, 36 MoE experts on CPU, 700K ctx] E[CIRU ROCmFPX llama.cpp frank-default-large, port 52625] F[jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge -256K ctx / 2 slots, native MTP, PromptForge DualView] +200K ctx / 2 slots, native MTP, PromptForge DualView] D --> E --> F end diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index c23884f..20c089c 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -39,14 +39,14 @@ ALIAS=frank-default-large BIND_ADDR=0.0.0.0 LISTEN_PORT=52625 -# Serving: full native 256K context PER SLOT (n_ctx_train = 262144), 2 -# slots. -c is the TOTAL context; llama.cpp splits it across the -np slots, -# so to give each slot the full 262144 we set CTX_SIZE = 2 x 262144 = -# 524288. f16 KV, context checkpoints, prompt cache, native MTP depth-4. -# The Strix Halo is unified memory; the addressable pool is 96 GiB (VRAM is -# a pointer). KV at 524288 x 64 KiB/cell is ~32 GiB; total sits inside the -# 96 GiB pool. -CTX_SIZE=524288 +# Serving: 200000 context PER SLOT (user request 2026-08-19; native max +# n_ctx_train = 262144), 2 slots. -c is the TOTAL context; llama.cpp +# splits it across the -np slots, so CTX_SIZE = 2 x 200000 = 400000. +# f16 KV, context checkpoints, prompt cache, native MTP depth-4. +# The Strix Halo is unified memory; rocm-smi reports the 64 GiB budget. +# KV at 400000 x 64 KiB/cell is ~24 GiB. The ~8 GiB saved versus 524288 +# adds OOM headroom (see the 2026-08-18 CTX_CHECKPOINTS 32 -> 16 change). +CTX_SIZE=400000 PARALLEL=2 BATCH_SIZE=2048 UBATCH_SIZE=2048