dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-large: 200000 context per slot (400000 total, 2 slots)

Browse Source 
User request 2026-08-19: lower per-stream context from 262144 to 200000.
CTX_SIZE is the total; llama.cpp splits it across the 2 parallel slots.
f16 KV drops from ~32 GiB to ~24 GiB, adding OOM headroom.

Docs: sync the 2026-08-18 service swap (frank-default-large on dankclaw,
frank-default-small on dankserver) across AGENTS.md, README.md,
deployment.md, ciru-activefpx-promptforge.md, and index.md. The pi
models.json contextWindow/maxTokens for frank-default-large move
262144 -> 200000 (separate file, outside the repo).
This commit is contained in:
Frank
2026-08-19 10:38:53 -07:00
parent 9fd82dc6e1
commit 8e4d63b68d
6 changed files with 32 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+8 -7
View File
@@ -1,7 +1,7 @@
# vLLM Scripts
Deployment scripts for the LLM inference fleet (2026-08-17 topology):
llama.cpp FastMTP on Dankserver and vLLM ROCm on Dankclaw.
Deployment scripts for the LLM inference fleet (2026-08-18 topology):
upstream llama.cpp on Dankserver and CIRU ROCmFPX llama.cpp on Dankclaw.
## Targets
@@ -14,12 +14,13 @@ llama.cpp FastMTP on Dankserver and vLLM ROCm on Dankclaw.
| Service | Host | Runtime | Model | Port |
|---------|------|---------|-------|------|
| frank-default-large | dankserver | llama.cpp (patched FastMTP build) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS, FastMTP 32K draft + ngram-mod, K q8_0 / V f16 KV, 128K ctx | 7770 |
| frank-default-small | dankclaw | vLLM ROCm (vllm-venv) | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, full 262144 ctx, no KV quantization, 4 seqs, batch 8192, ngram spec + tool calling | 52625 |
| frank-default-small | dankserver | llama.cpp (upstream b10121) | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU, 4 slots x ~175K ctx | 7770 |
| frank-default-large | dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK) | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView, 200K ctx / 2 slots | 52625 |
The 2026-08-17 redo retired the pipeline-parallel split (PP head + RPC
worker) and the FastFlowLM NPU endpoint. Their scripts are preserved in
`old/scripts/` for reference.
worker) and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the
vLLM service on Dankclaw and renamed the services (see AGENTS.md). Their
scripts are preserved in `old/scripts/` for reference.
## Structure
@@ -31,7 +32,7 @@ worker) and the FastFlowLM NPU endpoint. Their scripts are preserved in
See [docs/deployment.md](docs/deployment.md). Run
`FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver` for the llama.cpp box or
`FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw` for the vLLM box.
`FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw` for the CIRU box.
## Knowledge Base