Docs: reflect the 2026-08-18 second swap (rename + Qwen3.6 restore)
- AGENTS.md: topology table, second-swap note, benchmark policy service names, mark FastMTP section retired, update model-load quirk. - docs/index.md: intro, topology diagram, active-deployment + CIRU reference service names. - docs/deployment.md: overview table, manual-update paths, rewrite both service-unit descriptions (Dankserver=Qwen3.6 Q6_K_P frank-default-small, Dankclaw=CIRU frank-default-large), validation model name.
This commit is contained in:
Frank committed
1 parent
1ba9618fcc
commit
819b9d448a
3 files changed
+65
-64
No files matched your search
+32
-37
@@ -8,8 +8,8 @@ Two servers run LLM inference services. Each server pulls its scripts from this
|
||||
|
||||
| Target | Host | Runtime | Service | Port |
|
||||
|--------|------|---------|---------|------|
|
||||
| Dankserver | dank@dankserver | llama.cpp FastMTP (Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod) | frank-default-large | 7770 |
|
||||
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView) | frank-default-small | 52625 |
|
||||
| Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 |
|
||||
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView, 256K ctx / 2 slots) | frank-default-large | 52625 |
|
||||
|
||||
## Workflow
|
||||
|
||||
@@ -72,65 +72,60 @@ Health endpoints:
|
||||
|
||||
## Manual Update
|
||||
|
||||
### Update Dankserver (frank-default-large)
|
||||
### Update Dankserver (frank-default-small)
|
||||
|
||||
Run these commands on Dankserver:
|
||||
|
||||
```
|
||||
cd /home/dank/vLLM
|
||||
git pull origin main
|
||||
sudo ln -sf /home/dank/vLLM/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
|
||||
sudo ln -sf /home/dank/vLLM/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart frank-default-large
|
||||
sudo systemctl restart frank-default-small
|
||||
curl -sf http://127.0.0.1:7770/health
|
||||
```
|
||||
|
||||
### Update Dankclaw (frank-default-small)
|
||||
### Update Dankclaw (frank-default-large)
|
||||
|
||||
Run these commands on Dankclaw:
|
||||
|
||||
```
|
||||
cd /home/dank/vllmScripts
|
||||
git pull origin main
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart frank-default-small
|
||||
sudo systemctl restart frank-default-large
|
||||
curl -sf http://127.0.0.1:52625/health
|
||||
```
|
||||
|
||||
## Service Units
|
||||
|
||||
### Dankserver — frank-default-large
|
||||
### Dankserver — frank-default-small
|
||||
|
||||
The active service on Dankserver is llama.cpp serving
|
||||
`Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-IQ4_XS.gguf` under the alias
|
||||
`frank-default-large` on port 7770. It runs the patched FastMTP build
|
||||
(`/home/dank/llama.cpp-fastmtp/build/bin/llama-server`, commit 4df29be +
|
||||
the HauhauCS patch) with the FastMTP 32K draft sidecar (`--spec-type
|
||||
draft-mtp`, `--spec-draft-n-max 2`, `--spec-draft-ngl all`) and the
|
||||
ngram-mod fallback. Context 131072 (128K, single slot), KV K q8_0 + V f16,
|
||||
batch 2048 / ubatch 1024, `--no-mmap`, full GPU offload. The unit file is
|
||||
The active service on Dankserver is upstream llama.cpp (b10121) serving
|
||||
`Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q6_K_P.gguf` under the
|
||||
alias `frank-default-small` on port 7770. It runs the stock
|
||||
`/home/dank/llama.cpp/build/bin/llama-server` binary with the 36 MoE
|
||||
experts on CPU (`--n-cpu-moe 36`) and 7 CPU threads, ngram-mod
|
||||
speculative decoding (n_min 48 / n_max 64 / n_match 24), f16 KV for both
|
||||
K and V, context 700000 (4 slots of ~175K each), batch 8192 / ubatch 3072,
|
||||
`--no-mmap`. The unit file is `scripts/frank-default-small.service`. It
|
||||
launches through `scripts/serve-frank-small.sh`, which reads tunables from
|
||||
`scripts/frank-default-small.env`.
|
||||
|
||||
### Dankclaw — frank-default-large
|
||||
|
||||
The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
|
||||
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v2 GGUF + FFN/GDN
|
||||
companions) under the alias `frank-default-large` on port 52625. It runs
|
||||
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
|
||||
7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP
|
||||
(`--spec-type draft-mtp`), f16 KV for both K and V, context 262144 across
|
||||
2 parallel slots, batch 2048 / ubatch 2048, 32 context checkpoints, 8 GiB
|
||||
prompt cache, and reasoning in deepseek format. The unit file is
|
||||
`scripts/frank-default-large.service`. It launches through
|
||||
`scripts/serve-frank-large.sh`, which reads tunables from
|
||||
`scripts/frank-default-large.env`.
|
||||
|
||||
### Dankclaw — frank-default-small
|
||||
|
||||
The active service on Dankclaw is vLLM (ROCm) serving
|
||||
`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` under the served model name
|
||||
`frank-default-small` on port 52625. Full 262144 context, unquantized KV
|
||||
cache, `--max-num-seqs 4`, `--max-num-batched-tokens 8192`, ngram
|
||||
speculative decoding, `--reasoning-parser qwen3`, tool calling
|
||||
(`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with the custom
|
||||
chat template `/home/dank/models/qwen36-chat_template-tools.jinja`), and
|
||||
prefix caching (`--enable-prefix-caching`). The model is hybrid (GatedDeltaNet
|
||||
+ full attention); vLLM keeps prefix caching opt-in for hybrid models, so the
|
||||
flag is required. It reuses the KV of repeated prompt prefixes (pi re-sends
|
||||
the same system prompt + tool schema every turn), cutting follow-up TTFT
|
||||
roughly 4x (2023-token prefix: 4.6s cold -> 1.1s cached, 2026-08-17). The
|
||||
unit file is `scripts/frank-default-small.service`. It launches through
|
||||
`scripts/serve-frank-small.sh`, which reads tunables from
|
||||
`scripts/frank-default-small.env`.
|
||||
`scripts/frank-default-large.env`. See [[ciru-activefpx-promptforge]].
|
||||
|
||||
## Validation
|
||||
|
||||
@@ -139,7 +134,7 @@ The deploy script checks the health endpoint after a restart. For deeper checks,
|
||||
```
|
||||
curl -s http://127.0.0.1:7770/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"model":"frank-default-large","messages":[{"role":"user","content":"Say OK"}]}'
|
||||
-d '{"model":"frank-default-small","messages":[{"role":"user","content":"Say OK"}]}'
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
Reference in new issue
Block a user