Docs: reflect the 2026-08-18 second swap (rename + Qwen3.6 restore)

- AGENTS.md: topology table, second-swap note, benchmark policy service names, mark FastMTP section retired, update model-load quirk. - docs/index.md: intro, topology diagram, active-deployment + CIRU reference service names. - docs/deployment.md: overview table, manual-update paths, rewrite both service-unit descriptions (Dankserver=Qwen3.6 Q6_K_P frank-default-small, Dankclaw=CIRU frank-default-large), validation model name.
2026-08-18 01:29:21 -07:00
parent 1ba9618fcc
commit 819b9d448a
3 changed files with 65 additions and 64 deletions
@@ -7,16 +7,19 @@ local machines. Topology (2026-08-18):

 | Target | Host | Runtime | Service | Model | Port |
 |--------|------|---------|---------|-------|------|
-| Dankserver | dank@dankserver | llama.cpp (patched FastMTP build) | frank-default-large | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS + FastMTP 32K draft + ngram-mod | 7770 |
-| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-small | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP, PromptForge DualView | 52625 |
+| Dankserver | dank@dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
+| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP, PromptForge DualView, 256K context / 2 slots | 52625 |

 The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
-NPU endpoint. The RTX 3090 runs the whole 27B model standalone
-(no RPC worker). The 2026-08-18 swap retired the vLLM service on Dankclaw;
-frank-default-small now runs the CIRU ROCmFPX llama.cpp runtime on the
-Strix Halo iGPU (gfx1151). See docs/ciru-activefpx-promptforge.md.
-The old scripts live in `old/scripts/` for reference; they describe the
-retired PP topology.
+NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw; the
+CIRU ROCmFPX llama.cpp runtime now runs on the Strix Halo iGPU (gfx1151).
+A second 2026-08-18 swap renamed the services and restored the Qwen3.6 35B
+A3B Q6_K_P model on Dankserver: Dankserver is now `frank-default-small`
+(Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE experts on CPU) and Dankclaw is
+now `frank-default-large` (CIRU Qwen3.8-27B, 256K context / 2 slots). The
+retired Qwen3.8-27B IQ4_XS FastMTP files live in `.AGENTTRASH/`. See
+docs/ciru-activefpx-promptforge.md. The old scripts live in `old/scripts/`
+for reference; they describe the retired PP topology.

 **Sibling project:** [../serverAppConfigs](../serverAppConfigs) manages the
 Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its
@@ -33,8 +36,8 @@ when the bench finishes.
 Rules:

 1. Stop the service before any benchmark:
-   `sudo systemctl stop frank-default-large` (Dankserver) or
-   `sudo systemctl stop frank-default-small` (Dankclaw). Use the password
+   `sudo systemctl stop frank-default-small` (Dankserver) or
+   `sudo systemctl stop frank-default-large` (Dankclaw). Use the password
   `Assface.3241` with `sudo -S` when no tty is available.
 2. Run the bench on a scratch port (never the service ports 7770/52625).
 3. Restart the service when the bench completes:
@@ -83,9 +86,11 @@ graph LR

 ## Runtime & Troubleshooting

-### FastMTP patched llama.cpp (Dankserver)
+### FastMTP patched llama.cpp (retired, 2026-08-18)

-The HauhauCS FastMTP draft sidecar requires a llama.cpp build from commit
+Historical: the Qwen3.8-27B IQ4_XS FastMTP service was retired on 2026-08-18
+(files in `.AGENTTRASH/`). The build notes below are kept for reference.
+The HauhauCS FastMTP draft sidecar required a llama.cpp build from commit
 `4df29be4f4c3673f428170fda944a5b19f743bb8` plus the
 `HauhauCS-FastMTP-llama.cpp.patch` (from the HF repo). The build lives at
 `/home/dank/llama.cpp-fastmtp/build` on Dankserver. Loading the draft with
@@ -163,7 +168,7 @@ graph TD

 - `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
 - Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
- Model load is slow: Dankserver ~5 minutes cold (15.7 GB IQ4_XS through RAM, `--no-mmap`); Dankclaw CIRU ~5-10 minutes (34.9 GiB ActiveFPX payload). Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 10 minutes on Dankserver and 20 minutes on Dankclaw (see `HEALTH_WAIT`).
+- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw CIRU loads the 34.9 GiB ActiveFPX payload in ~5-10 minutes. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`).
 - A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
 - Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md).

@@ -8,8 +8,8 @@ Two servers run LLM inference services. Each server pulls its scripts from this

 | Target | Host | Runtime | Service | Port |
 |--------|------|---------|---------|------|
-| Dankserver | dank@dankserver | llama.cpp FastMTP (Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod) | frank-default-large | 7770 |
-| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView) | frank-default-small | 52625 |
+| Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 |
+| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView, 256K ctx / 2 slots) | frank-default-large | 52625 |

 ## Workflow

@@ -72,65 +72,60 @@ Health endpoints:

 ## Manual Update

-### Update Dankserver (frank-default-large)
+### Update Dankserver (frank-default-small)

 Run these commands on Dankserver:

 ```
 cd /home/dank/vLLM
 git pull origin main
-sudo ln -sf /home/dank/vLLM/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
+sudo ln -sf /home/dank/vLLM/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
 sudo systemctl daemon-reload
-sudo systemctl restart frank-default-large
+sudo systemctl restart frank-default-small
 curl -sf http://127.0.0.1:7770/health
 ```

-### Update Dankclaw (frank-default-small)
+### Update Dankclaw (frank-default-large)

 Run these commands on Dankclaw:

 ```
 cd /home/dank/vllmScripts
 git pull origin main
-sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
+sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
 sudo systemctl daemon-reload
-sudo systemctl restart frank-default-small
+sudo systemctl restart frank-default-large
 curl -sf http://127.0.0.1:52625/health
 ```

 ## Service Units

-### Dankserver — frank-default-large
+### Dankserver — frank-default-small

-The active service on Dankserver is llama.cpp serving
-`Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-IQ4_XS.gguf` under the alias
-`frank-default-large` on port 7770. It runs the patched FastMTP build
-(`/home/dank/llama.cpp-fastmtp/build/bin/llama-server`, commit 4df29be +
-the HauhauCS patch) with the FastMTP 32K draft sidecar (`--spec-type
-draft-mtp`, `--spec-draft-n-max 2`, `--spec-draft-ngl all`) and the
-ngram-mod fallback. Context 131072 (128K, single slot), KV K q8_0 + V f16,
-batch 2048 / ubatch 1024, `--no-mmap`, full GPU offload. The unit file is
+The active service on Dankserver is upstream llama.cpp (b10121) serving
+`Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-Q6_K_P.gguf` under the
+alias `frank-default-small` on port 7770. It runs the stock
+`/home/dank/llama.cpp/build/bin/llama-server` binary with the 36 MoE
+experts on CPU (`--n-cpu-moe 36`) and 7 CPU threads, ngram-mod
+speculative decoding (n_min 48 / n_max 64 / n_match 24), f16 KV for both
+K and V, context 700000 (4 slots of ~175K each), batch 8192 / ubatch 3072,
+`--no-mmap`. The unit file is `scripts/frank-default-small.service`. It
+launches through `scripts/serve-frank-small.sh`, which reads tunables from
+`scripts/frank-default-small.env`.
+
+### Dankclaw — frank-default-large
+
+The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
+`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v2 GGUF + FFN/GDN
+companions) under the alias `frank-default-large` on port 52625. It runs
+`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
+7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP
+(`--spec-type draft-mtp`), f16 KV for both K and V, context 262144 across
+2 parallel slots, batch 2048 / ubatch 2048, 32 context checkpoints, 8 GiB
+prompt cache, and reasoning in deepseek format. The unit file is
 `scripts/frank-default-large.service`. It launches through
 `scripts/serve-frank-large.sh`, which reads tunables from
-`scripts/frank-default-large.env`.
-
-### Dankclaw — frank-default-small
-
-The active service on Dankclaw is vLLM (ROCm) serving
-`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` under the served model name
-`frank-default-small` on port 52625. Full 262144 context, unquantized KV
-cache, `--max-num-seqs 4`, `--max-num-batched-tokens 8192`, ngram
-speculative decoding, `--reasoning-parser qwen3`, tool calling
-(`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with the custom
-chat template `/home/dank/models/qwen36-chat_template-tools.jinja`), and
-prefix caching (`--enable-prefix-caching`). The model is hybrid (GatedDeltaNet
-+ full attention); vLLM keeps prefix caching opt-in for hybrid models, so the
-flag is required. It reuses the KV of repeated prompt prefixes (pi re-sends
-the same system prompt + tool schema every turn), cutting follow-up TTFT
-roughly 4x (2023-token prefix: 4.6s cold -> 1.1s cached, 2026-08-17). The
-unit file is `scripts/frank-default-small.service`. It launches through
-`scripts/serve-frank-small.sh`, which reads tunables from
-`scripts/frank-default-small.env`.
+`scripts/frank-default-large.env`. See [[ciru-activefpx-promptforge]].

 ## Validation

@@ -139,7 +134,7 @@ The deploy script checks the health endpoint after a restart. For deeper checks,
 ```
 curl -s http://127.0.0.1:7770/v1/chat/completions \
  -H "Content-Type: application/json" \
-  -d '{"model":"frank-default-large","messages":[{"role":"user","content":"Say OK"}]}'
+  -d '{"model":"frank-default-small","messages":[{"role":"user","content":"Say OK"}]}'
 ```

 ## Troubleshooting
@@ -1,8 +1,8 @@
 # vLLM Scripts — Knowledge Base

 Documentation for the LLM inference fleet on two local machines
-(2026-08-18): llama.cpp FastMTP on Dankserver and CIRU ROCmFPX llama.cpp
-on Dankclaw.
+(2026-08-18): upstream llama.cpp on Dankserver (frank-default-small) and
+CIRU ROCmFPX llama.cpp on Dankclaw (frank-default-large).

 ## Deployment Topology

@@ -10,19 +10,19 @@ on Dankclaw.
 graph LR
    subgraph Dankserver[NVIDIA Dankserver - Ubuntu 24.04]
        A[RTX 3090 24GB + 62GB RAM]
-        B[llama.cpp FastMTP
-frank-default-large, port 7770]
-        C[Qwen3.8-27B-Aggressive IQ4_XS
-FastMTP 32K draft + ngram-mod, 128K ctx]
+        B[upstream llama.cpp b10121
+frank-default-small, port 7770]
+        C[Qwen3.6-35B-A3B-Uncensored Q6_K_P
+ngram-mod, 36 MoE experts on CPU, 700K ctx]
        A --> B --> C
    end

    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
-        D[gfx1151 iGPU unified memory]
+        D[gfx1151 iGPU 96GB unified memory]
        E[CIRU ROCmFPX llama.cpp
-frank-default-small, port 52625]
+frank-default-large, port 52625]
        F[jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge
-64K ctx, native MTP, PromptForge DualView]
+256K ctx / 2 slots, native MTP, PromptForge DualView]
        D --> E --> F
    end

@@ -42,21 +42,22 @@ frank-default-small, port 52625]
 - [[deployment]] — deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
 - [[ciru-activefpx-promptforge]] — the CIRU runtime on Dankclaw
-  (frank-default-small)
- [[Dankserver]] — the llama.cpp FastMTP box (frank-default-large)
- [[Dankclaw]] — the CIRU ROCmFPX box (frank-default-small)
+  (frank-default-large)
+- [[Dankserver]] — the upstream llama.cpp box (frank-default-small)
+- [[Dankclaw]] — the CIRU ROCmFPX box (frank-default-large)

 ## CIRU Reference (Dankclaw)

 - [[ciru-activefpx-promptforge]] — model, ROCm 7.15 toolchain, build,
-  and the frank-default-small service
+  and the frank-default-large service

 ## vLLM Reference (historical)

 - [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
 - [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
 - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
-  notes and sizing (the model family formerly served by frank-default-small)
+  notes and sizing (the model family now served by frank-default-small as
+  Q6_K_P)

 ## Memory Management