dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

fleet redo 2026-08-17: retire PP+NPU, new topology

Browse Source 
- Move all previous scripts to old/scripts/ (retired PP/NPU-era)
- frank-default-large: llama.cpp FastMTP on dankserver (HauhauCS
  Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod, K q8_0/V f16,
  128K ctx, batch 2048/ubatch 1024, port 7770)
- frank-default-small: vLLM ROCm on dankclaw (Avesed Qwen3.6-35B-A3B
  INT8 W8A8, full 262144 ctx, no KV quant, 4 seqs, batch 16384,
  ngram spec, port 52625)
- retarget deploy.sh (dankserver/dankclaw, inline sudo, no helpers)
- docs: deployment + index + AGENTS.md + README for new topology,
  superseded banners on retired docs, add frank-redo-2026-08-17 runbook
This commit is contained in:
Frank committed 2026-08-17 15:23:46 -07:00
1 parent 136ee080c9
commit d9af7f1d19
45 files changed
+961 -469
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/FastFlowLM NPU Inference.md
+4
View File
@@ -1,3 +1,7 @@
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
> [deployment.md](deployment.md) for the active configuration.
---
type: research
title: FastFlowLM NPU Inference
docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md
+4
View File
@@ -1,3 +1,7 @@
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
> [deployment.md](deployment.md) for the active configuration.
# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)
Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX
docs/dankserver-vram-growth.md
+4
View File
@@ -1,3 +1,7 @@
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
> [deployment.md](deployment.md) for the active configuration.
---
type: reference
title: Dankserver VRAM and RAM Growth Under Load
docs/deployment.md
+64 -179
View File
@@ -8,43 +8,8 @@ Two servers run LLM inference services. Each server pulls its scripts from this
| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B-Uncensored Q6_K, split 0.68 to the Dankserver worker) | frank-default-large | 7770 |
| Dankclaw | dank@dankclaw | FastFlowLM NPU (qwythos:9b, 262144 ctx) | frank-default-small | 52625 |
| Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090) | frank-pp-worker | 50052 |
The `frank-default-small` (NPU) unit is not in `deploy.sh` and is disabled for boot (see Boot Policy below). If the unit is still loaded (pre-reboot), restart it by hand with `sudo systemctl restart frank-default-small`. After a reboot, re-enable the unit first (see Boot Policy).
## Boot Policy (2026-08-17)
Only the pipeline-parallel pair starts at boot: `frank-default-large` (Dankclaw head) and `frank-pp-worker` (Dankserver worker). Every other inference unit is disabled. This was set on 2026-08-17.
Disabled units:
| Machine | Unit | Fate |
|---------|------|------|
| Dankclaw | `frank-default-small` (FastFlowLM NPU, port 52625) | Disabled; unit file removed from `/etc/systemd/system`. The service keeps running until stopped or rebooted. |
| Dankclaw | `frank-large-ling` (Ling trial) | Disabled. |
| Dankclaw | `vllm-thinkingcap` | Disabled (already). |
| Dankserver | `frank-default-small` (old llama.cpp 35B) | Disabled; unit file removed. |
| Dankserver | `llamaserver`, `ollama`, `vllm-awq-qwen36`, `vllm-qwen36` | Disabled. |
`systemctl disable` removes the symlinked unit file from `/etc/systemd/system` when the file is a symlink. The repo copies stay intact. To restore a disabled service, re-symlink the unit and enable it:
```
sudo ln -sf /home/dank/<repo>/scripts/<unit>.service /etc/systemd/system/<unit>.service
sudo systemctl daemon-reload
sudo systemctl enable <unit>
```
The NPU re-enable recipe uses the Dankclaw repo path `/home/dank/vllmScripts/scripts/frank-default-small.service`.
The Dankserver helper `frank-restart-small.sh` targets the removed `frank-default-small` unit. Do not use it. Restart the PP worker with an explicit command:
```
sudo systemctl restart frank-pp-worker
```
`scripts/deploy.sh dankserver` also targets the removed unit. Do not run it until the script is retargeted to `frank-pp-worker`. The `dankclaw` target works: it restarts the PP head through `frank-restart-large.sh`.
| Dankserver | dank@dankserver | llama.cpp FastMTP (Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod) | frank-default-large | 7770 |
| Dankclaw | dank@dankclaw | vLLM ROCm (Avesed Qwen3.6-35B-A3B INT8 W8A8, ngram spec) | frank-default-small | 52625 |
## Workflow
@@ -70,22 +35,22 @@ graph LR
The `scripts/deploy.sh` script executes steps 4 through 6. Run it from the repository root.
Run this command to deploy to Dankclaw:
```
./scripts/deploy.sh dankclaw
```
Run this command to deploy to Dankserver:
```
./scripts/deploy.sh dankserver
FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver
```
Run this command to deploy to Dankclaw:
```
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw
```
Add the `--dry-run` flag to preview without making changes:
```
./scripts/deploy.sh dankclaw --dry-run
./scripts/deploy.sh dankserver --dry-run
```
The script does these actions:
@@ -93,153 +58,81 @@ The script does these actions:
1. Verifies that the local working tree has no uncommitted changes.
2. Pushes the `main` branch to origin.
3. Pulls the repository on the target machine.
4. Restarts the service. When the passwordless helper is installed (see below), it runs the helper with plain `sudo`. Otherwise it falls back to the inline sudo commands (unit symlink, cache drop, daemon-reload, restart).
4. Symlinks the systemd unit, reloads systemd, enables, and restarts the service.
5. Waits for the health endpoint to respond (over ssh).
Set `FRANK_SUDO_PW` when running headless on a target that does not yet have the passwordless helper, so that remote sudo reads the password from stdin:
Set `FRANK_SUDO_PW` when running headless, so that remote sudo reads the
password from stdin (`sudo -S`). The passwordless restart helpers from the
retired topology are obsolete and are not used.
Health endpoints:
- Dankserver: `http://127.0.0.1:7770/health`
- Dankclaw: `http://127.0.0.1:52625/health`
## Manual Update
### Update Dankserver (frank-default-large)
Run these commands on Dankserver:
```
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw
cd /home/dank/vLLM
git pull origin main
sudo ln -sf /home/dank/vLLM/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:7770/health
```
## Passwordless Restart
Each target has a root-owned helper script that performs the full restart. The user `dank` runs it with `sudo` and no password via a sudoers drop-in.
| Target | Helper | Sudoers drop-in |
|--------|--------|-----------------|
| Dankclaw | `/usr/local/sbin/frank-restart-large.sh` | `/etc/sudoers.d/frank-large` |
| Dankserver | `/usr/local/sbin/frank-restart-small.sh` (obsolete, see below) | `/etc/sudoers.d/frank-small` |
The Dankclaw helper does these steps:
1. Runs `sync`.
2. Drops the page caches with `echo 3 > /proc/sys/vm/drop_caches`.
3. Symlinks the systemd unit.
4. Reloads systemd.
5. Enables and restarts the service.
The Dankserver helper `frank-restart-small.sh` is obsolete: it restarts the removed `frank-default-small` unit. Do not use it. The sudoers drop-in does not cover `frank-pp-worker`, so the worker restart needs the password (or a new helper). Restart the worker with `sudo systemctl restart frank-pp-worker`.
The sudoers drop-in contains exactly two rules:
```
dank ALL=(root) NOPASSWD: /usr/local/sbin/frank-restart-large.sh
dank ALL=(root) NOPASSWD: /usr/bin/systemctl status frank-default-large
```
Rules match the command and its arguments exactly. Do not append extra arguments. For example, `sudo systemctl status frank-default-large --no-pager` fails; use the exact form instead. Install or replace the helper and drop-in only through `sudo` on the target, and validate with `sudo visudo -c`.
To test that the setup works, run this command:
```
sudo -n /usr/local/sbin/frank-restart-large.sh
```
### Update Dankclaw
1. Pull the repository.
2. Symlink the unit file, if it changed.
3. Drop the page caches.
4. Reload and restart the service.
### Update Dankclaw (frank-default-small)
Run these commands on Dankclaw:
```
cd /home/dank/vllmScripts
git pull origin main
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-large.service /etc/systemd/system/frank-default-large.service
sync && echo 3 | sudo tee /proc/sys/vm/drop_caches
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
sudo systemctl daemon-reload
sudo systemctl restart frank-default-large
curl -sf http://127.0.0.1:7770/health
sudo systemctl restart frank-default-small
curl -sf http://127.0.0.1:52625/health
```
### Update Dankserver
Run these commands on Dankserver to update and restart the PP worker:
```
cd /home/dank/vLLM
git pull origin main
sudo ln -sf /home/dank/vLLM/scripts/frank-pp-worker.service /etc/systemd/system/frank-pp-worker.service
sudo systemctl daemon-reload
sudo systemctl restart frank-pp-worker
```
The worker has no HTTP health endpoint. Verify the listener instead:
```
ss -tln | grep :50052
```
The old passwordless helper `frank-restart-small.sh` targets the removed `frank-default-small` unit. Do not use it. See Boot Policy.
## Service Units
### Dankclaw — frank-default-large
### Dankserver — frank-default-large
The active service on Dankclaw is the llama.cpp **pipeline-parallel
head**. It serves `Qwen3.8-27B-Uncensored-Q6_K.gguf` (JonathanColetti,
imatrix) under the alias `frank-default-large` on port 7770. The head
splits the tensors `0.68,0.32` to the `ggml-rpc` worker on Dankserver
(RTX 3090, port 50052): 45 of 65 blocks (0-44) on the 3090 — the
maximum for this config (measured 2026-08-17; see the PP doc).
Speculative decoding runs **MTP** with a standalone draft head
(`--spec-draft-model`, `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`, n_max
4) plus the `--spec-default` ngram-mod fallback (48/64/24). The head GPU
side runs **ROCm0** (no `--device` flag; default device selection puts
RPC first and picks ROCm for the iGPU). **1 slot** at 262144 context
(pool 262144, `n_ctx_slot = 262144`), KV cache **K q8_0 + V f16**
(mismatched, 784 MiB per FA layer; RPC0 8624 + ROCm0 3920 MiB) with
Flash Attention retained via the GGML_CUDA_FA_ALL_QUANTS rebuild, full
layer offload (`--gpu-layers all`, `--fit off`), batch 8192 / ubatch
256, `--direct-io`. The unit file is
The active service on Dankserver is llama.cpp serving
`Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-IQ4_XS.gguf` under the alias
`frank-default-large` on port 7770. It runs the patched FastMTP build
(`/home/dank/llama.cpp-fastmtp/build/bin/llama-server`, commit 4df29be +
the HauhauCS patch) with the FastMTP 32K draft sidecar (`--spec-type
draft-mtp`, `--spec-draft-n-max 2`, `--spec-draft-ngl all`) and the
ngram-mod fallback. Context 131072 (128K, single slot), KV K q8_0 + V f16,
batch 2048 / ubatch 1024, `--no-mmap`, full GPU offload. The unit file is
`scripts/frank-default-large.service`. It launches through
`scripts/serve-pp-head.sh`, which reads tunables from
`scripts/serve-frank-large.sh`, which reads tunables from
`scripts/frank-default-large.env`.
> Note: 2026-08-17 — the Q6_K swap: model + standalone draft head,
> mismatched q8_0/f16 KV with FA (both binaries rebuilt with
> GGML_CUDA_FA_ALL_QUANTS at 9b05354ec), split 0.68 (was 0.53 with the
> Q8_K_XL). Worker at 23854/24576 MiB, ~273 MiB free. MTP acceptance
> 0.75-0.83 (mean len 4.0-4.3), stronger than the old bundled head.
### Dankclaw — frank-default-small
> Note: 2026-08-17 — single-slot config: `PARALLEL=1`, pool 262144
> (was 2 slots x 200700 at 401400), split 0.53 (was 0.40). The 3090
> landed at 23460/24576 MiB, ~667 MiB free. Same-day incident: a full
> Dankserver disk corrupted the worker RPC tensor cache -> degenerate
> output; fixed by clearing the cache + freeing the disk. Guard:
> `scripts/prune-rpc-cache.sh` (cron, 40 GiB cap).
> Note: 2026-08-16 — the pipeline-parallel split was promoted to
> production as `frank-default-large` (renamed from `frank-pp-head`);
> the standalone single-node service retired. MTP was re-enabled at
> `n_max 4` over the ngram-mod fallback. See
> [[Pipeline Parallelism/Qwen3.8 PP Deployment]] for the runbook.
> Note: 2026-08-15 — the spec went DSpark (crashed under two concurrent 16k
> prompts on ROCm, `ROCm error: an illegal memory access`, status 6/ABRT) →
> `--spec-default` (ngram-mod, parallel-safe, PARALLEL=2). The backend went
> ROCm → Vulkan0 after a 262k-ctx bench showed Vulkan decodes 48% faster
> (5.68 vs 3.83 t/s). That Vulkan0 verdict applied to the standalone
> configuration; the PP head runs ROCm0 by default (HIP registers before
> Vulkan). See [[qwen3.8-spec-matrix]] for both decisions and the bench
> table. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash
> runtime, and the llama-serve/vLLM units on Dankclaw.
### Dankserver — old llama.cpp 35B service (retired 2026-08-16)
Dankserver previously ran `frank-default-small`: llama.cpp serving the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` (ngram-mod spec, 4 slots x 175104 ctx, 36 of 40 MoE expert layers on CPU, mmproj vision). The service was verified `inactive` on 2026-08-16, and its unit file was removed on 2026-08-17 under the boot policy. The name `frank-default-small` now belongs to the FastFlowLM NPU endpoint on Dankclaw (see [[FastFlowLM NPU Inference]]). The RTX 3090 now runs the PP worker. See git history for the full retired configuration.
The active service on Dankclaw is vLLM (ROCm) serving
`Avesed/Qwen3.6-35B-A3B-INT8-W8A8` under the served model name
`frank-default-small` on port 52625. Full 262144 context, unquantized KV
cache, `--max-num-seqs 4`, `--max-num-batched-tokens 16384`, ngram
speculative decoding, `--reasoning-parser qwen3`. The unit file is
`scripts/frank-default-small.service`. It launches through
`scripts/serve-frank-small.sh`, which reads tunables from
`scripts/frank-default-small.env`.
## Validation
The deploy script checks the health endpoint after a restart. For deeper checks, run the validation script on the target.
Run this command on Dankserver to test vLLM throughput:
The deploy script checks the health endpoint after a restart. For deeper checks, run a completion request against the endpoint:
```
./scripts/validate-vllm.sh
curl -s http://127.0.0.1:7770/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"frank-default-large","messages":[{"role":"user","content":"Say OK"}]}'
```
## Troubleshooting
@@ -248,32 +141,24 @@ If the health check fails after a deploy:
1. Read the service logs with `journalctl -u <service> -n 200`.
2. Confirm that the model files exist in the cache.
3. Confirm that the GPU has free memory.
3. Confirm that the GPU has free memory (`nvidia-smi` on Dankserver, `rocm-smi` on Dankclaw).
4. Check the service status with `systemctl status <service>`.
## Incident: HIP Runtime Crash From the `TEMP` Env Variable (2026-08-13)
On 2026-08-13 the `frank-default-large` service crash-looped for hours. Every restart died with SIGSEGV about 20 seconds into model load, at a fixed offset in `libamdhip64` (NULL refcount unref during `ggml_backend_cuda_buffer_set_tensor`). The kernel log showed bursts of `amdgpu: SVM mapping failed, exceeds resident system memory limit`, which pointed at memory but was a red herring.
On 2026-08-13 the old `frank-default-large` service crash-looped for hours on Dankclaw. Root cause: the env files exported `TEMP=1.0` (a sampling default). The ROCm HIP runtime reads `TEMP` as its JIT kernel-cache directory, used the path `1.0`, failed, and segfaulted on a NULL object. Fix: rename the sampling variable to `TEMPERATURE` (commit `227ce01`).
Root cause: the env files exported `TEMP=1.0` (a sampling default added in commit `db3eb01`). The ROCm HIP runtime reads the `TEMP` environment variable as its JIT kernel-cache directory. HIP then used the path `1.0`, failed, and segfaulted on a NULL object. The crash was deterministic whenever the serve script sourced the env file, and absent when it did not.
Fix: rename the sampling variable to `TEMPERATURE` in the env files and serve scripts (commit `227ce01`). The serve script still passes `--temp "$TEMPERATURE"` to `llama-server`, so sampling defaults are unchanged.
If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Check with `strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|..."`. Do not name any tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`.
If a service segfaults at load with no obvious cause, compare the exported env against the HIP runtime's env reads. Do not name any tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`.
## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)
A user reported that a streaming job on `frank-default-large` was released early when another job started. The suspicion was speculative decoding. The investigation found three facts:
1. **The server does not preempt.** `llama-server` defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically with concurrent streaming jobs.
2. **The releases were client-side aborts.** The pi coding agent aborts a stream when the user interrupts it. The client records `stopReason: "aborted"`; the server logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`.
3. **Dankserver crashed under the parallel researcher load.** Five parallel research subagents hammered `frank-default-small` with back-to-back requests. The service aborted twice with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. That crash, not speculative decoding, killed the parallel jobs. The affected service is retired; see [[dankserver-vram-growth]] for the VRAM/RAM growth findings behind the crash class.
Historical: on the retired topology, parallel jobs on the Dankserver 35B service died with a cuBLAS crash under researcher load. The affected service is retired. A client abort logs `W srv stop: cancel task, id_task = N`; a natural stop logs only `slot release` with `truncated = 0`. The server never preempts a running generation.
## See Also
- [AGENTS.md](../AGENTS.md) — the git workflow rules
- [README.md](../README.md) — repository overview
- [index.md](index.md) — knowledge base index
- [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) — the full redo runbook
- [serverAppConfigs knowledge base](../serverAppConfigs/docs/index.md) —
the sibling Podman app stack on the same hosts; its OpenWebUI consumes
the `:7770` endpoint, and its Gitea instance hosts this repository
the sibling Podman app stack on the same hosts
docs/frank-default-large-benchmark.md
+4
View File
@@ -1,3 +1,7 @@
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
> [deployment.md](deployment.md) for the active configuration.
# frank-default-large — Backend and Spec Benchmark
This page records the benchmarks that chose the runtime for
docs/frank-redo-2026-08-17.md
+127
View File
@@ -0,0 +1,127 @@
# Frank Program Redo — 2026-08-17
This page records the full redo of the inference fleet on 2026-08-17.
The pipeline-parallel split and the FastFlowLM NPU endpoint were retired.
The old scripts moved to `old/scripts/` in this repository.
## New Topology
| Service | Host | Runtime | Model | Port |
|---------|------|---------|-------|------|
| frank-default-large | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
| frank-default-small | Dankclaw | vLLM ROCm | Avesed Qwen3.6-35B-A3B INT8 W8A8 | 52625 |
The RTX 3090 runs the whole 27B model standalone. No RPC worker, no
tensor split, no NPU.
## Models
### frank-default-large — HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF:IQ4_XS
- Base: Qwen3.8-27B (dense 27B, 64 layers: 48 GatedDeltaNet + 16
gated-attention, hidden 5120, 24 Q heads / 4 KV heads, head dim 256,
native 262144 ctx).
- Quant: IQ4_XS (4.60 BPW, 15.71 GB). Every GGUF in the repo preserves the
native embedded MTP/NextN head.
- FastMTP draft sidecar: `...FastMTP-32K.gguf` (903 MB). Requires the
patched llama.cpp build (commit `4df29be4f4c3673f428170fda944a5b19f743bb8`
+ `HauhauCS-FastMTP-llama.cpp.patch`). Without the patch, the draft load
fails with `expected 5120, 248320, got 5120, 32768`.
- Spec chain: FastMTP (`--spec-type draft-mtp`, `--spec-draft-n-max 2`,
`--spec-draft-ngl all`, `--spec-draft-p-min 0`) then ngram-mod fallback
(48/64/24).
### frank-default-small — Avesed/Qwen3.6-35B-A3B-INT8-W8A8
- Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40
layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx).
- Quant: compressed-tensors INT8 W8A8 (symmetric per-channel weights,
dynamic per-token activations). Expert FFNs quantized; attention, router,
DeltaNet, lm_head kept bf16. Weights ~38 GB.
- Served with vLLM ROCm, ngram speculative decoding.
## Context Math (frank-default-large, RTX 3090)
KV per token: 16 attention layers x 4 KV heads x 256 dim.
K q8_0 = 1088 B/token, V f16 = 2048 B/token, total 50176 B/token.
Budget (MiB): 24576 total - 14982 weights - 861 draft - 1024 fixed
(CUDA context, compute graph, scratch, draft KV) - headroom.
| Context | KV (MiB) | Free after load |
|---------|----------|-----------------|
| 114688 (112K) | 5488 | 2221 (2.17 GiB) |
| 122880 (120K) | 5880 | 1829 (1.79 GiB) |
| 131072 (128K) | 6272 | 1437 (1.40 GiB) |
| 147456 (144K) | 7056 | 653 (0.64 GiB) |
Production choice: **131072 (128K)** — the maximum that keeps about
1.4 GiB headroom for checkpoints. Lower `CTX_SIZE` in
`scripts/frank-default-large.env` to 122880 for ~1.8 GiB headroom.
## Install Runbook
### Dankserver (frank-default-large)
1. Stop the old worker: `sudo systemctl stop frank-pp-worker`.
2. Download the model and draft to `/home/dank/models/hauhaucs-qwen38/`.
3. Build the patched llama.cpp:
```
git clone https://github.com/ggerganov/llama.cpp /home/dank/llama.cpp-fastmtp
cd /home/dank/llama.cpp-fastmtp
git checkout 4df29be4f4c3673f428170fda944a5b19f743bb8
curl -L -o HauhauCS-FastMTP-llama.cpp.patch <HF patch URL>
git apply HauhauCS-FastMTP-llama.cpp.patch
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j$(nproc)
```
4. Symlink the unit, reload, enable, restart:
```
sudo ln -sf /home/dank/vLLM/scripts/frank-default-large.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable frank-default-large
sudo systemctl restart frank-default-large
```
5. Verify: `curl -sf http://127.0.0.1:7770/health` and a chat completion.
6. Confirm VRAM headroom with `nvidia-smi`.
### Dankclaw (frank-default-small)
1. Stop the old services: `sudo systemctl stop frank-default-large
frank-default-small` (retired PP head and NPU endpoint).
2. Install uv and create the venv:
```
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv --python /usr/bin/python3.12 /home/dank/vllm-venv
uv pip install --python /home/dank/vllm-venv vllm --torch-backend=auto
```
The resolver pulls the AMD ROCm wheel set (torch +
amd-torch-device-gfx1151 + the vllm rocm variant) from the AMD index.
3. Cache the model (default HF cache):
```
python3 -c "from huggingface_hub import snapshot_download; snapshot_download('Avesed/Qwen3.6-35B-A3B-INT8-W8A8')"
```
4. Symlink the unit, reload, enable, restart:
```
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable frank-default-small
sudo systemctl restart frank-default-small
```
5. Verify: `curl -sf http://127.0.0.1:52625/health` and a chat completion.
## Fallback: vLLM source build for gfx1151
If the prebuilt ROCm vLLM wheel cannot run on gfx1151, build from source.
The retired CIRU package documents the proven recipe:
`/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/scripts/build-vllm-gfx1151.sh`
(branch `ciru/ling3-gfx1151-rocm715`, index `https://rocm.nightlies.amd.com/whl-multi-arch/`).
The upstream starting point needed a five-line ROCm sampler fix to launch
on gfx1151; newer vLLM releases may already include it.
## Configuration
- `models.json` (pi): two providers — `llamacpp (dankserver)` at
http://192.168.0.235:7770/v1 and `vllm (dankclaw)` at
http://192.168.0.143:52625/v1.
- HF token: `~/.cache/huggingface/token` on both machines (mode 600).
- OpenWebUI (sibling repo): retarget `OPENAI_API_BASE_URL` to the
Dankserver endpoint.
docs/index.md
+42 -52
View File
@@ -1,8 +1,8 @@
# vLLM Scripts — Knowledge Base
Documentation for the LLM inference fleet on two local machines:
llama.cpp production (PP head + RPC worker), the FastFlowLM NPU endpoint,
plus a vLLM research archive.
Documentation for the LLM inference fleet on two local machines
(2026-08-17 redo): llama.cpp FastMTP on Dankserver and vLLM ROCm on
Dankclaw.
## Deployment Topology
@@ -10,44 +10,47 @@ plus a vLLM research archive.
graph LR
subgraph Dankserver[NVIDIA Dankserver - Ubuntu 24.04]
A[RTX 3090 24GB + 62GB RAM]
B[ggml-rpc worker
frank-pp-worker, port 50052]
A --> B
B[llama.cpp FastMTP
frank-default-large, port 7770]
C[Qwen3.8-27B-Aggressive IQ4_XS
FastMTP 32K draft + ngram-mod, 128K ctx]
A --> B --> C
end
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
D[gfx1151 APU 30GiB phys / 96GiB GPU-visible]
E[llama.cpp PP head
frank-default-large, port 7770]
F[Qwen3.8-27B-Uncensored Q6_K
split 0.68/0.32 to worker, standalone MTP draft + ngram-mod]
D[gfx1151 iGPU unified memory]
E[vLLM ROCm
frank-default-small, port 52625]
F[Avesed Qwen3.6-35B-A3B INT8 W8A8
full 262144 ctx, ngram spec, 4 seqs]
D --> E --> F
G[FastFlowLM NPU XDNA2 50 TOPS
frank-default-small, port 52625 - DISABLED for boot 2026-08-17]
H[qwythos:9b
Q4NX 7.76 GB, 262k ctx]
D --> G --> H
end
I[LLM Clients / API] -->|HTTP :7770| E
I -->|HTTP :52625| G
E -->|RPC layers 0.68| B
I[LLM Clients / API] -->|HTTP :7770| B
I -->|HTTP :52625| E
```
Boot policy (2026-08-17): only `frank-default-large` and `frank-pp-worker` are enabled. The NPU endpoint and all retired units are disabled; see [[deployment]] for the table and re-enable recipes.
## Servers
| Server | GPU | VRAM / Memory | OS |
| --------------------------------------- | ----------------------------- | ----------------------------------- | ------------ |
| [[Dankserver System Specs\|Dankserver]] | NVIDIA RTX 3090 | 24 GB VRAM, 62 GB RAM | Ubuntu 24.04 |
| [[Dankclaw System Specs\|Dankclaw]] | AMD Radeon 8060S (Strix Halo) | 30 GiB physical / 96 GiB GPU-visible | Ubuntu 26.04 |
| Server | GPU | VRAM / Memory | OS |
| --------------------------------------- | ----------------------------- | -------------------- | ------------ |
| [[Dankserver System Specs\|Dankserver]] | NVIDIA RTX 3090 | 24 GB VRAM, 62 GB RAM | Ubuntu 24.04 |
| [[Dankclaw System Specs\|Dankclaw]] | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 |
## vLLM Reference
## Active Deployment
- [[deployment]] — deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the full redo: topology change, model
choices, context math, and the two install runbooks
- [[Dankserver]] — the llama.cpp FastMTP box (frank-default-large)
- [[Dankclaw]] — the vLLM ROCm box (frank-default-small)
## vLLM Reference (Dankclaw)
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
- [[deployment]] — deploy and update process for both servers
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
notes and sizing (the model family served by frank-default-small)
## Memory Management
@@ -55,36 +58,23 @@ Boot policy (2026-08-17): only `frank-default-large` and `frank-pp-worker` are e
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
## Model Deployment
## Retired Topology (2026-08-17 redo)
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — MoE model (35B total, 3B active) on RTX 3090
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Dense model (27B) on Strix Halo APU
- [[frank-default-large-benchmark]] — backend/spec benchmark; chose MTP sidecar (2026-08-14) over DFlash (parallel premature-EOS bug)
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history + current production state: Uncensored Q6_K + standalone MTP draft, q8_0/f16 KV with FA (FA_ALL_QUANTS rebuild), 1 slot @ 262144, split 0.68 (2026-08-17)
- [[qwen3.8-strix-optimizations]] — Qwen3.8-27B Strix speed research: ranked levers that keep UD-Q8_K_XL (PP to 3090, MTP tuning, same-bit Q8_0 A/B, prefill path, OS), quant landscape, 2026-08-16 test matrix
- [[frank-default-small-benchmark]] — ncmoe and batch/ubatch sweep that drove the Dankserver state
- [[dankserver-vram-growth]] — VRAM/RAM growth investigation behind the recurring cuBLAS crash on frank-default-small
- [[Dankclaw Improvements]] — researched enhancements: self-hosted agentic tools, runtime tuning, server/OS settings
- [[FastFlowLM NPU Inference]] — FastFlowLM (FLM), the AMD NPU-first runtime. The 50 TOPS Strix Halo NPU on Dankclaw: model fit, vendor benchmarks, verified install state, test runbook (2026-08-16)
## Speculative Decoding & Quantization
- [[DFlash DSpark Speculative Decoding]] — block diffusion (DFlash) and semi-autoregressive (DSpark) drafting on vLLM and llama.cpp
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
- [[frank-default-large-benchmark]] — backend/spec benchmark behind the old head
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
## Pipeline Parallelism
## Pipeline Parallelism (archive)
- [[Pipeline Parallelism]] — index of multi-node inference research and sample scripts
- [[Pipeline Parallelism/llama.cpp RPC]] — ggml-rpc-server + `--rpc` distributed inference
- [[Pipeline Parallelism/vLLM Pipeline Parallelism]] — vLLM PP across machines
- [[Pipeline Parallelism/Ethernet Bandwidth Math]] — bytes per token and link feasibility
- [[Pipeline Parallelism/Mixed-Vendor Experiments]] — NVIDIA + Strix Halo real-world runs
- [[Pipeline Parallelism/Expected Performance]] — estimated tok/s over 1/2.5/10 GbE
- [[Pipeline Parallelism/Sample Script Set]] — educational two-node Qwen3.6-27B Q8 split
- [[Pipeline Parallelism/Qwen3.8 PP Deployment]] — the production deployment + runbook: Qwen3.8-27B head (Strix) / worker (RTX 3090) over RPC, b10433, scripts in scripts/; split limits, FA-all-quants rebuild, cache incident (2026-08-17)
- [[Pipeline Parallelism]] — index of multi-node inference research. The PP
pair is retired; these pages are research history.
## Sibling projects
- [serverAppConfigs](../serverAppConfigs/docs/index.md) — the Podman app
stack on the same two hosts. Its OpenWebUI consumes the `:7770`
endpoint; its Gitea instance hosts this repository.
stack on the same two hosts. Its OpenWebUI consumes the `:7770` endpoint
on Dankserver; its Gitea instance hosts this repository.
docs/qwen3.8-spec-matrix.md
+4
View File
@@ -1,3 +1,7 @@
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
> [deployment.md](deployment.md) for the active configuration.
# Qwen3.8-27B Speculative-Decoding Matrix
This page records the speculative-decoding benchmark history and the