fleet redo 2026-08-17: retire PP+NPU, new topology
- Move all previous scripts to old/scripts/ (retired PP/NPU-era) - frank-default-large: llama.cpp FastMTP on dankserver (HauhauCS Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod, K q8_0/V f16, 128K ctx, batch 2048/ubatch 1024, port 7770) - frank-default-small: vLLM ROCm on dankclaw (Avesed Qwen3.6-35B-A3B INT8 W8A8, full 262144 ctx, no KV quant, 4 seqs, batch 16384, ngram spec, port 52625) - retarget deploy.sh (dankserver/dankclaw, inline sudo, no helpers) - docs: deployment + index + AGENTS.md + README for new topology, superseded banners on retired docs, add frank-redo-2026-08-17 runbook
This commit is contained in:
Frank committed
1 parent
136ee080c9
commit
d9af7f1d19
45 files changed
+961
-469
No files matched your search
@@ -0,0 +1,127 @@
|
||||
# Frank Program Redo — 2026-08-17
|
||||
|
||||
This page records the full redo of the inference fleet on 2026-08-17.
|
||||
The pipeline-parallel split and the FastFlowLM NPU endpoint were retired.
|
||||
The old scripts moved to `old/scripts/` in this repository.
|
||||
|
||||
## New Topology
|
||||
|
||||
| Service | Host | Runtime | Model | Port |
|
||||
|---------|------|---------|-------|------|
|
||||
| frank-default-large | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
|
||||
| frank-default-small | Dankclaw | vLLM ROCm | Avesed Qwen3.6-35B-A3B INT8 W8A8 | 52625 |
|
||||
|
||||
The RTX 3090 runs the whole 27B model standalone. No RPC worker, no
|
||||
tensor split, no NPU.
|
||||
|
||||
## Models
|
||||
|
||||
### frank-default-large — HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF:IQ4_XS
|
||||
|
||||
- Base: Qwen3.8-27B (dense 27B, 64 layers: 48 GatedDeltaNet + 16
|
||||
gated-attention, hidden 5120, 24 Q heads / 4 KV heads, head dim 256,
|
||||
native 262144 ctx).
|
||||
- Quant: IQ4_XS (4.60 BPW, 15.71 GB). Every GGUF in the repo preserves the
|
||||
native embedded MTP/NextN head.
|
||||
- FastMTP draft sidecar: `...FastMTP-32K.gguf` (903 MB). Requires the
|
||||
patched llama.cpp build (commit `4df29be4f4c3673f428170fda944a5b19f743bb8`
|
||||
+ `HauhauCS-FastMTP-llama.cpp.patch`). Without the patch, the draft load
|
||||
fails with `expected 5120, 248320, got 5120, 32768`.
|
||||
- Spec chain: FastMTP (`--spec-type draft-mtp`, `--spec-draft-n-max 2`,
|
||||
`--spec-draft-ngl all`, `--spec-draft-p-min 0`) then ngram-mod fallback
|
||||
(48/64/24).
|
||||
|
||||
### frank-default-small — Avesed/Qwen3.6-35B-A3B-INT8-W8A8
|
||||
|
||||
- Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40
|
||||
layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx).
|
||||
- Quant: compressed-tensors INT8 W8A8 (symmetric per-channel weights,
|
||||
dynamic per-token activations). Expert FFNs quantized; attention, router,
|
||||
DeltaNet, lm_head kept bf16. Weights ~38 GB.
|
||||
- Served with vLLM ROCm, ngram speculative decoding.
|
||||
|
||||
## Context Math (frank-default-large, RTX 3090)
|
||||
|
||||
KV per token: 16 attention layers x 4 KV heads x 256 dim.
|
||||
K q8_0 = 1088 B/token, V f16 = 2048 B/token, total 50176 B/token.
|
||||
|
||||
Budget (MiB): 24576 total - 14982 weights - 861 draft - 1024 fixed
|
||||
(CUDA context, compute graph, scratch, draft KV) - headroom.
|
||||
|
||||
| Context | KV (MiB) | Free after load |
|
||||
|---------|----------|-----------------|
|
||||
| 114688 (112K) | 5488 | 2221 (2.17 GiB) |
|
||||
| 122880 (120K) | 5880 | 1829 (1.79 GiB) |
|
||||
| 131072 (128K) | 6272 | 1437 (1.40 GiB) |
|
||||
| 147456 (144K) | 7056 | 653 (0.64 GiB) |
|
||||
|
||||
Production choice: **131072 (128K)** — the maximum that keeps about
|
||||
1.4 GiB headroom for checkpoints. Lower `CTX_SIZE` in
|
||||
`scripts/frank-default-large.env` to 122880 for ~1.8 GiB headroom.
|
||||
|
||||
## Install Runbook
|
||||
|
||||
### Dankserver (frank-default-large)
|
||||
|
||||
1. Stop the old worker: `sudo systemctl stop frank-pp-worker`.
|
||||
2. Download the model and draft to `/home/dank/models/hauhaucs-qwen38/`.
|
||||
3. Build the patched llama.cpp:
|
||||
```
|
||||
git clone https://github.com/ggerganov/llama.cpp /home/dank/llama.cpp-fastmtp
|
||||
cd /home/dank/llama.cpp-fastmtp
|
||||
git checkout 4df29be4f4c3673f428170fda944a5b19f743bb8
|
||||
curl -L -o HauhauCS-FastMTP-llama.cpp.patch <HF patch URL>
|
||||
git apply HauhauCS-FastMTP-llama.cpp.patch
|
||||
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
|
||||
cmake --build build --config Release -j$(nproc)
|
||||
```
|
||||
4. Symlink the unit, reload, enable, restart:
|
||||
```
|
||||
sudo ln -sf /home/dank/vLLM/scripts/frank-default-large.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload && sudo systemctl enable frank-default-large
|
||||
sudo systemctl restart frank-default-large
|
||||
```
|
||||
5. Verify: `curl -sf http://127.0.0.1:7770/health` and a chat completion.
|
||||
6. Confirm VRAM headroom with `nvidia-smi`.
|
||||
|
||||
### Dankclaw (frank-default-small)
|
||||
|
||||
1. Stop the old services: `sudo systemctl stop frank-default-large
|
||||
frank-default-small` (retired PP head and NPU endpoint).
|
||||
2. Install uv and create the venv:
|
||||
```
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
uv venv --python /usr/bin/python3.12 /home/dank/vllm-venv
|
||||
uv pip install --python /home/dank/vllm-venv vllm --torch-backend=auto
|
||||
```
|
||||
The resolver pulls the AMD ROCm wheel set (torch +
|
||||
amd-torch-device-gfx1151 + the vllm rocm variant) from the AMD index.
|
||||
3. Cache the model (default HF cache):
|
||||
```
|
||||
python3 -c "from huggingface_hub import snapshot_download; snapshot_download('Avesed/Qwen3.6-35B-A3B-INT8-W8A8')"
|
||||
```
|
||||
4. Symlink the unit, reload, enable, restart:
|
||||
```
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload && sudo systemctl enable frank-default-small
|
||||
sudo systemctl restart frank-default-small
|
||||
```
|
||||
5. Verify: `curl -sf http://127.0.0.1:52625/health` and a chat completion.
|
||||
|
||||
## Fallback: vLLM source build for gfx1151
|
||||
|
||||
If the prebuilt ROCm vLLM wheel cannot run on gfx1151, build from source.
|
||||
The retired CIRU package documents the proven recipe:
|
||||
`/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/scripts/build-vllm-gfx1151.sh`
|
||||
(branch `ciru/ling3-gfx1151-rocm715`, index `https://rocm.nightlies.amd.com/whl-multi-arch/`).
|
||||
The upstream starting point needed a five-line ROCm sampler fix to launch
|
||||
on gfx1151; newer vLLM releases may already include it.
|
||||
|
||||
## Configuration
|
||||
|
||||
- `models.json` (pi): two providers — `llamacpp (dankserver)` at
|
||||
http://192.168.0.235:7770/v1 and `vllm (dankclaw)` at
|
||||
http://192.168.0.143:52625/v1.
|
||||
- HF token: `~/.cache/huggingface/token` on both machines (mode 600).
|
||||
- OpenWebUI (sibling repo): retarget `OPENAI_API_BASE_URL` to the
|
||||
Dankserver endpoint.
|
||||
Reference in new issue
Block a user