Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
228 lines
10 KiB
Markdown
228 lines
10 KiB
Markdown
---
|
||
type: reference
|
||
description: "The day the whole inference fleet was rebuilt: what came down, what replaced it, and why it was done that way."
|
||
tags:
|
||
- llm
|
||
- frank
|
||
- retired-endpoint
|
||
timestamp: 2026-09-03
|
||
---
|
||
|
||
# Frank Program Redo — 2026-08-17
|
||
|
||
This page records the full redo of the inference fleet on 2026-08-17.
|
||
The pipeline-parallel split and the FastFlowLM NPU endpoint were retired.
|
||
The old scripts moved to `old/scripts/` in this repository.
|
||
|
||
> **HISTORY — the topology below is two fleet generations dead.** Today the
|
||
> fleet is three endpoints, each owned by its page: [[frank-archon]]
|
||
> (`192.168.0.235:7770`), [[frank-dragoon]] (`192.168.0.143:7770`) and
|
||
> [[frank-zealot]] (`192.168.0.143:7771`). No vLLM and no ROCm endpoint serves
|
||
> anything; there is no pipeline parallelism and no NPU endpoint.
|
||
|
||
## New Topology
|
||
|
||
| Service | Host | Runtime | Model | Port |
|
||
|---------|------|---------|-------|------|
|
||
| frank-dragoon | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
|
||
| frank-zealot | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
|
||
|
||
Names as the 2026-08-17 commit wrote them: `frank-default-large` on Dankserver
|
||
(`git show d9af7f1:scripts/frank-default-large.env:20,22` → `ALIAS=frank-default-large`,
|
||
`LISTEN_PORT=7770`) and `frank-default-small` on Dankclaw
|
||
(`git show d9af7f1:scripts/frank-default-small.env:16,18` → `ALIAS=frank-default-small`,
|
||
`LISTEN_PORT=52625`); the Protoss ids date from 2026-10-06 → 10-10.
|
||
|
||
The RTX 3090 runs the whole 27B model standalone. No RPC worker, no
|
||
tensor split, no NPU.
|
||
|
||
## Models
|
||
|
||
### frank-dragoon — HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF:IQ4_XS
|
||
|
||
- Base: Qwen3.8-27B (dense 27B, 64 layers: 48 GatedDeltaNet + 16
|
||
gated-attention, hidden 5120, 24 Q heads / 4 KV heads, head dim 256,
|
||
native 262144 ctx).
|
||
- Quant: IQ4_XS (4.60 BPW, 15.71 GB). Every GGUF in the repo preserves the
|
||
native embedded MTP/NextN head.
|
||
- FastMTP draft sidecar: `...FastMTP-32K.gguf` (903 MB). Requires the
|
||
patched llama.cpp build (commit `4df29be4f4c3673f428170fda944a5b19f743bb8`
|
||
+ `HauhauCS-FastMTP-llama.cpp.patch`). Without the patch, the draft load
|
||
fails with `expected 5120, 248320, got 5120, 32768`.
|
||
- Spec chain: FastMTP (`--spec-type draft-mtp`, `--spec-draft-n-max 2`,
|
||
`--spec-draft-ngl all`, `--spec-draft-p-min 0`) then ngram-mod fallback
|
||
(48/64/24).
|
||
|
||
### frank-zealot — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
|
||
|
||
- Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40
|
||
layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx).
|
||
- Quant: AMD Quark W8A8 INT8 (per-channel static weights, per-token
|
||
dynamic activations). MoE experts stored as per-expert nn.Linear
|
||
triplets that match vLLM's FusedMoE mapping exactly. Weights ~35 GB;
|
||
the vision tower, router, lm_head stay BF16.
|
||
- Served with vLLM ROCm, ngram speculative decoding, and tool calling.
|
||
- Swap history: the initial deployment used
|
||
`Avesed/Qwen3.6-35B-A3B-INT8-W8A8` (compressed-tensors INT8). Its tool
|
||
calls were malformed (the model emitted `<function=get_weather(city=...)>`
|
||
instead of the Qwen3 XML structure). The Quark quant follows the format
|
||
cleanly; swapped in the evening of 2026-08-17.
|
||
|
||
## Context Math (frank-dragoon, RTX 3090)
|
||
|
||
KV per token: 16 attention layers x 4 KV heads x 256 dim.
|
||
K q8_0 = 1088 B per layer (4 KV heads × 256 dim × 1.0625 B), V f16 = 2048 B per
|
||
layer, ×16 layers = 50176 B/token total.
|
||
|
||
Budget (MiB): 24576 total - 14982 weights - 861 draft - 1024 fixed
|
||
(CUDA context, compute graph, scratch, draft KV) - headroom.
|
||
|
||
| Context | KV (MiB) | Free after load |
|
||
|---------|----------|-----------------|
|
||
| 114688 (112K) | 5488 | 2221 (2.17 GiB) |
|
||
| 122880 (120K) | 5880 | 1829 (1.79 GiB) |
|
||
| 131072 (128K) | 6272 | 1437 (1.40 GiB) |
|
||
| 147456 (144K) | 7056 | 653 (0.64 GiB) |
|
||
|
||
Production choice: **131072 (128K)** — the maximum that keeps about
|
||
1.4 GiB headroom for checkpoints. Lower `CTX_SIZE` in
|
||
`scripts/frank-dragoon.env` to 122880 for ~1.8 GiB headroom.
|
||
|
||
## Install Runbook
|
||
|
||
### Dankserver (frank-dragoon)
|
||
|
||
1. Stop the old worker: `sudo systemctl stop frank-pp-worker`.
|
||
2. Download the model and draft to `/home/dank/models/hauhaucs-qwen38/`.
|
||
3. Build the patched llama.cpp:
|
||
```
|
||
git clone https://github.com/ggerganov/llama.cpp /home/dank/llama.cpp-fastmtp
|
||
cd /home/dank/llama.cpp-fastmtp
|
||
git checkout 4df29be4f4c3673f428170fda944a5b19f743bb8
|
||
curl -L -o HauhauCS-FastMTP-llama.cpp.patch <HF patch URL>
|
||
git apply HauhauCS-FastMTP-llama.cpp.patch
|
||
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release \
|
||
-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON
|
||
cmake --build build --config Release -j$(nproc)
|
||
```
|
||
`GGML_CUDA_FA_ALL_QUANTS=ON` is mandatory. Without it, prefill collapses
|
||
~12x (the q8_0 KV + flash attention falls back to a slow path;
|
||
measured 95 t/s vs 1300+ t/s at 8K on the 3090).
|
||
4. Symlink the unit, reload, enable, restart:
|
||
```
|
||
sudo ln -sf /home/dank/vLLM/scripts/frank-dragoon.service /etc/systemd/system/
|
||
sudo systemctl daemon-reload && sudo systemctl enable frank-dragoon
|
||
sudo systemctl restart frank-dragoon
|
||
```
|
||
5. Verify: `curl -sf http://127.0.0.1:7770/health` and a chat completion.
|
||
6. Confirm VRAM headroom with `nvidia-smi`.
|
||
|
||
### Dankclaw (frank-zealot)
|
||
|
||
1. Stop the old services: `sudo systemctl stop frank-dragoon
|
||
frank-zealot` (retired PP head and NPU endpoint).
|
||
2. Install uv and create the venv:
|
||
```
|
||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||
uv venv --python /usr/bin/python3.12 /home/dank/vllm-venv
|
||
uv pip install --python /home/dank/vllm-venv vllm --torch-backend=auto
|
||
```
|
||
The resolver pulls the AMD ROCm wheel set (torch +
|
||
amd-torch-device-gfx1151 + the vllm rocm variant) from the AMD index.
|
||
3. Cache the model (default HF cache):
|
||
```
|
||
python3 -c "from huggingface_hub import snapshot_download; snapshot_download('Avesed/Qwen3.6-35B-A3B-INT8-W8A8')"
|
||
```
|
||
4. Install the ROCm wheel set and the flash-attention triton path:
|
||
```
|
||
uv pip install torch --torch-backend=auto
|
||
uv pip install "vllm==0.27.1+rocm723" --index-url https://wheels.vllm.ai/rocm/ \
|
||
--extra-index-url https://pypi.org/simple --index-strategy unsafe-best-match --no-build-isolation
|
||
cp -r <ciru-venv>/site-packages/flash_attn* <vllm-venv>/site-packages/
|
||
# add a flash_attn_2_cuda stub module (ROCm build has no compiled .so)
|
||
```
|
||
Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (env file). Without it the
|
||
vision encoder falls back to torch SDPA and engine init OOMs.
|
||
5. Symlink the unit, reload, enable, restart:
|
||
```
|
||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-zealot.service /etc/systemd/system/
|
||
sudo systemctl daemon-reload && sudo systemctl enable frank-zealot
|
||
sudo systemctl restart frank-zealot
|
||
```
|
||
6. Verify: `curl -sf http://127.0.0.1:52625/health` and a chat completion.
|
||
|
||
## Fallback: vLLM source build for gfx1151
|
||
|
||
If the prebuilt ROCm vLLM wheel cannot run on gfx1151, build from source.
|
||
The retired CIRU package documents the proven recipe:
|
||
`/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/scripts/build-vllm-gfx1151.sh`
|
||
(branch `ciru/ling3-gfx1151-rocm715`, index `https://rocm.nightlies.amd.com/whl-multi-arch/`).
|
||
The upstream starting point needed a five-line ROCm sampler fix to launch
|
||
on gfx1151; newer vLLM releases may already include it.
|
||
|
||
## Configuration
|
||
|
||
- `models.json` (pi): two providers — `llamacpp (dankserver)` at
|
||
http://192.168.0.235:7770/v1 and `vllm (dankclaw)` at
|
||
http://192.168.0.143:52625/v1. (The harness config today has three providers
|
||
and no vLLM: `~/.omp/agent/models.yml:14,94,173`.)
|
||
- HF token: `~/.cache/huggingface/token` on both machines (mode 600).
|
||
- OpenWebUI (sibling repo): retarget `OPENAI_API_BASE_URL` to the
|
||
Dankserver endpoint.
|
||
|
||
## Evening Session (2026-08-17): tool calling, model swap, batch 8192
|
||
|
||
Three changes landed after the initial deployment:
|
||
|
||
### 1. Tool calling on frank-zealot (HTTP 400 fix)
|
||
|
||
Clients (pi) send `tool_choice: "auto"`. vLLM rejects it with HTTP 400
|
||
unless the server enables tool calling. Fixes applied:
|
||
|
||
- `--enable-auto-tool-choice --tool-call-parser qwen3_xml` (the Avesed /
|
||
Quark chat template renders the Qwen3 XML tool format, and qwen3_xml
|
||
is the matching parser).
|
||
- Custom chat template `/home/dank/models/qwen36-chat_template-tools.jinja`:
|
||
derived from the repo's `chat_template.jinja` (the tokenizer_config.json
|
||
ships no template, so vLLM would use its built-in one, which does not
|
||
render the XML tool instructions). The custom template adds:
|
||
- a concrete WORKED EXAMPLE built from the actual tool schema
|
||
(name + parameter keys) — models follow concrete examples much
|
||
better than the generic placeholder;
|
||
- tool requests default to non-thinking (`enable_thinking` unset + tools
|
||
present -> empty think block). Qwen3 tool calls only work reliably
|
||
with thinking off; normal requests keep thinking.
|
||
|
||
Known pitfall: the earlier Avesed model still emitted malformed XML even
|
||
with the strong template (`<function=get_weather(city=...)>`); that was
|
||
model behavior, which motivated the swap.
|
||
|
||
### 2. Model swap: Avesed INT8 W8A8 -> Quark W8A8 INT8
|
||
|
||
`frank-zealot` now serves
|
||
`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` (AMD Quark 0.11.1, ~35 GB,
|
||
vLLM-native expert layout). The Avesed compressed-tensors quant's
|
||
Qwen3-XML tool fidelity was poor. Verified with the Quark model: plain
|
||
tool requests (no chat_template_kwargs) return clean
|
||
`tool_calls: [{"name": "get_weather", "arguments": {"city": "Paris"}}]`;
|
||
non-tool requests keep reasoning. The Avesed checkpoint stays in the HF
|
||
cache for rollback.
|
||
|
||
### 3. Batch 8192
|
||
|
||
`MAX_NUM_BATCHED_TOKENS` lowered from 16384 to 8192 (user request).
|
||
vLLM has no separate ubatch knob; the scheduler issues one batch per
|
||
step up to this token cap.
|
||
|
||
### 4. Prefix caching enabled
|
||
|
||
Added `--enable-prefix-caching`. The model is hybrid (GatedDeltaNet +
|
||
full attention); vLLM keeps prefix caching **opt-in** for hybrid models
|
||
(`default_prefix_caching = is_prefix_caching_supported and not
|
||
is_hybrid`), so it was off by default. Enabling it reuses the KV of
|
||
repeated prompt prefixes. Measured on a 2023-token prefix (2026-08-17
|
||
evening): 4.6s cold -> 1.1s cached (~4x TTFT), output unchanged. This is
|
||
the main TTFT lever for pi, which re-sends the same system prompt + tool
|
||
schema every turn. Toggled via `ENABLE_PREFIX_CACHING` in the env file.
|
||
---
|