Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
10 KiB
type, description, tags, timestamp
| type | description | tags | timestamp | |||
|---|---|---|---|---|---|---|
| reference | The day the whole inference fleet was rebuilt: what came down, what replaced it, and why it was done that way. |
|
2026-09-03 |
Frank Program Redo — 2026-08-17
This page records the full redo of the inference fleet on 2026-08-17.
The pipeline-parallel split and the FastFlowLM NPU endpoint were retired.
The old scripts moved to old/scripts/ in this repository.
HISTORY — the topology below is two fleet generations dead. Today the fleet is three endpoints, each owned by its page: frank-archon (
192.168.0.235:7770), frank-dragoon (192.168.0.143:7770) and frank-zealot (192.168.0.143:7771). No vLLM and no ROCm endpoint serves anything; there is no pipeline parallelism and no NPU endpoint.
New Topology
| Service | Host | Runtime | Model | Port |
|---|---|---|---|---|
| frank-dragoon | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
| frank-zealot | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
Names as the 2026-08-17 commit wrote them: frank-default-large on Dankserver
(git show d9af7f1:scripts/frank-default-large.env:20,22 → ALIAS=frank-default-large,
LISTEN_PORT=7770) and frank-default-small on Dankclaw
(git show d9af7f1:scripts/frank-default-small.env:16,18 → ALIAS=frank-default-small,
LISTEN_PORT=52625); the Protoss ids date from 2026-10-06 → 10-10.
The RTX 3090 runs the whole 27B model standalone. No RPC worker, no tensor split, no NPU.
Models
frank-dragoon — HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF:IQ4_XS
- Base: Qwen3.8-27B (dense 27B, 64 layers: 48 GatedDeltaNet + 16 gated-attention, hidden 5120, 24 Q heads / 4 KV heads, head dim 256, native 262144 ctx).
- Quant: IQ4_XS (4.60 BPW, 15.71 GB). Every GGUF in the repo preserves the native embedded MTP/NextN head.
- FastMTP draft sidecar:
...FastMTP-32K.gguf(903 MB). Requires the patched llama.cpp build (commit4df29be4f4c3673f428170fda944a5b19f743bb8HauhauCS-FastMTP-llama.cpp.patch). Without the patch, the draft load fails withexpected 5120, 248320, got 5120, 32768.
- Spec chain: FastMTP (
--spec-type draft-mtp,--spec-draft-n-max 2,--spec-draft-ngl all,--spec-draft-p-min 0) then ngram-mod fallback (48/64/24).
frank-zealot — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
- Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40 layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx).
- Quant: AMD Quark W8A8 INT8 (per-channel static weights, per-token dynamic activations). MoE experts stored as per-expert nn.Linear triplets that match vLLM's FusedMoE mapping exactly. Weights ~35 GB; the vision tower, router, lm_head stay BF16.
- Served with vLLM ROCm, ngram speculative decoding, and tool calling.
- Swap history: the initial deployment used
Avesed/Qwen3.6-35B-A3B-INT8-W8A8(compressed-tensors INT8). Its tool calls were malformed (the model emitted<function=get_weather(city=...)>instead of the Qwen3 XML structure). The Quark quant follows the format cleanly; swapped in the evening of 2026-08-17.
Context Math (frank-dragoon, RTX 3090)
KV per token: 16 attention layers x 4 KV heads x 256 dim. K q8_0 = 1088 B per layer (4 KV heads × 256 dim × 1.0625 B), V f16 = 2048 B per layer, ×16 layers = 50176 B/token total.
Budget (MiB): 24576 total - 14982 weights - 861 draft - 1024 fixed (CUDA context, compute graph, scratch, draft KV) - headroom.
| Context | KV (MiB) | Free after load |
|---|---|---|
| 114688 (112K) | 5488 | 2221 (2.17 GiB) |
| 122880 (120K) | 5880 | 1829 (1.79 GiB) |
| 131072 (128K) | 6272 | 1437 (1.40 GiB) |
| 147456 (144K) | 7056 | 653 (0.64 GiB) |
Production choice: 131072 (128K) — the maximum that keeps about
1.4 GiB headroom for checkpoints. Lower CTX_SIZE in
scripts/frank-dragoon.env to 122880 for ~1.8 GiB headroom.
Install Runbook
Dankserver (frank-dragoon)
- Stop the old worker:
sudo systemctl stop frank-pp-worker. - Download the model and draft to
/home/dank/models/hauhaucs-qwen38/. - Build the patched llama.cpp:
git clone https://github.com/ggerganov/llama.cpp /home/dank/llama.cpp-fastmtp cd /home/dank/llama.cpp-fastmtp git checkout 4df29be4f4c3673f428170fda944a5b19f743bb8 curl -L -o HauhauCS-FastMTP-llama.cpp.patch <HF patch URL> git apply HauhauCS-FastMTP-llama.cpp.patch cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release \ -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON cmake --build build --config Release -j$(nproc)GGML_CUDA_FA_ALL_QUANTS=ONis mandatory. Without it, prefill collapses ~12x (the q8_0 KV + flash attention falls back to a slow path; measured 95 t/s vs 1300+ t/s at 8K on the 3090). - Symlink the unit, reload, enable, restart:
sudo ln -sf /home/dank/vLLM/scripts/frank-dragoon.service /etc/systemd/system/ sudo systemctl daemon-reload && sudo systemctl enable frank-dragoon sudo systemctl restart frank-dragoon - Verify:
curl -sf http://127.0.0.1:7770/healthand a chat completion. - Confirm VRAM headroom with
nvidia-smi.
Dankclaw (frank-zealot)
- Stop the old services:
sudo systemctl stop frank-dragoon frank-zealot(retired PP head and NPU endpoint). - Install uv and create the venv:
The resolver pulls the AMD ROCm wheel set (torch + amd-torch-device-gfx1151 + the vllm rocm variant) from the AMD index.
curl -LsSf https://astral.sh/uv/install.sh | sh uv venv --python /usr/bin/python3.12 /home/dank/vllm-venv uv pip install --python /home/dank/vllm-venv vllm --torch-backend=auto - Cache the model (default HF cache):
python3 -c "from huggingface_hub import snapshot_download; snapshot_download('Avesed/Qwen3.6-35B-A3B-INT8-W8A8')" - Install the ROCm wheel set and the flash-attention triton path:
Set
uv pip install torch --torch-backend=auto uv pip install "vllm==0.27.1+rocm723" --index-url https://wheels.vllm.ai/rocm/ \ --extra-index-url https://pypi.org/simple --index-strategy unsafe-best-match --no-build-isolation cp -r <ciru-venv>/site-packages/flash_attn* <vllm-venv>/site-packages/ # add a flash_attn_2_cuda stub module (ROCm build has no compiled .so)FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE(env file). Without it the vision encoder falls back to torch SDPA and engine init OOMs. - Symlink the unit, reload, enable, restart:
sudo ln -sf /home/dank/vllmScripts/scripts/frank-zealot.service /etc/systemd/system/ sudo systemctl daemon-reload && sudo systemctl enable frank-zealot sudo systemctl restart frank-zealot - Verify:
curl -sf http://127.0.0.1:52625/healthand a chat completion.
Fallback: vLLM source build for gfx1151
If the prebuilt ROCm vLLM wheel cannot run on gfx1151, build from source.
The retired CIRU package documents the proven recipe:
/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/scripts/build-vllm-gfx1151.sh
(branch ciru/ling3-gfx1151-rocm715, index https://rocm.nightlies.amd.com/whl-multi-arch/).
The upstream starting point needed a five-line ROCm sampler fix to launch
on gfx1151; newer vLLM releases may already include it.
Configuration
models.json(pi): two providers —llamacpp (dankserver)at http://192.168.0.235:7770/v1 andvllm (dankclaw)at http://192.168.0.143:52625/v1. (The harness config today has three providers and no vLLM:~/.omp/agent/models.yml:14,94,173.)- HF token:
~/.cache/huggingface/tokenon both machines (mode 600). - OpenWebUI (sibling repo): retarget
OPENAI_API_BASE_URLto the Dankserver endpoint.
Evening Session (2026-08-17): tool calling, model swap, batch 8192
Three changes landed after the initial deployment:
1. Tool calling on frank-zealot (HTTP 400 fix)
Clients (pi) send tool_choice: "auto". vLLM rejects it with HTTP 400
unless the server enables tool calling. Fixes applied:
--enable-auto-tool-choice --tool-call-parser qwen3_xml(the Avesed / Quark chat template renders the Qwen3 XML tool format, and qwen3_xml is the matching parser).- Custom chat template
/home/dank/models/qwen36-chat_template-tools.jinja: derived from the repo'schat_template.jinja(the tokenizer_config.json ships no template, so vLLM would use its built-in one, which does not render the XML tool instructions). The custom template adds:- a concrete WORKED EXAMPLE built from the actual tool schema (name + parameter keys) — models follow concrete examples much better than the generic placeholder;
- tool requests default to non-thinking (
enable_thinkingunset + tools present -> empty think block). Qwen3 tool calls only work reliably with thinking off; normal requests keep thinking.
Known pitfall: the earlier Avesed model still emitted malformed XML even
with the strong template (<function=get_weather(city=...)>); that was
model behavior, which motivated the swap.
2. Model swap: Avesed INT8 W8A8 -> Quark W8A8 INT8
frank-zealot now serves
nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8 (AMD Quark 0.11.1, ~35 GB,
vLLM-native expert layout). The Avesed compressed-tensors quant's
Qwen3-XML tool fidelity was poor. Verified with the Quark model: plain
tool requests (no chat_template_kwargs) return clean
tool_calls: [{"name": "get_weather", "arguments": {"city": "Paris"}}];
non-tool requests keep reasoning. The Avesed checkpoint stays in the HF
cache for rollback.
3. Batch 8192
MAX_NUM_BATCHED_TOKENS lowered from 16384 to 8192 (user request).
vLLM has no separate ubatch knob; the scheduler issues one batch per
step up to this token cap.