serverAppConfigs/docs/frank-redo-2026-08-17.md

Frank Program Redo — 2026-08-17

This page records the full redo of the inference fleet on 2026-08-17. The pipeline-parallel split and the FastFlowLM NPU endpoint were retired. The old scripts moved to old/scripts/ in this repository.

New Topology

Service	Host	Runtime	Model	Port
frank-default-large	Dankserver	llama.cpp (patched FastMTP)	HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS	7770
frank-default-small	Dankclaw	vLLM ROCm	Avesed Qwen3.6-35B-A3B INT8 W8A8	52625

The RTX 3090 runs the whole 27B model standalone. No RPC worker, no tensor split, no NPU.

Models

frank-default-large — HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF:IQ4_XS

• Base: Qwen3.8-27B (dense 27B, 64 layers: 48 GatedDeltaNet + 16 gated-attention, hidden 5120, 24 Q heads / 4 KV heads, head dim 256, native 262144 ctx).
• Quant: IQ4_XS (4.60 BPW, 15.71 GB). Every GGUF in the repo preserves the native embedded MTP/NextN head.
• FastMTP draft sidecar: ...FastMTP-32K.gguf (903 MB). Requires the patched llama.cpp build (commit 4df29be4f4c3673f428170fda944a5b19f743bb8
  • HauhauCS-FastMTP-llama.cpp.patch). Without the patch, the draft load fails with expected 5120, 248320, got 5120, 32768.
• Spec chain: FastMTP (--spec-type draft-mtp, --spec-draft-n-max 2, --spec-draft-ngl all, --spec-draft-p-min 0) then ngram-mod fallback (48/64/24).

frank-default-small — Avesed/Qwen3.6-35B-A3B-INT8-W8A8

• Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40 layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx).
• Quant: compressed-tensors INT8 W8A8 (symmetric per-channel weights, dynamic per-token activations). Expert FFNs quantized; attention, router, DeltaNet, lm_head kept bf16. Weights ~38 GB.
• Served with vLLM ROCm, ngram speculative decoding.

Context Math (frank-default-large, RTX 3090)

KV per token: 16 attention layers x 4 KV heads x 256 dim. K q8_0 = 1088 B/token, V f16 = 2048 B/token, total 50176 B/token.

Budget (MiB): 24576 total - 14982 weights - 861 draft - 1024 fixed (CUDA context, compute graph, scratch, draft KV) - headroom.

Context	KV (MiB)	Free after load
114688 (112K)	5488	2221 (2.17 GiB)
122880 (120K)	5880	1829 (1.79 GiB)
131072 (128K)	6272	1437 (1.40 GiB)
147456 (144K)	7056	653 (0.64 GiB)

Production choice: 131072 (128K) — the maximum that keeps about 1.4 GiB headroom for checkpoints. Lower CTX_SIZE in scripts/frank-default-large.env to 122880 for ~1.8 GiB headroom.

Install Runbook

Dankserver (frank-default-large)

1. Stop the old worker: sudo systemctl stop frank-pp-worker.
2. Download the model and draft to /home/dank/models/hauhaucs-qwen38/.
3. Build the patched llama.cpp:

   git clone https://github.com/ggerganov/llama.cpp /home/dank/llama.cpp-fastmtp
   cd /home/dank/llama.cpp-fastmtp
   git checkout 4df29be4f4c3673f428170fda944a5b19f743bb8
   curl -L -o HauhauCS-FastMTP-llama.cpp.patch <HF patch URL>
   git apply HauhauCS-FastMTP-llama.cpp.patch
   cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release \
     -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON
   cmake --build build --config Release -j$(nproc)
   

   GGML_CUDA_FA_ALL_QUANTS=ON is mandatory. Without it, prefill collapses ~12x (the q8_0 KV + flash attention falls back to a slow path; measured 95 t/s vs 1300+ t/s at 8K on the 3090).
4. Symlink the unit, reload, enable, restart:

   sudo ln -sf /home/dank/vLLM/scripts/frank-default-large.service /etc/systemd/system/
   sudo systemctl daemon-reload && sudo systemctl enable frank-default-large
   sudo systemctl restart frank-default-large
   

5. Verify: curl -sf http://127.0.0.1:7770/health and a chat completion.
6. Confirm VRAM headroom with nvidia-smi.

Dankclaw (frank-default-small)

1. Stop the old services: sudo systemctl stop frank-default-large frank-default-small (retired PP head and NPU endpoint).
2. Install uv and create the venv:

   curl -LsSf https://astral.sh/uv/install.sh | sh
   uv venv --python /usr/bin/python3.12 /home/dank/vllm-venv
   uv pip install --python /home/dank/vllm-venv vllm --torch-backend=auto
   

   The resolver pulls the AMD ROCm wheel set (torch + amd-torch-device-gfx1151 + the vllm rocm variant) from the AMD index.
3. Cache the model (default HF cache):

   python3 -c "from huggingface_hub import snapshot_download; snapshot_download('Avesed/Qwen3.6-35B-A3B-INT8-W8A8')"
   

4. Install the ROCm wheel set and the flash-attention triton path:

   uv pip install torch --torch-backend=auto
   uv pip install "vllm==0.27.1+rocm723" --index-url https://wheels.vllm.ai/rocm/ \
     --extra-index-url https://pypi.org/simple --index-strategy unsafe-best-match --no-build-isolation
   cp -r <ciru-venv>/site-packages/flash_attn* <vllm-venv>/site-packages/
   # add a flash_attn_2_cuda stub module (ROCm build has no compiled .so)
   

   Set FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE (env file). Without it the vision encoder falls back to torch SDPA and engine init OOMs.
5. Symlink the unit, reload, enable, restart:

   sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/
   sudo systemctl daemon-reload && sudo systemctl enable frank-default-small
   sudo systemctl restart frank-default-small
   

6. Verify: curl -sf http://127.0.0.1:52625/health and a chat completion.

Fallback: vLLM source build for gfx1151

If the prebuilt ROCm vLLM wheel cannot run on gfx1151, build from source. The retired CIRU package documents the proven recipe: /home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/scripts/build-vllm-gfx1151.sh (branch ciru/ling3-gfx1151-rocm715, index https://rocm.nightlies.amd.com/whl-multi-arch/). The upstream starting point needed a five-line ROCm sampler fix to launch on gfx1151; newer vLLM releases may already include it.

Configuration

• models.json (pi): two providers — llamacpp (dankserver) at http://192.168.0.235:7770/v1 and vllm (dankclaw) at http://192.168.0.143:52625/v1.
• HF token: ~/.cache/huggingface/token on both machines (mode 600).
• OpenWebUI (sibling repo): retarget OPENAI_API_BASE_URL to the Dankserver endpoint.