Files

T

Frank 87721ab0fe rename frank-small → frank-secondary (Dankclaw llama.cpp :7771)

Cross-cutting rename of the current llama.cpp endpoint on the RTX 3090
(port 7771) from frank-small to frank-secondary:

- Service trio: scripts/{frank-small.env,frank-small.service,
  serve-frank-small.sh,download-frank-small.sh} → frank-secondary.*
- Test: tests/frank_small_unit.bats → tests/frank_secondary_unit.bats
- Runbooks: serverAppConfigs-docs/LLM/frank-small-*.md →
  frank-secondary-*.md (9 files); the FreeToken runbook stays
  freetoken-frank-small.md (it documents the retired CUDA-era service)
- Identifiers: FRANK_SMALL→FRANK_SECONDARY, llm_small→llm_secondary,
  LLM_SMALL_*→LLM_SECONDARY_*, deploy_llm_small→deploy_llm_secondary
- Historical references to the OLD frank-small (FreeToken/CUDA on the
  3090, and the Dankserver service that became frank-medium) are
  preserved verbatim — they describe what the service was called
  at the time.

2026-10-06 15:37:13 -07:00

9.3 KiB

Raw Blame History

type, description, tags, timestamp

type

description

Frank Program Redo — 2026-08-17

This page records the full redo of the inference fleet on 2026-08-17. The pipeline-parallel split and the FastFlowLM NPU endpoint were retired. The old scripts moved to old/scripts/ in this repository.

New Topology

Service	Host	Runtime	Model	Port
frank-large	Dankserver	llama.cpp (patched FastMTP)	HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS	7770
frank-secondary	Dankclaw	vLLM ROCm	nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8	52625

The RTX 3090 runs the whole 27B model standalone. No RPC worker, no tensor split, no NPU.

Models

frank-large — HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF:IQ4_XS

Base: Qwen3.8-27B (dense 27B, 64 layers: 48 GatedDeltaNet + 16 gated-attention, hidden 5120, 24 Q heads / 4 KV heads, head dim 256, native 262144 ctx).
Quant: IQ4_XS (4.60 BPW, 15.71 GB). Every GGUF in the repo preserves the native embedded MTP/NextN head.
FastMTP draft sidecar: ...FastMTP-32K.gguf (903 MB). Requires the patched llama.cpp build (commit 4df29be4f4c3673f428170fda944a5b19f743bb8
- HauhauCS-FastMTP-llama.cpp.patch). Without the patch, the draft load fails with expected 5120, 248320, got 5120, 32768.
Spec chain: FastMTP (--spec-type draft-mtp, --spec-draft-n-max 2, --spec-draft-ngl all, --spec-draft-p-min 0) then ngram-mod fallback (48/64/24).

frank-secondary — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8

Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40 layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx).
Quant: AMD Quark W8A8 INT8 (per-channel static weights, per-token dynamic activations). MoE experts stored as per-expert nn.Linear triplets that match vLLM's FusedMoE mapping exactly. Weights ~35 GB; the vision tower, router, lm_head stay BF16.
Served with vLLM ROCm, ngram speculative decoding, and tool calling.
Swap history: the initial deployment used Avesed/Qwen3.6-35B-A3B-INT8-W8A8 (compressed-tensors INT8). Its tool calls were malformed (the model emitted <function=get_weather(city=...)> instead of the Qwen3 XML structure). The Quark quant follows the format cleanly; swapped in the evening of 2026-08-17.

Context Math (frank-large, RTX 3090)

KV per token: 16 attention layers x 4 KV heads x 256 dim. K q8_0 = 1088 B/token, V f16 = 2048 B/token, total 50176 B/token.

Budget (MiB): 24576 total - 14982 weights - 861 draft - 1024 fixed (CUDA context, compute graph, scratch, draft KV) - headroom.

Context	KV (MiB)	Free after load
114688 (112K)	5488	2221 (2.17 GiB)
122880 (120K)	5880	1829 (1.79 GiB)
131072 (128K)	6272	1437 (1.40 GiB)
147456 (144K)	7056	653 (0.64 GiB)

Production choice: 131072 (128K) — the maximum that keeps about 1.4 GiB headroom for checkpoints. Lower CTX_SIZE in scripts/frank-large.env to 122880 for ~1.8 GiB headroom.

Install Runbook

Dankserver (frank-large)

Stop the old worker: sudo systemctl stop frank-pp-worker.
Download the model and draft to /home/dank/models/hauhaucs-qwen38/.

Build the patched llama.cpp:

git clone https://github.com/ggerganov/llama.cpp /home/dank/llama.cpp-fastmtp
cd /home/dank/llama.cpp-fastmtp
git checkout 4df29be4f4c3673f428170fda944a5b19f743bb8
curl -L -o HauhauCS-FastMTP-llama.cpp.patch <HF patch URL>
git apply HauhauCS-FastMTP-llama.cpp.patch
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release \
  -DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON
cmake --build build --config Release -j$(nproc)

GGML_CUDA_FA_ALL_QUANTS=ON is mandatory. Without it, prefill collapses ~12x (the q8_0 KV + flash attention falls back to a slow path; measured 95 t/s vs 1300+ t/s at 8K on the 3090).

Symlink the unit, reload, enable, restart:

sudo ln -sf /home/dank/vLLM/scripts/frank-large.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable frank-large
sudo systemctl restart frank-large

Verify: curl -sf http://127.0.0.1:7770/health and a chat completion.
Confirm VRAM headroom with nvidia-smi.

Dankclaw (frank-secondary)

Stop the old services: sudo systemctl stop frank-large frank-secondary (retired PP head and NPU endpoint).

Install uv and create the venv:

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv --python /usr/bin/python3.12 /home/dank/vllm-venv
uv pip install --python /home/dank/vllm-venv vllm --torch-backend=auto

The resolver pulls the AMD ROCm wheel set (torch + amd-torch-device-gfx1151 + the vllm rocm variant) from the AMD index.

Cache the model (default HF cache):

python3 -c "from huggingface_hub import snapshot_download; snapshot_download('Avesed/Qwen3.6-35B-A3B-INT8-W8A8')"

Install the ROCm wheel set and the flash-attention triton path:

uv pip install torch --torch-backend=auto
uv pip install "vllm==0.27.1+rocm723" --index-url https://wheels.vllm.ai/rocm/ \
  --extra-index-url https://pypi.org/simple --index-strategy unsafe-best-match --no-build-isolation
cp -r <ciru-venv>/site-packages/flash_attn* <vllm-venv>/site-packages/
# add a flash_attn_2_cuda stub module (ROCm build has no compiled .so)

Set FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE (env file). Without it the vision encoder falls back to torch SDPA and engine init OOMs.

Symlink the unit, reload, enable, restart:

sudo ln -sf /home/dank/vllmScripts/scripts/frank-secondary.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable frank-secondary
sudo systemctl restart frank-secondary

Verify: curl -sf http://127.0.0.1:52625/health and a chat completion.

Fallback: vLLM source build for gfx1151

If the prebuilt ROCm vLLM wheel cannot run on gfx1151, build from source. The retired CIRU package documents the proven recipe: /home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/scripts/build-vllm-gfx1151.sh (branch ciru/ling3-gfx1151-rocm715, index https://rocm.nightlies.amd.com/whl-multi-arch/). The upstream starting point needed a five-line ROCm sampler fix to launch on gfx1151; newer vLLM releases may already include it.

Configuration

models.json (pi): two providers — llamacpp (dankserver) at http://192.168.0.235:7770/v1 and vllm (dankclaw) at http://192.168.0.143:52625/v1.
HF token: ~/.cache/huggingface/token on both machines (mode 600).
OpenWebUI (sibling repo): retarget OPENAI_API_BASE_URL to the Dankserver endpoint.

Evening Session (2026-08-17): tool calling, model swap, batch 8192

Three changes landed after the initial deployment:

1. Tool calling on frank-secondary (HTTP 400 fix)

Clients (pi) send tool_choice: "auto". vLLM rejects it with HTTP 400 unless the server enables tool calling. Fixes applied:

--enable-auto-tool-choice --tool-call-parser qwen3_xml (the Avesed / Quark chat template renders the Qwen3 XML tool format, and qwen3_xml is the matching parser).
Custom chat template /home/dank/models/qwen36-chat_template-tools.jinja: derived from the repo's chat_template.jinja (the tokenizer_config.json ships no template, so vLLM would use its built-in one, which does not render the XML tool instructions). The custom template adds:
- a concrete WORKED EXAMPLE built from the actual tool schema (name + parameter keys) — models follow concrete examples much better than the generic placeholder;
- tool requests default to non-thinking (enable_thinking unset + tools present -> empty think block). Qwen3 tool calls only work reliably with thinking off; normal requests keep thinking.

Known pitfall: the earlier Avesed model still emitted malformed XML even with the strong template (<function=get_weather(city=...)>); that was model behavior, which motivated the swap.

2. Model swap: Avesed INT8 W8A8 -> Quark W8A8 INT8

frank-secondary now serves nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8 (AMD Quark 0.11.1, ~35 GB, vLLM-native expert layout). The Avesed compressed-tensors quant's Qwen3-XML tool fidelity was poor. Verified with the Quark model: plain tool requests (no chat_template_kwargs) return clean tool_calls: [{"name": "get_weather", "arguments": {"city": "Paris"}}]; non-tool requests keep reasoning. The Avesed checkpoint stays in the HF cache for rollback.

3. Batch 8192

MAX_NUM_BATCHED_TOKENS lowered from 16384 to 8192 (user request). vLLM has no separate ubatch knob; the scheduler issues one batch per step up to this token cap.

4. Prefix caching enabled

Added --enable-prefix-caching. The model is hybrid (GatedDeltaNet + full attention); vLLM keeps prefix caching opt-in for hybrid models (default_prefix_caching = is_prefix_caching_supported and not is_hybrid), so it was off by default. Enabling it reuses the KV of repeated prompt prefixes. Measured on a 2023-token prefix (2026-08-17 evening): 4.6s cold -> 1.1s cached (~4x TTFT), output unchanged. This is the main TTFT lever for pi, which re-sends the same system prompt + tool schema every turn. Toggled via ENABLE_PREFIX_CACHING in the env file.

9.3 KiB Raw Blame History