Frank
55f87fcdce
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
224 lines
9.6 KiB
Markdown
224 lines
9.6 KiB
Markdown
---
|
||
type: configuration
|
||
title: FreeToken frank-zealot Runbook
|
||
description: RETIRED 2026-09-15 - FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-zealot on Dankserver (RTX 3090 era).
|
||
tags:
|
||
- freetoken
|
||
- dankserver
|
||
- frank-zealot
|
||
- qwen3.6
|
||
- moe
|
||
- fp8
|
||
timestamp: 2026-08-30
|
||
---
|
||
|
||
# FreeToken frank-zealot Runbook
|
||
|
||
> **RETIRED 2026-09-15.** The RTX 3090 left Dankserver and the successor
|
||
> [[frank-medium-rocmfpx]] (charlie12345/ROCmFPX llama.cpp on 2x Radeon AI
|
||
> PRO R9700) took the :7770 endpoint and the port. This page is kept as the
|
||
> retirement record of the FreeToken/CUDA era; the unit was archived to
|
||
> `~/.AGENTTRASH/20260915-frank-zealot-retire/host-units/` and the repo files
|
||
> to `scripts/.AGENTTRASH/`.
|
||
|
||
The `frank-zealot` service on Dankserver is served by
|
||
**FreeToken v0.1.2** — an edge-native MoE serving engine that keeps
|
||
expert weights in host RAM with an LRU GPU cache. Model history on this
|
||
runtime: official `Qwen/Qwen3.6-35B-A3B-FP8` (since 2026-09-02) →
|
||
`batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (2026-08-30,
|
||
same format and pool budget; re-landed 2026-09-01 after two
|
||
compressed-tensors FP8 candidates failed to load — see Quantization
|
||
Compatibility). The engine itself replaced the llama.cpp
|
||
Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment.
|
||
|
||
## Model
|
||
|
||
| Field | Value |
|
||
|-------|-------|
|
||
| HuggingFace | `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` |
|
||
| Lineage | Qwen/Qwen3.6-35B-A3B → huihui-ai abliterated → this FP8 (quantizer batsclamp) |
|
||
| Architecture | `qwen3_5_moe` (Qwen3_5MoeForConditionalGeneration) |
|
||
| Quant | native modelopt block-fp8 e4m3, dynamic activations, 128×128 `weight_block_size`, bf16 `weight_scale_inv` — byte-format identical to the official FP8 repo |
|
||
| BF16 kept | lm_head, embed_tokens, all norms, router gates, mamba state params, visual encoder, MTP module |
|
||
| Experts | un-fused per-expert layout (`mlp.experts.{0..255}.*`) matching official Qwen FP8 — same FreeToken loader path |
|
||
| Size on disk | ~35 GB (26 safetensors shards) |
|
||
| Path | `/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8` |
|
||
| Layers | 40 (10 full-attention + 30 GDN/linear-attention) |
|
||
| Experts | 256 per layer, 8 used per token |
|
||
| KV heads | 2 × 256 dim |
|
||
| Context | 262,144 (model max) |
|
||
| Chat template | froggeric/Qwen-Fixed-Chat-Templates (fixed Qwen3.6 template, shipped in `tokenizer_config.json`; FreeToken reads it from the checkpoint — no override) |
|
||
|
||
**Important:** this is the Huihui *abliterated* finetune (refusals
|
||
removed) — not the base Qwen weights served before 2026-08-30, and not
|
||
the earlier Tiel-Coder finetune. Expect a direct-answers assistant.
|
||
|
||
## Runtime
|
||
|
||
| Field | Value |
|
||
|-------|-------|
|
||
| Engine | FreeToken v0.1.2 (`ft` CLI) |
|
||
| Venv | `/home/dank/freetoken/.venv` (uv-managed CPython 3.12) |
|
||
| Dependencies | torch 2.11.0+cu130, flashinfer 0.6.17, apache-tvm-ffi |
|
||
| CUDA toolchain | pip-installed nvidia-cuda-nvcc/crt/nvvm/runtime 13.2.86 |
|
||
| API | OpenAI-compatible (`/v1/chat/completions`, `/v1/models`) |
|
||
|
||
## Pool Configuration
|
||
|
||
The GPU budget (~17.6 GiB after weights + fixed caches) is split
|
||
MoE-first between expert residency and KV pages:
|
||
|
||
| Pool | Size | VRAM | Notes |
|
||
|------|------|------|-------|
|
||
| MoE cache | 2,048 slots (20%) | 6.0 GiB | LRU expert cache, out of 10,240 total |
|
||
| KV cache | 453,005 tokens | 8.6 GiB | ~20 KiB/token (10 attn layers only) |
|
||
| Mamba state | 24 slots | 1.4 GiB | Per-sequence GDN state (caps concurrency) |
|
||
|
||
### Live Resize
|
||
|
||
KV and MoE can be re-partitioned without restart:
|
||
|
||
```bash
|
||
ft ctl --base-url http://127.0.0.1:7770 cache --kv N --moe M --wait 300
|
||
```
|
||
|
||
- **Growing KV** (shrinking MoE): works live.
|
||
- **Shrinking KV** (growing MoE): requires restart (old + new pools
|
||
cannot coexist in VRAM during the swap → OOM).
|
||
- The fit check refuses overcommit: if the plan doesn't fit, the old
|
||
cache is retained and serving continues.
|
||
|
||
### Benchmark Results (2026-09-02, base FP8 checkpoint)
|
||
|
||
See [[freetoken-benchmark-sweep]] for the full bs1/bs3/bs4/bs6 sweep
|
||
across KV pool sizes 8k–700k.
|
||
|
||
At the production config (20% residency, bs4):
|
||
|
||
| Metric | Value |
|
||
|--------|-------|
|
||
| bs1 decode | ~30 tok/s |
|
||
| bs4 aggregate | ~87 tok/s |
|
||
| bs4 per-request | ~22 tok/s |
|
||
| Prefill (4k) | ~1,130 tok/s |
|
||
|
||
## Quantization Compatibility
|
||
|
||
FreeToken v0.1.2 supports these safetensor quants for `qwen3_5_moe`:
|
||
|
||
| Format | Detection | Works |
|
||
|--------|-----------|-------|
|
||
| BF16 (no quant config) | default | ✅ (experts too large for 60 GB RAM) |
|
||
| modelopt block-fp8 (128×128) | `quant_method: "fp8"` + `weight_block_size` | ✅ (current) |
|
||
| NVFP4 (modelopt) | `quant_algo: "NVFP4"` | ✅ (best for VRAM) |
|
||
| NVFP4 (compressed-tensors) | 4-bit float, group_size=16, strategy=tensor_group | ✅ |
|
||
| Per-tensor FP8 (modelopt) | scalar `weight_scale` | ✅ |
|
||
| Per-channel FP8 (compressed-tensors) | `weight_scale: [O, 1]` | ❌ loader bug |
|
||
| Block-fp8 128×128 (compressed-tensors) | `quant_method: "compressed-tensors"` + `block_structure: [128, 128]` | ❌ dequant shape bug |
|
||
| Quark W8A8 INT8 | per-channel scales | ❌ loader bug |
|
||
| GGUF | only `gemma4` arch | ❌ for qwen3_5_moe |
|
||
|
||
**2026-09-01 live verification (Dankserver):** both compressed-tensors
|
||
FP8 shapes fail on FreeToken v0.1.2, each a clean exit at load (37 GB
|
||
fits RAM — no swap-thrash). `ornith-ai/Ornith-1.5-35B-A3B-FP8`
|
||
(per-channel) aborts in expert-source resolution:
|
||
`ValueError: Missing MoE expert source layers: {'gate_up': ...}` — its
|
||
expert tensors do not follow the `mlp.experts.{N}.gate_proj/up_proj/
|
||
down_proj` naming the MoE provider scans for.
|
||
`lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8` (block
|
||
128×128, standard naming) resolves experts but aborts in dequant:
|
||
`RuntimeError: The size of tensor a (2048) must match the size of
|
||
tensor b (...)` in `_dequant_fp8_weight`, which multiplies
|
||
`weight × weight_scale` directly instead of expanding the block scales.
|
||
modelopt block-fp8 remains the only FP8 path that loads.
|
||
|
||
## Sampling
|
||
|
||
**frank-zealot samples by default, and our config never said so.**
|
||
`scripts/serve-frank-zealot.sh` passes no sampling arguments at all — only
|
||
`--model`, `--served-model-name`, `--host`, `--port`, `--moe-cache-size`,
|
||
`--num-tokens` and `--max-running-requests`. The behaviour comes from the
|
||
runtime's own default:
|
||
|
||
```python
|
||
# freetoken/server/args.py:31
|
||
sampling_defaults: str = "model"
|
||
```
|
||
|
||
`--sampling-defaults model` fills `temperature` / `top_k` / `top_p` from the
|
||
**checkpoint's** `generation_config.json` for any request that omits them.
|
||
This checkpoint ships:
|
||
|
||
```json
|
||
{ "do_sample": true, "temperature": 1.0, "top_k": 20, "top_p": 0.95 }
|
||
```
|
||
|
||
Verified behaviourally 2026-09-12 — three identical requests with **no
|
||
sampling parameters sent at all** produced three different reasoning traces
|
||
(sha `13ca02…`/487 chars, `d49928…`/540, `7b81b8…`/577). Greedy decoding
|
||
would be byte-identical.
|
||
|
||
### This is the opposite of frank-dragoon
|
||
|
||
| | frank-zealot (FreeToken) | frank-dragoon (halogen) |
|
||
|---|---|---|
|
||
| Silent client (no `temperature`) | **samples** at temp 1.0 / top_k 20 / top_p 0.95 | **greedy** — no server-side sampler defaults |
|
||
| Deterministic across runs | No | Yes |
|
||
| Where the default comes from | checkpoint `generation_config.json` | none; the request decides |
|
||
|
||
A client that talks to both endpoints and sends no sampler gets
|
||
non-deterministic answers from :7770 on Dankserver and reproducible ones from
|
||
:7770 on Dankclaw. If a workflow needs determinism from frank-zealot it must
|
||
send `temperature: 0` explicitly, or the service must be relaunched with
|
||
`--sampling-defaults none` (framework defaults only).
|
||
|
||
### Related request behaviour
|
||
|
||
- **Thinking consumes the budget.** At `max_tokens: 200` all three probe runs
|
||
returned `finish_reason: "length"` with **empty `content`** and the whole
|
||
budget spent in `reasoning_content`. Budget for thinking, or the answer
|
||
never appears.
|
||
- **The API does not validate unknown fields** — a request carrying
|
||
`totally_bogus_field: 123` returns 200, as does
|
||
`chat_template_kwargs: {"preserve_thinking": true}`. A 200 is not evidence
|
||
that a key was understood.
|
||
|
||
|
||
## Files
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `scripts/serve-frank-zealot.sh` | Serve script (launches `ft serve`) |
|
||
| `scripts/frank-zealot.env` | Tunables (model path, pool sizes) |
|
||
| `scripts/frank-zealot.service` | Systemd unit |
|
||
|
||
## Limitations
|
||
|
||
- **Model identity**: serves the Huihui abliterated FP8 finetune, not the Tiel-Coder coder finetune from the llama.cpp era.
|
||
- **JIT compilation**: first startup compiles flashinfer/tvm-ffi kernels
|
||
(~5–15 min cold). Cached afterward in `~/.cache/flashinfer` and
|
||
`~/.cache/tvm-ffi`.
|
||
- **CUDA toolchain**: driver-only install (no system CUDA toolkit). The
|
||
venv provides pip-installed nvcc 13.2.86 for JIT compilation. All
|
||
nvidia-cu13 packages must be at the same version.
|
||
- **Env var collisions**: never name tunables `TEMP`, `HOST`, or
|
||
`GPU_MAX_HEAP_SIZE` (HIP runtime reads these).
|
||
|
||
## Operational quirks
|
||
|
||
- **`model` field is required** in `/v1/chat/completions` bodies —
|
||
FreeToken rejects requests without it with a 422 (`Field required`),
|
||
unlike llama.cpp's server which defaults to the loaded model. Send
|
||
`"model": "frank-zealot"`.
|
||
- Load on this swap took well under the old llama.cpp numbers:
|
||
weights → experts (parallel build) → CUDA graph capture → Uvicorn
|
||
in ~2 minutes; the 2026-08-30 swap verified ~40 tok/s decode on a
|
||
282-token completion.
|
||
|
||
## Citations
|
||
|
||
- FreeToken paper: arXiv 2608.16157
|
||
- FreeToken PyPI: `freetoken[accel]` v0.1.2
|
||
- HuggingFace model: `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8`
|
||
- Upstream weights: `huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated`, `Qwen/Qwen3.6-35B-A3B-FP8`
|
||
- Fixed chat template: `froggeric/Qwen-Fixed-Chat-Templates` |