feat(llm): rename inference services dropping 'default' from names

frank-default-large -> frank-large (Dankclaw :7770), frank-default-small -> frank-small (Dankserver :7770), frank-default-sightsound -> frank-npu (Dankclaw :7771). Units, env files, serve scripts, deploy-plan mappings, deploy-remote dispatch (stop/disable old units + relink + enable new), and the test seams move together. Retired frank-default-npu.* artifacts moved to scripts/.AGENTTRASH/ (tracked-in-history; ignored on disk). Served aliases become frank-large / frank-small via --alias; the NPU endpoint serves the whole FLM registry, so its rename changes no served tags.
2026-09-03 16:15:43 -07:00
1 parent 1e491fa69f
commit 69f87c8483
@@ -0,0 +1,165 @@
+---
+type: configuration
+title: FreeToken frank-default-small Runbook
+description: FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-default-small on Dankserver (RTX 3090).
+tags:
+  - freetoken
+  - dankserver
+  - frank-default-small
+  - qwen3.6
+  - moe
+  - fp8
+timestamp: 2026-08-30
+---
+
+# FreeToken frank-default-small Runbook
+
+The `frank-default-small` service on Dankserver is served by
+**FreeToken v0.1.2** — an edge-native MoE serving engine that keeps
+expert weights in host RAM with an LRU GPU cache. Model history on this
+runtime: official `Qwen/Qwen3.6-35B-A3B-FP8` (since 2026-09-02) →
+`batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (2026-08-30,
+same format and pool budget; re-landed 2026-09-01 after two
+compressed-tensors FP8 candidates failed to load — see Quantization
+Compatibility). The engine itself replaced the llama.cpp
+Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment.
+
+## Model
+
+| Field | Value |
+|-------|-------|
+| HuggingFace | `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` |
+| Lineage | Qwen/Qwen3.6-35B-A3B → huihui-ai abliterated → this FP8 (quantizer batsclamp) |
+| Architecture | `qwen3_5_moe` (Qwen3_5MoeForConditionalGeneration) |
+| Quant | native modelopt block-fp8 e4m3, dynamic activations, 128×128 `weight_block_size`, bf16 `weight_scale_inv` — byte-format identical to the official FP8 repo |
+| BF16 kept | lm_head, embed_tokens, all norms, router gates, mamba state params, visual encoder, MTP module |
+| Experts | un-fused per-expert layout (`mlp.experts.{0..255}.*`) matching official Qwen FP8 — same FreeToken loader path |
+| Size on disk | ~35 GB (26 safetensors shards) |
+| Path | `/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8` |
+| Layers | 40 (10 full-attention + 30 GDN/linear-attention) |
+| Experts | 256 per layer, 8 used per token |
+| KV heads | 2 × 256 dim |
+| Context | 262,144 (model max) |
+| Chat template | froggeric/Qwen-Fixed-Chat-Templates (fixed Qwen3.6 template, shipped in `tokenizer_config.json`; FreeToken reads it from the checkpoint — no override) |
+
+**Important:** this is the Huihui *abliterated* finetune (refusals
+removed) — not the base Qwen weights served before 2026-08-30, and not
+the earlier Tiel-Coder finetune. Expect a direct-answers assistant.
+
+## Runtime
+
+| Field | Value |
+|-------|-------|
+| Engine | FreeToken v0.1.2 (`ft` CLI) |
+| Venv | `/home/dank/freetoken/.venv` (uv-managed CPython 3.12) |
+| Dependencies | torch 2.11.0+cu130, flashinfer 0.6.17, apache-tvm-ffi |
+| CUDA toolchain | pip-installed nvidia-cuda-nvcc/crt/nvvm/runtime 13.2.86 |
+| API | OpenAI-compatible (`/v1/chat/completions`, `/v1/models`) |
+
+## Pool Configuration
+
+The GPU budget (~17.6 GiB after weights + fixed caches) is split
+MoE-first between expert residency and KV pages:
+
+| Pool | Size | VRAM | Notes |
+|------|------|------|-------|
+| MoE cache | 2,048 slots (20%) | 6.0 GiB | LRU expert cache, out of 10,240 total |
+| KV cache | 453,005 tokens | 8.6 GiB | ~20 KiB/token (10 attn layers only) |
+| Mamba state | 24 slots | 1.4 GiB | Per-sequence GDN state (caps concurrency) |
+
+### Live Resize
+
+KV and MoE can be re-partitioned without restart:
+
+```bash
+ft ctl --base-url http://127.0.0.1:7770 cache --kv N --moe M --wait 300
+```
+
+- **Growing KV** (shrinking MoE): works live.
+- **Shrinking KV** (growing MoE): requires restart (old + new pools
+  cannot coexist in VRAM during the swap → OOM).
+- The fit check refuses overcommit: if the plan doesn't fit, the old
+  cache is retained and serving continues.
+
+### Benchmark Results (2026-09-02, base FP8 checkpoint)
+
+See [[freetoken-benchmark-sweep]] for the full bs1/bs3/bs4/bs6 sweep
+across KV pool sizes 8k–700k.
+
+At the production config (20% residency, bs4):
+
+| Metric | Value |
+|--------|-------|
+| bs1 decode | ~30 tok/s |
+| bs4 aggregate | ~87 tok/s |
+| bs4 per-request | ~22 tok/s |
+| Prefill (4k) | ~1,130 tok/s |
+
+## Quantization Compatibility
+
+FreeToken v0.1.2 supports these safetensor quants for `qwen3_5_moe`:
+
+| Format | Detection | Works |
+|--------|-----------|-------|
+| BF16 (no quant config) | default | ✅ (experts too large for 60 GB RAM) |
+| modelopt block-fp8 (128×128) | `quant_method: "fp8"` + `weight_block_size` | ✅ (current) |
+| NVFP4 (modelopt) | `quant_algo: "NVFP4"` | ✅ (best for VRAM) |
+| NVFP4 (compressed-tensors) | 4-bit float, group_size=16, strategy=tensor_group | ✅ |
+| Per-tensor FP8 (modelopt) | scalar `weight_scale` | ✅ |
+| Per-channel FP8 (compressed-tensors) | `weight_scale: [O, 1]` | ❌ loader bug |
+| Block-fp8 128×128 (compressed-tensors) | `quant_method: "compressed-tensors"` + `block_structure: [128, 128]` | ❌ dequant shape bug |
+| Quark W8A8 INT8 | per-channel scales | ❌ loader bug |
+| GGUF | only `gemma4` arch | ❌ for qwen3_5_moe |
+
+**2026-09-01 live verification (Dankserver):** both compressed-tensors
+FP8 shapes fail on FreeToken v0.1.2, each a clean exit at load (37 GB
+fits RAM — no swap-thrash). `ornith-ai/Ornith-1.5-35B-A3B-FP8`
+(per-channel) aborts in expert-source resolution:
+`ValueError: Missing MoE expert source layers: {'gate_up': ...}` — its
+expert tensors do not follow the `mlp.experts.{N}.gate_proj/up_proj/
+down_proj` naming the MoE provider scans for.
+`lamianlbe/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive-fp8` (block
+128×128, standard naming) resolves experts but aborts in dequant:
+`RuntimeError: The size of tensor a (2048) must match the size of
+tensor b (...)` in `_dequant_fp8_weight`, which multiplies
+`weight × weight_scale` directly instead of expanding the block scales.
+modelopt block-fp8 remains the only FP8 path that loads.
+
+## Files
+
+| File | Purpose |
+|------|---------|
+| `scripts/serve-frank-small.sh` | Serve script (launches `ft serve`) |
+| `scripts/frank-default-small.env` | Tunables (model path, pool sizes) |
+| `scripts/frank-default-small.service` | Systemd unit |
+
+## Limitations
+
+- **Model identity**: serves the Huihui abliterated FP8 finetune, not the Tiel-Coder coder finetune from the llama.cpp era.
+- **JIT compilation**: first startup compiles flashinfer/tvm-ffi kernels
+  (~5–15 min cold). Cached afterward in `~/.cache/flashinfer` and
+  `~/.cache/tvm-ffi`.
+- **CUDA toolchain**: driver-only install (no system CUDA toolkit). The
+  venv provides pip-installed nvcc 13.2.86 for JIT compilation. All
+  nvidia-cu13 packages must be at the same version.
+- **Env var collisions**: never name tunables `TEMP`, `HOST`, or
+  `GPU_MAX_HEAP_SIZE` (HIP runtime reads these).
+
+## Operational quirks
+
+- **`model` field is required** in `/v1/chat/completions` bodies —
+  FreeToken rejects requests without it with a 422 (`Field required`),
+  unlike llama.cpp's server which defaults to the loaded model. Send
+  `"model": "frank-default-small"`.
+- Load on this swap took well under the old llama.cpp numbers:
+  weights → experts (parallel build) → CUDA graph capture → Uvicorn
+  in ~2 minutes; the 2026-08-30 swap verified ~40 tok/s decode on a
+  282-token completion.
+
+## Citations
+
+- FreeToken paper: arXiv 2608.16157
+- FreeToken PyPI: `freetoken[accel]` v0.1.2
+- HuggingFace model: `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8`
+- Upstream weights: `huihui-ai/Huihui-Qwen3.6-35B-A3B-abliterated`, `Qwen/Qwen3.6-35B-A3B-FP8`
+- Fixed chat template: `froggeric/Qwen-Fixed-Chat-Templates`