feat: replace llama.cpp Tiel-Coder with FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8

Replace the llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment with
FreeToken v0.1.2 — an edge-native MoE serving engine that keeps expert
weights in host RAM with an LRU GPU cache and paged KV with radix prefix
sharing.

Model: Qwen/Qwen3.6-35B-A3B-FP8 (modelopt block-fp8, 128×128 weight_block_size).
Architecture: qwen3_5_moe — 40 layers (10 attention + 30 GDN), 256 experts/8 used.

Production config:
- 2048 MoE cache slots (20% of 10240 expert instances, ~6.0 GiB VRAM)
- 453005 KV tokens (~8.6 GiB, ~20 KiB/token for 10 attn layers)
- 4 max-running-requests (mamba slot pool: 24 slots)
- bs4 decode: ~87 tok/s aggregate, ~30 tok/s bs1
- Prefill: ~1,130 tok/s at 4k prompt

Quirk: FreeToken v0.1.2 supports modelopt block-fp8, NVFP4, and per-tensor
FP8, but NOT compressed-tensors per-channel FP8 (Huihui/Ornith both failed).
Live KV resize via ft ctl cache grows KV by shrinking MoE; reverse needs restart.

Old llama.cpp scripts archived to .AGENTTRASH/.
Full benchmark sweep (bs1/bs3/bs4/bs6, 8k–700k KV) documented in
docs/LLM/freetoken-benchmark-sweep.md.

scripts/serve-frank-small.sh

@@ -1,23 +1,25 @@
 #!/usr/bin/env bash
-# Serve peculiar-ragdoll/Tiel-Coder-35B-A3B (UD-Q6_K_XL) as
-# frank-default-small via stock llama.cpp on Dankserver (RTX 3090).
+# Serve Qwen/Qwen3.6-35B-A3B-FP8 as frank-default-small via FreeToken
+# on Dankserver (RTX 3090, 60 GB RAM).
 #
-# Runtime: upstream llama.cpp (latest master, CUDA build with
-# -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86).
-# The model is a 35B-total / 8-active MoE hybrid (qwen35moe arch) with
-# SSM/Mamba layers (full_attention_interval=4: 10 attention + 30 SSM
-# layers out of 40).  256 experts, 8 used per token.
+# Runtime: FreeToken v0.1.2 (edge-native MoE serving engine).
+#   - Experts live in host RAM; an LRU cache of expert slots is resident
+#     on the GPU. Misses are streamed over PCIe.
+#   - Paged KV cache (radix prefix sharing).  Only the 10 full-attention
+#     layers page KV (~20 KiB/token); the 30 GDN/linear-attention layers
+#     hold constant-size per-sequence state (mamba slots).
+#   - OpenAI-compatible API (/v1/chat/completions, /v1/models).
 #
-# Expert offload: --cpu-moe keeps all 256 expert weights in CPU RAM.
-# Non-expert weights (attention, SSM, shared expert, embeddings, output)
-# are ~1.6 GB and fit entirely on the 24 GB RTX 3090.
+# Model: Qwen/Qwen3.6-35B-A3B-FP8 (safetensors, modelopt block-fp8,
+# 128x128 weight_block_size).  Architecture: qwen3_5_moe — 40 layers,
+# full_attention_interval=4 (10 attention + 30 GDN), 256 experts / 8 used,
+# 2 KV heads × 256 dim.  35 GB on disk.
 #
-# KV cache: Q8_0 for both K and V.  With CTX_SIZE=200700 (66 900 per
-# slot × 3 parallel) and 10 attention layers (2 KV heads × 256 dim),
-# the KV cache is ~2 GB — the requested 2 GB VRAM buffer.
-#
-# Model is loaded with mmap (default).  The 30 GB Q6_K_XL file is
-# memory-mapped; only accessed pages enter RAM.
+# Pool sizing: the GPU budget is split MoE-first between expert residency
+# and KV pages.  --moe-cache-size pins the number of resident expert
+# slots; --num-tokens pins the KV pool capacity.  Live resize is possible
+# via `ft ctl cache --kv N --moe M` (grows KV by shrinking MoE; shrinking
+# KV requires a restart due to the swap-during-rebuild memory cost).
 #
 # Tunables come from frank-default-small.env unless FRANK_SMALL_ENV_FILE
 # overrides that path.
@@ -33,55 +35,39 @@ if [[ -f "$ENV_FILE" ]]; then
 fi
 
 # --- defaults (used when the env file does not set them) -----------------
-: "${MODEL_DIR:=/home/dank/models/tiel-coder-35b}"
-: "${MODEL_FILE:=Tiel-Coder-35B-A3B-UD-Q6_K_XL.gguf}"
-: "${LLAMA_SERVER:=/home/dank/llama.cpp/build/bin/llama-server}"
+: "${MODEL_DIR:=/home/dank/models/qwen36-35b-a3b-fp8}"
 : "${ALIAS:=frank-default-small}"
 : "${BIND_ADDR:=0.0.0.0}"
 : "${LISTEN_PORT:=7770}"
-: "${CTX_SIZE:=200700}"
-: "${PARALLEL:=3}"
-: "${CACHE_TYPE_K:=q8_0}"
-: "${CACHE_TYPE_V:=q8_0}"
-: "${N_GPU_LAYERS:=999}"
-: "${BATCH_SIZE:=4096}"
-: "${UBATCH_SIZE:=1024}"
-: "${VERBOSITY:=2}"
-: "${CPU_MOE:=1}"
-: "${TEMPERATURE:=1.0}"
-: "${TOP_K:=20}"
-: "${TOP_P:=0.95}"
-: "${MIN_P:=0.0}"
-: "${THREADS:=7}"
-: "${THREADS_HTTP:=8}"
+: "${FT_VENV:=/home/dank/freetoken/.venv}"
+: "${CUDA_HOME_DIR:=${FT_VENV}/lib/python3.12/site-packages/nvidia/cu13}"
 
-MODEL="$MODEL_DIR/$MODEL_FILE"
+# Pool sizing (expert residency vs KV cache tradeoff).
+: "${MOE_CACHE_SIZE:=2048}"
+: "${NUM_TOKENS:=453005}"
+: "${MAX_RUNNING_REQUESTS:=4}"
 
-if [[ ! -f "$MODEL" ]]; then
-    echo "missing required file: $MODEL" >&2
+# Paths
+FT_BIN="${FT_VENV}/bin/ft"
+MODEL="${MODEL_DIR}"
+
+if [[ ! -x "$FT_BIN" ]]; then
+    echo "missing FreeToken CLI: $FT_BIN" >&2
     exit 1
 fi
-if [[ ! -x "$LLAMA_SERVER" ]]; then
-    echo "missing llama-server binary: $LLAMA_SERVER" >&2
+if [[ ! -d "$MODEL" ]]; then
+    echo "missing model directory: $MODEL" >&2
     exit 1
 fi
 
-exec "$LLAMA_SERVER" \
+export PATH="${FT_VENV}/bin:${PATH}"
+export CUDA_HOME="${CUDA_HOME_DIR}"
+
+exec "$FT_BIN" serve \
     --model "$MODEL" \
-    $([ "$CPU_MOE" = "1" ] && printf -- "--cpu-moe") \
-    --cache-type-k "$CACHE_TYPE_K" \
-    --cache-type-v "$CACHE_TYPE_V" \
-    --ctx-size "$CTX_SIZE" \
-    --parallel "$PARALLEL" \
-    --gpu-layers "$N_GPU_LAYERS" \
-    --batch-size "$BATCH_SIZE" \
-    --ubatch-size "$UBATCH_SIZE" \
-    --temp "$TEMPERATURE" \
-    --top-k "$TOP_K" \
-    --top-p "$TOP_P" \
-    --min-p "$MIN_P" \
-    --threads "$THREADS" \
-    --threads-http "$THREADS_HTTP" \
-    --alias "$ALIAS" \
+    --served-model-name "$ALIAS" \
     --host "$BIND_ADDR" \
-    --port "$LISTEN_PORT"
+    --port "$LISTEN_PORT" \
+    --moe-cache-size "$MOE_CACHE_SIZE" \
+    --num-tokens "$NUM_TOKENS" \
+    --max-running-requests "$MAX_RUNNING_REQUESTS"