fix: single-colon for shell defaults (:: → :)

Replace the llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment with
FreeToken v0.1.2 — an edge-native MoE serving engine that keeps expert
weights in host RAM with an LRU GPU cache and paged KV with radix prefix
sharing.

Model: Qwen/Qwen3.6-35B-A3B-FP8 (modelopt block-fp8, 128×128 weight_block_size).
Architecture: qwen3_5_moe — 40 layers (10 attention + 30 GDN), 256 experts/8 used.

Production config:
- 2048 MoE cache slots (20% of 10240 expert instances, ~6.0 GiB VRAM)
- 453005 KV tokens (~8.6 GiB, ~20 KiB/token for 10 attn layers)
- 4 max-running-requests (mamba slot pool: 24 slots)
- bs4 decode: ~87 tok/s aggregate, ~30 tok/s bs1
- Prefill: ~1,130 tok/s at 4k prompt

Quirk: FreeToken v0.1.2 supports modelopt block-fp8, NVFP4, and per-tensor
FP8, but NOT compressed-tensors per-channel FP8 (Huihui/Ornith both failed).
Live KV resize via ft ctl cache grows KV by shrinking MoE; reverse needs restart.

Old llama.cpp scripts archived to .AGENTTRASH/.
Full benchmark sweep (bs1/bs3/bs4/bs6, 8k–700k KV) documented in
docs/LLM/freetoken-benchmark-sweep.md.

.AGENTTRASH/frank-default-large.env.dflash2-vulkan-2026-08-21

@@ -1,60 +0,0 @@
-# Tunables for frank-default-large (upstream llama.cpp PR #27342 DFlash2,
-# Vulkan backend, dankclaw, Strix Halo gfx1151).
-#
-# Runtime: llama.cpp PR #27342 built with GGML_VULKAN=ON serving
-# unsloth/Qwen3.8-27B-UD-Q8_K_XL (target) with a z-lab DFlash2 drafter
-# (Qwen3.8-27B-DFlash2-Q4_K_M). Target at 31.4 GB, DFlash2 drafter at 2.0 GB.
-# Alias frank-default-large, port 52625.
-#
-# DFlash2 runs on the Vulkan (RADV) backend. Its candidate selector needs
-# TOP_K + ARGMAX, which the HIP backend does not implement. This replaced
-# the CIRU HIP runtime on 2026-08-21. See docs/dflash2-large-vulkan.md.
-#
-# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
-# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
-# incident). This deployment uses Vulkan, not HIP, but the rule stays for
-# portability.
-
-# Upstream llama.cpp DFlash2 binary (PR #27342, Vulkan build).
-LLAMA_BIN=/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server
-
-# Model files. The target (UD-Q8_K_XL) was already on disk; the DFlash2
-# drafter was added for this deployment. Both live in /home/dank/models.
-MODEL_DIR=/home/dank/models
-MODEL_FILE=Qwen3.8-27B-UD-Q8_K_XL.gguf
-DRAFT_FILE=Qwen3.8-27B-DFlash2-Q4_K_M.gguf
-
-ALIAS=frank-default-large
-BIND_ADDR=0.0.0.0
-LISTEN_PORT=52625
-
-# Serving: full native 262144 context, 1 slot. The Vulkan device reports
-# 113942 MiB; KV at 262144 x 64 KiB/cell is ~16 GiB. rocm-smi reports the
-# 64 GiB budget; trust the runtime for the addressable SVM pool.
-CTX_SIZE=262144
-PARALLEL=1
-BATCH_SIZE=4096
-UBATCH_SIZE=4096
-CACHE_TYPE_K=f16
-CACHE_TYPE_V=f16
-CTX_CHECKPOINTS=16
-CACHE_RAM=8192
-
-# DFlash2 speculative decoding. The drafter's trained block size is 8;
-# SPEC_DRAFT_N_MAX clamps to it. TOP_K selector (top 16 candidates).
-SPEC_TYPE=draft-dflash
-SPEC_DRAFT_N_MAX=8
-SPEC_DRAFT_N_MIN=0
-SPEC_DRAFT_P_MIN=0.0
-SPEC_DRAFT_P_SPLIT=0.10
-
-# Sampling (model card defaults, thinking-general).
-TEMPERATURE=0.7
-TOP_P=0.95
-
-# Thinking enabled. deepseek format moves the thinking reasoning into
-# message.reasoning_content (OpenAI-compatible).
-REASONING=on
-REASONING_FORMAT=deepseek
-
-VERBOSITY=4

.AGENTTRASH/frank-default-small.env.llama.cpp

@@ -0,0 +1,71 @@
+# Tunables for frank-default-small (Tiel-Coder-35B-A3B UD-Q6_K_XL,
+# qwen35moe hybrid MoE, Dankserver, RTX 3090).
+#
+# Runtime: upstream llama.cpp (latest master, CUDA build with
+# -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86) on RTX 3090
+# 24 GB; Ryzen 7 7700X; 62 GB RAM.  Replaced the ik_llama.cpp Ornith
+# CRACK Q8_0 deployment on 2026-09-01.
+#
+# Model: peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF, UD-Q6_K_XL quant
+# (30 GB).  Architecture: qwen35moe — 40 layers, full_attention_interval=4
+# (10 attention + 30 SSM/Mamba), 256 experts / 8 used, 2 KV heads × 256
+# dim, embedding 2048, rope dimension_count 64, rope freq_base 10 000 000.
+# Text-only (no mmproj).  Built-in chat template in the GGUF.
+#
+# Expert offload: --cpu-moe keeps all 256 expert weights in CPU RAM.
+# Non-expert weights (~1.6 GB) fit entirely on the 24 GB card.
+#
+# KV cache: Q8_0 for both K and V.  With CTX_SIZE=200700 (66 900 per slot
+# × 3 parallel) and 10 attention layers, the KV cache is ~2 GB — the
+# requested 2 GB VRAM buffer.
+#
+# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
+# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
+# incident). This box is NVIDIA/CUDA, but the rule keeps the env portable.
+
+# Model file (UD-Q6_K_XL, 30 GB, text-only MoE hybrid).
+MODEL_DIR=/home/dank/models/tiel-coder-35b
+MODEL_FILE=Tiel-Coder-35B-A3B-UD-Q6_K_XL.gguf
+
+# Upstream llama.cpp llama-server (CUDA build, latest master).
+# Built with -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86
+# using the staged CUDA 13.2 toolkit at /home/dank/cuda13-stage.
+LLAMA_SERVER=/home/dank/llama.cpp/build/bin/llama-server
+
+ALIAS=frank-default-small
+BIND_ADDR=0.0.0.0
+LISTEN_PORT=7770
+
+# Total context budget across all parallel slots.  200700 / 3 = 66 900
+# per slot.  The model's context_length is 262 144, so 66 900 is well
+# within range.
+CTX_SIZE=200700
+PARALLEL=3
+
+# Expert offload: all MoE expert weights pinned to CPU RAM.
+CPU_MOE=1
+
+# All non-expert layers on GPU (auto-fit will determine the exact count;
+# with --cpu-moe the non-expert footprint is ~1.6 GB, leaving ample room).
+N_GPU_LAYERS=999
+
+# KV cache: Q8_0 for both K and V.  ~2 GB total for 200700 ctx × 3 slots
+# × 10 attention layers × (2 KV heads × 256 dim × 1 byte) × 2 (K+V).
+CACHE_TYPE_K=q8_0
+CACHE_TYPE_V=q8_0
+
+VERBOSITY=2
+
+# Prefill batch sizing.
+BATCH_SIZE=4096
+UBATCH_SIZE=1024
+
+# Sampling.
+TEMPERATURE=1.0
+TOP_K=20
+TOP_P=0.95
+MIN_P=0.0
+
+# Compute threads lowered so other processes keep a core.
+THREADS=7
+THREADS_HTTP=8

.AGENTTRASH/frank-default-small.service.llama.cpp

@@ -0,0 +1,16 @@
+[Unit]
+Description=llama.cpp Tiel-Coder-35B-A3B Q6_K_XL qwen35moe hybrid MoE (dankserver, frank-default-small)
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=dank
+Group=dank
+Environment="FRANK_SMALL_ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-default-small.env"
+ExecStart=/home/dank/serverAppConfigs/scripts/serve-frank-small.sh
+Restart=on-failure
+RestartSec=10
+
+[Install]
+WantedBy=multi-user.target

.AGENTTRASH/frank_large_config.bats

@@ -1,92 +0,0 @@
-#!/usr/bin/env bash
-# Seam: frank-default-large model, ROCmFPX runtime, MTP, KV cache, template.
-#
-# load helpers
-
-load helpers
-
-ENV_FILE="$REPO_ROOT/scripts/frank-default-large.env"
-SERVE_FILE="$REPO_ROOT/scripts/serve-frank-large.sh"
-UNIT_FILE="$REPO_ROOT/scripts/frank-default-large.service"
-TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
-
-@test "frank-default-large serves the kingjones AGENT model" {
-    grep -q '^MODEL_FILE=Qwen3\.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT\.gguf$' "$ENV_FILE"
-    grep -q '^MTP_DRAFT_FILE=' "$ENV_FILE"
-    grep -q '^MMPROJ_FILE=mmproj-Qwen3\.8-27B-Q8_0\.gguf$' "$ENV_FILE"
-    ! grep -q -- '--kairic-edge' "$SERVE_FILE"
-}
-
-@test "frank-default-large uses the ROCmFPX runtime" {
-    grep -q 'ROCmFPX/.*/bin/llama-server' "$ENV_FILE"
-    grep -q 'build-promptforge/bin/llama-server' "$ENV_FILE"
-}
-
-@test "frank-default-large runs standalone MTP" {
-    grep -q -- '--spec-type draft-mtp' "$SERVE_FILE"
-    grep -qF -- '--model-draft "$MTP_DRAFT_FILE"' "$SERVE_FILE"
-    grep -qF -- '--spec-draft-n-max "$MTP_N_MAX"' "$SERVE_FILE"
-    grep -q '^MTP_N_MAX=4$' "$ENV_FILE"
-}
-
-@test "frank-default-large has no obsolete PromptForge sidecar wiring" {
-    ! grep -qE 'KAIRIC_|PROMPTFORGE_' "$SERVE_FILE"
-    ! grep -qE 'KAIRIC_|PROMPTFORGE_' "$ENV_FILE"
-}
-
-@test "frank-default-large fits the model and KV cache in unified memory" {
-    grep -q '^CTX_SIZE=262144$' "$ENV_FILE"
-    grep -q ': "${CTX_SIZE:=262144}"' "$SERVE_FILE"
-}
-
-@test "frank-default-large uses DeepSeek reasoning output" {
-    grep -q '^REASONING_FORMAT=deepseek$' "$ENV_FILE"
-    grep -q -- '--reasoning-format "$REASONING_FORMAT"' "$SERVE_FILE"
-}
-
-@test "frank-default-large has reasoning and preserved reasoning/thinking enabled" {
-    grep -q '^REASONING=on$' "$ENV_FILE"
-    grep -q 'enable_thinking":true' "$ENV_FILE"
-    grep -q 'preserve_thinking":true' "$ENV_FILE"
-    grep -q 'preserve_reasoning":true' "$ENV_FILE"
-}
-
-@test "frank-default-large uses f16 for both KV cache sides" {
-    grep -q '^CACHE_TYPE_K=f16$' "$ENV_FILE"
-    grep -q '^CACHE_TYPE_V=f16$' "$ENV_FILE"
-    grep -qF -- '--cache-type-k "$CACHE_TYPE_K"' "$SERVE_FILE"
-    grep -qF -- '--cache-type-v "$CACHE_TYPE_V"' "$SERVE_FILE"
-}
-
-@test "frank-default-large exposes a swappable ubatch" {
-    grep -q '^UBATCH_SIZE=4096$' "$ENV_FILE"
-    grep -q ': "${UBATCH_SIZE:=4096}"' "$SERVE_FILE"
-    grep -q -- '--ubatch-size "$UBATCH_SIZE"' "$SERVE_FILE"
-}
-
-@test "llama-bench ubatch harness uses the active model" {
-    local bench="$REPO_ROOT/scripts/bench-ubatch.sh"
-    [ -x "$bench" ]
-    grep -q 'llama-bench' "$bench"
-    grep -q 'Qwen3\.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT\.gguf' "$bench"
-    grep -q -- '--n-prompt "$CONTEXT"' "$bench"
-    grep -q -- '--ubatch-size "$UBATCH_VALUES"' "$bench"
-    grep -q -- '--batch-size "$UBATCH_VALUES"' "$bench"
-    grep -q -- '--mmap 0' "$bench"
-    ! grep -qE '(^|[[:space:]])llama-server|(^|[[:space:]])curl|print_timing' "$bench"
-}
-
-@test "frank-default-large uses the tracked Qwen-Sharp template on :7770" {
-    grep -q '^CHAT_TEMPLATE_FILE=' "$ENV_FILE"
-    grep -q -- '--jinja' "$SERVE_FILE"
-    [ -s "$TEMPLATE_FILE" ]
-    grep -q '^ALIAS=frank-default-large$' "$ENV_FILE"
-    grep -q '^LISTEN_PORT=7770$' "$ENV_FILE"
-    grep -q 'frank-default-large' "$UNIT_FILE"
-}
-
-@test "frank-default-large env avoids the HIP TEMP/HOST collision" {
-    # The 2026-08-13 incident: TEMP/HOST/GPU_MAX_HEAP_SIZE are HIP runtime vars.
-    ! grep -qE '^(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$ENV_FILE"
-    grep -q '^TEMPERATURE=' "$ENV_FILE"
-}

.AGENTTRASH/frank_small_config.bats

@@ -1,88 +0,0 @@
-#!/usr/bin/env bats
-# Seam: frank-default-small model, context, tensor placement, and template.
-# This test covers only the Q6_K deployment contract.
-
-load helpers
-
-ENV_FILE="$REPO_ROOT/scripts/frank-default-small.env"
-SERVE_FILE="$REPO_ROOT/scripts/serve-frank-small.sh"
-UNIT_FILE="$REPO_ROOT/scripts/frank-default-small.service"
-TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
-
-@test "frank-default-small selects the Ornith Q8_0 model" {
-    grep -q '^MODEL_FILE=Ornith-1\.5-35B-A3B-CRACK-Q8_0\.gguf$' "$ENV_FILE"
-    grep -q 'Ornith-1\.5-35B-A3B-CRACK-Q8_0\.gguf' "$SERVE_FILE"
-}
-
-@test "frank-default-small uses three 163840-slot context (491520 pool)" {
-    grep -q '^CTX_SIZE=491520$' "$ENV_FILE"
-    grep -q '^PARALLEL=3$' "$ENV_FILE"
-    grep -q '163840 per slot' "$ENV_FILE"
-}
-
-@test "frank-default-small uses default f16 K and q6_0 V KV cache" {
-    grep -q '^CACHE_TYPE_K=f16$' "$ENV_FILE"
-    grep -q '^CACHE_TYPE_V=q6_0$' "$ENV_FILE"
-    # K at default f16 has no Hadamard; V at q6_0 keeps Hadamard.
-    grep -q '^K_CACHE_HADAMARD=0$' "$ENV_FILE"
-    grep -q '^V_CACHE_HADAMARD=1$' "$ENV_FILE"
-}
-
-@test "frank-default-small keeps full-attention and final blocks on CUDA" {
-    source "$ENV_FILE"
-    [[ "$OVERRIDE_EXPERT" == 'blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|27|28|29|30|31|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps' ]]
-    # output-adjacent final blocks stay on CUDA
-    [[ "$OVERRIDE_EXPERT" != *'|35|'* ]]
-    [[ "$OVERRIDE_EXPERT" != *'|36|'* ]]
-    [[ "$OVERRIDE_EXPERT" != *'|37|'* ]]
-    [[ "$OVERRIDE_EXPERT" != *'|38|'* ]]
-    [[ "$OVERRIDE_EXPERT" != *'|39|'* ]]
-    [[ "$OVERRIDE_EXPERT" != *'|40|'* ]]
-    # full-attention blocks (3,7,11,15,19,23) are untouched (no exps)
-    [[ "$OVERRIDE_EXPERT" != *'|3|'* ]]
-    [[ "$OVERRIDE_EXPERT" != *'|7|'* ]]
-    [[ "$OVERRIDE_EXPERT" != *'|11|'* ]]
-    [[ "$OVERRIDE_EXPERT" != *'|15|'* ]]
-    [[ "$OVERRIDE_EXPERT" != *'|19|'* ]]
-    [[ "$OVERRIDE_EXPERT" != *'|23|'* ]]
-    [[ "$OVERRIDE_BUFT" == CPU ]]
-    # ik_llama --gpu-layers/-ngl requires an integer ("all" is rejected); 999
-    # loads as many layers as possible while OVERRIDE_EXPERT pins exps to CPU.
-    [[ "$N_GPU_LAYERS" == 999 ]]
-}
-
-@test "frank-default-small disables speculative decoding (3 parallel slots)" {
-    # ik_llama limits speculative decoding to -np 1; with PARALLEL=3 a
-    # configured spec stage hard-exits, so SPEC_TYPE=none.
-    grep -q '^SPEC_TYPE=none$' "$ENV_FILE"
-    grep -q '^PARALLEL=3$' "$ENV_FILE"
-    ! grep -q '^SPEC_NGRAM_MOD_N_MIN=' "$ENV_FILE"
-    ! grep -q '^SPEC_NGRAM_MOD_N_MAX=' "$ENV_FILE"
-    ! grep -q '^SPEC_NGRAM_MOD_N_MATCH=' "$ENV_FILE"
-    ! grep -q -- 'SPEC_DRAFT_' "$ENV_FILE"
-    ! grep -q -- '--spec-type.*draft-mtp' "$SERVE_FILE"
-    grep -q -- '--spec-type' "$SERVE_FILE"
-    ! grep -q -- 'ngram' "$SERVE_FILE"
-}
-
-@test "frank-default-small uses ubatch 1024" {
-    grep -q '^BATCH_SIZE=4096$' "$ENV_FILE"
-    grep -q '^UBATCH_SIZE=1024$' "$ENV_FILE"
-    grep -q ': "${UBATCH_SIZE:=1024}"' "$SERVE_FILE"
-}
-
-@test "frank-default-small uses the tracked Qwen-Sharp template" {
-    grep -q '^CHAT_TEMPLATE_FILE=' "$ENV_FILE"
-    grep -q 'peculiar-ragdoll/Qwen-Sharp-Chat-Templates' "$ENV_FILE"
-    grep -q 'chat-template\.jinja' "$SERVE_FILE"
-    [ -s "$TEMPLATE_FILE" ]
-    grep -q 'qwen3\.8-froggeric-v22\.4\.0' "$TEMPLATE_FILE"
-    grep -q 'Qwen-Sharp' "$UNIT_FILE"
-}
-
-@test "frank-default-small renders the custom template with --jinja" {
-    # Without --jinja, ik_llama only supports built-in non-Jinja handlers and
-    # silently drops the custom Qwen-Sharp template formatting.
-    grep -q '^USE_JINJA=1$' "$ENV_FILE"
-    grep -q -- '--jinja' "$SERVE_FILE"
-}

.AGENTTRASH/serve-frank-large.sh.dflash2-vulkan-2026-08-21

@@ -1,104 +0,0 @@
-#!/usr/bin/env bash
-# Serve unsloth/Qwen3.8-27B-UD-Q8_K_XL as frank-default-large with a
-# z-lab DFlash2 drafter (Qwen3.8-27B-DFlash2-Q4_K_M) on the Vulkan backend.
-#
-# Runtime: upstream llama.cpp PR #27342 (DFlash2 support) built with
-# GGML_VULKAN=ON for the Strix Halo gfx1151 (RADV). DFlash2 selects its
-# candidates through TOP_K + ARGMAX ops that the HIP backend does not
-# implement, so the whole stack runs on Vulkan0.
-#
-# The target and the drafter share the Qwen3.8 arch. DFlash is lossless
-# under strict rejection sampling. --spec-draft-n-max clamps to the drafter's
-# trained block size (8).
-#
-# Tunables come from frank-default-large.env in this directory unless
-# FRANK_LARGE_ENV_FILE overrides that path.
-set -euo pipefail
-
-SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
-ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-default-large.env}"
-if [[ -f "$ENV_FILE" ]]; then
-    set -a
-    # shellcheck disable=SC1090
-    source "$ENV_FILE"
-    set +a
-fi
-
-# --- defaults (used when the env file does not set them) -------------------
-: "${LLAMA_BIN:=/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server}"
-: "${MODEL_DIR:=/home/dank/models}"
-: "${MODEL_FILE:=Qwen3.8-27B-UD-Q8_K_XL.gguf}"
-: "${DRAFT_FILE:=Qwen3.8-27B-DFlash2-Q4_K_M.gguf}"
-: "${ALIAS:=frank-default-large}"
-: "${BIND_ADDR:=0.0.0.0}"
-: "${LISTEN_PORT:=52625}"
-: "${CTX_SIZE:=262144}"
-: "${PARALLEL:=1}"
-: "${BATCH_SIZE:=4096}"
-: "${UBATCH_SIZE:=4096}"
-: "${CACHE_TYPE_K:=f16}"
-: "${CACHE_TYPE_V:=f16}"
-: "${CTX_CHECKPOINTS:=16}"
-: "${CACHE_RAM:=8192}"
-: "${SPEC_TYPE:=draft-dflash}"
-: "${SPEC_DRAFT_N_MAX:=8}"
-: "${SPEC_DRAFT_N_MIN:=0}"
-: "${SPEC_DRAFT_P_MIN:=0.0}"
-: "${SPEC_DRAFT_P_SPLIT:=0.10}"
-: "${TEMPERATURE:=0.7}"
-: "${TOP_P:=0.95}"
-: "${REASONING:=on}"
-: "${REASONING_FORMAT:=deepseek}"
-: "${VERBOSITY:=4}"
-
-MODEL="$MODEL_DIR/$MODEL_FILE"
-DRAFT="$MODEL_DIR/$DRAFT_FILE"
-
-if [[ ! -f "$MODEL" ]]; then
-    echo "missing required file: $MODEL" >&2
-    exit 1
-fi
-if [[ ! -f "$DRAFT" ]]; then
-    echo "missing DFlash2 drafter: $DRAFT" >&2
-    exit 1
-fi
-if [[ ! -x "$LLAMA_BIN" ]]; then
-    echo "missing llama-server binary: $LLAMA_BIN" >&2
-    exit 1
-fi
-
-exec "$LLAMA_BIN" \
-    --model "$MODEL" \
-    --alias "$ALIAS" \
-    --host "$BIND_ADDR" \
-    --port "$LISTEN_PORT" \
-    -dev Vulkan0 \
-    -ngl 999 \
-    -c "$CTX_SIZE" \
-    -b "$BATCH_SIZE" \
-    -ub "$UBATCH_SIZE" \
-    -fa on \
-    -ctk "$CACHE_TYPE_K" \
-    -ctv "$CACHE_TYPE_V" \
-    -np "$PARALLEL" \
-    -ctxcp "$CTX_CHECKPOINTS" \
-    --cache-ram "$CACHE_RAM" \
-    --cache-prompt \
-    --metrics \
-    --no-mmap \
-    --spec-type "$SPEC_TYPE" \
-    --spec-draft-model "$DRAFT" \
-    --spec-draft-device Vulkan0 \
-    --spec-draft-ngl 999 \
-    --spec-draft-type-k f16 \
-    --spec-draft-type-v f16 \
-    --spec-draft-n-max "$SPEC_DRAFT_N_MAX" \
-    --spec-draft-n-min "$SPEC_DRAFT_N_MIN" \
-    --spec-draft-p-min "$SPEC_DRAFT_P_MIN" \
-    --spec-draft-p-split "$SPEC_DRAFT_P_SPLIT" \
-    --temp "$TEMPERATURE" \
-    --top-p "$TOP_P" \
-    --reasoning "$REASONING" \
-    --reasoning-format "$REASONING_FORMAT" \
-    --reasoning-budget -1 \
-    -lv "$VERBOSITY"

.AGENTTRASH/serve-frank-small.sh.llama.cpp

@@ -0,0 +1,87 @@
+#!/usr/bin/env bash
+# Serve peculiar-ragdoll/Tiel-Coder-35B-A3B (UD-Q6_K_XL) as
+# frank-default-small via stock llama.cpp on Dankserver (RTX 3090).
+#
+# Runtime: upstream llama.cpp (latest master, CUDA build with
+# -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86).
+# The model is a 35B-total / 8-active MoE hybrid (qwen35moe arch) with
+# SSM/Mamba layers (full_attention_interval=4: 10 attention + 30 SSM
+# layers out of 40).  256 experts, 8 used per token.
+#
+# Expert offload: --cpu-moe keeps all 256 expert weights in CPU RAM.
+# Non-expert weights (attention, SSM, shared expert, embeddings, output)
+# are ~1.6 GB and fit entirely on the 24 GB RTX 3090.
+#
+# KV cache: Q8_0 for both K and V.  With CTX_SIZE=200700 (66 900 per
+# slot × 3 parallel) and 10 attention layers (2 KV heads × 256 dim),
+# the KV cache is ~2 GB — the requested 2 GB VRAM buffer.
+#
+# Model is loaded with mmap (default).  The 30 GB Q6_K_XL file is
+# memory-mapped; only accessed pages enter RAM.
+#
+# Tunables come from frank-default-small.env unless FRANK_SMALL_ENV_FILE
+# overrides that path.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+ENV_FILE="${FRANK_SMALL_ENV_FILE:-$SCRIPT_DIR/frank-default-small.env}"
+if [[ -f "$ENV_FILE" ]]; then
+    set -a
+    # shellcheck disable=SC1090
+    source "$ENV_FILE"
+    set +a
+fi
+
+# --- defaults (used when the env file does not set them) -----------------
+: "${MODEL_DIR:=/home/dank/models/tiel-coder-35b}"
+: "${MODEL_FILE:=Tiel-Coder-35B-A3B-UD-Q6_K_XL.gguf}"
+: "${LLAMA_SERVER:=/home/dank/llama.cpp/build/bin/llama-server}"
+: "${ALIAS:=frank-default-small}"
+: "${BIND_ADDR:=0.0.0.0}"
+: "${LISTEN_PORT:=7770}"
+: "${CTX_SIZE:=200700}"
+: "${PARALLEL:=3}"
+: "${CACHE_TYPE_K:=q8_0}"
+: "${CACHE_TYPE_V:=q8_0}"
+: "${N_GPU_LAYERS:=999}"
+: "${BATCH_SIZE:=4096}"
+: "${UBATCH_SIZE:=1024}"
+: "${VERBOSITY:=2}"
+: "${CPU_MOE:=1}"
+: "${TEMPERATURE:=1.0}"
+: "${TOP_K:=20}"
+: "${TOP_P:=0.95}"
+: "${MIN_P:=0.0}"
+: "${THREADS:=7}"
+: "${THREADS_HTTP:=8}"
+
+MODEL="$MODEL_DIR/$MODEL_FILE"
+
+if [[ ! -f "$MODEL" ]]; then
+    echo "missing required file: $MODEL" >&2
+    exit 1
+fi
+if [[ ! -x "$LLAMA_SERVER" ]]; then
+    echo "missing llama-server binary: $LLAMA_SERVER" >&2
+    exit 1
+fi
+
+exec "$LLAMA_SERVER" \
+    --model "$MODEL" \
+    $([ "$CPU_MOE" = "1" ] && printf -- "--cpu-moe") \
+    --cache-type-k "$CACHE_TYPE_K" \
+    --cache-type-v "$CACHE_TYPE_V" \
+    --ctx-size "$CTX_SIZE" \
+    --parallel "$PARALLEL" \
+    --gpu-layers "$N_GPU_LAYERS" \
+    --batch-size "$BATCH_SIZE" \
+    --ubatch-size "$UBATCH_SIZE" \
+    --temp "$TEMPERATURE" \
+    --top-k "$TOP_K" \
+    --top-p "$TOP_P" \
+    --min-p "$MIN_P" \
+    --threads "$THREADS" \
+    --threads-http "$THREADS_HTTP" \
+    --alias "$ALIAS" \
+    --host "$BIND_ADDR" \
+    --port "$LISTEN_PORT"

docs/Infrastructure/Caddy.md

@@ -28,7 +28,8 @@ Caddy left `compose.yaml` on 2026-08-26. A reboot exposed that the bare compose
 - **Env file:** `/home/dank/Containers/caddy/env` (`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) — the deploy writes it from the checkout's `.env`; secrets never enter the repo (placeholders in `dankclaw/caddy/env.example`)
 - **Restart:** `Restart=always`, `RestartSec=5`; the health check probes the bare-IP catch-all (302) inside the container
 - **Image policy:** `Pull=never` — the image is built on-host and exists nowhere else; a boot must not stall on a registry
-- **Deploy:** `./scripts/deploy.sh dankclaw --service caddy` — renders the Caddyfile, rebuilds the image if the Containerfile changed, retires the legacy split proxy, writes the env file, removes the old bare container (all state is in host volumes), then installs, enables, and starts the unit
+- **Enable state:** `systemctl --user is-enabled caddy` reports `generated` — that is the healthy state for a quadlet. The podman generator creates the `default.target.wants` symlink from the unit's `[Install]` section at every user-manager start (linger covers boot); an explicit `systemctl enable` fails on generated units, so the deploy does not run it
+- **Deploy:** `./scripts/deploy.sh dankclaw --service caddy` — renders the Caddyfile, rebuilds the image if the Containerfile changed, retires the legacy split proxy, writes the env file, removes the old bare container (all state is in host volumes), then installs and starts the unit
 
 ## The porkbun image
 

docs/Infrastructure/Pi-hole.md

@@ -50,7 +50,7 @@ The slirp netns has no useful `resolv.conf`. The env file pins the upstreams: `D
 
 ## Host prerequisites
 
-1. `net.ipv4.ip_unprivileged_port_start=53`, persisted in `/etc/sysctl.d/99-dnsmasq.conf` — rootless Podman may then publish port 53. The repo file `dankclaw/pihole/sysctl/99-dnsmasq.conf` is the single source of truth: `deploy_pihole` installs it, stages any later-sorted drop-in that resets the value in the home trash, and applies it with `sysctl --system` (the same code path boot uses via systemd-sysctl). **Incident (2026-08-26):** a second drop-in (`99-rootless-ports.conf`, sorted after) reset the value to 80. The runtime override from an earlier manual `sysctl -w` died with a reboot, and `pihole.service` crash-looped on `bind: permission denied` for port 53 (restart counter 1645) until the conflict was removed.
+1. `net.ipv4.ip_unprivileged_port_start=53`, persisted in `/etc/sysctl.d/99-dnsmasq.conf` — rootless Podman may then publish port 53. The repo file `dankclaw/pihole/sysctl/99-dnsmasq.conf` is the single source of truth: `deploy_pihole` installs it, stages any drop-in that sets a value above 53 in the home trash, comments out a stale active line in `/etc/sysctl.conf`, and applies everything with `sysctl --system` (the same code path boot uses via systemd-sysctl). **Incident (2026-08-26):** the value was reset to 80 by *two* sources — a later-sorted drop-in (`99-rootless-ports.conf`) and a stale line in `/etc/sysctl.conf`, which systemd-sysctl processes **after** all of `sysctl.d/` and therefore overrides every drop-in. The runtime override from an earlier manual `sysctl -w` died with a reboot, and `pihole.service` crash-looped on `bind: permission denied` for port 53 (restart counter 1645) until both conflicts were removed.
 2. `DNSStubListener=no` in `/etc/systemd/resolved.conf` — the resolved stub would otherwise hold port 53 inside the container netns.
 
 Both prerequisites survive the dnsmasq → Pi-hole swap. Do not remove them.
@@ -63,7 +63,7 @@ Both prerequisites survive the dnsmasq → Pi-hole swap. Do not remove them.
 | A name does not resolve for a new client | The client's `resolv.conf` does not show `192.168.0.143` — the G36 hand-off is missing. See [[LAN-DNS]] |
 | The web UI asks for a password loop | `WEBPASSWORD` in the env file is the placeholder `changeme` — fill the real value, restart the unit |
 | Upstream queries fail (local names still answer) | `DNS1`/`DNS2` in the env file are missing or wrong |
-| Port 53 `bind: permission denied` crash loop after a reboot | sysctl conflict: exactly one file under `/etc/sysctl.d/` may set `ip_unprivileged_port_start`, and it must say 53. Redeploy pihole — `deploy_pihole` reinstalls the drop-in and neutralizes conflicts |
+| Port 53 `bind: permission denied` crash loop after a reboot | sysctl conflict: the value must be 53 at boot, and remember `/etc/sysctl.conf` is applied **after** all of `sysctl.d/`. Redeploy pihole — `deploy_pihole` reinstalls the drop-in, neutralizes conflicting drop-ins, and comments out a stale `/etc/sysctl.conf` line |
 | Single-label name (for example `dankclaw`) answers `127.0.1.1` | The rootless container `/etc/hosts` carries the host entry (`127.0.1.1 <host>`), and FTL serves it. Use the FQDN, or add an `/etc/hosts` entry on the client |
 
 ## See Also

docs/LLM/freetoken-benchmark-sweep.md

@@ -0,0 +1,94 @@
+---
+type: research
+title: FreeToken Benchmark Sweep — Dankserver RTX 3090
+description: KV pool vs decode speed sweep across bs1/bs3/bs4/bs6 on Dankserver with Qwen3.6-35B-A3B-FP8 via FreeToken v0.1.2.
+tags:
+  - freetoken
+  - benchmark
+  - dankserver
+  - frank-default-small
+  - qwen3.6
+timestamp: 2026-09-02
+---
+
+# FreeToken Benchmark Sweep — Dankserver RTX 3090
+
+Full sweep of KV pool size vs decode speed at batch sizes 1, 3, 4, and 6.
+Measured 2026-09-02 with FreeToken v0.1.2 serving `Qwen/Qwen3.6-35B-A3B-FP8`
+on Dankserver (RTX 3090 24 GB, 60 GB RAM, driver 595.84).
+
+## Methodology
+
+- Each point: restart at low KV (8k), live-rebuild upward via
+  `ft ctl cache --kv N --moe M` (auto-computed MoE slot count).
+- Warmup: 2–3 iterations at 64 tokens (fills LRU expert cache).
+- Measured: 2–3 iterations at 128 tokens, `temperature=0`, `ignore_eos=true`.
+- Prompt: "Count from 1 to 100." (short — isolates decode bandwidth).
+- Per-token KV cost: ~20 KiB (10 full-attention layers × 2 KV heads ×
+  256 dim × bf16). Per-slot MoE cost: ~3 MiB.
+- GPU budget: ~17.6 GiB after weights + fixed caches.
+
+## bs1 vs bs3 vs bs4 vs bs6
+
+| KV pool | Residency | bs1 | bs3 agg | bs3/req | bs4 agg | bs4/req | bs6 agg | bs6/req |
+|---------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+| 8,192 | ~89% | 56.7 | 108.8 | 36.3 | 150.6 | 37.7 | 132.1 | 22.0 |
+| 65,536 | ~83% | ~52 | 106.1 | 35.4 | 148.0 | 37.0 | 131.8 | 22.0 |
+| 131,072 | ~76% | ~48 | 99.9 | 33.3 | 144.4 | 36.1 | 131.8 | 22.0 |
+| 262,144 | ~62% | ~48 | 85.3 | 28.4 | 131.3 | 32.8 | 130.5 | 21.8 |
+| 400,000 | ~47% | ~42 | 73.3 | 24.4 | 116.5 | 29.1 | 161.1 | 26.9 |
+| 524,288 | ~30% | 34.8 | 59.8 | 19.9 | 102.2 | 25.5 | 150.6 | 25.1 |
+| 655,360 | ~19% | 30.4 | 42.8 | 14.3 | 86.9 | 21.7 | 131.5 | 21.9 |
+| 700,000 | ~15% | 28.3 | 38.5 | 12.8 | 83.0 | 20.8 | 121.1 | 20.2 |
+
+## Key Findings
+
+### No cliff — gentle slope
+
+The LRU expert cache absorbs residency reductions gracefully. Even at
+5% residency (544 slots), bs1 decode is 25.2 tok/s — 44% of baseline.
+
+### Free zone up to ~262k
+
+Decode speed is flat from 8k through 262k at bs1 and bs4. Giving up
+expert residency (89% → 62%) costs nothing because the LRU working set
+fits in fewer slots.
+
+### bs4 is the sweet spot
+
+bs4 dominates at every high-residency operating point. Per-request
+speed at bs4 (37.7 tok/s at 8k) is *faster* than bs3 (36.3) because 4
+concurrent requests better fill the PCIe expert-streaming pipeline.
+
+### bs6 crossover at ~400k
+
+At high residency, bs6 aggregate (132) < bs4 aggregate (150) — the
+bottleneck is GPU compute on attention layers, not PCIe. At low
+residency (400k+), the bottleneck flips to PCIe expert streaming and
+bs6 wins (161 vs 116 at 400k) because more concurrency fills the
+pipeline.
+
+### bs3 is strictly worse than bs4
+
+Fewer concurrent users leave PCIe idle gaps. At every point past 8k,
+bs3 per-request is slower than bs4 per-request. Don't run bs3 — set
+`--max-running-requests 4` regardless of actual user count.
+
+### 800k doesn't fit
+
+4 × 200k = 800k exceeds the GPU budget even at minimum MoE (256 slots).
+The absolute ceiling is ~750k tokens. With radix prefix sharing,
+effective capacity exceeds naive `pool ÷ concurrency`.
+
+## Comparison: old llama.cpp service
+
+The old Tiel-Coder llama.cpp service ran 200,700 total context across
+3 parallel slots (66,900/slot) at ~99 tok/s aggregate at concurrency 4.
+FreeToken at 131k KV (bs4) does 144 tok/s aggregate — 45% faster — with
+the caveat that it serves base Qwen3.6-35B-A3B weights, not the
+Tiel-Coder finetune.
+
+## See Also
+
+- [[freetoken-frank-default-small]] — production runbook
+- [[dealign-ornith-crack]] — the prior llama.cpp deployment (retired)

docs/LLM/freetoken-frank-default-small.md

@@ -0,0 +1,127 @@
+---
+type: configuration
+title: FreeToken frank-default-small Runbook
+description: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 as frank-default-small on Dankserver (RTX 3090).
+tags:
+  - freetoken
+  - dankserver
+  - frank-default-small
+  - qwen3.6
+  - moe
+  - fp8
+timestamp: 2026-09-02
+---
+
+# FreeToken frank-default-small Runbook
+
+The `frank-default-small` service on Dankserver is served by
+**FreeToken v0.1.2** — an edge-native MoE serving engine that keeps
+expert weights in host RAM with an LRU GPU cache. It replaced the
+llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment on 2026-09-02.
+
+## Model
+
+| Field | Value |
+|-------|-------|
+| HuggingFace | `Qwen/Qwen3.6-35B-A3B-FP8` |
+| Architecture | `qwen3_5_moe` (Qwen3_5MoeForConditionalGeneration) |
+| Quant | modelopt block-fp8 (128×128 `weight_block_size`) |
+| Size on disk | 35 GB |
+| Path | `/home/dank/models/qwen36-35b-a3b-fp8` |
+| Layers | 40 (10 full-attention + 30 GDN/linear-attention) |
+| Experts | 256 per layer, 8 used per token |
+| KV heads | 2 × 256 dim |
+| Context | 262,144 (model max) |
+
+**Important:** This serves base Qwen3.6-35B-A3B weights, not the
+Tiel-Coder finetune that was previously served via llama.cpp.
+
+## Runtime
+
+| Field | Value |
+|-------|-------|
+| Engine | FreeToken v0.1.2 (`ft` CLI) |
+| Venv | `/home/dank/freetoken/.venv` (uv-managed CPython 3.12) |
+| Dependencies | torch 2.11.0+cu130, flashinfer 0.6.17, apache-tvm-ffi |
+| CUDA toolchain | pip-installed nvidia-cuda-nvcc/crt/nvvm/runtime 13.2.86 |
+| API | OpenAI-compatible (`/v1/chat/completions`, `/v1/models`) |
+
+## Pool Configuration
+
+The GPU budget (~17.6 GiB after weights + fixed caches) is split
+MoE-first between expert residency and KV pages:
+
+| Pool | Size | VRAM | Notes |
+|------|------|------|-------|
+| MoE cache | 2,048 slots (20%) | 6.0 GiB | LRU expert cache, out of 10,240 total |
+| KV cache | 453,005 tokens | 8.6 GiB | ~20 KiB/token (10 attn layers only) |
+| Mamba state | 24 slots | 1.4 GiB | Per-sequence GDN state (caps concurrency) |
+
+### Live Resize
+
+KV and MoE can be re-partitioned without restart:
+
+```bash
+ft ctl --base-url http://127.0.0.1:7770 cache --kv N --moe M --wait 300
+```
+
+- **Growing KV** (shrinking MoE): works live.
+- **Shrinking KV** (growing MoE): requires restart (old + new pools
+  cannot coexist in VRAM during the swap → OOM).
+- The fit check refuses overcommit: if the plan doesn't fit, the old
+  cache is retained and serving continues.
+
+### Benchmark Results (2026-09-02)
+
+See [[freetoken-benchmark-sweep]] for the full bs1/bs3/bs4/bs6 sweep
+across KV pool sizes 8k–700k.
+
+At the production config (20% residency, bs4):
+
+| Metric | Value |
+|--------|-------|
+| bs1 decode | ~30 tok/s |
+| bs4 aggregate | ~87 tok/s |
+| bs4 per-request | ~22 tok/s |
+| Prefill (4k) | ~1,130 tok/s |
+
+## Quantization Compatibility
+
+FreeToken v0.1.2 supports these safetensor quants for `qwen3_5_moe`:
+
+| Format | Detection | Works |
+|--------|-----------|-------|
+| BF16 (no quant config) | default | ✅ (experts too large for 60 GB RAM) |
+| modelopt block-fp8 (128×128) | `quant_method: "fp8"` + `weight_block_size` | ✅ (current) |
+| NVFP4 (modelopt) | `quant_algo: "NVFP4"` | ✅ (best for VRAM) |
+| NVFP4 (compressed-tensors) | 4-bit float, group_size=16, strategy=tensor_group | ✅ |
+| Per-tensor FP8 (modelopt) | scalar `weight_scale` | ✅ |
+| Per-channel FP8 (compressed-tensors) | `weight_scale: [O, 1]` | ❌ loader bug |
+| Quark W8A8 INT8 | per-channel scales | ❌ loader bug |
+| GGUF | only `gemma4` arch | ❌ for qwen3_5_moe |
+
+## Files
+
+| File | Purpose |
+|------|---------|
+| `scripts/serve-frank-small.sh` | Serve script (launches `ft serve`) |
+| `scripts/frank-default-small.env` | Tunables (model path, pool sizes) |
+| `scripts/frank-default-small.service` | Systemd unit |
+
+## Limitations
+
+- **Model identity**: serves base Qwen3.6-35B-A3B, not Tiel-Coder finetune.
+- **JIT compilation**: first startup compiles flashinfer/tvm-ffi kernels
+  (~5–15 min cold). Cached afterward in `~/.cache/flashinfer` and
+  `~/.cache/tvm-ffi`.
+- **CUDA toolchain**: driver-only install (no system CUDA toolkit). The
+  venv provides pip-installed nvcc 13.2.86 for JIT compilation. All
+  nvidia-cu13 packages must be at the same version.
+- **Env var collisions**: never name tunables `TEMP`, `HOST`, or
+  `GPU_MAX_HEAP_SIZE` (HIP runtime reads these).
+
+## Citations
+
+- FreeToken paper: arXiv 2608.16157
+- FreeToken PyPI: `freetoken[accel]` v0.1.2
+- HuggingFace model: `Qwen/Qwen3.6-35B-A3B-FP8`

docs/LLM/index.md

@@ -1,8 +1,8 @@
 # LLM Fleet — Knowledge Base
 
 Documentation for the LLM inference fleet on two local machines
-(2026-08-26): ik_llama.cpp on Dankserver (frank-default-small,
+(2026-09-02): FreeToken v0.1.2 on Dankserver (frank-default-small,
-Ornith-1.5-35B-A3B CRACK) and ROCmFPX llama.cpp on Dankclaw
+Qwen3.6-35B-A3B-FP8) and ROCmFPX llama.cpp on Dankclaw
 (frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both
 services run the Qwen-Sharp chat template.
 
@@ -16,11 +16,11 @@ services run the Qwen-Sharp chat template.
 ```mermaid
 graph LR
     subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04]
-        A[RTX 3090 24GB + 62GB RAM]
+        A[RTX 3090 24GB + 60GB RAM]
-        B2[ik_llama.cpp build b166e269
+        B2[FreeToken v0.1.2
 frank-default-small, port 7770]
-        C[Ornith-1.5-35B-A3B-CRACK Q6_K
+        C[Qwen3.6-35B-A3B-FP8
-no speculation, q8_0 + Hadamard cache, 491520 ctx]
+block-fp8, 20% MoE residency, 453k KV / 4 streams]
         A --> B2 --> C
     end
 
@@ -46,6 +46,11 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
 
 ## Active Deployment
 
+- [[freetoken-frank-default-small]] — the active frank-default-small
+  runbook: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 on Dankserver
+  (2026-09-02)
+- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
+  (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
 - [[deployment]] — deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
 - [kingjones Qwen3.8 Uncensored ROCmFP4 STRIX MTP](https://huggingface.co/kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF) — active frank-default-large model on Dankclaw; Q6 AGENT main model with standalone MTP
@@ -54,9 +59,9 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
 - [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
 - [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
   on Dankclaw (retired)
-- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
+- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
-  Dankserver (frank-default-small, active 2026-08-24)
+  Dankserver (frank-default-small, retired 2026-09-02)
-- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for
+- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
   frank-default-small (2026-08-26): q8_0 + Hadamard KV cache,
   split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
 - [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP

docs/LLM/kingjones-strix-rocmfp4.md

@@ -0,0 +1,195 @@
+---
+type: reference
+title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
+description: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the ROCmFPX llama.cpp fork. 177B params (56B MoE + 51.2B PLE), single 113.47 GiB GGUF. frank-default-large.
+tags:
+  - llama.cpp
+  - rocm
+  - dankclaw
+  - gfx1151
+  - kingjones
+  - rocmfp4
+  - strix-halo
+  - qwen4exp
+timestamp: 2026-08-27
+---
+
+# kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
+
+> **Active since 2026-08-27.** Replaced the STRIX_LEAN deployment on the same
+> build. Same GPU memory profile, larger PLE table.
+
+`frank-default-large` (port 7770) serves
+`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX` from `kingjones777` on Hugging Face.
+A single GGUF file, 113.47 GiB. Text-only (no mmproj, no MTP).
+
+## Model Architecture
+
+Base model: Qwen3.8-Flash-Next (177B parameters total).
+
+| Component | Params | Details |
+|-----------|--------|---------|
+| MoE experts | 125.8B | 512 experts, 10 used per token |
+| Attention | dense | 48 layers (12 attn + 36 SSM) |
+| PLE table | 51.2B | `per_layer_token_embd`, Q8_0 |
+| lm head | ~0.3% | Q6_K |
+
+The PLE (per-layer embedding) n-gram table is the only thing that differs
+between STRIX, STRIX_LEAN, and FAST tiers. Everything else is byte-identical.
+
+## File Details
+
+| Field | Value |
+|-------|-------|
+| File | `Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` |
+| Size | 113.47 GiB |
+| Format | Single GGUF (not sharded) |
+| file_type | 105 (`Q4_0_ROCMFP4_STRIX`) |
+| General | 113.47 GiB, 5.51 bpw |
+
+Tensor composition:
+
+| Tensor group | Type |
+|---|---|
+| MoE expert weights (`ffn_*_exps`, 144) | TYPE_101 (ROCmFP4, 4.251 bpw) |
+| Shared expert (`ffn_*_shexp`, 144) | TYPE_101 |
+| Attention (`attn_*`, 120) | half TYPE_100, half TYPE_101 |
+| PLE (`per_layer_token_embd.weight`) | Q8_0 — 50.66 GiB |
+| `token_embd.weight` | Q6_K |
+| `output.weight` (lm head) | Q6_K |
+| Norms / biases | F32 |
+
+## File Type Reference
+
+| Tier | file_type | PLE table | Total size |
+|------|-----------|-----------|------------|
+| FAST | TYPE_106 | 25.34 GiB | 87.94 GiB |
+| STRIX_LEAN | TYPE_105 (old STRIX_LEAN) | 35.76 GiB | 98.49 GiB |
+| **STRIX** | **105 (`Q4_0_ROCMFP4_STRIX`)** | **50.66 GiB** | **113.47 GiB** |
+
+TYPE_100 = 4.506 bpw, TYPE_101 = 4.251 bpw (measured exactly by solving
+three files for two unknowns).
+
+## Memory
+
+The PLE table never enters GPU VRAM. It streams from the file through the OS
+page cache.
+
+| Context | GTT at load | GTT after prompt | GPU resident |
+|---------|-------------|-------------------|-------------|
+| 8,192 | 63.3 GiB | 63.6 GiB | ~63 GiB |
+| 32,768 | 64.1 GiB | 64.8 GiB | |
+| 65,536 | 65.1 GiB | 66.2 GiB | |
+| 131,072 | 67.2 GiB | 69.1 GiB | ~69 GiB |
+
+~63 GiB is the non-PLE mass plus compute buffers. **The PLE table does not
+grow GPU memory regardless of context depth.**
+
+Three critical consequences:
+
+1. **`--no-mmap` gets you OOM-killed.** It forces the PLE into anonymous memory
+   (neither file-backed nor reclaimable). The cgroup killer takes the process
+   with nothing in the server log — only `dmesg` shows the evidence.
+2. **Don't force PLE to CPU with `-ot`.** Measured generation dropped from
+   ~23 to 13.4 tok/s. The kernel streams it better than any offload hack.
+3. **A bigger tier costs disk, not GPU memory.** If you have storage, take
+   STRIX.
+
+## Performance
+
+Measured on Ryzen AI MAX+ 395 (gfx1151, ROCm 7.2.4), 49/49 offload. Prompts
+sized to 85% of context window:
+
+| Context | Prompt tokens | pp tok/s | gen tok/s |
+|---------|--------------|----------|-----------|
+| 8,192 | 6,963 | 279 | **22.02** |
+| 32,768 | 27,852 | 264 | **20.52** |
+| 65,536 | 55,705 | 234 | **18.23** |
+| 131,072 | 111,411 | 185 | **15.34** |
+
+Short prompts (~3,300 tokens): 381 tok/s prompt processing, 22.7 tok/s
+generation (median of 3 runs).
+
+## Serving Profile
+
+Runtime: charlie12345/ROCmFPX `llama-server` built at
+`/home/dank/GitPackages/ROCmFPX/build-qwen38`.
+
+Required flags:
+
+```
+llama-server \
+  --model Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf \
+  --host 0.0.0.0 --port 7770 --jinja \
+  --device ROCm0 --gpu-layers 999 \
+  --fit off --flash-attn on \
+  --ctx-size 524288 --parallel 2 \
+  --batch-size 4096 --ubatch-size 4096 \
+  --cache-type-k f16 --cache-type-v f16 \
+  --threads 16 --threads-batch 32 \
+  --ctx-checkpoints 16 --cache-ram 8192 \
+  --cache-prompt --cache-idle-slots --metrics \
+  --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 \
+  --presence-penalty 1.5 \
+  --reasoning on --reasoning-format deepseek --reasoning-budget -1 \
+  --chat-template-file chat-template.jinja \
+  --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}'
+```
+
+Key choices:
+
+- **`--fit off`** — auto-fit code segfaults on this model's qwen4exp
+  hyper-connection/indexer configuration.
+- **mmap (default)** — do NOT use `--no-mmap`. See memory section.
+- **`--gpu-layers 999`** — all layers on SVM-backed GPU pool.
+- **`--ctx-size 524288`** — above model's max (131072); llama.cpp clamps
+  internally.
+
+## Deployment
+
+### Download
+
+```bash
+# on Dankclaw
+TOKEN=$(cat ~/.cache/huggingface/token)
+DIR=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX
+mkdir -p "$DIR"
+curl -fsSL -C - -o "$DIR/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf" \
+  --retry 5 --retry-delay 3 --retry-max-time 7200 \
+  -H "Authorization: Bearer $TOKEN" \
+  "https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf"
+```
+
+The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.
+
+### Swap procedure
+
+1. Stop service: `sudo systemctl stop frank-default-large`
+2. Remove old model dir: `rm -rf /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN`
+3. Start service: `sudo systemctl start frank-default-large`
+4. Verify health: `curl http://192.168.0.143:7770/health`
+
+### Config files
+
+- Env: `scripts/frank-default-large.env`
+- Serve: `scripts/serve-frank-large.sh`
+- Chat template: `scripts/chat-template.jinja`
+
+## Known Issues
+
+1. **Patched llama.cpp required.** PR #27742 merged into the ROCmFPX fork.
+   Stock builds reject these files (TYPE_100/101 have no number in stock ggml).
+2. **Single-file payload.** Not sharded. Use `hf download` or a resumable
+   curl. Browser download unreliable for 113 GiB.
+3. **Model load is slow.** A few minutes cold for the 113 GiB file with a
+   fresh KV cache. Health returns 503 while loading. Do not kill the process.
+   `deploy.sh` waits up to 20 minutes.
+4. **Kernel spam about `amdgpu: SVM mapping failed`** is a known documented
+   symptom, not root cause. Investigate other causes before assuming memory
+   pressure.
+
+## Citations
+
+- [kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF)
+- [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
+- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — fork with PR #27742 (qwen4exp architecture)

docs/index.md

@@ -105,13 +105,17 @@ The LLM inference fleet on the two hosts. Full index:
 
 - [[deployment]] — unified deploy and update process for both servers
 - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
+- [[freetoken-frank-default-small]] — the active frank-default-small runbook: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 on Dankserver (2026-09-02)
+- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
 - [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (restored 2026-09-01)
+- [[kingjones-strix-rocmfp4]] — the active frank-default-large runbook:
+  kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF (177B params, 113.47 GiB)
 - [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
 - [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw
 - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
 - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
-- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (active)
+- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02)
-- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for frank-default-small (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26)
+- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for frank-default-small (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26)
 - [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22)
 - [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21)
 - [[Dankserver System Specs]] — the RTX 3090 box hardware

docs/log.md

@@ -30,10 +30,10 @@ timestamp: 2026-09-14
 - `/scripts/deploy.sh` and `/scripts/deploy-remote.sh` — Removed standalone proxy dispatch and added idempotent retirement before canonical Caddy deployment.
 - `/tests/caddyfile.bats`, `/tests/deployment_scoping.bats`, `/tests/deployment_flow.bats`, `/tests/frank_large_config.bats` — Added migration coverage and corrected the existing ubatch harness assertion.
 - `/docs/index.md` — Bundle index remains the root navigation entry for the updated pages.
-- **Boot-persistence audit (post-reboot, 2026-08-26 evening):** after Frank's reboot, Caddy and Pi-hole were both down. Caddy ran as a bare compose container (`restart: unless-stopped` does not survive shutdown's explicit stop; rootless Podman has no boot-time auto-start for bare containers). Pi-hole's quadlet existed but crash-looped (restart counter 1645): a later-sorted sysctl drop-in (`99-rootless-ports.conf`) reset `net.ipv4.ip_unprivileged_port_start` to 80, so the rootless port-53 publish failed with `bind: permission denied`.
+- **Boot-persistence audit (post-reboot, 2026-08-26 evening):** after Frank's reboot, Caddy and Pi-hole were both down. Caddy ran as a bare compose container (`restart: unless-stopped` does not survive shutdown's explicit stop; rootless Podman has no boot-time auto-start for bare containers). Pi-hole's quadlet existed but crash-looped (restart counter 1645): `net.ipv4.ip_unprivileged_port_start` was reset to 80 by two sources — a later-sorted sysctl drop-in (`99-rootless-ports.conf`) and a stale line in `/etc/sysctl.conf`, which systemd-sysctl applies **after** all of `sysctl.d/` (the first deploy pass only fixed the drop-in; the runtime value stayed 80 until the `/etc/sysctl.conf` line was neutralized too).
 - `/AGENTS.md` — new MANDATORY section "Boot Persistence & Self-Healing": every deployed, proven-working container must have a self-healing, boot-persistent quadlet unit (`WantedBy=default.target` + enabled + linger, `Restart=always`, repo-provenance `.container`), verified before a service is declared deployed.
 - `/dankclaw/caddy/quadlet/caddy.container`, `/dankclaw/caddy/env.example` — Caddy is now a boot-persistent quadlet (`caddy.service`): joins `dankclaw_default` so container-name upstreams keep resolving, `Pull=never` (local-only image), Porkbun credentials via `/home/dank/Containers/caddy/env` (deploy writes it from `.env`), bare-IP-catch-all health check. The caddy service was removed from `/dankclaw/compose.yaml`.
-- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` installs it to `/etc/sysctl.d/`, stages conflicting later-sorted drop-ins in the home trash, and applies `sysctl --system` (the boot code path).
+- `/dankclaw/pihole/sysctl/99-dnsmasq.conf` — repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` installs it to `/etc/sysctl.d/`, stages conflicting drop-ins (value > 53) in the home trash, comments out a stale `/etc/sysctl.conf` line (applied after all of sysctl.d), and applies `sysctl --system` (the boot code path).
-- `/scripts/deploy-remote.sh` — `deploy_caddy` installs/enables/starts the quadlet (retiring the bare container; all state is in host volumes); `deploy_pihole` enforces the sysctl prerequisite.
+- `/scripts/deploy-remote.sh` — `deploy_caddy` installs/starts the quadlet (retiring the bare container; all state is in host volumes). Boot persistence comes from podman's quadlet generator, which creates the `default.target.wants` symlink from the unit's `[Install]` section at every user-manager start — `is-enabled` reports `generated` for all healthy quadlets, and an explicit `systemctl enable` fails on generated units (learned live during the deploy). `deploy_pihole` enforces the sysctl prerequisites.
 - `/tests/caddy_quadlet.bats` (new seam), `/tests/caddyfile.bats`, `/tests/deployment_flow.bats`, `/tests/pihole.bats` — quadlet boot-persistence, self-healing, network, env, and sysctl coverage; compose assertions repointed from the removed caddy service to the quadlet.
 - `/docs/Infrastructure/Caddy.md`, `/docs/Infrastructure/Pi-hole.md`, `/docs/Dankclaw/Dankclaw Stack.md` — quadlet runtime facts, the 2026-08-26 incident, and the sysctl-conflict failure mode.

scripts/deploy-remote.sh

@@ -322,8 +322,13 @@ deploy_caddy() {
     # lives in host volumes, so removing the container loses nothing.
     podman stop caddy >/dev/null 2>&1 || true
     podman rm caddy >/dev/null 2>&1 || true
+    # install_quadlet runs daemon-reload, which re-runs podman's quadlet
+    # generator: it creates caddy.service AND the default.target.wants symlink
+    # from the unit's [Install] section (is-enabled reports "generated" —
+    # that is the healthy state for a quadlet; an explicit systemctl enable
+    # fails on generated units). Boot persistence comes from the generator,
+    # same as pihole/gitea/searxng/open-webui.
     install_quadlet caddy dankclaw/caddy/quadlet caddy
-    systemctl --user enable caddy.service
     systemctl --user restart caddy.service
     if podman container exists caddy; then
         local reloaded=0
@@ -449,16 +454,28 @@ deploy_open_webui() {
 
 deploy_pihole() {
     # Rootless port 53 needs net.ipv4.ip_unprivileged_port_start <= 53, and it
-    # must hold at boot (systemd-sysctl applies /etc/sysctl.d). The repo
+    # must hold at boot. The repo drop-in is the single source of truth.
-    # drop-in is the single source of truth; a later-sorted file that resets
+    # systemd-sysctl applies /etc/sysctl.d/*.conf (lexicographic), then
-    # the value to 80 took LAN DNS down after the 2026-08-26 reboot, so stage
+    # /run/sysctl.d, /usr/lib/sysctl.d, and finally /etc/sysctl.conf — later
-    # any such conflict in the home trash instead of deleting it.
+    # files override earlier ones. On 2026-08-26 a later-sorted drop-in (=80)
+    # AND a stale line in /etc/sysctl.conf (=80) both reset the value after a
+    # reboot and took LAN DNS down. Neutralize every conflicting source:
+    # stage sysctl.d files that set a value above 53 in the home trash, and
+    # comment out an active line in /etc/sysctl.conf (idempotent).
     run_privileged install -m 0644 dankclaw/pihole/sysctl/99-dnsmasq.conf \
         /etc/sysctl.d/99-dnsmasq.conf
-    if grep -qs 'ip_unprivileged_port_start' /etc/sysctl.d/99-rootless-ports.conf; then
+    local sysctl_file sysctl_val sysctl_trash
-        local sysctl_trash="$HOME/.AGENTTRASH/$(date +%Y%m%d)-sysctl-rootless-ports"
+    while IFS= read -r sysctl_file; do
-        mkdir -p "$sysctl_trash"
+        [[ "$sysctl_file" == */99-dnsmasq.conf ]] && continue
-        run_privileged mv /etc/sysctl.d/99-rootless-ports.conf "$sysctl_trash/"
+        sysctl_val="$(sed -nE 's/^[[:space:]]*net\.ipv4\.ip_unprivileged_port_start[[:space:]]*=[[:space:]]*([0-9]+).*/\1/p' "$sysctl_file" | head -1)"
+        if [[ "$sysctl_val" =~ ^[0-9]+$ ]] && (( sysctl_val > 53 )); then
+            sysctl_trash="$HOME/.AGENTTRASH/$(date +%Y%m%d)-sysctl-unprivileged-port"
+            mkdir -p "$sysctl_trash"
+            run_privileged mv "$sysctl_file" "$sysctl_trash/"
+        fi
+    done < <(grep -lE '^[[:space:]]*net\.ipv4\.ip_unprivileged_port_start[[:space:]]*=' /etc/sysctl.d/*.conf 2>/dev/null || true)
+    if [[ -f /etc/sysctl.conf ]] && grep -qE '^[[:space:]]*net\.ipv4\.ip_unprivileged_port_start[[:space:]]*=' /etc/sysctl.conf; then
+        run_privileged sed -i -E 's/^[[:space:]]*(net\.ipv4\.ip_unprivileged_port_start[[:space:]]*=)/# \1 managed by dankclaw\/pihole\/sysctl\/99-dnsmasq.conf/' /etc/sysctl.conf
     fi
     run_privileged sysctl --system >/dev/null
     mkdir -p /home/dank/Containers/pihole/etc-pihole /home/dank/Containers/pihole/etc-dockers

scripts/frank-default-large.env

@@ -1,5 +1,5 @@
 # Tunables for frank-default-large (kingjones777 Qwen3.8-Flash-Next
-# ROCmFP4 STRIX_LEAN, qwen4exp hybrid MoE, Dankclaw, Strix Halo gfx1151).
+# ROCmFP4 STRIX, qwen4exp hybrid MoE, Dankclaw, Strix Halo gfx1151).
 #
 # Runtime: ROCmFPX llama.cpp fork with PR #27742 (qwen4exp architecture)
 # merged and built at /home/dank/GitPackages/ROCmFPX/build-qwen38.
@@ -7,18 +7,22 @@
 # -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DGGML_CUDA_FA_ALL_QUANTS=ON
 # -DLLAMA_BUILD_WEBUI=OFF -DCMAKE_EXE_LINKER_FLAGS=-no-pie.
 # Replaced the kingjones Qwen3.8-27B Q6_0 AGENT deployment on 2026-09-01.
+# Replaced the STRIX_LEAN deployment on 2026-08-27.
 #
-# Model: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN-GGUF.
+# Model: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF.
-# 56B MoE (512 experts, 10 used), 48 layers (12 attention + 36 SSM,
+# 177B params (51.2B PLE n-gram table + 125.8B total MoE/attention).
-# full_attention_interval=4), 2 KV heads × 256 dim, embedding 2560,
+# Qwen3.8-Flash-Next base: 56B MoE (512 experts, 10 used), 48 layers
-# QSA indexer (4 heads, key_length 128, top_k 2048), hyper-connections
+# (12 attention + 36 SSM, full_attention_interval=4), 2 KV heads x 256 dim,
+# embedding 2560, QSA indexer (4 heads, top_k=2048), hyper-connections
 # (count=4, low_rank=320), PLE (layer 1, ngram_size=3, heads_per_ngram=8).
-# Q4_0_ROCMFP4_STRIX_LEAN quant, 98.5 GiB, 3-part GGUF split.
+# Q4_0_ROCMFP4_STRIX quant (file_type=105), 113.47 GiB, single GGUF file.
+# PLE table: 50.66 GiB Q8_0, streamed from page cache (never in GPU VRAM).
 # Text-only (no mmproj, no MTP draft head).
 #
-# Context: 131072 (128k) × 1 slot.  f16 KV cache.  --fit off because the
+# Context: 262144 (256k) x 2 slots.  f16 KV cache (~6.4 GiB per slot,
-# auto-fit code segfaults on this model's hyper-connection/indexer config;
+# ~12.8 GiB total).  --fit off because the auto-fit code segfaults on this
-# -ngl 999 places all layers on the SVM-backed GPU pool (116 GiB unified).
+# model's hyper-connection/indexer config; -ngl 999 places all layers on
+# the SVM-backed GPU pool (116 GiB unified).
 #
 # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
 # ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
@@ -29,9 +33,9 @@
 LLAMA_BIN=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server
 
 # Model files from
-# https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN-GGUF
+# https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF
-MODEL_DIR=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN
+MODEL_DIR=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX
-MODEL_FILE=Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN-00001-of-00003.gguf
+MODEL_FILE=Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf
 
 # Qwen-Sharp chat template (tracked in the repo, staged on the host). The
 # Sharp template drives thinking on by default (medium effort); the server
@@ -45,12 +49,13 @@ ALIAS=frank-default-large
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7770
 
-# Serving: 131072 context (128k) × 1 slot. f16 KV, context checkpoints,
+# Serving: f16 KV, context checkpoints, prompt cache. The model is 113.47
-# prompt cache. The model is 98.5 GiB loaded via mmap into the Strix Halo
+# GiB loaded via mmap into the Strix Halo unified memory pool (116 GiB
-# unified memory pool (116 GiB SVM-backed). --fit off avoids the auto-fit
+# SVM-backed). PLE table (50.66 GiB Q8_0) is streamed from page cache; never
+# enters GPU VRAM. GPU resident: ~63-69 GiB. --fit off avoids the auto-fit
 # segfault on this model's qwen4exp hyper-connection/indexer configuration.
-CTX_SIZE=131072
+CTX_SIZE=524288
-PARALLEL=1
+PARALLEL=2
 BATCH_SIZE=4096
 UBATCH_SIZE=4096
 NUM_THREADS=16

scripts/frank-default-small.env

@@ -1,71 +1,51 @@
-# Tunables for frank-default-small (Tiel-Coder-35B-A3B UD-Q6_K_XL,
+# Tunables for frank-default-small (Qwen3.6-35B-A3B-FP8,
-# qwen35moe hybrid MoE, Dankserver, RTX 3090).
+# qwen3_5_moe hybrid MoE, Dankserver, RTX 3090).
 #
-# Runtime: upstream llama.cpp (latest master, CUDA build with
+# Runtime: FreeToken v0.1.2 (edge-native MoE serving engine).
-# -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86) on RTX 3090
+# Installed via uv into /home/dank/freetoken/.venv (CPython 3.12).
-# 24 GB; Ryzen 7 7700X; 62 GB RAM.  Replaced the ik_llama.cpp Ornith
+# FreeToken keeps expert weights in host RAM with an LRU GPU cache;
-# CRACK Q8_0 deployment on 2026-09-01.
+# KV is paged with radix prefix sharing.  Only the 10 full-attention
+# layers page KV (~20 KiB/token); the 30 GDN layers use constant-size
+# per-sequence state (mamba slot pool).
 #
-# Model: peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF, UD-Q6_K_XL quant
+# Model: Qwen/Qwen3.6-35B-A3B-FP8 (HuggingFace safetensors, modelopt
-# (30 GB).  Architecture: qwen35moe — 40 layers, full_attention_interval=4
+# block-fp8 quant, 128x128 weight_block_size, 35 GB on disk).
-# (10 attention + 30 SSM/Mamba), 256 experts / 8 used, 2 KV heads × 256
+# Architecture: qwen3_5_moe — 40 layers (10 attention + 30 GDN),
-# dim, embedding 2048, rope dimension_count 64, rope freq_base 10 000 000.
+# 256 experts / 8 used, 2 KV heads × 256 dim, full_attention_interval=4.
-# Text-only (no mmproj).  Built-in chat template in the GGUF.
 #
-# Expert offload: --cpu-moe keeps all 256 expert weights in CPU RAM.
+# Replaced the llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment on
-# Non-expert weights (~1.6 GB) fit entirely on the 24 GB card.
+# 2026-09-02.  The old serve script and env are in .AGENTTRASH/.
 #
-# KV cache: Q8_0 for both K and V.  With CTX_SIZE=200700 (66 900 per slot
+# Pool sizing: GPU budget (~17.6 GiB after weights + fixed caches) is
-# × 3 parallel) and 10 attention layers, the KV cache is ~2 GB — the
+# split MoE-first.  MOE_CACHE_SIZE pins resident expert slots; NUM_TOKENS
-# requested 2 GB VRAM buffer.
+# pins the KV pool.  At 2048 slots / 453005 tokens the split is ~20%
+# residency (2048/10240) and ~8.6 GiB KV.  Live resize via
+# `ft ctl cache --kv N --moe M` can grow KV by shrinking MoE without
+# restart (the reverse requires a restart).
 #
-# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The
+# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE.
-# ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
+# The ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
-# incident). This box is NVIDIA/CUDA, but the rule keeps the env portable.
+# incident).  This box is NVIDIA/CUDA, but the rule keeps the env portable.
 
-# Model file (UD-Q6_K_XL, 30 GB, text-only MoE hybrid).
+# Model directory (safetensors checkpoint, 35 GB).
-MODEL_DIR=/home/dank/models/tiel-coder-35b
+MODEL_DIR=/home/dank/models/qwen36-35b-a3b-fp8
-MODEL_FILE=Tiel-Coder-35B-A3B-UD-Q6_K_XL.gguf
 
-# Upstream llama.cpp llama-server (CUDA build, latest master).
+# FreeToken venv (uv-managed CPython 3.12, ft CLI, flashinfer, torch+cu130).
-# Built with -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86
+FT_VENV=/home/dank/freetoken/.venv
-# using the staged CUDA 13.2 toolkit at /home/dank/cuda13-stage.
+CUDA_HOME_DIR=/home/dank/freetoken/.venv/lib/python3.12/site-packages/nvidia/cu13
-LLAMA_SERVER=/home/dank/llama.cpp/build/bin/llama-server
 
 ALIAS=frank-default-small
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=7770
 
-# Total context budget across all parallel slots.  200700 / 3 = 66 900
+# Expert residency: 2048 of 10240 total expert instances (20%).
-# per slot.  The model's context_length is 262 144, so 66 900 is well
+# Each slot ~3 MiB → ~6.0 GiB VRAM for the MoE cache.
-# within range.
+MOE_CACHE_SIZE=2048
-CTX_SIZE=200700
-PARALLEL=3
 
-# Expert offload: all MoE expert weights pinned to CPU RAM.
+# KV pool: 453005 tokens (~8.6 GiB at ~20 KiB/token).
-CPU_MOE=1
+# Covers 4 concurrent sessions at ~113k unique context each
+# (more with radix prefix sharing for common prefixes).
+NUM_TOKENS=453005
 
-# All non-expert layers on GPU (auto-fit will determine the exact count;
+# Max concurrent decoding requests (also caps mamba slot allocation).
-# with --cpu-moe the non-expert footprint is ~1.6 GB, leaving ample room).
+MAX_RUNNING_REQUESTS=4
-N_GPU_LAYERS=999
-
-# KV cache: Q8_0 for both K and V.  ~2 GB total for 200700 ctx × 3 slots
-# × 10 attention layers × (2 KV heads × 256 dim × 1 byte) × 2 (K+V).
-CACHE_TYPE_K=q8_0
-CACHE_TYPE_V=q8_0
-
-VERBOSITY=2
-
-# Prefill batch sizing.
-BATCH_SIZE=4096
-UBATCH_SIZE=1024
-
-# Sampling.
-TEMPERATURE=1.0
-TOP_K=20
-TOP_P=0.95
-MIN_P=0.0
-
-# Compute threads lowered so other processes keep a core.
-THREADS=7
-THREADS_HTTP=8

scripts/frank-default-small.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=llama.cpp Tiel-Coder-35B-A3B Q6_K_XL qwen35moe hybrid MoE (dankserver, frank-default-small)
+Description=FreeToken Qwen3.6-35B-A3B-FP8 qwen3_5_moe hybrid MoE (dankserver, frank-default-small)
 After=network-online.target
 Wants=network-online.target
 
@@ -13,4 +13,4 @@ Restart=on-failure
 RestartSec=10
 
 [Install]
-WantedBy=multi-user.target
+WantedBy=multi-user.target

scripts/serve-frank-large.sh

@@ -1,5 +1,5 @@
 #!/usr/bin/env bash
-# Serve kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN as
+# Serve kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX as
 # frank-default-large via the charlie12345/ROCmFPX llama.cpp fork with
 # PR #27742 (qwen4exp architecture) merged, on Dankclaw (Strix Halo
 # gfx1151, unified memory).
@@ -9,17 +9,17 @@
 # with -DGGML_HIP=ON -DGGML_HIP_ROCMI4_W4A4=ON
 # -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DLLAMA_BUILD_WEBUI=OFF.
 #
-# Model: 56B MoE (512 experts, 10 used), 48 layers (12 attention + 36
+# Model: 177B params (56B MoE + 51.2B PLE n-gram table). 512 experts,
-# SSM/Mamba, full_attention_interval=4), 2 KV heads × 256 dim, QSA
+# 10 used, 48 layers (12 attention + 36 SSM/Mamba,
+# full_attention_interval=4), 2 KV heads x 256 dim, QSA
 # indexer (4 heads, top_k=2048), hyper-connections (count=4, low_rank=320),
-# PLE (layer 1).  Q4_0_ROCMFP4_STRIX_LEAN quant, 98.5 GiB, 3-part split.
+# PLE (layer 1).  Q4_0_ROCMFP4_STRIX quant, 113.47 GiB, single file.
-#
+# PLE table (50.66 GiB Q8_0) streamed from page cache; never in GPU VRAM.
-# Context: 131072 (128k) × 1 slot.  f16 KV cache.  Model is loaded with
+# GPU resident: ~63-69 GiB. Context: up to 131072.
-# mmap (default) — the 98.5 GiB file is memory-mapped; the Strix Halo
+# Loaded with mmap (default) into the Strix Halo unified memory pool.
-# unified memory (116 GiB SVM-backed) provides the GPU address space.
 # --fit off is set because the auto-fit code segfaults on this model's
 # hyper-connection/indexer configuration; -ngl 999 places all layers on
-# the GPU (SVM-backed), which the unified memory pool handles.
+# the GPU (SVM-backed).
 #
 # The Qwen-Sharp chat template (scripts/chat-template.jinja) is applied at
 # runtime so the model emits reasoning in DeepSeek format; the reasoning
@@ -37,15 +37,15 @@ fi
 
 # --- defaults (used when the env file does not set them) -------------------
 : "${LLAMA_BIN:=/home/dank/GitPackages/ROCmFPX/build-qwen38/bin/llama-server}"
-: "${MODEL_DIR:=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN}"
+: "${MODEL_DIR:=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX}"
-: "${MODEL_FILE:=Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN-00001-of-00003.gguf}"
+: "${MODEL_FILE:=Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf}"
 : "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
 : "${CHAT_TEMPLATE_KWARGS:={}}"
 : "${ALIAS:=frank-default-large}"
 : "${BIND_ADDR:=0.0.0.0}"
 : "${LISTEN_PORT:=7770}"
-: "${CTX_SIZE:=131072}"
+: "${CTX_SIZE:=524288}"
-: "${PARALLEL:=1}"
+: "${PARALLEL:=2}"
 : "${BATCH_SIZE:=4096}"
 : "${UBATCH_SIZE:=4096}"
 : "${NUM_THREADS:=16}"

scripts/serve-frank-small.sh

@@ -1,23 +1,25 @@
 #!/usr/bin/env bash
-# Serve peculiar-ragdoll/Tiel-Coder-35B-A3B (UD-Q6_K_XL) as
+# Serve Qwen/Qwen3.6-35B-A3B-FP8 as frank-default-small via FreeToken
-# frank-default-small via stock llama.cpp on Dankserver (RTX 3090).
+# on Dankserver (RTX 3090, 60 GB RAM).
 #
-# Runtime: upstream llama.cpp (latest master, CUDA build with
+# Runtime: FreeToken v0.1.2 (edge-native MoE serving engine).
-# -DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86).
+#   - Experts live in host RAM; an LRU cache of expert slots is resident
-# The model is a 35B-total / 8-active MoE hybrid (qwen35moe arch) with
+#     on the GPU. Misses are streamed over PCIe.
-# SSM/Mamba layers (full_attention_interval=4: 10 attention + 30 SSM
+#   - Paged KV cache (radix prefix sharing).  Only the 10 full-attention
-# layers out of 40).  256 experts, 8 used per token.
+#     layers page KV (~20 KiB/token); the 30 GDN/linear-attention layers
+#     hold constant-size per-sequence state (mamba slots).
+#   - OpenAI-compatible API (/v1/chat/completions, /v1/models).
 #
-# Expert offload: --cpu-moe keeps all 256 expert weights in CPU RAM.
+# Model: Qwen/Qwen3.6-35B-A3B-FP8 (safetensors, modelopt block-fp8,
-# Non-expert weights (attention, SSM, shared expert, embeddings, output)
+# 128x128 weight_block_size).  Architecture: qwen3_5_moe — 40 layers,
-# are ~1.6 GB and fit entirely on the 24 GB RTX 3090.
+# full_attention_interval=4 (10 attention + 30 GDN), 256 experts / 8 used,
+# 2 KV heads × 256 dim.  35 GB on disk.
 #
-# KV cache: Q8_0 for both K and V.  With CTX_SIZE=200700 (66 900 per
+# Pool sizing: the GPU budget is split MoE-first between expert residency
-# slot × 3 parallel) and 10 attention layers (2 KV heads × 256 dim),
+# and KV pages.  --moe-cache-size pins the number of resident expert
-# the KV cache is ~2 GB — the requested 2 GB VRAM buffer.
+# slots; --num-tokens pins the KV pool capacity.  Live resize is possible
-#
+# via `ft ctl cache --kv N --moe M` (grows KV by shrinking MoE; shrinking
-# Model is loaded with mmap (default).  The 30 GB Q6_K_XL file is
+# KV requires a restart due to the swap-during-rebuild memory cost).
-# memory-mapped; only accessed pages enter RAM.
 #
 # Tunables come from frank-default-small.env unless FRANK_SMALL_ENV_FILE
 # overrides that path.
@@ -33,55 +35,39 @@ if [[ -f "$ENV_FILE" ]]; then
 fi
 
 # --- defaults (used when the env file does not set them) -----------------
-: "${MODEL_DIR:=/home/dank/models/tiel-coder-35b}"
+: "${MODEL_DIR:=/home/dank/models/qwen36-35b-a3b-fp8}"
-: "${MODEL_FILE:=Tiel-Coder-35B-A3B-UD-Q6_K_XL.gguf}"
-: "${LLAMA_SERVER:=/home/dank/llama.cpp/build/bin/llama-server}"
 : "${ALIAS:=frank-default-small}"
 : "${BIND_ADDR:=0.0.0.0}"
 : "${LISTEN_PORT:=7770}"
-: "${CTX_SIZE:=200700}"
+: "${FT_VENV:=/home/dank/freetoken/.venv}"
-: "${PARALLEL:=3}"
+: "${CUDA_HOME_DIR:=${FT_VENV}/lib/python3.12/site-packages/nvidia/cu13}"
-: "${CACHE_TYPE_K:=q8_0}"
-: "${CACHE_TYPE_V:=q8_0}"
-: "${N_GPU_LAYERS:=999}"
-: "${BATCH_SIZE:=4096}"
-: "${UBATCH_SIZE:=1024}"
-: "${VERBOSITY:=2}"
-: "${CPU_MOE:=1}"
-: "${TEMPERATURE:=1.0}"
-: "${TOP_K:=20}"
-: "${TOP_P:=0.95}"
-: "${MIN_P:=0.0}"
-: "${THREADS:=7}"
-: "${THREADS_HTTP:=8}"
 
-MODEL="$MODEL_DIR/$MODEL_FILE"
+# Pool sizing (expert residency vs KV cache tradeoff).
+: "${MOE_CACHE_SIZE:=2048}"
+: "${NUM_TOKENS:=453005}"
+: "${MAX_RUNNING_REQUESTS:=4}"
 
-if [[ ! -f "$MODEL" ]]; then
+# Paths
-    echo "missing required file: $MODEL" >&2
+FT_BIN="${FT_VENV}/bin/ft"
+MODEL="${MODEL_DIR}"
+
+if [[ ! -x "$FT_BIN" ]]; then
+    echo "missing FreeToken CLI: $FT_BIN" >&2
     exit 1
 fi
-if [[ ! -x "$LLAMA_SERVER" ]]; then
+if [[ ! -d "$MODEL" ]]; then
-    echo "missing llama-server binary: $LLAMA_SERVER" >&2
+    echo "missing model directory: $MODEL" >&2
     exit 1
 fi
 
-exec "$LLAMA_SERVER" \
+export PATH="${FT_VENV}/bin:${PATH}"
+export CUDA_HOME="${CUDA_HOME_DIR}"
+
+exec "$FT_BIN" serve \
     --model "$MODEL" \
-    $([ "$CPU_MOE" = "1" ] && printf -- "--cpu-moe") \
+    --served-model-name "$ALIAS" \
-    --cache-type-k "$CACHE_TYPE_K" \
-    --cache-type-v "$CACHE_TYPE_V" \
-    --ctx-size "$CTX_SIZE" \
-    --parallel "$PARALLEL" \
-    --gpu-layers "$N_GPU_LAYERS" \
-    --batch-size "$BATCH_SIZE" \
-    --ubatch-size "$UBATCH_SIZE" \
-    --temp "$TEMPERATURE" \
-    --top-k "$TOP_K" \
-    --top-p "$TOP_P" \
-    --min-p "$MIN_P" \
-    --threads "$THREADS" \
-    --threads-http "$THREADS_HTTP" \
-    --alias "$ALIAS" \
     --host "$BIND_ADDR" \
-    --port "$LISTEN_PORT"
+    --port "$LISTEN_PORT" \
+    --moe-cache-size "$MOE_CACHE_SIZE" \
+    --num-tokens "$NUM_TOKENS" \
+    --max-running-requests "$MAX_RUNNING_REQUESTS"

tests/deployment_flow.bats

@@ -56,10 +56,13 @@ load helpers
 
 @test "canonical Caddy deployment installs its boot-persistent quadlet" {
     # 2026-08-26: a bare compose container never came back after reboot.
-    # deploy_caddy must install, enable, and start the quadlet unit instead
+    # deploy_caddy must install and start the quadlet unit instead of
-    # of `podman-compose up`.
+    # `podman-compose up`. Boot persistence comes from the quadlet generator's
+    # default.target.wants symlink (is-enabled reports "generated"), so there
+    # is no explicit systemctl enable — it fails on generated units.
     grep -q 'install_quadlet caddy dankclaw/caddy/quadlet caddy' "$REPO_ROOT/scripts/deploy-remote.sh"
-    grep -q 'systemctl --user enable caddy.service' "$REPO_ROOT/scripts/deploy-remote.sh"
+    grep -q 'systemctl --user restart caddy.service' "$REPO_ROOT/scripts/deploy-remote.sh"
+    ! grep -q 'systemctl --user enable caddy.service' "$REPO_ROOT/scripts/deploy-remote.sh"
     ! grep -q 'podman-compose -f dankclaw/compose.yaml up -d --no-deps caddy' "$REPO_ROOT/scripts/deploy-remote.sh"
     ! grep -q 'no proxy was stopped or replaced' "$REPO_ROOT/scripts/deploy-remote.sh"
 }

tests/pihole.bats

@@ -62,6 +62,12 @@ HOSTS="$PIHOLE_DIR/addn-hosts"
     grep -q 'sysctl --system' "$REPO_ROOT/scripts/deploy-remote.sh"
 }
 
+@test "deploy_pihole neutralizes the sysctl.conf override (processed after sysctl.d)" {
+    # systemd-sysctl applies /etc/sysctl.conf LAST, so a stale line there
+    # beats every drop-in. The 2026-08-26 port-53 outage had one.
+    grep -q '/etc/sysctl.conf' "$REPO_ROOT/scripts/deploy-remote.sh"
+}
+
 @test "the pihole env example has placeholder values only" {
     [ -f "$ENV_EXAMPLE" ]
     grep -q '^WEBPASSWORD=changeme$' "$ENV_EXAMPLE"