Frank 6efb0fb231 feat: replace llama.cpp Tiel-Coder with FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 ...

Replace the llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment with
FreeToken v0.1.2 — an edge-native MoE serving engine that keeps expert
weights in host RAM with an LRU GPU cache and paged KV with radix prefix
sharing.

Model: Qwen/Qwen3.6-35B-A3B-FP8 (modelopt block-fp8, 128×128 weight_block_size).
Architecture: qwen3_5_moe — 40 layers (10 attention + 30 GDN), 256 experts/8 used.

Production config:
- 2048 MoE cache slots (20% of 10240 expert instances, ~6.0 GiB VRAM)
- 453005 KV tokens (~8.6 GiB, ~20 KiB/token for 10 attn layers)
- 4 max-running-requests (mamba slot pool: 24 slots)
- bs4 decode: ~87 tok/s aggregate, ~30 tok/s bs1
- Prefill: ~1,130 tok/s at 4k prompt

Quirk: FreeToken v0.1.2 supports modelopt block-fp8, NVFP4, and per-tensor
FP8, but NOT compressed-tensors per-channel FP8 (Huihui/Ornith both failed).
Live KV resize via ft ctl cache grows KV by shrinking MoE; reverse needs restart.

Old llama.cpp scripts archived to .AGENTTRASH/.
Full benchmark sweep (bs1/bs3/bs4/bs6, 8k–700k KV) documented in
docs/LLM/freetoken-benchmark-sweep.md.

2026-08-27 02:01:16 -07:00

..

bench-ubatch.sh

Set equal batch and ubatch at 8k context

2026-08-25 21:04:00 -07:00

chat-template.jinja

Swap LLM fleet to Ornith CRACK (small) and kingjones ROCmFP4 STRIX MTP (large); apply Qwen-Sharp chat template

2026-08-24 21:35:11 -07:00

deploy-open-webui-quadlet.sh

Deploy services independently by changed paths

2026-08-25 20:13:33 -07:00

deploy-pihole-quadlet.sh

Deploy services independently by changed paths

2026-08-25 20:13:33 -07:00

deploy-plan.py

hermes-webui: add Hermex server as loopback compose service on Dankclaw

2026-08-26 08:17:27 -07:00

deploy-remote.sh

caddy: drop the systemctl enable; the quadlet generator already enables it

2026-08-26 20:19:03 -07:00

deploy-searxng-quadlet.sh

Deploy services independently by changed paths

2026-08-25 20:13:33 -07:00

deploy.sh

deploy.sh: accept hermes-webui in the top-level service inventory

2026-08-26 08:22:23 -07:00

download-q8agent.sh

Upgrade frank-default-large to kingjones Q8_0 AGENT at full 262144 context

2026-08-25 07:12:02 -07:00

frank-default-large.env

feat: frank-default-large 2x 256k context slots (CTX_SIZE=524288, PARALLEL=2)

2026-08-26 21:26:39 -07:00

frank-default-large.service

feat: swap both LLM services + remove config-assertion tests

2026-08-26 20:37:46 -07:00

frank-default-npu.env

Reorganize: llm/scripts -> scripts (LLM serve scripts, units, env tunables)

2026-08-22 15:07:31 -07:00

frank-default-npu.service

scripts: unified deploy.sh; units + bench point at the merged checkout

2026-08-22 15:50:39 -07:00

frank-default-small.env

feat: replace llama.cpp Tiel-Coder with FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8

2026-08-27 02:01:16 -07:00

frank-default-small.service

feat: replace llama.cpp Tiel-Coder with FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8

2026-08-27 02:01:16 -07:00

homepage-status.py

Homepage (TDD slice 7): fix the /admin/ path + status.json perms

2026-08-23 16:49:33 -07:00

homepage-status.service

Homepage (TDD slice 6): the git-driven deploy

2026-08-23 12:59:56 -07:00

homepage-status.timer

Homepage (TDD slice 6): the git-driven deploy

2026-08-23 12:59:56 -07:00

mount-entropy.sh

Add CLI helper for entropy volume

2026-08-25 15:38:57 -07:00

serve-frank-default-npu.sh

Reorganize: llm/scripts -> scripts (LLM serve scripts, units, env tunables)

2026-08-22 15:07:31 -07:00

serve-frank-large.sh

feat: frank-default-large 2x 256k context slots (CTX_SIZE=524288, PARALLEL=2)

2026-08-26 21:26:39 -07:00

serve-frank-small.sh

feat: replace llama.cpp Tiel-Coder with FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8

2026-08-27 02:01:16 -07:00

test-changed.sh

feat: swap both LLM services + remove config-assertion tests

2026-08-26 20:37:46 -07:00

tmux-resurrect.sh

feat: add tmux session snapshot and restore utility

2026-08-25 17:04:15 -07:00