feat: replace llama.cpp Tiel-Coder with FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8
Replace the llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment with FreeToken v0.1.2 — an edge-native MoE serving engine that keeps expert weights in host RAM with an LRU GPU cache and paged KV with radix prefix sharing. Model: Qwen/Qwen3.6-35B-A3B-FP8 (modelopt block-fp8, 128×128 weight_block_size). Architecture: qwen3_5_moe — 40 layers (10 attention + 30 GDN), 256 experts/8 used. Production config: - 2048 MoE cache slots (20% of 10240 expert instances, ~6.0 GiB VRAM) - 453005 KV tokens (~8.6 GiB, ~20 KiB/token for 10 attn layers) - 4 max-running-requests (mamba slot pool: 24 slots) - bs4 decode: ~87 tok/s aggregate, ~30 tok/s bs1 - Prefill: ~1,130 tok/s at 4k prompt Quirk: FreeToken v0.1.2 supports modelopt block-fp8, NVFP4, and per-tensor FP8, but NOT compressed-tensors per-channel FP8 (Huihui/Ornith both failed). Live KV resize via ft ctl cache grows KV by shrinking MoE; reverse needs restart. Old llama.cpp scripts archived to .AGENTTRASH/. Full benchmark sweep (bs1/bs3/bs4/bs6, 8k–700k KV) documented in docs/LLM/freetoken-benchmark-sweep.md.
This commit is contained in:
1 parent
e219fdf939
commit
6efb0fb231
14 files changed
+494
-470
No files matched your search
+14
-9
@@ -1,8 +1,8 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on two local machines
|
||||
(2026-08-26): ik_llama.cpp on Dankserver (frank-default-small,
|
||||
Ornith-1.5-35B-A3B CRACK) and ROCmFPX llama.cpp on Dankclaw
|
||||
(2026-09-02): FreeToken v0.1.2 on Dankserver (frank-default-small,
|
||||
Qwen3.6-35B-A3B-FP8) and ROCmFPX llama.cpp on Dankclaw
|
||||
(frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both
|
||||
services run the Qwen-Sharp chat template.
|
||||
|
||||
@@ -16,11 +16,11 @@ services run the Qwen-Sharp chat template.
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04]
|
||||
A[RTX 3090 24GB + 62GB RAM]
|
||||
B2[ik_llama.cpp build b166e269
|
||||
A[RTX 3090 24GB + 60GB RAM]
|
||||
B2[FreeToken v0.1.2
|
||||
frank-default-small, port 7770]
|
||||
C[Ornith-1.5-35B-A3B-CRACK Q6_K
|
||||
no speculation, q8_0 + Hadamard cache, 491520 ctx]
|
||||
C[Qwen3.6-35B-A3B-FP8
|
||||
block-fp8, 20% MoE residency, 453k KV / 4 streams]
|
||||
A --> B2 --> C
|
||||
end
|
||||
|
||||
@@ -46,6 +46,11 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[freetoken-frank-default-small]] — the active frank-default-small
|
||||
runbook: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 on Dankserver
|
||||
(2026-09-02)
|
||||
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [kingjones Qwen3.8 Uncensored ROCmFP4 STRIX MTP](https://huggingface.co/kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF) — active frank-default-large model on Dankclaw; Q6 AGENT main model with standalone MTP
|
||||
@@ -54,9 +59,9 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
|
||||
on Dankclaw (retired)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-default-small, active 2026-08-24)
|
||||
- [[ikllama-ornith-dankserver]] — the ik_llama.cpp runtime for
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-default-small, retired 2026-09-02)
|
||||
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
|
||||
frank-default-small (2026-08-26): q8_0 + Hadamard KV cache,
|
||||
split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
|
||||
- [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP
|
||||
|
||||
Reference in new issue
Block a user