dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: record fleet double swap (large -> kingjones STRIX on pinned fork, small -> Huihui FP8 on FreeToken)

Browse Source 
- kingjones-strix-rocmfp4 reactivated as the ACTIVE large runbook:
  runtime build recipe (WEBUI=OFF gotcha), deployed serving profile,
  fa-on-works-unpatched evidence, MTP heads documented unloadable,
  swap procedure + 2026-08-30 validation record
- agentionai-fast-imatrix marked RETIRED (rollback record kept)
- freetoken-frank-default-small: model/lineage/template sections now
  Huihui abliterated FP8; new operational quirk: chat API 422 without
  explicit model field
- LLM index + AGENTS fleet table/runtime sections current; both
  services' history chains fixed (small was still Tiel in AGENTS.md)
- worklog entry added; models.json contextWindow for large corrected
  to the real per-slot 131072 (llama clamps n_ctx_train/parallel)
This commit is contained in:
Frank committed 2026-08-30 02:35:38 -07:00
1 parent aa9c1d065a
commit 0937cfb047
6 files changed
+268 -152
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+80 -53
View File
@@ -22,12 +22,12 @@ prefix, reorganized in follow-up commits). The retired target checkouts
archived. All services now run from `/home/dank/serverAppConfigs` on each
host.
### LLM fleet topology (2026-08-22)
### LLM fleet topology (2026-08-30)
| Target | Runtime | Service | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
| Dankclaw | charlie12345/ROCmFPX llama.cpp (build-qwen38) | frank-default-large | agentionai Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87 GiB Q4_0_ROCMFP4_FAST, PLE joined table in v2 GGUF), ngram-mod spec (MTP head on disk, incompatible with this build), 32K context / 1 slot, reasoning on | 7770 |
| Dankserver | FreeToken v0.1.2 | frank-default-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 |
| Dankclaw | kingjones30/ROCmFPX llama.cpp (card-pinned fork) | frank-default-large | kingjones777 Qwen3.8-Flash-Next-ROCmFP4-STRIX (113.47 GiB single GGUF, Q4_0_ROCMFP4_STRIX, PLE streams via mmap), 401408 ctx / 2 slots, fa on, no spec decode, reasoning on | 7770 |
History: the 2026-08-17 redo retired the pipeline-parallel split and the
FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
@@ -40,7 +40,12 @@ Qwen3.8-27B-Uncensored-Q6_0-ROCmFPX_AGENT model (standalone MTP draft,
Dankserver switched from ik_llama.cpp Ornith CRACK Q8_0 to stock llama.cpp
Tiel-Coder-35B (qwen35moe hybrid MoE), and Dankclaw switched from the Q6_0
AGENT model to the Flash-Next 56B MoE (qwen4exp, PR #27742 merged into
charlie's ROCmFPX fork, --fit off, 128k context). Runbooks:
charlie's ROCmFPX fork, --fit off, 128k context). On 2026-09-02
Dankserver moved to the FreeToken runtime serving the official
Qwen3.6-35B-A3B-FP8. On 2026-08-30 both services swapped again:
Dankclaw to the kingjones STRIX model on the card's pinned
kingjones30/ROCmFPX runtime, Dankserver to the Huihui abliterated
FP8 checkpoint on FreeToken. Runbooks:
[serverAppConfigs-docs/LLM/index.md](serverAppConfigs-docs/LLM/index.md).
Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`.
@@ -274,63 +279,85 @@ When you rename a service or move a model to a new host, update
## Runtime & Troubleshooting
### Qwen3.8-Flash-Next ROCmFP4-FAST imatrix llama.cpp (Dankclaw, ACTIVE)
### kingjones Qwen3.8-Flash-Next STRIX on kingjones30/ROCmFPX (Dankclaw, ACTIVE)
The large service serves
`agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF` (public,
87.06 GiB, Q4_0_ROCMFP4_FAST imatrix-calibrated) from
`/home/dank/models/agentionai-Qwen3.8-Flash-Next-ROCmFP4-FAST/` with a
charlie12345/ROCmFPX `llama-server` built at
`/home/dank/GitPackages/ROCmFPX/build-qwen38` (PR #27742 qwen4exp architecture
merged). The model is a 56B MoE (512 experts, 10 used), 48 layers (12 attn +
36 SSM, full_attention_interval=4), QSA indexer (4 heads, top_k=2048),
hyper-connections (count=4, low_rank=320), PLE (layer 1). The 51.2 B-parameter
n-gram table is a single joined tensor in the `v2/` GGUF we serve (the root
`-ple16` per-head split requires the LaurentZuijdwijk Vulkan fork and aborts
this loader's `GGML_ASSERT(qwen4exp is missing the PLE n-gram table)`) — fully
VRAM-resident, no sidecar, no `--ngram-on-disk`. The model's own MTP draft head
(`agentionai/Qwen3.8-Flash-Next-MTP-ROCmFP4-FAST-GGUF`, 2.28 GiB, ROCmFP4-FAST
quant matched to target) is on disk but does **not** load on this binary
(loader requires a full per-layer `blk.<il>.hc_attn_norm.weight` set; the head
carries only `blk.48`) — speculative decoding runs ngram-mod only. Vision
tower (f16) ships as mmproj/. It runs 32768 context / 1
slot (author quick-start; the fully GPU-resident 87 GiB model + f16 KV +
compute buffer OOMs the SVM pool beyond that — verified live: cudaMalloc
fails at 131072/262144), b 4096 / ub 4096, t 16 / tb 16, f16 KV cache,
`--fit off` (auto-fit segfaults on qwen4exp hyper-connection config),
`--flash-attn on` (restored 2026-08-29 via the QSA F32-mask patch,
`scripts/patches/qwen4exp-qsa-f32-mask.patch` — re-apply inside
`/home/dank/GitPackages/ROCmFPX` before any clean rebuild), and reasoning on
(Sharp template, budget -1, `--reasoning-format deepseek`; chat kwargs keep
`preserve_thinking` + `preserve_reasoning`). Env
`kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF` (public, 113.47
GiB single file, `Q4_0_ROCMFP4_STRIX` = file_type 105) from
`/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX/` with
the card's **pinned runtime**: the `kingjones30/ROCmFPX` fork
(charlie12345/ROCmFPX + qwen4exp-on-rocmfpx-d3ca537 patch — ROCmFP4
types and qwen4exp arch in one tree), built at
`/home/dank/GitPackages/kingjones-ROCmFPX/build` with
`-DGGML_HIP=ON -DGPU_TARGETS=gfx1151 -DGGML_NATIVE=ON
-DCMAKE_BUILD_TYPE=Release -DLLAMA_BUILD_WEBUI=OFF` (WebUI
provisioning needs npm/HF-bucket and aborts the link at 98% without
the OFF flag). Args: our validated profile — 401408 context / 2 slots,
b 4096 / ub 512 (ub 512 matches the card's re-measurement), t 16 / tb
16, f16 KV, `--fit off`, `--flash-attn on` (works UNPATCHED on this
model — the QSA F16 ggml_fill assert that needed
`scripts/patches/qwen4exp-qsa-f32-mask.patch` is specific to the
agentionai FAST layer set; if a future build aborts with
`GGML_ASSERT(a-type == GGML_TYPE_F32)`, port that fix), reasoning on
(Sharp template, deepseek format), no speculative decoding (card ships
no spec flags; Flash-Next MTP heads do not load on this loader family).
Two hard card constraints: **mmap must stay on** (`--no-mmap` forces
the 50.66 GiB Q8_0 PLE table into anonymous memory → silent cgroup
kill, only dmesg evidence; llama.cpp's "use_mmap=1" log line is
correct here — mmap pages ARE page cache) and **never pin the PLE
tensor with `-ot`** (23 → 13.4 t/s). The 51.2B-param PLE table never
enters GPU memory (~20 bytes/token-file per 500 decoded). On-host file
sha256 verified == HF LFS oid `c6770d74...49fa`. Env
`scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`.
**Serving status (2026-08-29):** deployed the agentionai ROCmFP4-FAST imatrix
model (v2 joined PLE layout), replacing the kingjones STRIX and jcbtc CIRU v1.1
deployments. The own-matched MTP draft head does NOT load on build-qwen38 —
verified against the v2 main itself (`missing tensor blk.0.hc_attn_norm.weight`),
third confirmation (EasiiX Q8_0 + agentionai FAST vs STRIX_LEAN earlier). The
head stays on disk for a future compatible binary. Same day, `--flash-attn on`
was restored: the QSA F32-mask patch (vendored
`scripts/patches/qwen4exp-qsa-f32-mask.patch`) fixes the F16-kq_mask
`ggml_fill` abort; the rebuilt fork binary carries it, verified live at 32K.
Runbook:
[serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md](serverAppConfigs-docs/LLM/kingjones-strix-rocmfp4.md).
**Serving status (2026-08-30):** swapped from the agentionai FAST
deployment. Validated live before the systemd handoff: clean load,
health 200, reasoning completions, and an operator session's ~111k
token prefill at 213–232 t/s (card rung: 196 at 131k). Note llama.cpp
clamps n_ctx_train 262144 → per-slot 131072 with parallel 2.
### agentionai ROCmFP4-FAST imatrix llama.cpp (Dankclaw, prior, 2026-08-29 to 2026-08-30)
Historical — rollback target. `agentionai/Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix-GGUF`
(87.06 GiB, Q4_0_ROCMFP4_FAST imatrix, v2/ joined-PLE layout only — the
root `-ple16` per-head split aborts this loader family) on the charlie
`build-qwen38` tree (`/home/dank/GitPackages/ROCmFPX`, PR #27742
merged) at 32768/1, ub 4096. That tree REQUIRES
`scripts/patches/qwen4exp-qsa-f32-mask.patch` for `--flash-attn on`
(F16-kq_mask ggml_fill abort otherwise). Flash-Next MTP heads do not
load on any current loader (require full per-layer
`blk.<il>.hc_attn_norm.weight`; heads carry only `blk.48` — verified
3x). Binary and payload retained. Runbook:
[serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md](serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md).
### Tiel-Coder-35B-A3B llama.cpp (Dankserver, ACTIVE)
### Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on FreeToken (Dankserver, ACTIVE)
The small service serves `peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF` (public,
30 GB, UD-Q6_K_XL quant) from `/home/dank/models/tiel-coder-35b/` with
upstream llama.cpp (latest master, CUDA build with
`-DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86`). The model is a
qwen35moe hybrid MoE: 40 layers (10 attention + 30 SSM/Mamba,
full_attention_interval=4), 256 experts / 8 used, 2 KV heads × 256 dim. It
runs Q8_0 KV cache, 200700 total context (66900/slot × 3 parallel), `--cpu-moe`
for expert offload, `-ngl 999` (non-expert on GPU), and mmap. Env
`scripts/frank-default-small.env`, serve `scripts/serve-frank-small.sh`.
The small service serves `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8`
(public, ~35 GB, 26 safetensors shards) from
`/home/dank/models/huihui-qwen36-35b-a3b-abliterated-fp8/` with
FreeToken v0.1.2 (`/home/dank/freetoken/.venv`, `ft serve`). Native
modelopt block-fp8 (e4m3, dynamic, 128×128 weight_block_size,
un-fused per-expert layout) — same format and loader path as the
official Qwen3.6-35B-A3B-FP8 it replaced on 2026-08-30, so the entire
memory/pool plan carries: experts host-RAM with 20% LRU GPU residency,
paged KV (only the 10 full-attention layers page; 30 GDN layers use
constant-size mamba slots), ~3.4 GiB VRAM headroom at boot, and the
validated `--memory-num-gpu-blocks-experts 2048
--memory-num-gpu-tokens-kv-cache 453005 --max-num-seqs 4` pool triple
(never grow the expert pool). 262144 context, reasoning on (the
checkpoint ships the froggeric fixed chat template — no override).
FreeToken's API REQUIRES an explicit `"model"` field in chat requests
(422 without it). Env `scripts/frank-default-small.env`, serve
`scripts/serve-frank-small.sh`. Runbook:
[serverAppConfigs-docs/LLM/freetoken-frank-default-small.md](serverAppConfigs-docs/LLM/freetoken-frank-default-small.md).
**Serving status (2026-09-01):** deployed the Tiel-Coder-35B model. The
previous ik_llama.cpp Ornith CRACK Q8_0 deployment is retired.
**Serving status (2026-08-30):** swapped the model within the FreeToken
runtime (official FP8 → Huihui abliterated FP8). ~40 tok/s verified on
a 282-token completion post-deploy. Prior: official Qwen FP8
(2026-09-02, runtime swap), Tiel-Coder-35B on llama.cpp (2026-09-01),
ik_llama.cpp Ornith CRACK (retired 2026-09-02).
### charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, prior v3, 2026-08-18 to 2026-08-25)