diff --git a/AGENTS.md b/AGENTS.md index dc1728b..8437997 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,14 +27,17 @@ host. | Target | Runtime | Service | Model | Port | |--------|---------|---------|-------|------| | Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 | -| Dankclaw | charlie12345/ROCmFPX llama.cpp, ROCmI4 W4A4 runtime | frank-default-large | caf-eemin Qwen3.8-27B-ROCmI4-MTP Q4_0_ROCMI4 (embedded NextN/MTP head, gfx1151 IU4 batched verification), 262K context / 1 slot | 7770 | +| Dankclaw | charlie12345/ROCmFPX llama.cpp | frank-default-large | kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT Q8_0_ROCMFPX (standalone MTP draft head, Turbo4 V cache, 262K context / 1 slot), reasoning on | 7770 | History: the 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw. On 2026-08-21 a DFlash2 Vulkan deployment briefly replaced CIRU, then was reverted the same day for speed. On 2026-08-22 the service briefly swapped to the Kairic Edge IU4 runtime, then was reverted to v3 the same day -(operator decision). Runbooks: [docs/LLM/index.md](docs/LLM/index.md). +(operator decision). On 2026-08-25 the service switched to the kingjones +Qwen3.8-27B-ROCmFPX-Q8_0-AGENT model (standalone MTP draft, 262144 context, +reasoning on); the previous Kairic Edge model was retired the same day. Runbooks: +[docs/LLM/index.md](docs/LLM/index.md). Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`. ## Repository Layout @@ -179,7 +182,21 @@ When you rename a service or move a model to a new host, update ## Runtime & Troubleshooting -### charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, ACTIVE) +### kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT llama.cpp (Dankclaw, ACTIVE) + +The large service serves `kingjones/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT` +(public) from `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` +with a charlie12345/ROCmFPX `llama-server` (build `build-kairic`). It runs a +standalone MTP draft head on ROCm0, f16 K + TurboQuant V KV cache, a full +262144 context / 1 slot, and reasoning on (Sharp template, budget -1, +`--reasoning-format none`; chat kwargs keep `preserve_thinking` + +`preserve_reasoning`). The payload (~28 GB main + 1.7 GB draft + 0.9 GB +mmproj, ~30 GB total) plus the 262k KV cache fit in the lifted 114 GiB TTM/GTT +budget (raised 2026-08-25). Env `scripts/frank-default-large.env`, serve +`scripts/serve-frank-large.sh`, runbook +[docs/LLM/kingjones-q8agent-rocmfpx.md](docs/LLM/kingjones-q8agent-rocmfpx.md). + +### charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, prior v3, 2026-08-18 to 2026-08-25) The v4 runtime serves `cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (`eemin` namespace, public) from `/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` @@ -300,9 +317,9 @@ graph TD - Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x `. - Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM - (`--no-mmap`, MoE experts on CPU) in a few minutes cold. Dankclaw v3 - loads the ~36.7 GiB five-file payload in ~18 s with a warm page cache, - a few minutes cold. Health returns 503 while loading. Do not kill the + (`--no-mmap`, MoE experts on CPU) in a few minutes cold. Dankclaw loads the + ~30 GiB Q8_0 AGENT payload (main + MTP draft + mmproj) a few minutes cold + with a fresh KV cache; ~18 s with a warm page cache. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `LLM_WAIT`). - A client abort and a server release look different in the journal. A diff --git a/docs/LLM/Dankclaw Improvements.md b/docs/LLM/Dankclaw Improvements.md index 7346e58..95521c9 100644 --- a/docs/LLM/Dankclaw Improvements.md +++ b/docs/LLM/Dankclaw Improvements.md @@ -21,20 +21,24 @@ marked "verify first". Re-verified 2026-08-15/16; corrections marked below. - **KV cache**: production runs `q8_0` K **and** `q8_0` V (both quantized). - **Spec**: DFlash/DSpark are retired. Production is `--spec-default` (ngram-mod) with draft-mtp first. See [[qwen3.8-spec-matrix]]. -- **TTM**: still 15.24 GiB as of 2026-08-16 — the modprobe fix is still - pending a reboot. +- **TTM**: raised to 114 GiB on 2026-08-25, set live (no reboot) and + persisted in `/etc/modprobe.d/ttm.conf`. The effective VRAM ceiling that + once held the service at ~15 GiB is gone. ## Verified Memory Picture (measured 2026-08-11) | Figure | Value | Source | | ------ | ----- | ------ | | OS physical RAM | 30.5 GiB | `/proc/meminfo` | -| TTM pages_limit | 15.24 GiB (3,995,281 x 4KiB) | `/sys/module/ttm/parameters/pages_limit` | +| TTM pages_limit | 114 GiB (29,883,536 x 4KiB) — raised 2026-08-25 | `/sys/module/ttm/parameters/pages_limit` | | ROCm VRAM total | 96.0 GiB | `rocm-smi --showmeminfo vram` | | ROCm VRAM used | 76.2 GiB | `rocm-smi` while serving 3x200700 | -The OS accounts only ~30.5 GiB physical RAM, and the TTM GPU-addressable pool -is capped at ~15.24 GiB. ROCm/SMI still reports a 96 GiB VRAM carve-out (VGM). +The OS accounts only ~30.5 GiB physical RAM. The TTM GPU-addressable pool +was capped at ~15.24 GiB (the real pre-2026-08-25 ceiling); that cap is now +lifted to 114 GiB. ROCm/SMI still reports a 96 GiB VRAM carve-out (VGM) — a +different, driver-reported figure that has always been larger than the kernel +cap. This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory limits documented in [[frank-default-large-benchmark]]. @@ -77,7 +81,7 @@ All require root and a reboot for the modprobe change. | Setting | Action | Effect | Priority | | ------- | ------ | ------ | -------- | -| TTM GPU pool | Set `amd-ttm --set 96` (write `/etc/modprobe.d/ttm.conf`), reboot | Lifts the 15 GiB TTM cap toward full RAM; more headroom for GPU buffers | High | +| TTM GPU pool | `/etc/modprobe.d/ttm.conf` → `options ttm pages_limit=29883536 page_pool_size=29883536` (applied 2026-08-25, set live then persisted) | Lifts the 15 GiB cap to 114 GiB; the full 262k KV + Q8 AGENT payload fit in one slot | Done | | Power profile | **N/A on this machine** (2026-08-15): no `power_profile` sysfs; GPU DPM already `performance` | none | ~~Medium~~ | | NVMe scheduler | Already `none` (`[none]` confirmed) | No change | Done | | BIOS VGM | Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS | Max dedicated VRAM for the GPU carve-out; verify in the live BIOS | Verify | diff --git a/docs/LLM/deployment.md b/docs/LLM/deployment.md index 2711ef8..895364b 100644 --- a/docs/LLM/deployment.md +++ b/docs/LLM/deployment.md @@ -137,24 +137,16 @@ from `peculiar-ragdoll/Qwen-Sharp-Chat-Templates`. See [[dealign-ornith-crack]]. ### Dankclaw — frank-default-large -The active service on Dankclaw is the charlie12345/ROCmFPX llama.cpp runtime -(with the opt-in `ROCmI4 W4A4` path) serving `cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` -(`eemin` namespace, `Q4_0_ROCMI4`, embedded NextN/MTP head) under the alias -`frank-default-large` on port 7770. It runs -`/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` (built -against system ROCm 7.2.4 / amdclang 22) with a **strict Qwen MTP** draft -(`--spec-type draft-mtp --spec-mtp-strict-qwen`) running on ROCm0 with all -layers (`--spec-draft-ngl all`) — no separate draft file, because the MTP head -is embedded in the main GGUF. K/V are f16, 65536 context (192K below the -native 262144 maximum; 1 slot, concurrent requests queue, the server never -preempts a running generation), -`-t 16 -tb 32`, batch 512 / ubatch 256 (ubatch is swappable; sweep it on 8k -context with `scripts/bench-ubatch-rocmi4.sh`), 16 context checkpoints, 8 GiB -prompt cache, and reasoning in deepseek format. The unit file is -`scripts/frank-default-large.service`. It launches through +The active service on Dankclaw uses the ROCmFPX llama.cpp runtime to serve +[kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF](https://huggingface.co/kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF) +under the alias `frank-default-large` on port 7770. The main model is +`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`; the standalone MTP draft is +`mtp-Qwen3.8-27B-Q4_0.gguf`, and the BF16 vision projector is loaded alongside +it. The K-side KV cache remains `f16`; the V-side cache uses `turbo4`. The +service keeps the existing 65536 context / 1-slot profile. The unit file is +`scripts/frank-default-large.service`; it launches through `scripts/serve-frank-large.sh`, which reads tunables from `scripts/frank-default-large.env`, applying the Qwen-Sharp chat template. -See [[eemin-rocmi4-mtp]]. > The kingjones ROCmFP4 STRIX MTP model served the slot from 2026-08-24; it > was replaced by the caf-eemin ROCmI4 embedded-MTP model on 2026-08-25. The diff --git a/docs/LLM/index.md b/docs/LLM/index.md index 94c488e..a358555 100644 --- a/docs/LLM/index.md +++ b/docs/LLM/index.md @@ -1,10 +1,10 @@ # LLM Fleet — Knowledge Base Documentation for the LLM inference fleet on two local machines -(2026-08-24): upstream llama.cpp on Dankserver (frank-default-small, -Ornith-1.5-35B-A3B CRACK) and CIRU ROCmFPX llama.cpp on Dankclaw -(frank-default-large, kingjones Qwen3.8-27B ROCmFP4). Both services -run the Qwen-Sharp chat template. +(2026-09-01): upstream llama.cpp on Dankserver (frank-default-small, +Ornith-1.5-35B-A3B CRACK) and ROCmFPX llama.cpp on Dankclaw +(frank-default-large, kingjones Qwen3.8-27B ROCmFPX Q8 AGENT). Both +services run the Qwen-Sharp chat template. > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -28,8 +28,8 @@ ngram-mod, selected experts offloaded via override, 491520 ctx] D[gfx1151 iGPU 64 GiB unified memory] E[CIRU ROCmFPX llama.cpp PromptForge v3 frank-default-large, port 7770] - F[Qwen3.8-27B-Uncensored-ROCmFP4 Q6_0 AGENT -MTP draft + Sharp template, 262K ctx / 1 slot] + F[Qwen3.8-27B ROCmFPX Q8_0 AGENT +MTP draft + Turbo4 V cache, 262K ctx / 1 slot] D --> E --> F end @@ -48,8 +48,10 @@ MTP draft + Sharp template, 262K ctx / 1 slot] - [[deployment]] — deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[eemin-rocmi4-mtp]] — the caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model - on Dankclaw (frank-default-large, active 2026-08-25) +- [kingjones Qwen3.8-27B ROCmFPX Q8 AGENT](https://huggingface.co/kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF) — active frank-default-large model on Dankclaw; standalone MTP and Turbo4 V cache +- [[kingjones-q8agent-rocmfpx]] — the Q8_0-AGENT runbook: serving profile, download, 262144 context, reasoning + preserve_thinking, 114 GiB TTM/GTT +- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model + on Dankclaw (retired) - [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (frank-default-small, active 2026-08-24) - [[kingjones-strix-mtp]] — the retired kingjones Qwen3.8-27B ROCmFP4 STRIX MTP diff --git a/docs/LLM/kingjones-q8agent-rocmfpx.md b/docs/LLM/kingjones-q8agent-rocmfpx.md new file mode 100644 index 0000000..1c71527 --- /dev/null +++ b/docs/LLM/kingjones-q8agent-rocmfpx.md @@ -0,0 +1,165 @@ +# frank-default-large: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT + +**Status:** ACTIVE on Dankclaw (since 2026-08-25). + +**Alias:** `frank-default-large`, port **7770**. + +The large service serves the kingjones `Qwen3.8-27B-ROCmFPX-Q8_0-AGENT` GGUF +from `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` with a +charlie12345/ROCmFPX `llama-server` (build `build-kairic`). It runs a +**standalone MTP draft head** (loaded separately, on ROCm0), **TurboQuant** V +KV cache, the **Sharp** chat template with **reasoning on**, and a **full +262144 context / 1 slot** profile. The whole payload plus the 262k KV cache +fit in the lifted **114 GiB TTM/GTT** budget (see the 114 GiB change in +[[Dankclaw Improvements]] and [[qwen3.8-strix-optimizations]]). + +| Figure | Value | +| ------ | ----- | +| Model | kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT (`Q8_0_ROCMFPX`) | +| HF repo | `kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF` (public) | +| Runtime | charlie12345/ROCmFPX `llama-server`, `build-kairic` | +| Draft head | standalone `mtp-Qwen3.8-27B-Q4_0.gguf` on ROCm0 | +| KV cache | f16 K, TurboQuant (`turbo4`) V, full 262144 context, 1 slot | +| Reasoning | on, `deepseek`, budget `-1`, `--reasoning-format none` | +| Chat template | Sharp (repo-tracked), `enable_thinking` + `preserve_thinking` on | +| Host path | `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` | +| Tunables | `scripts/frank-default-large.env` | +| Serve script | `scripts/serve-frank-large.sh` | + +## Model files + +Three files, from the HF repo above: + +| File | Size | Purpose | +| ---- | ---- | ------- | +| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | 28.21 GB | main weights | +| `mtp-Qwen3.8-27B-Q4_0.gguf` | 1.68 GB | standalone MTP draft head (loaded by the script) | +| `mmproj-Qwen3.8-27B-BF16.gguf` | 0.93 GB | vision projector (`--mmproj`) | + +The repo also has a `mtp-Qwen3.8-27B-Q8_0.gguf` (3.16 GB) draft; the env uses +the Q4_0 draft. Download them from the resolve URLs below (they are large — +use the token in `~/.cache/huggingface/token`). + +## Download + +```bash +# on Dankclaw +TOKEN=$(cat ~/.cache/huggingface/token) +REPO=kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF +DIR=/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT +mkdir -p "$DIR"; cd "$DIR" +dl(){ curl -fsSL -C - -o "$1" --retry 5 --retry-delay 3 --retry-max-time 3600 \ + -H "Authorization: Bearer $TOKEN" \ + "https://huggingface.co/$REPO/resolve/main/$1"; echo "done: $1"; } +dl Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf +dl mmproj-Qwen3.8-27B-BF16.gguf +dl mtp-Qwen3.8-27B-Q4_0.gguf +``` + +The serve script checks each artifact is present and readable before launch +(`scripts/frank-default-large.env` -> the four `MODEL_*` files plus the chat +template). A missing file fails the artifact check with `missing or +unreadable required artifact`. + +## Serving profile + +`scripts/serve-frank-large.sh` builds the `llama-server` call from the env: + +``` +/dev/llama-server -m
--alias frank-default-large + --host 0.0.0.0 --port 7770 --jinja + -dev ROCm0 -ngl 999 -np 1 -c 262144 + -b 512 -ub 512 -fa on + -ctk f16 -ctv turbo4 -t 16 -tb 32 + -ctxcp 32 --cache-ram 8192 --cache-prompt --cache-idle-slots --metrics + --spec-type draft-mtp --model-draft + --mmproj + --spec-draft-device ROCm0 --spec-draft-ngl 999 + --spec-draft-type-k f16 --spec-draft-type-v turbo4 + --spec-draft-n-max 4 --spec-draft-n-min 0 + --spec-draft-p-min 0.0 --spec-draft-p-split 0.10 + --spec-draft-backend-sampling + --temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 + --presence-penalty 1.5 + --reasoning on --reasoning-format none --reasoning-budget -1 +``` + +Key choices: + +- **Standalone MTP draft** (`--model-draft`, `--spec-type draft-mtp`). The + AGENT GGUF has no embedded draft head; the draft head is a separate file on + ROCm0. This matches the Qwen3.8-27B AGENT shape (`--spec-draft-type-k + f16 --spec-draft-type-v turbo4`). +- **f16 K + TurboQuant V** cache. f16 K is the safe long-ctx default; TurboQuant + halves the V cache at 262k while keeping quality (see the ubatch/threads-batch + notes in [[qwen3.8-strix-optimizations]]). +- **Full 262144 context, 1 slot.** The whole context KV cache plus the ~30 GB + payload (main + draft + mmproj) fit in the 114 GiB budget. Concurrent + requests queue; the server never preempts a running generation. +- **Reasoning on.** The Sharp template drives thinking (medium effort); + `--reasoning-format none` lets the template project the reasoning content into + `reasoning_content`. `--reasoning-budget -1` = no budget cap. + +## Reasoning + chat template + +The Sharp template (`scripts/chat-template.jinja`, repo-tracked) is a +Qwen-AGENT shape. Env `CHAT_TEMPLATE_KWARGS` keeps the reasoning trail in the +conversation: + +```json +{"enable_thinking": true, "preserve_thinking": true, "preserve_reasoning": true} +``` + +`preserve_thinking: true` keeps the model's prior thinking in the next prompt +(the user's "preserve thinking" request). `preserve_reasoning: true` keeps the +model's prior reasoning trace across requests too (OpenWebUI sends +`preserve_reasoning`/`preserve_thinking: true` when the conversation needs the +full trail). For multi-turn reasoning, the template stays in the reasoning +shape and `enable_thinking: true` per request. + +## Memory: the 114 GiB TTM/GTT requirement + +The 262k f16 KV cache is large (~a few GB) and the payload is ~30 GB. Both live +in the Strix Halo unified-memory GTT. The TTM `pages_limit` was raised to +114 GiB on 2026-08-25 (written live to +`/sys/module/ttm/parameters/pages_limit`, then persisted in +`/etc/modprobe.d/ttm.conf`). Without the raise the effective ceiling is ~15 GiB +and a 28 GB + 262k-KV load stalls. The GTT page-limit change is per the +`qwen3.8-strix-optimizations` table item 5. + +## Health + troubleshooting + +- Health: `curl http://192.168.0.143:7770/health` → `{"status":"ok"}`. +- `/api/models` reports `n_ctx:262144, n_ctx_train:262144, n_params:27.3B`. +- Model load is slow (a few minutes cold for the ~30 GB payload with a fresh + KV cache). Health returns 503 while loading. Do not kill the process. + `deploy.sh` waits up to 20 minutes (see `LLM_WAIT`). +- Kernel spam about `amdgpu: SVM mapping failed` is the known documented + symptom, not root cause (see the crash-loop procedure in the project + AGENTS.md). Investigate other causes before assuming memory pressure. +- A client abort logs `W srv stop: cancel task, id_task = N`; a natural stop + logs only `slot release` with `truncated = 0`. + +## Env tunables + +`scripts/frank-default-large.env`. Serving: `CTX_SIZE=262144`, +`PARALLEL=1`, `BATCH_SIZE=512`, `UBATCH_SIZE=512`, `NUM_THREADS=16`, +`THREAD_BATCH=32`, `CACHE_TYPE_K=f16`, `CACHE_TYPE_V=turbo4`, +`CTX_CHECKPOINTS=32`, `CACHE_RAM=8192`. Reasoning: `REASONING=on`, +`REASONING_FORMAT=none`. Sampling: `TEMPERATURE=0.7`, `TOP_K=20`, +`TOP_P=0.8`. MTP: `MTP_N_MAX=4`, `MTP_N_MIN=0`, `MTP_P_MIN=0.0`, +`MTP_P_SPLIT=0.10`. Strix Halo: `HSA_OVERRIDE_GFX_VERSION=11.5.1`, +`GGML_HIP_ENABLE_UNIFIED_MEMORY=1`. + +**Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`** — the ROCm HIP +runtime reads those bare vars (see the 2026-08-13 TEMP incident). llama.cpp +reads `--host/--port/--temp` as CLI flags, never as env vars. + +## History + +- **2026-08-25 (today):** switched the large service to the kingjones + Q8_0-AGENT model with 262144 context and reasoning on. Downloaded the three + payload files from the HF repo, raised the TTM/GTT to 114 GiB for the full + context, and persisted the env/serve script in this repo. +- Preceding model: Kairic Edge Qwen3.8-27B IU4 (deployed 2026-08-25 ~05:42, + retired the same day for the AGENT model). See [[kairic-edge]]. diff --git a/docs/LLM/qwen3.8-strix-optimizations.md b/docs/LLM/qwen3.8-strix-optimizations.md index a87ed8d..ef2eb3f 100644 --- a/docs/LLM/qwen3.8-strix-optimizations.md +++ b/docs/LLM/qwen3.8-strix-optimizations.md @@ -19,7 +19,7 @@ Related pages: | 2 | MTP `n_max` explicit 4–5 + Q4_0 draft head | +5–15% on the spec share | restart, 1.56 GB file | test 08-16 | | 3 | Plain `Q8_0` A/B vs UD-Q8_K_XL (same 8 bits) | dequant ALUs may differ | restart, 29 GB file | test 08-16 | | 4 | Prefill path: ROCm backend + tuned ubatch/threads-batch | 2–3× prefill for agentic turns | per-workload backend choice | partial data | -| 5 | TTM `pages_limit` raise (15.2 GiB → 90+) | removes a hard VRAM ceiling, stable 96 GiB budget | reboot | pending | +| 5 | TTM `pages_limit` raised to 114 GiB | removes the hard VRAM ceiling; the full 262k f16 KV plus the Q8 AGENT payload, MTP draft, and mmproj fit in one slot | modprobe, set live 2026-08-25 | applied (29,883,536 pages) | | 6 | Vulkan ICD pinning + loader-layer disable | robustness; small | service env vars | test 08-16 | Everything else (quantization downgrade, NVFP4, larger drafts) is documented @@ -166,11 +166,13 @@ hybrid state, upstream). Levers: ### 5. OS level (from the Improvements doc, re-verified 2026-08-15) -- **TTM `pages_limit` is still 3,995,281 pages ≈ 15.24 GiB** — verified - today, unchanged. The 96 GiB GTT budget only materializes if the TTM cap - is raised (kernel parameter, needs reboot). Until then the effective - ceiling is lower and the 28 GiB load-stall risk remains. **Highest-value - single OS change; schedule with the next reboot.** +- **TTM `pages_limit` is now 114 GiB (29,883,536 pages)** — applied 2026-08-25. + Written live to `/sys/module/ttm/parameters/pages_limit` (no reboot required) + and persisted in `/etc/modprobe.d/ttm.conf`. The full 262k f16 KV cache plus + the ~28 GB Q8 AGENT payload, MTP draft, and mmproj now fit in one slot. The + load-stall risk from the old 15 GiB cap is resolved. Note: 96 GiB is the + driver-reported VRAM carve-out (VGM/SVM pool); the TTM pages_limit is the + kernel page map that was the real ceiling. - `iommu=pt` — verified present on the kernel cmdline. Done. - **Vulkan ICD pinning** — `radeon_icd.json` exists; no `VK_ICD_FILENAMES` or loader-layer env is set. Pinning the radeon ICD and disabling loader diff --git a/docs/index.md b/docs/index.md index 1c64ad0..8667eaf 100644 --- a/docs/index.md +++ b/docs/index.md @@ -99,6 +99,7 @@ The LLM inference fleet on the two hosts. Full index: - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change - [[eemin-rocmi4-mtp]] — the caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw (frank-default-large, active) - [[kingjones-strix-mtp]] — the RETIRED kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (superseded 2026-08-25) +- [[kingjones-q8agent-rocmfpx]] — the Q8_0-AGENT runbook for frank-default-large (active): serving, download, 262144 context, reasoning + preserve_thinking, 114 GiB TTM/GTT - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (active) diff --git a/scripts/download-q8agent.sh b/scripts/download-q8agent.sh new file mode 100755 index 0000000..4634da5 --- /dev/null +++ b/scripts/download-q8agent.sh @@ -0,0 +1,63 @@ +#!/usr/bin/env bash +# Download the kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT payload for +# frank-default-large from HuggingFace into /home/dank/models/.../Q8_0-AGENT. +# +# Uses the token in ~/.cache/huggingface/token so the ~28 GB main file does not +# hit the anonymous 10 GB/30 min rate limit. Run on Dankclaw. +# +# bash scripts/download-q8agent.sh +# bash scripts/download-q8agent.sh --verify-only +set -euo pipefail + +REPO="kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF" +DIR="/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT" +EXPECTED=( + "Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf:28214425952" + "mmproj-Qwen3.8-27B-BF16.gguf:931145888" + "mtp-Qwen3.8-27B-Q4_0.gguf:1680271648" +) + +cd -- "$(dirname -- "$0")/.." + +main() { + local mode="download" + [[ "${1:-}" == "--verify-only" ]] && mode="verify" + mkdir -p "$DIR" + cd "$DIR" + + local token + token="$(cat "$HOME/.cache/huggingface/token")" + [[ -n "$token" ]] || { echo "no HF token in $HOME/.cache/huggingface/token" >&2; exit 1; } + + dl() { + local f="$1" + curl -fsSL -C - -o "$f" --retry 5 --retry-delay 3 --retry-max-time 3600 \ + -H "Authorization: Bearer $token" \ + "https://huggingface.co/$REPO/resolve/main/$f" + echo " done: $f ($(du -h "$f" | cut -f1))" + } + + if [[ "$mode" == "download" ]]; then + echo "downloading $REPO into $DIR" + dl "Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf" + dl "mmproj-Qwen3.8-27B-BF16.gguf" + dl "mtp-Qwen3.8-27B-Q4_0.gguf" + fi + + echo "verifying sizes" + local ok=1 f want act + for entry in "${EXPECTED[@]}"; do + f="${entry%%:*}"; want="${entry##*:}" + act="$(stat -c%s "$f")" + if [[ "$act" -eq "$want" ]]; then + echo " OK $f ($act bytes)" + else + echo " BAD $f ($act bytes, want $want)" >&2 + ok=0 + fi + done + [[ "$ok" -eq 1 ]] || exit 1 + echo "all files present and sized correctly" +} + +main "$@" diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index 11f5f30..193b7a8 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -1,65 +1,56 @@ -# Tunables for frank-default-large (CIRU/Kairic Edge ROCmFPX v1.2, dankclaw, -# Strix Halo gfx1151). +# Tunables for frank-default-large (kingjones Qwen3.8-27B ROCmFPX Q8 AGENT, +# dankclaw, Strix Halo gfx1151). # -# Runtime: ciru-ai/ROCmFPX tag kairic-edge-qwen38-27b-v1.2, commit -# 205a3e5f40e5542e2f2eb68e3d3f81f918b1d895, built with the pinned gfx1151 -# Prompt Forge / IU4 settings against /opt/rocm. Serves the Kairic Edge -# authoritative GGUF plus three required .pfs sidecars. Alias +# Runtime: ROCmFPX llama.cpp with TurboQuant support. Serves the kingjones +# AGENT GGUF plus its standalone MTP draft and vision projector. Alias # frank-default-large, port 7770. # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The ROCm # HIP runtime reads those bare env vars (see the 2026-08-13 TEMP incident). # llama.cpp reads --host/--port/--temp as CLI flags, never as env vars. -# ciru-ai/ROCmFPX Kairic Edge v1.2 build. +# ROCmFPX build with TurboQuant KV-cache support. LLAMA_BIN=/home/dank/ciru-qwen38/ROCmFPX/build-kairic/bin/llama-server -# Model files: authoritative GGUF plus all required Kairic Edge sidecars. -MODEL_DIR=/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge -MODEL_FILE=Qwen3.8-27B-IU4-Kairic-Edge.gguf -KAIRIC_FFN_SIDECAR=$MODEL_DIR/Qwen3.8-27B-Kairic-IU4-FFN.pfs -KAIRIC_GDN_SIDECAR=$MODEL_DIR/Qwen3.8-27B-Kairic-IU4-GDN.pfs -KAIRIC_GDN_OUTPUT_SIDECAR=$MODEL_DIR/Qwen3.8-27B-Kairic-IU4-GDN-Output.pfs -KAIRIC_EDGE_COMPATIBILITY_MODE=1 -# No separate MTP draft head: the MTP/NextN head is embedded in the main GGUF, -# so no --model-draft is passed. Kept for scripts that reference the var. -MTP_DRAFT_FILE= +# Model files from https://huggingface.co/kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF. +MODEL_DIR=/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT +MODEL_FILE=Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf +MTP_DRAFT_FILE=$MODEL_DIR/mtp-Qwen3.8-27B-Q4_0.gguf +MMPROJ_FILE=mmproj-Qwen3.8-27B-BF16.gguf # Qwen-Sharp chat template (tracked in the repo, staged on the host). The Sharp # template drives thinking on by default (medium effort) and produces the # peg-native + deepseek chat/reasoning shape the Pi agent expects. CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja -CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}' +CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' # Alias + network. ALIAS=frank-default-large BIND_ADDR=0.0.0.0 LISTEN_PORT=7770 -# Serving: 65536 context (192K below the model's 262144 maximum), 1 slot. f16 -# KV, context checkpoints, prompt cache, strict-MTP embedded draft on ROCm0. -# This leaves sufficient unified-memory headroom for the model plus KV cache. +# Serving: 262144 context (the model's maximum), 1 slot. f16 KV, context +# checkpoints, prompt cache, standalone MTP draft on ROCm0. The full 262144 +# context KV cache plus the ~28-30 GB Q8 model + MTP draft + mmproj fit in the +# lifted 114 GiB TTM/GTT budget on the Strix Halo (see the GTT 114 GiB change). # Concurrent requests queue; the server never preempts a running generation. -CTX_SIZE=65536 +CTX_SIZE=262144 PARALLEL=1 -# Batch/ubatch follow the qualified ROCmI4 W4A4 profile (see the bench in -# scripts/bench-ubatch-rocmi4.sh for the ubatch sweep on 8k contexts). +# Batch/ubatch follow the existing qualified Dankclaw profile. BATCH_SIZE=512 UBATCH_SIZE=512 NUM_THREADS=16 THREAD_BATCH=32 CACHE_TYPE_K=f16 -CACHE_TYPE_V=f16 +CACHE_TYPE_V=turbo4 CTX_CHECKPOINTS=32 CACHE_RAM=8192 -# Embedded-MTP (qwen35 NextN) qualified profile. The head is in the main GGUF, -# so the draft runs with all layers on ROCm0 (no --model-draft). +# Standalone MTP draft head, loaded from MTP_DRAFT_FILE by the serve script. MTP_N_MAX=4 MTP_N_MIN=0 MTP_P_MIN=0.0 MTP_P_SPLIT=0.10 -KAIRIC_EDGE_COMPATIBILITY_MODE=1 # Sampling (Sharp default). TEMPERATURE=0.7 @@ -68,9 +59,10 @@ TOP_P=0.8 MIN_P=0.0 PRESENCE_PENALTY=1.5 -# Thinking on (Sharp template drives it; reasoning format projects the tar into -# reasoning_content). -REASONING=off +# Reasoning on (Sharp template drives it; preserve_thinking + preserve_reasoning +# keep the reasoning trail in the conversation). reasoning-format none lets the +# template project the reasoning content into reasoning_content. +REASONING=on REASONING_FORMAT=none # Strix Halo unified memory + native gfx1151 detection for this ROCmFPX build. diff --git a/scripts/serve-frank-large.sh b/scripts/serve-frank-large.sh index 04be6cc..4961e74 100755 --- a/scripts/serve-frank-large.sh +++ b/scripts/serve-frank-large.sh @@ -1,7 +1,6 @@ #!/usr/bin/env bash -# Serve Qwen3.8-27B IU4 Kairic Edge v1.2 as frank-default-large. -# The Kairic runtime requires its authoritative GGUF and all three Prompt Forge -# sidecars. Compatibility mode is enabled for Qwen's recommended sampling. +# Serve the kingjones Qwen3.8-27B ROCmFPX Q8 AGENT model as frank-default-large. +# The deployment uses a standalone MTP draft head and Turbo4 V-side KV cache. set -euo pipefail SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd)" @@ -14,11 +13,10 @@ if [[ -f "$ENV_FILE" ]]; then fi : "${LLAMA_BIN:=/home/dank/ciru-qwen38/ROCmFPX/build-kairic/bin/llama-server}" -: "${MODEL_DIR:=/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge}" -: "${MODEL_FILE:=Qwen3.8-27B-IU4-Kairic-Edge.gguf}" -: "${KAIRIC_FFN_SIDECAR:=$MODEL_DIR/Qwen3.8-27B-Kairic-IU4-FFN.pfs}" -: "${KAIRIC_GDN_SIDECAR:=$MODEL_DIR/Qwen3.8-27B-Kairic-IU4-GDN.pfs}" -: "${KAIRIC_GDN_OUTPUT_SIDECAR:=$MODEL_DIR/Qwen3.8-27B-Kairic-IU4-GDN-Output.pfs}" +: "${MODEL_DIR:=/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT}" +: "${MODEL_FILE:=Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf}" +: "${MTP_DRAFT_FILE:=$MODEL_DIR/mtp-Qwen3.8-27B-Q4_0.gguf}" +: "${MMPROJ_FILE:=mmproj-Qwen3.8-27B-BF16.gguf}" : "${ALIAS:=frank-default-large}" : "${BIND_ADDR:=0.0.0.0}" : "${LISTEN_PORT:=7770}" @@ -29,7 +27,7 @@ fi : "${NUM_THREADS:=16}" : "${THREAD_BATCH:=32}" : "${CACHE_TYPE_K:=f16}" -: "${CACHE_TYPE_V:=f16}" +: "${CACHE_TYPE_V:=turbo4}" : "${CTX_CHECKPOINTS:=32}" : "${CACHE_RAM:=8192}" : "${MTP_N_MAX:=4}" @@ -43,11 +41,10 @@ fi : "${PRESENCE_PENALTY:=1.5}" : "${REASONING:=off}" : "${REASONING_FORMAT:=none}" -: "${KAIRIC_EDGE_COMPATIBILITY_MODE:=1}" : "${HSA_OVERRIDE_GFX_VERSION:=11.5.1}" MODEL="$MODEL_DIR/$MODEL_FILE" -for f in "$LLAMA_BIN" "$MODEL" "$KAIRIC_FFN_SIDECAR" "$KAIRIC_GDN_SIDECAR" "$KAIRIC_GDN_OUTPUT_SIDECAR"; do +for f in "$LLAMA_BIN" "$MODEL" "$MTP_DRAFT_FILE" "$MODEL_DIR/$MMPROJ_FILE" "$CHAT_TEMPLATE_FILE"; do if [[ ! -r "$f" ]]; then echo "missing or unreadable required artifact: $f" >&2 exit 1 @@ -57,11 +54,6 @@ if [[ ! -x "$LLAMA_BIN" ]]; then echo "missing executable runtime: $LLAMA_BIN" >&2 exit 1 fi -if [[ "$KAIRIC_EDGE_COMPATIBILITY_MODE" != 1 ]]; then - echo "KAIRIC_EDGE_COMPATIBILITY_MODE must be 1 for this deployment" >&2 - exit 1 -fi - SERVER_DIR="$(cd -- "$(dirname -- "$LLAMA_BIN")" && pwd)" ROCM_PATH="${ROCM_PATH:-/opt/rocm}" LIBRARY_PATH="$SERVER_DIR:$ROCM_PATH/lib:$ROCM_PATH/lib/rocm_sysdeps/lib:$ROCM_PATH/lib/llvm/lib:$ROCM_PATH/llvm/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" @@ -74,35 +66,14 @@ exec /usr/bin/env \ GGML_CUDA_GRAPH_OPT=0 \ LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0 \ LLAMA_MTP_CPU_ARGMAX_FASTPATH=1 \ - PROMPTFORGE_TARGET_ONLY=0 \ - PROMPTFORGE_MODE=iu4_ffn \ - PROMPTFORGE_IU4_SIDECAR="$KAIRIC_FFN_SIDECAR" \ - PROMPTFORGE_SIDECAR="$KAIRIC_FFN_SIDECAR" \ - PROMPTFORGE_IU4_HADAMARD=1 \ - PROMPTFORGE_IU4_SEGMENTED=0 \ - PROMPTFORGE_ENABLE_FFN_KEEPERS=0 \ - PROMPTFORGE_FFN_KEEPERS=late6 \ - PROMPTFORGE_ENABLE_GDN=1 \ - PROMPTFORGE_ENABLE_GDN_W8=0 \ - PROMPTFORGE_GDN_SIDECAR_OVERRIDE="$KAIRIC_GDN_SIDECAR" \ - PROMPTFORGE_GDN_SIDECAR="$KAIRIC_GDN_SIDECAR" \ - PROMPTFORGE_GDN_IU4_HADAMARD=1 \ - PROMPTFORGE_ENABLE_GDN_KEEPERS=0 \ - PROMPTFORGE_ENABLE_GDN_OUTPUT=1 \ - PROMPTFORGE_GDN_OUTPUT_SIDECAR="$KAIRIC_GDN_OUTPUT_SIDECAR" \ - PROMPTFORGE_GDN_OUTPUT_KEEPERS=v3_lateq6 \ - PROMPTFORGE_ENABLE_SMALLM_IU4=1 \ - PROMPTFORGE_ENABLE_SMALLM_GDN_IU4=1 \ - PROMPTFORGE_ENABLE_SMALLM_GDN_OUTPUT_IU4=0 \ - PROMPTFORGE_ENABLE_IU4_DECODE_M2_M5=0 \ - PROMPTFORGE_OUTPUT_K8_STRICT_GREEDY=0 \ "$LLAMA_BIN" -m "$MODEL" --alias "$ALIAS" \ --host "$BIND_ADDR" --port "$LISTEN_PORT" --jinja \ -dev ROCm0 -ngl 999 -np "$PARALLEL" -c "$CTX_SIZE" \ -b "$BATCH_SIZE" -ub "$UBATCH_SIZE" -fa on \ -ctk "$CACHE_TYPE_K" -ctv "$CACHE_TYPE_V" -t "$NUM_THREADS" -tb "$THREAD_BATCH" \ -ctxcp "$CTX_CHECKPOINTS" --cache-ram "$CACHE_RAM" --cache-prompt --cache-idle-slots --metrics \ - --kairic-edge --spec-type draft-mtp \ + --spec-type draft-mtp --model-draft "$MTP_DRAFT_FILE" \ + --mmproj "$MODEL_DIR/$MMPROJ_FILE" \ --spec-draft-device ROCm0 --spec-draft-ngl 999 \ --spec-draft-type-k "$CACHE_TYPE_K" --spec-draft-type-v "$CACHE_TYPE_V" \ --spec-draft-n-max "$MTP_N_MAX" --spec-draft-n-min "$MTP_N_MIN" \ diff --git a/tests/frank_large_config.bats b/tests/frank_large_config.bats index 2c36ee5..e320d92 100644 --- a/tests/frank_large_config.bats +++ b/tests/frank_large_config.bats @@ -1,6 +1,5 @@ #!/usr/bin/env bash -# Seam: frank-default-large model, runtime, Kairic Edge spec, template. -# Covers the Kairic Edge deployment contract (ciru-ai/ROCmFPX v1.2). +# Seam: frank-default-large model, ROCmFPX runtime, MTP, KV cache, template. # # load helpers @@ -11,34 +10,47 @@ SERVE_FILE="$REPO_ROOT/scripts/serve-frank-large.sh" UNIT_FILE="$REPO_ROOT/scripts/frank-default-large.service" TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja" -@test "frank-default-large serves the Kairic Edge model and sidecars" { - grep -q '^MODEL_FILE=Qwen3.8-27B-IU4-Kairic-Edge\.gguf$' "$ENV_FILE" - grep -q '^KAIRIC_FFN_SIDECAR=' "$ENV_FILE" - grep -q '^KAIRIC_GDN_SIDECAR=' "$ENV_FILE" - grep -q '^KAIRIC_GDN_OUTPUT_SIDECAR=' "$ENV_FILE" - grep -q -- '--kairic-edge' "$SERVE_FILE" +@test "frank-default-large serves the kingjones AGENT model" { + grep -q '^MODEL_FILE=Qwen3\.8-27B-Q8_0_ROCMFPX_AGENT\.gguf$' "$ENV_FILE" + grep -q '^MTP_DRAFT_FILE=' "$ENV_FILE" + ! grep -q -- '--kairic-edge' "$SERVE_FILE" } -@test "frank-default-large uses the Kairic Edge ROCmFPX v1.2 runtime" { - grep -q 'ROCmFPX/build-kairic/bin/llama-server' "$ENV_FILE" - grep -q '^KAIRIC_EDGE_COMPATIBILITY_MODE=1$' "$ENV_FILE" - grep -q '205a3e5f40e5542e2f2eb68e3d3f81f918b1d895' "$ENV_FILE" +@test "frank-default-large uses the ROCmFPX runtime" { + grep -q 'ROCmFPX/.*/bin/llama-server' "$ENV_FILE" + grep -q 'build-kairic/bin/llama-server' "$ENV_FILE" } -@test "frank-default-large runs Kairic Edge MTP in compatibility mode" { +@test "frank-default-large runs standalone MTP" { grep -q -- '--spec-type draft-mtp' "$SERVE_FILE" - grep -q -- '--kairic-edge' "$SERVE_FILE" + grep -qF -- '--model-draft "$MTP_DRAFT_FILE"' "$SERVE_FILE" grep -qF -- '--spec-draft-n-max "$MTP_N_MAX"' "$SERVE_FILE" - grep -qF -- '--spec-draft-p-split "$MTP_P_SPLIT"' "$SERVE_FILE" - grep -q '^KAIRIC_EDGE_COMPATIBILITY_MODE=1$' "$ENV_FILE" grep -q '^MTP_N_MAX=4$' "$ENV_FILE" } +@test "frank-default-large has no obsolete PromptForge sidecar wiring" { + ! grep -qE 'KAIRIC_|PROMPTFORGE_' "$SERVE_FILE" + ! grep -qE 'KAIRIC_|PROMPTFORGE_' "$ENV_FILE" +} + @test "frank-default-large fits the model and KV cache in unified memory" { - grep -q '^CTX_SIZE=65536$' "$ENV_FILE" + grep -q '^CTX_SIZE=262144$' "$ENV_FILE" grep -q ': "${CTX_SIZE:=262144}"' "$SERVE_FILE" } +@test "frank-default-large has reasoning and preserved reasoning/thinking enabled" { + grep -q '^REASONING=on$' "$ENV_FILE" + grep -q 'enable_thinking":true' "$ENV_FILE" + grep -q 'preserve_thinking":true' "$ENV_FILE" + grep -q 'preserve_reasoning":true' "$ENV_FILE" +} + +@test "frank-default-large uses Turbo4 only for the V cache" { + grep -q '^CACHE_TYPE_K=f16$' "$ENV_FILE" + grep -q '^CACHE_TYPE_V=turbo4$' "$ENV_FILE" + grep -qF -- '-ctk "$CACHE_TYPE_K" -ctv "$CACHE_TYPE_V"' "$SERVE_FILE" +} + @test "frank-default-large exposes a swappable ubatch" { grep -q '^UBATCH_SIZE=512$' "$ENV_FILE" grep -q ': "${UBATCH_SIZE:=512}"' "$SERVE_FILE"