diff --git a/AGENTS.md b/AGENTS.md index 735eb4d..dc1728b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,7 +27,7 @@ host. | Target | Runtime | Service | Model | Port | |--------|---------|---------|-------|------| | Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 | -| Dankclaw | CIRU ROCmFPX llama.cpp, PromptForge v3 runtime (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3 (Q8-FFNEQ-LateQ6, full-W8 views, Output-K8 MTP proxy, BF16 vision), 262K context / 1 slot | 7770 | +| Dankclaw | charlie12345/ROCmFPX llama.cpp, ROCmI4 W4A4 runtime | frank-default-large | caf-eemin Qwen3.8-27B-ROCmI4-MTP Q4_0_ROCMI4 (embedded NextN/MTP head, gfx1151 IU4 batched verification), 262K context / 1 slot | 7770 | History: the 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on @@ -179,17 +179,22 @@ When you rename a service or move a model to a new host, update ## Runtime & Troubleshooting -### v3 CIRU ActiveFPX PromptForge llama.cpp (Dankclaw, ACTIVE) +### charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, ACTIVE) + +The v4 runtime serves `cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (`eemin` namespace, +public) from `/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` +(charlie12345/ROCmFPX commit `c49ebdbd` built with `-DGGML_HIP_ROCMI4_W4A4=ON` +against system ROCm 7.2.4 / amdclang 22). The model is `Q4_0_ROCMI4` with an +embedded NextN/MTP head (no separate draft file); the gfx1151 native IU4 WMMA +instruction batch-verifies proposals. Profile: strict Qwen MTP +(`--spec-mtp-strict-qwen`, n_max 16, p_min 0.60, backend sampling, all layers +on ROCm0), `-t 16 -tb 32`, batch 512 / ubatch 256 (ubatch swappable via the +ubatch bench on 8k context), f16 KV, full 262144 context / 1 slot, TEMPERATURE +0.7 / TOP_P 0.95, reasoning on (deepseek, budget -1), `--no-mmap`. Two-file +payload (~14.4 GiB) in `/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP/`. Full +runbook: +[docs/LLM/eemin-rocmi4-mtp.md](docs/LLM/eemin-rocmi4-mtp.md). -The v3 runtime serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3` -from `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` -(PromptForge v2.3 + v3 Output-K8 runtime patch). The v3 profile: -`PROMPTFORGE_MODE=m2048_fused_tail1476`, full-W8 companion views, Output-K8 -MTP draft proxy, BF16 mmproj vision, TEMPERATURE 0.7 / TOP_P 0.95, -batch 4096 / ubatch 4096, 16 context checkpoints, reasoning on (deepseek, -budget -1), `--no-mmap`. Five-file payload (~36.7 GiB) in -`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`. Full runbook: -[docs/LLM/ciru-activefpx-promptforge.md](docs/LLM/ciru-activefpx-promptforge.md). ### Kairic Edge llama.cpp (Dankclaw, RETIRED experiment 2026-08-22) diff --git a/docs/LLM/deployment.md b/docs/LLM/deployment.md index 455360b..f9cf53f 100644 --- a/docs/LLM/deployment.md +++ b/docs/LLM/deployment.md @@ -18,7 +18,7 @@ repository to a running service. | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| | Dankserver | dank@dankserver | upstream llama.cpp (Ornith-1.5-35B-A3B-CRACK Q6_K, native MTP draft, selected MoE experts offloaded via --override-tensor) | frank-default-small | 7770 | -| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (Qwen3.8-27B-Uncensored-ROCmFP4 Q6 AGENT, standalone MTP Q4_0 head, 262K ctx / 1 slot) | frank-default-large | 7770 | +| Dankclaw | dank@dankclaw | charlie12345/ROCmFPX llama.cpp (caf-eemin Qwen3.8-27B ROCmI4 Q4_0, embedded MTP, ROCmI4 W4A4, 262K ctx / 1 slot) | frank-default-large | 7770 | ## Workflow @@ -137,24 +137,29 @@ from `peculiar-ragdoll/Qwen-Sharp-Chat-Templates`. See [[dealign-ornith-crack]]. ### Dankclaw — frank-default-large -The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving -`kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` (Q6_0 AGENT, dense) -under the alias `frank-default-large` on port 7770. It runs -`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm -7.15 SDK in the retired ling3-ciru venv) with a standalone MTP Q4_0 draft -head (`--model-draft` and `--spec-type draft-mtp`), f16 KV, full 262144 -context (1 slot. Concurrent requests queue; the server never preempts a -running generation), batch 4096 / ubatch 4096, 16 context checkpoints, -8 GiB prompt cache, and reasoning in deepseek format. The unit file is +The active service on Dankclaw is the charlie12345/ROCmFPX llama.cpp runtime +(with the opt-in `ROCmI4 W4A4` path) serving `cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` +(`eemin` namespace, `Q4_0_ROCMI4`, embedded NextN/MTP head) under the alias +`frank-default-large` on port 7770. It runs +`/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` (built +against system ROCm 7.2.4 / amdclang 22) with a **strict Qwen MTP** draft +(`--spec-type draft-mtp --spec-mtp-strict-qwen`) running on ROCm0 with all +layers (`--spec-draft-ngl all`) — no separate draft file, because the MTP head +is embedded in the main GGUF. K/V are f16, full 262144 context (1 slot, +concurrent requests queue, the server never preempts a running generation), +`-t 16 -tb 32`, batch 512 / ubatch 256 (ubatch is swappable; sweep it on 8k +context with `scripts/bench-ubatch-rocmi4.sh`), 16 context checkpoints, 8 GiB +prompt cache, and reasoning in deepseek format. The unit file is `scripts/frank-default-large.service`. It launches through `scripts/serve-frank-large.sh`, which reads tunables from `scripts/frank-default-large.env`, applying the Qwen-Sharp chat template. -See [[kingjones-strix-mtp]]. +See [[eemin-rocmi4-mtp]]. -> The kingjones Q6 AGENT model has served the slot since 2026-08-24; the -> v3 PromptForge payload is retired on this slot. -> Restored 2026-08-21. The intermediate DFlash2 Vulkan deployment was -> retired the same day. See [[dflash2-large-vulkan]] for the archived runbook. +> The kingjones ROCmFP4 STRIX MTP model served the slot from 2026-08-24; it +> was replaced by the caf-eemin ROCmI4 embedded-MTP model on 2026-08-25. The +> v3 PromptForge payload and the CIRU ROCmFPX runtime are retired on this slot. +> The intermediate DFlash2 Vulkan deployment was retired the same day (2026-08-21). +> See [[dflash2-large-vulkan]] for the archived runbook. ## Validation diff --git a/docs/LLM/eemin-rocmi4-mtp.md b/docs/LLM/eemin-rocmi4-mtp.md new file mode 100644 index 0000000..f2ca616 --- /dev/null +++ b/docs/LLM/eemin-rocmi4-mtp.md @@ -0,0 +1,184 @@ +--- +type: reference +title: caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw +description: caf-eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF (Q4_0_ROCMI4, embedded NextN/MTP head) served by the charlie12345/ROCmFPX llama.cpp runtime (ROCmI4 W4A4) on the Strix Halo iGPU. frank-default-large. +tags: + - llama.cpp + - rocm + - dankclaw + - gfx1151 + - rocmi4 + - mtp + - iu4 +timestamp: 2026-08-25 +--- + +# caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw + +> **Active 2026-08-25.** `frank-default-large` serves this model. It replaced +> the kingjones777 ROCmFP4 STRIX MTP payload on the same port and the same +> Strix Halo iGPU, running the new charlie12345/ROCmFPX ROCmI4 W4A4 build. See +> [[kingjones-strix-mtp]] for the retired page. + +`frank-default-large` (port 7770) serves +`cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (published under the public +`eemin` namespace) from the **charlie12345/ROCmFPX** llama.cpp fork built with +the opt-in `ROCmI4 W4A4` path. The model is `Q4_0_ROCMI4`: signed four-bit +weights packed two per byte, quantized from the BF16 Qwen3.8-27B and carrying +its **embedded NextN/MTP head** in the main GGUF — so there is no separate +draft file. The native gfx1151 `v_wmma_i32_16x16x16_iu4` matrix instruction +batch-verifies the MTP proposals for higher decode throughput. + +## What This Is + +**ROCmI4** is the GGUF weight format (signed 4-bit codes). **IU4** is the AMD +WMMA instruction spelling that consumes them. **W4A4** means four-bit weights +and four-bit activations during the accelerated matrix multiply. Ordinary +one-token decode stays memory-bandwidth-bound (MMVQ); batched prompt processing +and batched MTP target verification use the IU4 MMQ path. W4A4 is a lossy +activation-quant speed path — treat it as a speed/quality tradeoff. + +The model is a conversion/quantization of `Qwen/Qwen3.8-27B` (Unsloth lineage, +Apache-2.0), not a retrained model. It reports architecture `qwen35` +(M-RoPE), ~27.32B params, native 262144 context, 866 tensors, ftype 108. + +## Model Files + +| File | Size | SHA-256 | +|------|------|---------| +| `Qwen3.8-27B-Q4_0_ROCMI4.gguf` | 13.50 GiB (14,534,384,384 B) | `ad51ecbee55930ababc629c377943a7bbe50f8aafcf32340c7cdae6fa6b7bb3c` | +| `mmproj-BF16.gguf` (vision, optional) | 0.87 GiB (931,146,432 B) | `83ee4f4f205fa514161778c41df1ea14144faa0f713510893b63c2395f5c2d53` | + +Store under `/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP/`. The vision +projector is optional for text-only use. + +## Runtime Build + +The stock llama.cpp build does not understand `Q4_0_ROCMI4` or the qualified +IU4/MTP path. Use charlie12345/ROCmFPX at the pinned commit, built HIP-only +(so the combined HIP/Vulkan scheduler cannot route the ROCmI4 tensors to +Vulkan, which has no shader for this type) with the W4A4 option on. + +| Artifact | Path | +|----------|------| +| Fork + pinned commit | `/home/dank/rocmi4-qwen38/ROCmFPX` @ `c49ebdbd5c9f01ec242369f9e7f7967855f80cba` | +| Build | `cmake -S . -B build-rocmi4-w4a4 -G Ninja -DGGML_HIP=ON -DGGML_VULKAN=OFF -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DGGML_HIP_ROCMI4_W4A4=ON -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_WEBUI=OFF` | +| Binary | `/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` (version 257, `c49ebdb`) | +| ROCm | system `/opt/rocm-7.2.4`, amdclang 22 | + +Build a separate directory so the exact-int8 path (the default branch) stays +available. Rollback: disable `-DGGML_HIP_ROCMI4_W4A4=ON` and rebuild — no model +conversion or data change needed. The build prints +`ROCmI4 W4A4: enabled for device 0` when the accelerated path is active. + +## Model Download + +Downloaded from the public +`eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF` repo (the card's download section still +names the still-private `cafonz/…`; use `eemin`). Verify against the repo's +`checksums.sha256` after download: + +``` +cd /home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP +sha256sum -c checksums.sha256 +``` + +## Launch Profile + +The qualified MTP profile (from the model card and PR #94, verified on the +live service): strict Qwen MTP, backend sampling, `n_max 16`, `p_min 0.60`. + +```mermaid +graph TD + A[llama-server charlie12345 ROCmFPX c49ebdb
ROCmI4 W4A4 enabled] --> B[load Q4_0_ROCMI4 GGUF + mmproj
-ngl 999 -dev ROCm0] + B --> C[draft-mtp on ROCm0
--spec-draft-ngl all] + C --> D[strict Qwen MTP
n_max 16 / p_min 0.60
backend sampling] + D --> E[Sharp chat template
reasoning deepseek] +``` + +Key flags (see `scripts/serve-frank-large.sh`): + +``` +--spec-type draft-mtp --spec-mtp-strict-qwen +--spec-draft-device ROCm0 --spec-draft-ngl all +--spec-draft-type-k f16 --spec-draft-type-v f16 +--spec-draft-n-max 16 --spec-draft-n-min 0 +--spec-draft-p-min 0.60 --spec-draft-backend-sampling +``` + +Because the MTP head is **embedded**, no `--model-draft` is passed and the +draft uses all layers on ROCm0. `--spec-mtp-strict-qwen` preserves the target +path's greedy decisions; it does not remove the W4A4 activation approximation. + +### Serving settings + +- 262144 context (native max), 1 slot, f16 KV, context checkpoints, prompt + cache. Concurrent requests queue; the server never preempts a running + generation. +- `-t 16 -tb 32`, batch 512 / ubatch 256 (the qualified W4A4 profile). +- `--temp 0.7 --top-p 0.95`, reasoning on (`deepseek` puts `` into + `message.reasoning_content`). +- Qwen-Sharp chat template (`--chat-template-file`), `enable_thinking` + + `preserve_thinking` on, preserving the peg-native + deepseek shape the Pi + agent expects. +- Strix Halo unified memory + native gfx1151 detection: `HSA_OVERRIDE_GFX_VERSION=11.5.1`, + `GGML_HIP_ENABLE_UNIFIED_MEMORY=1`. + +### Service + +- Unit: `frank-default-large.service`, port 7770 +- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-default-large.env` + +## Tuning MTP + +MTP gains are content-dependent: predictable code, structured data, and lists +accept longer drafts than creative prose. The qualified start is `n16 / p0.60`. +If acceptance or stability is poor on another workload, lower +`--spec-draft-n-max` to 4 or 6 and remeasure. + +## ubatch Benchmark + +Decode throughput depends on the ubatch (per-eval token batch). Run the sweep +on 8k contexts, 512 → 4096 in steps of 512, with a fixed batch: + +``` +STOPSVC=1 ./scripts/bench-ubatch-rocmi4.sh +``` + +The script stops `frank-default-large` for a clean GPU, boots one server +instance per ubatch on a scratch port, reads authoritative `print_timing` tok/s, +and restarts the service. See [[frank-default-large-benchmark]]. + +## Known Issues + +1. **gfx1151 only** for the W4A4 path. Other GPUs fall back to exact int8 MMQ. +2. **Memory pressure:** the model is 13.5 GiB. Under `--no-mmap` with full + offload it needs the whole tensor set at once; leave room in the 64 GiB + unified pool (rocm-smi figure). Do not run a bench with the live service up + or it OOMs at load. +3. **`Q4_0_ROCMI4` + MTP flags need the ROCmFPX fork**; stock llama.cpp may + reject the file or drop the optimized path. +4. **W4A4 is lossy** (+~5.4% perplexity vs exact int8). Opt-in speed path. + +## Measured Results (qualification) + +From the model card, gfx1151, pinned build, W4A4 + strict MTP-16: + +| Result | Value | +|--------|-------| +| HumanEval pilot, W4A4 IU4 (mean) | 49.40 tok/s | +| Full HumanEval, W4A4 (mean / median) | 44.39 / 45.23 tok/s | +| HumanEval / HumanEval+ pass@1 | 94.5% / 91.5% | +| Non-speculative tg128 | ~13.8 tok/s | + +Local smoke test 2026-08-25 on the iGPU: model loaded, strict-MTP active, +draft acceptance 0.74 (mean 7.67), generation ~25 tok/s at a small output. +The ubatch sweep above gives the production ubatch choice. + +## See Also + +- [[deployment]] — deploy and update process +- [[kingjones-strix-mtp]] — the retired ROCmFP4 predecessor on this slot +- [[ciru-activefpx-promptforge]] — the retired v3 CIRU PromptForge page +- [[Dankclaw System Specs]] — the box +- [[frank-default-large-benchmark]] — backend/spec benchmark history diff --git a/docs/LLM/index.md b/docs/LLM/index.md index 73dbb70..94c488e 100644 --- a/docs/LLM/index.md +++ b/docs/LLM/index.md @@ -48,8 +48,12 @@ MTP draft + Sharp template, 262K ctx / 1 slot] - [[deployment]] — deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model - on Dankclaw (frank-default-large, active 2026-08-24) +- [[eemin-rocmi4-mtp]] — the caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model + on Dankclaw (frank-default-large, active 2026-08-25) +- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on + Dankserver (frank-default-small, active 2026-08-24) +- [[kingjones-strix-mtp]] — the retired kingjones Qwen3.8-27B ROCmFP4 STRIX MTP + model on Dankclaw (superseded 2026-08-25 by the ROCmI4 embedded-MTP model) - [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (frank-default-small, active 2026-08-24) - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime diff --git a/docs/LLM/kingjones-strix-mtp.md b/docs/LLM/kingjones-strix-mtp.md index 8ade2ac..441223e 100644 --- a/docs/LLM/kingjones-strix-mtp.md +++ b/docs/LLM/kingjones-strix-mtp.md @@ -15,10 +15,13 @@ timestamp: 2026-08-24 # kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw -> **Active 2026-08-24.** `frank-default-large` serves this model. It -> replaced the jcbtc v3 CIRU ActiveFPX PromptForge payload on the same -> ROCmFPX runtime and the same CIRU llama.cpp binary. See -> [[ciru-activefpx-promptforge]] for the retired v3 page. +> **Retired 2026-08-25.** The kingjones ROCmFP4 STRIX MTP payload has been +> replaced by the caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on the same +> port and iGPU, under a new runtime (charlie12345/ROCmFPX ROCmI4 W4A4). See +> [[eemin-rocmi4-mtp]]. The old kingjones GGUF + standalone MTP Q4_0 draft are +> retired on this slot. The CIRU ROCmFPX runtime/binary is also retired here. + +> _This page documents the retired model and its runbook for reference._ `frank-default-large` (port 7770) serves `Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` from `kingjones777` on Hugging diff --git a/docs/index.md b/docs/index.md index 8c7b0ec..c05b07d 100644 --- a/docs/index.md +++ b/docs/index.md @@ -96,6 +96,8 @@ The LLM inference fleet on the two hosts. Full index: - [[deployment]] — unified deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change +- [[eemin-rocmi4-mtp]] — the caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw (frank-default-large, active) +- [[kingjones-strix-mtp]] — the RETIRED kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (superseded 2026-08-25) - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (active) diff --git a/scripts/bench-ubatch-rocmi4.sh b/scripts/bench-ubatch-rocmi4.sh new file mode 100755 index 0000000..b2a8866 --- /dev/null +++ b/scripts/bench-ubatch-rocmi4.sh @@ -0,0 +1,218 @@ +#!/usr/bin/env bash +# bench-ubatch-rocmi4.sh — sweep ubatch size at 8k context on the +# frank-default-large ROCmI4 embedded-MTP stack (dankclaw, gfx1151). +# +# Boots one llama-server instance per ubatch on a scratch port with a fixed +# -b and -c 8192, runs a standardized ~75%-of-ctx prose workload, and reads +# authoritative prefill/generation tok/s from the server's own print_timing +# lines. Steps from UBATCH_MIN to UBATCH_MAX inclusive by UBATCH_STEP. +# +# Usage: +# ./scripts/bench-ubatch-rocmi4.sh +# UBATCH_MIN=512 UBATCH_MAX=4096 UBATCH_STEP=512 CTX=8192 RUNS=3 \ +# ./scripts/bench-ubatch-rocmi4.sh +# +# Benchmarking policy: stop the production service first so the GPU is not +# starved. With STOPSVC=1 the script restarts frank-default-large on exit. +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-default-large.env}" +if [[ -f "$ENV_FILE" ]]; then + set -a + # shellcheck disable=SC1090 + source "$ENV_FILE" + set +a +fi + +# --- sweep parameters ------------------------------------------------------- +: "${UBATCH_MIN:=512}" +: "${UBATCH_MAX:=4096}" +: "${UBATCH_STEP:=512}" +: "${CTX:=8192}" # 8k contexts +: "${BENCH_BATCH:=$CTX}" # fixed batch = ctx, so the full 8k prompt fits +: "${OUT_TOKENS:=256}" +: "${RUNS:=3}" # decode runs per ubatch +: "${PORT_BASE:=7894}" # scratch ports 7894.. +: "${STOPSVC:=0}" # stop/restart the live unit for a clean GPU + +# --- resolved paths (fall back to serve-script defaults) -------------------- +: "${LLAMA_BIN:=/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server}" +: "${MODEL_DIR:=/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP}" +: "${MODEL_FILE:=Qwen3.8-27B-Q4_0_ROCMI4.gguf}" +: "${MMPROJ_FILE:=mmproj-BF16.gguf}" +: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}" +: "${ALIAS:=frank-default-large}" +: "${HSA_OVERRIDE_GFX_VERSION:=11.5.1}" +: "${GGML_HIP_ENABLE_UNIFIED_MEMORY:=1}" +: "${MTP_N_MAX:=16}" +: "${MTP_P_MIN:=0.60}" + +MODEL="$MODEL_DIR/$MODEL_FILE" +MMPROJ="$MODEL_DIR/$MMPROJ_FILE" + +for f in "$MODEL" "$CHAT_TEMPLATE_FILE"; do + if [[ ! -f "$f" ]]; then + echo "missing required file: $f" >&2 + exit 1 + fi +done +if [[ ! -x "$LLAMA_BIN" ]]; then + echo "missing llama-server binary: $LLAMA_BIN" >&2 + exit 1 +fi + +MMPROJ_ARGS=() +if [[ -f "$MMPROJ" ]]; then + MMPROJ_ARGS=(--mmproj "$MMPROJ") +fi + +STopped=0 +cleanup() { + if [ "$STopped" = 1 ]; then + echo "=== restarting frank-default-large ===" + sudo systemctl restart frank-default-large || true + STopped=0 + fi +} +trap cleanup EXIT + +stop_live_service() { + if [ "$STOPSVC" = 1 ]; then + echo "=== stopping frank-default-large for a clean GPU ===" + sudo systemctl stop frank-default-large || true + STopped=1 + fi +} + +# wait_health — poll /health until ready or the server dies. +wait_health() { + local pid="$1" port="$2" i ready=0 crashed=0 + for i in $(seq 1 900); do + if ! kill -0 "$pid" 2>/dev/null; then crashed=1; break; fi + if curl -sf "http://127.0.0.1:${port}/health" >/dev/null 2>&1; then + ready=1; break + fi + [ $((i % 10)) = 0 ] && echo -n . + sleep 2 + done + [ "$crashed" = 1 ] && { echo " BACKEND_CRASHED (before ready)"; return 2; } + [ "$ready" = 1 ] || { echo " FAIL (timeout)"; return 1; } + echo " [ready]" + return 0 +} + +# run_one — boot one server instance, run +# the workload, parse print_timing, print a RESULT line. +run_one() { + local ubatch="$1" port="$2" out_tokens="$3" runs="$4" + local log="/tmp/ubatch-${ubatch}-p${port}.log" + : > "$log" + + echo "=== ubatch=${ubatch} ctx=${CTX} port=${port} runs=${runs} ===" + "$LLAMA_BIN" \ + --model "$MODEL" \ + "${MMPROJ_ARGS[@]}" \ + --chat-template-file "$CHAT_TEMPLATE_FILE" \ + --chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true}' \ + --alias bench-ubatch \ + --host 127.0.0.1 --port "$port" \ + -dev ROCm0 -ngl 999 -np 1 \ + -t 16 -tb 32 \ + -c "$CTX" -b "$BENCH_BATCH" -ub "$ubatch" \ + -fa on -ctk f16 -ctv f16 -ctxcp 16 \ + --cache-ram 8192 --cache-prompt --metrics --no-mmap \ + --spec-type draft-mtp --spec-mtp-strict-qwen \ + --spec-draft-device ROCm0 --spec-draft-ngl all \ + --spec-draft-type-k f16 --spec-draft-type-v f16 \ + --spec-draft-n-max "$MTP_N_MAX" --spec-draft-n-min 0 \ + --spec-draft-p-min "$MTP_P_MIN" \ + --spec-draft-backend-sampling \ + --temp 0.7 --top-p 0.95 \ + --reasoning on --reasoning-format deepseek --reasoning-budget -1 \ + -lv 4 >>"$log" 2>&1 & + local pid=$! + wait_health "$pid" "$port" || { tail -30 "$log"; return 2; } + + # one python pass per run; reuse the same boot across runs + local r + for r in $(seq 1 "$runs"); do + echo "--- run ${r}/${runs} ---" + python3 - "$ubatch" "$port" "$out_tokens" "$CTX" "$log" <<'PY' +import sys, time, json, urllib.request +ubatch, port, out_tokens, ctx, log = sys.argv[1], sys.argv[2], int(sys.argv[3]), int(sys.argv[4]), sys.argv[5] + +# Flowing prose at ~75% of ctx so MTP acceptance is realistic. +VAR = ( + "narrative nonfiction blends observation with reflection, ", + "and a careful writer lets each clause earn its place ", + "in the sentence's rising line, so meaning accumulates slowly. ", + "The reader is not hurried; the reader is led. Loose and periodic, ", + "the style turns on concrete nouns and verbs that carry weight, ", + "and it resists the abstract noun that hides a vague claim. ", + "A paragraph is not a list. It is a single turn of thought that ", + "opens, develops, and lands. This is the craft of prose. Choose, ", + "arrange, and above all omit. Omission is the discipline that ", + "keeps the whole clear, and clarity is the courtesy we owe the ", + "willing reader." +) +n = len(VAR) +parts = [] +t = 0 +while t * 76 < (ctx * 3) // 4: + for k in range(5): + parts.append(VAR[(t + k) % n]) + parts.append(f" This is cued by note {t}. ") + t += 1 +prompt = "".join(parts) +prompt += ("\n\nNow, in a detailed and flowing paragraph, continue this essay about " + "the philosophy of language and its influence on modern computation. ") + +body = json.dumps({ + "prompt": prompt, "max_tokens": out_tokens, + "temperature": 0.0, "stream": False, +}).encode() +req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/completions", + data=body, headers={"Content-Type": "application/json"}) +t0 = time.time() +with urllib.request.urlopen(req, timeout=7200) as resp: + out = json.load(resp) +dt = time.time() - t0 +pt = out.get("usage", {}).get("prompt_tokens", 0) +ct = out.get("usage", {}).get("completion_tokens", 0) + +lines = open(log, encoding="utf-8", errors="ignore").read().splitlines() +prefill = [l for l in lines if "prompt eval time" in l] +evals = [l for l in lines if "eval time" in l and "prompt eval" not in l] + +def grab(arr): + return arr[-1].strip() if arr else "n/a" + +g_tps = "" +if evals and ct: + # eval time line: "eval time = .. ms for N tokens" + ev = evals[-1].split("= ")[-1] + ms = float(ev.split(" for")[0].strip()) + g_tps = f"{ct / (ms / 1000):.2f}" +print(f"== RESULT ubatch={ubatch} run={r} " + f"PREFILL={grab(prefill)} GEN_TPS={g_tps} " + f"pt={pt} ct={ct} wall={dt:.1f}s") +PY + done + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true +} + +main() { + stop_live_service + local idx=0 ubatch port + printf "%-10s %-14s %-10s\n" "UBATCH" "PREFILL" "GEN_TPS" + for ubatch in $(seq "$UBATCH_MIN" "$UBATCH_STEP" "$UBATCH_MAX"); do + port=$((PORT_BASE + idx)) + run_one "$ubatch" "$port" "$OUT_TOKENS" "$RUNS" + idx=$((idx + 1)) + done + echo "=== sweep complete ===" +} + +main "$@" diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index 19409b2..3115e7d 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -1,75 +1,79 @@ -# Tunables for frank-default-large (CIRU llama.cpp, dankclaw, Strix Halo -# gfx1151). +# Tunables for frank-default-large (charlie12345/ROCmFPX, dankclaw, Strix Halo +# gfx1151, ROCmI4 embedded MTP). # -# Runtime: the PromptForge llama-server built from ciru-ai/ROCmFPX tag -# qwen3.8-activefpx-promptforge-v2.3 plus the v3 output-K8 runtime patch -# serving kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4 (dense, Q6 AGENT -# GGUF) with the standalone MTP Q4_0 draft head. Alias frank-default-large, -# port 7770. +# Runtime: charlie12345/ROCmFPX commit c49ebdbd built with +# -DGGML_HIP_ROCMI4_W4A4=ON against the system ROCm (/opt/rocm-7.2.4, amdclang +# 22). Serves caf-eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF (Q4_0_ROCMI4, embedded +# MTP, no separate draft head). Alias frank-default-large, port 7770. # -# This model replaced jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3) on -# 2026-08-24. The new model is a plain ROCmFP4 GGUF (no PromptForge .pfs -# sidecars, no Output-K8 proxy); the requested file is Q6_. +# This model replaced kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP on +# 2026-08-25. The old kingjones ROCmFP4 GGUF + standalone MTP Q4_0 draft are +# retired on this slot. # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The ROCm # HIP runtime reads those bare env vars (see the 2026-08-13 TEMP incident). # llama.cpp reads --host/--port/--temp as CLI flags, never as env vars. -# CIRU llama-server binary (ROCmFPX v2.3 + v3 output-K8 patch build, built -# against the private TheRock ROCm 7.15 SDK kept in the retired ling3-ciru -# venv). ROCm 7.15 is required for the advertised Strix Halo performance. -LLAMA_BIN=/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server +# charlie12345/ROCmFPX build with the ROCmI4 W4A4 path. +LLAMA_BIN=/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server -# Model files. Plain dense Qwen3.8-27B ROCmFP4 Q6 AGENT GGUF plus the standalone -# MTP draft head and the vision projector. Kingjones 2026-08-24. -MODEL_DIR=/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP -MODEL_FILE=Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf -MTP_DRAFT_FILE=mtp-Qwen3.8-27B-Q4_0.gguf -MMPROJ_FILE=mmproj-Qwen3.8-27B-Q8_0.gguf +# Model files: embedded-MTP ROCmI4 GGUF + optional BF16 vision projector. +MODEL_DIR=/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP +MODEL_FILE=Qwen3.8-27B-Q4_0_ROCMI4.gguf +MMPROJ_FILE=mmproj-BF16.gguf +# No separate MTP draft head: the MTP/NextN head is embedded in the main GGUF, +# so no --model-draft is passed. Kept for scripts that reference the var. +MTP_DRAFT_FILE= -# Qwen-Sharp chat template (tracked in the repo, staged on the host). +# Qwen-Sharp chat template (tracked in the repo, staged on the host). The Sharp +# template drives thinking on by default (medium effort) and produces the +# peg-native + deepseek chat/reasoning shape the Pi agent expects. CHAT_TEMPLATE_FILE=/home/dank/serverAppConfigs/scripts/chat-template.jinja -# enable_thinking + preserve_thinking on by default in the Sharp template; -# medium effort default. preserve_thinking keeps blocks in recorded -# assistant messages (set to false to strip them, true to always keep them). CHAT_TEMPLATE_KWARGS='{"enable_thinking":true,"preserve_thinking":true}' -# ROCm 7.15 SDK paths (inside the retired ling3-ciru venv). -ROCM_DEVEL=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_devel -ROCM_LIB=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_libraries -ROCM_CORE=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_core - +# Alias + network. ALIAS=frank-default-large BIND_ADDR=0.0.0.0 LISTEN_PORT=7770 -# Serving: 196608 (192K) context, 1 slot. f16 KV, context checkpoints, -# prompt cache, draft-mtp with standalone Q4_0 head, n_max 4 (the knee for -# the dense model). Unified memory; rocm-smi reports the 64 GiB budget. -CTX_SIZE=196608 +# Serving: full 262144 context, 1 slot. f16 KV, context checkpoints, prompt +# cache, strict-MTP embedded draft on ROCm0. Unified memory; rocm-smi reports +# the 64 GiB budget. Concurrent requests queue; the server never preempts a +# running generation. +CTX_SIZE=262144 PARALLEL=1 -BATCH_SIZE=8192 -UBATCH_SIZE=8192 +# Batch/ubatch follow the qualified ROCmI4 W4A4 profile (see the bench in +# scripts/bench-ubatch-rocmi4.sh for the ubatch sweep on 8k contexts). +BATCH_SIZE=512 +UBATCH_SIZE=256 +NUM_THREADS=16 +THREAD_BATCH=32 CACHE_TYPE_K=f16 CACHE_TYPE_V=f16 -CTX_CHECKPOINTS=32 +CTX_CHECKPOINTS=16 CACHE_RAM=8192 -# Dense-model MTP draft (standalone Q4_0 head, n_max 4). No PromptForge .pfs -# sidecars are loaded; the LLAMA_CPU_ARGMAX fast path stays off. -SPEC_DRAFT_N_MAX=4 -SPEC_DRAFT_N_MIN=0 -SPEC_DRAFT_P_MIN=0.0 -SPEC_DRAFT_P_SPLIT=0.10 +# Embedded-MTP (qwen35 NextN) qualified profile. The head is in the main GGUF, +# so the draft runs with all layers on ROCm0 (no --model-draft). +MTP_N_MAX=16 +MTP_N_MIN=0 +MTP_P_MIN=0.60 +SPEC_MTP_STRICT_QWEN=on +MTP_BACKEND_SAMPLING=on +SPEC_DRAFT_NGL=all -# Sampling (kingjones model card / Sharp default). +# Sampling (Sharp default). TEMPERATURE=1.0 TOP_K=20 TOP_P=0.95 -# Thinking on (Qwen-Sharp template drives thinking on; reasoning format -# projects the tar into reasoning_content). +# Thinking on (Sharp template drives it; reasoning format projects the tar into +# reasoning_content). REASONING=on REASONING_FORMAT=deepseek +# Strix Halo unified memory + native gfx1151 detection for this ROCmFPX build. +HSA_OVERRIDE_GFX_VERSION=11.5.1 +GGML_HIP_ENABLE_UNIFIED_MEMORY=1 + VERBOSITY=4 diff --git a/scripts/serve-frank-large.sh b/scripts/serve-frank-large.sh index c72c05f..5d83011 100755 --- a/scripts/serve-frank-large.sh +++ b/scripts/serve-frank-large.sh @@ -1,22 +1,18 @@ #!/usr/bin/env bash -# Serve kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP (dense) as -# frank-default-large via the CIRU ROCmFPX llama.cpp runtime on Dankclaw -# (Strix Halo gfx1151). +# Serve caf-eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF (Q4_0_ROCMI4, embedded MTP) as +# frank-default-large via the charlie12345/ROCmFPX llama.cpp runtime +# (ROCmI4 W4A4) on Dankclaw (Strix Halo gfx1151). # -# Runtime: the PromptForge llama-server built from ciru-ai/ROCmFPX tag -# qwen3.8-activefpx-promptforge-v2.3 (commit 103af8b41) plus the v3 output-K8 -# runtime patch (commit af0eac6), against the private TheRock ROCm 7.15 SDK -# (kept in the retired ling3-ciru venv). The model is the plain ROCmFP4 -# Q6 AGENT GGUF (no PromptForge .pfs sidecars) with the standalone MTP draft -# head mtp-Qwen3.8-27B-Q4_0.gguf. The Qwen-Sharp chat template is applied at -# runtime. +# Runtime: charlie12345/ROCmFPX pinned at commit +# c49ebdbd5c9f01ec242369f9e7f7967855f80cba, built with +# -DGGML_HIP_ROCMI4_W4A4=ON against the system ROCm (/opt/rocm-7.2.4, +# amdclang 22). The model has an embedded NextN/MTP head in the main GGUF, so +# a separate draft file is not loaded; the MTP draft runs on ROCm0 with all +# layers via --spec-draft-ngl all. # -# Spec: dense model -> native MTP draft at n_max 4 (the knee, per the model -# card). The draft head is a separate Q4_0 GGUF loaded via --model-draft. -# -# The model card requires ROCm 7.15 (not the system ROCm 7.2.4) for the -# advertised Strix Halo performance. LD_LIBRARY_PATH therefore points at the -# 7.15 SDK libraries first. +# The qualified MTP profile is strict Qwen MTP (--spec-mtp-strict-qwen) with +# backend sampling, n_max 16, p_min 0.60 (see the model card and the +# charlie12345/ROCmFPX ROCMI4 runbook). # # Tunables come from frank-default-large.env in this directory unless # FRANK_LARGE_ENV_FILE overrides that path. @@ -32,30 +28,32 @@ if [[ -f "$ENV_FILE" ]]; then fi # --- defaults (used when the env file does not set them) ------------------- -: "${LLAMA_BIN:=/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server}" -: "${MODEL_DIR:=/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP}" -: "${MODEL_FILE:=Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf}" -: "${MTP_DRAFT_FILE:=mtp-Qwen3.8-27B-Q4_0.gguf}" -: "${MMPROJ_FILE:=mmproj-Qwen3.8-27B-Q8_0.gguf}" +: "${LLAMA_BIN:=/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server}" +: "${MODEL_DIR:=/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP}" +: "${MODEL_FILE:=Qwen3.8-27B-Q4_0_ROCMI4.gguf}" +: "${MMPROJ_FILE:=mmproj-BF16.gguf}" : "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}" -: "${ROCM_DEVEL:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_devel}" -: "${ROCM_LIB:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_libraries}" -: "${ROCM_CORE:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_core}" : "${ALIAS:=frank-default-large}" : "${BIND_ADDR:=0.0.0.0}" : "${LISTEN_PORT:=7770}" : "${CTX_SIZE:=262144}" : "${PARALLEL:=1}" -: "${BATCH_SIZE:=4096}" -: "${UBATCH_SIZE:=4096}" +: "${BATCH_SIZE:=512}" +: "${UBATCH_SIZE:=256}" +: "${NUM_THREADS:=16}" +: "${THREAD_BATCH:=32}" : "${CACHE_TYPE_K:=f16}" : "${CACHE_TYPE_V:=f16}" : "${CTX_CHECKPOINTS:=16}" : "${CACHE_RAM:=8192}" -: "${SPEC_DRAFT_N_MAX:=4}" -: "${SPEC_DRAFT_N_MIN:=0}" -: "${SPEC_DRAFT_P_MIN:=0.0}" -: "${SPEC_DRAFT_P_SPLIT:=0.10}" +# Embedded-MTP (qwen35 NextN) qualified profile. The MTP head lives in the main +# GGUF, so no standalone draft file is referenced; the draft runs on ROCm0 with all layers. +: "${MTP_N_MAX:=16}" +: "${MTP_N_MIN:=0}" +: "${MTP_P_MIN:=0.60}" +: "${SPEC_MTP_STRICT_QWEN:=on}" +: "${MTP_BACKEND_SAMPLING:=on}" +: "${SPEC_DRAFT_NGL:=all}" : "${TEMPERATURE:=1.0}" : "${TOP_K:=20}" : "${TOP_P:=0.95}" @@ -63,20 +61,20 @@ fi : "${REASONING_FORMAT:=deepseek}" : "${CHAT_TEMPLATE_KWARGS:={}}" : "${VERBOSITY:=4}" +# Strix Halo unified memory + native gfx1151 detection for this ROCmFPX build. +: "${HSA_OVERRIDE_GFX_VERSION:=11.5.1}" +: "${GGML_HIP_ENABLE_UNIFIED_MEMORY:=1}" MODEL="$MODEL_DIR/$MODEL_FILE" -MTP_DRAFT="$MODEL_DIR/$MTP_DRAFT_FILE" +MMPROJ="$MODEL_DIR/$MMPROJ_FILE" -# Plain dense model: no PromptForge .pfs sidecars, no Output-K8 proxy, no IU4 -# lane. The v3 MTP is driven by the standalone Q4_0 draft head through the -# standard draft-mtp path. Unset any leftover PromptForge env lanes. -unset PROMPTFORGE_SIDECAR PROMPTFORGE_GDN_SIDECAR PROMPTFORGE_MTP_OUTPUT_K8_PROXY || true -unset PROMPTFORGE_MODE PROMPTFORGE_ENABLE_SMALLM_W8 || true -unset PROMPTFORGE_MTP_OUTPUT_K8 PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE || true -unset PROMPTFORGE_IU4_SIDECAR PROMPTFORGE_GDN_IU4_SIDECAR || true -unset PROMPTFORGE_ENABLE_NGRAM_MOD PROMPTFORGE_ENABLE_NGRAM_M65_IU4 || true +# Optional vision projector: load it only if the file exists. +MMPROJ_ARGS=() +if [[ -f "$MMPROJ" ]]; then + MMPROJ_ARGS=(--mmproj "$MMPROJ") +fi -for f in "$MODEL" "$MTP_DRAFT" "$MODEL_DIR/$MMPROJ_FILE" "$CHAT_TEMPLATE_FILE"; do +for f in "$MODEL" "$CHAT_TEMPLATE_FILE"; do if [[ ! -f "$f" ]]; then echo "missing required file: $f" >&2 exit 1 @@ -87,13 +85,22 @@ if [[ ! -x "$LLAMA_BIN" ]]; then exit 1 fi -# ROCm 7.15 SDK libraries must precede any system ROCm in LD_LIBRARY_PATH. -export ROCM_PATH="$ROCM_DEVEL" -export LD_LIBRARY_PATH="${ROCM_LIB}/lib:${ROCM_CORE}/lib:${ROCM_DEVEL}/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" +# Any leftover PromptForge env lanes from the previous (kingjones/v3) runtime +# must stay off; this slot now runs the plain strict-MTP draft-mtp path. +unset PROMPTFORGE_SIDECAR PROMPTFORGE_GDN_SIDECAR PROMPTFORGE_MTP_OUTPUT_K8_PROXY || true +unset PROMPTFORGE_MODE PROMPTFORGE_ENABLE_SMALLM_W8 || true +unset PROMPTFORGE_MTP_OUTPUT_K8 PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE || true +unset PROMPTFORGE_IU4_SIDECAR PROMPTFORGE_GDN_IU4_SIDECAR || true +unset PROMPTFORGE_ENABLE_NGRAM_MOD PROMPTFORGE_ENABLE_NGRAM_M65_IU4 || true +unset LLAMA_MTP_CPU_ARGMAX_FASTPATH LLAMA_MTP_BACKEND_GREEDY || true + +# Strix Halo unified memory + native gfx1151 detection for this ROCmFPX build. +export HSA_OVERRIDE_GFX_VERSION +export GGML_HIP_ENABLE_UNIFIED_MEMORY exec "$LLAMA_BIN" \ --model "$MODEL" \ - --mmproj "$MODEL_DIR/$MMPROJ_FILE" \ + "${MMPROJ_ARGS[@]}" \ --chat-template-file "$CHAT_TEMPLATE_FILE" \ --chat-template-kwargs "$CHAT_TEMPLATE_KWARGS" \ --alias "$ALIAS" \ @@ -101,32 +108,34 @@ exec "$LLAMA_BIN" \ --port "$LISTEN_PORT" \ -dev ROCm0 \ -ngl 999 \ + -np "$PARALLEL" \ + -t "$NUM_THREADS" \ + -tb "$THREAD_BATCH" \ -c "$CTX_SIZE" \ -b "$BATCH_SIZE" \ -ub "$UBATCH_SIZE" \ -fa on \ -ctk "$CACHE_TYPE_K" \ -ctv "$CACHE_TYPE_V" \ - -np "$PARALLEL" \ -ctxcp "$CTX_CHECKPOINTS" \ --cache-ram "$CACHE_RAM" \ --cache-prompt \ --metrics \ --no-mmap \ --spec-type draft-mtp \ - --model-draft "$MTP_DRAFT" \ + ${SPEC_MTP_STRICT_QWEN:+--spec-mtp-strict-qwen} \ --spec-draft-device ROCm0 \ - --spec-draft-ngl 999 \ - --spec-draft-type-k f16 \ - --spec-draft-type-v f16 \ - --spec-draft-n-max "$SPEC_DRAFT_N_MAX" \ - --spec-draft-n-min "$SPEC_DRAFT_N_MIN" \ - --spec-draft-p-min "$SPEC_DRAFT_P_MIN" \ - --spec-draft-p-split "$SPEC_DRAFT_P_SPLIT" \ + --spec-draft-ngl "$SPEC_DRAFT_NGL" \ + --spec-draft-type-k "$CACHE_TYPE_K" \ + --spec-draft-type-v "$CACHE_TYPE_V" \ + --spec-draft-n-max "$MTP_N_MAX" \ + --spec-draft-n-min "$MTP_N_MIN" \ + --spec-draft-p-min "$MTP_P_MIN" \ + ${MTP_BACKEND_SAMPLING:+--spec-draft-backend-sampling} \ --temp "$TEMPERATURE" \ --top-k "$TOP_K" \ --top-p "$TOP_P" \ --reasoning "$REASONING" \ --reasoning-format "$REASONING_FORMAT" \ --reasoning-budget -1 \ - -lv "$VERBOSITY" \ No newline at end of file + -lv "$VERBOSITY" diff --git a/tests/frank_large_config.bats b/tests/frank_large_config.bats new file mode 100644 index 0000000..f085fc9 --- /dev/null +++ b/tests/frank_large_config.bats @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +# Seam: frank-default-large model, runtime, embedded-MTP spec, template. +# Covers the ROCmI4 embedded-MTP deployment contract (charlie12345/ROCmFPX). +# +# load helpers + +load helpers + +ENV_FILE="$REPO_ROOT/scripts/frank-default-large.env" +SERVE_FILE="$REPO_ROOT/scripts/serve-frank-large.sh" +UNIT_FILE="$REPO_ROOT/scripts/frank-default-large.service" +TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja" + +@test "frank-default-large serves the eemin ROCmI4 embedded-MTP GGUF" { + grep -q '^MODEL_FILE=Qwen3.8-27B-Q4_0_ROCMI4\.gguf$' "$ENV_FILE" + grep -q 'Qwen3.8-27B-Q4_0_ROCMI4\.gguf' "$SERVE_FILE" + # The MTP head is embedded in the main GGUF: no separate draft file. + grep -q '^MTP_DRAFT_FILE=$' "$ENV_FILE" + ! grep -q -- '--model-draft' "$SERVE_FILE" +} + +@test "frank-default-large uses charlie12345 ROCmFPX with ROCmI4 W4A4" { + grep -q 'rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server' "$ENV_FILE" + grep -q 'eemin-Qwen3.8-27B-ROCMI4-MTP' "$ENV_FILE" +} + +@test "frank-default-large runs strict-Qwen embedded MTP with backend sampling" { + # embedded MTP: draft on ROCm0, all layers, no separate head + grep -q -- '--spec-type draft-mtp' "$SERVE_FILE" + grep -q -- '--spec-mtp-strict-qwen' "$SERVE_FILE" + grep -q -- '--spec-draft-ngl all' "$SERVE_FILE" + grep -q -- '--spec-draft-backend-sampling' "$SERVE_FILE" + grep -qF -- '--spec-draft-n-max "$MTP_N_MAX"' "$SERVE_FILE" + grep -qF -- '--spec-draft-p-min "$MTP_P_MIN"' "$SERVE_FILE" + # qualified profile tunables are exposed in the env + grep -q '^MTP_N_MAX=16$' "$ENV_FILE" + grep -q '^MTP_P_MIN=0.60$' "$ENV_FILE" + grep -q '^SPEC_MTP_STRICT_QWEN=on$' "$ENV_FILE" + grep -q '^MTP_BACKEND_SAMPLING=on$' "$ENV_FILE" +} + +@test "frank-default-large exposes a swappable ubatch" { + grep -q '^UBATCH_SIZE=256$' "$ENV_FILE" + grep -q ': "${UBATCH_SIZE:=256}"' "$SERVE_FILE" + grep -q -- '-ub "$UBATCH_SIZE"' "$SERVE_FILE" +} + +@test "frank-default-large uses the tracked Qwen-Sharp template on :7770" { + grep -q '^CHAT_TEMPLATE_FILE=' "$ENV_FILE" + grep -q 'chat-template\.jinja' "$SERVE_FILE" + [ -s "$TEMPLATE_FILE" ] + grep -q '^ALIAS=frank-default-large$' "$ENV_FILE" + grep -q '^LISTEN_PORT=7770$' "$ENV_FILE" + grep -q 'frank-default-large' "$UNIT_FILE" +} + +@test "frank-default-large env avoids the HIP TEMP/HOST collision" { + # The 2026-08-13 incident: TEMP/HOST/GPU_MAX_HEAP_SIZE are HIP runtime vars. + ! grep -qE '^(TEMP|HOST|GPU_MAX_HEAP_SIZE)=' "$ENV_FILE" + grep -q '^TEMPERATURE=' "$ENV_FILE" +}