Clarify llama-bench harness scope
This commit is contained in:
Frank
Vendored
+1
-1
@@ -5,4 +5,4 @@
|
||||
"attachmentFolderPath": "_attachments",
|
||||
"useMarkdownLinks": false,
|
||||
"showUnsupportedFiles": true
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -50,7 +50,7 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [kingjones Qwen3.8 Uncensored ROCmFP4 STRIX MTP](https://huggingface.co/kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF) — active frank-default-large model on Dankclaw; Q6 AGENT main model with standalone MTP
|
||||
- [[kingjones-strix-mtp]] — the active Q6 AGENT runbook: serving profile, standalone MTP, reasoning + preserve_thinking, 114 GiB TTM/GTT
|
||||
- [[ubatch-benchmark-large]] — active-model ubatch sweep plan and results
|
||||
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
|
||||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
|
||||
on Dankclaw (retired)
|
||||
|
||||
@@ -12,13 +12,13 @@ timestamp: 2026-09-01
|
||||
|
||||
# frank-default-large ubatch sweep
|
||||
|
||||
This benchmark sweeps the active `frank-default-large` model on Dankclaw using
|
||||
scratch ports and a 32K-context workload. The production service is stopped
|
||||
before the sweep and restarted by the benchmark's exit handler.
|
||||
This benchmark uses the active `frank-default-large` model on Dankclaw with
|
||||
`llama-bench`. It does not use `llama-server`, HTTP requests, or server-log
|
||||
scraping. Stop the production service before running it and start the service
|
||||
again afterward.
|
||||
|
||||
The 2026-09-01 run used `ctx=32768` and `batch=32768`. It completed through
|
||||
`ubatch=3072`; `ubatch=4096` failed during initialization with a ROCm
|
||||
allocation error, so 4096, 5120, and 6144 were not run.
|
||||
The old custom harnesses are retained in `.AGENTTRASH/bench-scripts-20260901/`
|
||||
for auditability only. Their results are invalid and must not be used.
|
||||
|
||||
## Sweep
|
||||
|
||||
@@ -30,32 +30,25 @@ requests by default.
|
||||
Run on Dankclaw from the repository checkout:
|
||||
|
||||
```bash
|
||||
STOPSVC=1 ./scripts/bench-ubatch-rocmi4.sh
|
||||
sudo systemctl stop frank-default-large
|
||||
./scripts/bench-ubatch.sh
|
||||
sudo systemctl start frank-default-large
|
||||
```
|
||||
|
||||
The script uses the active Q6 AGENT model, standalone MTP draft, a fixed
|
||||
32K batch, and the tracked Sharp
|
||||
template, and `--reasoning-format deepseek`. It reports server timing lines and
|
||||
per-run generation throughput. Run only when the production service can be
|
||||
interrupted; the GPU is shared with live workloads.
|
||||
The script uses the active Q6 AGENT model, a fixed 16K prompt workload, and
|
||||
`llama-bench` JSONL output. The initial ubatch-only sweep is:
|
||||
```text
|
||||
256, 512, 768, 1024, 1280, 1536, 1792, 2048, 3072, 4096, 5120, 6144
|
||||
```
|
||||
|
||||
The GPU is shared with live workloads; do not run with the production service
|
||||
up.
|
||||
|
||||
## Results
|
||||
|
||||
Record results below after the sweep, including failures or OOM points.
|
||||
|
||||
| ubatch | cold prefill | generation runs 1/2/3 | notes |
|
||||
| ---: | ---: | ---: | --- |
|
||||
| 256 | 214.73 tok/s | 17.68 / 17.15 / 17.12 tok/s | completed |
|
||||
| 512 | 213.47 tok/s | 17.91 / 17.29 / 17.72 tok/s | completed |
|
||||
| 768 | 215.60 tok/s | 17.96 / 18.17 / 17.98 tok/s | completed |
|
||||
| 1024 | 222.20 tok/s | 18.07 / 18.10 / 18.12 tok/s | completed |
|
||||
| 1280 | 219.13 tok/s | 17.47 / 17.69 / 17.49 tok/s | completed |
|
||||
| 1536 | 220.52 tok/s | 16.85 / 16.86 / 16.86 tok/s | completed |
|
||||
| 2048 | 60.97 tok/s | 6.24 / 4.94 / 11.19 tok/s | severe slowdown |
|
||||
| 3072 | 28.50 tok/s | 1.45 / 5.03 / 4.44 tok/s | severe slowdown |
|
||||
| 4096 | — | — | initialization OOM: rs-cache allocation |
|
||||
| 5120 | — | — | not reached |
|
||||
| 6144 | — | — | not reached |
|
||||
No valid results are recorded here. All results from the retired custom
|
||||
harnesses are explicitly discarded. Record the raw `llama-bench` JSONL output
|
||||
from the new harness here after a clean run.
|
||||
|
||||
## See also
|
||||
|
||||
|
||||
+1
-1
@@ -119,7 +119,7 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
### Benchmarks and Sizing
|
||||
|
||||
- [[frank-default-large-benchmark]] — backend/spec benchmark behind the old head
|
||||
- [[ubatch-benchmark-large]] — active-model ubatch sweep plan and results
|
||||
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
|
||||
- [[frank-default-small-benchmark]] — the small-model benchmark suite
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware notes
|
||||
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Qwen3.6-27B sizing on the iGPU
|
||||
|
||||
@@ -1,77 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Bench the Kairic Edge "fast greedy (default)" profile.
|
||||
# Usage: ./bench-fast-greedy.sh
|
||||
#
|
||||
# Config: reasoning off, argmax fastpath=1, deterministic sampler.
|
||||
# Sends a coding prompt, captures full stream, computes decode tokens and time.
|
||||
set -euo pipefail
|
||||
|
||||
HOST="127.0.0.1"
|
||||
PORT=7770
|
||||
ALIAS="frank-default-large"
|
||||
|
||||
# A coding prompt representative of the card's 164-task suite.
|
||||
PROMPT='### Instruction
|
||||
Write a Python class that implements a priority queue using a binary heap. Include methods for enqueue, dequeue, peek, and is_empty. Add type hints, docstrings, and a usage example. The heap should support arbitrary comparable elements.
|
||||
### Response'
|
||||
|
||||
DECODE_TOKENS=0
|
||||
START_SEC=0
|
||||
END_SEC=0
|
||||
|
||||
echo "=== Fast-greedy benchmark (FASTGREEDY=1) ==="
|
||||
echo "Model: $ALIAS on $HOST:$PORT"
|
||||
echo "Prompt tokens: $(echo "$PROMPT" | wc -c) chars"
|
||||
echo ""
|
||||
|
||||
# Use /v1/completions with streaming, capture tokens+time from usage.
|
||||
RESPONSE=$(curl -sf http://$HOST:$PORT/v1/completions \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "{
|
||||
\"model\": \"$ALIAS\",
|
||||
\"prompt\": $(python3 -c "import json,sys; print(json.dumps(sys.stdin.read()))" <<< "$PROMPT"),
|
||||
\"max_tokens\": 512,
|
||||
\"temperature\": 0,
|
||||
\"top_p\": 1.0,
|
||||
\"top_k\": 0,
|
||||
\"min_p\": 0.0,
|
||||
\"n\": 1,
|
||||
\"stream\": true
|
||||
}" 2>/dev/null)
|
||||
|
||||
# Extract final usage from the last JSON line (finish_reason=stop or length).
|
||||
# Also count output text tokens.
|
||||
LAST_LINE=$(echo "$RESPONSE" | grep 'finish_reason' | tail -1)
|
||||
|
||||
# Parse timing from the last chunk (has full usage)
|
||||
if echo "$LAST_LINE" | grep -q 'timings'; then
|
||||
COMP_TOKENS=$(echo "$LAST_LINE" | python3 -c "
|
||||
import sys, json
|
||||
data = json.loads(sys.stdin.read())
|
||||
print(data.get('usage', {}).get('completion_tokens', 0))
|
||||
print(data.get('timings', {}).get('predicted_per_second', 0))
|
||||
print(data.get('timings', {}).get('prompt_per_second', 0))
|
||||
" 2>/dev/null)
|
||||
echo "$COMP_TOKENS" | {
|
||||
read CT
|
||||
read TPS
|
||||
read PPS
|
||||
echo "Completion tokens: $CT"
|
||||
echo "Predicted (decode) tokens/sec: $TPS"
|
||||
echo "Prompt tokens/sec: $PPS"
|
||||
}
|
||||
else
|
||||
echo "WARNING: No timing data found in response"
|
||||
echo "$LAST_LINE" | head -5
|
||||
fi
|
||||
|
||||
# Count the actual text output
|
||||
OUTPUT_TEXT=$(echo "$RESPONSE" | grep '"text"' | head -1 | sed 's/.*"text":"//;s/","index.*//')
|
||||
OUTPUT_TOKENS=$(echo -n "$OUTPUT_TEXT" | wc -w)
|
||||
echo "Output text (rough token count): ~$OUTPUT_TOKENS words"
|
||||
echo ""
|
||||
|
||||
# Check journal for draft acceptance and final timing
|
||||
echo "=== Draft acceptance & final timing ==="
|
||||
sleep 2
|
||||
ssh -p 2222 dank@dankclaw "journalctl -u frank-default-large --since '1 minute ago' --no-pager 2>/dev/null | grep -E 'draft acceptance|print_timing.*n_decoded|statistics.*draft-mtp' | tail -4" 2>/dev/null || echo "(could not read journal)"
|
||||
@@ -1,153 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# bench-kairic-ab.sh — A/B perf isolation for the Kairic Edge deployment on
|
||||
# Dankclaw. Run this ON DANKCLAW (paths are absolute target paths).
|
||||
#
|
||||
# Configs:
|
||||
# runner pinned runner profile (b2048/ub512, reasoning off, target
|
||||
# greedy argmax fast path ON)
|
||||
# nommap runner + --no-mmap
|
||||
# nofast runner + fast path OFF
|
||||
# prod production profile (b4096/ub4096, --no-mmap, reasoning on/deepseek,
|
||||
# fast path OFF, preserve_thinking)
|
||||
# prod2048 prod with the pinned runner's batch sizes (b2048/ub512)
|
||||
# prodnommap (obsolete: identical to prod since ubatch 4096 + --no-mmap
|
||||
# moved into the production profile)
|
||||
#
|
||||
# Usage: ./scripts/bench-kairic-ab.sh <cfg> [ctx] [prompt_chars]
|
||||
# Example: ./scripts/bench-kairic-ab.sh prod 16384 53000
|
||||
#
|
||||
# The server runs on scratch port 7893. STOP frank-default-large before
|
||||
# benchmarking (see AGENTS.md). The server is killed on exit.
|
||||
set -uo pipefail
|
||||
CFG="${1:?runner|nofast|prod}"
|
||||
CTX="${2:-32768}"
|
||||
PCHARS="${3:-11000}"
|
||||
PORT=7893
|
||||
BIN=/home/dank/ciru-qwen38/ROCmFPX-kairic/build-kairic/bin/llama-server
|
||||
MDIR=/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge
|
||||
SDK=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages
|
||||
LOG=/tmp/kairic-ab-$CFG.log
|
||||
: > "$LOG"
|
||||
|
||||
export ROCM_PATH="$SDK/_rocm_sdk_devel"
|
||||
export LD_LIBRARY_PATH="$(dirname "$BIN"):$SDK/_rocm_sdk_libraries/lib:$SDK/_rocm_sdk_core/lib:$SDK/_rocm_sdk_devel/lib"
|
||||
export HIP_VISIBLE_DEVICES=0
|
||||
export HSA_OVERRIDE_GFX_VERSION=11.5.1
|
||||
export GGML_CUDA_GRAPH_OPT=0
|
||||
export LLAMA_MTP_CPU_ARGMAX_FASTPATH=1
|
||||
export PROMPTFORGE_TARGET_ONLY=0
|
||||
export PROMPTFORGE_MODE=iu4_ffn
|
||||
export PROMPTFORGE_IU4_SIDECAR="$MDIR/Qwen3.8-27B-Kairic-IU4-FFN.pfs"
|
||||
export PROMPTFORGE_SIDECAR="$MDIR/Qwen3.8-27B-Kairic-IU4-FFN.pfs"
|
||||
export PROMPTFORGE_IU4_HADAMARD=1
|
||||
export PROMPTFORGE_IU4_SEGMENTED=0
|
||||
export PROMPTFORGE_ENABLE_FFN_KEEPERS=0
|
||||
export PROMPTFORGE_FFN_KEEPERS=late6
|
||||
export PROMPTFORGE_ENABLE_GDN=1
|
||||
export PROMPTFORGE_ENABLE_GDN_W8=0
|
||||
export PROMPTFORGE_GDN_SIDECAR_OVERRIDE="$MDIR/Qwen3.8-27B-Kairic-IU4-GDN.pfs"
|
||||
export PROMPTFORGE_GDN_SIDECAR="$MDIR/Qwen3.8-27B-Kairic-IU4-GDN.pfs"
|
||||
export PROMPTFORGE_GDN_IU4_HADAMARD=1
|
||||
export PROMPTFORGE_ENABLE_GDN_KEEPERS=0
|
||||
export PROMPTFORGE_ENABLE_GDN_OUTPUT=1
|
||||
export PROMPTFORGE_GDN_OUTPUT_SIDECAR="$MDIR/Qwen3.8-27B-Kairic-IU4-GDN-Output.pfs"
|
||||
export PROMPTFORGE_GDN_OUTPUT_KEEPERS=v3_lateq6
|
||||
export PROMPTFORGE_ENABLE_SMALLM_IU4=1
|
||||
export PROMPTFORGE_ENABLE_SMALLM_GDN_IU4=1
|
||||
export PROMPTFORGE_ENABLE_SMALLM_GDN_OUTPUT_IU4=0
|
||||
export PROMPTFORGE_ENABLE_IU4_DECODE_M2_M5=0
|
||||
export PROMPTFORGE_OUTPUT_K8_STRICT_GREEDY=0
|
||||
|
||||
MMAP=""
|
||||
case "$CFG" in
|
||||
runner) export LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1; B=2048; UB=512; REA=off; RFMT=none; EXTRA="" ;;
|
||||
nommap) export LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1; B=2048; UB=512; REA=off; RFMT=none; EXTRA=""; MMAP="--no-mmap" ;;
|
||||
nofast) export LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0; B=2048; UB=512; REA=off; RFMT=none; EXTRA="" ;;
|
||||
prod) export LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0; B=4096; UB=4096; REA=on; RFMT=deepseek; EXTRA="--chat-template-kwargs {\"preserve_thinking\":\"true\"}"; MMAP="--no-mmap" ;;
|
||||
prod2048) export LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0; B=2048; UB=512; REA=on; RFMT=deepseek; EXTRA="--chat-template-kwargs {\"preserve_thinking\":\"true\"}"; MMAP="--no-mmap" ;;
|
||||
prodnommap) export LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=0; B=4096; UB=2048; REA=on; RFMT=deepseek; EXTRA="--chat-template-kwargs {\"preserve_thinking\":\"true\"}"; MMAP="--no-mmap" ;;
|
||||
esac
|
||||
|
||||
"$BIN" -m "$MDIR/Qwen3.8-27B-IU4-Kairic-Edge.gguf" \
|
||||
--alias bench --host 127.0.0.1 --port "$PORT" --jinja \
|
||||
-dev ROCm0 -ngl 999 -c "$CTX" -b "$B" -ub "$UB" -fa on -ctk f16 -ctv f16 \
|
||||
-t 16 -tb 32 -np 1 -ctxcp 32 --cache-ram 8192 --cache-prompt --cache-idle-slots $MMAP \
|
||||
--metrics --kairic-edge \
|
||||
--spec-type draft-mtp --spec-draft-device ROCm0 --spec-draft-ngl 999 \
|
||||
--spec-draft-type-k f16 --spec-draft-type-v f16 \
|
||||
--spec-draft-n-max 4 --spec-draft-n-min 0 --spec-draft-p-min 0.0 --spec-draft-p-split 0.10 \
|
||||
--spec-draft-backend-sampling \
|
||||
--temp 0 --top-p 1.0 --top-k 0 --min-p 0.0 \
|
||||
--reasoning "$REA" --reasoning-format "$RFMT" --reasoning-budget -1 \
|
||||
$EXTRA >> "$LOG" 2>&1 &
|
||||
SPID=$!
|
||||
trap 'kill $SPID 2>/dev/null; wait $SPID 2>/dev/null' EXIT
|
||||
|
||||
for i in $(seq 1 300); do
|
||||
kill -0 $SPID 2>/dev/null || { echo "SERVER DIED"; tail -20 "$LOG"; exit 2; }
|
||||
curl -sf "http://127.0.0.1:$PORT/health" >/dev/null 2>&1 && break
|
||||
sleep 2
|
||||
done
|
||||
echo "=== CFG=$CFG ready ==="
|
||||
|
||||
python3 - "$PORT" "$CFG" "$PCHARS" <<'PY'
|
||||
import sys, json, time, urllib.request
|
||||
port, cfg, pchars = sys.argv[1], sys.argv[2], int(sys.argv[3])
|
||||
VAR = [
|
||||
"narrative nonfiction blends observation with reflection, ",
|
||||
"and a careful writer lets each clause earn its place ",
|
||||
"in the sentence's rising line, so meaning accumulates slowly. ",
|
||||
"The reader is not hurried; the reader is led. Loose and periodic, ",
|
||||
"the style turns on concrete nouns and verbs that carry weight. ",
|
||||
"A paragraph is not a list but a single turn of thought that ",
|
||||
"opens, develops, and lands, and clarity is a courtesy owed ",
|
||||
"to the willing reader who gives the writer their attention. ",
|
||||
]
|
||||
parts = []
|
||||
t = 0
|
||||
while len("".join(parts)) < pchars:
|
||||
for k in range(3):
|
||||
parts.append(VAR[(t + k) % len(VAR)])
|
||||
parts.append(f" Note {t}. ")
|
||||
t += 1
|
||||
longtext = "".join(parts)
|
||||
prompt = longtext + "\n\nNow write a long, detailed continuation of this essay (at least five paragraphs) about the philosophy of language.\n"
|
||||
body = {
|
||||
"model": "bench",
|
||||
"prompt": prompt,
|
||||
"max_tokens": 512,
|
||||
"temperature": 0,
|
||||
"top_p": 1.0,
|
||||
"top_k": 0,
|
||||
"min_p": 0.0,
|
||||
"stream": False,
|
||||
"cache_prompt": True,
|
||||
}
|
||||
# ignore_eos is rejected by the exact-Q8 greedy argmax fast path (runner cfg).
|
||||
if cfg not in ("runner",):
|
||||
body["ignore_eos"] = True
|
||||
def run(tag):
|
||||
req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/completions",
|
||||
data=json.dumps(body).encode(),
|
||||
headers={"Content-Type": "application/json"})
|
||||
t0 = time.time()
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=3600) as resp:
|
||||
out = json.load(resp)
|
||||
except Exception as e:
|
||||
body_txt = getattr(e, "read", lambda: b"")()
|
||||
print(f"[{tag}] REQUEST FAILED: {e}")
|
||||
print(f"[{tag}] body: {body_txt.decode('utf-8', 'ignore')[:500]}")
|
||||
raise SystemExit(3)
|
||||
wall = time.time() - t0
|
||||
u = out.get("usage", {})
|
||||
tm = out.get("timings", {})
|
||||
pt = u.get("prompt_tokens", 0); ct = u.get("completion_tokens", 0)
|
||||
pps = tm.get("prompt_per_second", 0); tps = tm.get("predicted_per_second", 0)
|
||||
print(f"[{tag}] cfg={cfg} prompt={pt} completion={ct} prefill={pps:.1f} t/s decode={tps:.2f} t/s wall={wall:.1f}s")
|
||||
run("cold")
|
||||
run("warm")
|
||||
PY
|
||||
|
||||
grep -E "print_timing|statistics" "$LOG" | tail -6
|
||||
exit 0
|
||||
@@ -1,191 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# bench-spec-stack.sh — compare MTP-only vs MTP+ngram-mod spec on the v3
|
||||
# CIRU prompt-forge stack (Dankclaw, gfx1151), at 8k / 16k / 64k context.
|
||||
#
|
||||
# Reads authoritative prefill/generation tok/s from llama-server's own
|
||||
# `print_timing` lines (prompt eval time / eval time). Runs one server
|
||||
# instance per (spec, ctx) on a scratch port.
|
||||
#
|
||||
# Usage:
|
||||
# ./scripts/bench-spec-stack.sh <draft-mtp|draft-mtp,ngram-mod> <ctx> [out_tokens]
|
||||
# Example:
|
||||
# ./scripts/bench-spec-stack.sh draft-mtp,ngram-mod 16384 256
|
||||
set -euo pipefail
|
||||
|
||||
SPEC="${1:?usage: bench-spec-stack.sh <spec> <ctx> [out_tokens]}"
|
||||
CTX="${2:?usage: bench-spec-stack.sh <spec> <ctx> [out_tokens]}"
|
||||
OUT_TOKENS="${3:-256}"
|
||||
|
||||
ENV_FILE=/home/dank/serverAppConfigs/scripts/frank-default-large.env
|
||||
source "$ENV_FILE"
|
||||
|
||||
# This harness benchmarks the retired v3 stack. Force the v3 values so the
|
||||
# harness works regardless of the live service env (now Kairic).
|
||||
MODEL_DIR=/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3
|
||||
MODEL_FILE=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3-Q8-FFNEQ-LateQ6.gguf
|
||||
FFN_PFS=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs
|
||||
GDN_PFS=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs
|
||||
OUTPUT_K8_PFS=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-Output-K8.pfs
|
||||
MMPROJ_FILE=Qwen3.8-27B-mmproj-BF16.mmproj
|
||||
|
||||
# Derive the PromptForge sidecar paths from the env base values.
|
||||
export PROMPTFORGE_SIDECAR="$MODEL_DIR/$FFN_PFS"
|
||||
export PROMPTFORGE_GDN_SIDECAR="$MODEL_DIR/$GDN_PFS"
|
||||
export PROMPTFORGE_MTP_OUTPUT_K8_PROXY="$MODEL_DIR/$OUTPUT_K8_PFS"
|
||||
|
||||
MODEL="$MODEL_DIR/$MODEL_FILE"
|
||||
LLAMA_BIN=/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server
|
||||
PORT=7893
|
||||
PARALLEL=1
|
||||
BATCH=4096
|
||||
UBATCH=4096
|
||||
LOG="/tmp/specstack-$(echo "$SPEC" | tr ',' '-')-ctx$CTX.log"
|
||||
: > "$LOG"
|
||||
|
||||
for f in "$MODEL" "$PROMPTFORGE_SIDECAR" "$PROMPTFORGE_GDN_SIDECAR" "$PROMPTFORGE_MTP_OUTPUT_K8_PROXY" "$MODEL_DIR/$MMPROJ_FILE"; do
|
||||
[[ -f "$f" ]] || { echo "missing: $f" >&2; exit 1; }
|
||||
done
|
||||
[[ -x "$LLAMA_BIN" ]] || { echo "missing: $LLAMA_BIN" >&2; exit 1; }
|
||||
|
||||
export PROMPTFORGE_MODE="m2048_fused_tail1476"
|
||||
export GGML_CUDA_GRAPH_OPT=0
|
||||
export PROMPTFORGE_ENABLE_SMALLM_W8=1
|
||||
export PROMPTFORGE_MTP_OUTPUT_K8=1
|
||||
export PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE=0
|
||||
export PROMPTFORGE_MTP_OUTPUT_K8_PROXY
|
||||
unset LLAMA_MTP_CPU_ARGMAX_FASTPATH LLAMA_MTP_BACKEND_GREEDY || true
|
||||
unset PROMPTFORGE_IU4_SIDECAR PROMPTFORGE_GDN_IU4_SIDECAR || true
|
||||
unset PROMPTFORGE_ENABLE_NGRAM_MOD PROMPTFORGE_ENABLE_NGRAM_M65_IU4 || true
|
||||
|
||||
export ROCM_PATH="$ROCM_DEVEL"
|
||||
export LD_LIBRARY_PATH="${ROCM_LIB}/lib:${ROCM_CORE}/lib:${ROCM_DEVEL}/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
|
||||
|
||||
echo "=== spec-stack bench: spec=$SPEC ctx=$CTX model=$MODEL_FILE ==="
|
||||
echo "=== env: OUTPUT_K8=$PROMPTFORGE_MTP_OUTPUT_K8 smallm=$PROMPTFORGE_ENABLE_SMALLM_W8 ==="
|
||||
|
||||
# ngram-mod tuning matches the retired fallback line: n_min 48 / n_max 64 / n_match 24
|
||||
"$LLAMA_BIN" \
|
||||
--model "$MODEL" \
|
||||
--mmproj "$MODEL_DIR/$MMPROJ_FILE" \
|
||||
--alias bench \
|
||||
--host 127.0.0.1 \
|
||||
--port "$PORT" \
|
||||
-dev ROCm0 \
|
||||
-ngl 999 \
|
||||
-c "$CTX" \
|
||||
-b "$BATCH" \
|
||||
-ub "$UBATCH" \
|
||||
-fa on \
|
||||
-ctk "$CACHE_TYPE_K" \
|
||||
-ctv "$CACHE_TYPE_V" \
|
||||
-np "$PARALLEL" \
|
||||
-ctxcp "$CTX_CHECKPOINTS" \
|
||||
--cache-ram "$CACHE_RAM" \
|
||||
--cache-prompt \
|
||||
--no-mmap \
|
||||
--spec-type "$SPEC" \
|
||||
--spec-draft-device ROCm0 \
|
||||
--spec-draft-ngl 999 \
|
||||
--spec-draft-type-k f16 \
|
||||
--spec-draft-type-v f16 \
|
||||
--spec-draft-n-max 4 \
|
||||
--spec-draft-n-min 0 \
|
||||
--spec-draft-p-min 0.0 \
|
||||
--spec-draft-p-split 0.10 \
|
||||
--spec-draft-backend-sampling \
|
||||
--spec-ngram-mod-n-min 48 \
|
||||
--spec-ngram-mod-n-max 64 \
|
||||
--spec-ngram-mod-n-match 24 \
|
||||
--temp 0.7 \
|
||||
--top-p 0.95 \
|
||||
-lv 4 \
|
||||
>> "$LOG" 2>&1 &
|
||||
SERVER_PID=$!
|
||||
trap 'kill $SERVER_PID 2>/dev/null; wait $SERVER_PID 2>/dev/null' EXIT
|
||||
|
||||
echo "server pid $SERVER_PID on 127.0.0.1:$PORT"
|
||||
echo -n "waiting for server"
|
||||
ready=0
|
||||
crashed=0
|
||||
for i in $(seq 1 900); do
|
||||
if ! kill -0 "$SERVER_PID" 2>/dev/null; then
|
||||
crashed=1
|
||||
break
|
||||
fi
|
||||
if curl -sf "http://127.0.0.1:$PORT/health" >/dev/null 2>&1; then
|
||||
echo " [ready]"
|
||||
ready=1
|
||||
break
|
||||
fi
|
||||
[ $((i % 10)) = 0 ] && echo -n .
|
||||
sleep 2
|
||||
done
|
||||
if [ "$crashed" = "1" ]; then
|
||||
echo " BACKEND_CRASHED (server died before ready)"
|
||||
tail -30 "$LOG"
|
||||
exit 2
|
||||
fi
|
||||
[ "$ready" = "1" ] || { echo " FAIL (timeout)"; tail -25 "$LOG"; exit 1; }
|
||||
|
||||
# Print the implementations that initialized (feasibility evidence)
|
||||
grep -E "adding speculative implementation|ngram_mod|draft-mtp" "$LOG" | head -6
|
||||
|
||||
python3 - "$SPEC" "$CTX" "$PORT" "$OUT_TOKENS" "$LOG" <<'PY'
|
||||
import sys, time, json, urllib.request
|
||||
spec, ctx, port, out_tokens, log = sys.argv[1], int(sys.argv[2]), sys.argv[3], int(sys.argv[4]), sys.argv[5]
|
||||
|
||||
# Flowing prose (varied) at ~75% of ctx so MTP acceptance is realistic.
|
||||
VAR = (
|
||||
"narrative nonfiction blends observation with reflection, ",
|
||||
"and a careful writer lets each clause earn its place ",
|
||||
"in the sentence's rising line, so meaning accumulates slowly. ",
|
||||
"The reader is not hurried; the reader is led. Loose and periodic, ",
|
||||
"the style turns on concrete nouns and verbs that carry weight, ",
|
||||
"and it resists the abstract noun that hides a vague claim. ",
|
||||
"A paragraph is not a list. It is a single turn of thought that ",
|
||||
"opens, develops, and lands. This is the craft of prose. Choose, ",
|
||||
"arrange, and above all omit. Omission is the discipline that ",
|
||||
"keeps the whole clear, and clarity is the courtesy we owe the ",
|
||||
"willing reader."
|
||||
)
|
||||
n = len(VAR)
|
||||
parts = []
|
||||
t = 0
|
||||
while t * 76 < (ctx * 3) // 4:
|
||||
for k in range(5):
|
||||
parts.append(VAR[(t + k) % n])
|
||||
parts.append(f" This is cued by note {t}. ")
|
||||
t += 1
|
||||
prompt = "".join(parts)
|
||||
prompt += ("\n\nNow, in a detailed and flowing paragraph, continue this essay about "
|
||||
"the philosophy of language and its influence on modern computation. ")
|
||||
|
||||
body = json.dumps({
|
||||
"prompt": prompt,
|
||||
"max_tokens": out_tokens,
|
||||
"temperature": 0.0,
|
||||
"stream": False,
|
||||
}).encode()
|
||||
req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/completions",
|
||||
data=body, headers={"Content-Type": "application/json"})
|
||||
t0 = time.time()
|
||||
with urllib.request.urlopen(req, timeout=7200) as resp:
|
||||
out = json.load(resp)
|
||||
dt = time.time() - t0
|
||||
pt = out.get("usage", {}).get("prompt_tokens", 0)
|
||||
ct = out.get("usage", {}).get("completion_tokens", 0)
|
||||
print(f"request done: prompt_tokens={pt} completion_tokens={ct} wall={dt:.1f}s", flush=True)
|
||||
|
||||
lines = open(log, encoding="utf-8", errors="ignore").read().splitlines()
|
||||
prompt_evals = [l for l in lines if "prompt eval time" in l]
|
||||
evals = [l for l in lines if "eval time" in l and "prompt eval" not in l]
|
||||
|
||||
def grab(lines, label):
|
||||
if lines:
|
||||
print(label + ": " + lines[-1].strip())
|
||||
|
||||
print(f"== RESULT spec={spec} ctx={ctx} ==")
|
||||
grab(prompt_evals, "PREFILL")
|
||||
grab(evals, "GENERATION")
|
||||
print(f"prompt_tokens={pt} completion_tokens={ct} wall={dt:.1f}s")
|
||||
PY
|
||||
@@ -1,245 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# bench-ubatch-rocmi4.sh — sweep batch size with a fixed ubatch at 16k context
|
||||
# on the frank-default-large ROCmFPX standalone-MTP stack (dankclaw, gfx1151).
|
||||
#
|
||||
# Boots one llama-server instance per batch on a scratch port with a fixed
|
||||
# ubatch and -c 16384, runs a standardized ~75%-of-ctx prose workload, and reads
|
||||
# authoritative prefill/generation tok/s from the server's own print_timing
|
||||
# lines. Steps from UBATCH_MIN to UBATCH_MAX inclusive by UBATCH_STEP.
|
||||
#
|
||||
# Usage:
|
||||
# ./scripts/bench-ubatch-rocmi4.sh
|
||||
# UBATCH_MIN=512 UBATCH_MAX=4096 UBATCH_STEP=512 CTX=8192 RUNS=3 \
|
||||
# ./scripts/bench-ubatch-rocmi4.sh
|
||||
#
|
||||
# Benchmarking policy: stop the production service first so the GPU is not
|
||||
# starved. With STOPSVC=1 the script restarts frank-default-large on exit.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ENV_FILE="${FRANK_LARGE_ENV_FILE:-$SCRIPT_DIR/frank-default-large.env}"
|
||||
if [[ -f "$ENV_FILE" ]]; then
|
||||
set -a
|
||||
# shellcheck disable=SC1090
|
||||
source "$ENV_FILE"
|
||||
set +a
|
||||
fi
|
||||
|
||||
# --- sweep parameters -------------------------------------------------------
|
||||
: "${UBATCH_FIXED:=512}"
|
||||
: "${BATCH_VALUES:=512 1024 1536 2048}"
|
||||
: "${CTX:=16384}" # 16k context
|
||||
: "${OUT_TOKENS:=256}"
|
||||
: "${RUNS:=3}" # decode runs per ubatch
|
||||
: "${PORT_BASE:=7894}" # scratch ports 7894..
|
||||
: "${STOPSVC:=1}" # stop/restart the live unit for a clean GPU
|
||||
|
||||
# --- resolved paths (fall back to serve-script defaults) --------------------
|
||||
: "${LLAMA_BIN:=/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server}"
|
||||
: "${MODEL_DIR:=/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP}"
|
||||
: "${MODEL_FILE:=Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf}"
|
||||
: "${MTP_DRAFT_FILE:=$MODEL_DIR/mtp-Qwen3.8-27B-Q4_0.gguf}"
|
||||
: "${MMPROJ_FILE:=mmproj-Qwen3.8-27B-Q8_0.gguf}"
|
||||
: "${CHAT_TEMPLATE_FILE:=$SCRIPT_DIR/chat-template.jinja}"
|
||||
: "${ALIAS:=frank-default-large}"
|
||||
: "${HSA_OVERRIDE_GFX_VERSION:=11.5.1}"
|
||||
: "${GGML_HIP_ENABLE_UNIFIED_MEMORY:=1}"
|
||||
: "${MTP_N_MAX:=16}"
|
||||
: "${MTP_P_MIN:=0.60}"
|
||||
|
||||
MODEL="$MODEL_DIR/$MODEL_FILE"
|
||||
MMPROJ="$MODEL_DIR/$MMPROJ_FILE"
|
||||
|
||||
for f in "$MODEL" "$CHAT_TEMPLATE_FILE"; do
|
||||
if [[ ! -f "$f" ]]; then
|
||||
echo "missing required file: $f" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
if [[ ! -x "$LLAMA_BIN" ]]; then
|
||||
echo "missing llama-server binary: $LLAMA_BIN" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
MMPROJ_ARGS=()
|
||||
if [[ -f "$MMPROJ" ]]; then
|
||||
MMPROJ_ARGS=(--mmproj "$MMPROJ")
|
||||
fi
|
||||
|
||||
STopped=0
|
||||
SCRATCH_PID=""
|
||||
cleanup() {
|
||||
if [[ -n "$SCRATCH_PID" ]] && kill -0 "$SCRATCH_PID" 2>/dev/null; then
|
||||
echo "=== stopping scratch llama-server (pid $SCRATCH_PID) ==="
|
||||
kill "$SCRATCH_PID" 2>/dev/null || true
|
||||
wait "$SCRATCH_PID" 2>/dev/null || true
|
||||
SCRATCH_PID=""
|
||||
fi
|
||||
if [ "$STopped" = 1 ]; then
|
||||
echo "=== restarting frank-default-large ==="
|
||||
sudo systemctl restart frank-default-large || true
|
||||
STopped=0
|
||||
fi
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
stop_live_service() {
|
||||
if [ "$STOPSVC" = 1 ]; then
|
||||
echo "=== stopping frank-default-large for a clean GPU ==="
|
||||
sudo systemctl stop frank-default-large || true
|
||||
STopped=1
|
||||
fi
|
||||
}
|
||||
|
||||
# wait_health <pid> <port> — poll /health until ready or the server dies.
|
||||
wait_health() {
|
||||
local pid="$1" port="$2" i ready=0 crashed=0
|
||||
for i in $(seq 1 900); do
|
||||
if ! kill -0 "$pid" 2>/dev/null; then crashed=1; break; fi
|
||||
if curl -sf "http://127.0.0.1:${port}/health" >/dev/null 2>&1; then
|
||||
ready=1; break
|
||||
fi
|
||||
[ $((i % 10)) = 0 ] && echo -n .
|
||||
sleep 2
|
||||
done
|
||||
[ "$crashed" = 1 ] && { echo " BACKEND_CRASHED (before ready)"; return 2; }
|
||||
[ "$ready" = 1 ] || { echo " FAIL (timeout)"; return 1; }
|
||||
echo " [ready]"
|
||||
return 0
|
||||
}
|
||||
|
||||
# run_one <batch> <port> <out_tokens> <runs> — boot one server instance, run
|
||||
# the workload, parse print_timing, print a RESULT line.
|
||||
run_one() {
|
||||
local batch="$1" port="$2" out_tokens="$3" runs="$4"
|
||||
local ubatch="$UBATCH_FIXED"
|
||||
local log="/tmp/ubatch-${ubatch}-batch-${batch}-p${port}.log"
|
||||
: > "$log"
|
||||
|
||||
echo "=== ubatch=${ubatch} ctx=${CTX} batch=${batch} port=${port} runs=${runs} ==="
|
||||
"$LLAMA_BIN" \
|
||||
--model "$MODEL" \
|
||||
"${MMPROJ_ARGS[@]}" \
|
||||
--chat-template-file "$CHAT_TEMPLATE_FILE" \
|
||||
--chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
|
||||
--alias bench-ubatch \
|
||||
--host 127.0.0.1 --port "$port" \
|
||||
-dev ROCm0 -ngl 999 -np 1 \
|
||||
-t 16 -tb 32 \
|
||||
-c "$CTX" -b "$batch" -ub "$ubatch" \
|
||||
-fa on -ctk f16 -ctv f16 -ctxcp 16 \
|
||||
--cache-ram 8192 --cache-prompt --metrics --no-mmap \
|
||||
--spec-type draft-mtp --model-draft "$MTP_DRAFT_FILE" \
|
||||
--spec-draft-device ROCm0 --spec-draft-ngl 999 \
|
||||
--spec-draft-type-k f16 --spec-draft-type-v f16 \
|
||||
--spec-draft-n-max "$MTP_N_MAX" --spec-draft-n-min 0 \
|
||||
--spec-draft-p-min "$MTP_P_MIN" \
|
||||
--spec-draft-backend-sampling \
|
||||
--temp 0.7 --top-p 0.95 \
|
||||
--reasoning "$REASONING" --reasoning-format "$REASONING_FORMAT" --reasoning-budget -1 \
|
||||
-lv 4 >>"$log" 2>&1 &
|
||||
local pid=$!
|
||||
SCRATCH_PID="$pid"
|
||||
if ! wait_health "$pid" "$port"; then
|
||||
tail -30 "$log"
|
||||
kill "$pid" 2>/dev/null || true
|
||||
wait "$pid" 2>/dev/null || true
|
||||
SCRATCH_PID=""
|
||||
return 2
|
||||
fi
|
||||
|
||||
# one python pass per run; reuse the same boot across runs
|
||||
local r
|
||||
for r in $(seq 1 "$runs"); do
|
||||
echo "--- run ${r}/${runs} ---"
|
||||
if ! python3 - "$ubatch" "$port" "$out_tokens" "$CTX" "$log" "$r" <<'PY'
|
||||
import sys, time, json, urllib.request
|
||||
ubatch, port, out_tokens, ctx, log, run_number = sys.argv[1], sys.argv[2], int(sys.argv[3]), int(sys.argv[4]), sys.argv[5], sys.argv[6]
|
||||
|
||||
# Flowing prose at ~75% of ctx so MTP acceptance is realistic.
|
||||
VAR = (
|
||||
"narrative nonfiction blends observation with reflection, ",
|
||||
"and a careful writer lets each clause earn its place ",
|
||||
"in the sentence's rising line, so meaning accumulates slowly. ",
|
||||
"The reader is not hurried; the reader is led. Loose and periodic, ",
|
||||
"the style turns on concrete nouns and verbs that carry weight, ",
|
||||
"and it resists the abstract noun that hides a vague claim. ",
|
||||
"A paragraph is not a list. It is a single turn of thought that ",
|
||||
"opens, develops, and lands. This is the craft of prose. Choose, ",
|
||||
"arrange, and above all omit. Omission is the discipline that ",
|
||||
"keeps the whole clear, and clarity is the courtesy we owe the ",
|
||||
"willing reader."
|
||||
)
|
||||
n = len(VAR)
|
||||
parts = []
|
||||
t = 0
|
||||
while t * 76 < (ctx * 3) // 4:
|
||||
for k in range(5):
|
||||
parts.append(VAR[(t + k) % n])
|
||||
parts.append(f" This is cued by note {t}. ")
|
||||
t += 1
|
||||
prompt = "".join(parts)
|
||||
prompt += ("\n\nNow, in a detailed and flowing paragraph, continue this essay about "
|
||||
"the philosophy of language and its influence on modern computation. ")
|
||||
|
||||
body = json.dumps({
|
||||
"prompt": prompt, "max_tokens": out_tokens,
|
||||
"temperature": 0.0, "stream": False,
|
||||
}).encode()
|
||||
req = urllib.request.Request(f"http://127.0.0.1:{port}/v1/completions",
|
||||
data=body, headers={"Content-Type": "application/json"})
|
||||
t0 = time.time()
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=7200) as resp:
|
||||
out = json.load(resp)
|
||||
except Exception:
|
||||
print("request failed; preserving benchmark cleanup", file=sys.stderr)
|
||||
raise
|
||||
dt = time.time() - t0
|
||||
pt = out.get("usage", {}).get("prompt_tokens", 0)
|
||||
ct = out.get("usage", {}).get("completion_tokens", 0)
|
||||
|
||||
lines = open(log, encoding="utf-8", errors="ignore").read().splitlines()
|
||||
prefill = [l for l in lines if "prompt eval time" in l]
|
||||
evals = [l for l in lines if "eval time" in l and "prompt eval" not in l]
|
||||
|
||||
def grab(arr):
|
||||
return arr[-1].strip() if arr else "n/a"
|
||||
|
||||
g_tps = ""
|
||||
if evals and ct:
|
||||
# eval timing line: "eval time = .. ms / N tokens (... tokens per second)"
|
||||
import re
|
||||
match = re.search(r"eval time =\s*([0-9.]+) ms", evals[-1])
|
||||
if match:
|
||||
ms = float(match.group(1))
|
||||
g_tps = f"{ct / (ms / 1000):.2f}"
|
||||
print(f"== RESULT ubatch={ubatch} run={run_number} "
|
||||
f"PREFILL={grab(prefill)} GEN_TPS={g_tps} "
|
||||
f"pt={pt} ct={ct} wall={dt:.1f}s")
|
||||
PY
|
||||
then
|
||||
kill "$pid" 2>/dev/null || true
|
||||
wait "$pid" 2>/dev/null || true
|
||||
SCRATCH_PID=""
|
||||
return 2
|
||||
fi
|
||||
done
|
||||
kill "$pid" 2>/dev/null || true
|
||||
wait "$pid" 2>/dev/null || true
|
||||
SCRATCH_PID=""
|
||||
}
|
||||
|
||||
main() {
|
||||
stop_live_service
|
||||
local idx=0 ubatch port
|
||||
printf "%-10s %-14s %-10s\n" "UBATCH" "PREFILL" "GEN_TPS"
|
||||
for batch in $BATCH_VALUES; do
|
||||
port=$((PORT_BASE + idx))
|
||||
run_one "$batch" "$port" "$OUT_TOKENS" "$RUNS"
|
||||
idx=$((idx + 1))
|
||||
done
|
||||
echo "=== sweep complete ==="
|
||||
}
|
||||
|
||||
main "$@"
|
||||
Executable
+45
@@ -0,0 +1,45 @@
|
||||
#!/usr/bin/env bash
|
||||
# Benchmark ubatch with llama-bench for frank-default-large.
|
||||
#
|
||||
# This is intentionally a ubatch-only benchmark. It uses one llama-bench
|
||||
# process, a fixed 16K prompt workload, and a fixed batch size large enough to
|
||||
# hold that workload. No server process, HTTP requests, or log scraping are used.
|
||||
#
|
||||
# Run on Dankclaw after stopping frank-default-large:
|
||||
# sudo systemctl stop frank-default-large
|
||||
# ./scripts/bench-ubatch.sh
|
||||
# sudo systemctl start frank-default-large
|
||||
set -euo pipefail
|
||||
|
||||
: "${LLAMA_BENCH:=/home/dank/GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-bench}"
|
||||
: "${MODEL:=/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP/Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf}"
|
||||
: "${UBATCH_VALUES:=256,512,768,1024,1280,1536,1792,2048,3072,4096,5120,6144}"
|
||||
: "${CONTEXT:=16384}"
|
||||
: "${GEN_TOKENS:=128}"
|
||||
: "${REPETITIONS:=3}"
|
||||
: "${DEVICE:=ROCm0}"
|
||||
: "${OUTPUT:=jsonl}"
|
||||
|
||||
if [[ ! -x "$LLAMA_BENCH" ]]; then
|
||||
printf 'missing executable llama-bench: %s\n' "$LLAMA_BENCH" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ ! -r "$MODEL" ]]; then
|
||||
printf 'missing or unreadable model: %s\n' "$MODEL" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exec "$LLAMA_BENCH" \
|
||||
--model "$MODEL" \
|
||||
--n-prompt "$CONTEXT" \
|
||||
--n-gen "$GEN_TOKENS" \
|
||||
--batch-size "$CONTEXT" \
|
||||
--ubatch-size "$UBATCH_VALUES" \
|
||||
--device "$DEVICE" \
|
||||
--n-gpu-layers 999 \
|
||||
--cache-type-k f16 \
|
||||
--cache-type-v f16 \
|
||||
--flash-attn on \
|
||||
--mmap 0 \
|
||||
--repetitions "$REPETITIONS" \
|
||||
--output "$OUTPUT"
|
||||
@@ -63,28 +63,16 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
|
||||
grep -q -- '-ub "$UBATCH_SIZE"' "$SERVE_FILE"
|
||||
}
|
||||
|
||||
@test "large ubatch bench defaults to the active model and requested sweep" {
|
||||
local bench="$REPO_ROOT/scripts/bench-ubatch-rocmi4.sh"
|
||||
grep -q ': "${UBATCH_FIXED:=512}"' "$bench"
|
||||
grep -q 'BATCH_VALUES.*512 1024 1536 2048' "$bench"
|
||||
grep -q ': "${STOPSVC:=1}"' "$bench"
|
||||
grep -q ': "${CTX:=16384}"' "$bench"
|
||||
grep -q 'MODEL_FILE' "$bench"
|
||||
grep -q 'REASONING_FORMAT' "$bench"
|
||||
}
|
||||
|
||||
@test "large ubatch bench cleans up scratch server on request failure" {
|
||||
local bench="$REPO_ROOT/scripts/bench-ubatch-rocmi4.sh"
|
||||
grep -q 'SCRATCH_PID=' "$bench"
|
||||
grep -q 'kill.*SCRATCH_PID' "$bench"
|
||||
grep -qF 'eval time =\s' "$bench"
|
||||
}
|
||||
|
||||
@test "large ubatch bench supports fixed ubatch batch sweep" {
|
||||
local bench="$REPO_ROOT/scripts/bench-ubatch-rocmi4.sh"
|
||||
grep -q ': "${UBATCH_FIXED:=512}"' "$bench"
|
||||
grep -q 'BATCH_VALUES' "$bench"
|
||||
grep -q -- '-b "$batch" -ub "$ubatch"' "$bench"
|
||||
@test "llama-bench ubatch harness uses the active model" {
|
||||
local bench="$REPO_ROOT/scripts/bench-ubatch.sh"
|
||||
[ -x "$bench" ]
|
||||
grep -q 'llama-bench' "$bench"
|
||||
grep -q 'Qwen3\.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT\.gguf' "$bench"
|
||||
grep -q -- '--n-prompt "$CONTEXT"' "$bench"
|
||||
grep -q -- '--ubatch-size "$UBATCH_VALUES"' "$bench"
|
||||
grep -q -- '--batch-size "$CONTEXT"' "$bench"
|
||||
grep -q -- '--mmap 0' "$bench"
|
||||
! grep -qE '(^|[[:space:]])llama-server|(^|[[:space:]])curl|print_timing' "$bench"
|
||||
}
|
||||
|
||||
@test "frank-default-large uses the tracked Qwen-Sharp template on :7770" {
|
||||
|
||||
Reference in New Issue
Block a user