 Frank
|
fc75ca6ed8
|
frank-default-large: add MTP draft head (EasiiX Q8_0, draft-mtp+ngram-mod)
Add speculative decoding via EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF
Q8_0 sidecar (4.1 GB, qwen4exp architecture matching the main model).
- serve-frank-large.sh: SPEC_ARGS array, draft model existence check,
spec defaults (n_max 4, p_min 0.75, ngl 999, ROCm0)
- frank-default-large.env: MTP tunables section, fix 'no MTP' comment
- kingjones-strix-rocmfp4.md: MTP Draft Head section, spec flags in
serving profile, EasiiX + PR #27739 citations
Set SPEC_TYPE=none in the env to disable speculative decoding.
|
2026-08-27 23:52:42 -07:00 |
|
|
Frank
|
cadc7f2768
|
frank-default-large: threads-batch=16 confirmed optimal + decode benchmark results
Decode sweep (TB=8,16,32, 8K+ prompt, 256 gen tokens):
TB=8: 3.3 tok/s (76.6s)
TB=16: 3.4 tok/s (74.8s)
TB=32: 3.4 tok/s (75.0s)
Decode is flat; TB=16 wins on prefill (5.1% faster) with zero decode tradeoff.
Also:
- Fix :: syntax error in serve-frank-large.sh
- Document systemctl disable destroys unit file; use mask/unmask instead
|
2026-08-27 13:20:52 -07:00 |
|