frank-default-large: threads-batch=16 confirmed optimal + decode benchmark results

Decode sweep (TB=8,16,32, 8K+ prompt, 256 gen tokens):
  TB=8:  3.3 tok/s (76.6s)
  TB=16: 3.4 tok/s (74.8s)
  TB=32: 3.4 tok/s (75.0s)

Decode is flat; TB=16 wins on prefill (5.1% faster) with zero decode tradeoff.

Also:
- Fix :: syntax error in serve-frank-large.sh
- Document systemctl disable destroys unit file; use mask/unmask instead
This commit is contained in:
Frank
2026-08-27 13:20:52 -07:00
parent 26173df1fd
commit cadc7f2768
4 changed files with 50 additions and 6 deletions
+12 -3
View File
@@ -227,14 +227,23 @@ ALWAYS bench using llama-bench for llama.cpp or vllm bench for vllm runtimes.
DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE OFFICIAL BENCHES. ASK THE USER BEFORE DEVIATING FROM THIS DIRECTIVE.
1. Stop the service before any benchmark:
`sudo systemctl stop frank-default-small` (Dankserver) or
`sudo systemctl stop frank-default-large` (Dankclaw).
`sudo systemctl stop frank-default-large` (Dankclaw) or
`sudo systemctl stop frank-default-small` (Dankserver).
Then mask it to prevent the self-healing unit from restarting:
`sudo systemctl mask frank-default-large` (Dankclaw).
**CRITICAL WARNING:** Do NOT use `systemctl disable` on these service units.
On these hosts, `systemctl disable` doesn't just remove symlinks — it **deletes
the unit file itself** from `/etc/systemd/system/`. You'd have to manually
copy the unit file back from this repo, run `daemon-reload`, then enable and
start. Mask (`systemctl mask`) creates a symlink to `/dev/null` instead, which
is safe and reversible.
2. Run the bench (`llama-bench` for llama.cpp / vllm bench for vLLM).
Scratch ports are not used for tests or benches on the LLM hosts — a
test process cannot share the single GPU with the running service anyway
(the model + KV fill the card), so changes are validated directly against
the service itself.
3. Restart the service when the bench completes: `sudo systemctl start <svc>`.
3. Restart the service when the bench completes: unmask, then start:
`sudo systemctl unmask frank-default-large && sudo systemctl start frank-default-large` (Dankclaw).
4. Verify health before reporting:
`curl http://192.168.0.235:7770/health` or
`curl http://192.168.0.143:7770/health`.