frank-default-large: threads-batch=16 confirmed optimal + decode benchmark results
Decode sweep (TB=8,16,32, 8K+ prompt, 256 gen tokens): TB=8: 3.3 tok/s (76.6s) TB=16: 3.4 tok/s (74.8s) TB=32: 3.4 tok/s (75.0s) Decode is flat; TB=16 wins on prefill (5.1% faster) with zero decode tradeoff. Also: - Fix :: syntax error in serve-frank-large.sh - Document systemctl disable destroys unit file; use mask/unmask instead
This commit is contained in:
@@ -227,14 +227,23 @@ ALWAYS bench using llama-bench for llama.cpp or vllm bench for vllm runtimes.
|
||||
DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE OFFICIAL BENCHES. ASK THE USER BEFORE DEVIATING FROM THIS DIRECTIVE.
|
||||
|
||||
1. Stop the service before any benchmark:
|
||||
`sudo systemctl stop frank-default-small` (Dankserver) or
|
||||
`sudo systemctl stop frank-default-large` (Dankclaw).
|
||||
`sudo systemctl stop frank-default-large` (Dankclaw) or
|
||||
`sudo systemctl stop frank-default-small` (Dankserver).
|
||||
Then mask it to prevent the self-healing unit from restarting:
|
||||
`sudo systemctl mask frank-default-large` (Dankclaw).
|
||||
**CRITICAL WARNING:** Do NOT use `systemctl disable` on these service units.
|
||||
On these hosts, `systemctl disable` doesn't just remove symlinks — it **deletes
|
||||
the unit file itself** from `/etc/systemd/system/`. You'd have to manually
|
||||
copy the unit file back from this repo, run `daemon-reload`, then enable and
|
||||
start. Mask (`systemctl mask`) creates a symlink to `/dev/null` instead, which
|
||||
is safe and reversible.
|
||||
2. Run the bench (`llama-bench` for llama.cpp / vllm bench for vLLM).
|
||||
Scratch ports are not used for tests or benches on the LLM hosts — a
|
||||
test process cannot share the single GPU with the running service anyway
|
||||
(the model + KV fill the card), so changes are validated directly against
|
||||
the service itself.
|
||||
3. Restart the service when the bench completes: `sudo systemctl start <svc>`.
|
||||
3. Restart the service when the bench completes: unmask, then start:
|
||||
`sudo systemctl unmask frank-default-large && sudo systemctl start frank-default-large` (Dankclaw).
|
||||
4. Verify health before reporting:
|
||||
`curl http://192.168.0.235:7770/health` or
|
||||
`curl http://192.168.0.143:7770/health`.
|
||||
|
||||
Reference in New Issue
Block a user