frank-default-small: PARALLEL=3 160k, KV K q8_0 / V q5_0 + Hadamard
Reconfigure to 3 parallel 160000-ctx slots (was 1 slot). KV cache is now asymmetric: K at q8_0, V at q5_0, both with Hadamard. The 3 slots triple the KV, so expert blocks 27-34 are pinned back to CPU (freed ~8.7 GiB) leaving output-adjacent 35-40 on CUDA. Q8_0 model, split-mode none, spec disabled (ik limits spec to 1 slot). AGENTS.md: drop the scratch-port rule (test/bench processes cannot share the single GPU with the running service anyway). Env, serve script, bats, docs updated.
This commit is contained in:
1 parent
8c41c821c4
commit
04a8fb7806
5 files changed
+84
-68
No files matched your search
@@ -189,7 +189,11 @@ DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE
|
||||
1. Stop the service before any benchmark:
|
||||
`sudo systemctl stop frank-default-small` (Dankserver) or
|
||||
`sudo systemctl stop frank-default-large` (Dankclaw).
|
||||
2. Run the bench on a scratch port (never the LLM service port 7770).
|
||||
2. Run the bench (`llama-bench` for llama.cpp / vllm bench for vLLM).
|
||||
Scratch ports are not used for tests or benches on the LLM hosts — a
|
||||
test process cannot share the single GPU with the running service anyway
|
||||
(the model + KV fill the card), so changes are validated directly against
|
||||
the service itself.
|
||||
3. Restart the service when the bench completes: `sudo systemctl start <svc>`.
|
||||
4. Verify health before reporting:
|
||||
`curl http://192.168.0.235:7770/health` or
|
||||
@@ -406,7 +410,7 @@ two hosts, one Caddy, and one Gitea. One repo keeps them in one history.
|
||||
| Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before making changes. |
|
||||
| Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. |
|
||||
| Deploying a committed change | **Run `scripts/deploy.sh <target>`** — it pushes, pulls with fast-forward-only, plans changed paths, dispatches selected services, and checks their health. See [docs/LLM/deployment.md](docs/LLM/deployment.md). |
|
||||
| Benchmarking | **Stop the service first** — scratch port, restart, verify health. See the Benchmarking Policy above. |
|
||||
| Benchmarking | **Stop the service first**, restart, verify health. See the Benchmarking Policy above. |
|
||||
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
|
||||
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
|
||||
| Real secrets | **Never commit** — use host `.env` files. `.env.example` holds placeholders only. `FRANK_SUDO_PW` covers headless sudo. |
|
||||
Reference in new issue
Block a user