frank-default-small: PARALLEL=3 160k, KV K q8_0 / V q5_0 + Hadamard

Reconfigure to 3 parallel 160000-ctx slots (was 1 slot). KV cache is
now asymmetric: K at q8_0, V at q5_0, both with Hadamard. The 3 slots
triple the KV, so expert blocks 27-34 are pinned back to CPU (freed
~8.7 GiB) leaving output-adjacent 35-40 on CUDA. Q8_0 model,
split-mode none, spec disabled (ik limits spec to 1 slot). AGENTS.md:
drop the scratch-port rule (test/bench processes cannot share the
single GPU with the running service anyway). Env, serve script, bats,
docs updated.
This commit is contained in:
Frank committed 2026-08-26 05:43:35 -07:00
1 parent 8c41c821c4
commit 04a8fb7806
5 files changed
+84 -68

No files matched your search

+6 -2
View File
@@ -189,7 +189,11 @@ DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE
1. Stop the service before any benchmark:
`sudo systemctl stop frank-default-small` (Dankserver) or
`sudo systemctl stop frank-default-large` (Dankclaw).
2. Run the bench on a scratch port (never the LLM service port 7770).
2. Run the bench (`llama-bench` for llama.cpp / vllm bench for vLLM).
Scratch ports are not used for tests or benches on the LLM hosts — a
test process cannot share the single GPU with the running service anyway
(the model + KV fill the card), so changes are validated directly against
the service itself.
3. Restart the service when the bench completes: `sudo systemctl start <svc>`.
4. Verify health before reporting:
`curl http://192.168.0.235:7770/health` or
@@ -406,7 +410,7 @@ two hosts, one Caddy, and one Gitea. One repo keeps them in one history.
| Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before making changes. |
| Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. |
| Deploying a committed change | **Run `scripts/deploy.sh <target>`** — it pushes, pulls with fast-forward-only, plans changed paths, dispatches selected services, and checks their health. See [docs/LLM/deployment.md](docs/LLM/deployment.md). |
| Benchmarking | **Stop the service first** — scratch port, restart, verify health. See the Benchmarking Policy above. |
| Benchmarking | **Stop the service first**, restart, verify health. See the Benchmarking Policy above. |
| Deleting a file | **Move it to trash** — `.AGENTTRASH/` (in-repo) or `~/.AGENTTRASH/YYYYMMDD-topic`. Never `rm`. |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
| Real secrets | **Never commit** — use host `.env` files. `.env.example` holds placeholders only. `FRANK_SUDO_PW` covers headless sudo. |