docs(kv): add bit-density KV cache quant table across 3 runtimes
- Place ik's Trellis (IQ*_KT), IQK (IQ*_K), MXFP4, and dedicated fast KV type Q8_KV in a single high->low bits-per-value table with ROCmFPX TurboQuant KV. - Mark mainline-provided types (*), KV-native vs weight-family-reused-for-KV. - Hadamard annotated as a transform on top of a base quant, not a row. - bpw read from exact ggml block structs (QK_K=256, fp16 scale): type_size x 8 / QK, verified against the struct comment "6.5625 bits per weight" and the trait table. Notes where legacy nominal figures (4.0/4.25) diverge. - AGENTS.md: add benchmark directive (use llama-bench / vllm bench; ask before rolling your own).
This commit is contained in:
1 parent
4308f62004
commit
2277a2e9ac
2 files changed
+93
-1
No files matched your search
@@ -174,6 +174,9 @@ These servers serve real users (about 3 people). Running a bench while the
|
||||
service is up starves the GPU and corrupts results. Restart the service when
|
||||
the bench finishes.
|
||||
|
||||
ALWAYS bench using llama-bench for llama.cpp or vllm bench for vllm runtimes.
|
||||
DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE OFFICIAL BENCHES. ASK THE USER BEFORE DEVIATING FROM THIS DIRECTIVE.
|
||||
|
||||
1. Stop the service before any benchmark:
|
||||
`sudo systemctl stop frank-default-small` (Dankserver) or
|
||||
`sudo systemctl stop frank-default-large` (Dankclaw).
|
||||
|
||||
Reference in new issue
Block a user