dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: record the KV_MEM=0 passthrough trap in the frank-medium runbook

Browse Source
This commit is contained in:
Frank
2026-09-16 03:58:48 -07:00
parent df260dd419
commit 7ce6e96bc5
1 changed files with 1 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md
+1 -1
View File
@@ -148,7 +148,7 @@ Both replicas, uniform except GPU index and port:
|---|---|---|
| `TP` | 1 | one card per replica |
| `MAXLEN` | 114688 | the largest shape that boots on one card: 262144 needs 9.61 GiB of KV against 5.14 available, and 131072 needs 5.58 against 5.22 |
| `KV_MEM` | 0 | vLLM profiles this shape; the upstream pin (2x7551-32624) is a TP=2/262144 measurement and is invalid here |
| `KV_MEM` | 0 | passed through literally as `--kv-cache-memory 0`, which the image's patched KV sizing reads as "profile this shape"; the upstream pin (2x7551-32624) is a TP=2/262144 measurement and is invalid here. **Do not blank the variable instead of passing 0**: with the flag omitted vLLM sizes the cache from `GPU_UTIL` alone and the KV allocation dies with `torch.OutOfMemoryError` (measured: 2.16 GiB requested, 116 MiB free, 29.34 GiB already held) |
| `MAXSEQS` | 4 | matches the per-replica share of the fleet's concurrency |
| `CHUNK` | 8192 | prefill chunk (upstream default) |
| `GPU_UTIL` | 0.98 | upstream default; the card is dedicated |