From ff52543d8add6f3768141b09ef4bbe97995006a2 Mon Sep 17 00:00:00 2001 From: Frank Date: Wed, 16 Sep 2026 15:52:18 -0700 Subject: [PATCH] docs: frank-medium splits weights and KV across both cards, it does not replicate them MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The loader prints no per-device ledger at this verbosity, so the claim is measured rather than read: +131072 tokens of context added 4.83 GB to one card and 4.84 GB to the other (both carry their own attention layers' KV), and neither card's total usage could hold the 31.46 GB file. Worth writing down because the retired shape WAS two full copies — that is what a replica is, and it is why the 29.8 GB FP8 checkpoint could never run there. --- .../LLM/frank-medium-llamacpp-dflash2.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md index 314881c..e1d8f0d 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md +++ b/serverAppConfigs-docs/LLM/frank-medium-llamacpp-dflash2.md @@ -177,6 +177,15 @@ Deployed flags: slot: the raised window is real, not just declared, and it is 22 % beyond the 114688 the vLLM predecessor could offer. +- **Nothing is replicated.** The loader does not print a per-device ledger at + this verbosity, so this is measured instead of read: raising CTX_SIZE from + 131072 to 262144 added **4.83 GB to GPU 0 and 4.84 GB to GPU 1** — both cards + carry the KV for their own attention layers, about half each — and neither + card's *total* usage (24.94 / 29.42 GB at 262144) could hold the 31.46 GB + file, so no card carries a full copy of the weights. Compare the shape this + replaced: two vLLM replicas, i.e. two complete copies of the model, one per + card, which is exactly why a 29.8 GB FP8 checkpoint could not be served there + at any usable context. ### Measured throughput (2026-09-16, deployed service, `temperature 0`) Same filler corpus for every shape, so these are shapes rather than a