docs: frank-medium splits weights and KV across both cards, it does not replicate them

The loader prints no per-device ledger at this verbosity, so the claim is
measured rather than read: +131072 tokens of context added 4.83 GB to one card
and 4.84 GB to the other (both carry their own attention layers' KV), and
neither card's total usage could hold the 31.46 GB file. Worth writing down
because the retired shape WAS two full copies — that is what a replica is, and
it is why the 29.8 GB FP8 checkpoint could never run there.
This commit is contained in:
Frank
2026-09-16 15:52:18 -07:00
parent c97e359bc2
commit ff52543d8a
@@ -177,6 +177,15 @@ Deployed flags:
slot: the raised window is real, not just declared, and it is 22 % beyond the
114688 the vLLM predecessor could offer.
- **Nothing is replicated.** The loader does not print a per-device ledger at
this verbosity, so this is measured instead of read: raising CTX_SIZE from
131072 to 262144 added **4.83 GB to GPU 0 and 4.84 GB to GPU 1** — both cards
carry the KV for their own attention layers, about half each — and neither
card's *total* usage (24.94 / 29.42 GB at 262144) could hold the 31.46 GB
file, so no card carries a full copy of the weights. Compare the shape this
replaced: two vLLM replicas, i.e. two complete copies of the model, one per
card, which is exactly why a 29.8 GB FP8 checkpoint could not be served there
at any usable context.
### Measured throughput (2026-09-16, deployed service, `temperature 0`)
Same filler corpus for every shape, so these are shapes rather than a