docs: frank-medium splits weights and KV across both cards, it does not replicate them
The loader prints no per-device ledger at this verbosity, so the claim is measured rather than read: +131072 tokens of context added 4.83 GB to one card and 4.84 GB to the other (both carry their own attention layers' KV), and neither card's total usage could hold the 31.46 GB file. Worth writing down because the retired shape WAS two full copies — that is what a replica is, and it is why the 29.8 GB FP8 checkpoint could never run there.
This commit is contained in:
@@ -177,6 +177,15 @@ Deployed flags:
|
||||
slot: the raised window is real, not just declared, and it is 22 % beyond the
|
||||
114688 the vLLM predecessor could offer.
|
||||
|
||||
- **Nothing is replicated.** The loader does not print a per-device ledger at
|
||||
this verbosity, so this is measured instead of read: raising CTX_SIZE from
|
||||
131072 to 262144 added **4.83 GB to GPU 0 and 4.84 GB to GPU 1** — both cards
|
||||
carry the KV for their own attention layers, about half each — and neither
|
||||
card's *total* usage (24.94 / 29.42 GB at 262144) could hold the 31.46 GB
|
||||
file, so no card carries a full copy of the weights. Compare the shape this
|
||||
replaced: two vLLM replicas, i.e. two complete copies of the model, one per
|
||||
card, which is exactly why a 29.8 GB FP8 checkpoint could not be served there
|
||||
at any usable context.
|
||||
### Measured throughput (2026-09-16, deployed service, `temperature 0`)
|
||||
|
||||
Same filler corpus for every shape, so these are shapes rather than a
|
||||
|
||||
Reference in New Issue
Block a user