docs: the VRAM figures settle 0.2 GiB below the post-load reading
6.44 / 6.90 is the state straight after load; after the service has served traffic it sits at 6.23 / 6.64, with and without requests in flight. Both are ~5 GiB above the floor, but the runbook should not imply a load-dependent difference that the two readings do not actually show.
This commit is contained in:
@@ -282,9 +282,10 @@ larger, so the question is closed rather than pending.
|
||||
- Both cards hold the model and they are **balanced**: at 262144 with the
|
||||
deployed batch and split, `rocm-smi` reports **6.44 GiB free on GPU 0 and
|
||||
6.90 GiB on GPU 1** of 31.86 GiB each — reproducing the pre-deploy
|
||||
measurement of the same shape to 0.01 GiB. Under load (two concurrent
|
||||
requests, one per slot) they sit at **6.23 / 6.64**, and both are far above
|
||||
the operator's 1.5 GiB floor.
|
||||
measurement of the same shape to 0.01 GiB. That is the reading straight after
|
||||
load; once the service has served traffic it settles ~0.2 GiB lower
|
||||
(**6.23 / 6.64**, the same with two concurrent requests in flight and after
|
||||
them). Both figures are ~5 GiB above the operator's 1.5 GiB floor.
|
||||
- Drafting is live on **both slots**, which is what a 2-slot deployment has to
|
||||
prove: `slot print_timing` reported `draft acceptance = 0.62857 (44 accepted /
|
||||
70 generated), mean len = 5.40` and `0.48214 (27 / 56)` on slot 1, and
|
||||
|
||||
Reference in New Issue
Block a user