docs: the VRAM figures settle 0.2 GiB below the post-load reading

6.44 / 6.90 is the state straight after load; after the service has served
traffic it sits at 6.23 / 6.64, with and without requests in flight. Both are
~5 GiB above the floor, but the runbook should not imply a load-dependent
difference that the two readings do not actually show.
This commit is contained in:
Frank committed 2026-09-17 00:29:21 -07:00
1 parent 6472d706be
commit 24cdbb0e45
1 file changed
+4 -3
@@ -282,9 +282,10 @@ larger, so the question is closed rather than pending.
- Both cards hold the model and they are **balanced**: at 262144 with the - Both cards hold the model and they are **balanced**: at 262144 with the
deployed batch and split, `rocm-smi` reports **6.44 GiB free on GPU 0 and deployed batch and split, `rocm-smi` reports **6.44 GiB free on GPU 0 and
6.90 GiB on GPU 1** of 31.86 GiB each — reproducing the pre-deploy 6.90 GiB on GPU 1** of 31.86 GiB each — reproducing the pre-deploy
measurement of the same shape to 0.01 GiB. Under load (two concurrent measurement of the same shape to 0.01 GiB. That is the reading straight after
requests, one per slot) they sit at **6.23 / 6.64**, and both are far above load; once the service has served traffic it settles ~0.2 GiB lower
the operator's 1.5 GiB floor. (**6.23 / 6.64**, the same with two concurrent requests in flight and after
them). Both figures are ~5 GiB above the operator's 1.5 GiB floor.
- Drafting is live on **both slots**, which is what a 2-slot deployment has to - Drafting is live on **both slots**, which is what a 2-slot deployment has to
prove: `slot print_timing` reported `draft acceptance = 0.62857 (44 accepted / prove: `slot print_timing` reported `draft acceptance = 0.62857 (44 accepted /
70 generated), mean len = 5.40` and `0.48214 (27 / 56)` on slot 1, and 70 generated), mean len = 5.40` and `0.48214 (27 / 56)` on slot 1, and