frank-small: four FULL 262144-token slots — the max that fits the card
CTX_SIZE=1048576 / PARALLEL=4: every slot gets the model's full trained window. Measured fit arithmetic on the 3090: 1 full-context slot idles at 11,996 MiB, 4 at 22,346 -> 3,450 MiB per slot, ~8,546 MiB fixed. A 5th slot projects 25.8 GiB past the 24,576 MiB card, and the KV pool allocates upfront, so 4 is the ceiling. Explicit CTX_SIZE bypasses --fit on purpose: fit maximises the TOTAL pool and would divide a smaller window across slots at parallel>1. The slot-grid seam now pins both the full-window division and the budget inequality so a future capacity bump fails pre-deploy, not in the boot allocation. Audio verified while measuring: the mmproj's audio encoder transcribes input_audio parts verbatim (JFK WAV sample). Header comments updated.
This commit is contained in:
1 parent
87caca4894
commit
dbd764a493
3 files changed
+44
-28
No files matched your search
@@ -15,13 +15,16 @@
|
||||
# context-backed drafters cannot share one process, so this chain is the
|
||||
# only shape that carries both.
|
||||
#
|
||||
# Context: CTX_SIZE is left unset and --fit sizes the KV pool (q8_0 K and
|
||||
# V) against what the weights, projector, draft context and compute
|
||||
# buffers leave on the card; the chosen value lands in the startup log.
|
||||
# Context: CTX_SIZE is set explicitly to PARALLEL x 262144 so every slot
|
||||
# gets the model's FULL window; --fit (the CTX_SIZE-unset branch below)
|
||||
# maximises the TOTAL pool instead, which would divide a smaller window
|
||||
# across slots. The 4-slot layout is measured to fit the card (see the
|
||||
# SIZING note in frank-small.env); more slots would fail the KV allocation.
|
||||
#
|
||||
# Vision: --mmproj loads the BF16 projector (it also carries an audio
|
||||
# encoder; vision is the fleet contract). The fleet's image path lives
|
||||
# here, which is why halogen no longer sets HALOGEN_VISION_TOWER.
|
||||
# Vision + audio: --mmproj loads the BF16 projector, which also carries
|
||||
# an audio encoder — input_audio content parts transcribe verbatim
|
||||
# (verified 2026-09-17). The fleet's image path lives here, which is why
|
||||
# halogen no longer sets HALOGEN_VISION_TOWER.
|
||||
#
|
||||
# Sampling: greedy by default (the fleet contract). Clients that send their own
|
||||
# temperature get their own values, and the harness overrides per model from
|
||||
|
||||
Reference in new issue
Block a user