frank-small: four FULL 262144-token slots — the max that fits the card

CTX_SIZE=1048576 / PARALLEL=4: every slot gets the model's full trained
window. Measured fit arithmetic on the 3090: 1 full-context slot idles
at 11,996 MiB, 4 at 22,346 -> 3,450 MiB per slot, ~8,546 MiB fixed.
A 5th slot projects 25.8 GiB past the 24,576 MiB card, and the KV pool
allocates upfront, so 4 is the ceiling. Explicit CTX_SIZE bypasses --fit
on purpose: fit maximises the TOTAL pool and would divide a smaller
window across slots at parallel>1. The slot-grid seam now pins both the
full-window division and the budget inequality so a future capacity
bump fails pre-deploy, not in the boot allocation.

Audio verified while measuring: the mmproj's audio encoder transcribes
input_audio parts verbatim (JFK WAV sample). Header comments updated.
This commit is contained in:
Frank committed 2026-09-17 02:41:11 -07:00
1 parent 87caca4894
commit dbd764a493
3 files changed
+44 -28

No files matched your search

+9 -6
View File
@@ -15,13 +15,16 @@
# context-backed drafters cannot share one process, so this chain is the
# only shape that carries both.
#
# Context: CTX_SIZE is left unset and --fit sizes the KV pool (q8_0 K and
# V) against what the weights, projector, draft context and compute
# buffers leave on the card; the chosen value lands in the startup log.
# Context: CTX_SIZE is set explicitly to PARALLEL x 262144 so every slot
# gets the model's FULL window; --fit (the CTX_SIZE-unset branch below)
# maximises the TOTAL pool instead, which would divide a smaller window
# across slots. The 4-slot layout is measured to fit the card (see the
# SIZING note in frank-small.env); more slots would fail the KV allocation.
#
# Vision: --mmproj loads the BF16 projector (it also carries an audio
# encoder; vision is the fleet contract). The fleet's image path lives
# here, which is why halogen no longer sets HALOGEN_VISION_TOWER.
# Vision + audio: --mmproj loads the BF16 projector, which also carries
# an audio encoder — input_audio content parts transcribe verbatim
# (verified 2026-09-17). The fleet's image path lives here, which is why
# halogen no longer sets HALOGEN_VISION_TOWER.
#
# Sampling: greedy by default (the fleet contract). Clients that send their own
# temperature get their own values, and the harness overrides per model from