docs: the max-length sweep tabled; frank-small.env NGL/FLASH_ATTN restored
- runbook: identity now 4x262144, audio verified, throughput table with the OOM and negative-result rows, deployed flags updated to reality. - AGENTS.md frank-small paragraph matches the deployed shape. - log.md: the sweep entry. - scripts/frank-small.env: my earlier line-range edit had clobbered NGL=999 and FLASH_ATTN=on out of the file; the seams caught it. The live service was unaffected because the script's defaults match the declared values exactly, but the file contract is restored. Suite: ALL TESTS PASSED (321), vault 125 pages 0 unresolved.
This commit is contained in:
1 parent
24eaba1078
commit
17450dd172
4 files changed
+102
-34
No files matched your search
@@ -306,22 +306,20 @@ serving `HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced` at Q4_K_M.
|
||||
The MTP head is a SEPARATE assistant file (`mtp-gemma-4-12B-it.gguf`, 4
|
||||
nextn layers) loaded via `--model-draft`; the draftless **ngram-mod**
|
||||
matcher chains behind it in the same `--spec-type` value. BF16 mmproj,
|
||||
port 7771, q8_0 KV, `CTX_SIZE` unset so `--fit` sizes the pool, 1 slot,
|
||||
greedy default. Deployed flags are in `scripts/frank-small.env`; the serve
|
||||
script refuses to start if the weights, the projector, the sidecar or the
|
||||
binary are missing.
|
||||
port 7771, q8_0 KV, `CTX_SIZE=1048576` / `PARALLEL=4`
|
||||
(four FULL 262,144-token slots — the measured card ceiling), batch
|
||||
**8192/1024**, greedy default. Deployed flags are in
|
||||
`scripts/frank-small.env`; the serve script refuses to start if the
|
||||
weights, the projector, the sidecar or the binary are missing.
|
||||
|
||||
Verified at deploy (2026-09-17): `/v1/models` reports `frank-small`; a
|
||||
chat completion returns content plus `reasoning_content`; a tool call
|
||||
returns `tool_calls`; the slot timing lines show `draft acceptance =
|
||||
0.38–0.46, mean len ≈ 2.2` (MTP engaged); an image request is answered
|
||||
from the projector. `--fit` took the FULL window — `n_slots = 1,
|
||||
n_ctx_slot = 262144` — because Gemma4's sliding-window pattern keeps the
|
||||
q8 KV pool cheap: 11,996 MiB of the card's 24,576 MiB in use. The
|
||||
`Gemma4Assistant requires ctx_other to be set` error that prints at boot
|
||||
is the fit's own probe (the log calls it normal); the sidecar then loads
|
||||
against the real context. Gemma4's template accepts every
|
||||
`reasoning_effort` level (all seven probed: 200).
|
||||
returns `tool_calls`; an image request is answered from the projector;
|
||||
an `input_audio` WAV transcribes verbatim. Post-tuning live numbers at
|
||||
max length: pp@254k ~965 t/s, tg@253k 226–302 t/s with ~94% draft
|
||||
acceptance; idle 23,300 MiB of 24,576. The throughput sweep (and why
|
||||
ubatch 2048 OOMs the draft's pp buffer, and why draft depth 4 + poll
|
||||
100 is WORSE at long ctx) is tabled in the runbook.
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-small-gemma4-12b.md`.
|
||||
Rollback: [[frank-small-qwen38-gsq-rco]].
|
||||
|
||||
|
||||
Reference in new issue
Block a user