docs(vault): the work log keeps incidents and laws; the numbers move to the page that owns them
log.md drops 466 lines of per-swap ledger prose (1,169 lines out, 703 back) and keeps the causal stories: why the host went down, why a knob resets at boot, why a stale drop-in served a different model for 24 minutes, why the RAM was the fault. Date sections are unchanged (65 before, 65 after) -- this is compression inside sections, not deletion of days. Every number the compression would have orphaned was ported to its owner first, and a census of the deleted lines against the whole vault is what found them: - frank-zealot: the two boots that priced non-KV (20,190/KV 12,800 => 7,390 without the projector; 22,972/KV 14,113 => 8,859 with it, so the vision tower costs ~1.4 GiB for a 918 MB file), the 1,153 MiB-free 3-slot boot that made the seam price non-KV at 9 GiB, and the f16 wall's own figures (fit 33,198 vs 23,558 free on a 25,088.00 MiB KV request). - frank-archon: the pool-cut-is-headroom-not-arena law (2.75 GiB/rank bought zero slots; d(reserve) = d(arena) = +248 slots; 14.87 GiB / 6,496 slots against a 398,658-token pool), the MNBT compile-RANGE vs SEQS+1 law, and the 2 x 196,919 prefix-cache proof (393,837 tokens resident above the pool's 386,904; 94.1/102.3/117.6 t/s, acceptance 0.7284/0.7525). - frank-dragoon: the KL arm of the 2026-10-17 A/B (mean 0.155 / median 0.057, top-1 agreement 85.1 %) joins the PPL and NIAH figures already on the era row. - Payload and Disk Reclaim: the 121 GB layer census lesson -- a huge image layer can be the engine image, not a payload. Also fixes a wikilink the rewrite had hard-wrapped across two lines, which the link checker reads as a target that does not exist. New seam: 'one page per live LLM endpoint'. The live set is derived from scripts/ (an endpoint has both an env and a unit), and any <endpoint>-*.md page in the vault fails it -- the enforcement of the one-page-per-endpoint law, so a re-created era page cannot pass tests.
This commit is contained in:
Frank
@@ -193,6 +193,22 @@ Before 2026-09-26 the port belonged to the **`frank-medium`** service: [[frank-m
|
||||
on this card/model: stealing 2.14 GiB of weights to buy `MAX_MODEL_LEN 262144` cost 2.5 %
|
||||
decode; the other direction (`--expert-vs-cache-ratio` 0.82 → 0.84, +2.5 GiB of weights) cost
|
||||
8.9 %.
|
||||
- **A KV-pool cut is a runtime-headroom lever, never an arena lever.** The 6.1e9 → 3.15e9 B/rank pool cut
|
||||
(2.75 GiB/rank) bought **zero** extra expert-cache slots. The arena moves with the reserve, one-for-one
|
||||
(d(reserve) 0.57 GiB = d(arena) 0.57 GiB = **+248 slots**). The shape that pool cut replaced died with
|
||||
`0 bytes free` on a real 995-token request — that is headroom, not slot count. Read `expert cache space … / N
|
||||
slots` out of the boot ledger before trusting any pool arithmetic: the 2026-10-13 read was **14.87 GiB /
|
||||
6,496 slots** against `GPU KV cache size: 398,658 tokens`, and it is the POOL that caps resident
|
||||
conversations, not the arena.
|
||||
- **MNBT moves the compile RANGE, not the batch width.** The engine sets `num_reqs = min(max_num_batched_tokens,
|
||||
max_num_reqs)`, so the CUDA-graph compile width is **SEQS + 1** for any `MNBT >= SEQS` — 3 at SEQS 2, 4 at
|
||||
SEQS 3, exactly the constant printed in the engine's own message. MNBT only shifts the compiled range
|
||||
((1,1024) versus (1,2048)), which is why the identical capture failure appears at both rungs;
|
||||
`--enforce-eager` bypasses the path and loses CUDA graphs with it.
|
||||
- **The prefix cache proves itself at depth, not in the ledger.** Two concurrent conversations of 196,919 tokens
|
||||
each answered the marker buried at the head of **their own** prompt with **393,837 tokens resident** — above the
|
||||
fp8 pool's 386,904, so the answer came from a cached prefix and not a re-prefill. Same boot (2026-10-13):
|
||||
decode at 51.5k depth **94.1 cold / 102.3 warm / 117.6 prefix-hit t/s**, draft acceptance **0.7284 / 0.7525**.
|
||||
- **The host pool moves with the slab.** `vram_weights` and `host_pool_mib` serve the per-rank
|
||||
resident set *together*: at the fp8-KV pair (26071/3543) the ledger showed the slab full to
|
||||
259 KiB of slack, and cutting `vram-weights` by 3.8 GiB to pay for a bf16 pool spills ~3.8
|
||||
|
||||
@@ -69,7 +69,7 @@ rollback plan — recoverable only until the trash is purged.
|
||||
|
||||
| Window | Payload | Runtime | Shape | Replaced because | Rollback |
|
||||
|---|---|---|---|---|---|
|
||||
| 2026-10-06 → 10-17 | Swift1.5 abliterated v2 `.hgn` r2 66,687,678,464 B `9c880b8b…` + same trio | flash-server 0.16.2, same digest | 2 slots, ctx 262,144, pool 524,288 (14.6 GiB), MTP always-on | 2026-10-17 A/B at that shape: NIAH 18/18 vs 17/18, refusal census 0/5 on both, tool calls + red-PNG probe identical, needle hit at both depths of a 21.5k haystack, decode 50.0 vs 46.8 t/s, cold prefill 1,604 vs 1,425 t/s, PPL 3.968 vs 3.877 (finetune's edge gone: +2.3 %), thinking 1,476 vs 784 mean, 7.4 GiB RAM back | `/home/dank/models/halogen-flashnext-v2/` + `scripts/download-frank-dragoon-flashnext.sh` |
|
||||
| 2026-10-06 → 10-17 | Swift1.5 abliterated v2 `.hgn` r2 66,687,678,464 B `9c880b8b…` + same trio | flash-server 0.16.2, same digest | 2 slots, ctx 262,144, pool 524,288 (14.6 GiB), MTP always-on | 2026-10-17 A/B at that shape: NIAH 18/18 vs 17/18, refusal census 0/5 on both, tool calls + red-PNG probe identical, needle hit at both depths of a 21.5k haystack, decode 50.0 vs 46.8 t/s, cold prefill 1,604 vs 1,425 t/s, PPL 3.968 vs 3.877 (finetune's edge gone: +2.3 %), KL vs the swift15 logits mean 0.155 / median 0.057 with top-1 agreement 85.1 % — two sound models disagreeing on content, not a corrupt forward pass, thinking 1,476 vs 784 mean, 7.4 GiB RAM back | `/home/dank/models/halogen-flashnext-v2/` + `scripts/download-frank-dragoon-flashnext.sh` |
|
||||
| 2026-10-05 → 10-06 | peonist `halogen-qwen3.8-27b` `qwen3.8-27b-p1w4d-d2.hgn` 35,865,565,184 B `274c3fc7…` | halogen 0.1.4 `sha256:dc0a39a0…` | 1 slot × 200,700, DFlash2 drafter, `HALOGEN_CACHE_MB=16384`, text-only, censored base | flash-server v2 restored vision, abliteration and MTP | `/home/dank/models/halogen-qwen3.8-27b/` + `scripts/download-frank-dragoon-halogen.sh`; 0.1.4 digest still in podman storage |
|
||||
| 2026-10-03 → 10-05 | kingjones777 Uncensored `Q6_0_ROCMFPX_AGENT.gguf` 24,856,607,008 B `e966deb3…` + `mtp-Q4_0` `051a1764…` + BF16 mmproj | `kingjones30/ROCmFPX` @ `36e9acd` (`scripts/build-rocmfpx.sh`) | 1 × 262,144 f16 KV, `draft-mtp` n-max 4, vision on | halogen card claimed 31.7 t/s vs 27.6–28.1 measured | `/home/dank/models/kingjones-Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP/` + `/home/dank/GitPackages/kingjones-ROCmFPX/build` + `download-frank-dragoon-kingjones-uncensored.sh` |
|
||||
| 2026-10-02 → 10-03 | aligned `Q8_0_ROCMFPX_AGENT.gguf` 28,214,425,952 B `f3ef2bdb…` | same fork pin | 1 × 262,144 f16 KV, `draft-mtp` 4 (MTP +123 % decode) | the uncensored file was faster *and* uncensored | `/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/` + `download-frank-dragoon-kingjones.sh` |
|
||||
|
||||
@@ -118,6 +118,9 @@ re-run the same arms with the box quiet before a figure is carried anywhere else
|
||||
|
||||
- **Addressable ≠ advertised.** The card is 24,576 MiB but leaves **24,123 MiB**: quote `nvidia-smi`'s `free` field, never `total − used` — the driver's reservation is the difference.
|
||||
- **Exact KV and a 786,432 pool cannot coexist here**: 32,768 B/token ⇒ 24,576 MiB before a single weight loads (the retired "26,112 MiB" note doubled q8_0's scale overhead); the 27B analogue died at **25,088 MiB** with `cudaMalloc failed: out of memory`. Halving the pool funds exact KV; the per-slot window stops at the trained 262,144 (no YaRN here), so a third conversation is a KV-dtype decision, never more context.
|
||||
The wall's own figures: `cudaMalloc` failed on a **25,088.00 MiB** KV request when the engine's fit projection
|
||||
asked for **33,198 MiB** against **23,558 MiB** free. It booted f16 at all because the arm env's invented
|
||||
`KV_CACHE_TYPE` was swallowed by the serve script's `:=f16` default — so a failed window became the measurement.
|
||||
- **Exact KV reads 1.882× the bytes per generated token, not 2×** (32,768 vs 17,408 B/token, 2026-10-18): a decode drop of that order is EXPECTED and must be measured, while prefill should hold — and this service can afford the dtype precisely because its load is the compaction hub's ~170k-token prefill, which is weight-bandwidth bound and never re-reads the cache per generated token. The dtype is paid in decode and concurrency.
|
||||
- Never `--fit` with `parallel > 1` — it maximizes the *total* pool and shrinks each slot: set `CTX_SIZE` explicitly and keep it divisible by `PARALLEL` (the seam enforces it).
|
||||
- **KV rate law.** `full_layers × 4 KV heads × 256 dim` = **8,192 elements per token per cache** on this arch (the engine prints `n_embd_k_gqa = n_embd_v_gqa = 1024`), ×2 B (f16), ×1.0625 (q8_0 — 32 codes plus one 2-byte scale
|
||||
@@ -133,6 +136,11 @@ re-run the same arms with the box quiet before a figure is carried anywhere else
|
||||
`tests/frank_zealot_unit.bats`'s fit gate can see a boot that would die at `cudaMalloc`, so every retune keeps it
|
||||
green — and the first verbosity-4 ledger plus settled `nvidia-smi` of a new dtype belongs on this page, because
|
||||
projections drift.
|
||||
The two boots that priced non-KV: 2 × 262,144 **without** the projector — 20,190 used, KV 12,800 ⇒ non-KV
|
||||
**7,390 MiB**; 3 × 196,608 **with** it — 22,972 used, KV 14,113 ⇒ non-KV **8,859 MiB**. The difference prices the
|
||||
vision tower at **~1.4 GiB for a 918 MB file**, because the encoder's activation and compute buffers ride with it.
|
||||
The first 3-slot deploy (3 × 196,608) booted healthy and left only **1,153 MiB** free — that is how the 2–3 GiB
|
||||
band got violated in practice, and why the seam prices non-KV at the measured 9 GiB and refuses that shape.
|
||||
- **Ubatch is a prefill knob.** Decode never moves with it and the logical `--batch-size` is neutral; prefill climbs monotonically to **2048** on this CUDA card (+4.6 % at 8.9k depth; 710 → 1,188 t/s), a prompt shorter than the ubatch cannot see the knob at all (the control row), and a verdict never crosses backends — the 2× R9700 split peaks at 512. Each rung costs ~0.6–0.75 GiB of compute buffers; 2048 is the ceiling for a 27B-on-3090 shape.
|
||||
- **A quant rung is paid in decode**: Q5_K_M → Q6_K is +875 MiB of weights and **−10 %** rate on this bandwidth-bound card, and the byte-bandwidth law predicts a rung's rate within 1 %.
|
||||
- **Speculation pays only with room on the card.** The 2026-10-18 arm census (all q8_0 K+V at the 786,432 pool): 3 × 262,144 spec-OFF ubatch 2048 → 17,876 / 6,249 free, 167.2 t/s; spec-ON ubatch 2048 → 22,706 / 1,419, 109.2 t/s; spec-ON ubatch 1024 → 21,400 / 2,725, 213.5 t/s; 2 × 262,144 spec-ON → 213.2; 1 × 262,144 spec-ON → 214.2. That is **+28 % decode whenever the card has room**, and the doors back are one window per conversation, or q8_0 KV with ubatch 1024. The draft's own bill at that shape was 3,072.00 MiB of KV plus ~1,758 MiB of buffers.
|
||||
|
||||
@@ -196,6 +196,10 @@ whole host shares.
|
||||
process`); `podman system migrate` panics in 5.7.0.
|
||||
- A same-filesystem trash reclaims no space; root-owned leftovers need
|
||||
`sudo`; never `rm` without explicit permission.
|
||||
- **A giant image layer is not automatically a model blob.** The 2026-10-17 census found a 121 GB layer
|
||||
(`16b6663d…`) that read like an undeleted payload; it is the **engine image** — the radiance runtime itself.
|
||||
Map each layer to what the serving trio actually mounts before deleting anything: that census is both the
|
||||
disk-reclaim proof and the check that no deleted payload path still backs a live container.
|
||||
|
||||
## Related
|
||||
|
||||
|
||||
+703
-1169
File diff suppressed because it is too large
Load Diff
@@ -3,6 +3,9 @@
|
||||
# Every service defined in a compose.yaml must have a matching documentation
|
||||
# page in serverAppConfigs-docs/ and must be reachable from the index. This is the "docs first"
|
||||
# contract of this repo: no service ships without documentation.
|
||||
# The vault's second law, added 2026-10-18: ONE page per live LLM endpoint.
|
||||
# A payload swap is an Eras row on that endpoint's page, never a page of its
|
||||
# own -- a second file holding the same shape is the next thing to go stale.
|
||||
|
||||
load helpers
|
||||
|
||||
@@ -43,3 +46,31 @@ load helpers
|
||||
run "$REPO_ROOT/scripts/check-vault-links.py" "$REPO_ROOT/serverAppConfigs-docs"
|
||||
[ "$status" -eq 0 ] || { echo "$output"; false; }
|
||||
}
|
||||
|
||||
@test "one page per live LLM endpoint" {
|
||||
# The live set is derived, not hardcoded: an endpoint is live if scripts/
|
||||
# holds both its env and its unit (frank-tts has an env but no unit, so it
|
||||
# is not an inference endpoint; homepage-status is the reverse).
|
||||
local vault="$REPO_ROOT/serverAppConfigs-docs" derived=0 name extra
|
||||
for envf in "$REPO_ROOT"/scripts/frank-*.env; do
|
||||
name=$(basename "$envf" .env)
|
||||
[ -f "$REPO_ROOT/scripts/$name.service" ] || continue
|
||||
derived=$((derived + 1))
|
||||
[ -f "$vault/LLM/$name.md" ] || {
|
||||
echo "live endpoint '$name' has no canonical page: serverAppConfigs-docs/LLM/$name.md"
|
||||
return 1
|
||||
}
|
||||
extra=$(find "$vault" -name "${name}-*.md" -not -path '*/.AGENTTRASH/*' | sort)
|
||||
if [ -n "$extra" ]; then
|
||||
echo "endpoint '$name' owns exactly one page; these files re-create a second owner of its shape:"
|
||||
echo "$extra"
|
||||
echo "Fold them into LLM/$name.md (an Eras row plus, if needed, a Law) and trash the page."
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
# Guard against the derivation going vacuous -- an empty loop passes silently.
|
||||
[ "$derived" -ge 3 ] || {
|
||||
echo "expected at least 3 live endpoints derived from scripts/*.env + scripts/*.service, got $derived"
|
||||
return 1
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user