model cache cleared, re-fetched, and the update turned out to be real

Operator instruction: purge the cache so no run can serve a stale payload. The two
live payloads were cleared and then re-fetched rather than left absent — deleting
them without a re-fetch means neither endpoint starts.

halogen's re-fetch proved the point: its sidecars had been stale *in place*
(overlay.hgn 2,477,677,120 -> 2,572,466,560 bytes, overlay-speed.hgn
2,383,306,048 -> 2,478,095,488) while the 115.55 GiB checkpoint returned
byte-identical. frank-small's 27B and mmproj re-fetched to Hub-matching hashes,
unchanged.

Trap recorded: podman refuses a bind mount whose source directory is missing
(statfs ... no such file or directory), so frank-large needs /models to exist
*empty* for its first-start fetch to run.

Q4_K_S against the recorded Q4_K_M numbers: +2.1/+1.6/+2.6 % (iGPU-only pp512 /
pp16384 / tg64), +2.7 % tg32@64k, +4.1 %/+2.0 % for the two layer-split configs,
~0 on both pp512@64k, host RAM bottoming at 10.0 GiB against ~5 GiB. A memory
lever with a small speed dividend. IQ4_XS (97,473,155,360 bytes, sha256
b4787265...d0574f9d) is pulled and verified for the next comparison.

Also repairs an AGENTS.md edit that landed in the wrong section: the peer
session's commits shifted that file between my read and my write, so the
frank-large text briefly split the Model Config section.
This commit is contained in:
Frank committed 2026-09-16 22:07:04 -07:00
1 parent 77a07ffc30
commit a40a1aad92
4 files changed
+103 -11

No files matched your search

@@ -321,3 +321,31 @@ Operational notes for the next run:
the reboot: the same model, a more fragmented host afterwards — the engine
reported 240 free contiguous 2 MiB blocks against 475 earlier). Both endpoints
answered `/health` and served `/v1/models` under their names.
## Smaller-quant check: Q4_K_S against the recorded Q4_K_M numbers
Same host, same `build-dual`, same test shapes (`-r 1`, f16 KV, flash-attn
auto, `-d 65536` for depth). The repo ships i1-Q4_K_S as three `.partNofM`
fragments (35.00 + 35.00 + 34.09 GiB) to concatenate — 104.09 GiB against
110.97 for the Q4_K_M payload the earlier numbers came from.
| test | Q4_K_M | Q4_K_S | Δ |
|---|---|---|---|
| A\* iGPU-only pp512 / pp16384 / tg64 | 456.7 / 403.1 / 32.64 | **466.4 / 409.6 / 33.50** | +2.1 / +1.6 / +2.6 % |
| A\* iGPU-only pp512 / tg32 @64k | 231.8 / 25.77 | 232.0 / **26.47** | ~0 / +2.7 % |
| CUDA0/Vulkan1 split pp512 / tg32 @64k | 268.5 / 27.23 | 268.0 / **28.35** | ~0 / +4.1 % |
| Vulkan1/Vulkan0 split pp512 / tg32 @64k | 224.3 / 27.55 | **233.6 / 28.11** | +4.2 / +2.0 % |
Memory sampled through the run (`/proc/meminfo`, `mem_info_gtt_used` and
`nvidia-smi`, every 15 s): peak 3090 VRAM **13,466 MiB** (the CUDA-side split),
peak iGPU GTT **71.6 GiB** (A\*), and host RAM bottomed at **10.0 GiB
available** during A\* — against ~5 GiB for the Q4_K_M A\* run, so the 6.9 GiB
smaller file bought roughly that much headroom.
**Read it as a memory lever with a small speed dividend**: 2–4 % for 6.9 GiB.
Nothing structural moves — `-ot` still gates per-tensor placement and a layer
split still cannot consolidate KV on the 3090. If the aim were to *deploy*
mainline on this iGPU, `i1-IQ4_XS` (90.78 GiB, single file, sha256
`b4787265…d0574f9d`) is the rung that changes the memory arithmetic and q8_0 KV
is the lever that changes depth bandwidth; Q4_K_S is the conservative middle.