model cache cleared, re-fetched, and the update turned out to be real
Operator instruction: purge the cache so no run can serve a stale payload. The two live payloads were cleared and then re-fetched rather than left absent — deleting them without a re-fetch means neither endpoint starts. halogen's re-fetch proved the point: its sidecars had been stale *in place* (overlay.hgn 2,477,677,120 -> 2,572,466,560 bytes, overlay-speed.hgn 2,383,306,048 -> 2,478,095,488) while the 115.55 GiB checkpoint returned byte-identical. frank-small's 27B and mmproj re-fetched to Hub-matching hashes, unchanged. Trap recorded: podman refuses a bind mount whose source directory is missing (statfs ... no such file or directory), so frank-large needs /models to exist *empty* for its first-start fetch to run. Q4_K_S against the recorded Q4_K_M numbers: +2.1/+1.6/+2.6 % (iGPU-only pp512 / pp16384 / tg64), +2.7 % tg32@64k, +4.1 %/+2.0 % for the two layer-split configs, ~0 on both pp512@64k, host RAM bottoming at 10.0 GiB against ~5 GiB. A memory lever with a small speed dividend. IQ4_XS (97,473,155,360 bytes, sha256 b4787265...d0574f9d) is pulled and verified for the next comparison. Also repairs an AGENTS.md edit that landed in the wrong section: the peer session's commits shifted that file between my read and my write, so the frank-large text briefly split the Model Config section.
This commit is contained in:
1 parent
77a07ffc30
commit
a40a1aad92
4 files changed
+103
-11
No files matched your search
@@ -321,3 +321,31 @@ Operational notes for the next run:
|
||||
the reboot: the same model, a more fragmented host afterwards — the engine
|
||||
reported 240 free contiguous 2 MiB blocks against 475 earlier). Both endpoints
|
||||
answered `/health` and served `/v1/models` under their names.
|
||||
|
||||
|
||||
## Smaller-quant check: Q4_K_S against the recorded Q4_K_M numbers
|
||||
|
||||
Same host, same `build-dual`, same test shapes (`-r 1`, f16 KV, flash-attn
|
||||
auto, `-d 65536` for depth). The repo ships i1-Q4_K_S as three `.partNofM`
|
||||
fragments (35.00 + 35.00 + 34.09 GiB) to concatenate — 104.09 GiB against
|
||||
110.97 for the Q4_K_M payload the earlier numbers came from.
|
||||
|
||||
| test | Q4_K_M | Q4_K_S | Δ |
|
||||
|---|---|---|---|
|
||||
| A\* iGPU-only pp512 / pp16384 / tg64 | 456.7 / 403.1 / 32.64 | **466.4 / 409.6 / 33.50** | +2.1 / +1.6 / +2.6 % |
|
||||
| A\* iGPU-only pp512 / tg32 @64k | 231.8 / 25.77 | 232.0 / **26.47** | ~0 / +2.7 % |
|
||||
| CUDA0/Vulkan1 split pp512 / tg32 @64k | 268.5 / 27.23 | 268.0 / **28.35** | ~0 / +4.1 % |
|
||||
| Vulkan1/Vulkan0 split pp512 / tg32 @64k | 224.3 / 27.55 | **233.6 / 28.11** | +4.2 / +2.0 % |
|
||||
|
||||
Memory sampled through the run (`/proc/meminfo`, `mem_info_gtt_used` and
|
||||
`nvidia-smi`, every 15 s): peak 3090 VRAM **13,466 MiB** (the CUDA-side split),
|
||||
peak iGPU GTT **71.6 GiB** (A\*), and host RAM bottomed at **10.0 GiB
|
||||
available** during A\* — against ~5 GiB for the Q4_K_M A\* run, so the 6.9 GiB
|
||||
smaller file bought roughly that much headroom.
|
||||
|
||||
**Read it as a memory lever with a small speed dividend**: 2–4 % for 6.9 GiB.
|
||||
Nothing structural moves — `-ot` still gates per-tensor placement and a layer
|
||||
split still cannot consolidate KV on the 3090. If the aim were to *deploy*
|
||||
mainline on this iGPU, `i1-IQ4_XS` (90.78 GiB, single file, sha256
|
||||
`b4787265…d0574f9d`) is the rung that changes the memory arithmetic and q8_0 KV
|
||||
is the lever that changes depth bandwidth; Q4_K_S is the conservative middle.
|
||||
Reference in new issue
Block a user