model cache cleared, re-fetched, and the update turned out to be real
Operator instruction: purge the cache so no run can serve a stale payload. The two live payloads were cleared and then re-fetched rather than left absent — deleting them without a re-fetch means neither endpoint starts. halogen's re-fetch proved the point: its sidecars had been stale *in place* (overlay.hgn 2,477,677,120 -> 2,572,466,560 bytes, overlay-speed.hgn 2,383,306,048 -> 2,478,095,488) while the 115.55 GiB checkpoint returned byte-identical. frank-small's 27B and mmproj re-fetched to Hub-matching hashes, unchanged. Trap recorded: podman refuses a bind mount whose source directory is missing (statfs ... no such file or directory), so frank-large needs /models to exist *empty* for its first-start fetch to run. Q4_K_S against the recorded Q4_K_M numbers: +2.1/+1.6/+2.6 % (iGPU-only pp512 / pp16384 / tg64), +2.7 % tg32@64k, +4.1 %/+2.0 % for the two layer-split configs, ~0 on both pp512@64k, host RAM bottoming at 10.0 GiB against ~5 GiB. A memory lever with a small speed dividend. IQ4_XS (97,473,155,360 bytes, sha256 b4787265...d0574f9d) is pulled and verified for the next comparison. Also repairs an AGENTS.md edit that landed in the wrong section: the peer session's commits shifted that file between my read and my write, so the frank-large text briefly split the Model Config section.
This commit is contained in:
@@ -58,7 +58,7 @@ discovery, the `services.json` probe and the docs needed no rename.
|
||||
| file | size | role |
|
||||
|---|---|---|
|
||||
| `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint |
|
||||
| `qwen38-flash-next-w4b.overlay.hgn` | 2.31 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing |
|
||||
| `qwen38-flash-next-w4b.overlay.hgn` | 2.40 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing. **Refreshed 2026-09-16**: the sidecars are versioned *in place*, so a stale local copy is invisible from its file name — this one had read 2.31 GiB since it was first fetched while the Hub had moved on |
|
||||
| `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) |
|
||||
| `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount |
|
||||
|
||||
@@ -68,9 +68,23 @@ never re-downloads, and `huggingface_hub` resumes an interrupted pull. The
|
||||
mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in
|
||||
`/home/dank/models/halogen`.
|
||||
|
||||
`overlay-speed.hgn` (2.22 GiB) comes along with the repo but is unused: it
|
||||
trades the `o_proj` calibration for about 2 % of decode, and the quality
|
||||
sidecar is the default for a reason.
|
||||
**Refreshing the weights.** Delete the mount's contents (or the whole
|
||||
directory, then recreate it **empty**) and start the unit: the entrypoint sees
|
||||
the checkpoint absent and re-fetches the repo, resuming if interrupted. Two
|
||||
traps. First, the bind mount's source directory must *exist* — podman refuses
|
||||
the container with `Error: statfs /home/dank/models/halogen: no such file or
|
||||
directory` and the unit loops on `Restart=on-failure`; `mkdir -p` the empty
|
||||
directory and the fetch starts on the next start. Second, "refresh" only means
|
||||
"fetch what the Hub has now": on 2026-09-16 the re-fetch brought newer sidecars
|
||||
(`overlay.hgn` 2,477,677,120 → **2,572,466,560** bytes, `overlay-speed.hgn`
|
||||
2,383,306,048 → **2,478,095,488**) while the checkpoint returned
|
||||
byte-identical in size (124,068,083,904) — so an update can live entirely in
|
||||
the sidecars, and equal file sizes are not proof of an unchanged repo.
|
||||
|
||||
`overlay-speed.hgn` (2.31 GiB on the Hub; 2.22 GiB before the 2026-09-16
|
||||
refresh) comes along with the repo but is unused: it trades the `o_proj`
|
||||
calibration for about 2 % of decode, and the quality sidecar is the default for
|
||||
a reason.
|
||||
|
||||
## Deployed flags
|
||||
|
||||
|
||||
@@ -321,3 +321,31 @@ Operational notes for the next run:
|
||||
the reboot: the same model, a more fragmented host afterwards — the engine
|
||||
reported 240 free contiguous 2 MiB blocks against 475 earlier). Both endpoints
|
||||
answered `/health` and served `/v1/models` under their names.
|
||||
|
||||
|
||||
## Smaller-quant check: Q4_K_S against the recorded Q4_K_M numbers
|
||||
|
||||
Same host, same `build-dual`, same test shapes (`-r 1`, f16 KV, flash-attn
|
||||
auto, `-d 65536` for depth). The repo ships i1-Q4_K_S as three `.partNofM`
|
||||
fragments (35.00 + 35.00 + 34.09 GiB) to concatenate — 104.09 GiB against
|
||||
110.97 for the Q4_K_M payload the earlier numbers came from.
|
||||
|
||||
| test | Q4_K_M | Q4_K_S | Δ |
|
||||
|---|---|---|---|
|
||||
| A\* iGPU-only pp512 / pp16384 / tg64 | 456.7 / 403.1 / 32.64 | **466.4 / 409.6 / 33.50** | +2.1 / +1.6 / +2.6 % |
|
||||
| A\* iGPU-only pp512 / tg32 @64k | 231.8 / 25.77 | 232.0 / **26.47** | ~0 / +2.7 % |
|
||||
| CUDA0/Vulkan1 split pp512 / tg32 @64k | 268.5 / 27.23 | 268.0 / **28.35** | ~0 / +4.1 % |
|
||||
| Vulkan1/Vulkan0 split pp512 / tg32 @64k | 224.3 / 27.55 | **233.6 / 28.11** | +4.2 / +2.0 % |
|
||||
|
||||
Memory sampled through the run (`/proc/meminfo`, `mem_info_gtt_used` and
|
||||
`nvidia-smi`, every 15 s): peak 3090 VRAM **13,466 MiB** (the CUDA-side split),
|
||||
peak iGPU GTT **71.6 GiB** (A\*), and host RAM bottomed at **10.0 GiB
|
||||
available** during A\* — against ~5 GiB for the Q4_K_M A\* run, so the 6.9 GiB
|
||||
smaller file bought roughly that much headroom.
|
||||
|
||||
**Read it as a memory lever with a small speed dividend**: 2–4 % for 6.9 GiB.
|
||||
Nothing structural moves — `-ot` still gates per-tensor placement and a layer
|
||||
split still cannot consolidate KV on the 3090. If the aim were to *deploy*
|
||||
mainline on this iGPU, `i1-IQ4_XS` (90.78 GiB, single file, sha256
|
||||
`b4787265…d0574f9d`) is the rung that changes the memory arithmetic and q8_0 KV
|
||||
is the lever that changes depth bandwidth; Q4_K_S is the conservative middle.
|
||||
Reference in New Issue
Block a user