model cache cleared, re-fetched, and the update turned out to be real

Operator instruction: purge the cache so no run can serve a stale payload. The two
live payloads were cleared and then re-fetched rather than left absent — deleting
them without a re-fetch means neither endpoint starts.

halogen's re-fetch proved the point: its sidecars had been stale *in place*
(overlay.hgn 2,477,677,120 -> 2,572,466,560 bytes, overlay-speed.hgn
2,383,306,048 -> 2,478,095,488) while the 115.55 GiB checkpoint returned
byte-identical. frank-small's 27B and mmproj re-fetched to Hub-matching hashes,
unchanged.

Trap recorded: podman refuses a bind mount whose source directory is missing
(statfs ... no such file or directory), so frank-large needs /models to exist
*empty* for its first-start fetch to run.

Q4_K_S against the recorded Q4_K_M numbers: +2.1/+1.6/+2.6 % (iGPU-only pp512 /
pp16384 / tg64), +2.7 % tg32@64k, +4.1 %/+2.0 % for the two layer-split configs,
~0 on both pp512@64k, host RAM bottoming at 10.0 GiB against ~5 GiB. A memory
lever with a small speed dividend. IQ4_XS (97,473,155,360 bytes, sha256
b4787265...d0574f9d) is pulled and verified for the next comparison.

Also repairs an AGENTS.md edit that landed in the wrong section: the peer
session's commits shifted that file between my read and my write, so the
frank-large text briefly split the Model Config section.

AGENTS.md

@@ -264,17 +264,26 @@ of 2026-09-16 the image path is `frank-small` (`input: [text, image]`), and
 tower. There is **no audio** input anywhere in the fleet: that capability left
 with the FLM NPU endpoint on 2026-09-11.
 
-## Active Services
-
 ### frank-large — halogen-flash-server (Dankclaw, ACTIVE)
 
 halogen-flash-server 0.5.8 as a **rootless Quadlet again since 2026-09-16**
 (`dankclaw/frank-large/quadlet/frank-large.container`), serving
-`peonist-ai/halogen-qwen3.8-flash-next` (115.55 GiB w4b `.hgn` checkpoint + the
-2.31 GiB quality overlay) on port 7770 from the gfx1151 iGPU. Two conversations
-of 200,192 tokens resident (pool 400,384, 2 slots), greedy default, reasoning
-extracted to `reasoning_content`, and **no vision tower** — that moved to
-frank-small. It replaced ik_llama.cpp, which had replaced it for two days.
+`peonist-ai/halogen-qwen3.8-flash-next` (115.55 GiB w4b `.hgn` checkpoint + a
+quality overlay that re-fetched to 2.40 GiB on 2026-09-16) on port 7770 from
+the gfx1151 iGPU. Two conversations of 200,192 tokens resident (pool 400,384,
+2 slots), greedy default, reasoning extracted to `reasoning_content`, and **no
+vision tower** — that moved to frank-small. It replaced ik_llama.cpp, which had
+replaced it for two days.
+
+The engine **fetches its own weights on first start** (`HALOGEN_DOWNLOAD`, into
+the `/home/dank/models/halogen` bind mount): clearing that directory and
+restarting the unit is the supported way to pick up upstream updates, and it
+needs the directory to exist **empty** — a missing directory fails the
+container with `Error: statfs /home/dank/models/halogen: no such file or
+directory` and the unit loops on `Restart=on-failure` until `mkdir -p` brings
+it back. The sidecars are versioned in place, so a stale copy is invisible from
+the file name (both overlays were ~95 MB larger upstream than the copies this
+host had been serving).
 
 Measured boot ledger (2026-09-16, `HALOGEN_VERBOSE=1` ships in the unit):
 device budget `121 − 67.7 − 20 = 33.5 GiB` against a need of `~27.2 GiB`, so

serverAppConfigs-docs/LLM/halogen-flash-server.md

@@ -58,7 +58,7 @@ discovery, the `services.json` probe and the docs needed no rename.
 | file | size | role |
 |---|---|---|
 | `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint |
-| `qwen38-flash-next-w4b.overlay.hgn` | 2.31 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing |
+| `qwen38-flash-next-w4b.overlay.hgn` | 2.40 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing. **Refreshed 2026-09-16**: the sidecars are versioned *in place*, so a stale local copy is invisible from its file name — this one had read 2.31 GiB since it was first fetched while the Hub had moved on |
 | `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) |
 | `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount |
 
@@ -68,9 +68,23 @@ never re-downloads, and `huggingface_hub` resumes an interrupted pull. The
 mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in
 `/home/dank/models/halogen`.
 
-`overlay-speed.hgn` (2.22 GiB) comes along with the repo but is unused: it
-trades the `o_proj` calibration for about 2 % of decode, and the quality
-sidecar is the default for a reason.
+**Refreshing the weights.** Delete the mount's contents (or the whole
+directory, then recreate it **empty**) and start the unit: the entrypoint sees
+the checkpoint absent and re-fetches the repo, resuming if interrupted. Two
+traps. First, the bind mount's source directory must *exist* — podman refuses
+the container with `Error: statfs /home/dank/models/halogen: no such file or
+directory` and the unit loops on `Restart=on-failure`; `mkdir -p` the empty
+directory and the fetch starts on the next start. Second, "refresh" only means
+"fetch what the Hub has now": on 2026-09-16 the re-fetch brought newer sidecars
+(`overlay.hgn` 2,477,677,120 → **2,572,466,560** bytes, `overlay-speed.hgn`
+2,383,306,048 → **2,478,095,488**) while the checkpoint returned
+byte-identical in size (124,068,083,904) — so an update can live entirely in
+the sidecars, and equal file sizes are not proof of an unchanged repo.
+
+`overlay-speed.hgn` (2.31 GiB on the Hub; 2.22 GiB before the 2026-09-16
+refresh) comes along with the repo but is unused: it trades the `o_proj`
+calibration for about 2 % of decode, and the quality sidecar is the default for
+a reason.
 
 ## Deployed flags
 

serverAppConfigs-docs/LLM/x4-pair-experiment.md

@@ -321,3 +321,31 @@ Operational notes for the next run:
   the reboot: the same model, a more fragmented host afterwards — the engine
   reported 240 free contiguous 2 MiB blocks against 475 earlier). Both endpoints
   answered `/health` and served `/v1/models` under their names.
+
+
+## Smaller-quant check: Q4_K_S against the recorded Q4_K_M numbers
+
+Same host, same `build-dual`, same test shapes (`-r 1`, f16 KV, flash-attn
+auto, `-d 65536` for depth). The repo ships i1-Q4_K_S as three `.partNofM`
+fragments (35.00 + 35.00 + 34.09 GiB) to concatenate — 104.09 GiB against
+110.97 for the Q4_K_M payload the earlier numbers came from.
+
+| test | Q4_K_M | Q4_K_S | Δ |
+|---|---|---|---|
+| A\* iGPU-only pp512 / pp16384 / tg64 | 456.7 / 403.1 / 32.64 | **466.4 / 409.6 / 33.50** | +2.1 / +1.6 / +2.6 % |
+| A\* iGPU-only pp512 / tg32 @64k | 231.8 / 25.77 | 232.0 / **26.47** | ~0 / +2.7 % |
+| CUDA0/Vulkan1 split pp512 / tg32 @64k | 268.5 / 27.23 | 268.0 / **28.35** | ~0 / +4.1 % |
+| Vulkan1/Vulkan0 split pp512 / tg32 @64k | 224.3 / 27.55 | **233.6 / 28.11** | +4.2 / +2.0 % |
+
+Memory sampled through the run (`/proc/meminfo`, `mem_info_gtt_used` and
+`nvidia-smi`, every 15 s): peak 3090 VRAM **13,466 MiB** (the CUDA-side split),
+peak iGPU GTT **71.6 GiB** (A\*), and host RAM bottomed at **10.0 GiB
+available** during A\* — against ~5 GiB for the Q4_K_M A\* run, so the 6.9 GiB
+smaller file bought roughly that much headroom.
+
+**Read it as a memory lever with a small speed dividend**: 2–4 % for 6.9 GiB.
+Nothing structural moves — `-ot` still gates per-tensor placement and a layer
+split still cannot consolidate KV on the 3090. If the aim were to *deploy*
+mainline on this iGPU, `i1-IQ4_XS` (90.78 GiB, single file, sha256
+`b4787265…d0574f9d`) is the rung that changes the memory arithmetic and q8_0 KV
+is the lever that changes depth bandwidth; Q4_K_S is the conservative middle.

serverAppConfigs-docs/log.md

@@ -707,3 +707,44 @@ timestamp: 2026-09-14
 - Both endpoints restored and healthy: frank-large `400384 positions, ~27.2 GiB`
   against a 33.5 GiB budget (no clamp), frank-small 200 on 7771. Page: 
   [[x4-pair-experiment]] (rewritten).
+
+## 2026-09-16 (night) — model cache cleared on instruction, then re-fetched, and the update was real
+- **Operator instruction:** purge the model cache outright so no run can serve a
+  stale payload. Done with one-time `rm` authorisation, and the two *live*
+  payloads were cleared and immediately re-fetched — because deleting them
+  without a re-fetch means neither endpoint can start.
+- **Cleared:** the Uncensored i1-Q4_K_M payload (112G), the Q4_K_S part-fragments
+  (105G), the retired kingjones STRIX and agentionai ROCmFP4 payloads (114G +
+  863M), an unreferenced Qwen3.5 vision dir (8.8G), and all of `~/.AGENTTRASH`
+  including the ik part-files (119G). Disk went **298 GB → 895 GB free**; 814 GB
+  after re-fetching what the services need.
+- **The re-fetch produced genuine updates.** halogen's sidecars had been stale —
+  `overlay.hgn` 2,477,677,120 → **2,572,466,560** B, `overlay-speed.hgn`
+  2,383,306,048 → **2,478,095,488** B — while `w4b.hgn` came back the same size
+  (124,068,083,904). So "clear and re-pull" *is* an update mechanism here, and
+  the sidecars are versioned **in place**: a stale copy is invisible from the
+  file name. frank-small's 27B + mmproj re-fetched to hashes matching the Hub
+  exactly and are unchanged.
+- **Trap:** podman refuses a bind mount whose source directory is missing
+  (`Error: statfs /home/dank/models/halogen: no such file or directory`), so
+  frank-large looped on `Restart=on-failure` until `mkdir -p` recreated the
+  **empty** directory — then the engine fetched the repo itself
+  (`HALOGEN_DOWNLOAD`), which is the documented refresh path.
+- **Q4_K_S vs Q4_K_M, measured on the same host and build:** +2.1 / +1.6 / +2.6 %
+  (iGPU-only pp512 / pp16384 / tg64), +2.7 % tg32@64k, +4.1 % and +2.0 % for the
+  two layer-split configs, ~0 on both pp512@64k. Host RAM bottomed at **10.0 GiB
+  available** in A\* against ~5 GiB for Q4_K_M. A memory lever with a small speed
+  dividend — nothing structural moves.
+- **IQ4_XS pulled in parallel with the bench**, as asked: 97,473,155,360 bytes,
+  sha256 `b4787265…d0574f9d`, matching the Hub's LFS digest. Kept as the fresh
+  payload for the next comparison; Q4_K_S was cleared once the bench was done.
+- **Held back deliberately:** `models/checkpoints` (23 GB of Flux 2 Klein — no
+  unit mounts it, but its upstream is licence-gated, so it is not mine to
+  destroy) plus the small tool dirs under `models/` and `models/logs`, which is
+  the evidence for this page.
+- **Verified after the refresh:** `/v1/models` correct on 7770/7771, real
+  completions returning `pong` on both, halogen's ledger `68.0 GiB weights
+  locked / 11.0 GiB KV pool`, 2 slots over one 400384-position pool, and **307**
+  contiguous 2 MiB blocks left (240 before the refresh).
+- Runbook: [[halogen-flash-server]] gains a "Refreshing the weights" note; the
+  quant comparison is appended to [[x4-pair-experiment]].