diff --git a/AGENTS.md b/AGENTS.md index 2d0263b..43e0953 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -264,17 +264,26 @@ of 2026-09-16 the image path is `frank-small` (`input: [text, image]`), and tower. There is **no audio** input anywhere in the fleet: that capability left with the FLM NPU endpoint on 2026-09-11. -## Active Services - ### frank-large — halogen-flash-server (Dankclaw, ACTIVE) halogen-flash-server 0.5.8 as a **rootless Quadlet again since 2026-09-16** (`dankclaw/frank-large/quadlet/frank-large.container`), serving -`peonist-ai/halogen-qwen3.8-flash-next` (115.55 GiB w4b `.hgn` checkpoint + the -2.31 GiB quality overlay) on port 7770 from the gfx1151 iGPU. Two conversations -of 200,192 tokens resident (pool 400,384, 2 slots), greedy default, reasoning -extracted to `reasoning_content`, and **no vision tower** — that moved to -frank-small. It replaced ik_llama.cpp, which had replaced it for two days. +`peonist-ai/halogen-qwen3.8-flash-next` (115.55 GiB w4b `.hgn` checkpoint + a +quality overlay that re-fetched to 2.40 GiB on 2026-09-16) on port 7770 from +the gfx1151 iGPU. Two conversations of 200,192 tokens resident (pool 400,384, +2 slots), greedy default, reasoning extracted to `reasoning_content`, and **no +vision tower** — that moved to frank-small. It replaced ik_llama.cpp, which had +replaced it for two days. + +The engine **fetches its own weights on first start** (`HALOGEN_DOWNLOAD`, into +the `/home/dank/models/halogen` bind mount): clearing that directory and +restarting the unit is the supported way to pick up upstream updates, and it +needs the directory to exist **empty** — a missing directory fails the +container with `Error: statfs /home/dank/models/halogen: no such file or +directory` and the unit loops on `Restart=on-failure` until `mkdir -p` brings +it back. The sidecars are versioned in place, so a stale copy is invisible from +the file name (both overlays were ~95 MB larger upstream than the copies this +host had been serving). Measured boot ledger (2026-09-16, `HALOGEN_VERBOSE=1` ships in the unit): device budget `121 − 67.7 − 20 = 33.5 GiB` against a need of `~27.2 GiB`, so diff --git a/serverAppConfigs-docs/LLM/halogen-flash-server.md b/serverAppConfigs-docs/LLM/halogen-flash-server.md index c4bae0c..73d3192 100644 --- a/serverAppConfigs-docs/LLM/halogen-flash-server.md +++ b/serverAppConfigs-docs/LLM/halogen-flash-server.md @@ -58,7 +58,7 @@ discovery, the `services.json` probe and the docs needed no rename. | file | size | role | |---|---|---| | `qwen38-flash-next-w4b.hgn` | 115.55 GiB | the checkpoint | -| `qwen38-flash-next-w4b.overlay.hgn` | 2.31 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing | +| `qwen38-flash-next-w4b.overlay.hgn` | 2.40 GiB | **quality sidecar** — picked up automatically; the engine warns at startup if it is missing. **Refreshed 2026-09-16**: the sidecars are versioned *in place*, so a stale local copy is invisible from its file name — this one had read 2.31 GiB since it was first fetched while the Hub had moved on | | `qwen38-flash-next-vision.hgn` | 0.84 GiB | vision tower (see below) | | `tokenizer/` | ~21 MB | flat tokenizer dir; found beside the checkpoint, so no extra mount | @@ -68,9 +68,23 @@ never re-downloads, and `huggingface_hub` resumes an interrupted pull. The mount must be read-write for this. Roughly 127 GB; on Dankclaw it lands in `/home/dank/models/halogen`. -`overlay-speed.hgn` (2.22 GiB) comes along with the repo but is unused: it -trades the `o_proj` calibration for about 2 % of decode, and the quality -sidecar is the default for a reason. +**Refreshing the weights.** Delete the mount's contents (or the whole +directory, then recreate it **empty**) and start the unit: the entrypoint sees +the checkpoint absent and re-fetches the repo, resuming if interrupted. Two +traps. First, the bind mount's source directory must *exist* — podman refuses +the container with `Error: statfs /home/dank/models/halogen: no such file or +directory` and the unit loops on `Restart=on-failure`; `mkdir -p` the empty +directory and the fetch starts on the next start. Second, "refresh" only means +"fetch what the Hub has now": on 2026-09-16 the re-fetch brought newer sidecars +(`overlay.hgn` 2,477,677,120 → **2,572,466,560** bytes, `overlay-speed.hgn` +2,383,306,048 → **2,478,095,488**) while the checkpoint returned +byte-identical in size (124,068,083,904) — so an update can live entirely in +the sidecars, and equal file sizes are not proof of an unchanged repo. + +`overlay-speed.hgn` (2.31 GiB on the Hub; 2.22 GiB before the 2026-09-16 +refresh) comes along with the repo but is unused: it trades the `o_proj` +calibration for about 2 % of decode, and the quality sidecar is the default for +a reason. ## Deployed flags diff --git a/serverAppConfigs-docs/LLM/x4-pair-experiment.md b/serverAppConfigs-docs/LLM/x4-pair-experiment.md index 3a20c48..798136f 100644 --- a/serverAppConfigs-docs/LLM/x4-pair-experiment.md +++ b/serverAppConfigs-docs/LLM/x4-pair-experiment.md @@ -321,3 +321,31 @@ Operational notes for the next run: the reboot: the same model, a more fragmented host afterwards — the engine reported 240 free contiguous 2 MiB blocks against 475 earlier). Both endpoints answered `/health` and served `/v1/models` under their names. + + +## Smaller-quant check: Q4_K_S against the recorded Q4_K_M numbers + +Same host, same `build-dual`, same test shapes (`-r 1`, f16 KV, flash-attn +auto, `-d 65536` for depth). The repo ships i1-Q4_K_S as three `.partNofM` +fragments (35.00 + 35.00 + 34.09 GiB) to concatenate — 104.09 GiB against +110.97 for the Q4_K_M payload the earlier numbers came from. + +| test | Q4_K_M | Q4_K_S | Δ | +|---|---|---|---| +| A\* iGPU-only pp512 / pp16384 / tg64 | 456.7 / 403.1 / 32.64 | **466.4 / 409.6 / 33.50** | +2.1 / +1.6 / +2.6 % | +| A\* iGPU-only pp512 / tg32 @64k | 231.8 / 25.77 | 232.0 / **26.47** | ~0 / +2.7 % | +| CUDA0/Vulkan1 split pp512 / tg32 @64k | 268.5 / 27.23 | 268.0 / **28.35** | ~0 / +4.1 % | +| Vulkan1/Vulkan0 split pp512 / tg32 @64k | 224.3 / 27.55 | **233.6 / 28.11** | +4.2 / +2.0 % | + +Memory sampled through the run (`/proc/meminfo`, `mem_info_gtt_used` and +`nvidia-smi`, every 15 s): peak 3090 VRAM **13,466 MiB** (the CUDA-side split), +peak iGPU GTT **71.6 GiB** (A\*), and host RAM bottomed at **10.0 GiB +available** during A\* — against ~5 GiB for the Q4_K_M A\* run, so the 6.9 GiB +smaller file bought roughly that much headroom. + +**Read it as a memory lever with a small speed dividend**: 2–4 % for 6.9 GiB. +Nothing structural moves — `-ot` still gates per-tensor placement and a layer +split still cannot consolidate KV on the 3090. If the aim were to *deploy* +mainline on this iGPU, `i1-IQ4_XS` (90.78 GiB, single file, sha256 +`b4787265…d0574f9d`) is the rung that changes the memory arithmetic and q8_0 KV +is the lever that changes depth bandwidth; Q4_K_S is the conservative middle. \ No newline at end of file diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 4d776c1..83fa4ac 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -707,3 +707,44 @@ timestamp: 2026-09-14 - Both endpoints restored and healthy: frank-large `400384 positions, ~27.2 GiB` against a 33.5 GiB budget (no clamp), frank-small 200 on 7771. Page: [[x4-pair-experiment]] (rewritten). + +## 2026-09-16 (night) — model cache cleared on instruction, then re-fetched, and the update was real +- **Operator instruction:** purge the model cache outright so no run can serve a + stale payload. Done with one-time `rm` authorisation, and the two *live* + payloads were cleared and immediately re-fetched — because deleting them + without a re-fetch means neither endpoint can start. +- **Cleared:** the Uncensored i1-Q4_K_M payload (112G), the Q4_K_S part-fragments + (105G), the retired kingjones STRIX and agentionai ROCmFP4 payloads (114G + + 863M), an unreferenced Qwen3.5 vision dir (8.8G), and all of `~/.AGENTTRASH` + including the ik part-files (119G). Disk went **298 GB → 895 GB free**; 814 GB + after re-fetching what the services need. +- **The re-fetch produced genuine updates.** halogen's sidecars had been stale — + `overlay.hgn` 2,477,677,120 → **2,572,466,560** B, `overlay-speed.hgn` + 2,383,306,048 → **2,478,095,488** B — while `w4b.hgn` came back the same size + (124,068,083,904). So "clear and re-pull" *is* an update mechanism here, and + the sidecars are versioned **in place**: a stale copy is invisible from the + file name. frank-small's 27B + mmproj re-fetched to hashes matching the Hub + exactly and are unchanged. +- **Trap:** podman refuses a bind mount whose source directory is missing + (`Error: statfs /home/dank/models/halogen: no such file or directory`), so + frank-large looped on `Restart=on-failure` until `mkdir -p` recreated the + **empty** directory — then the engine fetched the repo itself + (`HALOGEN_DOWNLOAD`), which is the documented refresh path. +- **Q4_K_S vs Q4_K_M, measured on the same host and build:** +2.1 / +1.6 / +2.6 % + (iGPU-only pp512 / pp16384 / tg64), +2.7 % tg32@64k, +4.1 % and +2.0 % for the + two layer-split configs, ~0 on both pp512@64k. Host RAM bottomed at **10.0 GiB + available** in A\* against ~5 GiB for Q4_K_M. A memory lever with a small speed + dividend — nothing structural moves. +- **IQ4_XS pulled in parallel with the bench**, as asked: 97,473,155,360 bytes, + sha256 `b4787265…d0574f9d`, matching the Hub's LFS digest. Kept as the fresh + payload for the next comparison; Q4_K_S was cleared once the bench was done. +- **Held back deliberately:** `models/checkpoints` (23 GB of Flux 2 Klein — no + unit mounts it, but its upstream is licence-gated, so it is not mine to + destroy) plus the small tool dirs under `models/` and `models/logs`, which is + the evidence for this page. +- **Verified after the refresh:** `/v1/models` correct on 7770/7771, real + completions returning `pong` on both, halogen's ledger `68.0 GiB weights + locked / 11.0 GiB KV pool`, 2 slots over one 400384-position pool, and **307** + contiguous 2 MiB blocks left (240 before the refresh). +- Runbook: [[halogen-flash-server]] gains a "Refreshing the weights" note; the + quant comparison is appended to [[x4-pair-experiment]].