From a518fc7c22e16d992dc5d6a25b487c4b088e0e34 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 19 Oct 2026 01:52:30 -0700 Subject: [PATCH] docs(vault): de-stale the quant and swap method pages Parent-written (scout summaries were corroborated against the files, not trusted): every restatement of an endpoint shape now cites the endpoint page, and four claims contradicted by today's fleet state are corrected. - Quantization Production: the challenger-contract line quoted the retired compaction-hub window; now cites [[frank-zealot]]'s Current shape. The 3090 worked example is era-tagged. Sources listed [[frank-dragoon]] three times. - Quantization by Hardware Architecture: said the 3090 lives on Dankserver (it moved to Dankclaw 2026-09-16; the Eras table owns the move) and ranked the retired Ling-3.0 Quark INT4 service as the ACTIVE Dankclaw workload. The active service is [[frank-dragoon]] on a closed `.hgn` format. - GGUF Compatibility: Related claimed the ftype-115 ROCmFPX file was "in production"; that is an era now, so it points at [[kingjones-q8agent-rocmfpx]]. - Model Swap and Rollback: dropped the fragment byte-count ledger (AGENTS.md owns it), the Ampere FP8 hardware claim ([[Quantization by Hardware Architecture]] owns it), halogen's HTTP surface ([[frank-dragoon]]'s wire contract) and three brittle unit-file line refs (accepted loss: line numbers are derivable, the root-unit law stays). Fixed the per-slot example, which described the retired 3-slot arithmetic, and a Related bullet that called [[frank-archon]] retired while linking the same page twice. Census over the diff: 1 orphan line, the deleted unit line-number refs above. --- .../LLM/GGUF Compatibility.md | 3 +- .../LLM/Model Swap and Rollback.md | 33 +++++++++---------- .../LLM/Quantization Production.md | 18 +++++----- .../Quantization by Hardware Architecture.md | 17 ++++++---- 4 files changed, 38 insertions(+), 33 deletions(-) diff --git a/serverAppConfigs-docs/LLM/GGUF Compatibility.md b/serverAppConfigs-docs/LLM/GGUF Compatibility.md index 3bbd537..aae7866 100644 --- a/serverAppConfigs-docs/LLM/GGUF Compatibility.md +++ b/serverAppConfigs-docs/LLM/GGUF Compatibility.md @@ -119,5 +119,6 @@ re-download. - [[Llama Cpp Runtimes]] — which fork ships which custom types - [[Llama Cpp Code Diff]] — the enum tables and code-level differences -- [[frank-dragoon]] — the ftype-115 lock-in in production +* [[kingjones-q8agent-rocmfpx]] — the era that served the ftype-115 file, and + [[frank-dragoon]] for what runs there now - [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads diff --git a/serverAppConfigs-docs/LLM/Model Swap and Rollback.md b/serverAppConfigs-docs/LLM/Model Swap and Rollback.md index 1095166..74c49bd 100644 --- a/serverAppConfigs-docs/LLM/Model Swap and Rollback.md +++ b/serverAppConfigs-docs/LLM/Model Swap and Rollback.md @@ -40,9 +40,9 @@ curl -s # then a REAL completion ``` Deploy caveat: "healthy after Ns" = systemd ACTIVE, not model LOADED — -confirm with a real `/v1/chat/completions`. RTX 3090 (Ampere) has no -FP8/FP4 tensor cores: FP8 weights run via upcast but must still fit; choose -quant by size fit, then accept the compute path the card supports. Before +confirm with a real `/v1/chat/completions`. On Ampere an FP8 file buys memory +and no compute ([[Quantization by Hardware Architecture]]): choose the quant by +size fit, then accept the compute path the card supports. Before choosing a smaller quant, state WHICH of the two you are buying — headroom, or a design change (a smaller quant does not change what the runtime can express; per-tensor cross-device placement stays refused under @@ -60,8 +60,8 @@ express; per-tensor cross-device placement stays refused under ## 3. Fetch and verify (HF payloads) **Inventory by QUANT NAME, never by extension** — large quants ship as -`.partNofM` fragments (Q4_K_M = 37.00+37.00+36.97 GiB), so an -`endswith(".gguf")` filter makes Q4_K_S look nonexistent. Check both +`.partNofM` fragments, so an `endswith(".gguf")` filter makes a whole rung look +nonexistent (AGENTS.md carries the worked fragment ledger). Check both mradermacher repos (`…-i1-GGUF` imatrix, `…-GGUF` plain) and the model's own repo; the tree API's `blobs=true` gives `lfs.oid` per file. Gated mirrors return 403 even with a valid token until the account accepts the gate — @@ -150,7 +150,7 @@ recoverable, drift is not. entry in `serverAppConfigs-docs/log.md`. 2. `~/.omp/agent/models.yml`: `contextWindow` must equal the REAL per-slot context — llama.cpp logs `n_ctx_slot` per prompt (`--parallel N` divides - the pool; e.g. 262144 native / 2 slots = 131072 per slot); vLLM reports + the pool; e.g. a 524288 pool over 2 slots = 262144 per slot); vLLM reports `max-model-len` and its concurrency line. Overselling causes silent client-side overflow or a discovered-wall 400. Also `maxTokens` vs the engine's cap, `input` capabilities (text/image), and @@ -252,12 +252,10 @@ small/fast + image roles at the new endpoint; move the vision role WITH the vision tower (one owner per capability). `shared/homepage/services.json`: a card per endpoint, probed from the monitor's own host. Prometheus: only add a job if the runtime actually exposes Prometheus text — and re-measure rather -than repeat, because that answer moves. This bullet once said halogen serves -`/health`, `/cache`, `/v1/*` "and nothing else"; measured 2026-10-18 on -Dankclaw, `curl -s http://127.0.0.1:7770/metrics` returns Prometheus text -(`# HELP llamacpp:prompt_tokens_total …`, content-type -`text/plain; version=0.0.4`) while an invented path 404s — so the surface has -grown and a scrape target is expressible again. +than repeat, because that answer moves: halogen's HTTP surface has grown twice +since this page was written, and the endpoint list that matters for a scrape +target — including whether `/metrics` answers — is [[frank-dragoon]]'s wire +contract. **4. Cutover, then verify by driving it.** `deploy.sh --service --service ` (order matters); per endpoint: `/health`, @@ -265,10 +263,10 @@ grown and a scrape target is expressible again. `reasoning_content`, a `tools` request returns `tool_calls`, an image request is answered (or refused with a named reason), the runtime's own speculation counters move, and the unit is `enabled` and `active`. All three LLM -endpoints are **root** units today (`User=dank` + `WantedBy=multi-user.target` — -`scripts/frank-archon.service:30,71`, `scripts/frank-dragoon.service:8,31`, -`scripts/frank-zealot.service:8,26`), so linger is not what gates them; linger -matters for rootless quadlets. Read the engine's boot ledger for a silent clamp. Finally spend one +endpoints are **root** units today (`scripts/frank-archon.service`, +`scripts/frank-dragoon.service`, `scripts/frank-zealot.service`), so linger is +not what gates them; linger matters for rootless quadlets. Read the engine's boot +ledger for a silent clamp. Finally spend one real request through the harness per changed role — routing labels lie; the servers' own journals do not. @@ -285,4 +283,5 @@ predecessor "stopped" (the port race above). - [[Llama Cpp Serving Sizing]] — pricing the shape once it loads - [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic - [[LLM Bench Windows and Measurement]] — measuring the new deployment -- [[frank-archon]] — the retired tcclaviger vLLM endpoint (2026-09-29 → 10-05); the live Dankserver endpoint is [[frank-archon]] +* [[frank-archon]] — the live Dankserver endpoint; the tcclaviger vLLM era + (2026-09-29 → 10-05) is a row in its Eras table diff --git a/serverAppConfigs-docs/LLM/Quantization Production.md b/serverAppConfigs-docs/LLM/Quantization Production.md index ca4c366..dae540b 100644 --- a/serverAppConfigs-docs/LLM/Quantization Production.md +++ b/serverAppConfigs-docs/LLM/Quantization Production.md @@ -147,12 +147,14 @@ cut to GPTQ-w4 for the 3090 is a bench experiment that needs no new hardware; EXL3 (`convert.py` on the 3090, exllama serve) is the quality-per-bit extreme of the same menu. Caveats before any swap: the serving contract (OpenAI server, -`reasoning_content`, vision mmproj, compaction-hub window -200,704 ×2 slots) must be met by the challenger runtime, and the -24 GB card runs ONE server — benching follows the stop-and-mask law. +`reasoning_content`, vision mmproj, and the compaction-hub window the endpoint +declares in its Current shape — [[frank-zealot]]) must be met by the challenger +runtime, and the 24 GB card runs ONE server — benching follows the stop-and-mask +law. #### Worked example — the frank-archon w4dn8 recipe ported to the 3090 -(audit 2026-10-08, target `Qwen3.6-35B-A3B`) +(audit 2026-10-08, target `Qwen3.6-35B-A3B`; sized at that date's hub shape — +the window history is [[frank-zealot]]'s Eras table) **The methodology ports; the encoding must be translated.** llama.cpp decodes no GPTQ — the recipe re-expresses on GGUF-native terms: imatrix on our corpora (replaces the radiance grams; runs the 65 GiB @@ -253,8 +255,6 @@ top-8**, moe_intermediate 512, hidden 2048, MTP ×1, 262k ctx, vocab - Repo: `scripts/convert-swift15-w4dn8.sh`, `scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`, `scripts/kld-gate-swift15.sh`, `scripts/build-llama-rdna4.sh`, - `scripts/build-kairic-edge-gfx1151.sh`, [[frank-dragoon]], - [[frank-dragoon]], [[qwen3.8-strix-optimizations]], - [[frank-archon]], - [[frank-zealot]], - [[frank-dragoon]], [[GGUF Compatibility]]. + `scripts/build-kairic-edge-gfx1151.sh`; endpoints [[frank-archon]], + [[frank-dragoon]], [[frank-zealot]]; method [[qwen3.8-strix-optimizations]] + and [[GGUF Compatibility]]. diff --git a/serverAppConfigs-docs/LLM/Quantization by Hardware Architecture.md b/serverAppConfigs-docs/LLM/Quantization by Hardware Architecture.md index 6b549cc..6f21f75 100644 --- a/serverAppConfigs-docs/LLM/Quantization by Hardware Architecture.md +++ b/serverAppConfigs-docs/LLM/Quantization by Hardware Architecture.md @@ -108,9 +108,10 @@ Use ROCmFP4 only with the ROCmFPX fork of llama.cpp. Upstream llama.cpp cannot r ## Choices for This Project -### Dankserver (RTX 3090, 24 GB, Ampere) +### The RTX 3090 (24 GB, Ampere) — frank-zealot's card -The RTX 3090 has no native FP8 or FP4 tensor cores. Use: +The 3090 has no native FP8 or FP4 tensor cores. It has sat in Dankclaw since +2026-09-16; [[frank-zealot]]'s Eras table records the move off Dankserver. Use: | Ranking | Format | Why | |---------|--------|-----| @@ -121,13 +122,17 @@ The RTX 3090 has no native FP8 or FP4 tensor cores. Use: ### Dankclaw (Strix Halo gfx1151, RDNA 3.5) -RDNA 3.5 has no native FP8 tensor cores. The APU shares memory. The active Ling-3.0-flash service runs vLLM-ROCm with the AMD Quark INT4 (W4A16) format (the jcbtc CIRU Strix-native bundle). INT4 weights keep the 124B model inside the 96 GiB frame buffer with a full-precision KV cache. +RDNA 3.5 has no native FP8 tensor cores and the APU shares its memory, so the +GPU budget is the GTT aperture and every live figure on the box is +[[Dankclaw System Specs]]. What serves here today is [[frank-dragoon]]: a +vendor `.hgn` checkpoint, a format this page cannot rank because no public +quantizer exists for it ([[Quantization Production]]). | Ranking | Format | Why | |---------|--------|-----| -| 1 | INT4 W4A16 (Quark/vLLM) | active; fits 96 GiB; native ROCm kernels; concurrency | -| 2 | MXFP4 (Quark) | same-bit, higher accuracy; beta on gfx1151 (A/B experiment) | -| 3 | GGUF Q4_K_M / Q8_0 | works with upstream llama.cpp only | +| 1 | `.hgn` (deployed) | the halogen engine's own format; production door is closed | +| 2 | GGUF K/I quants, K_P blends | the only self-produced option on this card ([[Quantization Production]]) | +| 3 | INT4 W4A16 (Quark) / MXFP4 (Quark) | the retired Ling-3.0-flash vLLM era, measured under the firmware-VGM frame buffer: [[ling-3.0-flash-trial]] | | 4 | FP8 (vLLM) | software emulation; memory savings only; no compute win | ## See Also