docs(vault): de-stale the quant and swap method pages
Parent-written (scout summaries were corroborated against the files, not trusted): every restatement of an endpoint shape now cites the endpoint page, and four claims contradicted by today's fleet state are corrected. - Quantization Production: the challenger-contract line quoted the retired compaction-hub window; now cites [[frank-zealot]]'s Current shape. The 3090 worked example is era-tagged. Sources listed [[frank-dragoon]] three times. - Quantization by Hardware Architecture: said the 3090 lives on Dankserver (it moved to Dankclaw 2026-09-16; the Eras table owns the move) and ranked the retired Ling-3.0 Quark INT4 service as the ACTIVE Dankclaw workload. The active service is [[frank-dragoon]] on a closed `.hgn` format. - GGUF Compatibility: Related claimed the ftype-115 ROCmFPX file was "in production"; that is an era now, so it points at [[kingjones-q8agent-rocmfpx]]. - Model Swap and Rollback: dropped the fragment byte-count ledger (AGENTS.md owns it), the Ampere FP8 hardware claim ([[Quantization by Hardware Architecture]] owns it), halogen's HTTP surface ([[frank-dragoon]]'s wire contract) and three brittle unit-file line refs (accepted loss: line numbers are derivable, the root-unit law stays). Fixed the per-slot example, which described the retired 3-slot arithmetic, and a Related bullet that called [[frank-archon]] retired while linking the same page twice. Census over the diff: 1 orphan line, the deleted unit line-number refs above.
This commit is contained in:
Frank
@@ -119,5 +119,6 @@ re-download.
|
||||
|
||||
- [[Llama Cpp Runtimes]] — which fork ships which custom types
|
||||
- [[Llama Cpp Code Diff]] — the enum tables and code-level differences
|
||||
- [[frank-dragoon]] — the ftype-115 lock-in in production
|
||||
* [[kingjones-q8agent-rocmfpx]] — the era that served the ftype-115 file, and
|
||||
[[frank-dragoon]] for what runs there now
|
||||
- [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads
|
||||
|
||||
@@ -40,9 +40,9 @@ curl -s <health> # then a REAL completion
|
||||
```
|
||||
|
||||
Deploy caveat: "healthy after Ns" = systemd ACTIVE, not model LOADED —
|
||||
confirm with a real `/v1/chat/completions`. RTX 3090 (Ampere) has no
|
||||
FP8/FP4 tensor cores: FP8 weights run via upcast but must still fit; choose
|
||||
quant by size fit, then accept the compute path the card supports. Before
|
||||
confirm with a real `/v1/chat/completions`. On Ampere an FP8 file buys memory
|
||||
and no compute ([[Quantization by Hardware Architecture]]): choose the quant by
|
||||
size fit, then accept the compute path the card supports. Before
|
||||
choosing a smaller quant, state WHICH of the two you are buying — headroom,
|
||||
or a design change (a smaller quant does not change what the runtime can
|
||||
express; per-tensor cross-device placement stays refused under
|
||||
@@ -60,8 +60,8 @@ express; per-tensor cross-device placement stays refused under
|
||||
## 3. Fetch and verify (HF payloads)
|
||||
|
||||
**Inventory by QUANT NAME, never by extension** — large quants ship as
|
||||
`.partNofM` fragments (Q4_K_M = 37.00+37.00+36.97 GiB), so an
|
||||
`endswith(".gguf")` filter makes Q4_K_S look nonexistent. Check both
|
||||
`.partNofM` fragments, so an `endswith(".gguf")` filter makes a whole rung look
|
||||
nonexistent (AGENTS.md carries the worked fragment ledger). Check both
|
||||
mradermacher repos (`…-i1-GGUF` imatrix, `…-GGUF` plain) and the model's own
|
||||
repo; the tree API's `blobs=true` gives `lfs.oid` per file. Gated mirrors
|
||||
return 403 even with a valid token until the account accepts the gate —
|
||||
@@ -150,7 +150,7 @@ recoverable, drift is not.
|
||||
entry in `serverAppConfigs-docs/log.md`.
|
||||
2. `~/.omp/agent/models.yml`: `contextWindow` must equal the REAL per-slot
|
||||
context — llama.cpp logs `n_ctx_slot` per prompt (`--parallel N` divides
|
||||
the pool; e.g. 262144 native / 2 slots = 131072 per slot); vLLM reports
|
||||
the pool; e.g. a 524288 pool over 2 slots = 262144 per slot); vLLM reports
|
||||
`max-model-len` and its concurrency line. Overselling causes silent
|
||||
client-side overflow or a discovered-wall 400. Also `maxTokens` vs the
|
||||
engine's cap, `input` capabilities (text/image), and
|
||||
@@ -252,12 +252,10 @@ small/fast + image roles at the new endpoint; move the vision role WITH the
|
||||
vision tower (one owner per capability). `shared/homepage/services.json`: a
|
||||
card per endpoint, probed from the monitor's own host. Prometheus: only add
|
||||
a job if the runtime actually exposes Prometheus text — and re-measure rather
|
||||
than repeat, because that answer moves. This bullet once said halogen serves
|
||||
`/health`, `/cache`, `/v1/*` "and nothing else"; measured 2026-10-18 on
|
||||
Dankclaw, `curl -s http://127.0.0.1:7770/metrics` returns Prometheus text
|
||||
(`# HELP llamacpp:prompt_tokens_total …`, content-type
|
||||
`text/plain; version=0.0.4`) while an invented path 404s — so the surface has
|
||||
grown and a scrape target is expressible again.
|
||||
than repeat, because that answer moves: halogen's HTTP surface has grown twice
|
||||
since this page was written, and the endpoint list that matters for a scrape
|
||||
target — including whether `/metrics` answers — is [[frank-dragoon]]'s wire
|
||||
contract.
|
||||
|
||||
**4. Cutover, then verify by driving it.** `deploy.sh <target> --service
|
||||
<large> --service <small>` (order matters); per endpoint: `/health`,
|
||||
@@ -265,10 +263,10 @@ grown and a scrape target is expressible again.
|
||||
`reasoning_content`, a `tools` request returns `tool_calls`, an image
|
||||
request is answered (or refused with a named reason), the runtime's own
|
||||
speculation counters move, and the unit is `enabled` and `active`. All three LLM
|
||||
endpoints are **root** units today (`User=dank` + `WantedBy=multi-user.target` —
|
||||
`scripts/frank-archon.service:30,71`, `scripts/frank-dragoon.service:8,31`,
|
||||
`scripts/frank-zealot.service:8,26`), so linger is not what gates them; linger
|
||||
matters for rootless quadlets. Read the engine's boot ledger for a silent clamp. Finally spend one
|
||||
endpoints are **root** units today (`scripts/frank-archon.service`,
|
||||
`scripts/frank-dragoon.service`, `scripts/frank-zealot.service`), so linger is
|
||||
not what gates them; linger matters for rootless quadlets. Read the engine's boot
|
||||
ledger for a silent clamp. Finally spend one
|
||||
real request through the harness per changed role — routing labels lie; the
|
||||
servers' own journals do not.
|
||||
|
||||
@@ -285,4 +283,5 @@ predecessor "stopped" (the port race above).
|
||||
- [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
|
||||
- [[LLM Bench Windows and Measurement]] — measuring the new deployment
|
||||
- [[frank-archon]] — the retired tcclaviger vLLM endpoint (2026-09-29 → 10-05); the live Dankserver endpoint is [[frank-archon]]
|
||||
* [[frank-archon]] — the live Dankserver endpoint; the tcclaviger vLLM era
|
||||
(2026-09-29 → 10-05) is a row in its Eras table
|
||||
|
||||
@@ -147,12 +147,14 @@ cut to GPTQ-w4 for the 3090 is a bench experiment that needs no new
|
||||
hardware; EXL3 (`convert.py` on the 3090, exllama serve) is the
|
||||
quality-per-bit extreme of the same menu.
|
||||
Caveats before any swap: the serving contract (OpenAI server,
|
||||
`reasoning_content`, vision mmproj, compaction-hub window
|
||||
200,704 ×2 slots) must be met by the challenger runtime, and the
|
||||
24 GB card runs ONE server — benching follows the stop-and-mask law.
|
||||
`reasoning_content`, vision mmproj, and the compaction-hub window the endpoint
|
||||
declares in its Current shape — [[frank-zealot]]) must be met by the challenger
|
||||
runtime, and the 24 GB card runs ONE server — benching follows the stop-and-mask
|
||||
law.
|
||||
|
||||
#### Worked example — the frank-archon w4dn8 recipe ported to the 3090
|
||||
(audit 2026-10-08, target `Qwen3.6-35B-A3B`)
|
||||
(audit 2026-10-08, target `Qwen3.6-35B-A3B`; sized at that date's hub shape —
|
||||
the window history is [[frank-zealot]]'s Eras table)
|
||||
**The methodology ports; the encoding must be translated.** llama.cpp
|
||||
decodes no GPTQ — the recipe re-expresses on GGUF-native terms:
|
||||
imatrix on our corpora (replaces the radiance grams; runs the 65 GiB
|
||||
@@ -253,8 +255,6 @@ top-8**, moe_intermediate 512, hidden 2048, MTP ×1, 262k ctx, vocab
|
||||
- Repo: `scripts/convert-swift15-w4dn8.sh`,
|
||||
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`,
|
||||
`scripts/kld-gate-swift15.sh`, `scripts/build-llama-rdna4.sh`,
|
||||
`scripts/build-kairic-edge-gfx1151.sh`, [[frank-dragoon]],
|
||||
[[frank-dragoon]], [[qwen3.8-strix-optimizations]],
|
||||
[[frank-archon]],
|
||||
[[frank-zealot]],
|
||||
[[frank-dragoon]], [[GGUF Compatibility]].
|
||||
`scripts/build-kairic-edge-gfx1151.sh`; endpoints [[frank-archon]],
|
||||
[[frank-dragoon]], [[frank-zealot]]; method [[qwen3.8-strix-optimizations]]
|
||||
and [[GGUF Compatibility]].
|
||||
|
||||
@@ -108,9 +108,10 @@ Use ROCmFP4 only with the ROCmFPX fork of llama.cpp. Upstream llama.cpp cannot r
|
||||
|
||||
## Choices for This Project
|
||||
|
||||
### Dankserver (RTX 3090, 24 GB, Ampere)
|
||||
### The RTX 3090 (24 GB, Ampere) — frank-zealot's card
|
||||
|
||||
The RTX 3090 has no native FP8 or FP4 tensor cores. Use:
|
||||
The 3090 has no native FP8 or FP4 tensor cores. It has sat in Dankclaw since
|
||||
2026-09-16; [[frank-zealot]]'s Eras table records the move off Dankserver. Use:
|
||||
|
||||
| Ranking | Format | Why |
|
||||
|---------|--------|-----|
|
||||
@@ -121,13 +122,17 @@ The RTX 3090 has no native FP8 or FP4 tensor cores. Use:
|
||||
|
||||
### Dankclaw (Strix Halo gfx1151, RDNA 3.5)
|
||||
|
||||
RDNA 3.5 has no native FP8 tensor cores. The APU shares memory. The active Ling-3.0-flash service runs vLLM-ROCm with the AMD Quark INT4 (W4A16) format (the jcbtc CIRU Strix-native bundle). INT4 weights keep the 124B model inside the 96 GiB frame buffer with a full-precision KV cache.
|
||||
RDNA 3.5 has no native FP8 tensor cores and the APU shares its memory, so the
|
||||
GPU budget is the GTT aperture and every live figure on the box is
|
||||
[[Dankclaw System Specs]]. What serves here today is [[frank-dragoon]]: a
|
||||
vendor `.hgn` checkpoint, a format this page cannot rank because no public
|
||||
quantizer exists for it ([[Quantization Production]]).
|
||||
|
||||
| Ranking | Format | Why |
|
||||
|---------|--------|-----|
|
||||
| 1 | INT4 W4A16 (Quark/vLLM) | active; fits 96 GiB; native ROCm kernels; concurrency |
|
||||
| 2 | MXFP4 (Quark) | same-bit, higher accuracy; beta on gfx1151 (A/B experiment) |
|
||||
| 3 | GGUF Q4_K_M / Q8_0 | works with upstream llama.cpp only |
|
||||
| 1 | `.hgn` (deployed) | the halogen engine's own format; production door is closed |
|
||||
| 2 | GGUF K/I quants, K_P blends | the only self-produced option on this card ([[Quantization Production]]) |
|
||||
| 3 | INT4 W4A16 (Quark) / MXFP4 (Quark) | the retired Ling-3.0-flash vLLM era, measured under the firmware-VGM frame buffer: [[ling-3.0-flash-trial]] |
|
||||
| 4 | FP8 (vLLM) | software emulation; memory savings only; no compute win |
|
||||
|
||||
## See Also
|
||||
|
||||
Reference in New Issue
Block a user