docs(vault): de-stale the quant and swap method pages
Parent-written (scout summaries were corroborated against the files, not trusted): every restatement of an endpoint shape now cites the endpoint page, and four claims contradicted by today's fleet state are corrected. - Quantization Production: the challenger-contract line quoted the retired compaction-hub window; now cites [[frank-zealot]]'s Current shape. The 3090 worked example is era-tagged. Sources listed [[frank-dragoon]] three times. - Quantization by Hardware Architecture: said the 3090 lives on Dankserver (it moved to Dankclaw 2026-09-16; the Eras table owns the move) and ranked the retired Ling-3.0 Quark INT4 service as the ACTIVE Dankclaw workload. The active service is [[frank-dragoon]] on a closed `.hgn` format. - GGUF Compatibility: Related claimed the ftype-115 ROCmFPX file was "in production"; that is an era now, so it points at [[kingjones-q8agent-rocmfpx]]. - Model Swap and Rollback: dropped the fragment byte-count ledger (AGENTS.md owns it), the Ampere FP8 hardware claim ([[Quantization by Hardware Architecture]] owns it), halogen's HTTP surface ([[frank-dragoon]]'s wire contract) and three brittle unit-file line refs (accepted loss: line numbers are derivable, the root-unit law stays). Fixed the per-slot example, which described the retired 3-slot arithmetic, and a Related bullet that called [[frank-archon]] retired while linking the same page twice. Census over the diff: 1 orphan line, the deleted unit line-number refs above.
This commit is contained in:
1 parent
9f1a4e4e51
commit
a518fc7c22
4 files changed
+38
-33
No files matched your search
@@ -119,5 +119,6 @@ re-download.
|
|||||||
|
|
||||||
- [[Llama Cpp Runtimes]] — which fork ships which custom types
|
- [[Llama Cpp Runtimes]] — which fork ships which custom types
|
||||||
- [[Llama Cpp Code Diff]] — the enum tables and code-level differences
|
- [[Llama Cpp Code Diff]] — the enum tables and code-level differences
|
||||||
- [[frank-dragoon]] — the ftype-115 lock-in in production
|
* [[kingjones-q8agent-rocmfpx]] — the era that served the ftype-115 file, and
|
||||||
|
[[frank-dragoon]] for what runs there now
|
||||||
- [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads
|
- [[Llama Cpp Serving Sizing]] — pricing what a payload costs once it loads
|
||||||
@@ -40,9 +40,9 @@ curl -s <health> # then a REAL completion
|
|||||||
```
|
```
|
||||||
|
|
||||||
Deploy caveat: "healthy after Ns" = systemd ACTIVE, not model LOADED —
|
Deploy caveat: "healthy after Ns" = systemd ACTIVE, not model LOADED —
|
||||||
confirm with a real `/v1/chat/completions`. RTX 3090 (Ampere) has no
|
confirm with a real `/v1/chat/completions`. On Ampere an FP8 file buys memory
|
||||||
FP8/FP4 tensor cores: FP8 weights run via upcast but must still fit; choose
|
and no compute ([[Quantization by Hardware Architecture]]): choose the quant by
|
||||||
quant by size fit, then accept the compute path the card supports. Before
|
size fit, then accept the compute path the card supports. Before
|
||||||
choosing a smaller quant, state WHICH of the two you are buying — headroom,
|
choosing a smaller quant, state WHICH of the two you are buying — headroom,
|
||||||
or a design change (a smaller quant does not change what the runtime can
|
or a design change (a smaller quant does not change what the runtime can
|
||||||
express; per-tensor cross-device placement stays refused under
|
express; per-tensor cross-device placement stays refused under
|
||||||
@@ -60,8 +60,8 @@ express; per-tensor cross-device placement stays refused under
|
|||||||
## 3. Fetch and verify (HF payloads)
|
## 3. Fetch and verify (HF payloads)
|
||||||
|
|
||||||
**Inventory by QUANT NAME, never by extension** — large quants ship as
|
**Inventory by QUANT NAME, never by extension** — large quants ship as
|
||||||
`.partNofM` fragments (Q4_K_M = 37.00+37.00+36.97 GiB), so an
|
`.partNofM` fragments, so an `endswith(".gguf")` filter makes a whole rung look
|
||||||
`endswith(".gguf")` filter makes Q4_K_S look nonexistent. Check both
|
nonexistent (AGENTS.md carries the worked fragment ledger). Check both
|
||||||
mradermacher repos (`…-i1-GGUF` imatrix, `…-GGUF` plain) and the model's own
|
mradermacher repos (`…-i1-GGUF` imatrix, `…-GGUF` plain) and the model's own
|
||||||
repo; the tree API's `blobs=true` gives `lfs.oid` per file. Gated mirrors
|
repo; the tree API's `blobs=true` gives `lfs.oid` per file. Gated mirrors
|
||||||
return 403 even with a valid token until the account accepts the gate —
|
return 403 even with a valid token until the account accepts the gate —
|
||||||
@@ -150,7 +150,7 @@ recoverable, drift is not.
|
|||||||
entry in `serverAppConfigs-docs/log.md`.
|
entry in `serverAppConfigs-docs/log.md`.
|
||||||
2. `~/.omp/agent/models.yml`: `contextWindow` must equal the REAL per-slot
|
2. `~/.omp/agent/models.yml`: `contextWindow` must equal the REAL per-slot
|
||||||
context — llama.cpp logs `n_ctx_slot` per prompt (`--parallel N` divides
|
context — llama.cpp logs `n_ctx_slot` per prompt (`--parallel N` divides
|
||||||
the pool; e.g. 262144 native / 2 slots = 131072 per slot); vLLM reports
|
the pool; e.g. a 524288 pool over 2 slots = 262144 per slot); vLLM reports
|
||||||
`max-model-len` and its concurrency line. Overselling causes silent
|
`max-model-len` and its concurrency line. Overselling causes silent
|
||||||
client-side overflow or a discovered-wall 400. Also `maxTokens` vs the
|
client-side overflow or a discovered-wall 400. Also `maxTokens` vs the
|
||||||
engine's cap, `input` capabilities (text/image), and
|
engine's cap, `input` capabilities (text/image), and
|
||||||
@@ -252,12 +252,10 @@ small/fast + image roles at the new endpoint; move the vision role WITH the
|
|||||||
vision tower (one owner per capability). `shared/homepage/services.json`: a
|
vision tower (one owner per capability). `shared/homepage/services.json`: a
|
||||||
card per endpoint, probed from the monitor's own host. Prometheus: only add
|
card per endpoint, probed from the monitor's own host. Prometheus: only add
|
||||||
a job if the runtime actually exposes Prometheus text — and re-measure rather
|
a job if the runtime actually exposes Prometheus text — and re-measure rather
|
||||||
than repeat, because that answer moves. This bullet once said halogen serves
|
than repeat, because that answer moves: halogen's HTTP surface has grown twice
|
||||||
`/health`, `/cache`, `/v1/*` "and nothing else"; measured 2026-10-18 on
|
since this page was written, and the endpoint list that matters for a scrape
|
||||||
Dankclaw, `curl -s http://127.0.0.1:7770/metrics` returns Prometheus text
|
target — including whether `/metrics` answers — is [[frank-dragoon]]'s wire
|
||||||
(`# HELP llamacpp:prompt_tokens_total …`, content-type
|
contract.
|
||||||
`text/plain; version=0.0.4`) while an invented path 404s — so the surface has
|
|
||||||
grown and a scrape target is expressible again.
|
|
||||||
|
|
||||||
**4. Cutover, then verify by driving it.** `deploy.sh <target> --service
|
**4. Cutover, then verify by driving it.** `deploy.sh <target> --service
|
||||||
<large> --service <small>` (order matters); per endpoint: `/health`,
|
<large> --service <small>` (order matters); per endpoint: `/health`,
|
||||||
@@ -265,10 +263,10 @@ grown and a scrape target is expressible again.
|
|||||||
`reasoning_content`, a `tools` request returns `tool_calls`, an image
|
`reasoning_content`, a `tools` request returns `tool_calls`, an image
|
||||||
request is answered (or refused with a named reason), the runtime's own
|
request is answered (or refused with a named reason), the runtime's own
|
||||||
speculation counters move, and the unit is `enabled` and `active`. All three LLM
|
speculation counters move, and the unit is `enabled` and `active`. All three LLM
|
||||||
endpoints are **root** units today (`User=dank` + `WantedBy=multi-user.target` —
|
endpoints are **root** units today (`scripts/frank-archon.service`,
|
||||||
`scripts/frank-archon.service:30,71`, `scripts/frank-dragoon.service:8,31`,
|
`scripts/frank-dragoon.service`, `scripts/frank-zealot.service`), so linger is
|
||||||
`scripts/frank-zealot.service:8,26`), so linger is not what gates them; linger
|
not what gates them; linger matters for rootless quadlets. Read the engine's boot
|
||||||
matters for rootless quadlets. Read the engine's boot ledger for a silent clamp. Finally spend one
|
ledger for a silent clamp. Finally spend one
|
||||||
real request through the harness per changed role — routing labels lie; the
|
real request through the harness per changed role — routing labels lie; the
|
||||||
servers' own journals do not.
|
servers' own journals do not.
|
||||||
|
|
||||||
@@ -285,4 +283,5 @@ predecessor "stopped" (the port race above).
|
|||||||
- [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
|
- [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
|
||||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
|
- [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
|
||||||
- [[LLM Bench Windows and Measurement]] — measuring the new deployment
|
- [[LLM Bench Windows and Measurement]] — measuring the new deployment
|
||||||
- [[frank-archon]] — the retired tcclaviger vLLM endpoint (2026-09-29 → 10-05); the live Dankserver endpoint is [[frank-archon]]
|
* [[frank-archon]] — the live Dankserver endpoint; the tcclaviger vLLM era
|
||||||
|
(2026-09-29 → 10-05) is a row in its Eras table
|
||||||
@@ -147,12 +147,14 @@ cut to GPTQ-w4 for the 3090 is a bench experiment that needs no new
|
|||||||
hardware; EXL3 (`convert.py` on the 3090, exllama serve) is the
|
hardware; EXL3 (`convert.py` on the 3090, exllama serve) is the
|
||||||
quality-per-bit extreme of the same menu.
|
quality-per-bit extreme of the same menu.
|
||||||
Caveats before any swap: the serving contract (OpenAI server,
|
Caveats before any swap: the serving contract (OpenAI server,
|
||||||
`reasoning_content`, vision mmproj, compaction-hub window
|
`reasoning_content`, vision mmproj, and the compaction-hub window the endpoint
|
||||||
200,704 ×2 slots) must be met by the challenger runtime, and the
|
declares in its Current shape — [[frank-zealot]]) must be met by the challenger
|
||||||
24 GB card runs ONE server — benching follows the stop-and-mask law.
|
runtime, and the 24 GB card runs ONE server — benching follows the stop-and-mask
|
||||||
|
law.
|
||||||
|
|
||||||
#### Worked example — the frank-archon w4dn8 recipe ported to the 3090
|
#### Worked example — the frank-archon w4dn8 recipe ported to the 3090
|
||||||
(audit 2026-10-08, target `Qwen3.6-35B-A3B`)
|
(audit 2026-10-08, target `Qwen3.6-35B-A3B`; sized at that date's hub shape —
|
||||||
|
the window history is [[frank-zealot]]'s Eras table)
|
||||||
**The methodology ports; the encoding must be translated.** llama.cpp
|
**The methodology ports; the encoding must be translated.** llama.cpp
|
||||||
decodes no GPTQ — the recipe re-expresses on GGUF-native terms:
|
decodes no GPTQ — the recipe re-expresses on GGUF-native terms:
|
||||||
imatrix on our corpora (replaces the radiance grams; runs the 65 GiB
|
imatrix on our corpora (replaces the radiance grams; runs the 65 GiB
|
||||||
@@ -253,8 +255,6 @@ top-8**, moe_intermediate 512, hidden 2048, MTP ×1, 262k ctx, vocab
|
|||||||
- Repo: `scripts/convert-swift15-w4dn8.sh`,
|
- Repo: `scripts/convert-swift15-w4dn8.sh`,
|
||||||
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`,
|
`scripts/recipes/qwen4exp-swift15-w4dn8-i8-hc8m.recipe`,
|
||||||
`scripts/kld-gate-swift15.sh`, `scripts/build-llama-rdna4.sh`,
|
`scripts/kld-gate-swift15.sh`, `scripts/build-llama-rdna4.sh`,
|
||||||
`scripts/build-kairic-edge-gfx1151.sh`, [[frank-dragoon]],
|
`scripts/build-kairic-edge-gfx1151.sh`; endpoints [[frank-archon]],
|
||||||
[[frank-dragoon]], [[qwen3.8-strix-optimizations]],
|
[[frank-dragoon]], [[frank-zealot]]; method [[qwen3.8-strix-optimizations]]
|
||||||
[[frank-archon]],
|
and [[GGUF Compatibility]].
|
||||||
[[frank-zealot]],
|
|
||||||
[[frank-dragoon]], [[GGUF Compatibility]].
|
|
||||||
@@ -108,9 +108,10 @@ Use ROCmFP4 only with the ROCmFPX fork of llama.cpp. Upstream llama.cpp cannot r
|
|||||||
|
|
||||||
## Choices for This Project
|
## Choices for This Project
|
||||||
|
|
||||||
### Dankserver (RTX 3090, 24 GB, Ampere)
|
### The RTX 3090 (24 GB, Ampere) — frank-zealot's card
|
||||||
|
|
||||||
The RTX 3090 has no native FP8 or FP4 tensor cores. Use:
|
The 3090 has no native FP8 or FP4 tensor cores. It has sat in Dankclaw since
|
||||||
|
2026-09-16; [[frank-zealot]]'s Eras table records the move off Dankserver. Use:
|
||||||
|
|
||||||
| Ranking | Format | Why |
|
| Ranking | Format | Why |
|
||||||
|---------|--------|-----|
|
|---------|--------|-----|
|
||||||
@@ -121,13 +122,17 @@ The RTX 3090 has no native FP8 or FP4 tensor cores. Use:
|
|||||||
|
|
||||||
### Dankclaw (Strix Halo gfx1151, RDNA 3.5)
|
### Dankclaw (Strix Halo gfx1151, RDNA 3.5)
|
||||||
|
|
||||||
RDNA 3.5 has no native FP8 tensor cores. The APU shares memory. The active Ling-3.0-flash service runs vLLM-ROCm with the AMD Quark INT4 (W4A16) format (the jcbtc CIRU Strix-native bundle). INT4 weights keep the 124B model inside the 96 GiB frame buffer with a full-precision KV cache.
|
RDNA 3.5 has no native FP8 tensor cores and the APU shares its memory, so the
|
||||||
|
GPU budget is the GTT aperture and every live figure on the box is
|
||||||
|
[[Dankclaw System Specs]]. What serves here today is [[frank-dragoon]]: a
|
||||||
|
vendor `.hgn` checkpoint, a format this page cannot rank because no public
|
||||||
|
quantizer exists for it ([[Quantization Production]]).
|
||||||
|
|
||||||
| Ranking | Format | Why |
|
| Ranking | Format | Why |
|
||||||
|---------|--------|-----|
|
|---------|--------|-----|
|
||||||
| 1 | INT4 W4A16 (Quark/vLLM) | active; fits 96 GiB; native ROCm kernels; concurrency |
|
| 1 | `.hgn` (deployed) | the halogen engine's own format; production door is closed |
|
||||||
| 2 | MXFP4 (Quark) | same-bit, higher accuracy; beta on gfx1151 (A/B experiment) |
|
| 2 | GGUF K/I quants, K_P blends | the only self-produced option on this card ([[Quantization Production]]) |
|
||||||
| 3 | GGUF Q4_K_M / Q8_0 | works with upstream llama.cpp only |
|
| 3 | INT4 W4A16 (Quark) / MXFP4 (Quark) | the retired Ling-3.0-flash vLLM era, measured under the firmware-VGM frame buffer: [[ling-3.0-flash-trial]] |
|
||||||
| 4 | FP8 (vLLM) | software emulation; memory savings only; no compute win |
|
| 4 | FP8 (vLLM) | software emulation; memory savings only; no compute win |
|
||||||
|
|
||||||
## See Also
|
## See Also
|
||||||
|
|||||||
Reference in new issue
Block a user