dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(vault): strip restated fleet facts from 17 technique pages

Browse Source 
Enforces "one fact, one owner": technique pages keep mechanism, laws, traps
and verdicts; per-endpoint shapes, digests and era measurements are cited to
the endpoint page that owns them.

Also repairs the four pointers the pass left empty (each verified against the
cited owner before landing):
- EMULATION dequant doubles host RAM (118 GiB -> ~238 GiB) -> [[frank-archon]]
- 12.11 tok/s flat 304 -> 16,095 tokens -> [[Tcclaviger Fork Ops]]
- bundled-head acceptance ladder 0.739/0.554/0.427/0.340 -> [[Qwen3.8 PP Deployment]]
- llama.cpp PR #23897 dedup provenance -> [[qwen3.8-spec-matrix]]

Replaces the retired-shape illustration in Llama Cpp Serving Sizing with the
owned MiMo Q5_K_M row (6.40 GiB payload -> 20,530 MiB used), and unwraps a
hard-wrapped [[Dankserver System Specs]] link that failed link integrity.

Orphan census over the diff: 0 numbers lost (comma-insensitive match against
the whole vault plus AGENTS.md).
This commit is contained in:
Frank committed 2026-10-19 01:33:47 -07:00
1 parent eb21dbc705
commit 04f4e2c6d4
18 files changed
+348 -425
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/LLM/Dankclaw Improvements.md
+36 -42
View File
@@ -17,66 +17,59 @@ Dates: researched 2026-08. Hardware is verified on the live machine unless
marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
**Live today (2026-10-18).** The era described below — llama.cpp on the 8060S
iGPU behind :7770 — is retired. Dankclaw now serves
[[frank-dragoon]] (halogen-flash-server 0.16.2, port
7770) and [[frank-zealot]] (llama.cpp PrismML fork on
the RTX 3090, port 7771 — drift today: that unit is `failed` since 2026-10-18
03:40 and a hand-run 9B-MTP process answers the port), and boots `iommu=pt` since 2026-10-18
([[dankclaw-iommu-translation-off]]). The numbers below are the measurements of
the era they were taken in; the `ttm` ceiling and the NVMe scheduler rows were
re-checked 2026-10-18 and still hold.
iGPU — is retired; what runs on this box now (both endpoints, their engines and
ports) is [[frank-dragoon]]'s and [[frank-zealot]]'s pages. Drift worth knowing
before trusting either page: the frank-zealot unit is `failed` since 2026-10-18
03:40 and a hand-run 9B-MTP process answers its port. The OS facts are this
page's: `iommu=pt` since 2026-10-18 ([[dankclaw-iommu-translation-off]]), and
the `ttm` ceiling and NVMe scheduler rows below were re-checked 2026-10-18 and
still hold. Every benchmark number below is a measurement of the era it was
taken in.
## Corrections (re-verified 2026-08-15/16)
- **Backend**: that era's production was **Vulkan0**, not ROCm. Measured on this box
(b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx
Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs
302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not
hold on the 8060S. See [[frank-dragoon]].
- **Backend**: that era's production was **Vulkan0**, not ROCm. On this box ROCm
wins prefill and Vulkan wins decode, and the decode gap widens with context:
the 64k A/B is [[qwen3.8-strix-optimizations]] §4, the 262k A/B is
[[qwen3.8-spec-matrix]] §High-Context Backend Comparison. The "HIP
out-performs Vulkan for decode" claim did not hold on the 8060S.
- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
does not exist on this machine (checked 2026-08-15). GPU DPM is already
`performance`, forced level `auto`. This item is N/A.
- **KV cache**: production ran `q8_0` K **and** `q8_0` V (both quantized). The law
still holds on `frank-zealot` (`CACHE_TYPE_K=q8_0`, `CACHE_TYPE_V=q8_0` —
`scripts/frank-zealot.env:55-56`).
- **KV cache**: production ran `q8_0` K **and** `q8_0` V (both quantized). What
`frank-zealot` runs now is its own Current shape — this row is not it.
- **Spec**: DFlash/DSpark are retired. That era's production was `--spec-default`
(ngram-mod) with draft-mtp first. See [[qwen3.8-spec-matrix]]. Today `frank-zealot`
is configured with speculation **off** (`SPEC_TYPE=none` — `scripts/frank-zealot.env:67`).
- **TTM**: raised to 114 GiB on 2026-08-25, set live (no reboot) and
persisted in `/etc/modprobe.d/ttm.conf`. The effective VRAM ceiling that
once held the service at ~15 GiB is gone.
(ngram-mod) with draft-mtp first — timeline on [[qwen3.8-spec-matrix]]; the
spec setting each endpoint runs today is on its own page.
- **TTM**: the `pages_limit` raise that removed the ~15 GiB ceiling is recorded
once on [[Dankclaw System Specs]] (Memory).
## Verified Memory Picture (measured 2026-08-11)
| Figure | Value | Source |
| ------ | ----- | ------ |
| OS physical RAM | 30.5 GiB | `/proc/meminfo` |
| TTM pages_limit | 114 GiB (29,883,536 x 4KiB) — raised 2026-08-25 | `/sys/module/ttm/parameters/pages_limit` |
| ROCm VRAM total | 96.0 GiB | `rocm-smi --showmeminfo vram` under the then firmware VGM carve |
| ROCm VRAM used | 76.2 GiB | `rocm-smi` while serving 3x200700 |
That 2026-08 reading is the firmware-VGM era: the BIOS carved ~96 GiB as
"dedicated" iGPU VRAM, so the kernel saw only ~30 GiB (the same split is
recorded in [[ling-3.0-flash-trial]] §Why Vulkan + GTT). The OS now accounts
**121 GiB** (`MemTotal 127108080 kB`, 2026-10-18) and `rocm-smi
--showmeminfo vram` reports **2.0 GiB** (2,147,483,648 B) of dedicated VRAM —
the carve is gone and the GPU budget is the GTT aperture, not the SMI figure
([[Dankclaw System Specs]]).
The TTM row above is still live: `pages_limit = 29883536` (114 GiB) and
`/etc/modprobe.d/ttm.conf` still carries the `options ttm` line (both read
2026-10-18).
recorded in [[ling-3.0-flash-trial]] §Why Vulkan + GTT; the 76.2 GiB row is an
era [[frank-dragoon]] load reading). Every live number on this box — `MemTotal`,
the ~114 GiB GTT aperture that is the real GPU budget, the 2.0 GiB dedicated
carve and the `ttm` page cap — belongs to [[Dankclaw System Specs]], re-read
2026-10-18.
This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory
limits documented in [[frank-dragoon]].
## 1. Self-Hosted Agentic Tools
All of these connect to the local OpenAI-compatible endpoint
(`http://127.0.0.1:7770/v1`). Today that port is **halogen-flash-server**, not a
llama.cpp server, and the address must be the IPv4 literal: `curl
http://localhost:7770/v1/models` returns nothing while
`http://127.0.0.1:7770/v1/models` answers 200 (checked 2026-10-18 — the rootless
podman publish is IPv4-only). None need NVIDIA.
All of these connect to the local OpenAI-compatible endpoint on
`:7770` (which engine owns that port is [[frank-dragoon]]'s to say), and the
address must be the IPv4 literal: `curl http://localhost:7770/v1/models` returns
nothing while `http://127.0.0.1:7770/v1/models` answers 200 (checked 2026-10-18
— the rootless podman publish is IPv4-only). None need NVIDIA.
| Tool | Use | Endpoint config | Priority |
| ---- | --- | --------------- | -------- |
@@ -90,12 +83,13 @@ podman publish is IPv4-only). None need NVIDIA.
| Qdrant | Vector store for RAG | Standalone, CPU-native | Low |
The pragmatic first installs are Open WebUI, Cline, and Aider. They all work
with the endpoint on 7770 and need no extra GPU. The 7770 endpoint today is
halogen-flash-server, not llama.cpp.
with the endpoint on 7770 and need no extra GPU.
## 2. Inference Runtime Tuning (llama.cpp on gfx1151)
Verified against the running service and llama.cpp source.
Verified against that era's running service and llama.cpp source; the era is
retired ([[frank-dragoon]]), so this table is history and a method, not a live
TODO list.
| Change | Recommendation | Priority |
| ------ | -------------- | -------- |
@@ -104,8 +98,8 @@ Verified against the running service and llama.cpp source.
| Flash attention | Leave `--flash-attn auto`. It helps prompt processing, not decode. | Keep |
| Load mode | Start with mmap (default). Enable `--mlock` only if page-fault stutter appears. | Optional |
| Threads | Use `--threads 12 --threads-batch 16 --threads-http 1` for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver. | Medium |
| DFlash n_max | Current 8 (block 16) is high. Try `--spec-draft-n-max 5`; it may reduce rejection cost at the same throughput. | Verify |
| ngram layering | Measured slower than DFlash alone at 4k (16.9 vs 20.2 t/s). Not enabled. | Keep off |
| DFlash n_max | Depth 8 (block 16) was high; a shallower draft cuts rejection cost at the same throughput. Sweep depth per the A/B protocol in [[DFlash DSpark Speculative Decoding]]. | Verify |
| ngram layering | Layering ngram on the DFlash draft measured slower than the draft alone at 4k — figures on [[DFlash DSpark Speculative Decoding]]. | Keep off |
## 3. Server / OS / Hardware Settings
@@ -113,7 +107,7 @@ All require root and a reboot for the modprobe change.
| Setting | Action | Effect | Priority |
| ------- | ------ | ------ | -------- |
| TTM GPU pool | `/etc/modprobe.d/ttm.conf` → `options ttm pages_limit=29883536 page_pool_size=29883536` (applied 2026-08-25, set live then persisted) | Lifts the 15 GiB cap to 114 GiB; the full 262k KV + Q8 AGENT payload fit in one slot | Done |
| TTM GPU pool | Raised 2026-08-25; the knob, its value and its persistence file are [[Dankclaw System Specs]]'s (Memory row) | Lifted the ~15 GiB kernel-page ceiling so a full 262k KV plus a ~30 GB payload fits one slot | Done |
| Power profile | **N/A on this machine** (2026-08-15; re-checked 2026-10-18: no `power_profile` under `/sys/class/drm/card0/device/` or `card1/device/`); GPU DPM already `performance` | none | ~~Medium~~ |
| NVMe scheduler | Already `none` (`[none] mq-deadline` in `/sys/block/nvme0n1/queue/scheduler`, re-read 2026-10-18) | No change | Done |
| BIOS VGM | Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS | Max dedicated VRAM for the GPU carve-out; verify in the live BIOS | Verify |