dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(vault): fleet-wide fact re-verification pass (67 pages)

Browse Source 
The docs-fact census wave re-read the live hosts and re-keyed every page
it could: runtime/payload identities, ports, sizes, ledger figures, unit
state, and the retirement banners that drifted as the fleet moved. The
wave was interrupted mid-run, so this commit carries only what passed the
integrity gates: every touched page parses (front matter or H1+status
convention intact, balanced fences, no truncated tables, trailing
newline), the docs_completeness and link-integrity seams are green, and
bats is 384/384.

Numbers that this session proved on live hosts are keyed to the measured
value, not the page's earlier claim.
This commit is contained in:
Frank committed 2026-10-18 13:15:43 -07:00
1 parent 466fd477d8
commit e7f0d93c86
67 files changed
+2619 -1152
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

serverAppConfigs-docs/LLM/Dankclaw Improvements.md
+41 -17
View File
@@ -16,9 +16,19 @@ priority and the exact action.
Dates: researched 2026-08. Hardware is verified on the live machine unless
marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
**Live today (2026-10-18).** The era described below — llama.cpp on the 8060S
iGPU behind :7770 — is retired. Dankclaw now serves
[[frank-dragoon-flash-server-v2-native]] (halogen-flash-server 0.16.2, port
7770) and [[frank-zealot-llama-prism-bonsai2-pq20]] (llama.cpp PrismML fork on
the RTX 3090, port 7771 — drift today: that unit is `failed` since 2026-10-18
03:40 and a hand-run 9B-MTP process answers the port), and boots `iommu=pt` since 2026-10-18
([[dankclaw-iommu-translation-off]]). The numbers below are the measurements of
the era they were taken in; the `ttm` ceiling and the NVMe scheduler rows were
re-checked 2026-10-18 and still hold.
## Corrections (re-verified 2026-08-15/16)
- **Backend**: production is **Vulkan0**, not ROCm. Measured on this box
- **Backend**: that era's production was **Vulkan0**, not ROCm. Measured on this box
(b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx
Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs
302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not
@@ -26,9 +36,12 @@ marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
does not exist on this machine (checked 2026-08-15). GPU DPM is already
`performance`, forced level `auto`. This item is N/A.
- **KV cache**: production runs `q8_0` K **and** `q8_0` V (both quantized).
- **Spec**: DFlash/DSpark are retired. Production is `--spec-default`
(ngram-mod) with draft-mtp first. See [[qwen3.8-spec-matrix]].
- **KV cache**: production ran `q8_0` K **and** `q8_0` V (both quantized). The law
still holds on `frank-zealot` (`CACHE_TYPE_K=q8_0`, `CACHE_TYPE_V=q8_0` —
`scripts/frank-zealot.env:55-56`).
- **Spec**: DFlash/DSpark are retired. That era's production was `--spec-default`
(ngram-mod) with draft-mtp first. See [[qwen3.8-spec-matrix]]. Today `frank-zealot`
is configured with speculation **off** (`SPEC_TYPE=none` — `scripts/frank-zealot.env:67`).
- **TTM**: raised to 114 GiB on 2026-08-25, set live (no reboot) and
persisted in `/etc/modprobe.d/ttm.conf`. The effective VRAM ceiling that
once held the service at ~15 GiB is gone.
@@ -39,21 +52,31 @@ marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
| ------ | ----- | ------ |
| OS physical RAM | 30.5 GiB | `/proc/meminfo` |
| TTM pages_limit | 114 GiB (29,883,536 x 4KiB) — raised 2026-08-25 | `/sys/module/ttm/parameters/pages_limit` |
| ROCm VRAM total | 96.0 GiB | `rocm-smi --showmeminfo vram` |
| ROCm VRAM total | 96.0 GiB | `rocm-smi --showmeminfo vram` under the then firmware VGM carve |
| ROCm VRAM used | 76.2 GiB | `rocm-smi` while serving 3x200700 |
The OS accounts only ~30.5 GiB physical RAM. The TTM GPU-addressable pool
was capped at ~15.24 GiB (the real pre-2026-08-25 ceiling); that cap is now
lifted to 114 GiB. ROCm/SMI still reports a 96 GiB VRAM carve-out (VGM) — a
different, driver-reported figure that has always been larger than the kernel
cap.
That 2026-08 reading is the firmware-VGM era: the BIOS carved ~96 GiB as
"dedicated" iGPU VRAM, so the kernel saw only ~30 GiB (the same split is
recorded in [[ling-3.0-flash-trial]] §Why Vulkan + GTT). The OS now accounts
**121 GiB** (`MemTotal 127108080 kB`, 2026-10-18) and `rocm-smi
--showmeminfo vram` reports **2.0 GiB** (2,147,483,648 B) of dedicated VRAM —
the carve is gone and the GPU budget is the GTT aperture, not the SMI figure
([[Dankclaw System Specs]]).
The TTM row above is still live: `pages_limit = 29883536` (114 GiB) and
`/etc/modprobe.d/ttm.conf` still carries the `options ttm` line (both read
2026-10-18).
This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory
limits documented in [[frank-dragoon-benchmark]].
## 1. Self-Hosted Agentic Tools
All of these connect to the local llama.cpp OpenAI-compatible endpoint
(`http://localhost:7770/v1`). None need NVIDIA.
All of these connect to the local OpenAI-compatible endpoint
(`http://127.0.0.1:7770/v1`). Today that port is **halogen-flash-server**, not a
llama.cpp server, and the address must be the IPv4 literal: `curl
http://localhost:7770/v1/models` returns nothing while
`http://127.0.0.1:7770/v1/models` answers 200 (checked 2026-10-18 — the rootless
podman publish is IPv4-only). None need NVIDIA.
| Tool | Use | Endpoint config | Priority |
| ---- | --- | --------------- | -------- |
@@ -67,7 +90,8 @@ All of these connect to the local llama.cpp OpenAI-compatible endpoint
| Qdrant | Vector store for RAG | Standalone, CPU-native | Low |
The pragmatic first installs are Open WebUI, Cline, and Aider. They all work
with the current llama.cpp server and need no extra GPU.
with the endpoint on 7770 and need no extra GPU. The 7770 endpoint today is
halogen-flash-server, not llama.cpp.
## 2. Inference Runtime Tuning (llama.cpp on gfx1151)
@@ -90,8 +114,8 @@ All require root and a reboot for the modprobe change.
| Setting | Action | Effect | Priority |
| ------- | ------ | ------ | -------- |
| TTM GPU pool | `/etc/modprobe.d/ttm.conf` → `options ttm pages_limit=29883536 page_pool_size=29883536` (applied 2026-08-25, set live then persisted) | Lifts the 15 GiB cap to 114 GiB; the full 262k KV + Q8 AGENT payload fit in one slot | Done |
| Power profile | **N/A on this machine** (2026-08-15): no `power_profile` sysfs; GPU DPM already `performance` | none | ~~Medium~~ |
| NVMe scheduler | Already `none` (`[none]` confirmed) | No change | Done |
| Power profile | **N/A on this machine** (2026-08-15; re-checked 2026-10-18: no `power_profile` under `/sys/class/drm/card0/device/` or `card1/device/`); GPU DPM already `performance` | none | ~~Medium~~ |
| NVMe scheduler | Already `none` (`[none] mq-deadline` in `/sys/block/nvme0n1/queue/scheduler`, re-read 2026-10-18) | No change | Done |
| BIOS VGM | Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS | Max dedicated VRAM for the GPU carve-out; verify in the live BIOS | Verify |
| Firmware/EC | Check GMKtec for a newer EC/fan-curve firmware | Better sustained thermal behavior | Verify |
@@ -99,8 +123,8 @@ All require root and a reboot for the modprobe change.
- Test `--spec-draft-n-max 5` and re-benchmark acceptance; if throughput rises,
change the env tunable.
- Consider switching the disabled `vllm-thinkingcap.service` unit off or
removing it to avoid confusion (it is loaded in a failed state).
- The `vllm-thinkingcap.service` unit this item told you to clean up is gone:
`systemctl is-enabled vllm-thinkingcap.service` → `not-found` (2026-10-18).
- Profile generation under 3-slot concurrent load before touching boot
settings, so any tuning is measured against the real workload.