diff --git a/AGENTS.md b/AGENTS.md index b359f45..30114b5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -7,7 +7,7 @@ and the FrankBook-Pro Mac mini. It tracks the Podman container stacks and the LLM inference fleet. One repo, one checkout per host, one deploy script. - **Dankserver** (192.168.0.235) — media/library stack (RomM, Copyparty, - Calibre-Web, Jellyfin, Plex) as bare containers, plus the `frank-small` + Calibre-Web, Jellyfin, Plex) as bare containers, plus the `frank-medium` LLM service. - **Dankclaw** (192.168.0.143) — AI/search/development/observability stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, @@ -27,11 +27,11 @@ prefix, reorganized in follow-up commits). The retired target checkouts archived. All services now run from `/home/dank/serverAppConfigs` on each host. -### LLM fleet topology (2026-09-11) +### LLM fleet topology (2026-09-15) | Target | Runtime | Service | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | FreeToken v0.1.2 | frank-small | batsclamp Huihui-Qwen3.6-35B-A3B-abliterated-FP8 (block-fp8 e4m3 128x128, ~35 GB safetensors), experts host-RAM LRU, 262144 ctx / 4 concurrent, reasoning on | 7770 | +| Dankserver | charlie12345/ROCmFPX llama.cpp | frank-medium | kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF (Q8_0_ROCMFPX_AGENT ftype 115, 26.28 GiB, 8.39 bpw), 2x Radeon AI PRO R9700 layer split, 131072 ctx / 1 slot, f16 KV, MTP draft head on ROCm1, reasoning on | 7770 | |Dankclaw|halogen-flash-server 0.5.8 (rootless Quadlet container)|frank-large|peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn 115.55 GiB + quality overlay, ~68 GiB pinned, 262144 ctx / 3 slots, pool 401408 / max_tok 32768 / reserve 15, vision tower on, greedy for silent clients)|7770| | Dankclaw | *(retired 2026-09-11)* FastFlowLM v1.0.1 on the XDNA2 NPU | frank-npu | Gemma4-E4B-IT-NPU2 pack (vision + audio). Retired with the `amd_iommu=off` decision: the driver's SVA/PASID bind needs an enabled IOMMU | 7771 | @@ -58,7 +58,12 @@ llama.cpp deployment gave way to halogen-flash-server 0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next, now a rootless Quadlet container, and the kernel cmdline moved from `iommu=pt` to `amd_iommu=off` for 13–16 % of prefill — which retired `frank-npu` for good (see the Runtime & -Troubleshooting section and its runbook). Runbooks: +Troubleshooting section and its runbook). On 2026-09-15 the RTX 3090 left +Dankserver for two Radeon AI PRO R9700 cards (gfx1201): `frank-small` +(FreeToken/CUDA) retired and `frank-medium` (charlie12345/ROCmFPX llama.cpp, +kingjones Q8_0_ROCMFPX-AGENT) took the :7770 endpoint — layer split, not +tensor, because the RCCL-less build's butterfly allreduce collapses prefill +7x (see the runbook for the measured table). Runbooks: [serverAppConfigs-docs/LLM/index.md](serverAppConfigs-docs/LLM/index.md). Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`. @@ -127,7 +132,7 @@ selected service and records its successful revision outside the checkout. ./scripts/deploy.sh dankclaw ./scripts/deploy.sh dankserver ./scripts/deploy.sh dankclaw --service searxng -./scripts/deploy.sh dankserver --service frank-small +./scripts/deploy.sh dankserver --service frank-medium ./scripts/deploy.sh dankclaw --dry-run ``` @@ -306,11 +311,11 @@ DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE 1. Stop the service before any benchmark: `systemctl --user stop frank-large` (Dankclaw — a **rootless** Quadlet - service, so drop the `sudo`) or `sudo systemctl stop frank-small` + service, so drop the `sudo`) or `sudo systemctl stop frank-medium` (Dankserver — still a system unit). Then mask it to prevent the self-healing unit from restarting: `systemctl --user mask frank-large` (Dankclaw) or - `sudo systemctl mask frank-small` (Dankserver). + `sudo systemctl mask frank-medium` (Dankserver). **CRITICAL WARNING:** Do NOT use `systemctl disable` on the **system** units. On these hosts, `systemctl disable` doesn't just remove symlinks — it **deletes the unit file itself** from `/etc/systemd/system/`. You'd have to manually @@ -326,8 +331,8 @@ DO NOT ROLL YOUR OWN BENCH UNLESS YOU CANNOT TEST THE DESIRED PARAMETER WITH THE against the service itself. 3. Restart the service when the bench completes: unmask, then start: `systemctl --user unmask frank-large && systemctl --user start frank-large` - (Dankclaw) or `sudo systemctl unmask frank-small && sudo systemctl start - frank-small` (Dankserver). + (Dankclaw) or `sudo systemctl unmask frank-medium && sudo systemctl start + frank-medium` (Dankserver). 4. Verify health before reporting: `curl http://192.168.0.235:7770/health` or `curl http://192.168.0.143:7770/health`. @@ -342,7 +347,7 @@ served alias: | alias | endpoint | |---|---| -| `frank-small` | Dankserver `192.168.0.235:7770` | +| `frank-medium` | Dankserver `192.168.0.235:7770` | | `frank-large` | Dankclaw `192.168.0.143:7770` | | `gemma4-it:e4b` | *(removed 2026-09-11 with `frank-npu`)* | @@ -375,8 +380,8 @@ driver cannot bind SVA/PASID, so every open of `/dev/accel/accel0` fails with from both the harness and Hermes. **Audio input left the fleet with it and has no replacement**; image input survives on `frank-large` through halogen's vision tower. Route map: Hermes `vision` → `frank-large`, Hermes -`title_generation` → `frank-small`, harness `vision` → `frank-large`, harness -`tiny` → `frank-small`, harness `task` → `frank-large` at medium effort. +`title_generation` → `frank-medium`, harness `vision` → `frank-large`, harness +`tiny` → `frank-medium`, harness `task` → `frank-large` at medium effort. The retired runbook keeps the full record, including the original IOMMU/SVA root cause: [serverAppConfigs-docs/LLM/frank-npu.md](serverAppConfigs-docs/LLM/frank-npu.md). @@ -491,7 +496,33 @@ load on any current loader (require full per-layer 3x). Binary and payload retained. Runbook: [serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md](serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md). -### Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on FreeToken (Dankserver, ACTIVE) +### kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT on ROCmFPX llama.cpp (Dankserver, ACTIVE since 2026-09-15) + +The medium service serves `kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF` +(public, 26.28 GiB, 8.39 bpw, `Q8_0_ROCMFPX_AGENT` ftype 115 — stock +llama.cpp cannot load it) from +`/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` with +charlie12345/ROCmFPX llama.cpp `c49ebdb`, built for **gfx1201** with the +fork's self-contained ROCm 7.14 toolchain +(`env JOBS=8 ./scripts/build-rocmfp4-rocm714-local.sh` in +`/home/dank/GitPackages/charlie-ROCmFPX`; the binary bundles its ROCm runtime +via RPATH, so the host's ROCm packages never participate). `--split-mode +layer` across both R9700s — tensor parallel loses 7x prefill on this +RCCL-less build — 131072 context / 1 slot, f16 KV, flash-attn on, MTP draft +head (n_max 6, p_min 0.6, `--spec-draft-device ROCm1`), reasoning on +(deepseek format), sampler defaults from the checkpoint's instruct table. +Measured 2026-09-15: llama-bench pp4096 1991 t/s / tg128 18.8 t/s (tensor: +271.8 / 26.26; single card: 1160.9 / 19.42); served with MTP 35.7–46.2 tok/s +and 60–69 % draft acceptance; prefix caching verified working with the draft +head. Env `scripts/frank-medium.env`, serve `scripts/serve-frank-medium.sh`. +Runbook: +[serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md](serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md). + +### Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on FreeToken (Dankserver, RETIRED 2026-09-15) + +Retired with the RTX 3090: FreeToken/CUDA cannot run on the replacement +R9700 cards, and the FreeToken-era unit was archived to +`~/.AGENTTRASH/20260915-frank-small-retire/host-units/`. Historical record: The small service serves `batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8` (public, ~35 GB, 26 safetensors shards) from @@ -623,7 +654,7 @@ graph TD flags. A different outcome isolates the env file as the cause. 3. Run gdb for the backtrace with this command: ``` - gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-small.sh + gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-medium.sh ``` `llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost. Trust gdb, not the log. diff --git a/README.md b/README.md index cd48ec6..97c0e5c 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,7 @@ one deploy script. | Host | User | IP | Podman stack | LLM service | LLM port | |------|------|----|--------------|-------------|----------| -| dankserver | dank | 192.168.0.235 | media and library | frank-small (llama.cpp, Qwen3.6-35B-A3B) | 7770 | +| dankserver | dank | 192.168.0.235 | media and library | frank-medium (ROCmFPX llama.cpp, Qwen3.8-27B AGENT) | 7770 | | dankclaw | dank | 192.168.0.143 | AI, search, development, observability | frank-large (CIRU ROCmFPX, Qwen3.8-27B) | 7770 | ## Active Services @@ -22,7 +22,7 @@ one deploy script. | RomM | dankserver | romm.dankserver.local | 8085 | | Copyparty | dankserver | files.dankserver.local | 3923 | | Calibre-Web | dankserver | books.dankserver.local | 8083 | -| frank-small | dankserver | — (bare endpoint) | 7770 | +| frank-medium | dankserver | — (bare endpoint) | 7770 | | SearXNG | dankclaw | search.dankclaw.local | 8081 | | ComfyUI | dankclaw | comfyui.dankclaw.local | 8188 | | OpenWebUI | dankclaw | chat.dankclaw.local | 3000 | diff --git a/serverAppConfigs-docs/Architecture/Deployment Flow.md b/serverAppConfigs-docs/Architecture/Deployment Flow.md index 71b29f9..14bee6c 100644 --- a/serverAppConfigs-docs/Architecture/Deployment Flow.md +++ b/serverAppConfigs-docs/Architecture/Deployment Flow.md @@ -46,7 +46,7 @@ graph LR ./scripts/deploy.sh dankserver ./scripts/deploy.sh frankbook-pro --service qbittorrent # podman-machine target ./scripts/deploy.sh dankclaw --service searxng -./scripts/deploy.sh dankserver --service frank-small +./scripts/deploy.sh dankserver --service frank-medium ./scripts/deploy.sh dankclaw --dry-run ``` @@ -63,7 +63,7 @@ service on the other host. | Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. | | SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. | | `frank-large` | systemd + `systemctl` | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. | -| `frank-small` | systemd + `systemctl` | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. | +| `frank-medium` | systemd + `systemctl` | Retires the legacy frank-small unit on first deploy, then relinks and restarts only the inference service; waits up to 20 minutes for `/health`. | | Dankserver media stack (RomM trio, Jellyfin, Calibre-Web, Copyparty) | user Quadlet + `systemctl --user` | Installs `dankserver//quadlet/*.container`, restarts the generated unit. The first cutover per service is idempotent: it retires the compose-era bare container, rescues any anonymous volume into the data root, and merge-moves `/opt/` to `/home/dank/Containers/` (mapped-uid roots move without chown). Steady state is restart-only. | The shared chat template maps to the inference service on the selected target. diff --git a/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md b/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md index a3ba501..fe1a967 100644 --- a/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md +++ b/serverAppConfigs-docs/Dankclaw/Dankclaw Stack.md @@ -17,7 +17,7 @@ stack. Its LLM inference endpoint `frank-large` (`:7770`) is a ROCmFPX llama.cpp fork serving the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF on the card's pinned runtime; an NPU endpoint (`frank-npu`, `:7771`, FLM v1.0.1) serves sight+sound models. The completion endpoint -`frank-small` (FreeToken, `:7770`) runs on Dankserver (192.168.0.235). +`frank-medium` (ROCmFPX llama.cpp, `:7770`) runs on Dankserver (192.168.0.235). ```mermaid graph LR @@ -50,7 +50,7 @@ graph LR | Pi-hole | `pihole.franksdank.cloud` (basicauth edge, not the guard chain) | admin UI `8083`; DNS on `:53` LAN-wide | [[Pi-hole]] | | frank-large (host service) | — | `7770` | [[deployment]] | | frank-npu (host service) | — | `7771` (`/v1/version` health) | [[frank-npu]] | -| frank-small (host service, Dankserver) | — | `7770` | [[deployment]] | +| frank-medium (host service, Dankserver) | — | `7770` | [[frank-medium-rocmfpx]] | The LLM rows are bare-metal systemd services, not containers. This repo owns their scripts, units, and knowledge base (see @@ -58,8 +58,8 @@ repo owns their scripts, units, and knowledge base (see `frank-large` runs the kingjones Qwen3.8-Flash-Next-ROCmFP4-STRIX GGUF on the card's pinned `kingjones30/ROCmFPX` llama.cpp build; `frank-npu` runs FLM v1.0.1 on the XDNA2 NPU; -`frank-small` runs FreeToken v0.1.2 serving -Huihui-Qwen3.6-35B-A3B-abliterated-FP8 on Dankserver. +`frank-medium` runs charlie12345/ROCmFPX llama.cpp serving +kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF on Dankserver. ## Boot Persistence diff --git a/serverAppConfigs-docs/Dankclaw/Hermes.md b/serverAppConfigs-docs/Dankclaw/Hermes.md index ba96491..9dba014 100644 --- a/serverAppConfigs-docs/Dankclaw/Hermes.md +++ b/serverAppConfigs-docs/Dankclaw/Hermes.md @@ -39,7 +39,7 @@ is not durable. The tracked, sanitized configuration baseline is - Hermes has two named local providers: `dankclaw-large` reaches `http://192.168.0.143:7770/v1` and exposes `frank-large`; `dankserver-small` reaches `http://192.168.0.235:7770/v1` and exposes - `frank-small`. + `frank-medium` (renamed from `frank-small` on 2026-09-15). - The live `config.yaml` contains secrets and user state and is not committed; the tracked example is the sanitized backup of its non-secret settings. diff --git a/serverAppConfigs-docs/Dankclaw/OpenWebUI.md b/serverAppConfigs-docs/Dankclaw/OpenWebUI.md index 490f173..6c96c30 100644 --- a/serverAppConfigs-docs/Dankclaw/OpenWebUI.md +++ b/serverAppConfigs-docs/Dankclaw/OpenWebUI.md @@ -28,7 +28,7 @@ host; `frank-large` moved from `:52625` to `:7770` on 2026-08-22. | Endpoint | Model name | Host | Capability | |----------|-----------|------|------------| | `:7770` (ROCmFPX llama.cpp fork) | `frank-large` | Dankclaw | completion | -| `:7770` (FreeToken v0.1.2) | `frank-small` | Dankserver | completion, multimodal | +| `:7770` (ROCmFPX llama.cpp fork) | `frank-medium` | Dankserver | completion | The default connection points at the on-host endpoint (`frank-large`). To add the other backend, open **Admin → Connections** in the UI and point an diff --git a/serverAppConfigs-docs/Dankserver/Plex.md b/serverAppConfigs-docs/Dankserver/Plex.md index d825921..6c2de53 100644 --- a/serverAppConfigs-docs/Dankserver/Plex.md +++ b/serverAppConfigs-docs/Dankserver/Plex.md @@ -21,7 +21,7 @@ public traffic can be fronted by Caddy like every other vhost. | Identity | fleet rootless model — container root maps to the `dank` user (1000:1000); the deployer chowns the data root on first cutover | | Port | `32400/tcp` published on `0.0.0.0` — zero client reconfiguration | | Network | `dankserver_appnet`, alias `plex`; Tautulli reaches it at `http://plex:32400` | -| GPU | none by decision — CPU decode only; the RTX 3090 stays dedicated to `frank-small` | +| GPU | none by decision — CPU decode only; the two R9700s stay dedicated to `frank-medium` | | Deploy | `./scripts/deploy.sh dankserver --service plex` | | Docs | https://support.plex.tv | diff --git a/serverAppConfigs-docs/LLM/Dankserver System Specs.md b/serverAppConfigs-docs/LLM/Dankserver System Specs.md index 4aa0022..43052df 100644 --- a/serverAppConfigs-docs/LLM/Dankserver System Specs.md +++ b/serverAppConfigs-docs/LLM/Dankserver System Specs.md @@ -1,8 +1,24 @@ +--- +type: configuration +title: Dankserver System Specs +description: Dankserver hardware and OS state since the 2026-09-15 GPU swap (2x AMD Radeon AI PRO R9700, gfx1201) serving frank-medium. +tags: + - dankserver + - hardware + - r9700 + - gfx1201 + - rocm +timestamp: 2026-09-15 +--- + # Dankserver System Specs ## Server Overview -Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.04 (Resolute Raccoon) with an RTX 3090 and 62 GB of system RAM. +Dankserver is a media/library server and the `frank-medium` LLM host. The +host runs Ubuntu 26.04 (Resolute Raccoon). The RTX 3090 that served the +FreeToken/CUDA `frank-small` era was removed on 2026-09-15 and replaced with +two AMD Radeon AI PRO R9700 cards. ## Hardware Specifications @@ -13,42 +29,49 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0 | Architecture | x86_64 | | Vendor | AuthenticAMD | | Model | AMD Ryzen 7 7700X | -| Cores | 8 (1 socket) | -| NUMA Nodes | 1 (node 0, 63503 MB) | +| Cores | 8 cores / 8 threads (1 socket, SMT off) | +| NUMA Nodes | 1 (node 0) | -### GPU +### GPUs | Property | Value | |----------|-------| -| Model | NVIDIA GeForce RTX 3090 | -| VRAM | 24576 MiB (24 GB) | -| Driver | 595.84 | -| CUDA Version | 13.2 | -| CUDA Toolkit | 12.0 (nvcc) | +| Model | 2x AMD Radeon AI PRO R9700 (Navi 48, DID 0x7551, rev c0) | +| Architecture | gfx1201 (RDNA4) | +| VRAM | 32,624 MiB each | +| Link | PCIe, no XGMI; different IOMMU groups | +| PCI | 03:00.0 and 0b:00.0 | +| iGPU | Raphael (gfx1036) at 10:00.0 — enumerates as ROCm device 2, hidden from HIP | + +The RTX 3090 is gone; stale `nvidia-smi` binaries and the disabled NVIDIA +apt lists remain on disk but no NVIDIA PCI device exists. ### Memory | Property | Value | |----------|-------| -| System RAM | 62 GiB | -| Swap | 127 GiB | +| System RAM | 60 GiB | +| Swap | 63 GiB | | /dev/shm | 32 GiB | ### Storage | Filesystem | Size | Used | Avail | Use% | |------------|------|------|-------|------| -| / (nvme0n1p2) | 915G | 729G | 140G | 84% | +| / (nvme0n1p2) | 915G | ~640G | ~280G | ~70% | | /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% | +Models live under `/home/dank/models`; the ROCmFPX build tree under +`/home/dank/GitPackages/charlie-ROCmFPX` with its self-contained ROCm 7.14 +toolchain under `~/.local/share/rocm-7.14.0a20260624`. + ### Network | Interface | Type | Speed | IP | Notes | |-----------|------|-------|----|-------| -| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.235 (DHCP) | RPC worker endpoint (port 50052). Default route via 192.168.0.1. | -| enxe8ea6ab7ebca | USB-C class adapter | 2500 Mb/s | 192.168.0.252 | UNPLUGGED 2026-08-17. Was idle (unused by the RPC path). | +| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.235 (DHCP) | LLM endpoint (port 7770). Default route via 192.168.0.1. | +| enxe8ea6ab7ebca | USB-C class adapter | 2500 Mb/s | 192.168.0.252 | Unplugged since 2026-08-17. | | wlp8s0 | WiFi | - | - | Unused (carrier down). | -| docker0 | bridge | - | 172.17.0.1 | Down; internal docker traffic only. | ### OS and Software @@ -57,21 +80,35 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0 | OS | Ubuntu 26.04 LTS (Resolute Raccoon) | | Kernel | 7.0.0-30-generic | | Python | 3.14.4 | +| ROCm userspace | Ubuntu 7.1 packages in /usr (hipcc 7.1.1, rocminfo, rocm-smi 7.1.1) | +| Serving stack | charlie12345/ROCmFPX llama.cpp with its own ROCm 7.14 toolchain (self-contained, RPATH-bundled) | + +The stale AMD-repo ROCm 6.3.3 packages (167, `~24.04`-suffixed) that used to +hijack `libhsa-runtime64` were purged on 2026-09-15 — see +[[frank-medium-rocmfpx]]. ### Active Services -The active service on Dankserver is `frank-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors selected by `--override-tensor`; every full-attention block plus blocks 35–40 remain on CUDA. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]]. +The active inference service is `frank-medium.service`: charlie12345/ROCmFPX +llama-server on port 7770, layer-split across both R9700s with the MTP draft +head on ROCm1. See [[frank-medium-rocmfpx]]. The former +`frank-small.service` (FreeToken/CUDA) was retired with the GPU swap; its +unit is archived in `~/.AGENTTRASH/20260915-frank-small-retire/host-units/`. +The earlier PP worker (`frank-pp-worker.service`) was retired with the +2026-08-17 redo. See [[deployment]]. ## Hardware Relevance -- 24 GB VRAM hosts the Q6_K non-expert tensors and the full-attention/final-layer expert placement. The context pool contains 491520 f16 cells across three 163840-token slots. The service uses ngram-mod only and `ubatch=1024`. Verify actual headroom after each model restart with `nvidia-smi` and the service journal. -- 62 GB system RAM holds the remaining CPU-resident expert tensors with room to spare. -- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled. -- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total). +- 2x 32 GB VRAM: the 26.28 GiB Q8_0_ROCMFPX_AGENT checkpoint plus MTP draft + and a 131,072-token f16 KV cache fit comfortably; layer split gives each + card about half the layers and its own KV share. +- 60 GB system RAM is mostly headroom; nothing in the serving path is + CPU-resident. +- The iGPU is deliberately unused: `HIP_VISIBLE_DEVICES=0,1` keeps work off + the shared-memory Raphael device. ## See Also -- [[VLLM Overview]] -- [[Hybrid CPU GPU Processing Dankserver]] -- [[RAM CPU Offloading in VLLM]] -- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] \ No newline at end of file +- [[frank-medium-rocmfpx]] — the active serving runbook +- [[deployment]] — the deploy flow +- [[Shared Memory Architecture Dankclaw]] — for contrast, Dankclaw's unified memory diff --git a/serverAppConfigs-docs/LLM/deployment.md b/serverAppConfigs-docs/LLM/deployment.md index 9fa0b8f..33322cc 100644 --- a/serverAppConfigs-docs/LLM/deployment.md +++ b/serverAppConfigs-docs/LLM/deployment.md @@ -19,7 +19,7 @@ SearXNG, Caddy, or another container service does **not** restart inference. | Target | Host | Service | Port | |---|---|---|---:| -| Dankserver | `dank@dankserver` | `frank-small` | 7770 | +| Dankserver | `dank@dankserver` | `frank-medium` | 7770 | | Dankclaw | `dank@dankclaw` | `frank-large` | 7770 | | Dankclaw | `dank@dankclaw` | `frank-npu` (FLM on the XDNA2 NPU) | 7771 | @@ -39,7 +39,7 @@ graph LR Run the unified deployer from the repository root: ```bash -./scripts/deploy.sh dankserver --service frank-small +./scripts/deploy.sh dankserver --service frank-medium ./scripts/deploy.sh dankclaw --service frank-large ./scripts/deploy.sh dankclaw --dry-run ``` @@ -47,8 +47,8 @@ Run the unified deployer from the repository root: Without `--service`, `scripts/deploy-plan.py` maps changed files to the service on the selected host. The following files select the corresponding service: -- `scripts/frank-small.env`, `frank-small.service`, and - `serve-frank-small.sh` select `frank-small` on Dankserver. +- `scripts/frank-medium.env`, `frank-medium.service`, and + `serve-frank-medium.sh` select `frank-medium` on Dankserver. - `scripts/frank-large.env`, `frank-large.service`, and `serve-frank-large.sh` select `frank-large` on Dankclaw. - `scripts/frank-npu.env`, `frank-npu.service`, @@ -70,7 +70,7 @@ slow. A failed handler does not advance its per-service state. Both units point into the stable target checkout: ```text -/home/dank/serverAppConfigs/scripts/frank-small.service +/home/dank/serverAppConfigs/scripts/frank-medium.service /home/dank/serverAppConfigs/scripts/frank-large.service ``` @@ -105,14 +105,14 @@ change in that service's inputs. If the LLM handler fails after the target pull: -1. Inspect `journalctl -u frank-small -n 200` or +1. Inspect `sudo journalctl -u frank-medium -n 200` or `journalctl -u frank-large -n 200`. 2. Confirm the model files, custom runtime binary, and chat template are readable by `dank`. 3. Retry only the affected service: ```bash - ./scripts/deploy.sh dankserver --service frank-small + ./scripts/deploy.sh dankserver --service frank-medium # or ./scripts/deploy.sh dankclaw --service frank-large ``` diff --git a/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md b/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md new file mode 100644 index 0000000..8f1e0b6 --- /dev/null +++ b/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md @@ -0,0 +1,214 @@ +--- +type: configuration +title: frank-medium ROCmFPX Runbook +description: charlie12345/ROCmFPX llama.cpp serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF as frank-medium on Dankserver (2x AMD Radeon AI PRO R9700, gfx1201). +tags: + - frank-medium + - dankserver + - rocmfpx + - llama-cpp + - mtp + - gfx1201 + - r9700 +timestamp: 2026-09-15 +--- + +# frank-medium ROCmFPX Runbook + +`frank-medium` is the Dankserver LLM endpoint (port 7770) since 2026-09-15, +when the RTX 3090 left the box and two **AMD Radeon AI PRO R9700** cards +(Navi 48, RDNA4, `gfx1201`, 32,624 MiB each, PCIe 5.0 x16) took its place. +It replaced the retired FreeToken/CUDA `frank-small` +([[freetoken-frank-small]]), which cannot run on this hardware at all. + +Stack: **charlie12345/ROCmFPX llama.cpp** (`main`, `c49ebdb`, 2026-08-22) +serving **kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF** with the +companion MTP draft head. Stock llama.cpp cannot load this GGUF — +`Q8_0_ROCMFPX_AGENT` is ftype 115 / `GGML_TYPE_Q8_0_ROCMFPX` (type 103) and +exists only in the fork. + +## Hardware + +| Property | Value | +|----------|-------| +| GPUs | 2x AMD Radeon AI PRO R9700 (DID 0x7551), rev c0 | +| Architecture | gfx1201 (Navi 48, RDNA4) | +| VRAM | 32,624 MiB each (rocm-smi) | +| Link | PCIe, no XGMI; cards sit in different IOMMU groups | +| iGPU | Raphael (gfx1036) enumerates as ROCm device 2 — hidden from HIP | +| CPU / RAM | Ryzen 7 7700X (8c/8t), 60 GiB RAM, 63 GiB swap | +| Kernel driver | in-tree `amdgpu`; no new driver needed | + +`HIP_VISIBLE_DEVICES=0,1` hides the iGPU so nothing can drift onto shared +system RAM. Devices are addressed as `ROCm0,ROCm1` (llama-server) or +`ROCm0/ROCm1` (llama-bench multi-device syntax — see the bench gotcha below). + +## Runtime build (self-contained ROCm 7.14) + +The host ROCm install is quarantined rather than repaired: the build uses the +fork's self-contained toolchain, so the binary carries its own +`libamdhip64`/`libhsa-runtime64`/`rocblas`/`hipblaslt` with +`RPATH=$ORIGIN/../lib` and never touches system ROCm. + +```bash +cd /home/dank/GitPackages/charlie-ROCmFPX +env JOBS=8 ./scripts/build-rocmfp4-rocm714-local.sh +# -> build-rdna4-rocm714/bin/llama-server +``` + +The script downloads the gfx120X-all TheRock ROCm 7.14 nightly tarball into +`~/.local/share/rocm-7.14.0a20260624` (~17 GiB extracted; tarball kept next to +the build tree) and builds `gfx1200;gfx1201`. + +**gfx1201 is not interchangeable with gfx1200** — the fork's +`docs/BUILD-AMD-ARCHITECTURES.md` warns a wrong-sibling build links, loads the +model, then segfaults. `scripts/rocmfpx-hip-arch.sh` auto-detects and refuses. + +## Model + +| File | Size | Role | +|------|------|------| +| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | 28,214,425,952 B (26.28 GiB, 8.39 bpw) | target | +| `mtp-Qwen3.8-27B-Q4_0.gguf` | 1,680,271,648 B | MTP draft head | +| `mmproj-Qwen3.8-27B-BF16.gguf` | 931,145,888 B | vision tower — **unused** (`--no-mmproj`) | + +The AGENT routing keeps 13 of the 17 `*.output.weight` tensors at full Q8_0 +(12 more than the plain `Q8_0_ROCMFPX` build), which raises MTP draft +acceptance (0.953 vs 0.911 on the author's gfx1151 box). Take this variant +only when serving with MTP. Fetch with `scripts/download-q8agent.sh` +(resumable, size-verified; run `--verify-only` to re-check). + +The model card documents 131,072 context; llama.cpp reports +`n_ctx_train (262144)` from the GGUF, but the author never tested beyond 131k, +so the deployment uses 131,072. + +## Parallelism: layer split, not tensor (measured) + +llama-bench on the host (pp4096 / tg128, same at depth 32768, f16 KV, +flash-attn on, 3 reps): + +| Split | pp4096 | tg128 | pp4096 @d32768 | tg128 @d32768 | +|-------|-------:|------:|---------------:|--------------:| +| 1x R9700 (`-sm none`) | 1160.9 | 19.42 | ~512 | ~18.4 | +| **2x layer (`-sm layer`)** | **1991.2** | 18.76 | **859.1** | 17.64 | +| 2x tensor (`-sm tensor`) | 271.8 | 26.26 | 228.6 | 25.09 | + +Tensor parallel wins decode by ~40 % but collapses prefill **7x**: this build +ships without RCCL, so llama.cpp logs +`internal AllReduce init failed (n_devices != 2?); falling back to meta-backend +butterfly` and every layer pays a slow cross-card reduction. The workload is +prefill-heavy (agentic, long prompts, reasoning traces), so the operator +switched to layer split on 2026-09-15. Revisit if the fork ships a working +RCCL tensor path (`-DGGML_HIP_RCCL=ON`). + +Served-path comparison with MTP on (real completions): layer 35.7–46.2 tok/s +vs tensor 28.7 tok/s — layer wins decode too once the draft head is in play. + +### llama-bench multi-device gotcha + +`llama-bench` and `llama-server` disagree on device syntax. In llama-bench, +`-dev ROCm0,ROCm1` means *two separate single-card benches* (the value is +split on `,` into separate device sets, one model load each); a single split +bench needs `-dev ROCm0/ROCm1`. The invalid form silently produces +single-card numbers in a table that looks like a split test. + +### Tensor-mode draft abort (if tensor is ever revisited) + +Under `--split-mode tensor` the MTP draft head must be pinned to exactly one +device (`--spec-draft-device ROCm0`); with it empty the draft joins the +meta-device KV allocation and the server aborts while building the KV cache: + +``` +ggml/src/ggml-backend.cpp:119: GGML_ASSERT(buffer) failed + <- ggml_backend_meta_alloc_ctx_tensors_from_buft <- llama_kv_cache ctor +``` + +## Deployment + +| File | Role | +|------|------| +| `scripts/frank-medium.env` | all tunables (sampler, split, draft, context) | +| `scripts/serve-frank-medium.sh` | env-driven llama-server launcher | +| `scripts/frank-medium.service` | root systemd unit (relinked into /etc/systemd/system) | +| `scripts/download-q8agent.sh` | model fetch + size verification | + +```bash +./scripts/deploy.sh dankserver --service frank-medium +``` + +First deploy (2026-09-15) also ran `retire_root_llm_small()` in +`scripts/deploy-remote.sh`: the CUDA-era `frank-small` unit was stopped, +disabled and moved to `~/.AGENTTRASH/20260915-frank-small-retire/host-units` +(the host's `systemctl disable` has been observed to delete unit files, +2026-08-26 incident, so both outcomes are tolerated). + +Deployed flags (see `scripts/frank-medium.env` for the full set): +`--split-mode layer --device ROCm0,ROCm1 --gpu-layers 999 --fit off`, +`--ctx-size 131072 --parallel 1`, f16 K/V, flash-attn on, +`--spec-type draft-mtp` (n_max 6, p_min 0.6) with the draft pinned to +`ROCm1` — the card holding the target model's last layers, where the final +hidden state is produced. Reasoning on, deepseek format, unlimited budget; +sampler defaults are the checkpoint's instruct table (temp 0.7, top-p 0.8, +top-k 20, min-p 0, presence 1.5). Text only — vision stays on +`frank-large` (Dankclaw). + +## Verified serving behavior (2026-09-15) + +- Health `{"status":"ok"}` ~18 s after start; model load ~17 s (warm page cache). +- MTP engaged: draft acceptance 60–69 % across prompts + (`timings.draft_n` / `draft_n_accepted` in every chat response). +- Decode 35.7–46.2 tok/s on a 200-token budget; cold prompt 949 tokens + prefilled in 2036 ms (~466 tok/s at ubatch 512). +- **Prefix caching works with the draft head**: the model card's + `mtp-prompt-cache-fix.patch` defect (draft head disables prefix caching) is + already fixed in `c49ebdb` — the speculative boundary rides inside the + context checkpoint (`common_prompt_checkpoint::data_spec`). Measured: turn 2 + of a conversation reprocessed 22 tokens (`cache_n` 945) instead of 949. +- Quirk: the request-level `"speculative": {"n_max": 0}` override does **not** + disable drafting (still drafts and accepts). To serve without MTP, restart + with `SPEC_TYPE=none`. + +## Client contract + +OpenAI-compatible `/v1` on port 7770, alias `frank-medium`. Reasoning traces +are projected into `reasoning_content` (deepseek format); requests that send +no sampler fields get the server's CLI defaults, not greedy. Text only. +Harness routing after the cutover: blankpad `smol`/`tiny`/`commit`/`default` +→ `frank-medium`, Hermes `title_generation` (and its other dankserver-small +routes) → `frank-medium`, vision stays `frank-large`. + +## Host ROCm posture (post-cleanup) + +Historically the host had a broken mix: AMD-repo ROCm 6.3.3 in +`/opt/rocm-6.3.3` hijacked `libhsa-runtime64` via +`/etc/ld.so.conf.d/10-rocm-opencl.conf`, breaking Ubuntu's ROCm 7.1 userspace +(`undefined reference to 'hsa_amd_memory_get_preferred_copy_engine'`). +On 2026-09-15 the stale 6.3.3 packages (~24.04-suffixed, 167 pkgs) were +purged, the hijack conf and the `repo.radeon.com` apt pin moved to +`~/.AGENTTRASH/20260915-rocm633-cleanup/`, and `ldconfig` now resolves only +Ubuntu's ROCm 7.1 libs. `rocm-smi` is Ubuntu's own 7.1.1 package. The +self-contained toolchain in `~/.local/share/rocm-7.14.0a20260624` does not +depend on any of it. + +## Troubleshooting + +- **Wrong arch build**: a gfx1200 binary on this card links, loads, then + segfaults. Rebuild with the pinned script and check the "Build complete for + gfx1200;gfx1201" line plus the emitted code objects. +- **Crash at load**: read the journal (`sudo journalctl -u frank-medium -n 100`), + then run the serve script directly; the bundled libs make the script and the + raw binary behave identically. The known abort shape is the tensor-mode + draft KV abort above. +- **MTP "not engaged"**: check `timings.draft_n` in a response; zero means the + draft head is not loaded (look for `common_speculative_init: adding + speculative implementation 'draft-mtp'` in the journal). +- **Model files missing/truncated**: `bash scripts/download-q8agent.sh + --verify-only`. + +## See Also + +- [[freetoken-frank-small]] — the retired predecessor (FreeToken/CUDA on the 3090) +- [[kingjones-q8agent-rocmfpx]] — the gfx1151 investigation of this checkpoint family +- [[Llama Cpp Runtimes]] — where the ROCmFPX fork sits among the runtimes +- [[Dankserver System Specs]] — current hardware +- [[deployment]] — the deploy flow diff --git a/serverAppConfigs-docs/LLM/freetoken-frank-small.md b/serverAppConfigs-docs/LLM/freetoken-frank-small.md index b326e41..8f32d70 100644 --- a/serverAppConfigs-docs/LLM/freetoken-frank-small.md +++ b/serverAppConfigs-docs/LLM/freetoken-frank-small.md @@ -1,7 +1,7 @@ --- type: configuration title: FreeToken frank-small Runbook -description: FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-small on Dankserver (RTX 3090). +description: RETIRED 2026-09-15 - FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B-abliterated-FP8 as frank-small on Dankserver (RTX 3090 era). tags: - freetoken - dankserver @@ -14,6 +14,13 @@ timestamp: 2026-08-30 # FreeToken frank-small Runbook +> **RETIRED 2026-09-15.** The RTX 3090 left Dankserver and the successor +> [[frank-medium-rocmfpx]] (charlie12345/ROCmFPX llama.cpp on 2x Radeon AI +> PRO R9700) took the :7770 endpoint and the port. This page is kept as the +> retirement record of the FreeToken/CUDA era; the unit was archived to +> `~/.AGENTTRASH/20260915-frank-small-retire/host-units/` and the repo files +> to `scripts/.AGENTTRASH/`. + The `frank-small` service on Dankserver is served by **FreeToken v0.1.2** — an edge-native MoE serving engine that keeps expert weights in host RAM with an LRU GPU cache. Model history on this diff --git a/serverAppConfigs-docs/LLM/index.md b/serverAppConfigs-docs/LLM/index.md index 6045a18..0773c52 100644 --- a/serverAppConfigs-docs/LLM/index.md +++ b/serverAppConfigs-docs/LLM/index.md @@ -1,9 +1,10 @@ # LLM Fleet — Knowledge Base -Documentation for the LLM inference fleet on two local machines -(2026-09-11): FreeToken v0.1.2 on Dankserver (frank-small, -Huihui-Qwen3.6-35B-A3B-abliterated-FP8) and halogen-flash-server 0.5.8 on -Dankclaw (frank-large, peonist-ai Qwen3.8-Flash-Next w4b). +Documentation for the LLM inference fleet on the two hosts +(2026-09-15): charlie12345/ROCmFPX llama.cpp on Dankserver (frank-medium, +kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT on 2x Radeon AI PRO R9700) and +halogen-flash-server 0.5.8 on Dankclaw (frank-large, peonist-ai +Qwen3.8-Flash-Next w4b). > 2026-08-22: the former `vllmScripts` repository is merged into the > serverAppConfigs repository. This knowledge base moved from the old repo's @@ -14,12 +15,13 @@ Dankclaw (frank-large, peonist-ai Qwen3.8-Flash-Next w4b). ```mermaid graph LR - subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04] - A[RTX 3090 24GB + 60GB RAM] - B2[FreeToken v0.1.2 -frank-small, port 7770] - C[Huihui-Qwen3.6-35B-A3B-abliterated-FP8 -block-fp8, 20% MoE residency, 453k KV / 4 streams] + subgraph Dankserver[AMD Dankserver - Ubuntu 26.04] + A[2x Radeon AI PRO R9700 +gfx1201, 32 GiB each] + B2[ROCmFPX llama.cpp +frank-medium, port 7770] + C[Qwen3.8-27B-ROCmFPX-Q8_0-AGENT +layer split, 131k ctx / 1 slot, MTP] A --> B2 --> C end @@ -40,14 +42,18 @@ w4b .hgn + overlay, 262k ctx / 4 slots] | Server | GPU | VRAM / Memory | OS | | --------------------------------------- | ----------------------------- | -------------------- | ------------ | -| [[Dankserver System Specs\|Dankserver]] | NVIDIA RTX 3090 | 24 GB VRAM, 62 GB RAM | Ubuntu 26.04 | +| [[Dankserver System Specs\|Dankserver]] | 2x AMD Radeon AI PRO R9700 | 2x 32 GB VRAM, 60 GB RAM | Ubuntu 26.04 | | [[Dankclaw System Specs\|Dankclaw]] | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 | ## Active Deployment -- [[freetoken-frank-small]] — the active frank-small - runbook: FreeToken v0.1.2 serving batsclamp/Huihui-Qwen3.6-35B-A3B- - abliterated-FP8 on Dankserver (swapped 2026-08-30; same format/pools) +- [[frank-medium-rocmfpx]] — the ACTIVE frank-medium runbook: + charlie12345/ROCmFPX llama.cpp serving kingjones777/ + Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF (MTP draft on) on Dankserver's + 2x Radeon AI PRO R9700 (since 2026-09-15; replaced frank-small) +- [[freetoken-frank-small]] — RETIRED 2026-09-15 with the RTX 3090: the + FreeToken v0.1.2 frank-small runbook (batsclamp Huihui-Qwen3.6-35B-A3B- + abliterated-FP8), kept as the retirement record - [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090 - [[deployment]] — deploy and update process for both servers @@ -80,7 +86,7 @@ w4b .hgn + overlay, 262k ctx / 4 slots] on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model) - [[kairic-edge]] — the retired Kairic Edge IU4 experiment (deployed and reverted 2026-08-22) -- [[Dankserver System Specs]] — the RTX 3090 box (frank-small) +- [[Dankserver System Specs]] — the 2x R9700 box (frank-medium) - [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large, on halogen-flash-server since 2026-09-11) - [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint @@ -106,8 +112,8 @@ w4b .hgn + overlay, 262k ctx / 4 slots] - [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism - [[VLLM Index]] — quick reference with commands, endpoints, and parameter table - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware - notes and sizing (this architecture family is what frank-small - serves today in the Huihui abliterated FP8 variant — see + notes and sizing (this architecture family is what the retired frank-small + served in the Huihui abliterated FP8 variant — see [[freetoken-frank-small]]; the Ornith-1.5-35B-A3B CRACK checkpoint that preceded it is archived at [[dealign-ornith-crack]]) @@ -136,7 +142,7 @@ w4b .hgn + overlay, 262k ctx / 4 slots] - [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit - [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations -- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (3090) vs Dankclaw (Strix Halo) +- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo) ## Related diff --git a/serverAppConfigs-docs/LLM/kingjones-q8agent-rocmfpx.md b/serverAppConfigs-docs/LLM/kingjones-q8agent-rocmfpx.md index 01d98cc..31421f9 100644 --- a/serverAppConfigs-docs/LLM/kingjones-q8agent-rocmfpx.md +++ b/serverAppConfigs-docs/LLM/kingjones-q8agent-rocmfpx.md @@ -1,6 +1,12 @@ # frank-large: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT -**Status:** Deployed on Dankclaw (since 2026-08-25). Serving is **BLOCKED by +> **SUPERSEDED 2026-09-15.** The Q8_0_ROCMFPX-AGENT checkpoint (with the +> separate `draft-mtp` head) now serves Dankserver as **frank-medium** on the +> fork's fixed build — see [[frank-medium-rocmfpx]]. The failure below was +> specific to the gfx1151 legacy-read path and the Q6_0 single-file era. + +**Status (historical, 2026-09-13):** was to deploy on Dankclaw. Serving is +**BLOCKED by the `set_rows` build bug** — the model loads but crashes in `ggml_compute_forward_set_rows` before it can serve. All config goals are met; a build fix is required for the model to run. diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index eb86cd5..b7c9944 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -114,8 +114,11 @@ The LLM inference fleet on the two hosts. Full index: - [[deployment]] — unified deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[freetoken-frank-small]] — the active frank-small runbook: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 on Dankserver (2026-09-02) -- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090 +- [[frank-medium-rocmfpx]] — the ACTIVE frank-medium runbook: + charlie12345/ROCmFPX llama.cpp serving kingjones777/ + Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF on Dankserver's 2x Radeon AI PRO + R9700 (since 2026-09-15; replaced the FreeToken frank-small) +- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on the retired Dankserver RTX 3090 - [[kingjones-strix-mtp]] — the Qwen3.8-27B-Q6_0 ROCmFPX AGENT era with the standalone MTP draft (Dankclaw, 2026-08-25 → 2026-08-29; historical) - [[agentionai-fast-imatrix]] — the retired frank-large runbook: @@ -138,10 +141,11 @@ The LLM inference fleet on the two hosts. Full index: - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (retired 2026-09-02) -- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for frank-small (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26) +- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for the old frank-small (q8_0 + Hadamard KV cache, split-mode none; 2026-08-26) +- [[freetoken-frank-small]] — RETIRED 2026-09-15: the FreeToken/CUDA frank-small runbook, retired with the RTX 3090 - [[kairic-edge]] — the retired Kairic Edge IU4 experiment (2026-08-22) - [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment (2026-08-21) -- [[Dankserver System Specs]] — the RTX 3090 box hardware +- [[Dankserver System Specs]] — the 2x R9700 box hardware (frank-medium) - [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large) - [[Dankclaw Improvements]] — host-level changes on the Strix Halo box - [[Swap Configuration]] — the 64 GiB `/swap.img` host swapfile (raised from diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 2241ac6..fc195c7 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -11,6 +11,47 @@ timestamp: 2026-09-14 # Bundle Change Log +## 2026-09-15 — frank-medium: Dankserver GPU swap (2x R9700), frank-small retired +- The RTX 3090 left Dankserver; two AMD Radeon AI PRO R9700 cards (Navi 48, + gfx1201, 32,624 MiB each) replaced it. FreeToken/CUDA cannot run there, so + `frank-small` is retired and `frank-medium` took the :7770 endpoint. +- Runtime: charlie12345/ROCmFPX llama.cpp `c49ebdb` built for **gfx1201** with + the fork's self-contained ROCm 7.14 toolchain (TheRock nightly, RPATH-bundled + libs) — the host's broken mixed ROCm (AMD-repo 6.3.3 hijacking libhsa via + `/etc/ld.so.conf.d/10-rocm-opencl.conf`) is fully bypassed. Model: + kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF (ftype 115, 26.28 GiB) + + Q4_0 MTP draft head; mmproj unused (vision stays on frank-large). +- **Parallelism decision, measured not argued:** llama-bench with the correct + multi-device syntax (`-dev ROCm0/ROCm1` — comma form silently benches each + card separately): layer 1991/18.8 t/s (pp4096/tg128) vs tensor 271.8/26.3 vs + single card 1160.9/19.4. Tensor's decode win costs a 7x prefill collapse + (no RCCL → "meta-backend butterfly" allreduce); served with MTP, layer also + wins decode (35.7–46.2 tok/s vs 28.7). Operator picked layer ("pipeline + parallelism"); the data agrees on both axes. Draft head pinned to ROCm1 + (card holding the last target layers); under tensor mode an unpinned draft + aborts the KV build (GGML_ASSERT(buffer) in ggml-backend.cpp). +- Verified live before and after the systemd cutover: health ~18 s, MTP + acceptance 60–69 %, cold 949-token prefill in 2036 ms, and **prefix caching + works with the draft head** (turn 2 reprocessed 22 tokens, cache_n 945) — + the model card's mtp-prompt-cache-fix.patch defect is already fixed in + `c49ebdb` (`common_prompt_checkpoint::data_spec`). +- Deploy: `deploy.sh dankserver --service frank-medium` (new + `retire_root_llm_small()` archived the CUDA-era unit to + `~/.AGENTTRASH/20260915-frank-small-retire/host-units/`); e2e gate 30/30 + including 11-admin (the blankpad→192.168.0.112 path issue has cleared). +- Host cleanup (the "Both" half of the plan): purged 167 stale AMD-repo + `~24.04` ROCm 6.3.3 packages, removed the radeon apt pin and the + ld.so.conf hijack (staged in `~/.AGENTTRASH/20260915-rocm633-cleanup/`); + `libhsa-runtime64` now resolves only to Ubuntu's ROCm 7.1 and `rocm-smi` + 7.1.1 (Ubuntu package) works. frank-medium unaffected (self-contained). +- Client cutover: blankpad `models.yml`/`config.yml` renamed the model id to + `frank-medium` (ctx 131072, maxTokens 32768); Hermes live config on Dankclaw + repointed `dankserver-small` routes to `frank-medium`. +- New: `/serverAppConfigs-docs/LLM/frank-medium-rocmfpx.md` runbook; + updated AGENTS.md, README, docs indexes, Dankserver System Specs, + deployment/Hermes/OpenWebUI/Plex/Stack pages; freetoken-frank-small and + kingjones-q8agent marked retired/superseded. + ## 2026-09-12 — Marreta content-cleaning proxy deployed on Dankclaw - Added `manualdousuario/marreta` (Laravel / `serversideup/php:8.4-fpm-nginx`) as a rootless Quadlet pair: `marreta` + `marreta-browser` (Lightpanda),