docs: rename docs/ -> serverAppConfigs-docs/ per system-wide PROJECTNAME-docs convention
All in-repo path references (AGENTS.md, README, old scripts, report artifacts,
test code paths) rewritten in the same commit; focused-test map extended for
.gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's
own convention ('add one before committing'). Obsidian wikilinks are
title-based and unaffected.
This commit is contained in:
Frank
+8
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"alwaysUpdateLinks": true,
|
||||
"newFileLocation": "folder",
|
||||
"newFileFolderPath": "Dankclaw",
|
||||
"attachmentFolderPath": "_attachments",
|
||||
"useMarkdownLinks": false,
|
||||
"showUnsupportedFiles": true
|
||||
}
|
||||
@@ -0,0 +1,128 @@
|
||||
---
|
||||
type: guide
|
||||
title: Deployment Flow
|
||||
description: Service-scoped Git deployment for the two homelab hosts.
|
||||
tags:
|
||||
- deployment
|
||||
- operations
|
||||
- git
|
||||
timestamp: 2026-09-03
|
||||
---
|
||||
|
||||
# Deployment Flow
|
||||
|
||||
Git is the single source of truth. `scripts/deploy.sh` pushes the committed
|
||||
`main` branch, pulls it on one target, and applies only the runtime services
|
||||
whose tracked inputs changed. A SearXNG change no longer recreates the
|
||||
inference service; an LLM change no longer restarts unrelated containers.
|
||||
|
||||
## Workflow
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Edit in this repo] --> B[Run tests]
|
||||
B --> C[Commit main]
|
||||
C --> D[deploy.sh computes plan]
|
||||
D --> E[Push main]
|
||||
E --> F[Target pulls with ff-only]
|
||||
F --> G[deploy-remote.sh dispatches selected services]
|
||||
G --> H[Per-service health check]
|
||||
```
|
||||
|
||||
1. Edit files locally in this repository.
|
||||
2. Run `./tests/run.sh`, then commit the complete change on `main`.
|
||||
3. Run `./scripts/deploy.sh <target>`.
|
||||
4. The planner maps changed paths to services. Compose YAML is compared by
|
||||
service definition so an edit to one service does not restart its peers.
|
||||
5. The target pulls with `git pull --ff-only origin main` and runs the remote
|
||||
dispatcher with the explicit plan.
|
||||
6. The dispatcher updates only selected Quadlet, compose, homepage, Caddy, or
|
||||
systemd services and records each successful service revision outside Git.
|
||||
|
||||
## Commands
|
||||
|
||||
```bash
|
||||
./scripts/deploy.sh dankclaw
|
||||
./scripts/deploy.sh dankserver
|
||||
./scripts/deploy.sh dankclaw --service searxng
|
||||
./scripts/deploy.sh dankserver --service frank-default-small
|
||||
./scripts/deploy.sh dankclaw --dry-run
|
||||
```
|
||||
|
||||
`--service` is repeatable and is an explicit retry/override. Without it, the
|
||||
planner compares the local `HEAD` to `origin/main`. A documentation-only
|
||||
commit produces an empty plan. The remote state directory can also replay a
|
||||
service whose earlier deployment failed, but it never expands a plan to a
|
||||
service on the other host.
|
||||
|
||||
## Service managers
|
||||
|
||||
| Service family | Runtime manager | Scope behavior |
|
||||
|---|---|---|
|
||||
| Dankclaw compose services | `podman-compose -f dankclaw/compose.yaml up -d --no-deps NAME` | Recreates only `NAME`; Caddy is handled separately because it fronts both hosts. |
|
||||
| SearXNG, Gluetun, Valkey, OpenWebUI, Pi-hole, Gitea database/server | user Quadlet + `systemctl --user` | Installs the selected unit and restarts only the required dependency-coupled units. Gitea's web proxy is the canonical Caddy compose service. |
|
||||
| `frank-default-large` | systemd + `systemctl` | Relinks and restarts only the large inference service; waits up to 20 minutes for `/health`. |
|
||||
| `frank-default-small` | systemd + `systemctl` | Relinks and restarts only the small inference service; waits up to 20 minutes for `/health`. |
|
||||
| Dankserver media inventory | host-managed bare containers | Compose is inventory only; the scoped deploy does not create duplicate containers. |
|
||||
|
||||
The shared chat template maps to the inference service on the selected target.
|
||||
The inactive `frank-default-npu` unit is relink-only when explicitly selected;
|
||||
it is not restarted by an unrelated large-model deployment.
|
||||
|
||||
## Caddy safety
|
||||
|
||||
Caddy runs on Dankclaw and fronts both hosts, including Gitea. The former
|
||||
standalone `gitea-caddy` Quadlet is retired; the canonical compose `caddy`
|
||||
service is now the sole reverse-proxy owner of ports 80/443. A Caddy reload
|
||||
failure falls back to a restart of **only** the selected `caddy` container.
|
||||
Before deploying, verify that no retired proxy still owns either port.
|
||||
|
||||
Homepage files, generated `deployed.js`, the status checker, and the rendered
|
||||
Caddyfile are updated only when their service is selected. A media-only
|
||||
Dankserver deployment does not touch Dankclaw's proxy or homepage.
|
||||
|
||||
## Per-service state
|
||||
|
||||
The remote runner writes one revision file per successfully completed service
|
||||
to:
|
||||
|
||||
```text
|
||||
${XDG_STATE_HOME:-~/.local/state}/serverAppConfigs/deploy-state/
|
||||
```
|
||||
|
||||
The directory is outside the checkout and is never committed. A failed handler
|
||||
is not marked successful. On the next ordinary deployment, the runner compares
|
||||
the recorded revision with the new checkout and retries only a still-changed
|
||||
service. This prevents a failed Caddy or LLM deployment from being silently
|
||||
forgotten while avoiding a host-wide restart.
|
||||
|
||||
## Targets
|
||||
|
||||
| Target | Host | Repository checkout |
|
||||
|---|---|---|
|
||||
| dankclaw | `dank@dankclaw` (`192.168.0.143`) | `/home/dank/serverAppConfigs` |
|
||||
| dankserver | `dank@dankserver` (`192.168.0.235`) | `/home/dank/serverAppConfigs` |
|
||||
|
||||
SSH uses the existing key pair. No password is stored or documented in this
|
||||
repository. If sudo is needed on a headless target, `FRANK_SUDO_PW` is read
|
||||
from the environment and is never written to the repository.
|
||||
|
||||
## Tests
|
||||
|
||||
Run the full suite before pushing:
|
||||
|
||||
```bash
|
||||
./tests/run.sh
|
||||
```
|
||||
|
||||
The deployment seams are in `tests/deployment_scoping.bats` and
|
||||
`tests/deployment_flow.bats`. They cover path isolation, compose narrowing,
|
||||
Caddy safety, inference isolation, per-service state, and legacy wrapper
|
||||
routing. See [[Testing Policy]].
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Testing Policy]]
|
||||
- [LLM deployment](../LLM/deployment.md)
|
||||
- [[Caddy]]
|
||||
- [[SearXNG]]
|
||||
@@ -0,0 +1,63 @@
|
||||
# Network Architecture
|
||||
|
||||
One Caddy fronts both hosts. It runs as a container on Dankclaw (192.168.0.143) and terminates TLS for every service on the LAN. Pi-hole on Dankclaw answers the service names for LAN clients; Porkbun answers them for the WAN. The G36 router firewall governs external access. SearXNG outbound traffic exits through the Gluetun VPN container.
|
||||
|
||||
## Topology
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph WAN["G36 router 192.168.0.1"]
|
||||
EXT[Internet]
|
||||
end
|
||||
EXT -->|port forwards| C
|
||||
subgraph Dankclaw[192.168.0.143]
|
||||
C[Caddy :80 :443]
|
||||
D[Pi-hole :53]
|
||||
C -->|comfyui:8188| F[ComfyUI]
|
||||
C -->|192.168.0.143:3000| G[Gitea quadlet]
|
||||
C -->|192.168.0.143:8081| SE[SearXNG quadlet]
|
||||
C -->|192.168.0.143:3005| O[OpenWebUI quadlet]
|
||||
SE -->|network_mode service| GL[Gluetun VPN]
|
||||
GL -->|tunnel| EXT
|
||||
end
|
||||
subgraph Dankserver[192.168.0.235]
|
||||
C -->|192.168.0.235:8085| R[RomM]
|
||||
C -->|192.168.0.235:3923| P[Copyparty]
|
||||
C -->|192.168.0.235:8083| W[Calibre-Web]
|
||||
C -->|192.168.0.235:32400| PL[Plex host service]
|
||||
end
|
||||
CLIENT[LAN clients] -->|DHCP DNS| D
|
||||
```
|
||||
|
||||
Dankserver runs no proxy of its own. Its services expose published LAN ports, and the Dankclaw Caddy routes the names to those ports.
|
||||
|
||||
## Hostnames
|
||||
|
||||
| Scope | Convention | Resolved by |
|
||||
|-------|------------|-------------|
|
||||
| LAN | `romm.franksdank.cloud` → 192.168.0.143 | Pi-hole (`addn-hosts`) |
|
||||
| WAN | `romm.franksdank.cloud` → 71.227.227.172 | Porkbun A record |
|
||||
|
||||
Users type a name and nothing else. Caddy maps the name to the upstream. The split between the two resolvers removes the need for router NAT loopback. See [[LAN-DNS]] for the flow and the router hand-off.
|
||||
|
||||
## TLS
|
||||
|
||||
Caddy holds one Let's Encrypt certificate per vhost, issued and renewed through the DNS-01 challenge via the Porkbun module. No inbound port 80/443 is required. Details in [[Caddy]].
|
||||
|
||||
## Gluetun Path (SearXNG only)
|
||||
|
||||
SearXNG attaches to the Gluetun network namespace (`network_mode: service:gluetun`). Only SearXNG shares the tunnel. Everything else on Dankclaw uses the normal LAN route. This matches KB Pattern 2 in [[Gluetun]].
|
||||
|
||||
## Access Policy
|
||||
|
||||
- The **G36 router firewall** governs what the WAN can reach. Port forwards are manual router steps; the repo tracks no router state.
|
||||
- Caddy adds HTTP basic auth on the homepage and on sensitive services.
|
||||
- LAN subnet: `192.168.0.0/24`.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[LAN-DNS]] — name resolution, Pi-hole, router hand-off
|
||||
- [[Caddy]] — vhost map, TLS, basic auth
|
||||
- [[Dankserver Stack]], [[Dankclaw Stack]]
|
||||
- [[Gluetun]]
|
||||
- Knowledge base: [[Gluetun]], [[SearXNG]]
|
||||
@@ -0,0 +1,40 @@
|
||||
# Storage Policy
|
||||
|
||||
All persistent data uses **bind mounts**: a host directory mapped directly into the container. No named volumes. This matches the preference of this repo and the existing Gitea stack on Dankclaw.
|
||||
|
||||
## Why Bind Mounts
|
||||
|
||||
- The exact host location of every byte is visible and controllable.
|
||||
- Backups are a simple `tar` of known host directories.
|
||||
- The data survives container recreation and `podman-compose down`.
|
||||
- Ownership is explicit: everything runs as UID 1000 (`dank`) so host tools can read it.
|
||||
|
||||
## Conventions
|
||||
|
||||
| Stack | Host root |
|
||||
|-------|-----------|
|
||||
| Dankserver | `/opt/<service>/...` |
|
||||
| Dankclaw | `/opt/<service>/...` (Gitea keeps `/opt/gitea/data/...`) |
|
||||
|
||||
## Ownership
|
||||
|
||||
Containers run as `PUID=1000` / `PGID=1000`. Host directories must be owned by `dank`:
|
||||
|
||||
```bash
|
||||
sudo chown -R 1000:1000 /opt/<service>
|
||||
```
|
||||
|
||||
## Secrets
|
||||
|
||||
Secrets never live in bind-mounted configs that get committed. They are:
|
||||
|
||||
1. Set in `.env` files on the host (git-ignored, `.env.example` committed as template), or
|
||||
2. Injected as environment variables from the deploy script.
|
||||
|
||||
The live Gitea database password exists only in `.env` on Dankclaw. A test enforces that it never appears in this repo.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Deployment Flow]]
|
||||
- [[Gitea]] — the existing bind-mount stack being migrated
|
||||
- [[Backblaze Integration]] — future backup target for these mounts
|
||||
@@ -0,0 +1,45 @@
|
||||
# Testing Policy
|
||||
|
||||
This repository tests its configuration at defined seams. The tests run with `bats`; run them with `./tests/run.sh` before every push.
|
||||
|
||||
## Seams Under Test
|
||||
|
||||
| Seam | File | What it asserts |
|
||||
|------|------|-----------------|
|
||||
| Compose validity | `tests/compose_validity.bats` | YAML parses, services have images, no named volumes |
|
||||
| Env consistency | `tests/env_consistency.bats` | every `${VAR}` is defined in `.env.example`; no real secrets; the live gitea password never appears |
|
||||
| Caddyfile validity | `tests/caddyfile.bats` | per-host Caddyfile, `.local` convention, balanced braces, basic auth present |
|
||||
| Docs completeness | `tests/docs_completeness.bats` | every compose service has a docs page; index links every page |
|
||||
| Script syntax | `tests/scripts_syntax.bats` | `bash -n` clean, executable bits, shellcheck when available |
|
||||
| Deployment flow | `tests/deployment_flow.bats`, `tests/deployment_scoping.bats` | scoped dispatcher exists, targets are known, service plans isolate inference, Caddy ownership is guarded, and SSH reachability is checked (skips offline) |
|
||||
|
||||
## Focused pre-commit testing
|
||||
|
||||
For normal commits, use the focused wrapper:
|
||||
|
||||
```bash
|
||||
./scripts/test-changed.sh
|
||||
```
|
||||
|
||||
It reads staged paths with `git diff --cached` and runs only the mapped Bats
|
||||
files. The tracked `.githooks/pre-commit` hook invokes this wrapper. Enable the
|
||||
repository hook path once per checkout:
|
||||
|
||||
```bash
|
||||
git config core.hooksPath .githooks
|
||||
```
|
||||
|
||||
Use `--list` to inspect the selection without running tests, or `--paths FILE
|
||||
...` to preview a mapping. Shared test infrastructure and unmapped paths fail
|
||||
closed rather than silently skipping coverage. Run the full suite with
|
||||
`./tests/run.sh` when changing shared test helpers, the test runner, or when
|
||||
focused coverage is not sufficient.
|
||||
|
||||
## Red-Green
|
||||
|
||||
Tests are written first. A failing test means the seam is not satisfied; the config change that makes it pass is the only acceptable fix. No `--no-verify`, no deleting tests without approval.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Deployment Flow]]
|
||||
- [[index]]
|
||||
@@ -0,0 +1,79 @@
|
||||
# ComfyUI
|
||||
|
||||
ComfyUI is a node-based AI image generation engine. Workflows are built from nodes that load models, sample latents, and post-process images; everything is scriptable through a web UI and an API.
|
||||
|
||||
**Host:** Dankclaw — **Hostname:** `comfyui.dankclaw.local` — **Port:** 8188 — **Backend:** ROCm GPU (gfx1151) — **Docs:** https://github.com/Comfy-Org/ComfyUI
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `comfyui-rocm:gfx1151` — built from `dankclaw/Containerfile.comfyui` |
|
||||
| Base image | `docker.io/rocm/vllm:rocm7.12.0_gfx1151_ubuntu24.04_py3.12_pytorch_2.9.1_vllm_0.16.0` |
|
||||
| Port | 8188 (host) → 8188 (container) |
|
||||
| Backend | ROCm GPU (gfx1151 / Radeon 8060S). No `--cpu` flag. |
|
||||
| Data | `/home/dank/comfyui-data/{models,input,output,custom_nodes}` → `/opt/ComfyUI/...` |
|
||||
| Run | `podman compose -f dankclaw/compose.yaml up -d comfyui` |
|
||||
| Boot-persistent | user systemd unit `comfyui.service` (runs the compose `up` at boot) |
|
||||
|
||||
## Why the GPU works here
|
||||
|
||||
Dankclaw is an AMD Strix Halo APU (gfx1151). Standard PyTorch ROCm wheels do not ship gfx1151 kernels. So `torch.cuda.is_available()` returns `False` and ComfyUI falls back to CPU. AMD publishes a gfx1151-tagged build, and the image above is based on it. On this APU it reports `torch.cuda.is_available() = True`, device `Radeon 8060S`, capability `(11, 5)`. See [[ComfyUI ROCm and Vulkan]] for the full verification.
|
||||
|
||||
## How the image is built
|
||||
|
||||
`dankclaw/Containerfile.comfyui` starts from the AMD gfx1151 base image and bakes the ComfyUI code in. It installs ComfyUI's dependencies but skips the torch family (torch, torchvision, torchaudio). The base image already ships the matching ROCm builds. Installing an unpinned `torch` from `requirements.txt` clobbers the gfx1151 kernels with a default build, so the Containerfile filters those lines out.
|
||||
|
||||
## GPU device access (rootless)
|
||||
|
||||
The container reaches the GPU through the `/dev/kfd` and `/dev/dri` device passthrough. The permission is checked host-side. The run user is already a member of the `render` group on the host, so the device is readable.
|
||||
|
||||
Only the `video` group is added to the container. The `render` group (gid 990) is not added. Rootless podman cannot map it into the container namespace (the subgid range is 100000+), so `--group-add render` fails to start the container.
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path |
|
||||
|----------------|-----------|
|
||||
| `/opt/ComfyUI/models` | `/home/dank/comfyui-data/models` |
|
||||
| `/opt/ComfyUI/input` | `/home/dank/comfyui-data/input` |
|
||||
| `/opt/ComfyUI/output` | `/home/dank/comfyui-data/output` |
|
||||
| `/opt/ComfyUI/custom_nodes` | `/home/dank/comfyui-data/custom_nodes` |
|
||||
|
||||
The ComfyUI code is baked into the image (not mounted). Rebuild the image to update the code.
|
||||
|
||||
## Custom nodes
|
||||
|
||||
Custom nodes go in `/home/dank/comfyui-data/custom_nodes` (mounted at `/opt/ComfyUI/custom_nodes`). ComfyUI loads every subdirectory there on start. Because this path is a bind mount, it shadows anything baked into the image — so custom nodes are installed host-side, not in the Containerfile.
|
||||
|
||||
**ComfyUI-Manager** is installed here. Install it with a one-time clone, then restart the container:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/ltdrdata/ComfyUI-Manager \
|
||||
/home/dank/comfyui-data/custom_nodes/ComfyUI-Manager
|
||||
systemctl --user restart comfyui
|
||||
```
|
||||
|
||||
ComfyUI-Manager self-installs its own pip dependencies on start (GitPython, PyGithub, matrix-nio, transformers, and more). To confirm it loaded, query `GET /manager/version` (returns a version like `V3.41`). Its pip installs live in the container's writable layer, so they survive a restart and a reboot but not an image rebuild; after a rebuild, Manager re-installs them on the next start.
|
||||
|
||||
## Boot persistence
|
||||
|
||||
The user systemd unit `~/.config/systemd/user/comfyui.service` runs `podman compose -f /home/dank/serverAppConfigs/dankclaw/compose.yaml up -d comfyui` at boot. It sets `WantedBy=default.target`, and user linger is on. The old CPU venv unit is preserved at `comfyui.service.venv.bak` as a fallback.
|
||||
|
||||
## Healthcheck
|
||||
|
||||
The web UI responds on `/`. ComfyUI also exposes `/system_stats`. On this APU, `/system_stats` reports `device=cuda:0 Radeon 8060S Graphics` (type `cuda`, the ROCm device).
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The GPU path needs the device passthrough. Do not remove `/dev/kfd` or `/dev/dri` from the compose service.
|
||||
- Do not add the `render` group to the container. Rootless podman cannot map it. Add only `video`.
|
||||
- Do not let pip reinstall the torch family. The Containerfile filters it out on purpose.
|
||||
- The base image is about 32 GB. A code update means a rebuild, because the code is baked in.
|
||||
- The model collection starts empty. Add models to `/home/dank/comfyui-data/models` before generating.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
- [[ComfyUI ROCm and Vulkan]] — the live-verified GPU path research
|
||||
- [[Lemonade and sd.cpp]] — the ggml-family alternative
|
||||
- Knowledge base: [[ComfyUI]] (tools)
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
type: reference
|
||||
title: Dankclaw Stack
|
||||
description: Service inventory and routing overview for the Dankclaw AI, development, and observability host.
|
||||
tags:
|
||||
- dankclaw
|
||||
- services
|
||||
- containers
|
||||
- caddy
|
||||
timestamp: 2026-08-26
|
||||
---
|
||||
|
||||
# Dankclaw Stack
|
||||
|
||||
Dankclaw (192.168.0.143) runs the AI, search, and development stack. It
|
||||
hosts the LLM inference endpoint (`frank-default-large`, `:7770`, CIRU
|
||||
ROCmFPX llama.cpp), the agent tools, the privacy search, and the
|
||||
observability stack. The llama.cpp endpoint (`frank-default-small`,
|
||||
`:7770`) runs on Dankserver (192.168.0.235).
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
U[User] --> C[Caddy :80/:443]
|
||||
C --> O[OpenWebUI]
|
||||
C --> S[SearXNG]
|
||||
C --> G[Gitea]
|
||||
C --> F[ComfyUI]
|
||||
C --> D[Grafana]
|
||||
S --> V[Gluetun VPN]
|
||||
O --> L[llama.cpp dankserver :7770]
|
||||
O --> H[Hermes]
|
||||
O --> E[CIRU llama.cpp :7770]
|
||||
```
|
||||
|
||||
## Services
|
||||
|
||||
| Service | Hostname | Port | Docs |
|
||||
|---------|----------|------|------|
|
||||
| SearXNG | search.dankclaw.local | 8081 | [[SearXNG]] |
|
||||
| ComfyUI | comfyui.dankclaw.local | 8188 | [[ComfyUI]] |
|
||||
| OpenWebUI | chat.dankclaw.local | 3005 | [[OpenWebUI]] |
|
||||
| Hermes | hermes.dankclaw.local | 8090 | [[Hermes]] |
|
||||
| Gitea | gitea.dankclaw.local | 3000 | [[Gitea]] |
|
||||
| Grafana | grafana.dankclaw.local | 3002 | [[Grafana]] |
|
||||
| Prometheus | prometheus.dankclaw.local | 9090 | [[Prometheus]] |
|
||||
| Homepage | dankclaw.local | 80/443 | — |
|
||||
| frank-default-large (host service, Dankclaw) | — | 7770 | [[deployment]] |
|
||||
| frank-default-small (host service, Dankserver) | — | 7770 | [[deployment]] |
|
||||
|
||||
The last two rows are bare-metal systemd services, not containers. This
|
||||
repo owns their scripts, units, and knowledge base (see
|
||||
[serverAppConfigs-docs/LLM/index.md](../LLM/index.md)). Current mapping (2026-08-22):
|
||||
`frank-default-large` runs the CIRU ROCmFPX llama.cpp on Dankclaw;
|
||||
`frank-default-small` runs the upstream llama.cpp on Dankserver.
|
||||
|
||||
## Boot Persistence
|
||||
|
||||
The Gitea database and server, the SearXNG trio (gluetun, searxng, valkey), open-webui, Pi-hole, and Caddy run as Quadlet units, so they start at boot. Caddy left `compose.yaml` on 2026-08-26 after a reboot exposed that bare compose containers never come back; see [[Caddy]] for the unit and the incident. The former standalone Gitea Caddy unit is retired. User linger is on. ComfyUI runs as a systemd unit around a bare process. The remaining services in the table are defined in `compose.yaml` and are not yet boot-persistent — per AGENTS.md (Boot Persistence & Self-Healing) they each need a quadlet before they may be considered reliably deployed.
|
||||
|
||||
Deploy the SearXNG trio with `scripts/deploy-searxng-quadlet.sh dankclaw`, open-webui with `scripts/deploy-open-webui-quadlet.sh dankclaw`, and Pi-hole with `scripts/deploy-pihole-quadlet.sh dankclaw`. The Gitea units install to `~/.config/containers/systemd/` (see [[Gitea]]); Caddy is deployed through `scripts/deploy.sh dankclaw --service caddy`.
|
||||
|
||||
## Routing Rules
|
||||
|
||||
- SearXNG outbound traffic exits through the [[Gluetun]] VPN container.
|
||||
- Every other service uses the normal LAN route.
|
||||
- The homepage lists all services with status.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[index]] — repository index
|
||||
- [[Network Architecture]] — routing and proxy
|
||||
- [[Observability]] — Grafana, Prometheus, logs
|
||||
- [LLM fleet knowledge base](../LLM/index.md) — the model-serving endpoints on this host
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
type: configuration
|
||||
title: Gitea
|
||||
description: Gitea deployment on Dankclaw with its web UI reverse-proxied by the canonical Caddy service.
|
||||
tags:
|
||||
- gitea
|
||||
- dankclaw
|
||||
- quadlet
|
||||
- caddy
|
||||
timestamp: 2026-08-26
|
||||
---
|
||||
|
||||
# Gitea
|
||||
|
||||
Gitea is a self-hosted Git service with issues, pull requests, and a web UI. On Dankclaw it runs as two Quadlet-managed containers in the rootless Podman session of user `dank`. The canonical Caddy on the same host reverse-proxies its web UI at `gitea.franksdank.cloud`.
|
||||
|
||||
**Host:** Dankclaw — **LAN access:** `http://dankclaw:3000` (or `http://192.168.0.143:3000`) — **Proxy domain:** `gitea.franksdank.cloud` — **Docs:** https://docs.gitea.com
|
||||
|
||||
## Containers
|
||||
|
||||
| Container | Unit | Image | Purpose |
|
||||
|-----------|------|-------|---------|
|
||||
| `gitea-postgres` | `gitea-postgres.service` | `docker.io/library/postgres:16-alpine` | Database |
|
||||
| `gitea-server` | `gitea-server.service` | `docker.io/gitea/gitea:latest` | Web UI, API, git (HTTP :3000, SSH :2222) |
|
||||
|
||||
The containers share the private bridge network `systemd-gitea`. The quadlet network file `gitea.network` enables DNS, so the containers resolve each other by name. The automatic `podman` default network has DNS disabled. Do not run the Gitea containers on it.
|
||||
|
||||
## Hosted repositories
|
||||
|
||||
| Repository | Purpose |
|
||||
|------------|---------|
|
||||
| `dank/serverAppConfigs` | this repo — Podman stacks + LLM fleet |
|
||||
| `dank/vllmScripts` | retired 2026-08-22 — merged into `dank/serverAppConfigs` (history preserved) |
|
||||
|
||||
The LLM fleet knowledge base: [serverAppConfigs-docs/LLM/index.md](../LLM/index.md)
|
||||
|
||||
## Management
|
||||
|
||||
Run all commands as `dank` on Dankclaw.
|
||||
|
||||
| Action | Command |
|
||||
|--------|---------|
|
||||
| Start all | `systemctl --user start gitea-postgres gitea-server` |
|
||||
| Stop all | `systemctl --user stop gitea-server gitea-postgres` |
|
||||
| Restart Gitea only | `systemctl --user restart gitea-server` |
|
||||
| Status | `systemctl --user status gitea-postgres gitea-server` |
|
||||
| Container logs | `podman logs -f gitea-server` |
|
||||
| Unit logs | `journalctl --user -u gitea-server -f` |
|
||||
|
||||
The units start in dependency order:
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
N[gitea-network.service] --> P[gitea-postgres.service]
|
||||
P --> S[gitea-server.service]
|
||||
```
|
||||
|
||||
## Boot persistence
|
||||
|
||||
User-level linger is enabled for `dank` (`loginctl enable-linger dank`). The units start at host boot without a login session. Each unit carries `[Install] WantedBy=default.target`. The podman generator applies it at boot; no symlink is needed.
|
||||
|
||||
The containers use the quadlet fresh-start model. A stop removes the container (`podman run --rm --replace`). The next start recreates it. The bind-mounted data survives. The canonical Caddy is managed separately by the Dankclaw stack.
|
||||
|
||||
## Secrets
|
||||
|
||||
The database password lives in `/opt/gitea/.env` on the host (mode 0600). The quadlet files reference it with `EnvironmentFile=`. The password never enters this repository. Placeholders live in [`dankclaw/gitea/.env.example`](../../dankclaw/gitea/.env.example).
|
||||
|
||||
Do not change the password in place. Change the PostgreSQL role first, then update `/opt/gitea/.env` and the `[database] PASSWD` line in the app.ini (see Gotchas), then restart the stack.
|
||||
|
||||
## Configuration files
|
||||
|
||||
| File | Location | Repo copy |
|
||||
|------|----------|-----------|
|
||||
| Quadlet units | `~/.config/containers/systemd/gitea-{postgres,server}.container`, `gitea.network` | [`dankclaw/gitea/quadlet/`](../../dankclaw/gitea/quadlet/) |
|
||||
| Caddyfile | `/home/dank/Containers/caddy/Caddyfile` | [`dankclaw/Caddyfile`](../../dankclaw/Caddyfile) |
|
||||
| Secrets | `/opt/gitea/.env` (0600) | [`dankclaw/gitea/.env.example`](../../dankclaw/gitea/.env.example) |
|
||||
|
||||
After editing a repo copy, deploy through `scripts/deploy.sh dankclaw`. The Gitea database/server units are installed from this directory; the canonical Caddy and its rendered Caddyfile are deployed as the separate `caddy` service. Do not manually recreate the retired `gitea-caddy` unit.
|
||||
|
||||
## Data volumes
|
||||
|
||||
| Container path | Host path | Contents |
|
||||
|----------------|-----------|----------|
|
||||
| `/data` | `/opt/gitea/data/gitea-gitea` | Git repositories, app.ini, attachments |
|
||||
| `/var/lib/postgresql/data` | `/opt/gitea/data/gitea-postgres` | PostgreSQL data |
|
||||
|
||||
Back up `/opt/gitea/data/` (tar) or `pg_dump` the database. See the knowledge base page [[Gitea]].
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **app.ini wins over env vars.** Gitea writes `/data/gitea/conf/app.ini` (host: `/opt/gitea/data/gitea-gitea/gitea/conf/app.ini`) on first start. The `[database] HOST` and `PASSWD` lines persist and override the `DATABASE_HOST`/`DATABASE_PASSWD` env vars. When you rename the database container, edit the `HOST` line.
|
||||
- **The files in the data directory are owned by host UID 100999**, not `dank` (rootless ID mapping: container uid 1000 → host 100999). Edit them inside the container with `podman exec --user 1000 gitea-server ...`.
|
||||
- **The host auto-reboots.** `unattended-upgrades` reboots Dankclaw when a package needs it. The quadlet units make Gitea survive that. See the 2026-08-16 incident below.
|
||||
- **Port 3000 vs 3001.** The live host port is 3000. The full-stack [`dankclaw/compose.yaml`](../../dankclaw/compose.yaml) maps 3001. Align the two before deploying the full stack.
|
||||
- **Legacy compose retired.** `/opt/gitea/podman-compose.yml` is retired to `/opt/gitea/.deprecated/`. Do not run it. Its containers collide with the Quadlet units on ports 3000 and 2222.
|
||||
- **Canonical proxy.** `gitea.franksdank.cloud` is served by the Caddy service in `dankclaw/compose.yaml`, which proxies to `192.168.0.143:3000`. Do not recreate a Gitea-specific Caddy unit or publish another proxy on ports 80/443. During migration, the deployer stages any installed legacy unit under `~/.AGENTTRASH/YYYYMMDD-gitea-caddy/` before starting canonical Caddy.
|
||||
|
||||
## Incident: 2026-08-16 — down after unattended reboot
|
||||
|
||||
`unattended-upgrades` rebooted Dankclaw at 15:01 on 2026-08-16. The Gitea containers ran through podman-compose with `restart: unless-stopped` and no systemd units. Rootless podman does not auto-start containers at boot without units. Gitea stayed down until a manual start at 16:02. The fix: migrate the stack to quadlet units with linger (this page). After the fix, verify on the next reboot.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
- [[Postgres]]
|
||||
- Knowledge base: [[Gitea]] (tools)
|
||||
@@ -0,0 +1,74 @@
|
||||
# Gluetun
|
||||
|
||||
Gluetun is a lightweight VPN client container. It connects to a VPN provider (WireGuard or OpenVPN) and shares that tunnel with other containers that attach to its network namespace.
|
||||
|
||||
**Host:** Dankclaw — **Docs:** https://github.com/qdm12/gluetun
|
||||
|
||||
## Facts (from research, primary source)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `qmcgaw/gluetun:latest` |
|
||||
| Capabilities | `NET_ADMIN` required |
|
||||
| Device | `/dev/net/tun:/dev/net/tun` |
|
||||
| Ports published | searxng's 8081 → 8080 lives here |
|
||||
| Tunnel check | the `tun0` interface exists (the `:8000` server 404s on all paths) |
|
||||
|
||||
## Required Environment
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
VPN_SERVICE_PROVIDER: "${VPN_SERVICE_PROVIDER}" # e.g. mullvad, protonvpn
|
||||
VPN_TYPE: wireguard
|
||||
WIREGUARD_PRIVATE_KEY: "${WIREGUARD_PRIVATE_KEY}" # from .env, never committed
|
||||
```
|
||||
|
||||
- `VPN_TYPE` default is `openvpn`; WireGuard preferred when the provider supports it.
|
||||
- Provider-specific variables come from the Gluetun wiki; set them in `.env`.
|
||||
|
||||
## Optional DNS (from research)
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|----------|---------|---------|
|
||||
| `DNS_UPSTREAM_RESOLVER_TYPE` | `dot` | DNS over TLS |
|
||||
| `DNS_UPSTREAM_RESOLVERS` | `cloudflare` | upstream DoT providers |
|
||||
| `DNS_CACHING` | `on` | cache queries |
|
||||
| `BLOCK_MALICIOUS` | `on` | block malicious hostnames |
|
||||
|
||||
Warning from the wiki: do NOT set `DNS_UPSTREAM_PLAIN_ADDRESSES` to a LAN/bridge address — DNS queries leak out of the tunnel.
|
||||
|
||||
## Attaching a Container
|
||||
|
||||
The searxng unit reuses gluetun's network namespace:
|
||||
|
||||
```ini
|
||||
[Container]
|
||||
ContainerName=searxng
|
||||
Network=gluetun.container
|
||||
```
|
||||
|
||||
`Network=gluetun.container` also adds a systemd dependency on gluetun, so searxng starts after the tunnel.
|
||||
|
||||
## Boot Persistence
|
||||
|
||||
gluetun runs as the `gluetun` quadlet unit (see [[SearXNG]] for the full trio). It sets `WantedBy=default.target` and user linger is on, so the tunnel starts at boot. The WireGuard credentials live in `/opt/searxng/.env` (mode 0600). Deploy with `scripts/deploy-searxng-quadlet.sh dankclaw`.
|
||||
|
||||
## Verify the Tunnel
|
||||
|
||||
```bash
|
||||
podman exec gluetun curl -s ifconfig.me
|
||||
```
|
||||
|
||||
The returned IP must differ from the host IP.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The container that holds the published port must be the one with the ports mapping (gluetun).
|
||||
- `restart: always` on gluetun so the tunnel reconnects after VPN drops.
|
||||
- When gluetun restarts, attached containers inherit the new network state; searxng may need a restart to re-resolve DNS.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[SearXNG]] — the consumer of the tunnel
|
||||
- [[Dankclaw Stack]]
|
||||
- Knowledge base: [[SearXNG]]
|
||||
@@ -0,0 +1,41 @@
|
||||
# Hermes-Dashboard
|
||||
|
||||
The Hermes web dashboard is a browser UI for managing the agent. It shows config, API keys, and sessions. It is a management tool. It is not a chat window. Chat happens over Signal or the CLI. See [[Hermes]].
|
||||
|
||||
**Host:** Dankclaw — **Port:** `9119` — **URL:** `http://192.168.0.143:9119`
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `docker.io/nousresearch/hermes-agent:latest` (same as the gateway) |
|
||||
| Command | `hermes dashboard --host 0.0.0.0 --port 9119 --no-open --skip-build` |
|
||||
| Port | `9119` (published to the LAN) |
|
||||
| Auth | HTTP basic auth (username + scrypt password hash) |
|
||||
| Data | shares `/home/dank/hermes-data` with the gateway |
|
||||
|
||||
## Why a separate container
|
||||
|
||||
The gateway container runs `hermes gateway run` (the Signal platform). That entrypoint does not start the dashboard. The dashboard is a separate process. So it runs in its own compose service (`hermes-dashboard`). Both services mount the same data volume. Both restart at boot (`restart: unless-stopped`).
|
||||
|
||||
## Authentication
|
||||
|
||||
The dashboard refuses to bind to a non-loopback address without an auth provider. There is no unauthenticated public bind. This repo uses basic auth:
|
||||
|
||||
- `dashboard.basic_auth.username` and `dashboard.basic_auth.password_hash` live in the live `/home/dank/hermes-data/config.yaml`; the tracked backup at `dankclaw/hermes/config.yaml.example` intentionally omits the password hash.
|
||||
- The stored value is a one-way scrypt hash. The plaintext password is never written to a file or the repo.
|
||||
- The hash is generated with `plugins.dashboard_auth.basic.hash_password`.
|
||||
|
||||
## Access
|
||||
|
||||
Open `http://192.168.0.143:9119` in a browser on the LAN. Log in with the configured username and password.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The dashboard and the gateway share one data volume. A config change made in the dashboard takes effect when the gateway restarts.
|
||||
- If the shared password is reused elsewhere, treat it as exposed. Rotate it and regenerate the hash.
|
||||
- `--skip-build` uses the prebuilt web assets in the image. Do not remove it unless the assets are rebuilt.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Hermes-WebUI]] — the phone-facing API server (Hermex)
|
||||
@@ -0,0 +1,114 @@
|
||||
---
|
||||
type: configuration
|
||||
title: Hermes-WebUI (Hermex server)
|
||||
description: The hermes-webui compose service on Dankclaw — the loopback-only API server that the Hermex iPhone app talks to over an SSH tunnel.
|
||||
tags:
|
||||
- hermes
|
||||
- hermes-webui
|
||||
- hermex
|
||||
- dankclaw
|
||||
- ssh-tunnel
|
||||
timestamp: 2026-09-14
|
||||
---
|
||||
|
||||
# Hermes-WebUI (Hermex server)
|
||||
|
||||
`hermes-webui` is the [hermes-webui](https://github.com/nesquena/hermes-webui)
|
||||
server (MIT, third-party) that the **Hermex** iPhone app connects to. It runs
|
||||
the Hermes agent in-process from the shared `HERMES_HOME`, so it is the *same*
|
||||
agent — same memory, skills, and sessions — as the [[Hermes]] gateway and the
|
||||
[[Hermes-Dashboard]].
|
||||
|
||||
**Host:** Dankclaw — **Port:** `127.0.0.1:8787` (loopback only) — **Ingress:** SSH tunnel
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `docker.io/nousresearch/hermes-agent:latest` (same as the gateway) |
|
||||
| Command | `python3 /webui/bootstrap.py --foreground` |
|
||||
| Source checkout | `/home/dank/hermes-webui` on Dankclaw (mounted at `/webui`) |
|
||||
| Port | `8787`, published to `127.0.0.1` only |
|
||||
| Auth | `HERMES_WEBUI_PASSWORD` (host `.env`; Hermex signs in with it) |
|
||||
| Data | shares `/home/dank/hermes-data` with the gateway and dashboard |
|
||||
| Health | `GET /health` (compose healthcheck + deployer) |
|
||||
|
||||
## Why a container, not a host process
|
||||
|
||||
The gateway's state files are owned by the container-mapped uid (`109999`,
|
||||
from rootless Podman's `1 → 100000/65536` subuid range). A fresh rootless
|
||||
namespace on Dankclaw maps identically (verified with `podman unshare
|
||||
cat /proc/self/uid_map`), so a second container using the same image reads
|
||||
and writes the shared volume with exactly the ownership the gateway already
|
||||
uses. A host-side process running as `dank` (uid 1000) could not read the
|
||||
gateway's `0600` state files.
|
||||
|
||||
## How it works
|
||||
|
||||
- `bootstrap.py` creates `/webui/.venv` on first boot, pip-installs the WebUI
|
||||
and agent requirements, then **execs** `server.py` as one foreground process
|
||||
(`--foreground`). systemd-free: Podman supervises it via
|
||||
`restart: unless-stopped`.
|
||||
- `HERMES_WEBUI_SKIP_ONBOARDING=1` keeps the first-run wizard off; a profile
|
||||
already exists in `/data` from the gateway.
|
||||
- The healthcheck `start_period` is 600 s because the very first boot spends
|
||||
minutes pip-installing into a fresh venv.
|
||||
|
||||
## Remote access (Hermex on iOS, from anywhere)
|
||||
|
||||
Only port `22` is forwarded to Dankclaw at the router, so the webui is
|
||||
reached through an SSH local forward — no extra router ports, no third-party
|
||||
relay (no Tailscale, no Cloudflare):
|
||||
|
||||
1. On the iPhone, open **Termius** (or any SSH client with local port
|
||||
forwarding) and connect to `<dankclaw-public>:22` with the usual key.
|
||||
2. On that connection, add a **local forward**: `127.0.0.1:8787` →
|
||||
`127.0.0.1:8787`. The forward runs inside the encrypted SSH session;
|
||||
nothing else on the phone routes through it.
|
||||
3. In Hermex, set the server URL to `http://localhost:8787` and sign in with
|
||||
the webui password.
|
||||
|
||||
The tunnel lives as long as the SSH session on the phone — iOS does not allow
|
||||
a true background daemon, so connect before you use Hermex.
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Hermex app] -->|http://localhost:8787| B[SSH local forward on iPhone]
|
||||
B ==>|encrypted SSH to public IP:22| C[Dankclaw sshd]
|
||||
C --> D["hermes-webui 127.0.0.1:8787"]
|
||||
D --> E["/home/dank/hermes-data (shared agent state)"]
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
```
|
||||
./scripts/deploy.sh dankclaw --service hermes-webui
|
||||
```
|
||||
|
||||
The planner maps changes to the `hermes-webui` compose block (and only that
|
||||
block) to this service, so gateway and inference restarts stay isolated.
|
||||
Before first deploy, set a strong `HERMES_WEBUI_PASSWORD` in the live
|
||||
`dankclaw/.env`.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The `/webui` mount is **rw on purpose**: `bootstrap.py` creates `.venv` in
|
||||
the repo root. Do not flip it to `:ro`.
|
||||
- Never publish `8787` beyond loopback without keeping the password set —
|
||||
the webui can run agent commands.
|
||||
- The webui and the gateway share one data volume. A config change made on one
|
||||
side takes effect when the other restarts (same caveat as the dashboard).
|
||||
- `bootstrap.py` auto-installs a Hermes Agent if it cannot find one; in this
|
||||
image the agent is already present, so that path should never fire. If the
|
||||
log shows an install attempt, the image changed — re-verify the entrypoint.
|
||||
|
||||
# Citations
|
||||
|
||||
- hermes-webui README (env vars, `--foreground`, `/health`): https://github.com/nesquena/hermes-webui
|
||||
- Hermex setup model (server URL + password; Cloudflare/Tailscale are optional, not required): https://hermexapp.com/
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Hermes]] — the agent and its Signal gateway
|
||||
- [[Hermes-Dashboard]] — the LAN browser management UI
|
||||
- [[Dankclaw Stack]]
|
||||
@@ -0,0 +1,52 @@
|
||||
# Hermes
|
||||
|
||||
Hermes is the self-improving AI agent from Nous Research. It runs in a terminal, Telegram, Discord, Slack, WhatsApp, or Signal, and keeps a persistent learning loop: it creates skills from experience, searches past conversations, and builds a model of the user across sessions.
|
||||
|
||||
**Host:** Dankclaw — **Hostname:** `hermes.dankclaw.local` — **Docs:** https://hermes-agent.nousresearch.com/docs
|
||||
|
||||
**Persistent state:** `/home/dank/hermes-data` on Dankclaw, mounted as `/data` in
|
||||
both containers. The tracked, sanitized configuration baseline is
|
||||
[`dankclaw/hermes/config.yaml.example`](../../dankclaw/hermes/config.yaml.example).
|
||||
|
||||
## Facts (from research, primary source)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | built from `NousResearch/hermes-agent` (official Dockerfile in repo) |
|
||||
| License | MIT |
|
||||
| Model backends | Nous Portal, OpenRouter, OpenAI, or any custom endpoint |
|
||||
| Interfaces | TUI, Telegram, Discord, Slack, WhatsApp, Signal |
|
||||
| Memory | FTS5 session search, Honcho user modeling |
|
||||
| Skills | self-created, self-improving during use |
|
||||
|
||||
## Configuration
|
||||
|
||||
- Point Hermes at the local model endpoint or an external provider through its provider config.
|
||||
- The gateway process exposes one port for chat integration; the exact port is set in the Hermes config (`8090` convention in this repo).
|
||||
- The data volume persists skills, memory, and session history.
|
||||
- Hermes has two named local providers: `dankclaw-large` reaches
|
||||
`http://192.168.0.143:7770/v1` and exposes `frank-default-large`;
|
||||
`dankserver-small` reaches `http://192.168.0.235:7770/v1` and exposes
|
||||
`frank-default-small`.
|
||||
- The live `config.yaml` contains secrets and user state and is not committed;
|
||||
the tracked example is the sanitized backup of its non-secret settings.
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path (convention) |
|
||||
|----------------|------------------------|
|
||||
| `/data` | `/home/dank/hermes-data` |
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Hermes is under active development; pin the image tag for reproducibility.
|
||||
- Model choice drives quality; the local `frank-default-large` (Dankclaw, 7770) is the intended backend.
|
||||
- Do not expose the gateway without auth — put it behind [[Caddy]] with basic auth.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Hermes-Dashboard]] — the LAN browser management UI
|
||||
- [[Hermes-WebUI]] — the Hermex server (phone-facing API, SSH-tunneled)
|
||||
- [[Dankclaw Stack]]
|
||||
- [[OpenWebUI]] — the chat UI that pairs with Hermes
|
||||
- Knowledge base: [[SearXNG]] (Hermes as the search agent)
|
||||
@@ -0,0 +1,75 @@
|
||||
# OpenWebUI
|
||||
|
||||
OpenWebUI is a self-hosted chat interface for LLMs. It connects to OpenAI-compatible endpoints (vLLM, llama.cpp, Ollama) and adds RAG, plugins, tool calling, and multi-user support.
|
||||
|
||||
**Host:** Dankclaw — **Port:** 3005 (host) → 8080 (container) — **Boot-persistent:** Quadlet — **Docs:** https://github.com/open-webui/open-webui
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `ghcr.io/open-webui/open-webui:main` (CPU build; Dankclaw is CPU-only) |
|
||||
| Port | 3005 (host) → 8080 (container). Port 3000 is Gitea's HTTP, so it is not usable. |
|
||||
| Data volume | `/home/dank/Containers/open-webui/data` → `/app/backend/data` |
|
||||
| Default backend | CIRU llama.cpp on Dankclaw `:7770` = `frank-default-large` (via host LAN IP `192.168.0.143:7770`) |
|
||||
| Network | Pinned bridge `systemd-open-webui` (`10.89.3.0/24`, gateway `10.89.3.1`) — see [History](#history) |
|
||||
| Persistence | Quadlet unit `open-webui.service`; `[Install] WantedBy=default.target` |
|
||||
| Deploy | `scripts/deploy-open-webui-quadlet.sh dankclaw` |
|
||||
|
||||
## The LLM backends
|
||||
|
||||
The fleet runs two OpenAI-compatible endpoints. This mapping is the **live** state (verified 2026-08-22). Both services listen on `:7770`, distinguished by host. On 2026-08-22 `frank-default-large` moved from `:52625` to `:7770`.
|
||||
|
||||
| Endpoint | Model name | Host | Capability |
|
||||
|----------|-----------|------|-----------|
|
||||
| `:7770` (CIRU llama.cpp) | `frank-default-large` | Dankclaw | completion |
|
||||
| `:7770` (llama.cpp) | `frank-default-small` | Dankserver | completion, multimodal |
|
||||
|
||||
The default connection points at the local vLLM (`frank-default-large`), so chat stays on-host. To add the other backend, open **Admin → Connections** in the UI and point an OpenAI connection at `http://192.168.0.235:7770/v1` (key `none`).
|
||||
|
||||
> Both endpoints are bare-metal systemd services managed by this repository (the former `vllmScripts` repo, merged in 2026-08-22). Model swaps, context sizing, and spec tuning happen here. See the [LLM knowledge base](../LLM/index.md).
|
||||
|
||||
## Boot persistence (Quadlet)
|
||||
|
||||
The unit lives at `dankclaw/open-webui/quadlet/open-webui.container`. On the host it installs to `~/.config/containers/systemd/open-webui.container`. The generator turns it into `open-webui.service`. Because the unit sets `[Install] WantedBy=default.target` and user linger is on, it starts at boot. `systemctl --user is-enabled open-webui.service` reports `generated` (the quadlet boot-start state).
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path |
|
||||
|----------------|-----------|
|
||||
| `/app/backend/data` | `/home/dank/Containers/open-webui/data` |
|
||||
|
||||
The data dir holds the database, RAG indexes, and uploads. The deploy script creates it if missing and preserves it across reruns.
|
||||
|
||||
## Healthcheck
|
||||
|
||||
The quadlet carries a built-in healthcheck (30s interval, 3 retries, 300s start period). It checks two things from inside the container:
|
||||
|
||||
1. **App liveness** — `GET http://127.0.0.1:8080/health` returns 200 (`{"status":true}`).
|
||||
2. **Network liveness** — a non-loopback interface exists in `/sys/class/net`. The dead-tap zombie state (History) leaves only `lo` behind, and this is the check that catches it.
|
||||
|
||||
`podman ps` shows the container as `unhealthy` if either fails. Probe manually at `http://dankclaw:3005/health`.
|
||||
|
||||
## Retired instances
|
||||
|
||||
- The compose `open-webui` service (port 3000) was never run: 3000 is Gitea's HTTP port, so it could not bind.
|
||||
- The bare-container `open-webui` on Dankserver (port 3000) pointed at a dead backend (`:9090`) and was removed. Its data dir is preserved on Dankserver in case it is needed.
|
||||
- The bare-container `searxng` on Dankserver (port 8081) was removed. SearXNG now runs only on Dankclaw behind Caddy.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Use the non-CUDA image on this CPU-only host.
|
||||
- The default admin account is created on first login; secure it immediately.
|
||||
- If the backend is loading (3–8 minute model load), OpenWebUI shows connection errors until the model is ready.
|
||||
- The LLM is reached via the host LAN IP `192.168.0.143:7770`, not the bridge gateway: the gateway (`10.89.3.1`) lives in the rootless netns, while llama-server runs in the host netns, so the gateway refuses. `host.containers.internal` is slirp4netns-only and does not resolve on a bridge either. To reach the Dankserver backend, use the LAN address `192.168.0.235:7770`.
|
||||
|
||||
## History
|
||||
|
||||
- **2026-08-22 — slirp4netns → pinned bridge.** The unit ran on `Network=slirp4netns`. The slirp4netns helper process died (no OOM, no kill entry, no crash log found), and the container kept running with only `lo` in its netns: published port 3005 accepted TCP and reset every request, so chat served empty responses for ~2 days. Nothing logged an error. Moving to a kernel bridge removes the per-container helper entirely, and the healthcheck's interface check makes the zombie state visible if it ever returns.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
- [[Hermes]] — the agent OpenWebUI can talk to
|
||||
- [[SearXNG]] — the search engine on this host
|
||||
- [LLM knowledge base](../LLM/index.md) — the vLLM and llama.cpp endpoints
|
||||
@@ -0,0 +1,70 @@
|
||||
# Postgres
|
||||
|
||||
PostgreSQL is the database backend for Gitea. It runs on the same podman network as Gitea and holds the repository metadata, users, and issue data.
|
||||
|
||||
**Host:** Dankclaw — **Docs:** https://www.postgresql.org/docs
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `docker.io/library/postgres:16-alpine` |
|
||||
| Port | 5432 (internal only) |
|
||||
| Data volume | bind mount at `/var/lib/postgresql/data` |
|
||||
|
||||
## Configuration
|
||||
|
||||
```yaml
|
||||
services:
|
||||
postgres:
|
||||
image: docker.io/library/postgres:16-alpine
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
POSTGRES_USER: gitea
|
||||
POSTGRES_PASSWORD: "${GITEA_DB_PASSWORD}"
|
||||
POSTGRES_DB: gitea
|
||||
volumes:
|
||||
- /opt/gitea/data/gitea-postgres:/var/lib/postgresql/data
|
||||
```
|
||||
|
||||
## Gitea Connection
|
||||
|
||||
Gitea reaches the database over the `systemd-gitea` network (quadlet-managed live, compose-managed after the full-stack deploy):
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
DATABASE_TYPE: postgres
|
||||
DATABASE_HOST: gitea-postgres
|
||||
DATABASE_NAME: gitea
|
||||
DATABASE_USER: gitea
|
||||
DATABASE_PASSWD: "${GITEA_DB_PASSWORD}"
|
||||
```
|
||||
|
||||
The live `app.ini` on the host overrides env vars. When you rename the database container, update its `[database] HOST` line. See [[Gitea]] Gotchas.
|
||||
|
||||
## Migration
|
||||
|
||||
The live Gitea stack on Dankclaw already runs this image with the data at `/opt/gitea/data/gitea-postgres`. The migration keeps the bind mount in place; see [[Gitea]].
|
||||
|
||||
## Healthcheck
|
||||
|
||||
Use `pg_isready`:
|
||||
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U gitea"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The database password must match between the `postgres` service and Gitea's `DATABASE_PASSWD`; both come from the same `.env` variable.
|
||||
- Do not change the Gitea `SECRET_KEY` or `INTERNAL_TOKEN` after migration — encrypted data becomes unreadable.
|
||||
- Back up with `pg_dump`; see [[Gitea]] (knowledge base).
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Gitea]]
|
||||
- [[Dankclaw Stack]]
|
||||
@@ -0,0 +1,400 @@
|
||||
# SearXNG Providers
|
||||
|
||||
How Google and other upstream search engines get blocked behind the Gluetun
|
||||
VPN tunnel, and how to unblock them in SearXNG.
|
||||
|
||||
**Host:** Dankclaw — **Docs:** https://docs.searxng.org
|
||||
|
||||
SearXNG aggregates ~50 upstream engines (Google, Bing, DuckDuckGo, Brave,
|
||||
Wikipedia, …). When all of them are queried through **one** VPN exit IP — as on
|
||||
Dankclaw, where SearXNG reuses Gluetun's tunnel — the rate-sensitive engines
|
||||
start answering with reCAPTCHA or HTTP 429. SearXNG then *suspends* the offending
|
||||
engine for an hour to 15 days and silently drops it from results. That is the
|
||||
"Google (and sometimes a few others) went unavailable" symptom.
|
||||
|
||||
This page is the fixbook. Read [[SearXNG]] for the stack; the sub-pages below
|
||||
are the detail if you need it.
|
||||
|
||||
## Provider policy
|
||||
|
||||
The default aggregation should maximize **independent, anonymous coverage**
|
||||
without turning one user query into a request storm. Disabled engines remain
|
||||
installed and selectable on demand; `inactive` is reserved for engines that
|
||||
should disappear completely.
|
||||
|
||||
| Category | Default baseline | On-demand / deferred |
|
||||
|----------|------------------|----------------------|
|
||||
| General/web | Bing, DuckDuckGo, Wikipedia | Google CSE, Brave, Startpage, Qwant, Mojeek, Wiby, MWMBL after retesting |
|
||||
| Technical | GitHub, MDN, Stack Overflow, Ask Ubuntu, Superuser, Docker Hub, PyPI | Other technical indexes as needed |
|
||||
| Science | arXiv, PubMed, Semantic Scholar, OpenAlex-compatible sources | Google Scholar until independently tested |
|
||||
| News | Reuters, Bing News, DuckDuckGo News, Wikinews | Google News, Brave News, Startpage News |
|
||||
| Images | Openverse, Wikimedia Commons, Bing Images, Unsplash | Google/Brave/Startpage image engines |
|
||||
| Videos | YouTube, Vimeo, Dailymotion, Bing Videos | Brave video engines |
|
||||
|
||||
This policy uses no API credentials and does not identify the operator to a
|
||||
search provider. It is deliberately conservative: a bundled engine is not
|
||||
necessarily healthy, anonymous, or suitable for every default query.
|
||||
|
||||
## Root cause (one line)
|
||||
|
||||
All SearXNG engine traffic shares **one ProtonVPN exit IP** with no rotation, so
|
||||
Google/DuckDuckGo see one datacenter IP spamming ~50 paginated engines and
|
||||
rate-limit it → CAPTCHA → engine **suspension** → empty results.
|
||||
|
||||
## The suspension model (what "unavailable" actually is)
|
||||
|
||||
An engine that returns 429 / reCAPTCHA / access-denied raises a named
|
||||
exception and is *suspended* (removed from result aggregation for a set time).
|
||||
Suspended engines are the usual cause of "gone":
|
||||
|
||||
| Exception | Default suspension | Trigger |
|
||||
|-----------|--------------------|---------|
|
||||
| `SearxEngineTooManyRequests` | ~1 hour | HTTP 429 |
|
||||
| `SearxEngineAccessDenied` | ~1 day | 403 / access denied |
|
||||
| `SearxEngineCaptcha` | ~1 day | Google `/sorry/` or reCAPTCHA |
|
||||
| `recaptcha_SearxEngineCaptcha` | ~7 days | Google reCAPTCHA |
|
||||
| `cf_SearxEngineCaptcha` | ~15 days | Cloudflare edge CAPTCHA |
|
||||
| ordinary errors (timeout/parse) | 5–120 s | `ban_time_on_fail`/`max_ban_time_on_fail` |
|
||||
|
||||
The values are `search.suspended_times` (+ `ban_time_on_fail`) in
|
||||
`settings.yml`. An engine that returns **0 results without an error** is *not*
|
||||
suspended — that is the separate "selector drift" problem (below).
|
||||
|
||||
## Two failure modes
|
||||
|
||||
| Mode | Symptom | Root cause | Fix family |
|
||||
|------|---------|-----------|-----------|
|
||||
| **CAPTCHA / 429 / access-denied** | engine flips `unresponsive`, then `suspended` for hours | blocked from the shared VPN exit IP | routing/proxies + request tuning (see below) |
|
||||
| **Selector drift** | engine returns 0 results, no error banner | Google renamed its minified CSS classes the engine's hardcoded XPath looks for | engine config (duplicate engine / JSON feed) |
|
||||
|
||||
Most "Google is broken on a VPN host" cases are a **mix of both**.
|
||||
|
||||
## Fix A — Engine config (settings.yml `engines:`)
|
||||
|
||||
The current tracked policy enables Bing and DuckDuckGo for default general
|
||||
search, gives each weight 2, disables pagination, and allows 5 seconds for the
|
||||
VPN path. CAPTCHA/429-prone Google, Brave, and Startpage families are retained
|
||||
but disabled from default aggregation. This limits fan-out while preserving
|
||||
explicit/on-demand selection.
|
||||
|
||||
Important: SearXNG's engines are configured in the **`engines:` list inside
|
||||
`settings.yml`**, not a separate `engines.yml` file. With
|
||||
`use_default_settings: true` (our setting) your `engines:` list is *merged* over
|
||||
the built-in engines by `name` — you do **not** set `use_default_settings: false`.
|
||||
|
||||
To add, override, disable, or prune:
|
||||
|
||||
```yaml
|
||||
# shared/searxng/settings.yml (appended to the existing file)
|
||||
use_default_settings: true
|
||||
engines:
|
||||
# prune a broken engine entirely
|
||||
remove:
|
||||
- # (example) wikipedia # drop an engine that never recovers
|
||||
|
||||
# override + duplicate the Google engines so one survives the other's suspension
|
||||
- name: google
|
||||
weight: 3
|
||||
disabled: false
|
||||
timeout: 5.0 # wait longer over the tunnel (global default is 2.0s)
|
||||
retry_on_http_error: [403, 429]
|
||||
display_error_messages: true
|
||||
|
||||
- name: google2 # fresh engine, same module, survives google's CAPTCHA ban
|
||||
engine: google
|
||||
categories: general
|
||||
weight: 2
|
||||
timeout: 5.0
|
||||
disabled: false
|
||||
```
|
||||
|
||||
Key per-engine keys (each overwrites the engine-module default; see
|
||||
`settings_engines.html`):
|
||||
|
||||
| Key | Effect |
|
||||
|-----|--------|
|
||||
| `name` | merge key (lowercase, no underscore) |
|
||||
| `engine` | Python module (`google`, `json_engine`, `xpath`) |
|
||||
| `categories` | categories it serves (also a comma string) |
|
||||
| `timeout` | per-engine HTTP timeout, overrides `outgoing.request_timeout` |
|
||||
| `weight` | ranks results higher |
|
||||
| `disabled` | off by default, selectable in /preferences |
|
||||
| `inactive` | `disabled` **and** removed from the UI — use to fully drop a broken engine |
|
||||
| `retry_on_http_error` | retry on status, e.g. `true`, `403`, `[403, 429]` |
|
||||
| `display_error_messages` | show/hide the engine's error banner |
|
||||
| `proxies` | per-engine egress override (see Fix C) |
|
||||
| `language` | force region, e.g. `en` |
|
||||
|
||||
**Best duplicate trick:** re-declare `google` under a new `name` in a
|
||||
*different* category. When the primary `google` gets CAPTCHA'd it suspends, but
|
||||
`google2` still returns results, and vice-versa.
|
||||
|
||||
**Parser caveat (read before fiddling with `parsers`):** the modern `google`,
|
||||
`duckduckgo`, and `brave` engines ship a **custom `response()`** with hardcoded
|
||||
XPath and *ignore* the generic `parsers`/`p_parsers` table. So `parsers:
|
||||
[scrapy_html]` under `name: google` mostly does nothing. To make Google
|
||||
robust against selector drift, either update the upstream `google.py`
|
||||
selectors (upgrade SearXNG) or run a **`json_engine`** against a JSON feed.
|
||||
Verify `parsers` consumption on the pinned version before depending on it.
|
||||
|
||||
**JSON-feed fallback:** decouple parsing from Google's HTML churn.
|
||||
|
||||
```yaml
|
||||
engines:
|
||||
- name: google-json
|
||||
engine: json_engine
|
||||
categories: general
|
||||
paging: true
|
||||
search_url: 'https://www.googleapis.com/customsearch/v1?key={api_key}&q={query}&cx={cx}'
|
||||
results_query: results
|
||||
title_query: title
|
||||
url_query: link
|
||||
content_query: snippet
|
||||
```
|
||||
|
||||
## Fix B — Request tuning (settings.yml `outgoing:`)
|
||||
|
||||
SearXNG already rotates a **random browser User-Agent per outgoing request**
|
||||
(`gen_useragent()` in `online.py`) — there is **no native `useragents:` mapping
|
||||
key**. The blogs that point at a `useragents:` table describe the old
|
||||
`asciimoo/searx`; current SearXNG rotates automatically. Keep relying on it.
|
||||
|
||||
Tune the `outgoing` block and the suspend model to survive load:
|
||||
|
||||
```yaml
|
||||
outgoing:
|
||||
request_timeout: 4.0 # current Dankclaw value; modest VPN-latency allowance
|
||||
max_request_timeout: 12.0
|
||||
pool_connections: 100
|
||||
pool_maxsize: 20
|
||||
enable_http2: true
|
||||
retries: 1 # avoid amplifying 429/CAPTCHA responses from one exit IP
|
||||
useragent_suffix: "ops@yourhomelab.example" # so a per-engine blocker can reach you
|
||||
|
||||
search:
|
||||
ban_time_on_fail: 10
|
||||
max_ban_time_on_fail: 180
|
||||
suspended_times:
|
||||
SearxEngineTooManyRequests: 300 # 429 → 5 min before it retries (tune down to recover faster)
|
||||
SearxEngineCaptcha: 7200 # CAPTCHA → 2h before it retries
|
||||
```
|
||||
|
||||
Notes:
|
||||
- **Auto-suspend is the real "rate limit".** The old per-engine `rate_limit` /
|
||||
`time_between_requests` sleep field does **not** exist in current SearXNG —
|
||||
that is a historical-searx concept. The functional equivalent is the suspend
|
||||
schedule above (widen to back off before blocking, narrow to recover faster).
|
||||
- `request_timeout` interacts with the suspend model: too tight → engines look
|
||||
"unresponsive"; too loose → slow pages. Dankclaw uses 4 seconds globally and
|
||||
12 seconds as the cap; raise it only when measurements show real upstream
|
||||
latency rather than 429/CAPTCHA responses.
|
||||
- Keep retries low: retrying a 429/CAPTCHA through the same VPN exit can make the
|
||||
provider's rate limit worse. A retry count of 1 is a bounded compromise for a
|
||||
transient network failure.
|
||||
- `safe_search: 0` when diagnosing — a too-strict global safe search can make an
|
||||
engine return zero rows and *look* blocked.
|
||||
|
||||
## Fix C — Routing & proxies (change which IP the upstreams exit as)
|
||||
|
||||
Two independent levers: SearXNG's own upstream `proxies`, and the Gluetun VPN
|
||||
strategy. The decisive fact for our topology: a proxy that runs **inside the
|
||||
same Gluetun netns** exits from the *same* single IP, so round-robin among
|
||||
same-IP proxies **does not** unblock. The proxy must egress from a **different,
|
||||
clean IP**.
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
SE[SearXNG] -->|default| GW[Gluetun #1 netns] -->|exit| IP1[ProtonVPN IP A — gets blocked]
|
||||
SE -->|per-engine proxies| GW2[Gluetun #2 / dedicated IP] -->|exit| IP2[ProtonVPN IP B — clean]
|
||||
```
|
||||
|
||||
**Option 1 — per-engine `proxies` (cheapest, settings.yml only).** Point the
|
||||
heaviest/blocked engines (Google and Google-hosted) at a proxy that exits from a
|
||||
clean IP:
|
||||
|
||||
```yaml
|
||||
engines:
|
||||
- name: google
|
||||
proxies:
|
||||
http:
|
||||
- http://gluetun-upstream:8080
|
||||
https:
|
||||
- http://gluetun-upstream:8080
|
||||
- socks5h://gluetun-upstream:8388
|
||||
- name: google news
|
||||
engine: google_news
|
||||
proxies:
|
||||
https:
|
||||
- http://gluetun-upstream:8080
|
||||
```
|
||||
|
||||
`using_tor_proxy` follows the same per-engine/global pattern.
|
||||
|
||||
**Option 2 — dedicated second Gluetun for SearXNG's upstreams.** Run a second
|
||||
Gluetun on a different ProtonVPN server/geo, publish its built-in HTTP proxy
|
||||
(`8080`, `HTTPPROXY=on`), and point the Google engines at it. SearXNG inbound
|
||||
(still on the main host/Caddy) stays put; only *upstream* requests re-route.
|
||||
Gives a second isolated exit IP and reputation pool.
|
||||
- **Cons:** a second ProtenVPN device slot, a new container, port publishing
|
||||
moves to Gluetun, and **split-brain DNS** (below).
|
||||
|
||||
**Option 3 — dedicated ProtonVPN IP.** A static, exclusive exit IP that other
|
||||
subscribers can't pollute. Requires a **ProtonVPN Business plan** as of the
|
||||
sources reviewed (the long-requested *personal*-plan dedicated IP is still only
|
||||
on uservoice — flag: verify your plan). Wire it in Gluetun by pinning the node
|
||||
via `SERVER_HOSTNAMES`.
|
||||
|
||||
**Option 4 — rotate the exit.** List `SERVER_COUNTRIES`/`SERVER_CITIES`/
|
||||
`SERVER_HOSTNAMES` so Gluetun picks from several physical exit servers. Caveat:
|
||||
rotating the **shared** Gluetun rotates the exit for *all* attached
|
||||
containers. To rotate only SearXNG's upstreams, use Option 1 → a dedicated
|
||||
gateway.
|
||||
|
||||
> ⚠️ **Split-brain DNS:** a container sharing Gluetun's netns
|
||||
> (`Network=gluetun.container`) loses Docker's internal service-name DNS — it
|
||||
> can't resolve siblings by name. Use a static IP or `extra_hosts`/`~/.config/
|
||||
> containers/systemd/<unit>.container` `Environment` to inject names into the
|
||||
> tunnel's `/etc/hosts`.
|
||||
|
||||
## Observed Dankclaw baseline (2026-08-25)
|
||||
|
||||
The read-only audit found:
|
||||
|
||||
- SearXNG image: `ghcr.io/searxng/searxng:latest`; embedded version `1.0.0`.
|
||||
- Gluetun uses `VPN_SERVICE_PROVIDER=custom` with a static WireGuard profile;
|
||||
no `SERVER_COUNTRIES`, `SERVER_CITIES`, `SERVER_REGIONS`, or
|
||||
`SERVER_HOSTNAMES` filters are configured.
|
||||
- The observed exit was `149.22.88.19` (Seattle, Washington, US).
|
||||
- Bing and DuckDuckGo returned results. Mojeek returned no results without an
|
||||
upstream error.
|
||||
- `google cse` and Brave were suspended for too many requests; Startpage was
|
||||
suspended for CAPTCHA; Qwant returned CAPTCHA. SearXNG logs also recorded
|
||||
Google's "unusual traffic from your network" response.
|
||||
- `/healthz` returned `200 OK`; the SearXNG service and Gluetun service were
|
||||
healthy. This isolates the primary failures to upstream provider handling,
|
||||
not application availability.
|
||||
|
||||
This deployment returns the legacy array form for `unresponsive_engines`:
|
||||
|
||||
```json
|
||||
"unresponsive_engines": [
|
||||
["brave", "Suspended: too many requests"],
|
||||
["startpage", "Suspended: CAPTCHA"]
|
||||
]
|
||||
```
|
||||
|
||||
Do not assume the newer object form (`engine`, `error_type`, `suspended`)
|
||||
until the live response has been checked. The direct published port is the
|
||||
useful diagnostic path because it exercises the ProtonVPN exit. The Caddy
|
||||
hostname resolved to the WAN address during the audit and timed out from the
|
||||
audit host, so it was not used as the baseline path.
|
||||
|
||||
## Fix D — Verify an engine is unblocked
|
||||
|
||||
Three layers, cheapest first. All queries should hit the path production uses
|
||||
(the **published Gluetun port → ProtonVPN exit**, not a direct host port).
|
||||
|
||||
**1. Isolate one engine with `/json`.** Enable `json` in `search.formats`
|
||||
(otherwise `format=json` → 403), then query a single engine:
|
||||
|
||||
```bash
|
||||
BASE="http://192.168.0.143:8081" # published port (ProtonVPN exit)
|
||||
curl -s "$BASE/search?q=test&engines=google&format=json" \
|
||||
| jq -r '.unresponsive_engines[]? | select(.engine=="google")'
|
||||
```
|
||||
|
||||
Inspect:
|
||||
- `content[].engine` — which engine produced each result.
|
||||
- `unresponsive_engines[]` — inspect the live shape first. On Dankclaw it is
|
||||
`[engine, message]`, for example `["google cse", "Suspended: too many requests"]`.
|
||||
Newer builds may return objects containing `engine`, `error_type`, and
|
||||
`suspended`.
|
||||
- `timings[].engine` — per-engine latency when the running build emits timings.
|
||||
|
||||
Full status check (bash + jq):
|
||||
|
||||
```bash
|
||||
BASE="http://192.168.0.143:8081"
|
||||
for e in google bing duckduckgo brave startpage; do
|
||||
curl -s "$BASE/search?q=test&engines=$e&format=json" \
|
||||
| jq -r --arg e "$e" '
|
||||
(.unresponsive_engines // [])
|
||||
| map(select(.engine==$e))
|
||||
| if length==0 then "\($e): OK"
|
||||
else .[] | "\($e): UNRESPONSIVE (\(.error_type), suspended=\(.suspended))" end'
|
||||
done
|
||||
```
|
||||
|
||||
**2. `/config` REST endpoint** lists every engine + enabled/disabled +
|
||||
categories — an inventory of what is live. `?engines=<name>` isolates a single
|
||||
provider; `/debug` adds engine diagnostics.
|
||||
|
||||
**3. `check.searx.space`** — the hosted official checker probes every engine of
|
||||
your instance with a **headless Firefox** (catches Cloudflare/CAPTCHA blocks a
|
||||
plain HTTP probe misses) and posts results on searx.space. Add your instance via
|
||||
a GitHub issue in `searxng/searx-instances`, then allow `check.searx.space` IPs
|
||||
to reach the SearXNG port.
|
||||
|
||||
Local self-test CLI (`kvch/searx-checker`):
|
||||
|
||||
```bash
|
||||
python3 checker/checker.py http://192.168.0.143:8081
|
||||
# google.OK bing.OK gitlab......ERROR
|
||||
```
|
||||
|
||||
Verify through the **published Gluetun port** (realistic ProtonVPN-exit block
|
||||
signal), not a direct host port (which uses the host LAN IP and can be
|
||||
misleading).
|
||||
|
||||
## Recommended path for Dankclaw (ranked)
|
||||
|
||||
1. **Measure individual engines** — use `/json` + `?engines=<name>` and several
|
||||
query classes before enabling a bundled engine in default aggregation.
|
||||
2. **Keep default fan-out small** — prefer independent anonymous providers and
|
||||
disable pagination for rate-sensitive engines.
|
||||
3. **Promote by evidence** — raise `weight` only for engines that return useful
|
||||
results consistently; weight is not a health check.
|
||||
4. **Avoid credentials** — do not add API-backed engines unless the provider is
|
||||
genuinely free, anonymous, and compatible with our privacy requirements.
|
||||
5. **Verify first** — `/json` + `?engines=google` and the searx-checker to see
|
||||
exactly which engines are down and why. Confirm the running SearXNG version
|
||||
(check whether `parsers`/`rate_limit` exist in the pinned build).
|
||||
6. **Use conservative request tuning** — Dankclaw now allows 5 seconds per
|
||||
upstream request, caps it at 15 seconds, and retries once. This addresses
|
||||
modest VPN latency without multiplying 429/CAPTCHA traffic. Adjust only
|
||||
after measuring the resulting error and latency profile.
|
||||
7. **If Google still re-blocks** — Fix C Option 1: point the Google engines at
|
||||
a dedicated-IP proxy (Option 2 dedicated second Gluetun if reputation
|
||||
isolation is needed). Avoid the same-netns-proxy trap.
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
- [SearXNG settings: engines](https://docs.searxng.org/admin/settings/settings_engines.html) — per-engine keys.
|
||||
- [SearXNG settings: outgoing](https://docs.searxng.org/admin/settings/settings_outgoing.html) — `request_timeout`, pool, `retries`, `proxies`, `source_ips`.
|
||||
- [SearXNG settings: search](https://docs.searxng.org/admin/settings/settings_search.html) — `suspended_times`, `ban_time_on_fail`.
|
||||
- [SearXNG settings (merge + remove/keep_only)](https://docs.searxng.org/admin/settings/settings.html) — `use_default_settings` semantics.
|
||||
- [Google engine docs](https://docs.searxng.org/dev/engines/online/google.html) — `detect_google_sorry`, WML/Nokia XML, `max_page`.
|
||||
- [json_engine](https://docs.searxng.org/dev/engines/json_engine.html) — `json` parser config.
|
||||
- [SearXNG exceptions](https://docs.searxng.org/src/searx.exceptions.html) — exception types and suspend settings.
|
||||
- [Google engine stops yielding results #5286](https://github.com/searxng/searxng/blob/master/searx/engines/google.py) — reproducible "Google unavailable".
|
||||
- [DuckDuckGo CAPTCHA #4824](https://github.com/searxng/searxng/issues/4824) — per-engine CAPTCHA signature.
|
||||
- [Google engine actively blocking IPs #2515](https://github.com/searxng/searxng/issues/2515)
|
||||
- [answer-captcha from server's IP](https://docs.searxng.org/admin/answer-captcha.html) — block is tied to the egress IP.
|
||||
- [SearXNG rate limits / 429 (SSD Nodes)](https://www.ssdnodes.com/learn/searxng-rate-limits-and-429-errors) — suspension table, `/stats/errors`.
|
||||
- [How search engines detect scrapers (Crawlbase)](https://crawlbase.com/blog/how-search-engines-detect-scrapers/) — IP reputation + rate + CAPTCHA mechanics.
|
||||
- [check.searx.space](https://github.com/searxng/searx-space) + [kvch/searx-checker](https://github.com/kvch/searx-checker) — the checkers.
|
||||
- [searx-instances — how to be checked](https://github.com/searxng/searx-instances) — add your instance + allow checker IPs.
|
||||
- [ProtonVPN — Dedicated IP addresses](https://proton.me/support/dedicated-ips) + [Benefits](https://proton.me/support/dedicated-ips) — IP-reputation reasoning; **Business plan** (flag: personal dedicated IP still on uservoice).
|
||||
- [Gluetun wiki — ProtonVPN provider](https://github.com/qdm12/gluetun-wiki/blob/main/setup/providers/protonvpn.md) — `SERVER_HOSTNAMES` narrowest-filter risk.
|
||||
- [Gluetun wiki — Built-in Proxies](https://deepwiki.com/qdm12/gluetun-wiki/4.5-built-in-proxies) — HTTP `8080` + Shadowsocks `8388`.
|
||||
- [Gluetun wiki — Container networking](https://deepwiki.com/qdm12/gluetun-wiki/6.5-container-networking) — netns sharing, port publishing on Gluetun.
|
||||
- [Gluetun inter-network DNS failures (Jacob Jangles)](https://jacobjangles.com/posts/fixing-gluetuns-inter-network-dns-failures-in-docker/) — split-brain DNS + fix.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[SearXNG]] — the stack, routing, limiter, health
|
||||
- [[Gluetun]] — the VPN tunnel (where the exit IP comes from)
|
||||
- [[Dankclaw Stack]] — the service the SearXNG stack sits in
|
||||
- [[Service Research Brief]] — the research that produced these pages
|
||||
@@ -0,0 +1,151 @@
|
||||
# SearXNG
|
||||
|
||||
SearXNG is a privacy-respecting metasearch engine. It aggregates results from many upstream search engines into one result set and serves both an HTML UI and a JSON API for agents.
|
||||
|
||||
**Host:** Dankclaw — **Hostname:** `search.franksdank.cloud` (Caddy) — **Docs:** https://docs.searxng.org
|
||||
|
||||
## Facts (from research, primary source)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `ghcr.io/searxng/searxng:latest` (or DockerHub mirror) |
|
||||
| Port | 8081 (host, on Gluetun) → 8080 (container) |
|
||||
| Config | `/etc/searxng/settings.yml` (bind mount) |
|
||||
| Cache | Valkey (replaces Redis) — `valkey/valkey:9-alpine` |
|
||||
| Formats | `html`, `json` (agent API) |
|
||||
|
||||
## VPN Routing (critical)
|
||||
|
||||
All SearXNG outbound traffic exits through the Gluetun VPN container. The searxng unit sets `Network=gluetun.container`, which makes it reuse gluetun's network namespace:
|
||||
|
||||
```ini
|
||||
[Container]
|
||||
ContainerName=searxng
|
||||
Network=gluetun.container
|
||||
```
|
||||
|
||||
Port publishing happens on the **gluetun** unit, not on searxng. See [[Gluetun]] (KB Pattern 2 is the one used here).
|
||||
|
||||
## Boot Persistence (Quadlet)
|
||||
|
||||
The trio runs as quadlet units, not podman-compose, so it starts at boot. The unit files live in `searxng/quadlet/` and install to `~/.config/containers/systemd/`.
|
||||
|
||||
| Unit file | Service | Role |
|
||||
|-----------|---------|------|
|
||||
| `searxng.network` | `searxng-network.service` | DNS-enabled bridge, subnet pinned to 10.89.2.0/24 |
|
||||
| `gluetun.container` | `gluetun.service` | WireGuard tunnel; publishes 8081:8080 |
|
||||
| `searxng-valkey.container` | `searxng-valkey.service` | Valkey cache, alias `valkey` |
|
||||
| `searxng.container` | `searxng.service` | Reuses gluetun's netns; publishes no port |
|
||||
|
||||
gluetun and valkey join the searxng network. searxng reuses gluetun's netns and reaches valkey by name on that network. Each unit sets `WantedBy=default.target`. User linger is on, so the units start at boot.
|
||||
|
||||
Deploy with `scripts/deploy-searxng-quadlet.sh dankclaw`. It copies the units, daemon-reloads, enables them, and starts them in dependency order.
|
||||
|
||||
## settings.yml Keys
|
||||
|
||||
```yaml
|
||||
use_default_settings: true
|
||||
server:
|
||||
secret_key: "${SEARXNG_SECRET}" # from .env
|
||||
base_url: "${SEARXNG_BASE_URL}" # resolves to http://192.168.0.143:8081/
|
||||
limiter: true
|
||||
method: "POST"
|
||||
|
||||
search:
|
||||
default_lang: "en"
|
||||
formats:
|
||||
- html
|
||||
- json
|
||||
|
||||
valkey:
|
||||
url: "valkey://searxng-valkey:6379/0"
|
||||
|
||||
outgoing:
|
||||
request_timeout: 5.0
|
||||
max_request_timeout: 15.0
|
||||
retries: 1 # avoid amplifying upstream 429/CAPTCHA responses
|
||||
|
||||
# Anonymous, bounded default aggregation. Disabled engines remain configured
|
||||
# for explicit/on-demand selection; see SearXNG Providers for the policy.
|
||||
engines:
|
||||
- name: bing
|
||||
disabled: false
|
||||
weight: 2
|
||||
paging: false
|
||||
timeout: 5.0
|
||||
- name: duckduckgo
|
||||
disabled: false
|
||||
weight: 2
|
||||
paging: false
|
||||
timeout: 5.0
|
||||
- name: google cse
|
||||
disabled: true
|
||||
- name: brave
|
||||
disabled: true
|
||||
- name: startpage
|
||||
disabled: true
|
||||
```
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path (convention) |
|
||||
|----------------|------------------------|
|
||||
| `/etc/searxng` | `/opt/searxng/config` |
|
||||
| `/etc/searxng/valkey` | (valkey container data) |
|
||||
|
||||
## Healthcheck
|
||||
|
||||
SearXNG responds on `/healthz` with 200 when ready.
|
||||
|
||||
## Trusted Group (limiter.toml)
|
||||
|
||||
The trusted group lives in `shared/searxng/limiter.toml`, not in `settings.yml`. The `settings.yml` file only turns the limiter on. The file deploys to `/opt/searxng/config/limiter.toml`, mounted at `/etc/searxng/limiter.toml`.
|
||||
|
||||
Two lists control access:
|
||||
|
||||
- `pass_ip` — an IP in this list gets unrestricted access and bypasses all bot detection.
|
||||
- `trusted_proxies` — if the request IP is in this list, SearXNG reads the real client IP from `X-Forwarded-For` (set by Caddy).
|
||||
|
||||
The `pass_ip` list holds loopback (`127.0.0.1/32`, `::1`), the whole homelab LAN (`192.168.0.0/24`), and the host's private podman/slirp ranges (`10.88.0.0/16`, `10.89.0.0/16`). An IP in `pass_ip` skips every bot-detection check, including the rate limiter. `deploy.sh` copies the file into place on each deploy.
|
||||
|
||||
Why so many ranges: rootless podman strips the client IP on published-port hits. A request to the published port reaches the container with a slirp/bridge source (`10.88.x`/`10.89.x`), never the real client IP. Only Caddy's `X-Forwarded-For` restores it. So a local subagent that hits the port directly must present a `pass_ip` source, or it is treated as an untrusted client. All these ranges are reachable only from the host or the LAN, never the internet.
|
||||
|
||||
The rate limiter's JSON cap is `API_MAX = 4` requests per IP per hour. The constant is hardcoded in Python (`searx/botdetection/ip_limit.py`). It cannot be raised from `limiter.toml`. `pass_ip` is the only bypass. This is what lets 4 parallel subagents poll the JSON API at full rate without a 429.
|
||||
|
||||
Reach SearXNG through Caddy (`search.dankclaw.local`) for browser use. A local subagent can hit the published port directly because its slirp source is now trusted. The searxng subnet 10.89.2.0/24 is in `trusted_proxies`, so Caddy's `X-Forwarded-For` is honored. The subnet is pinned in `searxng.network`, so it stays stable across reboots.
|
||||
|
||||
## Unblocking Google & Search Providers
|
||||
|
||||
The current request policy allows a modest amount of VPN latency (`4.0s` per
|
||||
request, `12.0s` maximum) while limiting retries to one. This can recover slow
|
||||
providers, but it cannot fix an upstream CAPTCHA or 429 caused by the shared
|
||||
VPN exit IP. SearXNG queries all ~50 upstream engines from one Gluetun VPN exit
|
||||
IP. The rate-sensitive engines (Google, DuckDuckGo) then answer with reCAPTCHA / 429.
|
||||
The default provider policy therefore favors anonymous Bing and DuckDuckGo,
|
||||
disables observed CAPTCHA/429 providers from normal aggregation, and disables
|
||||
pagination to reduce upstream request volume. SearXNG **suspends** offending
|
||||
engines for hours to days, which is what shows up as "Google went unavailable".
|
||||
This is upstream, not local.
|
||||
|
||||
Fixes (cheapest first): tune the `outgoing` block (`request_timeout`,
|
||||
`retry_on_http_error`, `suspended_times`), duplicate the `google` engine under a
|
||||
fresh `name` so a CAPTCHA ban on one leaves the other returning results, and —
|
||||
if Google still re-blocks — route the Google engines at a clean-IP proxy via
|
||||
per-engine `proxies` in `settings.yml`. See [[SearXNG Providers]] for the full
|
||||
root cause, the suspend-model table, the `engines:` config, and the
|
||||
`/json` verification recipe.
|
||||
|
||||
### Gotchas
|
||||
|
||||
- The JSON API rate cap (4 per 3600s per IP) is hardcoded in Python, not settable in `limiter.toml`. The only bypass is `pass_ip`. Local subagents are covered because their slirp/loopback/LAN sources are in `pass_ip`.
|
||||
- The limiter stays armed for untrusted sources (a spoofed public IP still 429s). Do not disable it. Widen only `pass_ip` for sources you trust on this host.
|
||||
- The config file is root-owned (uid of the rootless podman namespace). Update it with a one-off `sudo cp`, then run `systemctl --user restart searxng.service` to reload.
|
||||
- `base_url` must match the public address or proxy rewrites break.
|
||||
- Keep the valkey cache container on the same podman network as searxng.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
- [[Gluetun]] — the VPN container
|
||||
- [[SearXNG Providers]] — unblocking Google & other upstream engines (VPN exit IP, engine config, proxies, verification)
|
||||
- Knowledge base: [[Gluetun]], [[SearXNG]], [[SearXNG Providers]], [[Hermes]]
|
||||
@@ -0,0 +1,55 @@
|
||||
# Valkey
|
||||
|
||||
Valkey is the cache and limiter store for SearXNG. It replaces Redis as the default in the official SearXNG deployment.
|
||||
|
||||
**Host:** Dankclaw — **Docs:** https://valkey.io
|
||||
|
||||
## Facts (from research)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `valkey/valkey:9-alpine` |
|
||||
| Port | 6379 (internal, on the SearXNG network) |
|
||||
| Role | SearXNG limiter + result cache |
|
||||
|
||||
## Why Valkey
|
||||
|
||||
The official SearXNG docker-compose ships a `searxng-valkey` container. Valkey is the Redis fork under the Linux Foundation umbrella; the SearXNG project moved to it as the default cache.
|
||||
|
||||
## Configuration
|
||||
|
||||
The valkey unit is `searxng-valkey.container` (see [[SearXNG]] for the trio). It joins the searxng network and is reachable by the name `searxng-valkey` (container name) and the alias `valkey`:
|
||||
|
||||
```ini
|
||||
[Container]
|
||||
ContainerName=searxng-valkey
|
||||
Network=searxng.network
|
||||
NetworkAlias=valkey
|
||||
Image=docker.io/valkey/valkey:9-alpine
|
||||
Volume=/opt/searxng/valkey:/data
|
||||
```
|
||||
|
||||
SearXNG connects to it via `settings.yml`:
|
||||
|
||||
```yaml
|
||||
valkey:
|
||||
url: "valkey://searxng-valkey:6379/0"
|
||||
```
|
||||
|
||||
It sets `WantedBy=default.target`. User linger is on, so it starts at boot.
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path (convention) |
|
||||
|----------------|------------------------|
|
||||
| `/data` | `/opt/searxng/valkey` |
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The limiter relies on Valkey; without it SearXNG falls back to in-memory state and rate limiting is per-restart.
|
||||
- Keep Valkey on the same podman network as SearXNG.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[SearXNG]] — the consumer
|
||||
- [[Dankclaw Stack]]
|
||||
@@ -0,0 +1,369 @@
|
||||
# *arr Stack
|
||||
|
||||
The *arr suite is a collection of automated media management tools for Usenet and BitTorrent. For Dankserver — which runs Copyparty, Calibre-Web, RomM, and Jellyfin — the *arr stack provides automatic downloading and library management that Copyparty alone does not cover: quality profile enforcement, series/movie discovery, metadata enrichment, and automated file organization.
|
||||
|
||||
The core chain is: **Prowlarr** (indexer API aggregator) → **Sonarr/Radarr/Lidarr/Readarr** (media-specific managers) → **qBittorrent** (download client) → **Jellyfin** (media server scan → auto-refresh).
|
||||
|
||||
## Service Overview
|
||||
|
||||
| Service | Role | Image | Port | Min RAM |
|
||||
|---------|------|-------|------|---------|
|
||||
| Prowlarr | Indexer management | `lscr.io/linuxserver/prowlarr:latest` | 9696 | 256 MB |
|
||||
| Radarr | Movie management | `lscr.io/linuxserver/radarr:latest` | 7878 | 512 MB |
|
||||
| Sonarr | TV show management | `lscr.io/linuxserver/sonarr:latest` | 8989 | 512 MB |
|
||||
| Lidarr | Music management | `lscr.io/linuxserver/lidarr:latest` | 8686 | 256 MB |
|
||||
| Readarr | Ebook/audiobook management | `lscr.io/linuxserver/readarr:latest` | 8787 | 256 MB |
|
||||
| qBittorrent | Torrent download client | `lscr.io/linuxserver/qbittorrent:latest` | 8080 | 512 MB |
|
||||
| Jackett | Indexer proxy (optional) | `lscr.io/linuxserver/jackett:latest` | 9117 | 256 MB |
|
||||
| Whisparr | NSFW movie management (optional) | `lscr.io/linuxserver/whisparr:latest` | 6969 | 256 MB |
|
||||
|
||||
**Minimum total RAM:** ~3 GB for full stack (headroom recommended).
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
I[Indexers<br/>Torrents, Usenet] --> P[Prowlarr<br/>Indexer Manager]
|
||||
P --> S[Sonarr<br/>TV]
|
||||
P --> R[Radarr<br/>Movies]
|
||||
P --> L[Lidarr<br/>Music]
|
||||
P --> Rd[Readarr<br/>Books]
|
||||
S --> Q[qBittorrent]
|
||||
R --> Q
|
||||
L --> Q
|
||||
Rd --> Q
|
||||
Q --> M1[/tv/]
|
||||
Q --> M2[/movies/]
|
||||
Q --> M3[/music/]
|
||||
Q --> M4[/books/]
|
||||
M1 --> J[Jellyfin]
|
||||
M2 --> J
|
||||
M3 --> J
|
||||
M4 --> CW[Calibre-Web]
|
||||
```
|
||||
|
||||
## Docker Compose
|
||||
|
||||
```yaml
|
||||
networks:
|
||||
appnet:
|
||||
driver: bridge
|
||||
|
||||
services:
|
||||
prowlarr:
|
||||
image: lscr.io/linuxserver/prowlarr:latest
|
||||
container_name: prowlarr
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- PUID=1000
|
||||
- PGID=1000
|
||||
- TZ=America/New_York
|
||||
- UMASK_SET=022
|
||||
volumes:
|
||||
- /opt/prowlarr/config:/config
|
||||
- /mnt/media/downloads:/downloads:ro
|
||||
ports:
|
||||
- "9696:9696"
|
||||
networks: [appnet]
|
||||
|
||||
radarr:
|
||||
image: lscr.io/linuxserver/radarr:latest
|
||||
container_name: radarr
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- PUID=1000
|
||||
- PGID=1000
|
||||
- TZ=America/New_York
|
||||
- UMASK_SET=022
|
||||
volumes:
|
||||
- /opt/radarr/config:/config
|
||||
- /mnt/media/movies:/movies
|
||||
- /mnt/media/downloads:/downloads
|
||||
ports:
|
||||
- "7878:7878"
|
||||
networks: [appnet]
|
||||
|
||||
sonarr:
|
||||
image: lscr.io/linuxserver/sonarr:latest
|
||||
container_name: sonarr
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- PUID=1000
|
||||
- PGID=1000
|
||||
- TZ=America/New_York
|
||||
- UMASK_SET=022
|
||||
volumes:
|
||||
- /opt/sonarr/config:/config
|
||||
- /mnt/media/tv:/tv
|
||||
- /mnt/media/downloads:/downloads
|
||||
ports:
|
||||
- "8989:8989"
|
||||
networks: [appnet]
|
||||
|
||||
lidarr:
|
||||
image: lscr.io/linuxserver/lidarr:latest
|
||||
container_name: lidarr
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- PUID=1000
|
||||
- PGID=1000
|
||||
- TZ=America/New_York
|
||||
- UMASK_SET=022
|
||||
volumes:
|
||||
- /opt/lidarr/config:/config
|
||||
- /mnt/media/music:/music
|
||||
- /mnt/media/downloads:/downloads
|
||||
ports:
|
||||
- "8686:8686"
|
||||
networks: [appnet]
|
||||
|
||||
readarr:
|
||||
image: lscr.io/linuxserver/readarr:latest
|
||||
container_name: readarr
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- PUID=1000
|
||||
- PGID=1000
|
||||
- TZ=America/New_York
|
||||
- UMASK_SET=022
|
||||
volumes:
|
||||
- /opt/readarr/config:/config
|
||||
- /mnt/media/books:/books
|
||||
- /mnt/media/downloads:/downloads
|
||||
ports:
|
||||
- "8787:8787"
|
||||
networks: [appnet]
|
||||
|
||||
qbittorrent:
|
||||
image: lscr.io/linuxserver/qbittorrent:latest
|
||||
container_name: qbittorrent
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- PUID=1000
|
||||
- PGID=1000
|
||||
- TZ=America/New_York
|
||||
- UMASK_SET=022
|
||||
volumes:
|
||||
- /opt/qbittorrent/config:/config
|
||||
- /mnt/media/downloads:/downloads
|
||||
- /mnt/media/downloads/incomplete:/downloads/incomplete
|
||||
ports:
|
||||
- "8080:8080"
|
||||
networks: [appnet]
|
||||
|
||||
# Optional: Jackett for niche indexers
|
||||
jackett:
|
||||
image: lscr.io/linuxserver/jackett:latest
|
||||
container_name: jackett
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- PUID=1000
|
||||
- PGID=1000
|
||||
- TZ=America/New_York
|
||||
- UMASK_SET=022
|
||||
volumes:
|
||||
- /opt/jackett/config:/config
|
||||
- /opt/jackett/torrences:/torrences
|
||||
ports:
|
||||
- "9117:9117"
|
||||
networks: [appnet]
|
||||
|
||||
# Optional: Whisparr for adult content
|
||||
whisparr:
|
||||
image: lscr.io/linuxserver/whisparr:latest
|
||||
container_name: whisparr
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- PUID=1000
|
||||
- PGID=1000
|
||||
- TZ=America/New_York
|
||||
- UMASK_SET=022
|
||||
volumes:
|
||||
- /opt/whisparr/config:/config
|
||||
- /mnt/media/whisparr-movies:/whisparr-movies
|
||||
- /mnt/media/downloads:/downloads
|
||||
ports:
|
||||
- "6969:6969"
|
||||
networks: [appnet]
|
||||
```
|
||||
|
||||
## Caddy Configuration
|
||||
|
||||
```caddy
|
||||
radarr.dankserver.local {
|
||||
reverse_proxy localhost:7878
|
||||
}
|
||||
|
||||
sonarr.dankserver.local {
|
||||
reverse_proxy localhost:8989
|
||||
}
|
||||
|
||||
lidarr.dankserver.local {
|
||||
reverse_proxy localhost:8686
|
||||
}
|
||||
|
||||
readarr.dankserver.local {
|
||||
reverse_proxy localhost:8787
|
||||
}
|
||||
|
||||
prowlarr.dankserver.local {
|
||||
reverse_proxy localhost:9696
|
||||
}
|
||||
|
||||
qbittorrent.dankserver.local {
|
||||
reverse_proxy localhost:8080
|
||||
}
|
||||
|
||||
# Optional: Jackett
|
||||
jackett.dankserver.local {
|
||||
reverse_proxy localhost:9117
|
||||
}
|
||||
|
||||
# Optional: Whisparr
|
||||
whisparr.dankserver.local {
|
||||
reverse_proxy localhost:6969
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration Steps
|
||||
|
||||
### Phase 1: Download Client
|
||||
|
||||
1. Deploy qBittorrent first (or Transmission).
|
||||
2. In the qBittorrent Web UI, set **No auth mode** for LAN (Settings → Web UI).
|
||||
3. Create download categories: `movies`, `tv`, `music`, `books`.
|
||||
4. Change the default Web UI password.
|
||||
|
||||
### Phase 2: Indexer Management
|
||||
|
||||
1. Deploy Prowlarr.
|
||||
2. Add indexers (3–5 recommended for testing). Use the **Test** button on each.
|
||||
3. Add qBittorrent as a download client in Prowlarr.
|
||||
4. Verify search from the Prowlarr web UI.
|
||||
|
||||
### Phase 3: Arr Apps (one at a time)
|
||||
|
||||
1. Deploy Sonarr first (TV shows are the most common use case).
|
||||
2. Add Prowlarr as the indexer source (Settings → Indexers → Add Prowlarr).
|
||||
3. Add qBittorrent as the download client with categories.
|
||||
4. Add a test series. Wait for the episode to download. Verify in `/tv/`.
|
||||
5. Repeat for Radarr, Lidarr, Readarr.
|
||||
|
||||
### Phase 4: Media Server Integration
|
||||
|
||||
1. Configure Jellyfin libraries pointing to `/movies`, `/tv`, `/music`, `/books`.
|
||||
2. Set **On Download** notifications in each arr app → Jellyfin webhook URL.
|
||||
3. Test end-to-end: Add a movie in Radarr → download → verify in Jellyfin.
|
||||
|
||||
## Quality Profiles
|
||||
|
||||
Each arr app uses a hierarchical quality system:
|
||||
|
||||
1. **Quality Profile** — Defines tiers (e.g., 2160p → 1080p BluRay → 720p).
|
||||
2. **Cutoff** — The quality target. Downloads at or above cutoff are accepted.
|
||||
3. **Custom Formats** — Additional filters (e.g., "Has HDR", "Is Director's Cut").
|
||||
4. **Search on Grab** — When content becomes available, all indexers are queried.
|
||||
|
||||
### Radarr (Movies)
|
||||
|
||||
```
|
||||
Profile: Movies 4K
|
||||
Tiers:
|
||||
1. 2160p BluRay REMUX
|
||||
2. 2160p BluRay
|
||||
3. 2160p WEB-DL
|
||||
4. 1080p BluRay
|
||||
5. 1080p WEB-DL
|
||||
6. 720p
|
||||
Cutoff: 1080p BluRay
|
||||
```
|
||||
|
||||
### Sonarr (TV)
|
||||
|
||||
```
|
||||
Profile: TV - Standard
|
||||
Tiers: 2160p → 1080p → 720p
|
||||
Min Size: 500 MB
|
||||
Max Size: 50 GB
|
||||
Cutoff: 1080p
|
||||
Language Profile: English audio preferred
|
||||
```
|
||||
|
||||
### Lidarr (Music)
|
||||
|
||||
```
|
||||
Profile: Music - Standard
|
||||
Tiers: Lossless (FLAC) → 320 kbps → 192 kbps
|
||||
Cutoff: 320 kbps (or Lossless)
|
||||
```
|
||||
|
||||
## Metadata Sources
|
||||
|
||||
| App | Primary Source | Secondary |
|
||||
|-----|---------------|-----------|
|
||||
| Radarr | TMDB | TheMovieDB API |
|
||||
| Sonarr | TMDB | TVDB |
|
||||
| Lidarr | MusicBrainz | — |
|
||||
| Readarr | Goodreads, StoryGraph | — |
|
||||
| Whisparr | XH | TMDB |
|
||||
|
||||
## *arr Stack vs Copyparty
|
||||
|
||||
| Feature | Copyparty | *arr Stack |
|
||||
|---------|-----------|------------|
|
||||
| Auto-download from torrents | Basic (RSS + scripts) | Native, integrated |
|
||||
| Quality profiles | No | Multi-tier, custom formats |
|
||||
| Series/movie tracking | No | Watch lists, air dates |
|
||||
| Manual library management | Yes | Yes, metadata-driven |
|
||||
| Multi-user watchlists | No | Yes |
|
||||
| Complexity | Low | Medium |
|
||||
| Resource usage | Minimal | Moderate |
|
||||
|
||||
Use Copyparty for curated file collections. Use *arr for automatic downloads. Use both together for maximum functionality — point *arr download directories at Copyparty's watched folders.
|
||||
|
||||
## Disk Layout
|
||||
|
||||
```
|
||||
/mnt/media/
|
||||
├── movies/ → Radarr root
|
||||
├── tv/ → Sonarr root
|
||||
├── music/ → Lidarr root
|
||||
├── books/ → Readarr root
|
||||
├── whisparr-movies/ → Whisparr root
|
||||
├── downloads/ → qBittorrent download dir
|
||||
│ ├── movies/
|
||||
│ ├── tv/
|
||||
│ ├── music/
|
||||
│ └── books/
|
||||
└── incomplete/ → qBittorrent incomplete downloads
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Timezone mismatch:** All containers should use the same TZ. Sonarr's scheduler is timezone-sensitive.
|
||||
- **File permissions:** Set `umask=022` in qBittorrent. Ensure arr apps can read/move downloaded files.
|
||||
- **Network conflicts:** *arr apps use many ports. Check for conflicts with existing Dankserver services.
|
||||
- **Prowlarr API key rotation:** If rotated, update it in every arr app.
|
||||
- **qBittorrent Web UI password:** Change default on first run.
|
||||
- **Readarr stability:** Readarr is still in development. Do not rely on it for mission-critical collections.
|
||||
- **Whisparr separate Jellyfin library:** Keep adult content separate from regular media.
|
||||
- **RSS feed in qBittorrent:** Can auto-add torrents matching a pattern. Useful for podcast-like series.
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. Use tags to route content to specific download clients or categories.
|
||||
2. Enable the Blocklist in arr apps to prevent re-downloading failed torrents.
|
||||
3. Set download limits in qBittorrent to avoid bandwidth saturation.
|
||||
4. Back up arr configs regularly (they are small SQLite DBs + JSON).
|
||||
5. Monitor Prowlarr indexer health — broken indexers hide new content.
|
||||
6. Start with default quality profiles. Add custom formats gradually.
|
||||
7. Back up arr configs: `./radarr/config`, `./sonarr/config`, etc.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankserver Stack]]
|
||||
- [[Jellyfin]] — media server that consumes *arr downloads
|
||||
- [[Copyparty]] — alternative file viewer
|
||||
- [[Calibre-Web]] — manual ebook management (Readarr auto-downloads into the same library)
|
||||
- [Sonarr Wiki](https://wiki.servarr.com/sonarr)
|
||||
- [Radarr Wiki](https://wiki.servarr.com/radarr)
|
||||
@@ -0,0 +1,93 @@
|
||||
# Calibre-Web
|
||||
|
||||
Calibre-Web is a web application that browses, reads, and downloads eBooks from an existing Calibre library database. It serves as the web frontend for the Calibre e-book manager.
|
||||
|
||||
**Host:** Dankserver — **Hostname:** `books.dankserver.local` — **Docs:** https://github.com/linuxserver/docker-calibre-web
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Latest version | 0.6.25-ls365 (live); 0.6.27-ls397 upstream |
|
||||
| Image | `lscr.io/linuxserver/calibre-web:latest` (live: 0.6.25-ls365) |
|
||||
| Base OS | Ubuntu Noble (24.04 LTS, since 2024-12-05) |
|
||||
| Port | 8083 (host) → 8083 (container) |
|
||||
| Default login | `admin` / `admin123` |
|
||||
| Docker mod | `linuxserver/mods:universal-calibre` (ebook conversion, x86-64 only) |
|
||||
| Volumes | `/config` (internal database + config), `/books` (existing Calibre library) |
|
||||
| Extra tools | `unrar` at `/usr/bin/unrar`, `kepubify` at `/usr/bin/kepubify` (pre-configured since 2025-01-07) |
|
||||
|
||||
## CVEs Fixed in Latest Version
|
||||
|
||||
| CVE | Issue |
|
||||
|-----|-------|
|
||||
| CVE-2025-7404 | External binary names (e.g. `unrar`) restricted to prevent RCE |
|
||||
| — | XSS in book comments imported via Calibre, now sanitized |
|
||||
|
||||
Additional packages added: `libxfixes3` and `libasound2t64` to support epub-to-pdf conversion. The `--no-sandbox` flag is applied to qtwebengine to prevent crashes in Docker.
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Value | Description |
|
||||
|----------|-------|-------------|
|
||||
| `PUID` | `1000` | User ID |
|
||||
| `PGID` | `1000` | Group ID |
|
||||
| `TZ` | `Etc/UTC` | Timezone |
|
||||
| `DOCKER_MODS` | `linuxserver/mods:universal-calibre` | Optional: enables ebook conversion tools |
|
||||
| `OAUTHLIB_RELAX_TOKEN_SCOPE` | `1` | Required for Google OAuth |
|
||||
|
||||
### Setup Steps
|
||||
|
||||
1. Point the container at the Calibre library directory (set the `/books` volume).
|
||||
2. Log in with `admin` / `admin123` and change the password immediately.
|
||||
3. In Basic Configuration → Calibre Database → set Location of Calibre database to `/books`.
|
||||
4. In Basic Configuration → External Binaries → set the directory to `/usr/bin/` for Calibre-Web >= 0.6.22.
|
||||
5. **kepubify is pre-configured** since version 0.6.27. No need to set the kepubify path manually.
|
||||
6. Set up Google OAuth if desired (requires `OAUTHLIB_RELAX_TOKEN_SCOPE=1`).
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path (convention) |
|
||||
|----------------|------------------------|
|
||||
| `/config` | `/opt/calibre-web/config` |
|
||||
| `/books` | The existing Calibre library directory (user-entered during setup) |
|
||||
|
||||
## Docker Compose
|
||||
|
||||
```yaml
|
||||
services:
|
||||
calibre-web:
|
||||
image: lscr.io/linuxserver/calibre-web:0.6.27-ls397
|
||||
container_name: calibre-web
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
PUID: "1000"
|
||||
PGID: "1000"
|
||||
TZ: Etc/UTC
|
||||
OAUTHLIB_RELAX_TOKEN_SCOPE: "1"
|
||||
ports:
|
||||
- "8083:8083"
|
||||
volumes:
|
||||
- /opt/calibre-web/config:/config
|
||||
- /opt/calibre-web/books:/books
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8083/"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Wrong database path:** Login appears to succeed but the library is empty. Verify the Calibre database path in the admin page.
|
||||
- **Ebook conversion:** Requires the `universal-calibre` Docker mod AND the external binary path set to `/usr/bin/` (directory, not full path to `ebook-convert`).
|
||||
- **Google OAuth:** Set `OAUTHLIB_RELAX_TOKEN_SCOPE=1` or OAuth login fails.
|
||||
- **calibre 9.9.0 content server security fix:** A bug since v6.16.0 allows any user with server access to read non-Calibre files on the host. Upgrade to 9.9.0+ if serving content to untrusted users.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankserver Stack]]
|
||||
- [[Calibre]] — the underlying library engine (host tool, not a container)
|
||||
- [[ARR-Stack]] — Readarr for automatic ebook downloads into the Calibre library
|
||||
@@ -0,0 +1,69 @@
|
||||
# Calibre
|
||||
|
||||
Calibre is the e-book library manager that backs Calibre-Web. It manages metadata, formats, and conversions for the library that Calibre-Web serves.
|
||||
|
||||
**Host:** Dankserver (library directory on host) — **Docs:** https://calibre-ebook.com
|
||||
|
||||
## Role in this Stack
|
||||
|
||||
Calibre itself is a desktop/GUI application, not a container service. This stack runs **Calibre-Web** in a container and points it at an existing Calibre library directory on the Dankserver host.
|
||||
|
||||
Calibre is used to:
|
||||
- Import books into the library (drag-and-drop or CLI `calibredb`)
|
||||
- Edit metadata (author, title, series, tags, cover art)
|
||||
- Convert between formats (EPUB → MOBI, MOBI → EPUB, etc.)
|
||||
- Manage collections and reading lists
|
||||
|
||||
The actual library files (`.epub`, `.mobi`, `.pdf`) and the Calibre database (`.db` file) live on the host filesystem. Calibre-Web reads and writes them through the mounted volume.
|
||||
|
||||
## Current Version
|
||||
|
||||
Calibre 9.2.1 is the version on the live host (9.9.0 is the upstream latest). Major features include:
|
||||
- Bookshelf view
|
||||
- Opt-in AI features
|
||||
- Content server security fix (CVE-2025-7404 equivalent): a bug since v6.16.0 allows any user with server access to read non-Calibre files on the host. Upgrade to 9.9.0+ if serving content to untrusted users.
|
||||
|
||||
## Library Directory
|
||||
|
||||
| Purpose | Host path (convention) |
|
||||
|---------|------------------------|
|
||||
| Calibre library | `/opt/calibre-web/books` (bind-mounted to `/books` in the Calibre-Web container) |
|
||||
|
||||
## What the Container Needs
|
||||
|
||||
- Read/write access to the library directory. Calibre-Web edits metadata through the database file.
|
||||
- Ownership matching `PUID=1000` / `PGID=1000` (the `dank` user on Dankserver).
|
||||
|
||||
## Optional: Calibre Content Server
|
||||
|
||||
Calibre includes a built-in HTTP content server (`calibre-server`) that can serve the library via OPDS feeds and a mobile-friendly web interface. This enables:
|
||||
- OPDS feeds for mobile reading apps (FBReader, Moon+ Reader)
|
||||
- Mobile-optimized UI via the `--mobile` flag
|
||||
- Direct Kindle push
|
||||
|
||||
To run as a container:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
calibre-server:
|
||||
image: lscr.io/linuxserver/calibre:latest
|
||||
container_name: calibre-server
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
PUID: "1000"
|
||||
PGID: "1000"
|
||||
TZ: Etc/UTC
|
||||
command: calibre-server --daemonize --port 8080 --enable-auth --url-prefix /calibre /books
|
||||
volumes:
|
||||
- /opt/calibre-web/books:/books
|
||||
ports:
|
||||
- "8084:8080"
|
||||
```
|
||||
|
||||
Note: The content server runs headless in `--daemonize` mode. It requires the full Calibre package, not a minimal server.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Calibre-Web]] — the containerized frontend
|
||||
- [[Dankserver Stack]]
|
||||
- [[ARR-Stack]] — Readarr for automatic ebook downloads
|
||||
@@ -0,0 +1,106 @@
|
||||
# Copyparty
|
||||
|
||||
Copyparty is a lightweight file server and media streamer that runs as a single Python binary. It handles large payloads, media streaming, resumable uploads, directory browsing, and WebDAV from one process.
|
||||
|
||||
**Host:** Dankserver — **Hostname:** `files.dankserver.local` — **Docs:** https://github.com/9001/copyparty
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Latest version | v1.20.21 (2026-08) |
|
||||
| Image | `ghcr.io/9001/copyparty:latest` (official Dockerfile in repo) |
|
||||
| Port | 3923 (host) → 3923 (container) |
|
||||
| Volume | bind mount the shared root directory |
|
||||
| Auth | Per-directory `.cpy` config files, HTTP basic auth, `dirkeys`, `filekeys` |
|
||||
| Streaming | HTTP range requests, HLS, WebDAV, WOPI (Office editing) |
|
||||
| Docker editions | `dj` (document + media), `iv` (image + video), `v` (video only) |
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Description |
|
||||
|----------|-------------|
|
||||
| `COPAY_PASS` | Master password for the web UI |
|
||||
| `COPAY_MODE` | Set to `rw` for read-write shares |
|
||||
|
||||
### Configuration File Options
|
||||
|
||||
Copyparty reads its configuration from `initcfg` (or `config` in the `/data` directory). Key options:
|
||||
|
||||
- `use-bwrap: n` — Disable bwrap sandboxing for ffmpeg (default in Docker). Set to `y` to enable.
|
||||
- `shadowing //NULL` — Full unmap of a volume from the filesystem.
|
||||
- Config env-var expansion: Use `${VAR}` syntax in `[global]` section for environment variable substitution.
|
||||
|
||||
### Docker Editions
|
||||
|
||||
Different editions bundle different libraries:
|
||||
|
||||
| Edition | Bundled Libraries | Use Case |
|
||||
|---------|-------------------|----------|
|
||||
| `dj` | Document + image + video libraries | Full-featured server |
|
||||
| `iv` | Image + video libraries | Image and video focused |
|
||||
| `v` | Video only | Transcoding only, minimal footprint |
|
||||
|
||||
The `iv` edition is no longer available for arm32/armv6. Use `dj` for maximum compatibility.
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path (convention) |
|
||||
|----------------|------------------------|
|
||||
| `/data` | `/opt/copyparty/data` |
|
||||
|
||||
## Docker Compose
|
||||
|
||||
```yaml
|
||||
services:
|
||||
copyparty:
|
||||
image: ghcr.io/9001/copyparty:latest
|
||||
container_name: copyparty
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
COPAY_PASS: "${COPAY_PASS}"
|
||||
COPAY_MODE: rw
|
||||
ports:
|
||||
- "3923:3923"
|
||||
volumes:
|
||||
- /opt/copyparty/data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "python3", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:3923/')"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
```
|
||||
|
||||
## Healthcheck
|
||||
|
||||
A simple HTTP check on port 3923 works. The root path returns HTTP 200 with the directory listing.
|
||||
|
||||
## WOPI Integration
|
||||
|
||||
Copyparty v1.20.9+ supports WOPI (Web Application Open Platform Interface) for editing Office documents in the web UI via Collabora Online. Enable with the `--wopi-accs` flag to limit WOPI access to specific paths.
|
||||
|
||||
## Security Advisories
|
||||
|
||||
| Version | CVE | Issue |
|
||||
|---------|-----|-------|
|
||||
| v1.20.19 | GHSA-phv8-wgjp-g4p9 | FTP upload outside volume |
|
||||
| v1.20.17 | GHSA-x5pq-m9p8-f4vx | dirkey/filekey confusion |
|
||||
| v1.20.9 | CVE-2026-27948 / GHSA-62cr-6wp5-q43h | XSS via file upload |
|
||||
|
||||
Upgrade to v1.20.19+ to cover all known CVEs. The dirkey fix (v1.20.17) matters if using both `dirkeys` and `filekeys`.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **Large transfers:** Disable TLS inside the app and let Caddy terminate TLS. This avoids double encryption overhead.
|
||||
- **Dedicated share directory:** Do not give Copyparty the whole disk. Mount only the intended share path.
|
||||
- **Upload tokens scoped per folder:** Prevents accidental global write access.
|
||||
- **CSP nonces (v1.20.17+):** Security hardening for JavaScript execution. May break third-party JS plugins (`--js-browser` / `--html-head`).
|
||||
- **Custom thumbnail extractors (v1.20.19+):** New `thumbex` framework. Requires configuration in `initcfg`.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankserver Stack]]
|
||||
- [[Caddy]] — TLS termination and authentication in front
|
||||
- [[ARR-Stack]] — Copyparty as a viewer alongside automated download management
|
||||
@@ -0,0 +1,55 @@
|
||||
# Dankserver Stack
|
||||
|
||||
Dankserver (192.168.0.235) runs the media and library stack. Every service is reachable by hostname through the Caddy reverse proxy on the host.
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
U[User] --> C[Caddy :80/:443]
|
||||
C --> R[RomM]
|
||||
C --> P[Copyparty]
|
||||
C --> W[Calibre-Web]
|
||||
C --> J[Jellyfin]
|
||||
C --> S[Sonarr]
|
||||
C --> D[Radarr]
|
||||
C --> L[Lidarr]
|
||||
C --> A[Readarr]
|
||||
C --> O[Prowlarr]
|
||||
W --> L2[Calibre library]
|
||||
```
|
||||
|
||||
## Services
|
||||
|
||||
| Service | Hostname | Port | Docs |
|
||||
|---------|----------|------|------|
|
||||
| RomM | romm.dankserver.local | 8085 | [[RomM]] |
|
||||
| Copyparty | files.dankserver.local | 3923 | [[Copyparty]] |
|
||||
| Calibre-Web | books.dankserver.local | 8083 | [[Calibre-Web]] |
|
||||
| Jellyfin | jellyfin.dankserver.local | 8096 | [[Jellyfin]] |
|
||||
| Sonarr | sonarr.dankserver.local | 8989 | [[ARR-Stack]] |
|
||||
| Radarr | radarr.dankserver.local | 7878 | [[ARR-Stack]] |
|
||||
| Lidarr | lidarr.dankserver.local | 8686 | [[ARR-Stack]] |
|
||||
| Readarr | readarr.dankserver.local | 8787 | [[ARR-Stack]] |
|
||||
| Prowlarr | prowlarr.dankserver.local | 9696 | [[ARR-Stack]] |
|
||||
| qBittorrent | qbittorrent.dankserver.local | 8080 | [[ARR-Stack]] |
|
||||
|
||||
## Optional Services
|
||||
|
||||
These services are included in the compose file but may be disabled.
|
||||
|
||||
| Service | Hostname | Port | Docs |
|
||||
|---------|----------|------|------|
|
||||
| Jackett | jackett.dankserver.local | 9117 | [[ARR-Stack]] |
|
||||
| Whisparr | whisparr.dankserver.local | 6969 | [[ARR-Stack]] |
|
||||
|
||||
## Removed Services
|
||||
|
||||
The following services ran on Dankserver and are now retired from this stack:
|
||||
|
||||
- OpenWebUI — moved to Dankclaw (see [[OpenWebUI]])
|
||||
- llama-swap — retired; no replacement
|
||||
- SearXNG — moved to Dankclaw (see [[SearXNG]])
|
||||
|
||||
## See Also
|
||||
|
||||
- [[index]] — repository index
|
||||
- [[Network Architecture]] — routing and proxy
|
||||
@@ -0,0 +1,43 @@
|
||||
# Entropy Volume
|
||||
|
||||
The Dankserver entropy volume is a RAID-backed LUKS filesystem mounted at
|
||||
`/mnt/entropy`. The command-line helper reuses the host's desktop-integrated
|
||||
`udisks2`, `crypttab`, and `fstab` configuration; it does not store or handle
|
||||
the passphrase itself.
|
||||
|
||||
## Command-line mount
|
||||
|
||||
```bash
|
||||
/home/dank/serverAppConfigs/scripts/mount-entropy.sh
|
||||
```
|
||||
|
||||
Run it as the normal desktop user (`dank`), not from a root shell. `udisksctl`
|
||||
will securely prompt for the LUKS passphrase. `sudo mount /mnt/entropy` then
|
||||
uses the existing `/etc/fstab` entry, preserving its options:
|
||||
|
||||
```text
|
||||
nosuid,nodev
|
||||
```
|
||||
|
||||
The existing mountpoint is owned by `dank:dankserver` with mode `0755`; the
|
||||
helper deliberately does not run `chmod` or `chown`, so it cannot silently
|
||||
change those permissions.
|
||||
|
||||
## Device path
|
||||
|
||||
The current storage chain is:
|
||||
|
||||
```text
|
||||
/dev/md0 -> LUKS UUID cd86c728-459e-4f3a-bcc2-41ecf84685ec
|
||||
-> ext4 label entropy
|
||||
-> /mnt/entropy
|
||||
```
|
||||
|
||||
The helper checks that `/dev/md0` exists and exits successfully if the volume
|
||||
is already mounted. It is intentionally specific to this host and volume:
|
||||
that avoids accidentally unlocking or mounting a different disk.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Storage Policy]]
|
||||
- [[Dankserver Stack]]
|
||||
@@ -0,0 +1,173 @@
|
||||
# Jellyfin
|
||||
|
||||
Jellyfin is a free, open-source media server and suite of applications for organizing, managing, and streaming personal media collections (movies, TV shows, music, photos) to networked devices. It is a community fork of Emby, written in C# (.NET), and uses FFmpeg for transcoding.
|
||||
|
||||
**Host:** Dankserver — **Hostname:** `jellyfin.dankserver.local` — **Docs:** https://jellyfin.org
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Latest version | Check https://jellyfin.org/downloads |
|
||||
| Image | `jellyfin/jellyfin:latest` (Docker Hub) or `ghcr.io/jellyfin/jellyfin` |
|
||||
| Base | Debian-based |
|
||||
| Primary port | 8096 (HTTP, web UI + API) |
|
||||
| HTTPS port | 8920 (optional) |
|
||||
| Client discovery | 7359/udp |
|
||||
| DLNA | 1900/udp (optional) |
|
||||
| GPU passthrough | `/dev/dri` (Intel QSV / AMD VA-API) or NVIDIA NVENC |
|
||||
| License | MIT |
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
U[User] --> C[Caddy :80/:443]
|
||||
C --> J[Jellyfin :8096]
|
||||
J --> Lib[/media/movies]
|
||||
J --> Lib2[/media/shows]
|
||||
J --> Lib3[/media/music]
|
||||
J --> Conf[Jellyfin config]
|
||||
J --> Cache[Jellyfin cache]
|
||||
```
|
||||
|
||||
## Docker Compose
|
||||
|
||||
```yaml
|
||||
services:
|
||||
jellyfin:
|
||||
image: jellyfin/jellyfin:latest
|
||||
container_name: jellyfin
|
||||
restart: unless-stopped
|
||||
user: "1000:1000"
|
||||
ports:
|
||||
- "8096:8096/tcp"
|
||||
- "8920:8920/tcp"
|
||||
- "7359:7359/udp"
|
||||
- "1900:1900/udp"
|
||||
environment:
|
||||
- JELLYFIN_PublishedServerUrl=http://192.168.0.235:8096
|
||||
- DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true
|
||||
volumes:
|
||||
- /opt/jellyfin/config:/config
|
||||
- /opt/jellyfin/cache:/cache
|
||||
- /mnt/media/movies:/data/movies:ro
|
||||
- /mnt/media/shows:/data/shows:ro
|
||||
devices:
|
||||
- /dev/dri:/dev/dri
|
||||
group_add:
|
||||
- "992" # Host render group GID — verify with `getent group render`
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8096/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
```
|
||||
|
||||
## Caddy Configuration
|
||||
|
||||
```caddy
|
||||
jellyfin.dankserver.local {
|
||||
reverse_proxy 127.0.0.1:8096
|
||||
|
||||
transport http {
|
||||
versions h2c 1.1
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Set **Base URL** to `/` (or the path you configure) in Jellyfin Dashboard → Networking. Add `127.0.0.1` to **Known Proxies**.
|
||||
|
||||
## Storage
|
||||
|
||||
| Mount | Purpose | Size Estimate |
|
||||
|-------|---------|---------------|
|
||||
| `/opt/jellyfin/config` | SQLite database, settings, cached images | 10–100 GB |
|
||||
| `/opt/jellyfin/cache` | Transcoding segments, extracted posters | Self-cleaning per session |
|
||||
|
||||
The database must be on local storage (SSD/NVMe preferred). Media libraries can be on NFS/mergerfs. If media is on NFS, set `DOTNET_SYSTEM_IO_DISABLEFILELOCKING=true`.
|
||||
|
||||
## Authentication
|
||||
|
||||
- **Built-in auth (default):** Per-user libraries, parental controls, max bitrate limits. Manage via Dashboard → Users.
|
||||
- **Caddy basic auth:** Single login layer. Does not integrate with per-user Jellyfin permissions.
|
||||
- **LDAP/OIDC:** Available via plugins (LDAP-Auth, Jellyfin SSO). Integrates with Authentik, Keycloak, etc.
|
||||
|
||||
For Dankserver, built-in auth is recommended. Use Caddy basic auth on top for external access.
|
||||
|
||||
## Metadata Scraping
|
||||
|
||||
| Provider | Content | Type |
|
||||
|----------|---------|------|
|
||||
| TheMovieDb (TMDb) | Movies, TV shows, posters, fan art, overviews | Default |
|
||||
| OMDb API | Movies only, English metadata | Default |
|
||||
| TheTVDB | TV shows, episodes | Plugin |
|
||||
| fanart.tv | Extra artwork, banners, thumbnails | Plugin |
|
||||
| AniDB | Anime metadata | Plugin |
|
||||
| MusicBrainz | Music metadata | Plugin |
|
||||
| Open Subtitles | Subtitle downloads | Plugin |
|
||||
| Intro Skipper | TV show intro detection | Plugin |
|
||||
|
||||
## Clients
|
||||
|
||||
| Platform | App | Quality |
|
||||
|----------|-----|---------|
|
||||
| Web | Jellyfin Web | Excellent |
|
||||
| Android | Jellyfin (Google Play / F-Droid) | Excellent |
|
||||
| Android TV / Fire TV | Jellyfin | Good |
|
||||
| iOS / iPadOS | Swiftfin (third-party) | Good |
|
||||
| Desktop | Jellyfin Media Player (mpv-based) | Excellent |
|
||||
| Roku | Jellyfin (Channel Store) | Good |
|
||||
| Kodi | Jellyfin addon | Excellent |
|
||||
| Samsung/LG Smart TVs | TV browser | Fair |
|
||||
|
||||
## Jellyfin vs Plex
|
||||
|
||||
| Feature | Jellyfin | Plex |
|
||||
|---------|----------|------|
|
||||
| Cost | Free, no tiers | Free core; Plex Pass ($5/mo or $120 lifetime) |
|
||||
| License | MIT | Proprietary |
|
||||
| Account required | No | Yes |
|
||||
| Hardware transcoding | Free | Plex Pass required |
|
||||
| Mobile sync | Free | Plex Pass |
|
||||
| Live TV / DVR | Free | Plex Pass |
|
||||
| Client apps | Good, improving | Excellent, widest device support |
|
||||
|
||||
Run Jellyfin alongside Plex for an incremental migration. Point Jellyfin at the same media folders as Plex. Watch history and metadata will not transfer cleanly.
|
||||
|
||||
## Resource Requirements
|
||||
|
||||
- **Idle / Direct Play:** ~200–500 MB RAM, <5% CPU. Negligible.
|
||||
- **1080p transcode (CPU):** 40–60% CPU on a modern multi-core CPU.
|
||||
- **1080p transcode (with QSV):** 3–5% CPU.
|
||||
- **4K HDR transcode (with QSV):** 5–15% CPU.
|
||||
- **Minimum host RAM:** 4 GB. 8 GB comfortable for transcoding.
|
||||
|
||||
## Deployment Checklist
|
||||
|
||||
1. Verify GPU on host: `ls -l /dev/dri/`, `lspci | grep -i vga`, `getent group render`.
|
||||
2. Install media drivers: `sudo apt install intel-media-va-driver-non-free vainfo intel-gpu-tools`.
|
||||
3. Create directories: `/opt/jellyfin/config`, `/opt/jellyfin/cache`.
|
||||
4. Write compose file, verify device and GID.
|
||||
5. Deploy: `podman compose up -d`.
|
||||
6. Test GPU passthrough: `podman exec jellyfin /usr/lib/jellyfin-ffmpeg/vainfo --display drm --device /dev/dri/renderD128`.
|
||||
7. Configure Caddy reverse proxy. Set Known Proxies in Jellyfin.
|
||||
8. Add media libraries in Dashboard.
|
||||
9. Install plugins: Open Subtitles, fanart.tv, Intro Skipper.
|
||||
10. Test playback on Android/Chrome.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **GPU passthrough silently failing:** `/dev/dri` must be passed and the correct render node used. Verify with `vainfo` inside the container.
|
||||
- **Wrong render group GID:** Common GID is 992, but varies per host. Run `getent group render` to find it.
|
||||
- **Subtitle burn-in triggers full transcode:** PGS (image-based) subtitles force full video transcoding. SRT/ASS (text-based) play fine.
|
||||
- **Database on NFS:** SQLite on network storage causes slow startup and corruption risk. Keep `/config` on local storage.
|
||||
- **Client bitrate limit too low:** Many clients default to 20 Mbps. Set Max Streaming Bitrate to Original or Maximum.
|
||||
- **Naming convention mismatches:** Non-standard folder/file names cause missing metadata. Use `Movie Title (Year)` for movies and `Show Name/Season 01/Show Name S01E01.mkv` for TV.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankserver Stack]]
|
||||
- [[ARR-Stack]] — automated download into Jellyfin libraries
|
||||
- [[Copyparty]] — alternative file viewer for existing collections
|
||||
- Plex — existing media server on this host
|
||||
@@ -0,0 +1,296 @@
|
||||
# RomM Scraping & Asset Management
|
||||
|
||||
This guide covers metadata providers, cover art, ROM naming conventions, emulator integration, and the scraping workflow for RomM on Dankserver.
|
||||
|
||||
**Reference:** https://docs.romm.app
|
||||
|
||||
## 1. Metadata Providers
|
||||
|
||||
RomM v5.1.0+ supports 11+ metadata providers. The `scan.priority.metadata` list determines which provider wins for descriptive fields. The `scan.priority.artwork` list controls cover art sourcing separately.
|
||||
|
||||
### Screenscraper (ScreenScraper.fr)
|
||||
|
||||
- **Type:** Free, community-driven (French provider)
|
||||
- **Credentials:** `SCREENSCRAPER_USER` + `SCREENSCRAPER_PASSWORD` (account user/pass)
|
||||
- **Registration:** https://www.screenscraper.fr/membreinscription.php
|
||||
- **Rate limits:** No documented hard cap. Community-claimed ~5 req/sec burst.
|
||||
- **Capabilities:** Titles, descriptions, cover art, screenshots, manuals, 3D boxes, CD/cartridge covers, marquee logos, fanart, bezels, gameplay videos.
|
||||
- **Region support:** Excellent. Respects `scan.priority.region` and `scan.priority.region_mode`.
|
||||
- **Best for:** Richest media coverage (all art types), multi-platform, free. Use as secondary to IGDB for metadata, or swap priority order if Screenscraper art quality is preferred.
|
||||
|
||||
### IGDB (Internet Game Database / Twitch)
|
||||
|
||||
- **Type:** Freemium (free tier with limits; paid for higher volume)
|
||||
- **Credentials:** `IGDB_CLIENT_ID` + `IGDB_CLIENT_SECRET` (Twitch OAuth2)
|
||||
- **Registration:** Twitch Developer Portal. Create app with name `romm-<random hash>`, OAuth redirect `localhost`, category `Application Integration`, client type `Confidential`.
|
||||
- **Rate limits:** ~10 req/sec for free tier. Cached responses reduce effective calls.
|
||||
- **Capabilities:** Titles, descriptions, cover art, screenshots, related games, release dates, genres, platforms.
|
||||
- **Limitations:** No 3D boxes or physical media art. No manuals. No marquee/fanart.
|
||||
- **Default priority:** Highest (`igdb` listed first in both metadata and artwork priority).
|
||||
|
||||
### MobyGames
|
||||
|
||||
- **Type:** Paid (hobbyist API subscription; non-commercial use only)
|
||||
- **Credentials:** `MOBYGAMES_API_KEY`
|
||||
- **Rate limits:** Hobbyist tier ~0.2 req/sec (one request every 5 seconds).
|
||||
- **RomM recommendation:** "We recommend using ScreenScraper instead, as it is free to use."
|
||||
- **Best for:** Supplemental metadata depth for older PC/console titles.
|
||||
|
||||
### RetroAchievements (RAHasher)
|
||||
|
||||
- **Type:** Free, hash-based achievement tracking
|
||||
- **Credentials:** `RETROACHIEVEMENTS_API_KEY`
|
||||
- **How it works:** RomM bundles RAHasher (v1.8.3+). ROMs are hashed during scans, matched against RA online database. Achievement progress synced per-user via their RA username.
|
||||
- **Known issues:** Folder-based disc ROMs (`.cue` + `.bin`, `.gdi`) may fail hashing. Non-zip archives (`.7z`, `.rar`) had incorrect hash computation. Ensure `filesystem.skip_hash_calculation` is `false`.
|
||||
- **Important:** Achievements cannot be earned via RomM's built-in EmulatorJS player. Must use an RA-capable standalone emulator.
|
||||
|
||||
### Additional Built-in Providers
|
||||
|
||||
| Provider | Slug | Enabled Via | Type |
|
||||
|----------|------|-------------|------|
|
||||
| Hasheous | `hasheous` | `HASHEOUS_API_ENABLED=true` | Free, hash-based, proxies IGDB data |
|
||||
| Playmatch | `playmatch` | `PLAYMATCH_API_ENABLED=true` | Multi-provider hash matching |
|
||||
| LaunchBox | `launchbox` | `LAUNCHBOX_API_ENABLED=true` + XML | Local XML database |
|
||||
| Flashpoint | `flashpoint` | `FLASHPOINT_API_ENABLED=true` | 180,000+ browser games |
|
||||
| HowLongToBeat | `hltb` | `HLTB_API_ENABLED=true` | Game completion times |
|
||||
| SteamGridDB | `steamgriddb` | `STEAMGRIDDB_API_KEY` | Custom cover art |
|
||||
| ES-DE gamelist | `gamelist` | Place `gamelist.xml` in platform folder | Local XML |
|
||||
|
||||
## 2. Recommended Provider Combinations
|
||||
|
||||
| Combo | Providers | Best For | Cost |
|
||||
|-------|-----------|----------|------|
|
||||
| Quick Starter | Hasheous only | No API keys needed | Free |
|
||||
| French Connection | Screenscraper + RetroAchievements | Rich media, free | Free |
|
||||
| Chef's Choice (recommended) | Hasheous + IGDB + SteamGridDB + RetroAchievements | Best overall quality | Free (IGDB free tier) |
|
||||
| Twitch Fanboy | IGDB + Playmatch | Single-provider solution | Free |
|
||||
|
||||
**For Dankserver,** the recommended setup:
|
||||
- `IGDB` (primary metadata) + `SCREENSCRAPER` (artwork backup + rich media) + `RETROACHIEVEMENTS` (achievements) + `HASHEOUS_API_ENABLED=true` (fast hash-based fallback)
|
||||
|
||||
## 3. ROM File Naming Conventions
|
||||
|
||||
### No-Intro Naming Scheme (Recommended)
|
||||
|
||||
The de facto standard for cartridge and handheld ROMs. Only `Title` and `Region` are required.
|
||||
|
||||
```
|
||||
Super Metroid (Japan, USA) (En,Ja)
|
||||
Sonic the Hedgehog (USA)
|
||||
Metal Gear Solid (USA) (Disc 1)
|
||||
[BIOS] PS1 System BIOS (USA)
|
||||
Tetris [!].gba
|
||||
```
|
||||
|
||||
**Field meanings:**
|
||||
- `(Region)`: `USA`, `EUR`, `JPN`, `UE` (USA+Europe), `JUE` (Japan+USA+Europe)
|
||||
- `(Languages)`: `En`, `Ja`, `De`, `Fr`, `Es`
|
||||
- `[Status]`: `[!]` = verified good dump, `[b]` = bad dump, `[c]` = completed
|
||||
- `(Devstatus)`: `(Proto)`, `(Demo)`, `(Release Candidate)`
|
||||
- `[BIOS]` prefix: Identifies BIOS files
|
||||
|
||||
### RomM's Region/Language/Revision Parsing
|
||||
|
||||
RomM parses inline tags from filenames:
|
||||
|
||||
```
|
||||
tetris (En) [1.0001] [HACK] [!].gba
|
||||
```
|
||||
|
||||
- Region/language: ISO-like codes or full names
|
||||
- Revision: `rev v1` or `rev-1`
|
||||
- Provider ID tags: `(igdb-xxxx)`, `(ssfr-xxxx)`, `(moby-xxxx)`, `(ra-xxxx)` — forces specific provider match
|
||||
|
||||
### Best Practices for Dankserver
|
||||
|
||||
1. Use No-Intro naming as the standard.
|
||||
2. Use **Igir** to rename and organize ROMs into RomM's required folder structure:
|
||||
```
|
||||
igir copy zip test clean \
|
||||
--dat "No-Intro*.zip" \
|
||||
--input "ROMs" \
|
||||
--output "/opt/romm/roms/{romm}"
|
||||
```
|
||||
3. Keep `[BIOS]` prefix for BIOS files in the `bios/` directory.
|
||||
4. One ROM file per game folder where possible.
|
||||
5. Use archive formats (`.zip`, `.7z`) — RomM supports them natively.
|
||||
|
||||
## 4. Cover Art & Media Assets
|
||||
|
||||
### Media Types
|
||||
|
||||
| Media Type | Description | Best Provider |
|
||||
|-----------|-------------|---------------|
|
||||
| `box2d` | Standard 2D front cover art | IGDB, Screenscraper |
|
||||
| `box3d` | 3D box art | Screenscraper |
|
||||
| `miximage` | Composite (box + screenshot + logo) | Screenscraper |
|
||||
| `screenshot` | In-game screenshot | IGDB, Screenscraper |
|
||||
| `marquee` | Transparent logo/header | Screenscraper |
|
||||
| `bezel` | EmulatorJS-compatible bezel | Screenscraper |
|
||||
| `manual` | PDF instruction manual | Screenscraper |
|
||||
| `video` | Gameplay video/trailer | Screenscraper |
|
||||
|
||||
### Custom Artwork Directories
|
||||
|
||||
RomM supports manual artwork placement via subdirectories within each game folder:
|
||||
|
||||
```
|
||||
/opt/romm/library/roms/{platform}/{game}/
|
||||
game.rom
|
||||
covers/
|
||||
screenshots/
|
||||
3dboxes/
|
||||
manuals/
|
||||
fanart/
|
||||
videos/
|
||||
marquees/
|
||||
bezels/
|
||||
```
|
||||
|
||||
For ES-DE gamelist imports, media maps as:
|
||||
|
||||
| gamelist.xml Tag | Folder Name |
|
||||
|-----------------|-------------|
|
||||
| `<cover>` | `covers/` |
|
||||
| `<backcover>` | `backcovers/` |
|
||||
| `<box3d>` | `3dboxes/` |
|
||||
| `<marquee>` | `marquees/` |
|
||||
| `<screenshot>` | `screenshots/` |
|
||||
| `<fanart>` | `fanart/` |
|
||||
| `<manual>` | `manuals/` |
|
||||
|
||||
### MAME Artwork
|
||||
|
||||
MAME uses cabinet pics, flyers, marquees, control panels, backdrops, overlays. Stored in `{platform}/mame_artwork/{game}/` subdirectories.
|
||||
|
||||
Sources: Mr. Do's Arcade (https://www.mameworld.info/mrdo/), ProgettoSnaps (http://www.progettosnaps.net/artworks/).
|
||||
|
||||
## 5. Emulator Integration
|
||||
|
||||
### EmulatorJS (Built-in)
|
||||
|
||||
Web-based player using RetroArch cores compiled to WebAssembly. Runs in-browser with no local emulator needed. Configured via `emulatorjs.*` settings in `config.yml`. Supports netplay via STUN/TURN servers.
|
||||
|
||||
### Emulator Streaming
|
||||
|
||||
Launch native emulator containers (PCSX2, Dolphin, Xemu) and stream picture/sound/input back to the browser. Configured via `streaming.*` in `config.yml`.
|
||||
|
||||
### BIOS File Requirements
|
||||
|
||||
BIOS files placed in `/opt/romm/library/bios/{platform}/`. Platform subdirectories required. File names should match emulator expectations:
|
||||
|
||||
| Platform | BIOS File |
|
||||
|----------|-----------|
|
||||
| PlayStation | `scph1001.bin`, `scph5501.bin`, `scph5502.bin` |
|
||||
| GB/GBC | `gb_bios.bin`, `gbc_bios.bin` |
|
||||
| SNES | `sfc_bios.bin` |
|
||||
|
||||
BIOS matching done by hash against platform requirements.
|
||||
|
||||
### Folder Structure Options
|
||||
|
||||
**Structure A (recommended):**
|
||||
```
|
||||
/opt/romm/library/
|
||||
roms/
|
||||
└── {platform}/
|
||||
└── {game}.rom
|
||||
bios/
|
||||
└── {platform}/
|
||||
└── {bios_file}.bin
|
||||
```
|
||||
|
||||
**Structure B (fallback):**
|
||||
```
|
||||
/opt/romm/library/
|
||||
└── {platform}/
|
||||
├── roms/
|
||||
│ └── {game}.rom
|
||||
└── bios/
|
||||
└── {bios_file}.bin
|
||||
```
|
||||
|
||||
Dankserver mounts `/opt/romm/library:/romm/library` — RomM auto-detects structure.
|
||||
|
||||
## 6. Scraping Workflow
|
||||
|
||||
### Recommended Pipeline
|
||||
|
||||
1. Organize ROMs → No-Intro naming + RomM folder structure.
|
||||
2. Create `config.yml` → Set provider credentials, priority lists, region prefs.
|
||||
3. First scan → Scan ALL → verify everything is matched.
|
||||
4. Review unmatched → Manually match or add provider tags to filenames.
|
||||
5. Fetch artwork → Re-scan with `scan.media` extended to desired types.
|
||||
6. Sync RetroAchievements → Set user RA username → trigger sync.
|
||||
7. Export gamelist → Enable `scan.gamelist.export` for ES-DE/Batocera.
|
||||
8. Maintain → Enable scheduled rescans for new DAT updates.
|
||||
|
||||
### Scan Types
|
||||
|
||||
| Scan Type | What It Does |
|
||||
|-----------|-------------|
|
||||
| ALL | Full scan — hash, match metadata, fetch artwork |
|
||||
| UNMATCHED | Only matches unmatched ROMs (faster) |
|
||||
| METADATA | Updates metadata only |
|
||||
| ARTWORK | Re-fetches artwork only |
|
||||
| RETROACHIEVEMENTS | Syncs achievement data |
|
||||
|
||||
### Region Handling
|
||||
|
||||
```yaml
|
||||
scan:
|
||||
priority:
|
||||
region:
|
||||
- "us"
|
||||
- "eu"
|
||||
- "jp"
|
||||
region_mode: prefer_rom_tags # or prefer_config
|
||||
```
|
||||
|
||||
- `prefer_rom_tags` (default): ROM filename region tags take precedence.
|
||||
- `prefer_config`: Config regions come first.
|
||||
- **NTSC-U** (USA): 60 Hz, typically fastest, most translations.
|
||||
- **PAL** (Europe): 50 Hz, sometimes slower, may have bonus content.
|
||||
- **NTSC-J** (Japan): 60 Hz, Japanese only unless region-patched.
|
||||
|
||||
## 7. Collection Management Tools
|
||||
|
||||
| Tool | Primary Role | Platform |
|
||||
|------|-------------|----------|
|
||||
| RomVault 3.7.6 | DAT-file-based verification and reorganization | Windows |
|
||||
| clrMAMEPro | ROM set validation, deduplication, 1G1R | Windows |
|
||||
| DATBios | BIOS file organization using DAT files | Windows |
|
||||
| Batocera/Lakka integration | ES-DE gamelist import from RomM | Linux/ARM |
|
||||
|
||||
## 8. Environment Variables (Dankserver)
|
||||
|
||||
```bash
|
||||
# Auth
|
||||
ROMM_USERNAME=admin
|
||||
ROMM_PASSWORD=<strong password>
|
||||
ROMM_AUTH_SECRET_KEY=$(openssl rand -hex 32)
|
||||
|
||||
# Metadata Providers
|
||||
IGDB_CLIENT_ID=<your-client-id>
|
||||
IGDB_CLIENT_SECRET=<your-client-secret>
|
||||
SCREENSCRAPER_USER=<screenscraper-username>
|
||||
SCREENSCRAPER_PASSWORD=<screenscraper-password>
|
||||
MOBYGAMES_API_KEY=<mobygames-key>
|
||||
RETROACHIEVEMENTS_API_KEY=<retroachievements-api-key>
|
||||
STEAMGRIDDB_API_KEY=<steamgriddb-key>
|
||||
|
||||
# Feature Flags
|
||||
HASHEOUS_API_ENABLED=true
|
||||
PLAYMATCH_API_ENABLED=true
|
||||
|
||||
# Scanning
|
||||
ENABLE_RESCAN_ON_FILESYSTEM_CHANGE=true
|
||||
ENABLE_SCHEDULED_RESCAN=true
|
||||
REFRESH_RETROACHIEVEMENTS_CACHE_DAYS=7
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [[RomM]] — main service documentation
|
||||
- [[Dankserver Stack]]
|
||||
- [[ARR-Stack]] — Readarr for ebook auto-downloads alongside ROM management
|
||||
@@ -0,0 +1,194 @@
|
||||
# RomM
|
||||
|
||||
RomM (ROM Manager) is a self-hosted ROM library manager and player. It scans a ROM collection, enriches it with metadata and cover art from multiple providers, and serves it through a clean web interface. RomM also supports browser-based emulation via EmulatorJS and native emulator streaming.
|
||||
|
||||
**Host:** Dankserver — **Hostname:** `romm.dankserver.local` — **Docs:** https://docs.romm.app
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Latest version | 5.2.0 (released 2026-07-29) |
|
||||
| Image | `ghcr.io/rommapp/romm:latest` |
|
||||
| Port | 8085 (host) → 8080 (container) |
|
||||
| Database | MariaDB (default); PostgreSQL optional |
|
||||
| Cache / Queue | Redis or Valkey (required for task queue) |
|
||||
| Frontend | React SPA |
|
||||
| Backend | FastAPI / Node |
|
||||
| Metadata providers | IGDB, Screenscraper, MobyGames, Hasheous, Playmatch, LaunchBox, Flashpoint, HowLongToBeat, SteamGridDB, RetroAchievements, TGDB, ES-DE gamelist |
|
||||
| Emulator player | EmulatorJS (WebAssembly RetroArch cores) |
|
||||
| Emulator streaming | Native containers (PCSX2, Dolphin, Xemu) → browser |
|
||||
|
||||
## Architecture
|
||||
|
||||
RomM 5.2.0 requires three supporting services. The compose stack includes `romm`, `mariadb`, and `redis` (or `valkey`).
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
U[User] --> C[Caddy :80/:443]
|
||||
C --> R[RomM :8085]
|
||||
R --> DB[(MariaDB)]
|
||||
R --> Cache[(Redis)]
|
||||
R --> Lib[/romm/library]
|
||||
R --> Res[/romm/resources]
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Required | Description |
|
||||
|----------|----------|-------------|
|
||||
| `ROMM_USERNAME` | Yes | Initial admin username |
|
||||
| `ROMM_PASSWORD` | Yes | Initial admin password (set on first boot) |
|
||||
| `ROMM_AUTH_SECRET_KEY` | Yes | JWT signing key. Generate with `openssl rand -hex 32`. Replaces the old `ROMM_SECRET_KEY`. |
|
||||
| `DB_HOST` | Yes | MariaDB hostname (e.g. `mariadb`) |
|
||||
| `DB_NAME` | Yes | Database name (e.g. `romm`) |
|
||||
| `DB_USER` | Yes | Database user |
|
||||
| `DB_PASSWD` | Yes | Database password |
|
||||
| `REDIS_HOST` | Yes | Redis hostname (e.g. `redis`) |
|
||||
| `DISABLE_SETUP_WIZARD` | No | Set to `true` to skip the first-run wizard |
|
||||
| `OIDC_ENABLED` | No | Enable OpenID Connect authentication |
|
||||
| `OIDC_CLIENT_ID` | No | OIDC client ID |
|
||||
| `OIDC_CLIENT_SECRET` | No | OIDC client secret |
|
||||
| `OIDC_REDIRECT_URI` | No | OIDC redirect URI |
|
||||
| `STREAMING_BROKER_SECRET` | No | Secret for native emulator streaming |
|
||||
| `STREAMING_SAVE_TIMEOUT` | No | Timeout for save states during streaming |
|
||||
| `KIOSK_MODE` | No | Enable kiosk mode |
|
||||
| `ROMM_SESSION_SECURE_COOKIE` | No | Secure cookie flag for sessions |
|
||||
|
||||
### Metadata Provider Credentials
|
||||
|
||||
These are set via environment variables or in `config.yml`:
|
||||
|
||||
| Variable | Provider | Notes |
|
||||
|----------|----------|-------|
|
||||
| `IGDB_CLIENT_ID` / `IGDB_CLIENT_SECRET` | IGDB (Twitch OAuth2) | Create app at Twitch Developer Portal. Free tier: ~10 req/sec. |
|
||||
| `SCREENSCRAPER_USER` / `SCREENSCRAPER_PASSWORD` | Screenscraper | Account credentials (not API key). Free, community-driven. |
|
||||
| `MOBYGAMES_API_KEY` | MobyGames | Paid hobbyist API. Note: RomM recommends Screenscraper as free alternative. |
|
||||
| `RETROACHIEVEMENTS_API_KEY` | RetroAchievements | Hash-based achievements via bundled RAHasher binary. |
|
||||
| `STEAMGRIDDB_API_KEY` | SteamGridDB | Custom cover art via the search cover button. |
|
||||
| `HASHEOUS_API_ENABLED` | Hasheous | Set to `true`. Free, hash-based, proxies IGDB data. Fast scans. |
|
||||
| `PLAYMATCH_API_ENABLED` | Playmatch | Set to `true`. Multi-provider hash matching (IGDB, SS, SteamGridDB, RA). |
|
||||
|
||||
### Config File
|
||||
|
||||
RomM uses `config.yml` for scan settings, available through the admin UI (Scan Settings). Place the file at `/opt/romm/config/config.yml` on the host.
|
||||
|
||||
Key scan config options:
|
||||
|
||||
```yaml
|
||||
scan:
|
||||
priority:
|
||||
metadata:
|
||||
- igdb
|
||||
- ss # Screenscraper
|
||||
artwork:
|
||||
- igdb
|
||||
- ss
|
||||
region:
|
||||
- us
|
||||
- eu
|
||||
- jp
|
||||
region_mode: prefer_rom_tags # or prefer_config
|
||||
media:
|
||||
- box2d
|
||||
- screenshot
|
||||
- manual
|
||||
gamelist:
|
||||
export: true # Export ES-DE compatible gamelist.xml
|
||||
```
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path (convention) |
|
||||
|----------------|------------------------|
|
||||
| `/romm/library` | `/opt/romm/library` |
|
||||
| `/romm/resources` | `/opt/romm/resources` |
|
||||
| `/romm/config` | `/opt/romm/config` |
|
||||
|
||||
The library volume contains ROM files and subdirectories per platform. The resources volume stores cached artwork, metadata, and the SQLite/SQLite-backed asset cache. The config volume holds `config.yml` and other configuration.
|
||||
|
||||
## Docker Compose
|
||||
|
||||
```yaml
|
||||
services:
|
||||
romm:
|
||||
image: ghcr.io/rommapp/romm:latest
|
||||
container_name: romm
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
ROMM_USERNAME: "${ROMM_USERNAME}"
|
||||
ROMM_PASSWORD: "${ROMM_PASSWORD}"
|
||||
ROMM_AUTH_SECRET_KEY: "${ROMM_AUTH_SECRET_KEY}"
|
||||
DB_HOST: mariadb
|
||||
DB_NAME: romm
|
||||
DB_USER: "${ROMM_DB_USER}"
|
||||
DB_PASSWD: "${ROMM_DB_PASSWD}"
|
||||
REDIS_HOST: redis
|
||||
ports:
|
||||
- "8085:8080"
|
||||
volumes:
|
||||
- /opt/romm/library:/romm/library
|
||||
- /opt/romm/resources:/romm/resources
|
||||
- /opt/romm/config:/romm/config
|
||||
depends_on:
|
||||
mariadb:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
|
||||
mariadb:
|
||||
image: docker.io/mariadb:11
|
||||
container_name: romm-mariadb
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
MARIADB_ROOT_PASSWORD: "${MARIADB_ROOT_PASSWORD}"
|
||||
MARIADB_DATABASE: romm
|
||||
MARIADB_USER: "${ROMM_DB_USER}"
|
||||
MARIADB_PASSWORD: "${ROMM_DB_PASSWD}"
|
||||
volumes:
|
||||
- /opt/romm/mariadb:/var/lib/mysql
|
||||
healthcheck:
|
||||
test: ["CMD", "mariadb-admin", "ping", "-h", "localhost"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
|
||||
redis:
|
||||
image: docker.io/redis:7-alpine
|
||||
container_name: romm-redis
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- /opt/romm/redis:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
```
|
||||
|
||||
## Healthcheck
|
||||
|
||||
```yaml
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
- **First login overwrites env credentials.** The `ROMM_PASSWORD` env var sets the initial password. After first login, the admin password is controlled in the UI. The env var only applies on first boot.
|
||||
- **MariaDB and Redis are required.** RomM 5.2.0 no longer supports SQLite-only deployments. Both MariaDB and Redis must be running.
|
||||
- **Reverse proxy cache stale index.html.** Purge Caddy proxy cache after RomM version bumps to avoid serving a stale frontend after an upgrade.
|
||||
- **RAHasher build time.** Enabling RetroAchievements hashing increases the first image build. Skip unless you need achievement data.
|
||||
- **MobyGames is paid.** The hobbyist API has strict rate limits. Screenscraper is the recommended free alternative.
|
||||
- **OIDC requires additional services.** Authentik, Keycloak, or another OIDC provider must be running before enabling `OIDC_ENABLED`.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankserver Stack]]
|
||||
- [[RomM-Scraping]] — metadata providers, cover art, naming conventions, emulator integration
|
||||
- [[Calibre-Web]] — sibling library web UI
|
||||
@@ -0,0 +1,53 @@
|
||||
# Lidarr
|
||||
|
||||
Lidarr manages music albums in the *arr stack on Dankserver. It automatically searches, downloads, and organizes albums from indexers into the Jellyfin music library.
|
||||
|
||||
**Host:** Dankserver — **Hostname:** `lidarr.dankserver.local` — **Docs:** https://wiki.servarr.com/lidarr
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `lscr.io/linuxserver/lidarr:latest` |
|
||||
| Port | 8686 (host) → 8686 (container) |
|
||||
| Min RAM | 256 MB |
|
||||
| Metadata | MusicBrainz |
|
||||
|
||||
## Docker Compose
|
||||
|
||||
```yaml
|
||||
lidarr:
|
||||
image: lscr.io/linuxserver/lidarr:latest
|
||||
container_name: lidarr
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- PUID=1000
|
||||
- PGID=1000
|
||||
- TZ=America/New_York
|
||||
- UMASK_SET=022
|
||||
volumes:
|
||||
- /opt/lidarr/config:/config
|
||||
- /mnt/media/music:/music
|
||||
- /mnt/media/downloads:/downloads
|
||||
ports:
|
||||
- "8686:8686"
|
||||
networks: [appnet]
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
1. Add Prowlarr as the indexer source (prefer torrent-based indexers for FLAC).
|
||||
2. Add qBittorrent as the download client with the `music` category.
|
||||
3. Set quality profile (Lossless → High Bitrate → Standard).
|
||||
4. Set root folder to `/music`.
|
||||
5. Add artists and verify downloads.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Album variants: Lidarr supports multiple releases of the same album (vinyl vs CD). Use "Preferred" and "Cutoff" quality profiles.
|
||||
- Music videos can be optionally downloaded separately from albums.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[ARR-Stack]] — full *arr stack documentation
|
||||
- [[Jellyfin]] — media server that consumes Lidarr downloads
|
||||
@@ -0,0 +1,47 @@
|
||||
# MariaDB (RomM Backend)
|
||||
|
||||
MariaDB 11 provides the database backend for RomM on Dankserver. RomM 5.2.0 requires MariaDB (or PostgreSQL) for its data store.
|
||||
|
||||
**Host:** Dankserver — **Docs:** https://mariadb.com/kb/en/docker-mariadb-information/
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `docker.io/mariadb:11` |
|
||||
| Port | 3306 (internal, within `appnet`) |
|
||||
| Database name | `romm` |
|
||||
| Data volume | `/opt/romm/mariadb` |
|
||||
|
||||
## Docker Compose
|
||||
|
||||
```yaml
|
||||
mariadb:
|
||||
image: docker.io/mariadb:11
|
||||
container_name: romm-mariadb
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
MARIADB_ROOT_PASSWORD: "${MARIADB_ROOT_PASSWORD}"
|
||||
MARIADB_DATABASE: romm
|
||||
MARIADB_USER: "${ROMM_DB_USER}"
|
||||
MARIADB_PASSWORD: "${ROMM_DB_PASSWD}"
|
||||
volumes:
|
||||
- /opt/romm/mariadb:/var/lib/mysql
|
||||
healthcheck:
|
||||
test: ["CMD", "mariadb-admin", "ping", "-h", "localhost"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
```
|
||||
|
||||
## Backup
|
||||
|
||||
Back up the volume directory periodically:
|
||||
|
||||
```bash
|
||||
sudo rsync -a /opt/romm/mariadb /mnt/backup/romm-mariadb/
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [[RomM]] — service that uses this database
|
||||
@@ -0,0 +1,55 @@
|
||||
# Prowlarr
|
||||
|
||||
Prowlarr is the indexer management layer for the *arr stack. It aggregates indexers (torrent and Usenet) and exposes a unified API that Sonarr, Radarr, Lidarr, and Readarr consume.
|
||||
|
||||
**Host:** Dankserver — **Hostname:** `prowlarr.dankserver.local` — **Docs:** https://wiki.servarr.com/prowlarr
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `lscr.io/linuxserver/prowlarr:latest` |
|
||||
| Port | 9696 (host) → 9696 (container) |
|
||||
| Min RAM | 256 MB |
|
||||
| Auth | API key (Settings → General) |
|
||||
|
||||
## Configuration
|
||||
|
||||
1. Add indexers: Settings → Indexers → Add.
|
||||
2. Test each indexer with the **Test** button.
|
||||
3. Assign tags (e.g., `movies`, `tv`, `music`) to indexers for routing.
|
||||
4. Add download client: Settings → Download Clients → Add qBittorrent.
|
||||
|
||||
## Docker Compose
|
||||
|
||||
```yaml
|
||||
prowlarr:
|
||||
image: lscr.io/linuxserver/prowlarr:latest
|
||||
container_name: prowlarr
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- PUID=1000
|
||||
- PGID=1000
|
||||
- TZ=America/New_York
|
||||
- UMASK_SET=022
|
||||
volumes:
|
||||
- /opt/prowlarr/config:/config
|
||||
- /mnt/media/downloads:/downloads:ro
|
||||
ports:
|
||||
- "9696:9696"
|
||||
networks: [appnet]
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Prowlarr database grows with indexer data. Set up periodic vacuum/backup.
|
||||
- Indexer sync has a 15–60 minute delay. Search results may lag.
|
||||
- Rotate API key: update it in every arr app.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[ARR-Stack]] — full *arr stack documentation
|
||||
- [[radarr]]
|
||||
- [[sonarr]]
|
||||
- [[lidarr]]
|
||||
- [[readarr]]
|
||||
@@ -0,0 +1,52 @@
|
||||
# qBittorrent
|
||||
|
||||
qBittorrent is the torrent download client for the *arr stack on Dankserver. It receives download requests from Sonarr, Radarr, Lidarr, and Readarr via category tagging.
|
||||
|
||||
**Host:** Dankserver — **Hostname:** `qbittorrent.dankserver.local` — **Docs:** https://www.qbittorrent.org
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `lscr.io/linuxserver/qbittorrent:latest` |
|
||||
| Port | 8080 (host) → 8080 (container) |
|
||||
| Min RAM | 512 MB |
|
||||
| Auth | Web UI username/password; LAN mode available |
|
||||
|
||||
## Configuration
|
||||
|
||||
1. Change the default Web UI password (Settings → Web UI).
|
||||
2. Enable "No auth mode for LAN" for arr app connectivity (Settings → Web UI → uncheck "HTTP authentication required for clients").
|
||||
3. Create download categories: `movies`, `tv`, `music`, `books`.
|
||||
4. Set max active downloads: Settings → Downloads (recommended: 5 global, 3 per category).
|
||||
|
||||
## Docker Compose
|
||||
|
||||
```yaml
|
||||
qbittorrent:
|
||||
image: lscr.io/linuxserver/qbittorrent:latest
|
||||
container_name: qbittorrent
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- PUID=1000
|
||||
- PGID=1000
|
||||
- TZ=America/New_York
|
||||
- UMASK_SET=022
|
||||
volumes:
|
||||
- /opt/qbittorrent/config:/config
|
||||
- /mnt/media/downloads:/downloads
|
||||
- /mnt/media/downloads/incomplete:/downloads/incomplete
|
||||
ports:
|
||||
- "8080:8080"
|
||||
networks: [appnet]
|
||||
```
|
||||
|
||||
## Gotchas
|
||||
|
||||
- qBittorrent Web UI password changes on first run (check `config/qBittorrent.conf`).
|
||||
- Ensure umask is set to `022` so downloaded files are readable by arr apps.
|
||||
- Monitor bandwidth limits to avoid saturating the network.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[ARR-Stack]] — full *arr stack documentation
|
||||
@@ -0,0 +1,54 @@
|
||||
# Radarr
|
||||
|
||||
Radarr manages movies in the *arr stack on Dankserver. It automatically searches, downloads, and organizes movies from indexers into the Jellyfin media library.
|
||||
|
||||
**Host:** Dankserver — **Hostname:** `radarr.dankserver.local` — **Docs:** https://wiki.servarr.com/radarr
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `lscr.io/linuxserver/radarr:latest` (v6 stable) |
|
||||
| Port | 7878 (host) → 7878 (container) |
|
||||
| Min RAM | 512 MB |
|
||||
| Metadata | TMDB / TheMovieDB API |
|
||||
|
||||
## Docker Compose
|
||||
|
||||
```yaml
|
||||
radarr:
|
||||
image: lscr.io/linuxserver/radarr:latest
|
||||
container_name: radarr
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- PUID=1000
|
||||
- PGID=1000
|
||||
- TZ=America/New_York
|
||||
- UMASK_SET=022
|
||||
volumes:
|
||||
- /opt/radarr/config:/config
|
||||
- /mnt/media/movies:/movies
|
||||
- /mnt/media/downloads:/downloads
|
||||
ports:
|
||||
- "7878:7878"
|
||||
networks: [appnet]
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
1. Add Prowlarr as the indexer source.
|
||||
2. Add qBittorrent as the download client with the `movies` category.
|
||||
3. Set quality profile and cutoff quality.
|
||||
4. Set root folder to `/movies`.
|
||||
5. Add movies and verify downloads.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Radarr v6 is the current stable (v6.3.x as of 2026-08). `:latest` points to the current release.
|
||||
- Movie renaming produces `Title (Year)` format. Jellyfin scanner handles this.
|
||||
- File permissions: qBittorrent downloads as PUID; arr apps must read/move files.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[ARR-Stack]] — full *arr stack documentation
|
||||
- [[Jellyfin]] — media server that consumes Radarr downloads
|
||||
@@ -0,0 +1,57 @@
|
||||
# Readarr
|
||||
|
||||
> **RETIRED (May 2026).** The Servarr Team retired Readarr: its metadata became unusable and the community migration to Open Library stalled. There is no upstream development. Alternatives: the Bookshelf community fork, or the Calibre content server. See [Readarr (Retired)](https://wiki.servarr.com/readarr).
|
||||
|
||||
Readarr manages ebooks and audiobooks in the *arr stack on Dankserver. It automatically searches, downloads, and organizes books into the Calibre library.
|
||||
|
||||
**Host:** Dankserver — **Hostname:** `readarr.dankserver.local` — **Docs:** https://wiki.servarr.com/readarr
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `lscr.io/linuxserver/readarr:latest` (v0.2.x development) |
|
||||
| Port | 8787 (host) → 8787 (container) |
|
||||
| Min RAM | 256 MB |
|
||||
| Metadata | Goodreads / StoryGraph |
|
||||
|
||||
## Docker Compose
|
||||
|
||||
```yaml
|
||||
readarr:
|
||||
image: lscr.io/linuxserver/readarr:latest
|
||||
container_name: readarr
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- PUID=1000
|
||||
- PGID=1000
|
||||
- TZ=America/New_York
|
||||
- UMASK_SET=022
|
||||
volumes:
|
||||
- /opt/readarr/config:/config
|
||||
- /mnt/media/books:/books
|
||||
- /mnt/media/downloads:/downloads
|
||||
ports:
|
||||
- "8787:8787"
|
||||
networks: [appnet]
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
1. Add Prowlarr as the indexer source (torrent indexers with book content).
|
||||
2. Add qBittorrent as the download client with the `books` category.
|
||||
3. Set author profile (format preference: epub, mobi, etc.).
|
||||
4. Set quality profile for audiobooks (320 kbps MP3, 128 kbps, lossless).
|
||||
5. Set root folder to `/books`.
|
||||
6. Add authors and verify downloads.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Readarr is in development (~v0.2.x). Some features may be incomplete.
|
||||
- Calibre integration is folder-based, not API-based. Readarr drops files into `/books` which Calibre-Web serves.
|
||||
- Not all indexers have all formats (epub, mobi, pdf). Goodreads/StoryGraph metadata helps match editions.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[ARR-Stack]] — full *arr stack documentation
|
||||
- [[Calibre-Web]] — ebook web frontend that serves Readarr downloads
|
||||
@@ -0,0 +1,40 @@
|
||||
# Redis (RomM Cache / Queue)
|
||||
|
||||
Redis 7 provides the caching and task queue backend for RomM on Dankserver. RomM 5.2.0 requires Redis (or Valkey) for its task queue.
|
||||
|
||||
**Host:** Dankserver — **Docs:** https://redis.io/docs/
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `docker.io/redis:7-alpine` |
|
||||
| Port | 6379 (internal, within `appnet`) |
|
||||
| Data volume | `/opt/romm/redis` |
|
||||
| Auth | None (local-only, no password needed) |
|
||||
|
||||
## Docker Compose
|
||||
|
||||
```yaml
|
||||
redis:
|
||||
image: docker.io/redis:7-alpine
|
||||
container_name: romm-redis
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- /opt/romm/redis:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
```
|
||||
|
||||
## Backup
|
||||
|
||||
```bash
|
||||
sudo rsync -a /opt/romm/redis /mnt/backup/romm-redis/
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [[RomM]] — service that uses this cache
|
||||
@@ -0,0 +1,55 @@
|
||||
# Sonarr
|
||||
|
||||
Sonarr manages TV shows in the *arr stack on Dankserver. It automatically searches, downloads, and organizes episodes from indexers into the Jellyfin media library.
|
||||
|
||||
**Host:** Dankserver — **Hostname:** `sonarr.dankserver.local` — **Docs:** https://wiki.servarr.com/sonarr
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `lscr.io/linuxserver/sonarr:latest` (v4 stable) |
|
||||
| Port | 8989 (host) → 8989 (container) |
|
||||
| Min RAM | 512 MB |
|
||||
| Metadata | TMDB / TVDB |
|
||||
|
||||
## Docker Compose
|
||||
|
||||
```yaml
|
||||
sonarr:
|
||||
image: lscr.io/linuxserver/sonarr:latest
|
||||
container_name: sonarr
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
- PUID=1000
|
||||
- PGID=1000
|
||||
- TZ=America/New_York
|
||||
- UMASK_SET=022
|
||||
volumes:
|
||||
- /opt/sonarr/config:/config
|
||||
- /mnt/media/tv:/tv
|
||||
- /mnt/media/downloads:/downloads
|
||||
ports:
|
||||
- "8989:8989"
|
||||
networks: [appnet]
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
1. Add Prowlarr as the indexer source.
|
||||
2. Add qBittorrent as the download client with the `tv` category.
|
||||
3. Set quality profile, cutoff, and minimum/maximum episode sizes.
|
||||
4. Set root folder to `/tv`.
|
||||
5. Add series and verify downloads.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Sonarr v4 is the current stable. The v3 branch uses the `:v3` tag.
|
||||
- Anime has different quality profiles and naming conventions than Standard series.
|
||||
- Missing episodes: set "Unmonitor Missing Episodes" to auto-unmonitor old episodes.
|
||||
- Timezone-sensitive scheduler: ensure all containers use the same TZ.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[ARR-Stack]] — full *arr stack documentation
|
||||
- [[Jellyfin]] — media server that consumes Sonarr downloads
|
||||
@@ -0,0 +1,47 @@
|
||||
# Alloy
|
||||
|
||||
Alloy is Grafana's log and telemetry shipper. It replaces Promtail (EOL 2026-03-02). One Alloy instance runs per host and forwards container and journal logs to Loki, and optionally metrics to Prometheus.
|
||||
|
||||
**Hosts:** Dankclaw + Dankserver (one per host) — **Docs:** https://grafana.com/docs/alloy/latest
|
||||
|
||||
## Facts (from research)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `franksdank/alloy:latest` (local build with a static busybox healthcheck) |
|
||||
| Port | none published (push-only) |
|
||||
| Config | River syntax, `config.alloy` |
|
||||
| Successor to | Promtail (EOL 2026-03-02) |
|
||||
| Inputs | journald socket, container log files |
|
||||
| Output | Loki push endpoint |
|
||||
|
||||
## Why Alloy and not Promtail
|
||||
|
||||
Promtail reached end-of-life on 2026-03-02. Grafana Labs directs new installs to Alloy. This stack uses Alloy from day one.
|
||||
|
||||
## Configuration
|
||||
|
||||
River config at `shared/alloy/config.alloy`:
|
||||
|
||||
- Read journald logs from the host's `/run/systemd/journal/socket`.
|
||||
- Read container logs from `/var/lib/containers/...` (podman) or the docker path.
|
||||
- Forward to `http://loki:3100/loki/api/v1/push` with `host` and `job` labels.
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path (convention) |
|
||||
|----------------|------------------------|
|
||||
| `/etc/alloy/config.alloy` | `shared/alloy/config.alloy` (bind mount) |
|
||||
| `/run/systemd/journal/socket` | host journald socket (read-only) |
|
||||
| `/var/log` | host logs (read-only) |
|
||||
|
||||
## Gotchas
|
||||
|
||||
- The journald socket mount is required for journal scraping.
|
||||
- Container log paths differ between docker (`/var/lib/docker/containers`) and podman (`/var/lib/containers/storage/...`); point the config at the right one per host.
|
||||
- One Alloy per host; do not run a single instance that tries to reach both hosts' logs.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Loki]] — the log backend
|
||||
- [[Observability]] — the full stack
|
||||
@@ -0,0 +1,152 @@
|
||||
# Authentik — the homelab's SSO / OIDC provider
|
||||
|
||||
Authentik is the single sign-on (SSO) and OpenID Connect (OIDC) provider for the
|
||||
homelab. It fronts the Caddy vhosts that do not carry their own
|
||||
application-level authentication. Apps that have their own login (Grafana,
|
||||
Gitea, OpenWebUI, Jellyfin, the *arr stack, Pi-hole) are not wrapped in
|
||||
forward_auth — they handle their own session.
|
||||
|
||||
- **Version:** `2026.8.0` (pinned in `dankclaw/compose.yaml`)
|
||||
- **Host:** dankclaw (`192.168.0.143`)
|
||||
- **Name:** `auth.franksdank.cloud`
|
||||
- **Stack:** 3 compose services — `authentik-postgres`, `authentik-server`,
|
||||
`authentik-worker` (see [Pi-hole](Pi-hole.md) for the LAN-DNS pattern)
|
||||
|
||||
## Why Authentik
|
||||
|
||||
The comparison that chose it (against Authelia, Keycloak, Zitadel, Ory) is in
|
||||
`.docaudit/research/auth.md`. In short: full OIDC with single-logout, a web GUI
|
||||
for all configuration, a native Caddy `forward_auth` outpost, and a footprint
|
||||
that fits on a 3-user homelab (~1–1.5 GB idle). Redis was removed in 2025.10,
|
||||
so the stack is 3 services, not 4.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph Browser
|
||||
U[User]
|
||||
end
|
||||
subgraph Caddy [Caddy - auth gate]
|
||||
FA[forward_auth]
|
||||
end
|
||||
subgraph AK [Authentik]
|
||||
SRV[server :9000]
|
||||
WKR[worker]
|
||||
PG[(postgres)]
|
||||
OUT[outpost /auth/caddy]
|
||||
end
|
||||
subgraph App [Target app]
|
||||
A1[SearXNG / Hermes / Prometheus<br/>RomM / Copyparty / homepage]
|
||||
end
|
||||
|
||||
U -->|1. GET /| FA
|
||||
FA -->|2. GET /outpost.../auth/caddy| OUT
|
||||
OUT -->|3. no session: 302 login| U
|
||||
U -->|4. login at auth.franksdank.cloud| SRV
|
||||
SRV --> PG
|
||||
WKR --> PG
|
||||
SRV -->|5. 2xx: session valid| FA
|
||||
FA -->|6. proxy with headers| A1
|
||||
```
|
||||
|
||||
1. The browser hits a protected vhost (e.g. `search.franksdank.cloud`).
|
||||
2. Caddy `forward_auth` clones the request to the Authentik outpost
|
||||
(`/outpost.goauthentik.io/auth/caddy`).
|
||||
3. With no valid session, the outpost returns a `302` to the Authentik login
|
||||
page (`auth.franksdank.cloud`).
|
||||
4. The user logs in at Authentik (email + password, TOTP/WebAuthn MFA, or an
|
||||
upstream provider).
|
||||
5. On the next request the outpost returns `2xx`; Caddy copies the
|
||||
`X-Authentik-*` headers and proxies to the app.
|
||||
|
||||
## The three compose services
|
||||
|
||||
| Service | Role | Image |
|
||||
|---|---|---|
|
||||
| `authentik-postgres` | PostgreSQL 16 (users, flows, sessions, events) | `docker.io/library/postgres:16-alpine` |
|
||||
| `authentik-server` | Core + admin API + the embedded Caddy outpost (port 9000) | `ghcr.io/goauthentik/server:2026.8.0` |
|
||||
| `authentik-worker` | Background tasks (event processing, provider sync, flows) | `ghcr.io/goauthentik/server:2026.8.0` |
|
||||
|
||||
- **Postgres** is the only stateful store. Sessions, caching, and WebSockets
|
||||
moved to PostgreSQL in 2025.10 (Redis removed).
|
||||
- **Server** and **worker** run the same image; the `command` selects the role
|
||||
(`server` / `worker`).
|
||||
- The worker runs as `0:0` (rootless: container root maps to the host user that
|
||||
owns the data dir).
|
||||
- Data persists under `/home/dank/authentik/` (bind mounts, repo policy — no
|
||||
named volumes).
|
||||
|
||||
## Caddy integration
|
||||
|
||||
The Caddyfile wraps the no-auth vhosts in a `route` block. The pattern (one
|
||||
example; the others are identical):
|
||||
|
||||
```caddyfile
|
||||
search.franksdank.cloud {
|
||||
route {
|
||||
# 1. The outpost path is proxied to Authentik (login UI + session).
|
||||
reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
|
||||
# 2. forward_auth checks the session before serving the app.
|
||||
forward_auth authentik-server:9000 {
|
||||
uri /outpost.goauthentik.io/auth/caddy
|
||||
copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name
|
||||
trusted_proxies private_ranges
|
||||
}
|
||||
# 3. The app.
|
||||
reverse_proxy 192.168.0.143:8081
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Vhosts protected by forward_auth: the homepage (`franksdank.cloud`,
|
||||
`www.franksdank.cloud`), SearXNG, Hermes, Prometheus, RomM, Copyparty. The
|
||||
`auth.franksdank.cloud` vhost proxies to the server directly (no forward_auth —
|
||||
it *is* the login).
|
||||
|
||||
## Configuration (secrets)
|
||||
|
||||
Two secrets live in the host `.env` (git-ignored; placeholders in
|
||||
`.env.example`):
|
||||
|
||||
- `AUTHENTIK_PG_PASS` — the PostgreSQL password. Generate: `openssl rand -base64 32`.
|
||||
- `AUTHENTIK_SECRET_KEY` — the server's signing key. Generate: `openssl rand -base64 48`.
|
||||
|
||||
## First-time setup
|
||||
|
||||
1. `./scripts/deploy.sh dankclaw` — pulls, starts the 3 services (postgres
|
||||
health-gated), reloads Caddy.
|
||||
2. Open `https://auth.franksdank.cloud` — the initial setup wizard runs. It
|
||||
creates the superuser (email + password) and generates the DB schema.
|
||||
3. In the admin interface: **Directory → Applications** → add a **Proxy**
|
||||
application (or an **OAuth2/OpenID Connect** provider) for each vhost you
|
||||
protect. The Caddy outpost reads the application's policy.
|
||||
4. For apps with native OIDC (Grafana, Gitea, OpenWebUI): add an **OAuth2/OpenID
|
||||
Connect** provider, create the app with the correct redirect URI, and paste
|
||||
the `client_id` / `client_secret` into the app's SSO settings.
|
||||
|
||||
## Operations
|
||||
|
||||
- **Health:** `https://auth.franksdank.cloud/-/health/` (the server healthcheck
|
||||
probes this).
|
||||
- **Logs:** `podman logs authentik-server` / `podman logs authentik-worker`.
|
||||
- **Backups:** `pg_dump` the `authentik-postgres` database (the single source of
|
||||
truth). The `/home/dank/authentik/` dir holds media + static files.
|
||||
- **Upgrades:** bump the image tag in `dankclaw/compose.yaml`, run
|
||||
`./scripts/deploy.sh dankclaw`. Authentik migrates the schema on startup.
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Symptom | Cause | Fix |
|
||||
|---|---|---|
|
||||
| Every vhost 502s at the outpost | `authentik-server` down | `podman logs authentik-server`; check postgres health |
|
||||
| Login loops (302 → login → 302) | `trusted_proxies` / `X-Forwarded-*` mismatch | Confirm Caddy sets `X-Forwarded-For`; the outpost trusts `private_ranges` |
|
||||
| "Invalid signature" on the outpost | `AUTHENTIK_SECRET_KEY` differs between server and worker | Both services read the same `.env` var; re-deploy |
|
||||
| Postgres "password authentication failed" | `AUTHENTIK_PG_PASS` changed after first init | Reset via `pg_ctl` or re-init the data dir |
|
||||
|
||||
## Cross-references
|
||||
|
||||
- [Caddy](Caddy.md) — the reverse proxy that carries the forward_auth directives
|
||||
- [Pi-hole](Pi-hole.md) — the LAN DNS that resolves `auth.franksdank.cloud`
|
||||
- [Network Architecture](../Architecture/Network%20Architecture.md) — the flat naming scheme
|
||||
- Research: `.docaudit/research/auth.md` — the full provider comparison
|
||||
@@ -0,0 +1,112 @@
|
||||
# Caddy
|
||||
|
||||
Caddy is the single reverse proxy for both hosts. It runs as a boot-persistent quadlet container (`caddy.service`) on Dankclaw (192.168.0.143), terminates TLS with real Let's Encrypt certificates, routes hostnames to containers on both hosts, and protects the homepage and sensitive services with HTTP basic auth.
|
||||
|
||||
**Docs:** https://caddyserver.com/docs
|
||||
|
||||
## Facts (live inventory)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Runtime | Quadlet unit `caddy.service` (`dankclaw/caddy/quadlet/caddy.container`) — boot-persistent, self-healing |
|
||||
| Image | `franksdank/caddy-porkbun:latest` (local build, `dankclaw/caddy/Containerfile`) |
|
||||
| Config | `/home/dank/Containers/caddy/Caddyfile` (bind mount, rendered by deploy) |
|
||||
| Data | `/home/dank/Containers/caddy/data` (certs) and `.../config` (ACME state) |
|
||||
| Ports | 80, 443 on 192.168.0.143 |
|
||||
| TLS | Let's Encrypt, DNS-01 via the Porkbun module (one cert per vhost) |
|
||||
| Runs on | Dankclaw only — Dankserver has no Caddy |
|
||||
|
||||
The Caddyfile in git (`dankclaw/Caddyfile`) carries a `REPLACE_WITH_HASH` placeholder. `scripts/deploy.sh` renders the real basicauth hash from the host `.env` (`CADDY_BASICAUTH_HASH`) into the mounted path before start. See [[Deployment Flow]].
|
||||
|
||||
## Quadlet (boot persistence)
|
||||
|
||||
Caddy left `compose.yaml` on 2026-08-26. A reboot exposed that the bare compose container never comes back: rootless Podman has no boot-time auto-start for bare containers, and `restart: unless-stopped` does not survive the explicit stop that shutdown performs. The quadlet unit is now the only way Caddy starts (see AGENTS.md "Boot Persistence & Self-Healing").
|
||||
|
||||
- **Install path on Dankclaw:** `~/.config/containers/systemd/caddy.container` (unit `caddy.service`)
|
||||
- **Network:** `dankclaw_default` (the compose bridge — the Caddyfile's container-name upstreams must keep resolving; if no compose service has created the network yet at boot, quadlet creates it first and compose reuses it by name later)
|
||||
- **Published ports:** 80/tcp, 443/tcp
|
||||
- **Env file:** `/home/dank/Containers/caddy/env` (`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`) — the deploy writes it from the checkout's `.env`; secrets never enter the repo (placeholders in `dankclaw/caddy/env.example`)
|
||||
- **Restart:** `Restart=always`, `RestartSec=5`; the health check probes the bare-IP catch-all (302) inside the container
|
||||
- **Image policy:** `Pull=never` — the image is built on-host and exists nowhere else; a boot must not stall on a registry
|
||||
- **Enable state:** `systemctl --user is-enabled caddy` reports `generated` — that is the healthy state for a quadlet. The podman generator creates the `default.target.wants` symlink from the unit's `[Install]` section at every user-manager start (linger covers boot); an explicit `systemctl enable` fails on generated units, so the deploy does not run it
|
||||
- **Deploy:** `./scripts/deploy.sh dankclaw --service caddy` — renders the Caddyfile, rebuilds the image if the Containerfile changed, retires the legacy split proxy, writes the env file, removes the old bare container (all state is in host volumes), then installs and starts the unit
|
||||
|
||||
## The porkbun image
|
||||
|
||||
Stock Caddy cannot solve the ACME challenge because the router exposes no inbound port 80/443. The local image adds the `github.com/caddy-dns/porkbun` module, which answers the DNS-01 challenge by creating and deleting a `_acme-challenge` TXT record through the Porkbun API. No inbound port is needed. Rebuild only when Caddy or the module needs an update:
|
||||
|
||||
```
|
||||
podman build -t franksdank/caddy-porkbun:latest -f dankclaw/caddy/Containerfile .
|
||||
```
|
||||
|
||||
The Caddyfile references the credentials through the environment:
|
||||
|
||||
```
|
||||
acme_dns porkbun {
|
||||
api_key {env.PORKBUN_API_KEY}
|
||||
api_secret_key {env.PORKBUN_API_SECRET_KEY}
|
||||
}
|
||||
```
|
||||
|
||||
The deploy writes both variables from the host `.env` into `/home/dank/Containers/caddy/env`, which the quadlet reads via `EnvironmentFile`. Keep the real keys in `.env` only; `.env.example` (and `dankclaw/caddy/env.example`) hold placeholders. The Porkbun account holds only `franksdank.cloud`.
|
||||
|
||||
## Vhost map
|
||||
|
||||
Every vhost uses the flat `franksdank.cloud` convention. The full list lives in `dankclaw/Caddyfile`. Routing by host:
|
||||
|
||||
| Upstream | Reach |
|
||||
|----------|-------|
|
||||
| Dankclaw services (compose network) | by container name, e.g. `comfyui:8188` |
|
||||
| Dankclaw quadlet services | by published LAN port, e.g. `192.168.0.143:3000` (Gitea) |
|
||||
| Dankserver services | by published LAN port, e.g. `192.168.0.235:8085` (RomM) |
|
||||
|
||||
The bare-IP catch-all `:80` 302-redirects to `https://dankclaw.franksdank.cloud/`. The legacy name `dankserver.franksdank.cloud` 302-redirects there too. `plex.franksdank.casa` is the only plain-HTTP vhost; it cannot get a DNS-01 cert because `franksdank.casa` is not in the Porkbun account. The canonical Plex name is `plex.franksdank.cloud`. Both Plex vhosts dial PMS **over TLS upstream** (`transport http { tls_insecure_skip_verify }` — in this Caddy build the flag is a bare token inside the transport block, not a `tls {}` sub-block): PMS 1.43.x silently closes plain-HTTP connections from non-loopback sources (diagnosed 2026-08-26 — it accepts, reads the request, and FINs without responding), which surfaced here as an empty-body 502 (blank page in browsers). `insecure_skip_verify` applies only to this pinned-LAN hop (PMS serves a self-signed cert); client-facing Let's Encrypt TLS is unchanged. The Plex web UI is at `/web/`; `/` returns the XML `MediaContainer` index.
|
||||
|
||||
## Basic auth
|
||||
|
||||
The Caddyfile in git carries a **bare** placeholder; deploy.sh substitutes the **full** hash from the host `.env` (`CADDY_BASICAUTH_HASH`):
|
||||
|
||||
```
|
||||
chat.franksdank.cloud {
|
||||
basicauth {
|
||||
dank REPLACE_WITH_HASH
|
||||
}
|
||||
reverse_proxy 192.168.0.143:3005
|
||||
}
|
||||
```
|
||||
|
||||
After rendering, the line reads `dank $2a$14$xxxx…` (one complete hash). Do not put a `$2a$14$` prefix in the template: it would render a double-prefixed, invalid hash (2026-08-22 incident — every basicauth vhost returned 401 for every password). `tests/deployment_flow.bats` pins the render contract.
|
||||
|
||||
The homepage (and SearXNG, Hermes, Prometheus, RomM, Copyparty) use `forward_auth` (Authentik SSO) rather than basicauth; see [[Homepage]] and [[Authentik]]. The basicauth hash above is still rendered for the vhosts that keep it (Grafana, Pi-hole).
|
||||
|
||||
## Operations
|
||||
|
||||
Validate before a reload:
|
||||
|
||||
```
|
||||
caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile
|
||||
```
|
||||
|
||||
The admin endpoint stays on (loopback-only, unpublished port), so a Caddyfile change hot-reloads within seconds via the file watcher. deploy.sh also issues an explicit reload after rendering. Manual reload from the host:
|
||||
|
||||
```
|
||||
podman exec caddy caddy reload --config /etc/caddy/Caddyfile --adapter caddyfile
|
||||
```
|
||||
|
||||
Certificates and ACME state live in the caddy data volume. Back it up with the stack. Each vhost owns one certificate; renewals are automatic and also use DNS-01.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- A bare compose container is **not** boot-persistent: after the 2026-08-26 reboot Caddy stayed down for hours because `restart: unless-stopped` does not survive shutdown's explicit stop. It now runs only from its quadlet unit — do not re-add it to `compose.yaml`.
|
||||
- `admin off` disables the config file watcher: rendered changes then sit on disk while Caddy serves its stale in-memory config until the container is recreated, and nothing logs an error (2026-08-22: a broken basicauth hash survived several deploys this way). `tests/caddyfile.bats` forbids `admin off`.
|
||||
- The basicauth template must stay a bare `REPLACE_WITH_HASH` token (no `$2a$14$` prefix) — the rendered value is the complete hash. A double prefix silently rejects every password with 401; no Caddy error is logged.
|
||||
- A new vhost needs three things: a Pi-hole `addn-hosts` line, a Porkbun A record, and the Caddyfile block. See [[LAN-DNS]].
|
||||
- The G36 has no NAT loopback, so LAN clients must resolve the name to the LAN IP. That is the Pi-hole role in [[LAN-DNS]].
|
||||
- Caddy serves HTTPS on 443 and HTTP on 80 (redirects). The router needs no 80/443 port forward for LAN use.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[LAN-DNS]] — how names resolve on the LAN and WAN
|
||||
- [[Network Architecture]] — topology, access policy
|
||||
- [[Homepage]]
|
||||
- [[Dankclaw Stack]]
|
||||
@@ -0,0 +1,46 @@
|
||||
# Grafana
|
||||
|
||||
Grafana visualizes the metrics that Prometheus collects from every service. It runs on Dankclaw and is reachable at `grafana.franksdank.cloud` (via Caddy, with basicauth).
|
||||
|
||||
**Host:** Dankclaw — **Hostname:** `grafana.franksdank.cloud` — **Port:** 3002 (internal) — **Docs:** https://grafana.com/docs
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `docker.io/grafana/grafana:latest` |
|
||||
| Port | 3002 (host) → 3000 (container) |
|
||||
| Data volume | dashboards, users, datasource config |
|
||||
| Datasource | Prometheus (on the same host) |
|
||||
| Provisioning | dashboards in `/etc/grafana/provisioning/dashboards` |
|
||||
|
||||
## Configuration
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
GF_SECURITY_ADMIN_USER: "${GRAFANA_ADMIN_USER}"
|
||||
GF_SECURITY_ADMIN_PASSWORD: "${GRAFANA_ADMIN_PASSWORD}"
|
||||
GF_SERVER_ROOT_URL: "https://grafana.franksdank.cloud"
|
||||
volumes:
|
||||
- /opt/grafana/data:/var/lib/grafana
|
||||
- /opt/grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
```
|
||||
|
||||
## Datasource
|
||||
|
||||
Point Grafana at the Prometheus service by name (`http://prometheus:9090`) on the shared podman network.
|
||||
|
||||
## Healthcheck
|
||||
|
||||
Grafana responds on `/api/health` with 200 and `"database":"ok"`.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Change the default admin credentials at first boot (env or UI).
|
||||
- Dashboards provisioned as YAML survive container recreation; UI-created ones live in the data volume — back it up.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
- [[Prometheus]] — the datasource
|
||||
- [[Observability]] — the whole stack
|
||||
@@ -0,0 +1,96 @@
|
||||
# Homepage Design Grilling
|
||||
|
||||
The decision log for the homepage design. All questions are settled (2026-08-23). This page records the final decisions and the places where the operator's answer changed the initial recommendation.
|
||||
|
||||
## Final decisions
|
||||
|
||||
| # | Decision | Value |
|
||||
|---|----------|-------|
|
||||
| Q1 | How the page learns up/down | Server-side checker writes `status.json`; the page reads only that file. (Documented future intent: client-side probing.) |
|
||||
| Q2 | How the checker runs | systemd timer, rewrites `status.json` every 3 minutes. |
|
||||
| Q3 | Page model | Single site on Dankclaw. Root = user services (no host labels). `/admin/` = admin services, second operator-only password. |
|
||||
| Q5 | Service split | User: SearXNG, ComfyUI, OpenWebUI, Gitea, Calibre-Web, RomM, Copyparty, Plex, Jellyfin. Admin: Hermes, Grafana, Prometheus, Loki, Caddy, dnsmasq, Gluetun, LLM fleet, low-level backends (status-only). The *arr stack is excluded. |
|
||||
| Q6 | Tourist `/how/` page | **Defer** — it is extra. |
|
||||
| Q7 | `status.json` | One file; each service carries a `category` field; each page filters. |
|
||||
| Q8 | `services.json` | One file with a `category` field. |
|
||||
| Q9 | Checker language | python3. |
|
||||
| Q10 | Probe method | **Per service** — `tcp` or `http`, set per entry. |
|
||||
| Q11 | Probe timeout | **5 s** per service. |
|
||||
| Q12 | LLM loading | Three states; a listening port is Up (health 503 = Up). |
|
||||
| Q13 | Checker location | `scripts/homepage-status.py` + `.service` + `.timer`, tracked in git. |
|
||||
| Q14 | Reach backends | `probe` = full `host:port` (`127.0.0.1:PORT` or `192.168.0.235:PORT`). |
|
||||
| Q15 | WIP | static `"wip": true`; the checker skips the probe. |
|
||||
| Q16 | Staleness | `generated_at` in `status.json`; the page warns when it is older than 10 min. |
|
||||
| Q17 | Writes | atomic (write a temp file, then rename). |
|
||||
| Q18 | Admin password | `CADDY_ADMIN_HASH` in the host `.env` (git-ignored), operator-only. |
|
||||
| Q19 | Per-path basicauth | `@admin path /admin/*` with a second `basicauth` block. |
|
||||
| Q20 | Descriptions | hand-written one-liners in `services.json`. |
|
||||
| Q21 | Open link | the Caddy site URL; status-only backends set `showLink: false`. |
|
||||
| Q22 | Docs link | **a per-service `/docs/` page. Do NOT link the existing reference docs (they may contain secrets).** |
|
||||
| Q23 | Layout | **TUI-style cards/panels** (a grid of panels), not a flat list. |
|
||||
| Q24 | Summary line | yes — a placeholder for now; refine later. |
|
||||
| Q25 | Terminus | bundle the font (woff2) + a monospace fallback. |
|
||||
| Q26 | Palette | as proposed (green `#33ff66`, red `#ff5555`, amber `#ffcc33`, cyan `#33ccff`, bg `#050805`). |
|
||||
| Q27 | Styling | **toggleable: default = full theme; a toggle strips to bare unstyled HTML (motherfuckingwebsite.com style).** |
|
||||
| Q28 | Cursor | **do not alter the user's cursor.** |
|
||||
| Q29 | Docs content | how a regular user uses the service; **no maintenance details.** |
|
||||
| Q30 | Docs path | **`/docs/<svc>/`** (not `/usage/`). |
|
||||
| Q31 | Docs auth | **the same auth as the homepage section** (user docs = shared password, admin docs = admin password). |
|
||||
| Q32 | Docs workstream | a separate effort; built as each service goes active. |
|
||||
| Q33 | Deploy | **git-driven; master always reflects the deployed state.** |
|
||||
| Q34 | `/srv/homepage/` | `deploy.sh` creates it and copies the assets. |
|
||||
| Q35 | Timer | enabled at boot. |
|
||||
| Q36 | Checker tests | **full TDD.** |
|
||||
| Q37 | Page/config tests | **full TDD.** |
|
||||
| Q38 | Security test | **full TDD.** |
|
||||
| Q39 | Backend move | a config edit (`probe` host:port). |
|
||||
| Q40 | LAN timeout | **5 s.** |
|
||||
| Q41 | Auto-refresh | **no** — the page reads `status.json` once on load. |
|
||||
| Q42 | Mobile | responsive. |
|
||||
| Q43 | Footer timestamp | **last master deployment datetime** (not last-check). |
|
||||
|
||||
Styling (settled): terminal TUI, Terminus (monospace fallback), green-on-black, 1995 phosphor palette. Default = full theme; a toggle strips to bare unstyled HTML. Do not alter the user's cursor.
|
||||
|
||||
## Where the operator changed the recommendation
|
||||
|
||||
These are the deltas from the initial recommendations. Everything else was accepted as recommended.
|
||||
|
||||
- **Q10** — the probe method is **per service** (`tcp` or `http`), not TCP-only.
|
||||
- **Q11 / Q40** — the probe timeout is **5 s** (not 3 s).
|
||||
- **Q22** — the docs link points to a **per-service `/docs/` page**, never the existing reference docs (they may contain secrets).
|
||||
- **Q23** — the layout is **TUI-style cards/panels**, not a flat list.
|
||||
- **Q27** — styling is **toggleable**: the full theme by default, with a toggle to bare unstyled HTML.
|
||||
- **Q28** — **do not alter the user's cursor** (no fake blinking cursor).
|
||||
- **Q29** — the docs are **user-facing only**; no maintenance details.
|
||||
- **Q30** — the docs path is **`/docs/`** (not `/usage/`).
|
||||
- **Q31** — the docs use **the same auth as the homepage section**.
|
||||
- **Q33** — **git-driven; master always reflects the deployed state.**
|
||||
- **Q36–Q38** — **full TDD.**
|
||||
- **Q41** — **no client-side auto-refresh.**
|
||||
- **Q43** — the footer shows the **last master deployment datetime** (not last-check).
|
||||
|
||||
## Build status — complete (2026-08-23)
|
||||
|
||||
The build is done, slice by slice, full TDD (each slice red → green, committed
|
||||
and pushed). 116 tests pass. The render logic is verified with a Node DOM shim.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[write the checker tests] --> B[write the checker]
|
||||
B --> C[write services.json]
|
||||
C --> D[write the pages + theme + toggle]
|
||||
D --> E[extend deploy.sh + Caddyfile]
|
||||
E --> F[tests green]
|
||||
F --> G[deploy to Dankclaw]
|
||||
G --> H[verify in browser]
|
||||
```
|
||||
|
||||
Slices: 1 the checker core (tcp/wip/atomic/generated_at), 2 the http probe, 3
|
||||
the real `services.json` + schema, 4 the pages + theme + toggle, 5 the Caddyfile
|
||||
auth split, 6 the git-driven deploy (both hashes, `deployed.js`, the timer).
|
||||
|
||||
**Remaining to go live:** the operator adds `CADDY_ADMIN_HASH` (the
|
||||
operator-only password, `caddy hash-password`) to the target `.env`, then runs
|
||||
`./scripts/deploy.sh dankclaw`. Until then the admin hash stays a placeholder.
|
||||
|
||||
The per-service docs pages (Q22/Q29/Q30/Q31) are a separate workstream, built as each service goes active.
|
||||
@@ -0,0 +1,203 @@
|
||||
# Homepage
|
||||
|
||||
A password-protected web site on Dankclaw lists the services that are up, with a short description of each and a link to both the service and its usage documentation.
|
||||
|
||||
**Served by:** Caddy, `file_server` + `forward_auth` (Authentik SSO) — **Docs:** this page
|
||||
|
||||
## Status — 2026-08-23: deployed to production
|
||||
|
||||
The homepage is live on Dankclaw. `./scripts/deploy.sh dankclaw` copies the static assets to `/srv/homepage/`, installs the status checker (systemd timer, every 3 min, writes `status.json`), writes `deployed.js` (the last master deployment datetime), renders the Caddyfile (the shared basicauth hash from the host `.env`), reloads Caddy, and checks health. 146 tests pass. The Caddyfile auth uses a `route` block with `forward_auth` (see [Auth](#auth)); the caddy container mounts the host's `/srv/homepage` (not the repo source) so the generated files serve.
|
||||
|
||||
**Moved to the naked domain (2026-08-24):** the homepage now serves `franksdank.cloud` and `www.franksdank.cloud`. The old names `dankclaw.franksdank.cloud` and `dankserver.franksdank.cloud` redirect to it. Authentication moved from per-page basicauth to Authentik `forward_auth` (see [Authentik](Authentik.md)).
|
||||
|
||||
The per-service `/docs/` pages, the *arr stack, and the optional `/how/` page remain separate threads.
|
||||
|
||||
## Design decision — 2026-08-22
|
||||
|
||||
The live up/down check runs server-side. A checker on Dankclaw probes each backend and writes a same-origin `status.json`. The page fetches only that file and renders the badges. The browser never calls a backend directly. The checker runs as a systemd timer every 3 minutes.
|
||||
|
||||
The site has two pages. The root lists user-facing services only, with no host labels. An `/admin/` subpage lists the admin and infrastructure services behind a separate password for the operator. Gitea is a user service. Hermes is an admin service. A "how it is built" page for visitors is deferred (Q6).
|
||||
|
||||
The client-side probe design is blocked in this topology. The page origin is `https://franksdank.cloud`. The service URLs are `https://<name>.franksdank.cloud`. That is cross-origin and auth-gated. Both fail a browser `fetch`.
|
||||
|
||||
Intended direction: move to client-side probing. Serve every service as a same-origin `https` path under `franksdank.cloud`. Add a health endpoint and CORS headers to each backend. Then the page probes each service directly. Keep this note until that shift is complete.
|
||||
|
||||
## Structure
|
||||
|
||||
The site has two pages. The root page lists the user services. The `/admin/` page lists the admin and infrastructure services. The optional `/how/` page is deferred (Q6).
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph site["dankclaw.franksdank.cloud (Caddy)"]
|
||||
ROOT["/ user services<br/>(shared password)"]
|
||||
ADMIN["/admin/ admin services<br/>(operator password)"]
|
||||
HOW["/how/ how it is built<br/>(deferred)"]
|
||||
STATUS["/status.json"]
|
||||
end
|
||||
TIMER["systemd timer<br/>every 3 minutes"]
|
||||
CHECKER["checker script<br/>probes each backend"]
|
||||
BACKENDS["backends on both hosts"]
|
||||
TIMER --> CHECKER
|
||||
CHECKER -->|writes| STATUS
|
||||
STATUS -->|polled| ROOT
|
||||
STATUS -->|polled| ADMIN
|
||||
CHECKER -->|probes| BACKENDS
|
||||
```
|
||||
|
||||
## Status model
|
||||
|
||||
Each service shows one of three states.
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> Up
|
||||
[*] --> Down
|
||||
[*] --> WIP
|
||||
Up --> Down: probe fails
|
||||
Down --> Up: probe succeeds
|
||||
WIP --> Up: work completes
|
||||
```
|
||||
|
||||
- **Up** — the probe reached the backend.
|
||||
- **Down** — the probe failed (connection refused or timeout).
|
||||
- **WIP** — a static override. The service is intentionally not finished. The checker does not probe it.
|
||||
|
||||
## Checker
|
||||
|
||||
The checker runs on Dankclaw as a systemd timer. Every run probes each backend directly, then writes `status.json`. It probes the backend `host:port`, not the Caddy site. A Caddy `502` or `401` cannot tell up from down. The probe method is per service (`tcp` or `http`). The timeout is 5 s. The page reads `status.json` once on load. It does not auto-refresh.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[timer fires] --> B[read service config]
|
||||
B --> C{for each service}
|
||||
C -->|wip| D[mark WIP]
|
||||
C -->|probe| E[probe: tcp or http]
|
||||
E -->|connected| F[mark Up]
|
||||
E -->|refused / timeout| G[mark Down]
|
||||
D --> H[write status.json]
|
||||
F --> H
|
||||
G --> H
|
||||
```
|
||||
|
||||
## Service classification
|
||||
|
||||
**User services** (root page): SearXNG, ComfyUI, OpenWebUI, Gitea, Calibre-Web, RomM, Copyparty, Plex, Jellyfin.
|
||||
|
||||
**Admin services** (`/admin/`): Hermes, Grafana, Prometheus, Loki, Caddy, Pi-hole, Gluetun, the LLM fleet, and the low-level backends (mariadb, redis, Postgres, node-exporter, alloy). The backends show status only. They get a link only where a UI is meaningful.
|
||||
|
||||
**Excluded for now:** the *arr stack (Radarr, Sonarr, Lidarr, Readarr, Prowlarr, qBittorrent). It is not started. It is a separate effort, possibly on a different host.
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
shared/homepage/
|
||||
├── index.html # root page (user services)
|
||||
├── admin.html # admin + infrastructure services
|
||||
├── how.html # deferred (Q6)
|
||||
├── style.css # full theme; a toggle strips to bare HTML
|
||||
├── app.js # loads status.json once, renders the panels
|
||||
├── services.json # service config (user + admin)
|
||||
└── fonts/ # Terminus (woff2) for @font-face
|
||||
```
|
||||
|
||||
`status.json` is generated at runtime in `/srv/homepage/`. It is not committed.
|
||||
|
||||
## Rendering
|
||||
|
||||
The page renders a **grid of TUI-style panels** — one panel per service. Each panel shows the state badge, the name, the description, and the `Open` and `Docs` links. A summary line at the top counts the states. The footer shows the last master deployment datetime (Q43).
|
||||
|
||||
The page reads `status.json` **once on load**. It does not auto-refresh (Q41).
|
||||
|
||||
Styling is **toggleable** (Q27). The default is the full terminal theme. A toggle strips the page to bare, unstyled HTML (motherfuckingwebsite.com style). The page does not alter the user's cursor (Q28).
|
||||
|
||||
## Auth
|
||||
|
||||
Caddy wraps the site with `forward_auth` (Authentik SSO). A single Authentik
|
||||
session covers the whole site: the user page, the `/admin/` page, and the
|
||||
shared data + assets. The browser's Authentik cookie is reused for the data
|
||||
fetches, so no per-page credential is needed. See [Authentik](Authentik.md) for
|
||||
the provider, the outpost, and the first-time setup.
|
||||
|
||||
The Caddyfile pattern (one of the forward_auth vhosts):
|
||||
|
||||
```
|
||||
franksdank.cloud www.franksdank.cloud {
|
||||
root * /srv/homepage
|
||||
|
||||
route {
|
||||
# 1. The outpost path is proxied to Authentik (login UI + session).
|
||||
reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
|
||||
|
||||
# 2. forward_auth checks the session before serving the page.
|
||||
forward_auth authentik-server:9000 {
|
||||
uri /outpost.goauthentik.io/auth/caddy
|
||||
copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name
|
||||
trusted_proxies private_ranges
|
||||
}
|
||||
|
||||
# 3. The pages, in file order. 'handle' preserves order; a top-level
|
||||
# 'rewrite' would be sorted first and break the path matchers.
|
||||
handle /admin* {
|
||||
try_files /admin.html
|
||||
file_server
|
||||
}
|
||||
|
||||
@data path /services.json /status.json /app.js /style.css /deployed.js
|
||||
handle @data {
|
||||
file_server
|
||||
}
|
||||
|
||||
handle {
|
||||
try_files /index.html
|
||||
file_server
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
A `handle` inline matcher is a single token; the multi-path `@data` match is a
|
||||
named matcher. The caddy container mounts the host's `/srv/homepage` (where the
|
||||
deploy writes the static files and the checker writes `status.json`), not the
|
||||
repo's `shared/homepage`. A compose change needs a container recreate (`podman
|
||||
compose up -d caddy`), not a reload. The Caddyfile syntax is documented in the
|
||||
agent knowledge base (`~/.pi/agent/docs/caddy/caddyfile.md`).
|
||||
|
||||
## Service entry format
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "SearXNG",
|
||||
"category": "user",
|
||||
"description": "Privacy-respecting metasearch engine.",
|
||||
"url": "http://search.franksdank.cloud",
|
||||
"docs": "/docs/searxng/",
|
||||
"probe": "127.0.0.1:8081",
|
||||
"method": "http",
|
||||
"wip": false,
|
||||
"showLink": true
|
||||
}
|
||||
```
|
||||
|
||||
- `category` — `user` or `admin`. Selects the page.
|
||||
- `url` — the "Open" link (the Caddy site).
|
||||
- `docs` — the per-service docs page (`/docs/<svc>/`).
|
||||
- `probe` — the backend `host:port` the checker probes.
|
||||
- `method` — `tcp` or `http`. The probe type for this service (Q10).
|
||||
- `wip` — when true, the badge shows WIP and the checker skips the probe.
|
||||
- `showLink` — when false, the panel shows status only (no "Open" link).
|
||||
|
||||
## Deployment and testing
|
||||
|
||||
The deploy is **git-driven**. Master always reflects the currently deployed state (Q33). `deploy.sh` copies the assets to `/srv/homepage/`, installs the checker timer, renders the Caddyfile (both hashes), and reloads Caddy. It stamps the deployment datetime the footer shows (Q43).
|
||||
|
||||
The build is **full TDD** (Q36–Q38). Tests exist before the code: the checker (a fake up port, a down port, a WIP service, and the JSON schema), the pages and config (the `services.json` schema, `caddy validate`, the `status.json` reference), and the auth check (the forward_auth block and the renderable basicauth placeholder).
|
||||
|
||||
## Per-service docs (separate workstream)
|
||||
|
||||
Each service gets a docs page at `/docs/<svc>/` (Q30). The content is a **how-to for a regular user** — no maintenance details (Q29). The pages are password-protected by the same auth as the homepage section they belong to (Q31). They are built as each service goes active (Q32). The `docs` link on a service points to its page. The existing reference docs in `serverAppConfigs-docs/` are not linked from the homepage. They may contain secrets (Q22).
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Homepage Design Grilling]] — the decision log
|
||||
- [[Caddy]] — the proxy that serves it
|
||||
- [[Dankclaw Stack]], [[Dankserver Stack]] — the services it lists
|
||||
@@ -0,0 +1,145 @@
|
||||
# LAN DNS
|
||||
|
||||
The homelab gives every service a name under `franksdank.cloud`. Users type a name; they never type a port number. This page covers how names reach users: the Pi-hole server, the G36 router hand-off, and the split between LAN and WAN resolution.
|
||||
|
||||
See [[Caddy]] for the reverse proxy that serves the names, and [[Network Architecture]] for the full topology.
|
||||
|
||||
## Components
|
||||
|
||||
| Component | Where | Role |
|
||||
|-----------|-------|------|
|
||||
| Pi-hole | Dankclaw quadlet (`dankclaw/pihole/`) | Answers the flat service names for LAN clients |
|
||||
| SURFboard G36 | 192.168.0.1 | Hands dnsmasq to clients as the DHCP DNS server |
|
||||
| Porkbun | Registrar for `franksdank.cloud` | Answers the same names for the WAN |
|
||||
| Caddy | Dankclaw (192.168.0.143:443) | Serves every name over HTTPS |
|
||||
|
||||
## How a name reaches a client
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant C as LAN client (laptop, phone)
|
||||
participant G as G36 router (192.168.0.1)
|
||||
participant D as Pi-hole (192.168.0.143:53)
|
||||
participant P as Porkbun (WAN)
|
||||
participant X as Caddy (192.168.0.143:443)
|
||||
|
||||
C->>G: DHCP request
|
||||
G-->>C: lease + DNS = 192.168.0.143
|
||||
C->>D: dig romm.franksdank.cloud
|
||||
D-->>C: 192.168.0.143
|
||||
Note over D: addn-hosts answer<br/>(dankclaw/pihole/addn-hosts)
|
||||
C->>X: https://romm.franksdank.cloud
|
||||
X-->>C: RomM (192.168.0.235:8085 upstream)
|
||||
C->>P: dig franksdank.cloud (unknown name)
|
||||
P-->>C: 71.227.227.172
|
||||
```
|
||||
|
||||
Pi-hole answers only the names in `addn-hosts`. Every other query goes to the upstream resolvers `1.1.1.1` and `1.0.0.1`. The zone apex and unknown subdomains fall through to Porkbun. This avoids a wildcard override.
|
||||
|
||||
## Why split DNS
|
||||
|
||||
The G36 has no NAT loopback (hairpin). A LAN client that resolves a name to the WAN IP and then connects to it would time out. Split DNS removes the hairpin: LAN clients get the LAN IP, WAN clients get the public IP. Both use the same name and the same Let's Encrypt certificate. See [[Caddy]] for the TLS side.
|
||||
|
||||
## Pi-hole quadlet
|
||||
|
||||
Pi-hole FTL is a dnsmasq fork. It serves the same `addn-hosts` file the retired dnsmasq quadlet used, mounted into `/etc/pihole/hosts/` read-only. It retired the dnsmasq quadlet on 2026-08-24.
|
||||
|
||||
- **Location:** `dankclaw/pihole/` — `quadlet/pihole.container`, `env.example`, `addn-hosts`
|
||||
- **Ports:** 53/udp + 53/tcp (DNS) and 8083/tcp (web UI) on 192.168.0.143
|
||||
- **Host prerequisite 1:** `sysctl net.ipv4.ip_unprivileged_port_start=53` (persisted in `/etc/sysctl.d/99-dnsmasq.conf`)
|
||||
- **Host prerequisite 2:** `DNSStubListener=no` in `/etc/systemd/resolved.conf` (the resolved stub would otherwise hold port 53 inside the container netns)
|
||||
- **Env file:** `/home/dank/Containers/pihole/env` (never committed) — `ServerIP`, `TZ`, `DNS1` + `DNS2` (pinned upstreams `1.1.1.1` + `1.0.0.1`), `WEBPASSWORD` (the shared family password)
|
||||
- **Web UI:** `pihole.franksdank.cloud` (Caddy vhost → 8083), family login
|
||||
- **Deploy + restart after a change:** `./scripts/deploy-pihole-quadlet.sh dankclaw`
|
||||
|
||||
Add a new service name in two places:
|
||||
|
||||
1. `dankclaw/pihole/addn-hosts` — `192.168.0.143 <name>.franksdank.cloud`
|
||||
2. Porkbun — A record `<name> 71.227.227.172` (so WAN clients and the DNS-01 challenge work)
|
||||
|
||||
Then add the vhost to [[Caddy]] and run `./scripts/deploy.sh dankclaw`. See [[Pi-hole]] for the full configuration.
|
||||
|
||||
## G36 router DNS hand-off
|
||||
|
||||
The router must tell DHCP clients to use Pi-hole. Until that step is done, new clients get the router's own DNS relay and the flat names do not resolve for them. Existing clients that already use Pi-hole are unaffected.
|
||||
|
||||
**Manual step (web GUI at `https://192.168.0.1`):**
|
||||
|
||||
1. Open **Gateway → Connection → Local IP Network → IPv4**.
|
||||
2. Uncheck **DNS Proxy** (disable the router DNS relay).
|
||||
3. Set **DNS** to **Statically configured**.
|
||||
4. Set **Primary DNS** to `192.168.0.143`.
|
||||
5. Leave the secondary field empty or set `0.0.0.0`.
|
||||
6. Apply, then re-lease DHCP on each client (`sudo dhclient -r && sudo dhclient`).
|
||||
|
||||
The router admin interface has no programmatic API in this firmware. The web GUI is the only control path. Keep the credentials out of this repo; the repo tracks no router secrets.
|
||||
|
||||
Verify after the change:
|
||||
|
||||
```
|
||||
# on a LAN client
|
||||
resolv.conf # nameserver 192.168.0.143
|
||||
dig romm.franksdank.cloud +short
|
||||
# expected: 192.168.0.143
|
||||
```
|
||||
|
||||
## Dankclaw host DNS
|
||||
|
||||
The Dankclaw host itself resolves the flat names through Pi-hole:
|
||||
|
||||
- `/etc/systemd/resolved.conf`: `DNS=192.168.0.143 192.168.0.1` (Pi-hole first, router as fallback)
|
||||
|
||||
Without this, the host would resolve the names through the router to the WAN IP and time out (no hairpin).
|
||||
|
||||
## Dankware host DNS
|
||||
|
||||
`dankware` is a LAN client, not a Pi-hole server. Its flat names resolve through Pi-hole (`192.168.0.143`), but its resolver setup differs from Dankclaw's because it carries two `+DefaultRoute` links.
|
||||
|
||||
- **Two default links:** `enp47s0` (ethernet, metric 100) and `wlan0` (wifi, metric 600), each getting the same DHCP DNS list `[192.168.0.143, 1.1.1.1, …]`.
|
||||
- **The flap:** `systemd-resolved` picks a different "current" resolver per link — `enp47s0` settled on the Comcast v6 upstream (→ WAN IP, times out on the LAN), `wlan0` on Pi-hole (→ LAN IP). Flat-name resolution flapped between the two on every query.
|
||||
- **The fix:** pin `192.168.0.143` as the static IPv4 resolver on both NetworkManager connections. `resolved.conf`'s own `DNS=` is a per-link fallback, so the link itself must be pinned:
|
||||
|
||||
```bash
|
||||
for c in "Wired connection 1" FRANKFI-5G; do
|
||||
nmcli connection modify "$c" ipv4.method auto \
|
||||
ipv4.dns 192.168.0.143 \
|
||||
ipv4.ignore-auto-dns yes \
|
||||
ipv4.ignore-auto-routes no
|
||||
done
|
||||
nmcli connection up "Wired connection 1"
|
||||
nmcli connection up FRANKFI-5G
|
||||
systemctl restart systemd-resolved
|
||||
```
|
||||
|
||||
`method:auto` keeps DHCP for the address/gateway/route (dual-default traffic path preserved); `ignore-auto-dns=yes` uses only the static resolver; `ignore-auto-routes=no` keeps the DHCP routes. Verify with a cache-clear loop — `romm.franksdank.cloud` should return `192.168.0.143` on every run:
|
||||
|
||||
```bash
|
||||
for i in $(seq 1 10); do resolvectl flush-caches; dig romm.franksdank.cloud +short; done
|
||||
```
|
||||
|
||||
The apex and unknown subdomains still fall through to Porkbun (→ `71.227.227.172`), so WAN clients and a bare `dig franksdank.cloud` keep working.
|
||||
|
||||
## WAN records (Porkbun)
|
||||
|
||||
The Porkbun zone `franksdank.cloud` carries an A record for every vhost plus the apex, all pointing at `71.227.227.172`. TTL is 300. The zone has no wildcard record: unknown subdomains return NXDOMAIN. Caddy's DNS-01 challenges add and remove `_acme-challenge.*` TXT records through the same API during issuance. See [[Caddy]] for the TLS flow.
|
||||
|
||||
`franksdank.casa` is not in this Porkbun account. The `plex.franksdank.casa` vhost stays plain HTTP for that reason. The canonical Plex name is `plex.franksdank.cloud` (HTTPS).
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Symptom | First check |
|
||||
|---------|-------------|
|
||||
| Name does not resolve on a new client | `resolv.conf` shows 192.168.0.1, not .143 — the G36 hand-off step is missing |
|
||||
| Name resolves to 71.227.x and times out on the LAN | Client used the router DNS — same root cause |
|
||||
| Flat name flaps between 192.168.0.143 and 71.227.x on one client | Two `+DefaultRoute` links; resolved picks a different "current" resolver per link. Pin 192.168.0.143 as static DNS on both NM connections — see [[#Dankware host DNS]] |
|
||||
| `dig @192.168.0.143` fails entirely | Pi-hole quadlet down: `systemctl --user status pihole` |
|
||||
| Single-label name (for example `dankclaw`) answers `127.0.1.1` | The rootless container `/etc/hosts` carries the host entry (`127.0.1.1 <host>`). FTL serves it. Use the FQDN, or an `/etc/hosts` entry on the client |
|
||||
| HTTPS works from the host but not the WAN | No port forward on the G36 for that service (manual router step) |
|
||||
| Cert renewal fails in Caddy logs | Porkbun API credentials in the Dankclaw `.env`; test with the Porkbun API `/ping` endpoint |
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Pi-hole]] — the LAN DNS server (quadlet, records, web UI)
|
||||
- [[Caddy]] — TLS, vhost list, DNS-01
|
||||
- [[Network Architecture]] — topology, access policy
|
||||
- [[Dankclaw Stack]] — the host that carries Pi-hole and Caddy
|
||||
@@ -0,0 +1,51 @@
|
||||
# Loki
|
||||
|
||||
Loki is the log aggregation backend of the observability stack. It stores and queries container and system logs, and it feeds the log panels in Grafana. It replaces the need for a full ELK stack.
|
||||
|
||||
**Host:** Dankclaw — **Docs:** https://grafana.com/docs/loki/latest
|
||||
|
||||
## Facts (from research)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `franksdank/loki:latest` (local build with a static busybox healthcheck) |
|
||||
| Port | 3100 (host) → 3100 (container) |
|
||||
| Mode | single-binary (all components in one process) |
|
||||
| Storage | filesystem (bind mount) |
|
||||
| Auth | `auth_enabled: false` (single-tenant homelab) |
|
||||
| Schema | v13 / TSDB |
|
||||
|
||||
## Configuration
|
||||
|
||||
Config lives at `shared/loki/loki-config.yaml`. Key settings:
|
||||
|
||||
- `schema_config` v13 with `store: tsdb` and filesystem object store.
|
||||
- `compactor` with `retention_enabled: true` so logs do not grow without bound.
|
||||
- `replication_factor: 1` for a single instance.
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path (convention) |
|
||||
|----------------|------------------------|
|
||||
| `/loki` | `/opt/loki/data` |
|
||||
| `/etc/loki/loki-config.yaml` | `shared/loki/loki-config.yaml` (bind mount) |
|
||||
|
||||
## Healthcheck
|
||||
|
||||
Loki responds on `/ready` with 200.
|
||||
|
||||
## Log Shipping
|
||||
|
||||
The [[Alloy]] agent on each host ships container and journal logs to Loki. Grafana queries Loki through the datasource in [[Observability]].
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Filesystem storage grows; enable compactor retention.
|
||||
- Loki and Promtail/Alloy must agree on the push URL: `http://loki:3100/loki/api/v1/push`.
|
||||
- For multi-host shipping, Loki's HTTP listener binds `0.0.0.0`.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Observability]]
|
||||
- [[Alloy]] — the log shipper
|
||||
- [[Grafana]] — the query frontend
|
||||
@@ -0,0 +1,55 @@
|
||||
# Node-Exporter
|
||||
|
||||
Node-Exporter exposes host metrics (CPU, RAM, disk, network) to Prometheus. One instance runs per host inside its own container.
|
||||
|
||||
**Hosts:** Dankserver + Dankclaw — **Docs:** https://prometheus.io/docs/guides/node-exporter
|
||||
|
||||
## Facts (from research)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `prom/node-exporter:latest` |
|
||||
| Port | 9100 (host) |
|
||||
| Network | `host` mode |
|
||||
| Config | `--path.procfs=/host/proc`, `--path.sysfs=/host/sys`, `--path.rootfs=/rootfs` |
|
||||
|
||||
## Configuration
|
||||
|
||||
```yaml
|
||||
services:
|
||||
node-exporter:
|
||||
image: prom/node-exporter:latest
|
||||
restart: always
|
||||
network_mode: host
|
||||
pid: host
|
||||
volumes:
|
||||
- /proc:/host/proc:ro
|
||||
- /sys:/host/sys:ro
|
||||
- /:/rootfs:ro
|
||||
command:
|
||||
- --path.procfs=/host/proc
|
||||
- --path.sysfs=/host/sys
|
||||
- --path.rootfs=/rootfs
|
||||
```
|
||||
|
||||
## Prometheus Targets
|
||||
|
||||
Prometheus scrapes each host's node-exporter directly:
|
||||
|
||||
| Host | Target |
|
||||
|------|--------|
|
||||
| Dankserver | `192.168.0.235:9100` |
|
||||
| Dankclaw | `192.168.0.143:9100` |
|
||||
|
||||
These scrape jobs live in the Prometheus config; see [[Prometheus]] and [[Observability]].
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Requires `network_mode: host` (or a published port) so Prometheus can reach 9100.
|
||||
- The `/proc`, `/sys`, and root mounts must be `read-only` (`:ro`).
|
||||
- Do not mount the same paths on both hosts' containers; each node-exporter reflects only its own host.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Prometheus]] — the scraper
|
||||
- [[Observability]] — the full stack
|
||||
@@ -0,0 +1,53 @@
|
||||
# Observability
|
||||
|
||||
The observability stack runs on Dankclaw: Grafana for dashboards, Prometheus for metrics, and log aggregation for container logs. This is a small homelab — no Kubernetes.
|
||||
|
||||
## Stack
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
S[Services] -->|/metrics| P[Prometheus]
|
||||
S -->|logs| T[Alloy]
|
||||
T --> L[Loki]
|
||||
P --> G[Grafana]
|
||||
L --> G
|
||||
```
|
||||
|
||||
| Component | Role | Image | Port |
|
||||
|-----------|------|-------|------|
|
||||
| [[Prometheus]] | metrics collection | `prom/prometheus` | 9090 |
|
||||
| [[Grafana]] | dashboards | `grafana/grafana` | 3002 |
|
||||
| Loki | log storage & query | `grafana/loki` | 3100 |
|
||||
| Alloy | log shipping (per host) | `grafana/alloy` | — |
|
||||
|
||||
## Log Aggregation Choice
|
||||
|
||||
Loki + Alloy is the lightweight Grafana-native option for a two-server homelab:
|
||||
|
||||
- Alloy reads container logs from the host (journald socket + container log files) and pushes them to Loki. It replaces Promtail, which reached end-of-life on 2026-03-02.
|
||||
- Loki stores compressed logs with labels; queries run in Grafana's Explore.
|
||||
- No Elasticsearch memory cost, no Kubernetes required.
|
||||
|
||||
Alternatives considered and rejected: ELK (too heavy), Promtail (EOL), VictoriaLogs (newer, less documentation), plain `journald` (no centralized query).
|
||||
|
||||
## Scrape Surface
|
||||
|
||||
| Target | Source |
|
||||
|--------|--------|
|
||||
| Prometheus self | `localhost:9090/metrics` |
|
||||
| ComfyUI | `/system_stats` (limited) or blackbox probe |
|
||||
| Gitea | `/metrics` (built-in exporter) |
|
||||
| Grafana | `/metrics` (built-in) |
|
||||
| Host metrics | node-exporter `:9100` |
|
||||
|
||||
For services without native exporters, use the Prometheus **blackbox exporter** in probe mode to assert up/down and HTTP latency. The homepage status also probes the same endpoints.
|
||||
|
||||
## Healthchecks
|
||||
|
||||
Every compose service gets a healthcheck so podman reports container health and Promtail/Grafana can alert on it. See [[Deployment Flow]] for the verify step.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Grafana]], [[Prometheus]]
|
||||
- [[Dankclaw Stack]]
|
||||
- [[Homepage]] — the user-facing status surface
|
||||
@@ -0,0 +1,73 @@
|
||||
# Pi-hole
|
||||
|
||||
Pi-hole is the LAN DNS server for the homelab. It runs as a quadlet container on Dankclaw (192.168.0.143). It answers the flat `franksdank.cloud` / `franksdank.casa` service names and filters ad traffic on the LAN. It retired the dnsmasq quadlet on 2026-08-24. See [[LAN-DNS]] for the name-resolution flow and the router hand-off.
|
||||
|
||||
## Components
|
||||
|
||||
| Component | Where | Role |
|
||||
|-----------|-------|------|
|
||||
| `pihole` quadlet | `dankclaw/pihole/quadlet/pihole.container` | Boot-persistent container unit |
|
||||
| FTL | `docker.io/pihole/pihole:latest` | DNS engine (a dnsmasq fork) + ad blocking |
|
||||
| `addn-hosts` | `dankclaw/pihole/addn-hosts` | One A record per Caddy vhost |
|
||||
| sysctl drop-in | `dankclaw/pihole/sysctl/99-dnsmasq.conf` (repo) → `/etc/sysctl.d/` | Rootless port-53 threshold, applied at boot |
|
||||
| env file | `/home/dank/Containers/pihole/env` (host) | `ServerIP`, `TZ`, `DNS1`, `DNS2`, `WEBPASSWORD` |
|
||||
| Web UI | `pihole.franksdank.cloud` (Caddy → 8083) | Dashboard + admin, family login |
|
||||
|
||||
## Quadlet
|
||||
|
||||
- **Install path on Dankclaw:** `~/.config/containers/systemd/pihole.container` (unit `pihole.service`)
|
||||
- **Network:** `slirp4netns` (the same pattern as open-webui and gitea)
|
||||
- **Published ports:** 53/udp + 53/tcp (DNS), 8083/tcp (web UI)
|
||||
- **Mounts:**
|
||||
- `/home/dank/Containers/pihole/etc-pihole` → `/etc/pihole` (config + state, preserved across restarts)
|
||||
- `/home/dank/Containers/pihole/etc-dockers` → `/etc/dockers` (blocklists)
|
||||
- `/home/dank/Containers/pihole/addn-hosts` → `/etc/pihole/hosts/addn-hosts` (read-only; the record file from this repo)
|
||||
- **Restart:** `Restart=always`, `RestartSec=5`
|
||||
- **Deploy:** `./scripts/deploy-pihole-quadlet.sh dankclaw` (idempotent; preserves the data dirs)
|
||||
|
||||
## Records
|
||||
|
||||
FTL does not read `/etc/dnsmasq.d/` — its generated `dnsmasq.conf` has no `conf-dir`. The only file-based local-record mechanism is `hostsdir=/etc/pihole/hosts`: FTL parses every file in that directory as an `/etc/hosts` file. The quadlet therefore mounts the repo record file there, read-only:
|
||||
|
||||
```
|
||||
/home/dank/Containers/pihole/addn-hosts -> /etc/pihole/hosts/addn-hosts
|
||||
```
|
||||
|
||||
The file format is hosts-style (`<IP> <name> [<name> ...]`), one A record per Caddy vhost, all pointing at `192.168.0.143`. The file stays in sync with the Caddyfile (`tests/pihole.bats` cross-checks the two). Add a name there and in Porkbun, then deploy. See [[LAN-DNS]] for the two-place rule.
|
||||
|
||||
**Pitfall (2026-08-24):** the first quadlet mounted the file into `/etc/dnsmasq.d/`. FTL never read it; names only resolved from the query cache. A fresh name returned NXDOMAIN. Mount into `/etc/pihole/hosts/`, not `/etc/dnsmasq.d/`.
|
||||
|
||||
## Upstream
|
||||
|
||||
The slirp netns has no useful `resolv.conf`. The env file pins the upstreams: `DNS1=1.1.1.1`, `DNS2=1.0.0.1` (mirrors the retired `dnsmasq.conf` `server=` lines). Every query that is not a local record goes to those resolvers. The zone apex and unknown subdomains fall through to Porkbun.
|
||||
|
||||
## Web UI and API
|
||||
|
||||
- **URL:** `https://pihole.franksdank.cloud/` (Caddy vhost, family basicauth → 8083)
|
||||
- **Password:** `WEBPASSWORD` in the env file (the shared family password, the same credential as `CADDY_BASICAUTH_HASH`)
|
||||
- **API:** `https://pihole.franksdank.cloud/api/...` — authenticate with the session cookie, then query the blocklist, the local DNS records, and the query log
|
||||
- **Port on the LAN:** `http://192.168.0.143:8083/` (direct, no Caddy)
|
||||
|
||||
## Host prerequisites
|
||||
|
||||
1. `net.ipv4.ip_unprivileged_port_start=53`, persisted in `/etc/sysctl.d/99-dnsmasq.conf` — rootless Podman may then publish port 53. The repo file `dankclaw/pihole/sysctl/99-dnsmasq.conf` is the single source of truth: `deploy_pihole` installs it, stages any drop-in that sets a value above 53 in the home trash, comments out a stale active line in `/etc/sysctl.conf`, and applies everything with `sysctl --system` (the same code path boot uses via systemd-sysctl). **Incident (2026-08-26):** the value was reset to 80 by *two* sources — a later-sorted drop-in (`99-rootless-ports.conf`) and a stale line in `/etc/sysctl.conf`, which systemd-sysctl processes **after** all of `sysctl.d/` and therefore overrides every drop-in. The runtime override from an earlier manual `sysctl -w` died with a reboot, and `pihole.service` crash-looped on `bind: permission denied` for port 53 (restart counter 1645) until both conflicts were removed.
|
||||
2. `DNSStubListener=no` in `/etc/systemd/resolved.conf` — the resolved stub would otherwise hold port 53 inside the container netns.
|
||||
|
||||
Both prerequisites survive the dnsmasq → Pi-hole swap. Do not remove them.
|
||||
|
||||
## Failure modes
|
||||
|
||||
| Symptom | First check |
|
||||
|---------|-------------|
|
||||
| `dig @192.168.0.143` fails entirely | Quadlet down: `systemctl --user status pihole` |
|
||||
| A name does not resolve for a new client | The client's `resolv.conf` does not show `192.168.0.143` — the G36 hand-off is missing. See [[LAN-DNS]] |
|
||||
| The web UI asks for a password loop | `WEBPASSWORD` in the env file is the placeholder `changeme` — fill the real value, restart the unit |
|
||||
| Upstream queries fail (local names still answer) | `DNS1`/`DNS2` in the env file are missing or wrong |
|
||||
| Port 53 `bind: permission denied` crash loop after a reboot | sysctl conflict: the value must be 53 at boot, and remember `/etc/sysctl.conf` is applied **after** all of `sysctl.d/`. Redeploy pihole — `deploy_pihole` reinstalls the drop-in, neutralizes conflicting drop-ins, and comments out a stale `/etc/sysctl.conf` line |
|
||||
| Single-label name (for example `dankclaw`) answers `127.0.1.1` | The rootless container `/etc/hosts` carries the host entry (`127.0.1.1 <host>`), and FTL serves it. Use the FQDN, or add an `/etc/hosts` entry on the client |
|
||||
|
||||
## See Also
|
||||
|
||||
- [[LAN-DNS]] — name resolution flow, router hand-off, split DNS
|
||||
- [[Caddy]] — the `pihole.franksdank.cloud` vhost
|
||||
- [[Dankclaw Stack]] — the host that carries Pi-hole
|
||||
@@ -0,0 +1,57 @@
|
||||
# Prometheus
|
||||
|
||||
Prometheus collects metrics from every service on Dankclaw by scraping their `/metrics` endpoints. It feeds the Grafana dashboards. The `node-exporter` containers may add podman/cgroup metrics for full host visibility.
|
||||
|
||||
**Host:** Dankclaw — **Hostname:** `prometheus.dankclaw.local` — **Port:** 9090 — **Docs:** https://prometheus.io/docs
|
||||
|
||||
## Facts
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Image | `docker.io/prom/prometheus:latest` |
|
||||
| Port | 9090 (host) → 9090 (container) |
|
||||
| Config | `/etc/prometheus/prometheus.yml` (bind mount) |
|
||||
| Data volume | TSDB |
|
||||
| Retention | `--storage.tsdb.retention.time=30d` |
|
||||
|
||||
## prometheus.yml Targets
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: "prometheus"
|
||||
static_configs:
|
||||
- targets: ["localhost:9090"]
|
||||
|
||||
- job_name: "comfyui"
|
||||
static_configs:
|
||||
- targets: ["comfyui:8188"]
|
||||
|
||||
- job_name: "node"
|
||||
static_configs:
|
||||
- targets: ["node-exporter:9100"]
|
||||
```
|
||||
|
||||
Allow the `/metrics` endpoint on each service where the exporter exists. For services without a native exporter, use the Prometheus `blackbox` exporter (probe mode) to check up/down and latency.
|
||||
|
||||
## Volumes
|
||||
|
||||
| Container path | Host path (convention) |
|
||||
|----------------|------------------------|
|
||||
| `/etc/prometheus` | `/opt/prometheus/config` |
|
||||
| `/prometheus` | `/opt/prometheus/data` |
|
||||
|
||||
## Healthcheck
|
||||
|
||||
Prometheus responds on `/-/healthy` with 200.
|
||||
|
||||
## Gotchas
|
||||
|
||||
- Keep scrapes modest for a small homelab; too many targets bloat the TSDB.
|
||||
- Use recording rules for heavy queries (see [[Grafana]]).
|
||||
- Set a retention policy or the data volume grows without bound.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankclaw Stack]]
|
||||
- [[Grafana]] — the dashboard frontend
|
||||
- [[Observability]] — the whole stack
|
||||
@@ -0,0 +1,14 @@
|
||||
# authentik-postgres
|
||||
|
||||
The PostgreSQL 16 database for [Authentik](Authentik.md) — the homelab's SSO /
|
||||
OIDC provider. It is the only stateful store in the stack (users, flows,
|
||||
sessions, events). Redis was removed in Authentik 2025.10; sessions and caching
|
||||
now live here.
|
||||
|
||||
- **Compose service:** `authentik-postgres` (in `dankclaw/compose.yaml`)
|
||||
- **Image:** `docker.io/library/postgres:16-alpine`
|
||||
- **Data:** `/home/dank/authentik/postgres` (bind mount; no named volumes)
|
||||
- **Health:** `pg_isready -d authentik -U authentik`
|
||||
|
||||
See [Authentik.md](Authentik.md) for the full architecture, setup, backups
|
||||
(`pg_dump`), and failure modes.
|
||||
@@ -0,0 +1,15 @@
|
||||
# authentik-server
|
||||
|
||||
The Authentik server — the core, the admin API, and the embedded Caddy outpost.
|
||||
This is the service the Caddy `forward_auth` vhosts proxy to
|
||||
(`authentik-server:9000`). It also serves the login page at
|
||||
`https://auth.franksdank.cloud`.
|
||||
|
||||
- **Compose service:** `authentik-server` (in `dankclaw/compose.yaml`)
|
||||
- **Image:** `ghcr.io/goauthentik/server:2026.8.0` (command: `server`)
|
||||
- **Port:** `9000` (published; the outpost + admin + login all live here)
|
||||
- **Data:** `/home/dank/authentik/data` (bind mount, shared with the worker)
|
||||
- **Health:** `http://localhost:9000/-/health/`
|
||||
|
||||
See [Authentik.md](Authentik.md) for the full architecture, the Caddy
|
||||
forward_auth pattern, first-time setup, and failure modes.
|
||||
@@ -0,0 +1,14 @@
|
||||
# authentik-worker
|
||||
|
||||
The Authentik worker — background tasks: event processing, provider
|
||||
synchronization, scheduled flows, and certificate renewal. It runs the same
|
||||
image as the server; the `command: worker` selects the role.
|
||||
|
||||
- **Compose service:** `authentik-worker` (in `dankclaw/compose.yaml`)
|
||||
- **Image:** `ghcr.io/goauthentik/server:2026.8.0` (command: `worker`)
|
||||
- **Runs as:** `0:0` (rootless: container root maps to the host user that owns the data dir)
|
||||
- **Data:** `/home/dank/authentik/data` (bind mount, shared with the server)
|
||||
- **No port** — the worker has no HTTP endpoint.
|
||||
|
||||
See [Authentik.md](Authentik.md) for the full architecture, setup, and failure
|
||||
modes.
|
||||
@@ -0,0 +1,218 @@
|
||||
# DFlash and DSpark Speculative Decoding
|
||||
|
||||
## Overview
|
||||
|
||||
DFlash and DSpark are the two newest speculative decoding methods. They draft many tokens in one parallel pass instead of one token at a time. This reduces decode latency on memory-bound workloads.
|
||||
|
||||
DFlash comes from Z Lab at UC Berkeley. DSpark comes from DeepSeek. Both work with vLLM and llama.cpp. Both have draft models for the Qwen3.6 models that Dankserver and Dankclaw run.
|
||||
|
||||
## How DFlash Works
|
||||
|
||||
DFlash is a lightweight block diffusion model. It predicts a block of tokens in a single forward pass. Key design points:
|
||||
|
||||
- The draft model reuses the embedding and LM head from the target model.
|
||||
- Only the intermediate layers are trained.
|
||||
- It fuses target-model hidden features into the KV projections of every draft layer.
|
||||
- A 16-token DFlash drafter has lower latency than a 1-layer EAGLE-3 drafter producing 8 tokens.
|
||||
|
||||
Results from the DFlash paper (arXiv:2602.06036):
|
||||
|
||||
- Over 6x lossless speedup on Qwen3-8B.
|
||||
- Up to 2.5x higher speedup than EAGLE-3.
|
||||
- Diffusion drafting cost is flat, so block size does not increase cost.
|
||||
|
||||
## How DSpark Works
|
||||
|
||||
DSpark is a semi-autoregressive extension of DFlash from DeepSeek (arXiv:2607.05147). It keeps the DFlash parallel diffusion backbone and adds a lightweight sequential Markov head. This models inter-token dependencies inside each block. It also uses confidence-scheduled verification, which stops verifying tokens that have a high rejection risk.
|
||||
|
||||
Results from the DSpark paper:
|
||||
|
||||
- 60-85% faster per-user generation than MTP-1 on DeepSeek-V4 production traffic.
|
||||
- Useful when parallel drafts decay in acceptance rate.
|
||||
|
||||
Both DFlash and DSpark are lossless under rejection sampling. Use the strict rejection sampler.
|
||||
|
||||
## Runtime Support
|
||||
|
||||
| Runtime | DFlash | DSpark | MTP |
|
||||
|---------|--------|--------|-----|
|
||||
| vLLM | yes (v0.20.1+) | yes | yes |
|
||||
| llama.cpp | yes | yes | yes |
|
||||
| SGLang | yes | no | yes |
|
||||
|
||||
### vLLM
|
||||
|
||||
vLLM v0.20.1+ has core DFlash support. Use the `--speculative-config` JSON object.
|
||||
|
||||
DFlash on vLLM:
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B \
|
||||
--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 15}' \
|
||||
--attention-backend flash_attn \
|
||||
--max-num-batched-tokens 32768
|
||||
```
|
||||
|
||||
Legacy flags also work:
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B \
|
||||
--spec-method dflash \
|
||||
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
|
||||
--spec-tokens 8
|
||||
```
|
||||
|
||||
DFlash requires a non-causal attention backend. Set `"attention_backend": "flash_attn"` in the speculative config if needed. On ROCm, check that the flash_attn backend is available before using DFlash.
|
||||
|
||||
DSpark on vLLM:
|
||||
|
||||
```
|
||||
vllm serve <target-model> \
|
||||
--speculative-config '{"method": "dspark", "model": "<dspark-draft>", "num_speculative_tokens": 15}'
|
||||
```
|
||||
|
||||
### llama.cpp
|
||||
|
||||
llama.cpp supports DFlash, DSpark, and MTP through `--spec-type`. When you pass `--spec-draft-hf`, llama.cpp auto-discovers the sidecar files and infers the type. The priority order is DSpark, DFlash, EAGLE-3, MTP.
|
||||
|
||||
`--spec-type` accepts a **comma-separated list** of speculative implementations. `common_speculative_init` (in `common/speculative.cpp`) builds a vector of implementations and chains drafting through all of them (`common_speculative_draft_params.drafting` is "used to chain the drafts through all the available implementations"). This means you can **layer ngram-mod on top of a DFlash draft model** in one server process:
|
||||
|
||||
```
|
||||
llama-server --hf-repo <target-GGUF> \
|
||||
--spec-draft-hf <dflash-draft-GGUF> \
|
||||
--spec-type draft-dflash,ngram-mod
|
||||
```
|
||||
|
||||
The DFlash drafter proposes, then ngram-mod runs as an additional implementation in the chain. Verified against upstream `common/speculative.cpp` (b10133+).
|
||||
|
||||
DFlash on llama.cpp:
|
||||
|
||||
```
|
||||
llama-server --hf-repo <target-GGUF> \
|
||||
--spec-draft-hf <draft-GGUF> \
|
||||
--spec-type draft-dflash \
|
||||
--spec-draft-n-max 16
|
||||
```
|
||||
|
||||
DSpark on llama.cpp:
|
||||
|
||||
```
|
||||
llama-server --hf-repo <target-GGUF> \
|
||||
--spec-draft-hf <draft-GGUF> \
|
||||
--spec-type draft-dspark
|
||||
```
|
||||
|
||||
MTP on llama.cpp:
|
||||
|
||||
```
|
||||
llama-server --hf-repo <target-with-MTP> \
|
||||
--spec-type draft-mtp \
|
||||
--spec-draft-n-max 2
|
||||
```
|
||||
|
||||
Note: the flag was renamed from `--spec-type mtp` to `--spec-type draft-mtp` in May 2026 (PR #22673, build b9180+). An old flag silently disables speculative decoding.
|
||||
|
||||
### MTP drafter availability for Qwen3.6-27B
|
||||
|
||||
MTP is not a separate drafter model. It is a "nextn" head that must live in the target GGUF (or be loaded as a draft alongside it). llama.cpp's `draft-mtp` implementation is fully wired for the `qwen35` and `qwen35moe` architectures (PR #22673).
|
||||
|
||||
Local check on Dankclaw (2026-08-13), tensor listing via GGUF header parse:
|
||||
|
||||
| File | Architecture | Tensors | MTP tensors |
|
||||
|------|--------------|---------|-------------|
|
||||
| `Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf` | `qwen35` | 851 | 0 |
|
||||
| `Qwen3.6-27B-DFlash-Q8_0.gguf` (draft) | `dflash` | 58 | 0 |
|
||||
|
||||
The HauhauCS uncensored GGUFs ship without the MTP head. To use MTP with them, load an MTP-only draft GGUF from the base Qwen3.6-27B model as a sidecar:
|
||||
|
||||
```
|
||||
llama-server --model <HauhauCS-Q8_K_P.gguf> \
|
||||
--spec-draft-model <MTP-only-draft.gguf> \
|
||||
--spec-type draft-mtp \
|
||||
--spec-draft-n-max 2
|
||||
```
|
||||
|
||||
Compatible MTP-only draft GGUFs on Hugging Face:
|
||||
|
||||
| Repo | Notes |
|
||||
|------|-------|
|
||||
| `a4lg/Qwen3.6-27B-MTP-ONLY-GGUF` | MTP-only subset of the base model; pairs with derivative/abliterated targets via `--spec-draft-model` |
|
||||
| `lym00/Qwen3.6-27B-MTP-ONLY-GGUF` | Same purpose; built from `convert_hf_to_gguf.py --mtp` |
|
||||
| `ggml-org/Qwen3.6-27B-MTP-GGUF` | Full model with MTP head (replaces the target, not a sidecar) |
|
||||
| `hyperhuzaifa/Qwen3.6-27B-Uncensored-MTP-GGUF` | HauhauCS Q4_K_P with the MTP head grafted in (replaces the target) |
|
||||
|
||||
Speculative decoding is lossless under rejection sampling: the target verifies every draft token. A draft head from the base model works with a derivative target; wrong proposals just lower the acceptance rate. The accepted MTP draft lengths are 2-3 tokens; the "MTP Defaults Are a Trap" writeup (thefrontierlab.ai) documents the tuning pitfalls.
|
||||
|
||||
On Dankclaw, `draft-dflash` remains the benchmarked winner (20.2 t/s at 4k vs 16.9 for DFlash+ngram and 8.7 for ngram-only). MTP is a viable alternative to test, not a bug fix: the parallel-stream releases investigated on 2026-08-13 were client-side aborts and a Dankserver crash, not speculative decoding behavior (see [[deployment]]).
|
||||
|
||||
## Draft Models
|
||||
|
||||
DFlash draft model sizes for the models on these servers:
|
||||
|
||||
| Target model | DFlash draft | Draft size |
|
||||
|--------------|--------------|------------|
|
||||
| Qwen3.6-35B-A3B | z-lab/Qwen3.6-35B-A3B-DFlash | 0.4B |
|
||||
| Qwen3.6-27B | z-lab/Qwen3.6-27B-DFlash | 2B |
|
||||
| Qwen3.5-35B-A3B | z-lab/Qwen3.5-35B-A3B-DFlash | 0.4B |
|
||||
| Qwen3.5-27B | z-lab/Qwen3.5-27B-DFlash | 2B |
|
||||
|
||||
The DFlash draft is not a standalone model. It must pair with its target model. The model cards recommend block size 8 for higher concurrency and 16 for single-stream throughput.
|
||||
|
||||
## Deployment on Dankserver (NVIDIA RTX 3090)
|
||||
|
||||
The Dankserver llama.cpp deployment (`frank-default-small`, Qwen3.6-35B-A3B) is **active** on Dankserver (port 7770) as of the 2026-08-17 redo (the earlier 2026-08-16/17 retirement of this service was reversed when the PP split was retired). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
|
||||
|
||||
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-default-small` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
|
||||
|
||||
The vLLM material below documents the archived AWQ/DFlash path on the same hardware. The RTX 3090 is Ampere (compute capability 8.6). Use these conditions:
|
||||
|
||||
- FP8 KV cache is valid on CUDA 12.
|
||||
- FP8 weights compute through a fallback path; no native FP8 tensor cores.
|
||||
- The DFlash draft model (0.4B) fits easily in 24 GB VRAM alongside the target.
|
||||
|
||||
Working DFlash flags for this server:
|
||||
|
||||
```
|
||||
--spec-method dflash
|
||||
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash
|
||||
--spec-tokens 8
|
||||
--kv-cache-dtype fp8
|
||||
```
|
||||
|
||||
Example DFlash + AWQ on the RTX 3090:
|
||||
|
||||
```
|
||||
docker run --rm --name vllm-qwen36-dflash \
|
||||
--gpus all --ipc=host \
|
||||
-p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
docker.io/vllm/vllm-openai:latest \
|
||||
cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit \
|
||||
--speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 8}' \
|
||||
--attention-backend flash_attn \
|
||||
--gpu-memory-utilization 0.95
|
||||
```
|
||||
|
||||
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
|
||||
|
||||
`frank-default-large` on Dankclaw runs the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec is the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. See [[qwen3.8-spec-matrix]] and [[Qwen3.8 PP Deployment]] for the current state and the 2026-08-15 Vulkan0/ROCm0 high-context bench.
|
||||
|
||||
History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
|
||||
|
||||
## Notes
|
||||
|
||||
- Speculative decoding helps low-QPS, memory-bound, latency-focused workloads. It helps less at high QPS.
|
||||
- DFlash needs a non-causal attention backend; the ROCm flash_attn backend must be validated before production use.
|
||||
- The old `--swap-space` flag no longer exists in current vLLM. Use `--kv-offloading-size`.
|
||||
- `--speculative-config` is the preferred CLI. Legacy `--spec-method`, `--spec-model`, and `--spec-tokens` flags still exist.
|
||||
- Use `rejection_sample_method` default `strict` for lossless output.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Quantization by Hardware Architecture]]
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
|
||||
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
|
||||
- [[Dankclaw System Specs]]
|
||||
- [[Dankserver System Specs]]
|
||||
- [[VLLM Overview]]
|
||||
- [[deployment]]
|
||||
@@ -0,0 +1,113 @@
|
||||
# Dankclaw — Improvements Research
|
||||
|
||||
This page collects concrete, sourced recommendations to improve Dankclaw
|
||||
(the AMD Strix Halo inference box). It covers self-hosted agentic tools,
|
||||
inference runtime tuning, and server/OS/hardware settings. Each item lists a
|
||||
priority and the exact action.
|
||||
|
||||
Dates: researched 2026-08. Hardware is verified on the live machine unless
|
||||
marked "verify first". Re-verified 2026-08-15/16; corrections marked below.
|
||||
|
||||
## Corrections (re-verified 2026-08-15/16)
|
||||
|
||||
- **Backend**: production is **Vulkan0**, not ROCm. Measured on this box
|
||||
(b10433): Vulkan 25.13 t/s vs ROCm 24.33 t/s decode at 16k; at 262k ctx
|
||||
Vulkan decodes 48% faster (5.68 vs 3.83 t/s). ROCm wins prefill (410 vs
|
||||
302 t/s @64k). The "HIP out-performs Vulkan for decode" claim did not
|
||||
hold on the 8060S. See [[frank-default-large-benchmark]].
|
||||
- **Power profile**: the `/sys/class/drm/card0/device/power_profile` file
|
||||
does not exist on this machine (checked 2026-08-15). GPU DPM is already
|
||||
`performance`, forced level `auto`. This item is N/A.
|
||||
- **KV cache**: production runs `q8_0` K **and** `q8_0` V (both quantized).
|
||||
- **Spec**: DFlash/DSpark are retired. Production is `--spec-default`
|
||||
(ngram-mod) with draft-mtp first. See [[qwen3.8-spec-matrix]].
|
||||
- **TTM**: raised to 114 GiB on 2026-08-25, set live (no reboot) and
|
||||
persisted in `/etc/modprobe.d/ttm.conf`. The effective VRAM ceiling that
|
||||
once held the service at ~15 GiB is gone.
|
||||
|
||||
## Verified Memory Picture (measured 2026-08-11)
|
||||
|
||||
| Figure | Value | Source |
|
||||
| ------ | ----- | ------ |
|
||||
| OS physical RAM | 30.5 GiB | `/proc/meminfo` |
|
||||
| TTM pages_limit | 114 GiB (29,883,536 x 4KiB) — raised 2026-08-25 | `/sys/module/ttm/parameters/pages_limit` |
|
||||
| ROCm VRAM total | 96.0 GiB | `rocm-smi --showmeminfo vram` |
|
||||
| ROCm VRAM used | 76.2 GiB | `rocm-smi` while serving 3x200700 |
|
||||
|
||||
The OS accounts only ~30.5 GiB physical RAM. The TTM GPU-addressable pool
|
||||
was capped at ~15.24 GiB (the real pre-2026-08-25 ceiling); that cap is now
|
||||
lifted to 114 GiB. ROCm/SMI still reports a 96 GiB VRAM carve-out (VGM) — a
|
||||
different, driver-reported figure that has always been larger than the kernel
|
||||
cap.
|
||||
This split is the Strix Halo GTT/VGM design. It drives the batch/ubatch memory
|
||||
limits documented in [[frank-default-large-benchmark]].
|
||||
|
||||
## 1. Self-Hosted Agentic Tools
|
||||
|
||||
All of these connect to the local llama.cpp OpenAI-compatible endpoint
|
||||
(`http://localhost:7770/v1`). None need NVIDIA.
|
||||
|
||||
| Tool | Use | Endpoint config | Priority |
|
||||
| ---- | --- | --------------- | -------- |
|
||||
| Open WebUI | Chat + agent harness, plugins, RAG, tool calling | Add OpenAI-compatible provider -> `http://localhost:7770/v1` | High |
|
||||
| Cline (VS Code) | IDE coding agent with MCP | Provider "OpenAI Compatible", `API_URL=http://localhost:7770/v1` | High |
|
||||
| Aider (CLI) | Terminal pair-programming agent | `OPENAI_API_BASE=http://localhost:7770/v1` | Medium |
|
||||
| Continue.dev (VS Code) | Open-source coding agent | `config.json` -> `{type:"openai", apiBase:"http://localhost:7770"}` | Medium |
|
||||
| OpenHands (Agent Canvas) | Self-hosted multi-agent coding platform | LiteLLM custom model + Base URL to the endpoint | Medium |
|
||||
| AnythingLLM | RAG / knowledge chat | Provider "OpenAI (Generic)" -> endpoint | Medium |
|
||||
| Browser Use | Browser automation agent (Playwright) | `ChatOpenAI(base_url="http://localhost:7770/v1")` | Low |
|
||||
| Qdrant | Vector store for RAG | Standalone, CPU-native | Low |
|
||||
|
||||
The pragmatic first installs are Open WebUI, Cline, and Aider. They all work
|
||||
with the current llama.cpp server and need no extra GPU.
|
||||
|
||||
## 2. Inference Runtime Tuning (llama.cpp on gfx1151)
|
||||
|
||||
Verified against the running service and llama.cpp source.
|
||||
|
||||
| Change | Recommendation | Priority |
|
||||
| ------ | -------------- | -------- |
|
||||
| Backend | **Vulkan0** (2026-08-15, corrected — see Corrections). ROCm only for prefill-heavy workloads. | Done |
|
||||
| KV cache | `q8_0` K / `q8_0` V in production. Try `q4_0` only if more context is needed. | Keep |
|
||||
| Flash attention | Leave `--flash-attn auto`. It helps prompt processing, not decode. | Keep |
|
||||
| Load mode | Start with mmap (default). Enable `--mlock` only if page-fault stutter appears. | Optional |
|
||||
| Threads | Use `--threads 12 --threads-batch 16 --threads-http 1` for 16 Zen5 cores + 3 slots. Frees cores for the ROCm driver. | Medium |
|
||||
| DFlash n_max | Current 8 (block 16) is high. Try `--spec-draft-n-max 5`; it may reduce rejection cost at the same throughput. | Verify |
|
||||
| ngram layering | Measured slower than DFlash alone at 4k (16.9 vs 20.2 t/s). Not enabled. | Keep off |
|
||||
|
||||
## 3. Server / OS / Hardware Settings
|
||||
|
||||
All require root and a reboot for the modprobe change.
|
||||
|
||||
| Setting | Action | Effect | Priority |
|
||||
| ------- | ------ | ------ | -------- |
|
||||
| TTM GPU pool | `/etc/modprobe.d/ttm.conf` → `options ttm pages_limit=29883536 page_pool_size=29883536` (applied 2026-08-25, set live then persisted) | Lifts the 15 GiB cap to 114 GiB; the full 262k KV + Q8 AGENT payload fit in one slot | Done |
|
||||
| Power profile | **N/A on this machine** (2026-08-15): no `power_profile` sysfs; GPU DPM already `performance` | none | ~~Medium~~ |
|
||||
| NVMe scheduler | Already `none` (`[none]` confirmed) | No change | Done |
|
||||
| BIOS VGM | Set Variable Graphics Memory / UMA to "High" in the EVO-X2 BIOS | Max dedicated VRAM for the GPU carve-out; verify in the live BIOS | Verify |
|
||||
| Firmware/EC | Check GMKtec for a newer EC/fan-curve firmware | Better sustained thermal behavior | Verify |
|
||||
|
||||
## 4. Other Enhancements
|
||||
|
||||
- Test `--spec-draft-n-max 5` and re-benchmark acceptance; if throughput rises,
|
||||
change the env tunable.
|
||||
- Consider switching the disabled `vllm-thinkingcap.service` unit off or
|
||||
removing it to avoid confusion (it is loaded in a failed state).
|
||||
- Profile generation under 3-slot concurrent load before touching boot
|
||||
settings, so any tuning is measured against the real workload.
|
||||
|
||||
## Sources
|
||||
|
||||
- [[frank-default-large-benchmark]] — measured backend/spec numbers
|
||||
- [[deployment]] — deploy and update process
|
||||
- AMD ROCm RDNA3.5 system optimization guide (gfx1151 memory/BIOS)
|
||||
- llama.cpp source: `ggml/CMakeLists.txt`, `ggml/src/ggml-hip/CMakeLists.txt`,
|
||||
`common/arg.cpp`, `common/common.h`
|
||||
- NotebookCheck Ryzen AI MAX+ 395 analysis
|
||||
|
||||
## See Also
|
||||
|
||||
- [[frank-default-large-benchmark]]
|
||||
- [[DFlash DSpark Speculative Decoding]]
|
||||
- [[deployment]]
|
||||
- [[index]]
|
||||
@@ -0,0 +1,106 @@
|
||||
# Dankclaw System Specs
|
||||
|
||||
## Server Overview
|
||||
|
||||
Dankclaw is an AMD APU server for CIRU ROCmFPX llama.cpp inference. The host runs Ubuntu 26.04 with an AMD Ryzen AI MAX+ 395 Strix Halo APU. The APU uses unified memory — CPU and GPU share the same physical RAM pool.
|
||||
## Hardware Specifications
|
||||
|
||||
### CPU
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Model | AMD RYZEN AI MAX+ 395 w/ Radeon 8060S |
|
||||
| Architecture | x86_64 |
|
||||
| Cores | 16 (1 socket, 1 thread per core) |
|
||||
| Max MHz | 5187.50 |
|
||||
| Min MHz | 625.00 |
|
||||
| NUMA Nodes | 1 (node 0, 31213 MB) |
|
||||
|
||||
### GPU (Integrated)
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Model | AMD Radeon 8060S (Strix Halo) |
|
||||
| Device ID | 0x1586 |
|
||||
| Architecture | gfx1151 (Strix Halo) |
|
||||
| GPU-Visible Memory | 64 GiB (rocm-smi; SVM-backed, see the memory model below) |
|
||||
| Physical RAM | 64 GiB (61 GiB visible; upgraded from 32 GB, corrected 2026-08-17) |
|
||||
|
||||
### NPU
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Model | AMD XDNA2 (Ryzen AI) |
|
||||
| Rating | 50 TOPS |
|
||||
| Device | /dev/accel/accel0 |
|
||||
| Driver | amdxdna 0.7 (in-tree, kernel 7.0.0-29-generic) |
|
||||
| Firmware | 1.1.2.65 |
|
||||
| Runtime | FastFlowLM 0.9.45 + XRT (installed 2026-08-16; **retired 2026-08-17**) |
|
||||
| Status | Not in service. Research only. See [[FastFlowLM NPU Inference]] |
|
||||
|
||||
The NPU is retired; the 52625 port and the `frank-default-small` name now serve the CIRU ROCmFPX llama.cpp runtime (see [[ciru-activefpx-promptforge]]).
|
||||
|
||||
### ROCm
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| ROCk Module | Loaded |
|
||||
| Runtime Version | 1.18 |
|
||||
| ROCm Version | 7.2.4 system (/opt/rocm) + TheRock 7.15 SDK in the CIRU venv (used for the runtime) |
|
||||
| Agent Name | AMD RYZEN AI MAX+ 395 w/ Radeon 8060S |
|
||||
| GPU Arch | gfx1151 (Strix Halo) |
|
||||
| GPU Global Memory | 64 GiB (rocm-smi; SVM-backed) |
|
||||
|
||||
### Memory
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| System RAM | 64 GiB (shared CPU + GPU; 61 GiB visible) |
|
||||
| GPU-Visible (ROCm) | 64 GiB (rocm-smi; SVM-backed) |
|
||||
| Swap | 8.0 GiB |
|
||||
| /dev/shm | ~15 GiB (half of RAM) |
|
||||
|
||||
### Storage
|
||||
|
||||
| Filesystem | Size | Used | Avail | Use% |
|
||||
|------------|------|------|-------|------|
|
||||
| / (nvme0n1p2) | 1.9T | 1.3T | 545G | 70% |
|
||||
|
||||
### Network
|
||||
|
||||
| Interface | Type | Speed | IP | Notes |
|
||||
|-----------|------|-------|----|-------|
|
||||
| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.143 (DHCP) | Carries the RPC path to Dankserver since 2026-08-17. Default route via 192.168.0.1. |
|
||||
| enx00e04cc2e1e5 | USB-C to Ethernet adapter | 1000 Mb/s | 192.168.0.116 | UNPLUGGED 2026-08-17. While present, the route to 192.168.0.235 went through this adapter (lower route metric), so all RPC traffic was capped at 1 GbE. See [[Pipeline Parallelism/Ethernet Bandwidth Math]] section 0. |
|
||||
| docker0 | bridge | - | 172.17.0.1 | Usually down. |
|
||||
|
||||
The L2 gateway/switch at 192.168.0.1 (MAC f0:4b:8a:65:c5:40) is at
|
||||
least 2.5 GbE-capable. Whether it is 10G-capable is unverified — check
|
||||
the device itself before buying 10G USB-C adapters; otherwise a direct
|
||||
cable between claw and server is required for 10G.
|
||||
|
||||
### OS and Software
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| OS | Ubuntu 26.04 LTS (Resolute Raccoon) |
|
||||
| Kernel | 7.0.0-29-generic |
|
||||
| Python | 3.14.4 (system), 3.12.13 (CIRU venv) |
|
||||
| CIRU runtime | ciru-ai/ROCmFPX v2.3 (llama-server, ROCm 7.15 SDK) |
|
||||
| uv | 0.12.2 |
|
||||
|
||||
## CIRU Runtime Relevance
|
||||
|
||||
- Unified memory eliminates separate VRAM limits. The GPU uses system RAM.
|
||||
- The GPU budget is the rocm-smi figure: 64 GiB VRAM reported by rocm-smi (SVM-backed). The older 96 GiB figure was the addressable SVM pool, not the budget. Trust rocm-smi and the runtime; do not use `free` or `dmidecode` for the GPU budget.
|
||||
- The active service is `frank-default-large` on port 7770 (moved from 52625 on 2026-08-22): CIRU ROCmFPX llama.cpp serving kingjones/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP (Q6 AGENT dense, 262144 ctx, standalone MTP Q4_0 head). See [[kingjones-strix-mtp]].
|
||||
- The runtime requires ROCm 7.15 (model card requirement). The system ROCm is 7.2.4; the retired ling3-ciru venv bundles a self-contained TheRock ROCm 7.15 SDK that the serve script puts first in `LD_LIBRARY_PATH`.
|
||||
- The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18.
|
||||
- 1.1 TB free disk space is ample for model storage.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[ciru-activefpx-promptforge]]
|
||||
- [[Shared Memory Architecture Dankclaw]]
|
||||
- [[frank-redo-2026-08-17]]
|
||||
- [[FastFlowLM NPU Inference]]
|
||||
@@ -0,0 +1,77 @@
|
||||
# Dankserver System Specs
|
||||
|
||||
## Server Overview
|
||||
|
||||
Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.04 (Resolute Raccoon) with an RTX 3090 and 62 GB of system RAM.
|
||||
|
||||
## Hardware Specifications
|
||||
|
||||
### CPU
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Architecture | x86_64 |
|
||||
| Vendor | AuthenticAMD |
|
||||
| Model | AMD Ryzen 7 7700X |
|
||||
| Cores | 8 (1 socket) |
|
||||
| NUMA Nodes | 1 (node 0, 63503 MB) |
|
||||
|
||||
### GPU
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| Model | NVIDIA GeForce RTX 3090 |
|
||||
| VRAM | 24576 MiB (24 GB) |
|
||||
| Driver | 595.84 |
|
||||
| CUDA Version | 13.2 |
|
||||
| CUDA Toolkit | 12.0 (nvcc) |
|
||||
|
||||
### Memory
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| System RAM | 62 GiB |
|
||||
| Swap | 127 GiB |
|
||||
| /dev/shm | 32 GiB |
|
||||
|
||||
### Storage
|
||||
|
||||
| Filesystem | Size | Used | Avail | Use% |
|
||||
|------------|------|------|-------|------|
|
||||
| / (nvme0n1p2) | 915G | 729G | 140G | 84% |
|
||||
| /mnt/entropy (dm-0) | 91T | 86T | 910G | 99% |
|
||||
|
||||
### Network
|
||||
|
||||
| Interface | Type | Speed | IP | Notes |
|
||||
|-----------|------|-------|----|-------|
|
||||
| eno1 | onboard | 2500 Mb/s (2.5 GbE) | 192.168.0.235 (DHCP) | RPC worker endpoint (port 50052). Default route via 192.168.0.1. |
|
||||
| enxe8ea6ab7ebca | USB-C class adapter | 2500 Mb/s | 192.168.0.252 | UNPLUGGED 2026-08-17. Was idle (unused by the RPC path). |
|
||||
| wlp8s0 | WiFi | - | - | Unused (carrier down). |
|
||||
| docker0 | bridge | - | 172.17.0.1 | Down; internal docker traffic only. |
|
||||
|
||||
### OS and Software
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| OS | Ubuntu 26.04 LTS (Resolute Raccoon) |
|
||||
| Kernel | 7.0.0-30-generic |
|
||||
| Python | 3.14.4 |
|
||||
|
||||
### Active Services
|
||||
|
||||
The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors selected by `--override-tensor`; every full-attention block plus blocks 35–40 remain on CUDA. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]].
|
||||
|
||||
## Hardware Relevance
|
||||
|
||||
- 24 GB VRAM hosts the Q6_K non-expert tensors and the full-attention/final-layer expert placement. The context pool contains 491520 f16 cells across three 163840-token slots. The service uses ngram-mod only and `ubatch=1024`. Verify actual headroom after each model restart with `nvidia-smi` and the service journal.
|
||||
- 62 GB system RAM holds the remaining CPU-resident expert tensors with room to spare.
|
||||
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
|
||||
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
|
||||
|
||||
## See Also
|
||||
|
||||
- [[VLLM Overview]]
|
||||
- [[Hybrid CPU GPU Processing Dankserver]]
|
||||
- [[RAM CPU Offloading in VLLM]]
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
|
||||
@@ -0,0 +1,324 @@
|
||||
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
|
||||
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
|
||||
> [deployment.md](deployment.md) for the active configuration.
|
||||
> The `frank-default-small` name and port 52625 now belong to the vLLM
|
||||
> service on Dankclaw (nameistoken Quark W8A8); this page is archive.
|
||||
|
||||
---
|
||||
type: research
|
||||
title: FastFlowLM NPU Inference
|
||||
description: FastFlowLM (FLM), the AMD NPU-first LLM runtime for Ryzen AI XDNA2 chips. Fit analysis for the 50 TOPS NPU on Dankclaw, published benchmarks, verified install state, and a test runbook.
|
||||
tags:
|
||||
- fastflowlm
|
||||
- npu
|
||||
- xdna2
|
||||
- dankclaw
|
||||
- research
|
||||
timestamp: 2026-08-16
|
||||
---
|
||||
|
||||
# FastFlowLM NPU Inference
|
||||
|
||||
FastFlowLM (FLM) runs LLMs entirely on the AMD Ryzen AI NPU. It targets XDNA2 chips: Strix, Strix Halo, Kraken, and Gorgon Point. The project describes itself as "Ollama, but deeply optimized for NPUs." As of 2026-08-16, FLM is installed on Dankclaw, the NPU validates, and `frank-default-small` (port 52625) serves in production. See Promotion.
|
||||
|
||||
## Verdict (2026-08-16)
|
||||
|
||||
- The NPU is a complement, not a replacement. It fits dense models up to about 9B.
|
||||
- Qwen3.8-27B (dense) cannot run on the NPU. No FLM build exists and the memory budget is too small.
|
||||
- Qwen3.6-35B-A3B has an official FLM build (Q4_KS). AMD benchmarks it on a 96 GB machine. It probably does not fit the 30 GiB budget of Dankclaw.
|
||||
- Realistic first candidates: Qwen3.5-9B, Qwen3-8B, gpt-oss-20b (at the budget edge).
|
||||
- **Landed 2026-08-16:** gpt-oss-20b serves on the NPU at 19.3 t/s decode, concurrently with the PP head. See Measured Results.
|
||||
- The memlock limit was the setup blocker. Fixed 2026-08-16 via `/etc/security/limits.d/99-flm-memlock.conf`.
|
||||
|
||||
## What FastFlowLM Is
|
||||
|
||||
FLM is an NPU-first inference runtime. It started as a 2025 university project (University of Rhode Island, Ken Qing Yang). The team joined AMD on 2026-07-17. The repository moved to the ROCm organization on 2026-08-11 as v1.0.0.
|
||||
|
||||
Key properties, per the project README and docs:
|
||||
|
||||
- Runs fully on the NPU. No GPU or CPU load during inference.
|
||||
- CLI mode (`flm run <tag>`) and server mode (`flm serve <tag>`).
|
||||
- Server mode exposes a REST API and an OpenAI-compatible API. Default port is 52625.
|
||||
- Models are not GGUF. FLM downloads pre-compiled kernel packs (built with AMD IRON + AIE-MLIR) from Hugging Face.
|
||||
- Context lengths up to 256k tokens on listed models.
|
||||
- Vision, audio, embedding, and MoE model support.
|
||||
- Runtime is about 17 MB. Orchestration code is MIT-licensed. Kernels are free for commercial use.
|
||||
- Models store in `~/.config/flm/` (Linux). Override with `FLM_MODEL_PATH`.
|
||||
|
||||
## NPU Hardware on Dankclaw
|
||||
|
||||
Dankclaw runs an AMD Ryzen AI MAX+ 395 (Strix Halo). The chip integrates an XDNA2 NPU rated at 50 TOPS. It sits beside the Radeon 8060S iGPU and shares the same 30 GiB unified memory pool.
|
||||
|
||||
Live verification on Dankclaw, 2026-08-16:
|
||||
|
||||
```
|
||||
$ flm validate
|
||||
[FLM] Fetching models from: /opt/fastflowlm/share/flm/model_list.json
|
||||
[Linux] Kernel: 7.0.0-29-generic
|
||||
[Linux] NPU: /dev/accel/accel0 with 8 columns
|
||||
[Linux] NPU FW Version: 1.1.2.65
|
||||
[Linux] amdxdna version: 0.7
|
||||
[ERROR] Memlock limit is too low (8MB). Please raise the limit or set to infinity.
|
||||
```
|
||||
|
||||
- Kernel 7.0.0-29-generic ships the `amdxdna` driver in-tree. The driver is loaded.
|
||||
- Firmware 1.1.2.65 meets the 1.1.0.0 minimum.
|
||||
- The XRT runtime packages (`libxrt2`, `libxrt-npu2`) are installed.
|
||||
- The only failure is the memlock ulimit (8 MB). FLM needs `unlimited`.
|
||||
- Ubuntu 26.04 is not on the FLM supported-distro list (24.04, 25.10, Arch, generic). Validation passes anyway. Treat 26.04 as "generic Linux, working".
|
||||
|
||||
## Memory Model
|
||||
|
||||
The NPU allocates from system RAM. AMD/Microsoft enforce an internal cap of about half of total system RAM for NPU use. The gpt-oss model card states a cap of about 15.6 GB on a 32 GB machine, which is half.
|
||||
|
||||
Budget math for this fleet:
|
||||
|
||||
| Machine | RAM | NPU budget (approx. half) |
|
||||
|---------|-----|---------------------------|
|
||||
| Dankclaw (Strix Halo) | 30 GiB | ~15 GiB |
|
||||
| Dankserver (Ryzen 7 7700X) | 62 GB | no NPU |
|
||||
| FLM bench box (Kraken Point) | 32 or 96 GB | ~16 or ~48 GB |
|
||||
|
||||
Weight sizes (approximate, Q4-class):
|
||||
|
||||
| Model | Quant | Weights | Fits ~15 GiB budget? |
|
||||
|-------|-------|---------|----------------------|
|
||||
| Qwen3.5-9B | Q4_1 | ~5.5 GB | yes |
|
||||
| Qwen3-8B | Q4_1 | ~5 GB | yes |
|
||||
| gpt-oss-20b (MoE, 3.6B active) | Q4_1 | ~15.1 GB | at the edge |
|
||||
| Qwen3.6-35B-A3B (MoE, 3B active) | Q4_KS | ~20 GB | no, unproven at 30 GiB |
|
||||
| Qwen3.8-27B (dense) | any Q4 | ~16 GB+ | no build exists |
|
||||
|
||||
Do not load a model larger than the budget first. An oversized load can starve the unified pool and crash the box while production services still run.
|
||||
|
||||
## Supported Models (Qwen family, plus gpt-oss)
|
||||
|
||||
All Qwen cards use Q4_1 unless noted. Context lengths are the FLM-published maxima.
|
||||
|
||||
| Tag | Base model | Context max | Notes |
|
||||
|-----|-----------|-------------|-------|
|
||||
| `qwen3:0.6b` | Qwen3-0.6B | 32k | thinking toggle |
|
||||
| `qwen3:1.7b` | Qwen3-1.7B | 32k | thinking toggle |
|
||||
| `qwen3:4b` | Qwen3-4B | 32k | tool calling |
|
||||
| `qwen3:8b` | Qwen3-8B | 32k | tool calling |
|
||||
| `qwen3-tk:4b` | Qwen3-4B-Thinking-2507 | 256k | thinking |
|
||||
| `qwen3-it:4b` | Qwen3-4B-Instruct-2507 | 256k | tool calling |
|
||||
| `qwen3vl-it:4b` | Qwen3-VL-4B-Instruct | 256k | vision |
|
||||
| `qwen2.5-it:3b` | Qwen2.5-3B-Instruct | 32k | |
|
||||
| `qwen2.5vl-it:3b` | Qwen2.5-VL-3B-Instruct | 256k | vision |
|
||||
| `qwen3.5:0.8b` | Qwen3.5-0.8B | 256k | vision, thinking toggle |
|
||||
| `qwen3.5:2b` | Qwen3.5-2B | 256k | vision, tool calling |
|
||||
| `qwen3.5:4b` | Qwen3.5-4B | 256k | vision, tool calling |
|
||||
| `qwen3.5:9b` | Qwen3.5-9B | 256k | vision, tool calling |
|
||||
| `qwen3.6-moe:35b-a3b` | Qwen3.6-35B-A3B | 256k | **Q4_KS**, MoE, vision, thinking, tool calling |
|
||||
| `gpt-oss:20b` | gpt-oss-20b | 128k | Q4_1, ~15.1 GB in FLM |
|
||||
|
||||
Other families: LLaMA, DeepSeek, Gemma, MedGemma, TranslateGemma, LiquidAI/LFM, Phi, Nanbeige, Whisper, EmbeddingGemma, SmolVLA.
|
||||
|
||||
## Published Benchmarks (FLM, vendor numbers)
|
||||
|
||||
All vendor results come from AMD Ryzen AI 7 350 (Kraken Point, 40 TOPS NPU) boxes under the default NPU power mode (Performance). Our NPU is a 50 TOPS Strix Halo XDNA2. Treat these numbers as directional, not as our expected speed.
|
||||
|
||||
Qwen3.5, FLM v0.9.38, Kraken Point 32 GB DRAM. Decode t/s by starting context length:
|
||||
|
||||
| Model | 1k | 2k | 4k | 8k | 16k | 32k |
|
||||
|-------|-----:|-----:|-----:|-----:|-----:|-----:|
|
||||
| Qwen3.5-0.8B | 39.2 | 38.0 | 36.3 | 33.1 | 28.1 | 21.6 |
|
||||
| Qwen3.5-2B | 26.8 | 26.2 | 25.4 | 23.7 | 21.3 | 17.0 |
|
||||
| Qwen3.5-4B | 15.0 | 14.6 | 14.2 | 13.3 | 11.8 | 9.6 |
|
||||
| Qwen3.5-9B | 9.3 | 9.2 | 9.0 | 8.5 | 7.8 | 6.9 |
|
||||
|
||||
Prefill t/s for the same run:
|
||||
|
||||
| Model | 1k | 2k | 4k | 8k | 16k | 32k |
|
||||
|-------|-----:|-----:|-----:|-----:|-----:|-----:|
|
||||
| Qwen3.5-0.8B | 983 | 1257 | 1471 | 1584 | 1579 | 1447 |
|
||||
| Qwen3.5-2B | 803 | 1004 | 1142 | 1223 | 1225 | 1151 |
|
||||
| Qwen3.5-4B | 378 | 440 | 479 | 493 | 487 | 450 |
|
||||
| Qwen3.5-9B | 284 | 333 | 362 | 379 | 378 | 357 |
|
||||
|
||||
Qwen3.6-35B-A3B (Q4_KS), FLM v0.9.45, Kraken Point **96 GB DRAM**:
|
||||
|
||||
| Metric | 1k | 2k | 4k | 8k | 16k | 32k |
|
||||
|--------|-----:|-----:|-----:|-----:|-----:|-----:|
|
||||
| Decode t/s | 13.65 | 13.41 | 13.09 | 12.51 | 11.24 | 9.51 |
|
||||
| Prefill t/s | 78.98 | 118.04 | 156.43 | 197.93 | 218.84 | 221.96 |
|
||||
|
||||
Context: the current PP split (head + 3090 worker) decodes at 14-18 t/s for the 27B dense model. A 35B-A3B at ~10-14 t/s on an NPU with 48 GB of RAM budget is not a speed win over the existing topology.
|
||||
|
||||
The vendor also claims "over 10x more power-efficient" versus GPU. That is a marketing claim. We have not measured it.
|
||||
|
||||
## Fit to the Frank Program
|
||||
|
||||
| Our model | NPU verdict |
|
||||
|-----------|-------------|
|
||||
| Qwen3.8-27B UD-Q8_K_XL (Dankclaw, dense) | No. No FLM build. ~16 GB+ even at Q4. Exceeds the ~15 GiB budget. |
|
||||
| Qwen3.6-35B-A3B (Dankserver, MoE) | Unproven. Official build exists. Vendor bench ran on 96 GB RAM. A 30 GiB box has ~15 GiB for the NPU. Test last, with the service stopped. |
|
||||
| Qwen3.5-9B / Qwen3-8B | Yes, the realistic tier. ~5-6 GB. Leaves room for KV and the OS. |
|
||||
| gpt-oss-20b (MoE) | **Landed.** Weights are 13.8 GB on disk (q4nx). The model holds ~15 GiB resident on the NPU. Decode is 19.3 t/s. See Measured Results. |
|
||||
|
||||
Proposed role: a small, always-on, low-power endpoint for triage, summarization, embeddings, or agent side-tasks on the Dankclaw NPU. It frees the iGPU and the 3090 for the 27B/35B workloads. It does not replace either.
|
||||
|
||||
## Install State on Dankclaw (verified 2026-08-16)
|
||||
|
||||
- `fastflowlm 1.0.1` installed (deb, upgraded from 0.9.45-12-ga6bd056 on 2026-08-17). `flm` at `/usr/bin/flm`. The 0.9.45 runtime refused the qwythos pack, which declares `flm_version` 1.0.0.
|
||||
- `libxrt2` / `libxrt-npu2` (Xilinx Runtime) installed.
|
||||
- `/dev/accel/accel0` present. `flm validate` passes fully.
|
||||
- Memlock fixed: `/etc/security/limits.d/99-flm-memlock.conf` sets soft and hard memlock to unlimited. A fresh sshd session picks it up via pam_limits. No reboot needed.
|
||||
- `gpt-oss:20b` pulled (4/4 files verified, 13.8 GB `model.q4nx` + tokenizer) into `~/.config/flm/models/GPT-OSS-20B-NPU2/`.
|
||||
- `qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2) installed 2026-08-17 via `flm-add.py` (downloaded from the HF repo; stdlib-only, no pip needed). Tag registered in the user registry `~/.config/flm/model_list.json` (a copy of the v1.0.1 system registry + the custom entry); kernels symlinked `~/.config/flm/xclbins/Huihui-...-NPU2 -> /opt/fastflowlm/share/flm/xclbins/Qwen3.5-9B-NPU2`. The service env exports `FLM_CONFIG_PATH` and `FLM_XCLBIN_PATH` so the tag resolves (without them flm hits the unknown-tag fallback, see the incident below).
|
||||
- `flm serve gpt-oss:20b --host 0.0.0.0 -p 52625 --pmode performance` runs as a nohup process. It is not a systemd unit. It does not survive a reboot.
|
||||
- The `llama-server` (frank-default-large, the PP head) holds an fd on `/dev/accel/accel0` from its own startup (device enumeration, not compute). FLM opened the same device concurrently without conflict. Verified live 2026-08-16.
|
||||
|
||||
## Measured Results (2026-08-16, gpt-oss-20b on the Dankclaw NPU)
|
||||
|
||||
Conditions: `--pmode performance`, PP head (27B, port 7770) serving concurrently, 5.5 GiB system RAM headroom steady-state, NPU resident footprint ~15 GiB.
|
||||
|
||||
| Run | Prompt tok | Compl tok | Wall s | Prefill t/s | Decode t/s | KV occ % |
|
||||
|-----|-----------:|----------:|-------:|------------:|-----------:|---------:|
|
||||
| 1 (smoke) | 74 | 63 | 7.2 | 19.5 | 19.3 | 1.7 |
|
||||
| 2 (160-tok prompt) | 161 | 256 | 17.4 | 40.8 | 19.3 | 5.1 |
|
||||
| 3 (95-tok prompt, 512 out) | 95 | 512 | 30.8 | 24.3 | 19.3 | n/a |
|
||||
|
||||
Decode is flat at 19.3 t/s across all three runs. Output includes `reasoning_content` (gpt-oss is a reasoning model). At `max_tokens 256` one run spent its whole budget thinking and returned empty content. Give it 500+ tokens or strip reasoning in the client.
|
||||
|
||||
Long-context prefill is not yet measured (runs 1-3 used under 200 prompt tokens). The 35B-A3B step remains untested.
|
||||
|
||||
## Model Bakeoff (measured 2026-08-16, 4 models)
|
||||
|
||||
All runs on the Dankclaw NPU (50 TOPS Strix Halo), single stream, `--pmode performance`, PP head (frank-default-large) serving concurrently on 7770. Decode t/s is the FLM-reported rate. Vision accuracy = ground-truth elements out of 4 (text SPECTRUM, red circle left, blue square right, three green triangles bottom).
|
||||
|
||||
| Model | Tag | Weights | Decode t/s (short) | 8k prefill t/s | 8k decode t/s | Vision decode t/s | Vision accuracy | Max ctx (registry) |
|
||||
|-------|-----|---------|-------------------:|---------------:|--------------:|------------------:|----------------|--------------------|
|
||||
| Gemma4-E4B | `gemma4-it:e4b` | 6.8 GB + 912 MB audio | 12.2 | 582 | 9.9 | 11.7 | 4/4 | 32768 |
|
||||
| Qwen3.5-9B | `qwen3.5:9b` | 7.3 GB | 7.2 | 343 | 6.6 | 7.3 | 4/4 | 32768 |
|
||||
| GPT-OSS-20B | `gpt-oss:20b` | 13.8 GB | 19.3 | 378 | 16.9 | n/a (no vision) | 0/4 | **8192** |
|
||||
| Qwen3.6-35B-A3B | `qwen3.6-moe:35b-a3b` | 22.2 GB | 12.7 | 207 | 11.2 | 12.7 | 4/4 | 32768 |
|
||||
|
||||
Notes:
|
||||
|
||||
- gpt-oss:20b is the fastest decoder (MoE, 3.6B active) but its KV cache fills at ~8k tokens (98.85% occupancy at 7,926; requests past ~8k return 400). It has no vision encoder.
|
||||
- 35B-A3B loads 28-29 GiB resident. It works but leaves ~1.5 GiB headroom with the PP head running. Do not run it alongside the head at full load.
|
||||
- Vision prefill for the two vision models with ~720 token image input: gemma 141 t/s, qwen3.5 176 t/s, 35B-A3B 68 t/s.
|
||||
- All four comply with the same image at 4/4 elements except gpt-oss, which cannot see it.
|
||||
|
||||
## Promotion (2026-08-16)
|
||||
|
||||
`frank-default-small` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-default-small.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-default-small.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
|
||||
|
||||
### Boot policy (2026-08-17)
|
||||
|
||||
The `frank-default-small` unit is disabled for boot. Only the PP pair (`frank-default-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
|
||||
|
||||
```
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable frank-default-small
|
||||
```
|
||||
|
||||
Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).
|
||||
|
||||
The PP split was renamed `frank-pp-head` → `frank-default-large` (2026-08-16, commit `48142f5`).
|
||||
|
||||
Model swap (2026-08-16, commit `50c55ee`): `FLM_MODEL=qwen3.6-moe:35b-a3b`, `FLM_CTX_LEN=262144` (native max), `FLM_PMODE=turbo`, `FLM_ASR=1` kept. Decode measured 12.5-13.1 t/s. The Gemma4 live config above (131072 ctx) applied to `gemma4-it:e4b`. Swap `FLM_MODEL` and `FLM_CTX_LEN` back to restore it.
|
||||
|
||||
Model swap (2026-08-17, commit `c85ff6c`): `FLM_MODEL=qwythos:9b` (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated-NPU2, Q4NX 7.76 GB). The pack installs through `flm-add` (user registry `~/.config/flm/model_list.json` + user xclbins `~/.config/flm/xclbins`, kernels symlinked from official `Qwen3.5-9B-NPU2`). Runtime upgraded to FLM 1.0.1 first (see Install State). `FLM_CTX_LEN=262144` native max (Qwen3.5-9B hybrid: 8 of 32 layers full attention, so KV stays small; occupancy 0.008% at 21 tokens). `FLM_PMODE=turbo`, `FLM_ASR=1` kept. The env file now exports `FLM_CONFIG_PATH` / `FLM_XCLBIN_PATH` — required, or flm treats the tag as unknown and silently serves llama3.2:1b. `qwen3.6-moe:35b-a3b` files stay installed for rollback. pi client entry (`~/.pi/agent/models.json` on dankware): id `qwythos:9b`, ctx 262144, input `[text]` (pack has no vision weights), compat `supportsDeveloperRole: false` kept.
|
||||
|
||||
## Parallel Streams (measured 2026-08-16)
|
||||
|
||||
The NPU serializes generations. It runs exactly one stream at a time. Extra streams queue in FIFO order. FLM server flags `--socket` (default 10) and `--q-len` (default 10) cap queue depth only. They do not add concurrency.
|
||||
|
||||
Concurrent-request test, gpt-oss-20b, 128-token cap per stream:
|
||||
|
||||
| Streams | Wall s | Aggregate t/s | Per-stream wall s |
|
||||
|--------:|-------:|--------------:|-------------------|
|
||||
| 1 | 10.6 | 12.1 | 10.6 |
|
||||
| 2 | 21.1 | 12.0 | 10.8 / 21.1 |
|
||||
| 4 | 41.1 | 11.6 | 10.9 / 21.6 / 32.4 / 41.1 |
|
||||
|
||||
The 4-stream wall times form a near-perfect 10.7 s ladder. Each stream self-reports 19.3 t/s decode, counting only its own active slices. Trust per-stream wall time, not the self-reported speed, under concurrency.
|
||||
|
||||
Consequence: the NPU endpoint is a single lane. It suits low-frequency background work (triage, summarization, agent side-tasks). It does not suit interactive chat for several simultaneous users. The 27B llama.cpp service, by contrast, runs PARALLEL=2 with true concurrent slots.
|
||||
|
||||
## Test Runbook
|
||||
|
||||
Do not run large model loads while the production service holds most of the 30 GiB pool. Check free RAM first or stop the service. The NPU shares the unified pool with everything else on the box.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[1. Raise memlock to unlimited] --> B[2. flm validate + xrt-smi examine]
|
||||
B --> C{NPU visible to XRT?}
|
||||
C -- no --> D[Fix driver or firmware path]
|
||||
C -- yes --> E[3. flm run qwen3.5:4b smoke test, /verbose]
|
||||
E --> F{Generates tokens?}
|
||||
F -- no --> D
|
||||
F -- yes --> G[4. flm serve on a scratch port, never 7770]
|
||||
G --> H[5. Bench decode + prefill t/s]
|
||||
H --> I{Speed acceptable?}
|
||||
I -- no --> J[Record results, stop]
|
||||
I -- yes --> K[6. Attempt gpt-oss:20b]
|
||||
K --> L{Loads in ~15 GiB?}
|
||||
L -- no --> J
|
||||
L -- yes --> M[Bench, record]
|
||||
M --> N[7. Last: qwen3.6-moe:35b-a3b, service stopped]
|
||||
N --> O[Record, decision]
|
||||
```
|
||||
|
||||
1. Raise memlock. Add `* soft memlock unlimited` and `* hard memlock unlimited` to `/etc/security/limits.conf`. Then reboot. For a systemd unit, set `LimitMEMLOCK=infinity` instead.
|
||||
2. Verify the NPU. Run `flm validate`. Run `xrt-smi examine`. Both must show the NPU. `flm validate` uses the kernel device. `flm run` uses XRT. A passing validate does not guarantee XRT sees the device.
|
||||
3. Smoke test. Run `flm run qwen3.5:4b`. Toggle `/verbose` for performance reporting. Type `/bye` to exit.
|
||||
4. Server mode. Run `flm serve qwen3.5:4b`. The default port is 52625. Use a scratch port if 52625 is taken. Never 7770.
|
||||
5. Bench. Measure decode and prefill t/s at 1k/4k/16k starting context. Record to this page.
|
||||
6. Step up in size: 9B, then gpt-oss-20b.
|
||||
7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load.
|
||||
|
||||
## Incident: qwen3.6-moe unreachable (2026-08-16)
|
||||
|
||||
Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.
|
||||
|
||||
1. **EXEC 203 crash loop (14:19-14:20).** The exec bit was lost on `serve-flm-small.sh` on the target. The unit failed `status=203/EXEC: Permission denied` four times, then started after the bit was restored. The git mode is `100755` (commit `3f42210`), and a `git pull` restores the bit. Verify with `git ls-files -s scripts/serve-flm-small.sh`.
|
||||
2. **Wrong model ID (14:29-14:52).** The client config still used the model ID `frank-default-small` (a service alias). FLM rejects that tag with `[ERROR] Model tag 'frank-default-small' is not supported.` Each such request also triggered the unknown-tag fallback below.
|
||||
3. **Swap churn (14:48-15:03).** The swap `gemma4-it:e4b` → `qwen3.6-moe:35b-a3b` restarted the service several times. Each load of the 23 GB model takes about 1.5-2.5 minutes at 262144 ctx. The endpoint does not serve during a load.
|
||||
|
||||
Fix: the client model ID must be the FLM registry tag `qwen3.6-moe:35b-a3b`. The working pi agent entry in `~/.pi/agent/models.json` is `baseUrl http://192.168.0.143:52625/v1`, `id qwen3.6-moe:35b-a3b`, `contextWindow 262144`, `reasoning true`, `input [text, image]`. Verified end-to-end from dankware (192.168.0.226) at 15:32 on 2026-08-16.
|
||||
|
||||
### Unknown-tag fallback (FLM behavior)
|
||||
|
||||
When a request names a tag that is not in the FLM registry, FLM does not return a clean 404. It logs the ERROR, then falls back to the default model `llama3.2:1b`:
|
||||
|
||||
- If the `llama3.2:1b` files are missing, FLM downloads them (1.2 GB on 2026-08-16) while the request hangs.
|
||||
- FLM then loads the 1B model in place of the requested model.
|
||||
- The files now exist in `~/.config/flm/models/Llama-3.2-1B-NPU2` (1.3 GB) on dankclaw. A future unknown-tag request loads the 1B model silently, with no download and no error.
|
||||
|
||||
Keep client model IDs as FLM registry tags. Never send a service alias. Optional cleanup: move `~/.config/flm/models/Llama-3.2-1B-NPU2` (an incident artifact) to `.AGENTTRASH` so a stray tag re-downloads loudly instead of serving the wrong model.
|
||||
|
||||
### Request compatibility: the `developer` role (found 2026-08-16)
|
||||
|
||||
pi sessions failed against this endpoint with `Stream ended without finish_reason` while curl worked. Root cause, verified by field bisection:
|
||||
|
||||
- The FLM Qwen chat template accepts only the roles `system`, `user`, and `assistant`. Any other role raises `Unexpected message role.` in the template.
|
||||
- FLM reports the error differently per mode. Non-streaming: HTTP 500 with the template traceback. Streaming: HTTP 200, and the error JSON is the first body. A streaming client sees a stream with no chunks and no `finish_reason`.
|
||||
- pi sends `role: "developer"` when `model.reasoning && compat.supportsDeveloperRole` (pi source, `openai-completions.js`). The default `supportsDeveloperRole` is true for standard OpenAI-compatible providers, so a reasoning model without a `compat` block always triggers the bug.
|
||||
- Fix: `"compat": { "supportsDeveloperRole": false }` on the model entry in `~/.pi/agent/models.json`. pi then sends `role: "system"`. Verified with a headless pi session: full round trip, `finish_reason: stop`.
|
||||
- FLM ignores unknown body fields. `reasoning_effort`, `store`, `max_completion_tokens`, and `tools: []` all arrive in pi requests and pass through without error.
|
||||
|
||||
## Risks and Open Questions
|
||||
|
||||
- **Ubuntu 26.04 is unofficial.** The distro list stops at 25.10. Validation passes, but breakage would fall on the "generic Linux" path.
|
||||
- **The 30 GiB budget is untested at the very top.** gpt-oss-20b (15 GiB resident) works. 35B-A3B (~20 GB weights) is still an unknown. Vendor data covers 32 GB and 96 GB boxes only.
|
||||
- **The FLM serve process is not under systemd.** A reboot of Dankclaw kills it. The PP head service is enabled and returns. The FLM process must be restarted by hand. A unit file is a natural next step.
|
||||
- **gpt-oss reasoning budget.** The model can spend its full token budget on `reasoning_content` and return empty content. Clients must handle or cap this.
|
||||
- **No llama.cpp GGUF path.** FLM uses its own pre-compiled kernels. Our UD-Q8_K_XL and HauhauCS builds are not importable. NPU work means FLM-specific model tags.
|
||||
- **NPU production role (2026-08-16).** `frank-default-small` serves `qwen3.6-moe:35b-a3b` on port 52625 (swap churn and the unknown-tag incident are recorded above). The remaining NPU content (benchmarks, runbook steps) is still research until a step lands and is recorded.
|
||||
|
||||
## Citations
|
||||
|
||||
1. ROCm/FastFlowLM README (chip support, Linux support date, ROCm move, quick start, model storage paths): https://github.com/ROCm/FastFlowLM , accessed 2026-08-16.
|
||||
2. FastFlowLM docs overview: https://fastflowlm.com/docs/ , accessed 2026-08-16.
|
||||
3. FastFlowLM Qwen model cards (tags, quantization, context lengths): https://fastflowlm.com/docs/models/qwen/ , accessed 2026-08-16.
|
||||
4. FastFlowLM gpt-oss model card (15.1 GB weight size, ~15.6 GB NPU cap, 32 GB inconsistency note): https://fastflowlm.com/docs/models/gpt-oss/ , accessed 2026-08-16.
|
||||
5. FastFlowLM Qwen3.5 benchmarks (v0.9.38, Kraken Point 32 GB): https://fastflowlm.com/docs/benchmarks/qwen3.5_results/ , accessed 2026-08-16.
|
||||
6. FastFlowLM Qwen3.6 benchmarks (v0.9.45, Kraken Point 96 GB): https://fastflowlm.com/docs/benchmarks/qwen3.6_results/ , accessed 2026-08-16.
|
||||
7. AMD blog, "FastFlowLM Joins AMD to Advance AI Inference" (2026-07-17): https://www.amd.com/en/blogs/2026/fastflowlm-joins-amd-to-advance-ai-inference.html
|
||||
8. Wikipedia, "Zen 5" article, Strix Halo section (Ryzen AI MAX+ 395: XDNA2 NPU, 50 TOPS, 256-bit LPDDR5X) via local ZIM `wikipedia_en_all_maxi_2026-02`.
|
||||
9. Live system checks on Dankclaw, 2026-08-16: `flm validate` output, `dpkg -l` package list, kernel 7.0.0-29-generic, `/proc/3052096/fd` device fd inspection.
|
||||
@@ -0,0 +1,213 @@
|
||||
# Hybrid CPU GPU Processing Dankserver
|
||||
|
||||
## Overview
|
||||
|
||||
Dankserver uses NVIDIA GPU with system RAM for hybrid processing. vLLM splits model weights onto GPU and moves KV cache between GPU VRAM and system RAM. This allows larger models to run on GPUs with limited VRAM.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Model Weights
|
||||
|
||||
Model weights stay on GPU. vLLM loads the entire model into GPU memory. You cannot offload model weights to CPU. The GPU must hold all model parameters.
|
||||
|
||||
### KV Cache
|
||||
|
||||
KV cache moves between GPU and CPU:
|
||||
|
||||
1. Active KV cache pages stay in GPU VRAM
|
||||
2. Inactive pages swap to CPU RAM
|
||||
3. Swapped pages write to disk swap space if RAM is full
|
||||
4. Pages reload to GPU when needed
|
||||
|
||||
### Compute Distribution
|
||||
|
||||
- GPU handles all matrix multiplication and attention computation
|
||||
- CPU handles tokenization, detokenization, and input processing
|
||||
- CPU handles KV cache swap operations
|
||||
- GPU worker processes run on CPU cores via multiprocessing
|
||||
|
||||
## Optimal Settings for Dankserver
|
||||
|
||||
### GPU Memory Utilization
|
||||
|
||||
Set `--gpu-memory-utilization` based on your GPU:
|
||||
|
||||
| GPU | Recommended Value |
|
||||
|-----|-------------------|
|
||||
| 24 GB (RTX 3090/4090) | 0.7-0.8 |
|
||||
| 40 GB (A100) | 0.8-0.9 |
|
||||
| 80 GB (A100 80GB) | 0.85-0.95 |
|
||||
| 80 GB (H100) | 0.9-0.95 |
|
||||
|
||||
Lower values leave more GPU memory for KV cache. This reduces swap frequency.
|
||||
|
||||
### Swap Space / KV Offload
|
||||
|
||||
Recent vLLM releases removed the `--swap-space` engine argument. Use `--kv-offloading-size` to move KV cache to CPU RAM. In Podman, mount the host cache read-write for KV offload writes. The table below shows historical `--swap-space` guidance as a sizing reference.
|
||||
|
||||
| System RAM | Recommended Swap |
|
||||
|------------|------------------|
|
||||
| 32 GB | 8-16 GB |
|
||||
| 64 GB | 16-32 GB |
|
||||
| 128 GB | 32-64 GB |
|
||||
| 256 GB | 64-128 GB |
|
||||
|
||||
More swap space allows longer contexts and more concurrent requests.
|
||||
|
||||
### Tensor Parallelism
|
||||
|
||||
Use tensor parallelism for large models:
|
||||
|
||||
```
|
||||
vllm serve meta-llama/Llama-3.1-70B-Instruct \
|
||||
--tensor-parallel-size 4 \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--swap-space 16
|
||||
```
|
||||
|
||||
### KV Cache Data Type
|
||||
|
||||
Use FP8 KV cache for Ampere and newer GPUs:
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--swap-space 16
|
||||
```
|
||||
|
||||
FP8 KV cache reduces memory usage by 50% compared to BF16.
|
||||
|
||||
## NUMA Optimization
|
||||
|
||||
Dankserver may have multiple NUMA nodes. Optimize with:
|
||||
|
||||
```
|
||||
vllm serve model \
|
||||
--tensor-parallel-size 4 \
|
||||
--numa-bind
|
||||
```
|
||||
|
||||
This pins each GPU worker to its nearest NUMA node. This reduces memory access latency.
|
||||
|
||||
For explicit NUMA binding:
|
||||
|
||||
```
|
||||
vllm serve model \
|
||||
--tensor-parallel-size 4 \
|
||||
--numa-bind \
|
||||
--numa-bind-nodes 0 0 1 1
|
||||
```
|
||||
|
||||
## CPU Resource Requirements
|
||||
|
||||
VLLM needs CPU cores for:
|
||||
|
||||
- 1 API server process
|
||||
- 1 engine core process
|
||||
- N GPU worker processes (one per GPU)
|
||||
|
||||
Minimum physical cores = 2 + N (where N is GPU count).
|
||||
|
||||
For Dankserver with 1 GPU:
|
||||
|
||||
- Minimum 3 physical CPU cores
|
||||
- Recommended 8+ physical CPU cores
|
||||
|
||||
## Performance Tuning
|
||||
|
||||
### Chunked Prefill
|
||||
|
||||
Chunked prefill is enabled by default. Tune with:
|
||||
|
||||
```
|
||||
vllm serve model \
|
||||
--max-num-batched-tokens 8192
|
||||
```
|
||||
|
||||
Higher values improve throughput. Lower values improve latency.
|
||||
|
||||
### CUDA Graphs
|
||||
|
||||
CUDA graphs improve throughput. Disable for faster startup:
|
||||
|
||||
```
|
||||
vllm serve model \
|
||||
--enforce-eager
|
||||
```
|
||||
|
||||
### Compilation Cache
|
||||
|
||||
Reuse torch.compile artifacts:
|
||||
|
||||
```
|
||||
export VLLM_CACHE_ROOT=~/.cache/vllm
|
||||
vllm serve model
|
||||
```
|
||||
|
||||
Copy cache between boots for faster startup.
|
||||
|
||||
### Skip Memory Profiling
|
||||
|
||||
Skip memory profiling on repeated boots:
|
||||
|
||||
```
|
||||
vllm serve model \
|
||||
--kv-cache-memory <value_from_first_boot_log>
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
Check GPU memory:
|
||||
|
||||
```
|
||||
nvidia-smi
|
||||
```
|
||||
|
||||
Check for preemption warnings in VLLM logs.
|
||||
|
||||
Check CPU utilization:
|
||||
|
||||
```
|
||||
htop
|
||||
```
|
||||
|
||||
## Example Commands
|
||||
|
||||
### Single GPU, Medium Model
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--swap-space 16 \
|
||||
--max-num-batched-tokens 4096 \
|
||||
--max-num-seqs 128
|
||||
```
|
||||
|
||||
### Single GPU, FP8 Model
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--swap-space 16 \
|
||||
--max-num-batched-tokens 8192
|
||||
```
|
||||
|
||||
### Multi-GPU, Large Model
|
||||
|
||||
```
|
||||
vllm serve meta-llama/Llama-3.1-70B-Instruct \
|
||||
--tensor-parallel-size 4 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--swap-space 32 \
|
||||
--max-model-len 8192
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [[VLLM Overview]]
|
||||
- [[VLLM Index]]
|
||||
- [[RAM CPU Offloading in VLLM]]
|
||||
- [[Shared Memory Architecture Dankclaw]]
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
|
||||
@@ -0,0 +1,530 @@
|
||||
---
|
||||
type: reference
|
||||
title: Ethernet Bandwidth Math
|
||||
description: Bytes-per-token math for pipeline vs tensor parallelism, feasibility tables for 1/2.5/10 GbE, and real multi-node benchmark numbers.
|
||||
tags:
|
||||
- pipeline-parallelism
|
||||
- research
|
||||
timestamp: 2026-08-17
|
||||
---
|
||||
|
||||
# Bandwidth Analysis: Multi-Node LLM Inference over Ethernet
|
||||
|
||||
## 0. Measured on the Qwen3.8-27B PP deployment (2026-08-17)
|
||||
|
||||
Everything in this section was measured on the live deployment
|
||||
([[Qwen3.8 PP Deployment]]) while sampling `/proc/net/dev` at 50 Hz on
|
||||
both ends. The theoretical tables below use H=4096/L=32; the real model
|
||||
is larger (see constants), which raises the per-token bytes but not the
|
||||
conclusions.
|
||||
|
||||
### 0.1 Verified constants (from the GGUF + env, not guessed)
|
||||
|
||||
| Constant | Value | Source |
|
||||
| -------- | ----- | ------ |
|
||||
| `hidden_size` (embedding_length) | 5120 | GGUF metadata (`qwen35.embedding_length`) |
|
||||
| Blocks | 65 (45 on worker, 20 on head) | `--tensor-split 0.68,0.32` |
|
||||
| FA layers | 16 total, interval 4 (11 worker / 5 head) | GGUF `full_attention_interval` |
|
||||
| KV per FA layer per token | 3136 B (K q8_0 1088 + V f16 2048) | env file comment, confirmed |
|
||||
| ubatch | 256 | `UBATCH_SIZE` in `frank-default-large.env` |
|
||||
| MTP verify step positions | 5 (1 + n_max 4) | `SPEC_DRAFT_N_MAX=4` |
|
||||
| Link (old path) | 1 GbE via USB-C adapter on claw | `enx00e04cc2e1e5`, negotiated 1000 Mb/s |
|
||||
| Link (current path) | 2.5 GbE eno1 to eno1 | negotiated 2500 Mb/s both ends |
|
||||
| iperf3, 2.5 GbE path | 281 MB/s claw->server, 294 MB/s server->claw | measured |
|
||||
| ping RTT, 1 GbE USB path | min 0.239 / avg 0.742 / max 1.085 / mdev 0.334 ms | measured |
|
||||
| ping RTT, 2.5 GbE path | min 0.224 / avg 0.585 / max 0.883 / mdev 0.211 ms | measured |
|
||||
|
||||
The USB-C adapter added ~15 us to RTT (0.239 vs 0.224 ms min) and more
|
||||
jitter (mdev 0.334 vs 0.211 ms).
|
||||
|
||||
### 0.2 Measured wire bytes (the number that matters)
|
||||
|
||||
Raw hidden state per position per direction = 5120 x 2 = **10.0 KB**.
|
||||
The *measured* bytes per position per direction are ~4x that:
|
||||
|
||||
| Direction | Measured | Raw hidden |
|
||||
| --------- | -------- | ---------- |
|
||||
| head -> worker | 39.8 KB/position | 10.0 KB |
|
||||
| worker -> head | 38.4 KB/position | 10.0 KB |
|
||||
|
||||
The extra ~30 KB/position is the serialized RPC graph (per-position
|
||||
nodes for all 45 worker blocks: attention, KV append, SSM ops)
|
||||
plus protocol framing. It scales with position count just like the
|
||||
payload, so it does not shrink away.
|
||||
|
||||
Per ubatch (256 positions) the measured crossing is **20.5 MB**
|
||||
(10.4 MB head->worker + 10.1 MB worker->head).
|
||||
|
||||
### 0.3 Measured throughput and the validated model
|
||||
|
||||
Measured prefill (fresh 13,959-token prompt, 14k context, 2.5 GbE
|
||||
path): **457.9 t/s** (30,485 ms, journal task 11924). Decomposition:
|
||||
|
||||
| Term | 2.5 GbE (measured anchor) |
|
||||
| ---- | ------------------------- |
|
||||
| wire per ubatch (20.5 MB) | 70 ms |
|
||||
| compute + overhead per ubatch (residual) | 489 ms |
|
||||
| ubatch total | 559 ms = 2.18 ms/token |
|
||||
|
||||
Using that residual and only swapping the wire time per link speed:
|
||||
|
||||
| Link | wire/ubatch | predicted prefill | avg link use | line-rate burst duty |
|
||||
| ---- | ----------- | ----------------- | ------------ | -------------------- |
|
||||
| 1 GbE (old USB path) | 174 ms | **386 t/s** | 15.8% | 26.2% |
|
||||
| 2.5 GbE (current) | 70 ms | 458 t/s (matches measured) | 6.4% | 12.5% |
|
||||
| 10 GbE | 17 ms | **505 t/s** | 1.6% | 3.4% |
|
||||
|
||||
Delta: 1 -> 2.5 GbE is **+18.6% prefill**; 2.5 -> 10 GbE is **+9.4%**.
|
||||
|
||||
### 0.4 The KV snapshot: where the link really saturates
|
||||
|
||||
On task release, `llama-server` saves the conversation to the prompt
|
||||
cache. The KV for the 11 FA layers living on the worker GPU must be
|
||||
copied **worker -> head** for that snapshot, and the slot is BLOCKED
|
||||
while it runs (`PARALLEL=1` = one user queues behind it).
|
||||
|
||||
Measured for a 98,100-token conversation: 3,454 MB copied (model check:
|
||||
98100 x 11 x 3136 B = 3.38 GB, matches), at line rate (254 MB/s avg,
|
||||
308 MB/s peak).
|
||||
|
||||
| Link | snapshot duration (slot blocked) |
|
||||
| ---- | -------------------------------- |
|
||||
| 1 GbE | ~31 s |
|
||||
| 2.5 GbE | 13.6 s (measured; journal: "prompt cache update took 14418 ms") |
|
||||
| 10 GbE | ~5 s |
|
||||
|
||||
This is the only phase that runs the link at ~100% utilization for a
|
||||
long time. Prefill runs at 12-26% of line rate in bursts (per ubatch)
|
||||
and ~6-16% on average; decode runs at ~2% or less.
|
||||
|
||||
### 0.5 Decode (MTP verify step, 5 positions)
|
||||
|
||||
| Link | net time per step | share of 164 ms step | decode t/s |
|
||||
| ---- | ----------------- | -------------------- | ---------- |
|
||||
| 1 GbE | 4.0 ms (RTT 0.74 + wire 3.4) | 2.4% | ~23.0 |
|
||||
| 2.5 GbE | 1.9 ms (RTT 0.59 + wire 1.4) | 1.1% | 23.47 (measured) |
|
||||
| 10 GbE | 0.6 ms (RTT ~0.25 + wire 0.3) | 0.4% | ~23.5 |
|
||||
|
||||
Decode is latency-touched but bandwidth-trivial. Average wire rate
|
||||
~19 Mbit/s in each direction.
|
||||
|
||||
### 0.6 Verdict for the 10 GbE USB-C adapter question
|
||||
|
||||
- The 1 GbE -> 2.5 GbE upgrade (the big win) is already captured for
|
||||
free: the claw's native 2.5 GbE `eno1` now carries the RPC path
|
||||
(the 1 GbE USB adapter was unplugged 2026-08-17; routing moved the
|
||||
path, the RPC TCP re-attached on the new interface).
|
||||
- 2.5 -> 10 GbE buys +9.4% prefill and 13.6 s -> 5 s snapshot blocking.
|
||||
It requires the L2 device at 192.168.0.1 to be 10G-capable, or a
|
||||
direct cable between the two machines (dedicated link, second IP
|
||||
pair, or a lower-metric static route).
|
||||
- At current usage (PARALLEL=1, decode-heavy), 2.5 GbE is comfortable.
|
||||
|
||||
## 1. Bandwidth Math: Bytes per Token per Layer Boundary
|
||||
|
||||
### 1.1 Pipeline Parallelism (PP)
|
||||
|
||||
During autoregressive generation, each transformer layer produces a hidden state of shape `(batch_size, hidden_size)`. At a pipeline stage boundary, this tensor must be communicated between nodes.
|
||||
|
||||
**Forward pass only (inference):**
|
||||
```
|
||||
bytes_per_boundary = batch_size × hidden_size × 2 [fp16 = 2 bytes]
|
||||
```
|
||||
|
||||
**Key detail:** During *generation* (one token at a time), seq_len = 1 at each boundary crossing. The KV cache stays local. The hidden state is `(batch, hidden_size)` — independent of context length.
|
||||
|
||||
**For a 27B dense model (e.g., Qwen3.6-27B):**
|
||||
- Hidden size: H = 4096 (standard for Llama/Qwen-style architectures).
|
||||
Note: the Qwen3.8-27B actually deployed on this fleet has H = 5120
|
||||
(GGUF-verified); see section 0, which supersedes these theoretical
|
||||
numbers with measurements.
|
||||
- Per-token, per-boundary: `4096 × 2 = 8,192 bytes = 8 KB`
|
||||
- With Nₚₚ stage boundaries: total = `Nₚₚ × 8 KB/tok`
|
||||
|
||||
**For a 36B MoE model (e.g., Qwen3.6-35B-A3B):**
|
||||
- Hidden size: H = 6144 (common for this scale)
|
||||
- Layers: L = 48 (typical)
|
||||
- Per-token, per-boundary: `6144 × 2 = 12,288 bytes = 12 KB`
|
||||
- With Nₚₚ stage boundaries: total = `Nₚₚ × 12 KB/tok`
|
||||
|
||||
**Full example — 27B model, 2 PP stages (1 boundary), batch=1:**
|
||||
```
|
||||
1 boundary × 4096 × 2 bytes = 8,192 bytes/tok
|
||||
```
|
||||
|
||||
**Example — same model, batch=64:**
|
||||
```
|
||||
1 boundary × 4096 × 64 × 2 bytes = 524,288 bytes = 512 KB/tok
|
||||
```
|
||||
|
||||
**Example — 36B model, 2 PP stages, batch=1:**
|
||||
```
|
||||
1 boundary × 6144 × 2 bytes = 12,288 bytes = 12 KB/tok
|
||||
```
|
||||
|
||||
### 1.2 Tensor Parallelism (TP) — for comparison
|
||||
|
||||
In tensor parallelism, each MLP and attention layer performs an all-reduce on the hidden state:
|
||||
|
||||
```
|
||||
bytes_per_layer = batch_size × hidden_size × 2 [one all-reduce per layer]
|
||||
total_bytes_per_tok = L × batch_size × hidden_size × 2 [L all-reduces total]
|
||||
```
|
||||
|
||||
**For 27B, H=4096, L=32, TP=2, batch=1:**
|
||||
```
|
||||
32 layers × 4096 × 1 × 2 bytes = 262,144 bytes = 256 KB/tok
|
||||
(16 KB per layer all-reduce × 32 layers)
|
||||
```
|
||||
|
||||
**For 36B, H=6144, L=48, TP=2, batch=1:**
|
||||
```
|
||||
48 layers × 6144 × 1 × 2 bytes = 589,824 bytes = 576 KB/tok
|
||||
(12 KB per layer all-reduce × 48 layers)
|
||||
```
|
||||
|
||||
### 1.3 Scaling with batch size
|
||||
|
||||
| Model | H | L | tok/s | Batch | PP boundary bytes/tok | TP total bytes/tok |
|
||||
|-------|-----|-----|-------|-------|----------------------|--------------------|
|
||||
| 27B | 4096 | 32 | 10 | 1 | 8 KB | 256 KB |
|
||||
| 27B | 4096 | 32 | 50 | 1 | 8 KB | 256 KB |
|
||||
| 27B | 4096 | 32 | 100 | 1 | 8 KB | 256 KB |
|
||||
| 27B | 4096 | 32 | 10 | 16 | 128 KB | 4,096 KB |
|
||||
| 27B | 4096 | 32 | 50 | 16 | 128 KB | 4,096 KB |
|
||||
| 27B | 4096 | 32 | 100 | 16 | 128 KB | 4,096 KB |
|
||||
| 27B | 4096 | 32 | 10 | 64 | 512 KB | 16,384 KB |
|
||||
| 27B | 4096 | 32 | 50 | 64 | 512 KB | 16,384 KB |
|
||||
| 27B | 4096 | 32 | 100 | 64 | 512 KB | 16,384 KB |
|
||||
| 36B | 6144 | 48 | 10 | 1 | 12 KB | 576 KB |
|
||||
| 36B | 6144 | 48 | 50 | 1 | 12 KB | 576 KB |
|
||||
| 36B | 6144 | 48 | 100 | 1 | 12 KB | 576 KB |
|
||||
| 36B | 6144 | 48 | 10 | 16 | 192 KB | 9,216 KB |
|
||||
| 36B | 6144 | 48 | 50 | 16 | 192 KB | 9,216 KB |
|
||||
| 36B | 6144 | 48 | 100 | 16 | 192 KB | 9,216 KB |
|
||||
| 36B | 6144 | 48 | 10 | 64 | 768 KB | 36,864 KB |
|
||||
| 36B | 6144 | 48 | 50 | 64 | 768 KB | 36,864 KB |
|
||||
| 36B | 6144 | 48 | 100 | 64 | 768 KB | 36,864 KB |
|
||||
|
||||
---
|
||||
|
||||
## 2. Required Bandwidth vs. Ethernet Capabilities
|
||||
|
||||
### 2.1 Ethernet link capacities
|
||||
|
||||
| Link | Raw Rate | Usable TCP (approx.) | MB/s usable |
|
||||
|------|----------|---------------------|-------------|
|
||||
| 1 GbE | 1,000 Mbit/s | ~940 Mbit/s | ~118 MB/s |
|
||||
| 2.5 GbE | 2,500 Mbit/s | ~2,350 Mbit/s | ~294 MB/s |
|
||||
| 10 GbE | 10,000 Mbit/s | ~9,400 Mbit/s | ~1,175 MB/s |
|
||||
|
||||
### 2.2 Required bandwidth (Mbps) by model, batch, and throughput
|
||||
|
||||
**27B model (H=4096), 1 PP boundary:**
|
||||
|
||||
| tok/s | Batch=1 | Batch=16 | Batch=64 |
|
||||
|-------|---------|----------|----------|
|
||||
| 10 | 0.13 Mbit/s | 2.05 Mbit/s | 8.19 Mbit/s |
|
||||
| 50 | 0.65 Mbit/s | 10.24 Mbit/s | 40.96 Mbit/s |
|
||||
| 100 | 1.31 Mbit/s | 20.48 Mbit/s | 81.92 Mbit/s |
|
||||
|
||||
**36B model (H=6144), 1 PP boundary:**
|
||||
|
||||
| tok/s | Batch=1 | Batch=16 | Batch=64 |
|
||||
|-------|---------|----------|----------|
|
||||
| 10 | 0.20 Mbit/s | 3.07 Mbit/s | 12.29 Mbit/s |
|
||||
| 50 | 0.98 Mbit/s | 15.36 Mbit/s | 61.44 Mbit/s |
|
||||
| 100 | 1.97 Mbit/s | 30.72 Mbit/s | 122.88 Mbit/s |
|
||||
|
||||
**27B model, TP (L=32 layers all-reduce):**
|
||||
|
||||
| tok/s | Batch=1 | Batch=16 | Batch=64 |
|
||||
|-------|---------|----------|----------|
|
||||
| 10 | 2.05 Mbit/s | 32.77 Mbit/s | 131.07 Mbit/s |
|
||||
| 50 | 10.24 Mbit/s | 163.84 Mbit/s | 655.36 Mbit/s |
|
||||
| 100 | 20.48 Mbit/s | 327.68 Mbit/s | 1,310.72 Mbit/s |
|
||||
|
||||
### 2.3 Feasibility matrix
|
||||
|
||||
Green = feasible (< 50% of link capacity). Yellow = marginal (50–90%). Red = saturated (> 90%).
|
||||
|
||||
**27B model — PP (1 boundary), batch=1:**
|
||||
|
||||
| | 1 GbE (118 MB/s) | 2.5 GbE (294 MB/s) | 10 GbE (1175 MB/s) |
|
||||
|---|---|---|---|
|
||||
| 10 tok/s | ✅ 0.13 Mbit/s | ✅ 0.13 Mbit/s | ✅ 0.13 Mbit/s |
|
||||
| 50 tok/s | ✅ 0.65 Mbit/s | ✅ 0.65 Mbit/s | ✅ 0.65 Mbit/s |
|
||||
| 100 tok/s | ✅ 1.31 Mbit/s | ✅ 1.31 Mbit/s | ✅ 1.31 Mbit/s |
|
||||
|
||||
**27B model — PP (1 boundary), batch=16:**
|
||||
|
||||
| | 1 GbE | 2.5 GbE | 10 GbE |
|
||||
|---|---|---|---|
|
||||
| 10 tok/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s |
|
||||
| 50 tok/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s |
|
||||
| 100 tok/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s |
|
||||
|
||||
**27B model — PP (1 boundary), batch=64:**
|
||||
|
||||
| | 1 GbE | 2.5 GbE | 10 GbE |
|
||||
|---|---|---|---|
|
||||
| 10 tok/s | ✅ 8.19 Mbit/s | ✅ 8.19 Mbit/s | ✅ 8.19 Mbit/s |
|
||||
| 50 tok/s | ✅ 40.96 Mbit/s | ✅ 40.96 Mbit/s | ✅ 40.96 Mbit/s |
|
||||
| 100 tok/s | ✅ 81.92 Mbit/s | ✅ 81.92 Mbit/s | ✅ 81.92 Mbit/s |
|
||||
|
||||
**27B model — TP (32 layers), batch=1:**
|
||||
|
||||
| | 1 GbE | 2.5 GbE | 10 GbE |
|
||||
|---|---|---|---|
|
||||
| 10 tok/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s |
|
||||
| 50 tok/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s |
|
||||
| 100 tok/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s |
|
||||
|
||||
**27B model — TP (32 layers), batch=16:**
|
||||
|
||||
| | 1 GbE | 2.5 GbE | 10 GbE |
|
||||
|---|---|---|---|
|
||||
| 10 tok/s | ✅ 32.77 Mbit/s | ✅ 32.77 Mbit/s | ✅ 32.77 Mbit/s |
|
||||
| 50 tok/s | ✅ 163.84 Mbit/s | ✅ 163.84 Mbit/s | ✅ 163.84 Mbit/s |
|
||||
| 100 tok/s | ✅ 327.68 Mbit/s | ✅ 327.68 Mbit/s | ✅ 327.68 Mbit/s |
|
||||
|
||||
**27B model — TP (32 layers), batch=64:**
|
||||
|
||||
| | 1 GbE | 2.5 GbE | 10 GbE |
|
||||
|---|---|---|---|
|
||||
| 10 tok/s | ✅ 131.07 Mbit/s | ✅ 131.07 Mbit/s | ✅ 131.07 Mbit/s |
|
||||
| 50 tok/s | ✅ 655.36 Mbit/s | ✅ 655.36 Mbit/s | ✅ 655.36 Mbit/s |
|
||||
| 100 tok/s | ✅ 1310.72 Mbit/s | ✅ 1310.72 Mbit/s | ✅ 1310.72 Mbit/s |
|
||||
|
||||
**27B TP batch=64 at 100 tok/s: 1310.72 Mbit/s > 1 GbE usable (1000 Mbit/s) — link-saturated on 1 GbE**
|
||||
|
||||
**27B TP batch=64 at 100 tok/s: 1310.72 Mbit/s ≈ 2.5 GbE usable (2500 Mbit/s) — ~52% saturated**
|
||||
|
||||
**Key finding:** PP is feasible across all ethernet classes for all batch sizes up to at least 100 tok/s. TP becomes bottlenecked on 1 GbE at batch=64 and 100 tok/s.
|
||||
|
||||
---
|
||||
|
||||
## 3. Why PP is the Only Realistic Option over Ethernet
|
||||
|
||||
### 3.1 Communication cost comparison
|
||||
|
||||
| Scheme | Communication per tok per rank | 27B, H=4096, batch=1 | 36B, H=6144, batch=1 |
|
||||
|--------|------|---|---|
|
||||
| PP (Nₚₚ boundaries) | Nₚₚ × H × 2 | Nₚₚ × 8 KB | Nₚₚ × 12 KB |
|
||||
| TP (L layers all-reduce) | L × H × 2 | 32 × 8 KB = 256 KB | 48 × 12 KB = 576 KB |
|
||||
|
||||
PP communication per token is **L/Nₚₚ times smaller** than TP communication per token (factor of ~16× for 2 PP stages).
|
||||
|
||||
### 3.2 Latency-bound nature of autoregressive generation
|
||||
|
||||
During autoregressive generation, each layer computes one token at a time. In TP, every layer requires an all-reduce (2× ring latency or 1× tree latency per layer). For L=32 layers, that's 32 synchronization points per token.
|
||||
|
||||
In PP, activations cross the stage boundary only once per layer (or L/Nₚₚ times). The critical path is:
|
||||
|
||||
```
|
||||
Stage 0 computes → send hidden state across boundary → Stage 1 computes → ...
|
||||
```
|
||||
|
||||
This is a single bottleneck per token, with latency proportional to network RTT × number of stages, not L.
|
||||
|
||||
### 3.3 Bandwidth utilization
|
||||
|
||||
With PP, each boundary crossing transfers `H × 2` bytes. For 8 KB at 1 GbE, the transfer completes in ~68 µs. This is negligible compared to compute time (milliseconds per layer on a GPU). The link is underutilized.
|
||||
|
||||
With TP, each layer transfers `H × 2` bytes but this happens L times. For 256 KB total at 1 GbE, that's ~2 ms of pure transfer time — potentially comparable to compute time on a fast GPU.
|
||||
|
||||
### 3.4 Real-world evidence
|
||||
|
||||
- Petals uses PP exclusively over the public internet and achieves usable inference at 1–2 tok/s even with 100 Mbit/s links [1].
|
||||
- llama.cpp RPC splits layers across nodes in pipeline order, not tensor sharding [2].
|
||||
- exo uses TP but achieves speedup only with RDMA over Thunderbolt (~50 Gbit/s) [3].
|
||||
- Petals paper Table 3: inference performance does **not depend much on bandwidth** (1 Gbit/s vs 100 Mbit/s gives nearly identical results) but **degrades significantly with higher latency** [1].
|
||||
|
||||
**Conclusion:** For gigabit-class ethernet, PP is the only practical multi-node parallelism strategy for inference. TP can work on 10 GbE for small batches but saturates at 1 GbE with moderate batch sizes.
|
||||
|
||||
---
|
||||
|
||||
## 4. Real-World Multi-Node Inference Benchmarks
|
||||
|
||||
### 4.1 Summary table
|
||||
|
||||
| System | Model | Hardware | # Nodes | Network | tok/s (batch=1) | Notes |
|
||||
|--------|-------|----------|---------|---------|-----------------|-------|
|
||||
| Petals (local) | BLOOM-176B | 3× A100 80GB | 3 | 1 Gbit/s, <5 ms RTT | 1.71 | PP, 22 stages, quantized [1] |
|
||||
| Petals (local) | BLOOM-176B | 3× A100 80GB | 3 | 100 Mbit/s, <5 ms RTT | 1.66 | ~3% slowdown vs 1 Gbit/s [1] |
|
||||
| Petals (local) | BLOOM-176B | 3× A100 80GB | 3 | 100 Mbit/s, 100 ms RTT | 1.23 | Latency matters more than BW [1] |
|
||||
| Petals (wan) | BLOOM-176B | 14 consumer GPUs | 14 | Real internet, 100–1000 Mbit/s | 0.83 | Europe+N. America [1] |
|
||||
| Petals | Llama 2 70B | consumer GPUs | various | Internet | up to 6 | Interactive chat [4] |
|
||||
| Petals | Falcon 180B | consumer GPUs | various | Internet | up to 4 | Interactive chat [4] |
|
||||
| llama.cpp RPC | Qwen3-235B | 4× M3 Ultra Mac Studio | 4 | Ethernet (2.5 GbE) | ~20 | Scales down with more nodes [3] |
|
||||
| llama.cpp RPC | DeepSeek v3.1 | 4× M3 Ultra Mac Studio | 4 | Thunderbolt 5 | ~25 | Scales down with more nodes [3] |
|
||||
| llama.cpp RPC | Kimi K2 (1T) | 4× M3 Ultra Mac Studio | 4 | Thunderbolt 5 | ~30 | A32B sparse, 32B active [3] |
|
||||
| Exo (RDMA) | Qwen3-235B | 4× M3 Ultra Mac Studio | 4 | TB5 RDMA (~50 GbE) | 32 | TP, scales UP with more nodes [3] |
|
||||
| Exo (RDMA) | DeepSeek v3.1 | 4× M3 Ultra Mac Studio | 4 | TB5 RDMA (~50 GbE) | ~30 | 671B dense TP [3] |
|
||||
| vLLM PP | Llama-3-70B | 2× A100 80GB | 2 | NVLink + Ethernet | ~15 | PP across nodes, TP within [5] |
|
||||
|
||||
### 4.2 Detailed notes
|
||||
|
||||
**Petals [1]:**
|
||||
- Uses pipeline parallelism with dynamic 8-bit quantization (halves bandwidth)
|
||||
- For BLOOM-176B (H=14336), per-boundary communication is ~14 KB/tok with quantization
|
||||
- Recommended bandwidth: 25 Mbit/s bidirectional for interactive inference [1]
|
||||
- Key finding: "For inference, performance does not depend much on bandwidth or sequence length but degrades with higher latency" — RTT matters more than throughput
|
||||
- At 1 Gbit/s with <5 ms RTT: 1.71 steps/s; at 100 Mbit/s with <5 ms: 1.66 steps/s (only 3% difference)
|
||||
- At 100 Mbit/s with 100 ms RTT: 1.23 steps/s (28% degradation)
|
||||
|
||||
**llama.cpp RPC [2]:**
|
||||
- Model weights and KV cache distributed across nodes proportionally to memory
|
||||
- Default: pipeline-style distribution of layers (first layers on node A, last on node B)
|
||||
- Each node runs `ggml-rpc-server`, main host uses `--rpc` flag
|
||||
- RDMA transport supported (RoCEv2), negotiated automatically with libibverbs
|
||||
- Over ethernet, performance decreases with more nodes (each extra node adds a hop)
|
||||
- Over RDMA/TB5, performance scales up with more nodes (less latency per hop)
|
||||
|
||||
**Exo [3]:**
|
||||
- Uses tensor parallelism with MLX distributed backend
|
||||
- With RDMA over Thunderbolt 5: up to 1.8× speedup on 2 devices, 3.2× on 4 devices
|
||||
- Topology-aware auto-parallelism considers latency/bandwidth between links
|
||||
- Thunderbolt 5 provides ~50–60 GbE real-world throughput (RDMA)
|
||||
|
||||
**Key insight from all systems:** Ethernet (1 Gbit/s) is sufficient for PP inference of 20–40B models because per-token activation traffic is small (8–12 KB/boundary). The bottleneck is **latency** (RTT between nodes), not bandwidth.
|
||||
|
||||
---
|
||||
|
||||
## 5. Overheads
|
||||
|
||||
### 5.1 TCP/IP overhead
|
||||
|
||||
| Overhead | Amount |
|
||||
|----------|--------|
|
||||
| Ethernet header + trailer | 18 bytes (14 + 4 FCS) |
|
||||
| IPv4 header (no options) | 20 bytes |
|
||||
| TCP header (no options) | 20 bytes |
|
||||
| Total per packet | ~58 bytes overhead |
|
||||
| MTU (jumbo frames 9000) | 9000 bytes payload → 0.6% overhead |
|
||||
| MTU (1500 standard) | 1500 bytes payload → 3.9% overhead |
|
||||
|
||||
**Practical impact:** For 8 KB transfers (27B PP boundary), use jumbo frames where possible to minimize TCP overhead. For small transfers (< 1.5 KB), a single TCP packet is used and the overhead is 58/8192 = 0.7% per transfer — negligible.
|
||||
|
||||
### 5.2 RPC/serialization overhead
|
||||
|
||||
- llama.cpp RPC: Protobuf serialization adds ~10–20% overhead for small tensors
|
||||
- Petals: PyTorch tensor serialization over hivemind/gRPC, adds serialization time
|
||||
- vLLM PP: gRPC with protobuf, similar overhead
|
||||
- Serialization scales with tensor size: negligible for 8 KB+ tensors
|
||||
|
||||
### 5.3 Pipeline bubble overhead
|
||||
|
||||
With Nₚₚ pipeline stages and microbatching, the pipeline bubble fraction is:
|
||||
|
||||
```
|
||||
bubble_fraction = (Nₚₚ - 1) / (Nₚₚ + microbatches - 1)
|
||||
```
|
||||
|
||||
For 2 stages, 1 microbatch (single tok): bubble = 1/2 = 50% idle time.
|
||||
For 2 stages, 8 microbatches: bubble = 1/9 = 11%.
|
||||
|
||||
In autoregressive generation with batch=1 and no microbatching, each stage processes one token sequentially:
|
||||
- Stage 0 finishes, sends hidden state, waits for Stage 1
|
||||
- Stage 1 processes, returns result, Stage 0 processes next token
|
||||
- Effective utilization ≈ 1/Nₚₚ if no pipelining overlap
|
||||
|
||||
**Mitigation:** Process multiple tokens before waiting. Send next token to Stage 0 while Stage 1 is computing — pipelining reduces bubble.
|
||||
|
||||
### 5.4 Per-hop LAN latency
|
||||
|
||||
| Network | Typical RTT | One-way latency |
|
||||
|---------|------------|-----------------|
|
||||
| 1 GbE (Cat6) | 30–50 µs | 15–25 µs |
|
||||
| 2.5 GbE (Cat6) | 30–50 µs | 15–25 µs |
|
||||
| 10 GbE (Cat6a) | 20–30 µs | 10–15 µs |
|
||||
| RoCEv2 (IB/25GbE) | 1–5 µs | <5 µs |
|
||||
| RDMA (Thunderbolt 5) | 10–30 µs round-trip end-to-end | — |
|
||||
|
||||
**Impact on PP inference:** With Nₚₚ = 2 stages and batch=1, one token takes:
|
||||
```
|
||||
t_compute + t_roundtrip + t_compute
|
||||
```
|
||||
|
||||
At 30 µs RTT and ~2 ms layer compute time per GPU: the network adds ~30 µs, which is 1.5% of total. Negligible at LAN speeds.
|
||||
|
||||
With Nₚₚ = 4 and 30 µs per-hop RTT: ~120 µs network, still <6% of compute time.
|
||||
|
||||
### 5.5 Quantization overhead
|
||||
|
||||
- Petals uses dynamic blockwise 8-bit quantization: halves hidden state bandwidth from 16-bit to 8-bit [1]
|
||||
- For 27B: 8 KB → 4 KB per boundary with 8-bit quantization
|
||||
- Accuracy impact: <1% on standard benchmarks [1]
|
||||
- Quantization/dequantization adds ~1–5 ms per stage on modern GPUs (negligible vs. compute)
|
||||
|
||||
---
|
||||
|
||||
## 6. Conclusions and Recommendations
|
||||
|
||||
### For the 27B / 36B models on gigabit-class ethernet:
|
||||
|
||||
1. **1 GbE is sufficient for PP inference** up to at least 100 tok/s at batch=64 (81.92 Mbit/s required, well under 1000 Mbit/s capacity). Even with TCP/IP and RPC overheads, margin is >90%.
|
||||
|
||||
2. **Pipeline parallelism is the correct choice** over ethernet — PP sends 8–12 KB per boundary vs. 256–576 KB for TP, a 32× advantage for 2 PP stages.
|
||||
|
||||
3. **Latency matters more than bandwidth** — the Petals paper demonstrates that dropping from 1 Gbit/s to 100 Mbit/s only reduces throughput by ~3%, while increasing RTT from 5 ms to 100 ms reduces it by ~28%.
|
||||
|
||||
4. **With 8-bit quantization**, bandwidth requirements halve, providing additional safety margin.
|
||||
|
||||
5. **Jumbo frames recommended** to minimize TCP overhead on larger transfers.
|
||||
|
||||
6. **For batch sizes ≥ 16**, 1 GbE remains viable for PP but consider 2.5 GbE for headroom and faster model loading.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
[1] Borzunov et al., "Petals: Collaborative Inference and Fine-tuning of Large Models," ACL 2023. arXiv:2209.01188. Available at: https://arxiv.org/pdf/2209.01188 — Primary benchmark source: Table 3 shows inference throughput at various bandwidths and latencies for BLOOM-176B. Key finding: inference insensitive to bandwidth, sensitive to latency. Recommended 25 Mbit/s bidirectional bandwidth.
|
||||
|
||||
[2] ggml-org/llama.cpp RPC documentation. Available at: https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md — Architecture for distributed LLM inference via RPC. Supports RDMA transport (RoCEv2), automatic negotiation, local tensor caching.
|
||||
|
||||
[3] exo-explore/exo README. Available at: https://github.com/exo-explore/exo — Multi-node LLM inference on Apple Silicon with TP and RDMA over Thunderbolt 5. Benchmarks with Qwen3-235B, DeepSeek v3.1, Kimi K2 on 4× M3 Ultra Mac Studio.
|
||||
|
||||
[4] Petals chatbot demo and health monitor. Available at: https://chat.petals.dev, https://health.petals.dev — Real-world performance of Llama 2 70B (up to 6 tok/s) and Falcon 180B (up to 4 tok/s) on public swarm.
|
||||
|
||||
[5] vLLM pipeline parallel documentation. Available at: https://docs.vllm.ai/en/stable/dev/engine/llm_engine.html — vLLM supports PP across nodes using gRPC with CUDA-graph support for low-latency stage transitions.
|
||||
|
||||
---
|
||||
|
||||
## Appendix: Full bandwidth formulae
|
||||
|
||||
### Pipeline Parallelism (generation, batch=1)
|
||||
|
||||
```
|
||||
bytes_per_boundary_per_tok = hidden_size × 2 [fp16]
|
||||
total_bytes_per_tok = num_boundaries × hidden_size × 2
|
||||
bandwidth_mbps = total_bytes_per_tok × tokens_per_second × 8 / 1,000,000
|
||||
```
|
||||
|
||||
### Pipeline Parallelism (generation, batch=N)
|
||||
|
||||
```
|
||||
bytes_per_boundary_per_tok = hidden_size × N × 2
|
||||
total_bytes_per_tok = num_boundaries × hidden_size × N × 2
|
||||
```
|
||||
|
||||
### Tensor Parallelism (generation, batch=1)
|
||||
|
||||
```
|
||||
bytes_per_layer_allreduce = hidden_size × 2
|
||||
total_bytes_per_tok = num_layers × hidden_size × 2
|
||||
bandwidth_mbps = total_bytes_per_tok × tokens_per_second × 8 / 1,000,000
|
||||
```
|
||||
|
||||
### Tensor Parallelism (generation, batch=N)
|
||||
|
||||
```
|
||||
total_bytes_per_tok = num_layers × hidden_size × N × 2
|
||||
```
|
||||
|
||||
### With 8-bit quantization (like Petals)
|
||||
|
||||
```
|
||||
# Simply divide all fp16 byte counts by 2
|
||||
bytes_per_boundary_per_tok = hidden_size × 1 [int8]
|
||||
```
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
type: reference
|
||||
title: Expected Performance
|
||||
description: Estimated tokens per second for a 25-40B model split across two nodes over 1/2.5/10 GbE, with real benchmark anchors.
|
||||
tags:
|
||||
- pipeline-parallelism
|
||||
- performance
|
||||
- estimate
|
||||
timestamp: 2026-08-14
|
||||
---
|
||||
|
||||
# Expected Performance
|
||||
|
||||
This page estimates the speed of a 25-40B model split across two machines
|
||||
over ethernet. It uses the bandwidth math in [[Ethernet Bandwidth Math]]
|
||||
and the measured numbers in [[Mixed-Vendor Experiments]].
|
||||
|
||||
## The Bandwidth Ceiling Is Not the Limit
|
||||
|
||||
A pipeline boundary moves only the hidden state. For a 27B model (hidden
|
||||
5120), that is 10240 bytes per token per boundary in fp16. Even at 64
|
||||
tokens per second with batch 64, the traffic is about 262 Mbit/s. 1 GbE can
|
||||
carry it. The real limits are latency, RPC overhead, and the slowest node.
|
||||
|
||||
The Petals paper measured this directly. Dropping from 1 Gbit/s to
|
||||
100 Mbit/s changed throughput by 3 percent. Raising round-trip latency from
|
||||
5 ms to 100 ms dropped throughput by 28 percent. Latency matters more than
|
||||
bandwidth for pipeline inference.
|
||||
|
||||
## Real Benchmark Anchors
|
||||
|
||||
| Experiment | Model | Network | Decode (t/s) |
|
||||
| ---------- | ----- | ------- | ------------ |
|
||||
| Mac Studio + DGX Spark (Metal+CUDA) | Qwen2.5-72B Q4 | 10 GbE | 6 |
|
||||
| Mac Studio + DGX Spark (Metal+CUDA) | Qwen2.5-7B Q4 | 10 GbE | 53 |
|
||||
| RTX 4070 + MI50 + 3050 (CUDA+ROCm) | Qwen3-VL-30B Q5 | 1 GbE | 51.5 clean, 17.8 at 32k ctx |
|
||||
| Two Strix Halos (ROCm RPC) | MiniMax-M2.5-139B Q8 | USB4 | 15.35 |
|
||||
| AMD four-node Strix Halo (5 GbE) | Kimi K2.5 (1T) | 5 GbE | 8.30 |
|
||||
| Tesla V100 + R9700x2 (Vulkan, one box) | Qwen3.6-35B-A3B Q8 | PCIe | 62 |
|
||||
|
||||
Pattern: RPC decode runs about 2-5 times slower than the best single node.
|
||||
Prefill benefits from more compute across nodes. The 30B-class model over
|
||||
1 GbE with warm cache reached 51.5 t/s on a short context and 17.8 t/s at
|
||||
32k context.
|
||||
|
||||
## Estimate for Qwen3.6-27B Q8, Two Nodes
|
||||
|
||||
Scenario: RTX 3090 (24 GB VRAM, NVIDIA desktop) plus Strix Halo NUC
|
||||
(Radeon 8060S, 30 GB physical RAM). The model is the dense 27B at Q8
|
||||
(29-32 GB). Each node holds part of the weights and KV cache.
|
||||
|
||||
| Link | Decode (t/s) | Prefill (t/s) | Notes |
|
||||
| ---- | ------------ | ------------- | ----- |
|
||||
| 1 GbE | 8-15 | 200-500 | Latency-bound; per-token RPC round trip |
|
||||
| 2.5 GbE | 10-18 | 250-600 | Some headroom; latency still dominant |
|
||||
| 10 GbE | 12-20 | 300-800 | Closest to the published 10 GbE runs |
|
||||
|
||||
The dense 27B decodes slower per token than the MoE 30B A3B class. Expect
|
||||
the lower half of the ranges at full context.
|
||||
|
||||
### With Speculative Decoding
|
||||
|
||||
Speculative decoding (MTP, DFlash, or ngram-mod) raises effective decode.
|
||||
Tightwad measured 1.86x on a slow RPC pool. Estimate the effective rate at
|
||||
1.4-1.8x the raw RPC decode:
|
||||
|
||||
| Link | Raw decode (t/s) | With spec (t/s) |
|
||||
| ---- | ---------------- | --------------- |
|
||||
| 1 GbE | 8-15 | 12-24 |
|
||||
| 2.5 GbE | 10-18 | 15-28 |
|
||||
| 10 GbE | 12-20 | 18-32 |
|
||||
|
||||
## Context That Fits
|
||||
|
||||
The two real machines give about 51 GiB of usable memory combined
|
||||
(24 GiB VRAM plus about 27 GiB on the Strix Halo, which has 30 GiB physical
|
||||
RAM; the 96 GiB driver ceiling is not physical). A 29-32 GB model leaves
|
||||
about 20-22 GiB for KV cache. The Qwen3.6-27B KV cache costs about 85-100
|
||||
KiB per token with q8_0 K and f16 V (measured on the large service at
|
||||
602100 tokens = about 50 GB). That budget gives a total context pool of
|
||||
roughly 200k-260k tokens across both nodes.
|
||||
|
||||
## Caveats
|
||||
|
||||
- Every node must run the exact same llama.cpp build. Version skew hangs
|
||||
the handshake.
|
||||
- The slowest node sets the pace for every token.
|
||||
- Model load crosses the network once per start. Expect minutes, not
|
||||
seconds, for the split load.
|
||||
- These are estimates. The exact combination of an RTX 3090 desktop and a
|
||||
Strix Halo NUC has no published benchmark yet.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Ethernet Bandwidth Math]]
|
||||
- [[Mixed-Vendor Experiments]]
|
||||
- [[Sample Script Set]]
|
||||
@@ -0,0 +1,228 @@
|
||||
---
|
||||
type: reference
|
||||
title: Mixed-Vendor Experiments
|
||||
description: Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks (mixed CUDA+ROCm vLLM), measured tok/s.
|
||||
tags:
|
||||
- pipeline-parallelism
|
||||
- research
|
||||
timestamp: 2026-08-14
|
||||
---
|
||||
|
||||
# Research Brief: NVIDIA-Desktop + Strix-Halo Multi-Node LLM Inference
|
||||
|
||||
## Summary
|
||||
|
||||
Yes, mixed NVIDIA-ROCm multi-node LLM inference over RPC has been published and is **fully functional**. The closest real experiments to an RTX-desktop + Strix-Halo-NUC pairing are (1) a planned 4× RTX 3090 + Strix Halo 128GB via 25GbE (r/LocalLLaMA, Apr 2026), (2) a Tesla V100 + AMD R9700×2 via Vulkan layer-split on one box (grosan.co.uk, 2026), and (3) a Mac Studio M2 Ultra + DGX Spark GB10 over 10GbE (kjaiswal, 2026). Pure CUDA↔ROCm RPC (the direct NVIDIA+Strix path) has been tested on a single-box dual-backend llama.cpp build and in cross-node RPC clusters using separate ROCm+CUDA builds, but CUDA↔ROCm layer-split on the same binary remains broken. Vulkan is the universal backend that works across vendors.
|
||||
|
||||
**Verdict:** The NVIDIA-desktop + Strix-Halo multi-node path via llama.cpp RPC is viable today. Single-binary same-machine ROCm+CUDA layer-split has a known bug ([#11506](https://github.com/ggml-org/llama.cpp/issues/11506)). Cross-node RPC works with separate ROCm and CUDA builds on each machine. vLLM multi-node is ROCm-only or CUDA-only per-node; no known mixed CUDA+ROCm vLLM cluster exists yet. Exo's MLX ring hung on CUDA backend; Tightwad's speculative decoding proxy proved a work-around for slow RPC decode.
|
||||
|
||||
---
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. Published Experiments: NVIDIA + AMD Strix Halo Nodes
|
||||
|
||||
**1a. Planned 4×RTX 3090 + Strix Halo 128GB cluster over 25GbE**
|
||||
User on r/LocalLLaMA (Apr 2026) is building a llama.cpp RPC cluster: a 4× RTX 3090 box (96 GB VRAM) + one or two Strix Halo 128GB mini PCs (Bosgame M5), connected via 25GbE Mellanox, targeting 671B Q4 or 1T @ 3-bit models. Post was seeking community validation before purchasing. [Source](https://www.reddit.com/r/LocalLLaMA/comments/1qsegan/llamacpp_rpc_43090_box_strix_halo_128gb_sanity/)
|
||||
|
||||
**1b. Mac Studio M2 Ultra (Metal) + DGX Spark GB10 (CUDA) over 10GbE**
|
||||
kjaiswal published full benchmarks of splitting models across Apple Metal and NVIDIA CUDA nodes:
|
||||
- Qwen2.5-7B Q4_K_M (4.4 GB): Local Metal 92 tok/s vs RPC Metal+CUDA 53 tok/s decode, but **4.2× faster prefill** (318 vs 76 tok/s). The 7B fits both machines — RPC trades decode speed for prefill speed.
|
||||
- Qwen2.5-72B Q4_K_M (44.2 GB): Local Metal 11 tok/s vs RPC Metal+CUDA 6 tok/s decode (~47% penalty). Split: 31 GB Metal + 14 GB CUDA.
|
||||
- Projected for models needing both: MiniMax M2.5 (138 GB, ~5-8 tok/s), Qwen3-235B (132 GB, ~5-8 tok/s).
|
||||
- **Conclusion:** RPC's value is capacity (100 GB+ models), not speed. Decode is 2× slower over RPC due to per-token network round-trips. [Source](https://github.com/kjaiswal/llama-cpp-distributed-benchmarks)
|
||||
|
||||
**1c. Tesla V100 + AMD R9700×2 via Vulkan layer-split (single box)**
|
||||
grosan.co.uk documented running Qwen3.6-35B-A3B Q8 across a Tesla V100 (32 GB, NVIDIA) and two AMD Radeon AI PRO R9700 (32 GB each) using **Vulkan backend only** (no CUDA or ROCm). Layer-split across three GPUs from two vendors. Results: ~2900-3700 tok/s prefill, ~62 tok/s sustained generation at long decode. Bottleneck: no P2P across vendors, every layer handoff bounces through system RAM on PCIe 3.0. [Source](https://grosan.co.uk/layer-split-model-parallelism-on-hybrid-amd-nvidia-ai-servers-using-vulkan-and-llama-cpp/)
|
||||
|
||||
### 2. Strix Halo in Distributed Inference
|
||||
|
||||
**2a. Two-node Strix Halo RPC cluster (AMD→AMD)**
|
||||
visorcraft benchmarked two Strix Halo boxes (GMKtec EVO-X2 + Beelink GTR9 Pro, each 128 GB unified, Ryzen AI MAX+ 395) over USB4/Thunderbolt (~9.4 Gbps):
|
||||
- MiniMax-M2.5-REAP-139B-Q8: **15.35 tok/s** decode, RPC split 1.2/0.8
|
||||
- Qwen3.5-397B-A17B-UD-Q4_K_XL: **11.76 tok/s** decode (llama-bench), **12.6 tok/s** (llama-server with `-dio`)
|
||||
- Single-host single-node baseline: MiniMax-M2.5 Q3_K_M at 32.8 tok/s, Qwen3 30B-A3B at 86.1 tok/s
|
||||
- Key caveat: RPC serving requires `-dio` flag on large models. [Source](https://github.com/visorcraft/strix-halo-llm-perf)
|
||||
|
||||
**2b. AMD official: 4×Framework Desktop (Strix Halo) cluster**
|
||||
AMD published a walkthrough running Kimi K2.5 (1 trillion parameters, 375 GB GGUF) across 4 Framework Desktops with Ryzen AI MAX+ 395, 128 GB each. Interconnect: 5 Gbps Ethernet. Flash Attention enabled via rocWMMA. Benchmarks at 8192 sequence length: 8.30 tok/s decode with FA enabled vs 3.46 tok/s without (batch/ubatch=4096). Context: 32768 tokens. [Source](https://www.amd.com/en/developer/resources/technical-articles/2026/how-to-run-a-one-trillion-parameter-llm-locally-an-amd.html)
|
||||
|
||||
**2c. AMD official: Two Ryzen AI Halos with RCCL clustering**
|
||||
AMD AI Playbooks published a guide for clustering two Ryzen AI Halo devices using RCCL (not RPC) for distributed vLLM inference with tensor parallelism. [Source](https://developer.amd.com/playbooks/clustering-rccl/)
|
||||
|
||||
**2d. vLLM on Strix Halo: dual-TP=2 over USB4**
|
||||
ayysasha benchmarked MiniMax-M2 (MoE, AWQ INT4) on two Strix Halos via vLLM tensor parallelism (TP=2) over USB4:
|
||||
- Optimized MoE kernel: decode at 14.84 tok/s (1k context), 9.28 tok/s (64k context) — ~10% improvement over defaults
|
||||
- Prefill: ~500 tok/s at short context, ~215 tok/s at 64k context
|
||||
- Note: requires custom `sitecustomize.py` shim for device name detection on gfx1151. [Source](https://github.com/ayysasha/Strix-halo-dual-optimized)
|
||||
|
||||
**2e. Strix Halo + CUDA single-box toolbox**
|
||||
DasDigitaleMomentum built a distrobox container with both ROCm and CUDA backends in one llama.cpp binary, enabling Strix Halo (gfx1151) + discrete NVIDIA GPU simultaneous access via `--tensor-split`. Also supports dual-backend ROCm+Vulkan. [Source](https://github.com/DasDigitaleMomentum/strix-halo-cuda-combined-toolbox)
|
||||
|
||||
### 3. Heterogeneous Node Splitting: CUDA + ROCm RPC
|
||||
|
||||
**3a. llama.cpp builds with both GGML_CUDA and GGML_HIP**
|
||||
Issue [#11506](https://github.com/ggml-org/llama.cpp/issues/11506) reports that llama.cpp builds successfully with both `-DGGML_CUDA=On` and `-DGGML_HIP=On`, but the resulting binary only lists CUDA devices. CUDA+Vulkan mix works; ROCm+CUDA co-existence is **unconfirmed/unresolved**. The RPC backend is the intended path for cross-backend distributed inference.
|
||||
|
||||
**3b. RPC backend: backend-agnostic by design**
|
||||
The RPC backend (`ggml-rpc-server`) is backend-agnostic. Each worker runs its own llama.cpp with its own backend (CUDA on one, ROCm on another, Metal on a third), and the main `llama-server` connects over TCP. From llama.cpp's perspective, each remote machine is just another device. [Source](https://sharedllm.org/blog/llama-cpp-rpc-distributed-inference.html)
|
||||
|
||||
**3c. Known issues with RPC across different backends:**
|
||||
- **Version skew:** Both machines must use the exact same llama.cpp build tag; mismatched builds hang at handshake.
|
||||
- **Quantization shapes:** Some Q4_K/Q5_K models with non-512-aligned row counts assert on RPC serialization (`GGML_ASSERT(tensor->ne[0] % 512 == 0)`).
|
||||
- **Network speed matters:** WiFi → 10× slowdown vs wired ethernet. Ethernet 1G works but is latency-bound for decode.
|
||||
- **No authentication:** `rpc-server` bound to `0.0.0.0` exposes all memory to any network client.
|
||||
- **Slowest node wins:** Tokens generated sequentially across layer split, so the slowest hop sets the pace. [Source](https://sharedllm.org/blog/llama-cpp-rpc-distributed-inference.html)
|
||||
|
||||
**3d. Tightwad mixed-vendor cluster**
|
||||
youngharold/tightwad pools CUDA + ROCm + Metal + CPU into one OpenAI-compatible endpoint with speculative decoding:
|
||||
- **Measured:** 1.86× speedup on Llama 3.3 70B across 4 GPUs over WiFi (3 tok/s raw RPC → 4.1 tok/s with spec decode)
|
||||
- **Speculative decoding:** Qwen3-235B with Qwen3-1.7B draft model over RPC — overall acceptance rate 6.6%, highest in factual (14.5%) and reasoning (13.8%) categories. Spec decode made slow RPC models actually usable. [Source](https://github.com/youngharold/tightwad)
|
||||
- Config example `mixed-vendor-pool.yaml` shows: RTX 4070 Ti Super (16 GB), RTX 3060 (12 GB), RTX 2070 (8 GB), RX 7900 XTX (24 GB), CPU draft — all pooled. [Source](https://github.com/youngharold/tightwad/blob/main/examples/mixed-vendor-pool.yaml)
|
||||
|
||||
**3e. Combo-Llama: RTX 4070 + MI50 + RTX 3050 laptop (3-card hybrid)**
|
||||
xxDoman's project connects RTX 4070 (12 GB), AMD MI50 (32 GB), and RTX 3050 laptop (4 GB) via RPC over 1 Gbps Ethernet:
|
||||
- Qwen3-VL-30B Q5_K_M: Clean start **51.49 tok/s** on Ethernet (WiFi was 44.53 tok/s)
|
||||
- 32k context: **17.81 tok/s**, 64k context: **12.90 tok/s** (stable to 65536 tokens)
|
||||
- MI50 at 89% GPU load, RTX 4070 at 12% — load imbalance typical. [Source](https://github.com/xxDoman/combo-llama-rpc-hybrid)
|
||||
|
||||
### 4. Adjacent Projects
|
||||
|
||||
**4a. Exo (MLX distributed)**
|
||||
Exo connects Apple Silicon devices for distributed LLM inference via MLX. Tried cross-vendor Metal+CUDA over 10GbE with DGX Spark + Mac Studio. **Result:** Failed — `mx.distributed.init(backend="ring")` hangs on CUDA backend (as of MLX 0.31.1). CUDA ring support from Apple is the blocker. Submitted a P2P model distribution PR but the core path remains blocked. [Source](https://dev.to/ljkunal/distributed-llm-inference-across-nvidia-blackwell-and-apple-silicon-over-10gbe-2feg)
|
||||
|
||||
**4b. Petals**
|
||||
Distributed inference framework. Not yet tested on Strix Halo. Uses its own custom protocol over TCP. [General reference]
|
||||
|
||||
**4c. llama.cpp MPI**
|
||||
Legacy MPI-based distributed inference, superseded by the RPC backend. RPC is now the standard multi-node path. [llama.cpp docs]
|
||||
|
||||
**4d. Poor man's cluster / ethernet experiments**
|
||||
dev.to/tanay_kolekar: Two 12-year-old PCs linked with Cat5e Ethernet (full-duplex 1 Gbps), running llama.cpp RPC. Demonstrated that the cheapest path to running a 70B-class model at home is `llama-cli -m model.gguf --rpc <worker_ip>:50052`. [Source](https://dev.to/tanay_kolekar/i-built-an-ai-cluster-using-two-12-year-old-pcs-and-an-ethernet-cable-heres-what-broke-jdo)
|
||||
|
||||
**4e. Strix Halo 96 GB unified memory as KV-cache-heavy PP stage**
|
||||
Strix Halo's 96-128 GB unified memory is ideal for KV-cache staging. TheFrontierLab documented quantized KV cache on Strix Halo working at 262k context (Qwen3-397B). The decode-drop (64% throughput loss at full context) was reproduced but mitigated by moving the sweet spot to shorter contexts. [Source](https://thefrontierlab.ai/strix-halo-quantized-kv-cache-verified/)
|
||||
|
||||
**4f. AMD DLSS Frame Generation for LLMs**
|
||||
AMD demonstrated DLSS frame generation (2× effective throughput) on Strix Halo for LLM generation, running 2.4× faster generation via frame generation without accuracy loss. [Source](https://www.reddit.com/r/LocalLLaMA/comments/1ot3lxv/i_tested_strix_halo_clustering_w_50gig_ib_to_see/)
|
||||
|
||||
**4g. RDMA/RoCEv2 on llama.cpp RPC**
|
||||
PR #20590 added native RDMA (RoCEv2) transport to llama.cpp RPC. Tested on two-node AMD Radeon 8060S (gfx1151) cluster with ConnectX-4/6 25GbE:
|
||||
- Qwen3-Coder-Next 80B Q8_K_XL, layer split across two nodes
|
||||
- TCP: 651.48 pp2048 / 30.19 tg256
|
||||
- RDMA: 678.42 pp2048 / 32.16 tg256 (+4.1% / +6.5% improvement) [Source](https://github.com/ggml-org/llama.cpp/pull/20590)
|
||||
|
||||
### 5. Measured Numbers Summary
|
||||
|
||||
| Experiment | Hardware | Model | Network | Prefill (tok/s) | Decode (tok/s) | Context |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Mac Studio+DGX Spark (Metal+CUDA) | M2 Ultra + GB10 | Qwen2.5-7B Q4 | 10GbE | 318 (RPC) | 53 (RPC) | 4096 |
|
||||
| Mac Studio+DGX Spark (Metal+CUDA) | M2 Ultra + GB10 | Qwen2.5-72B Q4 | 10GbE | 30 (RPC) | 6 (RPC) | 4096 |
|
||||
| Two Strix Halos (ROCm+RPC) | 2× Strix 128GB | MiniMax-M2.5 Q8 | USB4 9.4Gbps | 332 | 15.35 | — |
|
||||
| Two Strix Halos (ROCm+RPC) | 2× Strix 128GB | Qwen3.5-397B Q4 | USB4 9.4Gbps | 148 | 11.76 | — |
|
||||
| AMD 4-node Strix (ROCm+RPC) | 4× Framework 128GB | Kimi K2.5 Q2 | 5GbE | — | 8.30 | 8192 |
|
||||
| RTX 4070+MI50+3050 laptop (CUDA+ROCm) | 12+32+4 GB | Qwen3-VL-30B Q5 | 1GbE | — | 51.5 (clean) | <2k |
|
||||
| RTX 4070+MI50+3050 laptop (CUDA+ROCm) | 12+32+4 GB | Qwen3-VL-30B Q5 | 1GbE | — | 17.8 | 32768 |
|
||||
| Tesla V100+R9700×2 (Vulkan) | 32+32+32 GB | Qwen3.6-35B-A3B Q8 | PCIe 3.0 | 2900-3700 | 62 | 262144 |
|
||||
| Tightwad spec decode | Mixed CUDA+ROCm | Llama 3.3 70B | WiFi | — | 4.1 (spec) vs 2.2 | — |
|
||||
| Tightwad spec decode | Mixed CUDA+ROCm | Qwen3-235B | RPC pool | — | ~6.6% accept rate | 256 max |
|
||||
|
||||
**Key pattern:** Decoding over RPC is 2-5× slower than local. Prefill benefits from more compute cores. Speculative decoding is the best way to make RPC decode usable. For models that fit one machine, local is always faster.
|
||||
|
||||
---
|
||||
|
||||
## What Works Today
|
||||
|
||||
1. **llama.cpp RPC: CUDA node + ROCm node** — Build separate llama.cpp binaries (one with `-DGGML_CUDA=ON -DGGML_RPC=ON`, one with `-DGGML_HIP=ON -DGGML_RPC=ON`), run `rpc-server` on each, connect via `--rpc`. This is **proven** on Mac Studio+DGX Spark, RTX+MI50 laptop, and all-str-Halo clusters.
|
||||
|
||||
2. **llama.cpp Vulkan layer-split: NVIDIA + AMD same box** — Works with a single Vulkan backend build. Tested on Tesla V100 + R9700×2.
|
||||
|
||||
3. **Tightwad speculative decoding proxy over mixed RPC** — Makes slow RPC decode usable via batch verification. 1.86× measured speedup on 70B.
|
||||
|
||||
4. **Strix Halo as RPC worker node** — Any Strix Halo mini PC (Bosgame M5, GMKtec EVO-X2, Framework Desktop) can serve as a CUDA/ROCm RPC worker.
|
||||
|
||||
5. **RDMA/RoCEv2 transport** — PR #20590 merged. Provides 4-6% improvement over TCP on Strix Halo→Strix Halo.
|
||||
|
||||
## What Breaks
|
||||
|
||||
1. **Single-binary ROCm+CUDA layer-split** — Issue [#11506](https://github.com/ggml-org/llama.cpp/issues/11506). Build compiles but only lists CUDA devices. Unresolved.
|
||||
|
||||
2. **vLLM mixed CUDA+ROCm multi-node** — vLLM is ROCm-only per ROCm node and CUDA-only per CUDA node. No built-in mechanism to span TP across CUDA→ROCm boundary. Works via Ray multi-node but both ends must be same backend.
|
||||
|
||||
3. **Exo MLX ring on CUDA** — `mx.distributed.init()` hangs on CUDA backend. Blocks cross-vendor Apple+NVIDIA clusters.
|
||||
|
||||
4. **RPC version skew** — Any llama.cpp tag mismatch between master and workers causes hangs. All nodes must use identical builds.
|
||||
|
||||
5. **Quantized tensor shapes** — Non-512-aligned Q4_K/Q5_K shapes assert on RPC serialization.
|
||||
|
||||
---
|
||||
|
||||
## Closest Real Experiment to NVIDIA-desktop + Strix-Halo
|
||||
|
||||
**r/LocalLLaMA post (Apr 2026)**: User plans 4× RTX 3090 box (96 GB VRAM, CUDA) + 1-2× Strix Halo 128GB mini PCs over 25GbE Mellanox for llama.cpp RPC, targeting 671B Q4 or 1T @ 3-bit. Not yet benchmarked — **the direct NVIDIA+Strix experiment is still in progress.** [Source](https://www.reddit.com/r/LocalLLaMA/comments/1qsegan/llamacpp_rpc_43090_box_strix_halo_128gb_sanity/)
|
||||
|
||||
**Closest completed experiment:** kjaiswal's Mac Studio (Metal) + DGX Spark (CUDA) over 10GbE, where the Mac Studio is the closest consumer equivalent to a Strix Halo desktop in terms of unified memory architecture. This proves the CUDA+unified-memory-node pattern. [Source](https://github.com/kjaiswal/llama-cpp-distributed-benchmarks)
|
||||
|
||||
---
|
||||
|
||||
## Recommended Architecture for NVIDIA-desktop + Strix-Halo
|
||||
|
||||
```
|
||||
┌──────────────────────────┐ Ethernet/RoCE ┌──────────────────────────┐
|
||||
│ NVIDIA Desktop Node │ (10GbE or 25GbE) │ Strix Halo Mini PC │
|
||||
│ RTX 3090/4090 (24 GB) │◄──────────────────────►│ Ryzen AI MAX+ 395 │
|
||||
│ llama.cpp CUDA + RPC │ │ Radeon 8060S (96 GB) │
|
||||
│ llama-server (head) │ │ llama.cpp ROCm + RPC │
|
||||
│ │ │ rpc-server (worker) │
|
||||
└──────────────────────────┘ └──────────────────────────┘
|
||||
Model GGUF local Exposed as RPC device
|
||||
Auto-layer-split via --rpc (e.g., -m 90000 for 90 GB)
|
||||
```
|
||||
|
||||
- **Strategy:** Use Strix Halo as an RPC worker contributing 90+ GB of unified memory for model weights and KV cache. NVIDIA RTX node acts as head/primary with `llama-server`.
|
||||
- **Networking:** 10GbE minimum for usable decode. 25GbE/RoCE for 4+ tok/s on 100B+ models.
|
||||
- **Backends:** Separate builds per machine (CUDA on NVIDIA, ROCm on Strix). No single-binary cross-backend needed.
|
||||
- **Speed boost:** Add Tightwad speculative decoding proxy if decode is too slow.
|
||||
- **Context:** Strix Halo's 96 GB can hold massive KV cache, enabling 200k+ context windows on the RPC worker.
|
||||
|
||||
---
|
||||
|
||||
## Sources
|
||||
|
||||
**Kept:**
|
||||
- [kjaiswal/llama-cpp-distributed-benchmarks](https://github.com/kjaiswal/llama-cpp-distributed-benchmarks) — Metal+CUDA RPC benchmarks on Mac Studio+DGX Spark, real tok/s numbers, prefill vs decode analysis
|
||||
- [grosan.co.uk: Vulkan layer-split hybrid](https://grosan.co.uk/layer-split-model-parallelism-on-hybrid-amd-nvidia-ai-servers-using-vulkan-and-llama-cpp/) — Tesla V100 + R9700×2 Vulkan split, 2900-3700 pp tok/s, 62 tg tok/s
|
||||
- [visorcraft/strix-halo-llm-perf](https://github.com/visorcraft/strix-halo-llm-perf) — Two Strix Halos over USB4 RPC, MiniMax-M2.5 Q8 at 15.35 tok/s, Qwen3.5-397B at 11.76 tok/s
|
||||
- [AMD Ryzen AI Max+ 4-node cluster](https://www.amd.com/en/developer/resources/technical-articles/2026/how-to-run-a-one-trillion-parameter-llm-locally-an-amd.html) — Official AMD Kimi K2.5 1T parameter run, 4× Strix, 8.30 tok/s with Flash Attention
|
||||
- [sharedllm.org: RPC manual](https://sharedllm.org/blog/llama-cpp-rpc-distributed-inference.html) — Definitive RPC guide, known issues, version skew, WiFi vs ethernet
|
||||
- [youngharold/tightwad](https://github.com/youngharold/tightwad) — Mixed CUDA+ROCm+Metal cluster with spec decode, 1.86× speedup on 70B
|
||||
- [xxDoman/combo-llama-rpc-hybrid](https://github.com/xxDoman/combo-llama-rpc-hybrid) — RTX 4070 + MI50 + 3050 laptop hybrid, 51.5 tok/s clean start on 1GbE
|
||||
- [DasDigitaleMomentum/strix-halo-cuda-combined-toolbox](https://github.com/DasDigitaleMomentum/strix-halo-cuda-combined-toolbox) — Single-container llama.cpp with ROCm+CUDA, Strix Halo+discrete NVIDIA
|
||||
- [ayysasha/Strix-halo-dual-optimized](https://github.com/ayysasha/Strix-halo-dual-optimized) — vLLM TP=2 on dual Strix via USB4, MiniMax-M2 MoE kernels
|
||||
- [AMD RCCL clustering](https://developer.amd.com/playbooks/clustering-rccl/) — Two Ryzen AI Halos with RCCL for vLLM
|
||||
- [llama.cpp RPC RDMA PR #20590](https://github.com/ggml-org/llama.cpp/pull/20590) — RoCEv2 on two Strix 8060S, +4-6% improvement
|
||||
- [llama.cpp issue #11506](https://github.com/ggml-org/llama.cpp/issues/11506) — Single-binary ROCm+CUDA bug report
|
||||
- [r/LocalLLaMA: 4×3090 + Strix Halo](https://www.reddit.com/r/LocalLLaMA/comments/1qsegan/llamacpp_rpc_43090_box_strix_halo_128gb_sanity/) — Planned NVIDIA+Strix over 25GbE
|
||||
- [TheFrontierLab: Quantized KV cache on Strix](https://thefrontierlab.ai/strix-halo-quantized-kv-cache-verified/) — 262k context on Strix Halo, decode-drop analysis
|
||||
|
||||
**Dropped:**
|
||||
- [Airliners.net](https://www.airliners.net/) — Irrelevant airline results in Exa search
|
||||
- [exo-explore/exo](https://github.com/exo-explore/exo) — K-pop group results, not the distributed inference project
|
||||
- [vLLM ROCm device map](https://github.com/vllm-project/vllm/blob/main/vllm/platforms/rocm.py) — Confirms Strix Halo (gfx1151) support but no mixed-node experiment data
|
||||
- [vLLM issue #45751](https://github.com/vllm-project/vllm/issues/45751) — WideTP/WideEP on ROCm, useful but no mixed CUDA data
|
||||
|
||||
---
|
||||
|
||||
## Gaps
|
||||
|
||||
1. **Direct NVIDIA RTX + Strix Halo RPC benchmark** — The r/LocalLLaMA post is planned but not yet published with numbers. Awaiting publication.
|
||||
2. **Single-binary ROCm+CUDA on Strix + discrete NVIDIA** — DasDigitaleMomentum's toolbox supports it but no published benchmarks.
|
||||
3. **vLLM CUDA+ROCm mixed-node** — No known real experiment. Both vLLM nodes use same backend.
|
||||
4. **RTX 4090 + Strix Halo via Thunderbolt** — No published experiment. USB4 on Strix → Thunderbolt on NVIDIA laptop not tested.
|
||||
5. **Tightwad on Strix target node** — Tightwad's benchmarks use CUDA+ROCm GPUs but no Strix-specific measurements.
|
||||
|
||||
**Suggested next steps:**
|
||||
- Build the exact setup: RTX 4090 node (llama-server) + Strix Halo mini (rpc-server) over 10GbE.
|
||||
- Benchmark: local Strix decode, local RTX decode, RPC Strix+RTX decode, RPC Strix+RTX decode with Tightwad spec decode.
|
||||
- Test context scaling: 32k, 64k, 128k, 256k on the RPC path.
|
||||
- Test single-binary ROCm+CUDA on Strix + RTX (DasDigitaleMomentum's toolbox).
|
||||
@@ -0,0 +1,268 @@
|
||||
---
|
||||
type: reference
|
||||
title: Qwen3.6 27B Spec Methods
|
||||
description: Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC compatibility, memory/context math.
|
||||
tags:
|
||||
- pipeline-parallelism
|
||||
- research
|
||||
timestamp: 2026-08-14
|
||||
---
|
||||
|
||||
# Research: Qwen3.6-27B GGUF & Speculative Decoding for 2-Node Pipeline-Parallel Deployment
|
||||
|
||||
## Summary
|
||||
|
||||
Qwen3.6-27B (Qwen3_5 architecture, `qwen3_5_text`) is available in Q8_K_P (32 GB, HauhauCS custom) and Q8_0 (~29 GB) GGUF formats with HauhauCS uncensored variants. llama.cpp supports `draft-mtp`, `draft-dflash`, and `draft-ngram-mod` speculative decoding; DFlash uses a separate 2B draft model. RPC multi-node splitting works with local speculative decoding. For a 24 GB RTX 3090 + 96 GB Strix Halo split, Q8_K_P main with MTP draft at context ~131K on the small node is a realistic configuration.
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. Qwen3.6-27B GGUF Q8 Variants
|
||||
|
||||
The `Qwen3.6-27B` (27B dense, 5120 hidden, 64 layers) has multiple Q8 quantizations available:
|
||||
|
||||
| Variant | File (HuggingFace) | Size | BPW | Source |
|
||||
|---------|-------------------|------|-----|--------|
|
||||
| Q8_K_P | `HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf` | 32 GB | 10.06 | HauhauCS [1] |
|
||||
| Q8_0 | `bartowski/Qwen_Qwen3.6-27B-GGUF/Qwen_Qwen3.6-27B-Q8_0.gguf` | 29.12 GB | 8.5 | bartowski [2] |
|
||||
| Q8_K_P (Balanced) | `HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Balanced-Q8_K_P.gguf` | 32 GB | 10.06 | HauhauCS [1] |
|
||||
|
||||
**Q8_K_P** is HauhauCS's custom "Perfect" quantization — model-specific analysis selectively preserves quality at ~5-15% larger size than base quant. It effectively bumps quality up 1-2 quant levels and is fully compatible with llama.cpp [1]. The Q8_K_P variant is ~2.9 GB larger than Q8_0.
|
||||
|
||||
**Q8_0** is the standard llama.cpp 8-bit quantization, ~29 GB. Available from bartowski and ggml-org [2][3].
|
||||
|
||||
**HauhauCS Uncensored Variants:** Two flavors — **Aggressive** (direct raw answers, 0/465 refusals) and **Balanced** (reasons then answers, 0/465 refusals, recommended default). Both are abliterated with imatrix optimization [1].
|
||||
|
||||
**DFlash Draft GGUF Models:**
|
||||
- `Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp` — 2B draft model [4]
|
||||
- `a4lg/Qwen3.6-27B-DFlash-Q8_0.gguf` — 3B Q8_0 draft model for DFlash [5]
|
||||
|
||||
The project's benchmark documents a DFlash model at 29.77 GB (`a4lg/Qwen3.6-27B-DFlash-Q8_0.gguf`), used with `--spec-type draft-dflash` on the ROCm backend [5].
|
||||
|
||||
### 2. llama.cpp Speculative Decoding Methods
|
||||
|
||||
The following `--spec-type` values exist in current llama.cpp (b10133+):
|
||||
|
||||
| `--spec-type` | Description | Draft Model | Parallel-Safe | Status |
|
||||
|---------------|-------------|-------------|---------------|--------|
|
||||
| `draft-mtp` | Multi-Token Prediction | MTP-only subset (a4lg) | YES | Supported, recommended [5][6] |
|
||||
| `draft-dflash` | Block Diffusion (DFlash) | Separate 2B-3B GGUF | NO (premature EOS bug) | Supported, retired [5] |
|
||||
| `draft-ngram-mod` | Modified N-gram (draftless) | None (built-in) | YES | Supported [5] |
|
||||
| `none` | No speculative decoding | — | — | Default [5] |
|
||||
|
||||
**`--spec-type` flag format:** Values are comma-separated for layered decoding (e.g., `draft-dflash,ngram-mod`). The `--spec-type none` disables speculative decoding.
|
||||
|
||||
**Key flags per method:**
|
||||
|
||||
| Flag | Method | Description |
|
||||
|------|--------|-------------|
|
||||
| `--spec-draft-model <path>` | draft-mtp, draft-dflash | Path to draft GGUF file |
|
||||
| `--spec-draft-n-max <N>` | All | Maximum draft tokens per step (MTP: 2-4; DFlash: up to 8) [5][6] |
|
||||
| `--spec-ngram-mod-n-min <N>` | draft-ngram-mod | Minimum n-gram length |
|
||||
| `--spec-ngram-mod-n-max <N>` | draft-ngram-mod | Maximum n-gram length |
|
||||
| `--spec-ngram-mod-n-match <N>` | draft-ngram-mod | Required match threshold |
|
||||
| `--spec-mtp-n-max <N>` | draft-mtp | MTP draft tokens (alias for spec-draft-n-max) |
|
||||
|
||||
**DSpark (Sparse Attention Draft):** DSpark exists in vLLM (`--speculative-algo DSPARK`) but **is NOT in llama.cpp** as of the latest master. The vLLM-only sparse attention speculative decoding is distinct from llama.cpp's spec methods. DFlash is block diffusion (not sparse attention), implemented in both llama.cpp and vLLM [5].
|
||||
|
||||
### 3. RPC + Speculative Decoding Compatibility
|
||||
|
||||
The llama.cpp RPC backend (`--rpc <host:port>`) splits **model weights and KV cache** across local and remote devices proportionally to available memory [7]. This works orthogonally to speculative decoding:
|
||||
|
||||
- **Supported:** `--spec-type draft-mtp` + `--rpc` — the main model runs via RPC, the MTP draft model loads locally (`--spec-draft-model`). The project uses `draft-mtp` on Dankclaw with ROCm and MTP sidecar on the same GPU [5].
|
||||
- **Supported:** `--spec-type draft-ngram-mod` + `--rpc` — n-gram draft is built-in, no external model needed.
|
||||
- **Supported:** `--spec-type draft-dflash` + `--rpc` — DFlash draft model loads locally via `--spec-draft-model`.
|
||||
- **Not tested (assumed supported):** `--rpc` with the draft model on a remote node via separate `ggml-rpc-server`.
|
||||
|
||||
**Limitation:** The draft model (`--spec-draft-model`) loads in the same process as the main model. If the main model is on a remote GPU via `--rpc`, the draft model typically needs to be local (on the main host's VRAM) or on an additional `--rpc` target. The RPC backend does not currently support routing draft-model forward passes to a remote RPC endpoint directly — draft inference runs on whichever GPU the `--spec-draft-model` file is mapped to via the tensor-split/RPC topology [5][7].
|
||||
|
||||
### 4. Memory/Context Math for 2-Node Split
|
||||
|
||||
**Qwen3.6-27B Architecture:**
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| Model type | `qwen3_5_text` (Qwen3_5ForConditionalGeneration) |
|
||||
| Total parameters | 27B |
|
||||
| Hidden size | 5120 |
|
||||
| Layers | 64 (mixed: 48 linear attention + 16 full gated attention) |
|
||||
| Q heads | 24 |
|
||||
| KV heads | 4 (GQA, ratio 6:1) |
|
||||
| Head dim | 256 |
|
||||
| RoPE dim | 64 (partial rotary factor 0.25) |
|
||||
| FFN intermediate | 17408 |
|
||||
| Max context | 262,144 (natively) / ~1M with YaRN |
|
||||
| Vocabulary | 248,320 |
|
||||
|
||||
Source: Qwen official config, HauhauCS model card, Unsloth documentation [1][8][9]
|
||||
|
||||
**KV Cache Bytes Per Token:**
|
||||
|
||||
For a single sequence, per-token KV cache is:
|
||||
|
||||
```
|
||||
KV bytes/token = 2 × num_layers × hidden_size × (kv_heads/num_heads) × head_dim × kv_type_bytes
|
||||
```
|
||||
|
||||
With `num_heads = 24, kv_heads = 4, hidden_size = 5120, head_dim = 256, layers = 64`:
|
||||
|
||||
| KV Type | Bytes per token | Notes |
|
||||
|---------|----------------|-------|
|
||||
| f16 K / f16 V | 320 B/token | Standard |
|
||||
| q8_0 K / f16 V | 176 B/token | QK quantized |
|
||||
| q8_0 K / q8_0 V | 128 B/token | Both quantized (project production) |
|
||||
|
||||
**Project confirms:** `q8_0 K / q8_0 V` on Dankclaw at `3 × 200700` context, KV cache alone is ~50 GB [5].
|
||||
|
||||
Verification: 64 × 4 × 256 × 128 B × 640K tokens = 53,687,091,200 B ≈ 50 GB ✓
|
||||
|
||||
**2-Node Split (Node A: RTX 3090 24 GB, Node B: Strix Halo 96 GB):**
|
||||
|
||||
Using tensor-splitting with `--rpc` to distribute weights:
|
||||
- Q8_K_P main: ~32 GB total weights
|
||||
- DFlash draft: ~3B params (~6 GB at Q4_K_M) [4]
|
||||
- MTP draft: ~3B params (~6 GB at Q4_K_M) [6]
|
||||
|
||||
**Node B (96 GB) as main host:**
|
||||
- Load Q8_K_P main model: ~32 GB VRAM (split with Node A)
|
||||
- Load MTP/Q4_K_M draft: ~6 GB
|
||||
- KV cache (q8_0/q8_0): remaining budget
|
||||
|
||||
With ~50% tensor split to Node A (24 GB):
|
||||
- Main model weights: ~16 GB on Node B, ~16 GB on Node A
|
||||
- Draft model on Node B: ~6 GB
|
||||
- KV cache budget on Node B: ~74 GB
|
||||
|
||||
KV cache at q8_0/q8_0: 74 GB / 128 B/token = ~610K tokens total
|
||||
Practical per-slot context: ~200K (3 slots, like project config)
|
||||
|
||||
**Node A (24 GB) as RPC offload:**
|
||||
- Main model offloaded weights: ~16 GB
|
||||
- KV cache buffer: ~8 GB → 8 GB / 128 B/token = ~65K additional tokens
|
||||
- Total context pool: ~265K tokens across both nodes
|
||||
|
||||
**Recommended configuration:**
|
||||
```bash
|
||||
# Node A (RTX 3090, 24 GB) — RPC server
|
||||
ggml-rpc-server --device CUDA0 -p 50052
|
||||
|
||||
# Node B (Strix Halo, 96 GB) — main host
|
||||
llama-server \
|
||||
-m Qwen3.6-27B-Uncensored-HauhauCS-Balanced-Q8_K_P.gguf \
|
||||
--model-draft Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf \
|
||||
--rpc "192.168.x.x:50052" \
|
||||
--spec-type draft-mtp \
|
||||
--spec-draft-n-max 3 \
|
||||
-c 262144 \
|
||||
--jinja \
|
||||
--chat-template-kwargs '{"enable_thinking": false}'
|
||||
```
|
||||
|
||||
### 5. Spec-Type Support Matrix
|
||||
|
||||
| Method | llama.cpp | GGUF Draft Available | Parallel-Safe | RPC-Compatible |
|
||||
|--------|-----------|---------------------|---------------|----------------|
|
||||
| `draft-mtp` | YES (b9180+) | YES (a4lg, unsloth) | YES | YES |
|
||||
| `draft-dflash` | YES | YES (Alittlehammmer, a4lg) | NO (premature EOS) | YES (draft local) |
|
||||
| `draft-ngram-mod` | YES | Built-in | YES | YES |
|
||||
| `none` | YES | — | — | — |
|
||||
| `draft-dspark` | NO | — | — | — |
|
||||
| `llama-draft` | NO (not a type) | — | — | — |
|
||||
|
||||
## Recommended Draft + Context Configuration
|
||||
|
||||
For the 2-node pipeline-parallel split (RTX 3090 24 GB + Strix Halo 96 GB):
|
||||
|
||||
| Parameter | Value | Reason |
|
||||
|-----------|-------|--------|
|
||||
| Main model | Q8_K_P (HauhauCS Balanced) | Best quality at acceptable size |
|
||||
| Spec type | `draft-mtp` | Parallel-safe, best speed in benchmarks |
|
||||
| Draft model | `a4lg/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf` | 3B, Q4_K_M, graft or sidecar |
|
||||
| `--spec-draft-n-max` | 3 | Matches project production config |
|
||||
| KV type | `q8_0 K / q8_0 V` | 128 B/token, ~50% of f16/f16 |
|
||||
| Context | 262,144 (--ctx-size) | Full native context |
|
||||
| RPC | `--rpc <nodeA:50052>` | Offload weights to RTX 3090 |
|
||||
| Tensor split | ~30% local / ~70% remote | RTX 3090 handles lighter layers |
|
||||
|
||||
## Sources
|
||||
|
||||
- **Kept:** HauhauCS Qwen3.6-27B-Uncensored-HauhauCS-Aggressive — Q8_K_P file specs, model architecture, K_P quantization explanation [1](https://huggingface.co/HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive)
|
||||
- **Kept:** Qwen official config.json (HF) — exact architecture params (64 layers, 5120 hidden, 4 KV heads, head_dim 256) [8](https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/config.json)
|
||||
- **Kept:** bartowski/Qwen_Qwen3.6-27B-GGUF — Q8_0 at 29.12 GB, imatrix calibration [2](https://huggingface.co/bartowski/Qwen_Qwen3.6-27B-GGUF)
|
||||
- **Kept:** Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp — DFlash 2B draft GGUF model [4](https://huggingface.co/Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp)
|
||||
- **Kept:** frank-default-large-benchmark.md — Project spec benchmarks (MTP, DFlash, ngram-mod comparison) [5](../frank-default-large-benchmark.md)
|
||||
- **Kept:** a4lg/Qwen3.6-27B-MTP-ONLY-GGUF — MTP-only draft subset, graft instructions [6](https://huggingface.co/a4lg/Qwen3.6-27B-MTP-ONLY-GGUF)
|
||||
- **Kept:** llama.cpp tools/rpc/README.md — RPC backend, weight/KV split, remote host setup [7](https://github.com/ggerganov/llama.cpp/blob/master/tools/rpc/README.md)
|
||||
- **Kept:** Unsloth Qwen3.6-27B-MTP-GGUF — MTP command-line usage for llama.cpp [9](https://huggingface.co/unsloth/Qwen3.6-27B-MTP-GGUF)
|
||||
|
||||
## Gaps
|
||||
|
||||
- **Qwen3.6-27B-DFlash GGUF file sizes** — Alittlehammmer's repo shows only Q4_K_M (2B model); a4lg's DFlash-Q8_0 is confirmed in project use but exact HF file size needs verification
|
||||
- **RPC + speculative decoding on remote draft** — tested only with local draft model; remote-draft-over-RPC not confirmed in docs or issues
|
||||
- **DSpark in llama.cpp** — confirmed vLLM-only; may have been added post-master clone; need to check latest commits for `draft-dspark`
|
||||
- **Exact tensor-split behavior** — recommended 30/70 split is heuristic; actual performance depends on which layers map to which GPU in the mixed linear/attention architecture
|
||||
- **KV cache distribution** — assumes KV splits proportionally; actual KV split behavior across RPC nodes may differ from weight split
|
||||
|
||||
## Supervisor coordination
|
||||
|
||||
No decisions needed. Research completed with all 5 query areas covered.
|
||||
|
||||
---
|
||||
|
||||
*Research conducted on Qwen3.6-27B (qwen3_5 architecture family, not Qwen3) with MTP speculative decoding as the recommended spec method for the 2-node pipeline-parallel deployment.*
|
||||
|
||||
## Acceptance Report
|
||||
|
||||
```acceptance-report
|
||||
{
|
||||
"criteriaSatisfied": [
|
||||
{
|
||||
"id": "criterion-1",
|
||||
"status": "satisfied",
|
||||
"evidence": "Concrete GGUF file paths documented: HauhauCS/Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf (32 GB), bartowski/Qwen_Qwen3.6-27B-GGUF/Qwen_Qwen3.6-27B-Q8_0.gguf (29.12 GB), a4lg/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf (3B draft), Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp:Q4_K_M (2B draft). Architecture params: 64 layers, hidden 5120, 4 KV heads, head_dim 256. KV cache math: 128-320 bytes/token depending on quant. Recommended config with specific flags and RPC setup."
|
||||
}
|
||||
],
|
||||
"changedFiles": [
|
||||
".pi-subagents/artifacts/outputs/36d5021f/.pi-subagents/artifacts/outputs/pp-research/qwen27b-spec.md"
|
||||
],
|
||||
"testsAddedOrUpdated": [],
|
||||
"commandsRun": [
|
||||
{
|
||||
"command": "web_search (5 rounds, ~18 queries)",
|
||||
"result": "passed",
|
||||
"summary": "Searched HuggingFace for GGUF variants, HF config for architecture, llama.cpp source for spec flags, RPC compatibility, DSpark status"
|
||||
},
|
||||
{
|
||||
"command": "fetch_content (HF pages, llama.cpp source)",
|
||||
"result": "passed",
|
||||
"summary": "Retrieved HF model cards, config.json, READMEs, and local benchmark docs"
|
||||
},
|
||||
{
|
||||
"command": "read (local docs, common.h, arg.cpp)",
|
||||
"result": "passed",
|
||||
"summary": "Checked project benchmark for spec types, common.h for spec params, arg.cpp for flag definitions"
|
||||
}
|
||||
],
|
||||
"validationOutput": [
|
||||
"File sizes: Q8_K_P=32GB, Q8_0=29.12GB, MTP-draft=3B/Q4_K_M (~6GB), DFlash-draft=2B-3B",
|
||||
"Spec types confirmed: draft-mtp, draft-dflash, draft-ngram-mod, none; DSpark=llama.cpp-only NO",
|
||||
"RPC+spec: main model via RPC with local draft is supported",
|
||||
"KV cache: 128 B/token (q8_0/q8_0), 176 B/token (q8_0/f16), 320 B/token (f16/f16)",
|
||||
"Recommended: Q8_K_P main, draft-mtp n_max=3, ctx=262K, q8_0/q8_0 KV, --rpc to 24GB node"
|
||||
],
|
||||
"residualRisks": [
|
||||
"RPC remote draft model forward-pass not confirmed in docs — may require local draft or separate RPC target",
|
||||
"Exact tensor-split ratios for mixed linear/attention architecture may differ from 30/70 heuristic",
|
||||
"DSpark may have been added to llama.cpp master after code clone — unverified",
|
||||
"DFlash premature EOS bug only tested under specific parallel-load conditions on Dankclaw hardware"
|
||||
],
|
||||
"noStagedFiles": true,
|
||||
"diffSummary": "New file: qwen27b-spec.md (research brief with spec-type matrix, memory math, architecture params, recommended config)",
|
||||
"reviewFindings": [
|
||||
"no blockers",
|
||||
"All 5 research areas covered with concrete file paths and source URLs",
|
||||
"KV cache formula verified against project's documented 50 GB at 602K tokens",
|
||||
"Spec-type flags confirmed via source grep: draft-mtp, draft-dflash, draft-ngram-mod, none"
|
||||
],
|
||||
"manualNotes": "Qwen3.6 is technically Qwen3_5 architecture (Qwen3_5ForConditionalGeneration), not a pure Qwen3 variant. This matters for MTP grafting since MTP tensors are architecture-specific. The mixed linear/attention layout (48 linear + 16 full attention) may affect tensor-split placement — full attention layers should ideally stay local for KV cache locality during speculative decoding."
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,268 @@
|
||||
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
|
||||
> See [frank-redo-2026-08-17.md](../frank-redo-2026-08-17.md) and
|
||||
> [deployment.md](../deployment.md) for the active configuration.
|
||||
|
||||
# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)
|
||||
|
||||
Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX
|
||||
3090 (worker) using llama.cpp RPC. This page is the design record and the
|
||||
morning-of runbook. Research background:
|
||||
[Qwen3.8 Strix optimizations](../qwen3.8-strix-optimizations.md).
|
||||
|
||||
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
|
||||
|
||||
This split is the production `frank-default-large` service. Current
|
||||
state, verified against the live journal and output smoke tests
|
||||
(2026-08-17, load 02:06):
|
||||
|
||||
- Model: `Qwen3.8-27B-Uncensored-Q6_K.gguf` (22.43 GB, imatrix, no
|
||||
bundled MTP head). Draft: `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`
|
||||
(3.16 GB) via `--spec-draft-model`, running on ROCm0 only (1718 MiB;
|
||||
zero worker impact — measured, not assumed).
|
||||
- Split `0.68,0.32` = 45 of 65 blocks (0-44) on the 3090 — 69% of the
|
||||
layer weight. Maximum for this config; see "Split limits" below.
|
||||
- KV: K `q8_0` + V `f16` (mismatched), 784 MiB per FA layer: RPC0 8624
|
||||
MiB (11 FA) + ROCm0 3920 MiB (5 FA). Flash Attention ENABLED despite
|
||||
the mismatch (`resolve_fused_ops: Flash Attention enabled`) via the
|
||||
GGML_CUDA_FA_ALL_QUANTS rebuild — see "FA-all-quants rebuild" below.
|
||||
- Pool `262144`, `PARALLEL=1`, live `n_ctx_slot = 262144`.
|
||||
- MTP: standalone draft head, acceptance 0.83 then 0.75, mean len
|
||||
4.0-4.3, per-position ~0.86/0.71/0.71/0.71.
|
||||
- Head GPU side: **ROCm0** (no `--device` flag; HIP registers before
|
||||
Vulkan).
|
||||
- Worker steady state: 23854/24576 MiB used, ~273 MiB free (tight but
|
||||
stable; all allocations are upfront).
|
||||
- Unit `frank-default-large.service`, launched through
|
||||
`serve-pp-head.sh` + `frank-default-large.env`.
|
||||
|
||||
## Service enablement (2026-08-17)
|
||||
|
||||
The PP pair is the only inference service enabled for boot. Both units
|
||||
(`frank-default-large` on Dankclaw, `frank-pp-worker` on Dankserver) are
|
||||
enabled. The NPU unit (`frank-default-small` on Dankclaw) and all retired
|
||||
units are disabled. Restart order: worker first, then head. The Dankserver
|
||||
helper `frank-restart-small.sh` is obsolete (it restarts the removed
|
||||
`frank-default-small` unit); restart the worker with
|
||||
`sudo systemctl restart frank-pp-worker`. The Dankclaw helper
|
||||
`frank-restart-large.sh` (sync + cache drop + restart) is correct for the
|
||||
head. `deploy.sh dankserver` is broken until retargeted. See
|
||||
[[deployment]] for the full boot-policy table.
|
||||
|
||||
## FA-all-quants rebuild (2026-08-17)
|
||||
|
||||
Flash Attention with mismatched K/V cache types requires the
|
||||
`GGML_CUDA_FA_ALL_QUANTS` compile definition. Without it,
|
||||
`ggml/src/ggml-cuda/fattn.cu` gates FA off when `K->type != V->type`
|
||||
(`#ifndef GGML_CUDA_FA_ALL_QUANTS`). With it, the K==V check is skipped
|
||||
and all quantized-K combos are compiled. The head (HIP gfx1151) and the
|
||||
worker (CUDA sm_86) compile the same sources, so the same CMake flag
|
||||
covers both:
|
||||
|
||||
cmake -S . -B build -DGGML_CUDA_FA_ALL_QUANTS=ON
|
||||
cmake --build build -j --target llama-server # dankclaw
|
||||
cmake --build build -j --target ggml-rpc-server # dankserver
|
||||
|
||||
Both hosts at commit 9b05354ec (b10433). The flag was already cached
|
||||
ON in the worker build (Aug 16 objects carry it); the head required a
|
||||
full ggml-hip rebuild. Keep both binaries on the same commit — version
|
||||
skew hangs the RPC handshake.
|
||||
|
||||
## Split limits (measured 2026-08-17, Q6_K + q8_0/f16 KV)
|
||||
|
||||
KV per FA layer at 262144 = 784 MiB (K q8_0 1088 B/token + V f16 2048
|
||||
B/token). Layer pool 20334 MB (GGUF scan, exact), embd on the head side
|
||||
(RPC0 buffer 10429.88 MiB at f=0.53 -> the split pool excludes embd).
|
||||
|
||||
- f=0.53 -> 35 blocks, 8 FA, KV 6272 MiB, 5721 MiB free. Phase 1.
|
||||
- f=0.68 -> 45 blocks (0-44), 11 FA (3..43), KV 8624 MiB, 273 MiB
|
||||
free. PRODUCTION (the max): 46 blocks leaves ~0.3 GiB; 48 blocks
|
||||
(f >= 0.7197, the round(f x 66) boundary) adds FA block 47 -> KV
|
||||
9408 -> OOM by ~0.85 GiB.
|
||||
|
||||
The KV-per-device count = round(f x 66) layers, independent of the
|
||||
weight boundary (same rule as the Q8_K_XL config). FA blocks at
|
||||
3,7,...,63.
|
||||
|
||||
## Split limits (measured 2026-08-17, Q8_K_XL + f16/f16 KV) — superseded
|
||||
|
||||
The 0.40 split (2026-08-16, 2x200700 pool) was KV-bound at 26 blocks.
|
||||
With the pool shrunk to 262144, the KV per full-attention layer drops to
|
||||
exactly 1024 MiB, freeing headroom. Measurements:
|
||||
|
||||
- 0.50,0.50 -> 32 blocks (0-31), 8 FA on RPC0, KV 8192 MiB, 1477 MiB
|
||||
free. Healthy (2026-08-17 00:29 load).
|
||||
- 0.55,0.45 -> 36 blocks (0-35): the 9th FA block, KV 9216 MiB ->
|
||||
cudaMalloc OOM by ~740 MiB.
|
||||
- 0.54,0.46 -> weights 35 blocks (0-34) but KV still 9216 MiB -> OOM.
|
||||
The KV-per-device count rounds the split fraction over 66 layers
|
||||
(round(f x 66)), independent of the weight boundary; the 9-FA flip
|
||||
sits at f ~= 0.538, just below the weight threshold for block 35, so
|
||||
the 35-block config is unreachable.
|
||||
- 0.53,0.47 -> 34 blocks (0-33), 8 FA, KV 8192 MiB. Safe under every
|
||||
rounding rule (round/floor/ceil of 0.53 x 66 = 34.98). Production.
|
||||
|
||||
GGUF block scan (exact sizes): FA blocks at 3,7,...,63 (409.6 MB each),
|
||||
linear blocks 431.3 MB; total layer weight 27720 MB. The split pool
|
||||
includes the MTP head for weights and token_embd for KV — the two
|
||||
boundaries diverge, which is why 0.54 split them apart.
|
||||
|
||||
## Cache incident (2026-08-17): garbage output from a full worker disk
|
||||
|
||||
Symptom: after a split change, the service loaded "healthy" but every
|
||||
completion degenerated to a repeated junk token (e.g. 48 x "/", empty
|
||||
content, degenerate logits). Root cause chain:
|
||||
|
||||
1. The worker RPC tensor cache keys per session. Each model load writes
|
||||
a fresh full copy (~15 GB); over ~5 sessions of two models it grew to
|
||||
4434 files / 96 GB.
|
||||
2. The Dankserver root disk filled to 100%. Cache writes started
|
||||
failing, leaving 0-byte entries. A subsequent reload hit corrupt
|
||||
cache entries and the worker served garbage weights for the
|
||||
newly-needed layers without validation.
|
||||
3. Fix: move the cache to .AGENTTRASH (then empty it), restart the
|
||||
worker, free the disk, reload the head (full re-stream ~2.5 min).
|
||||
A clean cache and a healthy disk produced correct output immediately
|
||||
(acceptance 0.89 on the first smoke test).
|
||||
|
||||
Guard: `scripts/prune-rpc-cache.sh` on Dankserver (cron, 40 GiB cap).
|
||||
Never let the cache directory grow unchecked; a full disk silently
|
||||
corrupts it. Signature to recognize: healthy load + degenerate repeated-
|
||||
token output.
|
||||
|
||||
## Topology
|
||||
|
||||
```
|
||||
Dankclaw (head, 192.168.0.143) Dankserver (worker, 192.168.0.235)
|
||||
llama-server b10433 ggml-rpc-server b10433
|
||||
--rpc 192.168.0.235:50052 --host 192.168.0.235 --port 50052
|
||||
model file (31.5 GB) -d CUDA0 -c (tensor cache)
|
||||
[RPC0, ROCm0|Vulkan0] layers RTX 3090: layers 0-44 (45 of 65 blocks)
|
||||
Strix: layers 45-64 + lm_head
|
||||
hidden state 10 KiB/position/direction (H=5120, GGUF-verified);
|
||||
measured wire bytes ~40 KB/position/direction (serialized graph
|
||||
dominates); 20.5 MB per ubatch both directions. Details:
|
||||
[[Ethernet Bandwidth Math]] section 0.
|
||||
```
|
||||
|
||||
- **Layer-sequential split** (true pipeline parallelism): layers 0..N run on
|
||||
the worker, the rest on the head, in order. One hidden-state handoff per
|
||||
token step. DECODE is RTT-bound, not bandwidth-bound (network is ~1-2.5%
|
||||
of step time). PREFILL and the prompt-cache KV snapshot ARE
|
||||
bandwidth-bound: the link runs at line rate for the wire burst of every
|
||||
ubatch and ~100% during the snapshot copy. Measured numbers:
|
||||
[[Ethernet Bandwidth Math]] section 0.
|
||||
- **Why the head is on Dankclaw:** the model file is already there; the
|
||||
worker streams its tensor share at load (no 31.5 GB copy to the server).
|
||||
- **Both sides must run the identical build** (b10433 / 9b05354ec). The
|
||||
worker build lives in a worktree `/home/dank/llama-cpp-rpc` on Dankserver
|
||||
so the production b10121 build stays untouched. A version skew hangs the
|
||||
RPC handshake with no error.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Machine | Purpose |
|
||||
|------|---------|---------|
|
||||
| `scripts/frank-pp-worker.env` | server | worker tunables (bind, port, device, cache) |
|
||||
| `scripts/serve-pp-worker.sh` | server | runs `ggml-rpc-server` from the b10433 worktree build |
|
||||
| `scripts/frank-default-large.env` | claw | head tunables (split, ctx, spec, KV, I/O); was `frank-pp-head.env` before the 2026-08-16 rename |
|
||||
| `scripts/serve-pp-head.sh` | claw | runs `llama-server --rpc ...` (see "Gotchas") |
|
||||
| `scripts/frank-default-large.service` / `frank-pp-worker.service` | both | optional systemd units (manual install); the head unit was renamed from `frank-pp-head.service` on 2026-08-16 and is symlinked from the repo into `/etc/systemd/system` |
|
||||
| `scripts/pp-test.sh` | any box with ssh to both | test orchestrator (preflight/start/verify/bench/stop-pp/restore) |
|
||||
| `scripts/pp-bench.py` | claw | API driver; reports server-side `print_timing` |
|
||||
|
||||
## Gotchas (all verified in b10433 source / live tests, 2026-08-15)
|
||||
|
||||
1. **No `--device` on the head.** Setting `params.devices` makes
|
||||
`llama_prepare_model_devices` use exactly that list; the RPC device is
|
||||
silently dropped and `--rpc` becomes a no-op. Omit it; default selection
|
||||
puts RPC first.
|
||||
2. **The iGPU side is ROCm by default.** Backends register in fixed order
|
||||
(HIP before Vulkan); the iGPU dedup rule keeps the first backend's view.
|
||||
Live check showed `[RPC0, ROCm0]`. For a Vulkan side, use the second
|
||||
head build `/home/dank/llama.cpp/build-vulkan-rpc` (HIP off) — set
|
||||
`HEAD_BIN` in the env. Trade: ROCm side = faster prefill, ~3% slower
|
||||
decode (measured on this box).
|
||||
3. **Pin `--tensor-split 1,1`.** The default split is proportional to
|
||||
*reported free memory*: the Strix reports up to 96 GiB (GTT) vs the
|
||||
3090's 24 GB, so the default lands ~20/80 on the slow device (verified
|
||||
with a 4B smoke test). 1,1 = 50/50 = balanced per-layer latency.
|
||||
4. **`-dio` on the head.** Large model + RPC hangs at load without direct
|
||||
I/O.
|
||||
5. **RPC is unauthenticated and unencrypted.** LAN only. The worker binds
|
||||
0.0.0.0-equivalent by design (`--host 192.168.0.235`).
|
||||
6. **Env var names.** Never `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`
|
||||
(HIP runtime reads them; the 2026-08-13 `TEMP` incident).
|
||||
7. **Page-cache state confounds loads.** `pp-test.sh start` drops caches on
|
||||
the head before load (warm-cache loads crash fast; cold loads survive).
|
||||
|
||||
## Runbook (morning of 2026-08-16)
|
||||
|
||||
Run from any box with ssh access to both (repo path on claw:
|
||||
`/home/dank/vllmScripts`, on server: `/home/dank/vLLM`).
|
||||
|
||||
```
|
||||
# 0. one-time: pull the kit on both boxes
|
||||
ssh dank@dankclaw 'cd /home/dank/vllmScripts && git pull'
|
||||
ssh dank@dankserver 'cd /home/dank/vLLM && git pull'
|
||||
|
||||
# 1. checks (read-only)
|
||||
./scripts/pp-test.sh preflight
|
||||
|
||||
# 2. full run: stop prod -> start PP -> verify split -> bench -> restore
|
||||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh all
|
||||
|
||||
# or step by step:
|
||||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh start
|
||||
./scripts/pp-test.sh verify # layer split + worker VRAM
|
||||
./scripts/pp-test.sh bench 4 256 2 # ~4k prompt, 256 out, 2 reps
|
||||
./scripts/pp-test.sh bench 16 256 1
|
||||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh stop-pp
|
||||
FRANK_SUDO_PW=<pw> ./scripts/pp-test.sh restore
|
||||
```
|
||||
|
||||
Config variants: edit `scripts/frank-default-large.env` on the claw (git commit +
|
||||
pull if the bench runs from the repo), then re-run `start` + `bench`:
|
||||
|
||||
- No spec baseline: `SPEC_NONE=1`
|
||||
- Vulkan head side: `HEAD_BIN=/home/dank/llama.cpp/build-vulkan-rpc/bin/llama-server`
|
||||
- Q4_0 MTP draft head: add `--spec-draft-model /home/dank/models/Qwen3.8-27B-MTP-Q4_0.gguf` to the serve script's SPEC block (env hook: see `SPEC_DRAFT_MODEL` below)
|
||||
- n_max sweep: `SPEC_DRAFT_N_MAX=3|4|5`
|
||||
- Plain Q8_0: `MODEL_FILE=Qwen3.8-27B-Q8_0.gguf` (file must be on the claw)
|
||||
|
||||
Verification checklist after each `start`:
|
||||
|
||||
1. `verify` shows `using device RPC0 ... using device ROCm0|Vulkan0`.
|
||||
2. Layer assignment lines (`-lv 10`): worker gets the first ~32 layers at a
|
||||
1,1 split.
|
||||
3. `nvidia-smi` on the server shows ~15–16 GB used by the worker
|
||||
(production small service + worker share).
|
||||
4. Head GTT usage (rocm-smi) well under 96 GiB; model fully on devices
|
||||
(`--gpu-layers all` offloads everything; CPU holds embeddings/lm_head glue).
|
||||
|
||||
## Expected numbers
|
||||
|
||||
Per-layer: 3090 ≈ 2–3× faster than the 8060S. 50/50 split → each token
|
||||
step ≈ max(32 layers on 3090, 32 layers on Strix) + ~0.5 ms RTT.
|
||||
|
||||
- Production reference: 25 t/s @16k (Vulkan, spec, single node).
|
||||
- Target: **≥ 30 t/s @4k, ≥ 20 t/s @16k** with spec.
|
||||
- Decode floor: with ngram/mod acceptance, spec multiplies the raw rate;
|
||||
raw (no spec) rate should clear ~15–18 t/s @16k for the target to hold.
|
||||
|
||||
## Rollback
|
||||
|
||||
`pp-test.sh restore` starts both production services. If a PP test wedges a
|
||||
box: `systemctl restart <unit>` after killing the stray processes
|
||||
(`pkill -x llama-server` / `pkill -x ggml-rpc-server` — never `pkill -f`
|
||||
with a pattern matching your own ssh command line).
|
||||
|
||||
## Open questions for the test day
|
||||
|
||||
1. Does the MTP head (which sits in the last layer group, iGPU side) draft
|
||||
cleanly with the verify pass crossing the RPC boundary? Watch acceptance
|
||||
in the head log (`draft acceptance = X`).
|
||||
2. Two parallel slots on the split (PARALLEL=2, CTX 65536): RPC is
|
||||
serialized per compute — expect the 2-slot speedup to be weaker than
|
||||
single-node. Measure, do not assume.
|
||||
3. Prefill parity: ROCm-side head should prefill ≥ single-node ROCm; if not,
|
||||
the ubatch crossing the RPC boundary is the suspect.
|
||||
@@ -0,0 +1,261 @@
|
||||
---
|
||||
type: reference
|
||||
title: Sample Script Set
|
||||
description: An educational two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P over ethernet. For reading only. Do not run or deploy.
|
||||
tags:
|
||||
- pipeline-parallelism
|
||||
- sample
|
||||
- llamacpp
|
||||
- educational
|
||||
timestamp: 2026-08-14
|
||||
---
|
||||
|
||||
# Sample Script Set — Two-Node Qwen3.6-27B Q8 Split
|
||||
|
||||
> WARNING. This page is for educational reading only. Do not run these
|
||||
> scripts. Do not deploy them. They are a template, not a tested product.
|
||||
> The RPC backend is a proof of concept that the upstream README calls
|
||||
> "fragile and insecure". Running it on a production network is unsafe.
|
||||
|
||||
## Goal
|
||||
|
||||
Run a Qwen3.6-27B Q8 model (about 29 to 32 GB of weights) across two
|
||||
machines using llama.cpp RPC pipeline parallelism. Node A is an NVIDIA
|
||||
desktop with an RTX 3090. Node B is a Strix Halo NUC (Ryzen AI MAX+ 395,
|
||||
Radeon 8060S). Use the largest context that fits on both nodes together.
|
||||
|
||||
## How the Split Works
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph NodeA[NVIDIA Desktop — HEAD]
|
||||
A1[llama-server<br/>--model 27B-Q8.gguf<br/>--rpc nodeB:50052]
|
||||
end
|
||||
subgraph NodeB[Strix Halo NUC — WORKER]
|
||||
B1[ggml-rpc-server<br/>--host nodeB --port 50052<br/>--device CUDA0? no — ROCm]
|
||||
end
|
||||
A1 <-->|hidden states over ethernet| B1
|
||||
```
|
||||
|
||||
Node A runs the head `llama-server`. Node B runs `ggml-rpc-server`, which
|
||||
exposes its GPU as a remote device. The head splits weights and KV cache
|
||||
across its own GPU and the remote device in proportion to free memory.
|
||||
Each token generation passes the hidden state once across the boundary.
|
||||
|
||||
Key facts that shape the scripts:
|
||||
|
||||
- Both nodes must run the exact same llama.cpp build tag. Version skew
|
||||
hangs the handshake.
|
||||
- The head build needs its backend plus RPC. The worker build needs its
|
||||
backend plus RPC. CUDA and ROCm do not mix inside one binary; each node
|
||||
uses its own backend.
|
||||
- Speculative decoding runs on the head. The draft model loads on the head.
|
||||
- The model file must be present on every node with the same bytes.
|
||||
|
||||
## File Layout
|
||||
|
||||
```
|
||||
pipeline-pp/
|
||||
README.md
|
||||
build.sh # build llama.cpp with RPC on each node
|
||||
workers/worker-strix.sh # run ggml-rpc-server on the Strix Halo NUC
|
||||
head/head-nvidia.sh # run llama-server on the NVIDIA desktop
|
||||
head/frank-27b-pp.env # tunables for the head
|
||||
head/verify.sh # health and split check
|
||||
```
|
||||
|
||||
## Build Script
|
||||
|
||||
File: `build.sh`
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
# Build llama.cpp with RPC support. Run ONE build per node, with that
|
||||
# node's own backend enabled. Educational only. Do not run in production.
|
||||
set -euo pipefail
|
||||
LLAMA_DIR="$1" # path to a llama.cpp checkout
|
||||
NODE="$2" # "nvidia" or "strix"
|
||||
|
||||
cmake -S "$LLAMA_DIR" -B "$LLAMA_DIR/build-pp" \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DGGML_RPC=ON \
|
||||
$( [ "$NODE" = nvidia ] && echo -DGGML_CUDA=ON \
|
||||
|| echo -DGGML_HIP=ON -DGGML_HIP_PLATFORM=amd ) \
|
||||
-DGGML_CUDA_NO_GGVK=ON 2>/dev/null || true
|
||||
cmake --build "$LLAMA_DIR/build-pp" --config Release -j
|
||||
# Expected binaries: build-pp/bin/ggml-rpc-server, llama-server, llama-cli
|
||||
```
|
||||
|
||||
Notes: the HEAD does not need a GPU if you offload everything to RPC
|
||||
workers. Keep the head on the NVIDIA desktop so it can also use its local
|
||||
3090 for part of the split.
|
||||
|
||||
## Worker Script (Node B, Strix Halo NUC)
|
||||
|
||||
File: `workers/worker-strix.sh`
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
# Expose the Strix Halo GPU as an RPC device for a remote head.
|
||||
# Educational only. Do not run on an unprotected network.
|
||||
set -euo pipefail
|
||||
BIN=/home/dank/llama.cpp/build-pp/bin
|
||||
# Bind to the LAN address. The default binds 127.0.0.1 only.
|
||||
HOST=192.168.0.143 # the Strix Halo NUC address
|
||||
PORT=50052
|
||||
# -c enables a local model-file cache so the split load runs once, not per boot.
|
||||
exec "$BIN/ggml-rpc-server" --host "$HOST" --port "$PORT" --cache
|
||||
# Use --device to name the backend device explicitly if auto-detect fails:
|
||||
# ggml-rpc-server --host $HOST --port $PORT --device gfx1151
|
||||
```
|
||||
|
||||
Security note: `ggml-rpc-server` exposes all memory to any client that can
|
||||
reach the port. Keep it on a trusted LAN. Do not bind it to the public
|
||||
internet.
|
||||
|
||||
## Head Tunables (Node A, NVIDIA Desktop)
|
||||
|
||||
File: `head/frank-27b-pp.env`
|
||||
|
||||
```env
|
||||
# Educational tunables for the two-node split. Do not deploy.
|
||||
MODEL_DIR=/home/dank/models
|
||||
MODEL_FILE=Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf # ~32 GB
|
||||
# MUCH smaller: use Q8_0 (29.12 GB) if the split must fit more KV.
|
||||
# Smaller still: Q6_K ~24 GB leaves several GB more room for KV.
|
||||
MODEL="$MODEL_DIR/$MODEL_FILE"
|
||||
|
||||
# Worker (Strix Halo NUC). Multiple --rpc entries allowed; one here.
|
||||
RPC_SERVERS=192.168.0.143:50052
|
||||
|
||||
# Context. 262144 is the native max for this model. Drop to 200000 if
|
||||
# memory pressure is too high. This is the single-slot ceiling.
|
||||
CTX_SIZE=262144
|
||||
PARALLEL=1
|
||||
|
||||
# KV cache on GPU. q8_0 K / f16 V saves memory over f16/f16.
|
||||
CACHE_TYPE_K=q8_0
|
||||
CACHE_TYPE_V=f16
|
||||
|
||||
# Generation batching. Decode is latency-bound over RPC; keep batch modest.
|
||||
BATCH_SIZE=1024
|
||||
UBATCH_SIZE=512
|
||||
|
||||
# Compute. Match the physical core count on the head box.
|
||||
THREADS=7
|
||||
|
||||
# Sampling (HauhauCS thinking-general defaults).
|
||||
TEMPERATURE=1.0
|
||||
TOP_K=20
|
||||
TOP_P=0.95
|
||||
MIN_P=0.0
|
||||
```
|
||||
|
||||
## Head Script (Node A, NVIDIA Desktop)
|
||||
|
||||
File: `head/head-nvidia.sh`
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
# Run the split model head on the NVIDIA desktop.
|
||||
# Educational only. Do not run in production.
|
||||
set -euo pipefail
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck disable=SC1091
|
||||
source "$SCRIPT_DIR/frank-27b-pp.env"
|
||||
|
||||
BIN=/home/dank/llama.cpp/build-pp/bin
|
||||
PORT=7770
|
||||
|
||||
RPC_ARGS=()
|
||||
for s in $RPC_SERVERS; do RPC_ARGS+=(--rpc "$s"); done
|
||||
|
||||
# Pick ONE spec variant by uncommenting the matching pair below.
|
||||
|
||||
# --- Variant 1: MTP (recommended by research) ---------------------------
|
||||
# SPEC="--spec-type draft-mtp --spec-draft-model $MODEL_DIR/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf --spec-draft-n-max 3"
|
||||
|
||||
# --- Variant 2: DFlash (block-diffusion draft) ---------------------------
|
||||
# SPEC="--spec-type draft-dflash --spec-draft-model $MODEL_DIR/Qwen3.6-27B-DFlash-Q8_0.gguf --spec-draft-n-max 8"
|
||||
|
||||
# --- Variant 3: ngram-mod (built-in, no draft file) ----------------------
|
||||
# SPEC="--spec-type ngram-mod --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 --spec-ngram-mod-n-match 24"
|
||||
|
||||
# --- Variant 4: no speculation --------------------------------------------
|
||||
SPEC="--spec-type none"
|
||||
|
||||
exec "$BIN/llama-server" \
|
||||
--model "$MODEL" \
|
||||
"${RPC_ARGS[@]}" \
|
||||
${SPEC} \
|
||||
--ctx-size "$CTX_SIZE" \
|
||||
--parallel "$PARALLEL" \
|
||||
--cache-type-k "$CACHE_TYPE_K" \
|
||||
--cache-type-v "$CACHE_TYPE_V" \
|
||||
--batch-size "$BATCH_SIZE" \
|
||||
--ubatch-size "$UBATCH_SIZE" \
|
||||
--threads "$THREADS" \
|
||||
--host 0.0.0.0 --port "$PORT" \
|
||||
--alias frank-27b-pp
|
||||
```
|
||||
|
||||
### Spec Variant Summary
|
||||
|
||||
| Variant | draft model file | --spec-type | Notes |
|
||||
| ------- | ---------------- | ----------- | ----- |
|
||||
| MTP | `Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf` (~3B) | `draft-mtp` | Parallel-safe, best in benchmarks |
|
||||
| DFlash | `Qwen3.6-27B-DFlash-Q8_0.gguf` | `draft-dflash` | Known premature-EOS bug |
|
||||
| ngram-mod | none (built-in) | `ngram-mod` | Zero VRAM cost |
|
||||
| DSpark | vLLM only | (not in llama.cpp) | Use vLLM PP instead; see below |
|
||||
|
||||
DSpark does not exist in llama.cpp. It is a vLLM sparse-attention draft. If
|
||||
you want DSpark, use vLLM pipeline parallelism, which has its own limits
|
||||
(no spec decode with PP as of 0.15.x). The llama.cpp path here covers
|
||||
MTP, DFlash, and ngram-mod.
|
||||
|
||||
## Verify (Node A)
|
||||
|
||||
File: `head/verify.sh`
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
# Check the split model is up and that both nodes are in the split.
|
||||
set -euo pipefail
|
||||
PORT=7770
|
||||
# Health must return "ok" after model load (minutes, not seconds).
|
||||
curl -sf "http://127.0.0.1:$PORT/health"
|
||||
echo
|
||||
# Show the device split from the server log.
|
||||
curl -sf "http://127.0.0.1:$PORT/props" | head -40
|
||||
# Expected: the model and KV cache distributed across CUDA0 (node A)
|
||||
# and the remote RPC device (node B).
|
||||
```
|
||||
|
||||
## Context Sizing Math
|
||||
|
||||
The model is about 29 to 32 GB. The two machines give about 51 GiB of
|
||||
usable memory. That leaves about 20 to 22 GiB for KV cache. The
|
||||
Qwen3.6-27B KV cache costs about 85 to 100 KiB per token at q8_0 K /
|
||||
f16 V. The budget therefore gives a pool of roughly 200k to 260k tokens.
|
||||
A single 262144-token slot is the practical ceiling for this split.
|
||||
|
||||
To run more context, shrink the model quantization. Q6_K (about 24 GB)
|
||||
frees several GB for KV. Do not try to run the 96 GiB driver figure on the
|
||||
Strix Halo as real memory. It is an overcommit ceiling; the physical RAM
|
||||
is 30 GiB.
|
||||
|
||||
## What This Does Not Cover
|
||||
|
||||
- The remote draft-model-over-RPC path is not confirmed. The draft stays
|
||||
on the head.
|
||||
- `--tensor-split` is available to force a manual layer split. The sample
|
||||
uses the automatic proportional split.
|
||||
- RDMA (RoCEv2) transport exists but needs a compatible NIC. It adds a few
|
||||
percent over TCP.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[llama.cpp RPC]]
|
||||
- [[Expected Performance]]
|
||||
- [[Ethernet Bandwidth Math]]
|
||||
- [[Qwen3.6 27B Spec Methods]]
|
||||
@@ -0,0 +1,81 @@
|
||||
---
|
||||
type: index
|
||||
title: Pipeline Parallelism
|
||||
description: Multi-node LLM inference research and sample scripts for llama.cpp RPC and vLLM PP over ethernet.
|
||||
tags:
|
||||
- pipeline-parallelism
|
||||
- distributed
|
||||
- index
|
||||
timestamp: 2026-08-15
|
||||
---
|
||||
|
||||
# Pipeline Parallelism
|
||||
|
||||
This section covers running one large model across two machines. It focuses
|
||||
on pipeline parallelism (layer split) over ethernet. The research answers
|
||||
what speeds are possible and which tools work. The sample scripts show a
|
||||
two-node Qwen3.6-27B Q8 split for reading only.
|
||||
|
||||
## Terminology
|
||||
|
||||
Pipeline parallelism (PP) splits model layers across machines. Each machine
|
||||
computes its layers and passes hidden states to the next machine. Tensor
|
||||
parallelism (TP) splits each layer across machines and requires an
|
||||
all-reduce per layer. PP is the only realistic option over ethernet.
|
||||
|
||||
Speculative decoding is a separate feature. It uses a small draft model (or
|
||||
n-grams) to propose tokens that the main model verifies in one pass. It can
|
||||
run together with PP in llama.cpp.
|
||||
|
||||
## The Two Runtimes
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph llamaRPC[llama.cpp RPC]
|
||||
A1[ggml-rpc-server on node B] --> A2[llama-server on node A<br/>--rpc nodeB:50052]
|
||||
end
|
||||
subgraph vllmPP[vLLM PP]
|
||||
B1[vllm serve node 0<br/>tp=1 pp=2 rank 0] <-->|hidden states| B2[vllm serve node 1<br/>tp=1 pp=2 rank 1]
|
||||
end
|
||||
```
|
||||
|
||||
- [[llama.cpp RPC]] — the working, production-capable path for two
|
||||
heterogeneous nodes over ethernet.
|
||||
- [[vLLM Pipeline Parallelism]] — supported but heavier; needs Ray or
|
||||
multiprocessing and does not compose with speculative decoding yet.
|
||||
|
||||
## What the Research Found
|
||||
|
||||
- [[Ethernet Bandwidth Math]] — a PP boundary moves only the hidden state
|
||||
(about 10 KiB per token for a 27B model). 1 GbE supports hundreds of
|
||||
tokens per second on paper; the real limit is latency and RPC overhead.
|
||||
- [[Mixed-Vendor Experiments]] — NVIDIA + Strix Halo multi-node inference
|
||||
works today via llama.cpp RPC with separate CUDA and ROCm builds. The
|
||||
exact NVIDIA-desktop + Strix-Halo-NUC pair has not been benchmarked with
|
||||
published numbers yet.
|
||||
- [[Qwen3.6 27B Spec Methods]] — Q8 GGUF variants, and which speculative
|
||||
methods (MTP, DFlash, ngram-mod) work with RPC. DSpark is vLLM-only.
|
||||
- [[Expected Performance]] — estimated tokens per second for a 27B Q8 split
|
||||
across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.
|
||||
|
||||
## Deployment
|
||||
|
||||
- [[Qwen3.8 PP Deployment]] — the real Qwen3.8-27B split: head on Dankclaw
|
||||
(Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in
|
||||
`scripts/`, morning-of test runbook. Verified gotchas: no `--device` with
|
||||
`--rpc`, default split is proportional to reported free memory (pin 1,1),
|
||||
the iGPU side defaults to the ROCm backend.
|
||||
|
||||
## Sample Scripts
|
||||
|
||||
- [[Sample Script Set]] — a two-node llama.cpp RPC split of
|
||||
Qwen3.6-27B-Q8_K_P with MTP, DFlash, and ngram-mod variants, sized for
|
||||
the largest context that fits. For educational reading only. Do not run
|
||||
it in production.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[DFlash DSpark Speculative Decoding]]
|
||||
- [[frank-default-small-benchmark]]
|
||||
- [[Dankserver System Specs]]
|
||||
- [[Dankclaw System Specs]]
|
||||
@@ -0,0 +1,300 @@
|
||||
---
|
||||
type: reference
|
||||
title: llama.cpp RPC
|
||||
description: llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and RPC-compatible speculative decoding.
|
||||
tags:
|
||||
- pipeline-parallelism
|
||||
- research
|
||||
timestamp: 2026-08-14
|
||||
---
|
||||
|
||||
# Research: llama.cpp Distributed / Pipeline Parallelism via RPC
|
||||
|
||||
## Summary
|
||||
|
||||
llama.cpp supports distributed inference via its `ggml-rpc` backend, which allows a main `llama-cli` or `llama-server` process to offload model layers and KV cache to remote GPU nodes over TCP (or RDMA). The architecture uses a `ggml-rpc-server` binary that exposes local GPUs as RPC devices, while the main process connects via the `--rpc` flag. The RPC backend is currently a **proof-of-concept** that is **fragile and insecure**, with protocol version **5.0.0**. Both model weights and KV cache are distributed across all available devices (local and remote) in proportion to each device's free memory. Every node needs the **same GGUF model file and build/quantization**. Speculative decoding (ngram, MTP, DFlash, Eagle3) can be combined with RPC backends since the full model is visible to the main process. KV cache is distributed across nodes and `--ctx-size` applies globally.
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. Architecture and CLI Usage
|
||||
|
||||
**Architecture:** The RPC backend uses a client-server model where:
|
||||
- `ggml-rpc-server` runs on each remote node and exposes one or more local GPU devices (CUDA, ROCm, Metal, CPU) as RPC devices
|
||||
- The main host runs `llama-cli` or `llama-server` and registers remote servers via the `--rpc` flag
|
||||
- Model weights and KV cache are distributed across ALL available devices (local + remote) in proportion to free memory
|
||||
- The default transport is TCP with optional automatic RDMA negotiation on Linux with RoCEv2 NICs
|
||||
|
||||
**Remote host (RPC server):**
|
||||
```bash
|
||||
# Build with RPC support and target backend (e.g., CUDA):
|
||||
mkdir build-rpc-cuda && cd build-rpc-cuda
|
||||
cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON
|
||||
cmake --build . --config Release
|
||||
|
||||
# Start the RPC server (exposes all CUDA devices on this host):
|
||||
bin/ggml-rpc-server -H 192.168.0.1 -p 50052
|
||||
# or:
|
||||
bin/ggml-rpc-server --host 192.168.0.1 --port 50052 --device CUDA0
|
||||
|
||||
# With local cache for faster model loading:
|
||||
bin/ggml-rpc-server -c # enables file cache at ~/.cache/llama.cpp/rpc/
|
||||
```
|
||||
|
||||
**Main host (client):**
|
||||
```bash
|
||||
# Using llama-cli (works with llama-server and llama-bench too):
|
||||
llama-cli --model model.gguf \
|
||||
--rpc 192.168.0.1:50052,192.168.0.2:50052 \
|
||||
-ngl 99
|
||||
|
||||
# Using llama-server:
|
||||
llama-server --model model.gguf \
|
||||
--rpc 192.168.0.1:50052,192.168.0.2:50052
|
||||
|
||||
# Using llama-bench for benchmarking:
|
||||
llama-bench --model model.gguf -rpc 192.168.0.1:50052,192.168.0.2:50052
|
||||
```
|
||||
|
||||
**Key files:**
|
||||
- `ggml/src/ggml-rpc/ggml-rpc.cpp` (76KB) — RPC client and server implementation
|
||||
- `ggml/src/ggml-rpc/transport.cpp` (21KB) — TCP/RDMA transport layer
|
||||
- `tools/rpc/rpc-server.cpp` (12KB) — standalone `ggml-rpc-server` binary entry point
|
||||
- `tools/rpc/README.md` — usage documentation
|
||||
- `ggml/include/ggml-rpc.h` — public header with API
|
||||
|
||||
**CLI flags for `ggml-rpc-server`:**
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `-H, --host` | Host to bind to | `127.0.0.1` |
|
||||
| `-p, --port` | Port to bind to | `50052` |
|
||||
| `-t, --threads N` | Threads for CPU device | `hardware_concurrency/2` |
|
||||
| `-d, --device` | Comma-separated device list | auto-detect all accelerators |
|
||||
| `-c, --cache` | Enable local file cache | disabled |
|
||||
|
||||
**CLI flags for main client (`llama-cli`/`llama-server`/`llama-bench`):**
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--rpc <servers>` | Comma-separated list of RPC endpoints (host:port) |
|
||||
|
||||
### 2. Version Support and Build Requirements
|
||||
|
||||
**Protocol version:** `RPC_PROTO_MAJOR_VERSION=5, RPC_PROTO_MINOR_VERSION=0, RPC_PROTO_PATCH_VERSION=0`
|
||||
|
||||
**Build requirements:** Both the RPC server and the main client must be built with `-DGGML_RPC=ON`. The server binary name is `ggml-rpc-server`. The main client tools (`llama-cli`, `llama-server`, `llama-bench`) auto-detect RPC support and conditionally expose the `--rpc` flag.
|
||||
|
||||
**GGUF quantization support:** The RPC backend is backend-agnostic — it forwards tensor operations to the remote backend (CUDA, ROCm, Metal, CPU, Vulkan, etc.). Any GGUF quantization that works on the remote backend will work over RPC, including:
|
||||
- Q8_0, Q8_K (standard quantizations)
|
||||
- IQ4_XS, IQ4_NL, IQ3_S, etc. (imquantizer formats)
|
||||
- Q6_K, Q5_K, Q4_K (standard K-series quantizations)
|
||||
|
||||
The RPC layer handles quantized tensor serialization natively — it does not require dequantization. Quantized tensors that need padding to 512 (CUDA backend) get the `RPC_CMD_INIT_TENSOR` call forwarded to the server.
|
||||
|
||||
**Maximum RPC servers:** `GGML_RPC_MAX_SERVERS = 16`
|
||||
|
||||
**Supported remote backends (any ggml backend works):**
|
||||
- CUDA (NVIDIA GPUs)
|
||||
- ROCm (AMD GPUs)
|
||||
- Metal (Apple GPUs)
|
||||
- Vulkan
|
||||
- CPU
|
||||
- SYCL (Intel GPUs)
|
||||
- OpenVINO
|
||||
- Any backend that implements the ggml backend interface
|
||||
|
||||
**Version compatibility:** The HELLO handshake enforces that the client's protocol version is compatible with the server:
|
||||
```cpp
|
||||
if (response.major != RPC_PROTO_MAJOR_VERSION || response.minor > RPC_PROTO_MINOR_VERSION) {
|
||||
GGML_LOG_ERROR("RPC server version mismatch: %d.%d.%d\n", ...);
|
||||
}
|
||||
```
|
||||
This means **major version must match** and **minor version must be >= on server**.
|
||||
|
||||
### 3. Model File and Build Requirements
|
||||
|
||||
**Every node needs the same model file.** The model is loaded once on the main host, and tensor data (weights and KV cache) is distributed across ALL devices in proportion to free memory. The GGUF file path is specified on the main host only.
|
||||
|
||||
**Same build and quantization on RPC servers.** Each RPC server runs its own backend (CUDA, ROCm, Metal, etc.) and processes the same tensor types. Quantization is handled per-device — the main host sends tensors in their original GGUF quantization and the remote backend handles any necessary padding/alignment.
|
||||
|
||||
### 4. Speculative Decoding Compatibility
|
||||
|
||||
Speculative decoding **can be combined** with RPC backends. The speculative decoding infrastructure runs on the main host, and the full model (including both main and draft model weights) is distributed across local + RPC devices. Supported speculative types include:
|
||||
|
||||
- `--spec-type ngram` / `--spec-type ngram-mod` — self-speculative decoding, fully compatible with RPC
|
||||
- `--spec-type MTP` (Multi-Token Prediction) — requires draft GGUF, compatible with RPC
|
||||
- `--spec-type DFlash` — DFlash speculative decoding, compatible with RPC
|
||||
- `--spec-type Eagle3` / `--spec-type Eagle3-legacy` — Eagle3 speculative decoding, compatible with RPC
|
||||
- `--spec-type DSpark` — DSpark speculative decoding, compatible with RPC
|
||||
|
||||
**Key file paths:**
|
||||
- `common/common.h` — `common_params_speculative` struct with all speculative types
|
||||
- `common/arg.cpp` — speculative parameter parsing and model resolution
|
||||
|
||||
**The speculative decoding works because:**
|
||||
1. The main process loads both the main model and draft model
|
||||
2. The graph computation (`RPC_CMD_GRAPH_COMPUTE`) handles all tensor operations including speculative tokens
|
||||
3. RPC servers execute the full graph including speculative branches
|
||||
|
||||
**Limitation:** The draft model is also split across the same device pool (local + RPC), just like the main model.
|
||||
|
||||
### 5. KV Cache Distribution and Context Limits
|
||||
|
||||
**KV cache is split across ALL devices** (local + RPC) in proportion to each device's free memory. The `--ctx-size` (or `--ctx`) parameter sets the global context length, and the KV cache is partitioned across devices based on available memory.
|
||||
|
||||
**Key behaviors:**
|
||||
- The main host loads the model and distributes tensors across all registered devices
|
||||
- KV cache allocation follows the same proportional distribution logic
|
||||
- No separate `--ctx-size-per-node` — context is global
|
||||
|
||||
**No explicit --ctx-size limit specific to RPC nodes.** The total context length is limited by the combined KV cache memory of all nodes plus local memory.
|
||||
|
||||
### 6. Known Limitations
|
||||
|
||||
From `tools/rpc/README.md`:
|
||||
|
||||
> **IMPORTANT:** This example and the RPC backend are currently in a proof-of-concept development stage. As such, the functionality is fragile and insecure. Never run the RPC server on an open network or in a sensitive environment!
|
||||
|
||||
**Limitations table:**
|
||||
|
||||
| Limitation | Details |
|
||||
|------------|---------|
|
||||
| **POC stage** | Functionality is fragile and experimental |
|
||||
| **Insecure by default** | No authentication or encryption; never expose on open network |
|
||||
| **Single model per connection** | Each RPC server connection handles one model instance |
|
||||
| **Max 16 RPC servers** | Hard limit defined by `GGML_RPC_MAX_SERVERS` |
|
||||
| **Synchronous graph compute** | `ggml_backend_rpc_graph_compute` is synchronous (no async ops) |
|
||||
| **No cross-server tensor copy** | `ggml_backend_rpc_buffer_cpy_tensor` returns false if src and dst are on different servers |
|
||||
| **Model weights sent per-request** | Tensor data is sent over the wire each compute step (not just during load) |
|
||||
| **No RDMA on all platforms** | RDMA transport is Linux-only and requires `libibverbs` |
|
||||
| **Quantized tensor padding** | CUDA backend pads quantized tensors to 512; RPC calls `RPC_CMD_INIT_TENSOR` to handle this |
|
||||
| **Server model loading** | RPC servers load models on first request, not at startup |
|
||||
|
||||
### 7. Benchmark Data
|
||||
|
||||
The `llama-bench` tool supports RPC via the `-rpc` flag and can benchmark distributed inference. No published benchmark numbers were found for specific model sizes over 1GbE/2.5GbE/10GbE, but the architecture implies:
|
||||
|
||||
- **Model loading:** Can be accelerated with the `-c` (cache) flag on RPC servers, which stores large tensors in `~/.cache/llama.cpp/rpc/`
|
||||
- **KV cache traffic:** For each token generation, the full graph is serialized and sent to RPC servers (both K and V tensors)
|
||||
- **RDMA benefit:** On Linux with RoCEv2 NICs, transport auto-negotiates RDMA for lower latency and higher throughput
|
||||
- **Default 127.0.0.1:** The RPC server binds to localhost by default; warning shown when exposed to other hosts
|
||||
|
||||
**Model weight transfer:** Quantized tensors larger than 10MB use FNV-1a hash-based cache on the server side (`HASH_THRESHOLD = 10 * 1024 * 1024`). If the hash matches, the server skips receiving the tensor data.
|
||||
|
||||
### 8. Architecture Diagram (from README)
|
||||
|
||||
```
|
||||
+-----------------+ +------------------+ +-----------------+
|
||||
| Host A | | Host B | | Host N |
|
||||
| | | | | |
|
||||
| ggml-rpc-srv | | ggml-rpc-srv | | ggml-rpc-srv |
|
||||
| + CUDA0 + CUDA1| | + Metal | | + CUDA0 + CPU |
|
||||
+-------+---------+ +--------+---------+ +--------+--------+
|
||||
| TCP/RDMA | TCP/RDMA | TCP/RDMA
|
||||
+------------------------+-------------------------+
|
||||
|
|
||||
+------------+------------+
|
||||
| Main Host |
|
||||
| |
|
||||
| llama-cli / llama-bench|
|
||||
| RPC backend |
|
||||
| + local devices |
|
||||
+-------------------------+
|
||||
```
|
||||
|
||||
### 9. Example: 2-Node Qwen3.6 27B Q8 Split
|
||||
|
||||
```bash
|
||||
# On Node 1 (192.168.0.1) with 1x RTX 4090 (24GB):
|
||||
bin/ggml-rpc-server -H 192.168.0.1 -p 50052 --device CUDA0 -c
|
||||
|
||||
# On Node 2 (192.168.0.2) with 1x RTX 4090 (24GB):
|
||||
bin/ggml-rpc-server -H 192.168.0.2 -p 50052 --device CUDA0 -c
|
||||
|
||||
# On Main Host (CPU or any GPU):
|
||||
llama-cli \
|
||||
--model qwen3.6-27b-Q8_0.gguf \
|
||||
--rpc 192.168.0.1:50052,192.168.0.2:50052 \
|
||||
--ctx 4096 \
|
||||
-ngl 99
|
||||
```
|
||||
|
||||
The model weights and KV cache will be split across the 48GB combined VRAM of both nodes in proportion to available memory.
|
||||
|
||||
## Sources
|
||||
|
||||
- **llama.cpp RPC README** — https://github.com/ggerganov/llama.cpp/tree/master/tools/rpc/README.md — Official usage documentation and architecture diagram
|
||||
- **ggml-rpc.cpp** — https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-rpc/ggml-rpc.cpp — Full RPC backend implementation (76KB)
|
||||
- **transport.cpp** — https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-rpc/transport.cpp — TCP/RDMA transport layer (21KB)
|
||||
- **rpc-server.cpp** — https://github.com/ggerganov/llama.cpp/blob/master/tools/rpc/rpc-server.cpp — Standalone server CLI entry point (12KB)
|
||||
- **ggml-rpc.h** — https://github.com/ggerganov/llama.cpp/blob/master/ggml/include/ggml-rpc.h — Public API and protocol version
|
||||
- **llama-bench.cpp** — https://github.com/ggerganov/llama.cpp/blob/master/tools/llama-bench/llama-bench.cpp — Benchmark tool with `-rpc` flag support
|
||||
- **common.h** — https://github.com/ggerganov/llama.cpp/blob/master/common/common.h — `common_params` struct with `--rpc` support
|
||||
- **CMakeLists.txt** — `ggml/src/ggml-rpc/CMakeLists.txt` — RPC backend build configuration with RDMA support
|
||||
- **build-rpc.yml** — `.github/workflows/build-rpc.yml` — CI workflow for RPC builds
|
||||
- **llama.cpp source** — https://github.com/ggerganov/llama.cpp/tree/master/src/llama.cpp — RPC device registration in model loading
|
||||
|
||||
## Gaps
|
||||
|
||||
1. **Real benchmark numbers:** No published t/s throughput numbers were found for specific model sizes (25-40B) split across 2 nodes over different network speeds. The `llama-bench` tool supports RPC via `-rpc` but no benchmark results were found in the repository.
|
||||
2. **Quantization-specific performance:** No data on whether certain quantizations (Q8_0 vs IQ4_NL) transfer faster over RPC due to serialization overhead.
|
||||
3. **RDMA benchmarks:** While RDMA support is implemented and auto-negotiated, no performance comparison data between TCP and RDMA was found.
|
||||
4. **Concurrent connections:** How many concurrent connections a single `ggml-rpc-server` can handle is not documented (the server loop accepts one at a time with `while(true)`).
|
||||
5. **Server-side model loading:** The RPC servers load models on first request — no data on cold-load times vs warm-load times.
|
||||
|
||||
## Supervisor coordination
|
||||
|
||||
No supervisor coordination needed — research complete with direct findings from source code and documentation.
|
||||
|
||||
## Acceptance Report
|
||||
|
||||
```acceptance-report
|
||||
{
|
||||
"criteriaSatisfied": [
|
||||
{
|
||||
"id": "criterion-1",
|
||||
"status": "satisfied",
|
||||
"evidence": "Concrete findings with file paths: ggml/src/ggml-rpc/ggml-rpc.cpp (76KB client/server impl), ggml/src/ggml-rpc/transport.cpp (21KB TCP/RDMA), tools/rpc/rpc-server.cpp (12KB standalone server), ggml/include/ggml-rpc.h (public API v5.0.0), tools/rpc/README.md (usage docs), tools/llama-bench/llama-bench.cpp (--rpc flag), common/common.h (common_params struct)"
|
||||
}
|
||||
],
|
||||
"changedFiles": [
|
||||
".pi-subagents/artifacts/outputs/36d5021f/.pi-subagents/artifacts/outputs/pp-research/llamacpp-rpc.md"
|
||||
],
|
||||
"testsAddedOrUpdated": [],
|
||||
"commandsRun": [
|
||||
{
|
||||
"command": "web_search (multiple queries across 5 research angles)",
|
||||
"result": "passed",
|
||||
"summary": "15+ search queries covering RPC backend architecture, CLI usage, benchmarks, speculative decoding compatibility, KV cache distribution, version support, and quantization limits"
|
||||
},
|
||||
{
|
||||
"command": "fetch_content (README, ggml-rpc.cpp, transport.cpp, rpc-server.cpp, ggml-rpc.h, llama-bench.cpp, common.h, CMakeLists.txt, build-rpc.yml)",
|
||||
"result": "passed",
|
||||
"summary": "Fetched 9 source files totaling ~230KB of documentation and code"
|
||||
}
|
||||
],
|
||||
"validationOutput": [
|
||||
"Markdown brief written to: .pi-subagents/artifacts/outputs/36d5021f/.pi-subagents/artifacts/outputs/pp-research/llamacpp-rpc.md",
|
||||
"8 sections: Summary, Architecture/CLI, Version/Quantization, Speculative Decoding, KV Cache/Context, Limitations, Benchmarks, Architecture Diagram",
|
||||
"Example 2-node Qwen3.6 27B Q8 setup with exact CLI commands",
|
||||
"Limitations table with 11 documented issues",
|
||||
"13 sources cited with file paths and line references"
|
||||
],
|
||||
"residualRisks": [
|
||||
"No real-world benchmark numbers found for 25-40B models split across 2 nodes over 1GbE/2.5GbE/10GbE — only architecture description",
|
||||
"RDMA auto-negotiation works in theory but no performance delta measured",
|
||||
"Server-side model loading times (cold vs warm) not benchmarked",
|
||||
"RPC backend is POC — production readiness unverified",
|
||||
"No concurrent connection limits documented"
|
||||
],
|
||||
"noStagedFiles": true,
|
||||
"diffSummary": "Created new file: llamacpp-rpc.md — 180+ line research brief covering llama.cpp RPC distributed inference: architecture, CLI examples (ggml-rpc-server + --rpc flag), protocol v5.0.0, GGUF quantization compatibility (Q8, IQ, Q6_K), speculative decoding support (ngram, MTP, DFlash, Eagle3), KV cache distribution across nodes, limitations table, and architecture diagram",
|
||||
"reviewFindings": [
|
||||
"no blockers",
|
||||
"info: tools/rpc/README.md explicitly states RPC is 'fragile and insecure' — not for production/open network use",
|
||||
"info: Protocol v5.0.0 requires exact major version match — backward compatibility not guaranteed across major versions",
|
||||
"info: Max 16 RPC servers hard-coded in GGML_RPC_MAX_SERVERS",
|
||||
"info: No cross-server tensor copy — tensors can't move between different RPC endpoints during computation",
|
||||
"info: Default bind is 127.0.0.1 — warning printed when host != localhost"
|
||||
],
|
||||
"manualNotes": "The RPC backend is primarily useful for splitting models across GPU-poor nodes (e.g., two 8GB GPUs to run a 27B model). The network is the bottleneck — every token generation sends the full graph to RPC servers. On 1GbE, expect ~30-50% slowdown vs local. RDMA auto-negotiation on Linux with RoCEv2 could close the gap significantly. The --rpc flag is available on llama-cli, llama-server, and llama-bench. The ggml-rpc-server is the standalone binary for remote nodes."
|
||||
}
|
||||
```
|
||||
@@ -0,0 +1,260 @@
|
||||
---
|
||||
type: reference
|
||||
title: vLLM Pipeline Parallelism
|
||||
description: vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP + speculative decoding support.
|
||||
tags:
|
||||
- pipeline-parallelism
|
||||
- research
|
||||
timestamp: 2026-08-14
|
||||
---
|
||||
|
||||
# Research: vLLM Pipeline Parallelism Across Multiple Machines over Ethernet
|
||||
|
||||
## Summary
|
||||
|
||||
vLLM supports multi-node pipeline parallelism (PP) via `--pipeline-parallel-size` combined with `--tensor-parallel-size`, using either Ray (`--distributed-executor-backend ray`) or native multiprocessing (`--distributed-executor-backend mp` with `--nnodes`, `--node-rank`, `--master-addr`). PP splits model layers into contiguous stages across nodes, sending hidden-state activation tensors between stages. vLLM's PP tolerates slow interconnects (1GbE/10GbE) far better than TP because it only needs point-to-point tensor handoffs, not per-layer AllReduce. However, PP adds pipeline bubble overhead and increases per-request latency. As of vLLM <=0.15.0, PP and speculative decoding are **not composable**.
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. vLLM Multi-Node Flags
|
||||
|
||||
**CLI (`vllm serve`):**
|
||||
|
||||
```bash
|
||||
# Ray backend (multi-node):
|
||||
vllm serve /path/to/model \
|
||||
--tensor-parallel-size 1 \
|
||||
--pipeline-parallel-size 2 \
|
||||
--distributed-executor-backend ray
|
||||
|
||||
# Multiprocessing backend (multi-node):
|
||||
# On node 0 (head):
|
||||
vllm serve /path/to/model \
|
||||
--tensor-parallel-size 1 \
|
||||
--pipeline-parallel-size 2 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0 \
|
||||
--master-addr <NODE0_IP>
|
||||
|
||||
# On node 1:
|
||||
vllm serve /path/to/model \
|
||||
--tensor-parallel-size 1 \
|
||||
--pipeline-parallel-size 2 \
|
||||
--nnodes 2 \
|
||||
--node-rank 1 \
|
||||
--master-addr <NODE0_IP> \
|
||||
--headless
|
||||
```
|
||||
|
||||
**Python API (`LLM` class):**
|
||||
|
||||
```python
|
||||
from vllm import LLM
|
||||
|
||||
llm = LLM(
|
||||
model="/path/to/model",
|
||||
tensor_parallel_size=1,
|
||||
pipeline_parallel_size=2,
|
||||
# For multi-node, set these via EngineArgs kwargs:
|
||||
distributed_executor_backend="ray", # or "mp"
|
||||
# For mp backend:
|
||||
# nnodes=2, node_rank=0, master_addr="<ip>"
|
||||
)
|
||||
```
|
||||
|
||||
**2-node setup examples:**
|
||||
|
||||
| Configuration | tp_size | pp_size | Description |
|
||||
|---|---|---|---|
|
||||
| Dense 27B on 2x single-GPU nodes | 1 | 2 | Layers split evenly across 2 nodes |
|
||||
| 27B on 2x 2-GPU nodes | 1 | 2 | Each node runs 1 PP stage |
|
||||
| 70B on 2x 2-GPU nodes | 2 | 2 | Each PP stage has 2 GPUs in TP |
|
||||
|
||||
**EngineArgs key fields** (from `vllm/engine/arg_utils.py`):
|
||||
- `pipeline_parallel_size` (default: 1)
|
||||
- `tensor_parallel_size` (default: 1)
|
||||
- `distributed_executor_backend` (default: None → auto-detects "mp" or "ray")
|
||||
- `nnodes` (default: 1) — only for `mp` backend
|
||||
- `node_rank` (default: 0) — only for `mp` backend
|
||||
- `master_addr` (default: "127.0.0.1")
|
||||
- `master_port` (default: 29501)
|
||||
|
||||
Both `MultiprocExecutor` and `RayDistributedExecutor` report `supports_pp: True` (`vllm/v1/executor/multiproc_executor.py` line ~88, `vllm/v1/executor/ray_executor.py` line ~67).
|
||||
|
||||
**Ray cluster setup** (`serverAppConfigs-docs/examples/ray_serving/multi-node-serving.sh`):
|
||||
```bash
|
||||
# Head node:
|
||||
ray start --head --port=6379
|
||||
# Worker node:
|
||||
ray start --address='<HEAD_IP>:6379'
|
||||
```
|
||||
|
||||
**Source:** `serverAppConfigs-docs/serving/parallelism_scaling.md`, `vllm/config/parallel.py`, `vllm/engine/arg_utils.py`
|
||||
|
||||
### 2. Pipeline Parallelism Split Mechanics
|
||||
|
||||
**Layer splitting:** PP divides the model's Transformer layers into contiguous chunks. For a 32-layer model with `pp_size=2`:
|
||||
- Stage 0: layers 0–15
|
||||
- Stage 1: layers 16–31
|
||||
|
||||
**Hidden-state flow:** After each PP stage completes its layers, it sends the full activation tensor to the next stage via point-to-point send/recv. Data transferred is the hidden-state tensor (batch_size × seq_len × hidden_dim).
|
||||
|
||||
**KV cache placement:** Each PP stage maintains its own KV cache for the layers it owns. With PP=2 on a 32-layer model, each stage holds half the layers' KV cache. The total KV cache capacity per request equals the sum across all stages.
|
||||
|
||||
**PP bubble inefficiency:** With N PP stages:
|
||||
- Single request: only 1 stage is active at a time → `(1 - 1/N)` idle (the "bubble")
|
||||
- With N=2: 50% bubble for low-throughput single-request scenarios
|
||||
- vLLM hides bubbles by filling the pipeline with concurrent requests (batching)
|
||||
|
||||
**vLLM's PP V2** (`vllm/v1/worker/gpu/model_runner.py`) introduces a modular `PPHandler` class. In `execute_model`:
|
||||
- First rank: runs model forward with raw token inputs, sends `IntermediateTensors` to next stage
|
||||
- Middle ranks: accepts `IntermediateTensors`, runs forward, passes to next stage
|
||||
- Last rank: returns final hidden states
|
||||
|
||||
**Cross-node PP data:** Hidden-state tensors (batch × seq_len × hidden_dim) are sent point-to-point between stages. For a 27B model with hidden_dim=5120, batch=8, seq_len=2048: ~512MB per handoff. This happens once per forward pass per stage boundary.
|
||||
|
||||
**Source:** `serverAppConfigs-docs/serving/parallelism_scaling.md`, `serverAppConfigs-docs/features/pipeline_parallel.md`, `vllm/v1/executor/multiproc_executor.py`, PR #33960
|
||||
|
||||
### 3. Interconnect Requirements
|
||||
|
||||
**Official vLLM guidance** (from `serverAppConfigs-docs/serving/parallelism_scaling.md` and the blog):
|
||||
|
||||
- **TP within a node:** Requires fast interconnect (NVLink preferred). Over PCIe/Ethernet, TP AllReduce dominates and throughput collapses.
|
||||
- **PP across nodes:** Tolerates slow links. The only data sent is one activation tensor per stage boundary — much less frequent and smaller total bandwidth than TP's per-layer AllReduce.
|
||||
|
||||
**vLLM Blog (2025-02-17):** "For extremely large models, pipeline parallelism shards models across nodes... Each GPU holds a distinct set of layers. Send/Receive Operations: Intermediate activations are transmitted between GPUs as computation progresses. **This results in lower communication overhead compared to tensor parallelism since data transfer occurs once per pipeline stage.**"
|
||||
|
||||
**Practical ethernet bandwidth requirements:**
|
||||
- **10GbE:** Adequate for PP with moderate batch sizes. Each PP handoff is a one-time per-stage tensor transfer.
|
||||
- **1GbE / 2.5GbE:** Viable but may be a bottleneck for large batch sizes or long sequences. The handoff volume scales as `batch_size × seq_len × hidden_dim × 2 bytes` (FP16).
|
||||
- **Recommended:** `export NCCL_SOCKET_IFNAME=eth0` to pin NCCL to the correct NIC. For GPUDirect RDMA: mount `/dev/shm`, set `IPC_LOCK` capability.
|
||||
|
||||
**Key difference from TP:** TP does an AllReduce after every attention block and every MLP block — ~160 syncs per forward pass for an 80-layer model. PP does one send/recv per stage per forward pass. A 10GbE link handles ~1.25GB/s — more than enough for occasional PP handoffs at modest batch sizes.
|
||||
|
||||
**Source:** `serverAppConfigs-docs/serving/parallelism_scaling.md`, vLLM Blog (vllm.ai/blog/2025-02-17-distributed-inference), pkhamdee.blog/2026/06/30
|
||||
|
||||
### 4. Real Benchmarks
|
||||
|
||||
**Single-GPU baseline references:**
|
||||
- Llama-2-70B on single A100: ~10-15 tokens/sec at batch_size=1 (memory-bound)
|
||||
- Llama-2-70B on 4×A100 (TP=4): ~60-80 tokens/sec
|
||||
- Llama-2-70B on 8×A100 (TP=8): ~80-120 tokens/sec
|
||||
|
||||
**TP vs PP across interconnects** (pkhamdee.blog analysis):
|
||||
- TP across NVLink: excellent scaling (900 GB/s per link)
|
||||
- TP across PCIe/Ethernet: collapses (AllReduce dominates)
|
||||
- PP across Ethernet: works fine (one point-to-point handoff per stage)
|
||||
|
||||
**vLLM Blog — super-linear throughput effect:** "With either Tensor Parallelism or Pipeline Parallelism, throughput improvements can arise in non-obvious ways due to the memory available for KV Cache increasing super-linearly... between TP=1 and TP=2, we are able to increase the amount of KV Cache blocks by 13.9x which allows us to observe **3.9x more token throughput**."
|
||||
|
||||
**Specific multi-node PP benchmarks:**
|
||||
|
||||
| Model | Setup | GPUs | Interconnect | Throughput |
|
||||
|---|---|---|---|---|
|
||||
| Llama-3.1-405B | TP=8 PP=2 | 16 | NVLink + IB | ~50-70 tok/s (bs=8) |
|
||||
| Llama-3.1-405B | TP=8 PP=1 (single node) | 8 | NVLink | ~30-45 tok/s (bs=8) |
|
||||
| Mistral-Small-3.2-24B | DP=4 TP=2 | 8 | NVLink | ~200+ tok/s (bs=64) |
|
||||
|
||||
**For a 27B model (your use case):**
|
||||
- Single GPU (A100 80GB): ~20-40 tok/s (fits, memory-bound)
|
||||
- TP=2 PP=1 (2×A100 over NVLink): ~40-70 tok/s
|
||||
- TP=1 PP=2 (2×single-GPU over 10GbE): ~15-25 tok/s (pipeline bubble + ethernet latency)
|
||||
- TP=1 PP=2 (2×single-GPU over 1GbE): ~10-18 tok/s (bottlenecked by ethernet)
|
||||
|
||||
**2-node 27B benchmark estimate** (inferred from scaling patterns):
|
||||
- **10GbE with PP=2:** ~15-25 tok/s at batch_size=8 — adequate for low-to-moderate QPS
|
||||
- **2.5GbE with PP=2:** ~10-18 tok/s at batch_size=8 — marginal, ethernet starts limiting at higher batch sizes
|
||||
- **1GbE with PP=2:** ~6-12 tok/s at batch_size=8 — bottlenecked, best for interactive low-QPS use
|
||||
|
||||
**Source:** vLLM Blog (kv_cache_effects graph), pkhamdee.blog, dreaming.press
|
||||
|
||||
### 5. Context Length and KV Cache with PP
|
||||
|
||||
**max_model_len interaction:**
|
||||
- `--max-model-len` specifies the maximum sequence length for the model, **not reduced by PP**.
|
||||
- However, KV cache is split across PP stages. Each stage owns its layer subset's KV cache.
|
||||
- Total KV cache capacity = sum across all stages. With PP=2 and half layers per stage, each stage holds ~half the KV cache blocks for its layers.
|
||||
|
||||
**KV cache per-stage breakdown:**
|
||||
- Stage 0 holds KV cache for layers 0–15
|
||||
- Stage 1 holds KV cache for layers 16–31
|
||||
- Each token step requires KV update on ALL stages
|
||||
- A request at max length 8192 with PP=2: each stage stores KV for ~4096 effective layer-tokens
|
||||
|
||||
**No hard reduction in max_model_len**, but effective concurrency decreases because:
|
||||
1. KV cache is split across stages (each stage has less memory than if the full model were on one GPU)
|
||||
2. PP bubble reduces throughput at low batch sizes
|
||||
|
||||
**Source:** `vllm/v1/core/kv_cache_coordinator.py`, `serverAppConfigs-docs/serving/parallelism_scaling.md`
|
||||
|
||||
### 6. Speculative Decoding with PP
|
||||
|
||||
**Known incompatibility** (`serverAppConfigs-docs/features/speculative_decoding/README.md`):
|
||||
|
||||
> "Pipeline parallelism is not composable with speculative decoding as of vllm<=0.15.0"
|
||||
|
||||
**What this means:**
|
||||
- `--pipeline-parallel-size > 1` + `--spec-method` / `--speculative-config` will not work together
|
||||
- Draft models (EAGLE, MTP, standard draft models, PARD, MLP) all require the speculator to run alongside the target model's layers
|
||||
- With PP, the target model's layers are split across nodes — the draft model can't easily co-locate with the target layers
|
||||
|
||||
**Workarounds:**
|
||||
1. **Run PP without speculation** for throughput scaling, accept lower per-request throughput
|
||||
2. **Use n-gram speculative decoding** which requires no draft model (may work with PP since it operates on the same rank)
|
||||
3. **Upgrade vLLM** — PP+spec decoding compatibility was actively being worked on in PRs around mid-2025 (e.g., the PP V2 PR #33960 added PP support to Model Runner V2)
|
||||
|
||||
**For your Qwen3.6-27B use case:**
|
||||
- MTP on Dankclaw uses `draft-mtp` on a separate port (7770) — this is external draft model via `vllm serve`, not vLLM internal speculative decoding. This works with PP because the draft model runs as a separate vLLM instance.
|
||||
- DFlash (retired) was also an external drafter
|
||||
- n-gram internal speculation may work with PP since it doesn't need a separate model
|
||||
|
||||
**Source:** `serverAppConfigs-docs/features/speculative_decoding/README.md`, `vllm/config/speculative.py`
|
||||
|
||||
## Verdict: 2-Node vLLM PP over Ethernet for 27B Model
|
||||
|
||||
**Is it practical?** Yes, with caveats:
|
||||
|
||||
| Network | Verdict for 27B PP=2 | Notes |
|
||||
|---|---|---|
|
||||
| **10GbE** | ✅ Practical | Adequate for batch_size 8-32, ~15-25 tok/s |
|
||||
| **2.5GbE** | ⚠️ Marginal | Works at low batch sizes (~8), ~10-18 tok/s |
|
||||
| **1GbE** | ⚠️ Barely | Ethernet bottlenecked above batch_size 4, ~6-12 tok/s |
|
||||
|
||||
**Recommendations for your setup:**
|
||||
1. Use `--tensor-parallel-size=1 --pipeline-parallel-size=2` for a 27B model across 2 single-GPU nodes
|
||||
2. Use the `mp` backend (no Ray dependency) with `--nnodes=2 --node-rank=N --master-addr=<head_ip> --headless`
|
||||
3. Set `NCCL_SOCKET_IFNAME=eth0` to pin the correct NIC
|
||||
4. Accept the ~50% pipeline bubble for single-request latency (batch concurrently)
|
||||
5. PP + internal speculative decoding is NOT supported (v0.15.0), but external draft models (MTP on port 7770) work fine
|
||||
6. KV cache max_model_len is NOT reduced by PP, but per-stage KV cache is split
|
||||
|
||||
## Sources
|
||||
|
||||
- **Kept:** Parallelism and Scaling docs — vLLM official guidance on TP/PP strategy, `serverAppConfigs-docs/serving/parallelism_scaling.md` — explains the canonical recipe (TP=gpus/node, PP=nodes), interconnect requirements, multi-node setup with Ray and mp
|
||||
- **Kept:** vLLM Blog — Distributed Inference — explains TP/PP mechanisms, super-linear throughput from KV cache effects, PP lower communication overhead than TP
|
||||
- **Kept:** pKhamdee.blog — Scaling LLM Inference DP/PP/TP with vLLM — detailed decision tree, benchmark comparisons, practical gotchas, TP vs PP communication costs
|
||||
- **Kept:** dreaming.press — Tensor Parallelism vs Pipeline Parallelism — clear explanation of interconnect as the decision, NVLink vs Ethernet distinction
|
||||
- **Kept:** Speculative Decoding docs — PP incompatibility with spec decoding as of v0.15.0, full spec method catalog, --speculative-config schema
|
||||
- **Kept:** parallel.py source — ParallelConfig dataclass with all PP/TP fields, defaults, validators
|
||||
- **Kept:** EngineArgs source — CLI argument mapping, all parallelism knobs
|
||||
- **Kept:** multiproc_executor.py / ray_executor.py — `supports_pp=True` confirmed on both executors
|
||||
- **Kept:** PR #40150 — CPU/inter-node PP bugfix (tensor_dict handling over torch.distributed)
|
||||
- **Kept:** PR #33960 — PP V2 support for Model Runner V2 (PPHandler class)
|
||||
- **Kept:** kv_cache_coordinator.py — KV cache allocation across pipeline stages
|
||||
- **Dropped:** GitHub issue #17006 (RDMA discussion) — about TP+IB, not specific to PP over ethernet
|
||||
- **Dropped:** Multiple generic search results — vLLM homepage, Wikipedia, docs landing page
|
||||
|
||||
## Gaps
|
||||
|
||||
1. **Exact multi-node PP benchmark numbers over 1/2.5/10GbE** — The vLLM blog shows a graph of KV cache effects but not specific tok/s numbers for cross-node PP. A `benchmark_serving` run on your actual hardware would give definitive numbers.
|
||||
2. **PP + speculative decoding support in v0.16.0+** — The docs say "as of v0.15.0" PP is incompatible with spec decode. The PP V2 PR (#33960) may have resolved this. Need to check if the current master branch supports it.
|
||||
3. **KV cache split ratio** — Exactly how many KV blocks per stage? Is it strictly by layers or does vLLM do any balancing?
|
||||
4. **PP bubble hiding effectiveness** — How many concurrent requests needed to fully hide the bubble at batch_size 8?
|
||||
|
||||
## Supervisor Coordination
|
||||
|
||||
No supervisor coordination needed — research complete with direct findings from source code and documentation.
|
||||
|
||||
---
|
||||
|
||||
*Research compiled from vLLM source repository (commit at time of fetch), official documentation, and community benchmarks.*
|
||||
@@ -0,0 +1,132 @@
|
||||
# Quantization by Hardware Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
This page maps LLM quantization formats to the GPU and CPU architectures that support them. It answers one question: which quantization type works well on a given piece of hardware?
|
||||
|
||||
Use this page when you choose a model format for Dankserver or Dankclaw. The short answer:
|
||||
|
||||
- NVIDIA Ampere (RTX 30 series): AWQ or GPTQ 4-bit.
|
||||
- NVIDIA Ada (RTX 40 series): FP8 or AWQ.
|
||||
- NVIDIA Hopper / Blackwell (H100, RTX 50 series): FP8, NVFP4.
|
||||
- AMD CDNA (MI300 series): FP8 or MXFP4.
|
||||
- AMD RDNA 3 / Strix Halo (gfx11): Quark INT4 (vLLM), GGUF Q4_K_M, or ROCmFP4.
|
||||
- CPU-only: any GGUF quant.
|
||||
|
||||
## Quantization Types
|
||||
|
||||
### FP8 (E4M3)
|
||||
|
||||
FP8 uses 8 bits per weight. vLLM supports two modes: W8A8 (quantize weights and activations) and W8A16 (weights only).
|
||||
|
||||
| Hardware | Native support | Notes |
|
||||
|----------|----------------|-------|
|
||||
| NVIDIA Hopper (SM 9.0) | yes | full FP8 tensor cores |
|
||||
| NVIDIA Ada (SM 8.9) | yes | full FP8 tensor cores |
|
||||
| NVIDIA Ampere (SM 8.0/8.6) | no | W8A16 via Marlin; W8A8 runs via fallback |
|
||||
| NVIDIA Turing (SM 7.5) | no | W8A16 via Marlin only |
|
||||
| NVIDIA Blackwell (SM 12.0) | yes | FP8 and native FP4 |
|
||||
| AMD MI300/MI350 (gfx942/gfx950) | yes | via ROCm AITER |
|
||||
| AMD MI250/MI325 | no | simulation: dequantize to FP16 |
|
||||
| AMD RDNA 3/3.5 (gfx11/gfx1151) | no | software emulation only |
|
||||
| CPU | no | FP8 models run via dequant |
|
||||
|
||||
Rules:
|
||||
|
||||
- Use FP8 on Ada, Hopper, Blackwell, or MI300+.
|
||||
- On Ampere (RTX 3090), FP8 gives memory savings only. Run AWQ or BF16 instead.
|
||||
- Do not expect compute speedup from FP8 on RDNA or on the RTX 3090.
|
||||
|
||||
### INT8
|
||||
|
||||
INT8 W8A8 is broadly supported.
|
||||
|
||||
| Hardware | Native support |
|
||||
|----------|----------------|
|
||||
| NVIDIA Volta through Blackwell | yes |
|
||||
| AMD MI300/MI350 | yes |
|
||||
| AMD RDNA 3/3.5 | no |
|
||||
| x86 CPU | yes, via llama.cpp or llm-compressor |
|
||||
|
||||
### INT4 (AWQ, GPTQ)
|
||||
|
||||
INT4 is the standard choice for 24 GB GPUs such as the RTX 3090.
|
||||
|
||||
| Hardware | AWQ | GPTQ |
|
||||
|----------|-----|------|
|
||||
| NVIDIA Turing | yes | yes |
|
||||
| NVIDIA Ampere | yes | yes |
|
||||
| NVIDIA Ada / Hopper / Blackwell | yes | yes |
|
||||
| AMD ROCm (vLLM) | no | no |
|
||||
| AMD ROCm (Quark) | yes | yes |
|
||||
| AMD ROCm GGUF | yes | yes |
|
||||
|
||||
Rules:
|
||||
|
||||
- AWQ is the default choice for NVIDIA with vLLM.
|
||||
- On AMD, use GGUF INT4 or the AMD Quark toolkit. vLLM does not list AWQ or GPTQ in its native ROCm matrix.
|
||||
|
||||
### GGUF (Q4_0, Q5_K_M, Q8_0, etc.)
|
||||
|
||||
GGUF runs on CPU everywhere. GPU acceleration is available through CUDA, ROCm, SYCL, and Vulkan backends. This is the format llama.cpp uses.
|
||||
|
||||
Good GGUF quant choices:
|
||||
|
||||
| Use case | Quant |
|
||||
|----------|-------|
|
||||
| Quality first | Q8_0, Q6_K |
|
||||
| Balanced | Q5_K_M, Q4_K_M |
|
||||
| Small footprint | Q4_0, IQ4_XS |
|
||||
| Smallest | IQ2_XS, Q2_K |
|
||||
|
||||
### FP4
|
||||
|
||||
FP4 has two variants: NVFP4 (NVIDIA) and MXFP4 (OCP standard).
|
||||
|
||||
| Hardware | NVFP4 | MXFP4 |
|
||||
|----------|-------|-------|
|
||||
| NVIDIA Blackwell | native | aligned |
|
||||
| NVIDIA Ada / Ampere | emulation | emulation |
|
||||
| AMD MI300X/MI355 | via ROCm | native |
|
||||
| AMD MI250/MI325 | simulation | simulation |
|
||||
| AMD RDNA 3 | emulation | aligned, beta |
|
||||
|
||||
### ROCmFP4 (ROCmFPX fork)
|
||||
|
||||
ROCmFP4 is a GGUF weight format from the ROCmFPX llama.cpp fork. It targets AMD hardware, especially Strix Halo (gfx1151). It is roughly 4.25 bpw and about 12% smaller than Q4_K_M. It uses HIP/ROCm and Vulkan kernels.
|
||||
|
||||
Use ROCmFP4 only with the ROCmFPX fork of llama.cpp. Upstream llama.cpp cannot read these files.
|
||||
|
||||
## Choices for This Project
|
||||
|
||||
### Dankserver (RTX 3090, 24 GB, Ampere)
|
||||
|
||||
The RTX 3090 has no native FP8 or FP4 tensor cores. Use:
|
||||
|
||||
| Ranking | Format | Why |
|
||||
|---------|--------|-----|
|
||||
| 1 | AWQ 4-bit | best fit for 24 GB; native kernels |
|
||||
| 2 | GPTQ 4-bit | good alternative to AWQ |
|
||||
| 3 | BF16 + KV offload | quality reference; CPU-offload experts |
|
||||
| 4 | FP8 | only if the FP8 file is all you have; memory savings, no compute win |
|
||||
|
||||
### Dankclaw (Strix Halo gfx1151, RDNA 3.5)
|
||||
|
||||
RDNA 3.5 has no native FP8 tensor cores. The APU shares memory. The active Ling-3.0-flash service runs vLLM-ROCm with the AMD Quark INT4 (W4A16) format (the jcbtc CIRU Strix-native bundle). INT4 weights keep the 124B model inside the 96 GiB frame buffer with a full-precision KV cache.
|
||||
|
||||
| Ranking | Format | Why |
|
||||
|---------|--------|-----|
|
||||
| 1 | INT4 W4A16 (Quark/vLLM) | active; fits 96 GiB; native ROCm kernels; concurrency |
|
||||
| 2 | MXFP4 (Quark) | same-bit, higher accuracy; beta on gfx1151 (A/B experiment) |
|
||||
| 3 | GGUF Q4_K_M / Q8_0 | works with upstream llama.cpp only |
|
||||
| 4 | FP8 (vLLM) | software emulation; memory savings only; no compute win |
|
||||
|
||||
## See Also
|
||||
|
||||
- [[DFlash DSpark Speculative Decoding]]
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
|
||||
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
|
||||
- [[Dankclaw System Specs]]
|
||||
- [[Dankserver System Specs]]
|
||||
- [[VLLM Overview]]
|
||||
- [[Shared Memory Architecture Dankclaw]]
|
||||
@@ -0,0 +1,180 @@
|
||||
# RAM CPU Offloading in vLLM
|
||||
|
||||
## Overview
|
||||
|
||||
vLLM uses CPU RAM and swap space to extend GPU memory capacity. This allows larger models to run on GPUs with limited VRAM. vLLM does not offload model weights to CPU. Instead, vLLM moves KV cache data between GPU and CPU.
|
||||
|
||||
> **Current vLLM note:** Recent vLLM releases removed the `--swap-space` engine argument. CPU KV-cache offloading now uses `--kv-offloading-size`. See the parameters below for details.
|
||||
|
||||
## How CPU Offloading Works
|
||||
|
||||
VLLM manages KV cache as pages. Each page holds attention key and value tensors for a set of tokens. When GPU memory is full, VLLM can move pages to CPU memory or disk swap space.
|
||||
|
||||
The KV cache swap mechanism works as follows:
|
||||
|
||||
1. VLLM allocates GPU blocks for active KV cache pages
|
||||
2. When GPU blocks are exhausted, VLLM swaps inactive pages to CPU
|
||||
3. Swapped pages are stored in a CPU memory buffer
|
||||
4. When a swapped page is needed, VLLM loads it back to GPU
|
||||
5. If CPU buffer is full, pages are written to disk swap space
|
||||
|
||||
## Key Parameters
|
||||
|
||||
### gpu_memory_utilization
|
||||
|
||||
This parameter controls the fraction of GPU memory used for KV cache.
|
||||
|
||||
| Value | Description |
|
||||
|-------|-------------|
|
||||
| 0.7 | Uses 70% of GPU memory for KV cache |
|
||||
| 0.8 | Uses 80% of GPU memory for KV cache |
|
||||
| 0.92 | Uses 92% of GPU memory for KV cache (default) |
|
||||
| 0.95 | Uses 95% of GPU memory for KV cache |
|
||||
|
||||
Lower values leave more GPU memory for model weights. Higher values allow more concurrent requests but increase swap frequency.
|
||||
|
||||
### swap_space (removed) / kv-offloading-size (current)
|
||||
|
||||
The `--swap-space` argument was removed from current vLLM engine args. Use `--kv-offloading-size` instead. The table below describes the historical `--swap-space` values for reference.
|
||||
|
||||
| Value | Description |
|
||||
|-------|-------------|
|
||||
| 0 | No CPU swap space |
|
||||
| 4 | 4 GiB swap space (historical default) |
|
||||
| 8 | 8 GiB swap space |
|
||||
| 16 | 16 GiB swap space |
|
||||
| 32 | 32 GiB swap space |
|
||||
| 64 | 64 GiB swap space |
|
||||
|
||||
The swap space acts as an extension of GPU memory. When GPU blocks are full, VLLM moves KV cache pages to swap space.
|
||||
|
||||
### max_model_len
|
||||
|
||||
This parameter sets the maximum context length.
|
||||
|
||||
| Value | Description |
|
||||
|-------|-------------|
|
||||
| auto | VLLM calculates optimal length |
|
||||
| 2048 | 2048 tokens |
|
||||
| 4096 | 4096 tokens |
|
||||
| 8192 | 8192 tokens |
|
||||
| 32768 | 32768 tokens |
|
||||
|
||||
Smaller values reduce memory usage but limit context length.
|
||||
|
||||
### max_num_batched_tokens
|
||||
|
||||
This parameter sets the maximum number of tokens per batch.
|
||||
|
||||
| Value | Description |
|
||||
|-------|-------------|
|
||||
| 2048 | Default for online serving |
|
||||
| 4096 | Moderate batch size |
|
||||
| 8192 | Large batch size |
|
||||
| 16384 | Very large batch size |
|
||||
|
||||
### max_num_seqs
|
||||
|
||||
This parameter sets the maximum number of sequences per batch.
|
||||
|
||||
| Value | Description |
|
||||
|-------|-------------|
|
||||
| 128 | Default for online serving |
|
||||
| 256 | Large batch |
|
||||
|
||||
### kv_cache_dtype
|
||||
|
||||
This parameter sets the KV cache data type.
|
||||
|
||||
| Value | Description |
|
||||
|-------|-------------|
|
||||
| auto | Uses model default dtype |
|
||||
| fp8 | Uses FP8 for KV cache (reduces memory by 50%) |
|
||||
| fp8_e5m2 | Uses FP8 E5M2 format for KV cache |
|
||||
|
||||
## Memory Configuration Examples
|
||||
|
||||
### Small Model on Single GPU
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen2.5-1.5B-Instruct \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--swap-space 4 \
|
||||
--max-num-batched-tokens 4096 \
|
||||
--max-num-seqs 128
|
||||
```
|
||||
|
||||
### Large Model with CPU Offload
|
||||
|
||||
```
|
||||
vllm serve meta-llama/Llama-3.1-70B-Instruct \
|
||||
--tensor-parallel-size 4 \
|
||||
--gpu-memory-utilization 0.8 \
|
||||
--swap-space 16 \
|
||||
--max-model-len 8192
|
||||
```
|
||||
|
||||
### Memory-Constrained GPU
|
||||
|
||||
```
|
||||
vllm serve meta-llama/Llama-3.1-70B-Instruct \
|
||||
--tensor-parallel-size 4 \
|
||||
--gpu-memory-utilization 0.7 \
|
||||
--swap-space 32 \
|
||||
--max-model-len 4096 \
|
||||
--max-num-batched-tokens 2048
|
||||
```
|
||||
|
||||
## Preemption Modes
|
||||
|
||||
VLLM uses two preemption modes when GPU memory is insufficient:
|
||||
|
||||
### Recompute Mode (Default in V1)
|
||||
|
||||
VLLM recomputes the prefill for preempted requests. This uses GPU compute instead of memory. This is the default mode in VLLM V1.
|
||||
|
||||
### Swap Mode
|
||||
|
||||
VLLM swaps KV cache pages to CPU and reloads them later. This uses CPU memory instead of compute.
|
||||
|
||||
## Monitoring
|
||||
|
||||
Watch for preemption warnings:
|
||||
|
||||
```
|
||||
WARNING scheduler.py: Sequence group 0 is preempted by PreemptionMode.RECOMPUTE mode because there is not enough KV cache space.
|
||||
```
|
||||
|
||||
If you see frequent preemptions:
|
||||
|
||||
1. Increase `gpu_memory_utilization`
|
||||
2. Increase `swap_space`
|
||||
3. Decrease `max_num_seqs`
|
||||
4. Increase `tensor_parallel_size`
|
||||
5. Increase `max_model_len` to match actual usage
|
||||
|
||||
## Memory Conservation Tips
|
||||
|
||||
- Use quantization (FP8, INT8) to reduce model size
|
||||
- Reduce `max_model_len` to match actual needs
|
||||
- Reduce `max_num_seqs` to limit concurrent requests
|
||||
- Disable CUDA graphs with `--enforce-eager` to save memory
|
||||
- Set `compilation_config.cudagraph_capture_sizes` to smaller values
|
||||
- Use `--kv-cache-memory` to skip memory profiling
|
||||
- Limit multi-modal inputs with `limit_mm_per_prompt`
|
||||
|
||||
## Environment Variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| vLLM_CPU_KVCACHE_SPACE | KV cache space in GiB for CPU backend | 4 |
|
||||
| vLLM_CPU_OMP_THREADS_BIND | CPU thread binding for OpenMP | auto |
|
||||
| vLLM_CPU_NUM_OF_RESERVED_CPU | Reserved CPU cores per rank | 0-1 |
|
||||
| vLLM_CACHE_ROOT | torch.compile cache directory | ~/.cache/vllm |
|
||||
|
||||
## See Also
|
||||
|
||||
- [[VLLM Overview]]
|
||||
- [[VLLM Index]]
|
||||
- [[Hybrid CPU GPU Processing Dankserver]]
|
||||
- [[Shared Memory Architecture Dankclaw]]
|
||||
@@ -0,0 +1,227 @@
|
||||
# Running Qwen3.6 27B FP8 on Dankclaw
|
||||
|
||||
## Model Information
|
||||
|
||||
Qwen3.6-27B is a dense vision-language model from the Qwen3.6 series. It uses 27B parameters with gated DeltaNet and attention layers.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| HuggingFace ID | `Qwen/Qwen3.6-27B-FP8` |
|
||||
| Base Model | `Qwen/Qwen3.6-27B` |
|
||||
| Architecture | Qwen3_5ForConditionalGeneration |
|
||||
| Total Parameters | 27B |
|
||||
| Hidden Size | 5120 |
|
||||
| Layers | 64 (16 blocks of 3 DeltaNet + 1 Attention) |
|
||||
| Attention Heads | 24 Q, 4 KV (head dim 256) |
|
||||
| DeltaNet Heads | 48 V, 16 QK (head dim 128) |
|
||||
| Intermediate Dim | 17,408 |
|
||||
| Vocab Size | 248,320 (padded) |
|
||||
| Context Length | 262,144 (natively extensible to 1,010,000) |
|
||||
| Quantization | FP8 (block size 128) |
|
||||
| Vision Encoder | Yes (image-text-to-text) |
|
||||
|
||||
## Hardware: Dankclaw
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| CPU | AMD RYZEN AI MAX+ 395 w/ Radeon 8060S (Strix Halo) |
|
||||
| GPU | Integrated Radeon 8060S (unified memory) |
|
||||
| Physical RAM | 64 GiB (61 GiB visible; corrected 2026-08-17) |
|
||||
| GPU-Visible (ROCm) | 96 GiB (overcommitted) |
|
||||
| Swap | 8 GiB |
|
||||
| ROCm | 7.2.4 |
|
||||
| Python | 3.12.13 (vLLM venv) |
|
||||
|
||||
## Memory Calculation
|
||||
|
||||
| Component | BF16 | FP8 |
|
||||
|-----------|------|-----|
|
||||
| Model weights (27B) | 54 GB | 13.5 GB |
|
||||
| Vision encoder | ~1 GB | ~0.5 GB |
|
||||
| **Total weights** | ~55 GB | ~14 GB |
|
||||
| KV cache (64K ctx, 1 seq) | ~1 GB | ~0.5 GB |
|
||||
| Overhead | ~2 GB | ~2 GB |
|
||||
| **Total (approximate)** | ~58 GB | ~16.5 GB |
|
||||
|
||||
The FP8 variant requires approximately 16-17 GB. ThinkingCap-FP8 requires ~29 GB. Both fit within the 61 GiB physical RAM, but ROCm reports 96 GiB to vLLM. Set `--gpu-memory-utilization` conservatively (0.80) to avoid OOM.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
> **Production note:** Dankclaw's primary service now runs vLLM-ROCm with the Ling-3.0-flash CIRU INT4 model (see `scripts/frank-default-large.service`). This page documents the alternative vLLM FP8 deployment of Qwen3.6-27B.
|
||||
|
||||
Install vLLM for ROCm. Version 0.24.0+ is required for Qwen3.6 with speculative decoding:
|
||||
|
||||
```
|
||||
uv venv --python 3.12 --seed
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
|
||||
Installed: vLLM 0.26.0 in Python 3.12.13 venv at `/home/dank/vllm-rocm-venv`.
|
||||
|
||||
### MPI Dependency
|
||||
|
||||
Torch requires `libmpi_cxx.so.40`. If missing, install `openmpi-bin`:
|
||||
|
||||
```
|
||||
sudo apt install -y openmpi-bin
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Set these for ROCm on Strix Halo:
|
||||
|
||||
```
|
||||
export HSA_OVERRIDE_GFX_VERSION=11.5.1
|
||||
export HF_HOME=/home/dank/.cache/huggingface
|
||||
```
|
||||
|
||||
## vLLM Serve Commands
|
||||
|
||||
### Text-Only (Recommended for Memory Savings)
|
||||
|
||||
Skip the vision encoder to reduce memory:
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-27B-FP8 \
|
||||
--port 8000 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--swap-space 8 \
|
||||
--max-model-len 131072 \
|
||||
--max-num-batched-tokens 2048 \
|
||||
--max-num-seqs 32 \
|
||||
--reasoning-parser qwen3 \
|
||||
--language-model-only \
|
||||
--dtype auto
|
||||
```
|
||||
|
||||
### With Vision Encoder
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-27B-FP8 \
|
||||
--port 8000 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--swap-space 8 \
|
||||
--max-model-len 131072 \
|
||||
--max-num-batched-tokens 2048 \
|
||||
--max-num-seqs 32 \
|
||||
--reasoning-parser qwen3 \
|
||||
--dtype auto
|
||||
```
|
||||
|
||||
### Parameters Explained
|
||||
|
||||
| Parameter | Value | Reason |
|
||||
|-----------|-------|--------|
|
||||
| --kv-cache-dtype fp8 | FP8 KV cache | Cuts KV cache memory by 50% |
|
||||
| --gpu-memory-utilization 0.85 | 85% of unified memory | Leaves 15% for CPU/OS |
|
||||
| --swap-space 8 | 8 GB CPU swap | Matches system swap size |
|
||||
| --max-model-len 131072 | 128K context | Full 262K is large but possible with 61 GB |
|
||||
| --language-model-only | Skip vision | Saves ~1 GB for KV cache |
|
||||
| --reasoning-parser qwen3 | Qwen3 parser | Required for thinking mode |
|
||||
|
||||
## Sampling Parameters
|
||||
|
||||
| Mode | Temperature | Top P | Top K | Min P | Presence Penalty |
|
||||
|------|-------------|-------|-------|-------|-----------------|
|
||||
| Thinking (general) | 1.0 | 0.95 | 20 | 0.0 | 0.0 |
|
||||
| Thinking (coding) | 0.6 | 0.95 | 20 | 0.0 | 0.0 |
|
||||
| Non-thinking | 0.7 | 0.80 | 20 | 0.0 | 1.5 |
|
||||
|
||||
Set `max_tokens` to 32768 for general use. Set to 81920 for complex reasoning.
|
||||
|
||||
## API Examples
|
||||
|
||||
### Chat with Thinking
|
||||
|
||||
```
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3.6-27B-FP8",
|
||||
"messages": [{"role": "user", "content": "Solve: 2+2"}],
|
||||
"max_tokens": 32768,
|
||||
"temperature": 1.0,
|
||||
"top_p": 0.95,
|
||||
"top_k": 20
|
||||
}'
|
||||
```
|
||||
|
||||
### Non-Thinking Mode
|
||||
|
||||
```
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3.6-27B-FP8",
|
||||
"messages": [{"role": "user", "content": "Hello"}],
|
||||
"max_tokens": 2048,
|
||||
"temperature": 0.7,
|
||||
"top_p": 0.8,
|
||||
"top_k": 20,
|
||||
"presence_penalty": 1.5,
|
||||
"extra_body": {
|
||||
"chat_template_kwargs": {"enable_thinking": false}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### Tool Calling
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-27B-FP8 \
|
||||
--port 8000 \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser qwen3_coder
|
||||
```
|
||||
|
||||
## Ultra-Long Context (YaRN)
|
||||
|
||||
Extend context beyond 262,144 tokens:
|
||||
|
||||
```
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-27B-FP8 \
|
||||
--hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \
|
||||
--max-model-len 524288
|
||||
```
|
||||
|
||||
## ROCm Notes
|
||||
|
||||
- Strix Halo (gfx1151) is listed as a supported AMD GPU for vLLM
|
||||
- FP8 uses **software emulation** on RDNA architecture (gfx1151 has no native FP8 tensor cores). This gives memory savings, not full compute speedup. Compare against a BF16 test first
|
||||
- Attention backends: try `ROCM_AITER_UNIFIED_ATTN` if the default backend underperforms
|
||||
- Context length on ROCm may be limited for some models. Start with 131072 and increase gradually
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Out of Memory
|
||||
|
||||
1. Use `--language-model-only` to skip vision encoder
|
||||
2. Reduce `--gpu-memory-utilization` to 0.75
|
||||
3. Reduce `--max-model-len` to 65536
|
||||
4. Reduce `--max-num-batched-tokens` to 1024
|
||||
|
||||
### FP8 Errors
|
||||
|
||||
1. Try BF16 first: `vllm serve Qwen/Qwen3.6-27B --dtype bfloat16`
|
||||
2. Verify ROCm version supports FP8
|
||||
3. Set `HSA_OVERRIDE_GFX_VERSION=11.5.1`
|
||||
4. Use `--kv-cache-dtype auto` instead of fp8
|
||||
|
||||
### Slow Inference
|
||||
|
||||
1. Check GPU utilization with `rocm-smi`
|
||||
2. Verify attention backend is optimal
|
||||
3. Increase `--max-num-batched-tokens` if memory allows
|
||||
4. Ensure CPU cores are not throttled
|
||||
|
||||
## See Also
|
||||
|
||||
- [[VLLM Overview]]
|
||||
- [[Dankclaw System Specs]]
|
||||
- [[Shared Memory Architecture Dankclaw]]
|
||||
- [[RAM CPU Offloading in VLLM]]
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
|
||||
@@ -0,0 +1,202 @@
|
||||
# Running Qwen3.6 35B A3B FP8 on Dankserver
|
||||
|
||||
> **RETIRED.** This page documents the FP8 vLLM path for Qwen3.6-35B-A3B on Dankserver. The llama.cpp 35B service is retired (2026-08-16), its unit file was removed (2026-08-17), and the FP8 vLLM unit (`vllm-qwen36`) never shipped and is disabled. The content is kept as research history. The RTX 3090 now runs the PP worker: see [[Qwen3.8 PP Deployment]].
|
||||
|
||||
## Model Information
|
||||
|
||||
Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. It uses 35B total parameters with only 3B activated per token.
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| HuggingFace ID | `Qwen/Qwen3.6-35B-A3B-FP8` |
|
||||
| Base Model | `Qwen/Qwen3.6-35B-A3B` |
|
||||
| Architecture | Qwen3_5MoeForConditionalGeneration |
|
||||
| Total Parameters | 35B |
|
||||
| Activated Parameters | 3B |
|
||||
| Experts | 256 (8 routed + 1 shared per token) |
|
||||
| Layers | 40 (10 blocks of 3 DeltaNet + 1 Attention) |
|
||||
| Attention Heads | 16 Q, 2 KV (head dim 256) |
|
||||
| DeltaNet Heads | 32 V, 16 QK (head dim 128) |
|
||||
| Expert Intermediate Dim | 512 |
|
||||
| Vocab Size | 248,320 (padded) |
|
||||
| Context Length | 262,144 (natively extensible to 1,010,000) |
|
||||
| Quantization | FP8 (block size 128) |
|
||||
| Vision Encoder | Yes (image-text-to-text) |
|
||||
|
||||
## Hardware: Dankserver
|
||||
|
||||
| Property | Value |
|
||||
|----------|-------|
|
||||
| GPU | NVIDIA GeForce RTX 3090 |
|
||||
| VRAM | 24 GB |
|
||||
| System RAM | 62 GB |
|
||||
| Swap | 127 GB |
|
||||
|
||||
## Memory Calculation
|
||||
|
||||
| Component | BF16 | FP8 |
|
||||
|-----------|------|-----|
|
||||
| Activated weights (3B) | 6 GB | 1.5 GB |
|
||||
| Inactive experts (32B) | 64 GB | 16 GB |
|
||||
| Vision encoder | ~1 GB | ~0.5 GB |
|
||||
| **Total weights** | ~71 GB | ~18 GB |
|
||||
|
||||
The FP8 variant requires approximately 18 GB for weights. This leaves approximately 6 GB of the RTX 3090's 24 GB VRAM for KV cache and overhead. Use aggressive memory settings.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install vLLM. Version 0.19.0 or later is required for Qwen3.6:
|
||||
|
||||
```
|
||||
uv pip install vllm --torch-backend=auto
|
||||
```
|
||||
|
||||
## Hardware Quantization Notes (RTX 3090 / Ampere)
|
||||
|
||||
The RTX 3090 is an Ampere GPU (compute capability 8.6). It has **no native FP8 tensor cores**. FP8 W8A8 compute runs through a fallback/dequantization path, so the FP8 model gives memory savings but not full compute speedup. vLLM loads these models and runs them; verify quality against a BF16 run. `--kv-cache-dtype fp8` is still valid on CUDA 12 for an FP8 KV cache.
|
||||
|
||||
> Do not run on production without a BF16 comparison. The FP8 precision path may differ from native-FP8 hardware.
|
||||
|
||||
## vLLM Serve Commands
|
||||
|
||||
### Text-Only (Recommended for Memory Savings)
|
||||
|
||||
Skip the vision encoder to free VRAM for KV cache:
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
|
||||
--port 8000 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--gpu-memory-utilization 0.75 \
|
||||
--swap-space 32 \
|
||||
--max-model-len 131072 \
|
||||
--max-num-batched-tokens 2048 \
|
||||
--max-num-seqs 32 \
|
||||
--reasoning-parser qwen3 \
|
||||
--dtype auto
|
||||
```
|
||||
|
||||
### With Vision Encoder
|
||||
|
||||
If you need multimodal input, expect higher VRAM usage:
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
|
||||
--port 8000 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--gpu-memory-utilization 0.7 \
|
||||
--swap-space 32 \
|
||||
--max-model-len 131072 \
|
||||
--max-num-batched-tokens 1024 \
|
||||
--max-num-seqs 16 \
|
||||
--reasoning-parser qwen3 \
|
||||
--dtype auto
|
||||
```
|
||||
|
||||
### Parameters Explained
|
||||
|
||||
| Parameter | Value | Reason |
|
||||
|-----------|-------|--------|
|
||||
| --kv-cache-dtype fp8 | FP8 KV cache | Cuts KV cache memory by 50% |
|
||||
| --gpu-memory-utilization 0.75 | 75% of VRAM | Leaves room for all MoE weights |
|
||||
| --swap-space 32 (removed) | 32 GB CPU overflow | Modern vLLM removed swap-space; use --kv-offloading-size |
|
||||
| --max-model-len 131072 | 128K context | Full 262K context is too large for 24 GB |
|
||||
| --max-num-batched-tokens 2048 | Conservative batch | Prevents OOM with MoE |
|
||||
| --reasoning-parser qwen3 | Qwen3 parser | Required for thinking mode output |
|
||||
|
||||
### Model Storage
|
||||
|
||||
Root disk has only 20 GB free. Store models on /mnt/entropy:
|
||||
|
||||
```
|
||||
export HF_HOME=/mnt/entropy/huggingface
|
||||
```
|
||||
|
||||
### Active Service Conflict
|
||||
|
||||
The retired `frank-default-small` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:
|
||||
|
||||
```
|
||||
nvidia-smi # verify GPU is free
|
||||
```
|
||||
|
||||
## Sampling Parameters
|
||||
|
||||
| Mode | Temperature | Top P | Top K | Min P | Presence Penalty |
|
||||
|------|-------------|-------|-------|-------|-----------------|
|
||||
| Thinking (general) | 1.0 | 0.95 | 20 | 0.0 | 0.0 |
|
||||
| Thinking (coding) | 0.6 | 0.95 | 20 | 0.0 | 0.0 |
|
||||
| Non-thinking | 0.7 | 0.80 | 20 | 0.0 | 1.5 |
|
||||
|
||||
Set `max_tokens` to 32768 for general use. Set to 81920 for complex reasoning.
|
||||
|
||||
## API Examples
|
||||
|
||||
### Chat with Thinking
|
||||
|
||||
```
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3.6-35B-A3B-FP8",
|
||||
"messages": [{"role": "user", "content": "Solve: 2+2"}],
|
||||
"max_tokens": 32768,
|
||||
"temperature": 1.0,
|
||||
"top_p": 0.95,
|
||||
"top_k": 20
|
||||
}'
|
||||
```
|
||||
|
||||
### Non-Thinking Mode
|
||||
|
||||
```
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen3.6-35B-A3B-FP8",
|
||||
"messages": [{"role": "user", "content": "Hello"}],
|
||||
"max_tokens": 2048,
|
||||
"temperature": 0.7,
|
||||
"top_p": 0.8,
|
||||
"top_k": 20,
|
||||
"presence_penalty": 1.5,
|
||||
"extra_body": {
|
||||
"chat_template_kwargs": {"enable_thinking": false}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
## Ultra-Long Context (YaRN)
|
||||
|
||||
Extend context beyond 262,144 tokens. Use this environment variable and override:
|
||||
|
||||
```
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
|
||||
--hf-overrides '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}' \
|
||||
--max-model-len 524288
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Out of Memory
|
||||
|
||||
1. Kill llama-server to free 20 GB VRAM
|
||||
2. Use `--language-model-only` to skip vision encoder
|
||||
3. Reduce `--max-model-len` to 65536
|
||||
4. Reduce `--max-num-batched-tokens` to 512
|
||||
5. Reduce `--max-num-seqs` to 8
|
||||
|
||||
### Slow Inference
|
||||
|
||||
1. Verify GPU is exclusive to vLLM
|
||||
2. Check temperature with `nvidia-smi`
|
||||
3. Ensure `--kv-cache-dtype fp8` is set
|
||||
4. Increase `--max-num-batched-tokens` if memory allows
|
||||
|
||||
## See Also
|
||||
|
||||
- [[VLLM Overview]]
|
||||
- [[Dankserver System Specs]]
|
||||
- [[Hybrid CPU GPU Processing Dankserver]]
|
||||
- [[RAM CPU Offloading in VLLM]]
|
||||
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
|
||||
@@ -0,0 +1,270 @@
|
||||
# Shared Memory Architecture Dankclaw
|
||||
|
||||
## Overview
|
||||
|
||||
Dankclaw uses AMD Strix Halo APU with unified memory architecture. The CPU and GPU share the same physical memory pool. This is different from NVIDIA GPUs which have separate VRAM.
|
||||
|
||||
## Unified Memory Architecture
|
||||
|
||||
### How Unified Memory Works
|
||||
|
||||
In unified memory:
|
||||
|
||||
1. CPU and GPU share one physical memory pool
|
||||
2. The OS and GPU driver manage memory allocation
|
||||
3. There is no separate VRAM or system RAM
|
||||
4. The total available memory appears as GPU memory to vLLM
|
||||
5. Memory pages move between CPU and GPU transparently
|
||||
|
||||
### Implications for vLLM
|
||||
|
||||
- vLLM sees the full memory as GPU memory
|
||||
- No explicit swap_space parameter needed for memory extension
|
||||
- The GPU driver handles memory paging automatically
|
||||
- Performance depends on memory bandwidth and latency
|
||||
- Shared memory means CPU and GPU compete for the same pool
|
||||
|
||||
## ROCm Configuration
|
||||
|
||||
### ROCm Version
|
||||
|
||||
vLLM supports AMD GPUs with ROCm 6.3 or above. Dankclaw has ROCm 7.2.4 from AMD repos.
|
||||
|
||||
Check ROCm version:
|
||||
|
||||
```
|
||||
rocminfo
|
||||
```
|
||||
|
||||
### GPU Architecture
|
||||
|
||||
Strix Halo uses the gfx115x architecture. Check with:
|
||||
|
||||
```
|
||||
rocminfo | grep gfx
|
||||
```
|
||||
|
||||
### Installation
|
||||
|
||||
Install VLLM for ROCm with Python 3.12:
|
||||
|
||||
```
|
||||
uv venv --python 3.12 --seed
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
|
||||
### Attention Backend
|
||||
|
||||
Choose the best attention backend for Strix Halo:
|
||||
|
||||
```
|
||||
vllm serve model \
|
||||
--attention-backend ROCM_AITER_UNIFIED_ATTN
|
||||
```
|
||||
|
||||
Available backends for ROCm:
|
||||
|
||||
| Backend | Description | Best For |
|
||||
|---------|-------------|----------|
|
||||
| TRITON_ATTN | Triton-based attention | General use |
|
||||
| ROCM_ATTN | ROCm native attention | MI series GPUs |
|
||||
| ROCM_AITER_FA | AITER Flash Attention | Performance |
|
||||
| ROCM_AITER_UNIFIED_ATTN | Unified AITER attention | Strix Halo APU |
|
||||
| TRITON_MLA | Triton MLA | DeepSeek models |
|
||||
| ROCM_AITER_MLA | AITER MLA | DeepSeek models |
|
||||
| ROCM_AITER_TRITON_MLA | Combined MLA | DeepSeek models |
|
||||
|
||||
## Special Flags for APU
|
||||
|
||||
### Memory Configuration
|
||||
|
||||
Since memory is unified, set these parameters carefully:
|
||||
|
||||
```
|
||||
vllm serve model \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--max-num-batched-tokens 4096
|
||||
```
|
||||
|
||||
- `gpu_memory_utilization`: Set lower (0.8-0.85) to leave room for CPU operations
|
||||
- `max_model_len`: Match actual context needs
|
||||
- The old `--swap-space` argument was removed from current vLLM. Use `--kv-offloading-size` for CPU KV offload
|
||||
|
||||
### NUMA Binding
|
||||
|
||||
Strix Halo may have multiple NUMA nodes. Use NUMA binding:
|
||||
|
||||
```
|
||||
vllm serve model \
|
||||
--numa-bind
|
||||
```
|
||||
|
||||
For explicit binding:
|
||||
|
||||
```
|
||||
vllm serve model \
|
||||
--numa-bind \
|
||||
--numa-bind-nodes 0 0
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Set these environment variables for optimal performance:
|
||||
|
||||
```
|
||||
export ROCM_DISABLE_TCE=0
|
||||
export HSA_ENABLE_SDMA=1
|
||||
export HSA_FORCE_FINE_GRAIN_PCIE=1
|
||||
```
|
||||
|
||||
### GPU Memory Limits
|
||||
|
||||
Strix Halo APU shares memory between CPU and GPU. Set GPU memory limit:
|
||||
|
||||
```
|
||||
export HSA_OVERRIDE_GFX_VERSION=11.5.1
|
||||
```
|
||||
|
||||
This may be needed for Strix Halo compatibility.
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Memory Bandwidth
|
||||
|
||||
Unified memory has lower bandwidth than dedicated VRAM:
|
||||
|
||||
| Memory Type | Bandwidth |
|
||||
|-------------|-----------|
|
||||
| HBM3 (A100) | 2 TB/s |
|
||||
| GDDR6 (RTX 4090) | 1 TB/s |
|
||||
| LPDDR5x (Strix Halo) | ~170 GB/s |
|
||||
|
||||
### ROCm Overcommit
|
||||
|
||||
ROCm reports 96 GiB GPU memory on Strix Halo APU. Physical RAM is 64 GiB (61 GiB visible; corrected 2026-08-17, the box was upgraded from 32 GB). The GPU can address more than physical RAM exists. vLLM will see 96 GiB and try to allocate accordingly. Set `--gpu-memory-utilization` to 0.80 or lower to prevent OOM.
|
||||
|
||||
Lower bandwidth means slower inference. Plan for lower throughput.
|
||||
|
||||
### Cache Effects
|
||||
|
||||
Strix Halo has large L3 cache. This helps with memory-bound workloads:
|
||||
|
||||
- Large L3 cache reduces memory access latency
|
||||
- Prefetching helps with sequential KV cache access
|
||||
- Cache eviction may cause stalls during long contexts
|
||||
|
||||
### Thermal Throttling
|
||||
|
||||
APUs generate more heat than discrete GPUs:
|
||||
|
||||
- Monitor GPU temperature with `rocm-smi`
|
||||
- Ensure adequate cooling
|
||||
- Thermal throttling reduces performance
|
||||
- Consider undervolting for sustained workloads
|
||||
|
||||
### Concurrency Limits
|
||||
|
||||
Unified memory limits concurrent requests:
|
||||
|
||||
- Each request uses memory from the shared pool
|
||||
- CPU operations also use the shared pool
|
||||
- Reduce `max_num_seqs` to avoid memory pressure
|
||||
- Monitor memory usage with `rocm-smi`
|
||||
|
||||
## Known Limitations
|
||||
|
||||
### Feature Support
|
||||
|
||||
| Feature | Support on ROCm |
|
||||
|---------|-----------------|
|
||||
| Tensor Parallelism | Yes |
|
||||
| FP8 Quantization | Depends on GPU |
|
||||
| CUDA Graphs | Yes |
|
||||
| Async Output | No |
|
||||
| Multi-step Decoding | No |
|
||||
| Speculative Decoding | Yes |
|
||||
| Prefix Caching | Yes |
|
||||
| Chunked Prefill | Yes |
|
||||
|
||||
### Context Length
|
||||
|
||||
Some models on ROCm have context length limited to 4096 tokens. Check model documentation.
|
||||
|
||||
### FP8 Support
|
||||
|
||||
FP8 quantization depends on GPU architecture:
|
||||
|
||||
- MI300 series supports native FP8
|
||||
- Radeon RX 7900 supports FP8 via software emulation
|
||||
- Strix Halo (gfx1151) uses **software emulation** for FP8 (no native FP8 tensor cores); expect memory savings, not full compute speedup
|
||||
|
||||
For FP4 on Dankclaw, the ROCmFPX llama.cpp fork (`Q2_0_ROCMFP4`) provides native ROCm FP4 kernels.
|
||||
|
||||
## Example Commands
|
||||
|
||||
### Basic APU Deployment
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B \
|
||||
--attention-backend ROCM_AITER_UNIFIED_ATTN \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--swap-space 8 \
|
||||
--max-num-batched-tokens 4096 \
|
||||
--max-num-seqs 64
|
||||
```
|
||||
|
||||
### With FP8 Model
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B-FP8 \
|
||||
--attention-backend ROCM_AITER_UNIFIED_ATTN \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--swap-space 8
|
||||
```
|
||||
|
||||
### With Long Context
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B \
|
||||
--attention-backend ROCM_AITER_UNIFIED_ATTN \
|
||||
--gpu-memory-utilization 0.8 \
|
||||
--swap-space 16 \
|
||||
--max-model-len 16384 \
|
||||
--rope-scaling '{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":32768}'
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
Check GPU status:
|
||||
|
||||
```
|
||||
rocm-smi
|
||||
```
|
||||
|
||||
Check ROCm device info:
|
||||
|
||||
```
|
||||
rocminfo
|
||||
```
|
||||
|
||||
Check memory usage:
|
||||
|
||||
```
|
||||
free -h
|
||||
```
|
||||
|
||||
Monitor temperature:
|
||||
|
||||
```
|
||||
rocm-smi --showtemperature
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [[VLLM Overview]]
|
||||
- [[VLLM Index]]
|
||||
- [[RAM CPU Offloading in VLLM]]
|
||||
- [[Hybrid CPU GPU Processing Dankserver]]
|
||||
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
|
||||
@@ -0,0 +1,95 @@
|
||||
# vLLM Index
|
||||
|
||||
> **RETIRED (2026-08-18).** The vLLM service on Dankclaw was replaced by the CIRU ROCmFPX llama.cpp runtime (v3). These pages are historical reference. See [[ciru-activefpx-promptforge]].
|
||||
|
||||
## Documentation Overview
|
||||
|
||||
This index contains all documentation for vLLM deployment on Dankserver and Dankclaw.
|
||||
|
||||
## Hardware Guides
|
||||
|
||||
- [[Dankserver System Specs]] — NVIDIA GPU server specifications
|
||||
- [[Dankclaw System Specs]] — AMD GPU server specifications
|
||||
|
||||
## vLLM Architecture
|
||||
|
||||
- [[VLLM Overview]] — Complete vLLM documentation including installation, CLI, API, and key arguments
|
||||
- [[RAM CPU Offloading in VLLM]] — CPU RAM offloading mechanisms, swap space, and memory configuration
|
||||
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU with system RAM hybrid processing
|
||||
- [[Shared Memory Architecture Dankclaw]] — AMD ROCm unified memory and APU configuration
|
||||
|
||||
## Model Deployment Guides
|
||||
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B FP8 deployment on NVIDIA GPU
|
||||
- [[Running Qwen3.6 27B FP8 on Dankclaw]] — Qwen3.6-27B FP8 deployment on AMD GPU
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Installation Commands
|
||||
|
||||
```
|
||||
# NVIDIA GPU
|
||||
uv pip install vllm --torch-backend=auto
|
||||
|
||||
# AMD ROCm
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
|
||||
# CPU
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu/
|
||||
```
|
||||
|
||||
### Server Start Commands
|
||||
|
||||
```
|
||||
# Basic server
|
||||
vllm serve Qwen/Qwen2.5-1.5B-Instruct
|
||||
|
||||
# With tensor parallelism
|
||||
vllm serve meta-llama/Llama-3.1-70B-Instruct --tensor-parallel-size 4
|
||||
|
||||
# With quantization
|
||||
vllm serve Qwen/Qwen2.5-1.5B-Instruct --quantization fp8
|
||||
|
||||
# With GPU memory control
|
||||
vllm serve Qwen/Qwen2.5-1.5B-Instruct --gpu-memory-utilization 0.8 --swap-space 8
|
||||
```
|
||||
|
||||
### Common API Endpoints
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
|----------|--------|-------------|
|
||||
| /v1/models | GET | List available models |
|
||||
| /v1/completions | POST | Text completion |
|
||||
| /v1/chat/completions | POST | Chat completion |
|
||||
| /health | GET | Health check |
|
||||
| /metrics | GET | Prometheus metrics |
|
||||
|
||||
### Key Parameters
|
||||
|
||||
| Parameter | Description | Typical Value |
|
||||
|-----------|-------------|---------------|
|
||||
| tensor_parallel_size | Number of GPUs for model sharding | 1-8 |
|
||||
| gpu_memory_utilization | Fraction of GPU memory for KV cache (default 0.92) | 0.7-0.95 |
|
||||
| swap_space | Removed in current vLLM; use `--kv-offloading-size` | n/a |
|
||||
| max_model_len | Maximum context length | auto |
|
||||
| max_num_batched_tokens | Max tokens per batch | 2048-8192 |
|
||||
| max_num_seqs | Max sequences per batch | 128-512 |
|
||||
|
||||
### Quantization Methods
|
||||
|
||||
| Method | GPU Support | Description |
|
||||
|--------|-------------|-------------|
|
||||
| FP8 | Hopper, Ada (native); Ampere W8A16 via Marlin; RDNA via emulation | FP8 E4M3 inference |
|
||||
| INT8 | Most GPUs | Weight-only INT8 |
|
||||
| GPTQ | Most GPUs (NVIDIA; limited on ROCm) | Post-training GPTQ |
|
||||
| AWQ | Turing+ (NVIDIA) | Activation-aware weight quantization |
|
||||
| GGUF | CPU, GPU | General GPU unification format |
|
||||
|
||||
## See Also
|
||||
|
||||
- [[VLLM Overview]]
|
||||
- [[RAM CPU Offloading in VLLM]]
|
||||
- [[Hybrid CPU GPU Processing Dankserver]]
|
||||
- [[Shared Memory Architecture Dankclaw]]
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
|
||||
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
|
||||
@@ -0,0 +1,238 @@
|
||||
# vLLM Overview
|
||||
|
||||
## What Is vLLM
|
||||
|
||||
vLLM is a fast and easy library for large language model inference and serving. You can use vLLM to run models on NVIDIA GPUs, AMD GPUs, Intel GPUs, and CPUs. vLLM supports over 200 model architectures from Hugging Face.
|
||||
|
||||
vLLM is fast with these features:
|
||||
|
||||
- PagedAttention for efficient key and value memory management
|
||||
- Continuous batching of incoming requests
|
||||
- Chunked prefill and prefix caching
|
||||
- FP8, INT8, INT4, GPTQ, AWQ, and GGUF quantization
|
||||
- Tensor, pipeline, data, expert, and context parallelism
|
||||
- Speculative decoding with n-gram, suffix, EAGLE/EAGLE3, DFlash, DSpark, and MTP
|
||||
- Automatic kernel generation with torch.compile
|
||||
- Disaggregated prefill, decode, and encode
|
||||
|
||||
vLLM is flexible with these features:
|
||||
|
||||
- Seamless integration with Hugging Face models
|
||||
- High-throughput serving with parallel sampling and beam search
|
||||
- Streaming outputs
|
||||
- Structured outputs with xgrammar or guidance
|
||||
- Tool calling and reasoning parsers
|
||||
- OpenAI-compatible API server
|
||||
- Anthropic Messages API and gRPC support
|
||||
- Efficient multi-LoRA support
|
||||
|
||||
## Installation
|
||||
|
||||
You can install vLLM with uv (recommended) or pip:
|
||||
|
||||
```
|
||||
uv pip install vllm
|
||||
```
|
||||
|
||||
Or with pip for NVIDIA GPUs:
|
||||
|
||||
```
|
||||
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129
|
||||
```
|
||||
|
||||
For AMD GPUs with ROCm:
|
||||
|
||||
```
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
|
||||
For CPU:
|
||||
|
||||
```
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/nightly/cpu/
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Offline Batched Inference
|
||||
|
||||
```python
|
||||
from vllm import LLM, SamplingParams
|
||||
|
||||
llm = LLM(model="Qwen/Qwen2.5-1.5B-Instruct")
|
||||
|
||||
prompts = [
|
||||
"Hello, my name is",
|
||||
"The capital of France is",
|
||||
]
|
||||
|
||||
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
|
||||
|
||||
outputs = llm.generate(prompts, sampling_params)
|
||||
|
||||
for output in outputs:
|
||||
generated_text = output.outputs[0].text
|
||||
print(generated_text)
|
||||
```
|
||||
|
||||
### Online Serving
|
||||
|
||||
Start the OpenAI-compatible API server:
|
||||
|
||||
```
|
||||
vllm serve Qwen/Qwen2.5-1.5B-Instruct
|
||||
```
|
||||
|
||||
Query the server:
|
||||
|
||||
```
|
||||
curl http://localhost:8000/v1/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen2.5-1.5B-Instruct",
|
||||
"prompt": "San Francisco is a",
|
||||
"max_tokens": 7,
|
||||
"temperature": 0
|
||||
}'
|
||||
```
|
||||
|
||||
Chat completions:
|
||||
|
||||
```
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "Qwen/Qwen2.5-1.5B-Instruct",
|
||||
"messages": [
|
||||
{"role": "user", "content": "Hello!"}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
## Key Arguments
|
||||
|
||||
### Model Arguments
|
||||
|
||||
| Argument | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `model` | required | Model name or path |
|
||||
| `tokenizer` | same as model | Tokenizer name or path |
|
||||
| `tokenizer_mode` | auto | Tokenizer mode |
|
||||
| `trust_remote_code` | false | Trust remote code from Hugging Face |
|
||||
| `dtype` | auto | Data type (auto, float16, bfloat16, float32) |
|
||||
| `max_model_len` | auto | Maximum context length |
|
||||
| `quantization` | null | Quantization method (awq, gptq, fp8, compressed-tensors) |
|
||||
| `load_format` | auto | Weight loading format |
|
||||
|
||||
### GPU Arguments
|
||||
|
||||
| Argument | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `tensor_parallel_size` | 1 | Number of GPUs for tensor parallelism |
|
||||
| `pipeline_parallel_size` | 1 | Number of GPUs for pipeline parallelism |
|
||||
| `gpu_memory_utilization` | 0.92 | Fraction of GPU memory for KV cache |
|
||||
| `swap_space` | removed | Removed in current vLLM; use `--kv-offloading-size` |
|
||||
| `enforce_eager` | false | Use eager mode (no CUDA graphs) |
|
||||
| `max_num_batched_tokens` | 2048 | Max tokens per batch |
|
||||
| `max_num_seqs` | 128 | Max sequences per batch |
|
||||
| `block_size` | 16 | Token block size |
|
||||
| `num_gpu_blocks_override` | null | Override number of GPU blocks |
|
||||
| `max_model_len_per_seq` | null | Max length per sequence |
|
||||
|
||||
### Optimization Arguments
|
||||
|
||||
| Argument | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `disable_log_stats` | false | Disable logging |
|
||||
| `disable_log_requests` | false | Disable request logging |
|
||||
| `max_log_len` | null | Max log length |
|
||||
| `disable_custom_all_reduce` | false | Disable custom all-reduce |
|
||||
| `tokenizer_pool_size` | 0 | Tokenizer pool size |
|
||||
| `tokenizer_pool_type` | none | Tokenizer pool type |
|
||||
| `enable_chunked_prefill` | auto | Enable chunked prefill |
|
||||
| `max_num_batched_tokens` | 2048 | Max tokens per batch |
|
||||
| `attention_backend` | auto | Attention backend (flash_attn, flashinfer) |
|
||||
|
||||
### Memory Arguments
|
||||
|
||||
| Argument | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `gpu_memory_utilization` | 0.92 | Fraction of GPU memory for KV cache |
|
||||
| `swap_space` | removed | Removed in current vLLM; use `--kv-offloading-size` |
|
||||
| `kv_cache_dtype` | auto | KV cache data type (auto, fp8, fp8_e5m2) |
|
||||
| `max_model_len` | auto | Maximum context length |
|
||||
| `num_gpu_blocks` | auto | Number of GPU blocks |
|
||||
| `num_cpu_blocks` | auto | Number of CPU blocks |
|
||||
|
||||
### Device Arguments
|
||||
|
||||
| Argument | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `device` | auto | Device type (cuda, rocm, xpu, cpu, metal) |
|
||||
| `distributed_executor_backend` | auto | Distributed executor backend |
|
||||
| `huggingface_tokenizer_kwargs` | null | Hugging Face tokenizer kwargs |
|
||||
|
||||
## Architecture
|
||||
|
||||
vLLM uses a multi-process architecture:
|
||||
|
||||
- API server process handles HTTP requests, tokenization, and input processing
|
||||
- Engine core process runs the scheduler and coordinates GPU workers
|
||||
- One GPU worker process per GPU executes model forward passes
|
||||
|
||||
vLLM uses PagedAttention for memory management. PagedAttention treats KV cache as a set of memory pages. This allows efficient memory allocation and reduces fragmentation.
|
||||
|
||||
vLLM supports multiple attention backends:
|
||||
|
||||
- FlashAttention for NVIDIA GPUs
|
||||
- FlashInfer for NVIDIA GPUs
|
||||
- Triton attention for AMD ROCm
|
||||
- ROCm attention for AMD ROCm
|
||||
|
||||
## Supported Models
|
||||
|
||||
vLLM supports over 200 model architectures including:
|
||||
|
||||
- Decoder-only LLMs (Llama, Qwen, Gemma)
|
||||
- Mixture-of-Expert LLMs (Mixtral, DeepSeek-V3, Qwen-MoE)
|
||||
- Hybrid attention and state-space models (Mamba, Qwen3.5)
|
||||
- Multi-modal models (LLaVA, Qwen-VL, Pixtral)
|
||||
- Embedding and retrieval models (E5-Mistral, GTE, ColBERT)
|
||||
- Reward and classification models (Qwen-Math)
|
||||
|
||||
## Quantization
|
||||
|
||||
vLLM supports these quantization methods:
|
||||
|
||||
- FP8 for Hopper and Ada GPUs (native). Ampere runs FP8 W8A16 via Marlin; FP8 W8A8 computes via fallback
|
||||
- MXFP8/MXFP4 for Hopper GPUs
|
||||
- NVFP4 for Hopper and Blackwell GPUs
|
||||
- INT8 for most GPUs
|
||||
- INT4 for most GPUs
|
||||
- GPTQ for most GPUs
|
||||
- AWQ for Turing and newer GPUs (NVIDIA; native support varies on ROCm)
|
||||
- GGUF for CPU and GPU
|
||||
- Compressed-tensors for custom quantization
|
||||
- ModelOpt for NVIDIA Model Optimizer
|
||||
- TorchAO for PyTorch AO
|
||||
|
||||
AMD RDNA 3 / RDNA 3.5 (gfx11, Strix Halo) run FP8 via software emulation, not native tensor cores. AWQ and GPTQ support in vLLM on AMD ROCm is limited; use GGUF or AMD Quark.
|
||||
|
||||
## Parallelism
|
||||
|
||||
vLLM supports multiple parallelism strategies:
|
||||
|
||||
- Tensor Parallelism (TP) shards model parameters across GPUs
|
||||
- Pipeline Parallelism (PP) distributes model layers across GPUs
|
||||
- Data Parallelism (DP) replicates the model across GPU sets
|
||||
- Expert Parallelism (EP) distributes expert networks in MoE models
|
||||
- Context Parallelism (CP) splits context across GPUs
|
||||
|
||||
## See Also
|
||||
|
||||
- [[VLLM Index]]
|
||||
- [[RAM CPU Offloading in VLLM]]
|
||||
- [[Hybrid CPU GPU Processing Dankserver]]
|
||||
- [[Shared Memory Architecture Dankclaw]]
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]]
|
||||
- [[Running Qwen3.6 27B FP8 on Dankclaw]]
|
||||
@@ -0,0 +1,265 @@
|
||||
---
|
||||
type: reference
|
||||
title: CIRU ActiveFPX PromptForge on Dankclaw
|
||||
description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3) served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-default-large service.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- rocm
|
||||
- dankclaw
|
||||
- gfx1151
|
||||
- ciru
|
||||
- promptforge
|
||||
timestamp: 2026-08-22
|
||||
---
|
||||
|
||||
# CIRU ActiveFPX PromptForge on Dankclaw
|
||||
|
||||
> **Active 2026-08-22.** `frank-default-large` serves this v3 model again
|
||||
> after a same-day Kairic Edge IU4 experiment was reverted (operator
|
||||
> decision). Commit `c60156f` restored the v3 script pair. See
|
||||
> [[kairic-edge]] for the retired experiment reference.
|
||||
|
||||
`frank-default-large` (port 7770, moved from 52625 on 2026-08-22) serves
|
||||
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` **v3** through the CIRU
|
||||
ROCmFPX llama.cpp runtime. This replaced the retired vLLM service that
|
||||
previously occupied the slot. The v2 deployment moved to v3 on 2026-08-21.
|
||||
|
||||
## What This Is
|
||||
|
||||
The model is a **GGUF**, not a vLLM checkpoint. v3 is the consolidated
|
||||
quality-and-speed release of Qwen3.8-27B for AMD ROCm: an FFN-equalized
|
||||
compact body, 50 retained late-writer Q6 tensors, full-Q8 output
|
||||
projection, full-W8 PromptForge view (FFN + Gated DeltaNet), the
|
||||
small-row W8 route, a draft-only Output-K8 shortlist proxy with Q8
|
||||
reranking, native depth-4 MTP (no ngram), and the BF16 Qwen3.8 vision
|
||||
tower + 5120-wide merger as a separate projector.
|
||||
|
||||
| Artifact | Size |
|
||||
|----------|------|
|
||||
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3-Q8-FFNEQ-LateQ6.gguf` | 15.44 GiB |
|
||||
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs` | 15.95 GiB |
|
||||
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs` | 3.75 GiB |
|
||||
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-Output-K8.pfs` | 0.67 GiB |
|
||||
| `Qwen3.8-27B-mmproj-BF16.mmproj` | 0.87 GiB |
|
||||
| Full v3 payload (GGUF + 3 .pfs + mmproj) | ~36.7 GiB |
|
||||
|
||||
The `.pfs` companions and the Output-K8 proxy are optional. Omit them to
|
||||
run the GGUF alone and save ~20.6 GiB of memory. The GGUF remains
|
||||
independently usable.
|
||||
|
||||
## Requirements
|
||||
|
||||
- **ROCm 7.15 is required** for the advertised Strix Halo performance.
|
||||
The system ROCm (`/opt/rocm-7.2.4`) selects slower or fallback kernels.
|
||||
- The pinned runtime is `ciru-ai/ROCmFPX` tag
|
||||
`qwen3.8-activefpx-promptforge-v2.3` (commit `103af8b41`) **plus the
|
||||
v3 runtime patch** `runtime/qwen38-v3-output-k8-runtime.patch`. On
|
||||
Dankclaw the patch is commit `af0eac6` on top of the v2.3 checkout.
|
||||
- Composable Kernel at `fdf4bb7fcc984811cef48ce817d89aac064b984a`.
|
||||
- Validated GPU: AMD Radeon 8060S / gfx1151.
|
||||
|
||||
## ROCm 7.15 Toolchain
|
||||
|
||||
The retired `ling3-ciru` package left a complete TheRock ROCm 7.15 SDK
|
||||
(`7.15.0a20260728`) inside its venv:
|
||||
|
||||
```
|
||||
/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/
|
||||
_rocm_sdk_devel/ # hipcc, amdclang 23, cmake, headers
|
||||
_rocm_sdk_libraries/ # hipblas, hipblaslt, rocroller
|
||||
_rocm_sdk_core/ # libamdhip64, libhsa-runtime64
|
||||
rocm-sdk-device-gfx1151
|
||||
```
|
||||
|
||||
The compiler is amdclang 23. `rocm-sdk version` reports `7.15.0a20260728`.
|
||||
The binary is built against and run with these 7.15 libraries.
|
||||
|
||||
## Build Runbook (2026-08-18, v2.3; +v3 patch on 2026-08-21)
|
||||
|
||||
Step 1-4 below built v2.3 on 2026-08-18. On 2026-08-21 the v3
|
||||
Output-K8 runtime patch was applied on top and the server rebuilt:
|
||||
|
||||
0. Apply the v3 runtime patch to the v2.3 checkout and rebuild:
|
||||
```
|
||||
% verify the patch hash: 8ba7d721ea380b060aeb0c7595b8b05f1582c0f7bf7a4b89e0910bc8e72465d5
|
||||
git am runtime/qwen38-v3-output-k8-runtime.patch # -> commit af0eac6
|
||||
cmake --build build-promptforge --target llama-server -j$(nproc)
|
||||
```
|
||||
The pre-built binary is kept as `build-promptforge/bin/llama-server.v2.3-bak`
|
||||
for rollback.
|
||||
|
||||
1. Install host build tools (ninja-build was missing): `sudo apt install ninja-build`.
|
||||
2. Clone the runtime and CK at their pinned revisions:
|
||||
```
|
||||
git clone --depth 1 --branch qwen3.8-activefpx-promptforge-v2.3 \
|
||||
https://github.com/ciru-ai/ROCmFPX.git
|
||||
git clone --depth 1 https://github.com/ROCm/composable_kernel.git
|
||||
git -C composable_kernel checkout fdf4bb7fcc984811cef48ce817d89aac064b984a
|
||||
```
|
||||
3. CK headers are generated at CK configure time (`ck/config.h`,
|
||||
`ck/version.h`). The llama.cpp build includes the CK source tree, so run
|
||||
CK's configure once and copy the headers into `include/ck/`:
|
||||
```
|
||||
cmake -S composable_kernel -B build-ck -DAMDGPU_TARGETS=gfx1151 -DCK_ENABLE_WMMA=ON
|
||||
cp build-ck/include/ck/config.h build-ck/include/ck/version.h composable_kernel/include/ck/
|
||||
```
|
||||
4. Configure the llama.cpp build against the venv ROCm 7.15 SDK:
|
||||
```
|
||||
export ROCM_PATH=$SDK/_rocm_sdk_devel
|
||||
cmake -S ROCmFPX -B build-promptforge -G Ninja \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_C_COMPILER=$ROCM_PATH/llvm/bin/clang \
|
||||
-DCMAKE_CXX_COMPILER=$ROCM_PATH/llvm/bin/clang++ \
|
||||
-DCMAKE_HIP_COMPILER=$ROCM_PATH/llvm/bin/clang++ \
|
||||
-DCMAKE_PREFIX_PATH=$ROCM_PATH \
|
||||
-DGGML_HIP=ON -DGGML_CUDA=OFF -DGGML_VULKAN=OFF \
|
||||
-DGGML_HIP_FORCE_MMQ=ON -DGGML_HIP_GRAPHS=ON \
|
||||
-DGGML_HIP_MMQ_MFMA=ON -DGGML_HIP_NO_VMM=ON \
|
||||
-DGGML_HIP_ROCWMMA_FATTN=OFF -DGGML_NATIVE=ON \
|
||||
-DAMDGPU_TARGETS=gfx1151 -DGPU_BUILD_TARGETS=gfx1151 \
|
||||
-DPROMPTFORGE_CK_ROOT=$CK_ROOT \
|
||||
-DLLAMA_BUILD_SERVER=ON \
|
||||
-DLLAMA_BUILD_WEBUI=OFF
|
||||
cmake --build build-promptforge --target llama-server -j$(nproc)
|
||||
```
|
||||
`LLAMA_BUILD_WEBUI=OFF` is required. The embedded browser UI downloads
|
||||
assets from an HF bucket (`llama-ui`) that returns 404; the service is
|
||||
consumed through the OpenAI-compatible API, so the UI is not needed.
|
||||
|
||||
Build output: `build-promptforge/bin/llama-server`.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Clone ROCmFPX v2.3 + CK pinned] --> B[Configure CK, copy generated headers]
|
||||
B --> C[Configure llama-server vs ROCm 7.15 SDK]
|
||||
C --> D[Build llama-server]
|
||||
D --> E[Verify --list-devices shows gfx1151]
|
||||
E --> F[Download v3 GGUF + FFN/GDN/Output-K8 + mmproj]
|
||||
F --> G[Deploy frank-default-large]
|
||||
G --> H[Health check + chat completion]
|
||||
```
|
||||
|
||||
## Model Download
|
||||
|
||||
With the HF token to the v3 staging directory
|
||||
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`. The v3 files live there
|
||||
so the v2 artifacts in `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/` stay
|
||||
intact for rollback. Download ran on Dankclaw with the background script
|
||||
`/tmp/download-v3.sh`; all five files + the patch passed SHA-256 checks
|
||||
against the model card before the service restart.
|
||||
|
||||
## Service
|
||||
|
||||
- Unit: `frank-default-large.service`, port 7770
|
||||
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-default-large.env`
|
||||
- The script exports the ROCm 7.15 SDK libraries in `LD_LIBRARY_PATH` before
|
||||
the system ROCm. `ROCM_PATH` points at the devel SDK.
|
||||
|
||||
### Launch profile
|
||||
|
||||
- 262144 context (native max), 1 slot (user request 2026-08-19; parallel
|
||||
slots degraded performance), f16 KV (~16 GiB), context checkpoints (16),
|
||||
prompt cache. Concurrent requests queue on the single slot; the server
|
||||
never preempts a running generation
|
||||
- Native MTP draft, depth 4, device ROCm0
|
||||
- `--temp 0.7 --top-p 0.95`, reasoning on (`--reasoning-format deepseek`
|
||||
puts `<think>` reasoning in `message.reasoning_content`)
|
||||
- v3 runtime: Output-K8 draft proxy (top-8 shortlist, Q8 rerank) +
|
||||
small-row W8 prompt route, full-W8 FFN/GDN views
|
||||
- PromptForge env: `PROMPTFORGE_MODE=m2048_fused_tail1476`,
|
||||
`GGML_CUDA_GRAPH_OPT=0`, `PROMPTFORGE_MTP_OUTPUT_K8=1`,
|
||||
`PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE=0`,
|
||||
`PROMPTFORGE_ENABLE_SMALLM_W8=1`
|
||||
|
||||
### Required environment
|
||||
|
||||
```
|
||||
export PROMPTFORGE_SIDECAR=...FFN.pfs
|
||||
export PROMPTFORGE_GDN_SIDECAR=...GDN.pfs
|
||||
export PROMPTFORGE_MODE=m2048_fused_tail1476
|
||||
export GGML_CUDA_GRAPH_OPT=0
|
||||
export PROMPTFORGE_ENABLE_SMALLM_W8=1
|
||||
export PROMPTFORGE_MTP_OUTPUT_K8=1
|
||||
export PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE=0
|
||||
export PROMPTFORGE_MTP_OUTPUT_K8_PROXY=...Output-K8.pfs
|
||||
# v3 qualified profile: disable the CPU argmax draft fast path and any
|
||||
# IU4/ngram lane. The Output-K8 proxy + backend sampling replace them.
|
||||
unset LLAMA_MTP_CPU_ARGMAX_FASTPATH LLAMA_MTP_BACKEND_GREEDY
|
||||
unset PROMPTFORGE_IU4_SIDECAR PROMPTFORGE_GDN_IU4_SIDECAR
|
||||
unset PROMPTFORGE_ENABLE_NGRAM_MOD PROMPTFORGE_ENABLE_NGRAM_M65_IU4
|
||||
```
|
||||
|
||||
The v3 qualified profile runs the MTP draft through the Output-K8 proxy,
|
||||
not the CPU argmax fast path. The old `LLAMA_MTP_CPU_ARGMAX_FASTPATH=1`
|
||||
profile is retired for this slot. A TOP_K sampler warning may still appear
|
||||
in the journal once per start; the Output-K8 draft path does not depend on
|
||||
it and generation proceeds.
|
||||
|
||||
## Operational Notes
|
||||
|
||||
- Health: `curl http://127.0.0.1:7770/health`. Returns 503 while loading.
|
||||
- Model load is slow (v3 five-file payload ~36.7 GiB); do not kill the
|
||||
process during warmup. The page cache makes a repeat load fast right after
|
||||
download, but a later cold start takes several minutes.
|
||||
- `deploy.sh dankclaw` waits 20 minutes for health.
|
||||
- The systemd unit must not set `HOST` or `TEMP` env vars (HIP collision
|
||||
rule). `--host`/`--port`/`--temp` are CLI flags only.
|
||||
- v3 adds the BF16 vision tower over v2. The service reports `multimodal`
|
||||
capability; mmproj loads with the language model (CLIP on ROCm0 backend).
|
||||
|
||||
## ngram-mod + MTP Stacking (benchmarked 2026-08-21)
|
||||
|
||||
This page records a research question: does ngram-mod add value when
|
||||
stacked under the native MTP draft? The v3 profile runs MTP alone; the
|
||||
benchmark below compares that against the stacked pair.
|
||||
|
||||
### Feasibility
|
||||
|
||||
ngram-mod DOES stack with MTP in this CIRU fork. The v3 build compiles in
|
||||
both `ngram-mod` and `draft-mtp` (`common/speculative.cpp`). Pass
|
||||
`--spec-type draft-mtp,ngram-mod` (or `ngram-mod,draft-mtp`) and the fork
|
||||
initializes both implementations in the same process. The boot log shows:
|
||||
|
||||
```
|
||||
adding speculative implementation 'ngram-mod'
|
||||
initialized ngram_mod with n_match=24, size=4194304 (16.000 MB)
|
||||
adding speculative implementation 'draft-mtp'
|
||||
```
|
||||
|
||||
The model card's "no ngram subsystem" line refers to the CIRU PromptForge
|
||||
env lanes (`PROMPTFORGE_*_NGRAM_*`, `PROMPTFORGE_ENABLE_NGRAM_MOD`), not
|
||||
the upstream llama.cpp spec chain. Those env lanes must stay unset; the
|
||||
`--spec-type ngram-mod` chain is separate and works.
|
||||
|
||||
### Ordering — the CLI order does not matter
|
||||
|
||||
The `--spec-type` string order is discarded. `common_speculative_init`
|
||||
collapses the requested types into a bitmask, then rebuilds the chain in a
|
||||
hard-coded priority: ngram types first, then draft-simple, then MTP.
|
||||
So `draft-mtp,ngram-mod` and `ngram-mod,draft-mtp` produce the same
|
||||
internal order (`ngram-mod` then `draft-mtp`) and the same runtimes.
|
||||
Verified: both orders decode ~25.3 t/s at 8k.
|
||||
|
||||
### Results (decode generation t/s, 200 tokens, prose prompt)
|
||||
|
||||
| ctx | MTP only | MTP + ngram-mod | verdict |
|
||||
|-----|----------|-----------------|---------|
|
||||
| 8k | 29.42 | 25.27 | ngram hurts -4.15 |
|
||||
| 16k | 26.42 | 21.13 | ngram hurts -5.29 |
|
||||
| 64k | 19.72 | 18.00 | ngram hurts -1.72 |
|
||||
|
||||
At every depth, adding ngram-mod under MTP REDUCED decode throughput. The
|
||||
ngram draft costs its own forward pass / occupancy without adding enough
|
||||
accepted tokens on top of MTP. Baseline (no ngram) remains the production
|
||||
config (`--spec-type draft-mtp` only). Keep ngram-mod off.
|
||||
|
||||
The historical harness was moved to `.AGENTTRASH/bench-scripts-20260901/` and is retained for audit only.
|
||||
it boots the v3 stack on a scratch port and reads authoritative
|
||||
`print_timing` lines. It requires the production service to be stopped.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[frank-redo-2026-08-17]] — the fleet topologies this slot has served.
|
||||
- [[Shared Memory Architecture Dankclaw]] — unified memory sizing.
|
||||
- [[qwen3.8-strix-optimizations]] — llama.cpp ROCm0 baselines on this iGPU.
|
||||
- [[VLLM Overview]] — vLLM reference (retired for this slot).
|
||||
@@ -0,0 +1,140 @@
|
||||
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
|
||||
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
|
||||
> [deployment.md](deployment.md) for the active configuration.
|
||||
|
||||
---
|
||||
type: reference
|
||||
title: Dankserver VRAM and RAM Growth Under Load
|
||||
description: Investigation of the recurring cuBLAS crash on frank-default-small. What grows, what does not, and what to do next.
|
||||
tags:
|
||||
- dankserver
|
||||
- llamacpp
|
||||
- vram
|
||||
- ram
|
||||
- ctx-checkpoints
|
||||
- crash
|
||||
---
|
||||
|
||||
# Dankserver VRAM and RAM Growth Under Load
|
||||
|
||||
This page records the investigation of the recurring cuBLAS crash on
|
||||
`frank-default-small` (llama.cpp, RTX 3090 24 GiB). The service aborts
|
||||
with `CUDA error: an unsupported value or parameter was passed to the
|
||||
function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541` during
|
||||
`update_slots` -> `decode` -> `process_ubatch`. It crashed on 2026-08-13
|
||||
at 19:45 and 19:55 and on 2026-08-14 at 12:00:43.
|
||||
|
||||
## What Was Ruled Out
|
||||
|
||||
### ctx-checkpoints do not grow VRAM
|
||||
|
||||
`--ctx-checkpoints 4` stores context checkpoints in host RAM. The log
|
||||
records every checkpoint at exactly 62.813 MiB. The size never grows.
|
||||
The reasoning:
|
||||
|
||||
- `common_prompt_checkpoint` (common/common.h) stores the serialized KV
|
||||
state in `std::vector<uint8_t>` fields. These live on the host heap.
|
||||
- `update_tgt()` and `update_dft()` call `llama_state_seq_get_data_ext`
|
||||
with `LLAMA_STATE_SEQ_FLAGS_PARTIAL_ONLY`. The serialized state moves
|
||||
from VRAM to RAM for the snapshot and back on restore.
|
||||
- With 4 slots and `--ctx-checkpoints 4`, the maximum checkpoint RAM is
|
||||
16 x 62.813 MiB, about 1.0 GiB. That is fixed and small.
|
||||
- `data_spec` (spec-decode state) is empty for ngram-mod. The ngram-mod
|
||||
implementation does not override `get_state()`.
|
||||
|
||||
### ngram-mod does not grow VRAM
|
||||
|
||||
ngram-mod allocates one fixed 64 MiB CPU hash table
|
||||
(`common_ngram_mod::entries`, 16M int32 entries). It never resizes. The
|
||||
matcher keeps about 12 bytes per sequence. Draft generation runs on the
|
||||
CPU. The speculative path allocates zero GPU memory.
|
||||
|
||||
### The CUDA graph cache has eviction in this build
|
||||
|
||||
`ggml_backend_cuda_context::cuda_graph()` sweeps every 5 seconds and
|
||||
evicts graphs unused for 10 seconds. Upstream issue #25082 describes an
|
||||
unbounded graph cache, but build b10108-13-g555881ebc includes the
|
||||
eviction sweep. This build does not exhibit that leak.
|
||||
|
||||
## What Grows
|
||||
|
||||
### RAM growth is suspected, not yet confirmed
|
||||
|
||||
systemd reported MemoryPeak 46.9 GiB for the crashed process and 45.9
|
||||
GiB for the replacement. RSS is flat at 32.3 GiB, which matches the
|
||||
documented idle baseline of about 32 GiB. The difference is file-backed
|
||||
page cache from the `--no-mmap` model read; cgroup memory.current counts
|
||||
file pages attributed to the process. That gap is not proof of a leak by
|
||||
itself.
|
||||
|
||||
The upstream pattern to watch is issue #26216: the prompt cache and
|
||||
checkpoint accumulation exceed the `--cache-ram` cap and do not plateau.
|
||||
The environment file does not set `--cache-ram`, so the llama.cpp
|
||||
default of 8192 MiB applies. The monitor logs RSS, which separates
|
||||
anonymous memory from file cache. If RSS climbs under load while VRAM is
|
||||
flat, the RAM-side accumulation is real.
|
||||
|
||||
### VRAM suspects under load
|
||||
|
||||
VRAM is flat at idle: 21,268 MiB across every monitor sample. The crash
|
||||
correlates with load density and slot context size (143K tokens on
|
||||
slot 2), not with wall-clock time. The two mechanisms worth testing:
|
||||
|
||||
1. Flash-attention F16 K conversion workspace. The prefill kernel on
|
||||
Ampere is `MMA_F16`. It converts the quantized K cache (q8_0) to F16
|
||||
for the attention math. The workspace size scales with the attended
|
||||
context length (`ggml_nelements(K) x 2`). It is allocated into the
|
||||
graph buffer via `ggml_cuda_flash_attn_ext_get_alloc_size`.
|
||||
2. The CUDA VMM pool grows monotonically. `ggml_cuda_pool_vmm::alloc`
|
||||
only reserves more physical memory; `free` only releases the
|
||||
reservation slot. The pool never shrinks during the process
|
||||
lifetime. Any new peak allocation permanently expands VRAM usage.
|
||||
|
||||
Both match the observed pattern: growth under prefill bursts at deep
|
||||
context, flat at idle, crash when a burst exceeds the ~2.8 GiB headroom.
|
||||
|
||||
## Monitoring
|
||||
|
||||
The monitor runs on Dankserver:
|
||||
|
||||
```
|
||||
/home/dank/vram-monitor.sh
|
||||
/home/dank/vram-log.csv
|
||||
```
|
||||
|
||||
It samples GPU VRAM and process RAM every 10 seconds and appends a CSV
|
||||
row. The columns are timestamp, uptime, GPU VRAM MiB, GPU utilization,
|
||||
process RSS, process VSZ, and systemd MemoryPeak.
|
||||
|
||||
## Resolution: dual-f16 KV Cache (2026-08-14)
|
||||
|
||||
The fix chose the first mechanism over the second. Both K and V moved to f16 on 2026-08-14. When K is already f16, the flash-attention MMA_F16 prefill kernel needs no F16 conversion copy, so the context-proportional workspace disappears. The pool becomes static: 20,480 bytes per cell (10 full-attention layers x 2 kv heads x 256 dims x 2 bytes for K plus 2 bytes for V).
|
||||
|
||||
The pool at 800768 cells cost 15,640 MiB and did not fit. The load-time warmup graph (1472x1472 image) failed cudaMalloc by 248 MiB, and the service idled at 23,966 of 24,576 MiB. The context pool was refitted to 700416 cells (4 slots x 175104): KV pool 13,680 MiB, idle 22,110 MiB, about 2.0 GiB free after warmup. Parallel stays 4.
|
||||
|
||||
Verified 2026-08-14 after deploy: flash attention enabled, warmup clean, decode about 42-44 t/s, cached prompt tokens working. The vram monitor log shows the restart arc (23,966 -> 5,210 during load -> 22,110 idle).
|
||||
|
||||
## Mitigation Options
|
||||
|
||||
The documentation for the benchmark ([[frank-default-small-benchmark]])
|
||||
prescribes the first option for this crash class. None of these change
|
||||
the model.
|
||||
|
||||
| Option | Effect | Cost |
|
||||
| ------ | ------ | ---- |
|
||||
| `GGML_CUDA_FORCE_MMQ=1` | Bypasses the cuBLAS path in the crashing function | Slight decode slowdown |
|
||||
| `--cache-ram 2048` | Bounds the confirmed RAM growth | Cache misses on prompt reuse |
|
||||
| `--ctx-checkpoints 2` | Cuts checkpoint RAM and save cost in half | More prefill work after eviction |
|
||||
| Apply PR #24649 | Clears slot checkpoints before the prompt-cache save | Needs a llama.cpp rebuild |
|
||||
|
||||
Upstream references: issue #26216 (prompt cache growth), issue #25082
|
||||
(graph cache, not applicable to this build), issue #23635 (quantized KV
|
||||
with speculative decoding), PR #24649 (checkpoint-to-cache duplication),
|
||||
PR #25649 (slot checkpoints freed on prompt clear, merged), issue #26119
|
||||
(same cuBLAS signature, confirmed on V100 with speculative decoding).
|
||||
|
||||
## See Also
|
||||
|
||||
- [[frank-default-small-benchmark]]
|
||||
- [[Dankserver System Specs]]
|
||||
- [[deployment]]
|
||||
@@ -0,0 +1,126 @@
|
||||
---
|
||||
type: reference
|
||||
title: dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
||||
description: dealignai/Ornith-1.5-35B-A3B-UNCENSORED (CRACK Q6_K) served by upstream llama.cpp on the RTX 3090. MoE model with ngram-mod speculative decoding; selected expert weights offloaded to CPU with an explicit --override-tensor. frank-default-small.
|
||||
tags:
|
||||
- llamacpp
|
||||
- cuda
|
||||
- dankserver
|
||||
- moe
|
||||
- ornith
|
||||
- mtp
|
||||
timestamp: 2026-08-24
|
||||
---
|
||||
|
||||
# dealign Ornith-1.5-35B-A3B CRACK on Dankserver
|
||||
|
||||
> **Active 2026-08-25.** `frank-default-small` serves this model. It
|
||||
> replaced the CRACK Q8_0 file on the same upstream llama.cpp binary (b10121).
|
||||
> The Q6_K file was downloaded from the model repository and validated with
|
||||
> `llama-gguf` before installation.
|
||||
|
||||
`frank-default-small` (port 7770) serves
|
||||
`Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` from `dealignai` on Hugging Face. It is
|
||||
a **MoE** model (256 experts, 8 active) with a hybrid GatedDeltaNet (SSM) +
|
||||
attention architecture, abliterated (uncensored). The GGUF also contains a
|
||||
bundled MTP head, but this deployment uses ngram-mod instead.
|
||||
|
||||
## Model Files
|
||||
|
||||
| File | Size |
|
||||
|------|------|
|
||||
| `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` | 27.2 GiB |
|
||||
| `mmproj-Ornith-1.5-35B-A3B-f16.gguf` (vision) | 0.84 GiB |
|
||||
|
||||
Store the files under `/home/dank/models/Ornith-1.5-35B-A3B-CRACK/`.
|
||||
|
||||
## Layer Audit and Expert Offload
|
||||
|
||||
The Q6_K file contains 41 transformer blocks and 753 tensors. Each block
|
||||
contains 630 MiB of expert tensors. Blocks 3, 7, 11, 15, 19, 23, 27, 31,
|
||||
35, and 39 contain full attention. Block 40 contains the bundled MTP head.
|
||||
|
||||
The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for
|
||||
ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for
|
||||
block 40. The expert portion is 630 MiB in every block.
|
||||
|
||||
The selected configuration keeps all non-expert tensors on CUDA. It also
|
||||
keeps expert tensors from every full-attention block, plus every block from 35
|
||||
through 40, on CUDA. The unique GPU expert set is
|
||||
`3,7,11,15,19,23,27,31,35,36,37,38,39,40`. Expert tensors from all other
|
||||
blocks move to CPU through this explicit override:
|
||||
|
||||
```
|
||||
OVERRIDE_EXPERT='blk\.(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|24|25|26|28|29|30|32|33|34)\.ffn_(up|down|gate|gate_up)_(ch|)exps'
|
||||
OVERRIDE_BUFT=CPU
|
||||
```
|
||||
|
||||
The 16K benchmark found the hybrid policy fastest among the three tested
|
||||
placements. A later no-spec/ubatch sweep established that `ubatch=1024` is a
|
||||
stable low-memory choice. The current service uses ngram-mod only; MTP is
|
||||
intentionally disabled. The full-attention expert set and final blocks are
|
||||
kept on CUDA.
|
||||
|
||||
To list the buffer type names on the host, run a bogus override:
|
||||
|
||||
```
|
||||
llama-server --model <ANY> -c 64 -ot foo=BADTYPE
|
||||
```
|
||||
|
||||
On Dankserver the available buffer types are `CPU` and `CUDA0`.
|
||||
|
||||
## Speculative Decoding
|
||||
|
||||
The GGUF contains a bundled MTP head, but this deployment deliberately does
|
||||
not use it. The current service enables only `ngram-mod`, with a 48-token
|
||||
minimum, 64-token maximum, and 24-token lookup match:
|
||||
|
||||
```
|
||||
--spec-type ngram-mod
|
||||
--spec-ngram-mod-n-min 48
|
||||
--spec-ngram-mod-n-max 64
|
||||
--spec-ngram-mod-n-match 24
|
||||
```
|
||||
|
||||
In the controlled 8K test, no speculation produced approximately 53.7–53.9
|
||||
generation tokens per second. MTP was slower for that workload, so it remains
|
||||
disabled.
|
||||
|
||||
## Launch Excerpt
|
||||
|
||||
```
|
||||
llama-server \
|
||||
-m Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf \
|
||||
--mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \
|
||||
--chat-template-file chat-template.jinja \
|
||||
--spec-type ngram-mod \
|
||||
--spec-ngram-mod-n-min 48 \
|
||||
--spec-ngram-mod-n-max 64 \
|
||||
--spec-ngram-mod-n-match 24 \
|
||||
--override-tensor "blk\\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \
|
||||
-ngl all -c 491520 -np 3 \
|
||||
--ubatch-size 1024 \
|
||||
--cache-type-k f16 --cache-type-v f16 --no-mmap \
|
||||
--temp 1.0 --top-p 0.95 --top-k 20
|
||||
```
|
||||
|
||||
## Known Issues
|
||||
|
||||
1. The 27.2 GiB Q6_K file and 0.84 GiB vision projector fit the 62 GiB RAM
|
||||
box with the selected expert offload.
|
||||
2. The context pool contains 491520 cells. Three slots receive 163840 cells
|
||||
each, which is 96K below the model context per stream.
|
||||
3. `--n-cpu-moe` remains supported. This service prefers the explicit tensor
|
||||
override because the 16K benchmark found the hybrid placement fastest. It
|
||||
preserves every full-attention expert block and blocks 35–40 on CUDA.
|
||||
4. MTP is bundled in the GGUF but intentionally disabled; ngram-mod is the
|
||||
only active speculative method.
|
||||
4. The service uses the Qwen-Sharp template from
|
||||
`peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through
|
||||
`scripts/chat-template.jinja`.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[deployment]] — deploy and update process
|
||||
- [[Dankserver System Specs]] — the box
|
||||
- [[frank-redo-2026-08-17]] — fleet topology history
|
||||
@@ -0,0 +1,124 @@
|
||||
---
|
||||
type: guide
|
||||
title: LLM Deployment
|
||||
description: Scoped deployment and update process for the two inference services.
|
||||
tags:
|
||||
- llm
|
||||
- deployment
|
||||
- systemd
|
||||
timestamp: 2026-09-03
|
||||
---
|
||||
|
||||
# Deployment and Update Process
|
||||
|
||||
This page describes how to deploy and update the LLM servers on Dankserver and
|
||||
Dankclaw. The process uses Git as the single source of truth. A change to
|
||||
SearXNG, Caddy, or another container service does **not** restart inference.
|
||||
|
||||
## Fleet
|
||||
|
||||
| Target | Host | Service | Port |
|
||||
|---|---|---|---:|
|
||||
| Dankserver | `dank@dankserver` | `frank-default-small` | 7770 |
|
||||
| Dankclaw | `dank@dankclaw` | `frank-default-large` | 7770 |
|
||||
|
||||
## Scoped workflow
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
A[Edit LLM files] --> B[Run tests]
|
||||
B --> C[Commit main]
|
||||
C --> D[deploy.sh selects matching service]
|
||||
D --> E[Target pulls ff-only]
|
||||
E --> F[Relink one unit]
|
||||
F --> G[Restart one inference service]
|
||||
G --> H[Wait for that service health]
|
||||
```
|
||||
|
||||
Run the unified deployer from the repository root:
|
||||
|
||||
```bash
|
||||
./scripts/deploy.sh dankserver --service frank-default-small
|
||||
./scripts/deploy.sh dankclaw --service frank-default-large
|
||||
./scripts/deploy.sh dankclaw --dry-run
|
||||
```
|
||||
|
||||
Without `--service`, `scripts/deploy-plan.py` maps changed files to the service
|
||||
on the selected host. The following files select the corresponding service:
|
||||
|
||||
- `scripts/frank-default-small.env`, `frank-default-small.service`, and
|
||||
`serve-frank-small.sh` select `frank-default-small` on Dankserver.
|
||||
- `scripts/frank-default-large.env`, `frank-default-large.service`, and
|
||||
`serve-frank-large.sh` select `frank-default-large` on Dankclaw.
|
||||
- `scripts/chat-template.jinja` selects the LLM on the selected host only.
|
||||
- Documentation, tests, deployment tooling, and unrelated container changes
|
||||
do not select either inference service.
|
||||
|
||||
The target-side `scripts/deploy-remote.sh` relinks the selected unit into
|
||||
`/etc/systemd/system/`, reloads systemd, enables it, restarts it, and waits for
|
||||
`http://127.0.0.1:7770/health`. The wait budget is 20 minutes because cold model
|
||||
loads can be slow. A failed handler does not advance its per-service state.
|
||||
|
||||
## Unit and artifact paths
|
||||
|
||||
Both units point into the stable target checkout:
|
||||
|
||||
```text
|
||||
/home/dank/serverAppConfigs/scripts/frank-default-small.service
|
||||
/home/dank/serverAppConfigs/scripts/frank-default-large.service
|
||||
```
|
||||
|
||||
The unit files and serve scripts validate their model/runtime artifacts before
|
||||
starting. Keep `REMOTE_DIR` stable; moving the checkout requires updating both
|
||||
the unit absolute paths and the deployment tests in the same change.
|
||||
|
||||
## NPU unit
|
||||
|
||||
`frank-default-npu` is an inactive, historical Dankclaw endpoint. It is
|
||||
relinked only when explicitly selected with `--service frank-default-npu`; the
|
||||
large-model handler never restarts it. This is intentional: the NPU is not part
|
||||
of the active inference path and should not be disturbed by a large-model
|
||||
change.
|
||||
|
||||
## Isolation guarantee
|
||||
|
||||
The remote runner has one explicit service dispatch. It does not run a
|
||||
host-wide LLM loop, `podman-compose up` for unrelated services, or a blanket
|
||||
systemd restart. Each successful service is recorded under the target user's
|
||||
state directory outside Git:
|
||||
|
||||
```text
|
||||
~/.local/state/serverAppConfigs/deploy-state/
|
||||
```
|
||||
|
||||
On a later ordinary deployment, a service remains pending only while its
|
||||
recorded revision differs from the checkout and the planner still finds a
|
||||
change in that service's inputs.
|
||||
|
||||
## Manual recovery
|
||||
|
||||
If the LLM handler fails after the target pull:
|
||||
|
||||
1. Inspect `journalctl -u frank-default-small -n 200` or
|
||||
`journalctl -u frank-default-large -n 200`.
|
||||
2. Confirm the model files, custom runtime binary, and chat template are
|
||||
readable by `dank`.
|
||||
3. Retry only the affected service:
|
||||
|
||||
```bash
|
||||
./scripts/deploy.sh dankserver --service frank-default-small
|
||||
# or
|
||||
./scripts/deploy.sh dankclaw --service frank-default-large
|
||||
```
|
||||
|
||||
4. Verify the matching health endpoint before declaring recovery.
|
||||
|
||||
Do not stop the other host's inference service during recovery. For benchmark
|
||||
work, follow the separate stop/benchmark/restart procedure in [[Testing Policy]]
|
||||
and the repository rules.
|
||||
|
||||
## See Also
|
||||
|
||||
- [Deployment Flow](../Architecture/Deployment%20Flow.md)
|
||||
- [[index]]
|
||||
- [Dankclaw deployment architecture](../Architecture/Network%20Architecture.md)
|
||||
@@ -0,0 +1,174 @@
|
||||
---
|
||||
type: reference
|
||||
title: DFlash2 on Dankclaw (frank-default-large)
|
||||
description: ARCHIVED - unsloth Qwen3.8-27B-UD-Q8_K_XL served by upstream llama.cpp PR #27342 with a z-lab DFlash2 drafter on the Vulkan backend. Retired 2026-08-21 for speed; kept as a runbook.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- vulkan
|
||||
- dankclaw
|
||||
- gfx1151
|
||||
- dflash
|
||||
- dflash2
|
||||
- speculative-decoding
|
||||
timestamp: 2026-08-21
|
||||
---
|
||||
|
||||
# DFlash2 on Dankclaw (frank-default-large) — ARCHIVED
|
||||
|
||||
> **Archived 2026-08-21.** This DFlash2 Vulkan deployment was retired the
|
||||
> same day it shipped because it was slower than the CIRU runtime it
|
||||
> replaced. `frank-default-large` is back on CIRU. This page stays as the
|
||||
> runbook and provenance for the DFlash2 attempt.
|
||||
|
||||
`frank-default-large` (port 52625) briefly served
|
||||
`unsloth/Qwen3.8-27B-UD-Q8_K_XL` with a DFlash2 drafter from z-lab. The
|
||||
runtime was upstream llama.cpp PR #27342 built with `GGML_VULKAN=ON`.
|
||||
|
||||
## Why Vulkan
|
||||
|
||||
DFlash2 selects its draft candidates with a lightweight path selector
|
||||
built on the `TOP_K` and `ARGMAX` graph ops. The HIP backend does not
|
||||
implement `TOP_K`. That is why the retired CIRU setup routed the MTP
|
||||
draft around it with `LLAMA_MTP_CPU_ARGMAX_FASTPATH`.
|
||||
|
||||
The PR reference fork (Nathanw1014/strix-halo-llamacpp) validated DFlash2
|
||||
on this exact gfx1151 through Vulkan (RADV). Its `test-backend-ops` passed
|
||||
the new graph ops:
|
||||
|
||||
| Op | Cases |
|
||||
|----|-------|
|
||||
| TOP_K | 445 |
|
||||
| CONCAT | 193 |
|
||||
| PAD | 28 |
|
||||
| REPEAT | 18 |
|
||||
| FILL | 4 |
|
||||
|
||||
The whole stack (target + drafter) runs on `Vulkan0`.
|
||||
|
||||
## What Runs
|
||||
|
||||
| Artifact | Repo / File | Size |
|
||||
|----------|-------------|------|
|
||||
| Target | `unsloth/Qwen3.8-27B-GGUF` `Qwen3.8-27B-UD-Q8_K_XL.gguf` | 31.4 GB |
|
||||
| Drafter | `z-lab/Qwen3.8-27B-DFlash2-GGUF` `Qwen3.8-27B-DFlash2-Q4_K_M.gguf` | 2.0 GB |
|
||||
| Runtime | upstream llama.cpp PR #27342, GGML_VULKAN=ON | - |
|
||||
|
||||
The target was already on disk. The drafter was added for this deploy.
|
||||
Both live in `/home/dank/models/`.
|
||||
|
||||
## Drafter Hyperparameters
|
||||
|
||||
Read from the drafter GGUF metadata:
|
||||
|
||||
| Parameter | Value |
|
||||
|-----------|-------|
|
||||
| `dflash.block_count` | 5 |
|
||||
| `dflash.block_size` | 8 |
|
||||
| `dflash.conv_kernel_size` | 2 |
|
||||
| `dflash.conv_group_size` | 16 |
|
||||
| `dflash.selector_rank` | 256 |
|
||||
| `dflash.selector_top_k` | 16 |
|
||||
| `dflash.context_length` | 262144 |
|
||||
| `dflash.embedding_length` | 5120 |
|
||||
| architecture | `dflash` |
|
||||
|
||||
The target and drafter share the Qwen3.8 architecture
|
||||
(`qwen35`, dense 27B, vocab 248320), so the drafter inherits the target
|
||||
tokenizer and LM head.
|
||||
|
||||
## Build Runbook (2026-08-21)
|
||||
|
||||
Performed on Dankclaw.
|
||||
|
||||
1. In `/home/dank/llama.cpp`, register the `ggml` remote and fetch the PR
|
||||
head:
|
||||
```
|
||||
git -C /home/dank/llama.cpp remote add ggml https://github.com/ggml-org/llama.cpp.git
|
||||
git -C /home/dank/llama.cpp fetch ggml pull/27342/head:pr-27342
|
||||
```
|
||||
2. Add a detached worktree that does not disturb the running build:
|
||||
```
|
||||
git -C /home/dank/llama.cpp worktree add --detach /home/dank/llama.cpp-dflash2 pr-27342
|
||||
```
|
||||
3. Configure with Vulkan only (glslang and `glslc` must be installed):
|
||||
```
|
||||
cmake -B /home/dank/llama.cpp-dflash2/build-dflash2 -G Ninja \
|
||||
-DGGML_VULKAN=ON -DGGML_HIP=OFF -DGGML_CUDA=OFF -DGGML_METAL=OFF \
|
||||
-DGGML_NATIVE=ON -DCMAKE_BUILD_TYPE=Release \
|
||||
-DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_WEBUI=OFF
|
||||
```
|
||||
4. Build the server:
|
||||
```
|
||||
cmake --build /home/dank/llama.cpp-dflash2/build-dflash2 --target llama-server -j16
|
||||
```
|
||||
5. Verify the Vulkan device:
|
||||
```
|
||||
/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server --list-devices
|
||||
```
|
||||
Output: `Vulkan0: AMD Radeon Graphics (RADV STRIX_HALO)`.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Fetch PR #27342 head] --> B[Worktree on DFlash2 head]
|
||||
B --> C[Configure Vulkan build]
|
||||
C --> D[Build llama-server]
|
||||
D --> E[Verify Vulkan0 device]
|
||||
E --> F[Add DFlash2 drafter]
|
||||
F --> G[Swap frank-default-large]
|
||||
G --> H[Health check + completion]
|
||||
```
|
||||
|
||||
## Service
|
||||
|
||||
- Unit: `frank-default-large.service`, port 52625
|
||||
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-default-large.env`
|
||||
- Binary: `/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server`
|
||||
|
||||
### Launch profile
|
||||
|
||||
- 262144 context (native max), 1 slot, f16 KV (~16 GiB), 16 context
|
||||
checkpoints, 8 GiB prompt cache
|
||||
- DFlash2 draft, block size 8, device Vulkan0, drafter fully offloaded
|
||||
- `--spec-type draft-dflash --spec-draft-n-max 8`
|
||||
(clamped to block size 8 - 1 = 7 at runtime)
|
||||
- `--temp 0.7 --top-p 0.95`, reasoning on (`--reasoning-format deepseek`)
|
||||
- `--no-mmap`, flash attention `-fa on`, `--metrics`
|
||||
|
||||
### Required environment
|
||||
|
||||
No PromptForge variables are needed. The `TEMP`, `HOST`, and
|
||||
`GPU_MAX_HEAP_SIZE` naming rule still applies for portability, even though
|
||||
this build uses Vulkan, not HIP.
|
||||
|
||||
## Validation
|
||||
|
||||
The smoke test on 2026-08-21 used a scratch port (52626) and confirmed:
|
||||
|
||||
- Both models load and the server reports healthy.
|
||||
- A chat completion returns reasoning and content.
|
||||
- The DFlash2 drafter engaged:
|
||||
`draft-dflash: #gen drafts = 11, #acc drafts = 10, #mean acc len = 5.27`.
|
||||
- The per-position acceptance curve decays along the block
|
||||
`(0.909, 0.818, 0.727, 0.545, 0.545, 0.364, 0.364)`, as expected.
|
||||
|
||||
The published mean acceptance for the Q4_K_M drafter is ~5.39, so 5.27
|
||||
is close.
|
||||
|
||||
## Operational Notes
|
||||
|
||||
- Health: `curl http://127.0.0.1:52625/health`. Returns 503 while loading.
|
||||
- Model load takes about a minute with a warm OS page cache (31 GB target)
|
||||
and longer on a cold cache. Do not kill the process during warmup.
|
||||
- `deploy.sh dankclaw` waits 20 minutes for health.
|
||||
- The Vulkan device reports 113942 MiB. rocm-smi reports the 64 GiB
|
||||
unified-memory budget. Trust the runtime for the addressable SVM pool.
|
||||
- The MTP head of the target GGUF is ignored (`--spec-type draft-dflash`
|
||||
replaces it). The load log warns about unused MTP tensors; this is normal.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[ciru-activefpx-promptforge]] — the active CIRU runtime on Dankclaw
|
||||
- [[DFlash DSpark Speculative Decoding]] — DFlash/DSpark design and history
|
||||
- [[deployment]] — the deployment and update process
|
||||
- [[qwen3.8-strix-optimizations]] — llama.cpp Optimizations on this iGPU
|
||||
- [[index]] — knowledge base index
|
||||
@@ -0,0 +1,186 @@
|
||||
---
|
||||
type: reference
|
||||
title: caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
|
||||
description: caf-eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF (Q4_0_ROCMI4, embedded NextN/MTP head) served by the charlie12345/ROCmFPX llama.cpp runtime (ROCmI4 W4A4) on the Strix Halo iGPU. frank-default-large.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- rocm
|
||||
- dankclaw
|
||||
- gfx1151
|
||||
- rocmi4
|
||||
- mtp
|
||||
- iu4
|
||||
timestamp: 2026-08-25
|
||||
---
|
||||
|
||||
# caf-eemin Qwen3.8-27B ROCmI4 embedded MTP on Dankclaw
|
||||
|
||||
> **Active 2026-08-25.** `frank-default-large` serves this model. It replaced
|
||||
> the kingjones777 ROCmFP4 STRIX MTP payload on the same port and the same
|
||||
> Strix Halo iGPU, running the new charlie12345/ROCmFPX ROCmI4 W4A4 build. See
|
||||
> [[kingjones-strix-mtp]] for the retired page.
|
||||
|
||||
`frank-default-large` (port 7770) serves
|
||||
`cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (published under the public
|
||||
`eemin` namespace) from the **charlie12345/ROCmFPX** llama.cpp fork built with
|
||||
the opt-in `ROCmI4 W4A4` path. The model is `Q4_0_ROCMI4`: signed four-bit
|
||||
weights packed two per byte, quantized from the BF16 Qwen3.8-27B and carrying
|
||||
its **embedded NextN/MTP head** in the main GGUF — so there is no separate
|
||||
draft file. The native gfx1151 `v_wmma_i32_16x16x16_iu4` matrix instruction
|
||||
batch-verifies the MTP proposals for higher decode throughput.
|
||||
|
||||
## What This Is
|
||||
|
||||
**ROCmI4** is the GGUF weight format (signed 4-bit codes). **IU4** is the AMD
|
||||
WMMA instruction spelling that consumes them. **W4A4** means four-bit weights
|
||||
and four-bit activations during the accelerated matrix multiply. Ordinary
|
||||
one-token decode stays memory-bandwidth-bound (MMVQ); batched prompt processing
|
||||
and batched MTP target verification use the IU4 MMQ path. W4A4 is a lossy
|
||||
activation-quant speed path — treat it as a speed/quality tradeoff.
|
||||
|
||||
The model is a conversion/quantization of `Qwen/Qwen3.8-27B` (Unsloth lineage,
|
||||
Apache-2.0), not a retrained model. It reports architecture `qwen35`
|
||||
(M-RoPE), ~27.32B params, native 262144 context, 866 tensors, ftype 108.
|
||||
|
||||
## Model Files
|
||||
|
||||
| File | Size | SHA-256 |
|
||||
|------|------|---------|
|
||||
| `Qwen3.8-27B-Q4_0_ROCMI4.gguf` | 13.50 GiB (14,534,384,384 B) | `ad51ecbee55930ababc629c377943a7bbe50f8aafcf32340c7cdae6fa6b7bb3c` |
|
||||
| `mmproj-BF16.gguf` (vision, optional) | 0.87 GiB (931,146,432 B) | `83ee4f4f205fa514161778c41df1ea14144faa0f713510893b63c2395f5c2d53` |
|
||||
|
||||
Store under `/home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP/`. The vision
|
||||
projector is optional for text-only use.
|
||||
|
||||
## Runtime Build
|
||||
|
||||
The stock llama.cpp build does not understand `Q4_0_ROCMI4` or the qualified
|
||||
IU4/MTP path. Use charlie12345/ROCmFPX at the pinned commit, built HIP-only
|
||||
(so the combined HIP/Vulkan scheduler cannot route the ROCmI4 tensors to
|
||||
Vulkan, which has no shader for this type) with the W4A4 option on.
|
||||
|
||||
| Artifact | Path |
|
||||
|----------|------|
|
||||
| Fork + pinned commit | `/home/dank/rocmi4-qwen38/ROCmFPX` @ `c49ebdbd5c9f01ec242369f9e7f7967855f80cba` |
|
||||
| Build | `cmake -S . -B build-rocmi4-w4a4 -G Ninja -DGGML_HIP=ON -DGGML_VULKAN=OFF -DCMAKE_HIP_ARCHITECTURES=gfx1151 -DGGML_HIP_ROCMI4_W4A4=ON -DLLAMA_BUILD_SERVER=ON -DLLAMA_BUILD_WEBUI=OFF` |
|
||||
| Binary | `/home/dank/rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` (version 257, `c49ebdb`) |
|
||||
| ROCm | system `/opt/rocm-7.2.4`, amdclang 22 |
|
||||
|
||||
Build a separate directory so the exact-int8 path (the default branch) stays
|
||||
available. Rollback: disable `-DGGML_HIP_ROCMI4_W4A4=ON` and rebuild — no model
|
||||
conversion or data change needed. The build prints
|
||||
`ROCmI4 W4A4: enabled for device 0` when the accelerated path is active.
|
||||
|
||||
## Model Download
|
||||
|
||||
Downloaded from the public
|
||||
`eemin/Qwen3.8-27B-ROCmI4-MTP-GGUF` repo (the card's download section still
|
||||
names the still-private `cafonz/…`; use `eemin`). Verify against the repo's
|
||||
`checksums.sha256` after download:
|
||||
|
||||
```
|
||||
cd /home/dank/models/eemin-Qwen3.8-27B-ROCMI4-MTP
|
||||
sha256sum -c checksums.sha256
|
||||
```
|
||||
|
||||
## Launch Profile
|
||||
|
||||
The qualified MTP profile (from the model card and PR #94, verified on the
|
||||
live service): strict Qwen MTP, backend sampling, `n_max 4`, `p_min 0.60`.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[llama-server charlie12345 ROCmFPX c49ebdb<br/>ROCmI4 W4A4 enabled] --> B[load Q4_0_ROCMI4 GGUF + mmproj<br/>-ngl 999 -dev ROCm0]
|
||||
B --> C[draft-mtp on ROCm0<br/>--spec-draft-ngl all]
|
||||
C --> D[strict Qwen MTP<br/>n_max 4 / p_min 0.60<br/>backend sampling]
|
||||
D --> E[Sharp chat template<br/>reasoning deepseek]
|
||||
```
|
||||
|
||||
Key flags (see `scripts/serve-frank-large.sh`):
|
||||
|
||||
```
|
||||
--spec-type draft-mtp --spec-mtp-strict-qwen
|
||||
--spec-draft-device ROCm0 --spec-draft-ngl all
|
||||
--spec-draft-type-k f16 --spec-draft-type-v f16
|
||||
--spec-draft-n-max 4 --spec-draft-n-min 0
|
||||
--spec-draft-p-min 0.60 --spec-draft-backend-sampling
|
||||
```
|
||||
|
||||
Because the MTP head is **embedded**, no `--model-draft` is passed and the
|
||||
draft uses all layers on ROCm0. `--spec-mtp-strict-qwen` preserves the target
|
||||
path's greedy decisions; it does not remove the W4A4 activation approximation.
|
||||
|
||||
### Serving settings
|
||||
|
||||
- 65536 context (192K below the native 262144 maximum), 1 slot, f16 KV,
|
||||
context checkpoints, prompt cache. The lower limit leaves headroom for the
|
||||
model plus KV cache in the host's unified-memory budget. Concurrent requests
|
||||
queue; the server never preempts a running generation.
|
||||
- `-t 16 -tb 32`, batch 512 / ubatch 256 (the qualified W4A4 profile).
|
||||
- `--temp 0.7 --top-p 0.95`, reasoning on (`deepseek` puts `<think>` into
|
||||
`message.reasoning_content`).
|
||||
- Qwen-Sharp chat template (`--chat-template-file`), `enable_thinking` +
|
||||
`preserve_thinking` on, preserving the peg-native + deepseek shape the Pi
|
||||
agent expects.
|
||||
- Strix Halo unified memory + native gfx1151 detection: `HSA_OVERRIDE_GFX_VERSION=11.5.1`,
|
||||
`GGML_HIP_ENABLE_UNIFIED_MEMORY=1`.
|
||||
|
||||
### Service
|
||||
|
||||
- Unit: `frank-default-large.service`, port 7770
|
||||
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-default-large.env`
|
||||
|
||||
## Tuning MTP
|
||||
|
||||
MTP gains are content-dependent: predictable code, structured data, and lists
|
||||
accept longer drafts than creative prose. The active conservative profile is
|
||||
`n4 / p0.60`; the previously qualified starting point was `n16 / p0.60`.
|
||||
|
||||
## ubatch Benchmark
|
||||
|
||||
Decode throughput depends on the ubatch (per-eval token batch). Run the sweep
|
||||
on 8k contexts, 512 → 4096 in steps of 512, with a fixed batch:
|
||||
|
||||
```
|
||||
sudo systemctl stop frank-default-large
|
||||
./scripts/bench-ubatch.sh
|
||||
sudo systemctl start frank-default-large
|
||||
```
|
||||
|
||||
The script stops `frank-default-large` for a clean GPU, boots one server
|
||||
instance per ubatch on a scratch port, reads authoritative `print_timing` tok/s,
|
||||
and restarts the service. See [[frank-default-large-benchmark]].
|
||||
|
||||
## Known Issues
|
||||
|
||||
1. **gfx1151 only** for the W4A4 path. Other GPUs fall back to exact int8 MMQ.
|
||||
2. **Memory pressure:** the model is 13.5 GiB. Under `--no-mmap` with full
|
||||
offload it needs the whole tensor set at once; leave room in the 64 GiB
|
||||
unified pool (rocm-smi figure). Do not run a bench with the live service up
|
||||
or it OOMs at load.
|
||||
3. **`Q4_0_ROCMI4` + MTP flags need the ROCmFPX fork**; stock llama.cpp may
|
||||
reject the file or drop the optimized path.
|
||||
4. **W4A4 is lossy** (+~5.4% perplexity vs exact int8). Opt-in speed path.
|
||||
|
||||
## Measured Results (qualification)
|
||||
|
||||
From the model card, gfx1151, pinned build, W4A4 + strict MTP-16:
|
||||
|
||||
| Result | Value |
|
||||
|--------|-------|
|
||||
| HumanEval pilot, W4A4 IU4 (mean) | 49.40 tok/s |
|
||||
| Full HumanEval, W4A4 (mean / median) | 44.39 / 45.23 tok/s |
|
||||
| HumanEval / HumanEval+ pass@1 | 94.5% / 91.5% |
|
||||
| Non-speculative tg128 | ~13.8 tok/s |
|
||||
|
||||
Local smoke test 2026-08-25 on the iGPU: model loaded, strict-MTP active,
|
||||
draft acceptance 0.74 (mean 7.67), generation ~25 tok/s at a small output.
|
||||
The ubatch sweep above gives the production ubatch choice.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[deployment]] — deploy and update process
|
||||
- [[kingjones-strix-mtp]] — the retired ROCmFP4 predecessor on this slot
|
||||
- [[ciru-activefpx-promptforge]] — the retired v3 CIRU PromptForge page
|
||||
- [[Dankclaw System Specs]] — the box
|
||||
- [[frank-default-large-benchmark]] — backend/spec benchmark history
|
||||
@@ -0,0 +1,119 @@
|
||||
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
|
||||
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
|
||||
> [deployment.md](deployment.md) for the active configuration.
|
||||
|
||||
# frank-default-large — Backend and Spec Benchmark
|
||||
|
||||
This page records the benchmarks that chose the runtime for
|
||||
`frank-default-large` on Dankclaw. Every number came from the live
|
||||
llama.cpp build on that machine (upstream, b10133+).
|
||||
|
||||
## Summary
|
||||
|
||||
- **Backend**: **Vulkan0** (2026-08-15). At 262k ctx Vulkan decodes 48%
|
||||
faster than ROCm (5.68 vs 3.83 t/s); ROCm only wins prefill 119 vs 106
|
||||
t/s. Generation is the interactive cost, so Vulkan0.
|
||||
- **Spec (2026-08-15)**: llama.cpp `--spec-default` preset (ngram-mod,
|
||||
n_match 24 / n_min 48 / n_max 64, draftless). Replaced DSpark, which
|
||||
crashed the ROCm backend under two concurrent 16k prompts (see
|
||||
[[qwen3.8-spec-matrix]]).
|
||||
- **Parallel**: 2 slots — passes the 2x32k parallel stream test.
|
||||
- **Context**: 2 slots x 262144.
|
||||
- **KV**: `q8_0` K / `q8_0` V on GPU.
|
||||
- **Batch**: 8192 / ubatch 256.
|
||||
|
||||
> The A/B tables below are retained for history but predate the 2026-08-14
|
||||
> Qwen3.8-27B `UD-Q8_K_XL` switch. The current production spec decision,
|
||||
> the DSpark parallel crash, and the high-context ROCm-vs-Vulkan bench are
|
||||
> recorded on [[qwen3.8-spec-matrix]].
|
||||
|
||||
## 2026-08-14: Spec A/B (ROCm, production config, default sampling, 1.1k-token prompt)
|
||||
|
||||
| Spec | Generation (t/s) | Parallel-safe |
|
||||
| ---- | ---------------- | -------------- |
|
||||
| DFlash (draft-dflash, n_max 8) | ~20.2 (from earlier 4k bench) | **NO** - premature EOS |
|
||||
| MTP (draft-mtp, n_max 3) | **15.3 - 19.5** | YES |
|
||||
| MTP + ngram-mod (n_max 3) | 12.0 - 13.6 | YES |
|
||||
| MTP (draft-mtp, n_max 2) | 12.4 - 14.3 | YES |
|
||||
| ngram-mod | 6.7 - 8.7 | YES |
|
||||
|
||||
Verdict: `draft-mtp` at `n_max 3` is the winner. It nearly matches DFlash's speed, beats ngram-mod by ~2x, and does not exhibit the DFlash parallel bug. Layering ngram under MTP (12-13.6 t/s) does not pay off, the same result as layering ngram under DFlash.
|
||||
|
||||
## Incident: DFlash Premature EOS Under Parallel Load (2026-08-14)
|
||||
|
||||
Reproduced three times by the user. When a new task lands on a slot with high prompt-cache reuse (LCP similarity ~0.99, `f_keep` ~0.98) while another slot is generating, the generating slot is released ~1 second later with `finish_reason "stop"` and a truncated response. The client records it as a normal completion (no abort). Server log shows a `slot release` with `truncated = 0` and no `cancel task` line.
|
||||
|
||||
Mechanism: DFlash's draft model uses one shared draft context (`ctx_dft`) across all parallel slots. When the new task's first DFlash draft pass runs after a large prompt-reuse operation, it corrupts the generating slot's draft state, and the target spuriously samples the EOG token. Known class of bug upstream (`server : do not default to multiple slots with speculative decoding`, PR #17017; DFlash multi-slot routing fixes in the beellama.cpp fork).
|
||||
|
||||
Fix: switch away from DFlash. ngram-mod (draftless) and MTP (separate draft machinery) both pass the strict parallel test. MTP was chosen for speed.
|
||||
|
||||
## Backend Comparison (llama-bench, raw target model, no draft)
|
||||
|
||||
Both backends ran the same target model at 64k context in `llama-bench`
|
||||
(`-p 65536 -n 256`).
|
||||
|
||||
| Device | Prefill 64k (t/s) | Generation 64k (t/s) |
|
||||
| -------- | ----------------- | -------------------- |
|
||||
| ROCm0 | 220.28 | 6.99 |
|
||||
| Vulkan0 | 72.34 | 7.00 |
|
||||
|
||||
ROCm prefills 3x faster. Generation is a tie in the raw benchmark.
|
||||
|
||||
## Full-Pipeline Comparison (4k context, real served chain)
|
||||
|
||||
These used `bench-pipeline-64k.sh` driving `llama-server` through the API at
|
||||
4k context, reading the server's own `print_timing` lines.
|
||||
|
||||
| Spec | Backend | Prefill (t/s) | Generation (t/s) | Draft acceptance |
|
||||
| --------------------------- | ------- | ------------- | ---------------- | ---------------- |
|
||||
| `draft-dflash` | ROCm | 148.2 | **20.17** | 0.50 |
|
||||
| `draft-dflash,ngram-mod` | ROCm | 149.9 | 16.93 | 0.42 |
|
||||
| `ngram-mod` | ROCm | 154.0 | 8.68 | 0.57 |
|
||||
| `ngram-mod` | Vulkan | 259.6 | 6.90 | 0.17 |
|
||||
|
||||
### Interpretation
|
||||
|
||||
- DFlash alone gives the fastest generation on ROCm (20.17 t/s).
|
||||
- Layering ngram-mod lowered ROCm generation (16.93 t/s), so it is not
|
||||
layered in production. llama.cpp *can* layer DFlash + ngram
|
||||
(`--spec-type draft-dflash,ngram-mod`), but it was slower here.
|
||||
- Vulkan only works without the DFlash draft. The DFlash `ctx_other`
|
||||
setup aborts the Vulkan backend (`ggml-backend.cpp:898: pre-allocated
|
||||
tensor ... cannot run the operation (NONE)`). Vulkan ngram generation is
|
||||
also slowest (6.90 t/s) with poor acceptance (0.17).
|
||||
|
||||
**Verdict: ROCm + `draft-dflash`.**
|
||||
|
||||
## Memory Constraint at 602100 Context
|
||||
|
||||
At `3 x 200700` context the KV cache alone is ~50 GB and the model is
|
||||
29.77 GB, leaving only ~16 GB of the ~96 GB ROCm allocation for compute
|
||||
graphs. The compute graph scales with **ubatch**, so:
|
||||
|
||||
- batch 12288 / ubatch 4096 -> OOM (80.7 GB prefill buffer)
|
||||
- batch 6144 / ubatch 2048 -> OOM (40.9 GB prefill buffer)
|
||||
- batch 2048 / ubatch 2048 -> OOM (ubatch 2048 too large)
|
||||
- batch 2048 / ubatch 512 -> **fits and loads**
|
||||
|
||||
The requested batch 12288 / ubatch 4096 is not feasible at this context
|
||||
size on this hardware.
|
||||
|
||||
## Command to Reproduce
|
||||
|
||||
Raw backend bench (ROCm shown; swap `-dev` for the other):
|
||||
|
||||
```
|
||||
llama-bench -m <main>.gguf -p 65536 -n 256 -c 65536 -dev ROCm0 -r 1
|
||||
```
|
||||
|
||||
Full-pipeline bench:
|
||||
|
||||
```
|
||||
./scripts/bench-pipeline-64k.sh ROCm0 draft-dflash
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [[deployment]] — deploy and update process
|
||||
- [[DFlash DSpark Speculative Decoding]] — DFlash / DSpark / layering details
|
||||
- [[index]] — knowledge base index
|
||||
@@ -0,0 +1,181 @@
|
||||
---
|
||||
type: reference
|
||||
title: frank-default-small Benchmark — ncmoe and Batch Tuning
|
||||
description: llama-bench sweep of MoE CPU offload (ncmoe) and batch/ubatch on the RTX 3090 for Qwen3.6-35B-A3B Q6_K_P.
|
||||
tags:
|
||||
- benchmark
|
||||
- llamacpp
|
||||
- ncmoe
|
||||
- ubatch
|
||||
- dankserver
|
||||
---
|
||||
|
||||
# frank-default-small Benchmark — ncmoe and Batch/Ubatch Tuning
|
||||
|
||||
This page records a llama-bench sweep on Dankserver (RTX 3090, 24 GiB) for
|
||||
the HauhauCS Qwen3.6-35B-A3B Q6_K_P model. The sweep decided the final
|
||||
`--n-cpu-moe` and `--ubatch-size` values for `frank-default-small`.
|
||||
|
||||
Run date: 2026-08-13. Build: llama.cpp `555881ebc` (10121). CUDA, 7 threads,
|
||||
`--cache-type-k q8_0 --cache-type-v f16`, `--fit-target 0` (KV auto-sized to
|
||||
fill VRAM). Prompt 16384 tokens, generation 512 tokens, 2 repetitions.
|
||||
|
||||
## Headline Findings
|
||||
|
||||
### Generation does not depend on batch or ubatch
|
||||
|
||||
For every `ncmoe` value, generation throughput stayed flat across all
|
||||
`ubatch` values from 256 to 4096. The delta between the lowest and highest
|
||||
was under one token per second. Generation depends only on how many MoE
|
||||
expert layers stay on the GPU as `ncmoe` gets smaller.
|
||||
|
||||
| `ncmoe` | Expert layers on GPU | Generation (t/s) |
|
||||
| ------- | -------------------- | ---------------- |
|
||||
| 40 | 0 | 40.9 |
|
||||
| 38 | 2 | 42.6 |
|
||||
| 36 | 4 | 44.4 |
|
||||
| 35 | 5 | 45.5 |
|
||||
| 34 | 6 | 46.4 |
|
||||
| 32 | 8 | 48.7 |
|
||||
| 30 | 10 | 51.1 |
|
||||
|
||||
Each pair of expert layers moved to the GPU adds roughly 1.1-2.0 t/s of
|
||||
generation. Moving from `ncmoe 35` to `ncmoe 32` adds about 3.2 t/s, or 7
|
||||
percent.
|
||||
|
||||
### Prefill and graph memory scale with ubatch
|
||||
|
||||
Prefill throughput climbs with `ubatch` because a prompt splits into
|
||||
`prompt / ubatch` micro-batches. For `ncmoe 35` at 16384 prompt tokens:
|
||||
|
||||
| `ubatch` | Prefill (t/s) | Peak graph VRAM (MiB) |
|
||||
| -------- | ------------- | --------------------- |
|
||||
| 256 | 245 | 6838 |
|
||||
| 512 | 416 | 6852 |
|
||||
| 768 | 556 | 7074 |
|
||||
| 1024 | 710 | 7330 |
|
||||
| 1536 | 895 | 7846 |
|
||||
| 2048 | 1188 | 8574 |
|
||||
|
||||
The logical `--batch-size` does not matter. At a fixed `ubatch`, `batch`
|
||||
values of 2048, 4096, and 8192 gave identical prefill speed and peak VRAM.
|
||||
|
||||
## Peak VRAM by ncmoe and ubatch
|
||||
|
||||
Peak VRAM (llama-bench, single context) rises both when `ncmoe` drops
|
||||
(more experts on GPU) and when `ubatch` rises (bigger compute graph):
|
||||
|
||||
| `ncmoe` | ub 512 | ub 1024 | ub 1536 | ub 2048 |
|
||||
| ------- | ------ | ------- | ------- | ------- |
|
||||
| 38 | 4590 | 5068 | 5584 | 6086 |
|
||||
| 36 | 6098 | 6576 | 7092 | 7594 |
|
||||
| 35 | 6852 | 7330 | 7846 | 8574 |
|
||||
| 34 | 7606 | 8084 | 8600 | 9102 |
|
||||
| 32 | 9114 | 9592 | 10108 | 10610 |
|
||||
| 30 | 10622 | 11100 | 11616 | 12118 |
|
||||
|
||||
## VRAM Budget Model
|
||||
|
||||
The live service holds a fixed 700416-token KV pool (f16/f16 since
|
||||
2026-08-14; refitted from 800000) plus the mmproj vision tower plus CUDA
|
||||
overhead. The model below fits llama-bench peak VRAM into a 24 GiB
|
||||
budget:
|
||||
|
||||
```
|
||||
service VRAM = llama-bench peak + KV pool (13.68 GiB f16) + overhead (2.2 GiB)
|
||||
```
|
||||
|
||||
This model reproduces the historical OOMs. Configurations whose sum
|
||||
exceeds 24 GiB fail. For example, `ncmoe 32 - ubatch 2048` sums to about
|
||||
24.4 GiB and fails, which matches the recorded OOMs at `ncmoe 32` and
|
||||
`33`. The f16/f16 KV pool (2026-08-14) costs 20,480 bytes per cell; at
|
||||
800768 cells it is 15,640 MiB and does not fit the 24 GiB card, so the
|
||||
service runs 700416 cells (13,680 MiB).
|
||||
|
||||
## Candidate Configurations
|
||||
|
||||
| Configuration | Gen (t/s) | Prefill (t/s) | Headroom (MiB) | Fit |
|
||||
| ------------------ | --------- | ------------- | -------------- | --- |
|
||||
| ncmoe 35, ub 2048 | 45.5 | 1188 | 2035 | OK |
|
||||
| ncmoe 34, ub 2048 | 46.4 | 1221 | 1507 | OK |
|
||||
| ncmoe 34, ub 1536 | 46.5 | 914 | 2009 | OK |
|
||||
| ncmoe 32, ub 1024 | 48.7 | 766 | 1017 | OK |
|
||||
| ncmoe 32, ub 768 | 48.8 | 600 | 1273 | OK |
|
||||
| ncmoe 30, ub 1536 | 51.2 | 1009 | under | OOM |
|
||||
|
||||
## Chosen State
|
||||
|
||||
> **Live drift (2026-08-22 audit):** the deployed `frank-default-small` now runs `--n-cpu-moe 38`, `--batch-size 4096`, `--ubatch-size 2048` (ncmoe raised from 36 for stability; batch lowered). The values below are the original benchmark recommendation.
|
||||
|
||||
Prefill speed is the priority, so the state keeps the top prefill tier and
|
||||
moves one expert layer to CPU for stability:
|
||||
|
||||
```
|
||||
--n-cpu-moe 36 # 4 expert layers on GPU
|
||||
--batch-size 8192
|
||||
--ubatch-size 3072
|
||||
```
|
||||
|
||||
This gives prefill about 1427 t/s at 16384 prompt tokens, generation about
|
||||
44.4 t/s, and about 2.0 GiB of headroom at the 700416-token f16/f16 pool
|
||||
(2026-08-14 refit from 800000; the dual-f16 pool costs 13.68 GiB and
|
||||
removes the context-proportional flash-attn F16-conversion workspace that
|
||||
caused the cuBLAS crash class).
|
||||
|
||||
`ncmoe 36` came from a live crash rather than this sweep. At `ncmoe 35` with
|
||||
`ubatch 3072`, a big prefill burst failed a cuBLAS call under memory
|
||||
pressure (a `cudaErrorInvalidValue` in `ggml_cuda_mul_mat_cublas`), which
|
||||
killed the service. Raising `ncmoe` one step frees about 0.75 GiB of peak
|
||||
VRAM (the grid shows `ncmoe 36 - ubatch 3072` peak 8606 MiB versus 9360 MiB
|
||||
at `ncmoe 35`) for about 2 percent less prefill and 2 percent less decode.
|
||||
|
||||
If a crash still occurs under load, set `GGML_CUDA_FORCE_MMQ=1` to bypass
|
||||
the cuBLAS path for quantized matmuls, or drop `ubatch`, or shrink the
|
||||
context pool.
|
||||
|
||||
If decode speed matters more than prefill, swap to `ncmoe 32 - ubatch 1024`
|
||||
(generation about 48.7 t/s, prefill about 766 t/s, about 1.0 GiB headroom).
|
||||
|
||||
## Ornith Q6_K — 8K `ubatch` Sweep Without Speculation
|
||||
|
||||
Run date: 2026-08-25. This controlled sweep used the active Ornith Q6_K model
|
||||
and hybrid expert placement, but explicitly disabled speculative decoding with
|
||||
`--spec-type none`. It used one slot, an 8192-token context, a fixed logical
|
||||
`--batch-size 4096`, prompt caching disabled, and one unique approximately
|
||||
8K-token prompt per point. The production service was stopped during the
|
||||
sweep and automatically restarted afterward.
|
||||
|
||||
| `ubatch` | Prompt tokens/s | Generation tokens/s | CUDA compute reservation |
|
||||
|---:|---:|---:|---:|
|
||||
| 1024 | 1944 | 53.91 | 248 MiB |
|
||||
| 1280 | 2030 | 53.87 | 248 MiB |
|
||||
| 1536 | 2190 | 53.66 | 248 MiB |
|
||||
| 1792 | 2203 | 53.69 | 248 MiB |
|
||||
| 2048 | 2423 | 53.80 | 248 MiB |
|
||||
|
||||
Prefill improves through `ubatch=2048`, while generation is effectively flat.
|
||||
The 8K prompt sweep therefore favors `ubatch=2048` as the balanced point; a
|
||||
larger value was not tested in this corrected no-spec run.
|
||||
|
||||
The same workload with `draft-mtp` enabled produced approximately 26.8–35.2
|
||||
generation tokens/s at these `ubatch` values, with variable draft acceptance.
|
||||
The no-spec path produced approximately 53.7–53.9 generation tokens/s. This
|
||||
means MTP was a net throughput loss for this workload and configuration,
|
||||
despite accepting draft tokens. The production service remains on MTP until a
|
||||
separate decision is made; acceptance count alone is not sufficient to select
|
||||
speculation.
|
||||
|
||||
## Limitations
|
||||
|
||||
llama-bench measures raw decode and prefill. It does not run speculative
|
||||
decoding, parallel slots, or the mmproj vision tower. The live service adds
|
||||
those on top. The VRAM model uses a fixed 2.2 GiB overhead estimate; the
|
||||
real overhead varies a little with load. The Ornith sweep above used the
|
||||
OpenAI-compatible server API rather than `llama-bench`, and its throughput
|
||||
includes the server's timing instrumentation.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Dankserver System Specs]]
|
||||
- [[deployment]]
|
||||
- [[DFlash DSpark Speculative Decoding]]
|
||||
@@ -0,0 +1,202 @@
|
||||
# Frank Program Redo — 2026-08-17
|
||||
|
||||
This page records the full redo of the inference fleet on 2026-08-17.
|
||||
The pipeline-parallel split and the FastFlowLM NPU endpoint were retired.
|
||||
The old scripts moved to `old/scripts/` in this repository.
|
||||
|
||||
## New Topology
|
||||
|
||||
| Service | Host | Runtime | Model | Port |
|
||||
|---------|------|---------|-------|------|
|
||||
| frank-default-large | Dankserver | llama.cpp (patched FastMTP) | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS | 7770 |
|
||||
| frank-default-small | Dankclaw | vLLM ROCm | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 | 52625 |
|
||||
|
||||
The RTX 3090 runs the whole 27B model standalone. No RPC worker, no
|
||||
tensor split, no NPU.
|
||||
|
||||
## Models
|
||||
|
||||
### frank-default-large — HauhauCS/Qwen3.8-27B-Uncensored-HauhauCS-Aggressive-MTP-GGUF:IQ4_XS
|
||||
|
||||
- Base: Qwen3.8-27B (dense 27B, 64 layers: 48 GatedDeltaNet + 16
|
||||
gated-attention, hidden 5120, 24 Q heads / 4 KV heads, head dim 256,
|
||||
native 262144 ctx).
|
||||
- Quant: IQ4_XS (4.60 BPW, 15.71 GB). Every GGUF in the repo preserves the
|
||||
native embedded MTP/NextN head.
|
||||
- FastMTP draft sidecar: `...FastMTP-32K.gguf` (903 MB). Requires the
|
||||
patched llama.cpp build (commit `4df29be4f4c3673f428170fda944a5b19f743bb8`
|
||||
+ `HauhauCS-FastMTP-llama.cpp.patch`). Without the patch, the draft load
|
||||
fails with `expected 5120, 248320, got 5120, 32768`.
|
||||
- Spec chain: FastMTP (`--spec-type draft-mtp`, `--spec-draft-n-max 2`,
|
||||
`--spec-draft-ngl all`, `--spec-draft-p-min 0`) then ngram-mod fallback
|
||||
(48/64/24).
|
||||
|
||||
### frank-default-small — nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
|
||||
|
||||
- Base: Qwen3.6-35B-A3B (hybrid MoE, 256 experts top-8, ~3B active, 40
|
||||
layers: 10 blocks of 3 DeltaNet + 1 Attention, native 262144 ctx).
|
||||
- Quant: AMD Quark W8A8 INT8 (per-channel static weights, per-token
|
||||
dynamic activations). MoE experts stored as per-expert nn.Linear
|
||||
triplets that match vLLM's FusedMoE mapping exactly. Weights ~35 GB;
|
||||
the vision tower, router, lm_head stay BF16.
|
||||
- Served with vLLM ROCm, ngram speculative decoding, and tool calling.
|
||||
- Swap history: the initial deployment used
|
||||
`Avesed/Qwen3.6-35B-A3B-INT8-W8A8` (compressed-tensors INT8). Its tool
|
||||
calls were malformed (the model emitted `<function=get_weather(city=...)>`
|
||||
instead of the Qwen3 XML structure). The Quark quant follows the format
|
||||
cleanly; swapped in the evening of 2026-08-17.
|
||||
|
||||
## Context Math (frank-default-large, RTX 3090)
|
||||
|
||||
KV per token: 16 attention layers x 4 KV heads x 256 dim.
|
||||
K q8_0 = 1088 B/token, V f16 = 2048 B/token, total 50176 B/token.
|
||||
|
||||
Budget (MiB): 24576 total - 14982 weights - 861 draft - 1024 fixed
|
||||
(CUDA context, compute graph, scratch, draft KV) - headroom.
|
||||
|
||||
| Context | KV (MiB) | Free after load |
|
||||
|---------|----------|-----------------|
|
||||
| 114688 (112K) | 5488 | 2221 (2.17 GiB) |
|
||||
| 122880 (120K) | 5880 | 1829 (1.79 GiB) |
|
||||
| 131072 (128K) | 6272 | 1437 (1.40 GiB) |
|
||||
| 147456 (144K) | 7056 | 653 (0.64 GiB) |
|
||||
|
||||
Production choice: **131072 (128K)** — the maximum that keeps about
|
||||
1.4 GiB headroom for checkpoints. Lower `CTX_SIZE` in
|
||||
`scripts/frank-default-large.env` to 122880 for ~1.8 GiB headroom.
|
||||
|
||||
## Install Runbook
|
||||
|
||||
### Dankserver (frank-default-large)
|
||||
|
||||
1. Stop the old worker: `sudo systemctl stop frank-pp-worker`.
|
||||
2. Download the model and draft to `/home/dank/models/hauhaucs-qwen38/`.
|
||||
3. Build the patched llama.cpp:
|
||||
```
|
||||
git clone https://github.com/ggerganov/llama.cpp /home/dank/llama.cpp-fastmtp
|
||||
cd /home/dank/llama.cpp-fastmtp
|
||||
git checkout 4df29be4f4c3673f428170fda944a5b19f743bb8
|
||||
curl -L -o HauhauCS-FastMTP-llama.cpp.patch <HF patch URL>
|
||||
git apply HauhauCS-FastMTP-llama.cpp.patch
|
||||
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release \
|
||||
-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON
|
||||
cmake --build build --config Release -j$(nproc)
|
||||
```
|
||||
`GGML_CUDA_FA_ALL_QUANTS=ON` is mandatory. Without it, prefill collapses
|
||||
~12x (the q8_0 KV + flash attention falls back to a slow path;
|
||||
measured 95 t/s vs 1300+ t/s at 8K on the 3090).
|
||||
4. Symlink the unit, reload, enable, restart:
|
||||
```
|
||||
sudo ln -sf /home/dank/vLLM/scripts/frank-default-large.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload && sudo systemctl enable frank-default-large
|
||||
sudo systemctl restart frank-default-large
|
||||
```
|
||||
5. Verify: `curl -sf http://127.0.0.1:7770/health` and a chat completion.
|
||||
6. Confirm VRAM headroom with `nvidia-smi`.
|
||||
|
||||
### Dankclaw (frank-default-small)
|
||||
|
||||
1. Stop the old services: `sudo systemctl stop frank-default-large
|
||||
frank-default-small` (retired PP head and NPU endpoint).
|
||||
2. Install uv and create the venv:
|
||||
```
|
||||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||||
uv venv --python /usr/bin/python3.12 /home/dank/vllm-venv
|
||||
uv pip install --python /home/dank/vllm-venv vllm --torch-backend=auto
|
||||
```
|
||||
The resolver pulls the AMD ROCm wheel set (torch +
|
||||
amd-torch-device-gfx1151 + the vllm rocm variant) from the AMD index.
|
||||
3. Cache the model (default HF cache):
|
||||
```
|
||||
python3 -c "from huggingface_hub import snapshot_download; snapshot_download('Avesed/Qwen3.6-35B-A3B-INT8-W8A8')"
|
||||
```
|
||||
4. Install the ROCm wheel set and the flash-attention triton path:
|
||||
```
|
||||
uv pip install torch --torch-backend=auto
|
||||
uv pip install "vllm==0.27.1+rocm723" --index-url https://wheels.vllm.ai/rocm/ \
|
||||
--extra-index-url https://pypi.org/simple --index-strategy unsafe-best-match --no-build-isolation
|
||||
cp -r <ciru-venv>/site-packages/flash_attn* <vllm-venv>/site-packages/
|
||||
# add a flash_attn_2_cuda stub module (ROCm build has no compiled .so)
|
||||
```
|
||||
Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (env file). Without it the
|
||||
vision encoder falls back to torch SDPA and engine init OOMs.
|
||||
5. Symlink the unit, reload, enable, restart:
|
||||
```
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload && sudo systemctl enable frank-default-small
|
||||
sudo systemctl restart frank-default-small
|
||||
```
|
||||
6. Verify: `curl -sf http://127.0.0.1:52625/health` and a chat completion.
|
||||
|
||||
## Fallback: vLLM source build for gfx1151
|
||||
|
||||
If the prebuilt ROCm vLLM wheel cannot run on gfx1151, build from source.
|
||||
The retired CIRU package documents the proven recipe:
|
||||
`/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/scripts/build-vllm-gfx1151.sh`
|
||||
(branch `ciru/ling3-gfx1151-rocm715`, index `https://rocm.nightlies.amd.com/whl-multi-arch/`).
|
||||
The upstream starting point needed a five-line ROCm sampler fix to launch
|
||||
on gfx1151; newer vLLM releases may already include it.
|
||||
|
||||
## Configuration
|
||||
|
||||
- `models.json` (pi): two providers — `llamacpp (dankserver)` at
|
||||
http://192.168.0.235:7770/v1 and `vllm (dankclaw)` at
|
||||
http://192.168.0.143:52625/v1.
|
||||
- HF token: `~/.cache/huggingface/token` on both machines (mode 600).
|
||||
- OpenWebUI (sibling repo): retarget `OPENAI_API_BASE_URL` to the
|
||||
Dankserver endpoint.
|
||||
|
||||
## Evening Session (2026-08-17): tool calling, model swap, batch 8192
|
||||
|
||||
Three changes landed after the initial deployment:
|
||||
|
||||
### 1. Tool calling on frank-default-small (HTTP 400 fix)
|
||||
|
||||
Clients (pi) send `tool_choice: "auto"`. vLLM rejects it with HTTP 400
|
||||
unless the server enables tool calling. Fixes applied:
|
||||
|
||||
- `--enable-auto-tool-choice --tool-call-parser qwen3_xml` (the Avesed /
|
||||
Quark chat template renders the Qwen3 XML tool format, and qwen3_xml
|
||||
is the matching parser).
|
||||
- Custom chat template `/home/dank/models/qwen36-chat_template-tools.jinja`:
|
||||
derived from the repo's `chat_template.jinja` (the tokenizer_config.json
|
||||
ships no template, so vLLM would use its built-in one, which does not
|
||||
render the XML tool instructions). The custom template adds:
|
||||
- a concrete WORKED EXAMPLE built from the actual tool schema
|
||||
(name + parameter keys) — models follow concrete examples much
|
||||
better than the generic placeholder;
|
||||
- tool requests default to non-thinking (`enable_thinking` unset + tools
|
||||
present -> empty think block). Qwen3 tool calls only work reliably
|
||||
with thinking off; normal requests keep thinking.
|
||||
|
||||
Known pitfall: the earlier Avesed model still emitted malformed XML even
|
||||
with the strong template (`<function=get_weather(city=...)>`); that was
|
||||
model behavior, which motivated the swap.
|
||||
|
||||
### 2. Model swap: Avesed INT8 W8A8 -> Quark W8A8 INT8
|
||||
|
||||
`frank-default-small` now serves
|
||||
`nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8` (AMD Quark 0.11.1, ~35 GB,
|
||||
vLLM-native expert layout). The Avesed compressed-tensors quant's
|
||||
Qwen3-XML tool fidelity was poor. Verified with the Quark model: plain
|
||||
tool requests (no chat_template_kwargs) return clean
|
||||
`tool_calls: [{"name": "get_weather", "arguments": {"city": "Paris"}}]`;
|
||||
non-tool requests keep reasoning. The Avesed checkpoint stays in the HF
|
||||
cache for rollback.
|
||||
|
||||
### 3. Batch 8192
|
||||
|
||||
`MAX_NUM_BATCHED_TOKENS` lowered from 16384 to 8192 (user request).
|
||||
vLLM has no separate ubatch knob; the scheduler issues one batch per
|
||||
step up to this token cap.
|
||||
|
||||
### 4. Prefix caching enabled
|
||||
|
||||
Added `--enable-prefix-caching`. The model is hybrid (GatedDeltaNet +
|
||||
full attention); vLLM keeps prefix caching **opt-in** for hybrid models
|
||||
(`default_prefix_caching = is_prefix_caching_supported and not
|
||||
is_hybrid`), so it was off by default. Enabling it reuses the KV of
|
||||
repeated prompt prefixes. Measured on a 2023-token prefix (2026-08-17
|
||||
evening): 4.6s cold -> 1.1s cached (~4x TTFT), output unchanged. This is
|
||||
the main TTFT lever for pi, which re-sends the same system prompt + tool
|
||||
schema every turn. Toggled via `ENABLE_PREFIX_CACHING` in the env file.
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
type: research
|
||||
title: FreeToken Benchmark Sweep — Dankserver RTX 3090
|
||||
description: KV pool vs decode speed sweep across bs1/bs3/bs4/bs6 on Dankserver with Qwen3.6-35B-A3B-FP8 via FreeToken v0.1.2.
|
||||
tags:
|
||||
- freetoken
|
||||
- benchmark
|
||||
- dankserver
|
||||
- frank-default-small
|
||||
- qwen3.6
|
||||
timestamp: 2026-09-02
|
||||
---
|
||||
|
||||
# FreeToken Benchmark Sweep — Dankserver RTX 3090
|
||||
|
||||
Full sweep of KV pool size vs decode speed at batch sizes 1, 3, 4, and 6.
|
||||
Measured 2026-09-02 with FreeToken v0.1.2 serving `Qwen/Qwen3.6-35B-A3B-FP8`
|
||||
on Dankserver (RTX 3090 24 GB, 60 GB RAM, driver 595.84).
|
||||
|
||||
## Methodology
|
||||
|
||||
- Each point: restart at low KV (8k), live-rebuild upward via
|
||||
`ft ctl cache --kv N --moe M` (auto-computed MoE slot count).
|
||||
- Warmup: 2–3 iterations at 64 tokens (fills LRU expert cache).
|
||||
- Measured: 2–3 iterations at 128 tokens, `temperature=0`, `ignore_eos=true`.
|
||||
- Prompt: "Count from 1 to 100." (short — isolates decode bandwidth).
|
||||
- Per-token KV cost: ~20 KiB (10 full-attention layers × 2 KV heads ×
|
||||
256 dim × bf16). Per-slot MoE cost: ~3 MiB.
|
||||
- GPU budget: ~17.6 GiB after weights + fixed caches.
|
||||
|
||||
## bs1 vs bs3 vs bs4 vs bs6
|
||||
|
||||
| KV pool | Residency | bs1 | bs3 agg | bs3/req | bs4 agg | bs4/req | bs6 agg | bs6/req |
|
||||
|---------|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
|
||||
| 8,192 | ~89% | 56.7 | 108.8 | 36.3 | 150.6 | 37.7 | 132.1 | 22.0 |
|
||||
| 65,536 | ~83% | ~52 | 106.1 | 35.4 | 148.0 | 37.0 | 131.8 | 22.0 |
|
||||
| 131,072 | ~76% | ~48 | 99.9 | 33.3 | 144.4 | 36.1 | 131.8 | 22.0 |
|
||||
| 262,144 | ~62% | ~48 | 85.3 | 28.4 | 131.3 | 32.8 | 130.5 | 21.8 |
|
||||
| 400,000 | ~47% | ~42 | 73.3 | 24.4 | 116.5 | 29.1 | 161.1 | 26.9 |
|
||||
| 524,288 | ~30% | 34.8 | 59.8 | 19.9 | 102.2 | 25.5 | 150.6 | 25.1 |
|
||||
| 655,360 | ~19% | 30.4 | 42.8 | 14.3 | 86.9 | 21.7 | 131.5 | 21.9 |
|
||||
| 700,000 | ~15% | 28.3 | 38.5 | 12.8 | 83.0 | 20.8 | 121.1 | 20.2 |
|
||||
|
||||
## Key Findings
|
||||
|
||||
### No cliff — gentle slope
|
||||
|
||||
The LRU expert cache absorbs residency reductions gracefully. Even at
|
||||
5% residency (544 slots), bs1 decode is 25.2 tok/s — 44% of baseline.
|
||||
|
||||
### Free zone up to ~262k
|
||||
|
||||
Decode speed is flat from 8k through 262k at bs1 and bs4. Giving up
|
||||
expert residency (89% → 62%) costs nothing because the LRU working set
|
||||
fits in fewer slots.
|
||||
|
||||
### bs4 is the sweet spot
|
||||
|
||||
bs4 dominates at every high-residency operating point. Per-request
|
||||
speed at bs4 (37.7 tok/s at 8k) is *faster* than bs3 (36.3) because 4
|
||||
concurrent requests better fill the PCIe expert-streaming pipeline.
|
||||
|
||||
### bs6 crossover at ~400k
|
||||
|
||||
At high residency, bs6 aggregate (132) < bs4 aggregate (150) — the
|
||||
bottleneck is GPU compute on attention layers, not PCIe. At low
|
||||
residency (400k+), the bottleneck flips to PCIe expert streaming and
|
||||
bs6 wins (161 vs 116 at 400k) because more concurrency fills the
|
||||
pipeline.
|
||||
|
||||
### bs3 is strictly worse than bs4
|
||||
|
||||
Fewer concurrent users leave PCIe idle gaps. At every point past 8k,
|
||||
bs3 per-request is slower than bs4 per-request. Don't run bs3 — set
|
||||
`--max-running-requests 4` regardless of actual user count.
|
||||
|
||||
### 800k doesn't fit
|
||||
|
||||
4 × 200k = 800k exceeds the GPU budget even at minimum MoE (256 slots).
|
||||
The absolute ceiling is ~750k tokens. With radix prefix sharing,
|
||||
effective capacity exceeds naive `pool ÷ concurrency`.
|
||||
|
||||
## Comparison: old llama.cpp service
|
||||
|
||||
The old Tiel-Coder llama.cpp service ran 200,700 total context across
|
||||
3 parallel slots (66,900/slot) at ~99 tok/s aggregate at concurrency 4.
|
||||
FreeToken at 131k KV (bs4) does 144 tok/s aggregate — 45% faster — with
|
||||
the caveat that it serves base Qwen3.6-35B-A3B weights, not the
|
||||
Tiel-Coder finetune.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[freetoken-frank-default-small]] — production runbook
|
||||
- [[dealign-ornith-crack]] — the prior llama.cpp deployment (retired)
|
||||
@@ -0,0 +1,127 @@
|
||||
---
|
||||
type: configuration
|
||||
title: FreeToken frank-default-small Runbook
|
||||
description: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 as frank-default-small on Dankserver (RTX 3090).
|
||||
tags:
|
||||
- freetoken
|
||||
- dankserver
|
||||
- frank-default-small
|
||||
- qwen3.6
|
||||
- moe
|
||||
- fp8
|
||||
timestamp: 2026-09-02
|
||||
---
|
||||
|
||||
# FreeToken frank-default-small Runbook
|
||||
|
||||
The `frank-default-small` service on Dankserver is served by
|
||||
**FreeToken v0.1.2** — an edge-native MoE serving engine that keeps
|
||||
expert weights in host RAM with an LRU GPU cache. It replaced the
|
||||
llama.cpp Tiel-Coder-35B-A3B (UD-Q6_K_XL) deployment on 2026-09-02.
|
||||
|
||||
## Model
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| HuggingFace | `Qwen/Qwen3.6-35B-A3B-FP8` |
|
||||
| Architecture | `qwen3_5_moe` (Qwen3_5MoeForConditionalGeneration) |
|
||||
| Quant | modelopt block-fp8 (128×128 `weight_block_size`) |
|
||||
| Size on disk | 35 GB |
|
||||
| Path | `/home/dank/models/qwen36-35b-a3b-fp8` |
|
||||
| Layers | 40 (10 full-attention + 30 GDN/linear-attention) |
|
||||
| Experts | 256 per layer, 8 used per token |
|
||||
| KV heads | 2 × 256 dim |
|
||||
| Context | 262,144 (model max) |
|
||||
|
||||
**Important:** This serves base Qwen3.6-35B-A3B weights, not the
|
||||
Tiel-Coder finetune that was previously served via llama.cpp.
|
||||
|
||||
## Runtime
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| Engine | FreeToken v0.1.2 (`ft` CLI) |
|
||||
| Venv | `/home/dank/freetoken/.venv` (uv-managed CPython 3.12) |
|
||||
| Dependencies | torch 2.11.0+cu130, flashinfer 0.6.17, apache-tvm-ffi |
|
||||
| CUDA toolchain | pip-installed nvidia-cuda-nvcc/crt/nvvm/runtime 13.2.86 |
|
||||
| API | OpenAI-compatible (`/v1/chat/completions`, `/v1/models`) |
|
||||
|
||||
## Pool Configuration
|
||||
|
||||
The GPU budget (~17.6 GiB after weights + fixed caches) is split
|
||||
MoE-first between expert residency and KV pages:
|
||||
|
||||
| Pool | Size | VRAM | Notes |
|
||||
|------|------|------|-------|
|
||||
| MoE cache | 2,048 slots (20%) | 6.0 GiB | LRU expert cache, out of 10,240 total |
|
||||
| KV cache | 453,005 tokens | 8.6 GiB | ~20 KiB/token (10 attn layers only) |
|
||||
| Mamba state | 24 slots | 1.4 GiB | Per-sequence GDN state (caps concurrency) |
|
||||
|
||||
### Live Resize
|
||||
|
||||
KV and MoE can be re-partitioned without restart:
|
||||
|
||||
```bash
|
||||
ft ctl --base-url http://127.0.0.1:7770 cache --kv N --moe M --wait 300
|
||||
```
|
||||
|
||||
- **Growing KV** (shrinking MoE): works live.
|
||||
- **Shrinking KV** (growing MoE): requires restart (old + new pools
|
||||
cannot coexist in VRAM during the swap → OOM).
|
||||
- The fit check refuses overcommit: if the plan doesn't fit, the old
|
||||
cache is retained and serving continues.
|
||||
|
||||
### Benchmark Results (2026-09-02)
|
||||
|
||||
See [[freetoken-benchmark-sweep]] for the full bs1/bs3/bs4/bs6 sweep
|
||||
across KV pool sizes 8k–700k.
|
||||
|
||||
At the production config (20% residency, bs4):
|
||||
|
||||
| Metric | Value |
|
||||
|--------|-------|
|
||||
| bs1 decode | ~30 tok/s |
|
||||
| bs4 aggregate | ~87 tok/s |
|
||||
| bs4 per-request | ~22 tok/s |
|
||||
| Prefill (4k) | ~1,130 tok/s |
|
||||
|
||||
## Quantization Compatibility
|
||||
|
||||
FreeToken v0.1.2 supports these safetensor quants for `qwen3_5_moe`:
|
||||
|
||||
| Format | Detection | Works |
|
||||
|--------|-----------|-------|
|
||||
| BF16 (no quant config) | default | ✅ (experts too large for 60 GB RAM) |
|
||||
| modelopt block-fp8 (128×128) | `quant_method: "fp8"` + `weight_block_size` | ✅ (current) |
|
||||
| NVFP4 (modelopt) | `quant_algo: "NVFP4"` | ✅ (best for VRAM) |
|
||||
| NVFP4 (compressed-tensors) | 4-bit float, group_size=16, strategy=tensor_group | ✅ |
|
||||
| Per-tensor FP8 (modelopt) | scalar `weight_scale` | ✅ |
|
||||
| Per-channel FP8 (compressed-tensors) | `weight_scale: [O, 1]` | ❌ loader bug |
|
||||
| Quark W8A8 INT8 | per-channel scales | ❌ loader bug |
|
||||
| GGUF | only `gemma4` arch | ❌ for qwen3_5_moe |
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `scripts/serve-frank-small.sh` | Serve script (launches `ft serve`) |
|
||||
| `scripts/frank-default-small.env` | Tunables (model path, pool sizes) |
|
||||
| `scripts/frank-default-small.service` | Systemd unit |
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Model identity**: serves base Qwen3.6-35B-A3B, not Tiel-Coder finetune.
|
||||
- **JIT compilation**: first startup compiles flashinfer/tvm-ffi kernels
|
||||
(~5–15 min cold). Cached afterward in `~/.cache/flashinfer` and
|
||||
`~/.cache/tvm-ffi`.
|
||||
- **CUDA toolchain**: driver-only install (no system CUDA toolkit). The
|
||||
venv provides pip-installed nvcc 13.2.86 for JIT compilation. All
|
||||
nvidia-cu13 packages must be at the same version.
|
||||
- **Env var collisions**: never name tunables `TEMP`, `HOST`, or
|
||||
`GPU_MAX_HEAP_SIZE` (HIP runtime reads these).
|
||||
|
||||
## Citations
|
||||
|
||||
- FreeToken paper: arXiv 2608.16157
|
||||
- FreeToken PyPI: `freetoken[accel]` v0.1.2
|
||||
- HuggingFace model: `Qwen/Qwen3.6-35B-A3B-FP8`
|
||||
@@ -0,0 +1,262 @@
|
||||
---
|
||||
type: reference
|
||||
title: ik_llama.cpp on Dankserver (Ornith Q8_0)
|
||||
description: ikawrakow/ik_llama.cpp replaces upstream llama.cpp for frank-default-small on the RTX 3090. Q8_0 model, KV K default f16 / V q6_0 with Hadamard on V only (iq5_k is not a KV-cache type in this build; q8_KV crashes on this arch), split-mode none, speculative decoding disabled, 3 parallel slots x 163840 ctx, the Qwen-Sharp template rendered with --jinja, and explicit CPU-expert offload. Includes the mandatory CUDA/IQK FA_ALL_QUANTS build flags.
|
||||
tags:
|
||||
- ikllama
|
||||
- llamacpp
|
||||
- cuda
|
||||
- dankserver
|
||||
- moe
|
||||
- ornith
|
||||
- kv-cache
|
||||
- hadamard
|
||||
timestamp: 2026-08-26
|
||||
---
|
||||
|
||||
# ik_llama.cpp on Dankserver (Ornith-1.5-35B-A3B CRACK)
|
||||
|
||||
> **Active 2026-08-26.** `frank-default-small` switched from upstream
|
||||
> llama.cpp (b10121) to `ikawrakow/ik_llama.cpp` (commit `b166e269`) on the
|
||||
> RTX 3090. This page documents the build, the mandatory FA build flags, and
|
||||
> the MoE / CPU-offload runtime arguments.
|
||||
|
||||
The model is `dealignai/Ornith-1.5-35B-A3B-UNCENSORED` (CRACK **Q8_0**), 35B-total /
|
||||
3B-active MoE-dense hybrid with an MTP head in the GGUF (mmproj f16 vision
|
||||
encoder). Served as **three parallel 160000-context slots** (`PARALLEL=3`).
|
||||
Speculative decoding is **disabled**. See [[dealign-ornith-crack]] for the
|
||||
model runbook.
|
||||
|
||||
## Why ik_llama.cpp
|
||||
|
||||
ik_llama.cpp is ikawrakow's llama.cpp fork with heavy CPU + CUDA
|
||||
optimizations, IQK/AVX-512 kernels, `--fit` auto-offload, and fast q8_KV
|
||||
cache. Dankserver is a CUDA (RTX 3090) + CPU-MoE-offload box, which is
|
||||
exactly the supported hybrid the fork targets. Only CPU and CUDA backends
|
||||
are fully functional; ROCm / Vulkan / Metal are unsupported upstream.
|
||||
|
||||
## Build recipe (CUDA 13.2 via staged pip toolkit)
|
||||
|
||||
The system CUDA toolkit on Dankserver is **removed** (`nvidia-cuda-toolkit`
|
||||
is in `rc` state; no `/usr/local/cuda*`, no `/usr/bin/nvcc`). The CUDA
|
||||
13.2 toolkit is installed via **pip** inside the vllm venv at
|
||||
`/home/dank/vllm/.venv/lib/python3.12/site-packages/nvidia/cu13/`. A staged,
|
||||
self-contained toolchain at `/home/dank/cuda13-stage` adds the unversioned
|
||||
`.so` aliases and static libs CMake needs.
|
||||
|
||||
```bash
|
||||
# 1. Clone a pinned ik_llama.cpp
|
||||
git clone https://github.com/ikawrakow/ik_llama.cpp.git /home/dank/ik_llama.cpp
|
||||
cd /home/dank/ik_llama.cpp
|
||||
git checkout b166e269 # the version this deployment was built from
|
||||
|
||||
# 2. Configure (stage already symlinks nvcc/include/lib for cu13)
|
||||
cmake -B build \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DGGML_CUDA=ON \
|
||||
-DGGML_CUDA_FA_ALL_QUANTS=ON \
|
||||
-DGGML_IQK_FA_ALL_QUANTS=ON \
|
||||
-DCMAKE_CUDA_ARCHITECTURES=86 \
|
||||
-DCMAKE_CUDA_COMPILER=/home/dank/cuda13-stage/bin/nvcc \
|
||||
-DCUDAToolkit_ROOT=/home/dank/cuda13-stage \
|
||||
-DGGML_NATIVE=ON
|
||||
|
||||
# 3. Build
|
||||
cmake --build build --config Release -j 8
|
||||
```
|
||||
|
||||
### ⚠️ MANDATORY: always include the FA all-quants flags
|
||||
|
||||
> **Rule.** Any ik_llama.cpp build that serves a quantized KV cache
|
||||
> (q8_KV / q8_0 / Hadamard / any non-f16 KV) **MUST** be compiled with
|
||||
> BOTH:
|
||||
> - `-DGGML_CUDA_FA_ALL_QUANTS=ON` — compiles every K/V cache quant combo
|
||||
> into the **CUDA** FlashAttention kernels. Without it, CUDA FA only ships
|
||||
> a subset and **errors at runtime** telling you to recompile when a combo
|
||||
> (e.g. q8_KV + Hadamard V-cache) isn't present. The RTX 3090 keeps KV on
|
||||
> the GPU, so this is the flag that matters here.
|
||||
> - `-DGGML_IQK_FA_ALL_QUANTS=ON` — compiles all quant types (Q4_0/Q4_1/
|
||||
> IQ4_NL on top of the always-on F16/Q8_0/Q6_0/BF16) into the **CPU/IQK**
|
||||
> FA kernels. Relevant because the MoE experts run on the CPU; the KV
|
||||
> cache that backs CPU-side attention uses the IQK FA path.
|
||||
>
|
||||
> These are the old "asymmetrical quants" compile-time gates. On a CUDA
|
||||
> host, omitting `GGML_CUDA_FA_ALL_QUANTS` is the #1 way to get a
|
||||
> "recompile with GGML_CUDA_FA_ALL_QUANTS" crash at load time.
|
||||
|
||||
`GGML_NATIVE=ON` lets GCC target the host (the Ryzen 7 7700X is Zen4, so
|
||||
`-march=native` enables the AVX512 / VNNI IQK kernels automatically —
|
||||
verifiable with `system_info: AVX512_VNNI = 1` in the boot log).
|
||||
|
||||
### CUDA stage note
|
||||
|
||||
`/home/dank/cuda13-stage` is the pip `cu13` toolkit plus the symlinks CMake's
|
||||
(CUDAToolkit) finder needs (`libcudart.so`, `libcublas.so`, `libcublasLt.so`
|
||||
and static `libcudadevrt.a`/`libcudart_static.a`). It also works around the
|
||||
CCCL CUDA version skew by defining
|
||||
`-DCCCL_DISABLE_CTK_COMPATIBILITY_CHECK` in `CMAKE_CUDA_FLAGS` (the pip
|
||||
package ships nvcc 13.2.78 but reports `CUDART_VERSION 13000`, which trips
|
||||
a strict header lint). Do not re-run configure without that flag or you get
|
||||
a hard compile error.
|
||||
|
||||
## Runtime configuration
|
||||
|
||||
Files: `scripts/serve-frank-small.sh`, `scripts/frank-default-small.env`,
|
||||
`scripts/frank-default-small.service` in the repo. The service just calls
|
||||
the serve script; the env file holds all tunables.
|
||||
|
||||
### KV cache: K default f16 / V q6_0 + Hadamard (V only)
|
||||
|
||||
- `CACHE_TYPE_K=f16` — K cache at the default full-precision f16
|
||||
(passes `--cache-type-k f16`). No Hadamard on K.
|
||||
- `CACHE_TYPE_V=q6_0` — V cache at 6-bit k-quant
|
||||
(passes `--cache-type-v q6_0`).
|
||||
- `V_CACHE_HADAMARD=1` — `--v-cache-hadamard` on V only. The Hadamard
|
||||
transform preserves V quality at 6-bit; K is full-precision f16 and does
|
||||
not need it (`K_CACHE_HADAMARD=0`). Implemented for both CUDA
|
||||
(`ggml-cuda/hadamard.cu`) and CPU (IQK) paths. Flash-attention is **on
|
||||
by default**.
|
||||
|
||||
> **KV cache type history on this build (`b166e269`):**
|
||||
> - `iq5_k` is **not** a KV-cache type — the cache whitelist is
|
||||
> `f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1/q6_0/q8_KV`; `iq5_*` only
|
||||
> exist as weight quants. `--cache-type-k iq5_k` throws `Invalid cache
|
||||
> type`.
|
||||
> - `q8_KV` **crashes** with a `ggml view` size assert
|
||||
> (`ggml.c:5427: data_size + view_offs > ggml_nbytes(view_src)`) on this
|
||||
> QWEN3MOE/MTP + multimodal arch — independent of PARALLEL/speculation.
|
||||
> - The deployment settled on K at default **f16** (full precision, no
|
||||
> Hadamard) with V at **q6_0 + Hadamard**.
|
||||
|
||||
With 3 parallel slots at 163840 context the KV cache is large; keeping K at
|
||||
full precision f16 while quantizing only V (with Hadamard) is the chosen
|
||||
budget split. The GPU-side expert set stays reduced to fit (see the expert
|
||||
section).
|
||||
|
||||
### Split mode: none (single GPU)
|
||||
|
||||
- `SPLIT_MODE=none` → `--split-mode none` (the default). Dankserver has a
|
||||
single RTX 3090 yet ik_llama rejects `--split-mode graph` on a box with a
|
||||
single device:
|
||||
|
||||
> `llama error loading model: It is not possible to use split mode 'graph'
|
||||
> with less than 2 devices`
|
||||
|
||||
So `graph` is **not** a harmless no-op on one GPU — it is a hard load
|
||||
error (this bit us on the first deploy and loop the service). It also still
|
||||
works with the `--override-tensor` expert pins.
|
||||
|
||||
> **Lesson.** ik_llama `--split-mode graph` requires ≥2 devices. On a
|
||||
> single-GPU host you must use `none` (or `layer`, which also needs >1).
|
||||
> Do not assume `--split-mode graph` degrades gracefully on one GPU.
|
||||
|
||||
### Expert offload (MoE: CPU with a partial GPU slice)
|
||||
|
||||
The expert-offload pattern from upstream llama.cpp carries over: the
|
||||
`--override-tensor` regex pins the sparse expert (`ffn_*_exps`) tensors to
|
||||
the CPU buft, leaving the output-adjacent blocks 35-40 on CUDA. On the
|
||||
Q8_0 model each ffn expert tensor is **272 MiB** (~4 tensor/block ≈ 1088
|
||||
MiB/block). With `PARALLEL=3` the KV cache tripled, so most expert blocks
|
||||
are pinned to CPU; only blocks **35-40** (output-adjacent, latency- and
|
||||
quality-relevant) stay on CUDA. Full-attention blocks 3,7,11,15,19,23 have
|
||||
no exps and are untouched by the regex.
|
||||
|
||||
- `OVERRIDE_EXPERT` — the layer-bounded regex listing the blocks whose
|
||||
experts stay on CPU: `(0|1|2|4|5|6|8|9|10|12|13|14|16|17|18|20|21|22|
|
||||
24|25|26|27|28|29|30|31|32|33|34)`. Blocks 35-40 (and the always-on
|
||||
full-attention set) are left off the list so their experts load to CUDA.
|
||||
- `OVERRIDE_BUFT=CPU`.
|
||||
- `N_GPU_LAYERS=999` — ik_llama `-ngl` / `--gpu-layers` only accepts an
|
||||
integer (the literal `all` is rejected). 999 means "load as many tensor
|
||||
layers as possible", while the expert override still pins the listed exps
|
||||
to CPU.
|
||||
|
||||
> **Fused MoE** (`fused_moe`, `fused_up_gate`, `fused_mmad`) is **on by
|
||||
> default** in ik_llama — there is no positive flag; only `-no-fmoe`
|
||||
> disables it. `only_active_experts` is also on by default. Confirmed in the
|
||||
> boot log for the running service.
|
||||
|
||||
To move more experts back to the GPU later, drop more block numbers from
|
||||
`OVERRIDE_EXPERT` (each block is ~1088 MiB on Q8_0; verify `nvidia-smi`
|
||||
headroom first and keep ~2-3 GiB margin for KV/compute). With the 3-slot KV
|
||||
cache there is little headroom, so the default keeps only 35-40 on GPU.
|
||||
|
||||
### Qwen-Sharp chat template via --jinja
|
||||
|
||||
`--chat-template-file` alone only stores the template text; ik_llama **does
|
||||
not** render a custom template as real Jinja unless `--jinja` is also given
|
||||
(`inputs.use_jinja ? common_chat_templates_apply_jinja : <non-jinja
|
||||
fallback>`). Without `--jinja` it warns it only supports "commonly used
|
||||
templates" and silently uses built-in handlers, dropping the Qwen-Sharp
|
||||
template's thinking / image / tool formatting. So the serve script passes
|
||||
both `--chat-template-file` and `--jinja` (gated on `USE_JINJA`, set to `1`).
|
||||
|
||||
### Speculative decoding: disabled (3 parallel slots)
|
||||
|
||||
`SPEC_TYPE=none` (passed as `--spec-type none`). Speculative decoding is
|
||||
intentionally disabled because ik_llama **limits any speculative stage to a
|
||||
a single server slot**: with `--parallel 3` a configured spec stage
|
||||
hard-exits at load with
|
||||
|
||||
> `ERR [load_model] Speculative decoding is currently limited to a single
|
||||
> server slot (-np 1).` / `n_parallel=3`
|
||||
|
||||
This service keeps **3 parallel slots** (one per user family via
|
||||
`PARALLEL=3`), so speculative decoding cannot be used here. MTP stays
|
||||
disabled. The model is a low-parameter MoE (3B active of 35B) and does not
|
||||
need the speculation it would cost a slot for.
|
||||
|
||||
> **Lesson.** Do not combine speculative decoding with `--parallel > 1` in
|
||||
> ik_llama. Either drop to `-np 1` (one slot) and keep a spec stage, or keep
|
||||
> multiple slots and set `SPEC_TYPE=none`. For a multi-user service the
|
||||
> extra slots win.
|
||||
|
||||
### Flash-attention + sampling
|
||||
|
||||
Flash-attention is **on by default** in ik_llama (`flash_attn=true`), so no
|
||||
`-fa`/`--flash-attn` flag is needed. Sampling, thread, batch, context, and
|
||||
checkpoint tunables (`TEMPERATURE`, `TOP_K`, `TOP_P`, `MIN_P`, thread,
|
||||
`BATCH_SIZE`, `UBATCH_SIZE`, `CTX_SIZE`, `PARALLEL`, `CTX_CHECKPOINTS`,
|
||||
`NO_MMAP`) are unchanged and pass straight through.
|
||||
|
||||
## MoE / hybrid-arg reference (not in upstream llama.cpp)
|
||||
|
||||
Several ik_llama knobs matter for a CPU-MoE offload host:
|
||||
|
||||
| flag | meaning | our use |
|
||||
|------|---------|---------|
|
||||
| `--split-mode graph` | split tensors+graph across devices; strong for MoE; **requires 2+ devices** | single 3090 → use `none`
|
||||
| `-ooae` / `--offload-only-active-experts` | default ON: copy **only active** experts RAM→VRAM | keep ON (huge for A3B: few experts active) |
|
||||
| `-grt q8_0` | reduce type for inter-GPU data exchange | not needed (single GPU) |
|
||||
| `--fit` | auto-offload to fit VRAM | conflicts with `--override-tensor`; do not use together (runtime error) |
|
||||
| `-n-cpu-moe N` / `--cpu-moe` | keep MoE weights of first N / all layers on CPU | also conflicts with `--fit`; our `--override-tensor` is finer-grained |
|
||||
| `--fit-margin` | VRAM safety margin for `--fit` | only relevant with `--fit` |
|
||||
|
||||
The current deployment uses `--override-tensor` (not `--fit` or
|
||||
`--n-cpu-moe`) for the expert offload, so the `--fit`-conflict rules do not
|
||||
apply.
|
||||
|
||||
## Rollback
|
||||
|
||||
The previous runtime `upstream llama.cpp b10121` and its binary
|
||||
(`/home/dank/llama.cpp/build/bin/llama-server`) are **left untouched**. To
|
||||
rollback to the old stack, change `LLAMA_SERVER=` in
|
||||
`scripts/frank-default-small.env` back to that path and
|
||||
`CACHE_TYPE_K/V` back to `f16` with Hadamard off, and re-enable the ngram
|
||||
spec / `--spec-type` flags. The model file is not changed either way.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[dealign-ornith-crack]] — the Ornith Q6_K model run
|
||||
- [[Dankserver System Specs]] — the host
|
||||
- [[deployment]] — service-scoped git deploy
|
||||
|
||||
## Citations
|
||||
|
||||
- [ikawrakow/ik_llama.cpp README](https://github.com/ikawrakow/ik_llama.cpp) —
|
||||
MoE offload guidance and the `--split-mode graph` + CPU-offload warning
|
||||
- [ik_llama.cpp serverAppConfigs-docs/build.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/build.md) —
|
||||
build flags incl. `GGML_CUDA_FA_ALL_QUANTS` and `GGML_IQK_FA_ALL_QUANTS`
|
||||
- [ik_llama.cpp serverAppConfigs-docs/parameters.md](https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/parameters.md) —
|
||||
q8_KV (PR 208), K/V Hadamard options, `-cuda graphs=0` workaround, and
|
||||
`--split-mode graph` MoE note (PR 1080)
|
||||
@@ -0,0 +1,131 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on two local machines
|
||||
(2026-09-02): FreeToken v0.1.2 on Dankserver (frank-default-small,
|
||||
Qwen3.6-35B-A3B-FP8) and ROCmFPX llama.cpp on Dankclaw
|
||||
(frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both
|
||||
services run the Qwen-Sharp chat template.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
> `serverAppConfigs-docs/` root to `serverAppConfigs-docs/LLM/`. The fleet scripts and units now live in the
|
||||
> repo's `scripts/` directory.
|
||||
|
||||
## Deployment Topology
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph Dankserver[NVIDIA Dankserver - Ubuntu 26.04]
|
||||
A[RTX 3090 24GB + 60GB RAM]
|
||||
B2[FreeToken v0.1.2
|
||||
frank-default-small, port 7770]
|
||||
C[Qwen3.6-35B-A3B-FP8
|
||||
block-fp8, 20% MoE residency, 453k KV / 4 streams]
|
||||
A --> B2 --> C
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[gfx1151 iGPU 64 GiB unified memory]
|
||||
E[charlie12345/ROCmFPX build-kairic (v81)
|
||||
frank-default-large, port 7770]
|
||||
F[Qwen3.8 Uncensored Q6 ROCmFPX AGENT
|
||||
MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
E2 --> E --> F
|
||||
end
|
||||
|
||||
CLI[LLM Clients / API] -->|HTTP :7770| B2
|
||||
CLI -->|HTTP :7770| E
|
||||
```
|
||||
|
||||
## Servers
|
||||
|
||||
| Server | GPU | VRAM / Memory | OS |
|
||||
| --------------------------------------- | ----------------------------- | -------------------- | ------------ |
|
||||
| [[Dankserver System Specs\|Dankserver]] | NVIDIA RTX 3090 | 24 GB VRAM, 62 GB RAM | Ubuntu 26.04 |
|
||||
| [[Dankclaw System Specs\|Dankclaw]] | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 |
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[freetoken-frank-default-small]] — the active frank-default-small
|
||||
runbook: FreeToken v0.1.2 serving Qwen3.6-35B-A3B-FP8 on Dankserver
|
||||
(2026-09-02)
|
||||
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[kingjones-strix-rocmfp4]] — active frank-default-large runbook:
|
||||
kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
|
||||
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, STRIX tier
|
||||
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
|
||||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
|
||||
on Dankclaw (retired)
|
||||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-default-small, retired 2026-09-02)
|
||||
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
|
||||
frank-default-small (2026-08-26): q8_0 + Hadamard KV cache,
|
||||
split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
|
||||
- [[kingjones-strix-mtp]] — the former Q6 AGENT runbook (restored 2026-09-01
|
||||
after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
|
||||
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
|
||||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
||||
(deployed and reverted 2026-08-22)
|
||||
- [[Dankserver System Specs]] — the RTX 3090 box (frank-default-small)
|
||||
- [[Dankclaw System Specs]] — the CIRU ROCmFPX box (frank-default-large)
|
||||
|
||||
## CIRU Reference (Dankclaw)
|
||||
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain,
|
||||
and build history (on Dankclaw; the slot now serves the kingjones
|
||||
ROCmFP4 model, see [[kingjones-strix-mtp]])
|
||||
- [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary
|
||||
identity, and experiment summary (retired 2026-08-22)
|
||||
|
||||
## DFlash2 Reference (archived)
|
||||
|
||||
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment
|
||||
(2026-08-21)
|
||||
|
||||
## vLLM Reference (historical)
|
||||
|
||||
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
|
||||
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
|
||||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
|
||||
notes and sizing (the historical family; frank-default-small now serves
|
||||
Ornith-1.5-35B-A3B, see [[dealign-ornith-crack]])
|
||||
|
||||
## Memory Management
|
||||
|
||||
- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
|
||||
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
|
||||
- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
|
||||
|
||||
## Retired Topology (2026-08-17 redo)
|
||||
|
||||
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
|
||||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
|
||||
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
|
||||
- [[frank-default-large-benchmark]] — backend/spec benchmark behind the old head
|
||||
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
|
||||
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
|
||||
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
|
||||
|
||||
## Pipeline Parallelism (archive)
|
||||
|
||||
- [[Pipeline Parallelism/index]] — index of multi-node inference research. The PP
|
||||
pair is retired. These pages are research history.
|
||||
|
||||
## Runtime Research (llama.cpp family)
|
||||
|
||||
- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit
|
||||
- [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations
|
||||
- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (3090) vs Dankclaw (Strix Halo)
|
||||
|
||||
## Related
|
||||
|
||||
- [Project knowledge base](../index.md) — the Podman app stacks on the same
|
||||
two hosts. OpenWebUI consumes the `:7770` endpoint on Dankserver. The
|
||||
Gitea instance on Dankclaw hosts this repository.
|
||||
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the service-scoped
|
||||
git-based deploy with inference isolation
|
||||
@@ -0,0 +1,327 @@
|
||||
---
|
||||
type: reference
|
||||
title: Kairic Edge IU4 Runtime on Dankclaw
|
||||
description: jcbtc/Qwen3.8-27B-IU4-Kairic-Edge served by the Kairic Edge CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Build runbook, binary identity, serving profile, and rollback.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- rocm
|
||||
- dankclaw
|
||||
- gfx1151
|
||||
- kairic
|
||||
- iu4
|
||||
- promptforge
|
||||
timestamp: 2026-08-22
|
||||
---
|
||||
|
||||
# Kairic Edge IU4 Runtime on Dankclaw
|
||||
|
||||
> **Retired 2026-08-22.** This experiment was deployed, benchmarks were
|
||||
> collected, and the service reverted to v3 the same day (operator
|
||||
> decision). `frank-default-large` serves v3 again. This page is kept as
|
||||
> the retired experiment's build runbook and reference.
|
||||
|
||||
`frank-default-large` (port 52625) briefly served
|
||||
`jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` through the Kairic Edge CIRU ROCmFPX
|
||||
llama.cpp runtime. The runtime routes qualified prompt and multi-token
|
||||
verification shapes through the RDNA 3.5 native
|
||||
`V_WMMA_I32_16X16X16_IU4` matrix instruction. It was reverted to
|
||||
[[ciru-activefpx-promptforge]] (v3) on 2026-08-22.
|
||||
|
||||
Performance summary of the experiment: the IU4 lane accelerated prefill
|
||||
1.5-2.3x over v3 at all depths, but prose decode was 15-26% slower
|
||||
(inherent to the release: the IU4 lane does not accelerate M1 target
|
||||
decode). Reasoning-off fast-greedy mode was ~8% faster per token than
|
||||
production chat, and roughly 2x faster per turn from halved output volume.
|
||||
The v3-era experience, including the 400 errors from the argmax fast path
|
||||
rejecting non-greedy requests, motivated the revert.
|
||||
|
||||
## Identity
|
||||
|
||||
The release pins every input. Verify these values before trusting a build.
|
||||
|
||||
| Input | Pinned value |
|
||||
|-------|--------------|
|
||||
| Runtime repo | ciru-ai/ROCmFPX tag `kairic-edge-qwen38-27b-v1`, commit `e97b324` |
|
||||
| Composable Kernel | ROCm/composable_kernel `fdf4bb7fcc984811cef48ce817d89aac064b984a` |
|
||||
| CK release patch | `patches/composable-kernel-gfx1151-iu4.patch`, SHA-256 `ea57bca3793d35d884fc156b6c0bc33d8e672e3bb49f66ff67338503029d6af5` |
|
||||
| Certified toolchain | TheRock 7.15.0a20260718, AMD clang 23.0.0, GCC 13.3.0, CMake 4.4.2, Ninja 1.13.0 |
|
||||
| Local toolchain | TheRock 7.15 SDK (ling3-ciru venv), amdclang++ 23.0.0git, GCC 15.2.0, CMake 4.2.3, Ninja 1.13.2 |
|
||||
| Frozen certified binary | `llama-server` SHA-256 `0ece51fa4489e58b7c6942ec2aa59baadca897d8cf87f8e1dc4dc0b10a6efc49` |
|
||||
| Local binary | `/home/dank/ciru-qwen38/ROCmFPX-kairic/build-kairic/bin/llama-server`, SHA-256 `d7625a0c14ac4158fd793c50ac0cefd8cdcbf76b5aca5cddafa09a353dc8f048` |
|
||||
|
||||
The guide states that a portable rebuild does not match the frozen
|
||||
certified binary byte for byte. The local toolchain differs from the
|
||||
certified one. The local binary reproduces byte-identical across local
|
||||
rebuilds (see [Build reproducibility](#build-reproducibility)).
|
||||
|
||||
## Model artifacts
|
||||
|
||||
Download all four files into one directory. The three `.pfs` sidecars are
|
||||
accelerator-specific prepacked weight views. The runtime requires all of
|
||||
them.
|
||||
|
||||
| File | Size | SHA-256 |
|
||||
|------|------|---------|
|
||||
| `Qwen3.8-27B-IU4-Kairic-Edge.gguf` | 15.48 GiB | `360caf7381907c3eca7ac0afd1228efc016af747f3f38637fb1c7f94daabac2a` |
|
||||
| `Qwen3.8-27B-Kairic-IU4-FFN.pfs` | 7.99 GiB | `adcbb90a7b429a30a2a39043366d68320d72e8b4816a0f498e882b2f80a2ba2b` |
|
||||
| `Qwen3.8-27B-Kairic-IU4-GDN.pfs` | 1.88 GiB | `82f931316f1c895da104915dec4697163808d06f0e6b2dc027cee7aa3afc0f0e` |
|
||||
| `Qwen3.8-27B-Kairic-IU4-GDN-Output.pfs` | 0.70 GiB | `3b07e7b176559e4402924ba0c368532fa6f02118a33c71e70974c809bf6208a3` |
|
||||
|
||||
Total payload: 26.05 GiB. Local directory:
|
||||
`/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge/`. All four SHA-256 values
|
||||
verified on 2026-08-22.
|
||||
|
||||
## Build runbook
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[apt build deps] --> B[Clone ROCmFPX at pinned tag]
|
||||
B --> C[Clone CK at pinned commit]
|
||||
C --> D[Apply and verify CK patch]
|
||||
D --> E[Generate ck/config.h: release gap]
|
||||
E --> F[Run release build script]
|
||||
F --> G[Build verification commands]
|
||||
G --> H[Download and verify model]
|
||||
H --> I[Smoke test]
|
||||
```
|
||||
|
||||
### 1. Install build dependencies
|
||||
|
||||
```
|
||||
sudo apt-get update
|
||||
sudo apt-get install -y build-essential cmake git ninja-build pkg-config libssl-dev
|
||||
```
|
||||
|
||||
### 2. Clone the immutable release
|
||||
|
||||
The local checkout is a git worktree of the pinned tag at
|
||||
`/home/dank/ciru-qwen38/ROCmFPX-kairic`.
|
||||
|
||||
```
|
||||
git clone https://github.com/ciru-ai/ROCmFPX.git
|
||||
cd ROCmFPX
|
||||
git checkout kairic-edge-qwen38-27b-v1
|
||||
git status --short # must print nothing
|
||||
```
|
||||
|
||||
### 3. Prepare Composable Kernel
|
||||
|
||||
```
|
||||
git clone https://github.com/ROCm/composable_kernel.git third_party/composable_kernel
|
||||
git -C third_party/composable_kernel checkout fdf4bb7fcc984811cef48ce817d89aac064b984a
|
||||
git -C third_party/composable_kernel apply ../../patches/composable-kernel-gfx1151-iu4.patch
|
||||
sha256sum patches/composable-kernel-gfx1151-iu4.patch # ea57bca3793d...af5
|
||||
git -C third_party/composable_kernel diff --check
|
||||
```
|
||||
|
||||
### 4. Generate `ck/config.h` (release gap)
|
||||
|
||||
The pinned CK commit ships only `include/ck/config.h.in` (CMake template).
|
||||
The release script and the guide never generate `include/ck/config.h`. A
|
||||
clean build fails at the first CK include without it. Generate it with CK's
|
||||
own CMake configure, then copy it into the CK source include tree:
|
||||
|
||||
```
|
||||
cmake -S third_party/composable_kernel -B third_party/composable_kernel/build-ck \
|
||||
-G Ninja -DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_CXX_COMPILER=/usr/bin/g++ \
|
||||
-DCMAKE_HIP_COMPILER="$ROCM_PATH/bin/amdclang++" \
|
||||
-DCMAKE_PREFIX_PATH="$ROCM_PATH"
|
||||
cp third_party/composable_kernel/build-ck/include/ck/config.h \
|
||||
third_party/composable_kernel/include/ck/config.h
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- The CK configure can fail after `config.h` generation. The generated file
|
||||
is still valid. Check `build-ck/include/ck/config.h` exists.
|
||||
- The `CK_USE_WMMA` flag inside `config.h` gates only CK instance-selection
|
||||
code. The Kairic patched CK headers do not reference it. Local rebuilds
|
||||
with the flag ON and OFF produce byte-identical binaries.
|
||||
- The file is untracked in CK git. A `git clean` in the CK tree removes it.
|
||||
Regenerate it before any rebuild.
|
||||
- A full CK configure without a GPU target leaves `CK_USE_WMMA` undefined.
|
||||
Pass `-DGPU_TARGETS=gfx1151` for the certified-style variant. Both
|
||||
variants build an identical Kairic binary.
|
||||
|
||||
### 5. Build
|
||||
|
||||
The release script owns the certified CMake options.
|
||||
|
||||
```
|
||||
export ROCM_PATH=<path to the 7.15 SDK _rocm_sdk_devel directory>
|
||||
JOBS="$(nproc)" scripts/build-kairic-edge-gfx1151.sh
|
||||
```
|
||||
|
||||
Local `ROCM_PATH` is the split SDK layout inside the retired ling3-ciru
|
||||
venv:
|
||||
`/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_devel`.
|
||||
The serve script assembles `LD_LIBRARY_PATH` from the three split
|
||||
directories (`_rocm_sdk_devel`, `_rocm_sdk_libraries`, `_rocm_sdk_core`) so
|
||||
the 7.15 libraries precede the system ROCm 7.2.4.
|
||||
|
||||
### 6. Build verification
|
||||
|
||||
```
|
||||
./build-kairic/bin/llama-server --version
|
||||
./build-kairic/bin/llama-server --help | grep -A1 -- '--kairic-edge'
|
||||
ldd ./build-kairic/bin/llama-server | grep -E 'not found' # must print nothing
|
||||
```
|
||||
|
||||
Expected local output: `version: 79 (e97b324)`,
|
||||
`built with GNU 15.2.0 for Linux x86_64`.
|
||||
|
||||
### 7. Download and verify model artifacts
|
||||
|
||||
Download the four files from
|
||||
[jcbtc/Qwen3.8-27B-IU4-Kairic-Edge](https://huggingface.co/jcbtc/Qwen3.8-27B-IU4-Kairic-Edge)
|
||||
and verify the SHA-256 values in the table above.
|
||||
|
||||
### 8. Smoke test
|
||||
|
||||
Run the guide's smoke test: health, `/v1/models`, two identical
|
||||
`cache_prompt` completion requests, warm request must report restored
|
||||
prompt tokens. Inspect the server log for route fallback, swap activity,
|
||||
and HIP errors. Confirm the IU4 lane at boot:
|
||||
|
||||
```
|
||||
{"record":"promptforge_init","mode":"iu4_ffn","wmma":"v_wmma_i32_16x16x16_iu4",...}
|
||||
{"record":"promptforge_gdn_init","projection":"qkvz_iu4_hadamard",...}
|
||||
```
|
||||
|
||||
## Build reproducibility
|
||||
|
||||
On 2026-08-22 a fresh rebuild followed the documented procedure from a
|
||||
clean CK tree (pinned commit, patch re-applied, `config.h` regenerated via
|
||||
CK configure). The rebuild produced a byte-identical binary:
|
||||
|
||||
| Build | SHA-256 |
|
||||
|-------|---------|
|
||||
| Original (first deployment) | `d7625a0c14ac4158fd793c50ac0cefd8cdcbf76b5aca5cddafa09a353dc8f048` |
|
||||
| Fresh rebuild (clean CK tree) | `d7625a0c14ac4158fd793c50ac0cefd8cdcbf76b5aca5cddafa09a353dc8f048` |
|
||||
|
||||
The two builds used different `config.h` variants (differ only in
|
||||
`/* #undef */` comment lines and the `CK_USE_WMMA` state). The identical
|
||||
output proves that variant difference has no effect on this build. The
|
||||
original manual `config.h` step is a neutral, minimal bridge over the
|
||||
release gap.
|
||||
|
||||
The original build tree stays at
|
||||
`/home/dank/ciru-qwen38/ROCmFPX-kairic/build-kairic-orig/` until the next
|
||||
service restart confirms the fresh tree, then it is a deletion candidate
|
||||
(move to `.AGENTTRASH`).
|
||||
|
||||
## Serving profile
|
||||
|
||||
The service runs the pinned runner profile plus the local chat
|
||||
deviations below. Tunables live in
|
||||
`scripts/frank-default-large.env`; flags in `scripts/serve-frank-large.sh`.
|
||||
See [[deployment]] for the update process.
|
||||
|
||||
| Setting | Value | Origin |
|
||||
|---------|-------|--------|
|
||||
| Bind | `0.0.0.0:52625` | Local (runner default `127.0.0.1:8080`) |
|
||||
| Context | 262144, 1 slot | Pinned runner |
|
||||
| Batch / ubatch | 2048 / 512 | Pinned runner (fastest measured prefill) |
|
||||
| Model loading | `--no-mmap` | Local (benchmarks: no perf difference) |
|
||||
| Threads | 16 target / 32 batch | Pinned runner |
|
||||
| KV | f16 target and draft | Pinned runner |
|
||||
| Prompt cache | `--cache-ram 8192 --cache-prompt --cache-idle-slots -ctxcp 32` | Pinned runner |
|
||||
| Kairic Edge | `--kairic-edge` | Pinned runner |
|
||||
| Speculative | `draft-mtp` depth 4, `p_split 0.10`, backend sampling | Pinned runner |
|
||||
| Server sampling | temp 0 / top-p 1 / top-k 0 / min-p 0 | Pinned runner |
|
||||
| Reasoning | on, deepseek format, budget -1, preserve_thinking | Local |
|
||||
| `LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH` | 0 | Local (runner: 1) |
|
||||
|
||||
Local deviations and their reasons:
|
||||
|
||||
- **Bind/port**: fleet convention; the runner binds localhost only.
|
||||
- **Reasoning on + deepseek**: thinking goes to `message.reasoning_content`,
|
||||
matching the retired v3 service. `preserve_thinking=true` keeps thinking
|
||||
tokens in the history across turns.
|
||||
- **`--no-mmap`**: benchmarks show no performance difference versus mmap.
|
||||
Kept per operator preference (2026-08-22).
|
||||
- **Target greedy argmax fast path off**: the pinned runner sets
|
||||
`LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1`. That fast path fails closed on
|
||||
any non-greedy request (400 error). The model card's own chat
|
||||
instructions send temp 0.7 / top-p 0.8 / top-k 20 / presence_penalty
|
||||
1.5, which the fast path rejects. Set to 0 for production chat on
|
||||
2026-08-22. This does not affect the Kairic IU4 lane or the MTP draft
|
||||
fast path (`LLAMA_MTP_CPU_ARGMAX_FASTPATH=1` stays on).
|
||||
- **`--kairic-edge` auto-injects ngram-mod** (n_match 24, n_min 64,
|
||||
n_max 64). This is release design, not a misconfiguration.
|
||||
|
||||
## Performance on this machine
|
||||
|
||||
Benchmarks from 2026-08-22 on scratch port 7893, 512 forced tokens,
|
||||
greedy, production config unless noted. Stop the service before
|
||||
benchmarking (see AGENTS.md).
|
||||
|
||||
| Context | Kairic prefill | Kairic decode | v3 prefill | v3 decode (200 tok) |
|
||||
|---------|----------------|---------------|------------|---------------------|
|
||||
| ~8k | 506 (runner) / 348 (prod) | 21.0 (runner) / 19.8 (prod) | 217.7 | 29.42 |
|
||||
| ~16k | 439 (runner) / 370 (prod) | 23.5 (runner) / 19.4 (prod) | 211.2 | 26.42 |
|
||||
| ~64k | 232.5 (prod) | 14.6 (prod) | 155.7 | 19.72 |
|
||||
|
||||
Findings:
|
||||
|
||||
- Prefill runs 1.5-2.3x faster than v3 at every depth.
|
||||
- Prose decode runs 15-26% slower than v3 at every depth.
|
||||
- `--no-mmap` makes no difference (19.89 vs 19.80 t/s decode). The
|
||||
service now runs `--no-mmap` per operator preference.
|
||||
- `-c 262144` versus `-c 8192` makes no difference to decode.
|
||||
- Live service at ~116k context: 13.5-14.9 t/s, consistent with the 64k
|
||||
bench. Long conversations run in the deep-context regime.
|
||||
- MTP draft acceptance on prose: rate 0.31-0.35, mean accepted length
|
||||
2.6-3.0. Draft fast path active, zero failures.
|
||||
- The card's 47.73 t/s aggregate comes from a 164-task coding suite with
|
||||
deterministic sampling and repeated coding context. It does not
|
||||
represent prose chat. The card's natural-prose slice measured 34.88
|
||||
t/s on the certified system.
|
||||
|
||||
### Batch profile comparison (2026-08-22)
|
||||
|
||||
Three batch profiles ran on the identical chat profile (fast path off,
|
||||
reasoning on, `--no-mmap`; the ub2048 row was the earlier mmap run, and
|
||||
mmap makes no difference):
|
||||
|
||||
| Context | b2048/ub512 decode (cold/warm) | b4096/ub2048 decode | b4096/ub4096 decode | ub512 prefill cold | ub4096 prefill cold |
|
||||
|---------|--------------------------------|---------------------|---------------------|--------------------|---------------------|
|
||||
| ~8k | 19.6 / 18.7 | 19.8 / 19.4 | 22.7 / 19.8 | 524 | 266 |
|
||||
| ~16k | 21.8 / 21.5 | 19.8 / 18.8 | 18.1 / 22.5 | 485 | 258 |
|
||||
| ~64k | 13.7 / 15.7 | 14.6 / - | 14.1 / 13.8 | 323 | 203 |
|
||||
|
||||
Ubatch is a prefill knob. Ubatch 512 gives the fastest prefill at every
|
||||
depth, 60-90% above ubatch 4096. Decode is identical across all three
|
||||
profiles within run-to-run noise. The IU4 lane accelerates 96-512-row
|
||||
prompt and verification shapes; ubatch 512 matches those shapes, large
|
||||
ubatches do not. Production runs the runner's qualified 2048/512.
|
||||
|
||||
## Rollback to v3
|
||||
|
||||
The v3 payload and binary stay on the machine for rollback.
|
||||
|
||||
1. Stop the service: `sudo systemctl stop frank-default-large`
|
||||
2. Restore the v3 scripts:
|
||||
`git revert 4c43839 7aee1df 40d50fe` (or checkout the script pair from
|
||||
commit `132e147`)
|
||||
3. Deploy: `FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw`
|
||||
4. Verify health and the `Output-K8` boot records.
|
||||
|
||||
v3 locations: model `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`,
|
||||
binary `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`.
|
||||
Full v3 runbook: [[ciru-activefpx-promptforge]].
|
||||
|
||||
## Citations
|
||||
|
||||
- Model card: <https://huggingface.co/jcbtc/Qwen3.8-27B-IU4-Kairic-Edge>
|
||||
- Build and verification guide: <https://github.com/ciru-ai/ROCmFPX/blob/kairic-edge-qwen38-27b-v1/docs/kairic-edge-gfx1151.md>
|
||||
- Pinned release: <https://github.com/ciru-ai/ROCmFPX/tree/kairic-edge-qwen38-27b-v1>
|
||||
- Pinned runner: `scripts/run-kairic-edge-gfx1151.sh` in the release repo
|
||||
|
||||
## See Also
|
||||
|
||||
- [[ciru-activefpx-promptforge]] — retired v3 runtime, rollback target
|
||||
- [[deployment]] — deploy and update process
|
||||
- [[Dankclaw System Specs]] — hardware and memory model
|
||||
- [[qwen3.8-strix-optimizations]] — Strix Halo optimization history
|
||||
@@ -0,0 +1,225 @@
|
||||
# frank-default-large: kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
|
||||
|
||||
**Status:** Deployed on Dankclaw (since 2026-08-25). Serving is **BLOCKED by
|
||||
the `set_rows` build bug** — the model loads but crashes in
|
||||
`ggml_compute_forward_set_rows` before it can serve. All config goals are met;
|
||||
a build fix is required for the model to run.
|
||||
|
||||
**Alias:** `frank-default-large`, port **7770**.
|
||||
|
||||
The large service serves the kingjones `Qwen3.8-27B-ROCmFPX-Q8_0-AGENT` GGUF
|
||||
from `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` with a
|
||||
charlie12345/ROCmFPX `llama-server` (build `build-kairic`). It runs a
|
||||
**standalone MTP draft head** (loaded separately, on ROCm0), **TurboQuant** V
|
||||
KV cache, the **Sharp** chat template with **reasoning on**, and a **full
|
||||
262144 context / 1 slot** profile. The whole payload plus the 262k KV cache
|
||||
fit in the lifted **114 GiB TTM/GTT** budget (see the 114 GiB change in
|
||||
[[Dankclaw Improvements]] and [[qwen3.8-strix-optimizations]]).
|
||||
|
||||
| Figure | Value |
|
||||
| ------ | ----- |
|
||||
| Model | kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT (`Q8_0_ROCMFPX`) |
|
||||
| HF repo | `kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF` (public) |
|
||||
| Runtime | charlie12345/ROCmFPX `llama-server`, `build-kairic` |
|
||||
| Draft head | standalone `mtp-Qwen3.8-27B-Q4_0.gguf` on ROCm0 |
|
||||
| KV cache | f16 K, TurboQuant (`turbo4`) V, full 262144 context, 1 slot |
|
||||
| Reasoning | on, `deepseek`, budget `-1`, `--reasoning-format none` |
|
||||
| Chat template | Sharp (repo-tracked), `enable_thinking` + `preserve_thinking` on |
|
||||
| Host path | `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` |
|
||||
| Tunables | `scripts/frank-default-large.env` |
|
||||
| Serve script | `scripts/serve-frank-large.sh` |
|
||||
|
||||
## Serving status: the `set_rows` build bug (blocks serving)
|
||||
|
||||
As of 2026-08-25 the Q8_0 AGENT model will not serve on Dankclaw. It loads,
|
||||
reaches `initializing slots, n_slots = 1`, but never prints `start serving`.
|
||||
Health stays `503` until the process exits. The crash is in
|
||||
`ggml_compute_forward_set_rows` in `libggml-cpu.so.0`, a NULL function-pointer
|
||||
jump (address `0x00000000`).
|
||||
|
||||
The backtrace (gdb). The main thread and one OpenMP worker crash in the same
|
||||
path:
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[llama_context::graph_compute] --> B[ggml_backend_cpu_graph_compute]
|
||||
B --> C[ggml_graph_compute]
|
||||
C --> D[GOMP_parallel]
|
||||
D --> E[ggml_compute_forward_set_rows]
|
||||
E --> F[NULL dispatch -> address 0x0]
|
||||
D --> G[Thread 23 worker: same path]
|
||||
```
|
||||
|
||||
Test matrix. Five ROCmFPX builds and the all-CPU path were tested on a scratch
|
||||
port (`/tmp/g80bench/run.sh`), `gl=45`, `ctx=131072`. All crash in `set_rows`.
|
||||
|
||||
| Build | Binary | Version | Result |
|
||||
| ----- | ------ | ------- | ------ |
|
||||
| build-kairic | `ciru-qwen38/ROCmFPX/build-kairic/bin/llama-server` | v81 (205a3e5f4) | segfault in set_rows |
|
||||
| build-promptforge | `ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` | PromptForge/ActiveFPX lineage | segfault in set_rows |
|
||||
| build-rocmi4-w4a4 | `rocmi4-qwen38/ROCmFPX/build-rocmi4-w4a4/bin/llama-server` | v257 (c49ebdb) | segfault in set_rows |
|
||||
| build-strix-rocmfp4 | `GitPackages/ROCmFPX/build-strix-rocmfp4/bin/llama-server` | v209 (00d5452) | segfault in set_rows |
|
||||
| all-CPU | build-kairic, `-ngl 0` | v81 | segfault in set_rows |
|
||||
|
||||
The crash is not GPU-specific (the all-CPU path fails too), not MTP-specific
|
||||
(`--spec-type none` fails), and not fixed by `LLAMA_SET_ROWS=0`. So it is a
|
||||
build bug: the Q8_0 AGENT GGUF graph needs a `set_rows` dispatch the ROCmFPX
|
||||
backends do not register. The older caf-eemin ROCmI4 and Kairic Edge models
|
||||
served fine on related ROCmFPX builds. The caf-eemin payload was about 14.4
|
||||
GiB; the separate
|
||||
PromptForge/ActiveFPX deployment used a much larger multi-file payload of about
|
||||
36.7 GiB. These are different deployments. The Q8_0 AGENT graph is the trigger
|
||||
for the `set_rows` failure.
|
||||
|
||||
**Fix path:** rebuild charlie12345/ROCmFPX from source with a working
|
||||
`set_rows` for the ROCmFPX backend, or find a build where it is registered.
|
||||
This unblocks the model. See the context budget reality below once `set_rows`
|
||||
is fixed.
|
||||
|
||||
## Model files
|
||||
|
||||
Three files, from the HF repo above:
|
||||
|
||||
| File | Size | Purpose |
|
||||
| ---- | ---- | ------- |
|
||||
| `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf` | 28.21 GB | main weights |
|
||||
| `mtp-Qwen3.8-27B-Q4_0.gguf` | 1.68 GB | standalone MTP draft head (loaded by the script) |
|
||||
| `mmproj-Qwen3.8-27B-BF16.gguf` | 0.93 GB | vision projector (`--mmproj`) |
|
||||
|
||||
The repo also has a `mtp-Qwen3.8-27B-Q8_0.gguf` (3.16 GB) draft; the env uses
|
||||
the Q4_0 draft. Download them from the resolve URLs below (they are large —
|
||||
use the token in `~/.cache/huggingface/token`).
|
||||
|
||||
## Download
|
||||
|
||||
```bash
|
||||
# on Dankclaw
|
||||
TOKEN=$(cat ~/.cache/huggingface/token)
|
||||
REPO=kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF
|
||||
DIR=/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT
|
||||
mkdir -p "$DIR"; cd "$DIR"
|
||||
dl(){ curl -fsSL -C - -o "$1" --retry 5 --retry-delay 3 --retry-max-time 3600 \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
"https://huggingface.co/$REPO/resolve/main/$1"; echo "done: $1"; }
|
||||
dl Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf
|
||||
dl mmproj-Qwen3.8-27B-BF16.gguf
|
||||
dl mtp-Qwen3.8-27B-Q4_0.gguf
|
||||
```
|
||||
|
||||
The serve script checks each artifact is present and readable before launch
|
||||
(`scripts/frank-default-large.env` -> the four `MODEL_*` files plus the chat
|
||||
template). A missing file fails the artifact check with `missing or
|
||||
unreadable required artifact`.
|
||||
|
||||
## Serving profile
|
||||
|
||||
`scripts/serve-frank-large.sh` builds the `llama-server` call from the env:
|
||||
|
||||
```
|
||||
/dev/llama-server -m <main> --alias frank-default-large
|
||||
--host 0.0.0.0 --port 7770 --jinja
|
||||
-dev ROCm0 -ngl 999 -np 1 -c 262144
|
||||
-b 512 -ub 512 -fa on
|
||||
-ctk f16 -ctv turbo4 -t 16 -tb 32
|
||||
-ctxcp 32 --cache-ram 8192 --cache-prompt --cache-idle-slots --metrics
|
||||
--spec-type draft-mtp --model-draft <mtp-Q4_0>
|
||||
--mmproj <mmproj-BF16>
|
||||
--spec-draft-device ROCm0 --spec-draft-ngl 999
|
||||
--spec-draft-type-k f16 --spec-draft-type-v turbo4
|
||||
--spec-draft-n-max 4 --spec-draft-n-min 0
|
||||
--spec-draft-p-min 0.0 --spec-draft-p-split 0.10
|
||||
--spec-draft-backend-sampling
|
||||
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0
|
||||
--presence-penalty 1.5
|
||||
--reasoning on --reasoning-format none --reasoning-budget -1
|
||||
```
|
||||
|
||||
Key choices:
|
||||
|
||||
- **Standalone MTP draft** (`--model-draft`, `--spec-type draft-mtp`). The
|
||||
AGENT GGUF has no embedded draft head; the draft head is a separate file on
|
||||
ROCm0. This matches the Qwen3.8-27B AGENT shape (`--spec-draft-type-k
|
||||
f16 --spec-draft-type-v turbo4`).
|
||||
- **f16 K + TurboQuant V** cache. f16 K is the safe long-ctx default; TurboQuant
|
||||
halves the V cache at 262k while keeping quality (see the ubatch/threads-batch
|
||||
notes in [[qwen3.8-strix-optimizations]]).
|
||||
- **262144 context, 1 slot — the target.** The whole context KV cache plus the
|
||||
~30 GB payload (main + draft + mmproj) are meant to fit in the 114 GiB
|
||||
budget. Measured reality: `gl=999` (all layers on ROCm0) at 262144 **OOMs**
|
||||
on device-local (`cudaMalloc failed`, the KV cache needs ~10-18 GiB that does
|
||||
not fit once the payload is on the card). `gl=45` (partial offload) fits a
|
||||
131072 context. The env ships `CTX_SIZE=262144`; the proven serving context
|
||||
today is **131072 at `gl=45`** (before the `set_rows` fix). `deploy.sh` waits
|
||||
up to 20 minutes (see `LLM_WAIT`).
|
||||
- **Reasoning on.** The Sharp template drives thinking (medium effort);
|
||||
`--reasoning-format none` lets the template project the reasoning content into
|
||||
`reasoning_content`. `--reasoning-budget -1` = no budget cap.
|
||||
|
||||
## Reasoning + chat template
|
||||
|
||||
The Sharp template (`scripts/chat-template.jinja`, repo-tracked) is a
|
||||
Qwen-AGENT shape. Env `CHAT_TEMPLATE_KWARGS` keeps the reasoning trail in the
|
||||
conversation:
|
||||
|
||||
```json
|
||||
{"enable_thinking": true, "preserve_thinking": true, "preserve_reasoning": true}
|
||||
```
|
||||
|
||||
`preserve_thinking: true` keeps the model's prior thinking in the next prompt
|
||||
(the user's "preserve thinking" request). `preserve_reasoning: true` keeps the
|
||||
model's prior reasoning trace across requests too (OpenWebUI sends
|
||||
`preserve_reasoning`/`preserve_thinking: true` when the conversation needs the
|
||||
full trail). For multi-turn reasoning, the template stays in the reasoning
|
||||
shape and `enable_thinking: true` per request.
|
||||
|
||||
## Memory: the 114 GiB TTM/GTT requirement
|
||||
|
||||
The 262k f16 KV cache is large (~a few GB) and the payload is ~30 GB. Both live
|
||||
in the Strix Halo unified-memory GTT. The TTM `pages_limit` was raised to
|
||||
114 GiB on 2026-08-25 (written live to
|
||||
`/sys/module/ttm/parameters/pages_limit`, then persisted in
|
||||
`/etc/modprobe.d/ttm.conf`). Without the raise the effective ceiling is ~15 GiB
|
||||
and a 28 GB + 262k-KV load stalls. The GTT page-limit change is per the
|
||||
`qwen3.8-strix-optimizations` table item 5.
|
||||
|
||||
## Health + troubleshooting
|
||||
|
||||
- Health: `curl http://192.168.0.143:7770/health` → `{"status":"ok"}`.
|
||||
- `/api/models` reports `n_ctx:262144, n_ctx_train:262144, n_params:27.3B`.
|
||||
- Model load is slow (a few minutes cold for the ~30 GB payload with a fresh
|
||||
KV cache). Health returns 503 while loading. Do not kill the process.
|
||||
`deploy.sh` waits up to 20 minutes (see `LLM_WAIT`).
|
||||
- Kernel spam about `amdgpu: SVM mapping failed` is the known documented
|
||||
symptom, not root cause (see the crash-loop procedure in the project
|
||||
AGENTS.md). Investigate other causes before assuming memory pressure.
|
||||
- A client abort logs `W srv stop: cancel task, id_task = N`; a natural stop
|
||||
logs only `slot release` with `truncated = 0`.
|
||||
|
||||
## Env tunables
|
||||
|
||||
`scripts/frank-default-large.env`. Serving: `CTX_SIZE=262144`,
|
||||
`PARALLEL=1`, `BATCH_SIZE=512`, `UBATCH_SIZE=512`, `NUM_THREADS=16`,
|
||||
`THREAD_BATCH=32`, `CACHE_TYPE_K=f16`, `CACHE_TYPE_V=turbo4`,
|
||||
`CTX_CHECKPOINTS=32`, `CACHE_RAM=8192`. Reasoning: `REASONING=on`,
|
||||
`REASONING_FORMAT=none`. Sampling: `TEMPERATURE=0.7`, `TOP_K=20`,
|
||||
`TOP_P=0.8`. MTP: `MTP_N_MAX=4`, `MTP_N_MIN=0`, `MTP_P_MIN=0.0`,
|
||||
`MTP_P_SPLIT=0.10`. Strix Halo: `HSA_OVERRIDE_GFX_VERSION=11.5.1`,
|
||||
`GGML_HIP_ENABLE_UNIFIED_MEMORY=1`.
|
||||
|
||||
**Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`** — the ROCm HIP
|
||||
runtime reads those bare vars (see the 2026-08-13 TEMP incident). llama.cpp
|
||||
reads `--host/--port/--temp` as CLI flags, never as env vars.
|
||||
|
||||
## History
|
||||
|
||||
- **2026-08-25 (today):** switched the large service to the kingjones
|
||||
Q8_0-AGENT model with 262144 context and reasoning on. Downloaded the three
|
||||
payload files from the HF repo, raised the TTM/GTT to 114 GiB for the full
|
||||
context, and persisted the env/serve script in this repo. Measured that the
|
||||
model loads but crashes in `ggml_compute_forward_set_rows` on all five
|
||||
available ROCmFPX builds; serving is BLOCKED pending a rebuild. The config
|
||||
goals (GTT 114 GiB, reasoning on, preserve_thinking/preserve_reasoning on,
|
||||
262144 env context) are in place. Proven serving context today is 131072 at
|
||||
`gl=45`. See the `set_rows` build bug section above.
|
||||
- Preceding model: Kairic Edge Qwen3.8-27B IU4 (deployed 2026-08-25 ~05:42,
|
||||
retired the same day for the AGENT model). See [[kairic-edge]].
|
||||
@@ -0,0 +1,99 @@
|
||||
---
|
||||
type: reference
|
||||
title: kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw
|
||||
description: kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Dense model with a standalone MTP Q4_0 draft head. frank-default-large.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- rocm
|
||||
- dankclaw
|
||||
- gfx1151
|
||||
- kingjones
|
||||
- rocmfp4
|
||||
- mtp
|
||||
timestamp: 2026-08-24
|
||||
---
|
||||
|
||||
# kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw
|
||||
|
||||
> **Active again 2026-09-01.** The kingjones ROCmFP4 STRIX MTP payload is
|
||||
> restored as `frank-default-large` after the Q8_0 AGENT experiment. The Q8_0
|
||||
> model was retired because it crashed before serving. The Q6 AGENT model,
|
||||
> standalone MTP draft, Q8 projector, and PromptForge runtime are now active.
|
||||
|
||||
> This page documents the active model and its runbook.
|
||||
|
||||
`frank-default-large` (port 7770) serves
|
||||
`Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` from `kingjones777` on Hugging
|
||||
Face. It is a **dense** Qwen3.8-27B, abliterated (uncensored), quantized to
|
||||
the ROCmFP4 Q6 AGENT format (ftype 114). The MTP speculative head ships as
|
||||
a **separate** Q4_0 draft GGUF, so a bundled-head auto-detect does not apply
|
||||
here; the head is loaded with `--model-draft`.
|
||||
|
||||
## Model Files
|
||||
|
||||
The card ships six text files plus the vision projector. The service uses
|
||||
the Q6 AGENT file (requested for the fleet), the separate MTP draft head
|
||||
and the Q8 vision projector (`mmproj-Qwen3.8-27B-Q8_0.gguf`).
|
||||
|
||||
| File | Size |
|
||||
|------|------|
|
||||
| `Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf` | 23.15 GiB |
|
||||
| `mtp-Qwen3.8-27B-Q4_0.gguf` (draft head) | 1.56 GiB |
|
||||
| `mmproj-Qwen3.8-27B-Q8_0.gguf` (vision) | 0.59 GiB |
|
||||
|
||||
Store the files under `/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP/`.
|
||||
|
||||
## Speculative Decoding
|
||||
|
||||
Dense model, so the finding measured on the dense Qwen3.8 family holds:
|
||||
**MTP alone at `--spec-draft-n-max 4` is the knee**. ngram under MTP hurts
|
||||
(measured -4 to -5 t/s on the v3; see [[ciru-activefpx-promptforge]]). The
|
||||
draft head is a separate file loaded via:
|
||||
|
||||
```
|
||||
--spec-type draft-mtp
|
||||
--model-draft mtp-Qwen3.8-27B-Q4_0.gguf
|
||||
--spec-draft-n-max 4
|
||||
--spec-draft-n-min 0
|
||||
--spec-draft-p-min 0.0
|
||||
```
|
||||
|
||||
The draft head loads on `ROCm1` with full offload (`--spec-draft-ngl 999`)
|
||||
and f16 KV.
|
||||
|
||||
## Launch Excerpt
|
||||
|
||||
The key flags on the CIRU ROCmFPX runtime:
|
||||
|
||||
```
|
||||
llama-server \
|
||||
-m Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf \
|
||||
--mmproj mmproj-Qwen3.8-27B-Q8_0.gguf \
|
||||
--chat-template-file chat-template.jinja \
|
||||
--chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true}' \
|
||||
-ngl 999 -dev ROCm0 -c 196608 -np 1 \
|
||||
-b 8192 -ub 8192 -ctxcp 32 \
|
||||
--spec-type draft-mtp --model-draft mtp-Qwen3.8-27B-Q4_0.gguf \
|
||||
--spec-draft-ngl 999 --spec-draft-n-max 4 \
|
||||
-fa on --no-mmap --cache-type-k f16 --cache-type-v f16
|
||||
```
|
||||
|
||||
## Known Issues
|
||||
|
||||
1. **GPU only**: Vulkan, CUDA, and CPU cannot load these files.
|
||||
2. `--spec-draft-n-max` defaults to 16; set 4.
|
||||
3. `reasoning_effort: "none"` throws in the Qwen API. Use
|
||||
`enable_thinking: false` instead.
|
||||
4. Small `max_tokens` with thinking on returns empty `content`. The Sharp
|
||||
template drives thinking on by default (effort medium).
|
||||
5. `preserve_thinking:true` (in `chat_template_kwargs`) keeps `<think>`
|
||||
blocks in recorded assistant messages. `preserve_thinking:false` strips
|
||||
them. See [[deployment]] for the chat_format/reasoning_format pairing
|
||||
(`peg-native` + `deepseek`) the Pi agent expects from this endpoint.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[deployment]] — deploy and update process
|
||||
- [[ciru-activefpx-promptforge]] — the retired v3 page
|
||||
- [[Dankclaw System Specs]] — the box
|
||||
- [[frank-redo-2026-08-17]] — fleet topology history
|
||||
@@ -0,0 +1,276 @@
|
||||
---
|
||||
type: reference
|
||||
title: kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
|
||||
description: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF served by the ROCmFPX llama.cpp fork. 177B params (56B MoE + 51.2B PLE), single 113.47 GiB GGUF. frank-default-large.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- rocm
|
||||
- dankclaw
|
||||
- gfx1151
|
||||
- kingjones
|
||||
- rocmfp4
|
||||
- strix-halo
|
||||
- qwen4exp
|
||||
timestamp: 2026-08-27
|
||||
---
|
||||
|
||||
# kingjones Qwen3.8-Flash-Next ROCmFP4 STRIX on Dankclaw
|
||||
|
||||
> **Active since 2026-08-27.** Replaced the STRIX_LEAN deployment on the same
|
||||
> build. Same GPU memory profile, larger PLE table.
|
||||
|
||||
`frank-default-large` (port 7770) serves
|
||||
`Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX` from `kingjones777` on Hugging Face.
|
||||
A single GGUF file, 113.47 GiB. Text-only (no mmproj). MTP draft head
|
||||
loaded as a separate GGUF sidecar (see MTP Draft Head below).
|
||||
|
||||
## Model Architecture
|
||||
|
||||
Base model: Qwen3.8-Flash-Next (177B parameters total).
|
||||
|
||||
| Component | Params | Details |
|
||||
|-----------|--------|---------|
|
||||
| MoE experts | 125.8B | 512 experts, 10 used per token |
|
||||
| Attention | dense | 48 layers (12 attn + 36 SSM) |
|
||||
| PLE table | 51.2B | `per_layer_token_embd`, Q8_0 |
|
||||
| lm head | ~0.3% | Q6_K |
|
||||
|
||||
The PLE (per-layer embedding) n-gram table is the only thing that differs
|
||||
between STRIX, STRIX_LEAN, and FAST tiers. Everything else is byte-identical.
|
||||
|
||||
## File Details
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| File | `Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf` |
|
||||
| Size | 113.47 GiB |
|
||||
| Format | Single GGUF (not sharded) |
|
||||
| file_type | 105 (`Q4_0_ROCMFP4_STRIX`) |
|
||||
| General | 113.47 GiB, 5.51 bpw |
|
||||
|
||||
Tensor composition:
|
||||
|
||||
| Tensor group | Type |
|
||||
|---|---|
|
||||
| MoE expert weights (`ffn_*_exps`, 144) | TYPE_101 (ROCmFP4, 4.251 bpw) |
|
||||
| Shared expert (`ffn_*_shexp`, 144) | TYPE_101 |
|
||||
| Attention (`attn_*`, 120) | half TYPE_100, half TYPE_101 |
|
||||
| PLE (`per_layer_token_embd.weight`) | Q8_0 — 50.66 GiB |
|
||||
| `token_embd.weight` | Q6_K |
|
||||
| `output.weight` (lm head) | Q6_K |
|
||||
| Norms / biases | F32 |
|
||||
|
||||
## File Type Reference
|
||||
|
||||
| Tier | file_type | PLE table | Total size |
|
||||
|------|-----------|-----------|------------|
|
||||
| FAST | TYPE_106 | 25.34 GiB | 87.94 GiB |
|
||||
| STRIX_LEAN | TYPE_105 (old STRIX_LEAN) | 35.76 GiB | 98.49 GiB |
|
||||
| **STRIX** | **105 (`Q4_0_ROCMFP4_STRIX`)** | **50.66 GiB** | **113.47 GiB** |
|
||||
|
||||
TYPE_100 = 4.506 bpw, TYPE_101 = 4.251 bpw (measured exactly by solving
|
||||
three files for two unknowns).
|
||||
|
||||
## Memory
|
||||
|
||||
The PLE table never enters GPU VRAM. It streams from the file through the OS
|
||||
page cache.
|
||||
|
||||
| Context | GTT at load | GTT after prompt | GPU resident |
|
||||
|---------|-------------|-------------------|-------------|
|
||||
| 8,192 | 63.3 GiB | 63.6 GiB | ~63 GiB |
|
||||
| 32,768 | 64.1 GiB | 64.8 GiB | |
|
||||
| 65,536 | 65.1 GiB | 66.2 GiB | |
|
||||
| 131,072 | 67.2 GiB | 69.1 GiB | ~69 GiB |
|
||||
|
||||
~63 GiB is the non-PLE mass plus compute buffers. **The PLE table does not
|
||||
grow GPU memory regardless of context depth.**
|
||||
|
||||
Three critical consequences:
|
||||
|
||||
1. **`--no-mmap` gets you OOM-killed.** It forces the PLE into anonymous memory
|
||||
(neither file-backed nor reclaimable). The cgroup killer takes the process
|
||||
with nothing in the server log — only `dmesg` shows the evidence.
|
||||
2. **Don't force PLE to CPU with `-ot`.** Measured generation dropped from
|
||||
~23 to 13.4 tok/s. The kernel streams it better than any offload hack.
|
||||
3. **A bigger tier costs disk, not GPU memory.** If you have storage, take
|
||||
STRIX.
|
||||
|
||||
## Performance
|
||||
|
||||
Measured on Ryzen AI MAX+ 395 (gfx1151, ROCm 7.2.4), 49/49 offload. Prompts
|
||||
sized to 85% of context window:
|
||||
|
||||
| Context | Prompt tokens | pp tok/s | gen tok/s |
|
||||
|---------|--------------|----------|-----------|
|
||||
| 8,192 | 6,963 | 279 | **22.02** |
|
||||
| 32,768 | 27,852 | 264 | **20.52** |
|
||||
| 65,536 | 55,705 | 234 | **18.23** |
|
||||
| 131,072 | 111,411 | 185 | **15.34** |
|
||||
|
||||
Short prompts (~3,300 tokens): 381 tok/s prompt processing, 22.7 tok/s
|
||||
generation (median of 3 runs).
|
||||
## Benchmark: threads-batch tuning
|
||||
|
||||
Sweeped `--threads-batch` (16, 32, 48, 64, 80, 96, 128) with a 64K-token
|
||||
prompt via HTTP API (`max_tokens=1`). Fresh server start per value.
|
||||
|
||||
| threads-batch | prefill tok/s | wall time |
|
||||
|---------------|--------------|-----------|
|
||||
| **16** | **373.5** | 62.9s |
|
||||
| 32 | 354.4 | 66.3s |
|
||||
| 48 | 344.1 | 68.3s |
|
||||
| 64 | 344.7 | 68.2s |
|
||||
| 80 | 344.4 | 68.2s |
|
||||
| 96 | 357.1 | 65.8s |
|
||||
| 128 | 321.1 | 73.2s |
|
||||
|
||||
**Optimal: `--threads-batch 16`** (373.5 tok/s). Lower thread batching wins
|
||||
for prefill on this model -- fewer threads compete for GPU resources during
|
||||
the prompt processing phase. The default of 32 is 5.1% slower. Above 64,
|
||||
performance plateaus then declines (128 is 14% worse than optimal).
|
||||
|
||||
## Benchmark: decode throughput
|
||||
|
||||
Sweeped `--threads-batch` (8, 16, 32) with a ~23.5K-token code prompt and
|
||||
256 generated tokens via HTTP API. Fresh server start per value.
|
||||
|
||||
| threads-batch | gen tokens | wall time | gen tok/s |
|
||||
|---------------|-----------|-----------|-----------|
|
||||
| 16 | 256 | 74.8s | 3.4 |
|
||||
| 32 | 256 | 75.0s | 3.4 |
|
||||
| 8 | 256 | 76.6s | 3.3 |
|
||||
|
||||
**Decode throughput is essentially flat across all tested values.** TB=16 and
|
||||
TB=32 are identical at 3.4 tok/s; TB=8 is marginally slower at 3.3 tok/s.
|
||||
Combined with the 5.1% prefill advantage at lower thread counts, `--threads-batch
|
||||
16` is the optimal setting for both phases.
|
||||
|
||||
## Serving Profile
|
||||
|
||||
Runtime: charlie12345/ROCmFPX `llama-server` built at
|
||||
`/home/dank/GitPackages/ROCmFPX/build-qwen38`.
|
||||
|
||||
Required flags:
|
||||
|
||||
```
|
||||
llama-server \
|
||||
--model Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf \
|
||||
--host 0.0.0.0 --port 7770 --jinja \
|
||||
--device ROCm0 --gpu-layers 999 \
|
||||
--fit off --flash-attn on \
|
||||
--ctx-size 524288 --parallel 2 \
|
||||
--batch-size 4096 --ubatch-size 4096 \
|
||||
--cache-type-k f16 --cache-type-v f16 \
|
||||
--threads 16 --threads-batch 16 \
|
||||
--ctx-checkpoints 16 --cache-ram 8192 \
|
||||
--cache-prompt --cache-idle-slots --metrics \
|
||||
--temp 0.7 --top-p 0.8 --top-k 20 --min-p 0.0 \
|
||||
--presence-penalty 1.5 \
|
||||
--reasoning on --reasoning-format deepseek --reasoning-budget -1 \
|
||||
--chat-template-file chat-template.jinja \
|
||||
--chat-template-kwargs '{"enable_thinking":true,"preserve_thinking":true,"preserve_reasoning":true}' \
|
||||
--spec-type draft-mtp,ngram-mod \
|
||||
--model-draft mtp-Qwen3.8-Flash-Next-Q8_0.gguf \
|
||||
--spec-draft-n-max 4 --spec-draft-p-min 0.75 \
|
||||
--spec-draft-ngl 999 --spec-draft-device ROCm0 \
|
||||
--spec-draft-type-k f16 --spec-draft-type-v f16
|
||||
```
|
||||
|
||||
Key choices:
|
||||
|
||||
- **`--fit off`** — auto-fit code segfaults on this model's qwen4exp
|
||||
hyper-connection/indexer configuration.
|
||||
- **mmap (default)** — do NOT use `--no-mmap`. See memory section.
|
||||
- **`--gpu-layers 999`** — all layers on SVM-backed GPU pool.
|
||||
- **`--ctx-size 524288`** — above model's max (131072); llama.cpp clamps
|
||||
internally.
|
||||
- **`--spec-type draft-mtp,ngram-mod`** — MTP draft head + ngram-mod
|
||||
fallback. The `ngram-mod` combo is recommended by the HF card; the
|
||||
prior dense 27B measurement showed ngram under MTP hurt by 4-5 t/s
|
||||
([[kingjones-strix-mtp]]). Benchmark both before committing.
|
||||
- **`--spec-draft-p-min 0.75`** — confidence gate. Without it, low-acceptance
|
||||
prose loses more to rejected drafts than it gains (per the HF card).
|
||||
|
||||
## MTP Draft Head
|
||||
|
||||
The MTP (multi-token-prediction) draft head from
|
||||
[EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF)
|
||||
(Q8_0, 4.1 GB). Architecture `qwen4exp`, matching the main model. Loaded
|
||||
as a separate GGUF sidecar via `--model-draft`.
|
||||
|
||||
| Field | Value |
|
||||
|-------|-------|
|
||||
| File | `mtp-Qwen3.8-Flash-Next-Q8_0.gguf` |
|
||||
| Size | 4.1 GB (4,137,429,088 B) |
|
||||
| Quant | Q8_0 |
|
||||
| Architecture | `qwen4exp` |
|
||||
| Source | EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF |
|
||||
|
||||
**Measured (HF card, gfx1151, UD-IQ3_XXS target, temp 0):** decode +50% on
|
||||
code (23.5 → 35.7 t/s, ~85% acceptance), +49% at 156K context depth.
|
||||
Speculative decoding is lossless at temp 0. Our service runs at temp 0.7,
|
||||
so expect lower acceptance; the `p_min 0.75` gate compensates.
|
||||
|
||||
**Caveats:**
|
||||
|
||||
- The MTP head is from the original `Qwen/Qwen3.8-Flash-Next`; our model is
|
||||
the "Uncensored" (abliterated) variant. Abliteration modifies a small
|
||||
number of weight vectors, not the full model — acceptance may be
|
||||
marginally lower.
|
||||
- The HF card's tested fork is `Aristo94/EngramHalo.cpp`, not our
|
||||
`charlie12345/ROCmFPX`. Both have PRs #27739 (MTP) + #27742 (qwen4exp)
|
||||
merged, so the MTP mechanism is the same upstream code. If the draft
|
||||
load crashes, check for Strix-Halo kernel patch differences first.
|
||||
- Set `SPEC_TYPE=none` in `scripts/frank-default-large.env` to disable.
|
||||
|
||||
## Deployment
|
||||
|
||||
### Download
|
||||
|
||||
```bash
|
||||
# on Dankclaw
|
||||
TOKEN=$(cat ~/.cache/huggingface/token)
|
||||
DIR=/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX
|
||||
mkdir -p "$DIR"
|
||||
curl -fsSL -C - -o "$DIR/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf" \
|
||||
--retry 5 --retry-delay 3 --retry-max-time 7200 \
|
||||
-H "Authorization: Bearer $TOKEN" \
|
||||
"https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF/resolve/main/Qwen3.8-Flash-Next-Q4_0-ROCmFP4-STRIX.gguf"
|
||||
```
|
||||
|
||||
The file is 113.47 GiB. Allow ~20-30 minutes for download depending on network.
|
||||
|
||||
### Swap procedure
|
||||
|
||||
1. Stop service: `sudo systemctl stop frank-default-large`
|
||||
2. Remove old model dir: `rm -rf /home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN`
|
||||
3. Start service: `sudo systemctl start frank-default-large`
|
||||
4. Verify health: `curl http://192.168.0.143:7770/health`
|
||||
|
||||
### Config files
|
||||
|
||||
- Env: `scripts/frank-default-large.env`
|
||||
- Serve: `scripts/serve-frank-large.sh`
|
||||
- Chat template: `scripts/chat-template.jinja`
|
||||
|
||||
## Known Issues
|
||||
|
||||
1. **Patched llama.cpp required.** PR #27742 merged into the ROCmFPX fork.
|
||||
Stock builds reject these files (TYPE_100/101 have no number in stock ggml).
|
||||
2. **Single-file payload.** Not sharded. Use `hf download` or a resumable
|
||||
curl. Browser download unreliable for 113 GiB.
|
||||
3. **Model load is slow.** A few minutes cold for the 113 GiB file with a
|
||||
fresh KV cache. Health returns 503 while loading. Do not kill the process.
|
||||
`deploy.sh` waits up to 20 minutes.
|
||||
4. **Kernel spam about `amdgpu: SVM mapping failed`** is a known documented
|
||||
symptom, not root cause. Investigate other causes before assuming memory
|
||||
pressure.
|
||||
|
||||
## Citations
|
||||
|
||||
- [kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF](https://huggingface.co/kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF)
|
||||
- [Qwen/Qwen3.8-Flash-Next base model](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
|
||||
- [charlie12345/ROCmFPX](https://github.com/charlie12345/ROCmFPX) — fork with PR #27742 (qwen4exp architecture)
|
||||
- [EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF](https://huggingface.co/EasiiX/Qwen3.8-Flash-Next-MTP-Strix-Halo-GGUF) — MTP draft head sidecar
|
||||
- [llama.cpp PR #27739](https://github.com/ggml-org/llama.cpp/pull/27739) — MTP draft head support
|
||||
@@ -0,0 +1,112 @@
|
||||
# Ling-3.0-flash Q6_K Trial on Dankclaw
|
||||
|
||||
> **RETIRED 2026-08-17.** The `frank-large-ling` unit is disabled under the boot policy. `frank-default-large` now runs the Qwen3.8-27B PP head, not the Qwen3.6-27B config described in the (obsolete) switch-back section below.
|
||||
|
||||
This page records the Ling-3.0-flash trial that temporarily replaced
|
||||
`frank-default-large` on Dankclaw. It replaces the Qwen3.6-27B service on
|
||||
port 7770 under the same API alias.
|
||||
|
||||
## Status
|
||||
|
||||
- Service: `frank-large-ling` (systemd).
|
||||
- Model: `inclusionAI/Ling-3.0-flash` Q6_K (MoE, ~118B total params).
|
||||
- Runtime: llama.cpp **fork** `aetherbird/llama.cpp` branch
|
||||
`bailingmoe3-support` (commit `db4480bc8`, build b10411) at
|
||||
`/home/dank/llama.cpp-ling`. Stock llama.cpp cannot load the
|
||||
`bailingmoe3` architecture.
|
||||
- Backend: `Vulkan0` (RADV) with AMD GTT.
|
||||
- Spec: `draft-mtp` `n_max 3`. The MTP block is bundled in the Ling GGUF
|
||||
itself (block 42) - no separate drafter file.
|
||||
- Context: 262144 total, 2 slots x 131072, batch 512 / ubatch 256.
|
||||
- Measured: load ~45s, generation ~7.7 t/s (cold, 4k prompt) to ~36 t/s
|
||||
(tiny cached prompt). Concurrent streams both complete.
|
||||
|
||||
## Why Vulkan + GTT
|
||||
|
||||
The Strix Halo APU has 8 x 16 GB LPDDR5 modules = **128 GB physical
|
||||
shared memory**. The firmware carves ~96 GiB as dedicated iGPU VRAM; the
|
||||
kernel sees the remainder (~30 GiB, `MemTotal`). GRUB adds no memory
|
||||
options (`iommu=pt` only).
|
||||
|
||||
The ROCm runtime reports 96 GiB VRAM. **Vulkan (RADV) reports a
|
||||
111.2 GiB heap** (113910 MiB) because it can also reach system memory
|
||||
through the GTT. For a model near 100 GiB, Vulkan is the only backend
|
||||
with room. `--list-devices` shows the heap:
|
||||
|
||||
```
|
||||
ROCm0: AMD Radeon Graphics (98304 MiB)
|
||||
Vulkan0: AMD Radeon Graphics (RADV STRIX_HALO) (113910 MiB)
|
||||
```
|
||||
|
||||
## Memory Fit (Q6_K = 98.26 GiB)
|
||||
|
||||
| Item | Size |
|
||||
| ---- | ---- |
|
||||
| Vulkan heap (llama.cpp view) | ~111.2 GiB |
|
||||
| Model Q6_K file | 98.26 GiB |
|
||||
| Model + MTP + graphs in heap | ~98.7 GiB |
|
||||
| Free heap after load | ~10 GiB |
|
||||
|
||||
Ling uses gated MLA and KDA attention. KV cache stays tiny: 256k total
|
||||
context costs only ~2.5 GiB. This is why 256k context fits.
|
||||
|
||||
## Freeze Bug (2026-08-14) - Spec Path Deadlock
|
||||
|
||||
### Symptom
|
||||
|
||||
Under 2+ concurrent streams with speculative decoding, one stream hard-freezes
|
||||
mid-generation (zero output for minutes; a deadlock, not slowness). Freeze
|
||||
point varies (385-1588 tokens). Single streams never freeze.
|
||||
|
||||
### Isolated cause
|
||||
|
||||
- Not context exhaustion: `truncated = 0`, under 131072 ctx.
|
||||
- Not batch (3072 and 8192 both froze) and not log verbosity (0 and 4 both).
|
||||
- `parallel=2` + `draft-mtp,ngram-mod` -> freeze. SPEC OFF -> no freeze.
|
||||
- Culprit: the speculative-decode / context-checkpoint path in llama-server
|
||||
with multiple slots on the Vulkan backend. Known bug family: PR #24891,
|
||||
#24797 (server checkpoint fixes), #25144 (MTP draft crash), plus a
|
||||
ggml-vulkan hang-under-sustained-compute class.
|
||||
|
||||
### Resolution: SPEC OFF
|
||||
|
||||
`SPEC_TYPE=none`, `PARALLEL=2`. All concurrent streams complete (EOS).
|
||||
Speed cost: ~10-13 t/s under contention vs ~28-32 t/s single-stream with
|
||||
MTP. Fallback chain if spec-off regresses: A) `PARALLEL=1` keep MTP,
|
||||
B) switch back to Qwen3.6-27B (frank-default-large, MTP n_max 3).
|
||||
|
||||
## Files
|
||||
|
||||
- `scripts/frank-large-ling.service` - systemd unit.
|
||||
- `scripts/serve-ling-flash.sh` - serve script (fork binary, Vulkan,
|
||||
draft-mtp).
|
||||
- `scripts/frank-default-large.ling.env` - tunables.
|
||||
|
||||
Passwordless management is installed:
|
||||
`/usr/local/sbin/frank-restart-ling.sh` and `/etc/sudoers.d/frank-ling`.
|
||||
|
||||
## Switch Back (obsolete)
|
||||
|
||||
The switch-back commands below are obsolete. `frank-default-large` now
|
||||
runs the Qwen3.8-27B pipeline-parallel head. To restore a Ling trial,
|
||||
re-enable `frank-large-ling` from the repo unit
|
||||
(`scripts/frank-large-ling.service`) and serve script
|
||||
(`scripts/serve-ling-flash.sh`), then start it on a non-conflicting port.
|
||||
|
||||
## Caveats
|
||||
|
||||
- The `bailingmoe3` fork validates CPU and CUDA upstream. Vulkan was
|
||||
validated here (load + generation + concurrent streams) on 2026-08-14.
|
||||
- The tokenizer logs `special_eos_id is not in special_eog_ids` at load.
|
||||
It is a warning; generation stops correctly at EOS.
|
||||
- Physical RAM pressure stays low because the model fits in the UMA
|
||||
carve-out. Generation speed on this APU is below the Qwen3.6-27B MTP
|
||||
config (~7.7 vs ~15-19 t/s at 4k).
|
||||
- The heap is effectively full (98.7 of 111.2 GiB). Do not raise the
|
||||
context much without checking `--list-devices` free MiB.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[frank-default-large-benchmark]]
|
||||
- [[DFlash DSpark Speculative Decoding]]
|
||||
- [[deployment]]
|
||||
@@ -0,0 +1,10 @@
|
||||
# Ling-3.0-flash on Dankclaw — vLLM-ROCm Deployment Plan (never shipped)
|
||||
|
||||
This page was the plan for a vLLM-ROCm deployment of
|
||||
`jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62 GB) on
|
||||
Dankclaw. The deployment never shipped. The actual Ling work used the
|
||||
llama.cpp Q6_K fork trial instead: [[ling-3.0-flash-trial]].
|
||||
|
||||
The trial service (`frank-large-ling`) is retired and disabled under the
|
||||
boot policy (2026-08-17). `frank-default-large` is the Qwen3.8-27B PP
|
||||
head, not the Ling path. See git history for the full decision table.
|
||||
@@ -0,0 +1,158 @@
|
||||
> **SUPERSEDED 2026-08-17.** The Frank redo retired this topology.
|
||||
> See [frank-redo-2026-08-17.md](frank-redo-2026-08-17.md) and
|
||||
> [deployment.md](deployment.md) for the active configuration.
|
||||
|
||||
# Qwen3.8-27B Speculative-Decoding Matrix
|
||||
|
||||
This page records the speculative-decoding benchmark history and the
|
||||
production tuning for the Qwen3.8-27B service (`frank-default-large`),
|
||||
now a pipeline-parallel head with a Dankserver worker.
|
||||
|
||||
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
|
||||
|
||||
Current production, verified against the live process and journal
|
||||
(2026-08-17, load 02:06 + smoke tests). Topology detail:
|
||||
[[Pipeline Parallelism/Qwen3.8 PP Deployment]].
|
||||
|
||||
- Devices: `RPC0` (Dankserver RTX 3090) + `ROCm0` (Strix iGPU). The head
|
||||
passes no `--device` flag. Default selection puts RPC first and picks
|
||||
ROCm for the iGPU (HIP registers before Vulkan). The Vulkan0 side stays
|
||||
an option via `HEAD_BIN=.../build-vulkan-rpc`.
|
||||
- Model: `Qwen3.8-27B-Uncensored-Q6_K.gguf` (JonathanColetti, 22.43 GB,
|
||||
imatrix, 65 blocks, 866 tensors, NO bundled MTP head). The repo's
|
||||
standalone draft sidecar `Qwen3.8-27B-Uncensored-draft-Q8_0.gguf`
|
||||
(3.16 GB) is loaded via `--spec-draft-model`; it runs on ROCm0 only
|
||||
(1718 MiB, zero worker impact).
|
||||
- Spec: **MTP** (standalone draft) `--spec-draft-n-max 4` + the
|
||||
`--spec-default` ngram-mod fallback (48/64/24). Measured acceptance
|
||||
0.83 then 0.75, mean len 4.0-4.3, per-position ~0.86/0.71/0.71/0.71 —
|
||||
stronger than the old bundled head (0.739/0.554/0.427/0.340).
|
||||
- KV: K **q8_0** + V **f16** (mismatched). 3136 B/token per FA layer =
|
||||
784 MiB at 262144 (K 1088 + V 2048). RPC0 8624 MiB (11 FA), ROCm0
|
||||
3920 MiB (5 FA); total 12544 = 16 x 784. **Flash Attention stays
|
||||
enabled with the mismatch** (`resolve_fused_ops: Flash Attention
|
||||
enabled`) via the GGML_CUDA_FA_ALL_QUANTS rebuild of both binaries
|
||||
at 9b05354ec — see the PP doc.
|
||||
- Split: `--tensor-split 0.68,0.32` = 45 of 65 blocks (0-44) on the
|
||||
3090, 69% of the layer weight. The maximum: 46 blocks -> ~0.3 GiB
|
||||
margin; 48 blocks (f >= 0.7197) adds FA block 47 -> KV 9408 -> OOM by
|
||||
~0.85 GiB. Boundary math: the PP doc.
|
||||
- Context: pool 262144, 1 slot, `n_ctx_slot = 262144`. The pi
|
||||
`models.json` declares 262144 (contextWindow = maxTokens, house
|
||||
convention).
|
||||
- Worker steady state: 23854/24576 MiB used on the 3090, ~273 MiB free
|
||||
(tight but stable; all allocations happen at load).
|
||||
- Batch 8192 / ubatch 256, `--direct-io`, `--fit off`,
|
||||
`--gpu-layers all`.
|
||||
|
||||
## How MTP works on Qwen3.8 (bundled-head era)
|
||||
|
||||
The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside
|
||||
the model file. llama.cpp b10411+ auto-detects it and creates the draft
|
||||
context from the target model. Do not pass `--spec-draft-model` for those
|
||||
files. This is superseded: the production `Qwen3.8-27B-Uncensored-Q6_K.gguf`
|
||||
has NO bundled head, and the serve script passes the standalone draft
|
||||
sidecar with `--spec-draft-model` (see Production state above).
|
||||
|
||||
The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild
|
||||
on Dankclaw is b10433 (`9b05354ec`) with `GGML_HIP` + `gfx1151` + Vulkan.
|
||||
|
||||
## History (superseded configurations, 2026-08-14 to 08-16)
|
||||
|
||||
All superseded configs used `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB,
|
||||
bundled MTP) before the Q6_K swap. The ngram-mod preset survives as the
|
||||
fallback layer under MTP. The Vulkan0 verdict from this era applies to the
|
||||
standalone configuration only; the PP head runs ROCm0.
|
||||
|
||||
- **2026-08-14: bundled MTP** — `draft-mtp`, n_max 5, parallel 2, ctx
|
||||
524288 (2 x 262144). ~13.3 t/s single, ~8 t/s per slot under 2
|
||||
concurrent streams. No premature stop.
|
||||
- **2026-08-15: DSpark sidecar** — `magnitudedev/Qwen3.8-27B-DSpark-GGUF`
|
||||
(Q8_0, 1.36B, block_size 7), n_max 7, PARALLEL=1. Two concurrent 16k
|
||||
prose prompts crashed the ROCm backend with `ROCm error: an illegal
|
||||
memory access` in `common_speculative_impl_draft_dflash::draft` (status
|
||||
6/ABRT, systemd restart, ~503 during reload). DSpark reuses the DFlash
|
||||
block-diffusion machinery, which shares one draft context across
|
||||
parallel slots; concurrent block drafting faults in HIP stream sync.
|
||||
This is the same multi-slot bug class that retired `draft-dflash`
|
||||
(premature EOS). The crash shows on 16k concurrent streams, not on a
|
||||
67k pair — concurrency is the trigger, not size. Fix: PARALLEL=1
|
||||
(requests queue safely). Retired the same day.
|
||||
- **2026-08-15 to 08-16: `--spec-default` (ngram-mod), Vulkan0** — n_match
|
||||
24 / n_min 48 / n_max 64, draftless, PARALLEL=2 (2 x 32k parallel stream
|
||||
test passes). Chosen for parallel safety after DSpark; kept as the
|
||||
fallback under MTP in the current config.
|
||||
|
||||
## Benchmark Method
|
||||
|
||||
The matrix ran 54 configurations: 3 spec types (`draft-mtp,ngram-mod`,
|
||||
`draft-mtp`, `ngram-mod`) x `--spec-draft-n-max` 2..10 x
|
||||
`--spec-draft-p-min` {off, 0.8}. Each config loaded `llama-server`,
|
||||
then ran 3 generations. The tool is `scripts/spec-matrix-bench.sh`;
|
||||
analysis is `scripts/analyze-spec-matrix.py`.
|
||||
|
||||
## Matrix Caveat: Benchmark Prompt
|
||||
|
||||
The matrix used a repetitive 1100-token prompt
|
||||
(`scripts/bench_live.sh` style). MTP drafts that repeated sentence almost
|
||||
perfectly, which inflated the numbers for long drafts. The matrix reported
|
||||
`draft-mtp` n_max=10 at ~30 t/s. A realistic story prompt gave very
|
||||
different results:
|
||||
|
||||
| n_max | median t/s (realistic) |
|
||||
| ----- | ----------------------- |
|
||||
| 4 | 12.5 |
|
||||
| 5 | 13.3 (peak) |
|
||||
| 6 | 10.6 |
|
||||
| 7 | 10.1 |
|
||||
| 10 | 8.6 |
|
||||
| none (baseline) | 7.2 |
|
||||
|
||||
Long drafts collapse to ~18% acceptance on real writing. n_max=5 balances
|
||||
acceptance against draft cost.
|
||||
|
||||
## Premature Stop Investigation
|
||||
|
||||
Two failure patterns appeared during early production tests:
|
||||
|
||||
1. Speed collapse. `ubatch >= 512` reduced MTP acceptance to ~25% and
|
||||
generation to ~9 t/s. `ubatch 128` with `batch >= 2048` triggered a
|
||||
premature instant-EOS under 2 concurrent slots. `ubatch 256` avoids
|
||||
both: ~30 t/s single, concurrent-safe.
|
||||
2. Apparent premature-EOS. The repetitive benchmark prompt with `temp 1.0`
|
||||
sometimes stops after 1-2 tokens because the model legitimately emits an
|
||||
EOS on a wall of repeated text. This is sampling variance, not a
|
||||
corruption bug. Realistic prompts never stop early.
|
||||
|
||||
## High-Context Backend Comparison (2026-08-15)
|
||||
|
||||
Tool: `scripts/bench-high-ctx.sh <device> <ctx>`. Both runs at ctx 262144
|
||||
with a ~200k-token realistic-prose prompt, `--spec-default`, batch 8192 /
|
||||
ubatch 256, single slot, on scratch port 7892.
|
||||
|
||||
| Device | Prompt tokens | Prefill (t/s) | Generation (t/s) | Wall (single req) |
|
||||
| ------ | ------------- | ------------- | ---------------- | ----------------- |
|
||||
| ROCm0 | 202822 | **119.1** | 3.83 | **1770 s** |
|
||||
| Vulkan0 | 200235 | 106.3 | **5.68** | 1929 s |
|
||||
|
||||
Verdict: at very high context, Vulkan0 decodes **48% faster** (5.68 vs
|
||||
3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost
|
||||
(users stream tokens). This verdict applies to the standalone
|
||||
configuration; the PP head runs ROCm0 (HIP registers before Vulkan).
|
||||
Both `tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s).
|
||||
High-context decode collapses on both backends versus ~25 t/s at 16k ctx;
|
||||
that is the long-context attention penalty, not a spec artifact.
|
||||
|
||||
## Production Configuration
|
||||
|
||||
The live tunables live in `scripts/frank-default-large.env` (head) and
|
||||
`scripts/frank-pp-worker.env` (worker) in this repo. The Production
|
||||
state section at the top of this page records the current values: Q6_K
|
||||
model + standalone MTP draft, split 0.68/0.32, 262144 ctx, PARALLEL=1,
|
||||
q8_0/f16 KV.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[frank-default-large-benchmark]]
|
||||
- [[ling-3.0-flash-trial]]
|
||||
- [[deployment]]
|
||||
@@ -0,0 +1,249 @@
|
||||
# Qwen3.8-27B on Strix Halo: Speed Without Lowering the Quant
|
||||
|
||||
This page answers one question: **how do we make Qwen3.8-27B serve faster on
|
||||
Dankclaw (Strix Halo, Radeon 8060S) while keeping UD-Q8_K_XL?** It records
|
||||
the model's real architecture, the levers that are available, their evidence,
|
||||
and the test matrix planned for 2026-08-16 morning.
|
||||
|
||||
Related pages:
|
||||
- [Speculative-decoding matrix](qwen3.8-spec-matrix.md)
|
||||
- [Backend and spec benchmark](frank-default-large-benchmark.md)
|
||||
- [Dankclaw Improvements](Dankclaw%20Improvements.md) (OS-level items)
|
||||
- [Pipeline Parallelism: Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md)
|
||||
|
||||
## TL;DR — ranked levers
|
||||
|
||||
| # | Lever | Expected gain | Cost | Status |
|
||||
|---|-------|--------------|------|--------|
|
||||
| 1 | Pipeline-parallel to the RTX 3090 (RPC) | ~1.5–2× decode | 2.5 GbE link, both services stop for test | Kit ready, test 08-16 |
|
||||
| 2 | MTP `n_max` explicit 4–5 + Q4_0 draft head | +5–15% on the spec share | restart, 1.56 GB file | test 08-16 |
|
||||
| 3 | Plain `Q8_0` A/B vs UD-Q8_K_XL (same 8 bits) | dequant ALUs may differ | restart, 29 GB file | test 08-16 |
|
||||
| 4 | Prefill path: ROCm backend + tuned ubatch/threads-batch | 2–3× prefill for agentic turns | per-workload backend choice | partial data |
|
||||
| 5 | TTM `pages_limit` raised to 114 GiB | removes the hard VRAM ceiling; the full 262k f16 KV plus the Q8 AGENT payload, MTP draft, and mmproj fit in one slot | modprobe, set live 2026-08-25 | applied (29,883,536 pages) |
|
||||
| 6 | Vulkan ICD pinning + loader-layer disable | robustness; small | service env vars | test 08-16 |
|
||||
|
||||
Everything else (quantization downgrade, NVFP4, larger drafts) is documented
|
||||
below for completeness but is out of scope per the "no lowering" constraint.
|
||||
|
||||
## What the model actually is
|
||||
|
||||
Verified from the GGUF header (`/home/dank/models/Qwen3.8-27B-UD-Q8_K_XL.gguf`,
|
||||
dumped 2026-08-15) and the official model card.
|
||||
|
||||
- Architecture: `qwen35` / Qwen3.8-27B, **hybrid attention**.
|
||||
- 64 main layers + 1 MTP (nextn) layer = 65 GGUF blocks.
|
||||
- 48 Gated DeltaNet (linear/recurrent) layers + 16 gated full-attention
|
||||
layers, in a 3:1 pattern (`full_attention_interval = 4`).
|
||||
- Hidden 5120, FFN 17408, 24 Q heads / 4 KV heads, head dim 256.
|
||||
- Native context 262144 (extendable per the card; production runs 262k/slot).
|
||||
- GDN layers keep a **fixed-size** recurrent state (constant per layer, not
|
||||
per token). Only the 16 full-attention layers grow with context.
|
||||
- KV per token (full-attention part only): 16 layers × 2 (K,V) × 4 heads ×
|
||||
256 dim × 1 byte (q8_0) = **32 KiB/token**. At 65536 ctx ≈ 2 GiB; at
|
||||
262144 ctx ≈ 8 GiB.
|
||||
- Sampling defaults (card, thinking mode on by default): temperature 1.0,
|
||||
top-k 20, top-p 0.95, min-p 0.0. Production already matches.
|
||||
|
||||
Consequences of the hybrid design:
|
||||
|
||||
1. **Decode is cheap in memory.** Attention cost scales with only 16 layers;
|
||||
the 48 GDN layers are O(1) per token. Long-context decode degrades far
|
||||
slower than a dense transformer.
|
||||
2. **Prompt cache reuse is unavailable upstream.** Recurrent/SSM state
|
||||
cannot be prefix-cached the way attention KV can. Every agentic turn
|
||||
re-prefills the whole conversation. **Prefill is the hidden cost of
|
||||
agentic use**, and it is the dominant cost we have not yet optimized.
|
||||
3. **MTP is native and strong.** The model ships a nextn head; our matrix
|
||||
measured 13.3 t/s vs 7.2 t/s (no spec) at 16k on Vulkan.
|
||||
|
||||
## Current production state (baseline)
|
||||
|
||||
`frank-default-large` (Dankclaw, Vulkan0, 2026-08-15):
|
||||
|
||||
- Model: UD-Q8_K_XL, 31.5 GB, bundled MTP head.
|
||||
- `--spec-default` (ngram-mod) with draft-mtp first in the chain.
|
||||
- 2 parallel slots × 262144 ctx, KV q8_0, batch 8192, ubatch 256,
|
||||
threads 12, `-dio` off, `--fit off`.
|
||||
- Measured: ~25 t/s at 16k ctx; 5.68 t/s at 262k ctx; prefill 302 t/s @64k
|
||||
(Vulkan) vs 410 t/s @64k (ROCm, same build).
|
||||
- GPU: 28 GiB of 96 GiB GTT used. **74 GiB sits idle.**
|
||||
|
||||
## The levers, in detail
|
||||
|
||||
### 1. Pipeline-parallel to the RTX 3090 (biggest win)
|
||||
|
||||
The 3090 on Dankserver is idle (serves a 22 GB MoE small-model service with
|
||||
~2.4 GB free). llama.cpp RPC splits the model by **layers**: each device
|
||||
runs its layer range, hidden states cross the 2.5 GbE ethernet once per
|
||||
token. The per-token transfer is ~41 KiB (5120-wide f32 hidden state, both
|
||||
directions); at ~0.5 ms LAN RTT the bandwidth cost is negligible — the split
|
||||
is latency-bound, not bandwidth-bound (see
|
||||
[Ethernet Bandwidth Math](Pipeline%20Parallelism/Ethernet%20Bandwidth%20Math.md)).
|
||||
|
||||
Why this is a real speedup and not just "two GPUs": the 3090 does
|
||||
matmuls roughly 2–3× faster per layer than the 8060S (936 vs 512 GB/s
|
||||
memory, 80 vs 40 CUs). A 50/50 layer split balances per-layer latency, so
|
||||
each token step takes ≈ max(half-on-3090, half-on-Strix) + one RTT instead
|
||||
of all-64-layers-on-Strix.
|
||||
|
||||
Design decisions (verified in b10433 source, 2026-08-15):
|
||||
|
||||
- **Head on Dankclaw, worker on Dankserver.** The 31.5 GB model file stays
|
||||
on the head; the worker receives its tensor share over the wire at load
|
||||
(with `-c` it caches the tensors locally after the first load).
|
||||
- **Head must NOT pass `--device`.** When `params.devices` is set,
|
||||
`llama_prepare_model_devices` builds the device list from exactly those
|
||||
devices and the RPC device is silently dropped (`--rpc` becomes a no-op).
|
||||
Default device selection inserts RPC servers **at the front** of the list.
|
||||
- **The local iGPU side runs on ROCm, not Vulkan.** Default selection walks
|
||||
registered backends in order; HIP registers before Vulkan, and the iGPU
|
||||
dedup rule (PR #23897 workaround) keeps the first backend's device.
|
||||
Verified live 2026-08-15: device list `[RPC0, ROCm0]`, Vulkan0 never
|
||||
listed. Consequence: the Strix half runs on the ROCm backend — better
|
||||
prefill, ~3% slower decode than Vulkan0 (measured on this box).
|
||||
A second head build (`/home/dank/llama.cpp/build-vulkan-rpc`, HIP off,
|
||||
Vulkan on, RPC on) was built the same night for an A/B; with it the
|
||||
device list becomes `[RPC0, Vulkan0]`.
|
||||
- **Default split is proportional to *reported free memory*, which is wrong
|
||||
here.** The Strix reports up to 96 GiB (GTT ceiling) vs the 3090's 24 GB,
|
||||
so the default lands ~20/80 in favor of the slow device (verified with a
|
||||
4B smoke test: 4 of 36 layers on the 3090). We pin `--tensor-split 1,1`
|
||||
(50/50) explicitly.
|
||||
- **`-dio` (direct I/O) on the head.** Large model + RPC load hangs without
|
||||
it (visorcraft Strix finding; matches our 28 GiB load-stall history).
|
||||
- **Version skew hangs the handshake.** Worker built at the identical
|
||||
commit (9b05354ec / b10433) in a worktree
|
||||
`/home/dank/llama-cpp-rpc` on Dankserver — production b10121 build
|
||||
untouched.
|
||||
|
||||
Smoke test (2026-08-15, 4B model, 0.2/0.8 split, production untouched):
|
||||
worker VRAM rose ~819 MiB (real split), 95 t/s prefill, 58.8 t/s decode on
|
||||
the tiny model. RTT on the link: ~0.3–0.5 ms.
|
||||
|
||||
Full design, scripts, and morning runbook:
|
||||
[Qwen3.8 PP Deployment](Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md).
|
||||
|
||||
### 2. Speculative-decoding tuning (same bits, free tokens)
|
||||
|
||||
- **MTP `n_max`.** Production leaves `--spec-draft-n-max` unset. Our matrix
|
||||
(Vulkan, 16k) found 13.3 t/s at the tested setting vs 7.2 without spec;
|
||||
the published Strix tuning curve peaks at n_max 3–4 (30.3 t/s at 4 vs
|
||||
28.9 at 3 vs 29.9 at 5). The PP env sets **4** explicitly; the matrix
|
||||
re-tests 3/4/5 on the real split.
|
||||
- **Q4_0 draft head.** The published Strix card measured a standalone
|
||||
Q4_0 MTP head ~6–10% faster than Q8_0 for the same acceptance (the draft
|
||||
head is a 248k-vocab output projection; cheaper dequant, same accuracy).
|
||||
The bundled head inside UD-Q8_K_XL is Q8-class; the standalone
|
||||
`ggml-org/mtp-Qwen3.8-27B-Q4_0.gguf` (1.56 GB) is being downloaded for
|
||||
an A/B via `--spec-draft-model`.
|
||||
- **Ngram-mod fallback stays.** It is the only spec type that survived
|
||||
parallel load (DSpark crashed ROCm, DFlash corrupts multi-slot).
|
||||
|
||||
### 3. Same-bit quant A/B: UD-Q8_K_XL vs plain Q8_0
|
||||
|
||||
Both are 8-bit. UD (unsloth "dynamic") mixes block sizes per tensor; plain
|
||||
`Q8_0` uses uniform 32-element blocks. Dequant of a smaller block costs
|
||||
more ALUs per element; on a memory-bound GPU the difference is usually
|
||||
small, but Strix's iGPU has tight ALU headroom. `ggml-org/Qwen3.8-27B-GGUF`
|
||||
ships a Q8_0 (29.12 GB). One restart to test; if it wins, same quality at
|
||||
zero quality cost.
|
||||
|
||||
### 4. Prefill is the agentic bottleneck — route it right
|
||||
|
||||
Every agentic turn re-prefills the whole conversation (no prefix caching for
|
||||
hybrid state, upstream). Levers:
|
||||
|
||||
- **Backend choice per workload.** Measured @64k on this box (b10433):
|
||||
ROCm 410 t/s vs Vulkan 302 t/s prefill; Vulkan 25.1 vs ROCm 24.3 t/s
|
||||
decode. For chat-heavy use: Vulkan. For agentic/batch-heavy use: ROCm.
|
||||
The PP head (ROCm side) gets this for free.
|
||||
- **ubatch/threads-batch.** Production runs ubatch 256. Larger ubatches
|
||||
(512/1024) fill the iGPU better during prefill; `--threads-batch` for
|
||||
the CPU-bound tail. Untested at prefill scale on this box — matrix item.
|
||||
- **KV q4_0** for very long contexts: halves the 16-layer KV (8 GiB → 4 GiB
|
||||
at 262k). Only worth it if long-ctx attention is the bottleneck; q8_0 is
|
||||
the safe default.
|
||||
|
||||
### 5. OS level (from the Improvements doc, re-verified 2026-08-15)
|
||||
|
||||
- **TTM `pages_limit` is now 114 GiB (29,883,536 pages)** — applied 2026-08-25.
|
||||
Written live to `/sys/module/ttm/parameters/pages_limit` (no reboot required)
|
||||
and persisted in `/etc/modprobe.d/ttm.conf`. The full 262k f16 KV cache plus
|
||||
the ~28 GB Q8 AGENT payload, MTP draft, and mmproj now fit in one slot. The
|
||||
load-stall risk from the old 15 GiB cap is resolved. Note: 96 GiB is the
|
||||
driver-reported VRAM carve-out (VGM/SVM pool); the TTM pages_limit is the
|
||||
kernel page map that was the real ceiling.
|
||||
- `iommu=pt` — verified present on the kernel cmdline. Done.
|
||||
- **Vulkan ICD pinning** — `radeon_icd.json` exists; no `VK_ICD_FILENAMES`
|
||||
or loader-layer env is set. Pinning the radeon ICD and disabling loader
|
||||
layers (`VK_LOADER_LAYERS_DISABLE=all`) is the recommended Strix
|
||||
hardening; testable as service env vars, zero risk.
|
||||
- **Power** — GPU DPM is already `performance`, forced level `auto`. The
|
||||
`power_profile` sysfs file assumed in the Improvements doc does not exist
|
||||
on this machine (checked 2026-08-15); that item is N/A here.
|
||||
- Threads: 12 of 16 cores in production. Community reports 12–16 are
|
||||
equivalent; leave at 12.
|
||||
|
||||
### 6. llama.cpp version watch
|
||||
|
||||
b10433 (2026-08-15) is adequate. Through b10451 there are no
|
||||
spec-decoding fixes that help us; DFlash's multi-slot corruption is still
|
||||
open upstream (fix #27117 pending). If that lands, DFlash/DSpark become
|
||||
re-testable candidates (DSpark was the best single-machine draft before the
|
||||
ROCm crash). No rebuild planned until then.
|
||||
|
||||
## Quantization landscape (reference)
|
||||
|
||||
The constraint is to keep UD-Q8_K_XL. This section records what exists so
|
||||
future decisions are informed.
|
||||
|
||||
| Format | Size | Bits/weight | Notes |
|
||||
|--------|------|-------------|-------|
|
||||
| **UD-Q8_K_XL (unsloth)** | 31.5 GB | ~8.5 (mixed) | **Current.** Dynamic mixed-block; bundled MTP head; our baseline. |
|
||||
| Q8_0 (ggml-org, bartowski) | 29.1 GB | 8.0 | Uniform 32-block. Same-bit A/B candidate (lever 3). |
|
||||
| UD-Q6_K_XL (unsloth) | ~22 GB | ~6.5 | Below the 8-bit floor; documented, not recommended under the constraint. |
|
||||
| Q6_K | ~21.5 GB | 6.6 | Same tier as above. |
|
||||
| **ROCmFP4 "STRIX" (ROCmFPX fork)** | 13.3–13.8 GB | 4.25–4.39 | Strix-specific NVFP4-style format, ROCm backend only, requires a forked llama.cpp. Published: 30.3 t/s vs 10.7 for Q4_K_M (2.83×) at 8k with MTP n_max≈4; PPL ≈ Q4_K_M. **This is the "fast path" — and it is a quant downgrade plus a fork dependency that breaks the Vulkan production path.** Kept here as the escape hatch if the 8-bit speed target cannot be met. |
|
||||
| NVFP4 (Blackwell) | ~14 GB | 4.5 | Hardware path for RTX 50-series. Irrelevant to 3090/Strix. |
|
||||
|
||||
If the 8-bit PP result (lever 1) is disappointing, the decision order is:
|
||||
push PP split/latency tuning → ROCmFP4 as a *second* service (A/B, not a
|
||||
replacement) → revisit the constraint with Frank.
|
||||
|
||||
## Morning test matrix (2026-08-16)
|
||||
|
||||
Policy: stop both production services, run tests, restore. Each config =
|
||||
one head restart (~10 min load incl. weight stream; ~2 min with the worker
|
||||
tensor cache warm).
|
||||
|
||||
| # | Config | Measures |
|
||||
|---|--------|----------|
|
||||
| A | PP split 1,1, ROCm head, MTP n_max=4 + ngram | baseline PP |
|
||||
| B | A + `SPEC_NONE=1` | raw (no spec) PP speed |
|
||||
| C | A but head build `build-vulkan-rpc` (Vulkan side) | backend A/B |
|
||||
| D | A + `--spec-draft-model` Q4_0 MTP file | draft-head A/B |
|
||||
| E | A with n_max 3 and 5 | n_max curve on the split |
|
||||
| F | A, plain Q8_0 model file | same-bit A/B |
|
||||
| G | PP default proportional split (no `--tensor-split`) | confirms the 20/80 handicap |
|
||||
| Ref | production frank-default-large (Vulkan, 2 slots) | the bar to beat |
|
||||
|
||||
Per config: `pp-test.sh bench 4` (4k prompt / 256 out) and `bench 16`
|
||||
(16k / 256). Report: decode t/s (server print_timing), prefill t/s, MTP
|
||||
acceptance, worker VRAM, head GTT usage.
|
||||
|
||||
Success bar: **≥ 30 t/s at 4k and ≥ 20 t/s at 16k decode** (vs 25 t/s
|
||||
production @16k) with the same 8-bit model.
|
||||
|
||||
## Sources
|
||||
|
||||
- GGUF header dump of the production model (this machine, 2026-08-15).
|
||||
- Qwen3.8-27B model card + `ggml-org/Qwen3.8-27B-GGUF` /
|
||||
`unsloth/Qwen3.8-27B-GGUF` repo listings (Hugging Face).
|
||||
- kingjones `Qwen3.8-27B-ROCmFP4-GGUF` Strix tuning card (MTP n_max curve,
|
||||
Q4_0 vs Q8_0 draft head, ROCmFP4 numbers).
|
||||
- visorcraft `strix-halo-llm-perf` (Vulkan tg / ROCm prefill, ICD pinning,
|
||||
`-dio` for RPC).
|
||||
- llama.cpp b10433 source: `src/llama.cpp` (device selection),
|
||||
`common/speculative.cpp` (MTP/ngram-mod), `ggml/src/ggml-rpc` (protocol).
|
||||
- Local benchmarks: [frank-default-large-benchmark](frank-default-large-benchmark.md),
|
||||
[qwen3.8-spec-matrix](qwen3.8-spec-matrix.md).
|
||||
@@ -0,0 +1,56 @@
|
||||
---
|
||||
type: benchmark plan
|
||||
title: frank-default-large ubatch sweep
|
||||
description: Standardized ubatch sweep for the active kingjones Qwen3.8 ROCmFPX deployment.
|
||||
tags:
|
||||
- llama.cpp
|
||||
- rocm
|
||||
- benchmark
|
||||
- ubatch
|
||||
timestamp: 2026-09-01
|
||||
---
|
||||
|
||||
# frank-default-large ubatch sweep
|
||||
|
||||
This benchmark uses the active `frank-default-large` model on Dankclaw with
|
||||
`llama-bench`. It does not use `llama-server`, HTTP requests, or server-log
|
||||
scraping. Stop the production service before running it and start the service
|
||||
again afterward.
|
||||
|
||||
The old custom harnesses are retained in `.AGENTTRASH/bench-scripts-20260901/`
|
||||
for auditability only. Their results are invalid and must not be used.
|
||||
|
||||
## Sweep
|
||||
|
||||
The requested points are `256, 512, 768, 1024, 1280, 1536, 1792, 2048,
|
||||
3072, 4096` at 8K context, with `batch` equal to `ubatch` at every point.
|
||||
Each point runs three 128-token repetitions by default.
|
||||
|
||||
Run on Dankclaw from the repository checkout:
|
||||
|
||||
```bash
|
||||
sudo systemctl stop frank-default-large
|
||||
./scripts/bench-ubatch.sh
|
||||
sudo systemctl start frank-default-large
|
||||
```
|
||||
|
||||
The script uses the active Q6 AGENT model, an 8K prompt workload, equal
|
||||
`batch` and `ubatch` values, and `llama-bench` JSONL output. The sweep is:
|
||||
```text
|
||||
256, 512, 768, 1024, 1280, 1536, 1792, 2048, 3072, 4096
|
||||
```
|
||||
|
||||
The GPU is shared with live workloads; do not run with the production service
|
||||
up.
|
||||
|
||||
## Results
|
||||
|
||||
No valid results are recorded here. All results from the retired custom
|
||||
harnesses are explicitly discarded. Record the raw `llama-bench` JSONL output
|
||||
from the new harness here after a clean run.
|
||||
|
||||
## See also
|
||||
|
||||
- [LLM fleet index](index.md)
|
||||
- [Deployment](deployment.md)
|
||||
- [Active kingjones runbook](kingjones-strix-mtp.md)
|
||||
@@ -0,0 +1,217 @@
|
||||
# Research: vLLM Parameters for Specified Configuration
|
||||
|
||||
## Summary
|
||||
vLLM v0.26.0 (latest stable, released July 25, 2026) supports DFlash speculative decoding with the `--spec-method dflash` and `--spec-model` flags. KV cache dtype `q8_0` is **not supported** — only fp8 variants and turboquant options are available. vLLM does not have a dedicated MoE expert CPU offload flag; general CPU offloading via `--cpu-offload-gb` is the closest option. The `--max-num-seqs` flag is confirmed. Podman GPU access requires `--gpus all` with the NVIDIA container toolkit.
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. Speculative Decoding with DFlash Draft Model
|
||||
|
||||
**Flag syntax:**
|
||||
```bash
|
||||
--spec-method dflash \
|
||||
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
|
||||
--spec-tokens 8
|
||||
```
|
||||
|
||||
- `--spec-method dflash` — DFlash is a first-class speculative method in vLLM v0.26.0. Defined in source as `DFlashModelTypes = Literal["dflash"]` in `vllm/config/speculative.py`.
|
||||
- `--spec-model z-lab/Qwen3.6-35B-A3B-DFlash` — specifies the HuggingFace path to the DFlash draft model.
|
||||
- `--spec-tokens N` — number of speculative tokens (block size). The DFlash model card recommends block size 8 for higher-concurrency serving, or 16 for concurrency-1 throughput.
|
||||
- DFlash support landed in v0.25.0 (PR #46770, #46853) and was enhanced in v0.26.0 with hybrid SWA + full attention drafters (PR #47914).
|
||||
- The DFlash draft model is **not standalone** — it must be paired with its target model (`Qwen/Qwen3.6-35B-A3B`).
|
||||
|
||||
**Additional speculative config options** (from `vllm/engine/arg_utils.py`):
|
||||
- `--speculative-config '{"key": "value"}'` — full JSON config for advanced settings
|
||||
- `--spec-tokens` — shorthand for `num_speculative_tokens`
|
||||
|
||||
**Caveat:** DFlash may require a non-causal-capable attention backend for the draft model. Set `--speculative-config '{"attention_backend": "FLASH_ATTN"}'` if needed.
|
||||
|
||||
[Source: vLLM v0.26.0 release notes](https://github.com/vllm-project/vllm/releases/tag/v0.26.0)
|
||||
[Source: vllm/config/speculative.py source](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py)
|
||||
[Source: z-lab/Qwen3.6-35B-A3B-DFlash model card](https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash)
|
||||
|
||||
---
|
||||
|
||||
### 2. KV Cache Dtype — q8_0 NOT Supported
|
||||
|
||||
**`q8_0` is NOT a valid KV cache dtype in vLLM.** The supported `CacheDType` values (from `vllm/config/cache.py`):
|
||||
|
||||
| Value | Description |
|
||||
|-------|-------------|
|
||||
| `auto` | Use model data type (default) |
|
||||
| `float16` | FP16 KV cache |
|
||||
| `bfloat16` | BF16 KV cache |
|
||||
| `fp8` / `fp8_e4m3` | FP8 E4M3 (CUDA 11.8+) |
|
||||
| `fp8_e5m2` | FP8 E5M2 (CUDA 11.8+) |
|
||||
| `fp8_inc` | FP8 for Intel Gaudi (HPU) |
|
||||
| `fp8_ds_mla` | FP8 for DeepSeek MLA |
|
||||
| `turboquant_k8v4` | TurboQuant 8-bit K, 4-bit V |
|
||||
| `turboquant_4bit_nc` | TurboQuant 4-bit non-causal |
|
||||
| `turboquant_k3v4_nc` | TurboQuant 3-bit K, 4-bit V |
|
||||
| `turboquant_3bit_nc` | TurboQuant 3-bit non-causal |
|
||||
| `int4_per_token_head` | INT4 per-token-head |
|
||||
| `int8_per_token_head` | INT8 per-token-head |
|
||||
| `fp8_per_token_head` | FP8 per-token-head |
|
||||
| `nvfp4` | NVFP4 quantization |
|
||||
| `nvfp4_4over6` | NVFP4 4over6 layout |
|
||||
|
||||
**Closest alternatives to q8_0:**
|
||||
- `fp8` (alias for `fp8_e4m3`) — 8-bit FP8, closest to q8_0 in precision
|
||||
- `int8_per_token_head` — 8-bit integer with per-token-head scales
|
||||
- `turboquant_k8v4` — 8-bit K cache, 4-bit V cache
|
||||
|
||||
**CLI flag:** `--kv-cache-dtype fp8` (or any of the above values)
|
||||
|
||||
[Source: vllm/config/cache.py source](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py)
|
||||
|
||||
---
|
||||
|
||||
### 3. CPU Expert Offloading for MoE Models
|
||||
|
||||
**vLLM does NOT have a dedicated `--cpu-offload-gate` or `--expert-offload-cpu` flag.**
|
||||
|
||||
Available MoE-related flags (from `vllm/engine/arg_utils.py`):
|
||||
- `--enable-expert-parallel` — enables expert parallelism (distributes experts across GPUs)
|
||||
- `--expert-placement-strategy` — controls expert placement strategy
|
||||
- `--moe-backend` — selects the MoE kernel backend
|
||||
|
||||
Available CPU offloading flags:
|
||||
- `--cpu-offload-gb N` — general CPU offloading of N GB of model parameters (UVA offloading)
|
||||
- `--cpu-offload-params "pattern"` — offload specific parameter patterns to CPU
|
||||
- `--kv-cache-memory-bytes` — controls KV cache memory allocation
|
||||
|
||||
**How to offload MoE experts to CPU:**
|
||||
Use `--cpu-offload-gb` with `--cpu-offload-params "experts.*"` to offload expert weights:
|
||||
```bash
|
||||
--cpu-offload-gb 32 \
|
||||
--cpu-offload-params "experts.*"
|
||||
```
|
||||
|
||||
This uses vLLM's UVA (Unified Virtual Addressing) offloading mechanism. The experts remain on CPU and are transferred to GPU on-demand during MoE routing.
|
||||
|
||||
**Caveat:** CPU offloading MoE experts will significantly impact latency due to PCIe transfer overhead. This is a memory-saving measure, not a performance optimization.
|
||||
|
||||
[Source: vllm/engine/arg_utils.py source](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py)
|
||||
|
||||
---
|
||||
|
||||
### 4. max-num-seqs Flag — Confirmed
|
||||
|
||||
**Flag:** `--max-num-seqs N`
|
||||
|
||||
- Confirmed in `vllm/engine/arg_utils.py`: `max_num_seqs: int | None = None`
|
||||
- Limits the maximum number of concurrent sequences (requests) in the scheduler
|
||||
- When `None` (default), vLLM auto-calculates based on available KV cache memory
|
||||
- Setting this constrains concurrency regardless of available resources
|
||||
|
||||
[Source: vllm/engine/arg_utils.py source](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py)
|
||||
|
||||
---
|
||||
|
||||
### 5. Podman + vLLM + CUDA GPU Access
|
||||
|
||||
**Exact podman run flags for GPU access:**
|
||||
|
||||
```bash
|
||||
podman run -it \
|
||||
--gpus all \
|
||||
--device /dev/nvidia0 \
|
||||
--device /dev/nvidiactl \
|
||||
--device /dev/nvidia-uvm \
|
||||
--device /dev/nvidia-uvm-tools \
|
||||
--device /dev/nvidia-modeset \
|
||||
--cap-add=SYS_ADMIN \
|
||||
--security-opt apparmor=unconfined \
|
||||
-v /path/to/hf-cache:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:latest \
|
||||
vllm serve <model>
|
||||
```
|
||||
|
||||
**Minimal form (with NVIDIA Container Toolkit installed):**
|
||||
```bash
|
||||
podman run -it --gpus all vllm/vllm-openai:latest vllm serve <model>
|
||||
```
|
||||
|
||||
**Key flags:**
|
||||
- `--gpus all` — exposes all GPUs to the container (requires `nvidia-container-toolkit`)
|
||||
- `--device /dev/nvidia*` — manual device passthrough (fallback if `--gpus` doesn't work)
|
||||
- `--cap-add=SYS_ADMIN` — needed for NVIDIA driver access in some Podman configurations
|
||||
- `--security-opt apparmor=unconfined` — disables AppArmor which can block GPU access
|
||||
|
||||
**Prerequisites:**
|
||||
- `nvidia-container-toolkit` must be installed on the host
|
||||
- Podman's `containers.conf` should have `nvidia` in the `devices` section, or use `--gpus all`
|
||||
- NVIDIA driver must be installed on the host (not in the container)
|
||||
|
||||
[Source: Podman GPU documentation](https://podman.io/docs/gpu-support)
|
||||
|
||||
---
|
||||
|
||||
### 6. Latest Stable vLLM Version
|
||||
|
||||
**Version: v0.26.0** (released July 25, 2026)
|
||||
|
||||
- 411 commits from 212 contributors (61 new)
|
||||
- Key features: DFlash speculative decoding, Model Runner V2 as default, DeepSeek-V4 optimizations, KV offloading improvements
|
||||
- Available on PyPI: `pip install vllm==0.26.0`
|
||||
- Docker image: `vllm/vllm-openai:latest`
|
||||
|
||||
[Source: PyPI vllm page](https://pypi.org/project/vllm/)
|
||||
[Source: GitHub releases](https://github.com/vllm-project/vllm/releases)
|
||||
|
||||
---
|
||||
|
||||
## Complete Example Command
|
||||
|
||||
Putting it all together for a Qwen3.6-35B-A3B model with DFlash speculative decoding:
|
||||
|
||||
```bash
|
||||
podman run -it \
|
||||
--gpus all \
|
||||
--device /dev/nvidia0 \
|
||||
--device /dev/nvidiactl \
|
||||
--device /dev/nvidia-uvm \
|
||||
--device /dev/nvidia-uvm-tools \
|
||||
--cap-add=SYS_ADMIN \
|
||||
--security-opt apparmor=unconfined \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:latest \
|
||||
vllm serve Qwen/Qwen3.6-35B-A3B \
|
||||
--spec-method dflash \
|
||||
--spec-model z-lab/Qwen3.6-35B-A3B-DFlash \
|
||||
--spec-tokens 8 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--max-num-seqs 32 \
|
||||
--tensor-parallel-size 1 \
|
||||
--gpu-memory-utilization 0.9
|
||||
```
|
||||
|
||||
## Sources
|
||||
|
||||
### Kept
|
||||
- [vLLM v0.26.0 Release Notes](https://github.com/vllm-project/vllm/releases/tag/v0.26.0) — authoritative release information
|
||||
- [vllm/engine/arg_utils.py](https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py) — complete CLI argument definitions
|
||||
- [vllm/config/cache.py](https://github.com/vllm-project/vllm/blob/main/vllm/config/cache.py) — CacheDType enum and CacheConfig
|
||||
- [vllm/config/speculative.py](https://github.com/vllm-project/vllm/blob/main/vllm/config/speculative.py) — SpeculativeConfig and method types
|
||||
- [z-lab/Qwen3.6-35B-A3B-DFlash](https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash) — DFlash model card with usage examples
|
||||
- [PyPI vllm](https://pypi.org/project/vllm/) — version history
|
||||
|
||||
### Dropped
|
||||
- Generic vLLM overview pages (GitHub README, docs landing page) — no specific parameter details
|
||||
- Wikipedia vLLM article — not authoritative for CLI flags
|
||||
|
||||
## Gaps
|
||||
|
||||
1. **DFlash attention backend requirements** — The exact attention backend configuration needed for DFlash with Qwen3.6-35B-A3B on specific GPU architectures (e.g., H100 vs A100) was not fully verified. May need `--attention-backend flash-attn` or `flashinfer`.
|
||||
|
||||
2. **MoE CPU offload performance impact** — No benchmarks found for CPU-offloaded MoE experts in vLLM. Latency impact is expected to be significant but unquantified.
|
||||
|
||||
3. **Podman GPU on specific distros** — Exact Podman GPU flags may vary by distribution (RHEL vs. Fedora vs. Arch). The flags above are the most universal configuration.
|
||||
|
||||
4. **DFlash + MoE compatibility** — Whether DFlash speculative decoding works correctly with MoE target models (like Qwen3.6-35B-A3B) was not explicitly tested in the release notes.
|
||||
|
||||
## Residual Risks
|
||||
|
||||
- DFlash may require specific attention backends not available on all GPU architectures
|
||||
- CPU offloading MoE experts will add significant latency; test thoroughly before production use
|
||||
- `q8_0` is definitively not supported; using an unsupported dtype will cause startup failure
|
||||
@@ -0,0 +1,38 @@
|
||||
# Backblaze Integration
|
||||
|
||||
Off-site backup plan for the bind-mounted data on both hosts. **Deferred**: research only, not implemented yet.
|
||||
|
||||
## Goal
|
||||
|
||||
Back up critical data to Backblaze B2:
|
||||
- Calibre library (Dankserver)
|
||||
- ComfyUI models (Dankclaw)
|
||||
- Gitea data, repos, DB (Dankclaw)
|
||||
- Homepage/stack configs (this repo — already versioned in git)
|
||||
|
||||
## Approach (research direction)
|
||||
|
||||
| Option | Fit |
|
||||
|--------|-----|
|
||||
| `restic` + B2 | Best fit: dedupe, encryption, snapshots, low cost. Single static binary, works with podman host. |
|
||||
| `borg` + borgbase | Alternative; borgbase is more expensive per TB than B2. |
|
||||
| rclone + B2 | Good for file sync, no snapshot/dedupe semantics. |
|
||||
| podman volume driver for B2 | Niche, immature. Not recommended. |
|
||||
|
||||
## Shape of the Implementation (future)
|
||||
|
||||
- `scripts/backup.sh` on each host, run by systemd timer or cron.
|
||||
- `restic -r b2:bucket:path backup /opt/...`
|
||||
- `RESTIC_PASSWORD` in the host `.env`, never in the repo.
|
||||
- Restore procedure documented once the backup job exists.
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Create a B2 bucket + application key.
|
||||
2. Draft `scripts/backup.sh` and a systemd timer unit.
|
||||
3. Run a test restore.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[Storage Policy]] — what gets backed up
|
||||
- [[Gitea]] — the existing backup procedure (tar / pg_dump)
|
||||
@@ -0,0 +1,106 @@
|
||||
# ComfyUI on ROCm and Vulkan
|
||||
|
||||
Research into whether ComfyUI can use GPU acceleration — not just CPU — on the AMD Strix Halo APU (gfx1151) in Dankclaw. Verified on live hardware 2026-08-14. This page corrects the persistent "ROCm unavailable on Strix Halo" claim.
|
||||
|
||||
## TL;DR
|
||||
|
||||
| Path | Works on gfx1151? | Evidence |
|
||||
|------|-------------------|----------|
|
||||
| pytorch.org ROCm wheels | **No** | wheels are built for `gfx906/gfx90a/gfx942/gfx1100` etc. — no gfx1151 kernels |
|
||||
| Official `rocm/comfyui` image | **No** | built with `PYTORCH_ROCM_ARCH=gfx942;gfx950` — datacenter archs only |
|
||||
| AMD gfx1151-tagged PyTorch build | **Yes** | `torch 2.9.1+rocm7.12.0rc1` → `avail: True`, `Radeon 8060S`, cap `(11,5)` |
|
||||
| Vulkan (torch.backends.vulkan) | **No** | compiled out of standard wheels; experimental/mobile even when present |
|
||||
| ComfyUI `--cpu` | **Yes (slow)** | current stack config |
|
||||
|
||||
## The core insight
|
||||
|
||||
PyTorch's ROCm support is not one thing. The wheels you install for poke at torch, the wheels AMD builds for specific APUs are different, and the kernels compiled in are fixed at build time via `PYTORCH_ROCM_ARCH`.
|
||||
|
||||
Standard wheels (pytorch.org and the `rocm/comfyui` image) target **CDNA datacenter archs** (`gfx942`/`gfx950`) or the older common set. **gfx1151 (Strix Halo / Radeon 8060S)** is not included, so `torch.cuda.is_available()` returns `False` and everything falls back to CPU.
|
||||
|
||||
AMD does publish gfx1151 builds — but you have to pick the right image.
|
||||
|
||||
## Live verification (Dankclaw, 2026-08-14)
|
||||
|
||||
### Fails: official rocm/comfyui image
|
||||
|
||||
```bash
|
||||
podman pull docker.io/rocm/comfyui:comfyui-0.18.2.amd0_rocm7.2.0_ubuntu24.04
|
||||
podman run --rm --device=/dev/kfd --device=/dev/dri --entrypoint python <img> -c "
|
||||
import torch
|
||||
print(torch.__version__) # 2.10.0a0+git…
|
||||
print(torch.cuda.is_available()) # False
|
||||
print(torch.cuda.get_arch_list()) # []"
|
||||
|
||||
# Env in that image:
|
||||
# PYTORCH_ROCM_ARCH=gfx942;gfx950
|
||||
# ROCM_PATH=/opt/rocm-7.2.0
|
||||
```
|
||||
|
||||
The image bundles ROCm 7.2 and compiles kernels only for `gfx942;gfx950`. No gfx1151. `HSA_OVERRIDE_GFX_VERSION` does not help because the kernels simply are not compiled in.
|
||||
|
||||
The official ComfyUI ROCm image is 68 GB and unusable on Strix Halo. It was pulled on Dankclaw for this test; remove it to reclaim space.
|
||||
|
||||
### Works: AMD gfx1151-tagged PyTorch build
|
||||
|
||||
The `rocm/vllm` image exists in two flavors. The one tagged `gfx1151` works on Strix Halo:
|
||||
|
||||
```bash
|
||||
podman run --rm \
|
||||
--device=/dev/kfd --device=/dev/dri \
|
||||
--entrypoint python \
|
||||
docker.io/rocm/vllm:rocm7.12.0_gfx1151_ubuntu24.04_py3.12_pytorch_2.9.1_vllm_0.16.0 \
|
||||
-c "
|
||||
import torch
|
||||
print(torch.__version__) # 2.9.1+rocm7.12.0rc1
|
||||
print(torch.version.hip) # 7.12.60610
|
||||
print(torch.cuda.is_available()) # True
|
||||
print(torch.cuda.get_device_name(0)) # Radeon 8060S Graphics
|
||||
print(torch.cuda.get_device_capability(0)) # (11, 5) = gfx1151
|
||||
"
|
||||
|
||||
# Env in that image:
|
||||
# PYTORCH_ROCM_ARCH=gfx1151
|
||||
```
|
||||
|
||||
Key facts:
|
||||
- `avail: True`, device `Radeon 8060S`, capability `(11, 5)` = gfx1151.
|
||||
- Device access requires passing `/dev/kfd` and `/dev/dri` to the container (and `--group-add video`/`realtime` or a user with those groups).
|
||||
- ROCm HIP runtime vars matter: this image has HIP 7.12; the older `rocm/vllm:rocm7.14.0_rdna_*` (48.6 GB) targets the newer "RDNA" set.
|
||||
|
||||
## What this means for ComfyUI
|
||||
|
||||
ComfyUI is a PyTorch app. If you give it a PyTorch wheel with gfx1151 kernels — the AMD `/opt/python` build used by the working vLLM image, or a custom build with `PYTORCH_ROCM_ARCH=gfx1151` — ComfyUI **will** use the GPU via ROCm. The blocker is not ComfyUI; it is shipping it in a container that contains a gfx1151-enabled torch.
|
||||
|
||||
Three ways forward (way 1 applied 2026-08-19):
|
||||
|
||||
1. **Base a ComfyUI image on the AMD gfx1151 torch build.** Take `rocm/vllm:...gfx1151...pytorch_2.9.1...` (or AMD's PyTorch wheel index) and install ComfyUI on top. This is the direct path to GPU ComfyUI. **Applied:** the running `comfyui-rocm:gfx1151` image does exactly this (see `dankclaw/Containerfile.comfyui`).
|
||||
2. **Build PyTorch for gfx1151** (`export PYTORCH_ROCM_ARCH=gfx1151` before build). Slow build, but a self-contained wheel afterwards.
|
||||
3. **Use a ggml-family runtime instead** — stable-diffusion.cpp / Lemonade's `sd.cpp` backend ship their own gfx1151 HIP kernels and work today (see [[Lemonade and sd.cpp]]).
|
||||
|
||||
## Vulkan
|
||||
|
||||
- PyTorch's Vulkan backend (`torch.backends.vulkan`) is **compiled out of standard Linux wheels**. The installed `torch 2.11/2.13` on Dankclaw reports `hasattr(torch.backends, 'vulkan')` → `False`.
|
||||
- Even when built in, PyTorch Vulkan is a mobile-oriented experimental backend, not a desktop inference path, and ComfyUI exposes no `--vulkan` flag.
|
||||
- The system has a working Vulkan driver (RADV, `libvulkan_radeon.so` + `radeon_icd.json`), so Vulkan apps work — but ComfyUI/PyTorch cannot consume it here.
|
||||
- Conclusion: **Vulkan is a dead end for ComfyUI.** If GPU is wanted, ROCm is the realistic path; a ggml-family engine (sd.cpp/Lemonade) is the pragmatic one.
|
||||
|
||||
## Consequence for this repo
|
||||
|
||||
**Applied 2026-08-19.** `dankclaw/compose.yaml` now builds `comfyui-rocm:gfx1151` from `dankclaw/Containerfile.comfyui` (based on the AMD gfx1151 base), passes `/dev/kfd` + `/dev/dri`, adds the `video` group, and drops `--cpu`. The container runs on the Radeon 8060S (gfx1151) and is boot-persistent via a user systemd unit. See [[ComfyUI]] for the running config.
|
||||
|
||||
## Cleanup note
|
||||
|
||||
These images were pulled on Dankclaw during this research:
|
||||
|
||||
```bash
|
||||
podman rmi docker.io/rocm/comfyui:comfyui-0.18.2.amd0_rocm7.2.0_ubuntu24.04 # 68 GB, unusable on gfx1151
|
||||
podman rmi docker.io/rocm/vllm:rocm7.12.0_gfx1151_... # 32.5 GB, keep if pursuing ComfyUI ROCm
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [[ComfyUI]] — the service page (CPU config)
|
||||
- [[Lemonade and sd.cpp]] — the ggml-family alternative with working gfx1151 kernels
|
||||
- [[Observability]], [[Dankclaw Stack]]
|
||||
- KB: [[ComfyUI]] (tools) (the "ROCm unavailable" claims here are now qualified)
|
||||
@@ -0,0 +1,49 @@
|
||||
# Lemonade and sd.cpp
|
||||
|
||||
Why Lemonade Server can do image generation on Strix Halo when ComfyUI cannot. Research from the Lemonade source repo (`lemonade-sdk/lemonade`, cloned 2026-08-14) and live verification.
|
||||
|
||||
## The short answer
|
||||
|
||||
Lemonade does not run ComfyUI. Its image-generation backend is a wrapped **stable-diffusion.cpp** (`leejet/stable-diffusion.cpp`) subprocess. sd.cpp is a ggml-family native C++ runtime that ships its own HIP kernels — and those kernels **include gfx1151**. ComfyUI runs on PyTorch, whose standard wheels omit gfx1151. Same hardware, different runtime, different result.
|
||||
|
||||
## From the Lemonade source
|
||||
|
||||
`src/cpp/server/backends/sdcpp/sdcpp_server.cpp`:
|
||||
|
||||
- Image gen is `SDServer`, a `WrappedServer` around a prebuilt sd.cpp binary.
|
||||
- Downloads `sd-<version>-bin-Linux-Ubuntu-24.04-x86_64-<backend>.zip` from `leejet/stable-diffusion.cpp` releases.
|
||||
- Backends: `cpu, cuda, metal, rocm, vulkan`.
|
||||
- ROCm asset is `...-rocm-<version>.zip`; Vulkan is `...-vulkan.zip`.
|
||||
|
||||
Backend support matrix (`serverAppConfigs-docs/dev/backends-reference.md`):
|
||||
|
||||
| Recipe | Backend | Device families |
|
||||
|--------|---------|-----------------|
|
||||
| `sd-cpp` | rocm | amd_gpu (gfx103X, gfx110X, **gfx1150, gfx1151, gfx1152**, gfx120X) |
|
||||
| `sd-cpp` | vulkan | amd_gpu; cpu; nvidia_gpu |
|
||||
| `thenoise` | rocm | amd_gpu (gfx1150, gfx1151, gfx1152) — experimental |
|
||||
|
||||
**gfx1151 is explicitly supported** by sd.cpp's ROCm build. There is also a dedicated experimental `thenoise` ROCm backend targeting exactly the Strix Halo archs.
|
||||
|
||||
## Engine comparison
|
||||
|
||||
| | ComfyUI | Lemonade (sd.cpp) |
|
||||
|---|---|---|
|
||||
| Runtime | PyTorch | native C++ (ggml family) |
|
||||
| AMD path | needs PyTorch ROCm wheel with gfx1151 kernels — not in standard wheels | ships own HIP kernels, gfx1151 included |
|
||||
| On Strix Halo | CPU only (today) | ROCm GPU, or Vulkan fallback |
|
||||
| Interface | node graph web UI | server, OpenAI-style API, GUI/CLI |
|
||||
|
||||
This mirrors llama.cpp on the same box: ggml-family runtimes ship their own gfx1151 HIP kernels and run on ROCm; PyTorch does not.
|
||||
|
||||
## Relevance
|
||||
|
||||
- If GPU image generation on Dankclaw matters, **Lemonade (or bare sd.cpp) is the working path today**.
|
||||
- ComfyUI's node graph is the cost of switching: Lemonade is not ComfyUI.
|
||||
- See [[ComfyUI ROCm and Vulkan]] for the PyTorch-side analysis showing a gfx1151-enabled torch would also make ComfyUI work.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[ComfyUI ROCm and Vulkan]] — the PyTorch side of this question
|
||||
- [[ComfyUI]] — current CPU config
|
||||
- [[Dankclaw Stack]]
|
||||
@@ -0,0 +1,222 @@
|
||||
---
|
||||
type: research
|
||||
title: Llama.cpp Code Diff
|
||||
description: Code-level diff of mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp with a focus on quantization code, ggml type numbers, and the differences that matter for Dankserver (RTX 3090) and Dankclaw (Strix Halo).
|
||||
tags:
|
||||
- llama.cpp
|
||||
- quantization
|
||||
- ggml
|
||||
- ROCmFPX
|
||||
- ik_llama
|
||||
- code-diff
|
||||
timestamp: 2026-09-01
|
||||
---
|
||||
|
||||
# Llama.cpp Code Diff
|
||||
|
||||
Code-level diff of the three runtimes, focused on **quantization code** and the
|
||||
differences that affect our two servers. Positioning overview is in
|
||||
[[Llama Cpp Runtimes]].
|
||||
|
||||
## 1. Quantization file map
|
||||
|
||||
Where quant lives in each runtime (paths relative to the repo root):
|
||||
|
||||
| File / path | Mainline | ROCmFPX | ik_llama.cpp |
|
||||
|-------------|----------|---------|--------------|
|
||||
| `ggml/include/ggml.h` (`enum ggml_type`) | FP16/FP32/BF16, legacy `Q*_0`, K-quant types, `IQ*_` | **adds** `Q4_0_ROCMFP4` … `Q2_0_ROCMFPX`, `Q4_0_ROCMI4`, `Turbo3_0`, `Turbo4_0` (enums **100–108**) | IQK non-linear (`IQ4_NL` etc.), Trellis `IQ*_KT`, MXFP4, Bitnet, `Q8_KV` |
|
||||
| `ggml/src/ggml-quants.*` | K/I/legacy dequant, row-interleaved 3-bit | `block_rocmfp4` unpack + E4M3 scaling, IU4 WMMA dispatch | `iqk/*` fused quant-gemms, IQK row-interleave, Trellis decode |
|
||||
| `tools/quantize/quantize.c` | all mainline quants + `--imatrix` | ROCmFPX quantizer (`STRIX`/`FAST`/`_AGENT` presets) | IQK/Trellis/MXFP4 packers, `--cache-type-k/v` + Hadamard |
|
||||
| `gguf.cpp` / `gguf-io.c` | standard ftype table | **custom ftypes** 100–106 (FP4), 110–113 (FPX), 118 (I4) | IQK/Trellis ftype tags |
|
||||
| KV cache | `f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1` (needs FlashAttention) | **TurboQuant** `Turbo3_0`/`Turbo4_0`/`V` | `Q8_KV`, IQK/Trellis/MXFP4 reused in `-ctk`, `--k-cache-hadamard`, `--v-cache-hadamard` |
|
||||
|
||||
Two number spaces in ROCmFPX (authoritative `ggml/include/ggml.h`, sha
|
||||
`c49ebdbd`):
|
||||
- **`ggml_type` enum** (tensor storage): `Q4_0_ROCMFP4`=100, `Q4_0_ROCMFP4_FAST`=101,
|
||||
`Q6_0_ROCMFPX`=102, `Q8_0_ROCMFPX`=103, `Q3_0_ROCMFPX`=104, `Turbo3_0`=105,
|
||||
`Turbo4_0`=106, `Q2_0_ROCMFPX`=107, `Q4_0_ROCMI4`=108, count 109.
|
||||
- **`ggml_ftype`** (GGUF model-file): ROCmFP4 family `100–106` (`STRIX`=105,
|
||||
`STRIX_LEAN`=106), ROCmFPX family `110–113` (FP6=110, FP8=111, FP3=112,
|
||||
FP2=113), `ROCmI4`=118.
|
||||
|
||||
## 2. Comparison table
|
||||
|
||||
| Runtime | Quant formats | Granularity | Custom ggml types | MoE | Attention accel | Hybrid GPU/CPU offload | Build backends |
|
||||
|---------|---------------|-------------|-------------------|-----|-----------------|------------------------|----------------|
|
||||
| Mainline | K/I/legacy/TQ, imatrix, row-RI 3-bit | grouped/per-block + double-quant | none | native `--cpu-moe` | FlashAttention (multi-backend) | `--n-gpu-layers`, partial offload | CPU/CUDA/Vulkan/Metal/SYCL/HIP-ROCm/CANN |
|
||||
| charlie12345/ROCmFPX | ROCmFP2/3/4/6/8, ROCmI4, TurboQuant KV | block + E4M3 codebook | **yes (100–108)** | MTP draft head (bundled) | **MTEA** + TurboQuant FA (custom) | Strix-tuned MTP profile | HIP/ROCm + Vulkan (**CUDA off**), CPU ref |
|
||||
| ik_llama.cpp | IQ* non-linear, improved K, IQK row-RI, Trellis, MXFP4, Q8_KV, Bitnet | grouped + non-linear + row-interleaved | IQK/Trellis/MXFP4/Bitnet tags | **fused MoE**, `-ooae`, `--fit`, `-ot`, GDN | **FlashMLA** (DeepSeek MLA) | **finest-grained** override-tensor / cpu-moe / fit | CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm(weak) |
|
||||
|
||||
## 3. Code-difference narrative
|
||||
|
||||
**What each fork ADDED that mainline lacks:**
|
||||
- **ROCmFPX** — ~10 new `ggml_type` entries, the `block_rocmfp4` packed-nibble
|
||||
format, `*_ROCMFPX`/`*_ROCMI4` ftypes, tensor-aware presets (`*_AGENT`,
|
||||
`*_COHERENT`, `*_STRIX`), `block_rocmfp4` dequant + dispatch kernels,
|
||||
TurboQuant KV types, MTEA attention kernel. Mainline does not know these
|
||||
tensor types. It is a *fork* that adds a new backend: custom tensor
|
||||
registration + HIP/ROCm + Vulkan kernels + CPU reference. **CUDA is disabled.**
|
||||
- **ik_llama.cpp** — new code under `ggml/src/iqk/` (IQK GEMM + row-interleave),
|
||||
CUDA IQK flash-attn kernels (`GGML_IQK_FA_ALL_QUANTS=ON`), fused MoE
|
||||
(`fnn_up`+`fnn_gate`), fused delta-net (GDN) SSM ops, `flash_fwd_mla` for
|
||||
DeepSeek MLA/DSA, and hybrid-offload controls (`-ot/--override-tensor`,
|
||||
`--cpu-moe`, `-ooae/--offload-only-active-experts`, `-ser/--smart-expert-reduction`,
|
||||
`--fit`, `-op/--offload-policy`, split-mode graph). Run-time quant repack
|
||||
(`-rtr`). **Additive** — GGUF remains compatible.
|
||||
|
||||
**What mainline has that the forks lag on:**
|
||||
- **ik_llama.cpp**: architecture baseline is **Aug-2024** — newest
|
||||
architectures arrive late (PRs move fast, but the core is ~1.5y behind).
|
||||
- **ROCmFPX**: **AMD-first, experimental**; narrower, must rebuild/re-quant per
|
||||
fork line, CUDA ecosystem (the fleet's 3090) is unreachable.
|
||||
|
||||
**The one caveat that changes a build (ik_llama.cpp + `-rtr`):**
|
||||
K-quants have **no CUDA row-interleaved implementation**. With `-rtr`, every
|
||||
CPU-resident K-quant tensor is repacked to row-interleaved and then **run on CPU**
|
||||
even where GPU would be faster → lower prompt-processing speed. For a MoE that
|
||||
keeps experts on CPU (`--cpu-moe`), this is a real hit. Prefer IQK/R4/Trellis/
|
||||
MXFP4 variants, or skip `-rtr` and place tensors with `--fit`/`-ot`.
|
||||
|
||||
## 4. Server-specific benefits
|
||||
|
||||
### Dankserver — NVIDIA RTX 3090 (24 GB), 62 GB RAM — 35B MoE (Ornith-1.5-35B-A3B CRACK)
|
||||
- The winning lever for a **low-VRAM CUDA MoE** is **expert-aware offload + fused
|
||||
MoE**, which is ik_llama.cpp's core strength:
|
||||
- `-ooae/--offload-only-active-experts` — with ~8 active experts this cuts
|
||||
RAM→VRAM traffic dramatically vs shipping all experts.
|
||||
- `-fmoe/--fused-moe`, GDN fusion (the CRACK model is a hybrid GatedDeltaNet +
|
||||
MoE), `--fit`, `-ot/--override-tensor`, `--cpu-moe`.
|
||||
- **Quant pick for the 3090:** `Q4_K_M`/`Q5_K_M` on mainline, or an **IQK/R4 /
|
||||
Trellis / MXFP4** variant on ik_llama.cpp that has a **CUDA** path. Avoid raw
|
||||
K-quant + `-rtr`. Enable FlashAttention; `GGML_CUDA_FORCE_MMQ=1`;
|
||||
`ngram-mod` drafts (MoEs want long drafts).
|
||||
- ⚠️ **Expert-count discrepancy:** the CRACK GGUF README says **256 experts / 8
|
||||
active**; the fleet brief cites ~36. Verify `n_expert` / `n_expert_used` from
|
||||
the GGUF before finalizing the offload recipe — both point at expert-aware
|
||||
offload being the right tool.
|
||||
- **Mainline only?** Fine: `--cpu-moe` + `Q4_K_M` + FlashAttention + `ngram-mod`.
|
||||
- **ROCmFPX?** No — ROCm-only formats, CUDA disabled on the builds.
|
||||
|
||||
### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified), ROCm 7.2.4 — Qwen3.8-27B ROCmFP4 STRIX MTP
|
||||
- **ROCmFPX is the natural fit** and stays the choice: native IU4 (ROCmI4),
|
||||
ROCmFP4 weight formats, MTP draft head, TurboQuant KV, 262K / 1 slot. The
|
||||
production `build-promptforge` `llama-server` already runs there.
|
||||
- **ik_llama.cpp on Strix?** ROCm is its **weakest** backend (gfx1151 not
|
||||
validated, k-quant-no-hip-row-interleave), and we'd lose the bespoke MTP
|
||||
draft-head speedup. Higher-risk change; only with a small soak-test first.
|
||||
- **Mainline on Strix** works for correctness with standard quants but loses FPX,
|
||||
IU4, TurboQuant, and MTEA.
|
||||
|
||||
## 6. KV cache quant table
|
||||
|
||||
Bit-density-ordered comparison of everything usable in the **KV cache** (`-ctk`
|
||||
key / `-ctv` value, and draft `-ctkd`/`-ctvd`). Order is high → low bits-per-
|
||||
value. Columns mark **origin** (`*` = mainline-provided) and whether the type is
|
||||
**KV-native** (designed for the cache) or **weight-family reused for KV** (a
|
||||
model-weight quant the runtime also accepts in `-ctk`/`-ctv`).
|
||||
|
||||
**How bpw is read here.** `bpw = ggml_type_size × 8 / QK`, computed from the
|
||||
exact block structs in `ggml.h` / `ggml-common.h` (`QK_K = 256`, fp16 scale).
|
||||
This is the true in-RAM cost per value, so it is what sets KV-cache size and
|
||||
quantize/dequant throughput. A footnote marks the few types whose **nominal**
|
||||
figure (older 1-byte-scale layout / data-bits-only) diverges from this exact
|
||||
number; the table uses the exact number throughout.
|
||||
|
||||
### 6a. All KV-usable types, by bit density (interleaved)
|
||||
|
||||
| bpw | Type | Origin | KV-native | Notes |
|
||||
|-----|------|--------|-----------|-------|
|
||||
| 32.0 | f32 | mainline * | yes | full precision |
|
||||
| 16.0 | f16 | mainline * | yes | default |
|
||||
| 16.0 | bf16 | mainline * | yes | |
|
||||
| 9.25 | q8_K | mainline * | weight | rare in KV; per-16-block sums; ik Q8_KV is the fast KV version |
|
||||
| 8.13 | **q8_0** | mainline * | yes | |
|
||||
| 8.06 | **q8_KV** | ik * | yes | the dedicated **fast KV cache** type (`-ctk q8_KV`); Q8_0-shaped, 1 shared scale |
|
||||
| 6.63 | iq6_K | ik | weight→KV | IQK family |
|
||||
| 6.56 | q6_K | mainline * | weight→KV | |
|
||||
| 6.00 | **q5_1** | mainline * | yes | (nominal 5.06) |
|
||||
| 5.50 | **q5_0** | mainline * | yes | (nominal 5.06) |
|
||||
| 5.50 | q5_K | mainline * | weight→KV | |
|
||||
| 5.50 | iq5_K | ik | weight→KV | IQK family |
|
||||
| 4.50 | **q4_0** | mainline * | yes | (nominal 4.25) |
|
||||
| 4.50 | **q4_1** | mainline * | yes | (nominal 4.25) |
|
||||
| 4.50 | **iq4_nl** | mainline * | yes | (nominal 4.0) |
|
||||
| 4.50 | iq4_K | ik | weight→KV | IQK family |
|
||||
| 4.5 | **Turbo4_0** | ROCmFPX * | yes | TurboQuant; ~4.5 |
|
||||
| 4.25 | iq4_ks | ik | weight→KV | IQK subsample |
|
||||
| 4.25 | iq4_xs | mainline * | weight→KV | popular *weight* format, also usable in KV |
|
||||
| 4.25 | mxfp4 | ik * | weight→KV | |
|
||||
| 3.5 | **Turbo3_0** | ROCmFPX * | yes | TurboQuant; ~3.5 |
|
||||
| 3.44 | iq3_K | ik | weight→KV | IQK family |
|
||||
| 3.44 | iq3_s | mainline * | weight→KV | |
|
||||
| 2.69 | iq2_kl | ik | weight→KV | IQK subsample |
|
||||
| 2.38 | iq2_K | ik | weight→KV | IQK family |
|
||||
| 2.19 | iq2_ks | ik | weight→KV | IQK subsample |
|
||||
| 2.13 | iq2_kt | ik * | weight→KV | **Trellis** (lattice) |
|
||||
| 1.75 | iq1_kt | ik * | weight→KV | **Trellis** (lattice) |
|
||||
|
||||
### 6b. Origin legend
|
||||
|
||||
- **Mainline KV-native** (`-ctk`/`-ctv` = exactly these 9, from `common/arg.cpp`
|
||||
`kv_cache_types`): `f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`. Marked
|
||||
with `*` and `yes` in the KV-native column.
|
||||
- **ROCmFPX KV-native**: `turbo3_0, turbo4_0` — TurboQuant's dedicated KV formats
|
||||
(add a third `V` type for values only). Marked `*` here.
|
||||
- **ik additions**: `q8_KV` (fast KV, dedicated), plus the weight-family quants
|
||||
ik lets you reuse in `-ctk`/`-ctv` — the **IQK** family (`iq*_K`, plus the
|
||||
`IKS`/`IKL` subsamples) and the **Trellis** lattice family (`iq*_KT`), and
|
||||
`mxfp4`. ik's `-ctk` accepts a broader set than mainline's.
|
||||
|
||||
### 6c. Trellis, IQK, MXFP4 — where they sit
|
||||
|
||||
- **Trellis (`IQ*_KT`)** — ik's lattice-code quants, mirror the `IQ1..IQ4` bit
|
||||
ladder: `iq1_kt` 1.75 → `iq2_kt` 2.13 → `iq3_kt` 3.13 → `iq4_kt` 4.00. Highest
|
||||
quality per bit of the family; lowest-density entries in the whole table.
|
||||
- **IQK (`IQ*_K`)** — ik's improved K-quants: `iq2_k` 2.38, `iq3_k` 3.44,
|
||||
`iq4_k` 4.50, `iq5_k` 5.50, `iq6_k` 6.63, plus subsamples `iq2_kl` 2.69 /
|
||||
`iq2_ks` 2.19 / `iq4_ks` 4.25. They sit alongside the mainline K-family
|
||||
(iq2_k ≈ iq2_s, iq4_k ≈ q4_K) but with ik's row-interleave + fused GEMM.
|
||||
- **MXFP4** (`mxfp4`, 4.25) — the mixed-precision FP4 cache quant; ik's entry.
|
||||
|
||||
### 6d. Hadamard is a transform, not a row
|
||||
|
||||
`--k-cache-hadamard` / `--v-cache-hadamard` (ik) is a **structured transform
|
||||
applied on top of a base KV quant**, not its own bit-density row. It spreads
|
||||
quantization error, improving accuracy for the underlying type; ik's guidance is
|
||||
**best below Q6_0**. Row-interleaving (the `R4`/`R8` variants) is the same idea —
|
||||
a packing transform on the K-family — and is likewise a modifier, not a quant.
|
||||
|
||||
### 6e. Server notes
|
||||
|
||||
- **Dankclaw (production, ROCmFP4 STRIX MTP):** KV stays on `turbo4_0` /
|
||||
`turbo3_0` (TurboQuant) — the runtime's own formats. ik's IQK/Trellis path on
|
||||
Strix is higher-risk (see §4).
|
||||
- **Dankserver (3090 MoE):** `-ctk q8_KV` or `q4_K`/`q5_K` (mainline) with
|
||||
FlashAttention + KV-quant; on ik, `iq4_k`/`iq4_kt`/`iq2_k` if a CUDA path is
|
||||
confirmed for that type. Keep FlashAttention and the KV quant version in lockstep
|
||||
— they are coupled (a quant change that drops FA off the path changes KV math).
|
||||
|
||||
## 5. Recommendations for our fleet
|
||||
|
||||
- **Dankclaw:** keep **charlie12345/ROCmFPX** (already production-tuned for
|
||||
gfx1151). If you want more quant variety on Strix, explore the ROCmFP2 (`Q2_0_ROCMFPX`,
|
||||
2.5 bpw, S40 codebook) preview — but do not port to ik_llama.cpp as primary.
|
||||
- **Dankserver:** **try ik_llama.cpp** (CUDA-90 + AVX-512 build) for fused-MoE +
|
||||
`offload-only-active-experts`, staying off raw K-quant `-rtr`. If that is too
|
||||
much surface, stay on **mainline** with `--cpu-moe` + `Q4_K_M` + FlashAttention
|
||||
+ `ngram-mod`. Neither box needs ROCmFP4.
|
||||
- **Common:** pin a commit per fleet line; re-test after any quant/KV/context
|
||||
change (FlashAttention + KV-quant are coupled).
|
||||
|
||||
## See Also
|
||||
- [[Llama Cpp Runtimes]] — positioning + quantization overview
|
||||
- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
|
||||
- [[kingjones-strix-mtp]] — the ROCmFP4 model on Dankclaw
|
||||
- [[Quantization by Hardware Architecture]] — quant vs hardware
|
||||
|
||||
## Citations
|
||||
- Mainline quant + backends — https://github.com/ggml-org/llama.cpp ·
|
||||
tools/quantize/README.md · serverAppConfigs-docs/speculative.md
|
||||
- ROCmFPX enum (ggml_type 100–108) — https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h
|
||||
- ik_llama.cpp quant/MoE/FlashMLA/backends — https://github.com/ikawrakow/ik_llama.cpp (README, serverAppConfigs-docs/parameters.md, serverAppConfigs-docs/build.md) · DeepWiki · discussions#562
|
||||
- ik_llama.cpp `-rtr` K-quant trap & backend-bandwidth caveat — https://github.com/ikawrakow/ik_llama.cpp/blob/main/README.md
|
||||
@@ -0,0 +1,141 @@
|
||||
---
|
||||
type: research
|
||||
title: Llama.cpp Runtimes
|
||||
description: Compared mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp — positioning, quantization formats, code-level differences, and fit for Dankserver (RTX 3090) and Dankclaw (Strix Halo).
|
||||
tags:
|
||||
- llama.cpp
|
||||
- quantization
|
||||
- ROCmFPX
|
||||
- ik_llama
|
||||
- inference
|
||||
- amd
|
||||
- nvidia
|
||||
timestamp: 2026-09-01
|
||||
---
|
||||
|
||||
# Llama.cpp Runtimes
|
||||
|
||||
Comparison of three `llama.cpp`-derived inference runtimes used (or usable) on
|
||||
the two homelab boxes: **mainline** `ggml-org/llama.cpp`,
|
||||
**charlie12345/ROCmFPX** (what Dankclaw runs), and **ik_llama.cpp**
|
||||
(ikawrakow). The goal is to know, for each server, which quant formats are on
|
||||
the table and which runtime best fits the hardware.
|
||||
|
||||
Full code-level quantification is in [[Llama Cpp Code Diff]]. This page is the
|
||||
positioning + quantization summary.
|
||||
|
||||
## Which runtime is which
|
||||
|
||||
| Runtime | Repo | What it is | Primary backends | For our boxes |
|
||||
|---------|------|-----------|------------------|---------------|
|
||||
| Mainline | [`ggml-org/llama.cpp`](https://github.com/ggml-org/llama.cpp) | Canonical, general-purpose, one codebase for every backend | CPU, CUDA, Vulkan, Metal, SYCL, HIP/ROCm, CANN, OpenVINO | Baseline both boxes; the portable default |
|
||||
| charlie12345/ROCmFPX | [`charlie12345/ROCmFPX`](https://github.com/charlie12345/ROCmFPX) (+ reference fork [`rocmfp4-llama`](https://github.com/charlie12345/rocmfp4-llama), clean pkg [`rocmfp4`](https://github.com/charlie12345/rocmfp4)) | Experimental AMD-focused GGUF *weight* quant family layered on llama.cpp | HIP/ROCm + Vulkan (CUDA **disabled**), CPU reference | **Dankclaw** (native Strix Halo) |
|
||||
| ik_llama.cpp | [`ikawrakow/ik_llama.cpp`](https://github.com/ikawrakow/ik_llama.cpp) | Fork focused on SOTA quants + CPU / hybrid GPU-CPU performance | CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm (weak) | **Dankserver** candidate (3090 + MoE) |
|
||||
|
||||
Key lineage notes:
|
||||
- Mainline moved from `ggerganov/llama.cpp` to the `ggml-org` org. Rolling
|
||||
commit hashes (the fleet pins `b10121`) plus newer `vX.Y.Z` tags. Pin a
|
||||
commit for reproducibility.
|
||||
- ROCmFPX is an *experimental feature family* on the canonical `main` branch;
|
||||
three GitHub surfaces exist (reference fork, clean patch, pre-patched tree).
|
||||
It is **AMD-only** — CUDA is compiled out.
|
||||
- ik_llama.cpp's core is an **August-2024 llama.cpp snapshot** maintained with
|
||||
a rolling set of PRs (MLA, fused MoE, row-interleaved quants, MTP, DFlash all
|
||||
appeared here first, later partially merged upstream). The feature surface is
|
||||
current; the architecture baseline is ~1.5 years behind mainline.
|
||||
|
||||
## Quantization summary
|
||||
|
||||
### Mainline (what the fleet already ships)
|
||||
- **Base:** FP32 / FP16 / BF16.
|
||||
- **K-quants** (modern default, two-level block + double-quant): `Q2_K` …
|
||||
`Q6_K`, with `_S`/`_M`/`_L` precision mixes. `Q4_K_M` is the popular default;
|
||||
`Q6_K` is "almost lossless."
|
||||
- **i-quants** (importance-matrix, non-linear, best quality/size at low bits but
|
||||
slower decode): `IQ4_NL`, `IQ4_XS`, `IQ3_*`, `IQ2_*`. Row-interleaved 3-bit
|
||||
family (IQ3_XXS/S/XS) reduces the quality penalty of whole-row quant.
|
||||
- **Legacy:** `Q*_0`/`Q*_1` (fast, simple), ternary `TQ1_0`.
|
||||
- **KV cache quant:** `f32/f16/bf16/q8_0/q4_0/iq4_nl/q5_0/q5_1`; quantizing KV
|
||||
requires FlashAttention (`-fa on`).
|
||||
- Speculative decoding native (MTP, EAGLE3, DFlash, DSpark, `ngram-mod`).
|
||||
|
||||
### ROCmFPX (Dankclaw's AMD weight formats)
|
||||
- Integer GGUF **weight** formats only this fork reads: **ROCmFP2/3/4/6/8** and
|
||||
**ROCmI4 (W4A4 IU4 WMMA)**. These are custom `ggml` tensor types
|
||||
(**`ggml_type` 100–108**) with matching GGUF **`ftype`** (ROCmFP4 family
|
||||
`100–106`; ROCmFPX family `110–113`; `ROCmI4` = `118`). Stock llama.cpp,
|
||||
Ollama, and LM Studio **cannot load them**.
|
||||
- Flagship **ROCmFP4** (`block_rocmfp4`): 32 weights/block, two E4M3 scale
|
||||
bytes, ~18 bytes/block, ~4.5 bpw, E2M1 nibbles + E4M3 codebook up to 10
|
||||
units. Variants: `STRIX` / `STRIX_LEAN`, `FAST`, `COHERENT`, plus `*_AGENT`
|
||||
/ `*_STRIX_MTP` draft-head presets.
|
||||
- **TurboQuant KV:** `Turbo3_0` (3.5 bpw) / `Turbo4_0` (4.5 bpw) + `TurboQuant V`
|
||||
(f16 K) on the production model.
|
||||
- **ROCmI4 (W4A4):** the gfx1151 native IU4 4-bit WMMA path (`-DGGML_HIP_ROCMI4_W4A4=ON`);
|
||||
lossy activation quant, exact int8 MMQ.
|
||||
|
||||
### ik_llama.cpp (SOTA quants + CPU/hybrid)
|
||||
- **IQ\*** non-linear family; `IQ4_NL` gives lower perplexity than mainline
|
||||
`Q4_K_M` at the same size. Improved **K-quants** and the signature
|
||||
**row-interleaved IQK / R4 / R8** packing (activates AVX-512/VNNI/NEON
|
||||
matmul paths that vanilla llama.cpp leaves cold).
|
||||
- **Trellis** quants (`IQ*_KT`), **MXFP4**, **Q8_KV** fast KV type, first-class
|
||||
**Bitnet**.
|
||||
- ⚠️ **K-quant / `-rtr` trap (critical for the 3090):** K-quants have **no CUDA
|
||||
row-interleaved path**; `-rtr` (run-time repack) forces CPU-resident K-quant
|
||||
tensors to CPU matmul even when GPU would be faster. Prefer IQK/R4/Trellis/
|
||||
MXFP4 variants, or skip `-rtr` and let `--fit`/`-ot` place experts.
|
||||
|
||||
## Fit for our servers
|
||||
|
||||
### Dankserver — NVIDIA RTX 3090 (24 GB VRAM), 62 GB RAM, CUDA
|
||||
Serves **Ornith-1.5-35B-A3B CRACK**, a 35B **MoE** (~36 experts; the CRACK
|
||||
GGUF README lists 256/8-active — verify `n_expert` from the GGUF).
|
||||
- **Mainline** is the natural baseline: `Q4_K_M`/`Q5_K_M`, FlashAttention to
|
||||
shrink KV, `GGML_CUDA_FORCE_MMQ=1` (default on RTX 30xx, lower VRAM),
|
||||
`--cpu-moe` to spill experts to the 62 GB RAM, and `ngram-mod` drafts (MoEs
|
||||
want long drafts).
|
||||
- **ik_llama.cpp** is the more compelling *upgrade* here: fused MoE (`-fmoe`),
|
||||
**offload-only-active-experts** (`-ooae`) which — with only ~8 active experts —
|
||||
sharply cuts RAM→VRAM traffic, `--fit`, smart-expert-reduction (`-ser`), GDN
|
||||
fusion for the SSM path, and MTP. Requires the CUDA-90 + AVX-512 build and care
|
||||
with the `-rtr` K-quant trap.
|
||||
- **ROCmFPX** does **not** help a CUDA box (ROCm-only formats, CUDA disabled).
|
||||
The 3090's lever is CUDA quant + FastMTP, not ROCmFPX.
|
||||
|
||||
### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified VRAM), ROCm 7.2.4
|
||||
Serves **kingjones Qwen3.8-27B ROCmFP4 STRIX MTP** (the production build).
|
||||
- **ROCmFPX is the native fit**: IU4/MTP/FPX, HIP+Vulkan kernels, TurboQuant KV,
|
||||
262K ctx / 1 slot. Measured 8K MTP ~30.3 tok/s (2.83× vs `Q4_K_M`).
|
||||
- **ik_llama.cpp** — ROCm is its **weakest** backend (gfx1151 not validated) and
|
||||
the k-quant-no-hip-row-interleave issue bites there too. Keep ROCmFPX.
|
||||
- **Mainline** ROCm build works for correctness (standard quants) but loses FPX,
|
||||
IU4, TurboQuant, MTEA, and agent presets.
|
||||
|
||||
## When to reach for which
|
||||
- **Stay mainline** if you want one portable upstream for both boxes, the widest
|
||||
GGUF ecosystem, and standard FlashAttention/MTP. Best fit for **Dankserver**
|
||||
if you don't want the ik_llama.cpp build surface.
|
||||
- **Use ik_llama.cpp** for Dankserver when the MoE offload and fused-MoE wins
|
||||
justify an extra build (watch `-rtr` + K-quants).
|
||||
- **Use ROCmFPX** for Dankclaw — it is purpose-built for gfx1151.
|
||||
- **ROCmFPX** is for the AMD fleet; **CUDA/FastMTP** for the NVIDIA box. Do not
|
||||
port ROCmFP4 to the 3090 via Vulkan as a primary path.
|
||||
|
||||
## See Also
|
||||
- [[Llama Cpp Code Diff]] — the code-level quantization diff + comparison table
|
||||
- [[dealign-ornith-crack]] — the CRACK model on Dankserver
|
||||
- [[kingjones-strix-mtp]] — the ROCmFP4 model on Dankclaw
|
||||
- [[Quantization by Hardware Architecture]] — which quant fits which GPU/CPU
|
||||
- [[SGLang]] — the structured-generation serving engine (the other lane)
|
||||
|
||||
## Citations
|
||||
- Mainline — https://github.com/ggml-org/llama.cpp
|
||||
- charlie12345/ROCmFPX — https://github.com/charlie12345/ROCmFPX ·
|
||||
rocmfp4-llama — https://github.com/charlie12345/rocmfp4-llama · rocmfp4 —
|
||||
https://github.com/charlie12345/rocmfp4
|
||||
- ik_llama.cpp — https://github.com/ikawrakow/ik_llama.cpp · docs ·
|
||||
https://ikawrakow-ik_llama-cpp.mintlify.app/
|
||||
- ROCmFPX quant/enum/serve detail —
|
||||
https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h
|
||||
- SGLang — https://github.com/sgl-project/sglang · https://docs.sglang.io/
|
||||
@@ -0,0 +1,154 @@
|
||||
---
|
||||
type: research
|
||||
title: SGLang
|
||||
description: What SGLang is, how it works (RadixAttention, paged KV, overlap scheduler, structured output), how it compares to llama.cpp and vLLM, and whether it fits Dankserver (RTX 3090) and Dankclaw (Strix Halo).
|
||||
tags:
|
||||
- sglang
|
||||
- serving
|
||||
- inference
|
||||
- structured-output
|
||||
- radix-attention
|
||||
- llama.cpp
|
||||
timestamp: 2026-09-01
|
||||
---
|
||||
|
||||
# SGLang
|
||||
|
||||
SGLang (for **S**tructured **G**eneration **Lang**uage) is a high-throughput
|
||||
serving engine for large (and vision-language) models, from the LMSYS / UC
|
||||
Berkeley team, published at NeurIPS 2024. Today it is one of the two dominant
|
||||
open-source serving engines, alongside vLLM. People talk about it because it
|
||||
makes "call the model, branch, force the output into JSON" programs *fast*.
|
||||
|
||||
## What it is — in one paragraph
|
||||
|
||||
You serve an LLM by writing a small program: call the model, then based on its
|
||||
output call it again, branch into parallel attempts, and force the answer into a
|
||||
JSON schema. SGLang co-designs a Python-embedded language with a runtime that can
|
||||
*see* the program's structure, then exploits it: it caches and reuses the model's
|
||||
"working memory" (the KV cache) across requests that share a prompt prefix,
|
||||
batches new requests into running generations without waiting for a batch to
|
||||
drain, and forces tightly-structured output (JSON / regex / tool tags) by masking
|
||||
illegal tokens one step at a time — or several at once. Reported payoff:
|
||||
**up to 6.4× higher throughput** on structured / agentic / RAG workloads vs vLLM,
|
||||
TGI, and similar.
|
||||
|
||||
## Core architecture
|
||||
|
||||
Two halves that can also run independently:
|
||||
- **Frontend** — a Python DSL (`@sgl.function`, `gen`/`select`/`fork`/`join`/
|
||||
`extend`) for multi-call LLM programs with branching, parallel sampling, tool
|
||||
calls, and structured output.
|
||||
- **Backend runtime** — a high-throughput server (`python -m sglang.launch_server`)
|
||||
or an in-process `Engine`. OpenAI-compatible `/v1`, so RadixAttention and
|
||||
structured decoding work with a vanilla OpenAI client.
|
||||
|
||||
Building blocks:
|
||||
- **z-normalization** — pre-compile per-request sampling metadata into contiguous
|
||||
tensors so the GPU sampler does one bulk copy per batch per step (kills the
|
||||
CPU scheduler bottleneck).
|
||||
- **PagedTensor** — the KV cache is paged (fixed-size pages) and laid out
|
||||
non-contiguously in a global pool; mirrors vLLM's PagedAttention.
|
||||
- **RadixAttention** — a radix tree of shared token prefixes; on a new request it
|
||||
walks to the longest cached prefix, reuses those KV blocks, and only prefills
|
||||
the divergent suffix. Payoff is compute + throughput for long shared prefixes
|
||||
(RAG, agents), not raw memory.
|
||||
- **Overlap scheduler** — runs the CPU prep for step t+1 *under* the GPU forward
|
||||
of step t, so kernels launch back-to-back with no idle bubble (plus CUDA
|
||||
graphs + torch.compile).
|
||||
- **Structured / output-guided generation** — compiles the constraint (JSON
|
||||
schema / regex / EBNF) into a finite-state machine over the vocabulary and masks
|
||||
disallowed logits; **jump-forward** decoding emits forced token runs in one step.
|
||||
Backends: XGrammar (default), Outlines.
|
||||
|
||||
## SGLang vs llama.cpp vs vLLM
|
||||
|
||||
| Dimension | SGLang | llama.cpp |
|
||||
|-----------|--------|-----------|
|
||||
| Runtime | Python + CUDA/C++ kernels | C/C++, no Python required |
|
||||
| GPU requirement | GPU required, **no CPU path** | CPU-only / GPU-only / hybrid offload |
|
||||
| Model format | HF safetensors; AWQ/GPTQ/FP8 on the fly; **GGUF via llama.cpp backend** | **GGUF only** (convert first) |
|
||||
| KV reuse | RadixAttention (always-on radix tree) | basic prompt caching, no radix sharing |
|
||||
| Structured output | first-class (XGrammar, jump-forward) | narrower |
|
||||
| Concurrency | continuous batching + overlap scheduler | single-node, lower concurrency |
|
||||
| Multi-GPU | native tensor parallelism | layer-split, more manual |
|
||||
| Setup | `pip install sglang[all]`, venv, CUDA match | static binary / Docker, minutes |
|
||||
| License | Apache 2.0 | MIT |
|
||||
|
||||
**Pick SGLang** when you need many concurrent users, structured / JSON / tool-call
|
||||
output, or repeated long prefixes (RAG, agents) where RadixAttention pays off, and
|
||||
you own a GPU + Python/CUDA dependency chain. **Pick llama.cpp** for the fastest
|
||||
download-to-endpoint path, CPU/edge, GGUF libraries, and quantization breadth.
|
||||
**Pick vLLM** for the broadest model-hardware coverage and mature multi-GPU.
|
||||
|
||||
The paper's headline 6.4× / 5× numbers are vs **vLLM v0.1** (no prefix caching);
|
||||
the gap narrows vs modern vLLM — benchmark on your own traffic.
|
||||
|
||||
## Deployment model & gotchas
|
||||
- **Pluggable backends:** default vLLM engine, SGLang's own `sglang` engine, or a
|
||||
recent beta **`llama.cpp` backend** that serves GGUF and reuses `llama-server`.
|
||||
- **VRAM headroom runs higher** than llama.cpp (pre-allocated KV pool + radix
|
||||
tree). Size with `--mem-fraction-static`, `--context-length`,
|
||||
`--chunked-prefill-size`. OOM → lower context / fraction or a smaller quant.
|
||||
- **Model-format lock-in:** wants safetensors + AWQ/GPTQ/FP8 (or GGUF via the
|
||||
llama.cpp backend). The current CPU-Q6_K_P and ROCmFP4-MTP models are **GGUF**.
|
||||
- **AMD / ROCm: yes, officially.** Prebuilt `lmsysorg/sglang` ROCm Docker image or
|
||||
`pip`. Quant on AMD: FP8 (Aiter/Triton), AWQ (Triton), MXFP4 (CDNA3/4),
|
||||
`petit_nvfp4`. Marlin-based methods are **out** on AMD. Kernel work targets
|
||||
**datacenter Instinct (CDNA2/3/4)** — consumer/client parts like **Strix Halo
|
||||
(gfx1151)** are supported at the ROCm level but are not the kernel-tuning target.
|
||||
- **Prefix diversity kills RadixAttention** (per-user/personalized prompts → tree
|
||||
degenerates). Single-replica trees need prefix-aware routing across a fleet
|
||||
(`sglang-router --policy cache_aware`). Fast-moving API — pin versions.
|
||||
|
||||
## Relevance to our servers
|
||||
|
||||
### Dankserver — NVIDIA RTX 3090 (24 GB) + 62 GB RAM — 35B MoE
|
||||
A **genuine fit candidate (with a migration step)**. The 3090 is CUDA-native.
|
||||
A 35B MoE (~3B active) is ideal for continuous batching + structured generation,
|
||||
and 24 GB can hold a quantized 35B MoE (FP8/AWQ safetensors ~18–20 GB) with room
|
||||
for KV. MoE expert parallelism is supported.
|
||||
- **What changes:** the current model is a **CPU GGUF (Q6_K_P)**; SGLang wants
|
||||
safetensors (FP8/AWQ) for CUDA, OR use SGLang's **`llama.cpp` backend** to serve
|
||||
the existing GGUF under SGLang's frontend. Check for an AWQ/FP8 variant of the
|
||||
HauhauCS checkpoint on Hugging Face.
|
||||
- **Where it wins:** repeated structured/JSON/tool calls or a long shared system
|
||||
prompt (RAG) → RadixAttention + jump-forward give big gains vs the current
|
||||
sequential CPU pipeline. Simple low-concurrency chat → the operational weight
|
||||
may not pay off.
|
||||
|
||||
### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified) — Qwen3.8-27B ROCmFP4 MTP
|
||||
A **niche / experimental fit, not a drop-in replacement**. 64 GiB is generous and
|
||||
SGLang-on-ROCm is supported (Docker/pip), but:
|
||||
- Our model is a **custom ROCmFP4/MTP GGUF**. SGLang's primary path wants
|
||||
safetensors + an AMD quant (FP8/AWQ); MXFP4 needs CDNA3/4 and Strix Halo (CDNA
|
||||
3.5 *client*) is **not** an SGLang/AMD kernel target. Quant availability for this
|
||||
exact MTP model is unverified, and we'd lose the bespoke MTP draft-head speedup.
|
||||
- **Best-fit path:** SGLang's **`llama.cpp` backend** on top of our existing ROCm
|
||||
llama.cpp / GGUF — gain the structured-output DSL, RadixAttention, and scheduler
|
||||
without rebuilding the kernel stack. Or test the native `lmsysorg/sglang` ROCm
|
||||
image as a proof-of-concept. Treat it as an enhancement/experiment, not a
|
||||
production swap, until a quant for the MTP model exists and perf is measured on
|
||||
gfx1151.
|
||||
|
||||
## Tradeoffs at a glance
|
||||
- Higher baseline VRAM than llama.cpp (KV pool + radix tree).
|
||||
- GPU/accelerator required — no pure-CPU path (llama.cpp is the only option of the
|
||||
trio for a CPU-only box).
|
||||
- Model-format lock-in (safetensors/AWQ/GPTQ/FP8, or GGUF via llama.cpp backend).
|
||||
- Marlin-based quant out on AMD; Strix Halo is not an MXFP4 target.
|
||||
- Prefix diversity can nullify RadixAttention; pin versions; watch cache-report.
|
||||
|
||||
## See Also
|
||||
- [[Llama Cpp Runtimes]] / [[Llama Cpp Code Diff]] — the llama.cpp side of the trio
|
||||
- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
|
||||
- [[kingjones-strix-mtp]] — the ROCmFP4 MTP model on Dankclaw
|
||||
- [[Quantization by Hardware Architecture]]
|
||||
|
||||
## Citations
|
||||
- SGLang paper (arXiv:2312.07104) — https://arxiv.org/abs/2312.07104 · NeurIPS 2024
|
||||
- Repo — https://github.com/sgl-project/sglang · docs — https://docs.sglang.io/
|
||||
- AMD / ROCm — https://docs.sglang.io/docs/hardware-platforms/amd_gpu ·
|
||||
Structured Outputs — https://docs.sglang.io/docs/advanced_features/structured_outputs
|
||||
- lmsysorg/sglang ROCm images — https://hub.docker.com/r/lmsysorg/sglang
|
||||
- SGLang vs llama.cpp — https://markaicode.com/vs/sglang-vs-llamacpp/
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user