docs: rename docs/ -> serverAppConfigs-docs/ per system-wide PROJECTNAME-docs convention

All in-repo path references (AGENTS.md, README, old scripts, report artifacts, test code paths) rewritten in the same commit; focused-test map extended for .gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's own convention ('add one before committing'). Obsidian wikilinks are title-based and unaffected.
2026-08-28 02:20:06 -07:00
1 parent 6d4a96a423
commit 5829a0bfe8
@@ -0,0 +1,38 @@
+# Backblaze Integration
+
+Off-site backup plan for the bind-mounted data on both hosts. **Deferred**: research only, not implemented yet.
+
+## Goal
+
+Back up critical data to Backblaze B2:
+- Calibre library (Dankserver)
+- ComfyUI models (Dankclaw)
+- Gitea data, repos, DB (Dankclaw)
+- Homepage/stack configs (this repo — already versioned in git)
+
+## Approach (research direction)
+
+| Option | Fit |
+|--------|-----|
+| `restic` + B2 | Best fit: dedupe, encryption, snapshots, low cost. Single static binary, works with podman host. |
+| `borg` + borgbase | Alternative; borgbase is more expensive per TB than B2. |
+| rclone + B2 | Good for file sync, no snapshot/dedupe semantics. |
+| podman volume driver for B2 | Niche, immature. Not recommended. |
+
+## Shape of the Implementation (future)
+
+- `scripts/backup.sh` on each host, run by systemd timer or cron.
+- `restic -r b2:bucket:path backup /opt/...`
+- `RESTIC_PASSWORD` in the host `.env`, never in the repo.
+- Restore procedure documented once the backup job exists.
+
+## Next Steps
+
+1. Create a B2 bucket + application key.
+2. Draft `scripts/backup.sh` and a systemd timer unit.
+3. Run a test restore.
+
+## See Also
+
+- [[Storage Policy]] — what gets backed up
+- [[Gitea]] — the existing backup procedure (tar / pg_dump)
@@ -0,0 +1,106 @@
+# ComfyUI on ROCm and Vulkan
+
+Research into whether ComfyUI can use GPU acceleration — not just CPU — on the AMD Strix Halo APU (gfx1151) in Dankclaw. Verified on live hardware 2026-08-14. This page corrects the persistent "ROCm unavailable on Strix Halo" claim.
+
+## TL;DR
+
+| Path | Works on gfx1151? | Evidence |
+|------|-------------------|----------|
+| pytorch.org ROCm wheels | **No** | wheels are built for `gfx906/gfx90a/gfx942/gfx1100` etc. — no gfx1151 kernels |
+| Official `rocm/comfyui` image | **No** | built with `PYTORCH_ROCM_ARCH=gfx942;gfx950` — datacenter archs only |
+| AMD gfx1151-tagged PyTorch build | **Yes** | `torch 2.9.1+rocm7.12.0rc1` → `avail: True`, `Radeon 8060S`, cap `(11,5)` |
+| Vulkan (torch.backends.vulkan) | **No** | compiled out of standard wheels; experimental/mobile even when present |
+| ComfyUI `--cpu` | **Yes (slow)** | current stack config |
+
+## The core insight
+
+PyTorch's ROCm support is not one thing. The wheels you install for poke at torch, the wheels AMD builds for specific APUs are different, and the kernels compiled in are fixed at build time via `PYTORCH_ROCM_ARCH`.
+
+Standard wheels (pytorch.org and the `rocm/comfyui` image) target **CDNA datacenter archs** (`gfx942`/`gfx950`) or the older common set. **gfx1151 (Strix Halo / Radeon 8060S)** is not included, so `torch.cuda.is_available()` returns `False` and everything falls back to CPU.
+
+AMD does publish gfx1151 builds — but you have to pick the right image.
+
+## Live verification (Dankclaw, 2026-08-14)
+
+### Fails: official rocm/comfyui image
+
+```bash
+podman pull docker.io/rocm/comfyui:comfyui-0.18.2.amd0_rocm7.2.0_ubuntu24.04
+podman run --rm --device=/dev/kfd --device=/dev/dri --entrypoint python <img> -c "
+import torch
+print(torch.__version__)          # 2.10.0a0+git…
+print(torch.cuda.is_available())  # False
+print(torch.cuda.get_arch_list()) # []"
+
+# Env in that image:
+#   PYTORCH_ROCM_ARCH=gfx942;gfx950
+#   ROCM_PATH=/opt/rocm-7.2.0
+```
+
+The image bundles ROCm 7.2 and compiles kernels only for `gfx942;gfx950`. No gfx1151. `HSA_OVERRIDE_GFX_VERSION` does not help because the kernels simply are not compiled in.
+
+The official ComfyUI ROCm image is 68 GB and unusable on Strix Halo. It was pulled on Dankclaw for this test; remove it to reclaim space.
+
+### Works: AMD gfx1151-tagged PyTorch build
+
+The `rocm/vllm` image exists in two flavors. The one tagged `gfx1151` works on Strix Halo:
+
+```bash
+podman run --rm \
+  --device=/dev/kfd --device=/dev/dri \
+  --entrypoint python \
+  docker.io/rocm/vllm:rocm7.12.0_gfx1151_ubuntu24.04_py3.12_pytorch_2.9.1_vllm_0.16.0 \
+  -c "
+import torch
+print(torch.__version__)        # 2.9.1+rocm7.12.0rc1
+print(torch.version.hip)        # 7.12.60610
+print(torch.cuda.is_available()) # True
+print(torch.cuda.get_device_name(0))  # Radeon 8060S Graphics
+print(torch.cuda.get_device_capability(0)) # (11, 5)  = gfx1151
+"
+
+# Env in that image:
+#   PYTORCH_ROCM_ARCH=gfx1151
+```
+
+Key facts:
+- `avail: True`, device `Radeon 8060S`, capability `(11, 5)` = gfx1151.
+- Device access requires passing `/dev/kfd` and `/dev/dri` to the container (and `--group-add video`/`realtime` or a user with those groups).
+- ROCm HIP runtime vars matter: this image has HIP 7.12; the older `rocm/vllm:rocm7.14.0_rdna_*` (48.6 GB) targets the newer "RDNA" set.
+
+## What this means for ComfyUI
+
+ComfyUI is a PyTorch app. If you give it a PyTorch wheel with gfx1151 kernels — the AMD `/opt/python` build used by the working vLLM image, or a custom build with `PYTORCH_ROCM_ARCH=gfx1151` — ComfyUI **will** use the GPU via ROCm. The blocker is not ComfyUI; it is shipping it in a container that contains a gfx1151-enabled torch.
+
+Three ways forward (way 1 applied 2026-08-19):
+
+1. **Base a ComfyUI image on the AMD gfx1151 torch build.** Take `rocm/vllm:...gfx1151...pytorch_2.9.1...` (or AMD's PyTorch wheel index) and install ComfyUI on top. This is the direct path to GPU ComfyUI. **Applied:** the running `comfyui-rocm:gfx1151` image does exactly this (see `dankclaw/Containerfile.comfyui`).
+2. **Build PyTorch for gfx1151** (`export PYTORCH_ROCM_ARCH=gfx1151` before build). Slow build, but a self-contained wheel afterwards.
+3. **Use a ggml-family runtime instead** — stable-diffusion.cpp / Lemonade's `sd.cpp` backend ship their own gfx1151 HIP kernels and work today (see [[Lemonade and sd.cpp]]).
+
+## Vulkan
+
+- PyTorch's Vulkan backend (`torch.backends.vulkan`) is **compiled out of standard Linux wheels**. The installed `torch 2.11/2.13` on Dankclaw reports `hasattr(torch.backends, 'vulkan')` → `False`.
+- Even when built in, PyTorch Vulkan is a mobile-oriented experimental backend, not a desktop inference path, and ComfyUI exposes no `--vulkan` flag.
+- The system has a working Vulkan driver (RADV, `libvulkan_radeon.so` + `radeon_icd.json`), so Vulkan apps work — but ComfyUI/PyTorch cannot consume it here.
+- Conclusion: **Vulkan is a dead end for ComfyUI.** If GPU is wanted, ROCm is the realistic path; a ggml-family engine (sd.cpp/Lemonade) is the pragmatic one.
+
+## Consequence for this repo
+
+**Applied 2026-08-19.** `dankclaw/compose.yaml` now builds `comfyui-rocm:gfx1151` from `dankclaw/Containerfile.comfyui` (based on the AMD gfx1151 base), passes `/dev/kfd` + `/dev/dri`, adds the `video` group, and drops `--cpu`. The container runs on the Radeon 8060S (gfx1151) and is boot-persistent via a user systemd unit. See [[ComfyUI]] for the running config.
+
+## Cleanup note
+
+These images were pulled on Dankclaw during this research:
+
+```bash
+podman rmi docker.io/rocm/comfyui:comfyui-0.18.2.amd0_rocm7.2.0_ubuntu24.04   # 68 GB, unusable on gfx1151
+podman rmi docker.io/rocm/vllm:rocm7.12.0_gfx1151_...                         # 32.5 GB, keep if pursuing ComfyUI ROCm
+```
+
+## See Also
+
+- [[ComfyUI]] — the service page (CPU config)
+- [[Lemonade and sd.cpp]] — the ggml-family alternative with working gfx1151 kernels
+- [[Observability]], [[Dankclaw Stack]]
+- KB: [[ComfyUI]] (tools) (the "ROCm unavailable" claims here are now qualified)
@@ -0,0 +1,49 @@
+# Lemonade and sd.cpp
+
+Why Lemonade Server can do image generation on Strix Halo when ComfyUI cannot. Research from the Lemonade source repo (`lemonade-sdk/lemonade`, cloned 2026-08-14) and live verification.
+
+## The short answer
+
+Lemonade does not run ComfyUI. Its image-generation backend is a wrapped **stable-diffusion.cpp** (`leejet/stable-diffusion.cpp`) subprocess. sd.cpp is a ggml-family native C++ runtime that ships its own HIP kernels — and those kernels **include gfx1151**. ComfyUI runs on PyTorch, whose standard wheels omit gfx1151. Same hardware, different runtime, different result.
+
+## From the Lemonade source
+
+`src/cpp/server/backends/sdcpp/sdcpp_server.cpp`:
+
+- Image gen is `SDServer`, a `WrappedServer` around a prebuilt sd.cpp binary.
+- Downloads `sd-<version>-bin-Linux-Ubuntu-24.04-x86_64-<backend>.zip` from `leejet/stable-diffusion.cpp` releases.
+- Backends: `cpu, cuda, metal, rocm, vulkan`.
+- ROCm asset is `...-rocm-<version>.zip`; Vulkan is `...-vulkan.zip`.
+
+Backend support matrix (`serverAppConfigs-docs/dev/backends-reference.md`):
+
+| Recipe | Backend | Device families |
+|--------|---------|-----------------|
+| `sd-cpp` | rocm | amd_gpu (gfx103X, gfx110X, **gfx1150, gfx1151, gfx1152**, gfx120X) |
+| `sd-cpp` | vulkan | amd_gpu; cpu; nvidia_gpu |
+| `thenoise` | rocm | amd_gpu (gfx1150, gfx1151, gfx1152) — experimental |
+
+**gfx1151 is explicitly supported** by sd.cpp's ROCm build. There is also a dedicated experimental `thenoise` ROCm backend targeting exactly the Strix Halo archs.
+
+## Engine comparison
+
+| | ComfyUI | Lemonade (sd.cpp) |
+|---|---|---|
+| Runtime | PyTorch | native C++ (ggml family) |
+| AMD path | needs PyTorch ROCm wheel with gfx1151 kernels — not in standard wheels | ships own HIP kernels, gfx1151 included |
+| On Strix Halo | CPU only (today) | ROCm GPU, or Vulkan fallback |
+| Interface | node graph web UI | server, OpenAI-style API, GUI/CLI |
+
+This mirrors llama.cpp on the same box: ggml-family runtimes ship their own gfx1151 HIP kernels and run on ROCm; PyTorch does not.
+
+## Relevance
+
+- If GPU image generation on Dankclaw matters, **Lemonade (or bare sd.cpp) is the working path today**.
+- ComfyUI's node graph is the cost of switching: Lemonade is not ComfyUI.
+- See [[ComfyUI ROCm and Vulkan]] for the PyTorch-side analysis showing a gfx1151-enabled torch would also make ComfyUI work.
+
+## See Also
+
+- [[ComfyUI ROCm and Vulkan]] — the PyTorch side of this question
+- [[ComfyUI]] — current CPU config
+- [[Dankclaw Stack]]
@@ -0,0 +1,222 @@
+---
+type: research
+title: Llama.cpp Code Diff
+description: Code-level diff of mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp with a focus on quantization code, ggml type numbers, and the differences that matter for Dankserver (RTX 3090) and Dankclaw (Strix Halo).
+tags:
+  - llama.cpp
+  - quantization
+  - ggml
+  - ROCmFPX
+  - ik_llama
+  - code-diff
+timestamp: 2026-09-01
+---
+
+# Llama.cpp Code Diff
+
+Code-level diff of the three runtimes, focused on **quantization code** and the
+differences that affect our two servers. Positioning overview is in
+[[Llama Cpp Runtimes]].
+
+## 1. Quantization file map
+
+Where quant lives in each runtime (paths relative to the repo root):
+
+| File / path | Mainline | ROCmFPX | ik_llama.cpp |
+|-------------|----------|---------|--------------|
+| `ggml/include/ggml.h` (`enum ggml_type`) | FP16/FP32/BF16, legacy `Q*_0`, K-quant types, `IQ*_` | **adds** `Q4_0_ROCMFP4` … `Q2_0_ROCMFPX`, `Q4_0_ROCMI4`, `Turbo3_0`, `Turbo4_0` (enums **100–108**) | IQK non-linear (`IQ4_NL` etc.), Trellis `IQ*_KT`, MXFP4, Bitnet, `Q8_KV` |
+| `ggml/src/ggml-quants.*` | K/I/legacy dequant, row-interleaved 3-bit | `block_rocmfp4` unpack + E4M3 scaling, IU4 WMMA dispatch | `iqk/*` fused quant-gemms, IQK row-interleave, Trellis decode |
+| `tools/quantize/quantize.c` | all mainline quants + `--imatrix` | ROCmFPX quantizer (`STRIX`/`FAST`/`_AGENT` presets) | IQK/Trellis/MXFP4 packers, `--cache-type-k/v` + Hadamard |
+| `gguf.cpp` / `gguf-io.c` | standard ftype table | **custom ftypes** 100–106 (FP4), 110–113 (FPX), 118 (I4) | IQK/Trellis ftype tags |
+| KV cache | `f32/f16/bf16/q8_0/q4_0/q4_1/iq4_nl/q5_0/q5_1` (needs FlashAttention) | **TurboQuant** `Turbo3_0`/`Turbo4_0`/`V` | `Q8_KV`, IQK/Trellis/MXFP4 reused in `-ctk`, `--k-cache-hadamard`, `--v-cache-hadamard` |
+
+Two number spaces in ROCmFPX (authoritative `ggml/include/ggml.h`, sha
+`c49ebdbd`):
+- **`ggml_type` enum** (tensor storage): `Q4_0_ROCMFP4`=100, `Q4_0_ROCMFP4_FAST`=101,
+  `Q6_0_ROCMFPX`=102, `Q8_0_ROCMFPX`=103, `Q3_0_ROCMFPX`=104, `Turbo3_0`=105,
+  `Turbo4_0`=106, `Q2_0_ROCMFPX`=107, `Q4_0_ROCMI4`=108, count 109.
+- **`ggml_ftype`** (GGUF model-file): ROCmFP4 family `100–106` (`STRIX`=105,
+  `STRIX_LEAN`=106), ROCmFPX family `110–113` (FP6=110, FP8=111, FP3=112,
+  FP2=113), `ROCmI4`=118.
+
+## 2. Comparison table
+
+| Runtime | Quant formats | Granularity | Custom ggml types | MoE | Attention accel | Hybrid GPU/CPU offload | Build backends |
+|---------|---------------|-------------|-------------------|-----|-----------------|------------------------|----------------|
+| Mainline | K/I/legacy/TQ, imatrix, row-RI 3-bit | grouped/per-block + double-quant | none | native `--cpu-moe` | FlashAttention (multi-backend) | `--n-gpu-layers`, partial offload | CPU/CUDA/Vulkan/Metal/SYCL/HIP-ROCm/CANN |
+| charlie12345/ROCmFPX | ROCmFP2/3/4/6/8, ROCmI4, TurboQuant KV | block + E4M3 codebook | **yes (100–108)** | MTP draft head (bundled) | **MTEA** + TurboQuant FA (custom) | Strix-tuned MTP profile | HIP/ROCm + Vulkan (**CUDA off**), CPU ref |
+| ik_llama.cpp | IQ* non-linear, improved K, IQK row-RI, Trellis, MXFP4, Q8_KV, Bitnet | grouped + non-linear + row-interleaved | IQK/Trellis/MXFP4/Bitnet tags | **fused MoE**, `-ooae`, `--fit`, `-ot`, GDN | **FlashMLA** (DeepSeek MLA) | **finest-grained** override-tensor / cpu-moe / fit | CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm(weak) |
+
+## 3. Code-difference narrative
+
+**What each fork ADDED that mainline lacks:**
+- **ROCmFPX** — ~10 new `ggml_type` entries, the `block_rocmfp4` packed-nibble
+  format, `*_ROCMFPX`/`*_ROCMI4` ftypes, tensor-aware presets (`*_AGENT`,
+  `*_COHERENT`, `*_STRIX`), `block_rocmfp4` dequant + dispatch kernels,
+  TurboQuant KV types, MTEA attention kernel. Mainline does not know these
+  tensor types. It is a *fork* that adds a new backend: custom tensor
+  registration + HIP/ROCm + Vulkan kernels + CPU reference. **CUDA is disabled.**
+- **ik_llama.cpp** — new code under `ggml/src/iqk/` (IQK GEMM + row-interleave),
+  CUDA IQK flash-attn kernels (`GGML_IQK_FA_ALL_QUANTS=ON`), fused MoE
+  (`fnn_up`+`fnn_gate`), fused delta-net (GDN) SSM ops, `flash_fwd_mla` for
+  DeepSeek MLA/DSA, and hybrid-offload controls (`-ot/--override-tensor`,
+  `--cpu-moe`, `-ooae/--offload-only-active-experts`, `-ser/--smart-expert-reduction`,
+  `--fit`, `-op/--offload-policy`, split-mode graph). Run-time quant repack
+  (`-rtr`). **Additive** — GGUF remains compatible.
+
+**What mainline has that the forks lag on:**
+- **ik_llama.cpp**: architecture baseline is **Aug-2024** — newest
+  architectures arrive late (PRs move fast, but the core is ~1.5y behind).
+- **ROCmFPX**: **AMD-first, experimental**; narrower, must rebuild/re-quant per
+  fork line, CUDA ecosystem (the fleet's 3090) is unreachable.
+
+**The one caveat that changes a build (ik_llama.cpp + `-rtr`):**
+K-quants have **no CUDA row-interleaved implementation**. With `-rtr`, every
+CPU-resident K-quant tensor is repacked to row-interleaved and then **run on CPU**
+even where GPU would be faster → lower prompt-processing speed. For a MoE that
+keeps experts on CPU (`--cpu-moe`), this is a real hit. Prefer IQK/R4/Trellis/
+MXFP4 variants, or skip `-rtr` and place tensors with `--fit`/`-ot`.
+
+## 4. Server-specific benefits
+
+### Dankserver — NVIDIA RTX 3090 (24 GB), 62 GB RAM — 35B MoE (Ornith-1.5-35B-A3B CRACK)
+- The winning lever for a **low-VRAM CUDA MoE** is **expert-aware offload + fused
+  MoE**, which is ik_llama.cpp's core strength:
+  - `-ooae/--offload-only-active-experts` — with ~8 active experts this cuts
+    RAM→VRAM traffic dramatically vs shipping all experts.
+  - `-fmoe/--fused-moe`, GDN fusion (the CRACK model is a hybrid GatedDeltaNet +
+    MoE), `--fit`, `-ot/--override-tensor`, `--cpu-moe`.
+- **Quant pick for the 3090:** `Q4_K_M`/`Q5_K_M` on mainline, or an **IQK/R4 /
+  Trellis / MXFP4** variant on ik_llama.cpp that has a **CUDA** path. Avoid raw
+  K-quant + `-rtr`. Enable FlashAttention; `GGML_CUDA_FORCE_MMQ=1`;
+  `ngram-mod` drafts (MoEs want long drafts).
+- ⚠️ **Expert-count discrepancy:** the CRACK GGUF README says **256 experts / 8
+  active**; the fleet brief cites ~36. Verify `n_expert` / `n_expert_used` from
+  the GGUF before finalizing the offload recipe — both point at expert-aware
+  offload being the right tool.
+- **Mainline only?** Fine: `--cpu-moe` + `Q4_K_M` + FlashAttention + `ngram-mod`.
+- **ROCmFPX?** No — ROCm-only formats, CUDA disabled on the builds.
+
+### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified), ROCm 7.2.4 — Qwen3.8-27B ROCmFP4 STRIX MTP
+- **ROCmFPX is the natural fit** and stays the choice: native IU4 (ROCmI4),
+  ROCmFP4 weight formats, MTP draft head, TurboQuant KV, 262K / 1 slot. The
+  production `build-promptforge` `llama-server` already runs there.
+- **ik_llama.cpp on Strix?** ROCm is its **weakest** backend (gfx1151 not
+  validated, k-quant-no-hip-row-interleave), and we'd lose the bespoke MTP
+  draft-head speedup. Higher-risk change; only with a small soak-test first.
+- **Mainline on Strix** works for correctness with standard quants but loses FPX,
+  IU4, TurboQuant, and MTEA.
+
+## 6. KV cache quant table
+
+Bit-density-ordered comparison of everything usable in the **KV cache** (`-ctk`
+key / `-ctv` value, and draft `-ctkd`/`-ctvd`). Order is high → low bits-per-
+value. Columns mark **origin** (`*` = mainline-provided) and whether the type is
+**KV-native** (designed for the cache) or **weight-family reused for KV** (a
+model-weight quant the runtime also accepts in `-ctk`/`-ctv`).
+
+**How bpw is read here.** `bpw = ggml_type_size × 8 / QK`, computed from the
+exact block structs in `ggml.h` / `ggml-common.h` (`QK_K = 256`, fp16 scale).
+This is the true in-RAM cost per value, so it is what sets KV-cache size and
+quantize/dequant throughput. A footnote marks the few types whose **nominal**
+figure (older 1-byte-scale layout / data-bits-only) diverges from this exact
+number; the table uses the exact number throughout.
+
+### 6a. All KV-usable types, by bit density (interleaved)
+
+| bpw | Type | Origin | KV-native | Notes |
+|-----|------|--------|-----------|-------|
+| 32.0 | f32 | mainline * | yes | full precision |
+| 16.0 | f16 | mainline * | yes | default |
+| 16.0 | bf16 | mainline * | yes | |
+| 9.25 | q8_K | mainline * | weight | rare in KV; per-16-block sums; ik Q8_KV is the fast KV version |
+| 8.13 | **q8_0** | mainline * | yes | |
+| 8.06 | **q8_KV** | ik * | yes | the dedicated **fast KV cache** type (`-ctk q8_KV`); Q8_0-shaped, 1 shared scale |
+| 6.63 | iq6_K | ik | weight→KV | IQK family |
+| 6.56 | q6_K | mainline * | weight→KV | |
+| 6.00 | **q5_1** | mainline * | yes | (nominal 5.06) |
+| 5.50 | **q5_0** | mainline * | yes | (nominal 5.06) |
+| 5.50 | q5_K | mainline * | weight→KV | |
+| 5.50 | iq5_K | ik | weight→KV | IQK family |
+| 4.50 | **q4_0** | mainline * | yes | (nominal 4.25) |
+| 4.50 | **q4_1** | mainline * | yes | (nominal 4.25) |
+| 4.50 | **iq4_nl** | mainline * | yes | (nominal 4.0) |
+| 4.50 | iq4_K | ik | weight→KV | IQK family |
+| 4.5 | **Turbo4_0** | ROCmFPX * | yes | TurboQuant; ~4.5 |
+| 4.25 | iq4_ks | ik | weight→KV | IQK subsample |
+| 4.25 | iq4_xs | mainline * | weight→KV | popular *weight* format, also usable in KV |
+| 4.25 | mxfp4 | ik * | weight→KV | |
+| 3.5 | **Turbo3_0** | ROCmFPX * | yes | TurboQuant; ~3.5 |
+| 3.44 | iq3_K | ik | weight→KV | IQK family |
+| 3.44 | iq3_s | mainline * | weight→KV | |
+| 2.69 | iq2_kl | ik | weight→KV | IQK subsample |
+| 2.38 | iq2_K | ik | weight→KV | IQK family |
+| 2.19 | iq2_ks | ik | weight→KV | IQK subsample |
+| 2.13 | iq2_kt | ik * | weight→KV | **Trellis** (lattice) |
+| 1.75 | iq1_kt | ik * | weight→KV | **Trellis** (lattice) |
+
+### 6b. Origin legend
+
+- **Mainline KV-native** (`-ctk`/`-ctv` = exactly these 9, from `common/arg.cpp`
+  `kv_cache_types`): `f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1`. Marked
+  with `*` and `yes` in the KV-native column.
+- **ROCmFPX KV-native**: `turbo3_0, turbo4_0` — TurboQuant's dedicated KV formats
+  (add a third `V` type for values only). Marked `*` here.
+- **ik additions**: `q8_KV` (fast KV, dedicated), plus the weight-family quants
+  ik lets you reuse in `-ctk`/`-ctv` — the **IQK** family (`iq*_K`, plus the
+  `IKS`/`IKL` subsamples) and the **Trellis** lattice family (`iq*_KT`), and
+  `mxfp4`. ik's `-ctk` accepts a broader set than mainline's.
+
+### 6c. Trellis, IQK, MXFP4 — where they sit
+
+- **Trellis (`IQ*_KT`)** — ik's lattice-code quants, mirror the `IQ1..IQ4` bit
+  ladder: `iq1_kt` 1.75 → `iq2_kt` 2.13 → `iq3_kt` 3.13 → `iq4_kt` 4.00. Highest
+  quality per bit of the family; lowest-density entries in the whole table.
+- **IQK (`IQ*_K`)** — ik's improved K-quants: `iq2_k` 2.38, `iq3_k` 3.44,
+  `iq4_k` 4.50, `iq5_k` 5.50, `iq6_k` 6.63, plus subsamples `iq2_kl` 2.69 /
+  `iq2_ks` 2.19 / `iq4_ks` 4.25. They sit alongside the mainline K-family
+  (iq2_k ≈ iq2_s, iq4_k ≈ q4_K) but with ik's row-interleave + fused GEMM.
+- **MXFP4** (`mxfp4`, 4.25) — the mixed-precision FP4 cache quant; ik's entry.
+
+### 6d. Hadamard is a transform, not a row
+
+`--k-cache-hadamard` / `--v-cache-hadamard` (ik) is a **structured transform
+applied on top of a base KV quant**, not its own bit-density row. It spreads
+quantization error, improving accuracy for the underlying type; ik's guidance is
+**best below Q6_0**. Row-interleaving (the `R4`/`R8` variants) is the same idea —
+a packing transform on the K-family — and is likewise a modifier, not a quant.
+
+### 6e. Server notes
+
+- **Dankclaw (production, ROCmFP4 STRIX MTP):** KV stays on `turbo4_0` /
+  `turbo3_0` (TurboQuant) — the runtime's own formats. ik's IQK/Trellis path on
+  Strix is higher-risk (see §4).
+- **Dankserver (3090 MoE):** `-ctk q8_KV` or `q4_K`/`q5_K` (mainline) with
+  FlashAttention + KV-quant; on ik, `iq4_k`/`iq4_kt`/`iq2_k` if a CUDA path is
+  confirmed for that type. Keep FlashAttention and the KV quant version in lockstep
+  — they are coupled (a quant change that drops FA off the path changes KV math).
+
+## 5. Recommendations for our fleet
+
+- **Dankclaw:** keep **charlie12345/ROCmFPX** (already production-tuned for
+  gfx1151). If you want more quant variety on Strix, explore the ROCmFP2 (`Q2_0_ROCMFPX`,
+  2.5 bpw, S40 codebook) preview — but do not port to ik_llama.cpp as primary.
+- **Dankserver:** **try ik_llama.cpp** (CUDA-90 + AVX-512 build) for fused-MoE +
+  `offload-only-active-experts`, staying off raw K-quant `-rtr`. If that is too
+  much surface, stay on **mainline** with `--cpu-moe` + `Q4_K_M` + FlashAttention
+  + `ngram-mod`. Neither box needs ROCmFP4.
+- **Common:** pin a commit per fleet line; re-test after any quant/KV/context
+  change (FlashAttention + KV-quant are coupled).
+
+## See Also
+- [[Llama Cpp Runtimes]] — positioning + quantization overview
+- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
+- [[kingjones-strix-mtp]] — the ROCmFP4 model on Dankclaw
+- [[Quantization by Hardware Architecture]] — quant vs hardware
+
+## Citations
+- Mainline quant + backends — https://github.com/ggml-org/llama.cpp ·
+  tools/quantize/README.md · serverAppConfigs-docs/speculative.md
+- ROCmFPX enum (ggml_type 100–108) — https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h
+- ik_llama.cpp quant/MoE/FlashMLA/backends — https://github.com/ikawrakow/ik_llama.cpp (README, serverAppConfigs-docs/parameters.md, serverAppConfigs-docs/build.md) · DeepWiki · discussions#562
+- ik_llama.cpp `-rtr` K-quant trap & backend-bandwidth caveat — https://github.com/ikawrakow/ik_llama.cpp/blob/main/README.md
@@ -0,0 +1,141 @@
+---
+type: research
+title: Llama.cpp Runtimes
+description: Compared mainline llama.cpp, charlie12345/ROCmFPX, and ik_llama.cpp — positioning, quantization formats, code-level differences, and fit for Dankserver (RTX 3090) and Dankclaw (Strix Halo).
+tags:
+  - llama.cpp
+  - quantization
+  - ROCmFPX
+  - ik_llama
+  - inference
+  - amd
+  - nvidia
+timestamp: 2026-09-01
+---
+
+# Llama.cpp Runtimes
+
+Comparison of three `llama.cpp`-derived inference runtimes used (or usable) on
+the two homelab boxes: **mainline** `ggml-org/llama.cpp`,
+**charlie12345/ROCmFPX** (what Dankclaw runs), and **ik_llama.cpp**
+(ikawrakow). The goal is to know, for each server, which quant formats are on
+the table and which runtime best fits the hardware.
+
+Full code-level quantification is in [[Llama Cpp Code Diff]]. This page is the
+positioning + quantization summary.
+
+## Which runtime is which
+
+| Runtime | Repo | What it is | Primary backends | For our boxes |
+|---------|------|-----------|------------------|---------------|
+| Mainline | [`ggml-org/llama.cpp`](https://github.com/ggml-org/llama.cpp) | Canonical, general-purpose, one codebase for every backend | CPU, CUDA, Vulkan, Metal, SYCL, HIP/ROCm, CANN, OpenVINO | Baseline both boxes; the portable default |
+| charlie12345/ROCmFPX | [`charlie12345/ROCmFPX`](https://github.com/charlie12345/ROCmFPX) (+ reference fork [`rocmfp4-llama`](https://github.com/charlie12345/rocmfp4-llama), clean pkg [`rocmfp4`](https://github.com/charlie12345/rocmfp4)) | Experimental AMD-focused GGUF *weight* quant family layered on llama.cpp | HIP/ROCm + Vulkan (CUDA **disabled**), CPU reference | **Dankclaw** (native Strix Halo) |
+| ik_llama.cpp | [`ikawrakow/ik_llama.cpp`](https://github.com/ikawrakow/ik_llama.cpp) | Fork focused on SOTA quants + CPU / hybrid GPU-CPU performance | CPU (AVX-512/VNNI), CUDA (Turing+), Vulkan, ROCm (weak) | **Dankserver** candidate (3090 + MoE) |
+
+Key lineage notes:
+- Mainline moved from `ggerganov/llama.cpp` to the `ggml-org` org. Rolling
+  commit hashes (the fleet pins `b10121`) plus newer `vX.Y.Z` tags. Pin a
+  commit for reproducibility.
+- ROCmFPX is an *experimental feature family* on the canonical `main` branch;
+  three GitHub surfaces exist (reference fork, clean patch, pre-patched tree).
+  It is **AMD-only** — CUDA is compiled out.
+- ik_llama.cpp's core is an **August-2024 llama.cpp snapshot** maintained with
+  a rolling set of PRs (MLA, fused MoE, row-interleaved quants, MTP, DFlash all
+  appeared here first, later partially merged upstream). The feature surface is
+  current; the architecture baseline is ~1.5 years behind mainline.
+
+## Quantization summary
+
+### Mainline (what the fleet already ships)
+- **Base:** FP32 / FP16 / BF16.
+- **K-quants** (modern default, two-level block + double-quant): `Q2_K` …
+  `Q6_K`, with `_S`/`_M`/`_L` precision mixes. `Q4_K_M` is the popular default;
+  `Q6_K` is "almost lossless."
+- **i-quants** (importance-matrix, non-linear, best quality/size at low bits but
+  slower decode): `IQ4_NL`, `IQ4_XS`, `IQ3_*`, `IQ2_*`. Row-interleaved 3-bit
+  family (IQ3_XXS/S/XS) reduces the quality penalty of whole-row quant.
+- **Legacy:** `Q*_0`/`Q*_1` (fast, simple), ternary `TQ1_0`.
+- **KV cache quant:** `f32/f16/bf16/q8_0/q4_0/iq4_nl/q5_0/q5_1`; quantizing KV
+  requires FlashAttention (`-fa on`).
+- Speculative decoding native (MTP, EAGLE3, DFlash, DSpark, `ngram-mod`).
+
+### ROCmFPX (Dankclaw's AMD weight formats)
+- Integer GGUF **weight** formats only this fork reads: **ROCmFP2/3/4/6/8** and
+  **ROCmI4 (W4A4 IU4 WMMA)**. These are custom `ggml` tensor types
+  (**`ggml_type` 100–108**) with matching GGUF **`ftype`** (ROCmFP4 family
+  `100–106`; ROCmFPX family `110–113`; `ROCmI4` = `118`). Stock llama.cpp,
+  Ollama, and LM Studio **cannot load them**.
+- Flagship **ROCmFP4** (`block_rocmfp4`): 32 weights/block, two E4M3 scale
+  bytes, ~18 bytes/block, ~4.5 bpw, E2M1 nibbles + E4M3 codebook up to 10
+  units. Variants: `STRIX` / `STRIX_LEAN`, `FAST`, `COHERENT`, plus `*_AGENT`
+  / `*_STRIX_MTP` draft-head presets.
+- **TurboQuant KV:** `Turbo3_0` (3.5 bpw) / `Turbo4_0` (4.5 bpw) + `TurboQuant V`
+  (f16 K) on the production model.
+- **ROCmI4 (W4A4):** the gfx1151 native IU4 4-bit WMMA path (`-DGGML_HIP_ROCMI4_W4A4=ON`);
+  lossy activation quant, exact int8 MMQ.
+
+### ik_llama.cpp (SOTA quants + CPU/hybrid)
+- **IQ\*** non-linear family; `IQ4_NL` gives lower perplexity than mainline
+  `Q4_K_M` at the same size. Improved **K-quants** and the signature
+  **row-interleaved IQK / R4 / R8** packing (activates AVX-512/VNNI/NEON
+  matmul paths that vanilla llama.cpp leaves cold).
+- **Trellis** quants (`IQ*_KT`), **MXFP4**, **Q8_KV** fast KV type, first-class
+  **Bitnet**.
+- ⚠️ **K-quant / `-rtr` trap (critical for the 3090):** K-quants have **no CUDA
+  row-interleaved path**; `-rtr` (run-time repack) forces CPU-resident K-quant
+  tensors to CPU matmul even when GPU would be faster. Prefer IQK/R4/Trellis/
+  MXFP4 variants, or skip `-rtr` and let `--fit`/`-ot` place experts.
+
+## Fit for our servers
+
+### Dankserver — NVIDIA RTX 3090 (24 GB VRAM), 62 GB RAM, CUDA
+Serves **Ornith-1.5-35B-A3B CRACK**, a 35B **MoE** (~36 experts; the CRACK
+GGUF README lists 256/8-active — verify `n_expert` from the GGUF).
+- **Mainline** is the natural baseline: `Q4_K_M`/`Q5_K_M`, FlashAttention to
+  shrink KV, `GGML_CUDA_FORCE_MMQ=1` (default on RTX 30xx, lower VRAM),
+  `--cpu-moe` to spill experts to the 62 GB RAM, and `ngram-mod` drafts (MoEs
+  want long drafts).
+- **ik_llama.cpp** is the more compelling *upgrade* here: fused MoE (`-fmoe`),
+  **offload-only-active-experts** (`-ooae`) which — with only ~8 active experts —
+  sharply cuts RAM→VRAM traffic, `--fit`, smart-expert-reduction (`-ser`), GDN
+  fusion for the SSM path, and MTP. Requires the CUDA-90 + AVX-512 build and care
+  with the `-rtr` K-quant trap.
+- **ROCmFPX** does **not** help a CUDA box (ROCm-only formats, CUDA disabled).
+  The 3090's lever is CUDA quant + FastMTP, not ROCmFPX.
+
+### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified VRAM), ROCm 7.2.4
+Serves **kingjones Qwen3.8-27B ROCmFP4 STRIX MTP** (the production build).
+- **ROCmFPX is the native fit**: IU4/MTP/FPX, HIP+Vulkan kernels, TurboQuant KV,
+  262K ctx / 1 slot. Measured 8K MTP ~30.3 tok/s (2.83× vs `Q4_K_M`).
+- **ik_llama.cpp** — ROCm is its **weakest** backend (gfx1151 not validated) and
+  the k-quant-no-hip-row-interleave issue bites there too. Keep ROCmFPX.
+- **Mainline** ROCm build works for correctness (standard quants) but loses FPX,
+  IU4, TurboQuant, MTEA, and agent presets.
+
+## When to reach for which
+- **Stay mainline** if you want one portable upstream for both boxes, the widest
+  GGUF ecosystem, and standard FlashAttention/MTP. Best fit for **Dankserver**
+  if you don't want the ik_llama.cpp build surface.
+- **Use ik_llama.cpp** for Dankserver when the MoE offload and fused-MoE wins
+  justify an extra build (watch `-rtr` + K-quants).
+- **Use ROCmFPX** for Dankclaw — it is purpose-built for gfx1151.
+- **ROCmFPX** is for the AMD fleet; **CUDA/FastMTP** for the NVIDIA box. Do not
+  port ROCmFP4 to the 3090 via Vulkan as a primary path.
+
+## See Also
+- [[Llama Cpp Code Diff]] — the code-level quantization diff + comparison table
+- [[dealign-ornith-crack]] — the CRACK model on Dankserver
+- [[kingjones-strix-mtp]] — the ROCmFP4 model on Dankclaw
+- [[Quantization by Hardware Architecture]] — which quant fits which GPU/CPU
+- [[SGLang]] — the structured-generation serving engine (the other lane)
+
+## Citations
+- Mainline — https://github.com/ggml-org/llama.cpp
+- charlie12345/ROCmFPX — https://github.com/charlie12345/ROCmFPX ·
+  rocmfp4-llama — https://github.com/charlie12345/rocmfp4-llama · rocmfp4 —
+  https://github.com/charlie12345/rocmfp4
+- ik_llama.cpp — https://github.com/ikawrakow/ik_llama.cpp · docs ·
+  https://ikawrakow-ik_llama-cpp.mintlify.app/
+- ROCmFPX quant/enum/serve detail —
+  https://github.com/charlie12345/ROCmFPX/blob/main/ggml/include/ggml.h
+- SGLang — https://github.com/sgl-project/sglang · https://docs.sglang.io/
@@ -0,0 +1,154 @@
+---
+type: research
+title: SGLang
+description: What SGLang is, how it works (RadixAttention, paged KV, overlap scheduler, structured output), how it compares to llama.cpp and vLLM, and whether it fits Dankserver (RTX 3090) and Dankclaw (Strix Halo).
+tags:
+  - sglang
+  - serving
+  - inference
+  - structured-output
+  - radix-attention
+  - llama.cpp
+timestamp: 2026-09-01
+---
+
+# SGLang
+
+SGLang (for **S**tructured **G**eneration **Lang**uage) is a high-throughput
+serving engine for large (and vision-language) models, from the LMSYS / UC
+Berkeley team, published at NeurIPS 2024. Today it is one of the two dominant
+open-source serving engines, alongside vLLM. People talk about it because it
+makes "call the model, branch, force the output into JSON" programs *fast*.
+
+## What it is — in one paragraph
+
+You serve an LLM by writing a small program: call the model, then based on its
+output call it again, branch into parallel attempts, and force the answer into a
+JSON schema. SGLang co-designs a Python-embedded language with a runtime that can
+*see* the program's structure, then exploits it: it caches and reuses the model's
+"working memory" (the KV cache) across requests that share a prompt prefix,
+batches new requests into running generations without waiting for a batch to
+drain, and forces tightly-structured output (JSON / regex / tool tags) by masking
+illegal tokens one step at a time — or several at once. Reported payoff:
+**up to 6.4× higher throughput** on structured / agentic / RAG workloads vs vLLM,
+TGI, and similar.
+
+## Core architecture
+
+Two halves that can also run independently:
+- **Frontend** — a Python DSL (`@sgl.function`, `gen`/`select`/`fork`/`join`/
+  `extend`) for multi-call LLM programs with branching, parallel sampling, tool
+  calls, and structured output.
+- **Backend runtime** — a high-throughput server (`python -m sglang.launch_server`)
+  or an in-process `Engine`. OpenAI-compatible `/v1`, so RadixAttention and
+  structured decoding work with a vanilla OpenAI client.
+
+Building blocks:
+- **z-normalization** — pre-compile per-request sampling metadata into contiguous
+  tensors so the GPU sampler does one bulk copy per batch per step (kills the
+  CPU scheduler bottleneck).
+- **PagedTensor** — the KV cache is paged (fixed-size pages) and laid out
+  non-contiguously in a global pool; mirrors vLLM's PagedAttention.
+- **RadixAttention** — a radix tree of shared token prefixes; on a new request it
+  walks to the longest cached prefix, reuses those KV blocks, and only prefills
+  the divergent suffix. Payoff is compute + throughput for long shared prefixes
+  (RAG, agents), not raw memory.
+- **Overlap scheduler** — runs the CPU prep for step t+1 *under* the GPU forward
+  of step t, so kernels launch back-to-back with no idle bubble (plus CUDA
+  graphs + torch.compile).
+- **Structured / output-guided generation** — compiles the constraint (JSON
+  schema / regex / EBNF) into a finite-state machine over the vocabulary and masks
+  disallowed logits; **jump-forward** decoding emits forced token runs in one step.
+  Backends: XGrammar (default), Outlines.
+
+## SGLang vs llama.cpp vs vLLM
+
+| Dimension | SGLang | llama.cpp |
+|-----------|--------|-----------|
+| Runtime | Python + CUDA/C++ kernels | C/C++, no Python required |
+| GPU requirement | GPU required, **no CPU path** | CPU-only / GPU-only / hybrid offload |
+| Model format | HF safetensors; AWQ/GPTQ/FP8 on the fly; **GGUF via llama.cpp backend** | **GGUF only** (convert first) |
+| KV reuse | RadixAttention (always-on radix tree) | basic prompt caching, no radix sharing |
+| Structured output | first-class (XGrammar, jump-forward) | narrower |
+| Concurrency | continuous batching + overlap scheduler | single-node, lower concurrency |
+| Multi-GPU | native tensor parallelism | layer-split, more manual |
+| Setup | `pip install sglang[all]`, venv, CUDA match | static binary / Docker, minutes |
+| License | Apache 2.0 | MIT |
+
+**Pick SGLang** when you need many concurrent users, structured / JSON / tool-call
+output, or repeated long prefixes (RAG, agents) where RadixAttention pays off, and
+you own a GPU + Python/CUDA dependency chain. **Pick llama.cpp** for the fastest
+download-to-endpoint path, CPU/edge, GGUF libraries, and quantization breadth.
+**Pick vLLM** for the broadest model-hardware coverage and mature multi-GPU.
+
+The paper's headline 6.4× / 5× numbers are vs **vLLM v0.1** (no prefix caching);
+the gap narrows vs modern vLLM — benchmark on your own traffic.
+
+## Deployment model & gotchas
+- **Pluggable backends:** default vLLM engine, SGLang's own `sglang` engine, or a
+  recent beta **`llama.cpp` backend** that serves GGUF and reuses `llama-server`.
+- **VRAM headroom runs higher** than llama.cpp (pre-allocated KV pool + radix
+  tree). Size with `--mem-fraction-static`, `--context-length`,
+  `--chunked-prefill-size`. OOM → lower context / fraction or a smaller quant.
+- **Model-format lock-in:** wants safetensors + AWQ/GPTQ/FP8 (or GGUF via the
+  llama.cpp backend). The current CPU-Q6_K_P and ROCmFP4-MTP models are **GGUF**.
+- **AMD / ROCm: yes, officially.** Prebuilt `lmsysorg/sglang` ROCm Docker image or
+  `pip`. Quant on AMD: FP8 (Aiter/Triton), AWQ (Triton), MXFP4 (CDNA3/4),
+  `petit_nvfp4`. Marlin-based methods are **out** on AMD. Kernel work targets
+  **datacenter Instinct (CDNA2/3/4)** — consumer/client parts like **Strix Halo
+  (gfx1151)** are supported at the ROCm level but are not the kernel-tuning target.
+- **Prefix diversity kills RadixAttention** (per-user/personalized prompts → tree
+  degenerates). Single-replica trees need prefix-aware routing across a fleet
+  (`sglang-router --policy cache_aware`). Fast-moving API — pin versions.
+
+## Relevance to our servers
+
+### Dankserver — NVIDIA RTX 3090 (24 GB) + 62 GB RAM — 35B MoE
+A **genuine fit candidate (with a migration step)**. The 3090 is CUDA-native.
+A 35B MoE (~3B active) is ideal for continuous batching + structured generation,
+and 24 GB can hold a quantized 35B MoE (FP8/AWQ safetensors ~18–20 GB) with room
+for KV. MoE expert parallelism is supported.
+- **What changes:** the current model is a **CPU GGUF (Q6_K_P)**; SGLang wants
+  safetensors (FP8/AWQ) for CUDA, OR use SGLang's **`llama.cpp` backend** to serve
+  the existing GGUF under SGLang's frontend. Check for an AWQ/FP8 variant of the
+  HauhauCS checkpoint on Hugging Face.
+- **Where it wins:** repeated structured/JSON/tool calls or a long shared system
+  prompt (RAG) → RadixAttention + jump-forward give big gains vs the current
+  sequential CPU pipeline. Simple low-concurrency chat → the operational weight
+  may not pay off.
+
+### Dankclaw — AMD gfx1151 Strix Halo (64 GiB unified) — Qwen3.8-27B ROCmFP4 MTP
+A **niche / experimental fit, not a drop-in replacement**. 64 GiB is generous and
+SGLang-on-ROCm is supported (Docker/pip), but:
+- Our model is a **custom ROCmFP4/MTP GGUF**. SGLang's primary path wants
+  safetensors + an AMD quant (FP8/AWQ); MXFP4 needs CDNA3/4 and Strix Halo (CDNA
+  3.5 *client*) is **not** an SGLang/AMD kernel target. Quant availability for this
+  exact MTP model is unverified, and we'd lose the bespoke MTP draft-head speedup.
+- **Best-fit path:** SGLang's **`llama.cpp` backend** on top of our existing ROCm
+  llama.cpp / GGUF — gain the structured-output DSL, RadixAttention, and scheduler
+  without rebuilding the kernel stack. Or test the native `lmsysorg/sglang` ROCm
+  image as a proof-of-concept. Treat it as an enhancement/experiment, not a
+  production swap, until a quant for the MTP model exists and perf is measured on
+  gfx1151.
+
+## Tradeoffs at a glance
+- Higher baseline VRAM than llama.cpp (KV pool + radix tree).
+- GPU/accelerator required — no pure-CPU path (llama.cpp is the only option of the
+  trio for a CPU-only box).
+- Model-format lock-in (safetensors/AWQ/GPTQ/FP8, or GGUF via llama.cpp backend).
+- Marlin-based quant out on AMD; Strix Halo is not an MXFP4 target.
+- Prefix diversity can nullify RadixAttention; pin versions; watch cache-report.
+
+## See Also
+- [[Llama Cpp Runtimes]] / [[Llama Cpp Code Diff]] — the llama.cpp side of the trio
+- [[dealign-ornith-crack]] — the CRACK MoE on Dankserver
+- [[kingjones-strix-mtp]] — the ROCmFP4 MTP model on Dankclaw
+- [[Quantization by Hardware Architecture]]
+
+## Citations
+- SGLang paper (arXiv:2312.07104) — https://arxiv.org/abs/2312.07104 · NeurIPS 2024
+- Repo — https://github.com/sgl-project/sglang · docs — https://docs.sglang.io/
+- AMD / ROCm — https://docs.sglang.io/docs/hardware-platforms/amd_gpu ·
+  Structured Outputs — https://docs.sglang.io/docs/advanced_features/structured_outputs
+- lmsysorg/sglang ROCm images — https://hub.docker.com/r/lmsysorg/sglang
+- SGLang vs llama.cpp — https://markaicode.com/vs/sglang-vs-llamacpp/
@@ -0,0 +1,45 @@
+# Service Research Brief
+
+Consolidated research from the `frank-default-small` research subagents (2026-08-14) plus the pi agent knowledge base. This is the source material behind the per-service docs pages.
+
+## Method
+
+Four parallel researchers fetched primary sources: official Docker docs, GitHub READMEs, container source (ComfyUI `cli_args.py`, OpenWebUI `env.py`), the Gluetun wiki, and Grafana/Loki/Prometheus docs. The first parallel run failed with `503 Loading model` (llama.cpp on Dankserver reloading); the retry completed for three of four, and the infra brief completed solo.
+
+## Key Findings
+
+### Dankserver stack
+
+- **RomM** — `ghcr.io/rommapp/romm:latest`; FastAPI+React; SQLite data; needs `ROMM_USERNAME`/`ROMM_PASSWORD`/`ROMM_SECRET_KEY` at first boot; volumes `/romm/library` + `/romm/resources`.
+- **Copyparty** — `ghcr.io/9001/copyparty:latest`; single-file Python server; port 3923; per-folder `.cpy` auth; `--resumable` for large uploads.
+- **Calibre-Web** — `lscr.io/linuxserver/calibre-web:latest` (matches the live container); `PUID=1000 PGID=1000 TZ=Etc/UTC`; `/config` + `/books`; external binary paths `unrar`/`kepubify`/`ebook-convert`; library path must be set in admin UI; DB path wrong ⇒ login succeeds but library empty.
+- **Calibre** — host tool, not a container; Calibre-Web points at its library directory.
+
+### Dankclaw stack
+
+- **SearXNG** — `ghcr.io/searxng/searxng:latest`; `network_mode: service:gluetun` (KB Pattern 2); Valkey cache replaces Redis; `settings.yml` keys: `server.secret_key`, `base_url`, `limiter: true`, `valkey.url`, `outgoing.*`, `search.formats: [html, json]`; JSON API rate-limited (4/h untrusted) — trusted-group the agent IP.
+- **Gluetun** — `qmcgaw/gluetun:latest`; `NET_ADMIN` + `/dev/net/tun`; port publishing moves to gluetun; `VPN_SERVICE_PROVIDER`, `VPN_TYPE`, `WIREGUARD_PRIVATE_KEY`; DNS: prefer `dot` upstreams, never `DNS_UPSTREAM_PLAIN_ADDRESSES` to a LAN address (leaks); verify with `podman exec gluetun curl -s ifconfig.me`.
+- **ComfyUI** — CPU backend required: ROCm wheels have no gfx1151 (Strix Halo) kernels; `--cpu` flag confirmed in `cli_args.py` ("To use the CPU for everything (slow)"); do NOT use CUDA images; models/input/output persisted.
+- **OpenWebUI** — `ghcr.io/open-webui/open-webui:main` (NOT `:cuda` on AMD); port 8080 in container; OpenAI-compatible endpoint to `host.containers.internal:7770/v1`; `/health`.
+- **Hermes** — Nous Research self-improving agent; MIT; official image `docker.io/nousresearch/hermes-agent:latest` (verified pullable 2026-08-14); TUI + Telegram/Discord/etc.; points at local llama.cpp `:7770/v1`; data volume for skills/memory.
+- **Gitea** — migrate live `/opt/gitea/podman-compose.yml` (postgres:16-alpine + gitea + caddy); **never change SECRET_KEY/INTERNAL_TOKEN after migration**; keep bind mounts `/opt/gitea/data/*`; DB password must move to `.env`.
+
+### Infrastructure
+
+- **Caddy** — `caddy:2-alpine`; directive is `basicauth` (renamed from `basic_auth` in v2.8+); hash with `caddy hash-password`; homepage via `root * /srv/http` + `file_server`; `admin off` recommended.
+- **Grafana/Prometheus** — `grafana/grafana:latest` + `prom/prometheus:latest`; prometheus.yml scrapes node-exporters on both hosts (192.168.0.143/235:9100) + Gitea/Grafana/Loki metrics; Grafana datasource provisioning `apiVersion: 1` with `access: proxy`.
+- **Node-Exporter** — `prom/node-exporter:latest`; `network_mode: host`, `pid: host`, read-only `/proc /sys /` mounts.
+- **Logs** — **Promtail is EOL 2026-03-02** ⇒ use **Grafana Alloy** (`grafana/alloy:latest`, River config) as the shipper; Loki `grafana/loki:latest` single-binary mode, TSDB schema v13, filesystem storage, compactor retention 30d; health `/ready`.
+
+## Open Items
+
+- VPN provider choice for Gluetun (`.env` on Dankclaw).
+- Calibre library path (user-entered later).
+- Caddy bcrypt hashes at deploy (`caddy hash-password`).
+- B2 bucket/app key for Backblaze backups (deferred).
+- Public DNS for `*.franksdank.cloud` domains (pending).
+
+## See Also
+
+- Per-service pages: [[SearXNG]] [[Gluetun]] [[ComfyUI]] [[OpenWebUI]] [[Hermes]] [[Gitea]] [[RomM]] [[Copyparty]] [[Calibre-Web]] [[Caddy]] [[Grafana]] [[Prometheus]] [[Loki]] [[Alloy]] [[Node-Exporter]]
+- [[Observability]] — the stack this research feeds
@@ -0,0 +1,24 @@
+# Research
+
+Subagent research briefs for the two hosts. Each page is a focused research
+findings doc; the project root index ([serverAppConfigs-docs/index.md](../index.md)) links back.
+
+## LLM Inference
+
+- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit
+- [[Llama Cpp Code Diff]] — code-level quantization diff + comparison table + server recommendations
+- [[SGLang]] — the structured-generation serving engine (llama.cpp/vLLM comparison, our- servers fit)
+
+## Image / Media
+
+- [[ComfyUI ROCm and Vulkan]] — live-verified GPU path for ComfyUI on Strix Halo
+- [[Lemonade and sd.cpp]] — the ggml-family image-generation alternative
+- [[Jellyfin]] — media server research brief
+
+## Backups
+
+- [[Backblaze Integration]] — off-site backup plan (future)
+
+## Legacy
+
+- [[Service Research Brief]] — consolidated subagent research behind these pages