## System & Hardware Topology
- **dankserver**: Dual AMD R9700 GPUs. 
  - *Issue*: RCCL fails on chipset-attached PCIe (PCIEX16_2) because lanes lack required atomic-op routing. 
  - *Fix*: Use a $30 M.2→PCIe x4 riser to move card #2 to CPU-attached M.2_2 slot.
- **dankclaw**: RTX 3090 (PCIe 4.0 x4 dock) + AMD Strix Halo (gfx1151, Vulkan). 
  - *Kernel*: Ubuntu 24.04, `amd_iommu=off`.

## LLM & Software Stack
- **vLLM / ROCm**: Version 0.27.1 / torch 2.11.0 / ROCm 7.14. 
  - *Constraint*: HIP runtime reads bare `TEMP` and `HOST` env vars; naming these explicitly can cause collisions.
  - *Constraint*: `GGML_CUDA_FA_ALL_QUANTS` required for flash-attention with quantized KV.
- **llama.cpp (ik_llama.cpp)**: 
  - *Model*: Qwen3.8-Flash-Next-Uncensored-i1-GGUF (i1-Q4_K_M).
  - *Performance*: ~3.8-4.0 t/s decode (bottlenecked by PCIe x4 sync/graph splits), ~192 t/s prefill.
  - *Note*: 30 t/s decode is unreachable via current PCIe x4/split-mode setup; requires mainline PR #28901 for capture-safe graphs.

## Service Management (Quadlet & Podman)
- **Persistence**: `restart: unless-stopped` survives crashes, not explicit stops. Every container must have a Quadlet `.container` file for boot-persistence.
- **frank-large (HALOGEN)**: Rootless Quadlet. Weights pinned (~68 GiB). KV pool: 524288 positions (~14.4 GiB). 
  - *Stall Behavior*: Multi-minute full-CPU stalls (no disk/output) are documented behavior under co-tenancy with `HALOGEN_FLASH_PIN_TRUNK=1`. Do not restart.
  - *First-touch*: 45-60s silence after restart as engine first-touches fresh KV pool.
  - *Constraints*: `maxTokens` <= 65536. Prompt + max_tokens must fit KV pool.
- **frank-small**: Defaults to sampling (temp 1.0, top_k 20, top_p 0.95).

## Networking & Auth
- **Marreta Auth Chain**: Caddy TLS → Anubis (:8923) → Authentik Outpost → Marreta (:8017).
  - *Fix*: Populated `invalidation_flow`, `authentication_flow`, and `authorization_flow` to prevent Rust outpost serde crashes.
  - *Probing*: Host-to-listener returns HTTP 000 (Caddy netns); probe inside netns or grep config for internal status.

## Operational Lessons
- **Memory**: `free` command lies about available memory (doesn't account for pinned/reserved weights). 
- **Skills Discovery**: `omp` scans `~/.omp/agent/skills/`. 
- **Root-Cause**: PCIe x4 dock provides ~29 GB/s, not 130 GB/s. 
- **Logs**: Use `podman logs` or `journalctl --user -u` for rootless Quadlets.