Frank
135bb2194d
- config.yml: default/commit -> frank-medium (dankserver); task/tiny/smol/vision -> frank-small (dankclaw-3090); vision xhigh - models.yml: add llamacpp (dankclaw-3090) provider; frank-large preserve_thinking + context notes - niri: focus-ring/border/shadow layout, hot-corners off, re-add noctalia.kdl include; blankpad eDP-1 scale 1.0 - memories: flush rollouts; retire frank_large + marreta_authentik skills; add auth-chain/kv-pool/quadlet-log skills - managed-skills: 40+ new fleet runbook skills - agent/skills: import built-in skill set
2.4 KiB
2.4 KiB
System & Hardware Topology
- dankserver: Dual AMD R9700 GPUs.
- Issue: RCCL fails on chipset-attached PCIe (PCIEX16_2) because lanes lack required atomic-op routing.
- Fix: Use a $30 M.2→PCIe x4 riser to move card #2 to CPU-attached M.2_2 slot.
- dankclaw: RTX 3090 (PCIe 4.0 x4 dock) + AMD Strix Halo (gfx1151, Vulkan).
- Kernel: Ubuntu 24.04,
amd_iommu=off.
- Kernel: Ubuntu 24.04,
LLM & Software Stack
- vLLM / ROCm: Version 0.27.1 / torch 2.11.0 / ROCm 7.14.
- Constraint: HIP runtime reads bare
TEMPandHOSTenv vars; naming these explicitly can cause collisions. - Constraint:
GGML_CUDA_FA_ALL_QUANTSrequired for flash-attention with quantized KV.
- Constraint: HIP runtime reads bare
- llama.cpp (ik_llama.cpp):
- Model: Qwen3.8-Flash-Next-Uncensored-i1-GGUF (i1-Q4_K_M).
- Performance: ~3.8-4.0 t/s decode (bottlenecked by PCIe x4 sync/graph splits), ~192 t/s prefill.
- Note: 30 t/s decode is unreachable via current PCIe x4/split-mode setup; requires mainline PR #28901 for capture-safe graphs.
Service Management (Quadlet & Podman)
- Persistence:
restart: unless-stoppedsurvives crashes, not explicit stops. Every container must have a Quadlet.containerfile for boot-persistence. - frank-large (HALOGEN): Rootless Quadlet. Weights pinned (~68 GiB). KV pool: 524288 positions (~14.4 GiB).
- Stall Behavior: Multi-minute full-CPU stalls (no disk/output) are documented behavior under co-tenancy with
HALOGEN_FLASH_PIN_TRUNK=1. Do not restart. - First-touch: 45-60s silence after restart as engine first-touches fresh KV pool.
- Constraints:
maxTokens<= 65536. Prompt + max_tokens must fit KV pool.
- Stall Behavior: Multi-minute full-CPU stalls (no disk/output) are documented behavior under co-tenancy with
- frank-small: Defaults to sampling (temp 1.0, top_k 20, top_p 0.95).
Networking & Auth
- Marreta Auth Chain: Caddy TLS → Anubis (:8923) → Authentik Outpost → Marreta (:8017).
- Fix: Populated
invalidation_flow,authentication_flow, andauthorization_flowto prevent Rust outpost serde crashes. - Probing: Host-to-listener returns HTTP 000 (Caddy netns); probe inside netns or grep config for internal status.
- Fix: Populated
Operational Lessons
- Memory:
freecommand lies about available memory (doesn't account for pinned/reserved weights). - Skills Discovery:
ompscans~/.omp/agent/skills/. - Root-Cause: PCIe x4 dock provides ~29 GB/s, not 130 GB/s.
- Logs: Use
podman logsorjournalctl --user -ufor rootless Quadlets.