kb: retire pi-installation association (OMP is the primary harness)

Purge live-state claims that ~/.pi is our harness; keep general
pi documentation in omp-docs/ and dated historical records.

- README.md: single config source; Legacy Notes reworded
- AGENTS.md: drop 'like ~/.pi did' framing
- APPEND_SYSTEM.md: okf-spec -> omp-docs/references/okf-spec.md
- dotfiles/zsh/.zshrc: systemprompt alias -> APPEND_SYSTEM.md
- managed skill llm-fleet-model-swap + model_config_sync skill:
  single-source models.yml; validation script YAML-only
- memories (MEMORY.md, memory_summary.md, learned.md): single source
- omp-docs: .okf root -> ~/.omp/omp-docs; de-live pi-era claims in
  pi/index.md, pi-web-access.md, engineering/agents-files.md
This commit is contained in:
Frank committed 2026-09-17 19:24:02 -07:00
1 parent eebcfbf5ac
commit c3f30bdbd0
23 files changed
+81 -76

No files matched your search

@@ -1,34 +1,26 @@
## System & Hardware Topology
- **dankserver**: Dual AMD R9700 GPUs.
- *Issue*: RCCL fails on chipset-attached PCIe (PCIEX16_2) because lanes lack required atomic-op routing.
- *Fix*: Use a $30 M.2→PCIe x4 riser to move card #2 to CPU-attached M.2_2 slot.
- **dankclaw**: RTX 3090 (PCIe 4.0 x4 dock) + AMD Strix Halo (gfx1151, Vulkan).
- *Kernel*: Ubuntu 24.04, `amd_iommu=off`.
## System Topology & Hardware
- **dankclaw**: AMD Strix Halo APU (gfx1151) + RTX 3090 in M.2 dock (PCIe x4, ~29 GB/s sequential). Combined GPU budget ~88 GiB. OS: Ubuntu 24.04.
- **dankserver**: ASUS TUF GAMING B650-E WIFI. Dual AMD R9700 GPUs.
- **RCCL Constraint**: Chipset-attached PCIe lanes fail RCCL atomics (broadcast/all-reduce/send-recv). Fix: Move card #2 to CPU-attached M.2_2 slot via PCIe x4 riser to restore atomic-op routing.
## LLM & Software Stack
- **vLLM / ROCm**: Version 0.27.1 / torch 2.11.0 / ROCm 7.14.
- *Constraint*: HIP runtime reads bare `TEMP` and `HOST` env vars; naming these explicitly can cause collisions.
- *Constraint*: `GGML_CUDA_FA_ALL_QUANTS` required for flash-attention with quantized KV.
- **llama.cpp (ik_llama.cpp)**:
- *Model*: Qwen3.8-Flash-Next-Uncensored-i1-GGUF (i1-Q4_K_M).
- *Performance*: ~3.8-4.0 t/s decode (bottlenecked by PCIe x4 sync/graph splits), ~192 t/s prefill.
- *Note*: 30 t/s decode is unreachable via current PCIe x4/split-mode setup; requires mainline PR #28901 for capture-safe graphs.
## LLM Model Portfolio
- **frank-large (halogen)**: Qwen3.8-27B-ROCmFPX-Q8_0. Runs on Strix Halo (gfx1151).
- *KV Pool*: 524288 positions (~14.4 GiB). Admission is pool-bound, not slot-bound (caps at 3 simultaneous full-length requests).
- *Behavior*: Defaults to greedy. Stalls (full-CPU, no disk activity, no output) under co-tenancy with pinned trunk are documented behavior, not crashes.
- *Constraints*: maxTokens <= 65536. First request after restart may be silent ~45-60s while first-touching KV pool.
- **frank-small**: Qwen3.8-27B GSQ-RCO IQ3_S. Runs on RTX 3090 (M.2 dock). Defaults to sampling (temp 1.0, top_k 20, top_p 0.95).
- **ik_llamacpp**: mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF (i1-Q4_K_M).
- *Backend*: CUDA (3090) + Vulkan (Strix Halo) split.
- *Performance*: ~3.8-4.0 t/s decode, ~192 t/s prefill.
- *Bottleneck*: PCIe x4 launch/sync overhead; CUDA graphs currently abort on hc ops (ggml_repeat_4d).
## Service Management (Quadlet & Podman)
- **Persistence**: `restart: unless-stopped` survives crashes, not explicit stops. Every container must have a Quadlet `.container` file for boot-persistence.
- **frank-large (HALOGEN)**: Rootless Quadlet. Weights pinned (~68 GiB). KV pool: 524288 positions (~14.4 GiB).
- *Stall Behavior*: Multi-minute full-CPU stalls (no disk/output) are documented behavior under co-tenancy with `HALOGEN_FLASH_PIN_TRUNK=1`. Do not restart.
- *First-touch*: 45-60s silence after restart as engine first-touches fresh KV pool.
- *Constraints*: `maxTokens` <= 65536. Prompt + max_tokens must fit KV pool.
- **frank-small**: Defaults to sampling (temp 1.0, top_k 20, top_p 0.95).
## Software Infrastructure
- **Podman**: Rootless deployment using Quadlet units for boot persistence. Note: `restart: unless-stopped` survives crashes, but not explicit stops (kernel power-off).
- **Authentik**: Anubis guard chain. Marreta app connected via ProxyProvider (fixed null flow crash).
- **AGENTS.md**: Structured as (1) Active-topology table, (2) Runbook pointers, (3) Key Lessons Learned.
## Networking & Auth
- **Marreta Auth Chain**: Caddy TLS → Anubis (:8923) → Authentik Outpost → Marreta (:8017).
- *Fix*: Populated `invalidation_flow`, `authentication_flow`, and `authorization_flow` to prevent Rust outpost serde crashes.
- *Probing*: Host-to-listener returns HTTP 000 (Caddy netns); probe inside netns or grep config for internal status.
## Operational Lessons
- **Memory**: `free` command lies about available memory (doesn't account for pinned/reserved weights).
- **Skills Discovery**: `omp` scans `~/.omp/agent/skills/`.
- **Root-Cause**: PCIe x4 dock provides ~29 GB/s, not 130 GB/s.
- **Logs**: Use `podman logs` or `journalctl --user -u` for rootless Quadlets.
## Durable Lessons
- **ROCm**: Use layer-split-not-tensor; mmap/PLE constraints apply; HIP env collisions (`TEMP`/`HOST`).
- **Memory**: `free` command lies about available memory (doesn't account for pinned/reserved weights).
- **PCIe**: M.2 x4 dock is ~29 GB/s; do not assume x16 bandwidth.
- **Skill Discovery**: `omp` scans `~/.omp/agent/skills/` only. New skills surface on the next session.
@@ -1 +1 @@
dankclaw/dankserver systems featuring ROCmFPX and llama.cpp. frank-large (HALOGEN) uses a 524288 KV pool and is managed via rootless Quadlets (mandatory for persistence). Key hardware fix: M.2 riser for R9700 chipset PCIe issues. Key software behavior: Halogen stalls are documented (don't restart), first-request silence is normal, and HIP env vars (TEMP/HOST) must be handled carefully. Marreta is secured via an Authentik/Anubis auth chain.
Host: dankclaw (Strix Halo + 3090) & dankserver (B650-E + R9700s). Models: frank-large (halogen, 524k KV pool, greedy), frank-small (sampling), ik_llamacpp (CUDA/Vulkan split). Infrastructure: Podman Quadlets (persistence), Authentik (Marreta chain). Key Fixes: RCCL requires CPU-attached lanes (M.2_2 riser). Halogen stalls are not crashes. `free` lies about pinned memory.
@@ -1,5 +1,15 @@
# Raw Memories
## 01a0ac72-2620-750f-82dd-79c84e00e98d
updated_at: 1789631572
DANKCLAW HARDWARE: AMD Strix Halo APU (gfx1151), Radeon 8060S iGPU. Has an M.2 dock slot used to carry an RTX 3090 at PCIe x4 (link downgraded from x16). This is distinct from dankserver (ASUS TUF GAMING B650-E WIFI) which has a free CPU-attached M.2_2 slot for an R9700 riser (~$30 M.2→PCIe x4 riser).
DANKCLAW LLM DEPLOYMENTS:
- frank-small: Qwen3.8-27B GSQ-RCO IQ3_S + MTP on RTX 3090 (M.2 dock, PCIe x4). Decode 61-82 t/s, prefill 430-1013 t/s. VRAM at 92% (22,614/24,576 MiB). PCIe utilization only ~3% during decode (compute-bound, not transport-bound). --parallel 1, 8 CPU threads.
- frank-large (halogen): runs on the iGPU (gfx1151), decode 37-48 t/s, MTP commit 1.60-1.88 tokens/round. Prefix caching confirmed working.
UNANSWERED QUESTION: User asked specifically about the M.2 slot in context of a Qwen Flash deployment running with pipeline parallelism on dankclaw. The assistant's response was aborted (OMP process exited) before answering. This likely refers to a llama.cpp pipeline-parallel setup where the Qwen Flash model's layers are split across the iGPU and the 3090 in the M.2 dock, using the M.2 slot as the interconnect between the two compute devices.
## 01a0ac6b-61a9-7505-b00a-52dd4cbb4ff3
updated_at: 1789599423
## Rootless Podman restart policy pitfall
@@ -0,0 +1,4 @@
thread_id: 01a0ac72-2620-750f-82dd-79c84e00e98d
updated_at: 1789631572
User asked about M.2 slots on dankclaw, clarified they meant the Qwen Flash pipeline-parallel deployment. Dankclaw is an AMD Strix Halo APU (gfx1151) with an M.2 dock carrying an RTX 3090 at PCIe x4. The M.2 slot docs in the repo are split: dankserver (B650-E, M.2_2 free for R9700 riser) vs dankclaw (M.2 dock for 3090). The Qwen Flash pipeline-parallel M.2 question was aborted before being answered.
@@ -1,4 +0,0 @@
Verifying the Marreta auth chain:
1. Probe `http://localhost:8017/` from host (expected 000, listener is in Caddy netns).
2. Check `authentik-proxy-provider-app-link` for null flows.
3. Ensure non-browser clients can bypass JS challenges or use backend datasource URLs.
@@ -1,4 +0,0 @@
Verify if the request fits the allocated KV pool:
1. Run `podman logs frank-large | grep 'kv pool:'`.
2. Compare 'reserved' count to 'requested' count.
3. Note: /health does not prove pool size; absence of WARNING is not evidence of headroom.
@@ -1,4 +1,7 @@
To retrieve logs for a rootless Quadlet service (e.g., frank-large):
1. Use `podman logs frank-large` for container-specific output.
2. Use `journalctl --user -u frank-large` for systemd-integrated output.
3. Use `podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'` to filter for memory/allocations.
Retrieving logs for a rootless Quadlet container.
Specific considerations:
1. Use `podman logs <container>` for standard output.
2. Use `journalctl --user -u <unit_name>` for systemd-integrated logs.
3. Filter specifically for dmalloc/kv pool events to monitor memory state.
4. Note that rootless quadlet log retrieval has specific gotchas regarding standard output capturing in nested namespaces.