command-center/agent/memories/--home-blank-Projects-serverAppConfig--/raw_memories.md

Raw Memories

01a0ac6b-61a9-7505-b00a-52dd4cbb4ff3

updated_at: 1789599423

Rootless Podman restart policy pitfall

• restart: unless-stopped only survives crashes, not explicit stops (e.g., kernel power-off sends a polite stop signal).
• Result: bare containers without a Quadlet unit are gone after reboot with no alert.

Repo serverAppConfig deployment pattern

• Every container must have a Quadlet .container file (git-tracked) to be boot-persistent and self-healing.
• The unit file is the container's birth certificate + pension plan + ghost in one.

Historical incidents (from story)

• Aug 26: Caddy never returned after reboot (no unit).
• Sep 11: Five observability services found unitless after reboot.

Persona note

• Model: frank-medium via llamacpp (dankserver). Persona: Soviet-style computer systems engineer with Munroe-esque humor. Uses "Comrade" address.

01a0a7ad-42e2-739d-bc1c-dc405ca02cbd

updated_at: 1789574671

Dankserver Dual R9700 RCCL Chipset Issue

Hardware: 2x AMD R9700 on ASUS board. Card #1 on CPU-attached PCIe x16. Card #2 on chipset-attached PCIe x16 slot (PCIEX16_2, which is x1 mode per ASUS spec).

Problem: RCCL (ROCm Collective Communications Library) fails on the chipset-attached card. Bare 2-rank torch.distributed probe results:

• broadcast: OK (CPU card) / FAIL (chipset card)
• all_reduce: OK / FAIL
• barrier: OK / FAIL
• send/recv (P2P): hang / FAIL

Implication: send/recv failure kills pipeline parallelism too (vLLM's PP handoff IS that primitive). No software-only escape.

Root cause: Chipset-attached PCIe lanes don't route atomic operations the way RCCL's kernels require. CPU-attached lanes do.

Fix (recommended): M.2→PCIe x4 riser (~$30) to move card #2 to M.2_2 slot. ASUS spec confirms M.2_1 and M.2_2 are CPU-attached; M.2_2 is free/empty. Restores atomic-op routing → RCCL works → TP=2 functional.

Alternative: Board with CPU x8/x8 bifurcation (~$250-400).

Fallback: llama.cpp layer split (no collectives needed, but needs GGUF and loses vLLM batching).

Bandwidth analysis: 27B TP=2 allreduce at batch 4 ≈ 10 MB per token-step ≈ 1.4 ms at Gen4 x4. Bandwidth is NOT the bottleneck; atomics routing is.

Software Stack

• vLLM 0.27.1 / torch 2.11.0 / ROCm 7.14
• Shipped RCCL exposes no ncclDevComm* symbols → torch's symmetric-memory path can't run
• Open hypothesis (unverified): recompiling RCCL with -mprintf-kind=buffered might work, but no live printf/assert/__assertfail found in device source

Operational Notes

• dankserver went down during probing; needs physical power cycle
• Boot persistence should auto-restore: frank-medium trio, media stack, node-exporter
• To distinguish 'off' from 'wedged': ip neigh from peer (FAILED = off/NIC dead; wedged kernel still answers ARP)
• AGENTS.md benchmarking policy updated to cover any GPU experiment, not just benchmarks

Model Target

• kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF
• Fork: charlie12345/rocmfpx (llama.cpp ROCm fork)

Skill Note

• 'grilling' skill does NOT exist in available skills list (returned 'Unknown skill: grilling')

01a0a87f-6b17-7492-995c-8c3e7b37e241

updated_at: 1789552363

Hardware Topology (dankclaw)

• RTX 3090 (24 GiB VRAM) in PCIe 4.0 x4 M.2 dock (downgraded from x16, measured ~29 GB/s sequential)
• AMD Strix Halo gfx1151 (64 GiB SVM-backed unified memory, RADV Vulkan driver; ROCm wheels do NOT support gfx1151)
• Combined GPU budget: ~88 GiB
• OS: Ubuntu 24.04
• Kernel cmdline: amd_iommu=off (for prefill gains; retired frank-npu)

ik_llama.cpp Build & Deployment

• Built with CUDA (for 3090) + Vulkan (for Strix Halo) backends
• Serves mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-GGUF at i1-Q4_K_M
• Deployed as root systemd unit (Restart=always) on port 7770, NOT a quadlet
• Context window: 262144 tokens
• Device split: --split-mode layer with -dev CUDA0,Vulkan0
• Shipped config: T16 (16 threads), no -ncmoe, no -thp
• Commits: 795b0b3, ad7a9c4
• Runbook: serverAppConfigs-docs/LLM/frank-large-ik-uncensored.md

Performance Envelope (verified single-instance)

Config	Decode (t/s)	Prefill (t/s)	Notes
Shipped T16	3.8-4.0	~192 (1.8k-32k ctx)	Production config
T8/T4/T2/T1	4.18/4.20/4.26/3.97	small-prompt only	Thread count barely matters
-ncmoe 42	4.54	—	22.0/24.6 GiB VRAM, fragile at 262k ctx
-ncmoe 48 -thp	4.60	—	110 GiB anon host buffer, co-tenant killer
--prefetch-experts	4.03	—	No effect (cache already resident)
Mainline llama.cpp 37b53fd	1.01	—	Ruled out entirely

Root-Cause: Why 30 t/s Decode Is Unreachable

1. Decode issues ~100 graph splits per token
2. Each split = GPU launch + sync across PCIe x4 (3090 in dock)
3. GPU SM utilization during decode: ~5%
4. ~90% of cycles = pause (thread barrier spin); T1 ≈ T16
5. CUDA graphs would collapse the overhead BUT capture ABORTS: qwen4exp hc ops use per-layer ggml_repeat_4d which is capture-illegal
6. Speculative decoding unavailable: GGUF has no MTP head (mradermacher stripped NextN from the checkpoint)
7. All experts on GPU: 84 GiB > 24 GiB 3090 VRAM — impossible
8. Path forward: mainline llama.cpp PR #28901 (hc-ops + capture-safe graphs) — re-evaluate when it lands

Prefill Ceiling

• ~192 t/s is the ceiling: compute-bound on CPU AVX512-VNNI
• Expert reads amortize across the batch; math saturates cores
• Halogen hit 30 t/s decode by holding entire 115.55 GiB w4b checkpoint resident in LPDDR inside one proprietary engine; 24 GB GPU + CPU split cannot replicate that residency with any public runtime

Operational Traps

• Unit has Restart=always: pkill'd servers respawn mid-experiment → phantom instances invalidate benchmarks. Always verify single-instance before measuring.
• Concurrent -j16 builds can starve probe processes
• PCIe x4 dock: do NOT assume x16 bandwidth; measured ~29 GB/s (not 130 GB/s)
• -ncmoe 42: +15-20% decode but only 2.5 GiB VRAM margin at 262k ctx — OOM risk
• -ncmoe 48 -thp: 110 GiB anon host buffer — will kill co-tenants

Trash (staged for user approval)

• dankclaw ~/.AGENTTRASH/20260916-ik-partfiles/ — 111 GiB (3 download parts, superseded)
• dankclaw ~/.AGENTTRASH/20260915-ik-swap/ — retired halogen quadlet
• repo tests/.AGENTTRASH/frank_large_quadlet.bats, scripts/.AGENTTRASH/frank-small*

01a0a902-7f58-77f4-9392-764f6cfbd59e

updated_at: 1789544414

AGENTS.md Rewrite Pattern (serverAppConfigs)

• Collapse chronological swap history into: (1) active-topology table (current state only), (2) runbook pointer table mapping each retired runtime → its docs/LLM/ runbook path.
• Add a 'Key Lessons Learned' section containing ONLY durable technical constraints/pitfalls (not narrative).
• In config sections (e.g. halogen), keep only invariants that survive any config change (pool/reserve matched pair, client contract, memory-model rule); push sizing history to runbook.
• Fix structural bugs (e.g. broken merged table rows) as part of the cleanup.

Durable LLM/ROCm/Quadlet Lessons (from AGENTS.md distillation)

• Layer-split-not-tensor: for multi-GPU ROCm, split by layer not tensor-parallel.
• mmap/PLE constraints apply to model loading on ROCm.
• HIP env collision: TEMP and HOST env vars can conflict with HIP runtime.
• GGML_CUDA_FA_ALL_QUANTS env var needed for flash-attention with quantized KV.
• MTP (Multi-Token Prediction) draft head has loader incompatibility on ROCm.
• For crash diagnosis: use gdb over logs (logs are often incomplete/misleading for segfaults).
• Swap-resize safe ordering: resize before use, not after.
• free command lies about available memory (doesn't account for reserved/pinned).
• systemctl disable can delete unit files (not just disable them).
• pkill -x for exact-name process kill (avoids substring matches).
• Rootless quadlet log retrieval has specific gotchas (see skill: rootless-quadlet-log-retrieval).

omp Skill Discovery

• omp ONLY discovers user skills at ~/.omp/agent/skills/.
• pi-era locations ~/.pi/agent/skills/ and ~/.agents/skills/ (vercel-labs skills CLI layout) are NOT scanned by omp.
• npx skills CLI auto-detects the wrong agent (claude) and doesn't know omp — bypass it, copy SKILL.md dirs directly.
• Newly installed skills surface on the NEXT session, not the current one.
• Skill sources: github.com/mattpocock/skills (grilling, tdd, implement, prototype, teach, diagnosing-bugs, code-review, handoff), github.com/vercel-labs/skills (find-skills).
• 4 skills unrecoverable without dankware (192.168.0.226) online: obsidian-vault, writing-great-skills, writing-guidelines, zim-knowledge-base.

01a09e67-630f-74ed-a910-e997e01fbe54

updated_at: 1789368883 Source: serverAppConfigs-docs/LLM/halogen-flash-server.md (halogen-flash-server runbook), sections Memory model, Two failure modes, Co-tenancy, and troubleshooting table.

Core stall advice: With HALOGEN_FLASH_PIN_TRUNK=1 (default), the halogen container holds most host memory. If a co-tenant takes the remainder and host memory runs out, both the engine and the co-tenant can stall for minutes at full CPU with no disk activity and no output. This is documented behavior, not a crash, and it needs no restart. Do not treat full-CPU/no-output multi-minute stalls as OOM/crash/restart cases.

Last resort: HALOGEN_FLASH_PIN_TRUNK=0 gives the pinned memory back to the host but costs several times the decode speed; use only as last resort, not routine tuning.

Diagnostics/monitoring traps: free, MemAvailable, and Grafana can read wrong by about 68 GiB because locked weights are counted as reclaimable. A request that never returns then returns 503 engine_busy is a different admission issue: the pool could not reserve prompt + max_tokens.

Restart first-touch behavior: after restart, the first request can make the engine silent for ~45-60 s (has not answered PING, /health engine_unresponsive) while it first-touches the fresh KV pool; it self-recovers. HealthStartPeriod=300s covers this; do not restart the unit; retrying after the window succeeds.

KV pool verification: use podman logs frank-large | grep 'kv pool:' and match reserved count to requested count; /health does not prove pool size. If under budget, nothing may print; absence of WARNING is not evidence of headroom.

Memory freeing/trimming: cut the KV pool or arena (examples: 262144 hands back ~3.6 GiB from 401408; MAX_TOK 16384 = ~8.4 GiB). Measured host-left gain can be less than modelled because the engine keeps freed space as working slack, but contiguous block count may improve substantially; stall warning keys on that. Example verified trim: kv_pool_positions=401408 exact/no clamp, held 100.0 GiB (68.0 weights + 11.0 KV + 21.0 working), host left 9.1 GiB / 1396 contiguous blocks vs previous 7.1 GiB / 310, stall NOTE no longer printed, health ok, real completion succeeded.

Client config constraint (blankpad OMP ~/.omp/agent/): frank-large maxTokens must be <=65536 because halogen returns 400 above cap; prompt + max_tokens must fit the KV pool or the request queues ~3600 s then 503s. Retired dead flm/dankclaw-npu provider on :7771; roles: vision -> frank-large, tiny -> frank-small, task -> frank-large:medium.

Deployment/environment notes: generated Quadlet units cannot be systemctl --user enable on this host; boot persistence uses [Install] WantedBy=default.target plus linger. Before replacing the halogen container/unit, remove the old Quadlet unit from ~/.config/containers/systemd/ to avoid duplicate port 7770 claims. Known environmental e2e breakage: blankpad -> 192.168.0.112 Sonarr path issue; deploy used --skip-e2e. Tests: 285+ green except known pre-existing test-73 ComfyUI selector bug.

01a09835-91cf-72e7-8ece-e5316d4de48b

updated_at: 1789286336 Expanded frank-large KV pool from 262144 to 524288 positions (~14.4 GiB). Reduced HALOGEN_KV_SLOTS from 4 to 3 (~111 MiB freed per slot). Held memory stable at 103.4 GiB (68 weights + 14.4 KV pool + 21 working) within 121 GiB host budget. Admission is pool-bound, not slot-bound; caps at 3 simultaneous full-length requests.

frank-small defaults to sampling (checkpoint values: do_sample=true, temperature=1.0, top_k=20, top_p=0.95), creating asymmetry with frank-large which defaults to greedy. To force determinism on frank-small: send temperature: 0 per request or relaunch with --sampling-defaults none.

preserve_thinking cannot be set server-side. Enumerated 156 HALOGEN_* env vars; none control thinking/preserve. Must be sent per-request in chat_template_kwargs by the client.

API/Host quirks: frank-small returns 200 for unknown fields (200 proves nothing). max_tokens: 200 with empty content indicates thinking consumed budget. host_reserve_gib=24 protects n-gram table; file-cache shortage causes uninterruptible sleep, not OOM.

Tests: 285/286 pass (test 73 comfyui pre-existing selector bug).

01a09648-8eb8-7093-876a-116b11d4c5fb

updated_at: 1789246675 Deployment Stack: Podman compose on dankclaw. Marreta shares gluetun egress with searxng. Caddy TLS edge routes to Anubis (:8923) -> Authentik embedded outpost -> internal :8017 listener -> marreta upstream.

Auth Chain Workflow: Provider-first creation -> Route flip (vhost becomes reverse_proxy anubis:8923, internal @svc handler) -> Deploy. Verified via unauthenticated challenge probe, post-auth config grep, outpost DNS reachability.

Resolved Failure: Marreta login-wall due to ProxyProvider.application = None on marreta-proxy. Null invalidation_flow crashed Rust outpost serde. Null auth/authz flows compounded issue.

Fix Applied: Populated invalidation_flow, authentication_flow, authorization_flow + linked Application(name=Marreta, slug=marreta, provider=marreta-proxy). Restarted authentik-server + authentik-outpost.

Pitfalls & Constraints: Probe http://localhost:8017/ from host returns HTTP 000 (listener in Caddy netns); probe inside netns or grep config. Non-browser clients cannot solve JS challenge; audit service-to-service traffic for backend datasource URLs hitting public guarded hostnames. Exempt-by-construction: IdP (auth.), git smart-HTTP (Gitea), self-auth services. Authentik providers/applications are live-only (ak shell); no repo blueprint (captured in authentik-proxy-provider-app-link skill).

Verification: E2E regression test (tests/e2e/specs/16-marreta.spec.ts) asserts homepage tile passes through guard chain to Marreta app title, avoiding login-wall parking.

01a09686-789e-7042-8774-55507faa78b2

updated_at: 1789232480 SERVICE: frank-large (rootless Quadlet, boot-persistent) on dankclaw. Unit: frank-large.service | Container: frank-large. LOG_RETRIEVAL: podman logs frank-large (full); podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):' (filtered); journalctl --user -u frank-large (journald). PORTS: Engine :8730 (container loopback, unauth), Front-end :8731, Published :7770. CONSTRAINTS: Weights pinned (~68 GiB). Env vars to avoid naming: TEMP, HOST, GPU_MAX_HEAP_SIZE (HIP runtime reads bare TEMP for JIT cache path). TEST_ENV: pyyaml missing for test suite (uv available, pip missing on system python3). Most suite failures are pre-existing environment issues, not commit regressions.