chore(memory): snapshot auto-learned memory and renamed workstation skills
--home-blank-.omp--: MEMORY.md +140 / learned.md / memory_summary.md / raw_memories.md +1002 rewritten by auto-learn after the model rename, frank-medium split-balance and bottleneck studies. Skills renamed to single-topic buckets: dotfile-git-workflow -> ghostty-text-aa + niri-display-rotation + noctalia-theme, handy-service-mgmt -> handy-flm-inference, inference-runtime -> llama-cpp-parallelism. --home-blank-Projects-serverAppConfig--: 38 new rollout summaries, raw_memories.md/MEMORY.md refreshed, skills re-bucketed (bench-protocol, llm-fleet-model-swap, rootless-quadlet-log-retrieval, orchestration-quadlet). Per-worktree bucket (wt-20261006-122204) is now gitignored, not committed.
This commit is contained in:
Frank committed
1 parent
a34322aeb0
commit
5b08896e4e
72 files changed
+1461
-175
No files matched your search
@@ -1,32 +1,108 @@
|
||||
## Fleet Infrastructure & Hardware
|
||||
- **dankclaw (192.168.0.143):** AMD Strix Halo APU (gfx1151) + RTX 3090 (PCIe x4 M.2 dock). Ubuntu 24.04. Combined GPU budget ~88 GiB.
|
||||
- **dankserver (192.168.0.235):** ASUS B650-E with 2x AMD R9700 GPUs (gfx1201). Ubuntu 24.04. RCCL atomics require M.2 riser to move card #2 to CPU-attached M.2_2 slot.
|
||||
- **the workstation / blankpad:** Ryzen 5 PRO 5650U, RTX 4070 Max-Q. Passwordless SSH to fleet.
|
||||
# Frank Fleet Memory Consolidation
|
||||
|
||||
## LLM Model Fleet
|
||||
- **frank-medium (dankserver :7770):**
|
||||
- Model: Qwen3.8-27B (MXFP6 / Q8_K_XL).
|
||||
- Architecture: Layer-split (35:30) across 2x R9700.
|
||||
- Drafter: DFlash2 (Q8_0).
|
||||
- Key Specs: n_slots=2, n_ctx_slot=196608, CACHE_RAM=16384 MiB (handles ~12.8 GiB per slot).
|
||||
- **frank-large (dankclaw :7770):**
|
||||
- Model: Halogen (iGPU/3090 split).
|
||||
- KV Pool: 524,288 positions (~14.4 GiB).
|
||||
- MaxTokens: 57344 (prevents 400-error 'dead zone').
|
||||
- Performance: ~3.8-4.0 t/s decode, ~192 t/s prefill.
|
||||
- **frank-small (dankclaw :7771):**
|
||||
- Model: Gemma4-12B (Q4_K_M) via vLLM.
|
||||
- Mode: Sampling (Temp 1.0, Top_k 20, Top_p 0.95).
|
||||
## 1. Fleet Topology & Identity
|
||||
|
||||
## Deployment System (serverAppConfigs)
|
||||
- **Architecture:** Quadlet-based podman units (boot-persistent).
|
||||
- **Pipeline:** `deploy.sh` (pre-flight) $\rightarrow$ `deploy-plan.py` (path-aware mapping) $\rightarrow$ `deploy-remote.sh` (revision-scoped pull/restart) $\rightarrow$ Playwright e2e gate.
|
||||
- **Homepage:** `shared/homepage/` with category-based rendering (`admin` vs `user`), status-aware badges, and bare mode (`?bare=1`).
|
||||
- **Persistence:** OMP Session Persistence (JSONL) with blob externalization for images/long strings. HistoryStorage (SQLite) for prompt-history UX.
|
||||
### Hosts
|
||||
- **dankclaw** (192.168.0.143): Strix Halo APU (gfx1151, ~114 GiB GTT) + M.2 docked RTX 3090 (24 GiB, PCIe x4). Hosts `frank-large` (Halogen) and `frank-small` (Llama.cpp).
|
||||
- **dankserver** (192.168.0.235): ASUS B650-E, 2x AMD R9700 (gfx1201, 32 GiB each), 4x DIMMs DDR5. Hosts `frank-primary` (VLLM/Radiance) and `frank-medium` (Llama.cpp).
|
||||
- **frankbook-pro** (192.168.0.112): M-series Mac mini (~30 GiB RAM). Role unclear/secondary.
|
||||
|
||||
## Core Lessons & Pitfalls
|
||||
- **RCCL Atomics:** Chipset-attached PCIe lacks atomics; use CPU-attached lanes for TP/PP functionality.
|
||||
- **Halogen Co-tenancy:** `HALOGEN_FLASH_PIN_TRUNK=1` holds host memory; multi-minute full-CPU stalls are documented behavior, not crashes.
|
||||
- **Cache Logic:** A refused RAM cache state forces a full prefill next turn (critical for hybrid DeltaNet models).
|
||||
- **Plex/Entropy:** Use `GroupAdd=keep-groups` in quadlets to pass host groups into containers without modifying filesystem permissions.
|
||||
- **Calibre-Web:** Use `8090` (internal), `Digest` (auth), and `printf` (password) to avoid common traps.
|
||||
### Services (Live as of 2026-10-05)
|
||||
- **frank-large** (:7770): Halogen-flash-server v2, Qwen3.8-27B Uncensored (Q6_0/Q8_0/ROCmFP4), MTP/DFlash2 drafter, HALOGEN_CTX=262144, KV pool 524288 pos.
|
||||
- **frank-small** (:7771): Llama.cpp, MiMo-V2.6-9B (Q6_K/Q5_K_M) or Qwen3.8-9B (IQ3_S), 2x200704 f16-KV.
|
||||
- **frank-primary** (:7770): tcclaviger VLLM fork, Qwen3.8-Flash-Next FP8, TP=2, SEQS=3, MNBT=2048, MAXLEN=200704, Arena ~17.77 GiB.
|
||||
- **frank-medium** (:7770): Llama.cpp, Qwen3.8-27B UD-Q8_K_XL, Tensor Split 35:30, CACHE_RAM 16384 MiB.
|
||||
- **Authentik**: `ridley` shape template, Django ORM creation, Group-based ACLs.
|
||||
- **ComfyUI**: Strix Halo iGPU, FP8/Q6 quantization, Ghostty image protocol.
|
||||
|
||||
## 2. Hardware & Memory Constraints
|
||||
|
||||
### RAM Stability (dankserver)
|
||||
- **Critical**: 4x DIMMs at 4800 MT/s caused silent corruption (NaN/garbage) at ~40k tokens in `frank-primary`. **Fix**: Drop to 4600 MT/s.
|
||||
- **Lesson**: Consumer non-ECC RAM raises no MCE for silent flips. Short in-band checks miss intermittent corruption.
|
||||
- **Shape E**: 3.15e9 B KV pool, 17.77 GiB Arena, UTIL 0.982.
|
||||
|
||||
### GPU & PCIe
|
||||
- **dankclaw 3090**: PCIe x4 dock (~29 GB/s). Decode bottlenecked by graph splits + PCIe sync. 30 t/s unreachable without mainline PR #28901.
|
||||
- **dankserver R9700**: Chipset-attached PCIe (Gen4 x1) fails RCCL atomics. **Fix**: M.2→PCIe x4 riser for 2nd card.
|
||||
- **Strix Halo**: 64 GiB Unified, RADV Vulkan. HALOGEN_CTX must be 262144 (native Qwen3.8-Flash-Next).
|
||||
|
||||
### KV & Context
|
||||
- **frank-large**: `maxTokens` 57344 (harness) + HALOGEN_CTX 262144. 400 error if `prompt + max > ctx`.
|
||||
- **frank-medium**: `CACHE_RAM` 16384 MiB required for 196608 ctx DeltaNet state. 8192 MiB caused refusals.
|
||||
- **frank-small**: 2x200704 f16-KV slots.
|
||||
|
||||
## 3. Runtime Environments
|
||||
|
||||
### Halogen (dankclaw)
|
||||
- **Container**: `halogen-flash-server v2` (Swift1.5 Abliterated).
|
||||
- **Env**: `HALOGEN_CTX=262144`, `HALOGEN_KV_POOL_POSITIONS=524288`.
|
||||
- **Pitfalls**: `HALOGEN_FLASH_PIN_TRUNK=1` causes multi-minute CPU stalls under co-tenancy. `HALOGEN_FLASH_PIN_TRUNK=0` as last resort.
|
||||
- **Boot**: First request may be silent ~45-60s (first-touch KV pool).
|
||||
|
||||
### Llama.cpp (dankclaw/dankserver)
|
||||
- **Forks**: `ik_llama.cpp` (CUDA+Vulkan), `rocmfpx` (ROCmFP4).
|
||||
- **Flags**: `--cache-ram 16384` (medium), `--expert-precise-memory` (primary), `--max-model-len 200704`.
|
||||
- **MTP**: Qwen3.8-Flash-Next MTP head needs full per-layer loading (Flash-Next specific).
|
||||
|
||||
### VLLM (dankserver)
|
||||
- **Fork**: tcclaviger RDNA4 vLLM.
|
||||
- **Seqs**: `SEQS=3` mandatory (SEQS=2 fails `torch.fx` ConstraintViolationError).
|
||||
- **MTP**: Speculative decoding disabled/optional.
|
||||
|
||||
## 4. Operational Hygiene
|
||||
|
||||
### Bench Protocol
|
||||
1. `sudo systemctl stop <service>`
|
||||
2. Run benchmark
|
||||
3. `sudo systemctl start <service>`
|
||||
4. `curl -s <port>/health` → 200
|
||||
5. **Trap**: `trap EXIT 'systemctl start <service>'; curl -s <port>/health` for self-restoring harnesses.
|
||||
|
||||
### Quadlet Persistence
|
||||
- **Rootless Podman**: `restart: unless-stopped` only survives crashes. Bare containers vanish on reboot.
|
||||
- **Pattern**: Git-tracked `.container` files in `~/.config/containers/systemd/`.
|
||||
- **Historical**: Caddy lost after reboot (Aug 26); 5 observability services lost (Sep 11).
|
||||
|
||||
### Git & Deploy
|
||||
- **Repo**: `/home/blank/Projects/serverAppConfig`.
|
||||
- **Gate**: `deploy.sh` rejects dirty trees (untracked/modified files).
|
||||
- **Pattern**: `git stash` → deploy → `git stash pop`.
|
||||
|
||||
## 5. Troubleshooting Knowledge Base
|
||||
|
||||
### max_tokens 400 Overflow
|
||||
- **Symptom**: `prompt is X tokens and the context is Y, leaving room for Z`. `X+max > Y`.
|
||||
- **Fix**: Adjust `maxTokens` in `models.yml` (e.g., 57344) or `HALOGEN_CTX` (262144).
|
||||
- **Note**: OMP snapshots at process start; existing sessions need restart (`omp -r`).
|
||||
|
||||
### Orphaned `llama-server` (frank-medium)
|
||||
- **Symptom**: Unit stopped, GPUs busy.
|
||||
- **Diag**: `ps -eo pid,ppid,args | grep llama`, `rocm-smi --showmeminfo`.
|
||||
- **Fix**: `kill <pid>`, `systemctl start`, verify geometry.
|
||||
|
||||
### Context Bloat
|
||||
- **Source**: 175 managed-skills descriptions (~38k nonMessageTokens).
|
||||
- **Fix**: Distill to vault reference pages, move dirs to `~/.AGENTTRASH/`, purge on approval.
|
||||
|
||||
### Authentik User Creation
|
||||
- **Method**: `ak shell` on `dankclaw`, Django ORM (`User.objects.create`...), `ak shell` sets password, assign groups (`Non-Admin Users`, `jellyfin_users`).
|
||||
- **ACLs**: Group membership is sole gate.
|
||||
|
||||
### ComfyUI (iGPU)
|
||||
- **Target**: Strix Halo iGPU (3090 occupied).
|
||||
- **Quant**: FP8 or Q6.
|
||||
- **Display**: Ghostty `\e_G` base64 PNG, or `chafa` helper.
|
||||
|
||||
## 6. Skills & Playbooks
|
||||
|
||||
- **llm-fleet-model-swap**: Check HF model card, download sha256, commit → push → deploy.sh → health.
|
||||
- **halogen-flash-server-memory**: Memory model, two failure modes (stall, 400), co-tenancy.
|
||||
- **frank-primary-ram-stability**: DDR5 4800 vs 4600 MT/s, chunk boundaries, prefix cache eviction.
|
||||
- **frank-medium-bench-protocol**: Restore half optional? No. Self-restoring harness recommended.
|
||||
- **authentik-user-creation**: Django ORM workflow, group-based access.
|
||||
- **orchestration-quadlet**: Rootless Podman boot persistence.
|
||||
- **context-bloat-distillation**: Skill directory to vault migration.
|
||||
- **comfyui-qwen-workflow**: iGPU image generation, ghostty display.
|
||||
- **expert-arena-definition**: `arena = budget - non-expert - kv_reserve - graphs - overhead`.
|
||||
- **skill-routing-pitfall**: `html-change-report` for git commits, not creative writing.
|
||||
Reference in new issue
Block a user