dank/command-center
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
d767348cfd02cade80788b0b0f39b7f930c1c98a
command-center/agent/memories/--home-blank-Projects-serverAppConfig--/MEMORY.md
T
Frank 5b08896e4e chore(memory): snapshot auto-learned memory and renamed workstation skills 
--home-blank-.omp--: MEMORY.md +140 / learned.md / memory_summary.md /
raw_memories.md +1002 rewritten by auto-learn after the model rename,
frank-medium split-balance and bottleneck studies. Skills renamed to
single-topic buckets: dotfile-git-workflow -> ghostty-text-aa +
niri-display-rotation + noctalia-theme, handy-service-mgmt ->
handy-flm-inference, inference-runtime -> llama-cpp-parallelism.

--home-blank-Projects-serverAppConfig--: 38 new rollout summaries,
raw_memories.md/MEMORY.md refreshed, skills re-bucketed (bench-protocol,
llm-fleet-model-swap, rootless-quadlet-log-retrieval, orchestration-quadlet).

Per-worktree bucket (wt-20261006-122204) is now gitignored, not committed.
2026-10-09 13:17:53 -07:00

5.8 KiB
Raw Blame History

Frank Fleet Memory Consolidation

1. Fleet Topology & Identity

Hosts

  • dankclaw (192.168.0.143): Strix Halo APU (gfx1151, ~114 GiB GTT) + M.2 docked RTX 3090 (24 GiB, PCIe x4). Hosts frank-large (Halogen) and frank-small (Llama.cpp).
  • dankserver (192.168.0.235): ASUS B650-E, 2x AMD R9700 (gfx1201, 32 GiB each), 4x DIMMs DDR5. Hosts frank-primary (VLLM/Radiance) and frank-medium (Llama.cpp).
  • frankbook-pro (192.168.0.112): M-series Mac mini (~30 GiB RAM). Role unclear/secondary.

Services (Live as of 2026-10-05)

  • frank-large (:7770): Halogen-flash-server v2, Qwen3.8-27B Uncensored (Q6_0/Q8_0/ROCmFP4), MTP/DFlash2 drafter, HALOGEN_CTX=262144, KV pool 524288 pos.
  • frank-small (:7771): Llama.cpp, MiMo-V2.6-9B (Q6_K/Q5_K_M) or Qwen3.8-9B (IQ3_S), 2x200704 f16-KV.
  • frank-primary (:7770): tcclaviger VLLM fork, Qwen3.8-Flash-Next FP8, TP=2, SEQS=3, MNBT=2048, MAXLEN=200704, Arena ~17.77 GiB.
  • frank-medium (:7770): Llama.cpp, Qwen3.8-27B UD-Q8_K_XL, Tensor Split 35:30, CACHE_RAM 16384 MiB.
  • Authentik: ridley shape template, Django ORM creation, Group-based ACLs.
  • ComfyUI: Strix Halo iGPU, FP8/Q6 quantization, Ghostty image protocol.

2. Hardware & Memory Constraints

RAM Stability (dankserver)

  • Critical: 4x DIMMs at 4800 MT/s caused silent corruption (NaN/garbage) at ~40k tokens in frank-primary. Fix: Drop to 4600 MT/s.
  • Lesson: Consumer non-ECC RAM raises no MCE for silent flips. Short in-band checks miss intermittent corruption.
  • Shape E: 3.15e9 B KV pool, 17.77 GiB Arena, UTIL 0.982.

GPU & PCIe

  • dankclaw 3090: PCIe x4 dock (~29 GB/s). Decode bottlenecked by graph splits + PCIe sync. 30 t/s unreachable without mainline PR #28901.
  • dankserver R9700: Chipset-attached PCIe (Gen4 x1) fails RCCL atomics. Fix: M.2→PCIe x4 riser for 2nd card.
  • Strix Halo: 64 GiB Unified, RADV Vulkan. HALOGEN_CTX must be 262144 (native Qwen3.8-Flash-Next).

KV & Context

  • frank-large: maxTokens 57344 (harness) + HALOGEN_CTX 262144. 400 error if prompt + max > ctx.
  • frank-medium: CACHE_RAM 16384 MiB required for 196608 ctx DeltaNet state. 8192 MiB caused refusals.
  • frank-small: 2x200704 f16-KV slots.

3. Runtime Environments

Halogen (dankclaw)

  • Container: halogen-flash-server v2 (Swift1.5 Abliterated).
  • Env: HALOGEN_CTX=262144, HALOGEN_KV_POOL_POSITIONS=524288.
  • Pitfalls: HALOGEN_FLASH_PIN_TRUNK=1 causes multi-minute CPU stalls under co-tenancy. HALOGEN_FLASH_PIN_TRUNK=0 as last resort.
  • Boot: First request may be silent ~45-60s (first-touch KV pool).

Llama.cpp (dankclaw/dankserver)

  • Forks: ik_llama.cpp (CUDA+Vulkan), rocmfpx (ROCmFP4).
  • Flags: --cache-ram 16384 (medium), --expert-precise-memory (primary), --max-model-len 200704.
  • MTP: Qwen3.8-Flash-Next MTP head needs full per-layer loading (Flash-Next specific).

VLLM (dankserver)

  • Fork: tcclaviger RDNA4 vLLM.
  • Seqs: SEQS=3 mandatory (SEQS=2 fails torch.fx ConstraintViolationError).
  • MTP: Speculative decoding disabled/optional.

4. Operational Hygiene

Bench Protocol

  1. sudo systemctl stop <service>
  2. Run benchmark
  3. sudo systemctl start <service>
  4. curl -s <port>/health → 200
  5. Trap: trap EXIT 'systemctl start <service>'; curl -s <port>/health for self-restoring harnesses.

Quadlet Persistence

  • Rootless Podman: restart: unless-stopped only survives crashes. Bare containers vanish on reboot.
  • Pattern: Git-tracked .container files in ~/.config/containers/systemd/.
  • Historical: Caddy lost after reboot (Aug 26); 5 observability services lost (Sep 11).

Git & Deploy

  • Repo: /home/blank/Projects/serverAppConfig.
  • Gate: deploy.sh rejects dirty trees (untracked/modified files).
  • Pattern: git stash → deploy → git stash pop.

5. Troubleshooting Knowledge Base

max_tokens 400 Overflow

  • Symptom: prompt is X tokens and the context is Y, leaving room for Z. X+max > Y.
  • Fix: Adjust maxTokens in models.yml (e.g., 57344) or HALOGEN_CTX (262144).
  • Note: OMP snapshots at process start; existing sessions need restart (omp -r).

Orphaned llama-server (frank-medium)

  • Symptom: Unit stopped, GPUs busy.
  • Diag: ps -eo pid,ppid,args | grep llama, rocm-smi --showmeminfo.
  • Fix: kill <pid>, systemctl start, verify geometry.

Context Bloat

  • Source: 175 managed-skills descriptions (~38k nonMessageTokens).
  • Fix: Distill to vault reference pages, move dirs to ~/.AGENTTRASH/, purge on approval.

Authentik User Creation

  • Method: ak shell on dankclaw, Django ORM (User.objects.create...), ak shell sets password, assign groups (Non-Admin Users, jellyfin_users).
  • ACLs: Group membership is sole gate.

ComfyUI (iGPU)

  • Target: Strix Halo iGPU (3090 occupied).
  • Quant: FP8 or Q6.
  • Display: Ghostty \e_G base64 PNG, or chafa helper.

6. Skills & Playbooks

  • llm-fleet-model-swap: Check HF model card, download sha256, commit → push → deploy.sh → health.
  • halogen-flash-server-memory: Memory model, two failure modes (stall, 400), co-tenancy.
  • frank-primary-ram-stability: DDR5 4800 vs 4600 MT/s, chunk boundaries, prefix cache eviction.
  • frank-medium-bench-protocol: Restore half optional? No. Self-restoring harness recommended.
  • authentik-user-creation: Django ORM workflow, group-based access.
  • orchestration-quadlet: Rootless Podman boot persistence.
  • context-bloat-distillation: Skill directory to vault migration.
  • comfyui-qwen-workflow: iGPU image generation, ghostty display.
  • expert-arena-definition: arena = budget - non-expert - kv_reserve - graphs - overhead.
  • skill-routing-pitfall: html-change-report for git commits, not creative writing.
Reference in New Issue View Git Blame Copy Permalink