docs: distill the managed skill library into the vault (waves 1-7)

All ~173 managed skills read in full and merged, one polished reference
page per skill-cluster:

- LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving
  Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4
  Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot
  Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM
  Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops,
  Heterogeneous Split Triage, FreeToken Builds + deltas folded into the
  retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V
  qualification pipeline, FLM pack registration, ik hybrid envelope,
  frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch
  split section, bench-window D-state/stale-read notes, second-endpoint
  bring-up).
- Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State
  Triage.
- Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim,
  Container Host Reachability, Fleet Ops Playbook, Fleet Audit and
  Posture, SSH and Remote Access.
- Infrastructure: Observability Provisioning + Homepage Status Probes;
  updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun
  zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page
  preserved and extended).
- FrankBook-Pro: ARR-Stack filled (API split, schema-first probing,
  auth-posture, auth-config trap); qBittorrent E2E verification.
- Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and
  Edits, Command Center Dotfiles, PICO-8 Asset Standards.
- Indexes: LLM Cross-Cutting References section, Operations/ and Agent/
  index pages, root-index Skill Distillation section, AGENTS.md pointer.
- log.md carries the wave plan.

check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.
This commit is contained in:
Frank committed 2026-10-05 21:19:15 -07:00
1 parent 8618ec6dbe
commit 33eae71d33
43 files changed
+5436 -2

No files matched your search

@@ -271,6 +271,83 @@ graph TD
6. Step up in size: 9B, then gpt-oss-20b.
7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load.
## Registering packs absent from the catalog (NPU2)
Use when adding a non-catalog NPU2 pack (an extraction or fine-tuned model)
to the FLM registry. Registry facts: user registry
`/home/dank/.config/flm/model_list.json`, shape
`{"model_path": "models", "models": {"<tag>": {"<size>": {name, url,
file_url, size, flm_min_version, files[], vlm, default_context_length,
max_prefill_len, details{format:"NPU2", family, …}, label[]}}}}`; serve
resolves tags through `FLM_CONFIG_PATH` at startup, and `flm list` with the
env var set lists custom tags (`⏬` = not downloaded).
1. Back up the registry JSON first (`cp -a`).
2. Insert the entry via a python json edit + atomic `os.replace`. Set
`files` to the FULL HF file list — **including `vision_weight.q4nx` even
for text-only packs**: the qwen3.5-family loader opens it unconditionally
and a missing file makes the first request HANG silently (no error log).
3. **Do not use `flm pull <tag>` for custom entries** — it throws
`json.exception out_of_range key '<tag>:<size>' not found` while building
the download list. Place files manually into
`/home/dank/.config/flm/models/<PackName>/` with
`curl -sfL "https://huggingface.co/<repo>/resolve/main/<file>?download=true"`
— plain `raw/main/<file>` returns 135-byte LFS pointer stubs, not
weights; verify byte counts against the HF tree API before trusting a
download.
4. Kernel packs: copy from the same-shape family pack (shape-match
`config.json` on hidden_size/num_hidden_layers/head_dim first) into
`/opt/fastflowlm/share/flm/xclbins/<PackName>/` via `sudo cp -a`
(root-owned dir).
5. Deploy/restart the serve unit (the deploy path, not ad-hoc restarts) so
the new process loads with all files present.
6. Verify with a REAL completion request (`model: "<tag>:<size>"`) —
`/v1/models` alone proves registration, not loadability.
Hard constraints: only ONE `flm serve` process can hold the NPU hardware
context — a second concurrent serve process fails every load with
`DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status
0x2000003); never run scratch/verify instances while production holds the
NPU. Tag switching happens INSIDE one serve process; the first request
after a model switch pays ~6–11 s cold-load cost (visible in
`usage.load_duration`). Over ssh, never `pkill -f "<pattern>"` with a
pattern present in your own command line — use bracket-guarded patterns
(`[q]wen3.5:4b`) or exact PIDs.
## Tag load-failure diagnosis (200 with an error body)
`POST /v1/chat/completions` answers **HTTP 200** with a flat
`{"error":"Failed to load <tag> model!"}` and no `choices`; `/v1/completions`
returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP
200. A status-code-only client reads this as a successful empty completion —
**always check for the `error` key**. Two distinct causes, distinguished via
`journalctl -u <flm-service> --full`:
1. **Unknown tag** (not in the registry): immediate error, nothing loaded.
Correct behavior, not a bug.
2. **Registered tag whose kernel pack is missing**: FLM resolves the tag,
downloads missing weight files on the fly (first request hangs ~90–110 s),
then fails `Failed to load model: No such file
'/home/dank/.config/flm/xclbins/<PackName>/<kernel>.xclbin'`; subsequent
attempts fail fast (~2 s). Root cause: the serve script exports
`FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so every
kernel lookup resolves under the user dir, which only contains symlinks
someone linked. **A tag listed in `/v1/models` is NOT proof it is
loadable on this host.** Fix: one symlink per missing pack
(`ln -s /opt/fastflowlm/share/flm/xclbins/<PackName>
/home/dank/.config/flm/xclbins/<PackName>`) — verified live: after the
symlink the first request loaded and answered in 7.4 s wall.
Switch-cost behavior: the `model` field routes per-request (no restart
needed); a request whose model differs from the resident one triggers a full
reload (7–11 s wall for an 8-token completion on ~4B-class packs);
alternating tags pays the swap each time. Single-lane NPU: concurrent
requests queue FIFO behind the current lock holder (`NPU Locked!` →
`NPU Lock Released!`). Probe recipe: `/v1/models` for registered tags; grep
the journal for `Failed to load model` (the quoted xclbin path names the
missing pack dir exactly); diff `ls ~/.config/flm/xclbins/` (symlinks =
installed) against `/opt/fastflowlm/share/flm/xclbins/` (available).
## Incident: qwen3.6-moe unreachable (2026-08-16)
Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.