docs: distill the managed skill library into the vault (waves 1-7)
All ~173 managed skills read in full and merged, one polished reference page per skill-cluster: - LLM cross-cutting (17 pages): GGUF Compatibility, Llama Cpp Serving Sizing, Llama Cpp Multi-Device Splits, PCIe Link and Atomics, RDNA4 Container Serving, RDNA4 MoE Expert Offload VRAM Budget, vLLM Boot Forensics, vLLM Porting and Compile Triage, LLM Bench Windows, LLM Performance Diagnosis, Model Swap and Rollback, Tcclaviger Fork Ops, Heterogeneous Split Triage, FreeToken Builds + deltas folded into the retired runbooks (GLM DFlash2 port laws, Lucebox dual-R9700, R9V qualification pipeline, FLM pack registration, ik hybrid envelope, frank-medium split rebalance, llama.cpp GGUF preflight, batch-ubatch split section, bench-window D-state/stale-read notes, second-endpoint bring-up). - Architecture (3): Stale Signal Audit, Quadlet Lifecycle, Deploy State Triage. - Operations (6 + index): Remote Job Runner, Payload and Disk Reclaim, Container Host Reachability, Fleet Ops Playbook, Fleet Audit and Posture, SSH and Remote Access. - Infrastructure: Observability Provisioning + Homepage Status Probes; updates to Authentik, Anubis, LAN-DNS (per-client DoH bypass, Porkbun zone ops), Swap Configuration, OpenWebUI, Hermes (pre-existing page preserved and extended). - FrankBook-Pro: ARR-Stack filled (API split, schema-first probing, auth-posture, auth-config trap); qBittorrent E2E verification. - Desktop/Agent/Research: Noctalia and Niri, OMP Harness, Agent Docs and Edits, Command Center Dotfiles, PICO-8 Asset Standards. - Indexes: LLM Cross-Cutting References section, Operations/ and Agent/ index pages, root-index Skill Distillation section, AGENTS.md pointer. - log.md carries the wave plan. check-vault-links.py 0 unresolved (185 pages); ./tests/run.sh 366/366.
This commit is contained in:
Frank
@@ -271,6 +271,83 @@ graph TD
|
||||
6. Step up in size: 9B, then gpt-oss-20b.
|
||||
7. Attempt `qwen3.6-moe:35b-a3b` only after the smaller runs succeed, and only with the production service stopped. Watch `free -h` during the model download and load.
|
||||
|
||||
## Registering packs absent from the catalog (NPU2)
|
||||
|
||||
Use when adding a non-catalog NPU2 pack (an extraction or fine-tuned model)
|
||||
to the FLM registry. Registry facts: user registry
|
||||
`/home/dank/.config/flm/model_list.json`, shape
|
||||
`{"model_path": "models", "models": {"<tag>": {"<size>": {name, url,
|
||||
file_url, size, flm_min_version, files[], vlm, default_context_length,
|
||||
max_prefill_len, details{format:"NPU2", family, …}, label[]}}}}`; serve
|
||||
resolves tags through `FLM_CONFIG_PATH` at startup, and `flm list` with the
|
||||
env var set lists custom tags (`⏬` = not downloaded).
|
||||
|
||||
1. Back up the registry JSON first (`cp -a`).
|
||||
2. Insert the entry via a python json edit + atomic `os.replace`. Set
|
||||
`files` to the FULL HF file list — **including `vision_weight.q4nx` even
|
||||
for text-only packs**: the qwen3.5-family loader opens it unconditionally
|
||||
and a missing file makes the first request HANG silently (no error log).
|
||||
3. **Do not use `flm pull <tag>` for custom entries** — it throws
|
||||
`json.exception out_of_range key '<tag>:<size>' not found` while building
|
||||
the download list. Place files manually into
|
||||
`/home/dank/.config/flm/models/<PackName>/` with
|
||||
`curl -sfL "https://huggingface.co/<repo>/resolve/main/<file>?download=true"`
|
||||
— plain `raw/main/<file>` returns 135-byte LFS pointer stubs, not
|
||||
weights; verify byte counts against the HF tree API before trusting a
|
||||
download.
|
||||
4. Kernel packs: copy from the same-shape family pack (shape-match
|
||||
`config.json` on hidden_size/num_hidden_layers/head_dim first) into
|
||||
`/opt/fastflowlm/share/flm/xclbins/<PackName>/` via `sudo cp -a`
|
||||
(root-owned dir).
|
||||
5. Deploy/restart the serve unit (the deploy path, not ad-hoc restarts) so
|
||||
the new process loads with all files present.
|
||||
6. Verify with a REAL completion request (`model: "<tag>:<size>"`) —
|
||||
`/v1/models` alone proves registration, not loadability.
|
||||
|
||||
Hard constraints: only ONE `flm serve` process can hold the NPU hardware
|
||||
context — a second concurrent serve process fails every load with
|
||||
`DRM_IOCTL_AMDXDNA_CREATE_HWCTX err=-22` (aie2 mgmt msg opcode 0x2 status
|
||||
0x2000003); never run scratch/verify instances while production holds the
|
||||
NPU. Tag switching happens INSIDE one serve process; the first request
|
||||
after a model switch pays ~6–11 s cold-load cost (visible in
|
||||
`usage.load_duration`). Over ssh, never `pkill -f "<pattern>"` with a
|
||||
pattern present in your own command line — use bracket-guarded patterns
|
||||
(`[q]wen3.5:4b`) or exact PIDs.
|
||||
|
||||
## Tag load-failure diagnosis (200 with an error body)
|
||||
|
||||
`POST /v1/chat/completions` answers **HTTP 200** with a flat
|
||||
`{"error":"Failed to load <tag> model!"}` and no `choices`; `/v1/completions`
|
||||
returns a nested OpenAI-shaped `server_error` (code 500) also under HTTP
|
||||
200. A status-code-only client reads this as a successful empty completion —
|
||||
**always check for the `error` key**. Two distinct causes, distinguished via
|
||||
`journalctl -u <flm-service> --full`:
|
||||
|
||||
1. **Unknown tag** (not in the registry): immediate error, nothing loaded.
|
||||
Correct behavior, not a bug.
|
||||
2. **Registered tag whose kernel pack is missing**: FLM resolves the tag,
|
||||
downloads missing weight files on the fly (first request hangs ~90–110 s),
|
||||
then fails `Failed to load model: No such file
|
||||
'/home/dank/.config/flm/xclbins/<PackName>/<kernel>.xclbin'`; subsequent
|
||||
attempts fail fast (~2 s). Root cause: the serve script exports
|
||||
`FLM_XCLBIN_PATH=/home/dank/.config/flm/xclbins` (user dir), so every
|
||||
kernel lookup resolves under the user dir, which only contains symlinks
|
||||
someone linked. **A tag listed in `/v1/models` is NOT proof it is
|
||||
loadable on this host.** Fix: one symlink per missing pack
|
||||
(`ln -s /opt/fastflowlm/share/flm/xclbins/<PackName>
|
||||
/home/dank/.config/flm/xclbins/<PackName>`) — verified live: after the
|
||||
symlink the first request loaded and answered in 7.4 s wall.
|
||||
|
||||
Switch-cost behavior: the `model` field routes per-request (no restart
|
||||
needed); a request whose model differs from the resident one triggers a full
|
||||
reload (7–11 s wall for an 8-token completion on ~4B-class packs);
|
||||
alternating tags pays the swap each time. Single-lane NPU: concurrent
|
||||
requests queue FIFO behind the current lock holder (`NPU Locked!` →
|
||||
`NPU Lock Released!`). Probe recipe: `/v1/models` for registered tags; grep
|
||||
the journal for `Failed to load model` (the quoted xclbin path names the
|
||||
missing pack dir exactly); diff `ls ~/.config/flm/xclbins/` (symlinks =
|
||||
installed) against `/opt/fastflowlm/share/flm/xclbins/` (available).
|
||||
|
||||
## Incident: qwen3.6-moe unreachable (2026-08-16)
|
||||
|
||||
Symptom: a client on dankware could not reach `qwen3.6-moe:35b-a3b` on port 52625. The journal shows three separate causes in the 14:19-15:03 window.
|
||||
|
||||
@@ -0,0 +1,97 @@
|
||||
---
|
||||
type: reference
|
||||
title: FreeToken Builds
|
||||
description: Building FreeToken (the official franksdank fork of FreeTok Tokenworks' freetoken vLLM-MTP runtime) on Dankclaw — the CUDA vLLM venv build and the ROCm build, with the dependency traps each one hit.
|
||||
tags: [freetoken, build, dankclaw, vllm, mtp]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# FreeToken Builds
|
||||
|
||||
FreeToken = the official franksdank fork of FreeTok Tokenworks' freetoken
|
||||
vLLM-MTP runtime. Two build paths exist, both on **Dankclaw**, with
|
||||
different dependency traps. Both were verified end-to-end 2026-09-11/12.
|
||||
|
||||
## Common verification (do not skip)
|
||||
|
||||
**Verify the build by real completion, not by version string.** A version
|
||||
bump alone does not move data through the model — a build can print a new
|
||||
version and still not have completed a real token. Before declaring a
|
||||
FreeToken build deployable, run an actual completion (any tiny prompt) and
|
||||
read the returned content:
|
||||
|
||||
```python
|
||||
import json, urllib.request
|
||||
req = urllib.request.Request("http://<host>:<port>/v1/chat/completions",
|
||||
json.dumps({"model": "frank-small", "messages": [{"role": "user",
|
||||
"content": "Hi"}]}).encode(), {"Content-Type": "application/json"})
|
||||
print(json.load(urllib.request.urlopen(req, timeout=600)))
|
||||
```
|
||||
|
||||
A 200 with a real `choices[0].message.content` is the only build green.
|
||||
|
||||
## CUDA build (the torch/vLLM dependency trap)
|
||||
|
||||
Both paths build on **Dankclaw** with the same preinstall state: Python
|
||||
3.12.10 in the system venv and **Torch 2.10.0.dev** (nightly, repo
|
||||
download.pytorch.org/whl/nightly/cu130, installed with
|
||||
`uv pip install --index-strategy unsafe-best-match "torch==2.10.0.dev<date>+cu130"
|
||||
--index-url https://download.pytorch.org/whl/nightly/cu130` — use
|
||||
`--index-strategy unsafe-best-match`, `uv` otherwise refuses the dev
|
||||
version). Verify the exact pair first; mismatched preinstalled deps are the
|
||||
#1 build-failure cause.
|
||||
|
||||
Then install the **matching vllm dev wheel**:
|
||||
|
||||
```bash
|
||||
uv pip install --index-strategy unsafe-best-match \
|
||||
--index-url https://wheels.vllm.ai/<commit-sha>/vllm-0.6.6.dev0+g<shortsha>.cu130-cp312-cp312-linux_x86_64.whl \
|
||||
--extra-index-url https://wheels.vllm.ai/nightly/cu130
|
||||
```
|
||||
|
||||
The wheel URL pattern is fixed:
|
||||
`https://wheels.vllm.ai/<full-commit>/vllm-0.6.6.dev0+g<8-char>.cu130-cp312-cp312-linux_x86_64.whl`
|
||||
— read it back from vLLM's CI-run page (`run details →
|
||||
build_and_install_package` job, "install the vllm wheel" step). **The trap:
|
||||
vLLM refuses versions < its install_requires floor at runtime, so the wheel
|
||||
version must be ≥ the runtime's floor.** A wheel built at a lower base
|
||||
(0.5.x) downloads but `import vllm` fails with a version constraint error —
|
||||
that failure looks like a build failure and is not one; upgrade the wheel,
|
||||
not the code.
|
||||
|
||||
If `uv pip install -e . --no-deps` refuses (some repos pin the
|
||||
`--no-deps` branch off), fall back to `python3 setup.py` inside the venv,
|
||||
after `python -m venv --copies .venv && source .venv/bin/activate` and
|
||||
`uv pip install --no-build-isolation "torch==…"` — the venv must NOT
|
||||
re-resolve torch (a second torch from PyPI d1 breaks the CUDA extension ABI).
|
||||
|
||||
## ROCm build (gfx1151 via TheRock nightly)
|
||||
|
||||
Same venv discipline, ROCm wheels from TheRock's nightly index:
|
||||
|
||||
```bash
|
||||
uv pip install --index-strategy unsafe-best-match \
|
||||
--index-url https://rocm.nightlies.amd.com/v2/gfx1151-dcgpu-supports_fbsgens_split_0717-rc1/ \
|
||||
"torch==2.10.0.dev<date>+rocm" "triton==3.6.0" "pytorch-triton==3.6.0"
|
||||
```
|
||||
|
||||
Then `vllm` itself (`uv pip install -e . --no-build-isolation --no-deps`,
|
||||
repo branch `specs-mtp-dev`, commit `2ee440d`), after copying the two
|
||||
prebuilt extension wheels into the tree (the repo's `setup.py` skips
|
||||
compiling them when found). The ROCm build also needs
|
||||
`install-fastrak-gfx1151.sh` from the repo's scripts and
|
||||
`FASTRACK_HOME=/home/dank/FastTrack-runtime` in the unit env. FreeToken's
|
||||
x86_64-gfx1151 prebuilt extension (`freetoken-ops`) ships on HF hub
|
||||
`franksdank/freetoken-ops-gfx1151` — fetch it once and point
|
||||
`FREETOKEN_OPS_HOME` at it; never rebuild the extension from source unless
|
||||
the extension API changed.
|
||||
|
||||
`--index-strategy unsafe-best-match` is load-bearing on both paths: the
|
||||
nightly indexes carry version pairs that `uv`'s strict resolver otherwise
|
||||
rejects, and the error (`No candidate found for pytorch-triton`) reads like
|
||||
a missing package.
|
||||
|
||||
## Related
|
||||
|
||||
- [[Model Swap and Rollback]] — the swap/deploy story these builds serve
|
||||
- [[Llama Cpp Serving Sizing]] — the harness measurement discipline
|
||||
@@ -0,0 +1,159 @@
|
||||
---
|
||||
type: reference
|
||||
title: Heterogeneous Split Triage
|
||||
description: Triage before building when someone asks to split a model across unequal accelerators on one host (24 GB GDDR6X + LPDDR APU, PCIe x4 link) — link characterisation, the real bottleneck, what placement never buys, UMA pools, drafter splits, and the verdict ranking.
|
||||
tags: [heterogeneous, multi-device, llama-cpp, triage, pcie]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Heterogeneous Split Triage
|
||||
|
||||
Produced the 2026-09-16 no-go verdict on splitting Qwen3.8-Flash-Next across
|
||||
the RTX 3090 + Strix Halo pair. Use it to answer "could we split model X
|
||||
across device A and device B" with arithmetic instead of enthusiasm.
|
||||
|
||||
## 1. Characterise the link (bytes vs latency)
|
||||
|
||||
```bash
|
||||
lspci -vv -s <bdf> | grep -E "LnkCap:|LnkSta:" # capability vs actual
|
||||
nvidia-smi --query-gpu=pcie.link.gen.max,pcie.link.gen.current,pcie.link.width.current --format=csv
|
||||
```
|
||||
|
||||
Expect **idle-downshift**: a Gen4 x4 dock idles at Gen1 x4 (~1 GB/s) and
|
||||
upshifts under load. Measure while the device is busy, or you size the design
|
||||
against the wrong number. Per-hop cost against the hidden state
|
||||
(`n_embd × 2 B` for f16): a 5120-dim model moves 10 KiB/hop ≈ 1.3 µs at
|
||||
Gen4 x4.
|
||||
|
||||
- **Bytes per token ≈ hops/token × hidden state** — typically ~1 MiB/token
|
||||
for 2 hops/layer × 64 layers ≈ 0.3 % of a 40 ms token. **Bytes are almost
|
||||
never the constraint.**
|
||||
- The real budget line is **hop COUNT**: every device boundary is a
|
||||
host-side sync. A runtime that splits per *op* hits ~100 splits/token and
|
||||
pins the GPU at ~5 % SM. Measure `graph splits` in the loader log; never
|
||||
assume.
|
||||
- **Design rule: partition by where the weights live, never by where the
|
||||
activations flow.** Resident weights, activations crossing.
|
||||
|
||||
## 2. Get the workload's real bottleneck (KV is usually it)
|
||||
|
||||
```
|
||||
KV/token = kv_bytes_per_position × context
|
||||
```
|
||||
|
||||
Attention reads the whole cached context every token, so KV/token scales
|
||||
with context — compare it against expert/weight traffic per token and total
|
||||
token time. At long context on a hybrid arch (few attention layers, many
|
||||
positions), **KV dominates**. Consequences:
|
||||
|
||||
- The highest-leverage placement is **KV cache on the fastest-memory device,
|
||||
together with its attention compute** — never KV alone: separating them
|
||||
puts KV/token over the link, which is fatal (GBs across 7.88 GB/s =
|
||||
hundreds of ms/token).
|
||||
- Check the fit: `kv_positions_per_slot × slots × bytes_per_position +
|
||||
attention_trunk ≤ VRAM`.
|
||||
|
||||
## 3. What placement never buys
|
||||
|
||||
Prefill is compute-bound and expert weights are read once per batch, so
|
||||
moving them to faster memory buys nothing while adding a serial stage.
|
||||
Expect a regression, and size it from calibration: on Strix Halo + RTX 3090
|
||||
(Qwen3.8-Flash-Next, ik_llama.cpp) moving 12.5 % of the expert mass to the
|
||||
GPU gave **decode +12 %, prefill 192 → 32 tok/s**. A
|
||||
`1/(f_big/BW_slow + f_fast/BW_fast)` harmonic calculation reproduces that
|
||||
decode gain to within ~2 points — usable for decode projections, and it
|
||||
assumes sequential stages (true at batch 1; no pipelining).
|
||||
|
||||
## 4. Is the second device even a separate pool?
|
||||
|
||||
On a UMA APU, check before assuming additivity:
|
||||
|
||||
```bash
|
||||
cat /sys/class/drm/card*/device/mem_info_vram_total # BIOS carve-out (small)
|
||||
cat /sys/class/drm/card*/device/mem_info_gtt_total # aperture onto SYSTEM RAM
|
||||
rocm-smi --showmeminfo all | grep -i gtt # used == what the CPU also sees
|
||||
```
|
||||
|
||||
A large GTT window means the "GPU memory" is the *same* LPDDR under one
|
||||
memory controller: adding a stage there adds a compute engine, not capacity
|
||||
or bandwidth. Only the discrete card's VRAM is additive. Fencing more
|
||||
(`amdgpu.gttsize`, BIOS UMA size) just removes bytes from CPU-visible RAM.
|
||||
|
||||
## 5. Speculative decoding as the "cheap" split
|
||||
|
||||
A drafter on the *other* device is bandwidth-free (token ids per round, tens
|
||||
of bytes), so it is tempting. Before proposing it:
|
||||
|
||||
- the drafter needs the target's **exact tokenizer** for a plain draft model
|
||||
(compare GGUF `tokenizer.ggml.tokens`, never trust the family name), or it
|
||||
must be a head **trained for that target** (EAGLE/DFlash/MTP are
|
||||
conditioned on the target's hidden states and are not portable between
|
||||
models);
|
||||
- a plain drafter costs **K sequential forwards per round**, so it must be
|
||||
10–50× faster than the target. Break-even with `r = t_draft/t_target`:
|
||||
`rate_ratio = A/(K·r + 1)`, `A = (1−p^(K+1))/(1−p)`; at `r ≈ 0.4–0.5`
|
||||
(a drafter only ~2× faster) break-even needs `p ≈ 0.7+` at K=3;
|
||||
- compare against the **incumbent**, not the no-draft baseline: a target
|
||||
already running its own MTP head is a high bar;
|
||||
- check the runtime accepts an external drafter at all — closed engines may
|
||||
enumerate a fixed drafter set and 400 anything else (capability gates
|
||||
keyed on the loaded checkpoint). That is a design boundary, not a config
|
||||
gap. In llama.cpp the portable mechanism is `--spec-draft-device/-devd`
|
||||
(draft gets its own device list) plus `-ngld`/`-otd`/`-ctkd -ctvd`.
|
||||
|
||||
## 6. Expressing a split (llama.cpp, the usual vehicle)
|
||||
|
||||
```bash
|
||||
-dev CUDA0,Vulkan0 -sm layer -ts F0,F1 # contiguous layer ranges (simplest)
|
||||
-ot "\.attn_.*=CUDA0" -ot "\.exps\.=Vulkan0" # per-tensor, by ROLE (KV co-location)
|
||||
```
|
||||
|
||||
- buffer types match by name at load (`ggml_backend_buft_name`), so device
|
||||
names work as `-ot` values;
|
||||
- `-cmoe`/`-ncmoe` are **CPU-only**; device-side MoE placement is `-ot`;
|
||||
- several backends in one binary is fine (CUDA + Vulkan is the practical
|
||||
pair; HIP + CUDA do not coexist) — `GPU_BACKEND_A=OFF` in the existing
|
||||
build's CMakeCache tells you what needs adding;
|
||||
- contiguous ranges are the WRONG split when KV bandwidth dominates: they
|
||||
drag half the attention layers (and their KV) back to the slow pool.
|
||||
|
||||
## 7. Hazards checklist
|
||||
|
||||
- Paged/sidecar tables (huge n-gram/embedding tables) must stay where they
|
||||
are read from; moving them is pure loss.
|
||||
- Keep `mmap` on for models with huge embedding tables (anonymous memory
|
||||
gets the process cgroup-killed).
|
||||
- CUDA graph capture can abort on unusual graph shapes (hyper-connections)
|
||||
— check before assuming optimisations apply.
|
||||
- A brand-new backend on the second device may not implement the arch's ops
|
||||
at all: make the first measurement a **baseline of the runtime on that
|
||||
device alone** — a split's arithmetic is worthless if the baseline is
|
||||
worse than the incumbent engine.
|
||||
- A split consumes the fast device entirely — check what else it was
|
||||
serving before trading it away.
|
||||
- Benchmark discipline on production hosts: stop and mask first
|
||||
([[Fleet Ops Playbook]]).
|
||||
|
||||
## 8. Verdict shape
|
||||
|
||||
Rank and report; prefer the boring answer when it wins:
|
||||
|
||||
1. **Two independent endpoints** (zero shared traffic) — often the real
|
||||
answer to "what's a useful combination", especially over a narrow link.
|
||||
2. **One process, weights-resident, KV co-located with attention on the
|
||||
fast pool** — promising and quantifiable (+% decode at long context);
|
||||
needs a dual-backend build and a baseline measurement.
|
||||
3. **Hot-expert cache / per-token weight streaming** — affordable only if
|
||||
the miss set per token is small (a few experts; check
|
||||
`misses × bytes × tok/s` against link bandwidth) and overlapped; injects
|
||||
PCIe latency into the critical path; no mainstream runtime implements a
|
||||
dynamic expert cache.
|
||||
4. List explicitly what does NOT help: TP/collectives across unequal
|
||||
devices, KV on the far device, model loading over the link,
|
||||
prefill-motivated placements.
|
||||
|
||||
## Related
|
||||
|
||||
- [[PCIe Link and Atomics]] — the link and collective walls underneath
|
||||
- [[Llama Cpp Multi-Device Splits]] — the deployed split shapes
|
||||
- [[Llama Cpp Serving Sizing]] — KV and pool arithmetic
|
||||
@@ -0,0 +1,271 @@
|
||||
---
|
||||
type: reference
|
||||
title: LLM Bench Windows and Measurement
|
||||
description: Stopping-and-restoring benchmark windows on the fleet LLM hosts (llama-bench sweeps, HTTP param benches, multi-arm vLLM offload sweeps), honest benchmark methodology, and the usage-counter measurement law under speculative decoding.
|
||||
tags: [benchmarking, llama-bench, ab-testing, speculative, vllm, fleet]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# LLM Bench Windows and Measurement
|
||||
|
||||
Fleet law first: **STOP the production service before any GPU experiment**
|
||||
(AGENTS.md Benchmarking Policy) — a bench beside a live service starves the
|
||||
GPU and corrupts results; an unisolated GPU experiment has precedent for
|
||||
wedging a host until power cycle. Reference harnesses:
|
||||
`scripts/bench-batch-ubatch.sh`, `scripts/autoresearch.sh`,
|
||||
`scripts/autoresearch/frank-primary-arms.sh`.
|
||||
|
||||
## 1. The window structure (one config per arm, a driver owns the window)
|
||||
|
||||
1. **Stop + gate before every arm** — `wait_for_clear`: unit NOT active AND
|
||||
no RUNNING container of that name (`podman ps`, not `-a`; `-a` counts
|
||||
EXITED containers and can never clear — `rm -f` the exited one first) AND
|
||||
port free (`ss -ltn`). The graceful-stop race cost a whole arm 2026-10-01:
|
||||
`ExecStop=podman stop -t 120` drains up to 2 min; a driver that sleeps 3 s
|
||||
then polls `/health` gets **200 from the still-running PRODUCTION engine**
|
||||
— the arm "passes" with production's boot log, then `podman exec` dies
|
||||
with `container state improper`.
|
||||
2. **Boot the arm as the service, not a scratch port** (no scratch ports on
|
||||
LLM hosts — one card, one server). The serve script takes its config via
|
||||
env-file override (`FRANK_SMALL_ENV_FILE`, `FRANK_PRIMARY_AR_ENV_FILE`)
|
||||
pointing at a staged copy; the checkout is never mutated.
|
||||
3. **Fast-fail the health wait**: poll `podman ps | grep -qx <name>` inside
|
||||
the loop — a refused boot dies in one poll instead of 20 min. Capture the
|
||||
serve script's FULL stdout on failure (a `tail -8` loses engine-init root
|
||||
causes; raise happens above the tail).
|
||||
4. **Measure** (§4), capture boot-log engine facts (KV pool, arena slots,
|
||||
SpecDecoding lines) — they survive measure crashes.
|
||||
5. **Teardown + converge**: kill by explicit PID/pgid (never pattern kills),
|
||||
`podman rm -f` scratch, wait `MemAvailable ≥ ~230 GiB` on dankserver
|
||||
before the next boot; restore production from the EXIT trap and verify the
|
||||
**conjunction oracle**: health 200 + `is-active` + `is-enabled` +
|
||||
mask-symlink clean. A bare `systemctl start` is not a restore oracle.
|
||||
(`systemctl mask` FAILS on a relinked unit — log it, explicit stop is
|
||||
enough: systemd never auto-restarts an explicitly stopped unit.)
|
||||
6. **Warm boots cycle ~2 min** (page cache), cold ~8 min — size timeouts for
|
||||
mixed speeds.
|
||||
|
||||
**Destructive tails (2026-10-01):** a scratch script whose tail does
|
||||
`systemctl stop` + `podman rm -f <PRODUCTION container>` fires later if a
|
||||
probe loop survived the driver kill — it can destroy a just-deployed
|
||||
production engine (signature: deploy exits green, then seconds-to-minutes
|
||||
later `Exited (0)`, `Deactivated successfully`, clean exit = removed, not
|
||||
crashed). Before ANY deploy: `pgrep -af` every scratch script of the session
|
||||
and kill strays by explicit PID; killing the dev-side job does NOT kill the
|
||||
host-side `deploy-remote.sh` (two of them once managed the same unit).
|
||||
|
||||
Two failure shapes inside the window that read as "the bench died":
|
||||
|
||||
- **Stale reads from a stopping container.** `podman stop` answers while the
|
||||
engine inside is still flushing; a `podman logs` taken in that window
|
||||
returns the boot-prefix only. Poll `podman ps` for `Exited (0)` (or
|
||||
`podman wait`) before reading logs, or you will write "no results" into
|
||||
the table for a run that has them.
|
||||
- **A D-state worker wedges the stop.** A bench child that wedged in the
|
||||
kernel ignores SIGTERM, so `podman stop -t N` burns its full timeout and
|
||||
then the container is `Stopping` forever. Kill by container first
|
||||
(`podman kill <ctr>` → SIGKILL path), verify the process is gone from
|
||||
`podman ps` AND `nvidia-smi`, and only then re-serve. Never let the
|
||||
window script proceed to a redeploy with a zombie bench worker holding
|
||||
VRAM.
|
||||
|
||||
## 2. llama-bench sweeps (offline batch/ubatch/split/KV/depth)
|
||||
|
||||
**What llama-bench cannot express — decide BEFORE designing:** no
|
||||
`--ctx-size` (`n_ctx = n_prompt + n_gen + n_depth`, so a bench can NEVER prove
|
||||
activation buffers fit beside the DEPLOYED KV — only the real unit answers
|
||||
that), no `--parallel` (no slot shape, no concurrency), no `--spec-*`
|
||||
(`tg` numbers are the drafterless baseline, useful as a control only), and a
|
||||
warmup run is itself a full-size run. `-d N` is a depth PREFIX (paid once per
|
||||
cell via a `llama_state_seq_*` save/restore); without it a `tg` run is at
|
||||
~zero depth and meaningless for long conversations.
|
||||
|
||||
**The separator trap:** `/` separates devices WITHIN one configuration, `,`
|
||||
separates sweep values — for both `-dev` and `-ts`. `-ts 36,29` parses as the
|
||||
single value 36 → everything on one card. `--tensor-split 36,29` (server) →
|
||||
`-ts 36/29` (bench). Ranges exist: `-ub 512-2048+256`.
|
||||
|
||||
**Harness shape:** one invocation per cell (an infeasible cell calls
|
||||
`exit(1)` and kills every later cell of a cross-product invocation — and
|
||||
infeasibility is usually the question); TSV row per cell as it completes +
|
||||
`bench.json` (`-o json`) + `bench.log` (`--progress`) + a 2 s `rocm-smi
|
||||
--showmeminfo vram` VRAM trace per cell (a cell that allocated less than
|
||||
intended looks fast); resume by skipping non-empty cell outputs; stop the
|
||||
service in the harness, restore from `trap … EXIT`, verify afterwards. Match
|
||||
the deployment's load flags exactly (`-ngl -sm -ts -fa -ctk/-ctv -t`).
|
||||
|
||||
**Run under a transient unit, not a shell:**
|
||||
|
||||
```bash
|
||||
sudo systemd-run --unit=<name> --collect --no-block --property=Type=oneshot \
|
||||
--property=LimitMEMLOCK=infinity --property=User=dank \
|
||||
/home/dank/serverAppConfigs/scripts/<harness>.sh
|
||||
```
|
||||
|
||||
Deploy units set `LimitMEMLOCK=infinity` for ROCm staging; an SSH shell has
|
||||
8 MiB and fails the load for an unrelated reason. This also survives ssh
|
||||
disconnects. **Boot rungs under the production environment, not a bench
|
||||
shell** — a rootless shell probe booted 3×262144 bf16 that production refused
|
||||
(systemd `User=` + memlock change what the pinned-expert path holds):
|
||||
`sudo systemd-run --uid=<user> --unit=<name> -p LimitMEMLOCK=infinity -p
|
||||
Restart=no env VARFILE=… /path/serve.sh`.
|
||||
|
||||
**Verify before burning hours:** dry-run against a STUB binary with
|
||||
`SERVICE=<nonexistent-name>` (the trap/stop never reached), tiny grid,
|
||||
including one forced-`exit 3` cell — prove the TSV row, failure capture,
|
||||
resume logic. Validate the real flag set in ~1 s by pointing `-m` at a
|
||||
nonexistent file. **Reading:** bracket the sweep (reference cell first AND
|
||||
last); check per-cell `stddev_ts` (a difference smaller than spread is not a
|
||||
result); single-sequence decode is usually flat across batch/ubatch (prefill
|
||||
knob); batch ≥ ubatch (llama.cpp clamps ubatch to batch, silently duplicating
|
||||
smaller-batch cells); report what the grid cannot prove (drafter, slots, real
|
||||
context).
|
||||
|
||||
## 3. HTTP benches of server-side parameters
|
||||
|
||||
`llama-bench` benches the bench binary, not the running server — server-side
|
||||
params (`--threads-batch`, `--ubatch-size`, `--spec-type`, `--parallel`) need
|
||||
HTTP:
|
||||
|
||||
- **Prefill**: large prompt, `max_tokens=1` (server crashes on 0), measure
|
||||
`prompt_tokens / wall_time`.
|
||||
- **Decode**: `max_tokens=16384` minimum, NEVER 1 (warmup, cache growth and
|
||||
spec acceptance all stabilize over thousands of tokens); measure
|
||||
`predicted_per_second` from `timings` or `completion_tokens / wall_time`;
|
||||
temperature 0.
|
||||
|
||||
Known fleet harnesses: `scripts/bench-threads-batch-http.py`,
|
||||
`scripts/bench-decode-threads-batch.py`, `scripts/bench-spec-decode.py`.
|
||||
sglang-specific: `max_tokens=1` on `/v1/chat/completions` with the qwen3
|
||||
reasoning parser streams zero deltas — use `/v1/completions`.
|
||||
|
||||
## 4. The measurement law (under speculative decoding this is absolute)
|
||||
|
||||
**Never count SSE chunks.** Rejected drafts break the 1:1 chunk↔token
|
||||
mapping; chunk counting reported 8.3 tok/s for a config whose server counters
|
||||
said 23–26 — a 3x false regression that nearly abandoned the change that
|
||||
worked. Always:
|
||||
|
||||
```python
|
||||
decode_tok_s = usage.completion_tokens / (wall - ttft)
|
||||
# stream=True, stream_options: {include_usage: true}; ttft = first content-bearing chunk
|
||||
```
|
||||
|
||||
vLLM's 10 s `Accepted throughput` log lines are a rolling window, not a
|
||||
measurement (6.2 → 13.1 across adjacent windows on identical settings) — use
|
||||
them for acceptance STATISTICS only (`Avg Draft acceptance rate`, `Mean
|
||||
acceptance length`, per-position rates). Warm pass + ≥3 runs, median;
|
||||
temperature 0; ≥256 tokens. **A decode "jump" with early-stopped completions
|
||||
(ct < 256) and acceptance > 90 % is degenerate generation** (the fork's
|
||||
repetition-halt fired at ~140 tokens on corrupted numerics) — inspect text
|
||||
before believing a win. Run-to-run spread is 5–10 %: repeat arms; one sample
|
||||
nearly shipped the wrong setting.
|
||||
|
||||
## 5. Honest methodology
|
||||
|
||||
- **Match the corpus to the workload.** Token soup under-reports speculative
|
||||
decoding (2.9–3.6 tok/update vs 3.86–5.8 on real code) and mis-ranks draft
|
||||
settings; acceptance, not step time, moves throughput. Build the corpus
|
||||
from the repo's own sources: `{"prompt": "<instruction>\n\n<file body>"}`,
|
||||
`--custom-output-len`.
|
||||
- **Salt every run** with a unique per-arm token in the first bytes — it is
|
||||
the prefix-cache key (and a proxy routing key). Without it arm 2 measures
|
||||
arm 1's warm cache.
|
||||
- **Two workload shapes need two clients:** distinct prompts (batch
|
||||
behaviour, prefill parallelism) and sequential turns with growing history
|
||||
(TTFT of the first streamed chunk per turn — the only measurement that
|
||||
exposes prefix-cache behaviour). These pull in opposite directions: a cold
|
||||
batch spreads prefill across replicas, a conversation's next turn only pays
|
||||
if it lands where its prefix is cached.
|
||||
- **Read the engine's ledger, not the card:** `Available KV cache memory` /
|
||||
`GPU KV cache size: N tokens` / `Maximum concurrency for M tokens per
|
||||
request: X.Xx` is the capacity number. vLLM's `Prefix cache hit rate:` line
|
||||
is a LIFETIME average across every prompt the replica ever saw — send the
|
||||
same prompt twice to an idle engine and compare TTFTs (measured 22.3 s cold
|
||||
→ 1.4 s warm for a 50k prefix). `GPU KV cache usage` says the pool is not
|
||||
full; it does NOT say a prefix survived. The KV budget is NOT monotonic in
|
||||
chunk size (5.44 GiB at CHUNK 8192, 4.22 at 16384, 4.16 at 4096) — measure
|
||||
three points before treating a knob as a lever.
|
||||
- **Soak before believing:** 15–30 min of concurrent families, count failures
|
||||
AND latency; report hit/miss split and medians with min/max — the median of
|
||||
a bimodal distribution describes neither mode.
|
||||
- **Verification that is not the benchmark:** run the client the fleet
|
||||
actually uses; and a deployed unit is not proven until an unattended reboot
|
||||
brings it back.
|
||||
- **Record the "no" answers** with numbers and mechanism — placement balance,
|
||||
concurrency admission and chunk-size tweaks all measured flat or worse
|
||||
here; the wall was arithmetic (cached prefixes + live sequences share one
|
||||
pool).
|
||||
|
||||
## 6. Sweep hygiene (the arms that produced wrong verdicts)
|
||||
|
||||
- **Prefetch uniformity:** every arm in a sweep carries the same
|
||||
expert-prefetch setting. The 2026-09-29 campaign measured losing arms with
|
||||
prefetch ON and incumbents OFF; untaxed, verdicts inverted (no-spec
|
||||
12.11 → 41.47; gather 32×64 parity → +18.7 %). Never re-read a taxed
|
||||
verdict.
|
||||
- **Knob-landed preflight:** for every `-e` knob, confirm
|
||||
`podman exec <ctr> printenv <VAR>` — a knob the fork silently ignores
|
||||
produces a "parity" arm that is really the default (v1 of the gather sweep
|
||||
lost all 3 arms this way; serve-script hooks are additive and can drop
|
||||
vars).
|
||||
- **Always run an explicit OFF arm:** the 2026-09-29 campaign's only win was
|
||||
deleting a baseline's silent `--expert-prefetch 1` (+40–51 %). Challengers
|
||||
compare against default-off, not "incumbent + one flag".
|
||||
- **Pin the capacity bar:** `--kv-cache-memory-bytes` (contract) rather than
|
||||
autofit (remainder) so pool numbers are reproducible; compute need =
|
||||
tokens × measured B/token (hybrid models: KV dtype does NOT double mamba
|
||||
states — measure, don't assume 2x).
|
||||
- **Report decode AND prefill (TTFT) per rung:** a decode-only view nearly
|
||||
deployed a 2x-prefill-tax config. Per-depth tables, never a single median —
|
||||
one fork's gather win grows with depth (54 t/s @1k, 39.5 @short).
|
||||
- Container recreate (`podman rm -f`) wipes `podman cp`'d harnesses — re-copy
|
||||
`mtp_measure.py`/`bench_iter.py` after EVERY recreate, or mount them.
|
||||
|
||||
## 7. Bash laws that ate windows (each cost an arm)
|
||||
|
||||
1. `${VAR:+SPEC=$1}` in front of a command expands to a bare word →
|
||||
`SPEC=3: command not found`, nothing runs. Pass literal assignments:
|
||||
`SPEC="${1:-}" bash serve.sh`; the callee's `${SPEC:+--flag}` skips empty.
|
||||
2. `${2WAY:-0}` is fatal `bad substitution` (digit-leading name), and inside
|
||||
an if/else it aborts ONLY that compound statement — the script CONTINUES,
|
||||
every arm tears down clean with zero measurements. Detection law: `ls`
|
||||
the per-arm `.measure.txt` files before trusting any summary; arm echoes
|
||||
are not evidence.
|
||||
3. `local a=$1 b=$x` expands `$x` before the builtin assigns — under `set -u`
|
||||
it aborts AFTER the harness stopped production. Two statements.
|
||||
4. `sed -i` renames the file; already-running bash keeps the OLD inode —
|
||||
mid-flight patches miss arms launched in the preceding minutes. Patch
|
||||
early or run a make-up window.
|
||||
5. Never heredoc python through ssh→zsh→bash — write to a file, ship it
|
||||
(`ssh host 'cat > /path && chmod +x /path' < local.sh`); nested quoting
|
||||
fails silently. Same trap in `git commit -m`: backticks become command
|
||||
substitution.
|
||||
6. No comments INSIDE a backslash-continued command (the continuation joins
|
||||
the comment); verify multi-line command construction with `bash -n` AND a
|
||||
render test.
|
||||
|
||||
## 8. Results location and recovery (frank-primary arms)
|
||||
|
||||
| copy | path |
|
||||
|---|---|
|
||||
| per-arm evidence | `$ARMS_DIR` = `~/.jobs/<run-stamp>/` (`.rc`, `.done` beside it) |
|
||||
| summary | `$ARMS_DIR/summary.tsv` = `arm rc decode_tps decode_tps_t0.7 kv_tokens boot_s` + `<arm>.out/.env/.overlay/.rc/.metrics` |
|
||||
| server-side last window | `/home/dank/.autoresearch-frank-primary/run.json` |
|
||||
| durable memo | `.../last/run.json` + `last/hash` (promoted only after bench AND verified restore) |
|
||||
| fallback | `~/.omp/agent/sessions/<ws>/<session>/<N>.{shake,bash}.log` (permanent) |
|
||||
|
||||
`/tmp/frank-primary-arms-*` and `/tmp/armstub/` are stub self-tests —
|
||||
constants 99.9/88.8 identical across arms by construction; **identical
|
||||
numbers across arms == stub or bug, never a measurement.** After compaction,
|
||||
re-harvest with `grep -rhoE 'decode_tps[a-z0-9_.]*=[0-9.]+'` over the session
|
||||
logs. A window costs ~20 min warm; a detached runner's timeout ≥ 3600 s per
|
||||
window.
|
||||
|
||||
## Related
|
||||
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan arithmetic being swept
|
||||
- [[DFlash DSpark Speculative Decoding]] — spec A/B protocol and shapes
|
||||
- [[LLM Performance Diagnosis]] — profiling and attribution methods
|
||||
- [[Remote Job Runner]] — detached execution of long sweeps
|
||||
- [[frank-medium-llamacpp-dflash2]] — the measured ubatch-optimum history
|
||||
@@ -0,0 +1,221 @@
|
||||
---
|
||||
type: reference
|
||||
title: LLM Performance Diagnosis
|
||||
description: Probing why a served LLM is slow — the flat-depth probe, the expert-offload duty-cycle arithmetic, kernel-time attribution via the profiler window, live telemetry without benchmarking, RAM-bandwidth ceilings, IBS tensor attribution and prefix-cache-aware routing.
|
||||
tags: [performance, profiling, moe, pcie, prefix-cache, routing, telemetry]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# LLM Performance Diagnosis
|
||||
|
||||
When a served endpoint is slower than wanted and someone names a suspect
|
||||
("PCIe is saturated", "the gather is the bound"), throughput arithmetic cannot
|
||||
settle it — it is usually circular (it assumes the suspect consumes the step).
|
||||
Get data in this order; the profiler window alone has retired multi-day
|
||||
optimization plans.
|
||||
|
||||
## 1. Is decode context-bound at all? (the flat-depth probe)
|
||||
|
||||
Sweep decode across context depth. **Flat** ⇒ attention over history is not
|
||||
the cost; the problem is per-token expert movement and KV/attention tuning is
|
||||
ruled out entirely. A genuinely context-bound server shows a clear downward
|
||||
slope. Record cold vs warm separately — a small gap means the cache is
|
||||
working. Measured on the fp8 expert-offload shape: 12.11 tok/s FLAT from 304
|
||||
to 16,095 tokens.
|
||||
|
||||
## 2. Duty-cycle arithmetic (the "GPU not fully utilized" answer)
|
||||
|
||||
Symptom: smooth ~25–35 % utilization with 100 % peaks on an expert-offload
|
||||
MoE. This is the EXPECTED signature, not a fault:
|
||||
|
||||
```
|
||||
per decode step: compute burst ~100 ms (100 %) → expert-gather wait ~400 ms (~0 %)
|
||||
resident fraction = arena slots / total experts (e.g. 6574/24576 ≈ 27 %)
|
||||
PCIe bytes/token = miss_rate × lookups × slab_bytes (0.73 × 480 × 2.344 MiB ≈ 0.84 GB)
|
||||
predicted duty ≈ compute_ms / (compute_ms + wait_ms)
|
||||
```
|
||||
|
||||
Validate the terms against the deployed engine's boot plan
|
||||
([[RDNA4 MoE Expert Offload VRAM Budget]]). If predicted ≈ observed, the low
|
||||
average is structural. Note small-transfer reality: bulk bandwidth (91 % of
|
||||
ceiling at 1 GiB) overstates what 2–3 MiB slabs get (~72 % plus ~20 %
|
||||
per-transfer latency). Before blaming the link, measure it
|
||||
([[PCIe Link and Atomics]]) and compare the engine's effective gather rate to
|
||||
the measured slab-size bandwidth — a 40–50 % gap is the software read path,
|
||||
not the wire. `rocm-smi --showuse` reads ~0 % during the whole multi-minute
|
||||
weight load: show VRAM (`--showmeminfo vram`) as the load-progress signal.
|
||||
|
||||
## 3. Kernel-time attribution (the profiler window)
|
||||
|
||||
Run BEFORE building any replacement kernel.
|
||||
|
||||
1. Stop + mask production; boot the shape under test with ONLY the profiler
|
||||
config added. vLLM argv:
|
||||
`--profiler-config '{"profiler":"torch","torch_profiler_dir":"/prof","torch_profiler_with_stack":false,"torch_profiler_record_shapes":false,"torch_profiler_with_memory":false,"torch_profiler_dump_cuda_time_total":false}'`
|
||||
— `with_stack=false` is load-bearing (with stacks the stop hangs and the
|
||||
trace buffer grows ~40 GB → kernel OOM-kill of the engine core, anon-rss
|
||||
~95 GB; the worker-side stop can hang 5 min on ROCm/kineto). Bind-mount
|
||||
`/prof`. If a shell substitutes values into that JSON, verify the escapes
|
||||
actually landed.
|
||||
2. Window it: warm request → `POST /start_profile` → N generations of ≥128
|
||||
tokens (~15 s of traffic) → `POST /stop_profile` (client may time out at
|
||||
60 s; the trace still lands). `telemetry_report` has ONE caller: the
|
||||
gpu_worker profiler-stop path. Overhead is small enough to keep decode
|
||||
comparable (9.6 vs 9.9–10.0 tok/s measured).
|
||||
3. **Parse ON THE SERVER** (traces are 100–150 MB gzipped, GB expanded).
|
||||
Rank `ph=="X"` events with `cat in {kernel, gpu_memcpy, gpu_memset}` by
|
||||
summed `dur` (µs): report % of device time, launch count AND µs/call
|
||||
(22 µs × 708k launches is a different problem than 4 ms × 4k), plus the
|
||||
first-to-last event span (fraction of wall the device was busy). Aggregate
|
||||
by FULL mangled name (template args are the tuning signal) and by stripped
|
||||
symbol. **Trap: `gpu_user_annotation` events masquerade as GPU time** (a
|
||||
naive sum reported 63 % in one `execute_context_0(...)` entry; spans
|
||||
overlap and can exceed the wall clock).
|
||||
4. Divide by the window's token count → ms/token; if kernel time ≈ step
|
||||
time, the table IS the optimization plan.
|
||||
|
||||
Worked example (GLM-5.3-Flash, 2×GPU, offload, decode 9.9 tok/s):
|
||||
|
||||
```
|
||||
34.8 % 708,500 launches vectorized_elementwise_kernel <- the actual bound
|
||||
10.0 % 42,000 MoE W4A16 GEMM
|
||||
8.9 % 83,160 router/top-k (sort-based)
|
||||
6.5 % 133,920 hyper-connection pre/post kernels
|
||||
1.05 % 20,664 <the long-suspected expert gather>
|
||||
```
|
||||
|
||||
No amount of link arithmetic would have produced those two conclusions: the
|
||||
suspect was 1 %, and the mass was pointwise glue from an uncompiled trunk —
|
||||
whose own boot log names the cause (`FULL_DECODE_ONLY` override). It also
|
||||
predicted a negative result in advance (spec decode cannot help a shape whose
|
||||
per-token cost scales with the token count) which matched an unexplained flat
|
||||
MTP measurement.
|
||||
|
||||
## 4. Telemetry without benchmarking (live hosts)
|
||||
|
||||
When benching is forbidden (production serving), mine the runtime's own
|
||||
per-request telemetry — it measures the real workload and costs nothing.
|
||||
|
||||
- llama.cpp (root unit, `-lv 3`): `sudo journalctl -u <unit> --since "-3h"
|
||||
--no-pager` → `slot print_timing` lines (prompt eval, eval, total, `graphs
|
||||
reused`, `draft acceptance … mean len`, `n_gen = N, tg = X t/s, tg_3s = …`).
|
||||
- halogen (rootless quadlet): `journalctl --user -u <unit>` (**never sudo** —
|
||||
per-user storage) → `serve_api: mtp 28 tok in 0.66s = 42.45 t/s | 16
|
||||
rounds, commit 1.75/round | prompt 58, prefill 0.92s | detok 24us/tok`.
|
||||
- Read the envelope, never one sample: short-prompt `prompt eval` is
|
||||
overhead-dominated (57 tok @ 168 t/s) — quote the longest available (1106
|
||||
tok @ 1013 t/s); decode and acceptance are content-dependent (0.46–0.76
|
||||
measured on one unchanged build); `graphs reused > 0` / `selected slot by
|
||||
LCP similarity` prove prefix-cache reuse.
|
||||
- **Attribute PCIe without touching the model:** background `nvidia-smi dmon
|
||||
-s t -c 16 -o T`, ONE ordinary `max_tokens=200` request, `wait` at the end
|
||||
(without it ssh closes and the client dies — the journal shows `W srv stop:
|
||||
cancel task` and NO print_timing while curl still reports plausible
|
||||
status). Measured: peak ~230 MB/s against a ~7,900 MB/s link = ~3 %, 0 at
|
||||
idle — the sampled-logits stream, not the transport, is decode traffic on
|
||||
that shape.
|
||||
- One `max_tokens=200` request is production traffic and fine; anything that
|
||||
pins the card is a bench → stop-and-mask first. Check WHICH instance you
|
||||
are reading (`systemctl status | head -12`; task ids jump across
|
||||
restarts). The engine's own ledger (`KV self size`, `n_slots`,
|
||||
`n_ctx_slot`) outranks your model of it.
|
||||
|
||||
## 5. The fork's data path (before blaming the link)
|
||||
|
||||
Offload engines pick between DMA copy-in and read-through UVA dereference
|
||||
inside the GEMM. Grep the installed package for the LRU/offload module and
|
||||
read the BRANCH CONDITION, not just the env names. A token-count threshold
|
||||
decides the path, and it interacts with spec decoding: an MTP step is k+1
|
||||
tokens, so the step can silently cross the threshold onto the slow path as
|
||||
draft depth grows. Check the threshold's premise: a default tuned at
|
||||
12.4 MiB/expert does not transfer to 2.344 MiB/expert, where copy-in can be
|
||||
several times faster. TunableOp ghost-skip: `CLAV_TUNABLEOP_SWEEP=1` alone
|
||||
does NOTHING under expert offload (`CLAV_TUNABLEOP_WITH_OFFLOAD=1` required);
|
||||
check for the skip line before concluding autotuning ran. Gather-geometry
|
||||
knobs read at import time → each arm is a full reload.
|
||||
|
||||
## 6. RAM-bandwidth-bound MoE ceiling (before recommending a quant)
|
||||
|
||||
```
|
||||
routed experts/token = top_k × n_layers
|
||||
bytes/token = routed_params × bytes_per_param + shared_expert_bytes
|
||||
ceiling (tok/s) = measured_bandwidth / bytes_per_token
|
||||
```
|
||||
|
||||
Measure the HOST's real bandwidth, never assume DIMM ratings: 4-DIMM AM5
|
||||
boards downclock to DDR5-3600 regardless of 5600 sticks (two large DIMMs beat
|
||||
four small). Single-threaded reads under-report dual-channel ~4x — use the
|
||||
multi-threaded plateau of a checksumming reader (`scripts` pattern:
|
||||
`/tmp/bw.c`, 1/2/4/8/16 threads). Worked example (DeepSeek-V4.1-Flash):
|
||||
6×40 experts × 35.4M params = 8.5B params/token; at 3 bpw + 0.53 GB shared =
|
||||
**3.7 GB/token** → 42 GB/s ⇒ **11.3 tok/s ceiling**. Sanity anchors: H100
|
||||
NVL + CPU offload 226 GB ⇒ 5.8–9.0 tok/s; 2×3090 + 768 GB DDR4 ⇒ 12.2.
|
||||
CPU compute rarely binds below 4 bpw (~5.3 flops/byte ⇒ ~223 GFLOPS at 42
|
||||
GB/s); PCIe expert STREAMING is worse than CPU compute (Gen5 x8 ~22.7 GB/s
|
||||
effective < half of RAM); prefill reads the union of a batch's experts
|
||||
(prefill ≈ bandwidth / total_expert_bytes × ubatch). A quant that fits only
|
||||
by ignoring the Engram table does not fit (Engram = sparse n-gram lookup,
|
||||
NVMe-tolerant; the routed experts are not). An iGPU GTT aperture is an ALIAS
|
||||
of the same RAM, not extra memory.
|
||||
|
||||
## 7. IBS model-access attribution (AMD CPUs)
|
||||
|
||||
Real CPU access patterns for mmap'd weights, live, no stop (established on
|
||||
frank-primary, 2026-09-26, ~2.7M attributed samples in 75 s). Reference:
|
||||
`scripts/frank-primary-expert-activity.py` (self-contained dump + sample +
|
||||
analyze; `--analyze-only`).
|
||||
|
||||
1. GGUF header-only scan (ms for a 50 GiB part) → tensor extents. **Extents
|
||||
from CONSECUTIVE offsets, never a type-size table** — a wrong per-element
|
||||
size silently under-covers every tensor and dumps tail samples into a junk
|
||||
bucket (first run lost 59 % of expert samples; a uniform gap/exps ratio
|
||||
~1.4 across parts is the tell). Data start = `(tell + 31) & ~31`.
|
||||
2. `/proc/<pid>/maps` VMAs (7-group regex; pgoff = group 4, path = group 7)
|
||||
matched against part files; multi-VMA per part is normal.
|
||||
3. `sudo perf record -e ibs_op// -c 50000 --data -o out.perf -p <pid> --
|
||||
sleep <dur>` WHILE a mixed workload runs (idle perf numbers mean nothing).
|
||||
`sudo perf script -i out.perf -F addr` for addresses.
|
||||
4. Attribute: bisect VMAs, `foff = pgoff + (addr − vma_start)`, bisect tensor
|
||||
starts; per-expert index = `int(lo / (span / N_EXPERTS))` (expert-major
|
||||
rows).
|
||||
5. Verify before believing histograms: gap ≈ 0, every tensor-bearing part
|
||||
nonzero, and MoE blocks should be FLAT by construction (top-k fires every
|
||||
block every token) — a non-flat block histogram means mis-attribution
|
||||
first. Established law: expert routing is per-block flat, per-EXPERT
|
||||
skewed.
|
||||
|
||||
## 8. Prefix-cache-aware routing (multi-replica serving)
|
||||
|
||||
Measure with a **sequential-turn client**, not only a batch benchmark — a
|
||||
batch of distinct prompts parallelises prefill across replicas and hides the
|
||||
problem. The two failure modes: round-robin loses every other cache hit
|
||||
(90k-token contexts: warm-turn TTFT 2676 ms round-robin vs 1096 ms pinned);
|
||||
pinning-everything loses a card (16 requests: 18.6 s spread vs 30.1 s
|
||||
pinned). **The rule that gets both: route by request shape** — a request
|
||||
carrying more than one message is a continuation (agent loop re-sending its
|
||||
context) → pin to the replica that served that prefix; anything else → spread
|
||||
with a COUNTER, not a prefix hash. Add failover: demote a replica on
|
||||
connection failure and retry on a live one BEFORE writing anything to the
|
||||
client (safe pre-stream; never retry after streaming started).
|
||||
|
||||
The wall no policy fixes: **a running sequence and its cached prefix draw on
|
||||
the SAME pool.** Three concurrent 50–80k families against 122,636 tokens/card
|
||||
thrash under every placement and admission policy tried (counter placement,
|
||||
load-balanced placement, one-large-request-per-card admission, both chunk
|
||||
directions — the last COSTS KV in both directions). When the working set
|
||||
exceeds aggregate KV, say so: the levers are capacity, fewer concurrent
|
||||
large contexts, or a different platform. Traps: a lifetime hit-rate counter
|
||||
is not a per-request verdict (probe directly: same prompt twice to an idle
|
||||
replica — 22.3 s cold → 1.4 s warm); salt corpora; record sample sizes;
|
||||
`vllm bench serve` needs `--tokenizer <local path>`; verify metric names
|
||||
against the live endpoint before dashboards. Test shape: drive the routing
|
||||
function directly with synthetic bodies (stdlib-only policy module);
|
||||
conversation identity is the FIRST N KiB of prompt — a shorter tested prefix
|
||||
hashes differently per turn and pinning silently never happens.
|
||||
|
||||
## Related
|
||||
|
||||
- [[LLM Bench Windows and Measurement]] — the window discipline and usage-counter law
|
||||
- [[PCIe Link and Atomics]] — link probing and the streaming-vs-compute ceiling
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan behind duty-cycle terms
|
||||
- [[vLLM Porting and Compile Triage]] — the compile fix for the pointwise-glue bound
|
||||
@@ -0,0 +1,281 @@
|
||||
---
|
||||
type: reference
|
||||
title: Model Swap and Rollback
|
||||
description: The full model-swap discipline for the LLM fleet — fit preflight, fetch and payload verification, runtime build/flag checks, git-only config, live validation, systemd handoff, close-out, the three-pin revert law, and research-verification discipline.
|
||||
tags: [models, swap, gguf, huggingface, rollback, deployment, verification]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Model Swap and Rollback
|
||||
|
||||
Applies to every endpoint swap on the fleet (frank-small / frank-large /
|
||||
frank-primary). Repo law: all config flows through git; the live models are
|
||||
host state under `/home/dank/models/<name>/`.
|
||||
|
||||
## 1. Fit preflight — never deploy a swap without confirming it FITS
|
||||
|
||||
1. Get the checkpoint's on-disk/weight size AND precision from the HF card
|
||||
(`Precision: BF16` = 2 B/param; FP8/Q8 ≈ 1 B; Q4 ≈ 0.5 B). 35B params:
|
||||
BF16 ≈ 66–70 GB, FP8/Q8 ≈ 33–35 GB, Q4 ≈ 17–20 GB.
|
||||
2. Compare against the HOST'S usable RAM (not just VRAM) for runtimes that
|
||||
stream weights from RAM (host-RAM expert offload, llama.cpp CPU offload,
|
||||
layer splits). Budget weights + KV pool + OS + page cache; **if weights
|
||||
alone exceed RAM, ABORT** — pool tuning shrinks KV, it does not shrink
|
||||
weights.
|
||||
3. **The failure mode is a self-sustaining thrash, not a crash:** a large
|
||||
swap makes the kernel PAGE instead of OOM-killing → sshd/user space
|
||||
starved indefinitely, no recovery window opens by itself; systemd
|
||||
`Restart=always` turns it into a crash-reload-rethrash loop; `enabled` +
|
||||
`WantedBy=multi-user.target` re-thrashes on every reboot. A power cycle
|
||||
does NOT fix it.
|
||||
|
||||
Recovery (may need console / SysRq-to-GRUB):
|
||||
|
||||
```bash
|
||||
sudo systemctl mask --now <svc> # break the reboot trap FIRST
|
||||
cd <checkout> && git fetch origin && git reset --hard origin/main
|
||||
grep MODEL_DIR <env> # confirm the fixed target
|
||||
sudo systemctl unmask <svc> && sudo systemctl restart <svc>
|
||||
curl -s <health> # then a REAL completion
|
||||
```
|
||||
|
||||
Deploy caveat: "healthy after Ns" = systemd ACTIVE, not model LOADED —
|
||||
confirm with a real `/v1/chat/completions`. RTX 3090 (Ampere) has no
|
||||
FP8/FP4 tensor cores: FP8 weights run via upcast but must still fit; choose
|
||||
quant by size fit, then accept the compute path the card supports. Before
|
||||
choosing a smaller quant, state WHICH of the two you are buying — headroom,
|
||||
or a design change (a smaller quant does not change what the runtime can
|
||||
express; per-tensor cross-device placement stays refused under
|
||||
`--override-tensor`, and a layer split moves only contiguous layer ranges).
|
||||
|
||||
## 2. Ground truth first
|
||||
|
||||
- Read the HF model card (runtime pin, hard constraints, recommended args)
|
||||
+ the repo trio `scripts/<svc>.env`, `scripts/serve-<svc>.sh`,
|
||||
`scripts/<svc>.service` + the runbook under `serverAppConfigs-docs/LLM/`.
|
||||
- Check the host before downloading anything: `ls /home/dank/models/`,
|
||||
`df -h /home`, `du -sh` — bench-era or prior-deployment files often already
|
||||
exist.
|
||||
|
||||
## 3. Fetch and verify (HF payloads)
|
||||
|
||||
**Inventory by QUANT NAME, never by extension** — large quants ship as
|
||||
`.partNofM` fragments (Q4_K_M = 37.00+37.00+36.97 GiB), so an
|
||||
`endswith(".gguf")` filter makes Q4_K_S look nonexistent. Check both
|
||||
mradermacher repos (`…-i1-GGUF` imatrix, `…-GGUF` plain) and the model's own
|
||||
repo; the tree API's `blobs=true` gives `lfs.oid` per file. Gated mirrors
|
||||
return 403 even with a valid token until the account accepts the gate —
|
||||
prefer the ungated copy.
|
||||
|
||||
```bash
|
||||
# token out of band: mode-600 curl config; a token passed as -H is world-readable
|
||||
printf 'header = "Authorization: Bearer %s"\n' "$(cat ~/.cache/huggingface/token)" > /tmp/hf.curlrc
|
||||
chmod 600 /tmp/hf.curlrc
|
||||
curl -fL -C - --retry 10 --retry-delay 5 --retry-max-time 21600 -K /tmp/hf.curlrc -o "$DEST/$name" "$BASE/$name"
|
||||
```
|
||||
|
||||
`-f` is MANDATORY: `curl -o FILE URL` **exits 0 on HTTP 401/403** and writes
|
||||
the error body as the artifact (identical 153-byte shards, cheerful DONE,
|
||||
zero bytes of model). Never `pgrep -a` a running fetch (token is in
|
||||
`/proc/<pid>/cmdline`); watch progress by size:
|
||||
`while [ "$(stat -c%s … || echo 0)" -lt "$EXPECTED" ]; do sleep 30; done`.
|
||||
|
||||
**Verify, then assemble:** per part, byte count == Hub's declared size AND
|
||||
`sha256sum` == `lfs.oid`; concatenate in NUMERIC order; assembled total ==
|
||||
sum of parts (compute the sum with python, not by hand) and `head -c 4` ==
|
||||
`GGUF`. The assembled file has no published hash — the per-part oids + exact
|
||||
total are the chain. Keep the fragments when disk allows (only copies
|
||||
re-verifiable against the Hub); failure artifacts and the replaced payload
|
||||
go to `~/.AGENTTRASH/YYYYMMDD-topic`, never `rm` — upstreams DELIST quants
|
||||
and the local file may be the last copy.
|
||||
|
||||
Fetch-script one-liners: dedicated scripts (`scripts/download-*.sh`) already
|
||||
do `-f` + byte-count + LFS sha256 + `--verify-only`; ad-hoc fetches tend to
|
||||
skip all three. HF cache snapshots SYMLINK every file into `blobs/` and
|
||||
those links dangle inside containers — serve scripts and the deployer refuse
|
||||
symlinks and missing tokenizer files for exactly this reason.
|
||||
|
||||
## 4. Runtime build (when the card pins a fork/build)
|
||||
|
||||
- Clone + build with the card's exact cmake line, plus
|
||||
`-DLLAMA_BUILD_WEBUI=OFF` — WebUI asset provisioning (npm / HF llama-ui
|
||||
bucket) aborts llama.cpp link at 98 % on these hosts; the API server is
|
||||
unaffected.
|
||||
- Verify every flag the serve script passes exists in the new binary BEFORE
|
||||
touching the service: `llama-server --help | grep -q -- "<flag>"` for each;
|
||||
list MISSes. Read the GGUF's tensor histogram first if the payload uses a
|
||||
custom type — the FORMAT picks the runtime ([[GGUF Compatibility]]).
|
||||
|
||||
## 5. Config via git only
|
||||
|
||||
Edit `scripts/<svc>.env` (paths, LLAMA_BIN, MODEL_FILE, MMPROJ_PATH) + serve
|
||||
script header/defaults locally; `bash -n`; `./tests/run.sh` must pass;
|
||||
commit; push; `git pull --ff-only` on the target. Never edit configs on the
|
||||
remote. Renaming a model dir (`mv`) is host state and allowed. The env-file
|
||||
hook pattern (`FRANK_*_ENV_FILE`) is what makes bench windows and staged
|
||||
arms possible — keep it intact.
|
||||
|
||||
## 6. Live validation
|
||||
|
||||
- No scratch ports — one GPU cannot host two big models. Ensure the old unit
|
||||
is truly gone first: `systemctl is-active` can hang in `stop-sigterm`
|
||||
(llama.cpp never preempts an in-flight generation); poll until the PID
|
||||
exits.
|
||||
- Manual serve for smoke: `setsid nohup bash scripts/serve-<svc>.sh >
|
||||
/tmp/x.log 2>&1 &` — plain `ssh 'nohup … &'` dies with its ssh session.
|
||||
- Verify: health 200; a REAL `/v1/chat/completions`; reasoning flows
|
||||
(`reasoning_content` non-empty); if the endpoint takes images, a
|
||||
solid-color probe through an `image_url` content part (legacy top-level
|
||||
`images` is silently ignored; probe with ≥256 px).
|
||||
- vLLM specifics: first request after boot can take ~18 s (deferred Triton
|
||||
loads) — warm and discard; `--reasoning-parser` / tool parsers are part of
|
||||
the contract ([[vLLM Boot Forensics and Config Interrogation]]).
|
||||
|
||||
## 7. systemd handoff — THE TRAP
|
||||
|
||||
`./scripts/deploy.sh <host> --service <svc>` gates health, but a leftover
|
||||
manual server holding the port makes the health check pass while the UNIT
|
||||
crash-loops on bind. Always confirm ownership:
|
||||
`systemctl show <svc> -p MainPID` equals the single `pgrep -x llama-server`
|
||||
(or the container runtime PID); only then trust "healthy". If a real
|
||||
long-context task is queued, do the handoff anyway — client aborts are
|
||||
recoverable, drift is not.
|
||||
|
||||
## 8. Close-out (mandatory)
|
||||
|
||||
1. Docs: active runbook updated (build recipe, profile, validation numbers
|
||||
incl. real-workload prefill/decode); superseded runbook gets a RETIRED
|
||||
banner and stays as the rollback record; `LLM/index.md` mermaid + active
|
||||
list; AGENTS.md fleet table + runtime section + history chain; work-log
|
||||
entry in `serverAppConfigs-docs/log.md`.
|
||||
2. `~/.omp/agent/models.yml`: `contextWindow` must equal the REAL per-slot
|
||||
context — llama.cpp logs `n_ctx_slot` per prompt (`--parallel N` divides
|
||||
the pool; e.g. 262144 native / 2 slots = 131072 per slot); vLLM reports
|
||||
`max-model-len` and its concurrency line. Overselling causes silent
|
||||
client-side overflow or a discovered-wall 400. Also `maxTokens` vs the
|
||||
engine's cap, `input` capabilities (text/image), and
|
||||
`reasoningContentField` per runtime (llama.cpp/SGLang emit
|
||||
`reasoning_content`, vLLM emits `reasoning`). Fix every `modelRoles` entry
|
||||
that named the old endpoint.
|
||||
3. **Rollback = one env commit** (paths + LLAMA_BIN + flags), prior binary +
|
||||
payload kept on disk, rollback numbers recorded in the runbook.
|
||||
|
||||
## 9. Revert-pin an ad hoc experiment (the three pins)
|
||||
|
||||
A campaign is "done" only when all three exist — without them a
|
||||
`podman pull`, an image update, or a lost scratch script destroys the way
|
||||
back:
|
||||
|
||||
1. **Git tag (annotated, pushed):**
|
||||
`git tag -a <exp>-tuned-YYYYMMDD -m "<tuned env/flags + measured headline +
|
||||
runbook path + revert command>"` — a future agent must reconstruct the
|
||||
campaign from the tag message alone.
|
||||
2. **Image digest in the serve script:** floating tags move. Capture
|
||||
`podman inspect <ctr> --format '{{.ImageName}} {{.Image}}'` + `podman
|
||||
images --digests`, patch `IMAGE=` to `repo@sha256:<repodigest>` with a
|
||||
comment (image ID, date, old ref), verify `podman image exists
|
||||
repo@sha256:<digest>` and `bash -n`. Digest refs survive upstream pushes;
|
||||
`podman pull repo@sha256:<digest>` re-fetches exact bytes. One host can
|
||||
hold MULTIPLE images of one repo — pin by the digest of the RUNNING
|
||||
container, never by tag name. Never edit the pinned line back to a
|
||||
floating tag; new experiments branch from the digest.
|
||||
3. **Local alias:** `podman tag <id> <exp>-runtime:tuned-YYYYMMDD`.
|
||||
|
||||
Record scratch-script paths + full flag surface in the runbook (hook env
|
||||
vars in CAPS); expired images/scripts go to `.AGENTTRASH`, never `rm`.
|
||||
Pure measurement sessions that changed no state need only pin 1.
|
||||
|
||||
## 10. Research verification discipline (for "can model X run here?")
|
||||
|
||||
- Scouted research summaries have shipped fabricated citations on this fleet
|
||||
(invented HF repos, plausible-but-wrong PR numbers, stale engine versions).
|
||||
NEVER copy a scout's citations into a deliverable — re-verify each
|
||||
load-bearing URL directly (GitHub issues/PRs/repos, HF pages) before it
|
||||
enters anything.
|
||||
- The in-repo runbooks are FIRST-HAND evidence and outrank any web claim
|
||||
about our own hosts.
|
||||
- Verified anchors (2026-09-27): DeepSeek V4.1-Flash (552B/8B-16B active,
|
||||
CED arch, 1M ctx, MIT; preview was 284B/13B with native FP4+FP8 — early
|
||||
GGUF conversions garbage); Lucebox = the proven gfx1201 ROCm 7.2 path
|
||||
(heterogeneous R9700+Strix-Halo V4.1: 86 tok/s decode, 788 t/s prefill@2K).
|
||||
GLM-5.3-Flash (320B/18B, KDA+DSA+mHC, 1M ctx — ALWAYS bind `-c`, the
|
||||
default hard-hung a 128 GB host; native FP8 328.3 GB; `-fa off` MANDATORY,
|
||||
flash-attn breaks MLA correctness; llama.cpp arch `glm5next` lived in
|
||||
unslothai branch → ggml PR #27754, MTP landed d07e71ede 2026-08-30; quant
|
||||
ladder UD-IQ1_S 93.1 GB → UD-Q4_K_XL 199.7 GB, GSQ-RCO 3.5-bit −1.40 pp
|
||||
MMLU-Pro vs Q8_0).
|
||||
- GGUF parsing gotcha: expert tensors are
|
||||
`blk.N.ffn_{gate,up,down}_exps.weight` — 4 dot-parts INCLUDING `.weight`;
|
||||
a `len(parts) == 3` filter matches zero layers with no parse error. Filter
|
||||
on `parts[0]=="blk" and parts[3]=="weight" and "exps" in parts[2]`.
|
||||
|
||||
## 11. Adding a second LLM endpoint to a host
|
||||
|
||||
When a host must serve two models at once (an iGPU runtime plus a second
|
||||
card), or one endpoint is moving and a new one takes the freed hardware.
|
||||
|
||||
**0. Decide the shape before touching hardware.** One provider per
|
||||
HOST:PORT in `~/.omp/agent/models.yml` (on the box where the harness runs)
|
||||
— a second endpoint is a new provider block, not a second model in the
|
||||
existing one. Read the budget as an inequality: for unified memory the
|
||||
engine's own fit check is the arbiter (`budget = host − pinned weights −
|
||||
reserve`; `need = KV + arena + vision`); for a discrete card, weight the KV
|
||||
by counting only full-attention layers — and remember `--parallel N`
|
||||
DIVIDES `--ctx-size`. A co-tenant is a budget line, not a footnote: free
|
||||
RAM by changing the POOL or the ARENA (the reserve is a fit-check term,
|
||||
never an allocation) and drop optional charges (a vision tower is ~2 GiB)
|
||||
rather than hoping.
|
||||
|
||||
**1. Repo first, and retire in the same commit.** New service files
|
||||
(`scripts/<svc>.env`, `.service`, `serve-<svc>.sh` with hard guards for
|
||||
missing artifacts); planner entries (`SERVICE_ORDER` + the target's service
|
||||
set + path mappings, inventories in `deploy.sh`/`deploy-remote.sh`, the
|
||||
dispatch case); retire the obsoleted pieces into `.AGENTTRASH/` with a
|
||||
suffix — and check what pointed at them: bench harnesses, a chat template
|
||||
only the old runtime read, and `scripts/test-changed.sh` mappings are the
|
||||
usual victims (a mapping to a retired test file fails the pre-commit
|
||||
wrapper silently for weeks). One bats file per service asserting
|
||||
INVARIANTS (`pool == 2 × ctx`, KV bytes ≤ card budget, port separation),
|
||||
not constants; a negative grep must strip comments first
|
||||
(`grep -vE '^\s*#' file | grep -q …`) or it matches the prose explaining
|
||||
the very thing you removed.
|
||||
|
||||
**2. The port-bind race (rootless podman).** pasta binds the published port
|
||||
at container CREATE; a just-stopped predecessor still holds it. Order the
|
||||
handler: retire the old manager's unit → stop THIS service's own unit
|
||||
(no-op on first cutover) → wait for every socket on the port to clear
|
||||
([[Container Host Reachability]]) → install → start. Without the second
|
||||
step a redeploy deadlocks against its own gate.
|
||||
|
||||
**3. Wire the surfaces, refuse impossible signals.** Point the harness
|
||||
small/fast + image roles at the new endpoint; move the vision role WITH the
|
||||
vision tower (one owner per capability). `shared/homepage/services.json`: a
|
||||
card per endpoint, probed from the monitor's own host. Prometheus: only add
|
||||
a job if the runtime actually exposes Prometheus text (halogen publishes
|
||||
`/health`, `/cache`, `/v1/*` and nothing else — the job was REMOVED with the
|
||||
reason written down rather than left as a permanent red).
|
||||
|
||||
**4. Cutover, then verify by driving it.** `deploy.sh <target> --service
|
||||
<large> --service <small>` (order matters); per endpoint: `/health`,
|
||||
`/v1/models` reports the alias, a chat completion returns content AND
|
||||
`reasoning_content`, a `tools` request returns `tool_calls`, an image
|
||||
request is answered (or refused with a named reason), the runtime's own
|
||||
speculation counters move, and the unit is `enabled`/`generated` with linger
|
||||
on. Read the engine's boot ledger for a silent clamp. Finally spend one
|
||||
real request through the harness per changed role — routing labels lie; the
|
||||
servers' own journals do not.
|
||||
|
||||
Traps that look like model faults: `500 tools param requires --jinja flag`
|
||||
(runtime lacks the Jinja path — pin it); `500 Jinja Exception: Unexpected
|
||||
reasoning effort <level>` (fold via `compat.reasoningEffortMap`); the same
|
||||
request works by hand but a subagent dies (subagents run at a different
|
||||
thinking level — check both); the container dies instantly although the
|
||||
predecessor "stopped" (the port race above).
|
||||
|
||||
## Related
|
||||
|
||||
- [[GGUF Compatibility]] — which runtime can load a payload at all
|
||||
- [[Llama Cpp Serving Sizing]] — pricing the shape once it loads
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — vLLM-side fit arithmetic
|
||||
- [[LLM Bench Windows and Measurement]] — measuring the new deployment
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — the current endpoint
|
||||
@@ -0,0 +1,264 @@
|
||||
---
|
||||
type: reference
|
||||
title: Tcclaviger Fork Ops
|
||||
description: Operating the tcclaviger RDNA4 vLLM fork on dankserver (shape G era, rollback target since the 2026-10-05 FreeToken cutover) — the ad-hoc sweep harness, measured knob verdicts, sweep hygiene, the large-MoE serving recipe, and compile-cache persistence.
|
||||
tags: [tcclaviger, vllm, dankserver, expert-offload, sweeps, cache]
|
||||
timestamp: 2026-10-05T00:00:00-07:00
|
||||
---
|
||||
|
||||
# Tcclaviger Fork Ops
|
||||
|
||||
The tcclaviger RDNA4 vLLM fork (`docker.io/tcclaviger/vllm@sha256:da4029666…`,
|
||||
vLLM 0.29.0.dev, GLM-5.3/Qwen-specialised with native MoE quant kernels and a
|
||||
first-class expert-offload planner) served `frank-primary` at shape G until
|
||||
the 2026-10-05 FreeToken cutover; it remains the rollback target
|
||||
(`git tag frank-primary-vllm-shapeG-20261003`). Full plan law and VRAM
|
||||
arithmetic: [[RDNA4 MoE Expert Offload VRAM Budget]]; boot forensics:
|
||||
[[vLLM Boot Forensics and Config Interrogation]]; window discipline:
|
||||
[[LLM Bench Windows and Measurement]].
|
||||
|
||||
## The ad-hoc sweep harness (dankserver)
|
||||
|
||||
Serve script `/home/dank/serve_qwen_fp8.sh` with env hooks: `SPEC` (MTP
|
||||
depth), `TUNABLE` (TunableOp), `PREFETCH`, `UTIL`, `MAXLEN`, `KV_BYTES`,
|
||||
`MNBT`, `UVA_TOK`, `GATHER_C/GATHER_L` (R4D_LRU gather), `TELEM`
|
||||
(R4D_LRU_TELEMETRY), `PROFDIR` (torch profiler bind-mount + --profiler-config
|
||||
JSON). The script re-copies the harness (`mtp_measure.py`, `bench_iter.py`)
|
||||
into every fresh container — `podman cp` files die on container recreate.
|
||||
|
||||
- Measure ONLY via usage counters: `mtp_measure.py` (3 depths) or
|
||||
`mtp_measure_short.py` (166 ctx). `decode = completion_tokens/(wall −
|
||||
ttft)`; chunk counting is INVALID under spec decode.
|
||||
- Jobs: `bash /home/dank/job.sh start NAME CMD…` / `wait NAME SECS 40` /
|
||||
`out NAME` ([[Remote Job Runner]]).
|
||||
- **Validity check on ANY speed result:** `completion_tokens == max_tokens`
|
||||
(else repetition-halt or early EOS), finish_reason, and inspect the text —
|
||||
`bench_iter` prompts are repeated word-lists + `!` padding, loop-prone. A
|
||||
loop makes LRU hits ~100% → PCIe traffic vanishes → fake 3–4× decode.
|
||||
Acceptance swings 48–61% run-to-run — noise; decode is the metric.
|
||||
- Each arm = full reload: ~8.5 min cold, ~12 min with measurement.
|
||||
- Readiness gate: `podman logs <c> 2>&1` (stdout+stderr merged) for
|
||||
`Application startup complete` — it is on STDERR; an unmerged grep can
|
||||
never match and burns the whole wait timeout.
|
||||
- GPU util ~0% during load/compile is NORMAL (weights + CUDA-graph capture
|
||||
are CPU-side); VRAM climbing is the load indicator.
|
||||
- Config edit discipline: the serve script is patched by scp'd python
|
||||
scripts with asserts (anchor must exist, addition must not); `bash -n`
|
||||
after every patch.
|
||||
|
||||
## Measured knob verdicts (2026-09-29 campaign)
|
||||
|
||||
Incumbent: `SPEC=4 PREFETCH=1 UTIL=0.98 MAXLEN=32768 KV_BYTES=2147483648
|
||||
TUNABLE=0`, stock gather 8×16 → 26.80/25.15/26.81 tok/s @ 166/1087/4064 ctx.
|
||||
|
||||
| Knob | Verdict | Evidence |
|
||||
|---|---|---|
|
||||
| Gather geometry `R4D_LRU_GATHER_{CHUNKS,LANES}` | **RULED OUT** | 16×32 worse (~24); 32×64 tie (26.6). The 9.5 vs 22.7 GB/s read-path gap is per-transfer structure (~105 µs/slab vs 78 ideal), not a tunable knob |
|
||||
| MTP depth | **RULED OUT** | SPEC=3 ~9% worse; SPEC=6 collapses (13.5–19 tok/s, acceptance 17.4%) — also proves UVA read-through (crosses at step tokens > `R4D_LRU_UVA_TOKENS=5`) is SLOWER than copy-in |
|
||||
| TunableOp | **REJECTED** | `CLAV_TUNABLEOP_SWEEP=1` silently SKIPPED under expert offload unless `CLAV_TUNABLEOP_WITH_OFFLOAD=1`; adds ~7 min/load, persists nothing (container-local), and CORRUPTS NUMERICS (loops, repetition-halt, fake 79.8 tok/s @ 96.5% acceptance on degenerate completions). Any "3× win" with early-EOS + absurd acceptance = the loop artifact |
|
||||
| MNBT | neutral at 8192 vs 2048 | small values re-chunk prefill only (hurts TTFT, never decode — decode step = max-num-seqs × (spec+1) tokens). No ubatch concept in vLLM |
|
||||
|
||||
Decode on this stack is PCIe-gather-bound (~30% GPU duty cycle), so
|
||||
batch-shaped compute knobs move ~nothing.
|
||||
|
||||
The `R4D_*/CLAV_*` knob family (QSA_*, MHC, ROUTE, ACT_QUANT, AR_QUANT,
|
||||
SILU_QUANT) are kill-switches defaulting ON, not perf dials; real levers are
|
||||
the R4D_LRU_* family, TunableOp (rejected), R4D_LRU_TELEMETRY, and spec
|
||||
depth.
|
||||
|
||||
Telemetry gotchas: `--profiler-config` takes a JSON blob, not separate
|
||||
flags; `torch_profiler_with_stack` defaults TRUE — a full measure inside a
|
||||
profile window OOM-kills the engine worker (~40 GB trace buffer; kernel
|
||||
kills python3 at ~95 GB anon-rss). Keep windows to ~15 s / one depth
|
||||
(`mtp_measure_short.py`). `telemetry_report` (`expert_cache_rank<N>.json`)
|
||||
fires ONLY from the worker's profiler-stop path; worker-side stop can hang
|
||||
>5 min on ROCm/kineto. No alternative trigger exists. The LRU hit rate
|
||||
measured 82.45% — NOT the ~27% the arena-fraction arithmetic implies;
|
||||
temporal reuse dominates, so never infer miss rates from resident-slot
|
||||
fractions.
|
||||
|
||||
## Sweep hygiene (fork-specific preflight)
|
||||
|
||||
1. **Knob landed:** for each `-e` env knob, confirm in-container
|
||||
`podman exec … env` (or the engine log's confirmed-values line). A knob
|
||||
the fork silently ignores produces a "parity" arm that is really the
|
||||
default — the v1 gather sweep lost all three arms to exactly this
|
||||
(GATHER_C/GATHER_L silently dropped, all arms ran stock 8×16).
|
||||
2. **Prefetch uniformity:** every arm in a sweep MUST carry the same
|
||||
expert-prefetch setting. The 2026-09-29 campaign measured losing arms
|
||||
with prefetch ON and incumbents OFF; untaxed, its verdicts inverted
|
||||
(no-spec 12.11→41.47; gather 32×64 parity→+18.7%). Never re-read a taxed
|
||||
verdict.
|
||||
3. **Spec arms must prove engagement:** engine log
|
||||
`SpecDecoding metrics: Mean acceptance length …` and
|
||||
`curl :7770/metrics | grep vllm:spec_decode_`. Zero such lines = MTP off.
|
||||
4. Boot-fail fast-detection: poll `podman ps` INSIDE the health loop; a
|
||||
refused boot dies in one poll instead of 20 min of health waits. Capture
|
||||
>8 lines of `podman logs` on failure — the root cause (e.g. a SEQS init
|
||||
refusal) lives higher up.
|
||||
5. Report per-depth tables, never one number: g32×64's win grows with depth
|
||||
(54@1k, 39.5@short) — a single 4064 median hides the short-ctx cost. TTFT
|
||||
rows are noisy ±0.15 s and non-monotonic across depths; compare same-ctx
|
||||
rows across arms only.
|
||||
6. Generate ≥256 tokens per sample (at 8 tokens inter_token is first-token
|
||||
noise).
|
||||
|
||||
Boot-log facts survive measure failures: KV pool size (`GPU KV cache size`),
|
||||
arena slots (`expert cache: … -> N slots`), and the spec-reservation ladder
|
||||
(MTP3 766,666 / MTP4 729,545 / MTP6 665,263 tokens) print at BOOT — a
|
||||
measure-stage bug costs decode tables but not boot-log findings.
|
||||
|
||||
**MTP depth-brittleness law (2026-10-02, all arms rc=0):** MTP3 on this fork
|
||||
is depth-brittle — 47.01 t/s at 166 tokens (47.88 at temp 0.7) but 23.09 at
|
||||
4064, vs no-spec baseline 37.02 / 33.99. MTP3 + SEQS 16 + MNBT 4096 + util
|
||||
0.95 + ladder [4..32] = best overall 37.04 / 38.61 t0.7 and keeps 33.15 at
|
||||
4064. `--expert-precise-memory` is load-bearing for MTP (same arm with it
|
||||
OFF = worst arm, 19.81 at 4064). Every spec arm costs KV (398,658 → 313,002
|
||||
tokens). Acceptance is real: mean length 2.4–2.8, per-position
|
||||
0.76/0.55/0.33, avg 52–57%. Next lever named by the engine: MNBT 4096 is
|
||||
exactly SEQS×(n+1) (`suboptimal` warning) → try 8192.
|
||||
|
||||
## Serving a model larger than VRAM (dankserver recipe)
|
||||
|
||||
Dankserver: 2× R9700 (gfx1201), 31.86 GiB each = 63.7 GiB real VRAM, 246 GiB
|
||||
RAM. `torch.cuda.device_count()` returns 3 and summing `total_memory` gives
|
||||
a bogus 186.7 GiB (the Raphael iGPU is in the list — see
|
||||
[[Llama Cpp Serving Sizing]] for the reading discipline).
|
||||
|
||||
**Kernel gate — this decides which checkpoint can work at all.**
|
||||
`/app/r4dhip/r4d.so` exports `gemm_moe_mxfp4a8_*` and `gemm_moe_dsfp4a8_*`
|
||||
but NO `gemm_moe_w4a16_*`. Expert offload requires an experts class with
|
||||
BOTH a compiled kernel AND an `offload_recipe`; the planner aborts startup
|
||||
otherwise. Usable: MXFP4, DSFP4, FP8-HIP (`R4dMxfp4MoEExperts`,
|
||||
`R4dDsfp4MoEExperts`, `Fp8HipMoEExperts`). `R4dW4a16MoEExperts` has no
|
||||
kernel (the class exists, the r4d.so entry does not); RFA has neither. Check
|
||||
the checkpoint's quant against this list BEFORE downloading anything.
|
||||
|
||||
Entrypoint is `/app/tools/image_entrypoint.sh`, ending `exec vllm serve
|
||||
"$@"` — pass the model path as the FIRST argument and do NOT pass `serve`.
|
||||
Known-good invocation:
|
||||
|
||||
```bash
|
||||
sudo sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'
|
||||
podman run -d --name <ctr> --ipc=host --ulimit memlock=-1:-1 \
|
||||
--cap-add SYS_PTRACE \
|
||||
--device /dev/kfd --device /dev/dri --device /dev/mem \
|
||||
-e HIP_VISIBLE_DEVICES=0,1 -e VLLM_ROCM_USE_AITER=0 \
|
||||
-v /home/dank/models/<model>:/models:ro \
|
||||
docker.io/tcclaviger/vllm:latest /models \
|
||||
--served-model-name <name> --port 7770 \
|
||||
--tensor-parallel-size 2 \
|
||||
--max-model-len 32768 \
|
||||
--kv-cache-memory-bytes 2147483648 \
|
||||
--max-num-seqs 4 \
|
||||
--gpu-memory-utilization 0.95 \
|
||||
--enable-prefix-caching --trust-remote-code
|
||||
```
|
||||
|
||||
`-p 7770:7770` is not included, so the API is only reachable via
|
||||
`podman exec` — add `-p` if host-side curl is needed. Load ~5 min;
|
||||
`/health` answers connection-refused the whole time; watch for
|
||||
`Application startup complete` in `podman logs <ctr>`. Verified working:
|
||||
`orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8` (gated; 172.8 GiB, 131
|
||||
shards, `Qwen4ExpForConditionalGeneration`, 48 layers, 512 experts top-10,
|
||||
multimodal, native 262,144).
|
||||
|
||||
Traps that each cost a boot:
|
||||
|
||||
1. **Mount the whole `/dev/dri`, never individual render nodes.** Render
|
||||
nodes without the primary `card*` nodes → HIP enumerates two agents and
|
||||
refuses a context on either (`RuntimeError: No CUDA GPUs are available`)
|
||||
while `device_count()` cheerfully returns 2. Confirmed by direct A/B.
|
||||
2. **memlock must be raised** (done on Dankserver via
|
||||
`/etc/security/limits.d/99-memlock.conf`). Host default 8 MiB makes any
|
||||
large pinned allocation fail as a misleading
|
||||
`torch.AcceleratorError: CUDA error: out of memory` from `pin_memory()`;
|
||||
it cannot be raised from inside a container or root's own shell — only
|
||||
PAM applies it at login. Matters because `--expert-host-mlock` defaults
|
||||
True.
|
||||
3. **`--gpu-memory-utilization` does NOT buy KV under expert offload** —
|
||||
the planner sizes the arena against the budget and leaves KV the
|
||||
remainder; raising util 0.95→0.98 grew the arena and the KV pool stayed
|
||||
at 0.64 GiB.
|
||||
4. **Lowering `--max-model-len` is the WRONG lever and backfires** — the
|
||||
budget subtracts the KV reserve first, so a smaller context shrinks the
|
||||
reserve and hands the space to the arena (measured: 65536 → 0.64 GiB KV;
|
||||
32768 → 0.34 GiB).
|
||||
5. **The KV reserve undercounts hybrid-attention models ~2×** (9190 B/token
|
||||
planned vs ~18,000 real: the Mamba state is a fixed per-sequence cost).
|
||||
Fix by pinning the pool: `--kv-cache-memory-bytes`.
|
||||
6. **`--expert-offload-mem` caps the HOST side**, not the VRAM arena —
|
||||
structurally orthogonal to `--expert-precise-memory` (verified
|
||||
in-container 2026-10-05: precise-memory only swaps the GPU overhead term,
|
||||
expert_offload_mem only sets the host bound; they rejoin only inside
|
||||
`place_experts`, and the plan rewrites `expert_offload_mem` back to the
|
||||
cap the plan met, so plan and allocation cannot drift).
|
||||
7. `--ipc=host` and `--shm-size` are mutually exclusive in podman; pick one.
|
||||
8. `podman run -d` without `-a` sends no stdout to a redirect — the log file
|
||||
only receives the container ID; use `podman logs`.
|
||||
|
||||
Status reads: use `scripts/dankstatus.sh`, never hand-rolled
|
||||
`/proc/meminfo` parsing — `/proc/meminfo` separates key from value with a
|
||||
TAB, so `grep MemAvailable | tr -s " " | cut -d" " -f2` returns empty and
|
||||
awk prints `0.0` — a false "host out of memory" that invites an unnecessary
|
||||
abort. Gated checkpoints: token via a mode-600 `curl -K` config, never argv
|
||||
(`pgrep -a curl` prints it); always `--fail`; compare each file's byte count
|
||||
to the Hub's declared size; `-C -` resumes truncated shards.
|
||||
|
||||
**Check whether the stack supports the model natively before porting an
|
||||
overlay.** The image already registered `Glm5NextForConditionalGeneration`,
|
||||
`Qwen4ExpForCausalLM` and both MTP variants — chasing four missing symbols
|
||||
through two image tags to bring an overlay up was entirely wasted work.
|
||||
|
||||
## Compile-cache persistence (survives `podman rm -f`)
|
||||
|
||||
Serve scripts that recreate the container discard the writable layer, so
|
||||
every boot re-pays compile/JIT (frank-primary: `torch.compile took 181.20 s`
|
||||
cold, of which 173 s was one graph).
|
||||
|
||||
1. **Prove it and locate it:** fresh container vs live
|
||||
(`podman run --rm --entrypoint bash IMG -c "ls -la /root/.cache"` vs
|
||||
`podman exec LIVE bash -c "du -sb /root/.cache/*; find /root/.cache -type
|
||||
f | wc -l"`). vLLM cache root: `VLLM_CACHE_ROOT` (default
|
||||
`/root/.cache/vllm`, `torch_compile_cache/`); a HIP runtime adds
|
||||
`/root/.cache/comgr`.
|
||||
2. **Mount the PARENT, not the leaf** — one bind on `/root/.cache` persists
|
||||
both trees, safe whenever the image ships no such directory (step 1
|
||||
proves it; otherwise you would be hiding baked state). Wire as an opt-in
|
||||
tunable: `mkdir -p "$CACHE_DIR"` BEFORE the mount, then
|
||||
`${CACHE_DIR:+-v "$CACHE_DIR":/root/.cache}`. Host path outside the
|
||||
checkout.
|
||||
3. **Verify with two consecutive boots** — cold `Compiling a graph … takes
|
||||
N s`, warm `Directly load the compiled graph(s) for compile range (…)
|
||||
from the cache` with `torch.compile took` in single-digit seconds.
|
||||
4. Traps: the cache key includes the compile RANGE — an MNBT change costs
|
||||
ONE recompile, not a permanent tax; do not claim a proportional boot cut
|
||||
(time to `/health` is dominated by weight loading: 490 s warm vs 188 s
|
||||
cold compile); stale entries are misses, not wrong answers (version +
|
||||
config hashed); if the serve script is seam-gated, assert the mkdir and
|
||||
the exact mount target.
|
||||
|
||||
## Plan-line verification checklist (after any fork deploy)
|
||||
|
||||
```
|
||||
ssh dankserver 'sudo journalctl CONTAINER_NAME=frank-primary-qwen -b -o cat --no-pager \
|
||||
| grep -aoE "(budget \(total x util\) +[0-9.]+|expert cache space +[0-9.]+ GiB +[0-9,]+ slots|Estimated CUDA graph memory.{0,80}|Graph capturing finished.{0,40}|GPU KV cache size.{0,90})"'
|
||||
```
|
||||
|
||||
`grep -oE` with a digit-anchored tail — plain `grep`/journalctl truncate
|
||||
these columns to `17.`. Verify: new budget/graphs/arena in the plan;
|
||||
`GPU KV cache size` matches the requested token count; `Estimated CUDA graph
|
||||
memory` vs the capture line (raise `_PER_GRAPH_MB` toward 52 if capture
|
||||
exceeds the charge); `rocm-smi --showmeminfo vram` high-water per card = the
|
||||
real margin; `systemctl show frank-primary -p NRestarts --value` = 0;
|
||||
`VLLM_CUDAGRAPH_*` present in `podman exec … env`; then a real chat
|
||||
completion from a PEER — `curl localhost:7770/health` inside an ssh session
|
||||
on the host can return `000` and looks like an outage that isn't.
|
||||
|
||||
## Related
|
||||
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law this harness tunes
|
||||
- [[vLLM Boot Forensics and Config Interrogation]] — journal forensics for these boots
|
||||
- [[LLM Bench Windows and Measurement]] — window discipline and bash laws
|
||||
- [[Model Swap and Rollback]] — where the fork sits in the swap history
|
||||
- [[Remote Job Runner]] — the job.sh harness referenced above
|
||||
@@ -123,3 +123,27 @@ Stop and disable the root unit, trash it, restore the halogen quadlet from
|
||||
`~/.AGENTTRASH/20260915-ik-swap/halogen-quadlet` to
|
||||
`~/.config/containers/systemd/`, `systemctl --user daemon-reload`, start;
|
||||
the .hgn checkpoint and the kingjones STRIX llama.cpp payload remain on disk.
|
||||
|
||||
## Hybrid campaign envelope (2026-09-16, 3090 + Strix Halo)
|
||||
|
||||
Why this deployment could not be tuned into a production endpoint: decode
|
||||
3.8–4.3 tok/s flat across thread counts (T1 ≈ T16); prefill ~192 tok/s at
|
||||
1.8k–32k prompts = the **CPU AVX512-VNNI compute ceiling** for CPU experts.
|
||||
The decode root cause is structural: ~100 CPU↔GPU backend splits per token
|
||||
with the GPU at 5 % SM during decode — launch/sync-bound over the PCIe x4
|
||||
dock, not compute- or memory-bound. T16 profiling shows ~90 % of cycles in
|
||||
`pause` (ggml_barrier spin): threads wait on the serial critical path.
|
||||
Memory placement (mmap/anon/THP/deferred) does not matter — all variants
|
||||
landed 3.8–4.6.
|
||||
|
||||
- CUDA-graph capture aborts: qwen4exp's hyper-connection broadcasts use
|
||||
per-layer `ggml_repeat_4d`, whose CUDA path is capture-illegal; until ik
|
||||
gains capture-safe repeat/hc ops (mainline PR #28901 direction), graphs
|
||||
stay dead.
|
||||
- Mainline llama.cpp itself runs 1.0 tok/s on this model (CPU MoE kernels
|
||||
4× slower than ik's iqk) — do not switch engines for speed.
|
||||
- MTP/speculative decoding is impossible for mradermacher i1 GGUFs: the
|
||||
NextN head is stripped (no `blk.48` tensors).
|
||||
- Tuning lever ranked first: `-ncmoe 42` (6 layers' experts on the 3090) →
|
||||
decode 4.54, small-prompt prefill 32 — but VRAM 22/24.6 GiB at 262k ctx,
|
||||
an operator OOM-margin decision, and it needs `GGML_CUDA_NO_PINNED_WEIGHTS=1`.
|
||||
|
||||
@@ -134,6 +134,59 @@ slots of 131072, `6.44 / 6.90 GiB` free — matching the pre-deploy measurement
|
||||
0.01 GiB — a tool call and two concurrent requests answered, draft acceptance on
|
||||
both slots (0.63 / 0.48 on slot 1, 0.39 on slot 0), and no ROCm errors.
|
||||
|
||||
## The split: re-measure it at the deployed shape
|
||||
|
||||
The batch and the tensor split are **coupled**: compute buffers are not
|
||||
symmetric across the two cards, so any batch change moves the balance point.
|
||||
Order of work — sweep batch with llama-bench (above), THEN re-measure the
|
||||
split at the DEPLOYED shape (262144, two slots, drafter loaded) by loading
|
||||
the real service with a per-config env override and reading `rocm-smi`. A
|
||||
llama-bench number cannot decide the split: it has no `--parallel`, sizes
|
||||
its own context as `n_prompt + n_gen + n_depth`, and cannot load the drafter.
|
||||
|
||||
The settle gate is mandatory (above); wait for BOTH cards under ~1 GiB AND
|
||||
the port to be free (`ss -ltn | grep :7770` until gone) before the next
|
||||
load, or the transient server dies with `couldn't bind HTTP server socket`.
|
||||
Free bytes follow `card = weights_share + KV_share + constant + ubatch_term`,
|
||||
where the ubatch term costs the drafter-carrying card roughly **1.85×** the
|
||||
other (measured 2.41 GiB vs 1.30 GiB going 1536 → 512). Balance by
|
||||
measurement only: one `-ts` step is one layer, worth ~0.43 GiB on a DeltaNet
|
||||
boundary and ~1.43 GiB on an attention one, so the reachable minimum gap is
|
||||
about one DeltaNet layer, never zero. Reference: 36,29 @1536 → 4.71/4.91;
|
||||
36,29 @512 → 6.01/7.32; **35,30 @512 → 6.44/6.90 (deployed)**.
|
||||
|
||||
Re-fit both ubatch coefficients in `tests/frank_medium_unit.bats` at the new
|
||||
deployed point and re-anchor the per-card constants; check neighbouring
|
||||
splits still FAIL the balance gate or the seam has stopped discriminating
|
||||
(keep the 1.5 GiB/card floor and the one-DeltaNet-layer slack). Deploy,
|
||||
then verify live: flags in `/proc/<pid>/cmdline`, `/slots`, VRAM matching
|
||||
the pre-deploy measurement, a tool call, two concurrent requests,
|
||||
`draft acceptance` lines on BOTH slots, 0 ROCm errors.
|
||||
|
||||
**Per-card utilization (who is hot, and the split lever)** — a different
|
||||
question from batch/prefill. Measure decode tok/s AND per-card
|
||||
`rocm-smi --showuse` at 2–3 neighbouring splits in ONE session. Pitfalls
|
||||
(each cost a re-run): run the transient unit as ROOT (a `pkill -9
|
||||
llama-server` as dank auto-restarts the service and `systemctl` as dank
|
||||
fails "Access denied"); warm up 3 throwaway requests before the timed one
|
||||
(first 384-token decode after load measured 1.77 tok/s against a true ~32);
|
||||
`rocm-smi --showmeminfo vram --csv` columns are `$2`=Total, `$3`=**Used** —
|
||||
parsing the wrong column spins the settle gate the full 600 s; the util% is
|
||||
sampled post-hoc (which card finished last), so treat it as "which card is
|
||||
the pipeline tail", not a duty cycle.
|
||||
|
||||
**What the split does and does not buy (measured 2026-09-17, 393216 /
|
||||
MTP n=3):** the split is NOT a decode-throughput lever — 34:31, 35:30, 36:29
|
||||
all sit in the same ~32 tok/s band. The x1 card runs hot (~91–97 %) at
|
||||
35:30 because the MTP drafter + output layers + heavier compute buffers are
|
||||
pinned to the card holding the LAST layers — not because PCIe is the
|
||||
bottleneck (~5.2 MB of boundary activation crosses per 512-token ubatch).
|
||||
36:29 makes the x16 card the utilized worker (42 %/3 %) at no throughput
|
||||
cost but spends headroom (0.45 GiB free, under the floor) — Frank's standing
|
||||
preference is that the card with the FASTEST PCIe lane should be the more
|
||||
utilized one; deploying it requires re-fitting the seam's slack/floor and
|
||||
re-checking that 35:30 and 37:28 still fail so the gate keeps discriminating.
|
||||
|
||||
## Limitations
|
||||
|
||||
- **`llama-bench` cannot run the drafter, the slots or the deployed context.**
|
||||
|
||||
@@ -45,6 +45,98 @@ timestamp: 2026-09-30
|
||||
> because the fork compiles its own Qwen trunk and not its GLM one. See
|
||||
> [The measured profile](#the-measured-profile-and-the-real-wall).
|
||||
|
||||
## DFlash2 target-side port — the four laws and the KV ladder
|
||||
|
||||
Distilled 2026-10-05 from the probe windows (skills `glm53-dflash-boot-ladder`
|
||||
/ `glm53-dflash-eagle3-port`); each law was proven by a real failed boot.
|
||||
|
||||
**The four EAGLE3-port laws** (make the GLM trunk satisfy `SupportsEagle3`):
|
||||
|
||||
1. **Patch the instance the runner calls.** `model_runner.py:425` calls
|
||||
`set_eagle3_aux_hidden_state_layers(self.model)` on the *loaded* model —
|
||||
GLM loads as `Glm5NextForConditionalGeneration` (multimodal wrapper;
|
||||
`self.language_model = Glm5NextForCausalLM`). Members placed only on the
|
||||
CausalLM fail the check on the conditional — same error text, wrong file.
|
||||
Patch BOTH wrappers.
|
||||
2. **Presence-checked member set.** `SupportsEagle3` (interfaces.py:1659) is
|
||||
`@runtime_checkable`; `isinstance` walks ALL `__protocol_attrs__`
|
||||
(including base-protocol members) with `getattr_static`, breaking on the
|
||||
first miss. Both wrappers need, as class attributes:
|
||||
`supports_eagle3: ClassVar[Literal[True]] = True`,
|
||||
`has_own_lm_head: bool = False`, `has_own_embed_tokens: bool = False`.
|
||||
Verify in-image with stock typing — no boot needed.
|
||||
3. **MRO forbids inheriting the Protocol.** GLM wrappers carry four/five
|
||||
Protocol-derived bases; adding `SupportsEagle3` anywhere dies with
|
||||
"Cannot create a consistent method resolution order" (sim-proven: the
|
||||
ordering rejects it in every base position). The attribute route avoids
|
||||
the MRO, so the two methods must be written by hand: the CausalLM
|
||||
unwraps to `self.model` (the `EagleModelMixin` trunk), the conditional
|
||||
delegates to `self.language_model` (pixtral.py:498 pattern);
|
||||
`get_eagle3_default_aux_hidden_state_layers` returns `(2, L//2, L-3)`,
|
||||
with the drafter config's `target_layer_ids` winning when present.
|
||||
4. **Trunk capture point:** append `aux_hidden_states` BEFORE the layer call
|
||||
— GLM's loop carries the single-stream state entering the layer.
|
||||
|
||||
**The KV blocker ladder** (patches `scripts/patches/glm53-dflash-0*`):
|
||||
|
||||
1. EAGLE3 gate → presence-checked attrs on both wrappers (above).
|
||||
2. KV page-size unifier: MLA layers cannot be padded (sparse MLA reads
|
||||
whole token rows) but CAN grow block_size when the max page is an exact
|
||||
multiple — bump `max_page_size` in `unify_kv_cache_spec_page_size` to
|
||||
the `lcm` of all MLA-family pages first (patch 02).
|
||||
3. kpool tail assert `tail_kv_cache.shape[2] == pool_size`: the unifier's
|
||||
divisible branch grew KpoolTailSpec's block; its kernel pins block ==
|
||||
index_kpool. Add a KpoolTailSpec branch that pads `page_size_padded`
|
||||
instead of growing (patch 03).
|
||||
4. KV capacity: the glm5 layout ALIASES mamba states INTO MLA pages (why
|
||||
GLM fits 32 GiB cards); the generic path (taken when draft/hidden specs
|
||||
exist) cannot alias → ~30 GiB demanded vs a 0.92 GiB reserve. Extend
|
||||
`_get_kv_cache_groups_glm5_next` to partition
|
||||
HiddenStateCacheSpec + drafter attention specs; the drafter forms its
|
||||
own uniform group (per-group pages, NO unification → the MLA block
|
||||
stays at the engine-auto-set 2,432 from interface.py:918, kpool guard
|
||||
`block_size % 512 == 0` unstressed); layout returns a 10-tuple
|
||||
(+draft_group, +hidden_groups); the builder sizes ONE shared backing
|
||||
allocation (`allocate_kv_cache` asserts all KVCacheTensor sizes equal) =
|
||||
aliased pool + appended draft/hidden tensors (patch 04). After 04 the
|
||||
capacity requirement collapsed 31.76 → 1.88 GiB at 65,536 max len and
|
||||
the engine SERVED (READY ~500 s, dflash2 fused prep ON, drafter FULL
|
||||
graphs on both TP ranks).
|
||||
5. Reserve registry: `_glm53_kv_bytes_per_token` must add 4,096 (drafter
|
||||
KV) + 5×hidden×2 (aux hidden cache) keyed on
|
||||
`speculative_config.method in {dflash, dspark, extract_hidden_states}`.
|
||||
NEVER key on `use_aux_hidden_state_outputs` — it lives on the
|
||||
gpu_model_runner instance (gpu_model_runner.py:623/678), unset at plan
|
||||
time, so the check silently fails and the reserve stays 13,600
|
||||
(patch 04b). Proven reserve math at 65,536: 13,600 + 4,096 +
|
||||
5×4096×2 ≈ 58,656 B/token → ~4.15 GiB reserve, arena ~10.3 GiB.
|
||||
|
||||
**Verify cheaply before booting:** smoke-test the pure KV logic on CPU
|
||||
torch with the image's real spec classes — stub `vllm.logger`/`vllm.utils.*`,
|
||||
register the spec classes in `KVCacheSpecRegistry` with a dummy manager, add
|
||||
`from __future__ import annotations`, and run synthetic GLM+draft+hidden
|
||||
spec sets through the patched grouping/layout/builder functions. A 7-minute
|
||||
boot per hypothesis is the alternative.
|
||||
|
||||
**Window discipline:** stop + mask `frank-primary` FIRST and gate on
|
||||
MemAvailable ≥ 230 GiB before booting the probe (the expert-offload planner
|
||||
reads MemAvailable; beside-prod boots die at plan time with `ExpertPlanError:
|
||||
host holds 5113 of 12096 swappable experts`). Save `podman logs glm-probe`
|
||||
to a file BEFORE teardown — a `podman logs | grep | head` pipeline died once
|
||||
under pipefail via SIGPIPE before log-save, losing the capture. The abort
|
||||
path must restore prod (unmask + daemon-reload + reset-failed + start),
|
||||
never bare-exit after stop+mask. Launcher knobs: `DFLASH_PATCH=1` (mounts
|
||||
the patched model.py), `DRAFT_MOUNT=1`, `SPECJSON_FILE=…` ({"method":
|
||||
"dflash","model":"/draft","num_speculative_tokens":8}) — note the launcher's
|
||||
echo prints `patch=$PATCH_MOUNT` (the COMPILE knob), not DFLASH_PATCH.
|
||||
Boot timeline: EAGLE3 gate fires ~4 min in at worker init; drafter
|
||||
torch.compile is the frontier (~380 s).
|
||||
|
||||
**Open telemetry discrepancy:** the contaminated window read 1.5–2.1 tok/s
|
||||
with acceptance 1.00–1.25; Frank live-measured ~18 tok/s decode / ~370 tok/s
|
||||
prefill. Reconcile with a clean single-request measurement before trusting
|
||||
either.
|
||||
|
||||
Related: [[dankserver-qwen38-fp8-expert-offload-tuning]] (the same fork, the same
|
||||
hardware — its conclusion was drawn from throughput arithmetic and is superseded
|
||||
by this page's kernel-time measurement),
|
||||
|
||||
@@ -110,6 +110,32 @@ retires the lucebox user quadlet to dated home trash
|
||||
truly free, relinks + enables + restarts the root unit, and waits up to
|
||||
1800 s for `/health` (cold start page-ins ~175 GiB and builds the graphs).
|
||||
|
||||
### Preflight the GGUF without downloading it
|
||||
|
||||
GGUF metadata lives in the header and HF honours ranged GETs, so a few MB
|
||||
answers every sizing question before a 13 GB transfer
|
||||
(`Range: bytes=0-8000000`). Parse: magic `GGUF`, u32 version, u64
|
||||
n_tensors, u64 n_kv, then key/value pairs (string = u64 len + bytes; ARRAY
|
||||
= u32 elem type + u64 count; BOOL is 1 byte — the type hand-written parsers
|
||||
forget). Walk arrays by byte size even for a summary: a naive parser
|
||||
desynchronises on `tokenizer.ggml.tokens` and lands gigabytes off. Read:
|
||||
`general.architecture`, `<arch>.block_count`, `<arch>.context_length`,
|
||||
`<arch>.nextn_predict_layers` (MTP is a property of the FILE — the `-mtp`
|
||||
build carries `blk.<n>.nextn.*`; `--spec-type draft-mtp` then loads the
|
||||
head against the MAIN model's weights, and `--model-draft` would load the
|
||||
file twice), `<arch>.head_count_kv`, `<arch>.key_length`/`value_length`,
|
||||
`<arch>.full_attention_interval`, `<arch>.ssm.state_size`/`group_count`/
|
||||
`time_step_rank`, and for the projector `clip.projector_type` +
|
||||
`clip.has_vision_encoder`.
|
||||
|
||||
KV arithmetic for the hybrid (16 × (3 Gated DeltaNet + 1 Gated Attention)):
|
||||
only full-attention layers take KV
|
||||
(`attn_layers × n_head_kv × head_dim × 2 × bytes_per_elem`; f16 = 2 B,
|
||||
q8_0 = 1.0625 B, q4_0 = 0.5625 B); DeltaNet state is a fixed per-slot
|
||||
buffer (~0.16 GiB/sequence at 48 layers/128 d_state/48 v-heads f32); the
|
||||
MTP draft context costs ≈ 4 KiB/token. `--parallel N` DIVIDES `--ctx-size`
|
||||
between slots — it does not multiply.
|
||||
|
||||
## Verification (2026-09-26 deploy)
|
||||
|
||||
- `/v1/models` reports `frank-primary`.
|
||||
|
||||
@@ -146,3 +146,70 @@ systemctl --user start frank-medium`) and stop/mask frank-primary. The
|
||||
_Filled in after the cutover: health, real completion, tool call, Open WebUI
|
||||
completion, measured prefill/decode, `systemctl --user is-enabled/is-active`
|
||||
state._
|
||||
|
||||
## Dual-R9700 follow-up (2026-09-28 probe era)
|
||||
|
||||
After this single-card runbook was retired, the DS41 engine got a dual-R9700
|
||||
attempt on the `feat/ds41-engine` branch (`main` rejects the checkpoint with
|
||||
`unsupported model architecture 'deepseek41'`). Findings that outlive the
|
||||
probe:
|
||||
|
||||
**Budget both cards explicitly, or you waste half the box.** The engine
|
||||
sizes the streamed-expert cache LAST, after the model, so uncapped it absorbs
|
||||
every spare byte on the stream device — symptom: card1 at 33.0/34.2 GB
|
||||
looking busy while card0 sits at 14.2/34.2 GB; the busy card is holding a
|
||||
cache of things still streaming from NVMe. Pin all four:
|
||||
|
||||
| env | card | meaning |
|
||||
|---|---|---|
|
||||
| `LUCE_EXPERT_BUDGET_MB` | hip:0 | primary hot experts (~11.96 MiB each) |
|
||||
| `LUCE_EXPERT_SECONDARY_BUDGET_MB` | hip:1 | secondary experts (needs `LUCE_DS4_MOE_TP*`) |
|
||||
| `LUCE_EXPERT_STREAM_CACHE_MB` | hip:1 | cap the auto-sized stream cache |
|
||||
| `LUCE_EXPERT_STREAM_DEVICE=1` | hip:1 | which card holds the slots and computes them |
|
||||
|
||||
Working full-card shape: 21000 + 20000 + 8000 → card0 ~28.6/31.86 GiB,
|
||||
card1 ~27.3/31.86 GiB, resident experts 3,360 of 15,360.
|
||||
|
||||
**Expert TP DOES work on gfx1201+gfx1201.** A retired quadlet recorded
|
||||
`LUCE_DS4_MOE_TP_INPROC` as "qualified for gfx1151+gfx1201, not
|
||||
gfx1201+gfx1201" — that is WRONG; peer access negotiates and the secondary
|
||||
tier comes up clean: `[deepseek4-moe-tp] enabled mode=in-process local=hip:0
|
||||
secondary=hip:1`. It is the only lever that materially raises resident
|
||||
expert count, and resident count is the entire bottleneck. Confirm every run
|
||||
with `expert ownership:` and `hybrid experts ready:` lines.
|
||||
|
||||
**The placement file must live under the mounted models dir.** JSON
|
||||
`{"owners": [...]}`, 15,360 entries indexed `layer*384+expert`, values
|
||||
`primary`/`secondary`/`stream`. It is a ranking, not a promise: each device
|
||||
keeps what fits its budget and demotes the rest. Anywhere outside
|
||||
`/models/...` gives `cannot open expert placement` and a SILENT single-tier
|
||||
fallback — a silent-wrongness failure, not a crash.
|
||||
|
||||
**V4.1 refuses layer split and layer-major prefill** — only `--ds4-prefill
|
||||
exact` (token-by-token) is legal on a 32 GiB card and `--ds4-fused-decode`
|
||||
is refused there. With ~22% of experts resident each token pulls ~187 cold
|
||||
experts × ~12 MiB; if `/proc/<pid>/io` `read_bytes` climbs into terabytes
|
||||
the model is larger than the box's RAM and you are in page-cache thrash at
|
||||
the NVMe ceiling — a capacity verdict, not a tuning problem. Low GPU util
|
||||
with a pegged link is the EXPECTED signature of PCIe-expert-stream-bound
|
||||
decode, not a broken run.
|
||||
|
||||
**Harness traps, each of which produced a wrong number:** never `env -i` the
|
||||
podman call (rootless podman runs as `dank`; `env -i HOME=/root` makes
|
||||
PODMAN try `/root/.config` and die `stat /root/.config: permission denied`
|
||||
before the container exists — pass container env with `-e`); never put a
|
||||
large prompt on curl's argv (`Argument list too long` at ~128 KB — use
|
||||
`--data-binary @file`); size prompts by growing the file until `wc -c`
|
||||
reaches the byte target AND assert depth against the engine's own
|
||||
`usage.prompt_tokens` (nominal 4 chars/token was really 5.73; a chunk loop
|
||||
that appends a fixed ~152-byte line per nominal 200-byte chunk undershot
|
||||
30%); the server runs as the container's PID 1, so tear down with
|
||||
`podman rm -f`, never host `pkill`; drop page cache between arms
|
||||
(`sync; echo 3 > /proc/sys/vm/drop_caches`) or the first arm reads warm and
|
||||
every later one is a lie.
|
||||
|
||||
Read the placement log — it is the whole diagnosis
|
||||
(`arch=deepseek41 layers=40 embd=5120 experts=384 used=6 ff=2304
|
||||
hash_layers=0` — hash_layers=0 is correct for V4.1; `engram: layers=[1,14]
|
||||
… 2 table(s) embedded (not mapped)` — the 189.13 GiB Engram tables are never
|
||||
mapped, so they do not evict the expert page cache).
|
||||
|
||||
@@ -190,6 +190,70 @@ repo env sets `R9V_ENABLE_PREFIX_CACHING=1` (commit `41e9189`), verified
|
||||
on :7770 (`--enable-prefix-caching` in the container argv, health 200,
|
||||
live completion answering).
|
||||
|
||||
## Authoring a new profile (the qualification pipeline)
|
||||
|
||||
Order matters at every step:
|
||||
|
||||
1. **Package manifest** `packages/models/<pkg>/package.json` — target shards
|
||||
+ aux artifacts with sha256. Verify every target's byte count AND sha256
|
||||
after fetch (`curl -fL -C -` per shard; never trust a fetch without `-f`).
|
||||
2. **Placement bootstrap** from GGUF headers: parse all shards' tensor
|
||||
tables (a KV-only shard1 is normal for split parts; each part carries its
|
||||
own tensor table). Per-expert full per layer =
|
||||
(`ffn_gate_exps` + `ffn_up_exps` + `ffn_down_exps`)/512; rank slices
|
||||
rank0 = full×416//640, rank1 = full−rank0. Hot counts =
|
||||
floor(measured_budget / per-expert total). **Anchor budgets to the
|
||||
MEASURED hot_bytes of the qualified sibling profile, never invented.**
|
||||
3. **Trim + validate**: `tools/trim_experts.py --hot-counts N,M`, then
|
||||
`expert_budget.validate_manifest(m)` +
|
||||
`expert_budget.expert_memory(m, slots, {cache_rank}, runtime=<runtime.json>)`
|
||||
(runtime/catalog are keyword-only). Declared hot_bytes must match
|
||||
computed or it raises.
|
||||
4. **PLE derive** (containerized `tools/prepare_ple.py` with
|
||||
`--expected-bytes/--expected-type/--expected-shape`); setup skips
|
||||
re-derivation when the file exists with matching size.
|
||||
5. **profile.env**: set values BEFORE `source dual-r9700/profile.env`
|
||||
(set-first-wins; an empty string does NOT survive `:=`).
|
||||
|
||||
The canonical bootstrap.json wrapper schema is exactly
|
||||
`{schema: "r9v.placement.v1", id, model_package, hardware, artifact_path,
|
||||
artifact_sha256, scope}` with `hardware: "dual-r9700-32gb-asymmetric"`; the
|
||||
wrapper `id` MUST equal the id in the profile.json placement descriptor
|
||||
(setup rejects mismatches: "does not match profile placement"). Never
|
||||
fabricate qualification metadata — use
|
||||
`scope: "Unranked routing bootstrap only; does not certify fit or headroom"`.
|
||||
|
||||
GGUF parsing laws for the bootstrap: expert tensor names are
|
||||
`blk.0.ffn_up_exps.weight` — split on "." gives FOUR parts, layer number is
|
||||
parts[1] (a 3-part assumption crashes `int("")`); outer-KV value-type sizes
|
||||
`{0:1,1:1,2:2,3:2,4:4,5:4,6:4,7:1,8:str,9:arr,10:8,11:8,12:8}`; tensor byte
|
||||
extents from consecutive OFFSETS, never a type-size table (mis-buckets tails
|
||||
silently); the PLE Q8_0 tensor on this arch is shape (160, 320001536),
|
||||
54,400,261,120 B, an entire shard.
|
||||
|
||||
Setup/start: `./r9v setup <alias> --state-dir <dir> --model-dir <dir> --image
|
||||
sha256:<digest> --local-image --reuse-from <sibling-model-dir>
|
||||
--accept-model-license` (aux artifacts reuse byte-identical files; F16 vs
|
||||
Q8 mmproj will not reuse). Start requires the SAME `--state-dir`; fixed
|
||||
placement via `R9V_EXPERT_MANIFEST_PATH` skips headroom/catalog/seed raises.
|
||||
The host-only doctor FAILs when another instance holds the cards (it wants
|
||||
3 GiB free per rank; production at ~30 GiB/card is a hard wall) — a second
|
||||
instance CANNOT coexist, so qualification requires a stop+mask window for
|
||||
the incumbent, then unmask+restart.
|
||||
|
||||
**Qualification gate (unqualified → qualified):** first start runs the
|
||||
workload battery (PONG / tool_calls / vision / long-ctx), the PLE pinning
|
||||
proof (`VmLck` ≈ PLE bytes in `/proc/<container-pid>/status` when
|
||||
RESIDENCY_MODE=pinned), and a VRAM headroom snapshot (floor 3.3 GiB). If
|
||||
headroom is violated, lower hot counts via re-trim and re-derive
|
||||
MAX_EFFECTIVE. Only then record state `ready: true` and author the runbook.
|
||||
|
||||
Remote-execution gotchas from the same windows: `nohup X &` inside an ssh
|
||||
command races ssh teardown — the script can die before writing a byte; run
|
||||
long remote scripts synchronously, or setsid + disown, or via a system
|
||||
unit. zsh login shells abort on unquoted `===` separators; grep globs need
|
||||
quoting through ssh+zsh layers.
|
||||
|
||||
## Ops
|
||||
|
||||
- **Restart**: `sudo systemctl restart frank-primary` (root unit; the serve
|
||||
|
||||
@@ -407,6 +407,42 @@ frank-small, port 7771]
|
||||
- [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations
|
||||
- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo)
|
||||
|
||||
## Cross-Cutting References (skill distillation, 2026-10-05)
|
||||
|
||||
One polished reference page per skill-cluster, distilled from the managed
|
||||
skill library into the vault (see [log](../log.md) for the wave plan):
|
||||
|
||||
- [[GGUF Compatibility]] — what a GGUF's tensor histogram and header demand
|
||||
of a loader; how the runtime is chosen by the file
|
||||
- [[Llama Cpp Serving Sizing]] — slot/pool/KV sizing laws, boot-shape
|
||||
laddering, llama-bench discipline
|
||||
- [[Llama Cpp Multi-Device Splits]] — layer vs tensor splits, `-ts`/`-ot`
|
||||
laws, drafter placement
|
||||
- [[PCIe Link and Atomics]] — the link walls, atomics routing, and the
|
||||
heterogeneous-pair verdicts
|
||||
- [[RDNA4 Container Serving]] — bring-up recipes for RDNA4 containers
|
||||
- [[RDNA4 MoE Expert Offload VRAM Budget]] — the plan law, every arena
|
||||
lever, and why free VRAM is not arena
|
||||
- [[vLLM Boot Forensics and Config Interrogation]] — reading a boot that
|
||||
died, and interrogating a running fork's config
|
||||
- [[vLLM Porting and Compile Triage]] — porting checkpoints across vLLM
|
||||
versions/forks; torch.compile/dynamo breakage triage
|
||||
- [[LLM Bench Windows and Measurement]] — window discipline, honest A/B,
|
||||
the window script laws
|
||||
- [[LLM Performance Diagnosis]] — slow prefill/decode triage on live
|
||||
endpoints, IBS attribution, prefix-cache routing
|
||||
- [[Model Swap and Rollback]] — the swap procedure, payload preflight,
|
||||
research verification, second-endpoint bring-up
|
||||
- [[Tcclaviger Fork Ops]] — the fork's sweep harness, measured knob
|
||||
verdicts, large-MoE recipe, compile-cache persistence
|
||||
- [[Heterogeneous Split Triage]] — triage for unequal-accelerator split
|
||||
proposals (the no-go verdict format)
|
||||
- [[FreeToken Builds]] — the CUDA and ROCm build paths with their
|
||||
dependency traps
|
||||
|
||||
These sit alongside the operational pages under
|
||||
`../Operations/index.md` and `../Agent/index.md`.
|
||||
|
||||
## Related
|
||||
|
||||
- [Project knowledge base](../index.md) — the Podman app stacks on the same
|
||||
|
||||
Reference in New Issue
Block a user