Files

T

Frank a34322aeb0 refactor(skills): consolidate managed skill library 94 -> 25

The managed-skill library was regenerated: 94 narrow per-incident skills
replaced by 25 broader ones (radiance-*, llamacpp-*, homepage-*, fleet/
endpoint ops). Git renders this as mass delete + mass add because whole
directories changed name; old content stays in history at 12c5e88.

Surface check: the 25 on disk are exactly the skills the harness advertises
this session.

2026-10-09 13:17:48 -07:00

2.9 KiB

Raw Blame History

name, description

name	description
closed-engine-runtime-investigation	Recipe for diagnosing/tuning a closed-source LLM runtime (halogen-class) on this fleet: docs, env-lever enumeration from the binary, ledger scraping, cache-assisted A/B under the bench law.

Investigating a closed-source LLM runtime (halogen-class)

Use when a containerized closed-source engine (e.g. halogen on frank-large) underperforms and the repo docs don't explain it. Fleet law: stop the production unit BEFORE any GPU experiment (sudo systemctl stop <unit>; mask FAILS on relinked units — stop alone, redeploy after).

1. Ground truth before flags

Boot ledger + per-request timings live in podman logs <container> (container-lifetime only) UNLESS the serve script captures stdout on disk (foreground podman run). A podman run -d ... > LOG redirect captures ONLY the container ID — a one-line hex file is the tell.
serve_api:-style log lines carry per-request prompt/prefill/rounds/commit/detok — build the depth-vs-rate curve from REAL traffic before any bench. Shallow-healthy + deep-collapsed = a depth-linear term (KV reads), not a broken engine.
If the engine's public repo exists, read its FLAGS/docs (e.g. docs/FLAGS.md) AND the bench tool source (e.g. tools/halogen-bench.py) — bench tools reveal which regimes the published numbers never measured (halogen's tg rows are shallow-only).

2. Enumerate the undocumented levers

The binary self-announces every env var it reads. The image often lacks strings:

podman run --rm --entrypoint /bin/sh <image> -c \
  'grep -aoE "<PREFIX>_[A-Z0-9_]+" /usr/local/bin/<engine> | sort -u'

Classify by kernel family (attention vs linear-state vs GEMV) against the suspected term. Only test levers with public precedent (e.g. issue #12 proved HALOGEN_FA_TQ) or an obvious name match; defaults are the publisher's measured winners — an A/B must show >=~15% to adopt.

3. Cache-assisted A/B (cheap reps)

Boot each probe arm as its own container (env is read at startup) with the prompt cache ON (explicit MB >= one full-entry at ~70 KB/token): rep 1 pays the cold prefill, rep 2+ resumes (~2 s) — depth measurements then cost decode-time only. Publish the probe on host LOOPBACK (127.0.0.1:), run the driver ON the host (python3 stdlib is enough), read the engine's own timings (prompt_ms/cache_n/predicted_ms/draft_n_accepted). Never benchmark beside a live engine on the same GPU.

4. Adopt or revert

Winner: env line into the serve env + test-seam update + docs, then deploy.sh <target> --service <svc> (restart restores everything; no unmask needed after stop-only).
Loser: podman rm -f probes, start the unit, revert nothing.
Either way: record the measured before/after in the runbook; report the depth behavior upstream if the publisher invites benchmark data — their bench gaps (depth, carve-topologies) are exactly what issues like #12 close.

2.9 KiB Raw Blame History