Frank
a34322aeb0
The managed-skill library was regenerated: 94 narrow per-incident skills
replaced by 25 broader ones (radiance-*, llamacpp-*, homepage-*, fleet/
endpoint ops). Git renders this as mass delete + mass add because whole
directories changed name; old content stays in history at 12c5e88.
Surface check: the 25 on disk are exactly the skills the harness advertises
this session.
30 lines
2.9 KiB
Markdown
30 lines
2.9 KiB
Markdown
---
|
|
name: closed-engine-runtime-investigation
|
|
description: "Recipe for diagnosing/tuning a closed-source LLM runtime (halogen-class) on this fleet: docs, env-lever enumeration from the binary, ledger scraping, cache-assisted A/B under the bench law."
|
|
---
|
|
|
|
# Investigating a closed-source LLM runtime (halogen-class)
|
|
|
|
Use when a containerized closed-source engine (e.g. halogen on frank-large) underperforms and the repo docs don't explain it. Fleet law: stop the production unit BEFORE any GPU experiment (`sudo systemctl stop <unit>`; mask FAILS on relinked units — stop alone, redeploy after).
|
|
|
|
## 1. Ground truth before flags
|
|
- Boot ledger + per-request timings live in `podman logs <container>` (container-lifetime only) UNLESS the serve script captures stdout on disk (foreground `podman run`). A `podman run -d ... > LOG` redirect captures ONLY the container ID — a one-line hex file is the tell.
|
|
- `serve_api:`-style log lines carry per-request `prompt/prefill/rounds/commit/detok` — build the depth-vs-rate curve from REAL traffic before any bench. Shallow-healthy + deep-collapsed = a depth-linear term (KV reads), not a broken engine.
|
|
- If the engine's public repo exists, read its FLAGS/docs (e.g. `docs/FLAGS.md`) AND the bench tool source (e.g. `tools/halogen-bench.py`) — bench tools reveal which regimes the published numbers never measured (halogen's tg rows are shallow-only).
|
|
|
|
## 2. Enumerate the undocumented levers
|
|
The binary self-announces every env var it reads. The image often lacks `strings`:
|
|
```
|
|
podman run --rm --entrypoint /bin/sh <image> -c \
|
|
'grep -aoE "<PREFIX>_[A-Z0-9_]+" /usr/local/bin/<engine> | sort -u'
|
|
```
|
|
Classify by kernel family (attention vs linear-state vs GEMV) against the suspected term. Only test levers with public precedent (e.g. issue #12 proved HALOGEN_FA_TQ) or an obvious name match; defaults are the publisher's measured winners — an A/B must show >=~15% to adopt.
|
|
|
|
## 3. Cache-assisted A/B (cheap reps)
|
|
Boot each probe arm as its own container (env is read at startup) with the prompt cache ON (explicit MB >= one full-entry at ~70 KB/token): rep 1 pays the cold prefill, rep 2+ resumes (~2 s) — depth measurements then cost decode-time only. Publish the probe on host LOOPBACK (127.0.0.1:<scratch>), run the driver ON the host (python3 stdlib is enough), read the engine's own `timings` (prompt_ms/cache_n/predicted_ms/draft_n_accepted). Never benchmark beside a live engine on the same GPU.
|
|
|
|
## 4. Adopt or revert
|
|
- Winner: env line into the serve env + test-seam update + docs, then `deploy.sh <target> --service <svc>` (restart restores everything; no unmask needed after stop-only).
|
|
- Loser: `podman rm -f` probes, start the unit, revert nothing.
|
|
- Either way: record the measured before/after in the runbook; report the depth behavior upstream if the publisher invites benchmark data — their bench gaps (depth, carve-topologies) are exactly what issues like #12 close.
|