---
name: closed-engine-runtime-investigation
description: "Recipe for diagnosing/tuning a closed-source LLM runtime (halogen-class) on this fleet: docs, env-lever enumeration from the binary, ledger scraping, cache-assisted A/B under the bench law."
---

# Investigating a closed-source LLM runtime (halogen-class)

Use when a containerized closed-source engine (e.g. halogen on frank-large) underperforms and the repo docs don't explain it. Fleet law: stop the production unit BEFORE any GPU experiment (`sudo systemctl stop <unit>`; mask FAILS on relinked units — stop alone, redeploy after).

## 1. Ground truth before flags
- Boot ledger + per-request timings live in `podman logs <container>` (container-lifetime only) UNLESS the serve script captures stdout on disk (foreground `podman run`). A `podman run -d ... > LOG` redirect captures ONLY the container ID — a one-line hex file is the tell.
- `serve_api:`-style log lines carry per-request `prompt/prefill/rounds/commit/detok` — build the depth-vs-rate curve from REAL traffic before any bench. Shallow-healthy + deep-collapsed = a depth-linear term (KV reads), not a broken engine.
- If the engine's public repo exists, read its FLAGS/docs (e.g. `docs/FLAGS.md`) AND the bench tool source (e.g. `tools/halogen-bench.py`) — bench tools reveal which regimes the published numbers never measured (halogen's tg rows are shallow-only).

## 2. Enumerate the undocumented levers
The binary self-announces every env var it reads. The image often lacks `strings`:
```
podman run --rm --entrypoint /bin/sh <image> -c \
  'grep -aoE "<PREFIX>_[A-Z0-9_]+" /usr/local/bin/<engine> | sort -u'
```
Classify by kernel family (attention vs linear-state vs GEMV) against the suspected term. Only test levers with public precedent (e.g. issue #12 proved HALOGEN_FA_TQ) or an obvious name match; defaults are the publisher's measured winners — an A/B must show >=~15% to adopt.

## 3. Cache-assisted A/B (cheap reps)
Boot each probe arm as its own container (env is read at startup) with the prompt cache ON (explicit MB >= one full-entry at ~70 KB/token): rep 1 pays the cold prefill, rep 2+ resumes (~2 s) — depth measurements then cost decode-time only. Publish the probe on host LOOPBACK (127.0.0.1:<scratch>), run the driver ON the host (python3 stdlib is enough), read the engine's own `timings` (prompt_ms/cache_n/predicted_ms/draft_n_accepted). Never benchmark beside a live engine on the same GPU.

## 4. Adopt or revert
- Winner: env line into the serve env + test-seam update + docs, then `deploy.sh <target> --service <svc>` (restart restores everything; no unmask needed after stop-only).
- Loser: `podman rm -f` probes, start the unit, revert nothing.
- Either way: record the measured before/after in the runbook; report the depth behavior upstream if the publisher invites benchmark data — their bench gaps (depth, carve-topologies) are exactly what issues like #12 close.