frank-primary: fork-bomb hazard, patch 06 documented, retired probe

Recording the incident: a diagnostic sitecustomize.py mounted at /probe on PYTHONPATH
produced a self-replicating chain of python3 processes 19+ levels deep inside the probe
container (sitecustomize runs in EVERY interpreter; the hook imported torch/vLLM at
startup; something in that import chain spawned a helper python which inherited
PYTHONPATH and the hook, and spawned again). Found alive 18 minutes later, 33 probe-tagged
processes. Killed with 'podman rm -f glm-probe' (the container cgroup owns every
generation) and verified by counting /proc/<pid>/environ for the marker variable --
never by pattern-killing an interpreter name, since this host runs python in production
containers (romm, jellyfin, plex, glances). Fixed at the source with an env marker that
children inherit, verified with a two-generation test: gen 1 installs, gen 2 stays inert.

The runbook now carries the hazard, the guard, the kill recipe, the conjunction the
restore must verify, and the no-overlapping-jobs rule. Patch 06
(glm53-dflash-01-aux-hidden-states.patch) is documented in the DFlash2 section, and the
/probe diagnostic is marked retired -- TORCH_LOGS answers dynamo questions with no
startup hook.

tests/run.sh: 362 seams pass.
This commit is contained in:
Comrade Chief Designer
2026-09-30 16:24:16 -07:00
parent c7db6d94e9
commit 3cf543ead6
@@ -346,6 +346,23 @@ MTP4's acceptance was already excellent (3.76-4.90). The port is worth doing as
it is small), not as a performance fix, and the honest expectation on this shape is
flat decode with a prefill regression like MTP's until the pointwise mass is fused.
**The port exists now: `scripts/patches/glm53-dflash-01-aux-hidden-states.patch`** (WIP,
independent of the compile changelist — it patches the same file as compile-01, so
compile-01 goes first if both are applied). It implements exactly the contract read out
of this image: `EagleModelMixin` on `Glm5NextModel`, capture of the single-stream
`hidden_states` ENTERING each requested layer in the loop, the conditional
`(hidden_states, aux_hidden_states)` tuple return, and
`set_aux_hidden_state_layers` / `get_eagle3_default_aux_hidden_state_layers` on
`Glm5NextForCausalLM` (delegation shape from `kimi_k25.py:466`, `pixtral.py:503`;
capture shape from `deepseek_v2.py:1551-1585`; tuple consumption at
`v1/worker/gpu/cudagraph_utils.py:621`).
**Retired diagnostic (do not resurrect as-is): the `/probe` startup hook.** An
`op_probe.py` + `sitecustomize.py` pair was mounted to wrap vLLM module forwards and
torch's name-map lookups. It produced no evidence and caused a **fork bomb** (see the
hazard note under experiment-window discipline). Use `TORCH_LOGS` for dynamo questions
instead — it needs no startup hook.
Note also that the KV-cap patch's predecessor arithmetic in
[[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its
"gather-bound" conclusion came from the same throughput arithmetic that this page's
@@ -479,6 +496,30 @@ glm-probe` cleanup ran while an earlier job was still mid-flight and deleted its
container, so one whole round produced no evidence at all (`no container with name
glm-probe found`) — check `proc://` for running jobs before opening a window.
**Never put a diagnostic hook on `PYTHONPATH` without a marker guard — it is a fork
bomb.** On 2026-09-30 a `sitecustomize.py` mounted at `/probe` (to wrap dynamo's
name-map lookups and vLLM module forwards) produced a self-replicating chain of
`python3` processes 19+ levels deep inside the probe container: `sitecustomize` is
imported by *every* interpreter, the hook imported torch/vLLM at startup, something in
that import chain spawned a helper python, and the child inherited `PYTHONPATH` (hence
the hook) and spawned again. It was still alive 18 minutes later with the container
still showing `Up`. Symptoms: `ps -eo pid,ppid,comm --forest` shows a deep ladder of
identical interpreters, each child's PPID being the previous one. Guard (now in the
hook): arm once per environment via a marker in `os.environ`, which children inherit —
```python
if os.environ.get("VLLM_PROBE_ARMED") == "1":
pass # an ancestor already armed it; stay inert
else:
os.environ["VLLM_PROBE_ARMED"] = "1"
<install probes>
```
Kill recipe: `podman rm -f <experiment-container>` (the container cgroup owns every
generation), then verify with a precise count over `/proc/<pid>/environ` for the marker
variable — **never** pattern-kill an interpreter name, because the media stack on this
host (romm, jellyfin, plex, glances) runs python in production.
```bash
# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health