frank-primary: fork-bomb hazard, patch 06 documented, retired probe
Recording the incident: a diagnostic sitecustomize.py mounted at /probe on PYTHONPATH produced a self-replicating chain of python3 processes 19+ levels deep inside the probe container (sitecustomize runs in EVERY interpreter; the hook imported torch/vLLM at startup; something in that import chain spawned a helper python which inherited PYTHONPATH and the hook, and spawned again). Found alive 18 minutes later, 33 probe-tagged processes. Killed with 'podman rm -f glm-probe' (the container cgroup owns every generation) and verified by counting /proc/<pid>/environ for the marker variable -- never by pattern-killing an interpreter name, since this host runs python in production containers (romm, jellyfin, plex, glances). Fixed at the source with an env marker that children inherit, verified with a two-generation test: gen 1 installs, gen 2 stays inert. The runbook now carries the hazard, the guard, the kill recipe, the conjunction the restore must verify, and the no-overlapping-jobs rule. Patch 06 (glm53-dflash-01-aux-hidden-states.patch) is documented in the DFlash2 section, and the /probe diagnostic is marked retired -- TORCH_LOGS answers dynamo questions with no startup hook. tests/run.sh: 362 seams pass.
This commit is contained in:
@@ -346,6 +346,23 @@ MTP4's acceptance was already excellent (3.76-4.90). The port is worth doing as
|
||||
it is small), not as a performance fix, and the honest expectation on this shape is
|
||||
flat decode with a prefill regression like MTP's until the pointwise mass is fused.
|
||||
|
||||
**The port exists now: `scripts/patches/glm53-dflash-01-aux-hidden-states.patch`** (WIP,
|
||||
independent of the compile changelist — it patches the same file as compile-01, so
|
||||
compile-01 goes first if both are applied). It implements exactly the contract read out
|
||||
of this image: `EagleModelMixin` on `Glm5NextModel`, capture of the single-stream
|
||||
`hidden_states` ENTERING each requested layer in the loop, the conditional
|
||||
`(hidden_states, aux_hidden_states)` tuple return, and
|
||||
`set_aux_hidden_state_layers` / `get_eagle3_default_aux_hidden_state_layers` on
|
||||
`Glm5NextForCausalLM` (delegation shape from `kimi_k25.py:466`, `pixtral.py:503`;
|
||||
capture shape from `deepseek_v2.py:1551-1585`; tuple consumption at
|
||||
`v1/worker/gpu/cudagraph_utils.py:621`).
|
||||
|
||||
**Retired diagnostic (do not resurrect as-is): the `/probe` startup hook.** An
|
||||
`op_probe.py` + `sitecustomize.py` pair was mounted to wrap vLLM module forwards and
|
||||
torch's name-map lookups. It produced no evidence and caused a **fork bomb** (see the
|
||||
hazard note under experiment-window discipline). Use `TORCH_LOGS` for dynamo questions
|
||||
instead — it needs no startup hook.
|
||||
|
||||
Note also that the KV-cap patch's predecessor arithmetic in
|
||||
[[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its
|
||||
"gather-bound" conclusion came from the same throughput arithmetic that this page's
|
||||
@@ -479,6 +496,30 @@ glm-probe` cleanup ran while an earlier job was still mid-flight and deleted its
|
||||
container, so one whole round produced no evidence at all (`no container with name
|
||||
glm-probe found`) — check `proc://` for running jobs before opening a window.
|
||||
|
||||
**Never put a diagnostic hook on `PYTHONPATH` without a marker guard — it is a fork
|
||||
bomb.** On 2026-09-30 a `sitecustomize.py` mounted at `/probe` (to wrap dynamo's
|
||||
name-map lookups and vLLM module forwards) produced a self-replicating chain of
|
||||
`python3` processes 19+ levels deep inside the probe container: `sitecustomize` is
|
||||
imported by *every* interpreter, the hook imported torch/vLLM at startup, something in
|
||||
that import chain spawned a helper python, and the child inherited `PYTHONPATH` (hence
|
||||
the hook) and spawned again. It was still alive 18 minutes later with the container
|
||||
still showing `Up`. Symptoms: `ps -eo pid,ppid,comm --forest` shows a deep ladder of
|
||||
identical interpreters, each child's PPID being the previous one. Guard (now in the
|
||||
hook): arm once per environment via a marker in `os.environ`, which children inherit —
|
||||
|
||||
```python
|
||||
if os.environ.get("VLLM_PROBE_ARMED") == "1":
|
||||
pass # an ancestor already armed it; stay inert
|
||||
else:
|
||||
os.environ["VLLM_PROBE_ARMED"] = "1"
|
||||
<install probes>
|
||||
```
|
||||
|
||||
Kill recipe: `podman rm -f <experiment-container>` (the container cgroup owns every
|
||||
generation), then verify with a precise count over `/proc/<pid>/environ` for the marker
|
||||
variable — **never** pattern-kill an interpreter name, because the media stack on this
|
||||
host (romm, jellyfin, plex, glances) runs python in production.
|
||||
|
||||
```bash
|
||||
# boot: ~7 min warm page cache, longer cold; health returns 503 while loading
|
||||
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health
|
||||
|
||||
Reference in New Issue
Block a user