From 3cf543ead638d3fb95697b5a89e379ad07ddb8ec Mon Sep 17 00:00:00 2001 From: Comrade Chief Designer Date: Wed, 30 Sep 2026 16:24:16 -0700 Subject: [PATCH] frank-primary: fork-bomb hazard, patch 06 documented, retired probe Recording the incident: a diagnostic sitecustomize.py mounted at /probe on PYTHONPATH produced a self-replicating chain of python3 processes 19+ levels deep inside the probe container (sitecustomize runs in EVERY interpreter; the hook imported torch/vLLM at startup; something in that import chain spawned a helper python which inherited PYTHONPATH and the hook, and spawned again). Found alive 18 minutes later, 33 probe-tagged processes. Killed with 'podman rm -f glm-probe' (the container cgroup owns every generation) and verified by counting /proc//environ for the marker variable -- never by pattern-killing an interpreter name, since this host runs python in production containers (romm, jellyfin, plex, glances). Fixed at the source with an env marker that children inherit, verified with a two-generation test: gen 1 installs, gen 2 stays inert. The runbook now carries the hazard, the guard, the kill recipe, the conjunction the restore must verify, and the no-overlapping-jobs rule. Patch 06 (glm53-dflash-01-aux-hidden-states.patch) is documented in the DFlash2 section, and the /probe diagnostic is marked retired -- TORCH_LOGS answers dynamo questions with no startup hook. tests/run.sh: 362 seams pass. --- .../LLM/frank-primary-glm53-flash-w4a16.md | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) diff --git a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md index aae6cd7..6f43628 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md +++ b/serverAppConfigs-docs/LLM/frank-primary-glm53-flash-w4a16.md @@ -346,6 +346,23 @@ MTP4's acceptance was already excellent (3.76-4.90). The port is worth doing as it is small), not as a performance fix, and the honest expectation on this shape is flat decode with a prefill regression like MTP's until the pointwise mass is fused. +**The port exists now: `scripts/patches/glm53-dflash-01-aux-hidden-states.patch`** (WIP, +independent of the compile changelist — it patches the same file as compile-01, so +compile-01 goes first if both are applied). It implements exactly the contract read out +of this image: `EagleModelMixin` on `Glm5NextModel`, capture of the single-stream +`hidden_states` ENTERING each requested layer in the loop, the conditional +`(hidden_states, aux_hidden_states)` tuple return, and +`set_aux_hidden_state_layers` / `get_eagle3_default_aux_hidden_state_layers` on +`Glm5NextForCausalLM` (delegation shape from `kimi_k25.py:466`, `pixtral.py:503`; +capture shape from `deepseek_v2.py:1551-1585`; tuple consumption at +`v1/worker/gpu/cudagraph_utils.py:621`). + +**Retired diagnostic (do not resurrect as-is): the `/probe` startup hook.** An +`op_probe.py` + `sitecustomize.py` pair was mounted to wrap vLLM module forwards and +torch's name-map lookups. It produced no evidence and caused a **fork bomb** (see the +hazard note under experiment-window discipline). Use `TORCH_LOGS` for dynamo questions +instead — it needs no startup hook. + Note also that the KV-cap patch's predecessor arithmetic in [[dankserver-qwen38-fp8-expert-offload-tuning]] should be read the same way: its "gather-bound" conclusion came from the same throughput arithmetic that this page's @@ -479,6 +496,30 @@ glm-probe` cleanup ran while an earlier job was still mid-flight and deleted its container, so one whole round produced no evidence at all (`no container with name glm-probe found`) — check `proc://` for running jobs before opening a window. +**Never put a diagnostic hook on `PYTHONPATH` without a marker guard — it is a fork +bomb.** On 2026-09-30 a `sitecustomize.py` mounted at `/probe` (to wrap dynamo's +name-map lookups and vLLM module forwards) produced a self-replicating chain of +`python3` processes 19+ levels deep inside the probe container: `sitecustomize` is +imported by *every* interpreter, the hook imported torch/vLLM at startup, something in +that import chain spawned a helper python, and the child inherited `PYTHONPATH` (hence +the hook) and spawned again. It was still alive 18 minutes later with the container +still showing `Up`. Symptoms: `ps -eo pid,ppid,comm --forest` shows a deep ladder of +identical interpreters, each child's PPID being the previous one. Guard (now in the +hook): arm once per environment via a marker in `os.environ`, which children inherit — + +```python +if os.environ.get("VLLM_PROBE_ARMED") == "1": + pass # an ancestor already armed it; stay inert +else: + os.environ["VLLM_PROBE_ARMED"] = "1" + +``` + +Kill recipe: `podman rm -f ` (the container cgroup owns every +generation), then verify with a precise count over `/proc//environ` for the marker +variable — **never** pattern-kill an interpreter name, because the media stack on this +host (romm, jellyfin, plex, glances) runs python in production. + ```bash # boot: ~7 min warm page cache, longer cold; health returns 503 while loading curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:7770/health