Frank
55f87fcdce
The 2026-10-09 wire aliases become the service names everywhere: systemd units, serve scripts, env files and FRANK_* variables, deploy-plan/deploy-remote service keys, container names, Prometheus job names + alert rules, Grafana dashboard queries, bats seams, runbook filenames, homepage/AGENTS lore, and historical prose. Rollback tags renamed in lockstep. Residue in the tree: zero. Retired frank-medium/small/npu names are history and stay. Known casualty: scripts/patches/*.patch context lines were rewritten; those retired-era vLLM-fork patches no longer apply byte-exact.
66 lines
3.8 KiB
Diff
66 lines
3.8 KiB
Diff
# gpu_worker-kv-cap.patch — fork-local fix for the expert-offload KV cap.
|
|
#
|
|
# STATUS: NOT part of the deployed frank-archon shape. It exists because the
|
|
# fork's plan under-models this model's KV, and it is the enabler for a future
|
|
# MTP attempt (the production GLM shape runs without MTP and boots fine on the
|
|
# plan's own reserve).
|
|
#
|
|
# THE BUG: the plan models KV as 13600 B/token + a small mamba term, but
|
|
# GLM-5.3-Flash also carries a per-sequence KDA state (34 linear-attention
|
|
# layers) and, under MTP, the draft model's KV. gpu_worker then CAPS the KV
|
|
# cache at that reserve, so the plan's own arithmetic can make a legal
|
|
# --max-model-len unbootable. Measured 2026-09-30 on the pinned image:
|
|
# - no MTP, max_model_len 131072: reserve 1.74 GiB >= need 1.86 GiB -> boots
|
|
# - MTP4, max_model_len 131072: reserve 1.74 GiB < need 2.07 GiB -> refuses
|
|
# - MTP4, max_model_len 262144: reserve 3.43 GiB < need 3.62 GiB -> refuses
|
|
# ("To serve at least one request with the model's max seq len ...")
|
|
# Raising max_model_len does not fix it: the cap's slope (~13,845 B/token) only
|
|
# overtakes the need's (~12,700 B/token + 0.52 GiB fixed) around 440k tokens,
|
|
# where the KV reserve would eat a third of the expert arena.
|
|
#
|
|
# THE FIX: let an explicit operator KV request raise the cap. The compile and
|
|
# graph-capture headroom still comes off the top, so the blast radius is one
|
|
# flag the operator chose.
|
|
#
|
|
# APPLY (into a host dir that is then bind-mounted read-only over the image):
|
|
# IMAGE=docker.io/tcclaviger/vllm@sha256:da4029666668b90a0386c1ed23ea370c79916ad4e1a6ad1dd9db098b9fa069ea
|
|
# D=/home/dank/glm-patch; mkdir -p $D/vllm/v1/worker
|
|
# podman run --rm --entrypoint cat $IMAGE \
|
|
# /opt/vllm/lib/python3.14/site-packages/vllm/v1/worker/gpu_worker.py \
|
|
# > $D/vllm/v1/worker/gpu_worker.py
|
|
# ( cd $D && patch -p1 < .../scripts/patches/gpu_worker-kv-cap.patch )
|
|
# # then add to the podman run:
|
|
# # -v $D/vllm/v1/worker/gpu_worker.py:/opt/vllm/lib/python3.14/site-packages/vllm/v1/worker/gpu_worker.py:ro
|
|
# # -e CLAV_KV_RESERVE_FLOOR_GIB=<needed> (or pass --kv-cache-memory-bytes)
|
|
#
|
|
# Base file sha256 (the pristine file out of the pinned image):
|
|
# 1f3942b1a01af075317161b06077a2e658086f1da0566c9f297129ceebe37dd7
|
|
# Patched file sha256 (verified 2026-09-30, patch applies clean and reproduces
|
|
# these bytes exactly):
|
|
# 80cd160bbc01270b0851f7b6fe7552b06ae2fae2428134fc29238d7f452af840
|
|
#
|
|
--- a/vllm/v1/worker/gpu_worker.py
|
|
+++ b/vllm/v1/worker/gpu_worker.py
|
|
@@ -824,6 +824,21 @@
|
|
expert_cfg = self.vllm_config.offload_config.expert
|
|
if expert_cfg.enable_expert_offload and expert_cfg.planned_kv_reserve_bytes > 0:
|
|
cap = int(expert_cfg.planned_kv_reserve_bytes)
|
|
+ # LOCAL PATCH (GLM-5.3 KDA + MTP draft): the plan models KV as
|
|
+ # 13600 B/token + a small mamba term, but this model also carries a
|
|
+ # per-sequence KDA state and (under MTP) the draft's KV, so the
|
|
+ # reserve can sit BELOW what max_model_len legitimately needs -- and
|
|
+ # the cap then makes a legal config unbootable (measured: need
|
|
+ # 2.07 GiB vs cap 1.74 GiB at max_model_len 131072 with MTP). An
|
|
+ # explicit --kv-cache-memory-bytes, or CLAV_KV_RESERVE_FLOOR_GIB,
|
|
+ # is the operator's word: let it raise the cap. The compile/capture
|
|
+ # headroom still comes off the top.
|
|
+ _explicit = self.cache_config.kv_cache_memory_bytes
|
|
+ _floor = os.environ.get("CLAV_KV_RESERVE_FLOOR_GIB")
|
|
+ if _explicit:
|
|
+ cap = max(cap, int(_explicit))
|
|
+ if _floor:
|
|
+ cap = max(cap, int(float(_floor) * (1 << 30)))
|
|
if self.available_kv_cache_memory_bytes > cap:
|
|
logger.info_once(
|
|
"expert offload: KV capped at the plan's reserve %s GiB; %s GiB of "
|