From 713b1266397fedf2492fffff87c3203aac404a06 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 6 Oct 2026 01:06:11 -0700 Subject: [PATCH] =?UTF-8?q?frank-primary:=20cutover=20measured=20=E2=80=94?= =?UTF-8?q?=20docs=20reflect=20the=20live=20service?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit AGENTS.md: row + section carry the measured 177,109 MiB container, floor 177000, live cutover 2026-10-06. Runbook: the three convert laws the live runs proved (scale-name, file-shadow, symlink-through), the measured boot (125.26 GiB page-in 38.1 s, PLE pin 47.68 GiB/43.9 s, ~2 min to health), the MTP/vision/tool/reasoning contract results, and the SSD tier fix. --- AGENTS.md | 9 +-- ...rank-primary-radiance-qwen38-native-fp8.md | 63 +++++++++++++++++-- 2 files changed, 64 insertions(+), 8 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index a231c23..34105bc 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,7 +24,7 @@ merge absorbed the former `vllmScripts` repo, history preserved). | Target | Service | Runtime | Model | Port | |--------|---------|---------|-------|------| -| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (~161 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, **2 × 200,000** (first boot; native 262,144), froggeric v22.5 template, vision + tools + `reasoning_content` auto, `/metrics` scraped. Cutover 2026-10-05 (rollback `frank-primary-freetoken-pr587-20261005`, FreeToken one-card image + payload on disk) | 7770 | +| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *… | Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (35,865,565,184 B; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (measured: 8 slots at serial ~6 t/s/stream), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot AND `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` are both 400s naming the limit, never truncations. Serve passes NO `--ulimit nofile` (rootless runc cannot raise Dankclaw's 524288 hard limit — see Key Lessons). Cutover 2026-10-05|7770| | Dankclaw | `frank-small` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF **IQ3_S -mtp** (27B dense qwen3_5 hybrid, 12,120,016,896 B; a per-tensor allocation mix of stock types; the -mtp file's MTP head present but NOT drafted — **spec OFF** since 2026-10-06, ngram skipped on the recurrent-rollback law) + unsloth's Qwen3.8-27B **F16 mmproj** for **vision** (no Swift projector ships; the base vision stack was hash-proven byte-identical to Swift's), on the RTX 3090, **2 × 131,072** (Frank's 128k cap each; the tight fit he ordered 2026-10-06 — **1,397 MiB measured free**, the 2-3 GiB floor knowingly violated) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 512; **48.0 t/s** no-spec decode (MTP reference 58.9-66.9). Supersedes BOTH same-day payloads: the live MiMo-V2.6 Q6_K 2×200,704 f16-KV shape and the Ornith-1.5-9B Q8_0 swap that was committed (77decb5..380aafa) but never deployed. | 7771 | @@ -477,7 +477,8 @@ checkpoint at NATIVE precision** — modelopt block-FP8 throughout, no requantisation, no calibration. Plain `podman run` through `scripts/serve-frank-primary.sh` under the root systemd unit (`frank-primary.service`, `User=dank`, `LimitMEMLOCK=infinity`); the -container name is the unit name (`frank-primary`). Cutover 2026-10-05; +container name is the unit name (`frank-primary`). Cutover LIVE 2026-10-06 +00:55 (chain built 2026-10-05/06); rollback tag `frank-primary-freetoken-pr587-20261005` (the FreeToken trio + Dockerfile are in the tag; the image and the orcarouter payload stay on disk until the window closes), older rollback `frank-primary-vllm-shapeG-20261003`. @@ -492,8 +493,8 @@ decoded to one bf16 shard with the checkpoint's own bf16 scale copy for every weight, and the rtn rule re-encodes the bf16 PLE at that same fixed scale, an exact round-trip that restores the original codes and lands the plugin's one-element scale plane → -`/home/dank/models/rad/qwen4exp-fp8-native.rad` (~161 GiB, floor -`FRANK_PRIMARY_RAD_MIN_MIB=158000`). The staging drop is the plugin +`/home/dank/models/rad/qwen4exp-fp8-native.rad` (177,109 MiB measured, +floor `FRANK_PRIMARY_RAD_MIN_MIB=177000`). The staging drop is the plugin SELECTION law (spec §2.4): the descriptor is `(arch_id, quant)` matched exactly, no shipped plugin claims `(qwen4exp, "fp8")`, the shipped `qwen4exp_fp8` plugin claims `(qwen4exp, "")` — and the descriptor is only diff --git a/serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md b/serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md index 0efb5ca..4c9317f 100644 --- a/serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md +++ b/serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md @@ -38,7 +38,8 @@ same served alias (`frank-primary`), same container identity all 131 shards sha256-match the Hub LFS oids (2026-10-05, log `/home/dank/logs/verify-qwen38-official-fp8.log`). - **The container**: `/home/dank/models/rad/qwen4exp-fp8-native.rad` - (~161 GiB; floor `FRANK_PRIMARY_RAD_MIN_MIB=158000` in the env gates a + (177,109 MiB measured = 173.0 GiB; floor + `FRANK_PRIMARY_RAD_MIN_MIB=177000` in the env gates a truncated convert). Built by `scripts/build-frank-primary-rad.sh`: 1. stage `/home/dank/models/stage-qwen4exp-fp8` — `config.json` with **`quantization_config` dropped** + relative symlinks into the raw @@ -62,7 +63,26 @@ same served alias (`frank-primary`), same container identity exact round-trip (nearest-code distance 2⁻³ vs the bf16 rounding perturbation 2⁻⁹): the container gets the checkpoint's own PLE codes plus the one-element scale plane (`blk.1.ple_ngram.scale`, view {1}); - 4. `rad-info` summary + size floor. + 4. THREE LAWS the live runs proved (each cost a full verify+declare + cycle; all fixed in the two scripts, 2026-10-05/06): + - **Scale-name law**: `scale_name()` appends `_scale_inv` to the + TENSOR NAME verbatim — a stacked tensor named + `…experts.gate_up_proj` (no `.weight` suffix) has its scale plane + at `…experts.gate_up_proj_scale_inv`, NOT `…weight_scale_inv`. + - **File-shadow law**: the reader opens every index-named file in + SORTED order and emplaces EVERY tensor each file physically + holds, first-wins (`checkpoint.cpp add_shard`). The raw + model-000NN shards carry tensors NAMED exactly what the merged + PLE shard is named, so the merged file MUST sort first + (`00-ple.merged.safetensors`) or a raw fp8 shard wins the name + and the plan reads one shard as the codes. + - **Symlink-through law**: staging symlinks the raw index into the + stage; `open(path,"w")` writes THROUGH it and corrupts the raw + checkpoint (raw verify then fails 143/144). The stack script + writes index/config temp+`os.replace` — an atomic swap replaces + the symlink itself; the build script COPIES the index and + symlinks only the shards. + 5. `rad-info` summary + size floor. - **Why the staging dir is mandatory** (the descriptor law, spec §2.4): the plugin is selected by the exact `(arch_id, quant descriptor)` pair with no fallback. A raw official checkpoint carries @@ -122,8 +142,10 @@ KV from the `0.82`-side remainder of claimed VRAM at 96 MiB headroom. while serving, 503 only while stopping — **the listener binds AFTER the full load**, so a bare `curl -fsS http://127.0.0.1:7770/health` gate is honest (the deployer loops on it; the FreeToken-era `"status":"ok"` body - grep is gone). Boot budget 1800 s (publisher `start_period` 15 m); the - big terms are the ~161 GiB page-in and the host-pool pin. + grep is gone). Boot budget 1800 s (publisher `start_period` 15 m); + measured 2026-10-06: health 200 ~2-3 min after start (~120 s warm) — + the big terms are the 125.26 GiB page-in (38.1 s at 3.5 GB/s) and the + host-pool pin (PLE 47.68 GiB in 43.9 s). - **Sampling** (oai.cpp): `temperature, top_p, top_k, min_p, typical_p, presence_penalty, frequency_penalty, repetition_penalty, repeat_last_n, chat_template_kwargs, reasoning_effort, enable_thinking, @@ -174,6 +196,39 @@ Post-cutover checklist (each is observable, not assumed): `image_url` content part. 6. Prometheus target `frank-primary` green (not never-green). +## Cutover measured (2026-10-06, all values observed live) + +- Container: 1,857,127,997,440 B = 177,109 MiB = 173.0 GiB. Plan-only + priced 172.76 GiB — the DATA planes; the written container adds a + metadata tail. Floor gates must be set from the real file (the first + deploy refused on 178000 > 177109; floor is 177000). +- Boot: 125.26 GiB read in 38.1 s (3.5 GB/s), PLE pinned + "47.68 GiB in 43.9 s" (the 2 B scale plane with it), each rank's + mover holds 25.46 GiB static+experts (19.15 GiB resident experts, + 79 %), 16,820 pinned pool slots, health 200 ~2-3 min after start + (~120 s warm). No 7-minute vLLM-style boot. +- KV ledger: kv_attn 99,956 blocks = 2.48 GiB (399,824 tokens), + kv_qsa_bkey 317.24 MiB, kv_gdn_state 4 slots × 54 MiB, PLE conv + states < 1 MiB; linear checkpoints 56.37 MiB × 8 slots (3 of 8 + card-backed at boot, 5 withdrawn under the 208 MiB floor). +- MTP on real traffic (temp 0.7, 220-token ask): draft_n 225 / + draft_n_accepted 119 = 52.9 % acceptance, decode 23.8 t/s. ONE + sample — the shape's real number needs live-traffic measurement. +- Contracts: `reasoning_content` field (matches models.yml), usage + metering separates reasoning_tokens; vision — a 512² solid-red PNG + through `image_url` answers `Red` (279 prompt tokens; NOTE the + 32-token budget is eaten by thinking — give vision probes ≥256 + max_tokens); native `tool_calls` with finish_reason tool_calls. +- Boot defects fixed the same day: the SSD prefix tier opened only + after the serve script mkdir -p'd the full path (the engine's + create is NOT recursive: "cannot create /data/kvcache/..."); the + converter's scale-name law, the PLE file-shadow and the + symlink-through-raw-index bug are in the payload section above. +- Prometheus: `/metrics` answers 200 with vLLM-compatible families + (verified from Dankclaw); the `frank-primary` job needs a + prometheus-service deploy to go green (config pushed, reload + deferred behind a live parallel writer's dirty tree). + ## Tuning levers (in order of likely use) - **`FRANK_PRIMARY_MAX_SEQS`**: raise 2 → 3/4 only when the ledger's KV