dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: two stacking fixes from the first live run

Browse Source 
- release the numpy frombuffer view before closing the source mmap
  (Python 3.14 BufferError: cannot close exported pointers exist).
- stacked scale planes rename: checkpoint.cpp scale_name() appends
  _scale_inv to the STACKED tensor name verbatim (no .weight segment) -
  ...experts.gate_up_proj_scale_inv, not ...gate_up_proj.weight_scale_inv.
This commit is contained in:
Frank
2026-10-05 23:56:36 -07:00
parent 13112a93f5
commit 272acb4c7f
1 changed files with 11 additions and 9 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/stack-qwen38-official-fp8.py
+11 -9
View File
@@ -7,9 +7,11 @@ leaves the box). Two transforms, both VALUE-PRESERVING:
1. EXPERTS: the shipped plugin's stacked layout names ONE tensor per family
(`mlp.experts.gate_up_proj` [E, 2F, H], `mlp.experts.down_proj` [E, H, F])
plus a `_scale_inv` plane at the same index, while the official FP8 repo
ships per-expert unfused tensors (`experts.<e>.{gate,up,down}_proj.weight`
+ scale). Stacking is pure byte concatenation along dim 0 — gate rows
plus a `<family>_scale_inv` plane at the same index (NO `.weight` in the
stacked name — scale_name() appends _scale_inv to the tensor name verbatim),
while the official FP8 repo ships per-expert unfused tensors
(`experts.<e>.{gate,up,down}_proj.weight` + scale). Stacking is pure byte
concatenation along dim 0 — gate rows
first inside each expert's gate_up slab, which is the order the block
wants (rad_block_moe_fp8.h: "a stacked gate_up is ALREADY FUSED in the
order this block wants"). No requantisation: bytes are moved, not decoded.
@@ -195,10 +197,10 @@ def stack_prefix(job):
out = os.path.join(stage, f"stacked.{idx:03d}.safetensors")
tensors = [
(f"{pre}.mlp.experts.gate_up_proj", F8, [n_e, f2, h], n_e * f2 * h),
(f"{pre}.mlp.experts.gate_up_proj.weight_scale_inv", BF16,
(f"{pre}.mlp.experts.gate_up_proj_scale_inv", BF16,
[n_e, gs[0] * 2, gs[1]], n_e * gus_bytes),
(f"{pre}.mlp.experts.down_proj", F8, [n_e, h, f], n_e * h * f),
(f"{pre}.mlp.experts.down_proj.weight_scale_inv", BF16,
(f"{pre}.mlp.experts.down_proj_scale_inv", BF16,
[n_e, ds[0], ds[1]], n_e * ds_bytes),
]
bufs = iter([gu, gus, dn, dns])
@@ -410,9 +412,9 @@ def main():
for idx, pre in enumerate(prefixes):
fname = f"stacked.{idx:03d}.safetensors"
for t in (f"{pre}.mlp.experts.gate_up_proj",
f"{pre}.mlp.experts.gate_up_proj.weight_scale_inv",
f"{pre}.mlp.experts.gate_up_proj_scale_inv",
f"{pre}.mlp.experts.down_proj",
f"{pre}.mlp.experts.down_proj.weight_scale_inv"):
f"{pre}.mlp.experts.down_proj_scale_inv"):
new_wm[t] = fname
new_wm[merged_name] = out_name
total = sum(meta_size(meta, k) for k in new_wm
@@ -500,10 +502,10 @@ def verify(stage, prefixes, n_e, shapes, meta, shard_ids, rows, hd,
hlen2, hdr = parse_header(path)
want = {
f"{pre}.mlp.experts.gate_up_proj": (F8, [n_e, f2, h]),
f"{pre}.mlp.experts.gate_up_proj.weight_scale_inv":
f"{pre}.mlp.experts.gate_up_proj_scale_inv":
(BF16, [n_e, gs[0] * 2, gs[1]]),
f"{pre}.mlp.experts.down_proj": (F8, [n_e, h, f]),
f"{pre}.mlp.experts.down_proj.weight_scale_inv": (BF16, [n_e, ds[0], ds[1]]),
f"{pre}.mlp.experts.down_proj_scale_inv": (BF16, [n_e, ds[0], ds[1]]),
}
for name, (dt, shp) in want.items():
e = hdr[name]