frank-primary: never write the stage index through the raw symlink
Run 3 rewrote the staged index THROUGH the relative symlink into the raw checkpoint: the raw index.json was restamped with the stacked layout (605 KB smaller), the raw verify then failed 143/144 and the overnight run died before stacking. Two fixes: - stack-qwen38-official-fp8.py: index/config writes are now temp+os.replace (an atomic swap replaces the symlink itself; the raw tree stays pristine). - build-frank-primary-rad.sh: the stage's index.json is COPIED like config.json; only shards stay relative symlinks.
This commit is contained in:
@@ -98,13 +98,18 @@ dropped = cfg.pop("quantization_config", None)
|
||||
json.dump(cfg, open(dst, "w"), indent=2)
|
||||
print(f"staging config.json written (quantization_config dropped: {dropped is not None})")
|
||||
PY
|
||||
# Everything else is a RELATIVE symlink — relative so the targets resolve
|
||||
# inside the convert container (the stage's parent is mounted at /models).
|
||||
# The index is COPIED like config.json, never symlinked: the reshape step
|
||||
# rewrites the stage's index, and a symlink would write through to the
|
||||
# raw checkpoint (measured 2026-10-05: the raw index was restamped with
|
||||
# the stacked layout and the raw verify failed 143/144). Shards stay
|
||||
# relative symlinks — relative so the targets resolve inside the convert
|
||||
# container (the stage's parent is mounted at /models).
|
||||
for f in "$SRC"/*; do
|
||||
name="$(basename "$f")"
|
||||
[ "$name" = "config.json" ] && continue
|
||||
case "$name" in config.json|model.safetensors.index.json) continue ;; esac
|
||||
ln -sfn "../Qwen/Qwen3.8-Flash-Next-FP8/$name" "$STAGE/$name"
|
||||
done
|
||||
cp "$SRC/model.safetensors.index.json" "$STAGE/"
|
||||
echo "staging dir ready: $STAGE ($(ls "$STAGE" | wc -l) entries)"
|
||||
fi
|
||||
|
||||
|
||||
@@ -424,12 +424,17 @@ def main():
|
||||
index["weight_map"] = new_wm
|
||||
if "metadata" in index and isinstance(index["metadata"], dict):
|
||||
index["metadata"]["total_size"] = total
|
||||
with open(index_path, "w") as fh:
|
||||
json.dump(index, fh, indent=2)
|
||||
|
||||
# ATOMIC REPLACE, never open(path, "w"): the stage's index is a symlink
|
||||
# to the RAW checkpoint (staging symlinks everything but config.json) —
|
||||
# open("w") would rewrite the raw file through the link. os.replace
|
||||
# swaps the link itself for a real file; the raw tree stays pristine.
|
||||
for path, payload in ((index_path, json.dumps(index, indent=2)),
|
||||
(cfg_path, json.dumps(cfg, indent=2))):
|
||||
tmp = path + ".tmp"
|
||||
with open(tmp, "w") as fh:
|
||||
fh.write(payload)
|
||||
os.replace(tmp, path)
|
||||
n = set_split(cfg, 1)
|
||||
with open(cfg_path, "w") as fh:
|
||||
json.dump(cfg, fh, indent=2)
|
||||
log(f"config: split_ngram_parts=1 written at {n} location(s)")
|
||||
|
||||
with open(os.path.join(stage, "ple_scale.txt"), "w") as fh:
|
||||
|
||||
Reference in New Issue
Block a user