dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-primary: pin the drafter to ROCm1 (the last-layers card)

Browse Source 
The shared MTP head borrows the main model output head, which a -ts 1,1
layer split places on ROCm1. With tensor overrides set, n_copies==1 in the
draft's scheduler, and a borrowed output.weight in a ROCm1 buffer aborts the
draft graph (ggml-backend.cpp:941) when the draft context is on ROCm0 —
the unit crash-looped 4x at ~12 s per attempt during the 2026-09-26 cutover.
Probed live on the host: ROCm0 aborts identically, ROCm1 loads and serves.
This commit is contained in:
Frank
2026-09-26 20:49:04 -07:00
parent a9b94a79eb
commit 5d632bbaea
1 changed files with 9 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
scripts/frank-primary.env
+9 -3
View File
@@ -62,9 +62,15 @@ NGL=999
OT_OVERRIDES=exps=CPU,per_layer_token_embd.weight=CPU
TENSOR_SPLIT=1,1
# The drafter lives on the first R9700 (the card the load log shows holding
# the shared embedding borrow path).
SPEC_DRAFT_DEVICE=ROCm0
# The drafter MUST sit on the SECOND R9700 (ROCm1): with -ts 1,1 the main
# model's output head lands on the last-layers card, and the shared MTP head
# BORROWS it. --override-tensor sets has_tensor_overrides(), which pins
# n_copies==1 in the draft's scheduler — a borrowed output.weight sitting in
# a ROCm1 buffer then aborts the draft graph with ggml-backend.cpp:941
# "pre-allocated tensor (output.weight) in a buffer (ROCm1) that cannot run
# the operation" when the draft is elsewhere (verified live 2026-09-26: the
# ROCm0 placement crash-looped 4x at ~12 s, ROCm1 loaded and served).
SPEC_DRAFT_DEVICE=ROCm1
# Batch geometry: logical batch 8192 with ubatch 512 — the measured optimum
# class on this fleet's ROCm stack (a bigger logical batch is free; compute