frank-primary: wire --decode-context-parallel-size (opt-in)
DCP is passed only when FRANK_PRIMARY_DCP is set, so the deployed shape is unchanged. Verified in this fork: DCP reuses the TP ranks when PCP is disabled (tp % dcp == 0), so TP=2 + DCP=2 fits the two R9700s with no extra ranks; the flash_attn backend carries DCP support, the 36 linear-attention layers of this hybrid do not — which is what the A/B window tests.
This commit is contained in:
1 parent
667ec2987c
commit
148a5f2575
2 files changed
+5
No files matched your search
@@ -111,6 +111,7 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
|||||||
"$FRANK_PRIMARY_IMAGE" /models \
|
"$FRANK_PRIMARY_IMAGE" /models \
|
||||||
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
|
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
|
||||||
--tensor-parallel-size 2 \
|
--tensor-parallel-size 2 \
|
||||||
|
${FRANK_PRIMARY_DCP:+--decode-context-parallel-size "$FRANK_PRIMARY_DCP"} \
|
||||||
--max-model-len "$FRANK_PRIMARY_MAXLEN" \
|
--max-model-len "$FRANK_PRIMARY_MAXLEN" \
|
||||||
--max-num-seqs "$FRANK_PRIMARY_SEQS" \
|
--max-num-seqs "$FRANK_PRIMARY_SEQS" \
|
||||||
${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
|
${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
|
||||||
|
|||||||
@@ -113,6 +113,10 @@ DOWNLOAD="$REPO_ROOT/scripts/download-frank-primary.sh"
|
|||||||
grep -q -- '--tensor-parallel-size 2' "$SERVE"
|
grep -q -- '--tensor-parallel-size 2' "$SERVE"
|
||||||
grep -q -- '--enable-prefix-caching' "$SERVE"
|
grep -q -- '--enable-prefix-caching' "$SERVE"
|
||||||
grep -q -- '--trust-remote-code' "$SERVE"
|
grep -q -- '--trust-remote-code' "$SERVE"
|
||||||
|
# DCP is opt-in: the flag is passed only when FRANK_PRIMARY_DCP is set
|
||||||
|
# (DCP reuses the TP ranks when PCP is off, so TP=2 + DCP=2 fits two GPUs).
|
||||||
|
grep -q -- '--decode-context-parallel-size "$FRANK_PRIMARY_DCP"' "$SERVE"
|
||||||
|
! grep -q '^FRANK_PRIMARY_DCP=' "$ENV_FILE"
|
||||||
}
|
}
|
||||||
|
|
||||||
@test "tools and reasoning are wired — the client contract is served" {
|
@test "tools and reasoning are wired — the client contract is served" {
|
||||||
|
|||||||
Reference in new issue
Block a user