frank-primary: wire --decode-context-parallel-size (opt-in)
DCP is passed only when FRANK_PRIMARY_DCP is set, so the deployed shape is unchanged. Verified in this fork: DCP reuses the TP ranks when PCP is disabled (tp % dcp == 0), so TP=2 + DCP=2 fits the two R9700s with no extra ranks; the flash_attn backend carries DCP support, the 36 linear-attention layers of this hybrid do not — which is what the A/B window tests.
This commit is contained in:
@@ -111,6 +111,7 @@ podman run -d --name "$FRANK_PRIMARY_CONTAINER" \
|
||||
"$FRANK_PRIMARY_IMAGE" /models \
|
||||
--served-model-name "$FRANK_PRIMARY_SERVED_NAME" --port 7770 \
|
||||
--tensor-parallel-size 2 \
|
||||
${FRANK_PRIMARY_DCP:+--decode-context-parallel-size "$FRANK_PRIMARY_DCP"} \
|
||||
--max-model-len "$FRANK_PRIMARY_MAXLEN" \
|
||||
--max-num-seqs "$FRANK_PRIMARY_SEQS" \
|
||||
${FRANK_PRIMARY_MNBT:+--max-num-batched-tokens "$FRANK_PRIMARY_MNBT"} \
|
||||
|
||||
Reference in New Issue
Block a user