frank-medium: swap to ParoQuant MXFP6 (Swift Qwen3.8-27B) + Qwen-Sharp v22.5.0
- Model: hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (25.16 GiB, 18 shards, quant_method=paroquant_mxfp6, bits=6, group_size=128, krot=8, rotation on) - Image: ggz14/vllm-radiance-mxfp4:0.13.0-e060895 (PR #51 launch6_p kernel, built on the published 0.9.3 base; docker->podman via save/load) - GPU_UTIL 0.98 -> 0.99 (user request); KV_DTYPE=auto (unquantized bf16) - Entrypoint: mxfp6 branch (17 patches + paroquant kernel compile + sitecustomize registration); GPU_UTIL default for set -u robustness - Quadlet: SNAP/GPU_UTIL/Image/cache updated; new mxfp6 cache dir - deploy-remote.sh: R4D gate + paroquant_mxfp6; new image/SNAP/cache defaults - Chat template: Qwen-Sharp v22.5.0 (peculiar-ragdoll); v22.4.0 kept as rollback - Test 168: dflash branch comment quant-aware (mxfp4 + paroquant_mxfp6) - Runbook: MXFP6 active shape, MXFP4 rollback MXFP4 checkpoint remains on disk as the rollback target.
This commit is contained in:
Frank
@@ -4,18 +4,18 @@
|
||||
# Unit name: frank-medium.service (container name: frank-medium)
|
||||
#
|
||||
# The dankserver LLM endpoint (:7770, alias frank-medium): the vllm-radiance
|
||||
# stack (stilldeadcode/vllm-radiance:0.9.3 image) serving
|
||||
# just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4 / quark, W4A8, 19.37
|
||||
# GiB) on the 2x Radeon AI PRO R9700 (gfx1201) as ONE tensor-parallel process
|
||||
# stack (the 0.13.0-e060895 image, PR #51) serving
|
||||
# hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6, W4A8, 25.16 GiB)
|
||||
# on the 2x Radeon AI PRO R9700 (gfx1201) as ONE tensor-parallel process
|
||||
# (TP=2, R4D one-shot all-reduce) with the z-lab DFlash2 external drafter
|
||||
# (SPEC=7) and UNQUANTIZED bf16 KV (KV_DTYPE=auto).
|
||||
# The entrypoint is QUANTIZATION-AWARE: it reads the model's weight
|
||||
# quantization from config.json and serves the plain FP8 path (no MXFP4
|
||||
# patches, --quantization=fp8, ROCM_AITER unified attention) or the MXFP4
|
||||
# patch-and-serve path (the DFlash2 drafter). A model swap is now: change
|
||||
# SNAP (+ SPEC_METHOD/SPEC), restart. The FP8 checkpoint
|
||||
# (orcarouter/Qwen3.8-27B-Uncensored-FP8) is the on-disk rollback target — the
|
||||
# 2026-09-20 to 2026-09-21 serving shape, before this MXFP4 + clean-KV cut.
|
||||
# patches, --quantization=fp8, ROCM_AITER unified attention), the MXFP4
|
||||
# patch-and-serve path, or the ParoQuant MXFP6 stack (PR #51's launch6_p
|
||||
# kernel). A model swap is now: change SNAP (+ SPEC_METHOD/SPEC), restart.
|
||||
# The MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq, the
|
||||
# 2026-09-20 to 2026-09-21 serving shape) is the on-disk rollback target.
|
||||
#
|
||||
# SHAPE. The launcher the upstream repo ships binds the server with
|
||||
# --network=host (vLLM takes --host 0.0.0.0 --port 7770 directly), which is
|
||||
@@ -58,19 +58,19 @@
|
||||
# docker.io ref and re-tags it before the restart. Pull=never keeps boots off
|
||||
# the registry.
|
||||
#
|
||||
# KEEP IN SYNC: the radiance clone is pinned (97353293...); if it is updated,
|
||||
# KEEP IN SYNC: the radiance clone is pinned (e060895...); if it is updated,
|
||||
# re-diff dankserver/frank-medium/radiance-entrypoint.sh against the new
|
||||
# serve-mxfp4.sh container command and bump the pin deliberately.
|
||||
#
|
||||
# DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the
|
||||
# HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident).
|
||||
[Unit]
|
||||
Description=frank-medium — vllm-radiance MXFP4 Qwen3.8-27B-Uncensored on 2x R9700 (gfx1201)
|
||||
Description=frank-medium — vllm-radiance ParoQuant MXFP6 Swift Qwen3.8-27B on 2x R9700 (gfx1201)
|
||||
After=network-online.target
|
||||
|
||||
[Container]
|
||||
ContainerName=frank-medium
|
||||
Image=localhost/radiance-vllm:0.9.3
|
||||
Image=localhost/radiance-vllm:0.13.0-e060895
|
||||
# vLLM binds 0.0.0.0:7770 on the host network directly; no PublishPort.
|
||||
AddDevice=/dev/kfd
|
||||
AddDevice=/dev/dri
|
||||
@@ -85,14 +85,14 @@ Environment=HIP_VISIBLE_DEVICES=0,1
|
||||
# Drafting: the external z-lab DFlash2-FP8 drafter (SPEC_METHOD=dflash). The
|
||||
# plain-FP8 checkpoint (rollback) carries its own single-layer MTP head, but
|
||||
# DFlash2 beat MTP self-speculation on decode, so the drafter serves either.
|
||||
# Chat template: the Qwen-Sharp variant (froggeric v22.4.0) — the terse
|
||||
# Chat template: the Qwen-Sharp variant (peculiar-ragdoll v22.5.0) — the terse
|
||||
# system prompt (opt out per-request with chat_template_kwargs {"terse":
|
||||
# false}), message.reasoning support (vLLM's thinking field), and the
|
||||
# tightened tool-call rendering. The predecessor (qwen-fixed-v22.3.jinja in
|
||||
# the /patches clone) remains there as the rollback target: flip CHAT_TEMPLATE.
|
||||
Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq
|
||||
# tightened tool-call rendering. The predecessor (qwen-sharp-v22.4.0.jinja,
|
||||
# mounted beside it) remains as the rollback target: flip CHAT_TEMPLATE.
|
||||
Environment=SNAP=/models/Qwen3.8-27B-PARO-MXFP6
|
||||
Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8
|
||||
Environment=CHAT_TEMPLATE=/templates/qwen-sharp-v22.4.0.jinja
|
||||
Environment=CHAT_TEMPLATE=/templates/qwen-sharp-v22.5.0.jinja
|
||||
# Fleet identity: the id at /v1/models and in every response body. Every
|
||||
# client (Hermes, the harness, services.json) points at frank-medium.
|
||||
Environment=SERVED_NAMES=frank-medium
|
||||
@@ -116,7 +116,7 @@ Environment=MAXLEN=262144
|
||||
# through the Gemma4 merge path and it is not MXFP4-quantized, so vLLM serves
|
||||
# image input on either checkpoint. harness models.yml keeps frank-medium
|
||||
# input: [text, image].
|
||||
Environment=GPU_UTIL=0.98
|
||||
Environment=GPU_UTIL=0.99
|
||||
# Speculative decoding: DFlash2 external drafter at depth 7. A/B tested
|
||||
# against MTP self-speculation (depth 8) on this FP8 checkpoint: DFlash2
|
||||
# wins on decode (100.0 vs 85.9 t/s geomean, 287.1 vs 179.7 t/s burst-4)
|
||||
@@ -144,7 +144,7 @@ Volume=/home/dank/models:/models
|
||||
Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface
|
||||
# Compile cache. Keyed on model + stack version + every graph-changing knob;
|
||||
# never share it across configurations (upstream rule).
|
||||
Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache
|
||||
Volume=/home/dank/.radiance-cache-mxfp6-013-pq-nqft-fp8s-gnq:/cache
|
||||
# The pinned radiance clone: patches applied at every container start.
|
||||
Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z
|
||||
# The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The
|
||||
@@ -154,6 +154,7 @@ Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/qwen-sharp-v22.4.0.jinja:/templates/qwen-sharp-v22.4.0.jinja:ro
|
||||
Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/qwen-sharp-v22.5.0.jinja:/templates/qwen-sharp-v22.5.0.jinja:ro
|
||||
# The entrypoint must be the container command; the image's own entrypoint
|
||||
# (which the patched script execs at the end) must not run unpatched.
|
||||
PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh
|
||||
|
||||
@@ -0,0 +1,488 @@
|
||||
{%- set template_version = "qwen3.8-froggeric-v22.5.0" %}
|
||||
{%- set _tool_format = tool_call_format if tool_call_format is defined else 'xml' %}
|
||||
{%- set image_count = namespace(value=0) %}
|
||||
{%- set video_count = namespace(value=0) %}
|
||||
{%- set add_vision_id = add_vision_id if add_vision_id is defined else false %}
|
||||
{%- set enable_thinking = enable_thinking if enable_thinking is defined else true %}
|
||||
{%- set auto_disable_thinking_with_tools = auto_disable_thinking_with_tools if auto_disable_thinking_with_tools is defined else false %}
|
||||
{%- if preserve_reasoning is defined and preserve_reasoning is not none %}
|
||||
{%- set _preserve_thinking = preserve_reasoning %}
|
||||
{%- elif preserve_thinking is defined and preserve_thinking is not none %}
|
||||
{%- set _preserve_thinking = preserve_thinking %}
|
||||
{%- else %}
|
||||
{%- set _preserve_thinking = true %}
|
||||
{%- endif %}
|
||||
{%- set max_tool_arg_chars = max_tool_arg_chars if max_tool_arg_chars is defined else 0 %}
|
||||
{%- set max_tool_response_chars = max_tool_response_chars if max_tool_response_chars is defined else 0 %}
|
||||
{%- set _default_reasoning_effort = 'medium' %}
|
||||
{%- set _has_tools = (tools is defined and tools and tools is iterable and tools is not mapping) %}
|
||||
{%- set _effort_raw = (reasoning_effort | string | lower) if reasoning_effort is defined and reasoning_effort is not none else _default_reasoning_effort %}
|
||||
{%- set _initial_thinking = enable_thinking %}
|
||||
{%- if _effort_raw in ('none', 'off') %}
|
||||
{%- set _initial_thinking = false %}
|
||||
{%- set _initial_effort = 'medium' %}
|
||||
{%- elif _effort_raw in ('minimal', 'low') %}
|
||||
{%- set _initial_effort = 'low' %}
|
||||
{%- elif _effort_raw in ('high', 'xhigh', 'max', 'ultracode', 'extreme') %}
|
||||
{%- set _initial_effort = 'xhigh' %}
|
||||
{%- else %}
|
||||
{%- set _initial_effort = 'medium' %}
|
||||
{%- endif %}
|
||||
{%- set ns_state = namespace(thinking=_initial_thinking, effort=_initial_effort) %}
|
||||
{%- if auto_disable_thinking_with_tools and _has_tools %}
|
||||
{%- set ns_state.thinking = false %}
|
||||
{%- endif %}
|
||||
{%- for msg in messages %}
|
||||
{%- if msg.role == 'system' or msg.role == 'developer' or msg.role == 'user' %}
|
||||
{%- if msg.content is string %}
|
||||
{%- if '<|think_off|>' in msg.content %}
|
||||
{%- set ns_state.thinking = false %}
|
||||
{%- elif '<|think_on|>' in msg.content %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- elif '<|think_xhigh|>' in msg.content or '<|think_high|>' in msg.content or '<|think_ultracode|>' in msg.content or '<|think_extreme|>' in msg.content or '<|think_max|>' in msg.content %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'xhigh' %}
|
||||
{%- elif '<|think_low|>' in msg.content or '<|think_minimal|>' in msg.content %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'low' %}
|
||||
{%- elif '<|think_medium|>' in msg.content %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'medium' %}
|
||||
{%- endif %}
|
||||
{%- elif msg.content is iterable and msg.content is not mapping %}
|
||||
{%- for item in msg.content %}
|
||||
{%- if item is string %}
|
||||
{%- set _item_text = item %}
|
||||
{%- elif item is mapping and 'text' in item and item.text is string %}
|
||||
{%- set _item_text = item.text %}
|
||||
{%- else %}
|
||||
{%- set _item_text = '' %}
|
||||
{%- endif %}
|
||||
{%- if _item_text %}
|
||||
{%- if '<|think_off|>' in _item_text %}
|
||||
{%- set ns_state.thinking = false %}
|
||||
{%- elif '<|think_on|>' in _item_text %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- elif '<|think_xhigh|>' in _item_text or '<|think_high|>' in _item_text or '<|think_ultracode|>' in _item_text or '<|think_extreme|>' in _item_text or '<|think_max|>' in _item_text %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'xhigh' %}
|
||||
{%- elif '<|think_low|>' in _item_text or '<|think_minimal|>' in _item_text %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'low' %}
|
||||
{%- elif '<|think_medium|>' in _item_text %}
|
||||
{%- set ns_state.thinking = true %}
|
||||
{%- set ns_state.effort = 'medium' %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set reasoning_instructions = '' %}
|
||||
{%- if ns_state.thinking %}
|
||||
{%- if ns_state.effort == 'xhigh' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}
|
||||
{%- elif ns_state.effort == 'low' %}
|
||||
{%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
|
||||
{%- if content is string %}
|
||||
{{- content }}
|
||||
{%- elif content is iterable and content is not mapping %}
|
||||
{%- for item in content %}
|
||||
{%- if item is mapping %}
|
||||
{%- if item.type == 'image' or 'image' in item or 'image_url' in item %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain images.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set image_count.value = image_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Picture ' ~ image_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
|
||||
{%- elif item.type == 'video' or 'video' in item or 'video_url' in item %}
|
||||
{%- if is_system_content %}
|
||||
{{- raise_exception('System message cannot contain videos.') }}
|
||||
{%- endif %}
|
||||
{%- if do_vision_count %}
|
||||
{%- set video_count.value = video_count.value + 1 %}
|
||||
{%- endif %}
|
||||
{%- if add_vision_id %}
|
||||
{{- 'Video ' ~ video_count.value ~ ': ' }}
|
||||
{%- endif %}
|
||||
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
|
||||
{%- elif 'text' in item %}
|
||||
{{- item.text }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected item type in content.') }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- item | string }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- elif content is none or content is undefined %}
|
||||
{{- '' }}
|
||||
{%- else %}
|
||||
{{- raise_exception('Unexpected content type.') }}
|
||||
{%- endif %}
|
||||
{%- endmacro %}
|
||||
{%- if not messages %}
|
||||
{{- raise_exception('No messages provided.') }}
|
||||
{%- endif %}
|
||||
{%- set head = namespace(count=0, seen_non_system=false) %}
|
||||
{%- for message in messages %}
|
||||
{%- set _is_sys = (message.role == 'system' or message.role == 'developer') %}
|
||||
{%- if _is_sys and not head.seen_non_system %}
|
||||
{%- set head.count = head.count + 1 %}
|
||||
{%- else %}
|
||||
{%- set head.seen_non_system = true %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set sys_state = namespace(content='') %}
|
||||
{%- for message in messages[:head.count] %}
|
||||
{%- set _part = render_content(message.content, false, true) | trim %}
|
||||
{%- if '<|think_off|>' in _part %}{%- set _part = _part.split('<|think_off|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_on|>' in _part %}{%- set _part = _part.split('<|think_on|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_xhigh|>' in _part %}{%- set _part = _part.split('<|think_xhigh|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_high|>' in _part %}{%- set _part = _part.split('<|think_high|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_ultracode|>' in _part %}{%- set _part = _part.split('<|think_ultracode|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_extreme|>' in _part %}{%- set _part = _part.split('<|think_extreme|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_max|>' in _part %}{%- set _part = _part.split('<|think_max|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_medium|>' in _part %}{%- set _part = _part.split('<|think_medium|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_low|>' in _part %}{%- set _part = _part.split('<|think_low|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_minimal|>' in _part %}{%- set _part = _part.split('<|think_minimal|>') | join('') | trim %}{%- endif %}
|
||||
{%- if _part %}
|
||||
{%- if sys_state.content %}
|
||||
{%- set sys_state.content = sys_state.content ~ '\n\n' ~ _part %}
|
||||
{%- else %}
|
||||
{%- set sys_state.content = _part %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- set _sc = sys_state.content %}
|
||||
{%- set _msgs = messages[head.count:] %}
|
||||
{%- if ns_state.thinking %}
|
||||
{%- set _terse_lead = 'Answer directly, after thinking. Lead with the answer, then only what it needs to be correct and usable.' %}
|
||||
{%- else %}
|
||||
{%- set _terse_lead = 'Answer directly and concisely. Give the answer with only what it needs to be correct and usable.' %}
|
||||
{%- endif %}
|
||||
{%- set _terse_core %}
|
||||
Never: open with preamble or pleasantries; restate the question; add filler transitions; hedge with niceties; or repeat a point you've already made.
|
||||
Always: keep essential steps, caveats, uncertainties, and specifics — never drop correctness or a needed warning for brevity. Keep the final answer lean. Use the least structure that conveys it (plain prose when short; lists or code only when they earn their place). If genuinely uncertain, say so and explain why — never omit uncertainty for the sake of brevity.
|
||||
If a user request is genuinely ambiguous, ask a sharp question, don't guess.
|
||||
{%- endset %}
|
||||
{%- set _terse = _terse_lead ~ '\n' ~ (_terse_core | trim) %}
|
||||
{#- `terse` chat-template kwarg: default true, so behaviour is unchanged unless a caller opts out.
|
||||
Pass {"terse": false} via chat_template_kwargs to serve the model with only its own system
|
||||
prompt -- useful for A/B-ing the terseness effect, or when a downstream prompt conflicts. #}
|
||||
{%- set _terse_on = terse if terse is defined else true %}
|
||||
{%- if _terse_on %}
|
||||
{%- if not _sc %}
|
||||
{%- set _sc = _terse | trim %}
|
||||
{%- else %}
|
||||
{%- set _sc = (_sc | trim) ~ '\n\n' ~ (_terse | trim) %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{#- Some runtimes (LM Studio's MLX backend) have no native tool-call parser and prepend their own
|
||||
protocol into the system content. Emitting ours as well gives the model two contradictory tool
|
||||
protocols and it follows one at random. Detect that marker and stand down; the
|
||||
`suppress_tool_instructions` kwarg overrides either way where a runtime can pass template kwargs. #}
|
||||
{%- set _runtime_tool_protocol = (_sc is defined and _sc and '[TOOL_REQUEST]' in _sc) %}
|
||||
{%- set _suppress_tools = suppress_tool_instructions if suppress_tool_instructions is defined else _runtime_tool_protocol %}
|
||||
{%- if _has_tools and not _suppress_tools %}
|
||||
{{- '<|im_start|>system\n' }}
|
||||
{%- if reasoning_instructions %}
|
||||
{{- reasoning_instructions + '\n\n' }}
|
||||
{%- endif %}
|
||||
{{- '# Tools\n\nYou have access to the following functions:\n\n<tools>' }}
|
||||
{%- for tool in tools %}
|
||||
{{- '\n' }}
|
||||
{{- tool | tojson }}
|
||||
{%- endfor %}
|
||||
{{- '\n</tools>' }}
|
||||
{%- if _tool_format == 'json' %}
|
||||
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }}
|
||||
{%- if ns_state.thinking %}
|
||||
{{- '<think>\nBrief explanation of tool call\n</think>\n' }}
|
||||
{%- endif %}
|
||||
{{- '<tool_call>\n{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1", "example_parameter_2": "This is the value for the second parameter"}}\n</tool_call>\n\n<IMPORTANT>\nReminder:\n' }}
|
||||
{%- if ns_state.thinking %}
|
||||
{{- '- You can use the <think></think> block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the <think></think> block.\n' }}
|
||||
{%- endif %}
|
||||
{{- '- Function calls MUST follow the specified format: a single JSON object with "name" and "arguments" keys inside <tool_call></tool_call> XML tags.\n' }}
|
||||
{%- if ns_state.thinking %}
|
||||
{{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY after thinking, with NO conversational text before it.\n' }}
|
||||
{%- else %}
|
||||
{{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY, with NO conversational text before it.\n' }}
|
||||
{%- endif %}
|
||||
{{- '- The <tool_call> tag MUST be at the very beginning of a new line, with NO spaces or indentation before it.\n- To call multiple functions, output a separate, completely closed <tool_call></tool_call> block for EACH function. Do NOT nest <tool_call> blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n</IMPORTANT>' }}
|
||||
{%- else %}
|
||||
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }}
|
||||
{%- if ns_state.thinking %}
|
||||
{{- '<think>\nBrief explanation of tool call\n</think>\n' }}
|
||||
{%- endif %}
|
||||
{{- '<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n' }}
|
||||
{%- if ns_state.thinking %}
|
||||
{{- '- You can use the <think></think> block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the <think></think> block.\n' }}
|
||||
{%- endif %}
|
||||
{{- '- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags.\n' }}
|
||||
{%- if ns_state.thinking %}
|
||||
{{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY after thinking, with NO conversational text before it.\n' }}
|
||||
{%- else %}
|
||||
{{- '- If you choose to call a tool, you MUST output the <tool_call> block IMMEDIATELY, with NO conversational text before it.\n' }}
|
||||
{%- endif %}
|
||||
{{- '- The <tool_call> and <function> tags MUST be at the very beginning of a new line, with NO spaces or indentation before them.\n- To call multiple functions, output a separate, completely closed <tool_call></tool_call> block for EACH function. Do NOT nest <tool_call> blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n</IMPORTANT>' }}
|
||||
{%- endif %}
|
||||
{%- if _sc %}
|
||||
{{- '\n\n' + _sc }}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{%- if _sc %}
|
||||
{{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + _sc + '<|im_end|>\n' }}
|
||||
{%- elif reasoning_instructions %}
|
||||
{{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set _last_idx = _msgs | length - 1 %}
|
||||
{%- set ns = namespace(multi_step_tool=true, last_query_index=_last_idx) %}
|
||||
{%- for message in _msgs[::-1] %}
|
||||
{%- set index = (_msgs | length - 1) - loop.index0 %}
|
||||
{%- if ns.multi_step_tool and message.role == 'user' %}
|
||||
{%- set _rc = render_content(message.content, false) | trim %}
|
||||
{%- if not (_rc.startswith('<tool_response>') and _rc.endswith('</tool_response>')) %}
|
||||
{%- set ns.multi_step_tool = false %}
|
||||
{%- set ns.last_query_index = index %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- if ns.multi_step_tool %}
|
||||
{%- if _last_idx > 50 %}
|
||||
{%- set ns.last_query_index = _last_idx %}
|
||||
{%- else %}
|
||||
{%- set ns.last_query_index = 0 %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set ns2 = namespace(prev_role='', consecutive_failures=0) %}
|
||||
{%- for message in _msgs %}
|
||||
{%- set is_system = (message.role == "system" or message.role == "developer") %}
|
||||
{%- set content = render_content(message.content, true, is_system) | trim %}
|
||||
{%- if is_system or message.role == 'user' %}
|
||||
{%- if '<|think_off|>' in content %}{%- set content = content.split('<|think_off|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_on|>' in content %}{%- set content = content.split('<|think_on|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_xhigh|>' in content %}{%- set content = content.split('<|think_xhigh|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_high|>' in content %}{%- set content = content.split('<|think_high|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_ultracode|>' in content %}{%- set content = content.split('<|think_ultracode|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_extreme|>' in content %}{%- set content = content.split('<|think_extreme|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_max|>' in content %}{%- set content = content.split('<|think_max|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_medium|>' in content %}{%- set content = content.split('<|think_medium|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_low|>' in content %}{%- set content = content.split('<|think_low|>') | join('') | trim %}{%- endif %}
|
||||
{%- if '<|think_minimal|>' in content %}{%- set content = content.split('<|think_minimal|>') | join('') | trim %}{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if is_system %}
|
||||
{{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
|
||||
{%- elif message.role == 'user' %}
|
||||
{%- set ns2.consecutive_failures = 0 %}
|
||||
{{- '<|im_start|>user\n' + content + '<|im_end|>\n' }}
|
||||
{%- elif message.role == 'assistant' %}
|
||||
{%- set reasoning_content = '' %}
|
||||
{%- set _explicit_reasoning = '' %}
|
||||
{%- if message.reasoning_content is defined and message.reasoning_content is not none %}
|
||||
{%- if message.reasoning_content is string %}
|
||||
{%- set _explicit_reasoning = message.reasoning_content %}
|
||||
{%- else %}
|
||||
{%- set _explicit_reasoning = message.reasoning_content | string %}
|
||||
{%- endif %}
|
||||
{%- elif message.thinking is defined and message.thinking is not none %}
|
||||
{%- if message.thinking is string %}
|
||||
{%- set _explicit_reasoning = message.thinking %}
|
||||
{%- else %}
|
||||
{%- set _explicit_reasoning = message.thinking | string %}
|
||||
{%- endif %}
|
||||
{%- elif message.reasoning is defined and message.reasoning is not none %}
|
||||
{%- if message.reasoning is string %}
|
||||
{%- set _explicit_reasoning = message.reasoning %}
|
||||
{%- else %}
|
||||
{%- set _explicit_reasoning = message.reasoning | string %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- if _explicit_reasoning %}
|
||||
{%- set _lead_end = '' %}
|
||||
{%- if content.startswith('<think>') and '</think>' in content %}
|
||||
{%- set _lead_end = '</think>' %}
|
||||
{%- elif content.startswith('<thinking>') and '</thinking>' in content %}
|
||||
{%- set _lead_end = '</thinking>' %}
|
||||
{%- elif content.startswith('</think>') %}
|
||||
{%- set _lead_end = '</think>' %}
|
||||
{%- elif content.startswith('</thinking>') %}
|
||||
{%- set _lead_end = '</thinking>' %}
|
||||
{%- endif %}
|
||||
{%- if _lead_end %}
|
||||
{%- set content = content.split(_lead_end)[-1].lstrip('\n') %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = _explicit_reasoning %}
|
||||
{%- else %}
|
||||
{%- set _think_end = '' %}
|
||||
{%- if content.startswith('</think>') %}
|
||||
{%- set _think_end = '</think>' %}
|
||||
{%- elif content.startswith('</thinking>') %}
|
||||
{%- set _think_end = '</thinking>' %}
|
||||
{%- elif '\n</think>' in content %}
|
||||
{%- set _think_end = '\n</think>' %}
|
||||
{%- elif '\n</thinking>' in content %}
|
||||
{%- set _think_end = '\n</thinking>' %}
|
||||
{%- elif '\n</ think>' in content %}
|
||||
{%- set _think_end = '\n</ think>' %}
|
||||
{%- elif '\n</think >' in content %}
|
||||
{%- set _think_end = '\n</think >' %}
|
||||
{%- elif content.startswith('<think>') and '</think>' in content %}
|
||||
{%- set _think_end = '</think>' %}
|
||||
{%- elif content.startswith('<thinking>') and '</thinking>' in content %}
|
||||
{%- set _think_end = '</thinking>' %}
|
||||
{%- endif %}
|
||||
{%- if _think_end %}
|
||||
{%- if 'thinking' in _think_end %}
|
||||
{%- set _think_start = '<thinking>' %}
|
||||
{%- else %}
|
||||
{%- set _think_start = '<think>' %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = content.split(_think_end)[0].rstrip('\n') %}
|
||||
{%- if _think_start in reasoning_content %}
|
||||
{%- set reasoning_content = reasoning_content.split(_think_start)[-1].lstrip('\n') %}
|
||||
{%- endif %}
|
||||
{%- set content = content.split(_think_end)[-1].lstrip('\n') %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- set reasoning_content = reasoning_content | trim %}
|
||||
{%- if (_preserve_thinking or loop.index0 > ns.last_query_index) %}
|
||||
{{- '<|im_start|>assistant\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>assistant\n' + content }}
|
||||
{%- endif %}
|
||||
{%- if message.tool_calls is defined and message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
|
||||
{%- for tool_call in message.tool_calls %}
|
||||
{%- if tool_call.function is defined and tool_call.function is not none %}
|
||||
{%- set tc = tool_call.function %}
|
||||
{%- else %}
|
||||
{%- set tc = tool_call %}
|
||||
{%- endif %}
|
||||
{%- set tc_name = tc.name if (tc.name is defined and tc.name is not none) else '' %}
|
||||
{%- if _tool_format == 'json' %}
|
||||
{%- if loop.first %}
|
||||
{%- if content | trim %}
|
||||
{{- '\n\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '\n' }}
|
||||
{%- endif %}
|
||||
{%- set _args = '{}' %}
|
||||
{%- if tc.arguments is defined and tc.arguments is not none %}
|
||||
{%- if tc.arguments is mapping %}
|
||||
{%- set _args = tc.arguments | tojson %}
|
||||
{%- elif tc.arguments is string %}
|
||||
{%- if tc.arguments %}
|
||||
{%- set _args = tc.arguments %}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{%- set _args = tc.arguments | tojson %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{{- '<tool_call>\n{"name": ' }}{{- tc_name | tojson }}{{- ', "arguments": ' }}{{- _args }}{{- '}\n</tool_call>' }}
|
||||
{%- else %}
|
||||
{%- if loop.first %}
|
||||
{%- if content | trim %}
|
||||
{{- '\n\n<tool_call>\n<function=' + tc_name + '>\n' }}
|
||||
{%- else %}
|
||||
{{- '<tool_call>\n<function=' + tc_name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '\n<tool_call>\n<function=' + tc_name + '>\n' }}
|
||||
{%- endif %}
|
||||
{%- if tc.arguments is defined and tc.arguments is not none %}
|
||||
{%- if tc.arguments is mapping %}
|
||||
{%- for args_name, args_value in tc.arguments.items() %}
|
||||
{{- '<parameter=' + args_name + '>\n' }}
|
||||
{%- if args_value is string %}
|
||||
{%- set _av = args_value %}
|
||||
{%- else %}
|
||||
{%- set _av = args_value | tojson %}
|
||||
{%- endif %}
|
||||
{%- if max_tool_arg_chars > 0 and _av | length > max_tool_arg_chars %}
|
||||
{{- _av[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_av | length | string) ~ ' chars]' }}
|
||||
{%- else %}
|
||||
{{- _av }}
|
||||
{%- endif %}
|
||||
{{- '\n</parameter>\n' }}
|
||||
{%- endfor %}
|
||||
{%- else %}
|
||||
{%- if tc.arguments is string %}
|
||||
{%- set _raw_args = tc.arguments %}
|
||||
{%- else %}
|
||||
{%- set _raw_args = tc.arguments | tojson %}
|
||||
{%- endif %}
|
||||
{%- if _raw_args %}
|
||||
{%- if max_tool_arg_chars > 0 and _raw_args | length > max_tool_arg_chars %}
|
||||
{{- _raw_args[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_raw_args | length | string) ~ ' chars]' }}
|
||||
{%- else %}
|
||||
{{- _raw_args }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{{- '</function>\n</tool_call>' }}
|
||||
{%- endif %}
|
||||
{%- endfor %}
|
||||
{%- endif %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- elif message.role == 'tool' %}
|
||||
{%- set _content_lower = content | lower %}
|
||||
{%- set _content_head = _content_lower[:120] %}
|
||||
{%- set _is_code_or_grep = ('throw new ' in _content_lower or 'throw error' in _content_lower or 'console.error' in _content_lower or 'logger.error' in _content_lower or 'logging.error' in _content_lower or 'import ' in _content_head or 'def ' in _content_head or 'function ' in _content_head) %}
|
||||
{%- set _exit_code_zero = ('exit code: 0' in _content_head or 'process exited with code 0' in _content_head) %}
|
||||
{%- set _error_field_ok = ('"error": null' in _content_head or '"error":null' in _content_head or '"error": false' in _content_head or '"error":false' in _content_head or '"error": ""' in _content_head or '"error":""' in _content_head) %}
|
||||
{%- set _strong_error = (('"error":' in _content_head and not _error_field_ok) or '"status": "error"' in _content_head or '"status":"error"' in _content_head or 'traceback (most recent call last):' in _content_head or 'command not found' in _content_head or 'invalid syntax' in _content_head or 'fatal:' in _content_head or (('exit code: ' in _content_head or 'process exited with code' in _content_head) and not _exit_code_zero) or _content_head.startswith('exception:') or _content_head.startswith('failed to ')) %}
|
||||
{%- set _weak_error = ('error:' in _content_head or 'err!' in _content_head) %}
|
||||
{%- set _weak_suppressed = ('$ ' in _content_head or 'took ' in _content_head or content | length >= 600) %}
|
||||
{%- if not _is_code_or_grep and (_strong_error or (_weak_error and not _weak_suppressed)) %}
|
||||
{%- set ns2.consecutive_failures = ns2.consecutive_failures + 1 %}
|
||||
{%- else %}
|
||||
{%- set ns2.consecutive_failures = 0 %}
|
||||
{%- endif %}
|
||||
{%- if ns2.prev_role != 'tool' %}
|
||||
{{- '<|im_start|>user' }}
|
||||
{%- endif %}
|
||||
{%- set _is_json_payload = (_tool_format == 'json' and content | trim | length > 0 and (content | trim)[:1] in ('{', '[')) %}
|
||||
{%- if not _is_json_payload and max_tool_response_chars > 0 and content | length > max_tool_response_chars %}
|
||||
{%- set content = content[:max_tool_response_chars] + '\n[TRUNCATED - original length ' ~ (content | length | string) ~ ' chars]' %}
|
||||
{%- endif %}
|
||||
{{- '\n<tool_response>\n' + content }}
|
||||
{%- if ns2.consecutive_failures >= 2 %}
|
||||
{{- '\n\n⚠️ SYSTEM WARNING: ' ~ ns2.consecutive_failures ~ ' consecutive tool errors detected. Your previous approach is incorrect. You MUST use a fundamentally different approach or corrected arguments.' }}
|
||||
{%- elif ns2.consecutive_failures == 1 %}
|
||||
{{- '\n\n⚠️ SYSTEM WARNING: The previous tool call returned an error. Diagnose the failure and retry with completely corrected arguments.' }}
|
||||
{%- endif %}
|
||||
{{- '\n</tool_response>' }}
|
||||
{%- if loop.last %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- else %}
|
||||
{%- set _next_role = _msgs[loop.index0 + 1].role %}
|
||||
{%- if _next_role != 'tool' %}
|
||||
{{- '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
{%- else %}
|
||||
{{- '<|im_start|>user\n[' + message.role + ']: ' + content + '<|im_end|>\n' }}
|
||||
{%- endif %}
|
||||
{%- set ns2.prev_role = message.role %}
|
||||
{%- endfor %}
|
||||
{%- if add_generation_prompt %}
|
||||
{{- '<|im_start|>assistant\n' }}
|
||||
{%- if not ns_state.thinking %}
|
||||
{{- '<think>\n\n</think>\n\n' }}
|
||||
{%- else %}
|
||||
{{- '<think>\n' }}
|
||||
{%- endif %}
|
||||
{%- endif %}
|
||||
@@ -1,6 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
# frank-medium container entrypoint: vllm-radiance MXFP4 on 2x Radeon AI PRO
|
||||
# R9700 (gfx1201), serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq.
|
||||
# frank-medium container entrypoint: vllm-radiance on 2x Radeon AI PRO R9700
|
||||
# (gfx1201). Quantization-aware (config.json quant_method): fp8 -> the plain
|
||||
# FP8 path; mxfp4/quark -> the MXFP4 patch-and-serve path; paroquant_mxfp6 ->
|
||||
# the ParoQuant W4A8 stack + the MXFP6 kernels (PR #51's launch6_p path).
|
||||
#
|
||||
# This is a fork of the patch-and-serve block inside radiance-vllm-mxfp4's
|
||||
# serve-mxfp4.sh (clone at /home/dank/GitPackages/radiance-vllm-mxfp4, pinned
|
||||
@@ -42,17 +44,25 @@ if [ "$SPEC_METHOD" = dflash ]; then : "${SPEC:=7}"; else : "${SPEC:=4}"; fi
|
||||
: "${MIN_M:=0}" # 0 = the W4A8 kernel serves every M (MIN_M=1 still routes M=1 to aiter)
|
||||
: "${R4D_ATTN:=1}"
|
||||
: "${ASYNC:=0}"
|
||||
: "${GPU_UTIL:=0.99}"
|
||||
|
||||
# Quantization-aware: read the model's weight quantization from config.json.
|
||||
# fp8 -> the plain FP8 path (no MXFP4 patches, --quantization=fp8, MTP
|
||||
# self-speculation, ROCM_AITER unified attention with the R4D all-reduce).
|
||||
# mxfp4/quark (or unknown) -> the MXFP4 patch-and-serve path (the default).
|
||||
QUANT="$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1])).get("quantization_config",{}).get("quant_method",""))' "$SNAP/config.json" 2>/dev/null || true)"
|
||||
IS_MXFP6=0
|
||||
case "$QUANT" in
|
||||
fp8) IS_MXFP4=0 ;;
|
||||
paroquant_mxfp6) IS_MXFP4=1; IS_MXFP6=1 ;;
|
||||
mxfp4|quark|"") IS_MXFP4=1 ;;
|
||||
*) IS_MXFP4=1; echo "[radiance] WARNING: quantization '$QUANT'; defaulting to the MXFP4 path" >&2 ;;
|
||||
esac
|
||||
# MXFP6 (ParoQuant) requires TP>=2 (rad_require_tp2 in the canonical launcher);
|
||||
# ${TP:-1} keeps the check set -u safe when the quadlet omits TP.
|
||||
if [ "$IS_MXFP6" = "1" ]; then
|
||||
[ "${TP:-1}" -ge 2 ] || { echo "frank-medium: paroquant_mxfp6 requires TP>=2 (got ${TP:-1})" >&2; exit 1; }
|
||||
fi
|
||||
|
||||
if [ "$IS_MXFP4" = "0" ]; then ATTN=ROCM_AITER_UNIFIED_ATTN
|
||||
elif [ "$R4D_ATTN" = "1" ]; then ATTN=R4D
|
||||
@@ -96,6 +106,47 @@ export RADIANCE_DFLASH_SELECTOR_TOPK="${RADIANCE_DFLASH_SELECTOR_TOPK:-}"
|
||||
export RADIANCE_R4D_REPORT=1
|
||||
export RADIANCE_PRESHUFFLE=${RADIANCE_PRESHUFFLE:-1}
|
||||
export RADIANCE_MXFP4_SANITIZE="${RADIANCE_MXFP4_SANITIZE:-0}"
|
||||
# ---------------------------------------------------------------- mxfp6 profile
|
||||
# The ParoQuant W4A8 stack (PR #51, the launch6_p kernel). Mirrors
|
||||
# paroquant/run_paroquant.sh's env. Runs BEFORE the MXFP4 base block below so
|
||||
# the base block's ${VAR:-default} guards preserve these values where they
|
||||
# conflict (GDN_MERGE_INPROJ=0, MXFP4_DECODE_MAX_M=64); the base block fills
|
||||
# the rest. Gated on IS_MXFP6 so the mxfp4/fp8 paths are untouched.
|
||||
if [ "$IS_MXFP6" = "1" ]; then
|
||||
export RADIANCE_PAROQUANT=1
|
||||
export RADIANCE_GDN_MERGE_INPROJ=${RADIANCE_GDN_MERGE_INPROJ:-0}
|
||||
export RADIANCE_GDN_FUSED_MAX_ITEMS=${RADIANCE_GDN_FUSED_MAX_ITEMS:-32}
|
||||
export RADIANCE_USE_R4D_AR_QUANT=${RADIANCE_USE_R4D_AR_QUANT:-1}
|
||||
export RADIANCE_FUSE_RMS_QUANT=${RADIANCE_FUSE_RMS_QUANT:-1}
|
||||
export RADIANCE_KV_GROUP_OPT=${RADIANCE_KV_GROUP_OPT:-1}
|
||||
export RADIANCE_MXFP4_DECODE_MAX_M=${RADIANCE_MXFP4_DECODE_MAX_M:-64}
|
||||
export RADIANCE_PQ_CHECKALL=${RADIANCE_PQ_CHECKALL:-}
|
||||
export RADIANCE_PQM_CHECKALL=${RADIANCE_PQM_CHECKALL:-}
|
||||
export RADIANCE_PQM_CHECK_MAX_M=${RADIANCE_PQM_CHECK_MAX_M:-128}
|
||||
export RADIANCE_PQ_CHECK_MAX_M=${RADIANCE_PQ_CHECK_MAX_M:-128}
|
||||
export RADIANCE_PQ_DECODE_MAX_M=${RADIANCE_PQ_DECODE_MAX_M:-64}
|
||||
export RADIANCE_PQ_WPERM=${RADIANCE_PQ_WPERM:-1}
|
||||
export RADIANCE_PQ_DECODE_NT=${RADIANCE_PQ_DECODE_NT:-1}
|
||||
export RADIANCE_PQ_ATILED=${RADIANCE_PQ_ATILED:-1}
|
||||
export RADIANCE_PQ_AT_LBK=${RADIANCE_PQ_AT_LBK:-128}
|
||||
export RADIANCE_PQ_AT_HOIST=${RADIANCE_PQ_AT_HOIST:-1}
|
||||
export RADIANCE_PQ_PTOK=${RADIANCE_PQ_PTOK:-1}
|
||||
export RADIANCE_PQ_FUSED_TOKQ=${RADIANCE_PQ_FUSED_TOKQ:-1}
|
||||
export RADIANCE_PQ_I8=${RADIANCE_PQ_I8:-0}
|
||||
export RADIANCE_PQ_PG=${RADIANCE_PQ_PG:-0}
|
||||
export RADIANCE_PQ_ZPE=${RADIANCE_PQ_ZPE:-0}
|
||||
export RADIANCE_PQ_PG_PRODUCER=${RADIANCE_PQ_PG_PRODUCER:-3}
|
||||
export RADIANCE_PQ_ROT_STREAM=${RADIANCE_PQ_ROT_STREAM:-1}
|
||||
export RADIANCE_PQ_ROT_STREAM2=${RADIANCE_PQ_ROT_STREAM2:-1}
|
||||
export RADIANCE_PQ_ROT_STREAM3=${RADIANCE_PQ_ROT_STREAM3:-0}
|
||||
export RADIANCE_PQ_AR_CHECK=${RADIANCE_PQ_AR_CHECK:-0}
|
||||
export RADIANCE_PQ_AR_FALLBACK=${RADIANCE_PQ_AR_FALLBACK:-0}
|
||||
export RADIANCE_PQ_ROT_V2=${RADIANCE_PQ_ROT_V2:-1}
|
||||
export RADIANCE_PQ_HIPCC_FLAGS=${RADIANCE_PQ_HIPCC_FLAGS:-}
|
||||
export RADIANCE_PQM_FUSED_TOKQ=${RADIANCE_PQM_FUSED_TOKQ:-1}
|
||||
export RADIANCE_PQM_SINGLE_LAUNCH=${RADIANCE_PQM_SINGLE_LAUNCH:-1}
|
||||
export RADIANCE_DFLASH_CAPTURE_DIR=${RADIANCE_DFLASH_CAPTURE_DIR:-}
|
||||
fi
|
||||
if [ "$IS_MXFP4" = "1" ]; then
|
||||
export RADIANCE_MXFP4=1 RADIANCE_MXFP4_W4A8=1 RADIANCE_MXFP4_W4A8_MIN_M="$MIN_M"
|
||||
else
|
||||
@@ -193,39 +244,95 @@ if [ "$IS_MXFP4" = "1" ]; then
|
||||
# ---------------------------------------------------------------- patch block
|
||||
# Byte-for-byte the launcher's container-start block. The patches come from
|
||||
# the /patches mount (the pinned radiance clone). SP must match the image's
|
||||
# venv layout; the image tag is pinned in the quadlet.
|
||||
# venv layout; the image tag is pinned in the quadlet. The mxfp6 branch
|
||||
# mirrors paroquant/run_paroquant.sh (17 patches, 10 modules, + the paroquant
|
||||
# kernel module + the sitecustomize registration); the mxfp4 branch keeps the
|
||||
# existing 20-patch / 11-module list untouched.
|
||||
SP=/opt/vllm/lib/python3.12/site-packages
|
||||
cd /patches
|
||||
python3 patch_quark_mxfp4.py
|
||||
python3 patch_nvfp4_mxfp4.py
|
||||
python3 patch_tp3_pad.py
|
||||
python3 patch_ar_maxbytes.py
|
||||
python3 patch_topk_triton_rows.py
|
||||
python3 patch_dflash_calib.py
|
||||
python3 patch_dflash_mxfp4_kv.py
|
||||
python3 patch_rmsquant_fusion.py
|
||||
python3 patch_verify_head.py
|
||||
python3 patch_kv_group_size.py
|
||||
python3 patch_topk_composite.py
|
||||
python3 patch_gdn_shared_build.py
|
||||
python3 patch_dflash_selector_topk.py
|
||||
python3 patch_gdn_merge_inproj.py
|
||||
python3 patch_dynwidth.py
|
||||
python3 patch_async_dynwidth.py
|
||||
python3 patch_step_trace.py
|
||||
python3 patch_ar_geometry.py
|
||||
python3 patch_ar_3rank.py
|
||||
python3 patch_gdn_glue.py
|
||||
# Non-fatal: fixes content=null on thinking-off requests; not required to serve.
|
||||
python3 patch_qwen3_thinkoff.py \
|
||||
if [ "$IS_MXFP6" = "1" ]; then
|
||||
python3 patch_quark_mxfp4.py
|
||||
python3 patch_ar_maxbytes.py
|
||||
python3 patch_topk_triton_rows.py
|
||||
python3 patch_dflash_calib.py
|
||||
python3 patch_dflash_mxfp4_kv.py
|
||||
python3 patch_rmsquant_fusion.py
|
||||
python3 patch_verify_head.py
|
||||
python3 patch_kv_group_size.py
|
||||
python3 patch_topk_composite.py
|
||||
python3 patch_gdn_shared_build.py
|
||||
python3 patch_async_dynwidth.py
|
||||
python3 patch_step_trace.py
|
||||
python3 patch_skinny_gemm.py
|
||||
python3 patch_dflash_selector_topk.py
|
||||
python3 patch_dynwidth.py
|
||||
python3 patch_ar_geometry.py
|
||||
python3 patch_gdn_merge_inproj.py
|
||||
# Non-fatal: fixes content=null on thinking-off requests; not required to serve.
|
||||
python3 patch_qwen3_thinkoff.py \
|
||||
|| echo "[radiance] WARNING: thinkoff patch did not apply; thinking-off requests will return empty content"
|
||||
cp mxfp4-configs/*.json "$SP"/aiter/ops/triton/configs/gemm/
|
||||
cp radiance_preamble.py /opt/radiance_preamble.py
|
||||
cp radiance_nvfp4.py radiance_mxfp4.py radiance_gdn.py radiance_rmsquant.py radiance_drafthead.py \
|
||||
cp mxfp4-configs/*.json "$SP"/aiter/ops/triton/configs/gemm/
|
||||
cp radiance_preamble.py /opt/radiance_preamble.py
|
||||
cp radiance_mxfp4.py radiance_gemm.py radiance_gdn.py radiance_gdnmerge.py radiance_rmsquant.py \
|
||||
radiance_drafthead.py radiance_verifyhead.py radiance_aroverlap.py radiance_topk.py \
|
||||
radiance_arnq.py "$SP"/
|
||||
hipcc -O3 -w -std=c++17 -fPIC -shared --offload-arch=gfx1201 $(python3 -m pybind11 --includes) \
|
||||
radiance_mxfp4_fp8.hip -o "$SP"/radiance_mxfp4_fp8.so
|
||||
# ---- paroquant: build the MXFP6 kernel module and register the quant method ----
|
||||
# RADIANCE_PQ_HIPCC_FLAGS: extra compile flags for A/B builds of the kernel
|
||||
# module only (e.g. -DPQ_HW_CVT=0). Empty in prod.
|
||||
hipcc -O3 -w -std=c++17 -fPIC -shared --offload-arch=gfx1201 $(python3 -m pybind11 --includes) \
|
||||
${RADIANCE_PQ_HIPCC_FLAGS:-} paroquant/radiance_paroquant.hip -o "$SP"/radiance_paroquant_kernel.so
|
||||
cp paroquant/radiance_paroquant.py paroquant/radiance_paroquant_mxfp4.py "$SP"/
|
||||
cp radiance_dflash_capture.py "$SP"/ 2>/dev/null || true
|
||||
# NB: appended to the STDLIB sitecustomize, not site-packages -- Ubuntu ships
|
||||
# /usr/lib/python3.12/sitecustomize.py and it shadows any site-packages one, so
|
||||
# a file dropped there is silently never imported. Guarded: an unguarded >>
|
||||
# appends another copy on every restart of an existing container.
|
||||
if ! grep -q radiance_paroquant /usr/lib/python3.12/sitecustomize.py; then
|
||||
printf "%s\n" \
|
||||
"try:" \
|
||||
" import radiance_paroquant # registers the paroquant quantization config" \
|
||||
" import radiance_paroquant_mxfp4 # and the MXFP4-weights variant (paroquant_mxfp4)" \
|
||||
" import radiance_dflash_capture # drafter training-data capture (inert unless RADIANCE_DFLASH_CAPTURE_DIR)" \
|
||||
"except Exception as e:" \
|
||||
" import sys" \
|
||||
" sys.stderr.write(\"[radiance.paroquant] registration failed: %r\\n\" % (e,))" \
|
||||
>> /usr/lib/python3.12/sitecustomize.py
|
||||
fi
|
||||
else
|
||||
python3 patch_quark_mxfp4.py
|
||||
python3 patch_nvfp4_mxfp4.py
|
||||
python3 patch_tp3_pad.py
|
||||
python3 patch_ar_maxbytes.py
|
||||
python3 patch_topk_triton_rows.py
|
||||
python3 patch_dflash_calib.py
|
||||
python3 patch_dflash_mxfp4_kv.py
|
||||
python3 patch_rmsquant_fusion.py
|
||||
python3 patch_verify_head.py
|
||||
python3 patch_kv_group_size.py
|
||||
python3 patch_topk_composite.py
|
||||
python3 patch_gdn_shared_build.py
|
||||
python3 patch_dflash_selector_topk.py
|
||||
python3 patch_gdn_merge_inproj.py
|
||||
python3 patch_dynwidth.py
|
||||
python3 patch_async_dynwidth.py
|
||||
python3 patch_step_trace.py
|
||||
python3 patch_ar_geometry.py
|
||||
python3 patch_ar_3rank.py
|
||||
python3 patch_gdn_glue.py
|
||||
# Non-fatal: fixes content=null on thinking-off requests; not required to serve.
|
||||
python3 patch_qwen3_thinkoff.py \
|
||||
|| echo "[radiance] WARNING: thinkoff patch did not apply; thinking-off requests will return empty content"
|
||||
cp mxfp4-configs/*.json "$SP"/aiter/ops/triton/configs/gemm/
|
||||
cp radiance_preamble.py /opt/radiance_preamble.py
|
||||
cp radiance_nvfp4.py radiance_mxfp4.py radiance_gdn.py radiance_rmsquant.py radiance_drafthead.py \
|
||||
radiance_verifyhead.py radiance_gdnmerge.py radiance_aroverlap.py radiance_topk.py \
|
||||
radiance_arnq.py radiance_tp3pad.py "$SP"/
|
||||
hipcc -O3 -w -std=c++17 -fPIC -shared --offload-arch=gfx1201 $([ "${MXFP4_CUMODE:-0}" = 1 ] && echo -mcumode) \
|
||||
cp radiance_dflash_capture.py "$SP"/ 2>/dev/null || true
|
||||
hipcc -O3 -w -std=c++17 -fPIC -shared --offload-arch=gfx1201 $([ "${MXFP4_CUMODE:-0}" = 1 ] && echo -mcumode) \
|
||||
$(python3 -m pybind11 --includes) radiance_mxfp4_fp8.hip -o "$SP"/radiance_mxfp4_fp8.so
|
||||
fi
|
||||
if [ -n "${R4D_SO:-}" ] && [ -f /r4d/r4d.so ]; then
|
||||
cp /r4d/r4d.so "$SP"/r4d.so
|
||||
echo "[radiance] using patched r4d.so from $R4D_SO"
|
||||
@@ -234,8 +341,9 @@ fi
|
||||
# the one just compiled into site-packages (cwd precedes it on sys.path).
|
||||
cd /
|
||||
fi
|
||||
# The weight-quantization flag. MXFP4 is auto-detected from the checkpoint;
|
||||
# FP8 is passed explicitly (the documented plain-FP8 path).
|
||||
# The weight-quantization flag. MXFP4 and paroquant_mxfp6 are auto-detected
|
||||
# from the checkpoint's quant_method; FP8 is passed explicitly (the documented
|
||||
# plain-FP8 path).
|
||||
QUANT_FLAG=""
|
||||
if [ "$IS_MXFP4" = "0" ]; then QUANT_FLAG="--quantization fp8"; fi
|
||||
exec /opt/radiance_entrypoint.sh "$SNAP" --served-model-name $SERVED_NAMES \
|
||||
|
||||
+14
-12
@@ -1178,13 +1178,15 @@ deploy_llm_small() {
|
||||
wait_for_health frank-small http://127.0.0.1:7771/health 900
|
||||
}
|
||||
|
||||
# deploy_llm_medium — frank-medium is the vllm-radiance MXFP4 stack as a
|
||||
# deploy_llm_medium — frank-medium is the vllm-radiance stack as a
|
||||
# rootless Quadlet since the 2026-09-20 cutover: one tensor-parallel (TP=2)
|
||||
# engine across both R9700s, R4D one-shot all-reduce, DFlash2-FP8 drafter.
|
||||
# It replaced mainline llama.cpp's layer split, which held the endpoint since
|
||||
# 2026-09-16; the conversation battery measured radiance at 2.1x decode and a
|
||||
# third of the time-to-first-token with 4x the concurrent throughput
|
||||
# (scripts/bench-conversation.py, 2026-09-20).
|
||||
# The entrypoint is quantization-aware (config.json quant_method): the plain
|
||||
# FP8 path, the MXFP4 patch-and-serve path, and the ParoQuant MXFP6 stack
|
||||
# (PR #51's launch6_p kernel). It replaced mainline llama.cpp's layer split,
|
||||
# which held the endpoint since 2026-09-16; the conversation battery measured
|
||||
# radiance at 2.1x decode and a third of the time-to-first-token with 4x the
|
||||
# concurrent throughput (scripts/bench-conversation.py, 2026-09-20).
|
||||
#
|
||||
# WHY THE PAIR IS BACK AND THE TRIO IS NOT: the 2026-09-16 cutover retired this
|
||||
# unit because TP=2 could not dispatch a collective on the second card (B650
|
||||
@@ -1197,13 +1199,13 @@ deploy_llm_small() {
|
||||
# pinned radiance patch clone, the patched libr4d build and the Triton compile
|
||||
# cache live outside the checkout (see the unit file's mount comments). The
|
||||
# gates below refuse a deploy that would start a container guaranteed to die.
|
||||
LLM_MEDIUM_IMAGE="${LLM_MEDIUM_IMAGE:-docker.io/stilldeadcode/vllm-radiance:0.9.3}"
|
||||
LLM_MEDIUM_IMAGE_LOCAL="${LLM_MEDIUM_IMAGE_LOCAL:-localhost/radiance-vllm:0.9.3}"
|
||||
LLM_MEDIUM_SNAP="${LLM_MEDIUM_SNAP:-/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq}"
|
||||
LLM_MEDIUM_IMAGE="${LLM_MEDIUM_IMAGE:-ggz14/vllm-radiance-mxfp4:0.13.0-e060895}"
|
||||
LLM_MEDIUM_IMAGE_LOCAL="${LLM_MEDIUM_IMAGE_LOCAL:-localhost/radiance-vllm:0.13.0-e060895}"
|
||||
LLM_MEDIUM_SNAP="${LLM_MEDIUM_SNAP:-/home/dank/models/Qwen3.8-27B-PARO-MXFP6}"
|
||||
LLM_MEDIUM_DRAFTER="${LLM_MEDIUM_DRAFTER:-/home/dank/models/Qwen3.8-27B-DFlash2-FP8}"
|
||||
LLM_MEDIUM_PATCHES="${LLM_MEDIUM_PATCHES:-/home/dank/GitPackages/radiance-vllm-mxfp4}"
|
||||
LLM_MEDIUM_R4D="${LLM_MEDIUM_R4D:-/home/dank/.cache/radiance-libr4d/b9e42ab-rx6}"
|
||||
LLM_MEDIUM_CACHE="${LLM_MEDIUM_CACHE:-/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq}"
|
||||
LLM_MEDIUM_CACHE="${LLM_MEDIUM_CACHE:-/home/dank/.radiance-cache-mxfp6-013-pq-nqft-fp8s-gnq}"
|
||||
|
||||
# The three replica/proxy user units from the 2026-09-16 era are retired HERE,
|
||||
# never left to rot: they must not exist alongside the TP=2 unit (the proxy
|
||||
@@ -1256,8 +1258,8 @@ deploy_llm_medium() {
|
||||
# of truth), so a model swap is just "edit the quadlet + deploy". The
|
||||
# container /models mount maps to /home/dank/models on the host. The
|
||||
# drafter gate applies only to the dflash path; the patched libr4d gate
|
||||
# applies only to the mxfp4/quark path (the plain FP8 path uses the
|
||||
# image's built-in r4d.so and skips the MXFP4 patches).
|
||||
# applies to the mxfp4/quark/paroquant_mxfp6 path (the plain FP8 path uses
|
||||
# the image's built-in r4d.so and skips the MXFP4 patches).
|
||||
local quadlet_file="dankserver/frank-medium/quadlet/frank-medium.container"
|
||||
local snap_c snap spec_method quant
|
||||
snap_c="$(sed -nE 's/^Environment=SNAP=//p' "$quadlet_file" | head -1)"
|
||||
@@ -1294,7 +1296,7 @@ deploy_llm_medium() {
|
||||
# NaNs the MXFP4 model, so the patched build is not optional there. The
|
||||
# plain FP8 path uses the image's built-in r4d.so (no MXFP4 patches).
|
||||
case "$quant" in
|
||||
mxfp4|quark)
|
||||
mxfp4|quark|paroquant_mxfp6)
|
||||
if [[ ! -f "$LLM_MEDIUM_R4D/r4d.so" ]]; then
|
||||
echo "ERROR: patched libr4d absent at $LLM_MEDIUM_R4D" >&2
|
||||
missing=1
|
||||
|
||||
@@ -1,48 +1,47 @@
|
||||
---
|
||||
type: configuration
|
||||
title: frank-medium vllm-radiance MXFP4 Runbook
|
||||
description: ACTIVE frank-medium runbook — vllm-radiance serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8, DFlash2 drafter, unquantized bf16 KV) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). Returned to MXFP4 on 2026-09-21 with KV_DTYPE=auto (clean KV) to test whether the endpoint's historical fp8 KV had been degrading W4A8 attention quality; the FP8 checkpoint (orcarouter/Qwen3.8-27B-Uncensored-FP8, the 2026-09-20 to 09-21 shape) is the on-disk rollback. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era.
|
||||
title: frank-medium vllm-radiance MXFP6 Runbook
|
||||
description: ACTIVE frank-medium runbook — vllm-radiance serving hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6 W4A8, DFlash2 drafter, unquantized bf16 KV) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). Cut to MXFP6 on 2026-09-21 (PR #51's launch6_p kernel, image 0.13.0-e060895); the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq, the 2026-09-21 shape) is the on-disk rollback. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era.
|
||||
tags:
|
||||
- frank-medium
|
||||
- dankserver
|
||||
- vllm
|
||||
- vllm-radiance
|
||||
- mxfp6
|
||||
- mxfp4
|
||||
- gfx1201
|
||||
- r9700
|
||||
- rccl
|
||||
- pcie-atomics
|
||||
- quadlet
|
||||
timestamp: 2026-09-20
|
||||
timestamp: 2026-09-21
|
||||
---
|
||||
|
||||
# frank-medium vllm-radiance MXFP4 Runbook
|
||||
# frank-medium vllm-radiance MXFP6 Runbook
|
||||
|
||||
The ACTIVE frank-medium deployment: the `stilldeadcode/vllm-radiance:0.9.3`
|
||||
image (tagged locally as `localhost/radiance-vllm:0.9.3`) plus the
|
||||
`ggz14/radiance-vllm-mxfp4` patches, serving
|
||||
`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` as **ONE
|
||||
tensor-parallel process across both Radeon AI PRO R9700 (gfx1201)**.
|
||||
It replaced the mainline-llama.cpp layer-split deployment
|
||||
([[frank-medium-llamacpp-dflash2]], SUPERSEDED 2026-09-20); the same image and
|
||||
model had earlier run here as two single-GPU replicas, which is the shape most
|
||||
of this page describes.
|
||||
The ACTIVE frank-medium deployment: the `ggz14/vllm-radiance-mxfp4:0.13.0-e060895`
|
||||
image (built from PR #51's head, tagged locally as
|
||||
`localhost/radiance-vllm:0.13.0-e060895`) plus the `radiance-vllm-mxfp4`
|
||||
patches at e060895, serving `hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6`
|
||||
(ParoQuant MXFP6 W4A8, 25.16 GiB, 18 shards) as **ONE tensor-parallel
|
||||
process across both Radeon AI PRO R9700 (gfx1201)**. It replaced the
|
||||
mainline-llama.cpp layer-split deployment ([[frank-medium-llamacpp-dflash2]],
|
||||
SUPERSEDED 2026-09-20); the same image and model had earlier run here as two
|
||||
single-GPU replicas, which is the shape most of this page describes.
|
||||
|
||||
**Serving shape since 2026-09-21: MXFP4 with unquantized (bf16) KV.** This
|
||||
endpoint briefly served `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3,
|
||||
`--quantization fp8`, DFlash2 drafter) from 2026-09-20 to 2026-09-21; it has
|
||||
since been cut back to the MXFP4 checkpoint above, now with
|
||||
`--kv-cache-dtype auto` (unquantized bf16 KV) instead of the endpoint's
|
||||
historical `fp8`. Hypothesis: fp8 KV sat on *both* the FP8 and MXFP4 arms (the
|
||||
flag was on the common exec path), so it could not explain the FP8-vs-MXFP4
|
||||
*throughput* gap — but MXFP4's weights are already W4A8, and fp8 KV on top
|
||||
compounds attention-path precision loss, which shows up as degraded *output
|
||||
quality*, not t/s. **Measured on deploy**: `kv_cache_dtype=auto`, no fp8
|
||||
banner, pool **486,464 tokens** (`Maximum concurrency for 262,144 tokens per
|
||||
request: 1`) — exactly half MXFP4's 943,581-token fp8 pool, still clearing the
|
||||
full 262,144 window so `MAXLEN` stays native. No quality verdict yet; this is
|
||||
the setup. Rollback: flip `SNAP` + `SPEC` back to the FP8 path with
|
||||
`KV_DTYPE=fp8` (that shape auto-sized to ~384k and held the full window).
|
||||
**Serving shape since 2026-09-21: ParoQuant MXFP6 with unquantized (bf16) KV.**
|
||||
This endpoint cut to the MXFP6 checkpoint above (PR #51's `launch6_p` kernel,
|
||||
`quant_method=paroquant_mxfp6`, bits=6, group_size=128, krot=8, rotation on,
|
||||
e4m3fn scale) with `--kv-cache-dtype auto` (unquantized bf16 KV) and
|
||||
`--gpu-memory-utilization 0.99`. The Swift variant is a finetune of
|
||||
`ukisai/Swift-Qwen3.8-27b` (a different base from `Qwen/Qwen3.8-27B`), so the
|
||||
DFlash2-FP8 drafter's hidden-state distribution is a mismatch — acceptance may
|
||||
drop from the MXFP4 era's 4.24/7, but it is still a net win over no spec
|
||||
decode. **The MXFP4 checkpoint is the on-disk rollback**: flip `SNAP` back to
|
||||
`/models/Qwen3.8-27B-Uncensored-MXFP4-awq`, `GPU_UTIL` to 0.98, and the image
|
||||
back to `localhost/radiance-vllm:0.9.3` (re-pull `docker.io/stilldeadcode/
|
||||
vllm-radiance:0.9.3` if it is no longer on disk; the MXFP4 shape auto-sized to
|
||||
486,464 tokens with bf16 KV and held the full 262,144 window).
|
||||
|
||||
**Deployed shape since 2026-09-20: one rootless Quadlet, `--tensor-parallel-size 2`.**
|
||||
The ProArt B850-Creator WiFi NEO board swap put both R9700s on CPU GPP root
|
||||
@@ -59,31 +58,30 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`:
|
||||
|
||||
| Setting | Value |
|
||||
|---|---|
|
||||
| Model | `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4 / quark, W4A8, 19.37 GiB single shard) |
|
||||
| Model | `hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6` (ParoQuant MXFP6 W4A8, 25.16 GiB, 18 shards; a finetune of `ukisai/Swift-Qwen3.8-27b`) |
|
||||
| Speculative | `--speculative-config {"method":"dflash","model":"/models/Qwen3.8-27B-DFlash2-FP8","num_speculative_tokens":7}` (the z-lab DFlash2 external drafter; mean acceptance 4.24/7. A/B tested against MTP self-speculation — DFlash2 wins on decode, see below) |
|
||||
| Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) |
|
||||
| Context | `--max-model-len 262144` |
|
||||
| Concurrency | `--max-num-seqs 8`, chunk 8192 |
|
||||
| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool 486,464 tokens**; the endpoint's historical fp8 KV halved precision on top of MXFP4's W4A8 (rollback: `KV_DTYPE=fp8` -> 943,581 tokens) |
|
||||
| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool ~730k tokens** (estimated at 0.99 fill: MXFP6's 25.16 GiB weights are ~12.58 GiB per card, leaving ~19.2 GiB per card for KV+activations); rollback: `KV_DTYPE=fp8` |
|
||||
| Prompt cache | `--enable-prefix-caching` |
|
||||
| GPU fill | `--gpu-memory-utilization 0.98` |
|
||||
| GPU fill | `--gpu-memory-utilization 0.99` |
|
||||
| Parsers | `--enable-auto-tool-choice --tool-call-parser qwen3_coder`, `--reasoning-parser qwen3` |
|
||||
| Sampler floor | `--override-sampling-effects` (temp 0.7 / top_p 0.95 / top_k 20) |
|
||||
| Multimodal | `--limit-mm-per-prompt '{"image": 8}'` |
|
||||
| Chat template | `qwen-sharp-v22.4.0.jinja` (froggeric Qwen-Sharp v22.4.0, repo-tracked at `dankserver/frank-medium/`, mounted at `/templates/`; terse system prompt by default — opt out per-request with `chat_template_kwargs {"terse": false}`; rollback: flip `CHAT_TEMPLATE` back to `/patches/qwen-fixed-v22.3.jinja`) |
|
||||
| Chat template | `qwen-sharp-v22.5.0.jinja` (peculiar-ragdoll Qwen-Sharp v22.5.0, repo-tracked at `dankserver/frank-medium/`, mounted at `/templates/`; terse system prompt by default — opt out per-request with `chat_template_kwargs {"terse": false}`; rollback: flip `CHAT_TEMPLATE` back to `/templates/qwen-sharp-v22.4.0.jinja`) |
|
||||
|
||||
**The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin
|
||||
(`KV_MEM=0`): the pool comes up at whatever VRAM the weights + drafter leave.
|
||||
MXFP4's weights are smaller than FP8's (19.4 vs 30.9 GiB), which frees VRAM for
|
||||
the pool — so even with unquantized bf16 KV the full window fits. The 2026-09-21
|
||||
clean-KV start auto-sized to **486,464 tokens** (`GPU KV cache size: 486,464
|
||||
tokens`, `Maximum concurrency for 262,144 tokens per request: 1`) — 1.86x the
|
||||
full window, and exactly half MXFP4's fp8-KV pool of 943,581 (the 2x f16-vs-fp8
|
||||
KV ratio). For reference the FP8 checkpoint (now the rollback) auto-sized to
|
||||
384,238 with fp8 KV (1.47x) and would only reach ~192,000 unquantized — which
|
||||
is why FP8 and MXFP4 differ on the window, and why MXFP4 is the shape that
|
||||
affords clean KV at the native 262,144. After the first healthy start,
|
||||
recalibrate with `calibrate-kv.sh`.
|
||||
MXFP6's weights are 25.16 GiB (~12.58 GiB per card at TP=2), which at 0.99 fill
|
||||
(34.2 GiB per card) leaves ~19.2 GiB per card for KV + activations. The
|
||||
estimated pool is **~730k tokens** (the actual size is determined at boot; the
|
||||
full 262,144 window fits with ~2.7x headroom). The DFlash2-FP8 drafter costs
|
||||
2.1 GiB of VRAM on top. For reference the MXFP4 checkpoint (now the rollback)
|
||||
auto-sized to 486,464 tokens with bf16 KV (1.86x the window), and the FP8
|
||||
checkpoint would only reach ~192,000 unquantized — which is why MXFP6's larger
|
||||
weights still afford clean KV at the native 262,144 (the 0.99 fill buys the
|
||||
pool back). After the first healthy start, recalibrate with `calibrate-kv.sh`.
|
||||
|
||||
## DFlash2 vs MTP — A/B on the FP8 checkpoint (2026-09-20)
|
||||
|
||||
|
||||
@@ -95,10 +95,11 @@ env_str() {
|
||||
# DFlash2 drafts at most 7 tokens per step; deeper is a different
|
||||
# drafter's contract and costs acceptance, not just VRAM.
|
||||
[ "$spec" -eq 7 ]
|
||||
# The MXFP4 decode kernel band covers MAXSEQS x (SPEC+1) rows, and the
|
||||
# shipped default covers 64. Exceeding it without the entrypoint's >8
|
||||
# bump is a launch failure at the worst moment (first concurrent
|
||||
# request).
|
||||
# The MXFP4 / ParoQuant-MXFP6 decode kernel band covers MAXSEQS x
|
||||
# (SPEC+1) rows, and the shipped default covers 64 (the mxfp6 profile
|
||||
# block exports the same RADIANCE_MXFP4_DECODE_MAX_M=64). Exceeding it
|
||||
# without the entrypoint's >8 bump is a launch failure at the worst
|
||||
# moment (first concurrent request).
|
||||
[ $(( maxseqs * (spec + 1) )) -le 64 ]
|
||||
grep -q 'RADIANCE_MXFP4_DECODE_MAX_M' "$ENTRY"
|
||||
;;
|
||||
|
||||
Reference in New Issue
Block a user