From e9a01cd013698a0ff7650ece40d665b7d5af1b5a Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 21 Sep 2026 22:46:44 -0700 Subject: [PATCH] frank-medium: swap to ParoQuant MXFP6 (Swift Qwen3.8-27B) + Qwen-Sharp v22.5.0 - Model: hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (25.16 GiB, 18 shards, quant_method=paroquant_mxfp6, bits=6, group_size=128, krot=8, rotation on) - Image: ggz14/vllm-radiance-mxfp4:0.13.0-e060895 (PR #51 launch6_p kernel, built on the published 0.9.3 base; docker->podman via save/load) - GPU_UTIL 0.98 -> 0.99 (user request); KV_DTYPE=auto (unquantized bf16) - Entrypoint: mxfp6 branch (17 patches + paroquant kernel compile + sitecustomize registration); GPU_UTIL default for set -u robustness - Quadlet: SNAP/GPU_UTIL/Image/cache updated; new mxfp6 cache dir - deploy-remote.sh: R4D gate + paroquant_mxfp6; new image/SNAP/cache defaults - Chat template: Qwen-Sharp v22.5.0 (peculiar-ragdoll); v22.4.0 kept as rollback - Test 168: dflash branch comment quant-aware (mxfp4 + paroquant_mxfp6) - Runbook: MXFP6 active shape, MXFP4 rollback MXFP4 checkpoint remains on disk as the rollback target. --- .../quadlet/frank-medium.container | 37 +- .../frank-medium/qwen-sharp-v22.5.0.jinja | 488 ++++++++++++++++++ .../frank-medium/radiance-entrypoint.sh | 178 +++++-- scripts/deploy-remote.sh | 26 +- .../LLM/frank-medium-radiance-mxfp4.md | 82 ++- tests/frank_medium_unit.bats | 9 +- 6 files changed, 709 insertions(+), 111 deletions(-) create mode 100644 dankserver/frank-medium/qwen-sharp-v22.5.0.jinja diff --git a/dankserver/frank-medium/quadlet/frank-medium.container b/dankserver/frank-medium/quadlet/frank-medium.container index d8913c5..3e503b8 100644 --- a/dankserver/frank-medium/quadlet/frank-medium.container +++ b/dankserver/frank-medium/quadlet/frank-medium.container @@ -4,18 +4,18 @@ # Unit name: frank-medium.service (container name: frank-medium) # # The dankserver LLM endpoint (:7770, alias frank-medium): the vllm-radiance -# stack (stilldeadcode/vllm-radiance:0.9.3 image) serving -# just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4 / quark, W4A8, 19.37 -# GiB) on the 2x Radeon AI PRO R9700 (gfx1201) as ONE tensor-parallel process +# stack (the 0.13.0-e060895 image, PR #51) serving +# hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6, W4A8, 25.16 GiB) +# on the 2x Radeon AI PRO R9700 (gfx1201) as ONE tensor-parallel process # (TP=2, R4D one-shot all-reduce) with the z-lab DFlash2 external drafter # (SPEC=7) and UNQUANTIZED bf16 KV (KV_DTYPE=auto). # The entrypoint is QUANTIZATION-AWARE: it reads the model's weight # quantization from config.json and serves the plain FP8 path (no MXFP4 -# patches, --quantization=fp8, ROCM_AITER unified attention) or the MXFP4 -# patch-and-serve path (the DFlash2 drafter). A model swap is now: change -# SNAP (+ SPEC_METHOD/SPEC), restart. The FP8 checkpoint -# (orcarouter/Qwen3.8-27B-Uncensored-FP8) is the on-disk rollback target — the -# 2026-09-20 to 2026-09-21 serving shape, before this MXFP4 + clean-KV cut. +# patches, --quantization=fp8, ROCM_AITER unified attention), the MXFP4 +# patch-and-serve path, or the ParoQuant MXFP6 stack (PR #51's launch6_p +# kernel). A model swap is now: change SNAP (+ SPEC_METHOD/SPEC), restart. +# The MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq, the +# 2026-09-20 to 2026-09-21 serving shape) is the on-disk rollback target. # # SHAPE. The launcher the upstream repo ships binds the server with # --network=host (vLLM takes --host 0.0.0.0 --port 7770 directly), which is @@ -58,19 +58,19 @@ # docker.io ref and re-tags it before the restart. Pull=never keeps boots off # the registry. # -# KEEP IN SYNC: the radiance clone is pinned (97353293...); if it is updated, +# KEEP IN SYNC: the radiance clone is pinned (e060895...); if it is updated, # re-diff dankserver/frank-medium/radiance-entrypoint.sh against the new # serve-mxfp4.sh container command and bump the pin deliberately. # # DO NOT name an environment variable TEMP, HOST or GPU_MAX_HEAP_SIZE: the # HIP runtime reads bare TEMP for its JIT cache path (2026-08-13 incident). [Unit] -Description=frank-medium — vllm-radiance MXFP4 Qwen3.8-27B-Uncensored on 2x R9700 (gfx1201) +Description=frank-medium — vllm-radiance ParoQuant MXFP6 Swift Qwen3.8-27B on 2x R9700 (gfx1201) After=network-online.target [Container] ContainerName=frank-medium -Image=localhost/radiance-vllm:0.9.3 +Image=localhost/radiance-vllm:0.13.0-e060895 # vLLM binds 0.0.0.0:7770 on the host network directly; no PublishPort. AddDevice=/dev/kfd AddDevice=/dev/dri @@ -85,14 +85,14 @@ Environment=HIP_VISIBLE_DEVICES=0,1 # Drafting: the external z-lab DFlash2-FP8 drafter (SPEC_METHOD=dflash). The # plain-FP8 checkpoint (rollback) carries its own single-layer MTP head, but # DFlash2 beat MTP self-speculation on decode, so the drafter serves either. -# Chat template: the Qwen-Sharp variant (froggeric v22.4.0) — the terse +# Chat template: the Qwen-Sharp variant (peculiar-ragdoll v22.5.0) — the terse # system prompt (opt out per-request with chat_template_kwargs {"terse": # false}), message.reasoning support (vLLM's thinking field), and the -# tightened tool-call rendering. The predecessor (qwen-fixed-v22.3.jinja in -# the /patches clone) remains there as the rollback target: flip CHAT_TEMPLATE. -Environment=SNAP=/models/Qwen3.8-27B-Uncensored-MXFP4-awq +# tightened tool-call rendering. The predecessor (qwen-sharp-v22.4.0.jinja, +# mounted beside it) remains as the rollback target: flip CHAT_TEMPLATE. +Environment=SNAP=/models/Qwen3.8-27B-PARO-MXFP6 Environment=DRAFTER=/models/Qwen3.8-27B-DFlash2-FP8 -Environment=CHAT_TEMPLATE=/templates/qwen-sharp-v22.4.0.jinja +Environment=CHAT_TEMPLATE=/templates/qwen-sharp-v22.5.0.jinja # Fleet identity: the id at /v1/models and in every response body. Every # client (Hermes, the harness, services.json) points at frank-medium. Environment=SERVED_NAMES=frank-medium @@ -116,7 +116,7 @@ Environment=MAXLEN=262144 # through the Gemma4 merge path and it is not MXFP4-quantized, so vLLM serves # image input on either checkpoint. harness models.yml keeps frank-medium # input: [text, image]. -Environment=GPU_UTIL=0.98 +Environment=GPU_UTIL=0.99 # Speculative decoding: DFlash2 external drafter at depth 7. A/B tested # against MTP self-speculation (depth 8) on this FP8 checkpoint: DFlash2 # wins on decode (100.0 vs 85.9 t/s geomean, 287.1 vs 179.7 t/s burst-4) @@ -144,7 +144,7 @@ Volume=/home/dank/models:/models Volume=/home/dank/.cache/huggingface:/root/.cache/huggingface # Compile cache. Keyed on model + stack version + every graph-changing knob; # never share it across configurations (upstream rule). -Volume=/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq:/cache +Volume=/home/dank/.radiance-cache-mxfp6-013-pq-nqft-fp8s-gnq:/cache # The pinned radiance clone: patches applied at every container start. Volume=/home/dank/GitPackages/radiance-vllm-mxfp4:/patches:z # The patched libr4d (pin b9e42ab + repo extras, key b9e42ab-rx6). The @@ -154,6 +154,7 @@ Volume=/home/dank/.cache/radiance-libr4d/b9e42ab-rx6:/r4d:z Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/radiance-entrypoint.sh:/usr/local/bin/frank-medium-entrypoint.sh:ro Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/frank-medium-healthcheck.py:/usr/local/bin/frank-medium-healthcheck.py:ro Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/qwen-sharp-v22.4.0.jinja:/templates/qwen-sharp-v22.4.0.jinja:ro +Volume=/home/dank/serverAppConfigs/dankserver/frank-medium/qwen-sharp-v22.5.0.jinja:/templates/qwen-sharp-v22.5.0.jinja:ro # The entrypoint must be the container command; the image's own entrypoint # (which the patched script execs at the end) must not run unpatched. PodmanArgs=--entrypoint /usr/local/bin/frank-medium-entrypoint.sh diff --git a/dankserver/frank-medium/qwen-sharp-v22.5.0.jinja b/dankserver/frank-medium/qwen-sharp-v22.5.0.jinja new file mode 100644 index 0000000..74465e7 --- /dev/null +++ b/dankserver/frank-medium/qwen-sharp-v22.5.0.jinja @@ -0,0 +1,488 @@ +{%- set template_version = "qwen3.8-froggeric-v22.5.0" %} +{%- set _tool_format = tool_call_format if tool_call_format is defined else 'xml' %} +{%- set image_count = namespace(value=0) %} +{%- set video_count = namespace(value=0) %} +{%- set add_vision_id = add_vision_id if add_vision_id is defined else false %} +{%- set enable_thinking = enable_thinking if enable_thinking is defined else true %} +{%- set auto_disable_thinking_with_tools = auto_disable_thinking_with_tools if auto_disable_thinking_with_tools is defined else false %} +{%- if preserve_reasoning is defined and preserve_reasoning is not none %} + {%- set _preserve_thinking = preserve_reasoning %} +{%- elif preserve_thinking is defined and preserve_thinking is not none %} + {%- set _preserve_thinking = preserve_thinking %} +{%- else %} + {%- set _preserve_thinking = true %} +{%- endif %} +{%- set max_tool_arg_chars = max_tool_arg_chars if max_tool_arg_chars is defined else 0 %} +{%- set max_tool_response_chars = max_tool_response_chars if max_tool_response_chars is defined else 0 %} +{%- set _default_reasoning_effort = 'medium' %} +{%- set _has_tools = (tools is defined and tools and tools is iterable and tools is not mapping) %} +{%- set _effort_raw = (reasoning_effort | string | lower) if reasoning_effort is defined and reasoning_effort is not none else _default_reasoning_effort %} +{%- set _initial_thinking = enable_thinking %} +{%- if _effort_raw in ('none', 'off') %} + {%- set _initial_thinking = false %} + {%- set _initial_effort = 'medium' %} +{%- elif _effort_raw in ('minimal', 'low') %} + {%- set _initial_effort = 'low' %} +{%- elif _effort_raw in ('high', 'xhigh', 'max', 'ultracode', 'extreme') %} + {%- set _initial_effort = 'xhigh' %} +{%- else %} + {%- set _initial_effort = 'medium' %} +{%- endif %} +{%- set ns_state = namespace(thinking=_initial_thinking, effort=_initial_effort) %} +{%- if auto_disable_thinking_with_tools and _has_tools %} + {%- set ns_state.thinking = false %} +{%- endif %} +{%- for msg in messages %} + {%- if msg.role == 'system' or msg.role == 'developer' or msg.role == 'user' %} + {%- if msg.content is string %} + {%- if '<|think_off|>' in msg.content %} + {%- set ns_state.thinking = false %} + {%- elif '<|think_on|>' in msg.content %} + {%- set ns_state.thinking = true %} + {%- elif '<|think_xhigh|>' in msg.content or '<|think_high|>' in msg.content or '<|think_ultracode|>' in msg.content or '<|think_extreme|>' in msg.content or '<|think_max|>' in msg.content %} + {%- set ns_state.thinking = true %} + {%- set ns_state.effort = 'xhigh' %} + {%- elif '<|think_low|>' in msg.content or '<|think_minimal|>' in msg.content %} + {%- set ns_state.thinking = true %} + {%- set ns_state.effort = 'low' %} + {%- elif '<|think_medium|>' in msg.content %} + {%- set ns_state.thinking = true %} + {%- set ns_state.effort = 'medium' %} + {%- endif %} + {%- elif msg.content is iterable and msg.content is not mapping %} + {%- for item in msg.content %} + {%- if item is string %} + {%- set _item_text = item %} + {%- elif item is mapping and 'text' in item and item.text is string %} + {%- set _item_text = item.text %} + {%- else %} + {%- set _item_text = '' %} + {%- endif %} + {%- if _item_text %} + {%- if '<|think_off|>' in _item_text %} + {%- set ns_state.thinking = false %} + {%- elif '<|think_on|>' in _item_text %} + {%- set ns_state.thinking = true %} + {%- elif '<|think_xhigh|>' in _item_text or '<|think_high|>' in _item_text or '<|think_ultracode|>' in _item_text or '<|think_extreme|>' in _item_text or '<|think_max|>' in _item_text %} + {%- set ns_state.thinking = true %} + {%- set ns_state.effort = 'xhigh' %} + {%- elif '<|think_low|>' in _item_text or '<|think_minimal|>' in _item_text %} + {%- set ns_state.thinking = true %} + {%- set ns_state.effort = 'low' %} + {%- elif '<|think_medium|>' in _item_text %} + {%- set ns_state.thinking = true %} + {%- set ns_state.effort = 'medium' %} + {%- endif %} + {%- endif %} + {%- endfor %} + {%- endif %} + {%- endif %} +{%- endfor %} +{%- set reasoning_instructions = '' %} +{%- if ns_state.thinking %} + {%- if ns_state.effort == 'xhigh' %} + {%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %} + {%- elif ns_state.effort == 'low' %} + {%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %} + {%- endif %} +{%- endif %} +{%- macro render_content(content, do_vision_count, is_system_content=false) %} + {%- if content is string %} + {{- content }} + {%- elif content is iterable and content is not mapping %} + {%- for item in content %} + {%- if item is mapping %} + {%- if item.type == 'image' or 'image' in item or 'image_url' in item %} + {%- if is_system_content %} + {{- raise_exception('System message cannot contain images.') }} + {%- endif %} + {%- if do_vision_count %} + {%- set image_count.value = image_count.value + 1 %} + {%- endif %} + {%- if add_vision_id %} + {{- 'Picture ' ~ image_count.value ~ ': ' }} + {%- endif %} + {{- '<|vision_start|><|image_pad|><|vision_end|>' }} + {%- elif item.type == 'video' or 'video' in item or 'video_url' in item %} + {%- if is_system_content %} + {{- raise_exception('System message cannot contain videos.') }} + {%- endif %} + {%- if do_vision_count %} + {%- set video_count.value = video_count.value + 1 %} + {%- endif %} + {%- if add_vision_id %} + {{- 'Video ' ~ video_count.value ~ ': ' }} + {%- endif %} + {{- '<|vision_start|><|video_pad|><|vision_end|>' }} + {%- elif 'text' in item %} + {{- item.text }} + {%- else %} + {{- raise_exception('Unexpected item type in content.') }} + {%- endif %} + {%- else %} + {{- item | string }} + {%- endif %} + {%- endfor %} + {%- elif content is none or content is undefined %} + {{- '' }} + {%- else %} + {{- raise_exception('Unexpected content type.') }} + {%- endif %} +{%- endmacro %} +{%- if not messages %} + {{- raise_exception('No messages provided.') }} +{%- endif %} +{%- set head = namespace(count=0, seen_non_system=false) %} +{%- for message in messages %} + {%- set _is_sys = (message.role == 'system' or message.role == 'developer') %} + {%- if _is_sys and not head.seen_non_system %} + {%- set head.count = head.count + 1 %} + {%- else %} + {%- set head.seen_non_system = true %} + {%- endif %} +{%- endfor %} +{%- set sys_state = namespace(content='') %} +{%- for message in messages[:head.count] %} + {%- set _part = render_content(message.content, false, true) | trim %} + {%- if '<|think_off|>' in _part %}{%- set _part = _part.split('<|think_off|>') | join('') | trim %}{%- endif %} + {%- if '<|think_on|>' in _part %}{%- set _part = _part.split('<|think_on|>') | join('') | trim %}{%- endif %} + {%- if '<|think_xhigh|>' in _part %}{%- set _part = _part.split('<|think_xhigh|>') | join('') | trim %}{%- endif %} + {%- if '<|think_high|>' in _part %}{%- set _part = _part.split('<|think_high|>') | join('') | trim %}{%- endif %} + {%- if '<|think_ultracode|>' in _part %}{%- set _part = _part.split('<|think_ultracode|>') | join('') | trim %}{%- endif %} + {%- if '<|think_extreme|>' in _part %}{%- set _part = _part.split('<|think_extreme|>') | join('') | trim %}{%- endif %} + {%- if '<|think_max|>' in _part %}{%- set _part = _part.split('<|think_max|>') | join('') | trim %}{%- endif %} + {%- if '<|think_medium|>' in _part %}{%- set _part = _part.split('<|think_medium|>') | join('') | trim %}{%- endif %} + {%- if '<|think_low|>' in _part %}{%- set _part = _part.split('<|think_low|>') | join('') | trim %}{%- endif %} + {%- if '<|think_minimal|>' in _part %}{%- set _part = _part.split('<|think_minimal|>') | join('') | trim %}{%- endif %} + {%- if _part %} + {%- if sys_state.content %} + {%- set sys_state.content = sys_state.content ~ '\n\n' ~ _part %} + {%- else %} + {%- set sys_state.content = _part %} + {%- endif %} + {%- endif %} +{%- endfor %} +{%- set _sc = sys_state.content %} +{%- set _msgs = messages[head.count:] %} +{%- if ns_state.thinking %} + {%- set _terse_lead = 'Answer directly, after thinking. Lead with the answer, then only what it needs to be correct and usable.' %} +{%- else %} + {%- set _terse_lead = 'Answer directly and concisely. Give the answer with only what it needs to be correct and usable.' %} +{%- endif %} +{%- set _terse_core %} +Never: open with preamble or pleasantries; restate the question; add filler transitions; hedge with niceties; or repeat a point you've already made. +Always: keep essential steps, caveats, uncertainties, and specifics — never drop correctness or a needed warning for brevity. Keep the final answer lean. Use the least structure that conveys it (plain prose when short; lists or code only when they earn their place). If genuinely uncertain, say so and explain why — never omit uncertainty for the sake of brevity. +If a user request is genuinely ambiguous, ask a sharp question, don't guess. +{%- endset %} +{%- set _terse = _terse_lead ~ '\n' ~ (_terse_core | trim) %} +{#- `terse` chat-template kwarg: default true, so behaviour is unchanged unless a caller opts out. + Pass {"terse": false} via chat_template_kwargs to serve the model with only its own system + prompt -- useful for A/B-ing the terseness effect, or when a downstream prompt conflicts. #} +{%- set _terse_on = terse if terse is defined else true %} +{%- if _terse_on %} + {%- if not _sc %} + {%- set _sc = _terse | trim %} + {%- else %} + {%- set _sc = (_sc | trim) ~ '\n\n' ~ (_terse | trim) %} + {%- endif %} +{%- endif %} +{#- Some runtimes (LM Studio's MLX backend) have no native tool-call parser and prepend their own + protocol into the system content. Emitting ours as well gives the model two contradictory tool + protocols and it follows one at random. Detect that marker and stand down; the + `suppress_tool_instructions` kwarg overrides either way where a runtime can pass template kwargs. #} +{%- set _runtime_tool_protocol = (_sc is defined and _sc and '[TOOL_REQUEST]' in _sc) %} +{%- set _suppress_tools = suppress_tool_instructions if suppress_tool_instructions is defined else _runtime_tool_protocol %} +{%- if _has_tools and not _suppress_tools %} + {{- '<|im_start|>system\n' }} + {%- if reasoning_instructions %} + {{- reasoning_instructions + '\n\n' }} + {%- endif %} + {{- '# Tools\n\nYou have access to the following functions:\n\n' }} + {%- for tool in tools %} + {{- '\n' }} + {{- tool | tojson }} + {%- endfor %} + {{- '\n' }} + {%- if _tool_format == 'json' %} + {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }} + {%- if ns_state.thinking %} + {{- '\nBrief explanation of tool call\n\n' }} + {%- endif %} + {{- '\n{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1", "example_parameter_2": "This is the value for the second parameter"}}\n\n\n\nReminder:\n' }} + {%- if ns_state.thinking %} + {{- '- You can use the block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the block.\n' }} + {%- endif %} + {{- '- Function calls MUST follow the specified format: a single JSON object with "name" and "arguments" keys inside XML tags.\n' }} + {%- if ns_state.thinking %} + {{- '- If you choose to call a tool, you MUST output the block IMMEDIATELY after thinking, with NO conversational text before it.\n' }} + {%- else %} + {{- '- If you choose to call a tool, you MUST output the block IMMEDIATELY, with NO conversational text before it.\n' }} + {%- endif %} + {{- '- The tag MUST be at the very beginning of a new line, with NO spaces or indentation before it.\n- To call multiple functions, output a separate, completely closed block for EACH function. Do NOT nest blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n' }} + {%- else %} + {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n' }} + {%- if ns_state.thinking %} + {{- '\nBrief explanation of tool call\n\n' }} + {%- endif %} + {{- '\n\n\nvalue_1\n\n\nThis is the value for the second parameter\nthat can span\nmultiple lines\n\n\n\n\n\nReminder:\n' }} + {%- if ns_state.thinking %} + {{- '- You can use the block to plan your next tool call OR to synthesize data and formulate your final response to the user.\n- ALL explanation and reasoning MUST be placed strictly inside the block.\n' }} + {%- endif %} + {{- '- Function calls MUST follow the specified format: an inner block must be nested within XML tags.\n' }} + {%- if ns_state.thinking %} + {{- '- If you choose to call a tool, you MUST output the block IMMEDIATELY after thinking, with NO conversational text before it.\n' }} + {%- else %} + {{- '- If you choose to call a tool, you MUST output the block IMMEDIATELY, with NO conversational text before it.\n' }} + {%- endif %} + {{- '- The and tags MUST be at the very beginning of a new line, with NO spaces or indentation before them.\n- To call multiple functions, output a separate, completely closed block for EACH function. Do NOT nest blocks.\n- If you have all necessary data, provide your final answer directly to the user without any tool call.\n' }} + {%- endif %} + {%- if _sc %} + {{- '\n\n' + _sc }} + {%- endif %} + {{- '<|im_end|>\n' }} +{%- else %} + {%- if _sc %} + {{- '<|im_start|>system\n' + (reasoning_instructions + '\n\n' if reasoning_instructions else '') + _sc + '<|im_end|>\n' }} + {%- elif reasoning_instructions %} + {{- '<|im_start|>system\n' + reasoning_instructions + '<|im_end|>\n' }} + {%- endif %} +{%- endif %} +{%- set _last_idx = _msgs | length - 1 %} +{%- set ns = namespace(multi_step_tool=true, last_query_index=_last_idx) %} +{%- for message in _msgs[::-1] %} + {%- set index = (_msgs | length - 1) - loop.index0 %} + {%- if ns.multi_step_tool and message.role == 'user' %} + {%- set _rc = render_content(message.content, false) | trim %} + {%- if not (_rc.startswith('') and _rc.endswith('')) %} + {%- set ns.multi_step_tool = false %} + {%- set ns.last_query_index = index %} + {%- endif %} + {%- endif %} +{%- endfor %} +{%- if ns.multi_step_tool %} + {%- if _last_idx > 50 %} + {%- set ns.last_query_index = _last_idx %} + {%- else %} + {%- set ns.last_query_index = 0 %} + {%- endif %} +{%- endif %} +{%- set ns2 = namespace(prev_role='', consecutive_failures=0) %} +{%- for message in _msgs %} + {%- set is_system = (message.role == "system" or message.role == "developer") %} + {%- set content = render_content(message.content, true, is_system) | trim %} + {%- if is_system or message.role == 'user' %} + {%- if '<|think_off|>' in content %}{%- set content = content.split('<|think_off|>') | join('') | trim %}{%- endif %} + {%- if '<|think_on|>' in content %}{%- set content = content.split('<|think_on|>') | join('') | trim %}{%- endif %} + {%- if '<|think_xhigh|>' in content %}{%- set content = content.split('<|think_xhigh|>') | join('') | trim %}{%- endif %} + {%- if '<|think_high|>' in content %}{%- set content = content.split('<|think_high|>') | join('') | trim %}{%- endif %} + {%- if '<|think_ultracode|>' in content %}{%- set content = content.split('<|think_ultracode|>') | join('') | trim %}{%- endif %} + {%- if '<|think_extreme|>' in content %}{%- set content = content.split('<|think_extreme|>') | join('') | trim %}{%- endif %} + {%- if '<|think_max|>' in content %}{%- set content = content.split('<|think_max|>') | join('') | trim %}{%- endif %} + {%- if '<|think_medium|>' in content %}{%- set content = content.split('<|think_medium|>') | join('') | trim %}{%- endif %} + {%- if '<|think_low|>' in content %}{%- set content = content.split('<|think_low|>') | join('') | trim %}{%- endif %} + {%- if '<|think_minimal|>' in content %}{%- set content = content.split('<|think_minimal|>') | join('') | trim %}{%- endif %} + {%- endif %} + {%- if is_system %} + {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }} + {%- elif message.role == 'user' %} + {%- set ns2.consecutive_failures = 0 %} + {{- '<|im_start|>user\n' + content + '<|im_end|>\n' }} + {%- elif message.role == 'assistant' %} + {%- set reasoning_content = '' %} + {%- set _explicit_reasoning = '' %} + {%- if message.reasoning_content is defined and message.reasoning_content is not none %} + {%- if message.reasoning_content is string %} + {%- set _explicit_reasoning = message.reasoning_content %} + {%- else %} + {%- set _explicit_reasoning = message.reasoning_content | string %} + {%- endif %} + {%- elif message.thinking is defined and message.thinking is not none %} + {%- if message.thinking is string %} + {%- set _explicit_reasoning = message.thinking %} + {%- else %} + {%- set _explicit_reasoning = message.thinking | string %} + {%- endif %} + {%- elif message.reasoning is defined and message.reasoning is not none %} + {%- if message.reasoning is string %} + {%- set _explicit_reasoning = message.reasoning %} + {%- else %} + {%- set _explicit_reasoning = message.reasoning | string %} + {%- endif %} + {%- endif %} + {%- if _explicit_reasoning %} + {%- set _lead_end = '' %} + {%- if content.startswith('') and '' in content %} + {%- set _lead_end = '' %} + {%- elif content.startswith('') and '' in content %} + {%- set _lead_end = '' %} + {%- elif content.startswith('') %} + {%- set _lead_end = '' %} + {%- elif content.startswith('') %} + {%- set _lead_end = '' %} + {%- endif %} + {%- if _lead_end %} + {%- set content = content.split(_lead_end)[-1].lstrip('\n') %} + {%- endif %} + {%- set reasoning_content = _explicit_reasoning %} + {%- else %} + {%- set _think_end = '' %} + {%- if content.startswith('') %} + {%- set _think_end = '' %} + {%- elif content.startswith('') %} + {%- set _think_end = '' %} + {%- elif '\n' in content %} + {%- set _think_end = '\n' %} + {%- elif '\n' in content %} + {%- set _think_end = '\n' %} + {%- elif '\n' in content %} + {%- set _think_end = '\n' %} + {%- elif '\n' in content %} + {%- set _think_end = '\n' %} + {%- elif content.startswith('') and '' in content %} + {%- set _think_end = '' %} + {%- elif content.startswith('') and '' in content %} + {%- set _think_end = '' %} + {%- endif %} + {%- if _think_end %} + {%- if 'thinking' in _think_end %} + {%- set _think_start = '' %} + {%- else %} + {%- set _think_start = '' %} + {%- endif %} + {%- set reasoning_content = content.split(_think_end)[0].rstrip('\n') %} + {%- if _think_start in reasoning_content %} + {%- set reasoning_content = reasoning_content.split(_think_start)[-1].lstrip('\n') %} + {%- endif %} + {%- set content = content.split(_think_end)[-1].lstrip('\n') %} + {%- endif %} + {%- endif %} + {%- set reasoning_content = reasoning_content | trim %} + {%- if (_preserve_thinking or loop.index0 > ns.last_query_index) %} + {{- '<|im_start|>assistant\n\n' + reasoning_content + '\n\n\n' + content }} + {%- else %} + {{- '<|im_start|>assistant\n' + content }} + {%- endif %} + {%- if message.tool_calls is defined and message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %} + {%- for tool_call in message.tool_calls %} + {%- if tool_call.function is defined and tool_call.function is not none %} + {%- set tc = tool_call.function %} + {%- else %} + {%- set tc = tool_call %} + {%- endif %} + {%- set tc_name = tc.name if (tc.name is defined and tc.name is not none) else '' %} + {%- if _tool_format == 'json' %} + {%- if loop.first %} + {%- if content | trim %} + {{- '\n\n' }} + {%- endif %} + {%- else %} + {{- '\n' }} + {%- endif %} + {%- set _args = '{}' %} + {%- if tc.arguments is defined and tc.arguments is not none %} + {%- if tc.arguments is mapping %} + {%- set _args = tc.arguments | tojson %} + {%- elif tc.arguments is string %} + {%- if tc.arguments %} + {%- set _args = tc.arguments %} + {%- endif %} + {%- else %} + {%- set _args = tc.arguments | tojson %} + {%- endif %} + {%- endif %} + {{- '\n{"name": ' }}{{- tc_name | tojson }}{{- ', "arguments": ' }}{{- _args }}{{- '}\n' }} + {%- else %} + {%- if loop.first %} + {%- if content | trim %} + {{- '\n\n\n\n' }} + {%- else %} + {{- '\n\n' }} + {%- endif %} + {%- else %} + {{- '\n\n\n' }} + {%- endif %} + {%- if tc.arguments is defined and tc.arguments is not none %} + {%- if tc.arguments is mapping %} + {%- for args_name, args_value in tc.arguments.items() %} + {{- '\n' }} + {%- if args_value is string %} + {%- set _av = args_value %} + {%- else %} + {%- set _av = args_value | tojson %} + {%- endif %} + {%- if max_tool_arg_chars > 0 and _av | length > max_tool_arg_chars %} + {{- _av[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_av | length | string) ~ ' chars]' }} + {%- else %} + {{- _av }} + {%- endif %} + {{- '\n\n' }} + {%- endfor %} + {%- else %} + {%- if tc.arguments is string %} + {%- set _raw_args = tc.arguments %} + {%- else %} + {%- set _raw_args = tc.arguments | tojson %} + {%- endif %} + {%- if _raw_args %} + {%- if max_tool_arg_chars > 0 and _raw_args | length > max_tool_arg_chars %} + {{- _raw_args[:max_tool_arg_chars] + '\n[TRUNCATED - original length ' ~ (_raw_args | length | string) ~ ' chars]' }} + {%- else %} + {{- _raw_args }} + {%- endif %} + {%- endif %} + {%- endif %} + {%- endif %} + {{- '\n' }} + {%- endif %} + {%- endfor %} + {%- endif %} + {{- '<|im_end|>\n' }} + {%- elif message.role == 'tool' %} + {%- set _content_lower = content | lower %} + {%- set _content_head = _content_lower[:120] %} + {%- set _is_code_or_grep = ('throw new ' in _content_lower or 'throw error' in _content_lower or 'console.error' in _content_lower or 'logger.error' in _content_lower or 'logging.error' in _content_lower or 'import ' in _content_head or 'def ' in _content_head or 'function ' in _content_head) %} + {%- set _exit_code_zero = ('exit code: 0' in _content_head or 'process exited with code 0' in _content_head) %} + {%- set _error_field_ok = ('"error": null' in _content_head or '"error":null' in _content_head or '"error": false' in _content_head or '"error":false' in _content_head or '"error": ""' in _content_head or '"error":""' in _content_head) %} + {%- set _strong_error = (('"error":' in _content_head and not _error_field_ok) or '"status": "error"' in _content_head or '"status":"error"' in _content_head or 'traceback (most recent call last):' in _content_head or 'command not found' in _content_head or 'invalid syntax' in _content_head or 'fatal:' in _content_head or (('exit code: ' in _content_head or 'process exited with code' in _content_head) and not _exit_code_zero) or _content_head.startswith('exception:') or _content_head.startswith('failed to ')) %} + {%- set _weak_error = ('error:' in _content_head or 'err!' in _content_head) %} + {%- set _weak_suppressed = ('$ ' in _content_head or 'took ' in _content_head or content | length >= 600) %} + {%- if not _is_code_or_grep and (_strong_error or (_weak_error and not _weak_suppressed)) %} + {%- set ns2.consecutive_failures = ns2.consecutive_failures + 1 %} + {%- else %} + {%- set ns2.consecutive_failures = 0 %} + {%- endif %} + {%- if ns2.prev_role != 'tool' %} + {{- '<|im_start|>user' }} + {%- endif %} + {%- set _is_json_payload = (_tool_format == 'json' and content | trim | length > 0 and (content | trim)[:1] in ('{', '[')) %} + {%- if not _is_json_payload and max_tool_response_chars > 0 and content | length > max_tool_response_chars %} + {%- set content = content[:max_tool_response_chars] + '\n[TRUNCATED - original length ' ~ (content | length | string) ~ ' chars]' %} + {%- endif %} + {{- '\n\n' + content }} + {%- if ns2.consecutive_failures >= 2 %} + {{- '\n\n⚠️ SYSTEM WARNING: ' ~ ns2.consecutive_failures ~ ' consecutive tool errors detected. Your previous approach is incorrect. You MUST use a fundamentally different approach or corrected arguments.' }} + {%- elif ns2.consecutive_failures == 1 %} + {{- '\n\n⚠️ SYSTEM WARNING: The previous tool call returned an error. Diagnose the failure and retry with completely corrected arguments.' }} + {%- endif %} + {{- '\n' }} + {%- if loop.last %} + {{- '<|im_end|>\n' }} + {%- else %} + {%- set _next_role = _msgs[loop.index0 + 1].role %} + {%- if _next_role != 'tool' %} + {{- '<|im_end|>\n' }} + {%- endif %} + {%- endif %} + {%- else %} + {{- '<|im_start|>user\n[' + message.role + ']: ' + content + '<|im_end|>\n' }} + {%- endif %} + {%- set ns2.prev_role = message.role %} +{%- endfor %} +{%- if add_generation_prompt %} + {{- '<|im_start|>assistant\n' }} + {%- if not ns_state.thinking %} + {{- '\n\n\n\n' }} + {%- else %} + {{- '\n' }} + {%- endif %} +{%- endif %} \ No newline at end of file diff --git a/dankserver/frank-medium/radiance-entrypoint.sh b/dankserver/frank-medium/radiance-entrypoint.sh index e543176..6b748c9 100755 --- a/dankserver/frank-medium/radiance-entrypoint.sh +++ b/dankserver/frank-medium/radiance-entrypoint.sh @@ -1,6 +1,8 @@ #!/usr/bin/env bash -# frank-medium container entrypoint: vllm-radiance MXFP4 on 2x Radeon AI PRO -# R9700 (gfx1201), serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq. +# frank-medium container entrypoint: vllm-radiance on 2x Radeon AI PRO R9700 +# (gfx1201). Quantization-aware (config.json quant_method): fp8 -> the plain +# FP8 path; mxfp4/quark -> the MXFP4 patch-and-serve path; paroquant_mxfp6 -> +# the ParoQuant W4A8 stack + the MXFP6 kernels (PR #51's launch6_p path). # # This is a fork of the patch-and-serve block inside radiance-vllm-mxfp4's # serve-mxfp4.sh (clone at /home/dank/GitPackages/radiance-vllm-mxfp4, pinned @@ -42,17 +44,25 @@ if [ "$SPEC_METHOD" = dflash ]; then : "${SPEC:=7}"; else : "${SPEC:=4}"; fi : "${MIN_M:=0}" # 0 = the W4A8 kernel serves every M (MIN_M=1 still routes M=1 to aiter) : "${R4D_ATTN:=1}" : "${ASYNC:=0}" +: "${GPU_UTIL:=0.99}" # Quantization-aware: read the model's weight quantization from config.json. # fp8 -> the plain FP8 path (no MXFP4 patches, --quantization=fp8, MTP # self-speculation, ROCM_AITER unified attention with the R4D all-reduce). # mxfp4/quark (or unknown) -> the MXFP4 patch-and-serve path (the default). QUANT="$(python3 -c 'import json,sys;print(json.load(open(sys.argv[1])).get("quantization_config",{}).get("quant_method",""))' "$SNAP/config.json" 2>/dev/null || true)" +IS_MXFP6=0 case "$QUANT" in fp8) IS_MXFP4=0 ;; + paroquant_mxfp6) IS_MXFP4=1; IS_MXFP6=1 ;; mxfp4|quark|"") IS_MXFP4=1 ;; *) IS_MXFP4=1; echo "[radiance] WARNING: quantization '$QUANT'; defaulting to the MXFP4 path" >&2 ;; esac +# MXFP6 (ParoQuant) requires TP>=2 (rad_require_tp2 in the canonical launcher); +# ${TP:-1} keeps the check set -u safe when the quadlet omits TP. +if [ "$IS_MXFP6" = "1" ]; then + [ "${TP:-1}" -ge 2 ] || { echo "frank-medium: paroquant_mxfp6 requires TP>=2 (got ${TP:-1})" >&2; exit 1; } +fi if [ "$IS_MXFP4" = "0" ]; then ATTN=ROCM_AITER_UNIFIED_ATTN elif [ "$R4D_ATTN" = "1" ]; then ATTN=R4D @@ -96,6 +106,47 @@ export RADIANCE_DFLASH_SELECTOR_TOPK="${RADIANCE_DFLASH_SELECTOR_TOPK:-}" export RADIANCE_R4D_REPORT=1 export RADIANCE_PRESHUFFLE=${RADIANCE_PRESHUFFLE:-1} export RADIANCE_MXFP4_SANITIZE="${RADIANCE_MXFP4_SANITIZE:-0}" +# ---------------------------------------------------------------- mxfp6 profile +# The ParoQuant W4A8 stack (PR #51, the launch6_p kernel). Mirrors +# paroquant/run_paroquant.sh's env. Runs BEFORE the MXFP4 base block below so +# the base block's ${VAR:-default} guards preserve these values where they +# conflict (GDN_MERGE_INPROJ=0, MXFP4_DECODE_MAX_M=64); the base block fills +# the rest. Gated on IS_MXFP6 so the mxfp4/fp8 paths are untouched. +if [ "$IS_MXFP6" = "1" ]; then + export RADIANCE_PAROQUANT=1 + export RADIANCE_GDN_MERGE_INPROJ=${RADIANCE_GDN_MERGE_INPROJ:-0} + export RADIANCE_GDN_FUSED_MAX_ITEMS=${RADIANCE_GDN_FUSED_MAX_ITEMS:-32} + export RADIANCE_USE_R4D_AR_QUANT=${RADIANCE_USE_R4D_AR_QUANT:-1} + export RADIANCE_FUSE_RMS_QUANT=${RADIANCE_FUSE_RMS_QUANT:-1} + export RADIANCE_KV_GROUP_OPT=${RADIANCE_KV_GROUP_OPT:-1} + export RADIANCE_MXFP4_DECODE_MAX_M=${RADIANCE_MXFP4_DECODE_MAX_M:-64} + export RADIANCE_PQ_CHECKALL=${RADIANCE_PQ_CHECKALL:-} + export RADIANCE_PQM_CHECKALL=${RADIANCE_PQM_CHECKALL:-} + export RADIANCE_PQM_CHECK_MAX_M=${RADIANCE_PQM_CHECK_MAX_M:-128} + export RADIANCE_PQ_CHECK_MAX_M=${RADIANCE_PQ_CHECK_MAX_M:-128} + export RADIANCE_PQ_DECODE_MAX_M=${RADIANCE_PQ_DECODE_MAX_M:-64} + export RADIANCE_PQ_WPERM=${RADIANCE_PQ_WPERM:-1} + export RADIANCE_PQ_DECODE_NT=${RADIANCE_PQ_DECODE_NT:-1} + export RADIANCE_PQ_ATILED=${RADIANCE_PQ_ATILED:-1} + export RADIANCE_PQ_AT_LBK=${RADIANCE_PQ_AT_LBK:-128} + export RADIANCE_PQ_AT_HOIST=${RADIANCE_PQ_AT_HOIST:-1} + export RADIANCE_PQ_PTOK=${RADIANCE_PQ_PTOK:-1} + export RADIANCE_PQ_FUSED_TOKQ=${RADIANCE_PQ_FUSED_TOKQ:-1} + export RADIANCE_PQ_I8=${RADIANCE_PQ_I8:-0} + export RADIANCE_PQ_PG=${RADIANCE_PQ_PG:-0} + export RADIANCE_PQ_ZPE=${RADIANCE_PQ_ZPE:-0} + export RADIANCE_PQ_PG_PRODUCER=${RADIANCE_PQ_PG_PRODUCER:-3} + export RADIANCE_PQ_ROT_STREAM=${RADIANCE_PQ_ROT_STREAM:-1} + export RADIANCE_PQ_ROT_STREAM2=${RADIANCE_PQ_ROT_STREAM2:-1} + export RADIANCE_PQ_ROT_STREAM3=${RADIANCE_PQ_ROT_STREAM3:-0} + export RADIANCE_PQ_AR_CHECK=${RADIANCE_PQ_AR_CHECK:-0} + export RADIANCE_PQ_AR_FALLBACK=${RADIANCE_PQ_AR_FALLBACK:-0} + export RADIANCE_PQ_ROT_V2=${RADIANCE_PQ_ROT_V2:-1} + export RADIANCE_PQ_HIPCC_FLAGS=${RADIANCE_PQ_HIPCC_FLAGS:-} + export RADIANCE_PQM_FUSED_TOKQ=${RADIANCE_PQM_FUSED_TOKQ:-1} + export RADIANCE_PQM_SINGLE_LAUNCH=${RADIANCE_PQM_SINGLE_LAUNCH:-1} + export RADIANCE_DFLASH_CAPTURE_DIR=${RADIANCE_DFLASH_CAPTURE_DIR:-} +fi if [ "$IS_MXFP4" = "1" ]; then export RADIANCE_MXFP4=1 RADIANCE_MXFP4_W4A8=1 RADIANCE_MXFP4_W4A8_MIN_M="$MIN_M" else @@ -193,39 +244,95 @@ if [ "$IS_MXFP4" = "1" ]; then # ---------------------------------------------------------------- patch block # Byte-for-byte the launcher's container-start block. The patches come from # the /patches mount (the pinned radiance clone). SP must match the image's -# venv layout; the image tag is pinned in the quadlet. +# venv layout; the image tag is pinned in the quadlet. The mxfp6 branch +# mirrors paroquant/run_paroquant.sh (17 patches, 10 modules, + the paroquant +# kernel module + the sitecustomize registration); the mxfp4 branch keeps the +# existing 20-patch / 11-module list untouched. SP=/opt/vllm/lib/python3.12/site-packages cd /patches -python3 patch_quark_mxfp4.py -python3 patch_nvfp4_mxfp4.py -python3 patch_tp3_pad.py -python3 patch_ar_maxbytes.py -python3 patch_topk_triton_rows.py -python3 patch_dflash_calib.py -python3 patch_dflash_mxfp4_kv.py -python3 patch_rmsquant_fusion.py -python3 patch_verify_head.py -python3 patch_kv_group_size.py -python3 patch_topk_composite.py -python3 patch_gdn_shared_build.py -python3 patch_dflash_selector_topk.py -python3 patch_gdn_merge_inproj.py -python3 patch_dynwidth.py -python3 patch_async_dynwidth.py -python3 patch_step_trace.py -python3 patch_ar_geometry.py -python3 patch_ar_3rank.py -python3 patch_gdn_glue.py -# Non-fatal: fixes content=null on thinking-off requests; not required to serve. -python3 patch_qwen3_thinkoff.py \ - || echo "[radiance] WARNING: thinkoff patch did not apply; thinking-off requests will return empty content" -cp mxfp4-configs/*.json "$SP"/aiter/ops/triton/configs/gemm/ -cp radiance_preamble.py /opt/radiance_preamble.py -cp radiance_nvfp4.py radiance_mxfp4.py radiance_gdn.py radiance_rmsquant.py radiance_drafthead.py \ - radiance_verifyhead.py radiance_gdnmerge.py radiance_aroverlap.py radiance_topk.py \ - radiance_arnq.py radiance_tp3pad.py "$SP"/ -hipcc -O3 -w -std=c++17 -fPIC -shared --offload-arch=gfx1201 $([ "${MXFP4_CUMODE:-0}" = 1 ] && echo -mcumode) \ - $(python3 -m pybind11 --includes) radiance_mxfp4_fp8.hip -o "$SP"/radiance_mxfp4_fp8.so +if [ "$IS_MXFP6" = "1" ]; then + python3 patch_quark_mxfp4.py + python3 patch_ar_maxbytes.py + python3 patch_topk_triton_rows.py + python3 patch_dflash_calib.py + python3 patch_dflash_mxfp4_kv.py + python3 patch_rmsquant_fusion.py + python3 patch_verify_head.py + python3 patch_kv_group_size.py + python3 patch_topk_composite.py + python3 patch_gdn_shared_build.py + python3 patch_async_dynwidth.py + python3 patch_step_trace.py + python3 patch_skinny_gemm.py + python3 patch_dflash_selector_topk.py + python3 patch_dynwidth.py + python3 patch_ar_geometry.py + python3 patch_gdn_merge_inproj.py + # Non-fatal: fixes content=null on thinking-off requests; not required to serve. + python3 patch_qwen3_thinkoff.py \ + || echo "[radiance] WARNING: thinkoff patch did not apply; thinking-off requests will return empty content" + cp mxfp4-configs/*.json "$SP"/aiter/ops/triton/configs/gemm/ + cp radiance_preamble.py /opt/radiance_preamble.py + cp radiance_mxfp4.py radiance_gemm.py radiance_gdn.py radiance_gdnmerge.py radiance_rmsquant.py \ + radiance_drafthead.py radiance_verifyhead.py radiance_aroverlap.py radiance_topk.py \ + radiance_arnq.py "$SP"/ + hipcc -O3 -w -std=c++17 -fPIC -shared --offload-arch=gfx1201 $(python3 -m pybind11 --includes) \ + radiance_mxfp4_fp8.hip -o "$SP"/radiance_mxfp4_fp8.so + # ---- paroquant: build the MXFP6 kernel module and register the quant method ---- + # RADIANCE_PQ_HIPCC_FLAGS: extra compile flags for A/B builds of the kernel + # module only (e.g. -DPQ_HW_CVT=0). Empty in prod. + hipcc -O3 -w -std=c++17 -fPIC -shared --offload-arch=gfx1201 $(python3 -m pybind11 --includes) \ + ${RADIANCE_PQ_HIPCC_FLAGS:-} paroquant/radiance_paroquant.hip -o "$SP"/radiance_paroquant_kernel.so + cp paroquant/radiance_paroquant.py paroquant/radiance_paroquant_mxfp4.py "$SP"/ + cp radiance_dflash_capture.py "$SP"/ 2>/dev/null || true + # NB: appended to the STDLIB sitecustomize, not site-packages -- Ubuntu ships + # /usr/lib/python3.12/sitecustomize.py and it shadows any site-packages one, so + # a file dropped there is silently never imported. Guarded: an unguarded >> + # appends another copy on every restart of an existing container. + if ! grep -q radiance_paroquant /usr/lib/python3.12/sitecustomize.py; then + printf "%s\n" \ + "try:" \ + " import radiance_paroquant # registers the paroquant quantization config" \ + " import radiance_paroquant_mxfp4 # and the MXFP4-weights variant (paroquant_mxfp4)" \ + " import radiance_dflash_capture # drafter training-data capture (inert unless RADIANCE_DFLASH_CAPTURE_DIR)" \ + "except Exception as e:" \ + " import sys" \ + " sys.stderr.write(\"[radiance.paroquant] registration failed: %r\\n\" % (e,))" \ + >> /usr/lib/python3.12/sitecustomize.py + fi +else + python3 patch_quark_mxfp4.py + python3 patch_nvfp4_mxfp4.py + python3 patch_tp3_pad.py + python3 patch_ar_maxbytes.py + python3 patch_topk_triton_rows.py + python3 patch_dflash_calib.py + python3 patch_dflash_mxfp4_kv.py + python3 patch_rmsquant_fusion.py + python3 patch_verify_head.py + python3 patch_kv_group_size.py + python3 patch_topk_composite.py + python3 patch_gdn_shared_build.py + python3 patch_dflash_selector_topk.py + python3 patch_gdn_merge_inproj.py + python3 patch_dynwidth.py + python3 patch_async_dynwidth.py + python3 patch_step_trace.py + python3 patch_ar_geometry.py + python3 patch_ar_3rank.py + python3 patch_gdn_glue.py + # Non-fatal: fixes content=null on thinking-off requests; not required to serve. + python3 patch_qwen3_thinkoff.py \ + || echo "[radiance] WARNING: thinkoff patch did not apply; thinking-off requests will return empty content" + cp mxfp4-configs/*.json "$SP"/aiter/ops/triton/configs/gemm/ + cp radiance_preamble.py /opt/radiance_preamble.py + cp radiance_nvfp4.py radiance_mxfp4.py radiance_gdn.py radiance_rmsquant.py radiance_drafthead.py \ + radiance_verifyhead.py radiance_gdnmerge.py radiance_aroverlap.py radiance_topk.py \ + radiance_arnq.py radiance_tp3pad.py "$SP"/ + cp radiance_dflash_capture.py "$SP"/ 2>/dev/null || true + hipcc -O3 -w -std=c++17 -fPIC -shared --offload-arch=gfx1201 $([ "${MXFP4_CUMODE:-0}" = 1 ] && echo -mcumode) \ + $(python3 -m pybind11 --includes) radiance_mxfp4_fp8.hip -o "$SP"/radiance_mxfp4_fp8.so +fi if [ -n "${R4D_SO:-}" ] && [ -f /r4d/r4d.so ]; then cp /r4d/r4d.so "$SP"/r4d.so echo "[radiance] using patched r4d.so from $R4D_SO" @@ -234,8 +341,9 @@ fi # the one just compiled into site-packages (cwd precedes it on sys.path). cd / fi -# The weight-quantization flag. MXFP4 is auto-detected from the checkpoint; -# FP8 is passed explicitly (the documented plain-FP8 path). +# The weight-quantization flag. MXFP4 and paroquant_mxfp6 are auto-detected +# from the checkpoint's quant_method; FP8 is passed explicitly (the documented +# plain-FP8 path). QUANT_FLAG="" if [ "$IS_MXFP4" = "0" ]; then QUANT_FLAG="--quantization fp8"; fi exec /opt/radiance_entrypoint.sh "$SNAP" --served-model-name $SERVED_NAMES \ diff --git a/scripts/deploy-remote.sh b/scripts/deploy-remote.sh index 44c8de8..a6ba645 100755 --- a/scripts/deploy-remote.sh +++ b/scripts/deploy-remote.sh @@ -1178,13 +1178,15 @@ deploy_llm_small() { wait_for_health frank-small http://127.0.0.1:7771/health 900 } -# deploy_llm_medium — frank-medium is the vllm-radiance MXFP4 stack as a +# deploy_llm_medium — frank-medium is the vllm-radiance stack as a # rootless Quadlet since the 2026-09-20 cutover: one tensor-parallel (TP=2) # engine across both R9700s, R4D one-shot all-reduce, DFlash2-FP8 drafter. -# It replaced mainline llama.cpp's layer split, which held the endpoint since -# 2026-09-16; the conversation battery measured radiance at 2.1x decode and a -# third of the time-to-first-token with 4x the concurrent throughput -# (scripts/bench-conversation.py, 2026-09-20). +# The entrypoint is quantization-aware (config.json quant_method): the plain +# FP8 path, the MXFP4 patch-and-serve path, and the ParoQuant MXFP6 stack +# (PR #51's launch6_p kernel). It replaced mainline llama.cpp's layer split, +# which held the endpoint since 2026-09-16; the conversation battery measured +# radiance at 2.1x decode and a third of the time-to-first-token with 4x the +# concurrent throughput (scripts/bench-conversation.py, 2026-09-20). # # WHY THE PAIR IS BACK AND THE TRIO IS NOT: the 2026-09-16 cutover retired this # unit because TP=2 could not dispatch a collective on the second card (B650 @@ -1197,13 +1199,13 @@ deploy_llm_small() { # pinned radiance patch clone, the patched libr4d build and the Triton compile # cache live outside the checkout (see the unit file's mount comments). The # gates below refuse a deploy that would start a container guaranteed to die. -LLM_MEDIUM_IMAGE="${LLM_MEDIUM_IMAGE:-docker.io/stilldeadcode/vllm-radiance:0.9.3}" -LLM_MEDIUM_IMAGE_LOCAL="${LLM_MEDIUM_IMAGE_LOCAL:-localhost/radiance-vllm:0.9.3}" -LLM_MEDIUM_SNAP="${LLM_MEDIUM_SNAP:-/home/dank/models/Qwen3.8-27B-Uncensored-MXFP4-awq}" +LLM_MEDIUM_IMAGE="${LLM_MEDIUM_IMAGE:-ggz14/vllm-radiance-mxfp4:0.13.0-e060895}" +LLM_MEDIUM_IMAGE_LOCAL="${LLM_MEDIUM_IMAGE_LOCAL:-localhost/radiance-vllm:0.13.0-e060895}" +LLM_MEDIUM_SNAP="${LLM_MEDIUM_SNAP:-/home/dank/models/Qwen3.8-27B-PARO-MXFP6}" LLM_MEDIUM_DRAFTER="${LLM_MEDIUM_DRAFTER:-/home/dank/models/Qwen3.8-27B-DFlash2-FP8}" LLM_MEDIUM_PATCHES="${LLM_MEDIUM_PATCHES:-/home/dank/GitPackages/radiance-vllm-mxfp4}" LLM_MEDIUM_R4D="${LLM_MEDIUM_R4D:-/home/dank/.cache/radiance-libr4d/b9e42ab-rx6}" -LLM_MEDIUM_CACHE="${LLM_MEDIUM_CACHE:-/home/dank/.radiance-cache-w4a8-093-gdnm-nqft-fp8s-gnq}" +LLM_MEDIUM_CACHE="${LLM_MEDIUM_CACHE:-/home/dank/.radiance-cache-mxfp6-013-pq-nqft-fp8s-gnq}" # The three replica/proxy user units from the 2026-09-16 era are retired HERE, # never left to rot: they must not exist alongside the TP=2 unit (the proxy @@ -1256,8 +1258,8 @@ deploy_llm_medium() { # of truth), so a model swap is just "edit the quadlet + deploy". The # container /models mount maps to /home/dank/models on the host. The # drafter gate applies only to the dflash path; the patched libr4d gate - # applies only to the mxfp4/quark path (the plain FP8 path uses the - # image's built-in r4d.so and skips the MXFP4 patches). + # applies to the mxfp4/quark/paroquant_mxfp6 path (the plain FP8 path uses + # the image's built-in r4d.so and skips the MXFP4 patches). local quadlet_file="dankserver/frank-medium/quadlet/frank-medium.container" local snap_c snap spec_method quant snap_c="$(sed -nE 's/^Environment=SNAP=//p' "$quadlet_file" | head -1)" @@ -1294,7 +1296,7 @@ deploy_llm_medium() { # NaNs the MXFP4 model, so the patched build is not optional there. The # plain FP8 path uses the image's built-in r4d.so (no MXFP4 patches). case "$quant" in - mxfp4|quark) + mxfp4|quark|paroquant_mxfp6) if [[ ! -f "$LLM_MEDIUM_R4D/r4d.so" ]]; then echo "ERROR: patched libr4d absent at $LLM_MEDIUM_R4D" >&2 missing=1 diff --git a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md index 63a4fbf..8a03adb 100644 --- a/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md +++ b/serverAppConfigs-docs/LLM/frank-medium-radiance-mxfp4.md @@ -1,48 +1,47 @@ --- type: configuration -title: frank-medium vllm-radiance MXFP4 Runbook -description: ACTIVE frank-medium runbook — vllm-radiance serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8, DFlash2 drafter, unquantized bf16 KV) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). Returned to MXFP4 on 2026-09-21 with KV_DTYPE=auto (clean KV) to test whether the endpoint's historical fp8 KV had been degrading W4A8 attention quality; the FP8 checkpoint (orcarouter/Qwen3.8-27B-Uncensored-FP8, the 2026-09-20 to 09-21 shape) is the on-disk rollback. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era. +title: frank-medium vllm-radiance MXFP6 Runbook +description: ACTIVE frank-medium runbook — vllm-radiance serving hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6 (ParoQuant MXFP6 W4A8, DFlash2 drafter, unquantized bf16 KV) as ONE tensor-parallel process (TP=2) on Dankserver's two AMD Radeon AI PRO R9700 (gfx1201). Cut to MXFP6 on 2026-09-21 (PR #51's launch6_p kernel, image 0.13.0-e060895); the MXFP4 checkpoint (just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq, the 2026-09-21 shape) is the on-disk rollback. Model-swap process: [[frank-medium-model-swap]]. The sections below describe the 2026-09-16 two-replica era. tags: - frank-medium - dankserver - vllm - vllm-radiance + - mxfp6 - mxfp4 - gfx1201 - r9700 - rccl - pcie-atomics - quadlet -timestamp: 2026-09-20 +timestamp: 2026-09-21 --- -# frank-medium vllm-radiance MXFP4 Runbook +# frank-medium vllm-radiance MXFP6 Runbook -The ACTIVE frank-medium deployment: the `stilldeadcode/vllm-radiance:0.9.3` -image (tagged locally as `localhost/radiance-vllm:0.9.3`) plus the -`ggz14/radiance-vllm-mxfp4` patches, serving -`just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` as **ONE -tensor-parallel process across both Radeon AI PRO R9700 (gfx1201)**. -It replaced the mainline-llama.cpp layer-split deployment -([[frank-medium-llamacpp-dflash2]], SUPERSEDED 2026-09-20); the same image and -model had earlier run here as two single-GPU replicas, which is the shape most -of this page describes. +The ACTIVE frank-medium deployment: the `ggz14/vllm-radiance-mxfp4:0.13.0-e060895` +image (built from PR #51's head, tagged locally as +`localhost/radiance-vllm:0.13.0-e060895`) plus the `radiance-vllm-mxfp4` +patches at e060895, serving `hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6` +(ParoQuant MXFP6 W4A8, 25.16 GiB, 18 shards) as **ONE tensor-parallel +process across both Radeon AI PRO R9700 (gfx1201)**. It replaced the +mainline-llama.cpp layer-split deployment ([[frank-medium-llamacpp-dflash2]], +SUPERSEDED 2026-09-20); the same image and model had earlier run here as two +single-GPU replicas, which is the shape most of this page describes. -**Serving shape since 2026-09-21: MXFP4 with unquantized (bf16) KV.** This -endpoint briefly served `orcarouter/Qwen3.8-27B-Uncensored-FP8` (FP8 E4M3, -`--quantization fp8`, DFlash2 drafter) from 2026-09-20 to 2026-09-21; it has -since been cut back to the MXFP4 checkpoint above, now with -`--kv-cache-dtype auto` (unquantized bf16 KV) instead of the endpoint's -historical `fp8`. Hypothesis: fp8 KV sat on *both* the FP8 and MXFP4 arms (the -flag was on the common exec path), so it could not explain the FP8-vs-MXFP4 -*throughput* gap — but MXFP4's weights are already W4A8, and fp8 KV on top -compounds attention-path precision loss, which shows up as degraded *output -quality*, not t/s. **Measured on deploy**: `kv_cache_dtype=auto`, no fp8 -banner, pool **486,464 tokens** (`Maximum concurrency for 262,144 tokens per -request: 1`) — exactly half MXFP4's 943,581-token fp8 pool, still clearing the -full 262,144 window so `MAXLEN` stays native. No quality verdict yet; this is -the setup. Rollback: flip `SNAP` + `SPEC` back to the FP8 path with -`KV_DTYPE=fp8` (that shape auto-sized to ~384k and held the full window). +**Serving shape since 2026-09-21: ParoQuant MXFP6 with unquantized (bf16) KV.** +This endpoint cut to the MXFP6 checkpoint above (PR #51's `launch6_p` kernel, +`quant_method=paroquant_mxfp6`, bits=6, group_size=128, krot=8, rotation on, +e4m3fn scale) with `--kv-cache-dtype auto` (unquantized bf16 KV) and +`--gpu-memory-utilization 0.99`. The Swift variant is a finetune of +`ukisai/Swift-Qwen3.8-27b` (a different base from `Qwen/Qwen3.8-27B`), so the +DFlash2-FP8 drafter's hidden-state distribution is a mismatch — acceptance may +drop from the MXFP4 era's 4.24/7, but it is still a net win over no spec +decode. **The MXFP4 checkpoint is the on-disk rollback**: flip `SNAP` back to +`/models/Qwen3.8-27B-Uncensored-MXFP4-awq`, `GPU_UTIL` to 0.98, and the image +back to `localhost/radiance-vllm:0.9.3` (re-pull `docker.io/stilldeadcode/ +vllm-radiance:0.9.3` if it is no longer on disk; the MXFP4 shape auto-sized to +486,464 tokens with bf16 KV and held the full 262,144 window). **Deployed shape since 2026-09-20: one rootless Quadlet, `--tensor-parallel-size 2`.** The ProArt B850-Creator WiFi NEO board swap put both R9700s on CPU GPP root @@ -59,31 +58,30 @@ From `dankserver/frank-medium/quadlet/frank-medium.container`: | Setting | Value | |---|---| -| Model | `just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq` (MXFP4 / quark, W4A8, 19.37 GiB single shard) | +| Model | `hugypufy/Swift-Qwen3.8-27B-PARO-MXFP6` (ParoQuant MXFP6 W4A8, 25.16 GiB, 18 shards; a finetune of `ukisai/Swift-Qwen3.8-27b`) | | Speculative | `--speculative-config {"method":"dflash","model":"/models/Qwen3.8-27B-DFlash2-FP8","num_speculative_tokens":7}` (the z-lab DFlash2 external drafter; mean acceptance 4.24/7. A/B tested against MTP self-speculation — DFlash2 wins on decode, see below) | | Parallelism | `--tensor-parallel-size 2` (`ROCR_VISIBLE_DEVICES=0,1`) | | Context | `--max-model-len 262144` | | Concurrency | `--max-num-seqs 8`, chunk 8192 | -| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool 486,464 tokens**; the endpoint's historical fp8 KV halved precision on top of MXFP4's W4A8 (rollback: `KV_DTYPE=fp8` -> 943,581 tokens) | +| KV | auto = **unquantized bf16** (`--kv-cache-dtype auto`), **auto-sized — pool ~730k tokens** (estimated at 0.99 fill: MXFP6's 25.16 GiB weights are ~12.58 GiB per card, leaving ~19.2 GiB per card for KV+activations); rollback: `KV_DTYPE=fp8` | | Prompt cache | `--enable-prefix-caching` | -| GPU fill | `--gpu-memory-utilization 0.98` | +| GPU fill | `--gpu-memory-utilization 0.99` | | Parsers | `--enable-auto-tool-choice --tool-call-parser qwen3_coder`, `--reasoning-parser qwen3` | | Sampler floor | `--override-sampling-effects` (temp 0.7 / top_p 0.95 / top_k 20) | | Multimodal | `--limit-mm-per-prompt '{"image": 8}'` | -| Chat template | `qwen-sharp-v22.4.0.jinja` (froggeric Qwen-Sharp v22.4.0, repo-tracked at `dankserver/frank-medium/`, mounted at `/templates/`; terse system prompt by default — opt out per-request with `chat_template_kwargs {"terse": false}`; rollback: flip `CHAT_TEMPLATE` back to `/patches/qwen-fixed-v22.3.jinja`) | +| Chat template | `qwen-sharp-v22.5.0.jinja` (peculiar-ragdoll Qwen-Sharp v22.5.0, repo-tracked at `dankserver/frank-medium/`, mounted at `/templates/`; terse system prompt by default — opt out per-request with `chat_template_kwargs {"terse": false}`; rollback: flip `CHAT_TEMPLATE` back to `/templates/qwen-sharp-v22.4.0.jinja`) | **The KV pool is auto-sized, not pinned.** Production ships no `KV_MEM` pin (`KV_MEM=0`): the pool comes up at whatever VRAM the weights + drafter leave. -MXFP4's weights are smaller than FP8's (19.4 vs 30.9 GiB), which frees VRAM for -the pool — so even with unquantized bf16 KV the full window fits. The 2026-09-21 -clean-KV start auto-sized to **486,464 tokens** (`GPU KV cache size: 486,464 -tokens`, `Maximum concurrency for 262,144 tokens per request: 1`) — 1.86x the -full window, and exactly half MXFP4's fp8-KV pool of 943,581 (the 2x f16-vs-fp8 -KV ratio). For reference the FP8 checkpoint (now the rollback) auto-sized to -384,238 with fp8 KV (1.47x) and would only reach ~192,000 unquantized — which -is why FP8 and MXFP4 differ on the window, and why MXFP4 is the shape that -affords clean KV at the native 262,144. After the first healthy start, -recalibrate with `calibrate-kv.sh`. +MXFP6's weights are 25.16 GiB (~12.58 GiB per card at TP=2), which at 0.99 fill +(34.2 GiB per card) leaves ~19.2 GiB per card for KV + activations. The +estimated pool is **~730k tokens** (the actual size is determined at boot; the +full 262,144 window fits with ~2.7x headroom). The DFlash2-FP8 drafter costs +2.1 GiB of VRAM on top. For reference the MXFP4 checkpoint (now the rollback) +auto-sized to 486,464 tokens with bf16 KV (1.86x the window), and the FP8 +checkpoint would only reach ~192,000 unquantized — which is why MXFP6's larger +weights still afford clean KV at the native 262,144 (the 0.99 fill buys the +pool back). After the first healthy start, recalibrate with `calibrate-kv.sh`. ## DFlash2 vs MTP — A/B on the FP8 checkpoint (2026-09-20) diff --git a/tests/frank_medium_unit.bats b/tests/frank_medium_unit.bats index 21a40d5..f290440 100755 --- a/tests/frank_medium_unit.bats +++ b/tests/frank_medium_unit.bats @@ -95,10 +95,11 @@ env_str() { # DFlash2 drafts at most 7 tokens per step; deeper is a different # drafter's contract and costs acceptance, not just VRAM. [ "$spec" -eq 7 ] - # The MXFP4 decode kernel band covers MAXSEQS x (SPEC+1) rows, and the - # shipped default covers 64. Exceeding it without the entrypoint's >8 - # bump is a launch failure at the worst moment (first concurrent - # request). + # The MXFP4 / ParoQuant-MXFP6 decode kernel band covers MAXSEQS x + # (SPEC+1) rows, and the shipped default covers 64 (the mxfp6 profile + # block exports the same RADIANCE_MXFP4_DECODE_MAX_M=64). Exceeding it + # without the entrypoint's >8 bump is a launch failure at the worst + # moment (first concurrent request). [ $(( maxseqs * (spec + 1) )) -le 64 ] grep -q 'RADIANCE_MXFP4_DECODE_MAX_M' "$ENTRY" ;;