Update AGENTS.md with target-specific branching and service status

- vllm-awq-qwen36-dflash-ampere.sh: AWQ-4bit Qwen3.6-35B-A3B with DFlash
  speculative decoding, optimized for Ampere hardware
  - Model: cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit (AWQ Marlin)
  - Draft: z-lab/Qwen3.6-35B-A3B-DFlash
  - --max-model-len 32768
  - --max-num-batched-tokens 1536
  - --cpu-offload-gb 40 (experts)
  - --reasoning-parser qwen3
  - --tool-call-parser qwen3_xml
  - Port: 34401
  - Note: prefix caching disabled (Mamba cache align conflict)

- scripts/validate-vllm.sh: Per-stream latency tests
  - Separate tests for 1, 2, 3, 4 streams
  - Different prompts per stream to avoid caching effects
  - Reports elapsed ms per stream count

AGENTS.md

@@ -0,0 +1,103 @@
+# vLLM Scripts — Agent Instructions
+
+## Project Overview
+Scripts for running large language models via vLLM on two local machines:
+- **dankserver**: NVIDIA GPU + System RAM (frank-default-small)
+- **dankclaw**: AMD Strix Halo Shared Memory (frank-default-large)
+
+## Directory Structure
+```
+vllmScripts/
+├── AGENTS.md          # This file — agent instructions
+├── README.md          # Human-readable project overview
+├── deploy.sh          # Automated deployment script
+├── scripts/           # Executable vLLM launch scripts
+│   ├── validate-vllm.sh  # Validation tool
+│   └── *.sh             # Inference server scripts
+├── tests/             # Test suite
+│   └── test_validator.sh
+├── docs/              # Knowledge base
+│   ├── README.md
+│   └── *.md           # Research notes
+└── failedAttempts/    # Archived scripts (for reference)
+```
+
+## Running Tests
+```bash
+# Test validator on dankserver
+bash tests/test_validator.sh dankserver
+
+# Test validator on dankclaw
+bash tests/test_validator.sh dankclaw
+
+# Run all tests
+bash tests/test_validator.sh dankserver && bash tests/test_validator.sh dankclaw
+```
+
+## Deployment
+```bash
+# Deploy all scripts to all targets
+bash deploy.sh all
+
+# Deploy specific script to dankserver
+bash deploy.sh dankserver vllm-qwen36.sh
+
+# Deploy specific script to dankclaw
+bash deploy.sh dankclaw vllm-thinkingcap.sh
+```
+
+## Git Workflow
+
+### Branching Strategy
+- `main`: Stable, working scripts only
+- `dankserver/*`: Server-specific development
+  - `dankserver/qwen36-awq`: Qwen3.6 AWQ experiments
+  - `dankserver/*`: Feature branches for dankserver
+- `dankclaw/*`: Claw-specific development
+  - `dankclaw/*`: Feature branches for dankclaw
+
+### Worktree Pattern
+For parallel development:
+```bash
+# Create worktree for feature
+git worktree add ../vllm-feature feature/qwen36-awq
+
+# When done, merge back
+git worktree remove ../vllm-feature
+```
+
+### Commit Guidelines
+- Use descriptive commit messages: `git commit -m "Add AWQ DFlash script"`
+- Commit before deploying: `git add -A && git commit -m "Update for deployment"`
+- Push to remote: `git push origin <branch>`
+
+### Deployment Flow
+1. Make changes locally
+2. Run tests: `bash tests/test_validator.sh dankserver`
+3. Commit: `git add -A && git commit -m "Description"`
+4. Deploy: `bash deploy.sh dankserver`
+5. Verify: `bash scripts/validate-vllm.sh -t dankserver`
+
+## Remote Servers
+
+### dankserver
+- SSH: `dank@dankserver`
+- Scripts: `~/vllm/`
+- Service: `vllm-awq-qwen36.service`
+- Model: frank-default-small (AWQ 4bit)
+- Port: 34401
+- Status: Disabled (pending bug fixes)
+
+### dankclaw
+- SSH: `dank@dankclaw`
+- Scripts: `~/vllm/`
+- Service: `vllm-thinkingcap.service`
+- Model: frank-default-large (FP8)
+- Port: 3090
+- Status: Running (you are the deployed inference)
+
+## Troubleshooting
+- Service failed: `systemctl status <service>`
+- Container stuck: `docker stop <name> && docker rm <name>`
+- OOM error: Reduce `--max-model-len` or increase `--cpu-offload-gb`
+- Prefix caching conflict: Remove `--enable-prefix-caching` from script

README.md

@@ -4,10 +4,27 @@ Scripts for running large language models via vLLM on two local machines.
 
 ## Targets
 
-| Host | User | Codename | Architecture |
-|------|------|----------|-------------|
-| dankserver | dank | DankServer | NVIDIA GPU + System RAM |
-| dankclaw | dank | DankClaw | AMD Strix Halo (Shared Memory) |
+| Host | User | Codename | Architecture | Service Status |
+|------|------|----------|-------------|----------------|
+| dankserver | dank | DankServer | NVIDIA GPU + System RAM | `vllm-awq-qwen36.service` (failed) |
+| dankclaw | dank | DankClaw | AMD Strix Halo (Shared Memory) | `vllm-thinkingcap.service` (failed) |
+
+## File Locations
+
+### Local (vllmScripts/)
+- `scripts/` — Active inference scripts
+- `tests/` — Test suite
+- `docs/` — Knowledge base
+- `failedAttempts/` — Archived scripts
+
+### Remote (dankserver)
+- `~/vllm/` — Deployed inference scripts
+- `~/vllm/failedAttempts/` — Archived scripts
+- `~/vLLM/` — Legacy project directory (vllm-qwen36.sh)
+
+### Remote (dankclaw)
+- `~/vllm/` — vLLM source code (not inference scripts)
+- `~/vLLM/` — Inference scripts (if deployed)
 
 ## Models
 
@@ -15,12 +32,74 @@ Scripts for running large language models via vLLM on two local machines.
 |-------|--------|--------|
 | Qwen3.6 35B A3B FP8 | dankserver | FP8 |
 | Qwen3.6 27B FP8 | dankclaw | FP8 |
+| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ 4bit |
+| Qwen3.6 35B A3B DFlash | dankserver | Draft model |
 
 ## Structure
 
 - `docs/` — Project knowledge base with research notes on vLLM configuration, hardware specifics, and deployment notes
-- `scripts/` — Executable vLLM launch scripts (coming soon)
+- `scripts/` — Executable vLLM launch scripts
+- `vllm-qwen36.sh` — FP8 inference script for dankserver
+- `vllm-awq-qwen36-dflash.sh` — AWQ 4bit with DFlash speculative decoding (docker)
+- `vllm-awq-qwen36-dflash-ampere.sh` — AWQ 4bit with DFlash speculative decoding (Ampere-optimized, systemd)
 
 ## Knowledge Base
 
 See [[docs/README.md]] for the project knowledge base index.
+
+## Lessons Learned
+
+### AWQ + DFlash on Ampere Hardware (2026-08-08)
+
+**What worked:**
+- `cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit` loads successfully on Ampere GPUs
+- AWQ Marlin kernel engaged automatically for weight conversion
+- DFlash draft model `z-lab/Qwen3.6-35B-A3B-DFlash` detected as EAGLE model
+- torch.compile with cudagraphs works with `--max-num-batched-tokens 1536`
+
+**What didn't work:**
+- `--enable-prefix-caching` conflicts with Mamba cache "align" mode — causes `AssertionError` in `unify_kv_cache_spec_page_size`
+- `--max-model-len 262144` causes OOM during AWQ→Marlin weight conversion on 24GB GPU
+- Nightly vLLM `--speculative-config` requires `num_speculative_tokens` when model is provided
+- `--speculative-model` flag removed in nightly; use `--speculative-config` JSON instead
+
+**Working configuration:**
+- `--max-model-len 32768` (32k context)
+- `--cpu-offload-gb 40` with `--cpu-offload-params "experts"`
+- `--gpu-memory-utilization 0.95`
+- `--performance-mode throughput`
+- `--enable-chunked-prefill`
+- `--max-num-batched-tokens 1536`
+- Thread binding: `OMP_NUM_THREADS=8`, `VLLM_CPU_OMP_THREADS_BIND=8`
+
+**Performance observations (no speculative decoding):**
+| Streams | Prompt Speed | Completion Speed | Total Time |
+|---------|-------------|-----------------|------------|
+| 1 | 4.27 tok/s | 21.42 tok/s | 11,951ms |
+| 2 | 2.44 tok/s | 12.25 tok/s | 20,891ms |
+| 3 | 1.35 tok/s | 6.76 tok/s | 37,849ms |
+| 4 | 1.43 tok/s | 7.19 tok/s | 35,606ms |
+
+Note: Speeds decrease with more streams due to KV cache contention.
+
+**Still needs work:**
+- DFlash speculative decoding integration (KV cache assertion bug in nightly)
+- Full context length support (32k is temporary workaround for OOM)
+
+## Validation
+
+Use `scripts/validate-vllm.sh` to test inference servers:
+
+```bash
+# Test dankserver (default)
+bash scripts/validate-vllm.sh -t dankserver
+
+# Test dankclaw
+bash scripts/validate-vllm.sh -t dankclaw
+
+# Custom host/port
+bash scripts/validate-vllm.sh -H 192.168.1.100 -P 8001 -M my-model
+
+# Run tests
+bash tests/test_validator.sh dankserver
+```

deploy.sh

@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# deploy.sh — Deploy inference scripts to remote servers
+#
+# Usage: ./deploy.sh [TARGET] [SCRIPT]
+#   TARGET: dankserver, dankclaw, or all
+#   SCRIPT: Script name to deploy (default: all scripts in scripts/)
+
+TARGET="${1:-all}"
+SCRIPT="${2:-}"
+
+# Remote configuration
+declare -A REMOTE_HOSTS
+REMOTE_HOSTS=(
+    ["dankserver"]="dank@dankserver"
+    ["dankclaw"]="dank@dankclaw"
+)
+
+declare -A REMOTE_DIRS
+REMOTE_DIRS=(
+    ["dankserver"]="/home/dank/vllm"
+    ["dankclaw"]="/home/dank/vllm"
+)
+
+declare -A REMOTE_SERVICES
+REMOTE_SERVICES=(
+    ["dankserver"]="vllm-awq-qwen36.service"
+    ["dankclaw"]="vllm-thinkingcap.service"
+)
+
+deploy_to() {
+    local target="$1"
+    local remote="${REMOTE_HOSTS[$target]}"
+    local remote_dir="${REMOTE_DIRS[$target]}"
+    local service="${REMOTE_SERVICES[$target]}"
+    
+    echo "=== Deploying to $target ==="
+    
+    # Create remote directory if it doesn't exist
+    sshpass -p 'Assface.3241' ssh "$remote" "mkdir -p $remote_dir"
+    
+    # Deploy scripts
+    if [ -n "$SCRIPT" ]; then
+        if [ -f "scripts/$SCRIPT" ]; then
+            echo "  Deploying scripts/$SCRIPT..."
+            sshpass -p 'Assface.3241' scp "scripts/$SCRIPT" "$remote:$remote_dir/$SCRIPT"
+        else
+            echo "  ERROR: scripts/$SCRIPT not found"
+            return 1
+        fi
+    else
+        for script in scripts/*.sh; do
+            if [ -f "$script" ]; then
+                local name=$(basename "$script")
+                echo "  Deploying $name..."
+                sshpass -p 'Assface.3241' scp "$script" "$remote:$remote_dir/$name"
+            fi
+        done
+    fi
+    
+    # Make scripts executable
+    if [ -n "$SCRIPT" ]; then
+        sshpass -p 'Assface.3241' ssh "$remote" "chmod +x $remote_dir/$SCRIPT"
+    else
+        sshpass -p 'Assface.3241' ssh "$remote" "chmod +x $remote_dir/*.sh"
+    fi
+    
+    # Restart service if it exists
+    echo "  Restarting $service..."
+    sshpass -p 'Assface.3241' ssh "$remote" "echo 'Assface.3241' | sudo -S systemctl restart $service 2>/dev/null || echo 'Service $service not found or failed'"
+    
+    echo "  Deploy to $target complete"
+    echo ""
+}
+
+case "$TARGET" in
+    dankserver|dankclaw)
+        deploy_to "$TARGET"
+        ;;
+    all)
+        for target in "${!REMOTE_HOSTS[@]}"; do
+            deploy_to "$target"
+        done
+        ;;
+    *)
+        echo "ERROR: Unknown target $TARGET"
+        echo "Usage: $0 [dankserver|dankclaw|all] [script]"
+        exit 1
+        ;;
+esac
+
+echo "=== Deployment complete ==="

docs/awq-dflash-ampere-notes.md

@@ -0,0 +1,83 @@
+# AWQ + DFlash on Ampere Hardware
+
+## Date: 2026-08-08
+
+## Objective
+Deploy Qwen3.6-35B-A3B with AWQ 4bit quantization and DFlash speculative decoding on NVIDIA Ampere-class GPU.
+
+## What Worked
+
+### Base AWQ Model
+- `cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit` loads successfully
+- AWQ Marlin kernel engaged automatically for weight conversion
+- `--cpu-offload-gb 40` with `--cpu-offload-params "experts"` helps fit model in memory
+- `--gpu-memory-utilization 0.95` is safe for this model
+
+### DFlash Draft Model
+- `z-lab/Qwen3.6-35B-A3B-DFlash` detected as EAGLE model
+- torch.compile with cudagraphs works with `--max-num-batched-tokens 1536`
+- Draft model shares embedding/lm_head weights with target model
+
+### Performance (no speculative decoding)
+| Streams | Prompt Speed | Completion Speed | Total Time |
+|---------|-------------|-----------------|------------|
+| 1 | 4.27 tok/s | 21.42 tok/s | 11,951ms |
+| 2 | 2.44 tok/s | 12.25 tok/s | 20,891ms |
+| 3 | 1.35 tok/s | 6.76 tok/s | 37,849ms |
+| 4 | 1.43 tok/s | 7.19 tok/s | 35,606ms |
+
+Note: Speeds decrease with more streams due to KV cache contention.
+
+## What Didn't Work
+
+### Prefix Caching + Mamba Cache Conflict
+- **Error**: `AssertionError` in `unify_kv_cache_spec_page_size`
+- **Cause**: Mamba cache "align" mode conflicts with prefix caching
+- **Fix**: Remove `--enable-prefix-caching` from command line
+
+### OOM with 262144 Context
+- **Error**: CUDA out of memory during AWQ→Marlin weight conversion
+- **Cause**: 24GB GPU cannot handle 262144 context + AWQ conversion
+- **Fix**: Use `--max-model-len 32768` (32k context)
+
+### Speculative Decoding KV Cache Bug
+- **Error**: `AssertionError` in nightly vLLM when using DFlash
+- **Cause**: Known bug in vLLM nightly with Mamba cache + speculative decoding
+- **Status**: Temporarily disabled speculative decoding to test base model
+
+## Systemd Service
+- Service: `vllm-awq-qwen36.service`
+- ExecStart: `/home/dank/vllm/vllm-awq-qwen36-dflash-ampere.sh`
+- Port: 34401 (to avoid conflicts with other services)
+- Status: Disabled (pending bug fixes)
+
+## Configuration Summary
+
+### Working Configuration
+```bash
+--max-model-len 32768
+--cpu-offload-gb 40
+--cpu-offload-params "experts"
+--gpu-memory-utilization 0.95
+--max-num-batched-tokens 1536
+--performance-mode throughput
+--enable-chunked-prefill
+# --enable-prefix-caching  # DISABLED - conflicts with Mamba
+```
+
+### Thread Binding
+```bash
+-e OMP_NUM_THREADS=8
+-e VLLM_CPU_OMP_THREADS_BIND=8
+```
+
+## Next Steps
+1. Monitor vLLM nightly for Mamba cache + speculative decoding fix
+2. Test with larger context windows when memory allows
+3. Benchmark with speculative decoding enabled (when bug is fixed)
+4. Consider using stable vLLM release instead of nightly
+
+## References
+- Model: https://huggingface.co/cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit
+- Draft: https://huggingface.co/z-lab/Qwen3.6-35B-A3B-DFlash
+- vLLM Docs: https://docs.vllm.ai/en/stable/

scripts/serve-vllm-qwen36-dflash.sh

@@ -32,6 +32,7 @@ echo ""
 podman rm -f "${CONTAINER_NAME}" 2>/dev/null || true
 
 # Run vLLM in Podman container (foreground mode for systemd)
+# Note: vllm/vllm-openai image has entrypoint "vllm serve", so we pass args directly
 exec podman run --name "${CONTAINER_NAME}" \
   --device /dev/nvidia0 \
   --device /dev/nvidiactl \
@@ -39,10 +40,10 @@ exec podman run --name "${CONTAINER_NAME}" \
   --device /dev/nvidia-uvm-tools \
   --device /dev/nvidia-modeset \
   -v "${GPU_LIBS}:/usr/lib/x86_64-linux-gnu:ro" \
-  -v "${HF_CACHE}:/root/.cache/huggingface:ro" \
+  -v "${HF_CACHE}:/root/.cache/huggingface" \
   -p "${PORT}:${PORT}" \
   docker.io/vllm/vllm-openai:latest \
-  vllm serve "${MODEL}" \
+  "${MODEL}" \
     --port "${PORT}" \
     --host "0.0.0.0" \
     --kv-cache-dtype "${KV_CACHE_DTYPE}" \

scripts/validate-vllm.sh

@@ -1,13 +1,65 @@
 #!/usr/bin/env bash
-# End-to-end validation test for vLLM Qwen3.6-35B-A3B-FP8
-# Tests single stream and 4 parallel streams
 set -euo pipefail
 
-HOST="${1:-127.0.0.1}"
-PORT="${2:-8001}"
-NUM_STREAMS=4
+# Configuration
+TARGET="${VLLM_TARGET:-dankserver}"
+HOST="${VLLM_HOST:-}"
+PORT="${VLLM_PORT:-}"
+MODEL="${VLLM_MODEL:-}"
 
-echo "=== Validating vLLM on ${HOST}:${PORT} ==="
+# Target presets
+case "$TARGET" in
+    dankserver)
+        HOST="${VLLM_HOST:-127.0.0.1}"
+        PORT="${VLLM_PORT:-34401}"
+        MODEL="${VLLM_MODEL:-frank-default-small}"
+        ;;
+    dankclaw)
+        HOST="${VLLM_HOST:-127.0.0.1}"
+        PORT="${VLLM_PORT:-3090}"
+        MODEL="${VLLM_MODEL:-frank-default-large}"
+        ;;
+    *)
+        echo "Unknown target: $TARGET"
+        echo "Usage: $0 [-t dankserver|dankclaw] [-H host] [-P port] [-M model]"
+        echo ""
+        echo "Targets:"
+        echo "  dankserver  - frank-default-small on port 34401"
+        echo "  dankclaw    - frank-default-large on port 3090"
+        echo ""
+        echo "Options:"
+        echo "  -t TARGET   Target preset (dankserver or dankclaw)"
+        echo "  -H HOST     Override host"
+        echo "  -P PORT     Override port"
+        echo "  -M MODEL    Override model name"
+        echo "  -n N        Number of streams to test (default: 4)"
+        exit 1
+        ;;
+esac
+
+# Parse arguments
+while getopts "t:H:P:M:n:" opt; do
+    case $opt in
+        t) TARGET="$OPTARG" ;;
+        H) HOST="$OPTARG" ;;
+        P) PORT="$OPTARG" ;;
+        M) MODEL="$OPTARG" ;;
+        n) MAX_STREAMS="$OPTARG" ;;
+    esac
+done
+
+# Default max streams if not set
+MAX_STREAMS="${MAX_STREAMS:-4}"
+
+# Different prompts for each stream to avoid prefix caching effects
+PROMPTS=(
+    "Explain the difference between supervised, unsupervised, and reinforcement learning in machine learning. For each type, provide a concrete real-world example, describe the training process, and discuss the advantages and limitations."
+    "Describe the architectural differences between RISC-V and x86 instruction set architectures. Include a discussion of register files, addressing modes, and instruction encoding. Compare their performance for embedded systems and server workloads."
+    "Analyze the economic impact of blockchain technology on supply chain management. Discuss the key benefits of transparency, immutability, and smart contracts. Compare implementation challenges of Bitcoin, Ethereum, and Hyperledger for enterprise use cases."
+    "Explain the principles of evolutionary algorithms in artificial intelligence. Describe how genetic algorithms, genetic programming, and evolutionary strategies differ in their approach to problem solving. Compare their performance on optimization problems and neural network architecture search."
+)
+
+echo "=== Validating vLLM on ${HOST}:${PORT} (${TARGET}) ==="
 
 # Test health endpoint
 echo "Health check..."
@@ -15,71 +67,60 @@ if ! curl -sf "http://${HOST}:${PORT}/health" > /dev/null 2>&1; then
     echo "FAIL: Health check failed"
     exit 1
 fi
+echo "OK"
 
-# Single stream test
-echo "Single stream test..."
-START=$(date +%s%N)
-RESPONSE=$(curl -s -X POST "http://${HOST}:${PORT}/v1/chat/completions" \
-    -H "Content-Type: application/json" \
-    -d '{
-        "model": "Qwen/Qwen3.6-35B-A3B-FP8",
-        "messages": [
-            {"role": "user", "content": "Explain the difference between supervised, unsupervised, and reinforcement learning in machine learning. For each type, provide a concrete real-world example, describe the training process, and discuss the advantages and limitations. Then compare them across three dimensions: data requirements, interpretability of results, and computational cost. Finally, describe a hybrid scenario where combining two of these approaches would outperform using either alone."}
-        ],
-        "max_tokens": 256
-    }')
-END=$(date +%s%N)
-ELAPSED_MS=$(( (END - START) / 1000000 ))
-PROMPT_TOKENS=$(echo "$RESPONSE" | python3 -c "import sys,json; print(json.load(sys.stdin)['usage']['prompt_tokens'])")
-COMPLETION_TOKENS=$(echo "$RESPONSE" | python3 -c "import sys,json; print(json.load(sys.stdin)['usage']['completion_tokens'])")
-SINGLE_SPEED=$(python3 -c "print(f'{${PROMPT_TOKENS}/${ELAPSED_MS}*1000:.2f}')")
-echo "Single stream: ${SINGLE_SPEED} tok/s (prompt)"
+# Test each stream count separately
+for N in $(seq 1 "$MAX_STREAMS"); do
+    echo ""
+    echo "--- ${N} stream(s) ---"
+    
+    TMPDIR=$(mktemp -d)
+    START=$(date +%s%N)
+    
+    # Launch N concurrent requests with different prompts
+    for i in $(seq 1 "$N"); do
+        PROMPT_IDX=$(( (i - 1) % ${#PROMPTS[@]} ))
+        PROMPT="${PROMPTS[$PROMPT_IDX]}"
         
-# Parallel streams test
-echo "Parallel streams test (${NUM_STREAMS} streams)..."
-START=$(date +%s%N)
-TMPDIR=$(mktemp -d)
-for i in $(seq 1 $NUM_STREAMS); do
         curl -s -X POST "http://${HOST}:${PORT}/v1/chat/completions" \
             -H "Content-Type: application/json" \
-        -d '{
-            "model": "Qwen/Qwen3.6-35B-A3B-FP8",
-            "messages": [
-                {"role": "user", "content": "Explain the difference between supervised, unsupervised, and reinforcement learning in machine learning. For each type, provide a concrete real-world example, describe the training process, and discuss the advantages and limitations. Then compare them across three dimensions: data requirements, interpretability of results, and computational cost. Finally, describe a hybrid scenario where combining two of these approaches would outperform using either alone."}
-            ],
-            "max_tokens": 256
-        }' > "$TMPDIR/$i.json" &
-done
-wait
-END=$(date +%s%N)
-ELAPSED_MS=$(( (END - START) / 1000000 ))
+            -d "{
+                \"model\": \"${MODEL}\",
+                \"messages\": [{\"role\": \"user\", \"content\": \"${PROMPT}\"}],
+                \"max_tokens\": 256
+            }" > "$TMPDIR/$i.json" &
+    done
+    wait
     
-# Aggregate results from all streams
-TOTAL_PROMPT=0
-TOTAL_COMPLETION=0
-for i in $(seq 1 $NUM_STREAMS); do
-    PT=$(python3 -c "import json; print(json.load(open('$TMPDIR/$i.json'))['usage']['prompt_tokens'])")
-    CT=$(python3 -c "import json; print(json.load(open('$TMPDIR/$i.json'))['usage']['completion_tokens'])")
-    TOTAL_PROMPT=$((TOTAL_PROMPT + PT))
-    TOTAL_COMPLETION=$((TOTAL_COMPLETION + CT))
-done
-rm -rf "$TMPDIR"
+    END=$(date +%s%N)
+    ELAPSED_MS=$(( (END - START) / 1000000 ))
     
-PARALLEL_SPEED=$(python3 -c "print(f'{${TOTAL_PROMPT}/${ELAPSED_MS}*1000:.2f}')")
+    # Validate all responses
+    ALL_OK=true
+    for i in $(seq 1 "$N"); do
+        if ! python3 -c "import json; d=json.load(open('$TMPDIR/$i.json')); assert len(d['choices']) > 0" 2>/dev/null; then
+            echo "  FAIL: Response $i invalid"
+            ALL_OK=false
+        fi
+    done
+    
+    # Calculate speeds from first response
+    if $ALL_OK; then
+        PROMPT_TOKENS=$(python3 -c "import json; print(json.load(open('$TMPDIR/1.json'))['usage']['prompt_tokens'])")
+        COMPLETION_TOKENS=$(python3 -c "import json; print(json.load(open('$TMPDIR/1.json'))['usage']['completion_tokens'])")
+        TOTAL_TOKENS=$((PROMPT_TOKENS + COMPLETION_TOKENS))
+        PROMPT_SPEED=$(python3 -c "print(f'{${PROMPT_TOKENS}/${ELAPSED_MS}*1000:.2f}')")
+        COMPLETION_SPEED=$(python3 -c "print(f'{${COMPLETION_TOKENS}/${ELAPSED_MS}*1000:.2f}')")
+        TOTAL_SPEED=$(python3 -c "print(f'{${TOTAL_TOKENS}/${ELAPSED_MS}*1000:.2f}')")
+        
+        echo "  OK - ${ELAPSED_MS}ms total"
+        echo "  Prompt tokens: ${PROMPT_TOKENS} (${PROMPT_SPEED} tok/s)"
+        echo "  Completion tokens: ${COMPLETION_TOKENS} (${COMPLETION_SPEED} tok/s)"
+        echo "  Total: ${TOTAL_TOKENS} tokens (${TOTAL_SPEED} tok/s)"
+    fi
+    
+    rm -rf "$TMPDIR"
+done
 
 echo ""
-echo "=== Results ==="
-echo "Single stream prompt speed:   ${SINGLE_SPEED} tok/s"
-echo "Parallel streams prompt speed: ${PARALLEL_SPEED} tok/s (${NUM_STREAMS} streams, ${TOTAL_PROMPT} total prompt tokens)"
-echo "Parallel completion speed:    $(python3 -c "print(f'{${TOTAL_COMPLETION}/${ELAPSED_MS}*1000:.2f}')") tok/s"
-echo "Total parallel time:          ${ELAPSED_MS}ms"
-echo ""
-
-# Fail if parallel speed < 5 tok/s
-if python3 -c "import sys; sys.exit(0 if ${TOTAL_PROMPT}/${ELAPSED_MS}*1000 >= 5 else 1)"; then
-    echo "PASS: Validation successful"
-    echo "$PARALLEL_SPEED" > /tmp/vllm_prompt_speed.txt
-else
-    echo "FAIL: Parallel prompt speed ${PARALLEL_SPEED} tok/s is below 5 tok/s threshold"
-    exit 1
-fi
+echo "=== Validation complete ==="

tests/test_validator.sh

@@ -0,0 +1,134 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Test suite for validate-vllm.sh
+# Usage: bash tests/test_validator.sh [TARGET]
+
+TARGET="${1:-dankserver}"
+SCRIPT="$(dirname "$0")/../scripts/validate-vllm.sh"
+PASS=0
+FAIL=0
+
+pass() {
+    echo "✓ PASS: $1"
+    ((PASS++))
+}
+
+fail() {
+    echo "✗ FAIL: $1"
+    ((FAIL++))
+}
+
+# Test 1: Script exists and is executable
+if [ -x "$SCRIPT" ]; then
+    pass "Script exists and is executable"
+else
+    fail "Script missing or not executable"
+    exit 1
+fi
+
+# Test 2: Script accepts -t flag
+if bash "$SCRIPT" -t dankserver --help 2>&1 | grep -q "frank-default-small"; then
+    pass "Script accepts -t dankserver flag"
+else
+    fail "Script does not accept -t dankserver flag"
+fi
+
+# Test 3: Script accepts -H flag
+if bash "$SCRIPT" -H 127.0.0.1 --help 2>&1 | grep -q "127.0.0.1"; then
+    pass "Script accepts -H host flag"
+else
+    fail "Script does not accept -H host flag"
+fi
+
+# Test 4: Script accepts -P flag
+if bash "$SCRIPT" -P 34401 --help 2>&1 | grep -q "34401"; then
+    pass "Script accepts -P port flag"
+else
+    fail "Script does not accept -P port flag"
+fi
+
+# Test 5: Script accepts -M flag
+if bash "$SCRIPT" -M frank-default-small --help 2>&1 | grep -q "frank-default-small"; then
+    pass "Script accepts -M model flag"
+else
+    fail "Script does not accept -M model flag"
+fi
+
+# Test 6: Script accepts -n flag
+if bash "$SCRIPT" -n 4 --help 2>&1 | grep -q "4"; then
+    pass "Script accepts -n streams flag"
+else
+    fail "Script does not accept -n streams flag"
+fi
+
+# Test 7: Health check endpoint is accessible
+if curl -sf "http://${TARGET}:34401/health" > /dev/null 2>&1; then
+    pass "Health check endpoint is accessible"
+else
+    fail "Health check endpoint is not accessible"
+fi
+
+# Test 8: Chat completions endpoint is accessible
+if curl -sf -X POST "http://${TARGET}:34401/v1/chat/completions" \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "frank-default-small",
+        "messages": [{"role": "user", "content": "Hello"}],
+        "max_tokens": 10
+    }' > /dev/null 2>&1; then
+    pass "Chat completions endpoint is accessible"
+else
+    fail "Chat completions endpoint is not accessible"
+fi
+
+# Test 9: Response contains expected fields
+RESPONSE=$(curl -sf -X POST "http://${TARGET}:34401/v1/chat/completions" \
+    -H "Content-Type: application/json" \
+    -d '{
+        "model": "frank-default-small",
+        "messages": [{"role": "user", "content": "Hello"}],
+        "max_tokens": 10
+    }')
+
+if echo "$RESPONSE" | python3 -c "import json; d=json.load(sys.stdin); assert 'choices' in d" 2>/dev/null; then
+    pass "Response contains 'choices' field"
+else
+    fail "Response missing 'choices' field"
+fi
+
+if echo "$RESPONSE" | python3 -c "import json; d=json.load(sys.stdin); assert 'usage' in d" 2>/dev/null; then
+    pass "Response contains 'usage' field"
+else
+    fail "Response missing 'usage' field"
+fi
+
+if echo "$RESPONSE" | python3 -c "import json; d=json.load(sys.stdin); assert 'prompt_tokens' in d['usage']" 2>/dev/null; then
+    pass "Response contains 'prompt_tokens' in usage"
+else
+    fail "Response missing 'prompt_tokens' in usage"
+fi
+
+if echo "$RESPONSE" | python3 -c "import json; d=json.load(sys.stdin); assert 'completion_tokens' in d['usage']" 2>/dev/null; then
+    pass "Response contains 'completion_tokens' in usage"
+else
+    fail "Response missing 'completion_tokens' in usage"
+fi
+
+# Test 10: Script runs successfully (full validation)
+if bash "$SCRIPT" -t "$TARGET" -n 1 2>&1 | grep -q "Validation complete"; then
+    pass "Full validation completes successfully"
+else
+    fail "Full validation fails"
+fi
+
+# Summary
+echo ""
+echo "=== Test Summary ==="
+echo "Passed: $PASS"
+echo "Failed: $FAIL"
+echo ""
+
+if [ "$FAIL" -gt 0 ]; then
+    exit 1
+fi

vllm-awq-qwen36-dflash-ampere.sh

@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+# vllm-awq-qwen36-dflash-ampere.sh
+#
+# Qwen3.6-35B-A3B AWQ 4bit with DFlash speculative decoding.
+# Optimized for Ampere-class GPUs.
+#
+# Model:     cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit
+# Draft:     z-lab/Qwen3.6-35B-A3B-DFlash
+# Port:      34401
+#
+set -euo pipefail
+
+MODEL="cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit"
+PORT=34401
+
+exec docker run --rm --name vllm-awq-qwen36-dflash-ampere \
+    --gpus all \
+    --privileged --ipc=host \
+    -p "${PORT}:8000" \
+    -e OMP_NUM_THREADS=8 \
+    -e VLLM_CPU_OMP_THREADS_BIND=8 \
+    -v ~/.cache/huggingface:/root/.cache/huggingface \
+    vllm/vllm-openai:nightly "${MODEL}" \
+    --served-model-name "frank-default-small" \
+    # --speculative-config '{"method": "dflash", "model": "z-lab/Qwen3.6-35B-A3B-DFlash", "num_speculative_tokens": 4}' \
+    --max-num-seqs 2 \
+    --max-model-len 32768 \
+    --tensor-parallel-size 1 \
+    --enable-chunked-prefill \
+    --max-num-batched-tokens 1536 \
+    --performance-mode throughput \
+    --enable-auto-tool-choice --tool-call-parser qwen3_xml \
+    --reasoning-parser qwen3 \
+    --trust-remote-code \
+    --gpu-memory-utilization 0.95 \
+    --cpu-offload-gb 13 \
+    --cpu-offload-params "experts" \
+    --default-chat-template-kwargs '{"enable_thinking": true, "preserve_thinking": true}'

vllm-awq-qwen36-dflash.sh

@@ -0,0 +1,31 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+MODEL="cyankiwi/Qwen3.6-35B-A3B-AWQ-4bit"
+DRAFT_MODEL="z-lab/Qwen3.6-35B-A3B-DFlash"
+PORT=8001
+
+docker run --rm --name vllm-awq-qwen36-dflash \
+    --gpus all \
+    -p ${PORT}:8000 \
+    -v ~/.cache/huggingface:/root/.cache/huggingface \
+    vllm/vllm-openai:nightly \
+    vllm serve "${MODEL}" \
+        --served-model-name "frank-default-small" \
+        --speculative-model "${DRAFT_MODEL}" \
+        --speculative-num-steps 1 \
+        --speculative-max-num-draft-tokens 4 \
+        --max-num-seqs 2 \
+        --max-model-len 262144 \
+        --max-num-batched-tokens 1536 \
+        --tensor-parallel-size 1 \
+        --enable-chunked-prefill \
+        --enable-prefix-caching \
+        --enable-auto-tool-choice \
+        --tool-call-parser qwen3_xml \
+        --reasoning-parser qwen3 \
+        --trust-remote-code \
+        --gpu-memory-utilization 0.95 \
+        --cpu-offload-gb 13 \
+        --cpu-offload-params "experts" \
+        --default-chat-template-kwargs '{"enable_thinking": true, "preserve_thinking": true}'

vllm-qwen36.sh

@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+MODEL="Qwen/Qwen3.6-35B-A3B-FP8"
+PORT=8001
+HOST="0.0.0.0"
+
+exec docker run --rm --name vllm-qwen36 \
+    --gpus all \
+    --privileged \
+    --ipc=host \
+    -p "${PORT}:${PORT}" \
+    -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
+    -v /home/dank/.cache/huggingface:/root/.cache/huggingface \
+    docker.io/vllm/vllm-openai:latest \
+    "${MODEL}" \
+    --port "${PORT}" \
+    --host "${HOST}" \
+    --kv-cache-dtype fp8 \
+    --gpu-memory-utilization 0.95 \
+    --max-num-seqs 4 \
+    --max-model-len 32768 \
+    --max-num-batched-tokens 1536 \
+    --trust-remote-code \
+    --cpu-offload-gb 40 \
+    --cpu-offload-params "experts" \
+    --enable-chunked-prefill \
+    --enable-prefix-caching \
+    --served-model-name frank-default-small \
+    --enable-auto-tool-choice \
+    --reasoning-parser qwen3 \
+    --tool-call-parser qwen3_xml