Frank
35bd22cd75
Add --served-model-name frank-default-small
2026-08-08 10:35:24 -07:00
Frank
0f97dcf449
Update validation script to test parallel streams (4 concurrent requests)
2026-08-08 00:30:08 -07:00
Frank
6a7025d01e
Iteration 2: Remove --enforce-eager to enable torch.compile (+14.6% prompt speed: 5.95 -> 6.82 tok/s)
2026-08-07 23:07:51 -07:00
Frank
1b90f421c6
Initial vLLM Qwen3.6-35B-A3B-FP8 serving with CPU expert offloading
2026-08-07 22:20:52 -07:00
Frank
9971bdef0f
feat: initial vLLM Qwen3.6-35B-A3B-FP8 serve script with DFlash drafting on dankserver
...
- Podman container with CUDA GPU access
- DFlash speculative decoding via --spec-method dflash
- FP8 KV cache, 4 parallel streams, 262K context
- CPU expert offloading via --cpu-offload-gb
- systemd service for persistent runtime
2026-08-07 18:28:38 -07:00
Frank
5452bb63f1
Add vLLM serve script, systemd service, and validation test for Qwen3.6-35B-A3B-FP8 on dankserver
2026-08-07 18:19:40 -07:00
Frank
663e47866b
Bump context to 600000 (200000/stream x 3 parallel)
2026-08-07 17:18:10 -07:00
Frank
a5f72627bb
Use --alias frank-default-large, multi-stream validation script
...
- Replace --override-kv with --alias for model naming
- Validation script tests 1, 2, 3 parallel streams sequentially
- Model name confirmed: frank-default-large
2026-08-07 17:15:09 -07:00
Frank
471e963350
Fix parallel validation, remove ngram-mod
...
- Rewrite validation script with temp files for true parallel curl requests
- Both prompts now complete 1024 tokens each (was cutting first short)
- Remove ngram-mod from spec pipeline (was -1.2 tok/s regression)
- True parallel throughput: 15.68 tok/s across 2 streams
2026-08-07 16:44:12 -07:00
Frank
2426ca563a
Add ngram-mod to spec pipeline: -1.2 tok/s regression
...
- DFlash+ngram-mod: 17.11 tok/s vs DFlash alone: 18.34 tok/s
- ngram overhead not offset by acceptance rate at current settings
- Keeping in place for now, may tune ngram-mod params later
2026-08-07 16:25:49 -07:00
Frank
764a65080f
Validation: 2 parallel prompts, ~18 tok/s with DFlash
...
- Add second parallel prompt (Byzantine Empire history vs ML)
- Both hit 1024 tokens, combined 2048 in 111.6s = 18.34 tok/s
- Parallel throughput matches single-stream baseline
2026-08-07 16:21:20 -07:00
Frank
25d57184a6
Enable DFlash speculative decoding, remove mlock
...
- Add --spec-type draft-dflash (was missing, DFlash was inactive)
- Change --load-mode from mlock to none (RLIMIT_MEMLOCK too small for 2.7GB)
- Increase validation max_tokens from 256 to 1024 with complex ML prompt
- Baseline: 18.76 tok/s (was 7.62 tok/s without DFlash)
2026-08-07 16:10:55 -07:00
Frank
1949f0651b
Pivot to llama.cpp with DFlash speculative decoding on ROCm
...
- Abandon vLLM/FP8 (Triton compilation hangs, ~0.5 tok/s on Strix Halo)
- llama.cpp v10133 with ROCm0 offload, DFlash draft model
- Main: unsloth/Qwen3.6-27B-MTP-GGUF:Q8_0 (~22GB)
- Draft: Alittlehammmer/Qwen3.6-27B-DFlash-GGUF-llama.cpp:Q8_0 (~1.8GB)
- 3 parallel streams, 131072 context/stream
- systemd service with minimal sandboxing (ROCm requires /dev/kfd access)
- Baseline: 7.62 tok/s (threshold 5.0 tok/s)
2026-08-07 15:41:27 -07:00
Frank
2bb1b2ae1d
Baseline: ThinkingCap 27B FP8 on dankclaw with enforce-eager and validation timeout
2026-08-07 08:58:20 -07:00
Frank
e4fba35e89
Baseline: ThinkingCap 27B FP8 on dankclaw
...
- vLLM 0.26.0+rocm723
- --kv-cache-dtype fp8 --gpu-memory-utilization 0.80
- --max-model-len 32768 --max-num-batched-tokens 2048
- --max-num-seqs 32 --reasoning-parser qwen3 --language-model-only
- Baseline response time: 219s for 50 tokens
2026-08-07 08:49:59 -07:00
Frank
7dce375ab4
docs: fix dankclaw specs (30GiB RAM, 96GiB GPU-visible, ROCm 7.2.4) + add validate.sh
2026-08-06 22:40:44 -07:00
Frank
5076ef828f
chore: fix typo in dankclaw specs, format index table
2026-08-06 22:29:23 -07:00
Frank
dc356acce7
docs: add top-level knowledge base index
2026-08-06 00:16:47 -07:00
Frank
671129ea69
chore: ignore obsidian workspace files
2026-08-06 00:14:06 -07:00
Frank
39db415348
feat: initialize vLLM scripts knowledge base
...
- VLLM Overview: installation, CLI, API, architecture, quantization, parallelism
- VLLM Index: quick reference with commands, endpoints, parameters
- Dankserver System Specs: RTX 3090 24GB, 62GB RAM, Ubuntu 24.04
- Dankclaw System Specs: AMD Strix Halo APU, 61GB unified, ROCm 7.1
- RAM/CPU Offloading in VLLM: KV cache swap, gpu_memory_utilization, preemption
- Hybrid CPU/GPU Processing (Dankserver): NVIDIA GPU + system RAM configuration
- Shared Memory Architecture (Dankclaw): AMD unified memory and ROCm setup
- Running Qwen3.6 35B A3B FP8 on Dankserver: MoE model deployment guide
- Running Qwen3.6 27B FP8 on Dankclaw: dense model deployment guide
2026-08-05 23:49:02 -07:00
Frank
Frank