diff --git a/docs/LLM/Dankserver System Specs.md b/docs/LLM/Dankserver System Specs.md index 8437b9f..9ebe2d4 100644 --- a/docs/LLM/Dankserver System Specs.md +++ b/docs/LLM/Dankserver System Specs.md @@ -60,12 +60,12 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 26.0 ### Active Services -The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors from blocks 0 through 32 held on CPU via `--override-tensor`. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]]. +The active service on Dankserver is `frank-default-small.service`: the llama.cpp Ornith-1.5-35B-A3B CRACK (Q6_K) on port 7770, with expert tensors selected by `--override-tensor`; full-attention blocks 27 and 31 plus output-adjacent blocks 33–40 remain on CUDA. The earlier PP worker (`frank-pp-worker.service`) was retired with the 2026-08-17 redo. See [[deployment]]. ## Hardware Relevance -- 24 GB VRAM hosts the Q6_K non-expert tensors and expert tensors from blocks 33 through 40. The provisional context pool contains 491520 f16 cells across three 163840-token slots. The configuration will be rebalanced after measuring the newly available headroom. Verify the actual headroom after each model restart with `nvidia-smi` and the service journal. -- 62 GB system RAM holds approximately 19.67 GiB of CPU expert tensors with room to spare. +- 24 GB VRAM hosts the Q6_K non-expert tensors and the measured hybrid expert placement. The context pool contains 491520 f16 cells across three 163840-token slots. Verify actual headroom after each model restart with `nvidia-smi` and the service journal. +- 62 GB system RAM holds the remaining CPU-resident expert tensors with room to spare. - 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled. - Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total). diff --git a/docs/LLM/dealign-ornith-crack.md b/docs/LLM/dealign-ornith-crack.md index e4e4c0c..998c15f 100644 --- a/docs/LLM/dealign-ornith-crack.md +++ b/docs/LLM/dealign-ornith-crack.md @@ -45,21 +45,23 @@ The non-expert tensors total 2.02 GiB. The layer totals are 660.96 MiB for ordinary SSM blocks, 655.81 MiB for full-attention blocks, and 662.40 MiB for block 40. The expert portion is 630 MiB in every block. -The provisional service keeps all non-expert tensors on CUDA. It also keeps -expert tensors from blocks 33 through 40 on CUDA. Those blocks sit next to the -output and include the bundled MTP head. Expert tensors from blocks 0 through -32 move to CPU through this explicit override: +The measured hybrid configuration keeps all non-expert tensors on CUDA. It +also keeps expert tensors from full-attention blocks 27 and 31, plus the +output-adjacent blocks 33 through 40, on CUDA. Block 40 includes the bundled +MTP head. Expert tensors from all other blocks move to CPU through this +explicit override: ``` -OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps' +OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps' OVERRIDE_BUFT=CPU ``` -The previous five-block configuration placed 3.69 GiB of expert weights on -CUDA and 22.13 GiB on CPU. This provisional eight-block configuration adds -approximately 1.89 GiB of expert weights to CUDA while reducing the context -pool by 98304 cells. The final placement will be selected after measuring this -new headroom and discussing layer priorities. +The measured benchmark compared three policies with a 16K-token prompt: +full-attention blocks plus block 40, contiguous top blocks 33–40, and this +hybrid. The hybrid was fastest at approximately 33.6 generated tokens per +second, versus 33.1 for the contiguous-top policy and 30.3 for the +full-attention-only policy. Its first-request prompt throughput was about +2711 tokens per second. To list the buffer type names on the host, run a bogus override: @@ -89,7 +91,7 @@ llama-server \ --mmproj mmproj-Ornith-1.5-35B-A3B-f16.gguf \ --chat-template-file chat-template.jinja \ --spec-type draft-mtp \ - --override-tensor "blk\\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \ + --override-tensor "blk\\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\\.ffn_(up|down|gate|gate_up)_(ch|)exps=CPU" \ -ngl all -c 491520 -np 3 \ --cache-type-k f16 --cache-type-v f16 --no-mmap \ --temp 1.0 --top-p 0.95 --top-k 20 @@ -102,8 +104,9 @@ llama-server \ 2. The provisional context pool contains 491520 cells. Three slots receive 163840 cells each, which is 96K below the model context per stream. 3. `--n-cpu-moe` remains supported. This service currently prefers the - explicit tensor override because it preserves the final eight expert - blocks on CUDA pending the next layer-priority decision. + explicit tensor override because the 16K benchmark found this hybrid + placement fastest. It preserves the final eight blocks plus full-attention + blocks 27 and 31 on CUDA. 4. The service uses the Qwen-Sharp template from `peculiar-ragdoll/Qwen-Sharp-Chat-Templates` through `scripts/chat-template.jinja`. diff --git a/docs/LLM/deployment.md b/docs/LLM/deployment.md index a200899..0fed439 100644 --- a/docs/LLM/deployment.md +++ b/docs/LLM/deployment.md @@ -124,10 +124,11 @@ curl -sf http://127.0.0.1:7770/health The active service on Dankserver is upstream llama.cpp (b10121) serving `Ornith-1.5-35B-A3B-CRACK-Q6_K.gguf` under the alias `frank-default-small` on port 7770. It runs the stock `/home/dank/llama.cpp/build/bin/llama-server` -binary with expert tensors from blocks 0 through 32 pinned to CPU by an -explicit `--override-tensor` regex. Expert tensors from blocks 33 through 40 -stay on CUDA. The service uses 7 CPU threads, native `draft-mtp` -speculative decoding, f16 KV, context 491520 (3 slots of 163840 each), batch +binary with expert tensors selected by an explicit `--override-tensor` +regex. Expert tensors from full-attention blocks 27 and 31 and the +output-adjacent blocks 33 through 40 stay on CUDA; other expert tensors are +CPU-resident. The service uses 7 CPU threads, native `draft-mtp` speculative +decoding, f16 KV, context 491520 (3 slots of 163840 each), batch 4096, ubatch 2048, and `--no-mmap`. The unit is `scripts/frank-default-small.service`. It launches through `scripts/serve-frank-small.sh`, which reads tunables from diff --git a/scripts/frank-default-small.env b/scripts/frank-default-small.env index 8bb90b6..e0dfdf3 100644 --- a/scripts/frank-default-small.env +++ b/scripts/frank-default-small.env @@ -42,11 +42,12 @@ SPEC_DRAFT_N_MIN=0 SPEC_DRAFT_P_MIN=0.0 # Expert offload via explicit tensor override (replaces --n-cpu-moe). The Q6_K -# layer audit found 630 MiB of expert tensors per block. Keep the four -# output-adjacent blocks 33-40 on CUDA, including the bundled MTP block 40. -# Move expert tensors from blocks 0-32 to CPU. All non-expert tensors stay on -# the 24 GB card. Buffer type "CPU" is the CPU buft name on this host. -OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps' +# layer audit found 630 MiB of expert tensors per block. Keep the selected +# full-attention blocks 27 and 31 plus output-adjacent blocks 33-40 on CUDA, +# including the bundled MTP block 40. Move all other expert tensors to CPU. +# All non-expert tensors stay on the 24 GB card. Buffer type "CPU" is the CPU +# buft name on this host. +OVERRIDE_EXPERT='blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps' OVERRIDE_BUFT=CPU N_GPU_LAYERS=all diff --git a/scripts/serve-frank-small.sh b/scripts/serve-frank-small.sh index 2903611..6cad613 100755 --- a/scripts/serve-frank-small.sh +++ b/scripts/serve-frank-small.sh @@ -7,8 +7,9 @@ # 3B-active MoE-dense hybrid with the MTP speculative head bundled in the GGUF. # # Expert offload: the expert (FFN exps) tensors selected by the layer-bounded -# --override-tensor pattern are pinned to CPU. The final output-adjacent -# blocks stay on CUDA with all non-expert tensors, including the MTP head. +# --override-tensor pattern are pinned to CPU. Selected full-attention and +# final output-adjacent blocks stay on CUDA with all non-expert tensors, +# including the MTP head. # # Spec: MoE model with a bundled MTP head -> native draft-mtp (the head is # auto-detected from the target model; b10121 creates the MTP draft context @@ -49,7 +50,7 @@ fi # Expert offload: --override-tensor regex pinned to the CPU buft. Same pattern # the internal --cpu-moe / --n-cpu-moe use; explicit here so we only move the # MoE expert weights and leave every other tensor on the GPU. -: "${OVERRIDE_EXPERT:=blk\.(0|1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|27|28|29|30|31|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps}" +: "${OVERRIDE_EXPERT:=blk\.(0|1|2|3|4|5|6|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25|26|28|29|30|32)\.ffn_(up|down|gate|gate_up)_(ch|)exps}" : "${OVERRIDE_BUFT:=CPU}" : "${N_GPU_LAYERS:=all}" : "${BATCH_SIZE:=4096}" diff --git a/tests/frank_small_config.bats b/tests/frank_small_config.bats index 2577276..c9c35f1 100644 --- a/tests/frank_small_config.bats +++ b/tests/frank_small_config.bats @@ -20,9 +20,13 @@ TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja" grep -q '^PARALLEL=3$' "$ENV_FILE" } -@test "frank-default-small keeps final eight expert blocks on CUDA" { - grep -q '^OVERRIDE_EXPERT=' "$ENV_FILE" - grep -Fq '31|32)\.ffn_' "$ENV_FILE" +@test "frank-default-small uses the measured hybrid expert placement" { + override=$(grep '^OVERRIDE_EXPERT=' "$ENV_FILE") + [[ "$override" == *'26|28|29|30|32)'* ]] + [[ "$override" != *'27'* ]] + [[ "$override" != *'31'* ]] + [[ "$override" != *'33'* ]] + [[ "$override" != *'40'* ]] grep -q '^OVERRIDE_BUFT=CPU$' "$ENV_FILE" grep -q '^N_GPU_LAYERS=all$' "$ENV_FILE" }