dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
133 Commits 42 Branches 8 Tags
02a1447602e149bd5d4e8c205656da0ecbd66968
Commit Graph
133 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Frank 02a1447602 head alias: frank-default -> frank-default-large (match renamed service) 2026-08-16 14:27:48 -07:00
Frank d0ade28f6d docs: fleet topology update — PP split renamed frank-default-large, NPU endpoint promoted to frank-default-small, 4-model NPU bakeoff matrix (2026-08-16) 2026-08-16 14:22:13 -07:00
Frank 3f42210922 fix: mark serve-flm-small.sh executable (unit failed EXEC 203) 2026-08-16 14:20:17 -07:00
Frank f6737f234d promote NPU endpoint to frank-default-small: FLM serve unit + wrapper + env (gemma4-it:e4b default), LimitMEMLOCK=infinity (2026-08-16) 2026-08-16 14:19:37 -07:00
Frank 48142f5d8e rename frank-pp-head to frank-default-large: PP split is the new production large endpoint (2026-08-16) 2026-08-16 13:44:31 -07:00
Frank 79c913aaa7 docs: NPU parallelism measured — single-lane FIFO serialization, aggregate t/s flat at N=1/2/4 2026-08-16 13:12:46 -07:00
Frank d48227c7f7 docs: gpt-oss-20b landed on the Dankclaw NPU — 19.3 t/s decode measured, memlock fix, serve state (2026-08-16) 2026-08-16 12:57:44 -07:00
Frank 0799fc5eb4 docs: FastFlowLM NPU research — AMD NPU runtime, 50 TOPS Strix Halo fit analysis, verified install state, test runbook (2026-08-16) 2026-08-16 12:29:11 -07:00
Frank 934c6d8be1 pp-head: 2x200700 f16 KV (ctx 401400, parallel 2), split 0.40,0.60 = max 3090 capacity 2026-08-16 12:23:08 -07:00
Frank 3ad4c6921c pp-head: alias frank-default, verbosity 4 (interactive use) 2026-08-16 10:42:29 -07:00
Frank 750730e4d9 pp-test: restore drops caches before production load; all-mode EXIT trap auto-restores; ssh ConnectTimeout 2026-08-16 10:17:01 -07:00
Frank 1716bf37c9 pp-test: preflight production check via health endpoint (no sudo needed) 2026-08-16 05:48:17 -07:00
Frank 9544c36afc PP kit + Strix research: Qwen3.8-27B RPC split (Dankclaw head / Dankserver worker) 
- scripts: serve-pp-head/worker, envs, systemd units, pp-test.sh orchestrator, pp-bench.py
- head design verified in b10433 source + live 4B smoke test:
  * no --device with --rpc (RPC device silently dropped otherwise)
  * default device selection = [RPC0, ROCm0] (HIP registers before Vulkan)
  * default split proportional to reported free mem = ~20/80 here, pin 1,1
  * -dio required for large model over RPC
- docs: qwen3.8-strix-optimizations (ranked no-quant-downgrade levers,
  quant landscape, 2026-08-16 test matrix), PP deployment runbook,
  index updates, Dankclaw Improvements corrections (Vulkan0, no power_profile sysfs)
- worker built at 9b05354ec in /home/dank/llama-cpp-rpc worktree (CUDA+RPC)
- second head build build-vulkan-rpc (HIP off) for backend A/B
 2026-08-16 05:40:27 -07:00
Frank c775077016 dankclaw: add MTP=1 toggle - prepend draft-mtp before --spec-default ngram-mod fallback 2026-08-15 22:31:29 -07:00
Frank b6f2ee2945 docs: record --spec-default + Vulkan0 + PARALLEL=2 final state; 262k-ctx ROCm-vs-Vulkan bench 2026-08-15 17:13:18 -07:00
Frank 5296cf11d7 dankclaw: switch backend to Vulkan0 (48% faster decode at 262k ctx); add bench-high-ctx.sh 2026-08-15 17:03:01 -07:00
Frank 1dbc70cb32 dankclaw: retire DSpark, use llama.cpp --spec-default (ngram-mod) preset; PARALLEL=2 trial 2026-08-15 15:15:06 -07:00
Frank 020e874a5c docs: record DSpark single-slot switch + ROCm parallel crash (2026-08-15) across KB; refs AGENTS/deployment/benchmark/matrix/index 2026-08-15 14:36:34 -07:00
Frank d04cc69e09 dankclaw: add accurate DSpark stream/queue test (PARALLEL=1) - reports true prompt_tokens; single+concurrent 16k pass 2026-08-15 14:32:21 -07:00
Frank 6266e43f10 dankclaw: DSpark parallel crash - ROCm illegal memory access under 2x16k prose, pin PARALLEL=1 per user 2026-08-15 14:24:27 -07:00
Frank b94bad066b dankclaw: switch frank-default-large spec to DSpark sidecar draft (magnitudedev Qwen3.8-27B-DSpark Q8_0, n_max 7) 2026-08-15 14:00:30 -07:00
Frank 40b079fe1e unit: refresh frank-default-large description (Qwen3.8-27B UD-Q8_K_XL, MTP+ngram-mod+ngram-map-k4v) 2026-08-14 21:01:22 -07:00
Frank 8d64adbcbe dankclaw: production spec -> 3-layer chain draft-mtp,ngram-mod,ngram-map-k4v (user request; ~equal real-prose speed) 2026-08-14 21:00:05 -07:00
Frank 2d7290a2a8 AGENTS: MANDATORY benchmarking policy - stop production service before any bench (serves ~3 real users); restart + health-check after 2026-08-14 20:54:34 -07:00
Frank 0eda208705 docs: Qwen3.8 spec-matrix benchmark page; add matrix tools; final env n_max=5 
Recording the 54-config matrix, the realistic-prompt correction (n_max=5 not
10), the ubatch findings (256 is the safe/fast point), and the premature-stop
sampling-variance investigation.
 2026-08-14 14:28:51 -07:00
Frank a818be70dc dankclaw Qwen3.8: winner n_max=5 (13.3 t/s real), not matrix n_max=10 
Realistic-prompt tuning: n_max=5 peaks ~13.3 t/s (baseline 7.2). matrix used a
repetitive prompt that inflated MTP. n_max=10 is only ~8.6 t/s on real writing.
 2026-08-14 14:26:14 -07:00
Frank 75bb9bb53e dankclaw Qwen3.8: batch 8192/ubatch 256 - fixes MTP collapse + premature-EOS 
Found: large ubatch (512) collapses MTP acceptance to ~25% (gen ~9 t/s);
ubatch 128 with batch>=2048 triggers premature instant-EOS under 2 concurrent
slots.  ubatch 256 = ~30 t/s gen, concurrent-safe, ~330 t/s prefill.
 2026-08-14 13:59:20 -07:00
Frank 57b0be037c dankclaw: switch frank-default-large to unsloth Qwen3.8-27B UD-Q8_K_XL 
- rebuild llama.cpp stock to b10433 (9b05354ec) for bundled-MTP support
- MODEL_FILE -> Qwen3.8-27B-UD-Q8_K_XL.gguf; DRAFT_FILE empty (MTP bundled)
- SPEC: draft-mtp, n_max=10, p_min off (~30 t/s stable) - matrix winner
- CTX 524288 / parallel 2 = 262144/slot (matches pi contextWindow)
- serve script: bundled-MTP (no --spec-draft-model), optional --spec-draft-p-min
 2026-08-14 13:26:17 -07:00
Frank d72092b2c7 docs: record dual-f16 KV refit on frank-default-small (2026-08-14) 2026-08-14 12:59:10 -07:00
Frank f96aa2b692 frank-default-small: refit ctx 800000->700000 for dual-f16 KV pool 
The f16/f16 KV pool costs 15640 MiB at 800768 cells. At full 800K the
3090 idled at 23966/24576 MiB and the 1472x1472 image warmup graph
failed cudaMalloc (needed +248 MiB). 700416 cells (175104/slot, 4 slots)
leaves ~2.3 GiB free after warmup. Parallel stays 4.
 2026-08-14 12:55:44 -07:00
Frank f70460b002 frank-default-small: KV K+V to f16, add verbosity tunable (F16 KV diag) 
F16 K removes the flash-attn F16-conversion workspace (MMA_F16 prefill
kernel needs no copy when K is already f16). Trades static KV pool size
for the context-proportional workspace growth implicated in the
cublasSgemm_v2 crash class. VERBOSITY=4 run logs real allocation sizes;
refit CTX_SIZE if the pool does not fit on the 24 GiB 3090.
 2026-08-14 12:50:59 -07:00
Frank bea52138d2 docs: record Dankserver VRAM/RAM growth investigation (2026-08-14) 2026-08-14 12:44:24 -07:00
Frank 9d7643ec5d frank-default-small: ncmoe 35->36 for prefill stability 
Fix a live crash: cuBLAS cudaErrorInvalidValue in ggml_cuda_mul_mat_cublas
killed the service during a 145k-token prefill burst at ncmoe35/ub3072.

ncmoe 36 moves one more expert layer to CPU: frees ~0.75 GiB peak VRAM
(grid: peak 8606 vs 9360 MiB at ub3072) for -2% prefill / -2% decode.
Keeps prefill priority (top ubatch tier). Docs synced; noted fallback
GGML_CUDA_FORCE_MMQ=1 if pressure persists.
 2026-08-14 10:56:40 -07:00
Frank 7daf9896e5 docs: record Ling freeze diagnosis - spec path deadlock, resolution spec-off parallel=2 2026-08-14 03:49:42 -07:00
Frank 139686043b ling B-test: spec off (--spec-type none), keep parallel=2 - isolate spec path as freeze trigger 2026-08-14 03:39:15 -07:00
Frank 7ba86812f7 ling A/B: verbosity 4->0, drop log file to isolate hang trigger (match known-good run) 2026-08-14 02:32:08 -07:00
Frank 6a6d673dca ling A/B: revert batch 8192->3072 to isolate mid-stream freeze (suspect batch) 2026-08-14 02:20:52 -07:00
Frank a5444e2c4d ling: batch 8192 (ubatch 3072); add -lv verbosity + log file for stop-investigation 2026-08-14 02:12:35 -07:00
Frank 321f946660 ling: ubatch 512 -> 3072 (batch 3072) - using heap headroom, watch for OOM 2026-08-14 01:46:36 -07:00
Frank a5b12e1062 ling: keep draft-mtp,ngram-mod active (A/B wash vs MTP alone, parallel-safe) 2026-08-14 01:45:22 -07:00
Frank 6647690ebd ling A/B: bench draft-mtp alone vs draft-mtp,ngram-mod 2026-08-14 01:43:53 -07:00
Frank 3204b4b29e ling trial: layer ngram-mod under draft-mtp; add ngram args to serve script 2026-08-14 01:40:46 -07:00
Frank c275cbaf57 docs: Ling-3.0-flash Q6_K trial page (Vulkan+GTT memory research, switch-back steps) 2026-08-14 01:22:02 -07:00
Frank 923e775354 dankclaw: Ling-3.0-flash Q6_K trial config (Vulkan+GTT+draft-mtp) 
Trial replaces frank-default-large temporarily via frank-large-ling.service
(port 7770, same alias). Requires the aetherbird bailingmoe3-support llama.cpp
fork at /home/dank/llama.cpp-ling (stock llama.cpp cannot load bailingmoe3).
Memory: 8x16GB LPDDR5 = 128GB physical; Vulkan exposes ~111.2GiB heap (UMA
96GiB + GTT) vs ROCm's 96GiB. Q6_K = 98.26GiB; MLA/KDA KV is tiny so 256k
ctx x 2 slots fits with ~10GiB headroom. Verified: loads ~45s, concurrent
streams both complete (no DFlash-style kill), ~7.7 t/s gen.
Restore Qwen: systemctl stop frank-large-ling; systemctl start
frank-default-large.
 2026-08-14 01:20:30 -07:00
Frank db69fc8192 docs: pipeline parallelism research + educational sample script set 
Compiles the 5-subagent research fan-out (llama.cpp RPC, vLLM PP,
bandwidth math, mixed-vendor experiments, Qwen3.6-27B spec methods) into
the KB under docs/Pipeline Parallelism/ with an index, expected
performance estimates (1/2.5/10 GbE), and an EDUCATIONAL-ONLY two-node
Qwen3.6-27B Q8 split script set (MTP/DFlash/ngram-mod variants, DSpark
noted as vLLM-only). Nothing here is deployed or intended to run.
 2026-08-14 00:22:23 -07:00
Frank 8d1d9baa82 docs: MTP wins the DFlash replacement; record parallel premature-EOS incident 
- benchmark doc: spec A/B table (MTP 15.3-19.5, MTP+ngram 12-13.6,
  MTP n_max2 12.4-14.3, ngram 6.7-8.7, DFlash retired), the DFlash
  shared-draft-context premature-EOS incident, updated batch/ubatch.
- deployment.md: Dankclaw now draft-mtp n_max 3 + MTP-ONLY-Q8_0 sidecar,
  batch 8192/ubatch 8192; note Dankserver bench drift.
- index.md/AGENTS.md: corrected stale topology (Ling-3.0-flash, DFlash).
- DFlash doc: Dankclaw section rewritten for MTP; DFlash retired.
 2026-08-13 23:27:06 -07:00
Frank a27839519e dankclaw: FINAL spec - draft-mtp n_max 3, ubatch 8192 
n_max 2 benched 12.4-14.3 t/s (slower than n_max 3 at 15.3-19.5);
n_max 3 is the MTP sweet spot. ubatch 8192 (==batch) uses the VRAM
headroom (65.5 GiB of 96 GiB) safely. MTP beats ngram-mod (~7 t/s) and
matches DFlash (~20 t/s) without the parallel premature-EOS bug.
 2026-08-13 23:07:38 -07:00
Frank 01eb957f90 frank-default-small: prefill-max ncmoe 35 / ubatch 3072 
Prefill is the priority for this workload. Switch from decode-max
(ncmoe 32 / ubatch 1024) to prefill-max:
- n-cpu-moe 35, ubatch 3072: prefill ~1458 t/s (top tier), gen ~45.4 t/s,
  ~1.2 GiB headroom at the 800k pool.
- Gen is independent of ubatch; ub 3072 reaches best prefill with no
  decode cost. Docs synced (benchmark Chosen State, deployment, specs).
 2026-08-13 23:07:30 -07:00
Frank 2b4e58a926 dankclaw: MTP tuning - spec-draft-n-max 2 
Retest MTP at n_max 2 (vs 3) for the acceptance/speed tradeoff, keeping
MTP-alone + ubatch 8192. Re-benches gen t/s and re-runs the strict
parallel-safety test.
 2026-08-13 23:02:07 -07:00
Frank 3c2db64a78 frank-default-small: ncmoe 32 + ubatch 1024 from llama-bench sweep 
Swept ncmoe(30..40) x ubatch(256..4096) on the RTX 3090 (Qwen3.6-35B-A3B
Q6_K_P, pp16384/tg512, fit-target 0). Key findings:
- Generation depends only on ncmoe: 40.9 (ncmoe40) .. 51.1 (ncmoe30);
  ubatch/batch do not affect decode at all.
- Prefill + graph VRAM scale with ubatch; logical batch is irrelevant.
- VRAM model (peak + 11.44GiB KV pool + 2.2GiB overhead) reproduces the
  historical OOMs: ncmoe32-ub2048 sums ~24.4GiB > 24.

Chosen state: ncmoe 32 (8 expert layers on GPU), batch 8192, ubatch 1024.
  gen ~48.7 t/s (+7% vs prior ncmoe 35), prefill ~766 t/s, ~1.0GiB headroom.
Alternates: ncmoe34-ub2048 (gen 46.4, pp 1221, 1.5GiB) for prefill-heavy;
  ncmoe32-ub768 (1.3GiB) if headroom matters most.
Docs: new frank-default-small-benchmark.md; synced deployment.md + specs.
 2026-08-13 22:56:24 -07:00