dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
141 Commits 42 Branches 8 Tags
accf90b6faa31449aaa24ea848a20b1acb789262
Commit Graph
141 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Frank accf90b6fa frank-default-large: split 0.55,0.45 = 35 blocks on the 3090 (true max) 
GGUF block scan (2026-08-17): FA at 3,7,...,63; linear blocks 383 MiB,
FA blocks 446 MiB. 35 blocks (0-34) keep 8 FA layers on RPC0 -> KV stays
8192 MiB, predicted ~0.8 GiB free. Block 35 is FA: +1 GiB KV -> OOM.
Measured at 0.50: 22650/24576 used, 1477 free (the earlier 714 MiB
projection was conservative).
 2026-08-17 00:34:58 -07:00
Frank 53ada02a63 frank-default-large: PARALLEL=1, CTX 262144, split 0.50 (max 3090 weight) 
Single full-context slot instead of two half-context slots. The 3090 takes
~33 of 66 layers (up from 26): the VRAM budget frees up because the f16 KV
pool shrinks 401400 -> 262144. 33 layers is the max with a safety margin
(34 -> 244 MiB headroom, 35 -> OOM). Verified math: /tmp/vram-math.py from
live journal buffers.
 2026-08-16 23:53:09 -07:00
Frank 7fe271e65d docs: llama.cpp stack at production state (2026-08-16) + sibling-project links 
- spec matrix: new 'Production state' section — PP head (RPC0+ROCm0,
  split 0.40/0.60), pool 401400 / n_ctx_slot 200704 live, MTP n_max 4
  over ngram-mod fallback (journal acceptance stats), f16 KV.
  2026-08-15 standalone state marked superseded (Vulkan0 verdict was
  standalone-only; PP head runs ROCm0 by default).
- deployment.md: fix frank-default-large section (MTP not 'no draft',
  ROCm0 not Vulkan0, 200704/slot not 262144, f16 not q8_0 KV,
  serve-pp-head.sh not serve-llama-frank-large.sh).
- PP deployment doc: production-state section; test-day open questions
  resolved with journal evidence.
- index.md: NPU line now qwen3.6-moe:35b-a3b @262k ctx; Sibling
  projects section -> serverAppConfigs.
- AGENTS.md: overview reflects PP+MTP reality; sibling pointer.
 2026-08-16 23:19:21 -07:00
Frank f5a78db017 docs: FLM rejects developer role — root cause of pi 'Stream ended without finish_reason' 
Streaming requests with role=developer get HTTP 200 + error JSON as
first body; pi sees a chunkless stream. Fix: compat.supportsDeveloperRole
false on the model entry. Verified by field bisection + headless pi run.
 2026-08-16 20:54:13 -07:00
Frank 9ec5678ef0 docs: qwen3.6-moe unreachable incident (2026-08-16) + topology reconciliation 
- FastFlowLM doc: incident section (EXEC-203 exec-bit loop, client model
  ID 'frank-default-small' rejected, swap churn 14:48-15:03), FLM
  unknown-tag fallback behavior (silently loads llama3.2:1b default),
  working models.json entry recorded, promotion status -> qwen3.6-moe
- AGENTS.md: Overview + target map now reflect qwen3.6-moe:35b-a3b on
  the NPU endpoint (Dankserver llama.cpp 35B service stopped)
- deployment.md: topology table current; Dankserver frank-default-small
  section marked STOPPED 2026-08-16 (alias moved to the NPU endpoint)
 2026-08-16 15:50:15 -07:00
Frank 50c55ee220 frank-default-small: swap to qwen3.6-moe:35b-a3b at 262144 ctx (native max), keep turbo+asr 2026-08-16 14:48:33 -07:00
Frank 2f1999b2e9 docs: frank-default-small live config — turbo, ASR, 131072 ctx verified 2026-08-16 14:38:37 -07:00
Frank fd4746422d frank-default-small: pmode turbo, ASR (Whisper-V3-Turbo) on, ctx-len 131072 (max verified) 2026-08-16 14:37:32 -07:00
Frank 02a1447602 head alias: frank-default -> frank-default-large (match renamed service) 2026-08-16 14:27:48 -07:00
Frank d0ade28f6d docs: fleet topology update — PP split renamed frank-default-large, NPU endpoint promoted to frank-default-small, 4-model NPU bakeoff matrix (2026-08-16) 2026-08-16 14:22:13 -07:00
Frank 3f42210922 fix: mark serve-flm-small.sh executable (unit failed EXEC 203) 2026-08-16 14:20:17 -07:00
Frank f6737f234d promote NPU endpoint to frank-default-small: FLM serve unit + wrapper + env (gemma4-it:e4b default), LimitMEMLOCK=infinity (2026-08-16) 2026-08-16 14:19:37 -07:00
Frank 48142f5d8e rename frank-pp-head to frank-default-large: PP split is the new production large endpoint (2026-08-16) 2026-08-16 13:44:31 -07:00
Frank 79c913aaa7 docs: NPU parallelism measured — single-lane FIFO serialization, aggregate t/s flat at N=1/2/4 2026-08-16 13:12:46 -07:00
Frank d48227c7f7 docs: gpt-oss-20b landed on the Dankclaw NPU — 19.3 t/s decode measured, memlock fix, serve state (2026-08-16) 2026-08-16 12:57:44 -07:00
Frank 0799fc5eb4 docs: FastFlowLM NPU research — AMD NPU runtime, 50 TOPS Strix Halo fit analysis, verified install state, test runbook (2026-08-16) 2026-08-16 12:29:11 -07:00
Frank 934c6d8be1 pp-head: 2x200700 f16 KV (ctx 401400, parallel 2), split 0.40,0.60 = max 3090 capacity 2026-08-16 12:23:08 -07:00
Frank 3ad4c6921c pp-head: alias frank-default, verbosity 4 (interactive use) 2026-08-16 10:42:29 -07:00
Frank 750730e4d9 pp-test: restore drops caches before production load; all-mode EXIT trap auto-restores; ssh ConnectTimeout 2026-08-16 10:17:01 -07:00
Frank 1716bf37c9 pp-test: preflight production check via health endpoint (no sudo needed) 2026-08-16 05:48:17 -07:00
Frank 9544c36afc PP kit + Strix research: Qwen3.8-27B RPC split (Dankclaw head / Dankserver worker) 
- scripts: serve-pp-head/worker, envs, systemd units, pp-test.sh orchestrator, pp-bench.py
- head design verified in b10433 source + live 4B smoke test:
  * no --device with --rpc (RPC device silently dropped otherwise)
  * default device selection = [RPC0, ROCm0] (HIP registers before Vulkan)
  * default split proportional to reported free mem = ~20/80 here, pin 1,1
  * -dio required for large model over RPC
- docs: qwen3.8-strix-optimizations (ranked no-quant-downgrade levers,
  quant landscape, 2026-08-16 test matrix), PP deployment runbook,
  index updates, Dankclaw Improvements corrections (Vulkan0, no power_profile sysfs)
- worker built at 9b05354ec in /home/dank/llama-cpp-rpc worktree (CUDA+RPC)
- second head build build-vulkan-rpc (HIP off) for backend A/B
 2026-08-16 05:40:27 -07:00
Frank c775077016 dankclaw: add MTP=1 toggle - prepend draft-mtp before --spec-default ngram-mod fallback 2026-08-15 22:31:29 -07:00
Frank b6f2ee2945 docs: record --spec-default + Vulkan0 + PARALLEL=2 final state; 262k-ctx ROCm-vs-Vulkan bench 2026-08-15 17:13:18 -07:00
Frank 5296cf11d7 dankclaw: switch backend to Vulkan0 (48% faster decode at 262k ctx); add bench-high-ctx.sh 2026-08-15 17:03:01 -07:00
Frank 1dbc70cb32 dankclaw: retire DSpark, use llama.cpp --spec-default (ngram-mod) preset; PARALLEL=2 trial 2026-08-15 15:15:06 -07:00
Frank 020e874a5c docs: record DSpark single-slot switch + ROCm parallel crash (2026-08-15) across KB; refs AGENTS/deployment/benchmark/matrix/index 2026-08-15 14:36:34 -07:00
Frank d04cc69e09 dankclaw: add accurate DSpark stream/queue test (PARALLEL=1) - reports true prompt_tokens; single+concurrent 16k pass 2026-08-15 14:32:21 -07:00
Frank 6266e43f10 dankclaw: DSpark parallel crash - ROCm illegal memory access under 2x16k prose, pin PARALLEL=1 per user 2026-08-15 14:24:27 -07:00
Frank b94bad066b dankclaw: switch frank-default-large spec to DSpark sidecar draft (magnitudedev Qwen3.8-27B-DSpark Q8_0, n_max 7) 2026-08-15 14:00:30 -07:00
Frank 40b079fe1e unit: refresh frank-default-large description (Qwen3.8-27B UD-Q8_K_XL, MTP+ngram-mod+ngram-map-k4v) 2026-08-14 21:01:22 -07:00
Frank 8d64adbcbe dankclaw: production spec -> 3-layer chain draft-mtp,ngram-mod,ngram-map-k4v (user request; ~equal real-prose speed) 2026-08-14 21:00:05 -07:00
Frank 2d7290a2a8 AGENTS: MANDATORY benchmarking policy - stop production service before any bench (serves ~3 real users); restart + health-check after 2026-08-14 20:54:34 -07:00
Frank 0eda208705 docs: Qwen3.8 spec-matrix benchmark page; add matrix tools; final env n_max=5 
Recording the 54-config matrix, the realistic-prompt correction (n_max=5 not
10), the ubatch findings (256 is the safe/fast point), and the premature-stop
sampling-variance investigation.
 2026-08-14 14:28:51 -07:00
Frank a818be70dc dankclaw Qwen3.8: winner n_max=5 (13.3 t/s real), not matrix n_max=10 
Realistic-prompt tuning: n_max=5 peaks ~13.3 t/s (baseline 7.2). matrix used a
repetitive prompt that inflated MTP. n_max=10 is only ~8.6 t/s on real writing.
 2026-08-14 14:26:14 -07:00
Frank 75bb9bb53e dankclaw Qwen3.8: batch 8192/ubatch 256 - fixes MTP collapse + premature-EOS 
Found: large ubatch (512) collapses MTP acceptance to ~25% (gen ~9 t/s);
ubatch 128 with batch>=2048 triggers premature instant-EOS under 2 concurrent
slots.  ubatch 256 = ~30 t/s gen, concurrent-safe, ~330 t/s prefill.
 2026-08-14 13:59:20 -07:00
Frank 57b0be037c dankclaw: switch frank-default-large to unsloth Qwen3.8-27B UD-Q8_K_XL 
- rebuild llama.cpp stock to b10433 (9b05354ec) for bundled-MTP support
- MODEL_FILE -> Qwen3.8-27B-UD-Q8_K_XL.gguf; DRAFT_FILE empty (MTP bundled)
- SPEC: draft-mtp, n_max=10, p_min off (~30 t/s stable) - matrix winner
- CTX 524288 / parallel 2 = 262144/slot (matches pi contextWindow)
- serve script: bundled-MTP (no --spec-draft-model), optional --spec-draft-p-min
 2026-08-14 13:26:17 -07:00
Frank d72092b2c7 docs: record dual-f16 KV refit on frank-default-small (2026-08-14) 2026-08-14 12:59:10 -07:00
Frank f96aa2b692 frank-default-small: refit ctx 800000->700000 for dual-f16 KV pool 
The f16/f16 KV pool costs 15640 MiB at 800768 cells. At full 800K the
3090 idled at 23966/24576 MiB and the 1472x1472 image warmup graph
failed cudaMalloc (needed +248 MiB). 700416 cells (175104/slot, 4 slots)
leaves ~2.3 GiB free after warmup. Parallel stays 4.
 2026-08-14 12:55:44 -07:00
Frank f70460b002 frank-default-small: KV K+V to f16, add verbosity tunable (F16 KV diag) 
F16 K removes the flash-attn F16-conversion workspace (MMA_F16 prefill
kernel needs no copy when K is already f16). Trades static KV pool size
for the context-proportional workspace growth implicated in the
cublasSgemm_v2 crash class. VERBOSITY=4 run logs real allocation sizes;
refit CTX_SIZE if the pool does not fit on the 24 GiB 3090.
 2026-08-14 12:50:59 -07:00
Frank bea52138d2 docs: record Dankserver VRAM/RAM growth investigation (2026-08-14) 2026-08-14 12:44:24 -07:00
Frank 9d7643ec5d frank-default-small: ncmoe 35->36 for prefill stability 
Fix a live crash: cuBLAS cudaErrorInvalidValue in ggml_cuda_mul_mat_cublas
killed the service during a 145k-token prefill burst at ncmoe35/ub3072.

ncmoe 36 moves one more expert layer to CPU: frees ~0.75 GiB peak VRAM
(grid: peak 8606 vs 9360 MiB at ub3072) for -2% prefill / -2% decode.
Keeps prefill priority (top ubatch tier). Docs synced; noted fallback
GGML_CUDA_FORCE_MMQ=1 if pressure persists.
 2026-08-14 10:56:40 -07:00
Frank 7daf9896e5 docs: record Ling freeze diagnosis - spec path deadlock, resolution spec-off parallel=2 2026-08-14 03:49:42 -07:00
Frank 139686043b ling B-test: spec off (--spec-type none), keep parallel=2 - isolate spec path as freeze trigger 2026-08-14 03:39:15 -07:00
Frank 7ba86812f7 ling A/B: verbosity 4->0, drop log file to isolate hang trigger (match known-good run) 2026-08-14 02:32:08 -07:00
Frank 6a6d673dca ling A/B: revert batch 8192->3072 to isolate mid-stream freeze (suspect batch) 2026-08-14 02:20:52 -07:00
Frank a5444e2c4d ling: batch 8192 (ubatch 3072); add -lv verbosity + log file for stop-investigation 2026-08-14 02:12:35 -07:00
Frank 321f946660 ling: ubatch 512 -> 3072 (batch 3072) - using heap headroom, watch for OOM 2026-08-14 01:46:36 -07:00
Frank a5b12e1062 ling: keep draft-mtp,ngram-mod active (A/B wash vs MTP alone, parallel-safe) 2026-08-14 01:45:22 -07:00
Frank 6647690ebd ling A/B: bench draft-mtp alone vs draft-mtp,ngram-mod 2026-08-14 01:43:53 -07:00
Frank 3204b4b29e ling trial: layer ngram-mod under draft-mtp; add ngram args to serve script 2026-08-14 01:40:46 -07:00