Commit Graph
13 Commits
Author SHA1 Message Date
Frank bea52138d2 docs: record Dankserver VRAM/RAM growth investigation (2026-08-14) 2026-08-14 12:44:24 -07:00
Frank db69fc8192 docs: pipeline parallelism research + educational sample script set
Compiles the 5-subagent research fan-out (llama.cpp RPC, vLLM PP,
bandwidth math, mixed-vendor experiments, Qwen3.6-27B spec methods) into
the KB under docs/Pipeline Parallelism/ with an index, expected
performance estimates (1/2.5/10 GbE), and an EDUCATIONAL-ONLY two-node
Qwen3.6-27B Q8 split script set (MTP/DFlash/ngram-mod variants, DSpark
noted as vLLM-only). Nothing here is deployed or intended to run.
2026-08-14 00:22:23 -07:00
Frank 8d1d9baa82 docs: MTP wins the DFlash replacement; record parallel premature-EOS incident
- benchmark doc: spec A/B table (MTP 15.3-19.5, MTP+ngram 12-13.6,
  MTP n_max2 12.4-14.3, ngram 6.7-8.7, DFlash retired), the DFlash
  shared-draft-context premature-EOS incident, updated batch/ubatch.
- deployment.md: Dankclaw now draft-mtp n_max 3 + MTP-ONLY-Q8_0 sidecar,
  batch 8192/ubatch 8192; note Dankserver bench drift.
- index.md/AGENTS.md: corrected stale topology (Ling-3.0-flash, DFlash).
- DFlash doc: Dankclaw section rewritten for MTP; DFlash retired.
2026-08-13 23:27:06 -07:00
Frank 3c2db64a78 frank-default-small: ncmoe 32 + ubatch 1024 from llama-bench sweep
Swept ncmoe(30..40) x ubatch(256..4096) on the RTX 3090 (Qwen3.6-35B-A3B
Q6_K_P, pp16384/tg512, fit-target 0). Key findings:
- Generation depends only on ncmoe: 40.9 (ncmoe40) .. 51.1 (ncmoe30);
  ubatch/batch do not affect decode at all.
- Prefill + graph VRAM scale with ubatch; logical batch is irrelevant.
- VRAM model (peak + 11.44GiB KV pool + 2.2GiB overhead) reproduces the
  historical OOMs: ncmoe32-ub2048 sums ~24.4GiB > 24.

Chosen state: ncmoe 32 (8 expert layers on GPU), batch 8192, ubatch 1024.
  gen ~48.7 t/s (+7% vs prior ncmoe 35), prefill ~766 t/s, ~1.0GiB headroom.
Alternates: ncmoe34-ub2048 (gen 46.4, pp 1221, 1.5GiB) for prefill-heavy;
  ncmoe32-ub768 (1.3GiB) if headroom matters most.
Docs: new frank-default-small-benchmark.md; synced deployment.md + specs.
2026-08-13 22:56:24 -07:00
Frank f20d6ce288 docs: add Dankclaw improvements research (agentic tools, runtime tuning, server/OS) 2026-08-11 18:14:44 -07:00
Frank e92c72fd11 docs: frank-default-large llama.cpp draft-dflash deployment + benchmark results; retire Ling vLLM references 2026-08-11 17:24:59 -07:00
Frank 35c44c4123 docs+deploy: Dankserver now llama.cpp frank-default-small (HauhauCS Q6_K_P + DFlash)
Retire vllm-awq-qwen36 (AWQ) from target maps, README, deployment, KB index,
Dankserver specs, DFlash and FP8 docs. New dankserver runtime: upstream
llama.cpp, port 7770, alias frank-default-small, 3x262144 ctx, DFlash Q8_0
draft, MoE experts offloaded to CPU, attention+KV on GPU (K q8_0, V f16).
deploy.sh now maps dankserver to frank-default-small and symlinks its unit.
models.json: dankserver baseUrl -> :7770, contextWindow -> 262144.
2026-08-11 01:18:29 -07:00
Frank 903d54c389 docs: reflect Ling-3 vLLM (frank-default-large) on Dankclaw
Update target maps (AGENTS, README, deployment, deploy.sh) from the retired
ROCmFPX llama.cpp stack to the active vLLM-ROCm CIRU INT4 service on port 7770
with MTP speculative decoding. Align Dankserver to the running
vllm-awq-qwen36 (AWQ, port 34401). Refresh KB pages: Dankclaw system specs
(gpu-mem-util 0.95, CIRU venv), quantization choices (INT4 primary, MXFP4
experiment), DFlash/DSpark Dankclaw section, and the KB index topology.
2026-08-10 23:37:27 -07:00
Frank 993b6af576 docs: add DFlash/DSpark speculative decoding research + quantization-by-hardware guide
- DFlash DSpark Speculative Decoding.md: verified method details (DFlash
  block diffusion from Z Lab arXiv:2602.06036; DSpark semi-autoregressive from
  DeepSeek arXiv:2607.05147), runtime support matrix (vLLM/llama.cpp/SGLang),
  exact vLLM --speculative-config recipes (incl. dflash/dspark), llama.cpp
  --spec-type recipes with sidecar auto-discovery, draft model table
  (z-lab Qwen3.6-27B-DFlash 2B, Qwen3.6-35B-A3B-DFlash 0.4B), deployment
  guidance for RTX 3090 (AWQ+Dflash) and Strix Halo (ROCmFPX MTP), measured
  45-50 t/s MTP decode on Strix Halo
- Quantization by Hardware Architecture.md: FP8/INT8/INT4/GPTQ/AWQ/GGUF/FP4/
  ROCmFP4 support per NVIDIA (Turing/Ampere/Ada/Hopper/Blackwell), AMD
  (CDNA MI250/MI300/MI350, RDNA 3/3.5/4), and CPU; recommended picks per
  Dankserver (AWQ/GPTQ 4-bit) and Dankclaw (ROCmFP4/Q4_K_M)
- index.md: link both new pages under Speculative Decoding & Quantization
2026-08-10 21:09:17 -07:00
Frank 90c4883f89 feat: add consistent deployment/update process (deploy.sh + docs + AGENTS.md)
- scripts/deploy.sh: target-aware deploy (dankclaw/dankserver). Pushes main,
  pulls on the target, installs/enables systemd unit, restarts service, and
  checks the health endpoint; --dry-run preview; pre-flight dirty-tree check
- docs/deployment.md: full deployment/update process with mermaid workflow,
  target map, manual update commands, service units, and troubleshooting
- AGENTS.md: add Deployment & Update Process section (deploy.sh usage, target
  map, mermaid workflow) and a Summary table row
- docs/index.md: crosslink deployment doc
2026-08-10 21:04:37 -07:00
Frank 3a64c8f3dd docs: fact-check knowledge base against live machines and vLLM source
Correct stale and incorrect facts after exhaustive verification:
- Dankclaw: NUMA node 0 = 31213 MB; disk 661G/1.1T/38%; vLLM 0.26.0 in
  /home/dank/vllm-rocm-venv (not vllm-env); clarify 96 GiB GPU-visible is a
  driver overcommit ceiling over 30 GiB physical (confirmed via sysfs
  mem_info_vram_total=103079215104)
- Dankserver: disk 729G/140G free/84%; /mnt/entropy 910G free; active service
  is vLLM EngineCore (22 GB), not llama-server; CPU model Ryzen 7 7700X
- index.md: Dankclaw 30GiB phys / 96GiB GPU-visible (was '61 GB unified');
  add deployment topology mermaid diagram
- vLLM defaults (from source): gpu_memory_utilization=0.92, max_num_seqs=128,
  swap_space removed (now --kv-offloading-size) across Overview/Index/RAM/Hybrid
- Quantization: FP8 not native on Ampere (RTX 3090) or RDNA 3.5 (gfx1151);
  add per-model hardware quantization notes; AWQ/GPTQ limited on ROCm
- Speculative decoding: add DFlash, DSpark, MTP, EAGLE3 to feature list
- Confirm Qwen3.6-27B and Qwen3.6-35B-A3B specs are correct (HF-verified)
- Production note: Dankclaw primary llama.cpp service runs ROCmFPX
  Ling-3.0-flash RocmFP4 MTP, not vLLM Qwen3.6-27B
2026-08-10 20:59:33 -07:00
Frank 5076ef828f chore: fix typo in dankclaw specs, format index table 2026-08-06 22:29:23 -07:00
Frank dc356acce7 docs: add top-level knowledge base index 2026-08-06 00:16:47 -07:00