dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity
210 Commits 43 Branches 8 Tags
fe71f7516d0ea28012e7b63a54f361a3b60a76a8
Commit Graph
210 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Frank 8c42771877 docs: mark retired FastFlowLM/NPU pages with the port/name handoff; tighten Dankclaw system specs runtime row 2026-08-17 18:10:57 -07:00
Frank 115b0af8dc docs: reflect the 2026-08-17 evening changes (Quark model swap, tool calling, batch 8192) 2026-08-17 18:01:02 -07:00
Frank 1cebbf47f6 frank-default-small: swap to nameistoken Quark W8A8 INT8, batch 8192 
- Model: nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8 (AMD Quark W8A8,
  vLLM-native expert layout). The Avesed quant emitted malformed Qwen3
  XML tool calls; the Quark model follows the format cleanly.
- max-num-batched-tokens lowered 16384 -> 8192 (user request).
- Chat template: neutral path qwen36-chat_template-tools.jinja (same
  base template; adds a concrete worked tool-call example).
 2026-08-17 17:49:27 -07:00
Frank a17e638a1f frank-default-small: stronger tool template with a concrete worked example 
The Avesed template only shows a generic format; the Qwen3.5-MoE
model kept emitting <function=get_weather(city=...)> (args baked into
the name). Add a WORKED EXAMPLE block derived from the actual tool
schema (name + parameter keys) to the system prompt.
 2026-08-17 17:38:00 -07:00
Frank 20108e5b7a frank-default-small: use the repo chat_template.jinja so tools render in Qwen3 XML 
The Avesed tokenizer_config.json ships no chat_template, so vLLM used its
built-in template, which does not render the <tool_call> instructions;
the model then emitted malformed XML (arguments baked into the function
name) and the qwen3_xml parser could not extract clean calls.
 2026-08-17 17:13:35 -07:00
Frank 5108af11aa frank-default-small: enable tool calling (--enable-auto-tool-choice --tool-call-parser qwen3_xml) 
Clients send tool_choice=auto; without these flags vLLM rejects with
HTTP 400. The Avesed chat template renders the Qwen3 XML tool format,
so qwen3_xml is the matching parser.
 2026-08-17 16:50:23 -07:00
Frank 57264f7faa record performance fixes: FA_ALL_QUANTS build flag (dankserver), RDNA triton FA env (dankclaw) 
- frank-default-large: GGML_CUDA_FA_ALL_QUANTS=ON is mandatory in the
  FastMTP build; without it q8_0 KV + FA falls back to a slow path and
  prefill collapses ~12x (95 -> 1312 t/s at 8K, measured)
- frank-default-small: FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE + pure-python
  flash_attn (triton_amd) + stub; without it the vision encoder warmup
  OOMs via torch SDPA
 2026-08-17 16:30:22 -07:00
Frank 27788f6f2c fix: make serve scripts + deploy.sh executable 2026-08-17 15:24:40 -07:00
Frank d9af7f1d19 fleet redo 2026-08-17: retire PP+NPU, new topology 
- Move all previous scripts to old/scripts/ (retired PP/NPU-era)
- frank-default-large: llama.cpp FastMTP on dankserver (HauhauCS
  Qwen3.8-27B IQ4_XS + FastMTP 32K draft + ngram-mod, K q8_0/V f16,
  128K ctx, batch 2048/ubatch 1024, port 7770)
- frank-default-small: vLLM ROCm on dankclaw (Avesed Qwen3.6-35B-A3B
  INT8 W8A8, full 262144 ctx, no KV quant, 4 seqs, batch 16384,
  ngram spec, port 52625)
- retarget deploy.sh (dankserver/dankclaw, inline sudo, no helpers)
- docs: deployment + index + AGENTS.md + README for new topology,
  superseded banners on retired docs, add frank-redo-2026-08-17 runbook
 2026-08-17 15:23:46 -07:00
Frank 136ee080c9 docs: measured PP link math (H=5120 verified, 20.5 MB/ubatch wire), NIC inventory 
- Ethernet Bandwidth Math: new section 0 with live measurements
  (iperf 281-294 MB/s, ping RTT both paths, prefill 457.9 t/s at
  2.5GbE, per-ubatch wire 20.5 MB, 98k KV snapshot 3.45 GB blocking
  the slot 13.6 s, validated link-speed model: 386/458/505 t/s)
- System specs: NIC tables for both boxes; the 1 GbE USB-C adapter on
  the claw was silently routing the RPC path (unplugged 2026-08-17)
- PP Deployment: fix hidden-state size (10 KiB/direction, H=5120) and
  RTT-vs-bandwidth claim (decode RTT-bound; prefill+snapshot BW-bound)
 2026-08-17 05:06:41 -07:00
Frank 968066ff23 docs: boot policy (only PP units enabled), deploy.sh dankserver hazard, stale-claim cleanup 
- deployment.md: Boot Policy (2026-08-17) section (disabled units table,
  re-enable recipes, stale frank-restart-small.sh helper, broken
  deploy.sh dankserver target); Update Dankserver -> frank-pp-worker;
  condensed retired 35B history block and 2026-08-13 incident
- AGENTS.md: boot-policy note, target map fix (Q6_K, split 0.68),
  deploy.sh dankserver warning, obsolete quirk entries condensed
- README.md: rewrite active services/models tables (PP head + worker)
- index.md: topology fix (qwythos:9b, Q6_K split 0.68, NPU disabled)
- qwen3.8-spec-matrix.md: superseded configs merged into compact
  history; stale inline env block removed; bundled-head MTP note
- FastFlowLM, PP Deployment: boot-policy + enablement notes
- System Specs x2, DFlash doc, 35B FP8 doc, Ling docs: stale active
  claims fixed, retired banners, never-shipped plan stubbed
 2026-08-17 03:30:00 -07:00
Frank 3f88b8e1ed docs + script defaults: Q6_K swap, q8_0/f16 KV, split 0.68, FA rebuild 
- spec matrix + PP doc + deployment.md + AGENTS.md + index.md: current
  production = Uncensored Q6_K + standalone MTP draft, mismatched
  q8_0/f16 KV with FA retained (GGML_CUDA_FA_ALL_QUANTS rebuild at
  9b05354ec), split 0.68 (45 blocks), KV 8624/3920, 273 MiB free.
- PP doc: 'FA-all-quants rebuild' section (flag, cmake commands, both
  hosts) + new-model split-limits measurements; old Q8_K_XL limits
  marked superseded.
- serve-pp-head.sh defaults: MODEL_FILE -> Q6_K, CACHE_TYPE_V -> f16
  (env already correct; defaults now match production).
 2026-08-17 02:08:08 -07:00
Frank 11f37dafd8 frank-default-large: split 0.68,0.32 = 45 blocks on 3090 (Q6_K phase 2) 
Max for the Q6_K + q8_0/f16 KV config: 45 blocks (0-44), 11 FA layers
(3..43), KV 8624 MiB, ~0.7 GiB free. 46 layers -> ~0.4 GiB margin; 48
layers (f>=0.7197) adds FA block 47 -> KV 9408 -> OOM by ~0.85 GiB.
Draft model measured on ROCm0 only (no worker impact). Phase 1 (0.53)
verified: acceptance 0.833, KV 6272/6272, flash_attn auto, healthy.
 2026-08-17 02:05:04 -07:00
Frank 1bc39c43ce frank-default-large: swap to Qwen3.8-27B-Uncensored Q6_K, q8_0/f16 KV 
- MODEL_FILE -> Qwen3.8-27B-Uncensored-Q6_K.gguf (22.43 GB, no bundled
  MTP head; repo ships a standalone draft sidecar)
- SPEC_DRAFT_MODEL -> Qwen3.8-27B-Uncensored-draft-Q8_0.gguf (3.16 GB)
- CACHE_TYPE_K=q8_0, CACHE_TYPE_V=f16 (mismatched, FA retained via the
  GGML_CUDA_FA_ALL_QUANTS rebuild of head HIP + worker CUDA at 9b05354ec)
- Split kept at 0.53 for phase 1 (Q6_K frees ~4.7 GiB on the worker at
  this split); re-maximize after measuring draft placement
- KV math: K 1088 B/token + V 2048 B/token = 784 MiB per FA layer
 2026-08-17 02:01:22 -07:00
Frank e9ec5cd13f docs: NPU endpoint now serves qwythos:9b (2026-08-17 swap, FLM 1.0.1) 
- AGENTS.md + deployment.md target maps: qwen3.6-moe:35b-a3b -> qwythos:9b
- FLM doc: FLM 1.0.1 upgrade record, flm-add install state, model swap
  paragraph (commit c85ff6c), FLM_CONFIG_PATH/FLM_XCLBIN_PATH requirement
- pi client entry note: id qwythos:9b, text-only input, developer-role compat kept
 2026-08-17 01:24:43 -07:00
Frank aa1941d592 docs + guard: single 262k slot state, split-limits analysis, cache incident 
- spec matrix + PP doc: production state = 1 slot @ 262144, split 0.53
  (34 blocks), KV 8192/8192, 3090 at 23460/24576. New 'Split limits
  (measured 2026-08-17)' section: 0.50/0.54/0.55 boundary data, the
  KV-per-device rounding rule (round(f x 66)), and why 35 blocks is
  unreachable (9-FA flip at f~0.538).
- PP doc: cache incident writeup - full Dankserver disk -> 0-byte cache
  entries -> corrupt weights -> degenerate output; fix + signature.
- deployment.md + AGENTS.md + index.md: numbers brought current.
- scripts/prune-rpc-cache.sh: cron guard capping the worker cache at
  40 GiB (oldest first) to prevent the disk-full recurrence.
 2026-08-17 01:24:06 -07:00
Frank c85ff6c6fd frank-default-small: NPU model swap qwen3.6-moe:35b-a3b -> qwythos:9b (Huihui-Qwythos-9B-Claude-Mythos-5-1M-abliterated) 
- FLM_MODEL=qwythos:9b, CTX 262144, turbo+asr kept
- Add FLM_CONFIG_PATH/FLM_XCLBIN_PATH: the custom tag lives in the user
  registry (~/.config/flm/model_list.json) + user xclbins; without them
  flm hits the unknown-tag fallback and serves llama3.2:1b silently
- Note FLM 0.9.45 -> 1.0.1 runtime upgrade (pack declares flm_version 1.0.0)
 2026-08-17 01:22:32 -07:00
Frank 2b96043740 frank-default-large: split 0.53,0.47 - provably-safe max (34 blocks) 
0.54 measured: weight boundary moved to 35 blocks but the KV-per-device
count (round(f x 66), independent of weight boundary) flipped to 9 FA ->
KV 9216 MiB -> OOM by ~740 MiB. The flip sits at f ~0.538, just below
the weight threshold for block 35 -> the 35-block config is unreachable.
0.53 gives 34 blocks (0-33), 8 FA, KV 8192 MiB under every rounding rule
(round/floor/ceil of 0.53x66 = 34.98 -> 34 or 35, never 36), predicted
~1.1 GiB free.
 2026-08-17 01:21:21 -07:00
Frank 193034fdb8 frank-default-large: split 0.54,0.46 - 35 blocks on 3090 (0.55 overran to 36 -> OOM) 
0.55 measured: 36 blocks (0-35) -> 9th FA block, KV 9216 MiB, cudaMalloc
OOM by ~740 MiB, crash loop. Exact GGUF math: layer pool 27720 MB + MTP
head; split assigns blocks while cumulative <= fraction x pool; cum(34)=
53.83%, cum(35)=55.31%. 0.54 lands 35 blocks (0-34), 8 FA layers, KV
8192 MiB, ~0.66 GiB free.
 2026-08-17 00:58:06 -07:00
Frank accf90b6fa frank-default-large: split 0.55,0.45 = 35 blocks on the 3090 (true max) 
GGUF block scan (2026-08-17): FA at 3,7,...,63; linear blocks 383 MiB,
FA blocks 446 MiB. 35 blocks (0-34) keep 8 FA layers on RPC0 -> KV stays
8192 MiB, predicted ~0.8 GiB free. Block 35 is FA: +1 GiB KV -> OOM.
Measured at 0.50: 22650/24576 used, 1477 free (the earlier 714 MiB
projection was conservative).
 2026-08-17 00:34:58 -07:00
Frank 53ada02a63 frank-default-large: PARALLEL=1, CTX 262144, split 0.50 (max 3090 weight) 
Single full-context slot instead of two half-context slots. The 3090 takes
~33 of 66 layers (up from 26): the VRAM budget frees up because the f16 KV
pool shrinks 401400 -> 262144. 33 layers is the max with a safety margin
(34 -> 244 MiB headroom, 35 -> OOM). Verified math: /tmp/vram-math.py from
live journal buffers.
 2026-08-16 23:53:09 -07:00
Frank 7fe271e65d docs: llama.cpp stack at production state (2026-08-16) + sibling-project links 
- spec matrix: new 'Production state' section — PP head (RPC0+ROCm0,
  split 0.40/0.60), pool 401400 / n_ctx_slot 200704 live, MTP n_max 4
  over ngram-mod fallback (journal acceptance stats), f16 KV.
  2026-08-15 standalone state marked superseded (Vulkan0 verdict was
  standalone-only; PP head runs ROCm0 by default).
- deployment.md: fix frank-default-large section (MTP not 'no draft',
  ROCm0 not Vulkan0, 200704/slot not 262144, f16 not q8_0 KV,
  serve-pp-head.sh not serve-llama-frank-large.sh).
- PP deployment doc: production-state section; test-day open questions
  resolved with journal evidence.
- index.md: NPU line now qwen3.6-moe:35b-a3b @262k ctx; Sibling
  projects section -> serverAppConfigs.
- AGENTS.md: overview reflects PP+MTP reality; sibling pointer.
 2026-08-16 23:19:21 -07:00
Frank f5a78db017 docs: FLM rejects developer role — root cause of pi 'Stream ended without finish_reason' 
Streaming requests with role=developer get HTTP 200 + error JSON as
first body; pi sees a chunkless stream. Fix: compat.supportsDeveloperRole
false on the model entry. Verified by field bisection + headless pi run.
 2026-08-16 20:54:13 -07:00
Frank 9ec5678ef0 docs: qwen3.6-moe unreachable incident (2026-08-16) + topology reconciliation 
- FastFlowLM doc: incident section (EXEC-203 exec-bit loop, client model
  ID 'frank-default-small' rejected, swap churn 14:48-15:03), FLM
  unknown-tag fallback behavior (silently loads llama3.2:1b default),
  working models.json entry recorded, promotion status -> qwen3.6-moe
- AGENTS.md: Overview + target map now reflect qwen3.6-moe:35b-a3b on
  the NPU endpoint (Dankserver llama.cpp 35B service stopped)
- deployment.md: topology table current; Dankserver frank-default-small
  section marked STOPPED 2026-08-16 (alias moved to the NPU endpoint)
 2026-08-16 15:50:15 -07:00
Frank 50c55ee220 frank-default-small: swap to qwen3.6-moe:35b-a3b at 262144 ctx (native max), keep turbo+asr 2026-08-16 14:48:33 -07:00
Frank 2f1999b2e9 docs: frank-default-small live config — turbo, ASR, 131072 ctx verified 2026-08-16 14:38:37 -07:00
Frank fd4746422d frank-default-small: pmode turbo, ASR (Whisper-V3-Turbo) on, ctx-len 131072 (max verified) 2026-08-16 14:37:32 -07:00
Frank 02a1447602 head alias: frank-default -> frank-default-large (match renamed service) 2026-08-16 14:27:48 -07:00
Frank d0ade28f6d docs: fleet topology update — PP split renamed frank-default-large, NPU endpoint promoted to frank-default-small, 4-model NPU bakeoff matrix (2026-08-16) 2026-08-16 14:22:13 -07:00
Frank 3f42210922 fix: mark serve-flm-small.sh executable (unit failed EXEC 203) 2026-08-16 14:20:17 -07:00
Frank f6737f234d promote NPU endpoint to frank-default-small: FLM serve unit + wrapper + env (gemma4-it:e4b default), LimitMEMLOCK=infinity (2026-08-16) 2026-08-16 14:19:37 -07:00
Frank 48142f5d8e rename frank-pp-head to frank-default-large: PP split is the new production large endpoint (2026-08-16) 2026-08-16 13:44:31 -07:00
Frank 79c913aaa7 docs: NPU parallelism measured — single-lane FIFO serialization, aggregate t/s flat at N=1/2/4 2026-08-16 13:12:46 -07:00
Frank d48227c7f7 docs: gpt-oss-20b landed on the Dankclaw NPU — 19.3 t/s decode measured, memlock fix, serve state (2026-08-16) 2026-08-16 12:57:44 -07:00
Frank 0799fc5eb4 docs: FastFlowLM NPU research — AMD NPU runtime, 50 TOPS Strix Halo fit analysis, verified install state, test runbook (2026-08-16) 2026-08-16 12:29:11 -07:00
Frank 934c6d8be1 pp-head: 2x200700 f16 KV (ctx 401400, parallel 2), split 0.40,0.60 = max 3090 capacity 2026-08-16 12:23:08 -07:00
Frank 3ad4c6921c pp-head: alias frank-default, verbosity 4 (interactive use) 2026-08-16 10:42:29 -07:00
Frank 750730e4d9 pp-test: restore drops caches before production load; all-mode EXIT trap auto-restores; ssh ConnectTimeout 2026-08-16 10:17:01 -07:00
Frank 1716bf37c9 pp-test: preflight production check via health endpoint (no sudo needed) 2026-08-16 05:48:17 -07:00
Frank 9544c36afc PP kit + Strix research: Qwen3.8-27B RPC split (Dankclaw head / Dankserver worker) 
- scripts: serve-pp-head/worker, envs, systemd units, pp-test.sh orchestrator, pp-bench.py
- head design verified in b10433 source + live 4B smoke test:
  * no --device with --rpc (RPC device silently dropped otherwise)
  * default device selection = [RPC0, ROCm0] (HIP registers before Vulkan)
  * default split proportional to reported free mem = ~20/80 here, pin 1,1
  * -dio required for large model over RPC
- docs: qwen3.8-strix-optimizations (ranked no-quant-downgrade levers,
  quant landscape, 2026-08-16 test matrix), PP deployment runbook,
  index updates, Dankclaw Improvements corrections (Vulkan0, no power_profile sysfs)
- worker built at 9b05354ec in /home/dank/llama-cpp-rpc worktree (CUDA+RPC)
- second head build build-vulkan-rpc (HIP off) for backend A/B
 2026-08-16 05:40:27 -07:00
Frank c775077016 dankclaw: add MTP=1 toggle - prepend draft-mtp before --spec-default ngram-mod fallback 2026-08-15 22:31:29 -07:00
Frank b6f2ee2945 docs: record --spec-default + Vulkan0 + PARALLEL=2 final state; 262k-ctx ROCm-vs-Vulkan bench 2026-08-15 17:13:18 -07:00
Frank 5296cf11d7 dankclaw: switch backend to Vulkan0 (48% faster decode at 262k ctx); add bench-high-ctx.sh 2026-08-15 17:03:01 -07:00
Frank 1dbc70cb32 dankclaw: retire DSpark, use llama.cpp --spec-default (ngram-mod) preset; PARALLEL=2 trial 2026-08-15 15:15:06 -07:00
Frank 020e874a5c docs: record DSpark single-slot switch + ROCm parallel crash (2026-08-15) across KB; refs AGENTS/deployment/benchmark/matrix/index 2026-08-15 14:36:34 -07:00
Frank d04cc69e09 dankclaw: add accurate DSpark stream/queue test (PARALLEL=1) - reports true prompt_tokens; single+concurrent 16k pass 2026-08-15 14:32:21 -07:00
Frank 6266e43f10 dankclaw: DSpark parallel crash - ROCm illegal memory access under 2x16k prose, pin PARALLEL=1 per user 2026-08-15 14:24:27 -07:00
Frank b94bad066b dankclaw: switch frank-default-large spec to DSpark sidecar draft (magnitudedev Qwen3.8-27B-DSpark Q8_0, n_max 7) 2026-08-15 14:00:30 -07:00
Frank 40b079fe1e unit: refresh frank-default-large description (Qwen3.8-27B UD-Q8_K_XL, MTP+ngram-mod+ngram-map-k4v) 2026-08-14 21:01:22 -07:00
Frank 8d64adbcbe dankclaw: production spec -> 3-layer chain draft-mtp,ngram-mod,ngram-map-k4v (user request; ~equal real-prose speed) 2026-08-14 21:00:05 -07:00