dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

frank-default-large: cap CTX_SIZE at 131072 — 262144 OOMs SVM pool (cudaMalloc 103176 MiB failed with fully GPU-resident 87 GiB model)

Browse Source
This commit is contained in:
Frank committed 2026-08-29 22:48:17 -07:00
1 parent 2f153c89f4
commit 7ac172c4f4
4 files changed
+19 -9
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+4 -3
View File
@@ -27,7 +27,7 @@ host.
| Target | Runtime | Service | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
| Dankclaw | charlie12345/ROCmFPX llama.cpp (build-qwen38) | frank-default-large | agentionai Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87 GiB Q4_0_ROCMFP4_FAST, PLE joined table in v2 GGUF), ngram-mod spec (MTP head on disk, incompatible with this build), 262K context / 1 slot, reasoning on | 7770 |
| Dankclaw | charlie12345/ROCmFPX llama.cpp (build-qwen38) | frank-default-large | agentionai Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87 GiB Q4_0_ROCMFP4_FAST, PLE joined table in v2 GGUF), ngram-mod spec (MTP head on disk, incompatible with this build), 131K context / 1 slot, reasoning on | 7770 |
History: the 2026-08-17 redo retired the pipeline-parallel split and the
FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
@@ -293,8 +293,9 @@ VRAM-resident, no sidecar, no `--ngram-on-disk`. The model's own MTP draft head
quant matched to target) is on disk but does **not** load on this binary
(loader requires a full per-layer `blk.<il>.hc_attn_norm.weight` set; the head
carries only `blk.48`) — speculative decoding runs ngram-mod only. Vision
tower (f16) ships as mmproj/. It runs 262144 context / 1
slot, b 4096 / ub 4096, t 16 / tb 16, f16 KV cache, `--fit off` (auto-fit
tower (f16) ships as mmproj/. It runs 131072 context / 1
slot (capped: the GPU-resident 87 GiB model + 262144 f16 KV + compute
buffer OOMs the SVM pool — verified live, cudaMalloc 103176 MiB), b 4096 / ub 4096, t 16 / tb 16, f16 KV cache, `--fit off` (auto-fit
segfaults on qwen4exp hyper-connection config), `--flash-attn off` (required:
fa on aborts graph build on the QSA indexer path — F32 `ggml_fill` on the F16
kq mask, ggml.c:5371), and reasoning on (Sharp