diff --git a/AGENTS.md b/AGENTS.md index c4069b2..a231215 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,7 +27,7 @@ host. | Target | Runtime | Service | Model | Port | |--------|---------|---------|-------|------| | Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 | -| Dankclaw | charlie12345/ROCmFPX llama.cpp (build-qwen38) | frank-default-large | agentionai Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87 GiB Q4_0_ROCMFP4_FAST, PLE joined table in v2 GGUF), ngram-mod spec (MTP head on disk, incompatible with this build), 131K context / 1 slot, reasoning on | 7770 | +| Dankclaw | charlie12345/ROCmFPX llama.cpp (build-qwen38) | frank-default-large | agentionai Qwen3.8-Flash-Next-ROCmFP4-FAST-imatrix (87 GiB Q4_0_ROCMFP4_FAST, PLE joined table in v2 GGUF), ngram-mod spec (MTP head on disk, incompatible with this build), 32K context / 1 slot, reasoning on | 7770 | History: the 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on @@ -293,12 +293,13 @@ VRAM-resident, no sidecar, no `--ngram-on-disk`. The model's own MTP draft head quant matched to target) is on disk but does **not** load on this binary (loader requires a full per-layer `blk..hc_attn_norm.weight` set; the head carries only `blk.48`) — speculative decoding runs ngram-mod only. Vision -tower (f16) ships as mmproj/. It runs 131072 context / 1 -slot (capped: the GPU-resident 87 GiB model + 262144 f16 KV + compute -buffer OOMs the SVM pool — verified live, cudaMalloc 103176 MiB), b 4096 / ub 4096, t 16 / tb 16, f16 KV cache, `--fit off` (auto-fit -segfaults on qwen4exp hyper-connection config), `--flash-attn off` (required: -fa on aborts graph build on the QSA indexer path — F32 `ggml_fill` on the F16 -kq mask, ggml.c:5371), and reasoning on (Sharp +tower (f16) ships as mmproj/. It runs 32768 context / 1 +slot (author quick-start; the fully GPU-resident 87 GiB model + f16 KV + +compute buffer OOMs the SVM pool beyond that — verified live: cudaMalloc +fails at 131072/262144), b 4096 / ub 4096, t 16 / tb 16, f16 KV cache, +`--fit off` (auto-fit segfaults on qwen4exp hyper-connection config), +`--flash-attn off` (required: fa on aborts graph build on the QSA indexer +path — F32 `ggml_fill` on the F16 kq mask, ggml.c:5371), and reasoning on (Sharp template, budget -1, `--reasoning-format deepseek`; chat kwargs keep `preserve_thinking` + `preserve_reasoning`). Env `scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`. diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index c989520..bbaa99c 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -60,11 +60,14 @@ LISTEN_PORT=7770 # hyper-connection/indexer config; -ngl 999 puts every layer on the # SVM-backed GPU pool. # -# CTX_SIZE is capped at 131072: the fully GPU-resident 87 GiB model plus a -# 262144-token f16 KV cache and the worst-case compute buffer exceed the -# Strix Halo SVM pool — verified live (cudaMalloc 103176 MiB OOM at -# 262144). 131072 is the proven STRIX-family value on this box. -CTX_SIZE=131072 +# CTX_SIZE is capped at 32768 (the author's own quick-start). This model is +# fully GPU-resident (87.06 GiB) — the PLE n-gram table no longer streams +# from RAM like STRIX_LEAN did, so there is no room for a huge context +# buffer: HIP heap is 119.3 GiB, and the context buffer alone is 50.8 GiB +# at 131072 and 103 GiB at 262144 (verified live: +# `cudaMalloc … failed: out of memory` on first graph alloc). At 32768 the +# context buffer drops to ~13 GiB (~100 GiB total) and loads clean. +CTX_SIZE=32768 PARALLEL=1 BATCH_SIZE=4096 UBATCH_SIZE=4096 diff --git a/scripts/serve-frank-large.sh b/scripts/serve-frank-large.sh index 04c0148..a394349 100755 --- a/scripts/serve-frank-large.sh +++ b/scripts/serve-frank-large.sh @@ -45,7 +45,7 @@ fi : "${ALIAS:=frank-default-large}" : "${BIND_ADDR:=0.0.0.0}" : "${LISTEN_PORT:=7770}" -: "${CTX_SIZE:=131072}" +: "${CTX_SIZE:=32768}" : "${PARALLEL:=1}" : "${BATCH_SIZE:=4096}" : "${UBATCH_SIZE:=4096}" diff --git a/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md b/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md index 5d5fa3c..2487312 100644 --- a/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md +++ b/serverAppConfigs-docs/LLM/agentionai-fast-imatrix.md @@ -89,7 +89,7 @@ llama-server \ --host 0.0.0.0 --port 7770 --jinja \ --device ROCm0 --gpu-layers 999 \ --split-mode none --fit off --flash-attn off \ - --ctx-size 131072 --parallel 1 \ + --ctx-size 32768 --parallel 1 \ --batch-size 4096 --ubatch-size 4096 \ --cache-type-k f16 --cache-type-v f16 \ --threads 16 --threads-batch 16 \ @@ -119,12 +119,13 @@ Key choices and deviations from the CIRU deployment: mask and aborts graph build (`GGML_ASSERT(a->type == GGML_TYPE_F32)`, ggml.c:5371, backtrace through `build_attn_qsa`). Verified live on this v2 main — loads and serves fine with fa off. -- Context capped at 131072 × 1 slot. The 262144 window does NOT fit: the - fully GPU-resident 87 GiB model plus a 262144-token f16 KV cache plus the - worst-case compute buffer exceeds the Strix Halo SVM pool (verified live: - `cudaMalloc 103176 MiB failed: out of memory` on first graph alloc at - 262144). 131072 is the proven STRIX-family value on this box. (The author's - quick-start is 32768 — the full 131K is comfortable.) +- Context capped at **32768 × 1 slot — the author's own quick-start**. This + model is fully GPU-resident (87.06 GiB): the PLE n-gram table sits inside + the GGUF as one tensor, unlike STRIX_LEAN which streamed it from RAM. + HIP heap is 119.3 GiB; the context buffer is 50.8 GiB at 131072 and + 103 GiB at 262144, so either window OOMs (`cudaMalloc … failed: out of + memory`, verified live on first graph alloc). At 32768 the buffer is + ~13 GiB (~100 GiB total) and loads clean. - MTP is disabled (ngram-mod only) — see the MTP section below. ## MTP draft head — INCOMPATIBLE on build-qwen38