diff --git a/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md b/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md index aecbdb9..20864f9 100644 --- a/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md +++ b/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md @@ -108,6 +108,24 @@ architecture, driver and queue budget. - **Memory:** gufo's own table puts the 8-bit target at **50.58 GiB** peak with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture. +### Measured at the 2026-09-28 cutover + +Live on Dankclaw after `deploy.sh dankclaw --service frank-large`: + +- **13 s** from `load_started` to `listening` (target 8 s, drafter 2 s, + session 3 s), against a 20-minute `LLM_WAIT` budget — a redeploy no longer + sits in a load wait. +- **RSS 30.6 GiB**, VSZ 87.3 GiB; host `used` 81 GiB of 121 GiB, i.e. the + same order as the llama.cpp deployment it replaced. +- A short-prompt live request returned + `Server-Timing: ttft;dur=549.8, inter_token;dur=24.2, max_inter_token;dur=166.9` + (≈41 tok/s). Short-prompt decode is gufo's *winning* regime; see the depth + table above before quoting it as the endpoint's number. +- Contract battery, all green: `/health` 200, `/ready` 200, + `/v1/models` → `frank-large` with `context_length 262144`, a chat + completion carrying both `content` and `reasoning_content`, a `tool_calls` + response with `finish_reason: tool_calls`, and SSE streaming. + ## Client contract — what actually changed Everything the fleet depended on still holds: port 7770, alias `frank-large`, diff --git a/serverAppConfigs-docs/log.md b/serverAppConfigs-docs/log.md index 993b937..dd384c3 100644 --- a/serverAppConfigs-docs/log.md +++ b/serverAppConfigs-docs/log.md @@ -2,6 +2,80 @@ ## 2026-09-28 +- **frank-large runtime swapped to gufo** (production change, Dankclaw). + Frank asked for `github.com/gufo-org/gufo` — a vertical inference engine + built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S, + `gfx1151`) — in place of the stock llama.cpp prebuilt. Payload, port 7770, + alias `frank-large`, one 262,144-token session and the client contract are + all unchanged; only the engine is new. + - **The runtime is built from source on the host.** gufo ships no prebuilt + Linux binary (the v0.1.0/v0.1.1 release assets are empty) and the host + has no Nix. Pinned at `v0.1.1` / `b0f8673`, installed to + `/home/dank/GitPackages/gufo-v0.1.1/` by the new + `/scripts/build-gufo.sh`, which is reproducible and fails closed on a + moved tag. Unlike the unsloth prebuilt it replaces, gufo **links the host + ROCm 7.2.4** through its RPATH, so the HIP dev packages must stay. + - **Two host workarounds, both encoded in the build script.** Ubuntu's + `librocwmma-dev` ships a truncated rocWMMA header set into + `/usr/include` that shadows AMD's complete one; the build died on + `'internal/accessors.hpp' file not found`, and a `-I` cannot fix it + because in HIP mode the host compilation pass searches the system + directories first. The package has no reverse dependencies; it is + removed. Second, Ubuntu 26.04 ships `/usr/include/c++/16` and the ROCm + clang prefers the newest tree, whose `` needs a + `std::__detail::__notify_impl` symbol version the installed + `libstdc++.so.6` (GLIBCXX_3.4.35) does not export — the link failed until + `--gcc-install-dir` was pinned to GCC 15. + - `gufo diagnose` reports `[PASS]` on `gfx1151` (ROCm 7.2.3, amdgpu + loaded, 20 CUs). It self-caps compute queues at 2 resident for + `serve llm` after reading `/sys/class/kfd/kfd/proc/*/queues/*/type`, + which matters because ComfyUI holds queues on the same device. + - **Shape:** one session, `--context 262144`, DFlash2 with the **adaptive** + block policy capped at `--draft-tokens 7` (llama.cpp's fixed depth 4 is + gone), greedy, `--think auto --preserve-thinking on`, text-only. + - **Client-contract deltas, all in the runbook:** the chat template is + compiled into the binary (no `--chat-template-file`, so the Qwen-Sharp + 73k-token-prompt failure mode cannot recur); `reasoning_effort` now + accepts the full OpenAI set, so `models.yml`'s `reasoningEffortMap` fold + is no longer required — **kept anyway** so this commit does not also move + every `high` role from `xhigh` to `high` effort; and there is no + KV-quantization or prompt-cache-RAM flag. + - **Measured at cutover:** 13 s from `load_started` to listening (target + 8 s, drafter 2 s, session 3 s) against a 20-minute `LLM_WAIT` budget; + RSS 30.6 GiB; a live short-prompt request returned + `Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (~41 tok/s). + Verified live: `/health` 200, `/ready` 200, `/v1/models` → + `frank-large` at `context_length 262144`, a chat completion carrying + `reasoning_content`, a `tool_calls` response with + `finish_reason: tool_calls`, and SSE streaming. + - **The trade, stated plainly.** gufo's own Q8_K_XL tables against + llama.cpp b11069: prefill +37 % to +80 %, cold load 4.5× faster, decode + +7 % at zero depth — but **−14 % at 65,536 tokens of depth and −24 % at + 131,072**. On the 8-bit target the crossover is ~32k and frank-large runs + one 262,144-token conversation, so long agent sessions sit on the losing + side of it. The Q4_K_XL target shows no such regression; revisiting the + quantization is a separate decision. + - `/tests/frank_large_unit.bats` — Rewritten around behaviour: the seam + runs the serve script against a stub engine and asserts on the argv it + actually execs, instead of grepping the script's text. A caveat found + while writing it: **bats 1.13 ignores a failing `A && B` in the middle of + a test** (bash exempts the non-final element of an AND-list from + `errexit`), so every such assertion was split into standalone `[ ]` lines. + Mutations confirmed the seam bites: dead runtime gate, `DRAFT_TOKENS=0`, + a wrong context/sessions/port, an alias swap, a moved drafter, a removed + missing-asset message and a neutered commit pin each fail it. + - `/scripts/deploy-remote.sh` — `deploy_llm_large` gates on the runtime + binary as well as the payload and returns **before** it stops the serving + unit, so a host whose gufo was never built reports the failure instead of + taking the endpoint down on the way there. + - Docs: new `/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`; + `frank-large-unsloth-qwen38-27b-q8xl.md` marked RETIRED (still the + rollback target — the prebuilt and the payload are both intact on disk); + AGENTS.md fleet table, frank-large section, bench policy and runbook + index; both vault indexes; `Dankclaw System Specs.md`; `README.md`. + - **e2e gate: 32/33.** The one failure is `192.168.0.112` (the FrankBook-Pro + Mac mini) at 100 % packet loss — offline, and the same failure recorded + in `564a781`. Not related to this change; no frank-large test failed. - **Frontier-MoE feasibility study on Dankserver** (research + build, no production change). Frank asked which of DeepSeek-V4.1-Flash, MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash would run best on the 2x R9700 box,