docs(llm): record the gufo cutover verification
Live numbers from the 2026-09-28 deploy: 13 s load to listening, RSS 30.6 GiB, ttft 549.8 ms / inter-token 24.2 ms on a short prompt, and a green contract battery (health, ready, /v1/models at 262,144, content + reasoning_content, tool_calls, SSE, and a two-turn tool loop). Also recorded: gufo accepts every request field the harness sends in extraBody and the full reasoning_effort set, both verified 200 — which is why models.yml's reasoningEffortMap fold is now optional rather than required, and why it is being kept anyway.
This commit is contained in:
Frank
@@ -108,6 +108,24 @@ architecture, driver and queue budget.
|
||||
- **Memory:** gufo's own table puts the 8-bit target at **50.58 GiB** peak
|
||||
with context capacity 262,144, against the iGPU's ~114 GiB GTT aperture.
|
||||
|
||||
### Measured at the 2026-09-28 cutover
|
||||
|
||||
Live on Dankclaw after `deploy.sh dankclaw --service frank-large`:
|
||||
|
||||
- **13 s** from `load_started` to `listening` (target 8 s, drafter 2 s,
|
||||
session 3 s), against a 20-minute `LLM_WAIT` budget — a redeploy no longer
|
||||
sits in a load wait.
|
||||
- **RSS 30.6 GiB**, VSZ 87.3 GiB; host `used` 81 GiB of 121 GiB, i.e. the
|
||||
same order as the llama.cpp deployment it replaced.
|
||||
- A short-prompt live request returned
|
||||
`Server-Timing: ttft;dur=549.8, inter_token;dur=24.2, max_inter_token;dur=166.9`
|
||||
(≈41 tok/s). Short-prompt decode is gufo's *winning* regime; see the depth
|
||||
table above before quoting it as the endpoint's number.
|
||||
- Contract battery, all green: `/health` 200, `/ready` 200,
|
||||
`/v1/models` → `frank-large` with `context_length 262144`, a chat
|
||||
completion carrying both `content` and `reasoning_content`, a `tool_calls`
|
||||
response with `finish_reason: tool_calls`, and SSE streaming.
|
||||
|
||||
## Client contract — what actually changed
|
||||
|
||||
Everything the fleet depended on still holds: port 7770, alias `frank-large`,
|
||||
|
||||
@@ -2,6 +2,80 @@
|
||||
|
||||
|
||||
## 2026-09-28
|
||||
- **frank-large runtime swapped to gufo** (production change, Dankclaw).
|
||||
Frank asked for `github.com/gufo-org/gufo` — a vertical inference engine
|
||||
built for exactly this hardware (Ryzen AI MAX+ 395 / Radeon 8060S,
|
||||
`gfx1151`) — in place of the stock llama.cpp prebuilt. Payload, port 7770,
|
||||
alias `frank-large`, one 262,144-token session and the client contract are
|
||||
all unchanged; only the engine is new.
|
||||
- **The runtime is built from source on the host.** gufo ships no prebuilt
|
||||
Linux binary (the v0.1.0/v0.1.1 release assets are empty) and the host
|
||||
has no Nix. Pinned at `v0.1.1` / `b0f8673`, installed to
|
||||
`/home/dank/GitPackages/gufo-v0.1.1/` by the new
|
||||
`/scripts/build-gufo.sh`, which is reproducible and fails closed on a
|
||||
moved tag. Unlike the unsloth prebuilt it replaces, gufo **links the host
|
||||
ROCm 7.2.4** through its RPATH, so the HIP dev packages must stay.
|
||||
- **Two host workarounds, both encoded in the build script.** Ubuntu's
|
||||
`librocwmma-dev` ships a truncated rocWMMA header set into
|
||||
`/usr/include` that shadows AMD's complete one; the build died on
|
||||
`'internal/accessors.hpp' file not found`, and a `-I` cannot fix it
|
||||
because in HIP mode the host compilation pass searches the system
|
||||
directories first. The package has no reverse dependencies; it is
|
||||
removed. Second, Ubuntu 26.04 ships `/usr/include/c++/16` and the ROCm
|
||||
clang prefers the newest tree, whose `<atomic_wait.h>` needs a
|
||||
`std::__detail::__notify_impl` symbol version the installed
|
||||
`libstdc++.so.6` (GLIBCXX_3.4.35) does not export — the link failed until
|
||||
`--gcc-install-dir` was pinned to GCC 15.
|
||||
- `gufo diagnose` reports `[PASS]` on `gfx1151` (ROCm 7.2.3, amdgpu
|
||||
loaded, 20 CUs). It self-caps compute queues at 2 resident for
|
||||
`serve llm` after reading `/sys/class/kfd/kfd/proc/*/queues/*/type`,
|
||||
which matters because ComfyUI holds queues on the same device.
|
||||
- **Shape:** one session, `--context 262144`, DFlash2 with the **adaptive**
|
||||
block policy capped at `--draft-tokens 7` (llama.cpp's fixed depth 4 is
|
||||
gone), greedy, `--think auto --preserve-thinking on`, text-only.
|
||||
- **Client-contract deltas, all in the runbook:** the chat template is
|
||||
compiled into the binary (no `--chat-template-file`, so the Qwen-Sharp
|
||||
73k-token-prompt failure mode cannot recur); `reasoning_effort` now
|
||||
accepts the full OpenAI set, so `models.yml`'s `reasoningEffortMap` fold
|
||||
is no longer required — **kept anyway** so this commit does not also move
|
||||
every `high` role from `xhigh` to `high` effort; and there is no
|
||||
KV-quantization or prompt-cache-RAM flag.
|
||||
- **Measured at cutover:** 13 s from `load_started` to listening (target
|
||||
8 s, drafter 2 s, session 3 s) against a 20-minute `LLM_WAIT` budget;
|
||||
RSS 30.6 GiB; a live short-prompt request returned
|
||||
`Server-Timing: ttft;dur=549.8, inter_token;dur=24.2` (~41 tok/s).
|
||||
Verified live: `/health` 200, `/ready` 200, `/v1/models` →
|
||||
`frank-large` at `context_length 262144`, a chat completion carrying
|
||||
`reasoning_content`, a `tool_calls` response with
|
||||
`finish_reason: tool_calls`, and SSE streaming.
|
||||
- **The trade, stated plainly.** gufo's own Q8_K_XL tables against
|
||||
llama.cpp b11069: prefill +37 % to +80 %, cold load 4.5× faster, decode
|
||||
+7 % at zero depth — but **−14 % at 65,536 tokens of depth and −24 % at
|
||||
131,072**. On the 8-bit target the crossover is ~32k and frank-large runs
|
||||
one 262,144-token conversation, so long agent sessions sit on the losing
|
||||
side of it. The Q4_K_XL target shows no such regression; revisiting the
|
||||
quantization is a separate decision.
|
||||
- `/tests/frank_large_unit.bats` — Rewritten around behaviour: the seam
|
||||
runs the serve script against a stub engine and asserts on the argv it
|
||||
actually execs, instead of grepping the script's text. A caveat found
|
||||
while writing it: **bats 1.13 ignores a failing `A && B` in the middle of
|
||||
a test** (bash exempts the non-final element of an AND-list from
|
||||
`errexit`), so every such assertion was split into standalone `[ ]` lines.
|
||||
Mutations confirmed the seam bites: dead runtime gate, `DRAFT_TOKENS=0`,
|
||||
a wrong context/sessions/port, an alias swap, a moved drafter, a removed
|
||||
missing-asset message and a neutered commit pin each fail it.
|
||||
- `/scripts/deploy-remote.sh` — `deploy_llm_large` gates on the runtime
|
||||
binary as well as the payload and returns **before** it stops the serving
|
||||
unit, so a host whose gufo was never built reports the failure instead of
|
||||
taking the endpoint down on the way there.
|
||||
- Docs: new `/serverAppConfigs-docs/LLM/frank-large-gufo-qwen38-27b.md`;
|
||||
`frank-large-unsloth-qwen38-27b-q8xl.md` marked RETIRED (still the
|
||||
rollback target — the prebuilt and the payload are both intact on disk);
|
||||
AGENTS.md fleet table, frank-large section, bench policy and runbook
|
||||
index; both vault indexes; `Dankclaw System Specs.md`; `README.md`.
|
||||
- **e2e gate: 32/33.** The one failure is `192.168.0.112` (the FrankBook-Pro
|
||||
Mac mini) at 100 % packet loss — offline, and the same failure recorded
|
||||
in `564a781`. Not related to this change; no frank-large test failed.
|
||||
- **Frontier-MoE feasibility study on Dankserver** (research + build, no
|
||||
production change). Frank asked which of DeepSeek-V4.1-Flash,
|
||||
MiMo-V2.6-Flash-MOPD and GLM-5.3-Flash would run best on the 2x R9700 box,
|
||||
|
||||
Reference in New Issue
Block a user