Files

T

Frank 8212d4451a docs: rename frank-default-large/small/sightsound to frank-large/small/npu across vault

Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.

2026-09-03 16:17:14 -07:00

4.4 KiB

Raw Blame History

Ling-3.0-flash Q6_K Trial on Dankclaw

RETIRED 2026-08-17. The frank-large-ling unit is disabled under the boot policy. At this page's retirement date frank-large ran the Qwen3.8-27B PP head (since superseded by the STRIX deployment — see kingjones-strix-rocmfp4), not the Qwen3.6-27B config described in the (obsolete) switch-back section below.

Status

Service: frank-large-ling (systemd).
Model: inclusionAI/Ling-3.0-flash Q6_K (MoE, ~118B total params).
Runtime: llama.cpp fork aetherbird/llama.cpp branch bailingmoe3-support (commit db4480bc8, build b10411) at /home/dank/llama.cpp-ling. Stock llama.cpp cannot load the bailingmoe3 architecture.
Backend: Vulkan0 (RADV) with AMD GTT.
Spec: draft-mtp n_max 3. The MTP block is bundled in the Ling GGUF itself (block 42) - no separate drafter file.
Context: 262144 total, 2 slots x 131072, batch 512 / ubatch 256.
Measured: load ~45s, generation ~7.7 t/s (cold, 4k prompt) to ~36 t/s (tiny cached prompt). Concurrent streams both complete.

Why Vulkan + GTT

The Strix Halo APU has 8 x 16 GB LPDDR5 modules = 128 GB physical shared memory. The firmware carves ~96 GiB as dedicated iGPU VRAM; the kernel sees the remainder (~30 GiB, MemTotal). GRUB adds no memory options (iommu=pt only).

The ROCm runtime reports 96 GiB VRAM. Vulkan (RADV) reports a 111.2 GiB heap (113910 MiB) because it can also reach system memory through the GTT. For a model near 100 GiB, Vulkan is the only backend with room. --list-devices shows the heap:

ROCm0:   AMD Radeon Graphics (98304 MiB)
Vulkan0: AMD Radeon Graphics (RADV STRIX_HALO) (113910 MiB)

Memory Fit (Q6_K = 98.26 GiB)

Item	Size
Vulkan heap (llama.cpp view)	~111.2 GiB
Model Q6_K file	98.26 GiB
Model + MTP + graphs in heap	~98.7 GiB
Free heap after load	~10 GiB

Ling uses gated MLA and KDA attention. KV cache stays tiny: 256k total context costs only ~2.5 GiB. This is why 256k context fits.

Freeze Bug (2026-08-14) - Spec Path Deadlock

Symptom

Under 2+ concurrent streams with speculative decoding, one stream hard-freezes mid-generation (zero output for minutes; a deadlock, not slowness). Freeze point varies (385-1588 tokens). Single streams never freeze.

Isolated cause

Not context exhaustion: truncated = 0, under 131072 ctx.
Not batch (3072 and 8192 both froze) and not log verbosity (0 and 4 both).
parallel=2 + draft-mtp,ngram-mod -> freeze. SPEC OFF -> no freeze.
Culprit: the speculative-decode / context-checkpoint path in llama-server with multiple slots on the Vulkan backend. Known bug family: PR #24891, #24797 (server checkpoint fixes), #25144 (MTP draft crash), plus a ggml-vulkan hang-under-sustained-compute class.

Resolution: SPEC OFF

SPEC_TYPE=none, PARALLEL=2. All concurrent streams complete (EOS). Speed cost: ~10-13 t/s under contention vs ~28-32 t/s single-stream with MTP. Fallback chain if spec-off regresses: A) PARALLEL=1 keep MTP, B) switch back to Qwen3.6-27B (frank-large, MTP n_max 3).

Files

scripts/frank-large-ling.service - systemd unit.
scripts/serve-ling-flash.sh - serve script (fork binary, Vulkan, draft-mtp).
scripts/frank-large.ling.env - tunables.

Passwordless management is installed: /usr/local/sbin/frank-restart-ling.sh and /etc/sudoers.d/frank-ling.

Switch Back (obsolete)

The switch-back commands below are obsolete. At retirement time frank-large ran the Qwen3.8-27B pipeline-parallel head. To restore a Ling trial, re-enable frank-large-ling from the repo unit (scripts/frank-large-ling.service) and serve script (scripts/serve-ling-flash.sh), then start it on a non-conflicting port.

Caveats

The bailingmoe3 fork validates CPU and CUDA upstream. Vulkan was validated here (load + generation + concurrent streams) on 2026-08-14.
The tokenizer logs special_eos_id is not in special_eog_ids at load. It is a warning; generation stops correctly at EOS.
Physical RAM pressure stays low because the model fits in the UMA carve-out. Generation speed on this APU is below the Qwen3.6-27B MTP config (~7.7 vs ~15-19 t/s at 4k).
The heap is effectively full (98.7 of 111.2 GiB). Do not raise the context much without checking --list-devices free MiB.

4.4 KiB Raw Blame History