Files

T

Frank 968066ff23 docs: boot policy (only PP units enabled), deploy.sh dankserver hazard, stale-claim cleanup

- deployment.md: Boot Policy (2026-08-17) section (disabled units table,
  re-enable recipes, stale frank-restart-small.sh helper, broken
  deploy.sh dankserver target); Update Dankserver -> frank-pp-worker;
  condensed retired 35B history block and 2026-08-13 incident
- AGENTS.md: boot-policy note, target map fix (Q6_K, split 0.68),
  deploy.sh dankserver warning, obsolete quirk entries condensed
- README.md: rewrite active services/models tables (PP head + worker)
- index.md: topology fix (qwythos:9b, Q6_K split 0.68, NPU disabled)
- qwen3.8-spec-matrix.md: superseded configs merged into compact
  history; stale inline env block removed; bundled-head MTP note
- FastFlowLM, PP Deployment: boot-policy + enablement notes
- System Specs x2, DFlash doc, 35B FP8 doc, Ling docs: stale active
  claims fixed, retired banners, never-shipped plan stubbed

2026-08-17 03:30:00 -07:00

4.5 KiB

Raw Blame History

Ling-3.0-flash Q6_K Trial on Dankclaw

RETIRED 2026-08-17. The frank-large-ling unit is disabled under the boot policy. frank-default-large now runs the Qwen3.8-27B PP head, not the Qwen3.6-27B config described in the (obsolete) switch-back section below.

This page records the Ling-3.0-flash trial that temporarily replaced frank-default-large on Dankclaw. It replaces the Qwen3.6-27B service on port 7770 under the same API alias.

Status

Service: frank-large-ling (systemd).
Model: inclusionAI/Ling-3.0-flash Q6_K (MoE, ~118B total params).
Runtime: llama.cpp fork aetherbird/llama.cpp branch bailingmoe3-support (commit db4480bc8, build b10411) at /home/dank/llama.cpp-ling. Stock llama.cpp cannot load the bailingmoe3 architecture.
Backend: Vulkan0 (RADV) with AMD GTT.
Spec: draft-mtp n_max 3. The MTP block is bundled in the Ling GGUF itself (block 42) - no separate drafter file.
Context: 262144 total, 2 slots x 131072, batch 512 / ubatch 256.
Measured: load ~45s, generation ~7.7 t/s (cold, 4k prompt) to ~36 t/s (tiny cached prompt). Concurrent streams both complete.

Why Vulkan + GTT

The Strix Halo APU has 8 x 16 GB LPDDR5 modules = 128 GB physical shared memory. The firmware carves ~96 GiB as dedicated iGPU VRAM; the kernel sees the remainder (~30 GiB, MemTotal). GRUB adds no memory options (iommu=pt only).

The ROCm runtime reports 96 GiB VRAM. Vulkan (RADV) reports a 111.2 GiB heap (113910 MiB) because it can also reach system memory through the GTT. For a model near 100 GiB, Vulkan is the only backend with room. --list-devices shows the heap:

ROCm0:   AMD Radeon Graphics (98304 MiB)
Vulkan0: AMD Radeon Graphics (RADV STRIX_HALO) (113910 MiB)

Memory Fit (Q6_K = 98.26 GiB)

Item	Size
Vulkan heap (llama.cpp view)	~111.2 GiB
Model Q6_K file	98.26 GiB
Model + MTP + graphs in heap	~98.7 GiB
Free heap after load	~10 GiB

Ling uses gated MLA and KDA attention. KV cache stays tiny: 256k total context costs only ~2.5 GiB. This is why 256k context fits.

Freeze Bug (2026-08-14) - Spec Path Deadlock

Symptom

Under 2+ concurrent streams with speculative decoding, one stream hard-freezes mid-generation (zero output for minutes; a deadlock, not slowness). Freeze point varies (385-1588 tokens). Single streams never freeze.

Isolated cause

Not context exhaustion: truncated = 0, under 131072 ctx.
Not batch (3072 and 8192 both froze) and not log verbosity (0 and 4 both).
parallel=2 + draft-mtp,ngram-mod -> freeze. SPEC OFF -> no freeze.
Culprit: the speculative-decode / context-checkpoint path in llama-server with multiple slots on the Vulkan backend. Known bug family: PR #24891, #24797 (server checkpoint fixes), #25144 (MTP draft crash), plus a ggml-vulkan hang-under-sustained-compute class.

Resolution: SPEC OFF

SPEC_TYPE=none, PARALLEL=2. All concurrent streams complete (EOS). Speed cost: ~10-13 t/s under contention vs ~28-32 t/s single-stream with MTP. Fallback chain if spec-off regresses: A) PARALLEL=1 keep MTP, B) switch back to Qwen3.6-27B (frank-default-large, MTP n_max 3).

Files

scripts/frank-large-ling.service - systemd unit.
scripts/serve-ling-flash.sh - serve script (fork binary, Vulkan, draft-mtp).
scripts/frank-default-large.ling.env - tunables.

Passwordless management is installed: /usr/local/sbin/frank-restart-ling.sh and /etc/sudoers.d/frank-ling.

Switch Back (obsolete)

The switch-back commands below are obsolete. frank-default-large now runs the Qwen3.8-27B pipeline-parallel head. To restore a Ling trial, re-enable frank-large-ling from the repo unit (scripts/frank-large-ling.service) and serve script (scripts/serve-ling-flash.sh), then start it on a non-conflicting port.

Caveats

The bailingmoe3 fork validates CPU and CUDA upstream. Vulkan was validated here (load + generation + concurrent streams) on 2026-08-14.
The tokenizer logs special_eos_id is not in special_eog_ids at load. It is a warning; generation stops correctly at EOS.
Physical RAM pressure stays low because the model fits in the UMA carve-out. Generation speed on this APU is below the Qwen3.6-27B MTP config (~7.7 vs ~15-19 t/s at 4k).
The heap is effectively full (98.7 of 111.2 GiB). Do not raise the context much without checking --list-devices free MiB.

4.5 KiB Raw Blame History