Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
124 lines
4.8 KiB
Markdown
124 lines
4.8 KiB
Markdown
---
|
|
type: reference
|
|
description: "Trial of a fork build for a new mixture-of-experts architecture: what loaded, what the harness refused, and the verdict."
|
|
tags:
|
|
- llm
|
|
timestamp: 2026-10-19
|
|
---
|
|
|
|
# Ling-3.0-flash Q6_K Trial on Dankclaw
|
|
|
|
> **RETIRED 2026-08-17.** The `frank-dragoon-ling` unit is disabled under the boot policy. At this page's retirement date `frank-dragoon` ran the Qwen3.8-27B PP head (since superseded by the STRIX deployment — see [[frank-dragoon]]), not the Qwen3.6-27B config described in the (obsolete) switch-back section below.
|
|
|
|
## Status
|
|
|
|
- Service: `frank-dragoon-ling` (systemd).
|
|
- Model: `inclusionAI/Ling-3.0-flash` Q6_K (MoE, ~118B total params).
|
|
- Runtime: llama.cpp **fork** `aetherbird/llama.cpp` branch
|
|
`bailingmoe3-support` (commit `db4480bc8`, build b10411) at
|
|
`/home/dank/llama.cpp-ling`. Stock llama.cpp cannot load the
|
|
`bailingmoe3` architecture.
|
|
- Backend: `Vulkan0` (RADV) with AMD GTT.
|
|
- Spec: `draft-mtp` `n_max 3`. The MTP block is bundled in the Ling GGUF
|
|
itself (block 42) - no separate drafter file.
|
|
- Context: 262144 total, 2 slots x 131072, batch 512 / ubatch 256.
|
|
- Measured: load ~45s, generation ~7.7 t/s (cold, 4k prompt) to ~36 t/s
|
|
(tiny cached prompt). Concurrent streams both complete.
|
|
|
|
## Why Vulkan + GTT
|
|
|
|
The Strix Halo APU has 8 x 16 GB LPDDR5 modules = **128 GB physical
|
|
shared memory**. The firmware carves ~96 GiB as dedicated iGPU VRAM; the
|
|
kernel sees the remainder (~30 GiB, `MemTotal`). GRUB adds no memory
|
|
options (`iommu=pt` only).
|
|
|
|
The ROCm runtime reports 96 GiB VRAM. **Vulkan (RADV) reports a
|
|
111.2 GiB heap** (113910 MiB) because it can also reach system memory
|
|
through the GTT. For a model near 100 GiB, Vulkan is the only backend
|
|
with room. `--list-devices` shows the heap:
|
|
|
|
```
|
|
ROCm0: AMD Radeon Graphics (98304 MiB)
|
|
Vulkan0: AMD Radeon Graphics (RADV STRIX_HALO) (113910 MiB)
|
|
```
|
|
|
|
## Memory Fit (Q6_K = 98.26 GiB)
|
|
|
|
| Item | Size |
|
|
| ---- | ---- |
|
|
| Vulkan heap (llama.cpp view) | ~111.2 GiB |
|
|
| Model Q6_K file | 98.26 GiB |
|
|
| Model + MTP + graphs in heap | ~98.7 GiB |
|
|
| Free heap after load | ~10 GiB |
|
|
|
|
Ling uses gated MLA and KDA attention. KV cache stays tiny: 256k total
|
|
context costs only ~2.5 GiB. This is why 256k context fits.
|
|
|
|
## Freeze Bug (2026-08-14) - Spec Path Deadlock
|
|
|
|
### Symptom
|
|
|
|
Under 2+ concurrent streams with speculative decoding, one stream hard-freezes
|
|
mid-generation (zero output for minutes; a deadlock, not slowness). Freeze
|
|
point varies (385-1588 tokens). Single streams never freeze.
|
|
|
|
### Isolated cause
|
|
|
|
- Not context exhaustion: `truncated = 0`, under 131072 ctx.
|
|
- Not batch (3072 and 8192 both froze) and not log verbosity (0 and 4 both).
|
|
- `parallel=2` + `draft-mtp,ngram-mod` -> freeze. SPEC OFF -> no freeze.
|
|
- Culprit: the speculative-decode / context-checkpoint path in llama-server
|
|
with multiple slots on the Vulkan backend. Known bug family: PR #24891,
|
|
#24797 (server checkpoint fixes), #25144 (MTP draft crash), plus a
|
|
ggml-vulkan hang-under-sustained-compute class.
|
|
|
|
### Resolution: SPEC OFF
|
|
|
|
`SPEC_TYPE=none`, `PARALLEL=2`. All concurrent streams complete (EOS).
|
|
Speed cost: ~10-13 t/s under contention vs ~28-32 t/s single-stream with
|
|
MTP. Fallback chain if spec-off regresses: A) `PARALLEL=1` keep MTP,
|
|
B) switch back to Qwen3.6-27B (frank-dragoon, MTP n_max 3).
|
|
|
|
## Files
|
|
|
|
- `scripts/frank-dragoon-ling.service` - systemd unit.
|
|
- `scripts/serve-ling-flash.sh` - serve script (fork binary, Vulkan,
|
|
draft-mtp).
|
|
- `scripts/frank-dragoon.ling.env` - tunables.
|
|
|
|
Passwordless management is installed:
|
|
`/usr/local/sbin/frank-restart-ling.sh` and `/etc/sudoers.d/frank-ling`.
|
|
|
|
## Switch Back (obsolete)
|
|
|
|
The switch-back commands below are obsolete. At retirement time
|
|
`frank-dragoon` ran the Qwen3.8-27B pipeline-parallel head. To restore a Ling trial,
|
|
re-enable `frank-dragoon-ling` from the repo unit
|
|
(`scripts/frank-dragoon-ling.service`) and serve script
|
|
(`scripts/serve-ling-flash.sh`), then start it on a non-conflicting port.
|
|
|
|
## Caveats
|
|
|
|
- The `bailingmoe3` fork validates CPU and CUDA upstream. Vulkan was
|
|
validated here (load + generation + concurrent streams) on 2026-08-14.
|
|
- The tokenizer logs `special_eos_id is not in special_eog_ids` at load.
|
|
It is a warning; generation stops correctly at EOS.
|
|
- Physical RAM pressure stays low because the model fits in the UMA
|
|
carve-out. Generation speed on this APU is below the Qwen3.6-27B MTP
|
|
config (~7.7 vs ~15-19 t/s at 4k).
|
|
- The heap is effectively full (98.7 of 111.2 GiB). Do not raise the
|
|
context much without checking `--list-devices` free MiB.
|
|
|
|
## The other path: vLLM-ROCm (never shipped)
|
|
|
|
A deployment plan existed for `jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native`
|
|
(INT4 W4A16, ~62 GB) served by vLLM-ROCm on Dankclaw. It never shipped; the
|
|
llama.cpp Q6_K fork trial above is what actually ran. The decision table the
|
|
plan carried is in git history.
|
|
|
|
## See Also
|
|
|
|
- [[frank-dragoon]]
|
|
- [[DFlash DSpark Speculative Decoding]]
|
|
- [[deployment]]
|