docs: frank-large flash-server v2 cutover — runbook, index, AGENTS.md
- New runbook: frank-large-flash-server-v2-swift15.md (engine, shape, memory model, engine laws, boot ledger, probes, rollback) - LLM/index.md: active entry + retired halogen 27B entry - Vault index: link the new runbook from the LLM Fleet section - AGENTS.md: fleet topology table, Model Config section, Runbook Index, retired table (halogen 27B → 10-06)
This commit is contained in:
Frank
@@ -12,8 +12,9 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
Calibre-Web, Jellyfin, Plex) plus the `frank-primary` LLM service.
|
||||
- **Dankclaw** (192.168.0.143) — AI/search/dev/observability stack (SearXNG
|
||||
via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus, Loki,
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (halogen container on
|
||||
the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the RTX 3090). Caddy on
|
||||
Alloy, Pi-hole) plus TWO LLM services: `frank-large` (flash-server
|
||||
container on the gfx1151 iGPU) and `frank-small` (llama.cpp CUDA on the
|
||||
RTX 3090). Caddy on
|
||||
Dankclaw reverse-proxies both hosts.
|
||||
- **FrankBook-Pro** (192.168.0.112) — macOS mini running the *arr stack
|
||||
(Prowlarr, Sonarr, Radarr, Lidarr, Readarr, qBittorrent) as quadlets inside
|
||||
@@ -25,7 +26,7 @@ merge absorbed the former `vllmScripts` repo, history preserved).
|
||||
| Target | Service | Runtime | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | `frank-primary` | **radiance** (StillDeadcode v1.0.7 line; image `stilldeadcode/radiance@sha256:4c3f484564…` digest-pinned INDEX digest, 287 MiB, `libr4d` gfx1201 kernels, plain `podman run`, root systemd unit) | `Qwen/Qwen3.8-Flash-Next-FP8` official (172.8 GB, 131 shards, sha256-verified) converted **no-recipe** to `qwen4exp-fp8-native.rad` (177,109 MiB measured = 173.0 GiB) — **native modelopt block-FP8, no requantisation** — on **BOTH R9700s TP=2** (`--tp-wire wht6`), experts in a **40 GiB/rank pinned host pool** + VRAM cache (`--placement expert_tiered`, `--expert-vs-cache-ratio 0.82` = the 99 %-hit point), **PLE 47.7 GiB pinned to host RAM** (`--ngram-placement ram`; root unit + `LimitMEMLOCK=infinity`), **fp8 KV 7.32 KiB/token**, **MTP spec decode depth 3**, *…
|
||||
| Dankclaw | `frank-large` | **halogen container** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | `peonist-ai/halogen-qwen3.8-27b` **`qwen3.8-27b-p1w4d-d2.hgn`** (35,865,565,184 B; the `.hgn` format loads in NO other engine — llama.cpp, vLLM and transformers all refuse it), **DFlash2 spec decode (`HALOGEN_DRAFTER=2`)**, **1 slot × 200,700 ctx** — `HALOGEN_KV_SLOTS=1` is load-bearing: batching XOR speculation, slots>1 silently disables the drafter (measured: 8 slots at serial ~6 t/s/stream), d2 KV (2× fp8, 64 KiB/token → 12.2 GiB pool, engine sizes it and prints a boot ledger naming any clamp), **TEXT-ONLY** (the engine ignores the vision encoder — vision is frank-small's job), **censored base** (the ROCmFPX AGENT rollback is the refusal escape hatch), **no server-side sampler: absent temperature is GREEDY**, so the card's thinking recipe (temp 1.0 / top_p 0.95) rides the wire in `models.yml`. Publisher card 31.7 t/s mean across eight text workloads vs the retired ROCmFPX endpoint's measured 27.6–28.1. Prompt over the slot AND `max_tokens` over `HALOGEN_MAX_TOKENS_CAP` are both 400s naming the limit, never truncations. Serve passes NO `--ulimit nofile` (rootless runc cannot raise Dankclaw's 524288 hard limit — see Key Lessons). Cutover 2026-10-05|7770|
|
||||
| Dankclaw | `frank-large` | **halogen-flash-server 0.16.2** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`, digest-pinned, plain `podman run` via `scripts/serve-frank-large.sh`, root systemd unit) | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated **`qwen38-flash-next-v2-swift15-abliterated.hgn`** (66,687,678,464 B = compacted r2 of the card's 89,775,868,032 B original, ~62 GiB resident; the v2 `.hgn` format loads in NO other engine), **+ REQUIRED `qwen38-flash-next-ngram.hgn`** (51,200,246,144 B, peonist @5cc17cea — read beside the checkpoint) **+ vision tower** (897,916,416 B, @e053f488 — **vision ON**, `image_url` parts only, ≥256 px probes), **MTP spec decode ALWAYS-ON (`HALOGEN_SPEC_ADAPT=0` — the adaptive default disables the head per request and HALVES decode)**, MTP head INSIDE the checkpoint (`HALOGEN_MTP_HEAD` unset; no sidecar fetched), **2 slots × 200,192 ctx** (KV pool 400,384 positions ≈ 10.7 GiB at ~28 KiB/position; the 27B slots>1-disables-drafting law does NOT carry over — batching and speculation coexist, the card's quickstart ships 2 with the head), **server-side sampler 1.0/0.95/20** (card recipe; models.yml extraBody matches), coupled cap/timeout 65536/7200, prompt cache ON by default (warm 100k turn ≈ 2 s vs ≈ 88 s cold) | 7770 |
|
||||
| Dankclaw | `frank-small` | llama.cpp master (0.5.0-dev, commit `5fc4f3c`), CUDA sm_86 build (root systemd unit, relinked) | ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF **RCO-IQ3_XXS -mtp** (27B dense qwen3_5 hybrid, 10,442,827,392 B; a per-tensor allocation mix of stock types; the MTP head in the file **drafted** — **spec ON** (draft-mtp depth 3) since 2026-10-06 evening, ngram skipped on the recurrent-rollback law) + the publisher's own **BF16 mmproj** for **vision**, on the RTX 3090, **2 × 131,072** (Frank's 128k cap each; the tightest fit in the endpoint's history — **489 MiB measured free**, the 2-3 GiB floor knowingly violated) with **q8_0 K + q8_0 V (FP8 class, 34 KiB/token)** under flash attention, batch 4096 / ubatch 512; **63.3 t/s recipe decode** (MTP acceptance 49.8 %; 48.0 no-spec). Supersedes the same-morning IQ3_S-mtp payloads: the two-slot no-spec shape and the one-slot MTP shape (both 2026-10-06; payload on disk).|7771|
|
||||
|
||||
Retired endpoints (`frank-npu`, and every prior runtime) are in
|
||||
@@ -234,7 +235,10 @@ Scratch ports are not used on LLM hosts — a
|
||||
test process cannot share the single GPU anyway; validate against the service
|
||||
itself.
|
||||
|
||||
(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05 and is the halogen container since, but the rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode settings, and will invert the ranking.**
|
||||
(A gufo-era law; frank-large ran ROCmFPX llama.cpp 2026-10-02 → 10-05, the
|
||||
halogen 27B container 2026-10-05 → 10-06, and is flash-server since, but the
|
||||
rollback tags keep it live.) **`gufo bench` cannot rank speculative-decode
|
||||
settings, and will invert the ranking.**
|
||||
Measured 2026-09-28 on frank-large at depth 32768: its corpus
|
||||
reports `acceptance=0.933 ± 0.000` with an **identical trace sha256** across
|
||||
draft widths, i.e. the same highly-predictable text every run, and 93 % of
|
||||
@@ -296,9 +300,10 @@ When renaming a service, retiring an endpoint, or moving a model: update the
|
||||
provider block, model `id`, `contextWindow`, `maxTokens`, `input` in
|
||||
`models.yml`, then fix every `modelRoles` entry that named the old endpoint —
|
||||
a role pointing at a provider the file does not define fails at call time. As
|
||||
of 2026-10-05 **two of the three endpoints accept images** (`input: [text,
|
||||
image]`) — `frank-large` went TEXT-ONLY with the halogen swap (the engine
|
||||
ignores the vision encoder; the BF16-mmproj era ended 2026-10-05) — while
|
||||
of 2026-10-06 **all three endpoints accept images** (`input: [text,
|
||||
image]`) — `frank-large` returned to vision with the flash-server v2 swap
|
||||
(the v2 vision tower loads beside the checkpoint; the text-only halogen
|
||||
27B window closed 2026-10-06) — while
|
||||
`frank-primary` takes them natively (radiance runs the checkpoint's
|
||||
27-layer Qwen-VL tower — patching, pad-token expansion and 3-part rotary
|
||||
are the engine core; `--mm-max-patches` available),
|
||||
@@ -318,82 +323,93 @@ endpoint on 2026-09-11, returned 2026-09-17 with the Gemma4 projector's
|
||||
audio encoder, and left again with the Gemma4 retirement on 2026-10-01
|
||||
(the MiMo tower is vision-only).
|
||||
|
||||
### frank-large — halogen container, Qwen3.8-27B + DFlash2 (Dankclaw, ACTIVE)
|
||||
### frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, ACTIVE)
|
||||
|
||||
The **halogen engine** (`ghcr.io/peonist-ai/halogen@sha256:dc0a39a0…`,
|
||||
digest-pinned, published tag 0.1.4) as a podman container driven by
|
||||
`scripts/serve-frank-large.sh` under a **root systemd unit**
|
||||
The **flash-server engine** (`ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84…`,
|
||||
digest-pinned, published tag 0.16.2, 3.6 GB image) as a podman container
|
||||
driven by `scripts/serve-frank-large.sh` under a **root systemd unit**
|
||||
(`scripts/frank-large.service`, `User=dank`, `Restart=always`,
|
||||
`ExecStop=podman
|
||||
stop -t 120 frank-large-halogen`, the script runs `podman run` FOREGROUND —
|
||||
the container's exit code is the service's exit so `Restart=always` stays
|
||||
real, and the engine ledger lands on disk) on port 7770 from the Strix Halo
|
||||
iGPU. Engine stdout goes to `/home/dank/logs/frank_large.log` — **the boot
|
||||
ledger (KV pool, prompt cache, drafter state, any clamp) lives there, not in
|
||||
the journal** (the 2026-10-05 `-d` + `podman wait` shape captured only the
|
||||
container ID — fixed 2026-10-06).
|
||||
Serving
|
||||
`peonist-ai/halogen-qwen3.8-27b`: `qwen3.8-27b-p1w4d-d2.hgn` (35,865,565,184 B
|
||||
≈ 33.4 GiB, sha256 `274c3fc7…`, 1352 tensors; p1 weights / w4 activations /
|
||||
d2 KV) + the flat `tokenizer/` directory (5 files). Cutover 2026-10-05 from the ROCmFPX llama.cpp fork
|
||||
([[frank-large-rocmfpx-q6agent-uncensored-mtp]], measured 27.6–28.1 t/s prose),
|
||||
chosen for the publisher's 31.7 t/s DFlash2 mean.
|
||||
`ExecStop=podman stop -t 60 frank-large-halogen`, the script runs `podman
|
||||
run` FOREGROUND — the container's exit code is the service's exit so
|
||||
`Restart=always` stays real, and the engine ledger lands on disk) on port
|
||||
7770 from the Strix Halo iGPU. Engine stdout goes to
|
||||
`/home/dank/logs/frank_large.log` — **the boot ledger (weights resident, KV
|
||||
pool, MTP head, prompt cache, any clamp) lives there, not in the journal**.
|
||||
Cutover 2026-10-06 from the halogen 27B-dense era
|
||||
([[frank-large-halogen27b-dflash2]], retired the same window).
|
||||
|
||||
**The format is the lock.** `.hgn` loads in nothing else — the runtime is
|
||||
chosen by the FILE, as ROCmFPX's tensor types were before it. This is NOT the
|
||||
2026-09 `halogen-flash-server` image (different repo, Flash-Next w4b era); the
|
||||
halogen-server repo restarted versioning at 0.1.x.
|
||||
Serving `Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`:
|
||||
`qwen38-flash-next-v2-swift15-abliterated.hgn` (66,687,678,464 B ≈ 62.1
|
||||
GiB, sha256 `9c880b8b…` — a compacted r2 of the card's 89,775,868,032 B
|
||||
append-only original; the abliteration workflow appends and the repo's
|
||||
tooling drops dead padding, live tensors byte-identical) + the **REQUIRED**
|
||||
n-gram lookup table `qwen38-flash-next-ngram.hgn` (51,200,246,144 B,
|
||||
peonist @5cc17cea) + the vision tower `qwen38-flash-next-vision.hgn`
|
||||
(897,916,416 B, @e053f488) + the flat `tokenizer/` directory (6 files,
|
||||
fetched fresh from peonist). Cutover 2026-10-06 from the halogen
|
||||
27B-dense shape, chosen by Frank for the ABLITERATED base.
|
||||
|
||||
**Two product regressions, both accepted by Frank before deploy:** the
|
||||
endpoint is **text-only** (README honest limits: the vision encoder is
|
||||
ignored — frank-small owns images now) and the checkpoint is a finetune of the
|
||||
**censored** `Qwen/Qwen3.8-27B` base (refusals can return; the rollback is one
|
||||
commit-revert away with the AGENT payload still on disk).
|
||||
**The format is the lock.** The v2 `.hgn` loads ONLY in the
|
||||
flash-server line — the 27B-era halogen 0.1.4 container refuses it, so the
|
||||
engine moved with the model. The MTP head is INSIDE the checkpoint
|
||||
(`HALOGEN_MTP_HEAD` stays unset; the separate `qwen38-flash-next-mtp.hgn`
|
||||
sidecar is deliberately not fetched).
|
||||
|
||||
**Engine laws that shape the config** (all in `scripts/frank-large.env` with
|
||||
evidence): `HALOGEN_KV_SLOTS=1` — batch XOR speculation, slots>1 turns the
|
||||
drafter off silently. `HALOGEN_SLOT_CTX=200700` — the per-conversation HARD
|
||||
ceiling (400 naming the limit, never truncation); `models.yml`
|
||||
`contextWindow: 200700` must match or the harness discovers the wall as a 400.
|
||||
`HALOGEN_MAX_TOKENS_CAP=65536` + `HALOGEN_QUEUE_TIMEOUT=7200` are a coupled
|
||||
pair; the harness `maxTokens: 57344` stays under the cap (over-cap is a 400
|
||||
naming the limit — README, never a 413). Thinking tokens COUNT against the
|
||||
budget: exhausted mid-thought is `finish_reason: "length"` with empty
|
||||
`content` and the reasoning in `reasoning_content`.
|
||||
`HALOGEN_DRAFTER=2` = DFlash2. **No server-side sampler**: an absent
|
||||
temperature is greedy, so the thinking recipe exists only in `models.yml`
|
||||
`extraBody`. `response_format` is unsupported (no constrained decoding).
|
||||
Thinking lands in **`reasoning_content`** (harness field unchanged); the same
|
||||
Qwen3.8 template rejects `reasoning_effort` outside `xhigh|medium|low`, so the
|
||||
`models.yml` fold stays mandatory.
|
||||
`HALOGEN_CACHE_MB=16384` pins the prompt cache (2026-10-06): auto is OFF
|
||||
on this box — the floor is 2 full-context entries (36.9 GB) and the boot
|
||||
ledger had 35.7 GB free — so every deep request re-prefilled its whole
|
||||
79-81k context (84.9-179.8 s per request for ~500-650 new tokens per
|
||||
turn) until the pin. Probe-verified: a 17.3k cold prefill of 27.0 s
|
||||
resumes in 1.7 s on the next turn (cache_n 16,384). Decode-at-depth is
|
||||
structural: measured 5.5 t/s at 46k → 3.3-3.8 at 80k vs 30-50 shallow
|
||||
(the KV pool reads at ~8 GB/s effective vs ~180-240 for the weights);
|
||||
`docs/FLAGS.md` documents no decode-affecting lever and 0.1.4 is the
|
||||
latest engine. The boot WARNING "smaller than ONE full-context entry
|
||||
(18.4 GB at kMaxCtx=262144)" is a false positive for our slot: the
|
||||
deepest possible entry is 200,700 × ~70 KB ≈ 14 GB, inside the pin, and
|
||||
the 262k entry it fears cannot exist (over-slot is a 400).
|
||||
**Product facts vs the 27B era, both regressions FIXED:** the base is
|
||||
**abliterated** (the 27B was a censored-base finetune; refusals were an
|
||||
accepted trade there) and **vision is ON** (the 27B engine silently
|
||||
ignored its encoder — frank-small owned images for exactly one day at this
|
||||
endpoint).
|
||||
|
||||
Fetch: `scripts/download-frank-large-halogen.sh` into
|
||||
`/home/dank/models/halogen-qwen3.8-27b/` (`-f` + byte count + LFS sha256 +
|
||||
`--verify-only`, no token on the command line; resume goes against the
|
||||
RESOLVED CDN URL because curl drops `Range` across HF's 302). The deployer
|
||||
and serve script both refuse a missing tokenizer file or **any symlink** —
|
||||
an HF cache snapshot symlinks each entry into `blobs/` and those links
|
||||
dangle inside the container mount.
|
||||
**Engine laws that shape the config** (all in `scripts/frank-large.env`
|
||||
with evidence): `HALOGEN_SPEC_ADAPT=0` is MANDATORY — the default adaptive
|
||||
policy disables the MTP head for whole requests after low acceptance and
|
||||
HALVES decode; the head stays on and temp-1.0 traffic pays some acceptance
|
||||
for it (accepted). `HALOGEN_CTX=200192` is the PER-SLOT hard ceiling (400
|
||||
naming the limit, never truncation); `models.yml` `contextWindow: 200192`
|
||||
must match, and compaction at 70 % = 140,134 + 57,344 budget = 197,478 fits.
|
||||
`HALOGEN_KV_SLOTS=2` is LEGAL here — the 27B "slots>1 disables the drafter"
|
||||
law does NOT carry over; batching and speculation coexist (the card's
|
||||
quickstart ships 2 slots with the head), so 2 is concurrent capacity.
|
||||
`HALOGEN_KV_POOL_POSITIONS=400384` (explicit 2 × ctx; ~28 KiB/position ⇒
|
||||
~10.7 GiB). `HALOGEN_MAX_TOK=8192` is only the per-request DEFAULT (the
|
||||
harness always sends 57,344); `HALOGEN_MAX_TOKENS_CAP=65536` +
|
||||
`HALOGEN_QUEUE_TIMEOUT=7200` are a coupled pair (FLAGS.md table).
|
||||
**A server-side sampler EXISTS** (`HALOGEN_TEMPERATURE/TOP_P/TOP_K` =
|
||||
1.0/0.95/20, the card's thinking recipe) — an absent temperature is no
|
||||
longer greedy; `models.yml` extraBody carries the same values, consistent
|
||||
not load-bearing. Prompt cache is ON by default (`PROMPT_CACHE=2`): a warm
|
||||
100k follow-up costs ~2 s vs ~88 s cold — the 27B engine's manual
|
||||
`HALOGEN_CACHE_MB` pin era is over. `response_format` is unsupported.
|
||||
Thinking lands in **`reasoning_content`**; the v2 chat template rejects
|
||||
`reasoning_effort` outside `xhigh|medium|low`, so the `models.yml` fold
|
||||
stays mandatory. Thinking tokens COUNT against the budget (empty content +
|
||||
`finish_reason: "length"` when exhausted mid-thought). Vision: OpenAI
|
||||
`image_url` content parts only; ≥256 px probe images or the model answers
|
||||
"video".
|
||||
|
||||
Runbook: `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md`.
|
||||
Rollback: revert the swap commit and redeploy — the ROCmFPX payload
|
||||
(`/home/dank/models/kingjones-Qwen3.8-27B-ROCmFP4-STRIX-MTP/`), the fork build
|
||||
and `scripts/build-rocmfpx.sh` (pin `36e9acd`) all remain on disk, seam-tested.
|
||||
The older gufo tag `frank-large-gufo-q4kxl-20261001` stays a **re-download**
|
||||
rollback (payloads removed 2026-10-03; `scripts/build-gufo.sh` stays).
|
||||
**Memory ledger** (121.2 GiB unified; the referee is the boot ledger):
|
||||
weights ~62 + KV ~10.7 + arena ~8 (`3.8 + 8.4 × MAX_TOK/16384`) + vision
|
||||
~2 ≈ 82.7 GiB engine-side + `HALOGEN_HOST_RESERVE_GIB=20` → ~18 GiB
|
||||
margin over baseline co-tenancy. The 51.2 GB n-gram table rides as mmap'd
|
||||
page cache beside the checkpoint — the card's quickstart arithmetic does
|
||||
not bill it as resident, but READ the boot ledger and re-size if the
|
||||
engine prints a clamp.
|
||||
|
||||
Fetch: `scripts/download-frank-large-flashnext.sh` into
|
||||
`/home/dank/models/halogen-flashnext-v2/` (`-f` + exact byte counts + LFS
|
||||
sha256 + `--verify-only`, no token on the command line; resume goes
|
||||
against the RESOLVED CDN URL because curl drops `Range` across HF's 302).
|
||||
The serve gates mirror the deployer EXACTLY (byte-exact checkpoint +
|
||||
n-gram + vision tower, flat tokenizer with all six files as regular
|
||||
files) — the 2026-10-01 crash-loop lesson.
|
||||
|
||||
Runbook:
|
||||
`serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md`.
|
||||
Rollback: revert the swap commit and redeploy — the 27B payload
|
||||
(`/home/dank/models/halogen-qwen3.8-27b/`), its fetch script and the
|
||||
halogen 0.1.4 image digest all remain on disk; the ROCmFPX door behind
|
||||
that (`scripts/build-rocmfpx.sh`, pin `36e9acd`) is untouched.
|
||||
|
||||
### frank-small — llama.cpp on the RTX 3090 (Dankclaw, ACTIVE)
|
||||
|
||||
@@ -997,7 +1013,7 @@ Active:
|
||||
|
||||
| Service | Runbook |
|
||||
|---------|---------|
|
||||
| frank-large (halogen container, Qwen3.8-27B .hgn + DFlash2, 1×200,700, text-only, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-halogen27b-dflash2.md` |
|
||||
| frank-large (flash-server 0.16.2, abliterated Swift1.5 v2 .hgn + MTP always-on, 2×200,192, vision ON, Dankclaw) | `serverAppConfigs-docs/LLM/frank-large-flash-server-v2-swift15.md` |
|
||||
| frank-small (llama.cpp CUDA, Swift-1.5-27B Uncensored RCO IQ3_XXS-mtp, 2×131,072 draft-mtp q8_0 KV, Dankclaw) | `serverAppConfigs-docs/LLM/frank-small-llama-swift15-unc-iq3xxs-mtp-131072.md` |
|
||||
| frank-primary (radiance StillDeadcode, Qwen3.8-Flash-Next-FP8 native, 2×R9700 TP=2, Dankserver) | `serverAppConfigs-docs/LLM/frank-primary-radiance-qwen38-native-fp8.md` |
|
||||
| Dankclaw kernel posture (`amd_iommu=off`) | `serverAppConfigs-docs/LLM/dankclaw-iommu-translation-off.md` |
|
||||
@@ -1007,6 +1023,7 @@ Retired (rollback targets and history):
|
||||
|
||||
| Runtime | Host | Window | Runbook |
|
||||
|---------|------|--------|---------|
|
||||
| halogen 0.1.4 Qwen3.8-27B p1w4d-d2 + DFlash2 (`frank-large`) | Dankclaw | 2026-10-05 → 10-06 (rollback target — payload + fetch script + image digest on disk; owns the prompt-cache pin and FA-tile lessons) | `frank-large-halogen27b-dflash2.md` |
|
||||
| ROCmFPX llama.cpp fork Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP (`frank-large`) | Dankclaw | 2026-10-03 → 10-05 (rollback target, payload + fork build on disk) | `frank-large-rocmfpx-q6agent-uncensored-mtp.md` |
|
||||
| tcclaviger vLLM fork Qwen3.8-Flash-Next-Uncensored FP8, TP=2, shape G (`frank-primary`) | Dankserver | 2026-09-29 → 10-05 (rollback target — tag `frank-primary-vllm-shapeG-20261003`; trio + env lore in the tag, image in rootless storage) | `dankserver-qwen38-fp8-expert-offload-tuning.md` |
|
||||
| FreeToken PR #587 ROCm, Qwen3.8-Flash-Next-Uncensored-FP8, ONE R9700 (`frank-primary`) | Dankserver | 2026-10-05 → 10-05 (rollback target — tag `frank-primary-freetoken-pr587-20261005`; trio + Dockerfile in the tag, image `localhost/frank-primary-freetoken:pr587-77a7201` and orcarouter payload on disk) | `frank-primary-freetoken-pr587.md` |
|
||||
|
||||
@@ -0,0 +1,178 @@
|
||||
# frank-large — halogen-flash-server 0.16.2, Qwen3.8-Flash-Next v2 Swift1.5 ABLITERATED (Dankclaw, Strix Halo iGPU)
|
||||
|
||||
**ACTIVE since 2026-10-06.** Rollback = revert the swap commit's trio
|
||||
(`scripts/frank-large.env`, `scripts/serve-frank-large.sh`,
|
||||
`scripts/frank-large.service`); the retired 27B payload stays on disk at
|
||||
`/home/dank/models/halogen-qwen3.8-27b/` and its fetch script
|
||||
(`scripts/download-frank-large-halogen.sh`) stays in the repo.
|
||||
|
||||
## Why this shape exists
|
||||
|
||||
Frank's ask: replace the frank-large model with
|
||||
`https://huggingface.co/Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated`.
|
||||
That payload is an **abliterated Swift1.5 finetune of peonist's
|
||||
Qwen3.8-Flash-Next v2 checkpoint in the v2 `.hgn` format** — a file that
|
||||
loads ONLY in the **halogen-flash-server** engine line, NOT in the halogen
|
||||
0.1.4 container that served the 27B dense checkpoint. The model move is
|
||||
therefore a **full engine migration**, and the accepted-product facts flip
|
||||
twice in the fleet's favour:
|
||||
|
||||
- the base is **abliterated** (the 27B halogen endpoint was a finetune of
|
||||
the censored `Qwen/Qwen3.8-27B`; refusals were an accepted regression there),
|
||||
- **vision is back** — the v2 vision tower loads beside the checkpoint and
|
||||
the engine runs it (the 27B halogen engine silently ignored its encoder),
|
||||
- **MTP speculative decoding returns** with the head inside the checkpoint
|
||||
itself (the 27B shipped a DFlash2 drafter instead; comparable purpose,
|
||||
different file plumbing),
|
||||
- the card claims **53–57 t/s greedy decode, ~1,640 t/s prefill at 34k, ~72 %
|
||||
fewer thinking tokens** than the censored 27B (905 mean), and **262,144
|
||||
native context**. Publisher-card numbers — verify against live traffic
|
||||
before quoting further (see Verification).
|
||||
|
||||
The engine is **closed-source**, ships as `ghcr.io/peonist-ai/
|
||||
halogen-flash-server:0.16.2` (the version the model card's quickstart pins),
|
||||
digest-pinned:
|
||||
|
||||
```
|
||||
ghcr.io/peonist-ai/halogen-flash-server@sha256:0c61bf84ac22308a53f5d1ca6b86806702d7039e5ebc51cae4c66621b92fe04a
|
||||
```
|
||||
|
||||
(3.6 GB image; RepoDigest captured at pull 2026-10-06. Card-tested fallback
|
||||
if a future pin misbehaves: 0.15.1.)
|
||||
|
||||
## The payload — a triangle of files, all sha256-verified at fetch
|
||||
|
||||
| File | Bytes | Source repo @ revision | sha256 (LFS) |
|
||||
|---|---|---|---|
|
||||
| `qwen38-flash-next-v2-swift15-abliterated.hgn` | 66,687,678,464 | Quat3rnion/halogen-swift1.5-qwen3.8-flash-next-v2-abliterated @ main (`69a85db`) | `9c880b8ba48deabfa06172eff1e7572e4fe465e09e97faec6e50c3f6fdcac037` |
|
||||
| `qwen38-flash-next-ngram.hgn` (REQUIRED) | 51,200,246,144 | peonist-ai/halogen-qwen3.8-flash-next @ `5cc17cea` | `3450acada94e19aabad88bd49b45eb70304820949c2240fc178067b20ae5dffe` |
|
||||
| `qwen38-flash-next-vision.hgn` (enabled) | 897,916,416 | peonist-ai/halogen-qwen3.8-flash-next @ `e053f488` | `d62e0ae553fe88afd3833733d4a4c669f34d20fd8dfce4b9610525bed2134b10` |
|
||||
| `tokenizer/` (6 files, flat dir) | 22.4 MB total | peonist @ `e053f488` | `tokenizer.json` is LFS `0997f410…` (byte-identical to the 27B-era copy); the rest are plain git blobs, size-checked |
|
||||
|
||||
Payload notes worth keeping:
|
||||
|
||||
- The deployed checkpoint is a **compacted r2** of the card's append-only
|
||||
original (89,775,868,032 B → 66,687,678,464 B): the abliteration workflow
|
||||
appends and the repo's `tooling/` compact drops dead padding; live tensors
|
||||
are byte-identical, the LFS sha256 is the contract, and the 24 GiB saving
|
||||
is why it fits.
|
||||
- **`HALOGEN_MTP_HEAD` stays UNSET**: the MTP head is INSIDE the v2
|
||||
checkpoint (the separate `qwen38-flash-next-mtp.hgn` sidecar is for a
|
||||
different assembly and is deliberately not fetched).
|
||||
- The **n-gram lookup table is REQUIRED**. The engine reads it beside the
|
||||
checkpoint (documented fallback); the card's quickstart also names it
|
||||
explicitly via `HALOGEN_NGRAM_TABLE`, and the serve script passes both —
|
||||
the env var is belt-and-braces over the documented beside-the-checkpoint
|
||||
default, and both mechanisms resolve to the same file.
|
||||
- The tokenizer is fetched fresh from peonist @ `e053f488` into
|
||||
`/home/dank/models/halogen-flashnext-v2/tokenizer/` — same flat-directory,
|
||||
no-symlink law as every other payload (an HF cache snapshot's symlinks
|
||||
dangle inside the container mount; both the serve gate and the deployer
|
||||
refuse them).
|
||||
|
||||
Fetch: `scripts/download-frank-large-flashnext.sh` (`--verify-only` to
|
||||
re-check; `-f` + exact byte counts + LFS sha256 + resolve-the-CDN-then-`-C -`
|
||||
resume law throughout; no token — the repos are public).
|
||||
|
||||
## Shape — the knobs that are not free
|
||||
|
||||
`scripts/frank-large.env` carries the full rationale inline; the table:
|
||||
|
||||
| Env | Value | Why |
|
||||
|---|---|---|
|
||||
| `HALOGEN_CTX` | 200192 | per-slot HARD ceiling (a longer prompt is a 400 naming the limit, never truncation). Harness compaction: 70 % = 140,134; 140,134 + 57,344 = 197,478 ≤ 200,192 |
|
||||
| `HALOGEN_KV_SLOTS` | 2 | the 27B engine's "slots>1 disables the drafter" law does NOT carry over — on this engine batching and speculation coexist (the card's quickstart ships 2 slots WITH the head). 2 slots = concurrent capacity |
|
||||
| `HALOGEN_KV_POOL_POSITIONS` | 400384 | explicit form of the default (2 × ctx), so the seam test pins the product; ~28 KiB/position ⇒ ~10.7 GiB KV |
|
||||
| `HALOGEN_HOST_RESERVE_GIB` | 20 | keeps the host honest (the w4b-era lesson) |
|
||||
| `HALOGEN_SPEC_ADAPT` | **0** | MANDATORY (card): the default adaptive policy turns the MTP head off for whole requests after low acceptance and **halves decode**. The head stays on; temp-1.0 traffic pays some acceptance for it — accepted trade |
|
||||
| `HALOGEN_MAX_TOK` / `HALOGEN_MAX_TOKENS_CAP` / `HALOGEN_QUEUE_TIMEOUT` | 8192 / 65536 / 7200 | 8192 is only the per-request DEFAULT (the harness always sends 57,344); the cap/timeout pair is COUPLED per FLAGS.md (65536 at ~10 t/s serial needs > 6550 s) |
|
||||
| `HALOGEN_TEMPERATURE` / `TOP_P` / `TOP_K` | 1.0 / 0.95 / 20 | the card's thinking recipe as SERVER-SIDE defaults — the engine HAS a sampler now, so an absent temperature is no longer greedy; models.yml extraBody matches it (consistent, not load-bearing) |
|
||||
| `HALOGEN_WEIGHTS_LOCK` | 1 | card's belt-and-braces; the engine pins weights by default on v2 (`HALOGEN_FLASH_PIN_TRUNK=0` is a w4b-only knob and is REFUSED by v2) |
|
||||
| `HALOGEN_MODEL_ID` / publish | frank-large / 7770:8731 | unchanged aliases; engine port 8730 never published |
|
||||
| MTP depth | default 2 | FLAGS.md: 2 is best all-round, 3 for code — Frank's traffic is agent/prose |
|
||||
|
||||
**Memory ledger** (121.2 GiB unified, the referee is the boot ledger):
|
||||
weights ~62 GiB resident + KV ~10.7 + arena ~8 (`3.8 + 8.4 × 8192/16384`)
|
||||
+ vision ~2 ≈ **82.7 GiB engine-side**, + 20 host reserve = ~103 with
|
||||
~18 GiB margin over the box's baseline co-tenancy. The 51.2 GB n-gram
|
||||
table rides as mmap'd page cache beside the checkpoint — the card's own
|
||||
quickstart arithmetic does not bill it as resident, but WATCH the ledger
|
||||
at boot and re-size if the engine prints a clamp. Prompt cache is ON by
|
||||
default (`PROMPT_CACHE=2`): a follow-up turn at 100k ctx costs ~2 s
|
||||
instead of ~88 s cold — the big-TTFT win the 27B engine needed a manual
|
||||
`HALOGEN_CACHE_MB` pin for is built in here.
|
||||
|
||||
## Engine contract facts (FLAGS.md of the flash-server repo)
|
||||
|
||||
- OpenAI chat **and** `/v1/responses` **and** `/v1/messages` **and**
|
||||
`/v1/completions`; tool calls verified on the card; `response_format`
|
||||
still unsupported (no constrained decoding).
|
||||
- Thinking → **`reasoning_content`** (same field as every engine this
|
||||
endpoint has run); the v2 chat template rejects `reasoning_effort`
|
||||
outside `xhigh|medium|low` — the models.yml fold (high→xhigh) stays
|
||||
mandatory.
|
||||
- Vision: OpenAI `image_url` content parts only (the fleet's standard
|
||||
contract; the legacy top-level `images` field is the frank-small trap).
|
||||
≥256 px probe images or the model answers "video".
|
||||
- `HALOGEN_MAX_TOKENS_CAP` over-cap is a 400 naming the limit — never a
|
||||
truncation. `max_tokens` INCLUDES thinking.
|
||||
- All flags read once at startup; every flag is BITWISE except
|
||||
`HALOGEN_CK_OVERLAY`, `HALOGEN_MATMUL_TUNING_FILE`,
|
||||
`HALOGEN_INDEXER_BUDGET`, `HALOGEN_COMPOSABLE_CONTEXT`. Unset
|
||||
`HALOGEN_CHECKPOINT` defaults to `/models/qwen38-flash-next-v2.hgn`
|
||||
with the lookup table beside it.
|
||||
- Prompt-lookup drafting (3,3 window) rides beside the MTP head by
|
||||
default and is byte-identical to greedy verification where it fires.
|
||||
- Watchdog: `HALOGEN_ENGINE_WATCHDOG_S=180` with defer-logic for memory
|
||||
stalls — on a shared host the defaults are the safe choice; the
|
||||
restart policy stays real through the root unit's `Restart=always`.
|
||||
|
||||
Kernel: `CONFIG_HSA_AMD_SVM=y` (CachyOS ships it); `amdgpu.noretry=-1`
|
||||
(auto) — for this IP auto resolves to retry-on, the same effective state
|
||||
the 27B engine ran on. Only revisit if the ledger shows SVM complaints.
|
||||
|
||||
## Mechanics
|
||||
|
||||
- **Root systemd unit** (`scripts/frank-large.service`, `User=dank`,
|
||||
`LimitMEMLOCK=infinity`, `Restart=always`, `ExecStop=podman stop -t 60
|
||||
frank-large-halogen`), driven by `scripts/serve-frank-large.sh` — the
|
||||
fleet's `podman run`-direct pattern. The script runs the container
|
||||
FOREGROUND so the boot ledger (weights resident, pool fit, MTP head,
|
||||
prompt cache, any clamp) lands in `/home/dank/logs/frank_large.log`.
|
||||
- Serve gates mirror the deployer EXACTLY (2026-10-01 crash-loop law):
|
||||
image digest in podman storage, byte-exact checkpoint + n-gram table +
|
||||
vision tower, flat tokenizer with all six files as regular files,
|
||||
GPU nodes present.
|
||||
- Engine stdout goes to the log, NOT the journal. The FIRST request after
|
||||
a (re)start can sit silent ~45–60 s on this engine class (health
|
||||
`engine_unresponsive`, 503s) and self-recovers — do not restart over it.
|
||||
|
||||
## Verification checklist (run after any redeploy)
|
||||
|
||||
1. Boot ledger greps in `/home/dank/logs/frank_large.log`: resident
|
||||
weights (~62 GiB), KV pool line, MTP/drafter line, prompt-cache line,
|
||||
**no clamp named**.
|
||||
2. `curl http://192.168.0.143:7770/health` → 200.
|
||||
3. Text completion probe: reasoning lands in `reasoning_content`,
|
||||
`timings` honest (generate ≥128 tokens; at 8 the inter_token number is
|
||||
first-token noise).
|
||||
4. Vision probe: a 512×512 solid-red PNG through an `image_url` part →
|
||||
"Red" (≥256 px, one word expected).
|
||||
5. Spec decode: `draft_n`/`draft_n_accepted` present in timings on
|
||||
real-shaped traffic (SPEC_ADAPT=0 means the head is always engaged).
|
||||
|
||||
## Client contract (harness)
|
||||
|
||||
`~/.omp/agent/models.yml` — provider `llamacpp (dankclaw)` @
|
||||
`http://192.168.0.143:7770/v1`: `contextWindow: 200192`,
|
||||
`maxTokens: 57344`, `input: [text, image]`,
|
||||
`reasoningContentField: "reasoning_content"`, the `high→xhigh` effort
|
||||
fold, `extraBody` temperature 1.0 / top_p 0.95. Compaction still rides
|
||||
frank-small (the `compactionModel` steering is unchanged).
|
||||
|
||||
## Rollback
|
||||
|
||||
Revert the swap commit and redeploy — the 27B trio is intact on disk
|
||||
(payload at `~/models/halogen-qwen3.8-27b/`, fetch script, and the halogen
|
||||
0.1.4 image digest still in podman storage). The older ROCmFPX rollback
|
||||
door (`build-rocmfpx.sh` pin) is untouched.
|
||||
@@ -10,11 +10,12 @@ experts in a 40 GiB/rank pinned host pool + VRAM cache (ratio 0.82), the
|
||||
47.7 GiB PLE table pinned to host RAM, fp8 KV, MTP spec decode depth 3,
|
||||
2 × 200,000 first boot. Rollback tags frank-primary-freetoken-pr587-20261005
|
||||
and frank-primary-vllm-shapeG-20261003),
|
||||
and Dankclaw runs two endpoints — **halogen container (frank-large, port 7770:
|
||||
ghcr.io/peonist-ai/halogen @ pinned digest serving peonist-ai
|
||||
/halogen-qwen3.8-27b, a 33.4 GiB .hgn checkpoint that loads in no other
|
||||
engine, DFlash2 spec decode, ONE slot at 200,700 ctx, text-only — vision
|
||||
lives on frank-small)** and
|
||||
and Dankclaw runs two endpoints — **halogen-flash-server 0.16.2
|
||||
(frank-large, port 7770: ghcr.io/peonist-ai/halogen-flash-server @ pinned
|
||||
digest serving the Quat3rnion ABLITERATED Swift1.5 finetune of
|
||||
Qwen3.8-Flash-Next v2, a ~62 GiB v2 .hgn that loads in no other engine,
|
||||
MTP spec decode always-on (HALOGEN_SPEC_ADAPT=0), TWO slots at 200,192
|
||||
ctx, vision ON)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: ajgazin Swift-1.5-Qwen3.8-27B
|
||||
Uncensored Dynamic-MTP RCO-IQ3_XXS GGUF — 27B (the MTP head in the file
|
||||
IS drafted, draft-mtp depth 3), TWO 131,072 conversations at Frank's
|
||||
@@ -45,10 +46,10 @@ PLE 47.7 GiB pinned RAM, fp8 KV 7.32 KiB/tok, MTP depth 3,
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[RTX 3090 (PCIe x4 dock) +
|
||||
gfx1151 iGPU 114 GiB GTT]
|
||||
E[halogen container gfx1151, digest-pinned image
|
||||
E[flash-server container gfx1151, digest-pinned image
|
||||
frank-large, port 7770]
|
||||
F[halogen-qwen3.8-27b .hgn p1w4d-d2 + DFlash2 drafter
|
||||
1 x 200700 ctx, d2 KV, text-only]
|
||||
F[flash-server v2 abliterated Swift1.5 .hgn + ngram table + vision tower
|
||||
2 x 200192 ctx, MTP always-on, image input]
|
||||
G[llama.cpp master CUDA sm_86
|
||||
frank-small, port 7771]
|
||||
H[Swift-1.5-27B Uncensored RCO-IQ3_XXS-mtp GGUF
|
||||
@@ -71,30 +72,37 @@ frank-small, port 7771]
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-large-halogen27b-dflash2]] — **ACTIVE since 2026-10-05**
|
||||
- [[frank-large-flash-server-v2-swift15]] — **ACTIVE since 2026-10-06**
|
||||
(root systemd unit + podman, port 7770): the digest-pinned
|
||||
`ghcr.io/peonist-ai/halogen` image serving `peonist-ai/halogen-qwen3.8-27b`
|
||||
(`qwen3.8-27b-p1w4d-d2.hgn`, 33.4 GiB) with the DFlash2 drafter on ONE slot
|
||||
of 200,700 ctx (slots>1 disables speculation — engine law). Publisher card
|
||||
31.7 t/s mean across eight text workloads. Accepted tradeoffs: TEXT-ONLY
|
||||
(the engine ignores the vision encoder; frank-small owns images) and the
|
||||
CENSORED base model (the ROCmFPX AGENT file was Uncensored by construction).
|
||||
No server-side sampler — the thinking recipe rides the wire in models.yml.
|
||||
Rollback: revert the swap commit; the ROCmFPX payload + fork build +
|
||||
`build-rocmfpx.sh` pin all remain on disk
|
||||
Tuned 2026-10-06: `HALOGEN_CACHE_MB=16384` pins the prompt cache —
|
||||
auto was OFF at the 36.9 GB two-entry floor, so every deep request
|
||||
re-prefilled its 79-81k context (85-180 s each); a 17.3k cold prefill
|
||||
of 27.0 s now resumes in 1.7 s on the warm turn (cache_n 16,384). The
|
||||
serve script captures the engine ledger on disk (foreground `podman
|
||||
run`; the `-d` era wrote only the container ID). Decode-at-depth is
|
||||
structural: 3.3-3.8 t/s at 80k vs 30-50 shallow; no supported flag
|
||||
touches it.
|
||||
The FA tile lever from upstream issue #12 (`HALOGEN_FA_TQ=256`) was
|
||||
A/B'd at 80k on 2026-10-06: decode ±0.3 % (rejected), prefill −9.5 %
|
||||
on this topology — the default tile stands. The deep harness roles
|
||||
(`slow`, `advisor`) rerouted to frank-primary; frank-large keeps
|
||||
OpenWebUI and shallow callers.
|
||||
`ghcr.io/peonist-ai/halogen-flash-server` 0.16.2 engine serving the
|
||||
Quat3rnion ABLITERATED Swift1.5 finetune of peonist's
|
||||
Qwen3.8-Flash-Next v2 checkpoint (`qwen38-flash-next-v2-swift15-abliterated.hgn`,
|
||||
66.7 GB compacted r2 of the card's 89.8 GB append-only original,
|
||||
sha256-verified) with the REQUIRED n-gram lookup table (51.2 GB,
|
||||
@peonist 5cc17cea) and the vision tower (897.9 MB, @e053f488) beside
|
||||
it. TWO slots of 200,192 ctx, MTP head ALWAYS ON (`HALOGEN_SPEC_ADAPT=0`
|
||||
— the adaptive default turns the head off per request and halves
|
||||
decode), server-side sampler at the card's 1.0/0.95/20, coupled
|
||||
cap/timeout 65536/7200, prompt cache ON by default (the 27B engine's
|
||||
manual `HALOGEN_CACHE_MB` pin era is over — warm 100k turn ≈ 2 s vs
|
||||
≈ 88 s cold). Vision ON (image_url parts only). Card: 53-57 t/s
|
||||
greedy, ~1,640 t/s prefill at 34k, ~72 % fewer thinking tokens.
|
||||
Rollback: revert the swap commit; the 27B payload + fetch script stay
|
||||
on disk
|
||||
- [[frank-large-halogen27b-dflash2]] — **RETIRED 2026-10-05 → 10-06
|
||||
(rollback target; payload + fetch script + image on disk)**: the
|
||||
digest-pinned `ghcr.io/peonist-ai/halogen` 0.1.4 engine serving
|
||||
`peonist-ai/halogen-qwen3.8-27b` (`qwen3.8-27b-p1w4d-d2.hgn`, 33.4 GiB)
|
||||
— DFlash2 drafter, ONE 200,700 slot, text-only (the engine ignored the
|
||||
vision encoder), censored base. Owns the measured lessons: the prompt
|
||||
cache pin (`HALOGEN_CACHE_MB=16384` — auto was OFF at the 36.9 GB
|
||||
two-entry floor; a 17.3k cold prefill of 27.0 s resumed in 1.7 s warm),
|
||||
the FA tile rejection (`HALOGEN_FA_TQ=256`: prefill −9.5 %, decode
|
||||
±0.3 %), and deep decode at 3.3-3.8 t/s@80k being structural. The
|
||||
foreground-`podman run` ledger capture and the deep-role rerouting
|
||||
(`slow`/`advisor` → frank-primary) carried forward. Superseded by the
|
||||
MODEL swap: the abliterated v2 .hgn loads only in flash-server, so the
|
||||
engine moved with the model
|
||||
- [[frank-small-llama-swift15-unc-iq3xxs-mtp-131072]] — **ACTIVE since
|
||||
2026-10-05 (evening); UNCECERSED RCO-IQ3_XXS with MTP back on since
|
||||
2026-10-06 evening** (root systemd unit, port 7771): llama.cpp master
|
||||
@@ -400,7 +408,7 @@ frank-small, port 7771]
|
||||
(deployed and reverted 2026-08-22)
|
||||
- [[Dankserver System Specs]] — the 2x R9700 box (frank-primary)
|
||||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on
|
||||
the unsloth b11030 runtime since 2026-09-21, frank-small on the RTX 3090)
|
||||
the flash-server v2 runtime since 2026-10-06, frank-small on the RTX 3090)
|
||||
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
|
||||
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
|
||||
the retirement record — the IOMMU/SVA err=-19 root cause and where
|
||||
|
||||
Reference in New Issue
Block a user