84 lines
5.0 KiB
Markdown
84 lines
5.0 KiB
Markdown
---
|
|
name: llamacpp-adapter-swap
|
|
description: "Add or roll back a GGUF LoRA adapter on a dank-fleet llama.cpp endpoint (e.g. frank-zealot): provenance census, fork-flag gate, fetcher+env+deployer surface, seam tests, and the behavioural before/after probe that is the only real proof."
|
|
---
|
|
|
|
# Applying a GGUF LoRA adapter to a fleet endpoint
|
|
|
|
An adapter is **payload, not shape**: it changes what the endpoint answers and nothing in
|
|
the KV/budget equation. Fleet law still applies — no benchmarks unless Frank orders one,
|
|
rollback is one env line, `./tests/run.sh` before commit, deployer health gate + e2e gate.
|
|
|
|
## 1. Census the artifact BEFORE writing any code
|
|
|
|
```bash
|
|
curl -sS "https://huggingface.co/api/models/<repo>?blobs=true" # size + LFS oid (None => plain git object)
|
|
curl -sSL -o /tmp/a.gguf "https://huggingface.co/<repo>/resolve/main/<file>" && sha256sum /tmp/a.gguf
|
|
```
|
|
|
|
Parse the GGUF KV section (15 lines of struct.unpack) and read **provenance**: abliteration
|
|
adapters carry `prism.abliterate.base_gguf` / `base_sha256`, `adapter.type`, `sites`,
|
|
`target_layers`, `target_embedding`. Compare the declared base against the file the endpoint
|
|
actually serves (its sha is in the endpoint's own fetcher script). A mismatch is normal for
|
|
sibling quantizations — the loader matches **tensor names, not hashes** — but it must be
|
|
stated in the docs, because it means the card's measured numbers are not this endpoint's
|
|
numbers.
|
|
|
|
## 2. Gate the runtime, from the binary's own `--help`
|
|
|
|
On the host: `/home/dank/GitPackages/<build>/llama-server --help | grep -iE 'lora|adapter'`.
|
|
Confirm `--lora-scaled FNAME:SCALE` exists (and `--lora-init-without-apply` + `POST
|
|
/lora-adapters` if a runtime toggle is ever wanted). Vendor forks gate adapters by metadata
|
|
version too — e.g. PrismML builds older than `prism-b10777` refuse packs with `unsupported
|
|
prism.hadamard.version: 2`. Read the card for the build it verified on and put that in the
|
|
fetcher header.
|
|
|
|
## 3. Surface (five files, copy the mmproj convention)
|
|
|
|
- `scripts/<svc>.env`: `LORA_FILE=<path>` + `LORA_SCALE=1`; document that **`LORA_FILE=` (empty)**
|
|
is the rollback and `LORA_SCALE=0` is the inert-but-present mode. Use `${VAR=default}` (no
|
|
colon) so an explicitly empty value survives — same reason `MMPROJ_FILE`/`PRESENCE_PENALTY` do.
|
|
- `scripts/serve-<svc>.sh`: `EXTRA_ARGS+=(--lora-scaled "${LORA_FILE}:${LORA_SCALE}")` — **one
|
|
argv token**, so a scale can never arrive without its file; refuse a missing file by name
|
|
(`Restart=always` turns an unfinished fetch into a crash loop that reads like a runtime fault).
|
|
- `scripts/download-<svc>-<adapter>.sh`: copy the sibling fetcher (`-fsSL --fail-with-body`, byte
|
|
count, sha256, `--verify-only`, idempotent). No LFS oid => pin the content sha. Header carries
|
|
the provenance finding.
|
|
- `scripts/deploy-remote.sh`: `LLM_<SVC>_LORA="${LLM_<SVC>_LORA:-<path>}"` + a check whose error
|
|
names **this** fetcher, not the payload's.
|
|
- `scripts/deploy-plan.py` needs nothing: fetchers are host-side state and select no service.
|
|
|
|
## 4. Seam tests (argv is the contract)
|
|
|
|
Stub-engine harness: create the adapter stub from the **shipped env's basename**, emit
|
|
`LORA_FILE`/`LORA_SCALE` in the generated test env, then assert on output only —
|
|
`--lora-scaled` value equals `<path>:<scale>`; a distinctive `LORA_SCALE` (0, 0.75) changes the
|
|
token; `LORA_FILE=` yields **absence** of the flag (`! grep -qxF -- '--lora-scaled'`); a missing
|
|
file exits non-zero naming that path; extend the env↔deployer anti-drift pair list.
|
|
|
|
## 5. Deploy and prove it behaviourally
|
|
|
|
Capture the **before** state on the live endpoint first, then: merge → on host `git pull` +
|
|
run the fetcher → `./scripts/deploy.sh <host> --service <svc>`.
|
|
|
|
```bash
|
|
sudo systemctl show <svc> -p DropInPaths --value # must be empty (bench-drop-in law)
|
|
sudo tr '\0' ' ' < /proc/$(sudo systemctl show <svc> -p MainPID --value)/cmdline
|
|
sudo journalctl -u <svc> --since <boot> | grep -iE 'adapter|lora' # llama_adapter_lora_init_impl prints prism.* keys
|
|
```
|
|
|
|
Refusal probe (the only proof of an abliteration adapter): same prompts, `temperature 0.1`,
|
|
**`max_tokens >= 2048`** — on thinking models a short budget is consumed by
|
|
`reasoning_content` and `content` comes back empty with `finish_reason: length`, which reads
|
|
as a refusal that never happened. Classify with a marker regex and report refusal-vs-complied;
|
|
do not paste generated harmful content into chat. Also run the standing trio: exact-phrase
|
|
chat, `tool_calls` with valid JSON, and a >=256 px solid-colour vision probe. A residual
|
|
`⚠️ Important Disclaimer` heading after the swap is style, not the refusal circuit.
|
|
|
|
## Docs
|
|
|
|
Endpoint page (`serverAppConfigs-docs/LLM/<svc>.md`): Adapter row, Fetch row, Flags row, one
|
|
Era row whose rollback is the env line, and a measured before/after paragraph. Laws worth
|
|
adding: provenance is metadata not a gate; an `token_embd` rank would crash, so an adapter is
|
|
not inert; the runtime pin may gain a second reason. Never restate the card's benchmark table.
|