diff --git a/llm/.gitignore b/llm/.gitignore deleted file mode 100644 index 15dea46..0000000 --- a/llm/.gitignore +++ /dev/null @@ -1,6 +0,0 @@ -.pi-subagents/ -.AGENTTRASH/ -__pycache__/ -*.pyc -.venv/ -.obsidian/ diff --git a/llm/AGENTS.md b/llm/AGENTS.md deleted file mode 100644 index 95fac4b..0000000 --- a/llm/AGENTS.md +++ /dev/null @@ -1,280 +0,0 @@ -# AGENTS.md — vllmScripts Agent Rules - -## Overview - -This project manages deployment scripts for the LLM inference fleet on two -local machines. Topology (2026-08-22): - -| Target | Host | Runtime | Service | Model | Port | -|--------|------|---------|---------|-------|------| -| Dankserver | dank@dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 | -| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp, PromptForge v3 runtime (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3 (Q8-FFNEQ-LateQ6, full-W8 views, Output-K8 MTP proxy, BF16 vision), 262K context / 1 slot | 52625 | - -The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM -NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw; -the CIRU ROCmFPX runtime then served the v3 ActiveFPX PromptForge model on -the Strix Halo iGPU (gfx1151). On 2026-08-22 the service briefly swapped to -the Kairic Edge IU4 runtime (pinned tag `kairic-edge-qwen38-27b-v1`), then -was reverted to v3 the same day (operator decision); the Kairic build and -model files remain on the machine. On 2026-08-21 a DFlash2 Vulkan -deployment briefly replaced CIRU, then was reverted the same day for -speed; the DFlash2 files live in `.AGENTTRASH/`. See -docs/ciru-activefpx-promptforge.md (active) and docs/kairic-edge.md -(retired experiment). The old scripts -live in `old/scripts/` for reference; they describe the retired PP -topology. - -**Sibling project:** [../serverAppConfigs](../serverAppConfigs) manages the -Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its -OpenWebUI consumes the `:7770` endpoint on Dankserver; its Gitea instance -hosts this repository. - -## Benchmarking Policy (MANDATORY) - -**STOP the production service BEFORE running any benchmark on a target.** -These servers serve real users (about 3 people). Running a bench while the -service is up starves the GPU and corrupts results. Restart the service -when the bench finishes. - -Rules: - -1. Stop the service before any benchmark: - `sudo systemctl stop frank-default-small` (Dankserver) or - `sudo systemctl stop frank-default-large` (Dankclaw). Use the password - `Assface.3241` with `sudo -S` when no tty is available. -2. Run the bench on a scratch port (never the service ports 7770/52625). -3. Restart the service when the bench completes: - `sudo systemctl start `. -4. Verify health before reporting: - `curl http://192.168.0.235:7770/health` or - `curl http://192.168.0.143:52625/health`. - -## Git Workflow - -All script edits must flow through git. Do not edit files on the remote machine directly. - -### Protocol - -1. **Source the script from the target machine** — fetch it from the server where it runs -2. **Place it in this repo** — commit the fetched version -3. **Edit here** — make all changes locally in the working tree -4. **Commit and push** — push to origin (Gitea on dankclaw) -5. **Pull on the target** — `git pull` on the remote machine before restarting -6. **Restart the service** — use the updated script on the target - -## Deletion & Trash Policy (MANDATORY) - -Never run `rm` to delete files. The command deletes permanently. Move every -item you would delete into `~/.AGENTTRASH/` instead. The trash keeps the -deletion recoverable and reviewable. - -Procedure: - -1. Create a dated staging directory: `mkdir -p ~/.AGENTTRASH/YYYYMMDD-topic` -2. Move each item: `mv ~/.AGENTTRASH/YYYYMMDD-topic/` -3. List the staged items and their sizes for the user. -4. Wait for explicit user permission before you empty the trash. - -Rules: - -- `rm`, `rm -rf`, and `git clean -f` are forbidden without explicit user - permission. -- `git * --no-verify` is forbidden. -- A moved item stays in the trash until the user approves its purge. -- The trash is a temporary hold, not a long-term archive. - -Use `mv`, never `rm`. This applies to every host, including Dankserver and -Dankclaw. - -## Deployment & Update Process - -Use `scripts/deploy.sh` to deploy committed changes to a target. It pushes `main`, pulls on the target, restarts the service, and checks the health endpoint. - -``` -FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver # llama.cpp FastMTP, port 7770 -FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw # CIRU llama.cpp, port 52625 -./scripts/deploy.sh dankserver --dry-run # preview without changes -``` - -Set `FRANK_SUDO_PW` for headless runs (remote sudo reads it from stdin). -The passwordless restart helpers installed for the retired topology are -obsolete; `deploy.sh` no longer uses them. - -Deployment workflow: - -```mermaid -graph LR - A[Edit locally] --> B[Commit] - B --> C[Push to origin] - C --> D[deploy.sh: pull on target] - D --> E[restart service] - E --> F[health check] -``` - -## Pi Model Config (`~/.pi/agent/models.json`) - -The Pi agent reaches the fleet through `/home/dank/.pi/agent/models.json`. -This file lives outside the repository. Each provider is one host endpoint. -Each model `id` is the served alias. The two aliases are -`frank-default-small` (dankserver `:7770`) and `frank-default-large` -(dankclaw `:52625`). - -The model `id` values are stable. The file -`/home/dank/.pi/agent/settings.json` references them. Two resolution rules -apply (verified in `@earendil-works/pi-coding-agent` `model-resolver.js`): - -- The default model resolves as a `defaultProvider` + `defaultModel` pair. - The `defaultProvider` must name the provider that serves the `defaultModel`. -- A subagent override uses a bare model `id`. A bare `id` resolves only when - it is unique across all providers. - -When you rename a service or move a model to a new host, update -`models.json`. Update the provider key, the model `id`, `contextWindow`, and -`input`. If the default model moved hosts, also update `defaultProvider` in -`settings.json`. Set `contextWindow` to the per-slot context (dankserver -`262144`, dankclaw `262144`). The full schema is in the agent knowledge base -page `~/.pi/agent/docs/pi/models.md`. - -## Runtime & Troubleshooting - -### FastMTP patched llama.cpp (retired, 2026-08-18) - -Historical: the Qwen3.8-27B IQ4_XS FastMTP service was retired on 2026-08-18 -(files in `.AGENTTRASH/`). The build notes below are kept for reference. -The HauhauCS FastMTP draft sidecar required a llama.cpp build from commit -`4df29be4f4c3673f428170fda944a5b19f743bb8` plus the -`HauhauCS-FastMTP-llama.cpp.patch` (from the HF repo). The build lives at -`/home/dank/llama.cpp-fastmtp/build` on Dankserver. Loading the draft with -a stock binary fails with `expected 5120, 248320, got 5120, 32768`. Do not -rebuild from another commit without re-applying the patch. - -**REQUIRED build flags:** `-DGGML_CUDA_FA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON`. -Without `GGML_CUDA_FA_ALL_QUANTS` the q8_0 KV cache + flash attention falls -back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s -vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README -for the base recipe. - -### v3 CIRU ActiveFPX PromptForge llama.cpp (Dankclaw, ACTIVE) - -The v3 runtime serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3` -from `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` -(PromptForge v2.3 + v3 Output-K8 runtime patch). The v3 profile: -`PROMPTFORGE_MODE=m2048_fused_tail1476`, full-W8 companion views, Output-K8 -MTP draft proxy, BF16 mmproj vision, TEMPERATURE 0.7 / TOP_P 0.95, -batch 4096 / ubatch 4096, 16 context checkpoints, reasoning on (deepseek, -budget -1), `--no-mmap`. Five-file payload (~36.7 GiB) in -`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`. Full runbook: -docs/ciru-activefpx-promptforge.md. - -### Kairic Edge llama.cpp (Dankclaw, RETIRED experiment 2026-08-22) - -The Kairic Edge llama-server serves `jcbtc/Qwen3.8-27B-IU4-Kairic-Edge` -from `/home/dank/ciru-qwen38/ROCmFPX-kairic/build-kairic/bin/llama-server` -(version 79, commit `e97b324`, pinned tag `kairic-edge-qwen38-27b-v1`). -It was deployed and reverted to v3 the same day (operator decision). -The runtime routes qualified prompt and verification shapes through the -RDNA 3.5 native IU4 matrix instruction. The build uses the release script -against the private TheRock ROCm 7.15 SDK kept in the retired ling3-ciru -venv. The model files (GGUF + FFN/GDN/GDN-Output .pfs, 26.05 GiB) live in -`/home/dank/models/Kairic-Qwen3.8-27B-IU4-Edge/`. - -**Release gap — CK `config.h`:** the pinned CK commit ships only -`config.h.in`; the release script and guide never generate -`third_party/composable_kernel/include/ck/config.h`. Generate it via a CK -CMake configure and copy it in before building (full runbook in -docs/kairic-edge.md). The file is untracked in CK git; a `git clean` -removes it. Local rebuilds are byte-identical (`d7625a0c...`). - -**Target greedy argmax fast path:** the pinned runner sets -`LLAMA_TARGET_GREEDY_ARGMAX_FASTPATH=1`. The fast path rejects non-greedy -requests (400) and the card's own chat sampling is non-greedy, so the -production attempt ran it at 0. It still rejects reasoning budgets unless -compatibility mode is used. Do not re-deploy Kairic without re-testing -chat first. - -**Rollback target:** the v3 model files remain in -`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/` and the v3 binary in -`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`. -Restored to v3 on 2026-08-22 (commit `c60156f`). - -**Env var naming rule still applies:** the HIP runtime reads `TEMP`, -`HOST`, and `GPU_MAX_HEAP_SIZE`. The serve script passes `--host`/`--port` -as CLI flags only; bind and port come from `BIND_ADDR`/`LISTEN_PORT`. - -### DFlash2 llama.cpp (Dankclaw, retired 2026-08-21) - -A DFlash2 Vulkan deployment briefly replaced CIRU on 2026-08-21, then was -reverted the same day for speed. Its serve script, env, and the build worktree -are archived in `.AGENTTRASH/` and documented in docs/dflash2-large-vulkan.md. - -### Env var naming rule — HIP runtime collision - -The ROCm HIP runtime reads the bare environment variables `TEMP`, `HOST`, -and `GPU_MAX_HEAP_SIZE`. The serve scripts export the env file with -`set -a`. A tunable named `TEMP` leaks into HIP. On 2026-08-13 -`TEMP=1.0` made HIP use the path `1.0` for its JIT kernel cache, then -segfaulted every model load. Commit `227ce01` fixed it by renaming the -sampling variable to `TEMPERATURE`. - -Never name a tunable `TEMP`, `HOST`, or `GPU_MAX_HEAP_SIZE`. Check for -collisions with this command: - -``` -strings /opt/rocm-*/lib/libamdhip64.so.* | grep -xE "TEMP|HOST|GPU_MAX_HEAP_SIZE" -``` - -### Crash-loop diagnostic procedure - -When a llama service segfaults during model load, follow these steps. - -```mermaid -graph TD - A[Service SEGVs at load] --> B[Read journal + kernel segfault line] - B --> C{Segfault in libamdhip64?} - C -- yes --> D[Bisect script vs raw binary] - C -- no --> E[Check config, memory, model files] - D --> F[Run gdb for backtrace] - F --> G[Find root cause] -``` - -1. Read the service journal with `journalctl -u -n 100`. Then read the kernel segfault line with `journalctl -k`. A segfault at address 8 in `libamdhip64` means a NULL unref in the HIP runtime. -2. Run the serve script directly. Then run the raw binary with the same flags. A different outcome isolates the env file sourcing. -3. Run gdb for the backtrace with this command: - ``` - gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-frank-large.sh - ``` - `llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost. Trust gdb, not the log. -4. Kernel spam about `amdgpu: SVM mapping failed` is a symptom, not the root cause. Investigate other causes before assuming memory pressure. - -### Operational quirks - -- `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '' | sudo -S `. -- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x `. -- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw v3 loads the ~36.7 GiB five-file payload (GGUF + FFN/GDN + Output-K8 .pfs + mmproj) in ~18 s with a warm page cache, a few minutes cold. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`). -- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md). -- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md). - -### Dankclaw memory model - -The GPU budget on Dankclaw is the rocm-smi figure: 64 GiB VRAM reported -by rocm-smi (SVM-backed; the docs' older 96 GiB figure was the addressable -SVM pool). The GPU node reports `local_mem_size 0`, so all allocations are -GTT/SVM-backed. Trust rocm-smi and the runtime. Do not use `free` or -`dmidecode` for the GPU budget; they show the CPU-side carve-out. This is -intentional. Do not change it. - -## Why - -Direct edits on the remote drift from version control. Future agents will not know what changed. Git is the single source of truth. - -> User request → Fetch script from remote → Commit here → Edit → Commit → Push → Pull on remote → Restart - -## Summary - -| Context | Action | -|---------|--------| -| Editing a deployment script | **Work through git** — fetch, commit, edit, push, pull, restart. Never edit on the remote directly. | -| Script exists only on remote | **Pull it into the repo first** — commit the fetched version before making changes. | -| Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. | -| Deploying a committed change | **Run `scripts/deploy.sh `** — it pushes, pulls on the target, restarts the service, and checks health. See [docs/deployment.md](docs/deployment.md). | -| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. | diff --git a/llm/README.md b/llm/README.md deleted file mode 100644 index 6febe2b..0000000 --- a/llm/README.md +++ /dev/null @@ -1,39 +0,0 @@ -# vLLM Scripts - -Deployment scripts for the LLM inference fleet (2026-08-18 topology): -upstream llama.cpp on Dankserver and CIRU ROCmFPX llama.cpp on Dankclaw. - -## Targets - -| Host | User | Codename | GPU | -|------|------|----------|-----| -| dankserver | dank | Dankserver | NVIDIA RTX 3090 24 GB + 62 GB RAM | -| dankclaw | dank | Dankclaw | AMD Strix Halo iGPU gfx1151 (unified memory) | - -## Active Services - -| Service | Host | Runtime | Model | Port | -|---------|------|---------|-------|------| -| frank-default-small | dankserver | llama.cpp (upstream b10121) | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU, 4 slots x ~175K ctx | 7770 | -| frank-default-large | dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK) | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView, 262K ctx / 1 slot | 52625 | - -The 2026-08-17 redo retired the pipeline-parallel split (PP head + RPC -worker) and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the -vLLM service on Dankclaw and renamed the services (see AGENTS.md). Their -scripts are preserved in `old/scripts/` for reference. - -## Structure - -- `docs/` — knowledge base: deployment runbook, model notes, research archive -- `scripts/` — serve scripts, env files, systemd units, deploy and test tools -- `old/scripts/` — retired PP/NPU-era scripts (reference only) - -## Deployment - -See [docs/deployment.md](docs/deployment.md). Run -`FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver` for the llama.cpp box or -`FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw` for the CIRU box. - -## Knowledge Base - -See [docs/index.md](docs/index.md) for the project knowledge base index.