agent: repoint FLM-NPU model roles off the retired provider; sync memory/skills
config.yml: vision -> llamacpp (dankclaw)/frank-large:auto, tiny -> llamacpp (dankserver)/frank-small:off, task -> llamacpp (dankclaw)/frank-large:medium (was flm dankclaw-npu), matching the flm (dankclaw-npu) provider removal in models.yml. Memories/managed skills: rollouts and learned notes from the frank-large KV-pool + sampler work and the noctalia recolor; merged the duplicate noctalia-palette-recolor skill into noctalia-custom-palette-recolor (adds the dotfiles-tracking step and the +show-config delta gotcha) and dropped the superseded legacy memory skills whose ghostty USR1 reload recipe is wrong for this fleet.
This commit is contained in:
Frank committed
1 parent
c32584e093
commit
872e86399a
56 files changed
+318
-178
No files matched your search
@@ -1,10 +1,22 @@
|
||||
SERVICE: frank-large (rootless Quadlet, boot-persistent) on dankclaw.
|
||||
Unit: frank-large.service | Container: frank-large.
|
||||
# dankclaw Deployment & Model Services
|
||||
|
||||
PORTS: Engine :8730 (container loopback, unauth), Front-end :8731, Published :7770.
|
||||
## frank-large (Quadlet Service)
|
||||
Deployed as a rootless Quadlet service (`frank-large.service`) on dankclaw. Hosts ~68 GiB weights and a 524,288-position KV pool (~14.4 GiB). HALOGEN_KV_SLOTS reduced to 3, stabilizing total memory at ~103.4 GiB within the 121 GiB budget. Admission is pool-bound, capping at 3 simultaneous full-length requests.
|
||||
|
||||
CONSTRAINTS: Weights pinned (~68 GiB). Env vars to avoid naming: TEMP, HOST, GPU_MAX_HEAP_SIZE (HIP runtime reads bare TEMP for JIT cache path).
|
||||
### Ports & Logging
|
||||
- Engine: :8730 (loopback), Front-end: :8731, Published: :7770
|
||||
- Logging: `podman logs frank-large` (full/filtered), `journalctl --user -u frank-large`
|
||||
- Env Vars: Avoid `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE` (HIP runtime reads bare `TEMP`).
|
||||
|
||||
TEST_ENV: pyyaml missing for test suite (uv available, pip missing on system python3). Most suite failures are pre-existing environment issues, not commit regressions.
|
||||
### Sampling & API Quirks
|
||||
- `frank-large` defaults to greedy; `frank-small` defaults to sampling (temp=1.0, top_k=20, top_p=0.95). Force small determinism via `temperature: 0` or `--sampling-defaults none`.
|
||||
- `preserve_thinking` is per-request (client-side `chat_template_kwargs`), not controlled by HALOGEN_* env vars.
|
||||
- `max_tokens: 200` with empty content indicates thinking consumed budget. frank-small returns 200 for unknown fields. File-cache shortage causes uninterruptible sleep, not OOM. `host_reserve_gib=24` protects n-gram table.
|
||||
- Tests: 285/286 pass (test 73 pre-existing comfyui selector bug). `pyyaml` missing on system python3; `uv` available.
|
||||
|
||||
LOG_RETRIEVAL: podman logs frank-large (full); podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):' (filtered); journalctl --user -u frank-large (journald).
|
||||
## Marreta & Authentik Guard Chain
|
||||
Deployed via podman compose, sharing gluetun egress with searxng. Caddy TLS edge routes through Anubis (:8923) → Authentik outpost → internal :8017 → marreta upstream.
|
||||
- Auth Flow: Provider-first creation → route flip (vhost `reverse_proxy anubis:8923`) → deploy.
|
||||
- Resolved Login-Wall: Null `ProxyProvider.application` and missing auth/authz/invalidation flows crashed the Rust outpost. Fix: populated flows, linked `Application(slug=marreta)`, restarted services.
|
||||
- Pitfalls: Host probe (`localhost:8017`) returns HTTP 000 due to Caddy netns; probe internally or grep config. Non-browser clients fail JS challenges. Audit service-to-service traffic for backend URLs hitting guarded hostnames. IdP, Gitea (git smart-HTTP), and self-auth services are exempt.
|
||||
- Verification: E2E test (`tests/e2e/specs/16-marreta.spec.ts`) validates homepage tile routing.
|
||||
@@ -1,3 +1,4 @@
|
||||
- frank-large (halogen-flash-server, Dankclaw) pool history and post-restart behavior, measured 2026-09-13: pool trimmed 524288 → 401408 (deployed, repo commits 91406af + 27e6335) to restore co-tenant headroom. Measured boot ledger at 401408/MAX_TOK=32768/reserve 15: 68.0 weights + 11.0 KV + 21.0 working = 100.0 GiB held, host left 9.1 GiB / 1396 contiguous 2 MiB blocks (vs 7.1 / 310 at 524288), engine printed need ~31.5 GiB vs modelled ~33.4, and the co-tenant stall NOTE no longer prints (threshold is somewhere below 9.1 GiB / 1396 blocks). TWO new facts: (1) On the FIRST request after any (re)start the engine can go silent ~45-60 s — /health flips to engine_unresponsive "no PONG within 30s" and the front-end 503s, then self-recovers. Do not restart the unit over it; the quadlet HealthStartPeriod=300s covers it. First completion attempt timed out during this window; a retry 20 s later returned instantly. (2) Host-left gain (+2.0 GiB) came in below the modelled +3.4 because the engine holds freed KV space as working slack (working 21.0 at 401k vs 20.6-ish at 524k with arena). The omp models.yml on blankpad (/home/blank/.omp/agent/models.yml) had frank-large maxTokens: 262144 — above halogen's 65536 cap and a 503-queue hazard after the pool cut; fixed to 65536, and the stale flm (dankclaw-npu) provider + its modelRoles (vision/tiny/task) were removed per the AGENTS.md route map (vision→frank-large, tiny→frank-small, task→frank-large:medium). _(context: frank-large KV pool trim to 401408 on Dankclaw, 2026-09-13, followed by OMP models.yml/config.yml cleanup on blankpad.)_
|
||||
- Dankclaw host swap (as of 2026-09-12): 64 GiB file at /swap.img, ext4 on /dev/nvme0n1p2, fallocated hole-free, driven by /etc/fstab line 12 (/swap.img none swap sw 0 0) which generates the swap.img.swap systemd unit — fstab is the only control point, there is no hand-written unit and no repo file, so deploy.sh cannot see swap state. vm.swappiness=60 (distro default, UNTESTED against this workload), zswap disabled. Raised from 8 GiB; 748 GiB still free on /. The old 8.1 GiB file is staged at dankclaw:~/.AGENTTRASH/20260912-swapfile-8g/swap.img awaiting Frank's approval to purge. Important framing: swap on this host protects the ~28 co-tenant containers, NOT frank-large — its ~68 GiB of pinned weights are not relieved by swap, and paging the KV pool (which IS anonymous) would be worse than the OOM kill it prevents, so if the box thrashes the knob is lower swappiness or a cgroup cap, not more swap. _(context: Raised swap on Dankclaw 2026-09-12 after the frank-large reserve-15 config left only ~7.1 GiB for co-tenants. Documented in serverAppConfigs-docs/Dankclaw/Swap Configuration.md and AGENTS.md memory-model section.)_
|
||||
- In the serverAppConfig repo, ./tests/run.sh exits 1 on a **clean tree** because of one pre-existing failure: tests/comfyui_rocm.bats test 73 "comfyui Containerfile is based on the gfx1151 torch image". It is a test-side selector bug, not a product regression — its find ... | head -1 picks dankclaw/alloy/Containerfile instead of the ComfyUI one, so the assertion reads the wrong file. Verify before blaming your own change: a good run reports 285 ok plus this single not ok 73. Confirmed present at HEAD across several sessions (2026-09-12 frank-large work), independent of frank-large/Caddy/quadlet edits. Do NOT treat a green deploy as blocked by it, and do NOT "fix" it by weakening the assertion — the correct fix is to target the actual ComfyUI Containerfile path. Note deploy.sh gates on its own per-service health plus the Playwright e2e suite, not on tests/run.sh, so this failure does not block deploys. _(context: Every ./tests/run.sh run during the 2026-09-12 frank-large KV-pool/reserve work returned exit=1 with exactly not ok 73 comfyui Containerfile is based on the gfx1151 torch image while 285 tests passed, including on commits where nothing ComfyUI-related changed.)_
|
||||
- Frank resolves contested tuning calls by **running the experiment**, not by reading the argument. On 2026-09-12 he overruled documentation-based pushback twice in a row with "lets just try it". Working with this, rather than against it: when he pushes back on a knob, state the prediction numerically up front, then run it and report measured vs predicted — he engages with that and it twice produced a better model than the docs (the arena linearity, and the vision-tower term). Corollaries: (1) get the rollback staged and named BEFORE the risky deploy, because the experiment may take production down — the 65536 trial OOMed frank-large for ~2 minutes; (2) use deploy.sh --skip-e2e for risky single-service experiments so the 4-minute e2e gate is not sitting on top of an outage; (3) after a failure, git revert sha + redeploy restores in ~10s, because the weights are already page-cached. Also learned the hard way: when a config experiment ends in a state that quietly reverses the user's original request (32768 + KV_POOL_FIT clamp meant the pool stayed 262144, i.e. the doubling never happened), say so plainly — do not let a green health check imply the goal was met. Separately, on log forensics: journalctl --user -u frank-large.service did NOT reach back to the 2026-09-11 boot even when asked with --since 2026-09-11 (rootless user journal retention is short). To compare against a container's ORIGINAL startup ledger, capture podman logs BEFORE the first deploy of the session — that is the only reliable copy of pre-change boot state. _(context: frank-large KV-pool doubling work on 2026-09-12: Frank overruled doc-based pushback twice ("let's just try it") — once raising HALOGEN_MAX_TOK to 65536 (OOMed the production service for ~2 min) and once back to 32768, which quietly un-did the doubling he had originally asked for. Also: journalctl --user retention for frank-large.service did not cover the 2026-09-11 boot even when queried with --si)_
|
||||
|
||||
@@ -1 +1 @@
|
||||
Frank-large rootless Quadlet service on dankclaw. Ports 7770/8731/8730. Weights ~68 GiB. Avoid TEMP/HOST/GPU_MAX_HEAP_SIZE env vars. Logs via podman/journalctl.
|
||||
dankclaw hosts frank-large (rootless Quadlet, 103.4 GiB RAM, 3 KV slots, greedy default) and marreta (podman compose, Caddy/Anubis/Authentik guard chain). Key ops: probe marreta internally to avoid netns HTTP 000; set temperature: 0 on frank-small for determinism; preserve_thinking is client-side per-request. Watch for file-cache uninterruptible sleep and missing pyyaml on system python3. Authentik flows must link Application(slug=marreta) to prevent Rust outpost crashes.
|
||||
@@ -1,5 +1,31 @@
|
||||
# Raw Memories
|
||||
|
||||
## 01a09835-91cf-72e7-8ece-e5316d4de48b
|
||||
updated_at: 1789286336
|
||||
Expanded frank-large KV pool from 262144 to 524288 positions (~14.4 GiB). Reduced HALOGEN_KV_SLOTS from 4 to 3 (~111 MiB freed per slot). Held memory stable at 103.4 GiB (68 weights + 14.4 KV pool + 21 working) within 121 GiB host budget. Admission is pool-bound, not slot-bound; caps at 3 simultaneous full-length requests.
|
||||
|
||||
frank-small defaults to sampling (checkpoint values: do_sample=true, temperature=1.0, top_k=20, top_p=0.95), creating asymmetry with frank-large which defaults to greedy. To force determinism on frank-small: send temperature: 0 per request or relaunch with --sampling-defaults none.
|
||||
|
||||
preserve_thinking cannot be set server-side. Enumerated 156 HALOGEN_* env vars; none control thinking/preserve. Must be sent per-request in chat_template_kwargs by the client.
|
||||
|
||||
API/Host quirks: frank-small returns 200 for unknown fields (200 proves nothing). max_tokens: 200 with empty content indicates thinking consumed budget. host_reserve_gib=24 protects n-gram table; file-cache shortage causes uninterruptible sleep, not OOM.
|
||||
|
||||
Tests: 285/286 pass (test 73 comfyui pre-existing selector bug).
|
||||
|
||||
## 01a09648-8eb8-7093-876a-116b11d4c5fb
|
||||
updated_at: 1789246675
|
||||
Deployment Stack: Podman compose on dankclaw. Marreta shares gluetun egress with searxng. Caddy TLS edge routes to Anubis (:8923) -> Authentik embedded outpost -> internal :8017 listener -> marreta upstream.
|
||||
|
||||
Auth Chain Workflow: Provider-first creation -> Route flip (vhost becomes reverse_proxy anubis:8923, internal @svc handler) -> Deploy. Verified via unauthenticated challenge probe, post-auth config grep, outpost DNS reachability.
|
||||
|
||||
Resolved Failure: Marreta login-wall due to ProxyProvider.application = None on marreta-proxy. Null invalidation_flow crashed Rust outpost serde. Null auth/authz flows compounded issue.
|
||||
|
||||
Fix Applied: Populated invalidation_flow, authentication_flow, authorization_flow + linked Application(name=Marreta, slug=marreta, provider=marreta-proxy). Restarted authentik-server + authentik-outpost.
|
||||
|
||||
Pitfalls & Constraints: Probe http://localhost:8017/ from host returns HTTP 000 (listener in Caddy netns); probe inside netns or grep config. Non-browser clients cannot solve JS challenge; audit service-to-service traffic for backend datasource URLs hitting public guarded hostnames. Exempt-by-construction: IdP (auth.), git smart-HTTP (Gitea), self-auth services. Authentik providers/applications are live-only (ak shell); no repo blueprint (captured in authentik-proxy-provider-app-link skill).
|
||||
|
||||
Verification: E2E regression test (tests/e2e/specs/16-marreta.spec.ts) asserts homepage tile passes through guard chain to Marreta app title, avoiding login-wall parking.
|
||||
|
||||
## 01a09686-789e-7042-8774-55507faa78b2
|
||||
updated_at: 1789232480
|
||||
SERVICE: frank-large (rootless Quadlet, boot-persistent) on dankclaw. Unit: frank-large.service | Container: frank-large.
|
||||
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
thread_id: 01a09648-8eb8-7093-876a-116b11d4c5fb
|
||||
updated_at: 1789246675
|
||||
|
||||
Deployed marreta on dankclaw via podman behind the Anubis+Authentik guard chain, sharing gluetun egress with searxng. Resolved login-wall caused by missing Application link in Authentik ProxyProvider (null flows crashing Rust outpost). Implemented provider-first route flip, netns-aware probing, and e2e regression test.
|
||||
+4
@@ -0,0 +1,4 @@
|
||||
thread_id: 01a09835-91cf-72e7-8ece-e5316d4de48b
|
||||
updated_at: 1789286336
|
||||
|
||||
Expanded frank-large KV pool to 524288 positions (~14.4 GiB) and reduced slots from 4 to 3. frank-large defaults to greedy; frank-small defaults to sampling (1.0 temp, top_k 20, top_p 0.95). preserve_thinking is per-request only. 285/286 tests pass.
|
||||
@@ -0,0 +1,17 @@
|
||||
# frank-large Operations
|
||||
Deployed as rootless Quadlet on dankclaw. ~68 GiB weights + 14.4 GiB KV pool (524k positions, 3 slots). Total ~103.4 GiB.
|
||||
## Ports
|
||||
- Engine: :8730 (loopback)
|
||||
- Front-end: :8731
|
||||
- Published: :7770
|
||||
## Logging & Env
|
||||
- `podman logs frank-large` / `journalctl --user -u frank-large`
|
||||
- Avoid `TEMP`, `HOST`, `GPU_MAX_HEAP_SIZE` (HIP bare TEMP)
|
||||
## Sampling & API
|
||||
- Large: greedy | Small: sampling (temp=1.0, top_k=20, top_p=0.95)
|
||||
- Small determinism: `temperature: 0` or `--sampling-defaults none`
|
||||
- `preserve_thinking`: client-side per-request (`chat_template_kwargs`)
|
||||
- `max_tokens: 200` w/ empty content = thinking consumed budget
|
||||
- File-cache shortage → uninterruptible sleep. `host_reserve_gib=24` protects n-gram table.
|
||||
## Tests
|
||||
- 285/286 pass (test 73 comfyui bug). `pyyaml` missing on system py3; `uv` available.
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"model": "frank-small",
|
||||
"temperature": 0,
|
||||
"chat_template_kwargs": {"preserve_thinking": true}
|
||||
}
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
#!/bin/bash
|
||||
# Verify KV pool slots and memory stability
|
||||
docker inspect frank-large --format '{{.Config.Env}}'
|
||||
podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'
|
||||
echo "Slots: HALOGEN_KV_SLOTS"
|
||||
echo "Memory: ~103.4 GiB (68W + 14.4KV + 21W)"
|
||||
+6
@@ -0,0 +1,6 @@
|
||||
[Service]
|
||||
ExecStart=/usr/bin/podman run ...
|
||||
Environment=TEMP=...
|
||||
Environment=HALOGEN_KV_SLOTS=3
|
||||
[Install]
|
||||
WantedBy=default.target
|
||||
-17
@@ -1,17 +0,0 @@
|
||||
Rootless Quadlet service 'frank-large' on dankclaw.
|
||||
|
||||
Ports:
|
||||
- Engine: :8730 (loopback, unauth)
|
||||
- Front-end: :8731
|
||||
- Published: :7770
|
||||
|
||||
Constraints:
|
||||
- Weights: Pinned (~68 GiB)
|
||||
- Env Vars: Avoid TEMP, HOST, GPU_MAX_HEAP_SIZE (HIP runtime issue)
|
||||
|
||||
Logging:
|
||||
- Use podman logs or journalctl.
|
||||
|
||||
Test Env:
|
||||
- pyyaml missing, uv available, pip missing.
|
||||
- Failures likely pre-existing.
|
||||
-11
@@ -1,11 +0,0 @@
|
||||
#!/bin/bash
|
||||
# Retrieve logs for frank-large service
|
||||
|
||||
# Full logs
|
||||
podman logs frank-large
|
||||
|
||||
# Filtered logs
|
||||
podman logs frank-large 2>&1 | grep -E '^(dmalloc|kv pool):'
|
||||
|
||||
# Journald logs
|
||||
journalctl --user -u frank-large
|
||||
+12
@@ -0,0 +1,12 @@
|
||||
# Marreta & Authentik Deployment
|
||||
Podman compose on dankclaw. Shares gluetun egress with searxng.
|
||||
## Guard Chain
|
||||
Caddy TLS → Anubis (:8923) → Authentik outpost → internal :8017 → marreta
|
||||
## Auth Fix & Pitfalls
|
||||
- Login-wall caused by null `ProxyProvider.application` & flows crashing Rust outpost.
|
||||
- Fix: Link `Application(slug=marreta)`, populate auth/authz/invalidation flows.
|
||||
- Probe: `localhost:8017` returns HTTP 000 (Caddy netns). Use internal probe or grep config.
|
||||
- Non-browser clients fail JS challenges. Audit service-to-service URLs for public guarded hostnames.
|
||||
- Exempt: IdP, Gitea (git smart-HTTP), self-auth services.
|
||||
## Verification
|
||||
- E2E: `tests/e2e/specs/16-marreta.spec.ts` asserts homepage tile passes guard chain.
|
||||
+2
@@ -0,0 +1,2 @@
|
||||
Run: npx playwright test tests/e2e/specs/16-marreta.spec.ts
|
||||
Assert: Homepage tile passes Anubis/ProxyProvider guard chain without login-wall parking.
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
#!/bin/bash
|
||||
# Health check through netns
|
||||
podman exec marreta_caddy wget -qO- http://localhost:8017/ || echo "HTTP 000 expected"
|
||||
# Grep config for route flip
|
||||
cat /etc/caddy/config.json | grep "reverse_proxy"
|
||||
+8
@@ -0,0 +1,8 @@
|
||||
application:
|
||||
name: Marreta
|
||||
slug: marreta
|
||||
provider: marreta-proxy
|
||||
flows:
|
||||
authentication: enabled
|
||||
authorization: enabled
|
||||
invalidation: enabled
|
||||
Reference in new issue
Block a user