live: heartbeat sync 2026-10-19T23:44:41-07:00 (Refs #11)

This commit is contained in:
Frank
2026-10-19 23:44:41 -07:00
parent 4023b21365
commit aec7f0f068
5 changed files with 108 additions and 38 deletions
@@ -1,49 +1,62 @@
---
name: dankclaw-pihole-lan-dns-record
description: "Add/repair a LAN DNS record for a new *.franksdank.cloud service on Dankclaw's Pi-hole v6 (hosts/custom.list path), plus the public Porkbun A record step and client cache gotchas."
description: "Add/repair a LAN DNS record for a new *.franksdank.cloud service on Dankclaw's Pi-hole v6 (hosts/custom.list + repo addn-hosts, plus the public Porkbun A record side) — and triage \"LAN names stopped resolving\" including the FTL boot-time record-load failure."
---
Use when a new `*.franksdank.cloud` service resolves publicly/LAN-needs a local address, on Dankclaw (Pi-hole v6 container, quadlet `pihole.container`).
# Dankclaw Pi-hole LAN DNS record
## Ground truth layout (verified 2026-10-11)
## What this fixes
- Quadlet: `~/.config/containers/systemd/pihole.container`; volumes `/home/dank/Containers/pihole/{etc-pihole,etc-dockers,addn-hosts}`.
- **LAN records live in `/home/dank/Containers/pihole/etc-pihole/hosts/custom.list`** — plain hosts format: `192.168.0.143 gitea.franksdank.cloud`. This is the boring, working path.
- `pihole.toml` `[[dns.records]]` array is EMPTY on this host — do not expect v6 UI-style records there.
- `gravity.db` exists but has NO `domain` table (v6 layout) — do not insert local records into SQLite.
A new `*.franksdank.cloud` service resolves publicly (via Porkbun → `franksdank.cloud` A record → Caddy on Dankclaw `192.168.0.143`) but **does not resolve on the LAN**. Pi-hole is the LAN DNS server (`dankclaw_default` bridge, published port 53) and serves local records only from files it imports.
## Procedure (30 s, proven)
## Live truth
```bash
ssh dankclaw 'echo "192.168.0.143 <name>.franksdank.cloud" | sudo tee -a /home/dank/Containers/pihole/etc-pihole/hosts/custom.list >/dev/null \
&& podman exec pihole pihole restartdns'
ssh dankclaw 'getent hosts <name>.franksdank.cloud' # must resolve
dig +short <name>.franksdank.cloud @192.168.0.143 # from any LAN box
- **The repo owns the records.** `dankclaw/pihole/addn-hosts` (hosts-style lines: `192.168.0.143 <name>`) is copied by `deploy-remote.sh` to `/home/dank/Containers/pihole/addn-hosts` and bind-mounted **read-only** into the container at `/etc/pihole/hosts/addn-hosts` (`Volume=/home/dank/Containers/pihole/addn-hosts:/etc/pihole/hosts/addn-hosts:ro` in `dankclaw/pihole/quadlet/pihole.container`). Add the name there, in the repo, then deploy — that is the durable path. `tests/pihole.bats` validates every line (IPv4, LAN range, name suffix, Caddyfile cross-check).
- `/home/dank/Containers/pihole/etc-pihole/hosts/custom.list` also exists on the host (older mechanism, mode `640` owned by the container uid, root reads it with `sudo`). It is **not** the repo's owner file — do not hand-edit it as the fix.
- `pihole.toml` carries a `dns.hosts = [ "192.168.0.143 …", … ]` array too (FTL's own in-config records). Both sources exist; if LAN names fail, neither is being applied.
- Pi-hole runs as a rootless quadlet named `pihole` — query it with `podman exec pihole …`, never `sudo podman` (per-user storage law).
- `pihole.franksdank.cloud` resolves through Pi-hole itself (loop/edge), so a vhost record pointing at `192.168.0.143` is correct for every Caddy-fronted service.
## Procedure (repo-first)
1. Append the line to `dankclaw/pihole/addn-hosts`:
```
192.168.0.143 <name>.franksdank.cloud
```
One record per vhost; keep it in sync with the Caddyfile (the bats seam cross-checks the two).
2. `./tests/run.sh` (the `pihole` seam validates the file), commit, push.
3. Deploy so the copy step runs: `./scripts/deploy.sh dankclaw --service pihole`.
4. Verify from the box: `getent hosts <name>.franksdank.cloud` → `192.168.0.143`.
5. Public edge record (optional; skip when every subscriber is LAN-only): Porkbun `POST /api/v3/json` with `apikey`/`secretapikey`/`command=add`/`type=A`/`hostname=<name>`/`data=76.49.118.136` (see `dankclaw-porkbun-edge`).
Client-side proof (avoid the resolver trap):
```
sudo resolvectl flush-caches # systemd-resolved otherwise keeps the negative cache
curl -sv --max-time 5 https://<name>.franksdank.cloud 2>&1 | grep 'Connected to'
```
Hosts files are read-only-readable only via sudo (dir owned by uid 100999). `sudo -n` works on dankclaw.
Expected: `Connected to <name>.franksdank.cloud (192.168.0.143)` — LAN route, not the public IP.
## API maze — DON'T re-explore
## Triage: every LAN name broke at once (2026-10-19 law)
- FTL API base is `/api` at the web port (8083) **with `Host: pi.hole` header**; `/admin/api/*` → bad_request hint; bare 127.0.0.1 without the vhost → 404 `not_found`.
- Login: `POST /api/login {"password": <WEBPASSWORD from /home/dank/Containers/pihole/env>}` → session in `X-Cors` header/cookie.
- `POST /api/apps/domains/custom` → **404 on this build** (FTL version predates it); `/api/customdns/custom` also absent. There is no container sqlite3. Use custom.list.
Signature: **all** `*.franksdank.cloud` return NXDOMAIN / Playwright `ERR_NAME_NOT_RESOLVED` while **external names still resolve**. The external-OK/local-fail split is the tell: FTL is answering external names from cache and holds **no local records at all** after a container/host restart — the mounts and files are intact and readable inside the container (`ls -l /etc/pihole/hosts/`, `grep -c . …/addn-hosts`), `dns.hosts` in `pihole.toml` is complete. The mechanism is unestablished; do not go hunting for a lost file.
## Public record (off-LAN reachability)
Porkbun API, keys in `/home/dank/Containers/caddy/env` (`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`):
```bash
curl -sL -X POST https://api.porkbun.com/api/json3/domain/api/create/franksdank.cloud \
-H "Content-Type: application/json" \
-d "{\"apikey\":\"$K\",\"secretapi\":\"$S\",\"name\":\"<name>\",\"type\":\"A\",\"content\":\"71.227.227.172\",\"ttl\":\"600\"}"
```
ssh dankclaw 'getent hosts grafana.franksdank.cloud' # empty => FTL, not the client
ssh dankclaw 'podman exec pihole pihole restartdns reload' # fixes it in seconds
```
- `json3` path 302-redirects → **`-L` required**; no-`-s` to see status.
- Porkbun edge (openresty) intermittently 503s — transient; retry later, don't burn turns. Public record is OPTIONAL when LAN-only subscribers exist.
- Wildcard `*.franksdank.cloud` TLS is already covered: Caddy + Porkbun DNS-01 (caddy-porkbun image), so a new subdomain needs NO new cert work — just the Caddyfile site block + reload.
Distinguish before acting: a **single** client failing while others work is FTL rate-limiting that client — `podman exec pihole grep Rate-limiting /var/log/pihole/FTL.log`; a `10.0.2.x` client address is a pasta container's DNS forwarder (SNAT), not a machine. Full causal page: `serverAppConfigs-docs/Infrastructure/Pi-hole.md` (**Records** pitfall + Failure modes table).
## Client-side cache gotchas
## Gotchas
- Workstation (systemd-resolved): negative cache survives Pi-hole changes; `sudo resolvectl flush-caches` needs a TTY (faillock-sensitive — no `-S` guessing). Non-interactive proof: `curl --resolve <name>.franksdank.cloud:443:192.168.0.143 https://<name>.franksdank.cloud/...`; cache self-expires (SOA negative TTL).
- Dankclaw/Dankserver resolve via Pi-hole directly — verify there first.
- Do NOT insert local records into SQLite. v6's `gravity.db` `domain` table has NO `type=2` custom-DNS rows in this build.
- `pihole restartdns reload` re-imports the hosts files **and** `dns.hosts`; that is the remedy for the boot-load failure above. Plain `pihole restartdns` also restarts the resolver — brief LAN-wide DNS blip.
- `dig @<ip> <name>` from the workstation bypasses systemd-resolved — use it to check FTL's answer directly (`+short @192.168.0.143`).
## Cross-reference
- Fleet DNS flow: `dankclaw/Caddyfile` (Caddy binds `:443` on the host, serves `*.franksdank.cloud` vhosts)
- Pi-hole service page: `serverAppConfigs-docs/Infrastructure/Pi-hole.md`, flow page `Infrastructure/LAN-DNS.md`
- Public edge: managed skill `dankclaw-porkbun-edge`
@@ -0,0 +1,57 @@
---
name: open-notebook-provision-model-rest
description: "Provision or repair an open-notebook model (LLM/TTS/embedding) over its REST API on Dankclaw: credential→model→defaults order, the defaults-store-the-ID trap, the /api/embed 500-that-succeeds, and the two-note ranking proof for embeddings."
---
# open-notebook model provisioning over REST
Use when adding/repairing a model row (embedding, LLM, TTS) in open-notebook on Dankclaw, when semantic
search is dead, or when an embed error appears in the worker log. Config lives in the **database**
(encrypted with `OPEN_NOTEBOOK_ENCRYPTION_KEY`) — no repo file mirrors it, REST or the UI are the only doors.
## Access (never put the bearer in argv)
The API bearer IS `OPEN_NOTEBOOK_PASSWORD`; the API publishes on loopback `127.0.0.1:5056` only, so run
from Dankclaw over ssh. `ssh dankclaw 'curl -H "Authorization: Bearer …"'` leaks the secret into
`/proc/<pid>/cmdline` — use a mode-600 curl config:
```bash
PW=$(grep -m1 '^OPEN_NOTEBOOK_PASSWORD=' /home/dank/Containers/open-notebook/env | cut -d= -f2- | tr -d '"')
cfg=$(mktemp); chmod 600 "$cfg"; printf 'header = "Authorization: Bearer %s"\n' "$PW" > "$cfg"
B=http://127.0.0.1:5056 # not 5055: seerr holds the host's 5055
```
## Order, and the address rule
`POST /api/credentials` → `POST /api/models` (`credential: <credential id>`) → `PUT /api/models/defaults`
→ any profile that names a model (speaker `voice_model`, episode `speaker_config`/`outline_llm`).
- Credential provider for any in-house OpenAI-compatible server: **`openai_compatible`** (not `openai`),
with a dummy `api_key` when the upstream does not authenticate.
- The app's URL guard refuses base URLs resolving in `169.254.0.0/16`, so `host.containers.internal`
cannot be stored; a **container name on `dankclaw_default`** (e.g. `http://frank-dragoon-halogen:8731/v1`)
or another machine's RFC1918 IP both pass.
## The two traps that eat the most time
1. **`PUT /api/models/defaults` takes the model ID, never the name.** `"default_embedding_model":
"qwen3-embedding-0.6b"` is accepted and echoed back, then every job dies in the worker with
`Model with ID qwen3-embedding-0.6b not found`. Use the row's `model:…` id (GET defaults first, change
one key, PUT the whole object back — the other keys in that response are ids and must be preserved).
2. **`POST /api/embed` answers 500 *after* the work succeeds.** Upstream `EmbedResponse.command_id`
declares `str` and the worker returns a `RecordID`, so FastAPI raises in serialization only. The truth is
`podman logs open-notebook | grep "Successfully embedded"`, never the HTTP code. Note saves
(`POST /api/notes`) submit `embed_note` themselves; `/api/embed` is only for backfills.
## Proof it actually works
- Per row: `POST /api/models/<id>/test` → `Embedding dimensions: 1024` (embeddings) or a real completion.
- Embeddings, discriminating test: create **two** unrelated notes, then
`POST /api/search` `{"query": "...", "type": "vector", "search_notes": true, "limit": 5}` — the enum is
`text|vector` (not `semantic`) and the score field is `similarity`. Each query must rank its own note
high and the other low (measured 0.659/0.292 vs 0.906/0.318). One note proves nothing: `total_count` is 1
either way. Delete the probe notes afterwards (`DELETE /api/notes/<id>`).
- Confirm which engine served it: `podman logs <engine container> | grep "POST /v1/embeddings"`.
Owner pages: `serverAppConfigs-docs/Dankclaw/Open Notebook.md` (app-side shapes), and the endpoint's own
page for whatever serves the model.
@@ -1,6 +1,6 @@
{
"v": 1,
"lastConsolidatedAt": 1792474614942,
"lastConsolidatedAt": 1792478514983,
"lastResult": {
"at": 1792433353188,
"sessions": 1,
@@ -1,6 +1,6 @@
{
"v": 1,
"lastConsolidatedAt": 1792474840929,
"lastConsolidatedAt": 1792478440973,
"lastResult": {
"at": 1792452243238,
"sessions": 1,
@@ -275,13 +275,13 @@ transition = [ "fade" ]
interval_seconds = 1200
[wallpaper.default]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_6kzekl.jpg"
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_4x17w3.jpg"
[wallpaper.last]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_6kzekl.jpg"
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_4x17w3.jpg"
[wallpaper.monitors.DP-1]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_6kzekl.jpg"
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_4x17w3.jpg"
[wallpaper.monitors.DP-2]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_oxex69.jpg"
@@ -290,7 +290,7 @@ transition = [ "fade" ]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_v9e2w3.jpg"
[wallpaper.monitors.eDP-1]
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_6kzekl.jpg"
path = "/home/blank/.local/share/noctalia/wallpapers/wallhaven_4x17w3.jpg"
[weather]
unit = "imperial"