live: heartbeat sync 2026-10-19T23:44:41-07:00 (Refs #11)
This commit is contained in:
Frank committed
1 parent
4023b21365
commit
aec7f0f068
5 files changed
+108
-38
No files matched your search
@@ -1,49 +1,62 @@
|
||||
---
|
||||
name: dankclaw-pihole-lan-dns-record
|
||||
description: "Add/repair a LAN DNS record for a new *.franksdank.cloud service on Dankclaw's Pi-hole v6 (hosts/custom.list path), plus the public Porkbun A record step and client cache gotchas."
|
||||
description: "Add/repair a LAN DNS record for a new *.franksdank.cloud service on Dankclaw's Pi-hole v6 (hosts/custom.list + repo addn-hosts, plus the public Porkbun A record side) — and triage \"LAN names stopped resolving\" including the FTL boot-time record-load failure."
|
||||
---
|
||||
|
||||
Use when a new `*.franksdank.cloud` service resolves publicly/LAN-needs a local address, on Dankclaw (Pi-hole v6 container, quadlet `pihole.container`).
|
||||
# Dankclaw Pi-hole LAN DNS record
|
||||
|
||||
## Ground truth layout (verified 2026-10-11)
|
||||
## What this fixes
|
||||
|
||||
- Quadlet: `~/.config/containers/systemd/pihole.container`; volumes `/home/dank/Containers/pihole/{etc-pihole,etc-dockers,addn-hosts}`.
|
||||
- **LAN records live in `/home/dank/Containers/pihole/etc-pihole/hosts/custom.list`** — plain hosts format: `192.168.0.143 gitea.franksdank.cloud`. This is the boring, working path.
|
||||
- `pihole.toml` `[[dns.records]]` array is EMPTY on this host — do not expect v6 UI-style records there.
|
||||
- `gravity.db` exists but has NO `domain` table (v6 layout) — do not insert local records into SQLite.
|
||||
A new `*.franksdank.cloud` service resolves publicly (via Porkbun → `franksdank.cloud` A record → Caddy on Dankclaw `192.168.0.143`) but **does not resolve on the LAN**. Pi-hole is the LAN DNS server (`dankclaw_default` bridge, published port 53) and serves local records only from files it imports.
|
||||
|
||||
## Procedure (30 s, proven)
|
||||
## Live truth
|
||||
|
||||
```bash
|
||||
ssh dankclaw 'echo "192.168.0.143 <name>.franksdank.cloud" | sudo tee -a /home/dank/Containers/pihole/etc-pihole/hosts/custom.list >/dev/null \
|
||||
&& podman exec pihole pihole restartdns'
|
||||
ssh dankclaw 'getent hosts <name>.franksdank.cloud' # must resolve
|
||||
dig +short <name>.franksdank.cloud @192.168.0.143 # from any LAN box
|
||||
- **The repo owns the records.** `dankclaw/pihole/addn-hosts` (hosts-style lines: `192.168.0.143 <name>`) is copied by `deploy-remote.sh` to `/home/dank/Containers/pihole/addn-hosts` and bind-mounted **read-only** into the container at `/etc/pihole/hosts/addn-hosts` (`Volume=/home/dank/Containers/pihole/addn-hosts:/etc/pihole/hosts/addn-hosts:ro` in `dankclaw/pihole/quadlet/pihole.container`). Add the name there, in the repo, then deploy — that is the durable path. `tests/pihole.bats` validates every line (IPv4, LAN range, name suffix, Caddyfile cross-check).
|
||||
- `/home/dank/Containers/pihole/etc-pihole/hosts/custom.list` also exists on the host (older mechanism, mode `640` owned by the container uid, root reads it with `sudo`). It is **not** the repo's owner file — do not hand-edit it as the fix.
|
||||
- `pihole.toml` carries a `dns.hosts = [ "192.168.0.143 …", … ]` array too (FTL's own in-config records). Both sources exist; if LAN names fail, neither is being applied.
|
||||
- Pi-hole runs as a rootless quadlet named `pihole` — query it with `podman exec pihole …`, never `sudo podman` (per-user storage law).
|
||||
- `pihole.franksdank.cloud` resolves through Pi-hole itself (loop/edge), so a vhost record pointing at `192.168.0.143` is correct for every Caddy-fronted service.
|
||||
|
||||
## Procedure (repo-first)
|
||||
|
||||
1. Append the line to `dankclaw/pihole/addn-hosts`:
|
||||
```
|
||||
192.168.0.143 <name>.franksdank.cloud
|
||||
```
|
||||
One record per vhost; keep it in sync with the Caddyfile (the bats seam cross-checks the two).
|
||||
2. `./tests/run.sh` (the `pihole` seam validates the file), commit, push.
|
||||
3. Deploy so the copy step runs: `./scripts/deploy.sh dankclaw --service pihole`.
|
||||
4. Verify from the box: `getent hosts <name>.franksdank.cloud` → `192.168.0.143`.
|
||||
5. Public edge record (optional; skip when every subscriber is LAN-only): Porkbun `POST /api/v3/json` with `apikey`/`secretapikey`/`command=add`/`type=A`/`hostname=<name>`/`data=76.49.118.136` (see `dankclaw-porkbun-edge`).
|
||||
|
||||
Client-side proof (avoid the resolver trap):
|
||||
|
||||
```
|
||||
sudo resolvectl flush-caches # systemd-resolved otherwise keeps the negative cache
|
||||
curl -sv --max-time 5 https://<name>.franksdank.cloud 2>&1 | grep 'Connected to'
|
||||
```
|
||||
|
||||
Hosts files are read-only-readable only via sudo (dir owned by uid 100999). `sudo -n` works on dankclaw.
|
||||
Expected: `Connected to <name>.franksdank.cloud (192.168.0.143)` — LAN route, not the public IP.
|
||||
|
||||
## API maze — DON'T re-explore
|
||||
## Triage: every LAN name broke at once (2026-10-19 law)
|
||||
|
||||
- FTL API base is `/api` at the web port (8083) **with `Host: pi.hole` header**; `/admin/api/*` → bad_request hint; bare 127.0.0.1 without the vhost → 404 `not_found`.
|
||||
- Login: `POST /api/login {"password": <WEBPASSWORD from /home/dank/Containers/pihole/env>}` → session in `X-Cors` header/cookie.
|
||||
- `POST /api/apps/domains/custom` → **404 on this build** (FTL version predates it); `/api/customdns/custom` also absent. There is no container sqlite3. Use custom.list.
|
||||
Signature: **all** `*.franksdank.cloud` return NXDOMAIN / Playwright `ERR_NAME_NOT_RESOLVED` while **external names still resolve**. The external-OK/local-fail split is the tell: FTL is answering external names from cache and holds **no local records at all** after a container/host restart — the mounts and files are intact and readable inside the container (`ls -l /etc/pihole/hosts/`, `grep -c . …/addn-hosts`), `dns.hosts` in `pihole.toml` is complete. The mechanism is unestablished; do not go hunting for a lost file.
|
||||
|
||||
## Public record (off-LAN reachability)
|
||||
|
||||
Porkbun API, keys in `/home/dank/Containers/caddy/env` (`PORKBUN_API_KEY`, `PORKBUN_API_SECRET_KEY`):
|
||||
|
||||
```bash
|
||||
curl -sL -X POST https://api.porkbun.com/api/json3/domain/api/create/franksdank.cloud \
|
||||
-H "Content-Type: application/json" \
|
||||
-d "{\"apikey\":\"$K\",\"secretapi\":\"$S\",\"name\":\"<name>\",\"type\":\"A\",\"content\":\"71.227.227.172\",\"ttl\":\"600\"}"
|
||||
```
|
||||
ssh dankclaw 'getent hosts grafana.franksdank.cloud' # empty => FTL, not the client
|
||||
ssh dankclaw 'podman exec pihole pihole restartdns reload' # fixes it in seconds
|
||||
```
|
||||
|
||||
- `json3` path 302-redirects → **`-L` required**; no-`-s` to see status.
|
||||
- Porkbun edge (openresty) intermittently 503s — transient; retry later, don't burn turns. Public record is OPTIONAL when LAN-only subscribers exist.
|
||||
- Wildcard `*.franksdank.cloud` TLS is already covered: Caddy + Porkbun DNS-01 (caddy-porkbun image), so a new subdomain needs NO new cert work — just the Caddyfile site block + reload.
|
||||
Distinguish before acting: a **single** client failing while others work is FTL rate-limiting that client — `podman exec pihole grep Rate-limiting /var/log/pihole/FTL.log`; a `10.0.2.x` client address is a pasta container's DNS forwarder (SNAT), not a machine. Full causal page: `serverAppConfigs-docs/Infrastructure/Pi-hole.md` (**Records** pitfall + Failure modes table).
|
||||
|
||||
## Client-side cache gotchas
|
||||
## Gotchas
|
||||
|
||||
- Workstation (systemd-resolved): negative cache survives Pi-hole changes; `sudo resolvectl flush-caches` needs a TTY (faillock-sensitive — no `-S` guessing). Non-interactive proof: `curl --resolve <name>.franksdank.cloud:443:192.168.0.143 https://<name>.franksdank.cloud/...`; cache self-expires (SOA negative TTL).
|
||||
- Dankclaw/Dankserver resolve via Pi-hole directly — verify there first.
|
||||
- Do NOT insert local records into SQLite. v6's `gravity.db` `domain` table has NO `type=2` custom-DNS rows in this build.
|
||||
- `pihole restartdns reload` re-imports the hosts files **and** `dns.hosts`; that is the remedy for the boot-load failure above. Plain `pihole restartdns` also restarts the resolver — brief LAN-wide DNS blip.
|
||||
- `dig @<ip> <name>` from the workstation bypasses systemd-resolved — use it to check FTL's answer directly (`+short @192.168.0.143`).
|
||||
|
||||
## Cross-reference
|
||||
|
||||
- Fleet DNS flow: `dankclaw/Caddyfile` (Caddy binds `:443` on the host, serves `*.franksdank.cloud` vhosts)
|
||||
- Pi-hole service page: `serverAppConfigs-docs/Infrastructure/Pi-hole.md`, flow page `Infrastructure/LAN-DNS.md`
|
||||
- Public edge: managed skill `dankclaw-porkbun-edge`
|
||||
@@ -0,0 +1,57 @@
|
||||
---
|
||||
name: open-notebook-provision-model-rest
|
||||
description: "Provision or repair an open-notebook model (LLM/TTS/embedding) over its REST API on Dankclaw: credential→model→defaults order, the defaults-store-the-ID trap, the /api/embed 500-that-succeeds, and the two-note ranking proof for embeddings."
|
||||
---
|
||||
|
||||
# open-notebook model provisioning over REST
|
||||
|
||||
Use when adding/repairing a model row (embedding, LLM, TTS) in open-notebook on Dankclaw, when semantic
|
||||
search is dead, or when an embed error appears in the worker log. Config lives in the **database**
|
||||
(encrypted with `OPEN_NOTEBOOK_ENCRYPTION_KEY`) — no repo file mirrors it, REST or the UI are the only doors.
|
||||
|
||||
## Access (never put the bearer in argv)
|
||||
|
||||
The API bearer IS `OPEN_NOTEBOOK_PASSWORD`; the API publishes on loopback `127.0.0.1:5056` only, so run
|
||||
from Dankclaw over ssh. `ssh dankclaw 'curl -H "Authorization: Bearer …"'` leaks the secret into
|
||||
`/proc/<pid>/cmdline` — use a mode-600 curl config:
|
||||
|
||||
```bash
|
||||
PW=$(grep -m1 '^OPEN_NOTEBOOK_PASSWORD=' /home/dank/Containers/open-notebook/env | cut -d= -f2- | tr -d '"')
|
||||
cfg=$(mktemp); chmod 600 "$cfg"; printf 'header = "Authorization: Bearer %s"\n' "$PW" > "$cfg"
|
||||
B=http://127.0.0.1:5056 # not 5055: seerr holds the host's 5055
|
||||
```
|
||||
|
||||
## Order, and the address rule
|
||||
|
||||
`POST /api/credentials` → `POST /api/models` (`credential: <credential id>`) → `PUT /api/models/defaults`
|
||||
→ any profile that names a model (speaker `voice_model`, episode `speaker_config`/`outline_llm`).
|
||||
|
||||
- Credential provider for any in-house OpenAI-compatible server: **`openai_compatible`** (not `openai`),
|
||||
with a dummy `api_key` when the upstream does not authenticate.
|
||||
- The app's URL guard refuses base URLs resolving in `169.254.0.0/16`, so `host.containers.internal`
|
||||
cannot be stored; a **container name on `dankclaw_default`** (e.g. `http://frank-dragoon-halogen:8731/v1`)
|
||||
or another machine's RFC1918 IP both pass.
|
||||
|
||||
## The two traps that eat the most time
|
||||
|
||||
1. **`PUT /api/models/defaults` takes the model ID, never the name.** `"default_embedding_model":
|
||||
"qwen3-embedding-0.6b"` is accepted and echoed back, then every job dies in the worker with
|
||||
`Model with ID qwen3-embedding-0.6b not found`. Use the row's `model:…` id (GET defaults first, change
|
||||
one key, PUT the whole object back — the other keys in that response are ids and must be preserved).
|
||||
2. **`POST /api/embed` answers 500 *after* the work succeeds.** Upstream `EmbedResponse.command_id`
|
||||
declares `str` and the worker returns a `RecordID`, so FastAPI raises in serialization only. The truth is
|
||||
`podman logs open-notebook | grep "Successfully embedded"`, never the HTTP code. Note saves
|
||||
(`POST /api/notes`) submit `embed_note` themselves; `/api/embed` is only for backfills.
|
||||
|
||||
## Proof it actually works
|
||||
|
||||
- Per row: `POST /api/models/<id>/test` → `Embedding dimensions: 1024` (embeddings) or a real completion.
|
||||
- Embeddings, discriminating test: create **two** unrelated notes, then
|
||||
`POST /api/search` `{"query": "...", "type": "vector", "search_notes": true, "limit": 5}` — the enum is
|
||||
`text|vector` (not `semantic`) and the score field is `similarity`. Each query must rank its own note
|
||||
high and the other low (measured 0.659/0.292 vs 0.906/0.318). One note proves nothing: `total_count` is 1
|
||||
either way. Delete the probe notes afterwards (`DELETE /api/notes/<id>`).
|
||||
- Confirm which engine served it: `podman logs <engine container> | grep "POST /v1/embeddings"`.
|
||||
|
||||
Owner pages: `serverAppConfigs-docs/Dankclaw/Open Notebook.md` (app-side shapes), and the endpoint's own
|
||||
page for whatever serves the model.
|
||||
Reference in new issue
Block a user