# AGENTS.md — serverAppConfigs Agent Rules

## Overview

This repository tracks the Podman container stacks for two homelab servers. Dankserver runs the media/library stack (RomM, Copyparty, Calibre-Web). Dankclaw runs the AI/search/development stack (SearXNG via Gluetun, ComfyUI, OpenWebUI, Hermes, Gitea, Grafana, Prometheus) plus the observability stack. Caddy reverse-proxies everything on both hosts.

## Git Workflow

All config edits flow through git. Do not edit files on the remote machine directly.

### Protocol

1. **Source the config from the target machine** — fetch the live compose/Caddyfile from the server where it runs
2. **Place it in this repo** — commit the fetched version (redact secrets into `.env`)
3. **Edit here** — make all changes locally in the working tree
4. **Run the tests** — `./tests/run.sh` must pass
5. **Commit and push** — push to origin (Gitea on dankclaw)
6. **Pull on the target** — `git pull` on the remote machine
7. **Restart the stack** — use `scripts/deploy.sh` to pull + restart + healthcheck

## Deployment

Use `scripts/deploy.sh` to deploy committed changes to a target.

```
./scripts/deploy.sh dankclaw        # Dankclaw stack
./scripts/deploy.sh dankserver      # Dankserver stack
./scripts/deploy.sh dankclaw --dry-run   # preview without changes
```

## Testing

Tests are mandatory before commit. `bats` must be installed on the development machine.

```
./tests/run.sh
```

Test seams: compose validity, env consistency (no leaked secrets), Caddyfile validity, docs completeness, script syntax, deployment flow. See [docs/Architecture/Testing Policy.md](docs/Architecture/Testing Policy.md).

## Secret Rules

- Never commit real secrets. The live Gitea database password is test-enforced as a leak check.
- `.env` files are git-ignored; commit only `.env.example` with placeholders.
- Do not document passwords in this repo, including sudo passwords. SSH keys are the auth mechanism.

## Runtime & Troubleshooting

- The LLM endpoints are bare-metal systemd services managed by the sibling `vllmScripts` repository (`../vllmScripts`), not by this repo: llama.cpp PP head `:7770` and FastFlowLM NPU `:52625` on Dankclaw, RPC worker `:50052` on Dankserver. Do not edit their scripts here.
- Model load on the LLM servers takes 3–8 minutes; `503 Loading model` is normal during load. Do not kill the process.
- `sudo` on both boxes needs a tty. Do not use `echo password | sudo -S` in committed scripts; run sudo interactively.
- Never run `pkill -f` with a pattern that appears in your own command line. Use `pkill -x <exact-name>`.
- ComfyUI on Dankclaw runs CPU-only (`--cpu`); ROCm wheels do not support gfx1151. Do not use CUDA images.
- SearXNG outbound traffic must stay on the Gluetun tunnel; do not move its ports off the gluetun service.

## Why

Direct edits on the remote drift from version control. Future agents will not know what changed. Git is the single source of truth.

## Summary

| Context | Action |
|---------|--------|
| Editing a compose/Caddyfile | **Work through git** — fetch, commit, edit, test, push, pull, restart. Never edit on the remote directly. |
| Config exists only on remote | **Pull it into the repo first** — commit the fetched version (redacted) before making changes. |
| Unstaged local changes | **Commit or discard** — check `git status` before starting work. Ask the user if unsure. |
| Deploying a committed change | **Run `scripts/deploy.sh <target>`** — it pulls on the target, restarts the stack, and checks health. |
| Tests exist | **Run them** — verify tests pass after every change. Do not commit failing tests. |
| Real secrets | **Never commit** — use host `.env` files; `.env.example` holds placeholders only. |