docs(operations): LLM usage-attribution research catalog

Deep-research reference for per-consumer tracking of the fleet endpoints:
measured baseline (no identity anywhere; frank-zealot /metrics 501), the
seven identity layers, engine-native/gateway/tracing/network catalogs with
citations, and a phased adoption plan.

serverAppConfigs-docs/Operations/LLM Usage Attribution.md

@@ -0,0 +1,199 @@
+---
+type: reference
+title: LLM Usage Attribution
+description: Research catalog of every viable way to answer "who is using the fleet LLMs" — engine-native identity, gateways, auth proxies, tracing platforms, network/host attribution — with the measured fleet baseline (2026-10-13) and a phased adoption plan.
+tags: [llm, observability, auth, gateway, litellm, tracking, identity]
+timestamp: 2026-10-14T00:00:00-07:00
+---
+
+# LLM Usage Attribution
+
+Deep-research catalog (2026-10-13/14): solutions for tracking WHO uses the
+fleet endpoints (`frank-archon`, `frank-dragoon`, `frank-zealot`). Full web
+citations; live-fleet measurements marked **(measured)**.
+
+## Baseline: what the fleet answers today
+
+- All three endpoints are `auth: none`, plain HTTP, LAN-only
+  (`models.yml` providers; no vhost on 7770/7771).
+- **(measured)** Live `/metrics` label census: `frank-archon` exports
+  `vllm:*`+`radiance:*` with labels only `{model_name, engine, le,
+  finished_reason, position, class}` — zero identity labels.
+  `frank-dragoon` exports `halogen:*`+`llamacpp:*`.
+  `frank-zealot` answers **501** — `serve-frank-zealot.sh` never passes
+  `--metrics`, so the Prometheus job is structurally never-green
+  (`LlmEndpointDown` fires). Fix: add `--metrics` to the ARGS array.
+- **(measured)** frank-zealot verbosity-4 journal lines carry slot/task/timing
+  only — no client IP, no user. Alloy ships journal + podman ctr.log to Loki,
+  so what lands there has no identity either.
+- Caddy has no `log` directive anywhere → no per-request access logs on the
+  OpenWebUI vhost.
+- Net: fleet-level token telemetry exists per ENDPOINT; per-CONSUMER
+  identity exists nowhere.
+
+## The identity layers (pick the layer, get its resolution)
+
+| # | Layer | Identity | Answers |
+|---|---|---|---|
+| L0 | DHCP/ARP | MAC, hostname, lease | which device was on the LAN |
+| L1 | DNS (Pi-hole v6) | device → queries | which device resolved/used a name |
+| L2 | L3/L4 (nftables/conntrack, JA3/JA4) | IP, TLS client lib | bytes per client, client software |
+| L3 | Transport (API keys, mTLS, SPIFFE, Tailscale) | key/cert/node | which credential connected |
+| L4 | App gateway (virtual keys → user/team/end_user) | named consumer | tokens/spend/cost per who |
+| L5 | Trace/content (Langfuse, Helicone, halogen records) | user_id, session, prompt text | what they asked, per session |
+| L6 | Client host (auditd, eBPF, netdata) | pid/uid/comm per connect | which process on which host |
+
+Keys/headers/`user` fields are CLIENT-DECLARED — they group traffic, they
+prove nothing about a human (halogen's own docs warn this verbatim:
+"they group requests, and they prove nothing about who sent them").
+
+## Catalog
+
+### A. Engine-native (per-consumer identity inside the inference server)
+
+| Engine | AuthN | Identity in logs | Per-identity metrics | OTel | Notes |
+|---|---|---|---|---|---|
+| llama.cpp | `--api-key` (multi, comma) / `--api-key-file` **(measured in the deployed build's `--help`)** | request `user` field drives slot assignment/eviction; `--log-requests` dumps bodies | `--metrics` exports `llamacpp:*`, NO key/user label | none upstream | [server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) |
+| vLLM | `--api-key` (multi) | request log; no user persistence | `vllm:*`; NO `api_key` label (cardinality, issue #53926) | native (`--otlp-traces-endpoint`) | [metrics design](https://github.com/vllm-project/vllm/blob/main/docs/design/metrics.md) |
+| radiance | no public auth docs **[unverified]** | — | **(measured)** no identity labels on live /metrics | none found | closed deployment surface; treat authN as absent until probed |
+| halogen-flash-server | `HALOGEN_API_KEY` / `HALOGEN_API_KEY_FILE` (multi, comma) | **`HALOGEN_RECORD_DIR` request records**: `turns/*.jsonl` per request — `client_family` (User-Agent), tokens, timings, finish, tool calls, plus HASHES of `user` / `prompt_cache_key` / `safety_identifier` / `metadata.user_id` (`.salt`-keyed); `units/` content-addressed text; caps `HALOGEN_RECORD_GIB`=2, `HALOGEN_RECORD_DAYS`=90; `HALOGEN_RECORD_TEXT=none` for metadata-only; `stats`/`prune` tools in-container | `halogen:*`+`llamacpp:*`, no per-key label | none | [configuration.md](https://github.com/peonist-ai/halogen-flash-server/blob/main/docs/configuration.md) — verbatim-verified 2026-10-13. Best native ledger of the fleet. |
+| SGLang | `--api-key` (single) | `--log-requests` | `sglang:*`, no key label | native (`--enable-trace`) | for any future swap |
+| Ollama | single key | request log | `ollama:` no key label | none | not in fleet |
+
+Wire fields usable from clients today (all client-declared): OpenAI
+`user`, `safety_identifier`, `prompt_cache_key`, Messages
+`metadata.user_id`; LiteLLM `end_user_id`; gateway header conventions
+(`X-Forwarded-User`, `Helicone-User-Id`, `cf-aig-metadata`).
+
+### B. LLM gateways (front door with virtual keys — the only layer with
+per-consumer TOKEN counts)
+
+| Solution | Identity | Usage lands | License | Footprint |
+|---|---|---|---|---|
+| **LiteLLM Proxy** | virtual keys → `user_id`/`team_id`/`end_user`; budgets, rate limits | Postgres spend ledger; `/metrics` labels `api_key_alias,user,team,end_user,model,client_ip,user_agent` (`litellm_*_tokens_metric`, spend, budget gauges); OTel/Langfuse callbacks | MIT core (SSO/RBAC/audit-log enterprise) | ~150–300 MiB + Postgres |
+| Kong AI Gateway | Consumer + key-auth/JWT/OIDC/mTLS; AI consumer groups | Postgres; `kong_*` per-consumer; file-log/zipkin/otel | OSS core; token-aware AI plugins enterprise | 0.5–1 GiB |
+| Apache APISIX | Consumer + key-auth/JWT | `apisix_llm_prompt_tokens`/`completion_tokens` with `consumer` label; http-logger/kafka | Apache-2.0 | 200–400 MiB |
+| Bifrost (maximhq) | virtual keys `sk-bf-*`/`x-bf-vk`, budgets | Prometheus/OTel, config store (SQLite/Postgres) | Apache-2.0, Go | ~150–300 MiB |
+| Portkey OSS gateway | virtual keys | logs/OTel | MIT | 200–400 MiB |
+| Higress AI GW | keys (+OIDC ent) | `higress_*`, ai-logging | Apache-2.0 CE | Istio-based, 0.5 GiB+ |
+| Envoy AI GW / Gateway API Inference Extension / LLM Gatekeeper | delegated auth | Envoy metrics/access logs | Apache-2.0 | K8s-native |
+| Gloo / KrakenD / Gravitee AI gateways | keys, OIDC | metrics/logs | OSS core, AI features enterprise | heavy |
+| Cloudflare AI Gateway | Access JWT `cf.user_id`, `cf-aig-metadata` | Cloudflare-managed analytics/logs | SaaS (exotic for LAN; Tunnel possible) | edge |
+
+([LiteLLM virtual keys](https://docs.litellm.ai/docs/proxy/virtual_keys),
+[prometheus](https://docs.litellm.ai/docs/proxy/prometheus),
+[APISIX ai-proxy](https://apisix.apache.org/docs/apisix/plugins/ai-proxy/),
+[Bifrost](https://github.com/maximhq/bifrost),
+[Kong AI](https://developer.konghq.com/index/ai-gateway/))
+
+### C. Auth proxies / identity infrastructure (identity without token counts)
+
+- **Ory Oathkeeper** (MIT, ~50–100 MiB): per-consumer api-key/JWT/
+  client-credentials → header mutation (`X-User: {{ .Subject }}`) +
+  decision logs.
+- **oauth2-proxy** (Apache-2.0): OIDC (Authentik) incl. client_credentials,
+  `--pass-user-headers` → `X-Forwarded-User`; ~100 MiB.
+- **Authentik itself**: proxy-provider custom headers (already injects
+  `X-Authentik-Username`/`-Groups`, spoof-stripped — the /whoami seam proves
+  the pattern on this host); event log + webhook forwarding.
+- **Caddy `tls client_auth`** (mTLS per client cert) + access-log fields —
+  Caddy already fronts both hosts.
+- **Smallstep step-ca**: short-lived per-device certs, ACME-style.
+- **SPIFFE/SPIRE**: per-workload SVIDs (k8s-flavored; heavy for quadlets).
+- **Tailscale / NetBird**: per-node identity, ACL + network-flow logs,
+  headscale for self-hosted control. Best for off-LAN laptops.
+- **ApiAuth** (nginx auth_request): the minimal per-consumer apikey sidecar.
+
+### D. LLM observability / tracing (prompt-level per-user audit)
+
+| Platform | Identity | Content | Self-host | Shape |
+|---|---|---|---|---|
+| **Langfuse** | `user_id`,`session_id`, tags | full input/output, masking hook | ✅ ClickHouse, Apache-2.0 | fed via LiteLLM callback / OTLP / SDK |
+| **Helicone** | api-key→user, `Helicone-User-Id` | full, tiered redaction | ✅ compose (ClickHouse+PG+Redis+Jawn), Apache-2.0 | PROXY — can be the front door itself |
+| Arize Phoenix | `user.id`,`session.id` | opt-in `gen_ai.*` | ✅, Apache-2.0 | OTLP collector |
+| OpenLIT | `user.id`,`session.id` | 3-tier capture | ✅, Apache-2.0 | OTLP+ClickHouse |
+| SigNoz LLM | OTel attrs | opt-in | ✅, Apache-2.0 | OTLP |
+| OpenLLMetry/Traceloop | assoc props | opt-in | ✅ Helm | OTel |
+| Lunary / W&B Weave / LangSmith | user/session | full | enterprise-gated | — |
+
+OTel GenAI semconv (Development-stability): `gen_ai.operation.name`,
+`gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`/
+`output_tokens`, `gen_ai.conversation.id`, content attrs opt-in; `user.id`
+from the registry. vLLM and SGLang emit OTLP natively; llama.cpp only via
+external SDK — **the fleet's engines are OTel-silent at request level**,
+which is why the gateway callback (LiteLLM→Langfuse) is the practical feed.
+
+### E. Network/host passive attribution (no client cooperation)
+
+- **Pi-hole v6** (already on Dankclaw): query log API per device,
+  `pihole6_exporter` + query-logger→Loki (GPLv3 exporter).
+- **Caddy access logs**: add a `log` directive (JSON) → Alloy→Loki; with the
+  auth chain it carries `X-Authentik-Username` for browser users.
+- **OpenWebUI admin**: per-user daily token stats, events/logs, analytics API
+  (`/api/v1/analytics/*`) — per-USER identity exists here already (MIT).
+- **auditd** connect-syscall rules (uid/pid/comm per connect to 7770/7771),
+  ship via audisp→Loki.
+- **eBPF**: Tetragon (process+socket events, policies), Coroot, DeepFlow,
+  bpftrace `tracepoint:syscalls:sys_enter_connect` one-liners. K8s-shaped
+  for single hosts, but agents.plugin-style tools work.
+- **nftables named counters** per consumer IP (bytes; reload resets).
+- **netdata apps.plugin**: per-process network bytes, auto-discovered.
+- **JA3/JA4 TLS fingerprinting**: client library only, TLS-only; the LAN is
+  plaintext HTTP today → use the HTTP **User-Agent census** instead (free
+  from any access log).
+- **DHCP/ARP census** (leases + `ip neigh`): device roster.
+
+## Phased plan for this fleet
+
+**Phase 0 — repo-only, no new services**
+1. frank-zealot: add `--metrics` (fixes the measured 501 + the
+   structurally-red Prometheus job + the firing `LlmEndpointDown`).
+2. frank-dragoon: set `HALOGEN_API_KEY_FILE` (multi-key per consumer) +
+   mount `HALOGEN_RECORD_DIR` (start `HALOGEN_RECORD_TEXT=none`; flip to
+   `all` if prompt audit is wanted) + Alloy ship of `turns/`.
+3. frank-zealot: `--api-key-file` (multi-key, verified in the deployed
+   build); radiance: probe for `--api-key` support (contract facts are
+   silent — add to `fleet-probes`).
+4. models.yml: per-consumer bearer keys (`auth: none` → key) — the provider
+   schema supports apiKey (pi-era conversion note).
+5. Caddy: `log` on `ai.franksdank.cloud` (OpenWebUI browser census, JSON →
+   Loki).
+   Limits: keys gate access, and only the engines' request records (halogen)
+   see them per-request; llama.cpp/radiance/vLLM metrics stay identity-blind.
+
+**Phase 1 — LiteLLM Proxy quadlet on Dankclaw = the front door.**
+One virtual key per consumer (omp-<workstation> ×N, openwebui, hermes,
+agents, bench). Per-key/user/team Prometheus labels into the existing
+Grafana+Alertmanager; Postgres spend ledger; budgets + rate limits per
+key. This is the ONLY layer that yields tokens-per-who without engine
+cooperation. (Bifrost is the lighter Apache-2.0 Go alternative; Helicone
+the proxy-native-observability alternative.)
+
+**Phase 2 — prompt-level audit.** Langfuse self-hosted fed by the LiteLLM
+callback: per-user traces, sessions, cost; masking + retention knobs.
+
+**Phase 3 — ambitious identity infra.** step-ca + Caddy mTLS per device;
+Tailscale (+flow logs) for off-LAN; auditd/eBPF process attribution on the
+agent hosts; Pi-hole query logger; nft counters per consumer IP. Browser
+users stay owned by OpenWebUI analytics + its per-user API keys; a pipeline
+can forward `X-Forwarded-User` so the gateway sees them too.
+
+## Cautions
+
+- Client-declared identity (keys, `user`, headers) groups traffic; it does
+  not authenticate humans. Authenticated identity = Authentik (browser) /
+  mTLS (device) / Tailscale (node).
+- Plain-HTTP LAN = bearer keys sniffable by any tenant; treat keys as
+  identity labels on the LAN, not secrets, until TLS/mTLS at the door.
+- Prompt-content logging stores a plaintext copy of everything — set
+  retention caps (halogen GIB/days; Langfuse masking; Loki retention) and
+  gitignore the record dirs.
+- Household-scale cardinality is safe for per-key Prometheus labels;
+  the engines' refusal of key labels is a multi-tenant concern, not ours.
+
+## Related
+- [[Fleet Audit and Posture]] — monitoring gaps census
+- [Observability](../Infrastructure/Observability.md) — the Loki/Prometheus
+  pipes every sink above uses
+- [LLM index](../LLM/index.md) — endpoint runbooks (env trios the Phase-0
+  edits touch)

serverAppConfigs-docs/Operations/index.md

@@ -23,6 +23,9 @@ Procedures for operating the fleet. The LLM-specific pages live under
   posture, whole-fleet audit, bulk redaction, stash audits
 - [[SSH and Remote Access]] — passwordless bootstrap without sshpass, fish
   login shells, GUI-hang diagnosis over ssh
+- [[LLM Usage Attribution]] — who-is-using-the-fleet-LLMs research catalog:
+  engine-native identity, gateways, tracing, network/host attribution, and
+  the phased plan
 
 ## Related
 - [Observability Provisioning](../Infrastructure/Observability%20Provisioning.md)

serverAppConfigs-docs/index.md

@@ -134,6 +134,9 @@ skill-cluster. See [log.md](log.md) for the wave plan and results.
   posture, whole-fleet audit, bulk redaction
 - [[SSH and Remote Access]] — passwordless bootstrap, fish login shells,
   GUI-hang diagnosis over ssh
+- [[LLM Usage Attribution]] — who-is-using-the-fleet-LLMs research catalog:
+  engine-native identity, gateways, tracing, network/host attribution,
+  phased plan
 
 **Architecture:**