diff --git a/serverAppConfigs-docs/Operations/LLM Usage Attribution.md b/serverAppConfigs-docs/Operations/LLM Usage Attribution.md new file mode 100644 index 0000000..b69a3ad --- /dev/null +++ b/serverAppConfigs-docs/Operations/LLM Usage Attribution.md @@ -0,0 +1,199 @@ +--- +type: reference +title: LLM Usage Attribution +description: Research catalog of every viable way to answer "who is using the fleet LLMs" — engine-native identity, gateways, auth proxies, tracing platforms, network/host attribution — with the measured fleet baseline (2026-10-13) and a phased adoption plan. +tags: [llm, observability, auth, gateway, litellm, tracking, identity] +timestamp: 2026-10-14T00:00:00-07:00 +--- + +# LLM Usage Attribution + +Deep-research catalog (2026-10-13/14): solutions for tracking WHO uses the +fleet endpoints (`frank-archon`, `frank-dragoon`, `frank-zealot`). Full web +citations; live-fleet measurements marked **(measured)**. + +## Baseline: what the fleet answers today + +- All three endpoints are `auth: none`, plain HTTP, LAN-only + (`models.yml` providers; no vhost on 7770/7771). +- **(measured)** Live `/metrics` label census: `frank-archon` exports + `vllm:*`+`radiance:*` with labels only `{model_name, engine, le, + finished_reason, position, class}` — zero identity labels. + `frank-dragoon` exports `halogen:*`+`llamacpp:*`. + `frank-zealot` answers **501** — `serve-frank-zealot.sh` never passes + `--metrics`, so the Prometheus job is structurally never-green + (`LlmEndpointDown` fires). Fix: add `--metrics` to the ARGS array. +- **(measured)** frank-zealot verbosity-4 journal lines carry slot/task/timing + only — no client IP, no user. Alloy ships journal + podman ctr.log to Loki, + so what lands there has no identity either. +- Caddy has no `log` directive anywhere → no per-request access logs on the + OpenWebUI vhost. +- Net: fleet-level token telemetry exists per ENDPOINT; per-CONSUMER + identity exists nowhere. + +## The identity layers (pick the layer, get its resolution) + +| # | Layer | Identity | Answers | +|---|---|---|---| +| L0 | DHCP/ARP | MAC, hostname, lease | which device was on the LAN | +| L1 | DNS (Pi-hole v6) | device → queries | which device resolved/used a name | +| L2 | L3/L4 (nftables/conntrack, JA3/JA4) | IP, TLS client lib | bytes per client, client software | +| L3 | Transport (API keys, mTLS, SPIFFE, Tailscale) | key/cert/node | which credential connected | +| L4 | App gateway (virtual keys → user/team/end_user) | named consumer | tokens/spend/cost per who | +| L5 | Trace/content (Langfuse, Helicone, halogen records) | user_id, session, prompt text | what they asked, per session | +| L6 | Client host (auditd, eBPF, netdata) | pid/uid/comm per connect | which process on which host | + +Keys/headers/`user` fields are CLIENT-DECLARED — they group traffic, they +prove nothing about a human (halogen's own docs warn this verbatim: +"they group requests, and they prove nothing about who sent them"). + +## Catalog + +### A. Engine-native (per-consumer identity inside the inference server) + +| Engine | AuthN | Identity in logs | Per-identity metrics | OTel | Notes | +|---|---|---|---|---|---| +| llama.cpp | `--api-key` (multi, comma) / `--api-key-file` **(measured in the deployed build's `--help`)** | request `user` field drives slot assignment/eviction; `--log-requests` dumps bodies | `--metrics` exports `llamacpp:*`, NO key/user label | none upstream | [server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) | +| vLLM | `--api-key` (multi) | request log; no user persistence | `vllm:*`; NO `api_key` label (cardinality, issue #53926) | native (`--otlp-traces-endpoint`) | [metrics design](https://github.com/vllm-project/vllm/blob/main/docs/design/metrics.md) | +| radiance | no public auth docs **[unverified]** | — | **(measured)** no identity labels on live /metrics | none found | closed deployment surface; treat authN as absent until probed | +| halogen-flash-server | `HALOGEN_API_KEY` / `HALOGEN_API_KEY_FILE` (multi, comma) | **`HALOGEN_RECORD_DIR` request records**: `turns/*.jsonl` per request — `client_family` (User-Agent), tokens, timings, finish, tool calls, plus HASHES of `user` / `prompt_cache_key` / `safety_identifier` / `metadata.user_id` (`.salt`-keyed); `units/` content-addressed text; caps `HALOGEN_RECORD_GIB`=2, `HALOGEN_RECORD_DAYS`=90; `HALOGEN_RECORD_TEXT=none` for metadata-only; `stats`/`prune` tools in-container | `halogen:*`+`llamacpp:*`, no per-key label | none | [configuration.md](https://github.com/peonist-ai/halogen-flash-server/blob/main/docs/configuration.md) — verbatim-verified 2026-10-13. Best native ledger of the fleet. | +| SGLang | `--api-key` (single) | `--log-requests` | `sglang:*`, no key label | native (`--enable-trace`) | for any future swap | +| Ollama | single key | request log | `ollama:` no key label | none | not in fleet | + +Wire fields usable from clients today (all client-declared): OpenAI +`user`, `safety_identifier`, `prompt_cache_key`, Messages +`metadata.user_id`; LiteLLM `end_user_id`; gateway header conventions +(`X-Forwarded-User`, `Helicone-User-Id`, `cf-aig-metadata`). + +### B. LLM gateways (front door with virtual keys — the only layer with +per-consumer TOKEN counts) + +| Solution | Identity | Usage lands | License | Footprint | +|---|---|---|---|---| +| **LiteLLM Proxy** | virtual keys → `user_id`/`team_id`/`end_user`; budgets, rate limits | Postgres spend ledger; `/metrics` labels `api_key_alias,user,team,end_user,model,client_ip,user_agent` (`litellm_*_tokens_metric`, spend, budget gauges); OTel/Langfuse callbacks | MIT core (SSO/RBAC/audit-log enterprise) | ~150–300 MiB + Postgres | +| Kong AI Gateway | Consumer + key-auth/JWT/OIDC/mTLS; AI consumer groups | Postgres; `kong_*` per-consumer; file-log/zipkin/otel | OSS core; token-aware AI plugins enterprise | 0.5–1 GiB | +| Apache APISIX | Consumer + key-auth/JWT | `apisix_llm_prompt_tokens`/`completion_tokens` with `consumer` label; http-logger/kafka | Apache-2.0 | 200–400 MiB | +| Bifrost (maximhq) | virtual keys `sk-bf-*`/`x-bf-vk`, budgets | Prometheus/OTel, config store (SQLite/Postgres) | Apache-2.0, Go | ~150–300 MiB | +| Portkey OSS gateway | virtual keys | logs/OTel | MIT | 200–400 MiB | +| Higress AI GW | keys (+OIDC ent) | `higress_*`, ai-logging | Apache-2.0 CE | Istio-based, 0.5 GiB+ | +| Envoy AI GW / Gateway API Inference Extension / LLM Gatekeeper | delegated auth | Envoy metrics/access logs | Apache-2.0 | K8s-native | +| Gloo / KrakenD / Gravitee AI gateways | keys, OIDC | metrics/logs | OSS core, AI features enterprise | heavy | +| Cloudflare AI Gateway | Access JWT `cf.user_id`, `cf-aig-metadata` | Cloudflare-managed analytics/logs | SaaS (exotic for LAN; Tunnel possible) | edge | + +([LiteLLM virtual keys](https://docs.litellm.ai/docs/proxy/virtual_keys), +[prometheus](https://docs.litellm.ai/docs/proxy/prometheus), +[APISIX ai-proxy](https://apisix.apache.org/docs/apisix/plugins/ai-proxy/), +[Bifrost](https://github.com/maximhq/bifrost), +[Kong AI](https://developer.konghq.com/index/ai-gateway/)) + +### C. Auth proxies / identity infrastructure (identity without token counts) + +- **Ory Oathkeeper** (MIT, ~50–100 MiB): per-consumer api-key/JWT/ + client-credentials → header mutation (`X-User: {{ .Subject }}`) + + decision logs. +- **oauth2-proxy** (Apache-2.0): OIDC (Authentik) incl. client_credentials, + `--pass-user-headers` → `X-Forwarded-User`; ~100 MiB. +- **Authentik itself**: proxy-provider custom headers (already injects + `X-Authentik-Username`/`-Groups`, spoof-stripped — the /whoami seam proves + the pattern on this host); event log + webhook forwarding. +- **Caddy `tls client_auth`** (mTLS per client cert) + access-log fields — + Caddy already fronts both hosts. +- **Smallstep step-ca**: short-lived per-device certs, ACME-style. +- **SPIFFE/SPIRE**: per-workload SVIDs (k8s-flavored; heavy for quadlets). +- **Tailscale / NetBird**: per-node identity, ACL + network-flow logs, + headscale for self-hosted control. Best for off-LAN laptops. +- **ApiAuth** (nginx auth_request): the minimal per-consumer apikey sidecar. + +### D. LLM observability / tracing (prompt-level per-user audit) + +| Platform | Identity | Content | Self-host | Shape | +|---|---|---|---|---| +| **Langfuse** | `user_id`,`session_id`, tags | full input/output, masking hook | ✅ ClickHouse, Apache-2.0 | fed via LiteLLM callback / OTLP / SDK | +| **Helicone** | api-key→user, `Helicone-User-Id` | full, tiered redaction | ✅ compose (ClickHouse+PG+Redis+Jawn), Apache-2.0 | PROXY — can be the front door itself | +| Arize Phoenix | `user.id`,`session.id` | opt-in `gen_ai.*` | ✅, Apache-2.0 | OTLP collector | +| OpenLIT | `user.id`,`session.id` | 3-tier capture | ✅, Apache-2.0 | OTLP+ClickHouse | +| SigNoz LLM | OTel attrs | opt-in | ✅, Apache-2.0 | OTLP | +| OpenLLMetry/Traceloop | assoc props | opt-in | ✅ Helm | OTel | +| Lunary / W&B Weave / LangSmith | user/session | full | enterprise-gated | — | + +OTel GenAI semconv (Development-stability): `gen_ai.operation.name`, +`gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`/ +`output_tokens`, `gen_ai.conversation.id`, content attrs opt-in; `user.id` +from the registry. vLLM and SGLang emit OTLP natively; llama.cpp only via +external SDK — **the fleet's engines are OTel-silent at request level**, +which is why the gateway callback (LiteLLM→Langfuse) is the practical feed. + +### E. Network/host passive attribution (no client cooperation) + +- **Pi-hole v6** (already on Dankclaw): query log API per device, + `pihole6_exporter` + query-logger→Loki (GPLv3 exporter). +- **Caddy access logs**: add a `log` directive (JSON) → Alloy→Loki; with the + auth chain it carries `X-Authentik-Username` for browser users. +- **OpenWebUI admin**: per-user daily token stats, events/logs, analytics API + (`/api/v1/analytics/*`) — per-USER identity exists here already (MIT). +- **auditd** connect-syscall rules (uid/pid/comm per connect to 7770/7771), + ship via audisp→Loki. +- **eBPF**: Tetragon (process+socket events, policies), Coroot, DeepFlow, + bpftrace `tracepoint:syscalls:sys_enter_connect` one-liners. K8s-shaped + for single hosts, but agents.plugin-style tools work. +- **nftables named counters** per consumer IP (bytes; reload resets). +- **netdata apps.plugin**: per-process network bytes, auto-discovered. +- **JA3/JA4 TLS fingerprinting**: client library only, TLS-only; the LAN is + plaintext HTTP today → use the HTTP **User-Agent census** instead (free + from any access log). +- **DHCP/ARP census** (leases + `ip neigh`): device roster. + +## Phased plan for this fleet + +**Phase 0 — repo-only, no new services** +1. frank-zealot: add `--metrics` (fixes the measured 501 + the + structurally-red Prometheus job + the firing `LlmEndpointDown`). +2. frank-dragoon: set `HALOGEN_API_KEY_FILE` (multi-key per consumer) + + mount `HALOGEN_RECORD_DIR` (start `HALOGEN_RECORD_TEXT=none`; flip to + `all` if prompt audit is wanted) + Alloy ship of `turns/`. +3. frank-zealot: `--api-key-file` (multi-key, verified in the deployed + build); radiance: probe for `--api-key` support (contract facts are + silent — add to `fleet-probes`). +4. models.yml: per-consumer bearer keys (`auth: none` → key) — the provider + schema supports apiKey (pi-era conversion note). +5. Caddy: `log` on `ai.franksdank.cloud` (OpenWebUI browser census, JSON → + Loki). + Limits: keys gate access, and only the engines' request records (halogen) + see them per-request; llama.cpp/radiance/vLLM metrics stay identity-blind. + +**Phase 1 — LiteLLM Proxy quadlet on Dankclaw = the front door.** +One virtual key per consumer (omp- ×N, openwebui, hermes, +agents, bench). Per-key/user/team Prometheus labels into the existing +Grafana+Alertmanager; Postgres spend ledger; budgets + rate limits per +key. This is the ONLY layer that yields tokens-per-who without engine +cooperation. (Bifrost is the lighter Apache-2.0 Go alternative; Helicone +the proxy-native-observability alternative.) + +**Phase 2 — prompt-level audit.** Langfuse self-hosted fed by the LiteLLM +callback: per-user traces, sessions, cost; masking + retention knobs. + +**Phase 3 — ambitious identity infra.** step-ca + Caddy mTLS per device; +Tailscale (+flow logs) for off-LAN; auditd/eBPF process attribution on the +agent hosts; Pi-hole query logger; nft counters per consumer IP. Browser +users stay owned by OpenWebUI analytics + its per-user API keys; a pipeline +can forward `X-Forwarded-User` so the gateway sees them too. + +## Cautions + +- Client-declared identity (keys, `user`, headers) groups traffic; it does + not authenticate humans. Authenticated identity = Authentik (browser) / + mTLS (device) / Tailscale (node). +- Plain-HTTP LAN = bearer keys sniffable by any tenant; treat keys as + identity labels on the LAN, not secrets, until TLS/mTLS at the door. +- Prompt-content logging stores a plaintext copy of everything — set + retention caps (halogen GIB/days; Langfuse masking; Loki retention) and + gitignore the record dirs. +- Household-scale cardinality is safe for per-key Prometheus labels; + the engines' refusal of key labels is a multi-tenant concern, not ours. + +## Related +- [[Fleet Audit and Posture]] — monitoring gaps census +- [Observability](../Infrastructure/Observability.md) — the Loki/Prometheus + pipes every sink above uses +- [LLM index](../LLM/index.md) — endpoint runbooks (env trios the Phase-0 + edits touch) diff --git a/serverAppConfigs-docs/Operations/index.md b/serverAppConfigs-docs/Operations/index.md index ec58fa1..d9ce3eb 100644 --- a/serverAppConfigs-docs/Operations/index.md +++ b/serverAppConfigs-docs/Operations/index.md @@ -23,6 +23,9 @@ Procedures for operating the fleet. The LLM-specific pages live under posture, whole-fleet audit, bulk redaction, stash audits - [[SSH and Remote Access]] — passwordless bootstrap without sshpass, fish login shells, GUI-hang diagnosis over ssh +- [[LLM Usage Attribution]] — who-is-using-the-fleet-LLMs research catalog: + engine-native identity, gateways, tracing, network/host attribution, and + the phased plan ## Related - [Observability Provisioning](../Infrastructure/Observability%20Provisioning.md) diff --git a/serverAppConfigs-docs/index.md b/serverAppConfigs-docs/index.md index c56d0f9..162fc49 100644 --- a/serverAppConfigs-docs/index.md +++ b/serverAppConfigs-docs/index.md @@ -134,6 +134,9 @@ skill-cluster. See [log.md](log.md) for the wave plan and results. posture, whole-fleet audit, bulk redaction - [[SSH and Remote Access]] — passwordless bootstrap, fish login shells, GUI-hang diagnosis over ssh +- [[LLM Usage Attribution]] — who-is-using-the-fleet-LLMs research catalog: + engine-native identity, gateways, tracing, network/host attribution, + phased plan **Architecture:**