docs(operations): LLM usage-attribution research catalog
Deep-research reference for per-consumer tracking of the fleet endpoints: measured baseline (no identity anywhere; frank-zealot /metrics 501), the seven identity layers, engine-native/gateway/tracing/network catalogs with citations, and a phased adoption plan.
This commit is contained in:
Frank
@@ -0,0 +1,199 @@
|
||||
---
|
||||
type: reference
|
||||
title: LLM Usage Attribution
|
||||
description: Research catalog of every viable way to answer "who is using the fleet LLMs" — engine-native identity, gateways, auth proxies, tracing platforms, network/host attribution — with the measured fleet baseline (2026-10-13) and a phased adoption plan.
|
||||
tags: [llm, observability, auth, gateway, litellm, tracking, identity]
|
||||
timestamp: 2026-10-14T00:00:00-07:00
|
||||
---
|
||||
|
||||
# LLM Usage Attribution
|
||||
|
||||
Deep-research catalog (2026-10-13/14): solutions for tracking WHO uses the
|
||||
fleet endpoints (`frank-archon`, `frank-dragoon`, `frank-zealot`). Full web
|
||||
citations; live-fleet measurements marked **(measured)**.
|
||||
|
||||
## Baseline: what the fleet answers today
|
||||
|
||||
- All three endpoints are `auth: none`, plain HTTP, LAN-only
|
||||
(`models.yml` providers; no vhost on 7770/7771).
|
||||
- **(measured)** Live `/metrics` label census: `frank-archon` exports
|
||||
`vllm:*`+`radiance:*` with labels only `{model_name, engine, le,
|
||||
finished_reason, position, class}` — zero identity labels.
|
||||
`frank-dragoon` exports `halogen:*`+`llamacpp:*`.
|
||||
`frank-zealot` answers **501** — `serve-frank-zealot.sh` never passes
|
||||
`--metrics`, so the Prometheus job is structurally never-green
|
||||
(`LlmEndpointDown` fires). Fix: add `--metrics` to the ARGS array.
|
||||
- **(measured)** frank-zealot verbosity-4 journal lines carry slot/task/timing
|
||||
only — no client IP, no user. Alloy ships journal + podman ctr.log to Loki,
|
||||
so what lands there has no identity either.
|
||||
- Caddy has no `log` directive anywhere → no per-request access logs on the
|
||||
OpenWebUI vhost.
|
||||
- Net: fleet-level token telemetry exists per ENDPOINT; per-CONSUMER
|
||||
identity exists nowhere.
|
||||
|
||||
## The identity layers (pick the layer, get its resolution)
|
||||
|
||||
| # | Layer | Identity | Answers |
|
||||
|---|---|---|---|
|
||||
| L0 | DHCP/ARP | MAC, hostname, lease | which device was on the LAN |
|
||||
| L1 | DNS (Pi-hole v6) | device → queries | which device resolved/used a name |
|
||||
| L2 | L3/L4 (nftables/conntrack, JA3/JA4) | IP, TLS client lib | bytes per client, client software |
|
||||
| L3 | Transport (API keys, mTLS, SPIFFE, Tailscale) | key/cert/node | which credential connected |
|
||||
| L4 | App gateway (virtual keys → user/team/end_user) | named consumer | tokens/spend/cost per who |
|
||||
| L5 | Trace/content (Langfuse, Helicone, halogen records) | user_id, session, prompt text | what they asked, per session |
|
||||
| L6 | Client host (auditd, eBPF, netdata) | pid/uid/comm per connect | which process on which host |
|
||||
|
||||
Keys/headers/`user` fields are CLIENT-DECLARED — they group traffic, they
|
||||
prove nothing about a human (halogen's own docs warn this verbatim:
|
||||
"they group requests, and they prove nothing about who sent them").
|
||||
|
||||
## Catalog
|
||||
|
||||
### A. Engine-native (per-consumer identity inside the inference server)
|
||||
|
||||
| Engine | AuthN | Identity in logs | Per-identity metrics | OTel | Notes |
|
||||
|---|---|---|---|---|---|
|
||||
| llama.cpp | `--api-key` (multi, comma) / `--api-key-file` **(measured in the deployed build's `--help`)** | request `user` field drives slot assignment/eviction; `--log-requests` dumps bodies | `--metrics` exports `llamacpp:*`, NO key/user label | none upstream | [server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) |
|
||||
| vLLM | `--api-key` (multi) | request log; no user persistence | `vllm:*`; NO `api_key` label (cardinality, issue #53926) | native (`--otlp-traces-endpoint`) | [metrics design](https://github.com/vllm-project/vllm/blob/main/docs/design/metrics.md) |
|
||||
| radiance | no public auth docs **[unverified]** | — | **(measured)** no identity labels on live /metrics | none found | closed deployment surface; treat authN as absent until probed |
|
||||
| halogen-flash-server | `HALOGEN_API_KEY` / `HALOGEN_API_KEY_FILE` (multi, comma) | **`HALOGEN_RECORD_DIR` request records**: `turns/*.jsonl` per request — `client_family` (User-Agent), tokens, timings, finish, tool calls, plus HASHES of `user` / `prompt_cache_key` / `safety_identifier` / `metadata.user_id` (`.salt`-keyed); `units/` content-addressed text; caps `HALOGEN_RECORD_GIB`=2, `HALOGEN_RECORD_DAYS`=90; `HALOGEN_RECORD_TEXT=none` for metadata-only; `stats`/`prune` tools in-container | `halogen:*`+`llamacpp:*`, no per-key label | none | [configuration.md](https://github.com/peonist-ai/halogen-flash-server/blob/main/docs/configuration.md) — verbatim-verified 2026-10-13. Best native ledger of the fleet. |
|
||||
| SGLang | `--api-key` (single) | `--log-requests` | `sglang:*`, no key label | native (`--enable-trace`) | for any future swap |
|
||||
| Ollama | single key | request log | `ollama:` no key label | none | not in fleet |
|
||||
|
||||
Wire fields usable from clients today (all client-declared): OpenAI
|
||||
`user`, `safety_identifier`, `prompt_cache_key`, Messages
|
||||
`metadata.user_id`; LiteLLM `end_user_id`; gateway header conventions
|
||||
(`X-Forwarded-User`, `Helicone-User-Id`, `cf-aig-metadata`).
|
||||
|
||||
### B. LLM gateways (front door with virtual keys — the only layer with
|
||||
per-consumer TOKEN counts)
|
||||
|
||||
| Solution | Identity | Usage lands | License | Footprint |
|
||||
|---|---|---|---|---|
|
||||
| **LiteLLM Proxy** | virtual keys → `user_id`/`team_id`/`end_user`; budgets, rate limits | Postgres spend ledger; `/metrics` labels `api_key_alias,user,team,end_user,model,client_ip,user_agent` (`litellm_*_tokens_metric`, spend, budget gauges); OTel/Langfuse callbacks | MIT core (SSO/RBAC/audit-log enterprise) | ~150–300 MiB + Postgres |
|
||||
| Kong AI Gateway | Consumer + key-auth/JWT/OIDC/mTLS; AI consumer groups | Postgres; `kong_*` per-consumer; file-log/zipkin/otel | OSS core; token-aware AI plugins enterprise | 0.5–1 GiB |
|
||||
| Apache APISIX | Consumer + key-auth/JWT | `apisix_llm_prompt_tokens`/`completion_tokens` with `consumer` label; http-logger/kafka | Apache-2.0 | 200–400 MiB |
|
||||
| Bifrost (maximhq) | virtual keys `sk-bf-*`/`x-bf-vk`, budgets | Prometheus/OTel, config store (SQLite/Postgres) | Apache-2.0, Go | ~150–300 MiB |
|
||||
| Portkey OSS gateway | virtual keys | logs/OTel | MIT | 200–400 MiB |
|
||||
| Higress AI GW | keys (+OIDC ent) | `higress_*`, ai-logging | Apache-2.0 CE | Istio-based, 0.5 GiB+ |
|
||||
| Envoy AI GW / Gateway API Inference Extension / LLM Gatekeeper | delegated auth | Envoy metrics/access logs | Apache-2.0 | K8s-native |
|
||||
| Gloo / KrakenD / Gravitee AI gateways | keys, OIDC | metrics/logs | OSS core, AI features enterprise | heavy |
|
||||
| Cloudflare AI Gateway | Access JWT `cf.user_id`, `cf-aig-metadata` | Cloudflare-managed analytics/logs | SaaS (exotic for LAN; Tunnel possible) | edge |
|
||||
|
||||
([LiteLLM virtual keys](https://docs.litellm.ai/docs/proxy/virtual_keys),
|
||||
[prometheus](https://docs.litellm.ai/docs/proxy/prometheus),
|
||||
[APISIX ai-proxy](https://apisix.apache.org/docs/apisix/plugins/ai-proxy/),
|
||||
[Bifrost](https://github.com/maximhq/bifrost),
|
||||
[Kong AI](https://developer.konghq.com/index/ai-gateway/))
|
||||
|
||||
### C. Auth proxies / identity infrastructure (identity without token counts)
|
||||
|
||||
- **Ory Oathkeeper** (MIT, ~50–100 MiB): per-consumer api-key/JWT/
|
||||
client-credentials → header mutation (`X-User: {{ .Subject }}`) +
|
||||
decision logs.
|
||||
- **oauth2-proxy** (Apache-2.0): OIDC (Authentik) incl. client_credentials,
|
||||
`--pass-user-headers` → `X-Forwarded-User`; ~100 MiB.
|
||||
- **Authentik itself**: proxy-provider custom headers (already injects
|
||||
`X-Authentik-Username`/`-Groups`, spoof-stripped — the /whoami seam proves
|
||||
the pattern on this host); event log + webhook forwarding.
|
||||
- **Caddy `tls client_auth`** (mTLS per client cert) + access-log fields —
|
||||
Caddy already fronts both hosts.
|
||||
- **Smallstep step-ca**: short-lived per-device certs, ACME-style.
|
||||
- **SPIFFE/SPIRE**: per-workload SVIDs (k8s-flavored; heavy for quadlets).
|
||||
- **Tailscale / NetBird**: per-node identity, ACL + network-flow logs,
|
||||
headscale for self-hosted control. Best for off-LAN laptops.
|
||||
- **ApiAuth** (nginx auth_request): the minimal per-consumer apikey sidecar.
|
||||
|
||||
### D. LLM observability / tracing (prompt-level per-user audit)
|
||||
|
||||
| Platform | Identity | Content | Self-host | Shape |
|
||||
|---|---|---|---|---|
|
||||
| **Langfuse** | `user_id`,`session_id`, tags | full input/output, masking hook | ✅ ClickHouse, Apache-2.0 | fed via LiteLLM callback / OTLP / SDK |
|
||||
| **Helicone** | api-key→user, `Helicone-User-Id` | full, tiered redaction | ✅ compose (ClickHouse+PG+Redis+Jawn), Apache-2.0 | PROXY — can be the front door itself |
|
||||
| Arize Phoenix | `user.id`,`session.id` | opt-in `gen_ai.*` | ✅, Apache-2.0 | OTLP collector |
|
||||
| OpenLIT | `user.id`,`session.id` | 3-tier capture | ✅, Apache-2.0 | OTLP+ClickHouse |
|
||||
| SigNoz LLM | OTel attrs | opt-in | ✅, Apache-2.0 | OTLP |
|
||||
| OpenLLMetry/Traceloop | assoc props | opt-in | ✅ Helm | OTel |
|
||||
| Lunary / W&B Weave / LangSmith | user/session | full | enterprise-gated | — |
|
||||
|
||||
OTel GenAI semconv (Development-stability): `gen_ai.operation.name`,
|
||||
`gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`/
|
||||
`output_tokens`, `gen_ai.conversation.id`, content attrs opt-in; `user.id`
|
||||
from the registry. vLLM and SGLang emit OTLP natively; llama.cpp only via
|
||||
external SDK — **the fleet's engines are OTel-silent at request level**,
|
||||
which is why the gateway callback (LiteLLM→Langfuse) is the practical feed.
|
||||
|
||||
### E. Network/host passive attribution (no client cooperation)
|
||||
|
||||
- **Pi-hole v6** (already on Dankclaw): query log API per device,
|
||||
`pihole6_exporter` + query-logger→Loki (GPLv3 exporter).
|
||||
- **Caddy access logs**: add a `log` directive (JSON) → Alloy→Loki; with the
|
||||
auth chain it carries `X-Authentik-Username` for browser users.
|
||||
- **OpenWebUI admin**: per-user daily token stats, events/logs, analytics API
|
||||
(`/api/v1/analytics/*`) — per-USER identity exists here already (MIT).
|
||||
- **auditd** connect-syscall rules (uid/pid/comm per connect to 7770/7771),
|
||||
ship via audisp→Loki.
|
||||
- **eBPF**: Tetragon (process+socket events, policies), Coroot, DeepFlow,
|
||||
bpftrace `tracepoint:syscalls:sys_enter_connect` one-liners. K8s-shaped
|
||||
for single hosts, but agents.plugin-style tools work.
|
||||
- **nftables named counters** per consumer IP (bytes; reload resets).
|
||||
- **netdata apps.plugin**: per-process network bytes, auto-discovered.
|
||||
- **JA3/JA4 TLS fingerprinting**: client library only, TLS-only; the LAN is
|
||||
plaintext HTTP today → use the HTTP **User-Agent census** instead (free
|
||||
from any access log).
|
||||
- **DHCP/ARP census** (leases + `ip neigh`): device roster.
|
||||
|
||||
## Phased plan for this fleet
|
||||
|
||||
**Phase 0 — repo-only, no new services**
|
||||
1. frank-zealot: add `--metrics` (fixes the measured 501 + the
|
||||
structurally-red Prometheus job + the firing `LlmEndpointDown`).
|
||||
2. frank-dragoon: set `HALOGEN_API_KEY_FILE` (multi-key per consumer) +
|
||||
mount `HALOGEN_RECORD_DIR` (start `HALOGEN_RECORD_TEXT=none`; flip to
|
||||
`all` if prompt audit is wanted) + Alloy ship of `turns/`.
|
||||
3. frank-zealot: `--api-key-file` (multi-key, verified in the deployed
|
||||
build); radiance: probe for `--api-key` support (contract facts are
|
||||
silent — add to `fleet-probes`).
|
||||
4. models.yml: per-consumer bearer keys (`auth: none` → key) — the provider
|
||||
schema supports apiKey (pi-era conversion note).
|
||||
5. Caddy: `log` on `ai.franksdank.cloud` (OpenWebUI browser census, JSON →
|
||||
Loki).
|
||||
Limits: keys gate access, and only the engines' request records (halogen)
|
||||
see them per-request; llama.cpp/radiance/vLLM metrics stay identity-blind.
|
||||
|
||||
**Phase 1 — LiteLLM Proxy quadlet on Dankclaw = the front door.**
|
||||
One virtual key per consumer (omp-<workstation> ×N, openwebui, hermes,
|
||||
agents, bench). Per-key/user/team Prometheus labels into the existing
|
||||
Grafana+Alertmanager; Postgres spend ledger; budgets + rate limits per
|
||||
key. This is the ONLY layer that yields tokens-per-who without engine
|
||||
cooperation. (Bifrost is the lighter Apache-2.0 Go alternative; Helicone
|
||||
the proxy-native-observability alternative.)
|
||||
|
||||
**Phase 2 — prompt-level audit.** Langfuse self-hosted fed by the LiteLLM
|
||||
callback: per-user traces, sessions, cost; masking + retention knobs.
|
||||
|
||||
**Phase 3 — ambitious identity infra.** step-ca + Caddy mTLS per device;
|
||||
Tailscale (+flow logs) for off-LAN; auditd/eBPF process attribution on the
|
||||
agent hosts; Pi-hole query logger; nft counters per consumer IP. Browser
|
||||
users stay owned by OpenWebUI analytics + its per-user API keys; a pipeline
|
||||
can forward `X-Forwarded-User` so the gateway sees them too.
|
||||
|
||||
## Cautions
|
||||
|
||||
- Client-declared identity (keys, `user`, headers) groups traffic; it does
|
||||
not authenticate humans. Authenticated identity = Authentik (browser) /
|
||||
mTLS (device) / Tailscale (node).
|
||||
- Plain-HTTP LAN = bearer keys sniffable by any tenant; treat keys as
|
||||
identity labels on the LAN, not secrets, until TLS/mTLS at the door.
|
||||
- Prompt-content logging stores a plaintext copy of everything — set
|
||||
retention caps (halogen GIB/days; Langfuse masking; Loki retention) and
|
||||
gitignore the record dirs.
|
||||
- Household-scale cardinality is safe for per-key Prometheus labels;
|
||||
the engines' refusal of key labels is a multi-tenant concern, not ours.
|
||||
|
||||
## Related
|
||||
- [[Fleet Audit and Posture]] — monitoring gaps census
|
||||
- [Observability](../Infrastructure/Observability.md) — the Loki/Prometheus
|
||||
pipes every sink above uses
|
||||
- [LLM index](../LLM/index.md) — endpoint runbooks (env trios the Phase-0
|
||||
edits touch)
|
||||
@@ -23,6 +23,9 @@ Procedures for operating the fleet. The LLM-specific pages live under
|
||||
posture, whole-fleet audit, bulk redaction, stash audits
|
||||
- [[SSH and Remote Access]] — passwordless bootstrap without sshpass, fish
|
||||
login shells, GUI-hang diagnosis over ssh
|
||||
- [[LLM Usage Attribution]] — who-is-using-the-fleet-LLMs research catalog:
|
||||
engine-native identity, gateways, tracing, network/host attribution, and
|
||||
the phased plan
|
||||
|
||||
## Related
|
||||
- [Observability Provisioning](../Infrastructure/Observability%20Provisioning.md)
|
||||
|
||||
@@ -134,6 +134,9 @@ skill-cluster. See [log.md](log.md) for the wave plan and results.
|
||||
posture, whole-fleet audit, bulk redaction
|
||||
- [[SSH and Remote Access]] — passwordless bootstrap, fish login shells,
|
||||
GUI-hang diagnosis over ssh
|
||||
- [[LLM Usage Attribution]] — who-is-using-the-fleet-LLMs research catalog:
|
||||
engine-native identity, gateways, tracing, network/host attribution,
|
||||
phased plan
|
||||
|
||||
**Architecture:**
|
||||
|
||||
|
||||
Reference in New Issue
Block a user