serverAppConfigs/serverAppConfigs-docs/Operations/LLM Usage Attribution.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	LLM Usage Attribution	Research catalog of every viable way to answer "who is using the fleet LLMs" — engine-native identity, gateways, auth proxies, tracing platforms, network/host attribution — with the measured fleet baseline (2026-10-13) and a phased adoption plan.	llm observability auth gateway litellm tracking identity	2026-10-14T00:00:00-07:00

LLM Usage Attribution

Deep-research catalog (2026-10-13/14): solutions for tracking WHO uses the fleet endpoints (frank-archon, frank-dragoon, frank-zealot). Full web citations; live-fleet measurements marked (measured).

Baseline: what the fleet answers today

• All three endpoints are auth: none, plain HTTP, LAN-only (models.yml providers; no vhost on 7770/7771).
• (measured) Live /metrics label census: frank-archon exports vllm:*+radiance:* with labels only {model_name, engine, le, finished_reason, position, class} — zero identity labels. frank-dragoon exports halogen:*+llamacpp:*. frank-zealot answers 501 — serve-frank-zealot.sh never passes --metrics, so the Prometheus job is structurally never-green (LlmEndpointDown fires). Fix: add --metrics to the ARGS array.
• (measured) frank-zealot verbosity-4 journal lines carry slot/task/timing only — no client IP, no user. Alloy ships journal + podman ctr.log to Loki, so what lands there has no identity either.
• Caddy has no log directive anywhere → no per-request access logs on the OpenWebUI vhost.
• Net: fleet-level token telemetry exists per ENDPOINT; per-CONSUMER identity exists nowhere.

The identity layers (pick the layer, get its resolution)

#	Layer	Identity	Answers
L0	DHCP/ARP	MAC, hostname, lease	which device was on the LAN
L1	DNS (Pi-hole v6)	device → queries	which device resolved/used a name
L2	L3/L4 (nftables/conntrack, JA3/JA4)	IP, TLS client lib	bytes per client, client software
L3	Transport (API keys, mTLS, SPIFFE, Tailscale)	key/cert/node	which credential connected
L4	App gateway (virtual keys → user/team/end_user)	named consumer	tokens/spend/cost per who
L5	Trace/content (Langfuse, Helicone, halogen records)	user_id, session, prompt text	what they asked, per session
L6	Client host (auditd, eBPF, netdata)	pid/uid/comm per connect	which process on which host

Keys/headers/user fields are CLIENT-DECLARED — they group traffic, they prove nothing about a human (halogen's own docs warn this verbatim: "they group requests, and they prove nothing about who sent them").

Catalog

A. Engine-native (per-consumer identity inside the inference server)

Engine	AuthN	Identity in logs	Per-identity metrics	OTel	Notes
llama.cpp	--api-key (multi, comma) / --api-key-file (measured in the deployed build's --help)	request user field drives slot assignment/eviction; --log-requests dumps bodies	--metrics exports llamacpp:*, NO key/user label	none upstream	server README
vLLM	--api-key (multi)	request log; no user persistence	vllm:*; NO api_key label (cardinality, issue #53926)	native (--otlp-traces-endpoint)	metrics design
radiance	no public auth docs [unverified]	—	(measured) no identity labels on live /metrics	none found	closed deployment surface; treat authN as absent until probed
halogen-flash-server	HALOGEN_API_KEY / HALOGEN_API_KEY_FILE (multi, comma)	HALOGEN_RECORD_DIR request records: turns/*.jsonl per request — client_family (User-Agent), tokens, timings, finish, tool calls, plus HASHES of user / prompt_cache_key / safety_identifier / metadata.user_id (.salt-keyed); units/ content-addressed text; caps HALOGEN_RECORD_GIB=2, HALOGEN_RECORD_DAYS=90; HALOGEN_RECORD_TEXT=none for metadata-only; stats/prune tools in-container	halogen:*+llamacpp:*, no per-key label	none	configuration.md — verbatim-verified 2026-10-13. Best native ledger of the fleet.
SGLang	--api-key (single)	--log-requests	sglang:*, no key label	native (--enable-trace)	for any future swap
Ollama	single key	request log	ollama: no key label	none	not in fleet

Wire fields usable from clients today (all client-declared): OpenAI user, safety_identifier, prompt_cache_key, Messages metadata.user_id; LiteLLM end_user_id; gateway header conventions (X-Forwarded-User, Helicone-User-Id, cf-aig-metadata).

B. LLM gateways (front door with virtual keys — the only layer with

per-consumer TOKEN counts)

Solution	Identity	Usage lands	License	Footprint
LiteLLM Proxy	virtual keys → user_id/team_id/end_user; budgets, rate limits	Postgres spend ledger; /metrics labels api_key_alias,user,team,end_user,model,client_ip,user_agent (litellm_*_tokens_metric, spend, budget gauges); OTel/Langfuse callbacks	MIT core (SSO/RBAC/audit-log enterprise)	~150–300 MiB + Postgres
Kong AI Gateway	Consumer + key-auth/JWT/OIDC/mTLS; AI consumer groups	Postgres; kong_* per-consumer; file-log/zipkin/otel	OSS core; token-aware AI plugins enterprise	0.5–1 GiB
Apache APISIX	Consumer + key-auth/JWT	apisix_llm_prompt_tokens/completion_tokens with consumer label; http-logger/kafka	Apache-2.0	200–400 MiB
Bifrost (maximhq)	virtual keys sk-bf-*/x-bf-vk, budgets	Prometheus/OTel, config store (SQLite/Postgres)	Apache-2.0, Go	~150–300 MiB
Portkey OSS gateway	virtual keys	logs/OTel	MIT	200–400 MiB
Higress AI GW	keys (+OIDC ent)	higress_*, ai-logging	Apache-2.0 CE	Istio-based, 0.5 GiB+
Envoy AI GW / Gateway API Inference Extension / LLM Gatekeeper	delegated auth	Envoy metrics/access logs	Apache-2.0	K8s-native
Gloo / KrakenD / Gravitee AI gateways	keys, OIDC	metrics/logs	OSS core, AI features enterprise	heavy
Cloudflare AI Gateway	Access JWT cf.user_id, cf-aig-metadata	Cloudflare-managed analytics/logs	SaaS (exotic for LAN; Tunnel possible)	edge

(LiteLLM virtual keys, prometheus, APISIX ai-proxy, Bifrost, Kong AI)

C. Auth proxies / identity infrastructure (identity without token counts)

• Ory Oathkeeper (MIT, ~50–100 MiB): per-consumer api-key/JWT/ client-credentials → header mutation (X-User: {{ .Subject }}) + decision logs.
• oauth2-proxy (Apache-2.0): OIDC (Authentik) incl. client_credentials, --pass-user-headers → X-Forwarded-User; ~100 MiB.
• Authentik itself: proxy-provider custom headers (already injects X-Authentik-Username/-Groups, spoof-stripped — the /whoami seam proves the pattern on this host); event log + webhook forwarding.
• Caddy tls client_auth (mTLS per client cert) + access-log fields — Caddy already fronts both hosts.
• Smallstep step-ca: short-lived per-device certs, ACME-style.
• SPIFFE/SPIRE: per-workload SVIDs (k8s-flavored; heavy for quadlets).
• Tailscale / NetBird: per-node identity, ACL + network-flow logs, headscale for self-hosted control. Best for off-LAN laptops.
• ApiAuth (nginx auth_request): the minimal per-consumer apikey sidecar.

D. LLM observability / tracing (prompt-level per-user audit)

Platform	Identity	Content	Self-host	Shape
Langfuse	user_id,session_id, tags	full input/output, masking hook	✅ ClickHouse, Apache-2.0	fed via LiteLLM callback / OTLP / SDK
Helicone	api-key→user, Helicone-User-Id	full, tiered redaction	✅ compose (ClickHouse+PG+Redis+Jawn), Apache-2.0	PROXY — can be the front door itself
Arize Phoenix	user.id,session.id	opt-in gen_ai.*	✅, Apache-2.0	OTLP collector
OpenLIT	user.id,session.id	3-tier capture	✅, Apache-2.0	OTLP+ClickHouse
SigNoz LLM	OTel attrs	opt-in	✅, Apache-2.0	OTLP
OpenLLMetry/Traceloop	assoc props	opt-in	✅ Helm	OTel
Lunary / W&B Weave / LangSmith	user/session	full	enterprise-gated	—

OTel GenAI semconv (Development-stability): gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens/ output_tokens, gen_ai.conversation.id, content attrs opt-in; user.id from the registry. vLLM and SGLang emit OTLP natively; llama.cpp only via external SDK — the fleet's engines are OTel-silent at request level, which is why the gateway callback (LiteLLM→Langfuse) is the practical feed.

E. Network/host passive attribution (no client cooperation)

• Pi-hole v6 (already on Dankclaw): query log API per device, pihole6_exporter + query-logger→Loki (GPLv3 exporter).
• Caddy access logs: add a log directive (JSON) → Alloy→Loki; with the auth chain it carries X-Authentik-Username for browser users.
• OpenWebUI admin: per-user daily token stats, events/logs, analytics API (/api/v1/analytics/*) — per-USER identity exists here already (MIT).
• auditd connect-syscall rules (uid/pid/comm per connect to 7770/7771), ship via audisp→Loki.
• eBPF: Tetragon (process+socket events, policies), Coroot, DeepFlow, bpftrace tracepoint:syscalls:sys_enter_connect one-liners. K8s-shaped for single hosts, but agents.plugin-style tools work.
• nftables named counters per consumer IP (bytes; reload resets).
• netdata apps.plugin: per-process network bytes, auto-discovered.
• JA3/JA4 TLS fingerprinting: client library only, TLS-only; the LAN is plaintext HTTP today → use the HTTP User-Agent census instead (free from any access log).
• DHCP/ARP census (leases + ip neigh): device roster.

Phased plan for this fleet

Phase 0 — repo-only, no new services

1. frank-zealot: add --metrics (fixes the measured 501 + the structurally-red Prometheus job + the firing LlmEndpointDown).
2. frank-dragoon: set HALOGEN_API_KEY_FILE (multi-key per consumer) + mount HALOGEN_RECORD_DIR (start HALOGEN_RECORD_TEXT=none; flip to all if prompt audit is wanted) + Alloy ship of turns/.
3. frank-zealot: --api-key-file (multi-key, verified in the deployed build); radiance: probe for --api-key support (contract facts are silent — add to fleet-probes).
4. models.yml: per-consumer bearer keys (auth: none → key) — the provider schema supports apiKey (pi-era conversion note).
5. Caddy: log on ai.franksdank.cloud (OpenWebUI browser census, JSON → Loki). Limits: keys gate access, and only the engines' request records (halogen) see them per-request; llama.cpp/radiance/vLLM metrics stay identity-blind.

Phase 1 — LiteLLM Proxy quadlet on Dankclaw = the front door. One virtual key per consumer (omp- ×N, openwebui, hermes, agents, bench). Per-key/user/team Prometheus labels into the existing Grafana+Alertmanager; Postgres spend ledger; budgets + rate limits per key. This is the ONLY layer that yields tokens-per-who without engine cooperation. (Bifrost is the lighter Apache-2.0 Go alternative; Helicone the proxy-native-observability alternative.)

Phase 2 — prompt-level audit. Langfuse self-hosted fed by the LiteLLM callback: per-user traces, sessions, cost; masking + retention knobs.

Phase 3 — ambitious identity infra. step-ca + Caddy mTLS per device; Tailscale (+flow logs) for off-LAN; auditd/eBPF process attribution on the agent hosts; Pi-hole query logger; nft counters per consumer IP. Browser users stay owned by OpenWebUI analytics + its per-user API keys; a pipeline can forward X-Forwarded-User so the gateway sees them too.

Cautions

• Client-declared identity (keys, user, headers) groups traffic; it does not authenticate humans. Authenticated identity = Authentik (browser) / mTLS (device) / Tailscale (node).
• Plain-HTTP LAN = bearer keys sniffable by any tenant; treat keys as identity labels on the LAN, not secrets, until TLS/mTLS at the door.
• Prompt-content logging stores a plaintext copy of everything — set retention caps (halogen GIB/days; Langfuse masking; Loki retention) and gitignore the record dirs.
• Household-scale cardinality is safe for per-key Prometheus labels; the engines' refusal of key labels is a multi-tenant concern, not ours.

Related

• Fleet Audit and Posture — monitoring gaps census
• Observability — the Loki/Prometheus pipes every sink above uses
• LLM index — endpoint runbooks (env trios the Phase-0 edits touch)