serverAppConfigs/serverAppConfigs-docs/Operations/LLM Usage Attribution.md

---
type: reference
title: LLM Usage Attribution
description: Research catalog of every viable way to answer "who is using the fleet LLMs" — engine-native identity, gateways, auth proxies, tracing platforms, network/host attribution — with the measured fleet baseline (2026-10-13) and a phased adoption plan.
tags: [llm, observability, auth, gateway, litellm, tracking, identity]
timestamp: 2026-10-14T00:00:00-07:00
---

# LLM Usage Attribution

Deep-research catalog (2026-10-13/14): solutions for tracking WHO uses the
fleet endpoints (`frank-archon`, `frank-dragoon`, `frank-zealot`). Full web
citations; live-fleet measurements marked **(measured)**.

## Baseline: what the fleet answers today

- All three endpoints are `auth: none`, plain HTTP, LAN-only
  (`models.yml` providers; no vhost on 7770/7771).
- **(measured)** Live `/metrics` label census: `frank-archon` exports
  `vllm:*`+`radiance:*` with labels only `{model_name, engine, le,
  finished_reason, position, class}` — zero identity labels.
  `frank-dragoon` exports `halogen:*`+`llamacpp:*`.
  `frank-zealot` answers **501** — `serve-frank-zealot.sh` never passes
  `--metrics`, so the Prometheus job is structurally never-green
  (`LlmEndpointDown` fires). Fix: add `--metrics` to the ARGS array.
- **(measured)** frank-zealot verbosity-4 journal lines carry slot/task/timing
  only — no client IP, no user. Alloy ships journal + podman ctr.log to Loki,
  so what lands there has no identity either.
- Caddy has no `log` directive anywhere → no per-request access logs on the
  OpenWebUI vhost.
- Net: fleet-level token telemetry exists per ENDPOINT; per-CONSUMER
  identity exists nowhere.

## The identity layers (pick the layer, get its resolution)

| # | Layer | Identity | Answers |
|---|---|---|---|
| L0 | DHCP/ARP | MAC, hostname, lease | which device was on the LAN |
| L1 | DNS (Pi-hole v6) | device → queries | which device resolved/used a name |
| L2 | L3/L4 (nftables/conntrack, JA3/JA4) | IP, TLS client lib | bytes per client, client software |
| L3 | Transport (API keys, mTLS, SPIFFE, Tailscale) | key/cert/node | which credential connected |
| L4 | App gateway (virtual keys → user/team/end_user) | named consumer | tokens/spend/cost per who |
| L5 | Trace/content (Langfuse, Helicone, halogen records) | user_id, session, prompt text | what they asked, per session |
| L6 | Client host (auditd, eBPF, netdata) | pid/uid/comm per connect | which process on which host |

Keys/headers/`user` fields are CLIENT-DECLARED — they group traffic, they
prove nothing about a human (halogen's own docs warn this verbatim:
"they group requests, and they prove nothing about who sent them").

## Catalog

### A. Engine-native (per-consumer identity inside the inference server)

| Engine | AuthN | Identity in logs | Per-identity metrics | OTel | Notes |
|---|---|---|---|---|---|
| llama.cpp | `--api-key` (multi, comma) / `--api-key-file` **(measured in the deployed build's `--help`)** | request `user` field drives slot assignment/eviction; `--log-requests` dumps bodies | `--metrics` exports `llamacpp:*`, NO key/user label | none upstream | [server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) |
| vLLM | `--api-key` (multi) | request log; no user persistence | `vllm:*`; NO `api_key` label (cardinality, issue #53926) | native (`--otlp-traces-endpoint`) | [metrics design](https://github.com/vllm-project/vllm/blob/main/docs/design/metrics.md) |
| radiance | no public auth docs **[unverified]** | — | **(measured)** no identity labels on live /metrics | none found | closed deployment surface; treat authN as absent until probed |
| halogen-flash-server | `HALOGEN_API_KEY` / `HALOGEN_API_KEY_FILE` (multi, comma) | **`HALOGEN_RECORD_DIR` request records**: `turns/*.jsonl` per request — `client_family` (User-Agent), tokens, timings, finish, tool calls, plus HASHES of `user` / `prompt_cache_key` / `safety_identifier` / `metadata.user_id` (`.salt`-keyed); `units/` content-addressed text; caps `HALOGEN_RECORD_GIB`=2, `HALOGEN_RECORD_DAYS`=90; `HALOGEN_RECORD_TEXT=none` for metadata-only; `stats`/`prune` tools in-container | `halogen:*`+`llamacpp:*`, no per-key label | none | [configuration.md](https://github.com/peonist-ai/halogen-flash-server/blob/main/docs/configuration.md) — verbatim-verified 2026-10-13. Best native ledger of the fleet. |
| SGLang | `--api-key` (single) | `--log-requests` | `sglang:*`, no key label | native (`--enable-trace`) | for any future swap |
| Ollama | single key | request log | `ollama:` no key label | none | not in fleet |

Wire fields usable from clients today (all client-declared): OpenAI
`user`, `safety_identifier`, `prompt_cache_key`, Messages
`metadata.user_id`; LiteLLM `end_user_id`; gateway header conventions
(`X-Forwarded-User`, `Helicone-User-Id`, `cf-aig-metadata`).

### B. LLM gateways (front door with virtual keys — the only layer with
per-consumer TOKEN counts)

| Solution | Identity | Usage lands | License | Footprint |
|---|---|---|---|---|
| **LiteLLM Proxy** | virtual keys → `user_id`/`team_id`/`end_user`; budgets, rate limits | Postgres spend ledger; `/metrics` labels `api_key_alias,user,team,end_user,model,client_ip,user_agent` (`litellm_*_tokens_metric`, spend, budget gauges); OTel/Langfuse callbacks | MIT core (SSO/RBAC/audit-log enterprise) | ~150–300 MiB + Postgres |
| Kong AI Gateway | Consumer + key-auth/JWT/OIDC/mTLS; AI consumer groups | Postgres; `kong_*` per-consumer; file-log/zipkin/otel | OSS core; token-aware AI plugins enterprise | 0.5–1 GiB |
| Apache APISIX | Consumer + key-auth/JWT | `apisix_llm_prompt_tokens`/`completion_tokens` with `consumer` label; http-logger/kafka | Apache-2.0 | 200–400 MiB |
| Bifrost (maximhq) | virtual keys `sk-bf-*`/`x-bf-vk`, budgets | Prometheus/OTel, config store (SQLite/Postgres) | Apache-2.0, Go | ~150–300 MiB |
| Portkey OSS gateway | virtual keys | logs/OTel | MIT | 200–400 MiB |
| Higress AI GW | keys (+OIDC ent) | `higress_*`, ai-logging | Apache-2.0 CE | Istio-based, 0.5 GiB+ |
| Envoy AI GW / Gateway API Inference Extension / LLM Gatekeeper | delegated auth | Envoy metrics/access logs | Apache-2.0 | K8s-native |
| Gloo / KrakenD / Gravitee AI gateways | keys, OIDC | metrics/logs | OSS core, AI features enterprise | heavy |
| Cloudflare AI Gateway | Access JWT `cf.user_id`, `cf-aig-metadata` | Cloudflare-managed analytics/logs | SaaS (exotic for LAN; Tunnel possible) | edge |

([LiteLLM virtual keys](https://docs.litellm.ai/docs/proxy/virtual_keys),
[prometheus](https://docs.litellm.ai/docs/proxy/prometheus),
[APISIX ai-proxy](https://apisix.apache.org/docs/apisix/plugins/ai-proxy/),
[Bifrost](https://github.com/maximhq/bifrost),
[Kong AI](https://developer.konghq.com/index/ai-gateway/))

### C. Auth proxies / identity infrastructure (identity without token counts)

- **Ory Oathkeeper** (MIT, ~50–100 MiB): per-consumer api-key/JWT/
  client-credentials → header mutation (`X-User: {{ .Subject }}`) +
  decision logs.
- **oauth2-proxy** (Apache-2.0): OIDC (Authentik) incl. client_credentials,
  `--pass-user-headers` → `X-Forwarded-User`; ~100 MiB.
- **Authentik itself**: proxy-provider custom headers (already injects
  `X-Authentik-Username`/`-Groups`, spoof-stripped — the /whoami seam proves
  the pattern on this host); event log + webhook forwarding.
- **Caddy `tls client_auth`** (mTLS per client cert) + access-log fields —
  Caddy already fronts both hosts.
- **Smallstep step-ca**: short-lived per-device certs, ACME-style.
- **SPIFFE/SPIRE**: per-workload SVIDs (k8s-flavored; heavy for quadlets).
- **Tailscale / NetBird**: per-node identity, ACL + network-flow logs,
  headscale for self-hosted control. Best for off-LAN laptops.
- **ApiAuth** (nginx auth_request): the minimal per-consumer apikey sidecar.

### D. LLM observability / tracing (prompt-level per-user audit)

| Platform | Identity | Content | Self-host | Shape |
|---|---|---|---|---|
| **Langfuse** | `user_id`,`session_id`, tags | full input/output, masking hook | ✅ ClickHouse, Apache-2.0 | fed via LiteLLM callback / OTLP / SDK |
| **Helicone** | api-key→user, `Helicone-User-Id` | full, tiered redaction | ✅ compose (ClickHouse+PG+Redis+Jawn), Apache-2.0 | PROXY — can be the front door itself |
| Arize Phoenix | `user.id`,`session.id` | opt-in `gen_ai.*` | ✅, Apache-2.0 | OTLP collector |
| OpenLIT | `user.id`,`session.id` | 3-tier capture | ✅, Apache-2.0 | OTLP+ClickHouse |
| SigNoz LLM | OTel attrs | opt-in | ✅, Apache-2.0 | OTLP |
| OpenLLMetry/Traceloop | assoc props | opt-in | ✅ Helm | OTel |
| Lunary / W&B Weave / LangSmith | user/session | full | enterprise-gated | — |

OTel GenAI semconv (Development-stability): `gen_ai.operation.name`,
`gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`/
`output_tokens`, `gen_ai.conversation.id`, content attrs opt-in; `user.id`
from the registry. vLLM and SGLang emit OTLP natively; llama.cpp only via
external SDK — **the fleet's engines are OTel-silent at request level**,
which is why the gateway callback (LiteLLM→Langfuse) is the practical feed.

### E. Network/host passive attribution (no client cooperation)

- **Pi-hole v6** (already on Dankclaw): query log API per device,
  `pihole6_exporter` + query-logger→Loki (GPLv3 exporter).
- **Caddy access logs**: add a `log` directive (JSON) → Alloy→Loki; with the
  auth chain it carries `X-Authentik-Username` for browser users.
- **OpenWebUI admin**: per-user daily token stats, events/logs, analytics API
  (`/api/v1/analytics/*`) — per-USER identity exists here already (MIT).
- **auditd** connect-syscall rules (uid/pid/comm per connect to 7770/7771),
  ship via audisp→Loki.
- **eBPF**: Tetragon (process+socket events, policies), Coroot, DeepFlow,
  bpftrace `tracepoint:syscalls:sys_enter_connect` one-liners. K8s-shaped
  for single hosts, but agents.plugin-style tools work.
- **nftables named counters** per consumer IP (bytes; reload resets).
- **netdata apps.plugin**: per-process network bytes, auto-discovered.
- **JA3/JA4 TLS fingerprinting**: client library only, TLS-only; the LAN is
  plaintext HTTP today → use the HTTP **User-Agent census** instead (free
  from any access log).
- **DHCP/ARP census** (leases + `ip neigh`): device roster.

## Phased plan for this fleet

**Phase 0 — repo-only, no new services**
1. frank-zealot: add `--metrics` (fixes the measured 501 + the
   structurally-red Prometheus job + the firing `LlmEndpointDown`).
2. frank-dragoon: set `HALOGEN_API_KEY_FILE` (multi-key per consumer) +
   mount `HALOGEN_RECORD_DIR` (start `HALOGEN_RECORD_TEXT=none`; flip to
   `all` if prompt audit is wanted) + Alloy ship of `turns/`.
3. frank-zealot: `--api-key-file` (multi-key, verified in the deployed
   build); radiance: probe for `--api-key` support (contract facts are
   silent — add to `fleet-probes`).
4. models.yml: per-consumer bearer keys (`auth: none` → key) — the provider
   schema supports apiKey (pi-era conversion note).
5. Caddy: `log` on `ai.franksdank.cloud` (OpenWebUI browser census, JSON →
   Loki).
   Limits: keys gate access, and only the engines' request records (halogen)
   see them per-request; llama.cpp/radiance/vLLM metrics stay identity-blind.

**Phase 1 — LiteLLM Proxy quadlet on Dankclaw = the front door.**
One virtual key per consumer (omp-<workstation> ×N, openwebui, hermes,
agents, bench). Per-key/user/team Prometheus labels into the existing
Grafana+Alertmanager; Postgres spend ledger; budgets + rate limits per
key. This is the ONLY layer that yields tokens-per-who without engine
cooperation. (Bifrost is the lighter Apache-2.0 Go alternative; Helicone
the proxy-native-observability alternative.)

**Phase 2 — prompt-level audit.** Langfuse self-hosted fed by the LiteLLM
callback: per-user traces, sessions, cost; masking + retention knobs.

**Phase 3 — ambitious identity infra.** step-ca + Caddy mTLS per device;
Tailscale (+flow logs) for off-LAN; auditd/eBPF process attribution on the
agent hosts; Pi-hole query logger; nft counters per consumer IP. Browser
users stay owned by OpenWebUI analytics + its per-user API keys; a pipeline
can forward `X-Forwarded-User` so the gateway sees them too.

## Cautions

- Client-declared identity (keys, `user`, headers) groups traffic; it does
  not authenticate humans. Authenticated identity = Authentik (browser) /
  mTLS (device) / Tailscale (node).
- Plain-HTTP LAN = bearer keys sniffable by any tenant; treat keys as
  identity labels on the LAN, not secrets, until TLS/mTLS at the door.
- Prompt-content logging stores a plaintext copy of everything — set
  retention caps (halogen GIB/days; Langfuse masking; Loki retention) and
  gitignore the record dirs.
- Household-scale cardinality is safe for per-key Prometheus labels;
  the engines' refusal of key labels is a multi-tenant concern, not ours.

## Related
- [[Fleet Audit and Posture]] — monitoring gaps census
- [Observability](../Infrastructure/Observability.md) — the Loki/Prometheus
  pipes every sink above uses
- [LLM index](../LLM/index.md) — endpoint runbooks (env trios the Phase-0
  edits touch)