Adobe Open-Sources Kubernetes Proxy to Isolate GPU Metrics Across Tenants

Adobe engineers have outlined an open-source approach that grants teams controlled visibility into their own Prometheus metrics within shared Kubernetes clusters while blocking access to other teams' data. The design targets GPU-intensive setups, where cost-conscious teams need to track whether their expensive accelerators are actually running useful work.

The Challenge: Multi-Tenant Metrics Without Spillover#

Centralised Prometheus instances holding thousands of namespaces create a security and performance dilemma. Unrestricted query access risks exposing one team's data to another; widespread usage of a shared store can also degrade performance and create noisy-neighbour interference. Adobe's answer: a tenant-aware proxy sitting between users and the central Prometheus, optionally paired with smaller per-tenant Prometheus instances.

How the Architecture Works#

The system leverages existing Kubernetes and CNCF components rather than introducing new metrics platforms. Incoming requests pass through NGINX and kube-rbac-proxy for authentication and authorisation before hitting a multi-tenant Prometheus proxy. That proxy then identifies the requesting tenant, finds the right Prometheus backend, and limits queries to the team's namespace only.

A critical piece is prom-label-proxy, which rewrites incoming PromQL queries to enforce namespace boundaries. This prevents the need to rely on developers manually adding the correct namespace—and stops malicious or careless query construction from leaking data. The enforcement happens at the proxy layer, before Prometheus even sees the request.

Teams also gain a new Kubernetes custom resource, MetricAccess, where they declare which metrics they need. Definitions can use exact names, regex patterns, or PromQL selectors, giving teams flexibility while keeping collection and access control with the platform team.

Per-Tenant Isolation and Storage#

For groups needing their own dashboards and alerts, the design can regularly push a filtered metric set into a tenant-specific Prometheus. When metricIsolation is turned on, only that tenant's series get stored, cutting storage overhead and narrowing what can be exposed. Adobe reports one case where stored series dropped from over 10,000 to around 300, reducing both disk and query load while creating an extra security boundary: metrics never collected into a tenant's store cannot leak from it later.