Mastering LLM FinOps: Tracking vLLM Inference Costs on Kubernetes with OpenCost

The rapid shift from proprietary SaaS APIs to self-hosted open-weight models has fundamentally altered the infrastructure landscape. As organizations move workloads like Llama 3 or Mistral onto their own Kubernetes clusters, they gain control over data privacy and model customization. However, they also inherit a significant challenge: the “AI Cost Fog.” Unlike the predictable per-token pricing of OpenAI or Anthropic, self-hosted LLMs run on expensive, high-performance GPU instances where the bill is tied to uptime, not necessarily usage.

Traditional cloud billing provides a blunt instrument. It tells you how much a g5.12xlarge instance cost over the last month, but it fails to explain which specific model, department, or internal application consumed that budget. This lack of visibility makes it nearly impossible for DevOps and FinOps teams to calculate the true Return on Investment (ROI) of their AI initiatives. To bridge this gap, OpenCost—an open-source project dedicated to Kubernetes cost monitoring—has collaborated with IBM Research to introduce granular inference tracking for vLLM, the industry-standard engine for high-throughput LLM serving.

The Architecture of Kubernetes Inference Tracking

Tracking the cost of an LLM request is significantly more complex than tracking a standard web request. In a standard microservice, you might measure CPU cycles and memory residency. In the world of LLMs, the primary cost driver is the GPU, and the primary unit of value is the token. To get a clear picture of spend, we need a technical stack that connects hardware costs to software throughput.

The architecture for this solution relies on three primary components: vLLM, Prometheus, and OpenCost.

vLLM as the Data Source

vLLM serves as the inference engine, optimized for high throughput using PagedAttention. Beyond its performance capabilities, vLLM acts as a metrics exporter. It exposes an endpoint (typically /metrics) that provides real-time data on token throughput, request latency, and GPU cache utilization. This is the “source of truth” for what the model is actually doing.

Prometheus as the Aggregator

Prometheus acts as the time-series database that scrapes these metrics from the vLLM pods. In a standard open-weight AI infrastructure, Prometheus is already monitoring cluster health. By adding vLLM metrics, it creates a unified repository of both infrastructure performance and model activity.

OpenCost as the Financial Engine

OpenCost sits at the top of this stack. It ingests infrastructure cost data from cloud provider APIs (AWS, Azure, GCP) or on-premise pricing sheets. By correlating this financial data with the resource metrics in Prometheus, OpenCost can assign a dollar value to specific Kubernetes objects.

One of the most critical features here is the use of SharedLabels. In a large cluster, there are often “tax” services—logging, monitoring, or service meshes—that support all models. OpenCost uses these labels to distribute the costs of these shared services across the active AI models, ensuring that the final cost-per-token reflects the total cost of ownership (TCO), not just the raw GPU time.

Deep Dive: vLLM Token Metrics and OpenCost Integration

To calculate the cost per million tokens, OpenCost must look beyond simple pod uptime. It specifically targets two counters exported by vLLM:

1. vllm:prompt_tokens_total: The number of tokens processed during the initial request (the “context”).
2. vllm:generation_tokens_total: The number of tokens generated by the model in response.

These metrics are crucial because prompt tokens and generation tokens have different computational weights. In many LLM architectures, processing a massive prompt is computationally different from the iterative process of generating new tokens.

Mapping Costs to Models

OpenCost maps these token counts to specific Kubernetes namespaces and pods. If you are running a multi-tenant cluster where the marketing namespace runs a Llama-3-70b model and the engineering namespace runs a CodeLlama model, OpenCost can isolate the GPU spend for each.

The mathematical formula for calculating the “Cost per Million Tokens” in this environment looks like this:

Cost per 1M Tokens = (Hourly Pod Cost / Total Tokens Generated in Hour) * 1,000,000

The “Hourly Pod Cost” is derived from the underlying node’s cost, divided by the percentage of the node’s resources (specifically VRAM and GPU cores) allocated to that pod. By integrating these metrics, FinOps teams can finally compare their internal costs directly against the pricing sheets of SaaS providers.

Allocation vs. Usage: The FinOps Dilemma

In traditional cloud computing, we often talk about “utilization.” In LLM FinOps, we must distinguish between Allocation and Usage. This distinction is where most organizations lose money.

Allocation-Based Costs: The Price of Readiness

Allocation is the cost of “keeping the lights on.” When you deploy a vLLM pod and request 4 NVIDIA A100 GPUs, you are billed for those GPUs from the moment the pod starts. Even if no one is sending requests to the model, the meter is running. This is the baseline cost of availability. In many enterprise environments, GPUs are allocated 24/7, leading to massive “idle” spend.

Usage-Based Costs: The Efficiency of Inference

Usage is the cost associated with active work—the tokens being processed and generated. If your model is highly active, your cost-per-token drops because the fixed allocation cost is spread across a larger volume of tokens.

The Silent Killer: Idle Hosting Costs

The “silent killer” of AI budgets is the gap between allocation and usage. If you allocate an H100 instance at $3.00/hour but only process a few hundred tokens, your effective cost per million tokens could be thousands of dollars—far exceeding any SaaS provider. OpenCost highlights this gap, allowing teams to see exactly how much they are paying for “idle” capacity. This data is the primary driver for implementing autoscaling or “scale-to-zero” strategies using tools like KEDA.

The Tipping Point: Self-Hosting vs. SaaS APIs

Choosing between a SaaS API (like OpenAI) and self-hosting on Kubernetes is a classic “Buy vs. Build” decision, but with a moving target. To make an informed choice, you need to understand the tipping point where self-hosting becomes cheaper.

Comparative Analysis

Consider the following comparison between a typical SaaS provider and a self-hosted vLLM instance on a cloud provider’s spot instances:

The 50% Utilization Rule

Based on the research and validation performed by IBM Research and the OpenCost team, the “tipping point” for cost-effectiveness usually occurs at approximately 50% GPU utilization.

If your inference service is consistently utilized at 50% or higher, the cost of the underlying GPU instance, when divided by the token throughput, typically results in a lower “cost per million tokens” than SaaS providers. If utilization is below 20%, you are almost certainly overpaying for the privilege of self-hosting, unless data privacy requirements override financial concerns.

Factors that influence this ROI include:

• Model Size: Larger models require more VRAM, which necessitates more expensive GPU nodes (e.g., moving from A10G to H100).
• Request Concurrency: vLLM excels at batching. Higher concurrency increases token throughput without increasing the hourly GPU cost, significantly lowering the per-token price.

Case Study: Validating Costs Across 109 GPUs

The integration of OpenCost and vLLM wasn’t just theoretical; it was validated in a massive production-grade environment by IBM Research. The test cluster consisted of 109 GPUs supporting 30 active AI models across various departments.

Multi-Model Showback

In a cluster of this size, the primary challenge was “showback”—the ability to show each department exactly what they were spending. By using OpenCost, the IBM team was able to:

1. Identify models that were over-provisioned (high allocation, low token throughput).
2. Attribute costs to specific experimental projects that were previously hidden in the general cluster spend.
3. Calculate the exact cost of “shared” infrastructure, such as the Kubernetes control plane and networking, and distribute it proportionally.

Key Findings

The validation phase revealed that without granular tracking, the organization was overestimating the cost of high-traffic models and underestimating the cost of “zombie” models—deployments that remained active but were no longer being used by developers. By identifying these zombie models, the team was able to reclaim nearly 15% of their GPU capacity, demonstrating the immediate financial impact of LLM FinOps.

Implementation Guide: Setting Up OpenCost with vLLM

For DevOps engineers looking to implement this, the setup involves configuring vLLM to export metrics and OpenCost to ingest them.

Prerequisites

• A Kubernetes cluster with NVIDIA GPUs.
• Prometheus Operator installed (for metric scraping).
• vLLM deployed as a Service.

Step 1: Configure vLLM Metrics

Ensure your vLLM deployment includes the --engine-use-ray (if multi-GPU) and that the metrics port is exposed. By default, vLLM exports metrics on port 8000.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama-3-serving
spec:
  template:
    spec:
      containers:
      - name: vllm-container
        image: vllm/vllm-openai
        args: ["--model", "meta-llama/Meta-Llama-3-8B", "--port", "8000"]
        ports:
        - containerPort: 8000
          name: metrics

Step 2: Create a ServiceMonitor

If you are using the Prometheus Operator, create a ServiceMonitor to tell Prometheus to scrape the vLLM metrics.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: vllm-monitor
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: vllm
  endpoints:
  - port: metrics
    path: /metrics
    interval: 30s

Step 3: Configure OpenCost

OpenCost needs to be pointed to your Prometheus instance. In your OpenCost configuration (often via Helm), ensure the following environment variables or settings are enabled:

• PROMETHEUS_SERVER_ENDPOINT: The URL of your Prometheus service.
• EMIT_VLLM_METRICS: Set to true to enable the specific LLM cost logic.

Once configured, you can query the OpenCost REST API to get the cost per model:

curl http://opencost.opencost.svc.cluster.local:9003/allocation/compute?window=1d&aggregate=namespace

This will return a JSON payload including the totalCost, vllm_prompt_tokens, and vllm_generation_tokens, allowing you to perform the final cost-per-token calculation in your reporting dashboard.

The Future of AI FinOps: Beyond Token Counting

As the field of AI infrastructure matures, cost tracking will move beyond simple token counting. We are entering an era where “GPU Efficiency” becomes a standard KPI for engineering teams.

Quantifying ‘Wasted’ Capacity

One area of active development within the OpenCost and IBM collaboration is the quantification of “wasted” GPU capacity. This involves analyzing GPU memory fragmentation—where a model occupies the VRAM but the GPU cores are idle. Future versions of OpenCost aim to provide a “Waste Score” for each model deployment, helping teams decide when to consolidate multiple small models onto a single large GPU using techniques like LoRA adapters.

Advanced Idle Detection

Current idle detection is often based on simple network traffic. However, LLMs have unique states, such as the KV (Key-Value) cache occupancy. A model might be “idle” in terms of requests but “active” in terms of holding a massive cache in memory to speed up future requests. Distinguishing between these states is essential for accurate FinOps.

The Role of Service Meshes

We also expect to see deeper integration with service meshes. For instance, using Linkerd for zero-downtime federation can provide even more granular traffic data. By combining service mesh telemetry with vLLM metrics, OpenCost can track the cost of a request from the moment it hits the cluster ingress to the moment the last token is generated.

The “AI Cost Fog” is beginning to lift. With the combination of vLLM’s detailed metrics and OpenCost’s financial rigor, organizations can finally treat AI infrastructure not as a mysterious black box, but as a manageable, optimized component of their modern technology stack. As GPU resources remain both expensive and scarce, the ability to track every cent spent on a token will be the difference between a successful AI transformation and a budgetary disaster.