The industry is currently moving past the initial excitement of Retrieval-Augmented Generation (RAG) and entering the era of “agentic loops.” We are no longer satisfied with an LLM that simply answers questions; we want agents that can plan, use tools, browse the web, and execute code to solve multi-step problems. However, as we move from simple inference to complex, long-running agentic workflows, we are discovering a fundamental truth: our current cloud infrastructure was never designed for this.

If you try to deploy a fleet of autonomous agents on a standard Kubernetes cluster or a fleet of serverless functions, you quickly run into the “Agentic Infrastructure Gap.” In a traditional microservices architecture, a service is either handling a request or sitting idle. But an agentic workload is different. An agent might spend 10 seconds reasoning, 30 seconds waiting for a third-party API to return data, and another 10 seconds executing a Python script.

During those 30 seconds of waiting, a standard container continues to consume expensive GPU or CPU resources just to keep its memory state alive. If you use serverless functions to save costs, you lose the “state”—the complex context of the agent’s reasoning process—forcing you to rebuild the entire context from a database on every invocation. This is the “idle compute” problem, and it is the primary reason why scaling agentic systems is currently prohibitively expensive and architecturally messy.

Enter AX. Developed by Google as an open orchestration layer, AX (and its underlying runtime, Agent Substrate) is designed to treat AI agents not as ephemeral requests, but as stateful, long-lived actors. It represents a shift toward “Agent-Native” infrastructure that prioritizes state persistence, sub-second resumption, and secure tool interaction.

The Agentic Infrastructure Gap: Why Kubernetes Isn’t Enough

To understand why AX is necessary, we have to look at the limitations of our existing toolset. Kubernetes is the gold standard for microservices, but it views the world through the lens of “containers” and “pods.”

The Persistence Problem

In an agentic loop, the “state” isn’t just a row in a SQL database. It is the entire conversation history, the current plan, the results of the last five tool calls, and the intermediate variables in a running code execution environment. When an agent waits for a human-in-the-loop approval or a slow external API, keeping a pod active just to hold that state in RAM is a waste of resources. Conversely, “hydrating” that state from a database every time the agent wakes up adds significant latency and complexity to the application logic.

The Idle Compute Problem

AI agents are “bursty.” They require intense compute for reasoning (LLM inference) and then virtually zero compute while waiting for environmental feedback. In a standard cloud environment, you pay for the “uptime” of the instance. If an agent spends 80% of its lifecycle waiting for tool outputs, you are effectively overpaying by 800% for the compute power required to simply “wait.”

The Context Engineering Challenge

Effective agents require highly tuned context. As discussed in our deep dive into context engineering for AI root cause analysis, the way information is structured and fed to an agent determines its success. Standard infrastructure doesn’t provide a native way to manage this evolving context over long periods; developers are forced to build custom “state machines” on top of existing tools, leading to brittle and hard-to-maintain codebases.

Introducing AX: The Stateful Actor Model for AI

AX solves these problems by moving away from the request-response model and adopting the Actor Model. In the world of AX, every agent is an “Actor”—a self-contained unit of execution that has its own state, its own logic, and its own way of communicating with other actors.

What is AX?

AX is an orchestration layer. It doesn’t replace your LLM; it manages the lifecycle of the agent using that LLM. It provides the control plane for deploying, scaling, and monitoring agents across a cluster.

Agent Substrate: The Compute Runtime

While AX handles the orchestration (the “where” and “when”), Agent Substrate is the underlying compute runtime (the “how”). It is a specialized environment designed specifically for stateful actor lifecycles. Unlike a generic Linux container, Agent Substrate is optimized for:

  1. Isolation: Ensuring that one agent’s execution cannot interfere with another.
  2. State Management: Automatically capturing the “snapshot” of an agent’s memory and execution pointer.
  3. Connectivity: Native support for protocols that allow agents to talk to the outside world.

By combining AX and Agent Substrate, Google has created a stack where the infrastructure understands what an agent is. It knows that an agent might sleep for an hour and wake up when a specific event occurs, and it manages the underlying hardware resources accordingly.

Sub-Second Resumption and the Magic of Checkpointing

The most technically impressive feature of AX is its ability to perform sub-second resumption through advanced checkpointing. This is the “secret sauce” that solves the idle compute problem.

How Checkpointing Works

In a standard system, if you want to stop a process and start it later, you usually have to save data to a database and then restart the application from scratch. AX takes a different approach. It performs a “deep checkpoint” of the entire agent state. This includes:

  • The current stack and heap of the agent’s execution environment.
  • The conversation history and internal “thought” process.
  • The status of any pending tool calls.

This snapshot is compressed and moved to high-speed storage. The compute resources (CPU/GPU) are then immediately freed up for other tasks.

Resumption on Event

When the external tool returns its data or the human provides the requested input, the AX control plane identifies the correct checkpoint, loads it back into Agent Substrate, and resumes execution exactly where it left off. Because this happens at the substrate level, the “wake up” time is sub-second.

To the agent, it feels like no time has passed. To the developer, it means they only paid for the milliseconds of active reasoning.

Feature Traditional Serverless Kubernetes Pods AX (Agent Substrate)
Statefulness Stateless (requires DB) Stateful (in-memory) Stateful (auto-checkpointed)
Idle Cost Low (but high cold start) High (always running) Low (active-on-event)
Startup Time Seconds (Cold start) Milliseconds (if warm) Sub-second (Resumption)
Complexity High (State management) Medium (Scaling/Orchestration) Low (Declarative)

Declarative Configuration: Managing Agents Like Kubernetes

One of the reasons Kubernetes became so successful was its declarative nature. You don’t tell the system how to start a container; you tell it what the desired state is via a YAML file. AX brings this same “Infrastructure-as-Code” (IaC) philosophy to AI agents.

Using the v1alpha1 API version, developers can define agents, their capabilities, and their resource constraints in a familiar manifest format.

apiVersion: ax.google.com/v1alpha1
kind: Agent
metadata:
  name: security-auditor-agent
spec:
  runtime: agent-substrate-v1
  model:
    provider: vertex-ai
    name: gemini-1.5-pro
  capabilities:
    - name: code-interpreter
      timeout: 30s
    - name: web-search
  triggers:
    - type: webhook
      endpoint: /v1/audit-trigger
  resumptionPolicy:
    type: OnEvent
    checkpointInterval: Always
  resources:
    limits:
      cpu: "2"
      memory: "4Gi"

The AX Control Plane

Similar to the Kubernetes API server, the AX control plane watches these manifests and ensures the actual state of the agent fleet matches the desired state. If an agent crashes, AX restarts it from the last checkpoint. If you need to update the model version for 100 running agents, you simply update the YAML and let AX handle the rolling transition.

This declarative approach allows DevOps teams to manage agents using the same CI/CD pipelines they use for their traditional backend services. It also makes it easier to implement security best practices, such as fixing JWT vulnerabilities in Node.js boilerplates or managing API keys, by injecting secrets directly into the agent’s environment through the manifest.

Connectivity and Security: MCP and Sandboxing

AI agents are inherently risky. To be useful, they need access to your tools, your data, and sometimes your internet connection. However, giving an LLM-driven agent the ability to execute code or call APIs is a recipe for disaster if not handled correctly. AX addresses this through two primary mechanisms: the Model Context Protocol (MCP) and Lightweight Sandboxing.

Model Context Protocol (MCP)

AX leverages MCP as a standardized way for agents to interact with external tools. Think of MCP as a universal interface—a “USB port” for AI. Instead of writing custom integration code for every tool (Jira, GitHub, Slack, SQL databases), developers can use MCP-compliant connectors.

This standardization allows AX to:

  1. Audit Tool Calls: Every interaction between the agent and a tool is logged and can be gated by the orchestrator.
  2. Standardize Context: MCP ensures that the data coming back from a tool is formatted in a way the agent can easily consume, reducing the need for manual prompt engineering.

Sandboxing and Network Fencing

Security in AX isn’t just about permissions; it’s about physical isolation. Agent Substrate uses lightweight sandboxing (often based on technologies like gVisor or WebAssembly) to ensure that even if an agent is “tricked” into running malicious code via a prompt injection attack, it cannot escape its environment.

Network Fencing is a key part of this. You can define strict egress rules in your AX manifest:

  • Allow the agent to talk to your internal PostgreSQL database.
  • Allow the agent to talk to the Gemini API.
  • Block all other outgoing traffic to prevent data exfiltration.

This level of granular control is essential for enterprise-grade AI, where the risk of an agent accidentally leaking sensitive data to a public API is a major barrier to adoption.

Practical Use Case: From Root Cause Analysis to Agentic Coding

To see the value of AX, let’s look at two high-stakes scenarios where traditional infrastructure struggles.

Scenario 1: Automated Root Cause Analysis (RCA)

When a production system goes down, the “Velocity Crisis” of modern development means there are often too many moving parts for a human to diagnose quickly. An AX-managed agent can be triggered by an alert.

  1. The agent wakes up (Sub-second resumption).
  2. It uses MCP to query logs and metrics.
  3. It realizes it needs to wait for a long-running database diagnostic script.
  4. AX checkpoints the agent, freeing up the GPU.
  5. Once the script finishes, AX resumes the agent, which then suggests a fix.

By using AX, the organization ensures that the agent’s context is preserved throughout the entire multi-hour investigation without paying for idle compute. This process relies heavily on the principles of context engineering to ensure the agent doesn’t get overwhelmed by “log noise.”

Scenario 2: Agentic Coding and the Velocity Crisis

In the world of agentic coding and code review, agents are now writing and reviewing pull requests. This requires an environment where the agent can safely compile and test code.

AX provides the perfect substrate for this:

  • Isolation: Each “coding session” happens in a clean sandbox.
  • Stateful Review: An agent can propose a change, wait for a human developer to comment, and then resume its “thought process” to address the feedback.
  • Resource Management: Coding agents often require significant CPU for running tests. AX can dynamically allocate these resources only when the tests are actually running.

The Future of Agentic Infrastructure: A New Standard?

AX is more than just a new Google Cloud product; it is a proposal for how we should build the next generation of the web. As agents become the primary way we interact with data and services, the “request-response” cycle of the 2010s will feel increasingly antiquated.

We are moving toward a future of Agent-Native infrastructure, where the focus shifts from managing servers to managing “agentic lifecycles.” AX provides a glimpse into this future by solving the three biggest hurdles to agent adoption:

  1. Cost: Through checkpointing and idle compute elimination.
  2. Complexity: Through declarative, Kubernetes-like orchestration.
  3. Security: Through sandboxing and standardized protocols like MCP.

The competition in this space is heating up. While specialized orchestrators like AX offer deep optimization, we will likely see general-purpose cloud providers trying to retrofit these features into their existing serverless and container offerings. However, the “clean slate” approach of AX—building on the Actor Model from day one—gives it a significant advantage in handling the unique, stateful demands of AI.

For DevOps engineers and AI architects, the message is clear: it’s time to stop treating agents like standard web apps. Whether AX becomes the industry-wide standard or paves the way for others, the move toward stateful, checkpointed, and sandboxed orchestration is the only way to bridge the gap between AI potential and production reality.