Self-Hosting a Private Terminal AI Coding Assistant: Ollama, Tailscale, and Qwen Code

The modern developer workstation is increasingly cluttered with AI coding assistants, but the underlying economics and infrastructure are starting to crack. If you manage a growing engineering team, you have likely run into the dual walls of SaaS subscription bloat and strict enterprise data governance. Per-seat SaaS pricing scales poorly as headcounts grow, turning basic developer tooling into a massive, recurring line item. More critically, strict regulatory frameworks and proprietary code policies often prohibit engineering teams from streaming internal codebases, proprietary algorithms, and sensitive git diffs to third-party cloud endpoints.

Fortunately, there is a pragmatic escape hatch. By combining open-weight models, local orchestration tools, and zero-trust networking, you can build a private, high-performance terminal AI coding assistant that keeps your code entirely on-premise. In this guide, we will walk through how to set up a distributed client-engine architecture using Ollama, Tailscale, and Qwen Code, allowing a team of approximately 10 developers to share a single internal GPU workstation with zero recurring API token costs.

The Distributed Client-Engine Architecture

To understand how a self-hosted AI coding assistant works in practice, we need to separate the user-facing interface from the heavy lifting of tensor processing. The architecture relies on a distributed client-engine pattern, splitting responsibilities between individual developer laptops and a centralized, headless GPU workstation.

+---------------------------------------------------+       +--------------------------------------------------+
|               Developer Laptop                    |       |              Centralized GPU Server              |
|                                                   |       |                                                  |
|  +---------------------------------------------+  |       |  +--------------------------------------------+  |
|  |          @qwen-code/qwen-code CLI           |  |       |  |                 Ollama                     |  |
|  |     (Node.js v20+ LTS / Terminal UI)        |  |       |  |   (Manages VRAM, Model Weights, Inference) |  |
|  +---------------------------------------------+  |       |  +--------------------------------------------+  |
|                         |                         |       |                         ^                        |
+-------------------------|-------------------------+       +-------------------------|------------------------|
                          |                                                           |
                          +========= Encrypted WireGuard Mesh (Tailscale) ============+

In this setup, the developer interacts with a local CLI client running on their machine. Instead of pointing that client to an external SaaS endpoint like OpenAI or Anthropic, the client fires standard, OpenAI-compatible v1 API requests directly over an encrypted network to your internal hardware.

At the core of this infrastructure is Tailscale, which creates a secure WireGuard mesh network connecting your engineers’ laptops to the headless GPU server. This completely eliminates the need for traditional, clunky VPN concentrators or exposing vulnerable ports to the public internet.

The hardware requirements are remarkably modest for small teams. By sharing a single idle GPU workstation—such as a machine equipped with an enterprise-grade NVIDIA GPU—around 10 developers can query the model throughout the day without stepping on each other’s toes, effectively bypassing per-seat licensing fees while maintaining absolute data privacy.

Setting Up the Centralized GPU Inference Server with Ollama

The first step in building your private AI stack is preparing the centralized inference server. We will use Ollama because it handles model weights, VRAM allocation, and provides an OpenAI-compatible API layer out of the box.

First, install Ollama on your headless GPU workstation following the official installation script for your Linux distribution:

curl -fsSL https://ollama.com/install.sh | sh

Once installed, pull the Qwen Code model variant. Depending on your GPU’s VRAM capacity, you can select the model size that best fits your hardware constraints:

ollama pull qwen2.5-coder

By default, Ollama binds strictly to 127.0.0.1, making it accessible only locally on the server. Because we want this server to accept requests from our team across the Tailscale mesh, we need to reconfigure Ollama to listen on the specific Tailscale network interface IP.

On Linux systemd-based systems, edit the Ollama service configuration:

sudo systemctl edit ollama.service

Add the following environment variable override under the [Service] block to bind Ollama to your Tailscale IP address (replace 100.x.x.x with your server’s actual Tailscale IP) and keep it open on port 11434:

[Service]
Environment="OLLAMA_HOST=100.64.0.10:11434"

Save the file and restart the Ollama service to apply the changes:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Verify that the server is up, healthy, and listening on the correct interface by hitting its headless health check and API endpoints from another machine on your mesh:

curl http://100.64.0.10:11434/v1/models

If configured correctly, you will receive a JSON payload listing your loaded and available models, proving the inference engine is ready to accept traffic.

Securing the Mesh: Connecting Laptops via Tailscale

Exposing an AI inference endpoint securely is historically where internal IT projects stall. Traditional approaches require configuring static IP whitelists, corporate VPN concentrators, or reverse proxies with mutual TLS. Tailscale bypasses this friction by building a zero-trust WireGuard mesh network directly on top of your existing infrastructure.

To connect your engineering team:

1. Install Tailscale on the centralized GPU server and authenticate it via your identity provider (such as Google Workspace, GitHub, or Okta).
2. Have each developer install the Tailscale client on their local laptop and authenticate using the same corporate SSO.
3. Verify connectivity by pinging the GPU server’s MagicDNS hostname or its direct Tailscale IP from a developer laptop:

tailscale ping gpu-server-name

Because Tailscale uses the WireGuard protocol, all traffic flowing between developer laptops and the GPU server is fully encrypted in transit. More importantly, your Ollama server remains entirely invisible to the public internet. There are no firewall ports to open on your office router or cloud provider security group, dramatically reducing your attack surface and satisfying strict compliance audits.

Integrating and Configuring the Qwen Code CLI

With the inference engine running and the secure network bridge established, it is time to set up the developer-facing client. We will use the @qwen-code/qwen-code terminal tool, which requires a modern JavaScript runtime environment.

System Prerequisites

Ensure that your development machines are running Node.js v20+ LTS, which is the minimum requirement for executing the CLI client reliably:

node -v
# Output should be v20.x.x or higher

Installing the CLI Tool

Install the Qwen Code CLI globally via your preferred package manager:

npm install -g @qwen-code/qwen-code

Configuring Environment Variables

The Qwen Code CLI is designed to communicate with OpenAI-compliant endpoints. To redirect requests away from commercial APIs and toward your private Tailscale-secured server, set the following environment variables in your local shell profile (~/.bashrc, ~/.zshrc, or project-specific .env files):

export OPENAI_API_BASE="http://100.64.0.10:11434/v1"
export OPENAI_API_KEY="ollama" # Ollama does not require a real token, but the client expects a string
export QWEN_MODEL="qwen2.5-coder"

Once configured, launch the terminal assistant:

qwen-code

The CLI will initiate, routing all prompt context and code generation requests securely over the WireGuard mesh directly to your team’s shared GPU workstation.

Managing Production Realities: Concurrency, VRAM, and Bottlenecks

While running a self-hosted AI stack offers incredible privacy and financial benefits, engineering managers and DevOps leads must account for a few operational bottlenecks when scaling this setup to a team of 10 engineers.

1. Concurrency and Silent Queueing

Unlike cloud hyperscalers with massive elastic clusters, a single shared workstation has finite compute resources. Ollama handles roughly 4 parallel inference requests concurrently on a standard enterprise GPU setup. When a 5th request comes in while the GPU is saturated, Ollama does not throw an error; instead, it silently queues incoming requests. Developers may notice slight latency spikes if multiple team members trigger heavy code completions simultaneously.

2. VRAM Management and Model Eviction

Be cautious about deploying multiple models on the same server. If a developer queries a secondary model (for instance, swapping from a coding model to a general-purpose language model), Ollama must evict the currently active model from the GPU VRAM and load the new weights from disk into memory. This eviction and reloading cycle introduces a 30 to 60-second delay for subsequent team queries. To maintain a smooth developer experience, standardize on a single primary coding model across the engineering team.

3. Monitoring Server Health

Because the infrastructure is shared, implement basic health checks and monitoring on the GPU server. Keep an eye on VRAM utilization using nvidia-smi and track service uptime to ensure unexpected crashes do not disrupt your team’s workflow.

watch -n 1 nvidia-smi

Future Outlook: The Shift to Localized Enterprise AI

The rapid advancement of open-weight models has fundamentally altered the economics of developer tooling. As open-source coding models continue to close the performance gap with proprietary closed-source alternatives, the justification for paying exorbitant per-seat SaaS fees is rapidly evaporating.

This technical shift mirrors broader macroeconomic movements in software engineering and IT infrastructure, where teams are actively reclaiming control over their tech stacks to protect margins and eliminate software bloat (as explored further in discussions on the AI deflationary spiral in IT outsourcing). By pairing high-performance local inference engines like Ollama with zero-trust networking layers like Tailscale, engineering organizations can bypass unpredictable subscription costs entirely.

Self-hosting your terminal AI coding assistant is no longer an academic exercise or a compromise in capability. It is a resilient, secure, and economically sound architecture that guarantees absolute data ownership and predictable infrastructure costs as your engineering team scales.