Serve One LLM Request on GKE Autopilot

This is the first milestone in the Multi-Tenant LLM Inference Platform project.

The goal sounds small: send one prompt to a language model running on a GPU and receive a response. But the request depends on several systems working together. Google Cloud must allow the GPU. GKE must find physical capacity. Kubernetes must schedule the Pod. The container must see the GPU. vLLM must download and load the model. Only then can it serve a request.

This field note explains that path slowly. It shows the important parts of the code, but not every line. The complete implementation is in GitHub.

When you are ready to reproduce it, use the companion guide: Run One LLM Request on GKE Autopilot. It gives you one command and one checkpoint at a time. The resource files are already in the repository.

What we wanted to prove

The completion test was precise:

1. Create a GKE Autopilot cluster from code.
2. Ask Kubernetes for one NVIDIA L4 GPU.
3. Run Qwen3-1.7B with vLLM.
4. send normal and streaming chat requests.
5. Prove what happens when a container does not request a GPU.
6. Save the evidence.
7. Delete every billable lab resource.

This milestone did not test multiple tenants, concurrent traffic, batching, fairness, autoscaling, or a public API. Those features would hide the first question under too many moving parts.

The system we built

Laptop
  |
  | kubectl port-forward
  v
Private Kubernetes Service
  |
  v
vLLM Pod
  |
  v
Qwen3-1.7B on one NVIDIA L4

Terraform created the Google Cloud foundation. Kubernetes manifests described the application inside the cluster. This boundary matters:

• Terraform owned the VPC, subnet, service account, permissions, and GKE cluster.
• Kubernetes owned the namespace, ComputeClass, Deployment, Pod, and Service.

The Kubernetes Service was private. kubectl port-forward created a temporary connection from the laptop to that Service. We did not create a public load balancer or expose the model to the internet.

Step 1: Prepare the Google Cloud project

What we did

We used the Google Cloud project tejo-llm-inference-lab. Billing was enabled. We also enabled three APIs:

• Compute Engine API, which manages virtual machines, networks, and GPUs
• Kubernetes Engine API, which manages GKE clusters
• Identity and Access Management API, which manages service accounts and permissions

Why this was necessary

A Google Cloud project is the administrative boundary for the lab. It holds billing, quotas, permissions, and resources. Enabling billing does not start a machine. It only lets the project create billable resources later.

Google Cloud services expose separate APIs. A project cannot create a GKE cluster just because it exists. The Kubernetes Engine API must also be enabled.

Terraform highlight

The configuration records the required APIs as a set:

locals {
  required_services = toset([
    "compute.googleapis.com",
    "container.googleapis.com",
    "iam.googleapis.com",
  ])
}

resource "google_project_service" "required" {
  for_each = local.required_services

  project            = var.project_id
  service            = each.value
  disable_on_destroy = false
}

for_each creates one Terraform-managed API resource for every value in the set. disable_on_destroy = false tells Terraform not to turn the APIs off during lab cleanup. Other resources in the same project may still need them.

The complete Terraform configuration is in GitHub.

Alternatives we considered

We could enable APIs through the Google Cloud console or with gcloud. Both work. Keeping the list in Terraform makes the requirement reviewable and repeatable.

What to remember

Billing, APIs, quotas, and running resources are different things. Having billing and an enabled API does not mean a GPU is running or costing money.

Step 2: Check both GPU quotas

What we did

We checked the regional NVIDIA L4 quota and the global all-GPUs quota. The project initially had:

regional L4 quota = 1
global all-GPUs quota = 0

We requested a global quota of one. Google approved it automatically.

Why this was necessary

The regional quota says how many L4 GPUs the project may use in a region. The global quota limits GPU use across all regions. Both limits must permit the request.

This created a useful failure. Looking only at the regional L4 quota suggested that the project was ready. It was not. The global limit still blocked every GPU allocation.

Alternatives we considered

Spot GPUs were also available in quota. They cost less, but Google can interrupt them. We used on-demand capacity because the first experiment needed a stable worker while we studied startup.

What to remember

Quota is permission, not inventory. A quota of one lets the project ask for one GPU. It does not reserve a physical GPU in a data center.

Step 3: Choose a region and record the cost

What we did

We compared us-central1, us-east1, us-west1, and us-west4. The first three had the same measured low G2 price during the experiment. We first selected us-central1 because it had three zones that listed G2 machines and the project had L4 quota there.

The first estimate used a g2-standard-4 machine:

one L4 GPU                    $0.560040239/hour
four vCPUs                    $0.099952848/hour
16 GiB memory                 $0.046839168/hour
Autopilot resource premiums   $0.084600000/hour
estimated total               $0.791432255/hour

Storage, network traffic, taxes, and future price changes were outside that estimate.

What actually happened

GKE could not create g2-standard-4 in any of the three tested us-central1 zones. It returned GCE out of resources.

We moved to us-east1. The smaller machine also failed in two tested zones. A g2-standard-8 with the same single L4 finally succeeded in us-east1-c.

The working shape was estimated at $0.955824271 per hour. The extra cost came from more CPU and memory, not from an additional GPU.

Why we kept the failed decision

ADR 0003 records the original us-central1 choice. ADR 0005 explains why it changed.

Keeping both records shows that the first decision was reasonable using price and quota data, but incomplete once we observed real capacity.

What to remember

Cloud capacity is a runtime condition. Region tables and quota pages cannot guarantee that a specific machine will be available when you need it.

Step 4: Tell Terraform which provider and inputs to use

What we did

Terraform needs a provider to communicate with Google Cloud. We pinned the Google provider to major version 7 and required Terraform 1.10 or newer.

terraform {
  required_version = ">= 1.10.0"

  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 7.0"
    }
  }
}

provider "google" {
  project = var.project_id
  region  = var.region
}

The version constraint ~> 7.0 accepts compatible 7.x releases but not version 8. That reduces the chance that a breaking provider change silently alters the lab.

The main inputs were simple:

variable "project_id" {
  type    = string
  default = "tejo-llm-inference-lab"
}

variable "region" {
  type    = string
  default = "us-east1"
}

The region variable also validates that the value is one of the low-cost regions we reviewed. This is a guardrail, not a claim that those regions will always be cheapest.

Alternatives we considered

We considered the Google Cloud console, gcloud scripts, Terraform, and OpenTofu. Terraform gave us a readable plan, dependency tracking, and one cleanup command. Its cost is another tool and a state file that we must protect.

For this single-person lab, state stayed local and outside Git. Before a second operator or CI system can apply infrastructure, we need a shared remote state backend with locking.

What to remember

Terraform code describes desired cloud state. The state file connects that description to real resource identities. The code is safe to publish. The state file may contain sensitive or environment-specific data and must not be committed.

Step 5: Create the network

What we did

We created a custom Virtual Private Cloud, or VPC. A VPC is the private network boundary for the cluster.

resource "google_compute_network" "platform" {
  name                    = "llm-inference-lab"
  auto_create_subnetworks = false
}

Disabling automatic subnet creation means Google does not create one subnet in every region. We create only the subnet this lab needs.

The regional subnet has one primary address range and two secondary ranges:

resource "google_compute_subnetwork" "platform" {
  region        = var.region
  network       = google_compute_network.platform.id
  ip_cidr_range = "10.10.0.0/20"

  secondary_ip_range {
    range_name    = "gke-pods"
    ip_cidr_range = "10.20.0.0/16"
  }

  secondary_ip_range {
    range_name    = "gke-services"
    ip_cidr_range = "10.30.0.0/20"
  }
}

The primary range serves the subnet itself. GKE uses one secondary range for Pod addresses and another for Service addresses. Keeping these ranges separate makes address ownership clear.

Alternatives we considered

We could use Google Cloud's default network. A dedicated VPC makes the lab boundary explicit and lets Terraform remove the entire network after the experiment. The tradeoff is more configuration.

What to remember

Pod and Service IP addresses must come from somewhere. In this cluster, the subnet gives GKE explicit ranges for both.

Step 6: Give GKE nodes an identity

What we did

Each GKE node needs a Google Cloud service account. A service account is the machine identity used when the node calls Google Cloud APIs.

resource "google_service_account" "gke_nodes" {
  account_id   = "gke-autopilot-nodes"
  display_name = "GKE Autopilot node service account"
}

resource "google_project_iam_member" "gke_nodes" {
  role   = "roles/container.defaultNodeServiceAccount"
  member = "serviceAccount:${google_service_account.gke_nodes.email}"
}

The IAM binding grants the standard role Google expects for a GKE node service account. We did not use a broad owner or editor role.

Why this was necessary

Human users and machines should not share identities. A dedicated node identity gives us a place to review and limit the permissions used by cluster machines.

What to remember

The Kubernetes service account used by a Pod and the Google Cloud service account used by a node are different identities. This milestone configured the node identity. It did not yet give the model application its own Google Cloud permissions.

Step 7: Create the GKE Autopilot cluster

What we did

The cluster resource joined the network, subnet, IP ranges, release channel, and node service account:

resource "google_container_cluster" "platform" {
  name     = var.cluster_name
  location = var.region

  enable_autopilot    = true
  deletion_protection = false

  network    = google_compute_network.platform.id
  subnetwork = google_compute_subnetwork.platform.id

  ip_allocation_policy {
    cluster_secondary_range_name  = "gke-pods"
    services_secondary_range_name = "gke-services"
  }

  release_channel {
    channel = "REGULAR"
  }
}

enable_autopilot = true is the key choice. We describe the Pod's resource needs, and GKE manages the worker nodes needed to run it.

deletion_protection = false was appropriate for a short-lived lab that had to be destroyed. A long-lived production cluster would need a separate decision.

The Regular release channel balances access to current Kubernetes versions with more rollout time than the Rapid channel.

Important detail

Creating the Autopilot cluster did not create a GPU node. The GPU node appeared only after Kubernetes saw a pending Pod that requested one.

Terraform output

Terraform also returned the exact command needed to configure local kubectl access:

output "connect_command" {
  value = "gcloud container clusters get-credentials ${google_container_cluster.platform.name} --region ${google_container_cluster.platform.location} --project ${var.project_id}"
}

What to remember

The cluster is the control environment. The GPU worker is workload capacity. In Autopilot, those two resources do not need to appear at the same time.

Step 8: Review and apply the Terraform plan

How we ran it

terraform init
terraform fmt -check
terraform validate
terraform plan
terraform apply

init downloaded the provider. fmt checked formatting. validate checked the configuration structure. plan showed the proposed changes without creating them. apply created the approved plan.

Terraform created eight managed items in total: three enabled APIs, one VPC, one subnet, one node service account, one IAM binding, and one GKE cluster.

A failure we observed

The local Terraform client lost its connection while Google Cloud was still processing the cluster operation. Terraform marked the cluster as tainted, which normally means it plans to replace the resource.

We checked Google Cloud first. The cluster existed and matched the configuration. We then removed the taint and ran another plan. Terraform reported no changes.

This was safer than immediately creating a second cluster or destroying a healthy one.

What to remember

A client timeout does not prove that the cloud-side operation failed. Check the provider's real state before retrying a long-running create operation.

Step 9: Create the Kubernetes resources

Terraform stopped at the cluster boundary. We then created five Kubernetes resources.

Namespace

apiVersion: v1
kind: Namespace
metadata:
  name: inference-system

The namespace groups the model-serving objects. It also gives us one clear cleanup and policy boundary inside the cluster.

ComputeClass

apiVersion: cloud.google.com/v1
kind: ComputeClass
metadata:
  name: l4-medium-us-east1-c
spec:
  priorityDefaults:
    location:
      zones: [us-east1-c]
  priorities:
    - machineType: g2-standard-8
      gpu:
        type: nvidia-l4
        count: 1
  nodePoolAutoCreation:
    enabled: true

A ComputeClass is a GKE resource that describes the machine shape and location to use for a workload. This one allowed GKE to create a node pool containing a g2-standard-8 with one L4.

We applied the ComputeClass before the Deployment. Applying both together failed because GKE checked the Deployment's ComputeClass reference before the custom resource was ready.

Deployment

A Deployment says how Kubernetes should keep an application running. The important scheduling section was:

spec:
  replicas: 1
  strategy:
    type: Recreate
  template:
    spec:
      nodeSelector:
        cloud.google.com/compute-class: l4-medium-us-east1-c
      containers:
        - name: vllm
          resources:
            limits:
              nvidia.com/gpu: "1"

The node selector chooses our ComputeClass. The GPU limit is also a GPU request. It tells the scheduler that the container must have one GPU device.

We used Recreate because two copies of this worker could otherwise briefly require two GPUs during an update. The project quota allowed only one.

Service

apiVersion: v1
kind: Service
metadata:
  name: vllm
  namespace: inference-system
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: vllm
  ports:
    - port: 8000
      targetPort: http

Pods are replaceable and their IP addresses may change. The Service provides one stable private name and address for Pods matching the label selector.

ClusterIP means the Service is reachable only inside the cluster unless we create a temporary tunnel such as port forwarding.

Kustomization

resources:
  - namespace.yaml
  - deployment.yaml
  - service.yaml

Kustomize groups related manifests so we can apply or delete them together. The ComputeClass stayed outside this list because it had to exist before admission checked the Deployment.

The complete Kubernetes manifests are in GitHub.

What to remember

These objects have different jobs: the Namespace groups, the ComputeClass chooses capacity, the Deployment maintains the application, the Pod runs it, and the Service gives it a stable private endpoint.

Step 10: Configure the vLLM container

What we did

The Deployment ran a pinned vLLM image and passed the model settings as arguments:

image: vllm/vllm-openai:v0.26.0@sha256:ffb2d59b...
args:
  - Qwen/Qwen3-1.7B
  - --served-model-name=qwen3-1.7b
  - --dtype=bfloat16
  - --max-model-len=8192
  - --gpu-memory-utilization=0.85
  - --generation-config=vllm

The digest identifies the exact container contents. A mutable tag alone can point to different bytes later.

The important runtime choices were:

• Qwen3-1.7B was small enough to leave clear GPU memory headroom.
• BF16 used a modern 16-bit numeric format supported by the L4.
• The 8,192-token context limit created a smaller, predictable memory boundary than the model's full supported context.
• The 0.85 memory setting let vLLM use most of the GPU while leaving some room outside its planned allocation.

The model cache used a 20 GiB emptyDir volume. This storage lives with the Pod. Replacing the Pod loses the cache and downloads the model again. That was simple for one experiment but is not a final production cache design.

/dev/shm used a memory-backed emptyDir. This gives processes fast shared memory for runtime communication.

Health probes

All three probes called /health:

• The startup probe allowed up to 30 minutes for the first model load.
• The readiness probe controlled whether the Service could send traffic to the Pod.
• The liveness probe restarted the container if a running server became unhealthy.

Startup, readiness, and liveness answer different questions. A slow model download should not look like a dead application.

Container security

The process ran as user 2000 instead of root. It could not gain extra privileges, and Linux capabilities were dropped. These settings reduce what a compromised model process can do inside the container.

They do not provide complete tenant isolation. This milestone had only one trusted workload.

Step 11: Fix the VLLM_PORT collision

What failed

The first vLLM process exited before loading the model. Kubernetes had created this environment variable because the Service was named vllm:

VLLM_PORT=tcp://...:8000

vLLM already uses VLLM_PORT and expects a number. It received a URI instead.

What we changed

spec:
  enableServiceLinks: false

Kubernetes stopped injecting legacy Service environment variables. Service discovery through DNS still worked.

Alternatives we considered

We could rename the Service or explicitly set VLLM_PORT=8000. Both solve this one collision. Disabling Service links removes the whole class of name collisions and keeps DNS as the normal discovery mechanism.

ADR 0006 records the decision.

What to remember

Environment variables can be part of an application's public interface even when you did not set them yourself. Inspect the final container environment when a process receives a surprising value.

Step 12: Watch the worker start

Once the fixed Pod was pending, Autopilot created a matching GPU node. The observed startup path was:

Pending Pod
  -> GPU node creation
  -> NVIDIA device becomes allocatable
  -> image pull
  -> model download
  -> weight load
  -> Torch compilation
  -> CUDA graph capture
  -> health check passes
  -> Pod becomes Ready

The main measurements were:

The image pull was fast even though the image was about 8.9 GB. That is an observation from this run, not a general guarantee.

GPU memory

The L4 reported 23,034 MiB total memory. After model startup, 20,124 MiB was in use.

The model weights used 3.22 GiB. vLLM assigned 14.85 GiB to the KV cache. The KV cache stores information from tokens already processed so the model does not recompute the whole conversation for every new token.

vLLM estimated room for 139,056 cached tokens and 16.97 concurrent requests at the configured 8,192-token maximum. These were memory estimates from the runtime, not measured throughput.

What to remember

Model weights are only one part of GPU memory use. Runtime workspaces, compiled graphs, and the KV cache can use much more memory than the weights.

Step 13: Send private normal and streaming requests

What we did

We opened a temporary local tunnel:

kubectl port-forward -n inference-system service/vllm 8000:8000

Then we sent an OpenAI-compatible chat request to http://127.0.0.1:8000/v1/chat/completions.

The request selected qwen3-1.7b, limited output to 64 tokens, set temperature to zero, and disabled the model's thinking mode. Disabling thinking made this small test easier to inspect. It was not an output-quality comparison.

Result

The normal request returned HTTP 200:

first byte:       0.985264 seconds
total time:       0.985650 seconds
prompt tokens:    29
completion tokens: 40

A warm streaming request also returned HTTP 200:

first byte: 0.333258 seconds
total time: 1.273798 seconds

The stream ended with the expected [DONE] event.

Important limit

curl measured the first byte of the server-sent event stream. That is close to, but not exactly, time to first generated content token. Milestone 2 needs a client that parses the stream and timestamps the first actual token.

What to remember

A successful HTTP response proves the serving path works. A few request timings do not prove latency, throughput, or capacity. That requires a controlled benchmark.

Step 14: Prove that the GPU request matters

What we did

We ran a deliberate failure Pod using the same CUDA-enabled image but without this resource request:

limits:
  nvidia.com/gpu: "1"

The check was small:

visible = torch.cuda.is_available()
print(f"cuda_available={visible}")
assert visible, "CUDA device is not visible; request nvidia.com/gpu"

Result

Kubernetes scheduled the Pod on a normal CPU node. The image contained CUDA software, but the container had no GPU device assigned. It printed:

cuda_available=False

and exited with code 1, as expected.

Why this failure matters

A CUDA-enabled container image does not give a Pod a GPU. The Pod must request the extended Kubernetes resource nvidia.com/gpu. The scheduler then places it on a compatible node, and the device plugin exposes the device to the container.

The complete missing-GPU experiment is reproducible from GitHub.

Step 15: Delete and verify the lab

What we did

We removed the application objects and ComputeClass. Then we reviewed a destroy plan and ran Terraform destroy.

kubectl delete -k infrastructure/kubernetes/base
kubectl delete -f infrastructure/kubernetes/base/compute-class.yaml
terraform plan -destroy
terraform destroy

Terraform reported:

0 added, 0 changed, 8 destroyed

We did not trust one success message. We checked the relevant systems again:

GKE clusters:                 0
Compute Engine instances:    0
lab networks:                0
Terraform state resources:   0

The Google Cloud project, billing connection, API enablement, and approved quotas remain. No cluster or GPU machine remains from the lab.

Why cleanup is part of the milestone

Stopping kubectl port-forward only closes the local tunnel. It does not stop the Pod, GPU node, or cluster. A cloud experiment is not complete until we prove that its billable resources are gone.

What to remember

Treat cleanup as a tested feature. Use both the infrastructure tool and read-only cloud checks to verify the result.

What changed in my understanding

I started with a simple model: if a region lists L4 machines and the project has quota, Autopilot should create the worker. The experiment showed why that model is incomplete.

Three separate conditions must hold:

the machine type is supported
          +
the project has quota
          +
physical capacity is available now
          =
the worker can be created

The run also showed that application startup can fail because of a Kubernetes-generated environment variable before any model code runs. The failure was not in CUDA, the model, or GPU capacity.

Finally, model weights were not the largest consumer of GPU memory. The 3.22 GiB of weights sat beside a 14.85 GiB KV cache and other runtime allocations. This is why choosing a GPU from model file size alone is not enough.

What this milestone proves—and what it does not

It proves that one Qwen3-1.7B worker can start on one L4 in GKE Autopilot and serve private normal and streaming requests using this configuration.

It does not prove:

• that us-east1-c will have an L4 tomorrow
• that the measured request times are stable
• how throughput changes with concurrency
• whether an L4 is the best GPU for this model
• how tenants should share the worker
• whether the worker recovers safely under load
• whether the current cache and startup design is suitable for production

Those limits define the next milestone: measure one worker under a controlled workload.

Repository map

Use this article to understand the implementation. Use GitHub when you want to reproduce or inspect it:

• Milestone definition
• Terraform files
• Kubernetes manifests
• Runbook
• Measured result
• Architecture decision records
• Draft pull request

The main lesson

One LLM request crosses more boundaries than it first appears to. It depends on cloud permissions, quotas, physical capacity, networking, scheduling, device assignment, container startup, model loading, GPU memory, service routing, and cleanup.

The useful habit is to make each boundary visible. Record its desired state, observe what actually happened, and keep the evidence. That turns a successful demo into something another engineer can understand and reproduce.