Run One LLM Request on GKE Autopilot

This is the hands-on companion to Serve One LLM Request on GKE Autopilot.

You will create the same Milestone 1 environment yourself. You will not write Terraform or Kubernetes resource files. The complete files already exist in GitHub. Your job is to run each step, inspect what happened, and decide whether it is safe to continue.

The lab creates a GKE Autopilot cluster and may create one NVIDIA L4 GPU worker. During the original run, the working g2-standard-8 worker was estimated at about $0.96 per running hour, before storage, network traffic, taxes, and price changes. Current prices and capacity may differ.

How to use this guide

Work through one numbered section at a time.

Each section contains:

• Run: the command to execute
• Understand: what the command does
• Check: what to inspect
• Continue when: the condition that must be true before moving on

Do not continue merely because a command printed no error. Check the resulting state.

1. Open a clean terminal

Use a normal terminal on your laptop. The commands use zsh or bash syntax. On Windows, use a Linux shell through WSL.

Do not run this guide from the tejo.dev repository. The infrastructure has its own repository and Terraform state.

2. Check the required tools

Run

gcloud version
terraform version
kubectl version --client
gke-gcloud-auth-plugin --version
git --version
curl --version
jq --version

Understand

You need:

• gcloud to authenticate and inspect Google Cloud
• Terraform 1.10 or newer to create and remove cloud resources
• kubectl to work with Kubernetes resources
• gke-gcloud-auth-plugin to authenticate kubectl to GKE
• Git to download the prepared files
• curl to send HTTP requests
• jq to format JSON responses

The GKE authentication plugin is separate from kubectl. A working kubectl installation does not prove that it can authenticate to GKE.

Check

Every command should print a version. Confirm that Terraform reports 1.10 or newer.

If the GKE plugin is missing and your Google Cloud CLI supports components, install it with:

gcloud components install gke-gcloud-auth-plugin

Use the official GKE cluster-access instructions if your package manager installed gcloud without the component manager.

Continue when

All seven version commands succeed.

3. Download the exact Milestone 1 implementation

Choose a directory where you keep projects.

Run

git clone https://github.com/tejokumar/multi-tenant-llm-inference-platform.git
cd multi-tenant-llm-inference-platform
git switch --detach 7ff43a2e1beb5de92b305ead6465c828b8da308f
git rev-parse --short HEAD
git status --short

Understand

The long value is the exact Git commit used for the first successful run. A detached checkout means you are inspecting that fixed version instead of following a branch that may change later.

The repository already contains:

infrastructure/terraform/          Google Cloud resources
infrastructure/kubernetes/base/    Kubernetes resources
experiments/001-missing-gpu/       deliberate failure
docs/runbooks/milestone-01.md       compact operator runbook
evidence/milestone-01/             original measurements

Check

The commit command should print:

7ff43a2

The status command should print nothing. That means there are no modified or untracked files.

If the directory already exists, do not clone over it. Enter the existing repository, run git status, and make sure you understand any local changes before switching commits.

Continue when

You are at commit 7ff43a2 and the working tree is clean.

4. Sign in to Google Cloud

Run

gcloud auth login
gcloud config set project tejo-llm-inference-lab
gcloud auth list --filter=status:ACTIVE --format='value(account)'
gcloud config get-value project

Understand

The first command signs your local Google Cloud CLI in. The second sets the default project for later commands. Neither command creates infrastructure.

The active account is your human identity. Terraform will use a short-lived access token from this login when it talks to Google Cloud.

Check

The active-account command should print the Google account you expect. The project command should print:

tejo-llm-inference-lab

Continue when

The correct account and project are active.

5. Confirm billing and API access

Run

gcloud billing projects describe tejo-llm-inference-lab \
  --format='table(projectId,billingEnabled,billingAccountName)'

gcloud services list --enabled \
  --filter='config.name:(compute.googleapis.com OR container.googleapis.com OR iam.googleapis.com)' \
  --format='table(config.name)'

Understand

Billing allows the project to create paid resources. The enabled APIs expose Compute Engine, GKE, and IAM operations.

These checks do not create resources and do not start billing by themselves.

Check

billingEnabled should be True. The API list should contain:

compute.googleapis.com
container.googleapis.com
iam.googleapis.com

If an API is absent, Terraform can enable it during the apply. Billing must already be enabled.

Continue when

Billing is enabled. It is safe to continue if an API is absent because the reviewed Terraform plan manages these three APIs.

6. Confirm both GPU quotas

Run

gcloud compute project-info describe \
  --project tejo-llm-inference-lab \
  --format=json | \
  jq '.quotas[] | select(.metric == "GPUS_ALL_REGIONS") | {metric, limit, usage}'

gcloud compute regions describe us-east1 \
  --project tejo-llm-inference-lab \
  --format=json | \
  jq '.quotas[] | select(.metric == "NVIDIA_L4_GPUS") | {metric, limit, usage}'

Understand

The first command checks the global limit across all GPU types and regions. The second checks the NVIDIA L4 limit in us-east1.

Both limits must allow one additional GPU. Calculate available quota as:

available = limit - usage

Check

Both rows should have at least one available unit. During the original run, both limits were one and usage was zero.

If either command prints no row, open the Google Cloud Quotas page and search for GPUS_ALL_REGIONS and NVIDIA_L4_GPUS. Quota metric names and CLI presentation can change.

Continue when

Both quotas have at least one available unit.

Do not continue if quota is zero. A Terraform apply may create the cluster, but the model Pod will not receive a GPU.

7. Validate the prepared Terraform configuration

Move into the Terraform working directory.

Run

cd infrastructure/terraform
terraform init
terraform fmt -check
terraform validate

Understand

terraform init downloads the pinned Google provider and prepares the working directory. It is safe to run more than once.

terraform fmt -check confirms standard formatting. terraform validate checks that the files form a valid Terraform configuration. None of these commands create cloud resources.

The configuration uses default values for this lab:

project: tejo-llm-inference-lab
region:  us-east1
cluster: llm-inference-lab

You do not need to create a .tfvars file.

Check

Validation should end with:

Success! The configuration is valid.

Continue when

Formatting and validation both pass.

8. Preview the cloud changes

This step is read-only against the cloud. It creates a local plan file but does not create infrastructure.

Run

GOOGLE_OAUTH_ACCESS_TOKEN="$(gcloud auth print-access-token)" \
  terraform plan -out=milestone-01.tfplan

Understand

Terraform compares the prepared configuration, its local state, and Google Cloud. The access token is short-lived and stays in the command environment.

The saved plan ensures that the next step applies the exact changes you reviewed.

Check

On a clean first run, expect Terraform to propose these resource types:

google_project_service                 three required APIs
google_compute_network                 one VPC
google_compute_subnetwork              one subnet
google_service_account                 one node identity
google_project_iam_member              one role binding
google_container_cluster               one Autopilot cluster

Previously enabled APIs may still appear as Terraform resources because Terraform is beginning to manage them. Do not approve unexpected deletions or replacements.

You can summarize the saved plan with:

terraform show -no-color milestone-01.tfplan | less

Press q to leave less.

Cost checkpoint

The plan creates a GKE cluster. The GPU worker is created later, when the model Pod requests it. Google Cloud pricing and any free-tier treatment can change. Check your billing budget before applying.

Continue when

The plan contains only the reviewed lab resources and no unexpected destroy or replace action.

9. Create the cloud foundation

This is the first billable action.

Run

GOOGLE_OAUTH_ACCESS_TOKEN="$(gcloud auth print-access-token)" \
  terraform apply milestone-01.tfplan

Understand

Terraform sends the approved operations to Google Cloud. Cluster creation can take several minutes. Do not close the terminal just because the command pauses without new output.

Check

At completion, Terraform should report a successful apply and print:

cluster_name   = "llm-inference-lab"
cluster_region = "us-east1"

Confirm the cluster directly:

gcloud container clusters list \
  --project tejo-llm-inference-lab \
  --filter='name=llm-inference-lab' \
  --format='table(name,location,status,currentMasterVersion)'

The status should become RUNNING.

If Terraform loses its connection, do not immediately rerun apply. Use the read-only gcloud container clusters list command first. A client failure does not prove that Google Cloud stopped the operation.

Continue when

The llm-inference-lab cluster is RUNNING and Terraform finished successfully. If Terraform failed but the cluster exists, stop and reconcile Terraform state before continuing.

10. Connect kubectl to the cluster

Run

gcloud container clusters get-credentials llm-inference-lab \
  --region us-east1 \
  --project tejo-llm-inference-lab

kubectl config current-context
kubectl get namespaces

Understand

get-credentials adds the cluster endpoint and authentication configuration to your local Kubernetes configuration. It also changes the current kubectl context.

The namespace query is the first end-to-end check from your terminal to the Kubernetes API.

Check

The current context should contain llm-inference-lab. The namespace list should include default, kube-system, kube-public, and kube-node-lease.

Continue when

kubectl get namespaces succeeds against the intended cluster.

11. Create the GPU ComputeClass first

Return to the repository root:

Run

cd ../..
kubectl apply -f infrastructure/kubernetes/base/compute-class.yaml
kubectl get computeclass l4-medium-us-east1-c -o yaml

Understand

The ComputeClass tells GKE which capacity to create for the model worker:

zone:          us-east1-c
machine type:  g2-standard-8
GPU:           one NVIDIA L4

We create it separately because GKE admission checks the Deployment's ComputeClass reference. In the original experiment, applying the ComputeClass and Deployment together caused an ordering failure.

This command creates a Kubernetes API object. It does not create the GPU node by itself.

Check

The returned object should show the name l4-medium-us-east1-c, machine type g2-standard-8, and GPU type nvidia-l4.

Continue when

The ComputeClass exists and contains the expected machine and GPU.

12. Create the model-serving resources

This is the GPU cost checkpoint. The Deployment requests one L4, so Autopilot will try to create a matching worker.

Run

kubectl apply -k infrastructure/kubernetes/base
kubectl get all -n inference-system

Understand

Kustomize applies the prepared Namespace, Deployment, and private Service. The Deployment creates a ReplicaSet, which creates one Pod.

The Pod asks for nvidia.com/gpu: 1 and selects the ComputeClass from the previous step. Autopilot sees the pending Pod and starts creating suitable capacity.

Check

Expect to see:

deployment.apps/vllm-qwen3
pod/vllm-qwen3-...
service/vllm

The Pod will normally begin as Pending. That is expected while GKE creates a GPU node.

Continue when

The Deployment, Pod, and Service exist. Do not wait for readiness in this terminal yet; the next step shows how to observe it.

13. Watch scheduling and startup

Use three terminals so you can observe different layers.

Terminal 1: Watch the Pod

kubectl get pods -n inference-system -w

Terminal 2: Watch Kubernetes events

kubectl get events -n inference-system --watch

Terminal 3: Follow application logs

kubectl logs -n inference-system deployment/vllm-qwen3 --follow

The logs command may initially report that the container is waiting. Run it again after the Pod reaches ContainerCreating or Running.

Understand

You are watching three different views:

• Pod state shows the high-level lifecycle.
• Events explain scheduling, node creation, image pulling, and failures.
• vLLM logs show model download, weight loading, compilation, memory allocation, and server startup.

Check

A healthy path looks roughly like:

Pending
  -> Scheduled
  -> ContainerCreating
  -> Running but not Ready
  -> Running and Ready

In another terminal, inspect the created node:

kubectl get nodes \
  -L cloud.google.com/gke-accelerator,node.kubernetes.io/instance-type,topology.kubernetes.io/zone

The model node should report nvidia-l4, g2-standard-8, and us-east1-c.

Wait explicitly for readiness:

kubectl wait -n inference-system \
  --for=condition=Ready \
  pod -l app.kubernetes.io/name=vllm \
  --timeout=30m

If capacity is unavailable

Events may report GCE out of resources or repeated autoscaler backoff. Quota cannot fix physical capacity.

Do not edit the resource files during this reproduction. Save the events:

kubectl get events -n inference-system --sort-by=.lastTimestamp

Then go to cleanup. A different machine shape, zone, or region is a design change and should be reviewed before retrying.

Continue when

The wait command reports that the Pod condition was met and kubectl get pods -n inference-system shows 1/1 Running.

14. Inspect the running GPU and server

Run

kubectl exec -n inference-system deployment/vllm-qwen3 -- \
  nvidia-smi --query-gpu=name,driver_version,memory.total,memory.used \
  --format=csv

kubectl logs -n inference-system deployment/vllm-qwen3 --tail=80

Understand

nvidia-smi talks to the GPU driver visible inside the container. It proves that Kubernetes assigned a GPU device to this Pod.

The log tail lets you inspect the completed vLLM startup without reading the entire log stream.

Check

The GPU name should be NVIDIA L4. The logs should show that the API server started and the model is ready. Exact driver versions, timings, and memory use may differ from the original run.

Continue when

The container sees one NVIDIA L4 and the Pod remains Ready.

15. Open a private tunnel

Keep this terminal running.

Run

kubectl port-forward -n inference-system service/vllm 8000:8000

Understand

Port forwarding connects local port 8000 to the private Kubernetes Service. It does not create a public endpoint.

The process owns the tunnel. Closing this terminal stops access but does not stop the cloud resources.

Check

Expect:

Forwarding from 127.0.0.1:8000 -> 8000

Continue when

The forwarding process is still running. Open a second terminal for the requests.

16. Check server health

Run

curl --fail --silent --show-error \
  http://127.0.0.1:8000/health

Understand

This calls the same health endpoint used by the Kubernetes probes. --fail makes curl return an error for an unsuccessful HTTP status.

Check

The command may return an empty body. Its success exit code is the important result. Confirm it with:

echo $?

The value should be 0.

Continue when

The health request succeeds.

17. Send one normal chat request

Run

curl --silent --show-error \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-1.7b","messages":[{"role":"user","content":"In one short sentence, explain what multi-tenancy means in an inference platform."}],"temperature":0,"max_tokens":64,"chat_template_kwargs":{"enable_thinking":false}}' \
  http://127.0.0.1:8000/v1/chat/completions | jq

Understand

The request uses vLLM's OpenAI-compatible chat endpoint. It selects the served model name, provides one user message, and limits the response to 64 tokens.

Thinking mode is disabled so the output is short and easy to inspect. This does not compare model quality.

Check

The JSON should contain:

• a model named qwen3-1.7b
• one assistant message under choices
• prompt and completion token counts under usage

Continue when

You receive one completed assistant message.

18. Send one streaming request

Run

curl --no-buffer --silent --show-error \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-1.7b","messages":[{"role":"user","content":"Name three signals we should measure for an LLM worker."}],"temperature":0,"max_tokens":64,"stream":true,"chat_template_kwargs":{"enable_thinking":false}}' \
  http://127.0.0.1:8000/v1/chat/completions

Understand

stream: true asks the server to return small server-sent events as generation proceeds. --no-buffer tells curl to print those events immediately.

Check

You should see several lines beginning with data:. The last event should be:

data: [DONE]

Continue when

The response arrives incrementally and ends with [DONE].

19. Run the deliberate missing-GPU failure

This experiment creates a small temporary Pod. It uses the same CUDA-enabled image but does not request a GPU.

Run

kubectl apply -f experiments/001-missing-gpu/pod.yaml

kubectl wait -n inference-system \
  --for=jsonpath='{.status.phase}'=Failed \
  pod/missing-gpu-check \
  --timeout=5m

kubectl logs -n inference-system missing-gpu-check
kubectl get pod -n inference-system missing-gpu-check -o wide

Understand

A container image can contain CUDA libraries without receiving a physical GPU. Kubernetes assigns a GPU only when the Pod requests the nvidia.com/gpu resource.

Without that request, the failure Pod can run on a normal node.

Check

The log should contain:

cuda_available=False

The Pod should be Failed. This is the expected result, not a broken lab.

Remove the experiment:

kubectl delete -f experiments/001-missing-gpu/pod.yaml

Continue when

You observed the expected failure and deleted the temporary Pod.

20. Review what is running before cleanup

Run

kubectl get all -n inference-system
kubectl get computeclass l4-medium-us-east1-c
gcloud compute instances list \
  --project tejo-llm-inference-lab \
  --format='table(name,zone,status,machineType.basename())'

Understand

This is your final inspection of the live environment. You should be able to connect each item to an earlier step.

Check

You should see the vLLM Deployment, Pod, private Service, ComputeClass, and at least one GKE-managed worker instance.

Continue when

You have finished the requests and saved any logs or notes you want to keep. The next steps delete the lab.

21. Delete the Kubernetes workload

Stop the port-forward process with Ctrl-C. Then run:

kubectl delete -k infrastructure/kubernetes/base
kubectl delete -f infrastructure/kubernetes/base/compute-class.yaml

Understand

This removes the namespace, model Deployment, Pod, private Service, and ComputeClass. Autopilot can then remove the GPU capacity that no workload needs.

Do not wait only for node scale-down. The next step removes the entire Terraform-managed cloud foundation.

Check

kubectl get namespace inference-system
kubectl get computeclass l4-medium-us-east1-c

Both commands should report NotFound after deletion completes.

Continue when

The workload namespace and ComputeClass are gone.

22. Preview and perform Terraform cleanup

Run

cd infrastructure/terraform

GOOGLE_OAUTH_ACCESS_TOKEN="$(gcloud auth print-access-token)" \
  terraform plan -destroy

Understand

The destroy plan shows which Terraform-managed objects will be removed. Review it before executing the destructive command.

The API resources use disable_on_destroy = false, so Terraform removes them from its state without turning the Google Cloud APIs off.

Check

The plan should target the lab cluster, network, subnet, node service account, IAM binding, and the three Terraform-managed API entries. It should not show unrelated resources.

When the plan is correct, run:

GOOGLE_OAUTH_ACCESS_TOKEN="$(gcloud auth print-access-token)" \
  terraform destroy

Terraform asks for confirmation. Read the final plan and type yes only if it contains the expected lab resources.

Continue when

Terraform reports a successful destroy.

23. Prove that cleanup finished

Run

gcloud container clusters list \
  --project tejo-llm-inference-lab \
  --filter='name=llm-inference-lab'

gcloud compute instances list \
  --project tejo-llm-inference-lab

gcloud compute networks list \
  --project tejo-llm-inference-lab \
  --filter='name=llm-inference-lab'

terraform state list

Understand

Terraform's success message is one source of evidence. These read-only checks ask Google Cloud and Terraform separately.

Check

The lab cluster query, instance list, lab-network query, and Terraform state should contain no lab resources.

If an unrelated VM exists in the project, the unfiltered instance list may not be empty. Do not delete it. Confirm only that no GKE worker from this lab remains.

Finish when

You can account for all four results:

lab cluster:                absent
lab GPU worker:             absent
lab network:                absent
Terraform-managed resources: 0

What you manually proved

You did more than send a prompt. You observed the complete path:

Terraform plan
  -> Google Cloud foundation
  -> Kubernetes desired state
  -> Autopilot GPU provisioning
  -> container and model startup
  -> private Service routing
  -> normal and streaming inference
  -> missing-GPU failure
  -> verified cleanup

You also saw why we separate explanation from execution. The Milestone 1 field note explains why each resource exists. This guide lets you operate those resources without copying hundreds of lines of configuration into a terminal.

Common stopping points

References

• Milestone 1 repository at the tested commit
• GKE: Deploy GPU workloads in Autopilot
• GKE: Install kubectl and configure cluster access
• Terraform CLI workflow
• Terraform plan reference
• Terraform destroy reference
• Project runbook