Serving one prompt is easy. Sharing limited GPU capacity between many users is not.
This project asks one question: How can several tenants share GPU workers without one workload making the platform slow or unreliable for everyone else?
I will answer it by building a real platform on Google Kubernetes Engine Autopilot. I will create failures, measure what happens, and publish the evidence in plain English.
The outcome
The finished platform will provide:
- an OpenAI-compatible streaming API
- tenant API keys and usage limits
- interactive and batch workloads
- bounded queues and fair scheduling
- routing across multiple model workers
- worker failure recovery and autoscaling
- metrics, traces, deployment files, and runbooks
The source and detailed evidence will live in the project repository. This page is the map.
How the project works
For each milestone, I will:
- Define the question and comparison criteria.
- Evaluate meaningful alternatives.
- Choose an approach and explain why.
- Build the smallest useful version.
- Break or stress it deliberately.
- Measure the result.
- Revisit the decision using evidence.
- Publish what I learned.
Important choices will become architecture decision records, or ADRs. ADRs will be published here in order. If a decision changes, the original ADR will remain and link to its replacement.
Project flow
- Serve one request. Run one model on one GPU worker in GKE Autopilot.
- Measure one worker. Record latency, throughput, queue time, and GPU use.
- Create overload. Add bounded queues, deadlines, cancellation, and admission control.
- Add tenants. Compare quotas, priority, and fair scheduling for interactive and batch traffic.
- Add workers. Route across multiple workers and recover when one fails.
- Add autoscaling. Compare request, queue, token, and GPU signals.
- Make it operable. Add dashboards, alerts, runbooks, rollout, and cost controls.
- Explain the system. Publish the final design and what the experiments changed.
Each milestone will produce working software, reproducible evidence, and a publication when there is a useful lesson to share.
Progress is based on completed functionality and verified learning outcomes, not time or target dates.
Completion test
The project is complete when another engineer can deploy it, create two tenants, send mixed workloads, observe fair sharing, test overload and worker failure, inspect scaling and traces, and destroy the environment cleanly.
The first version will not include training, fine-tuning, RAG, agents, custom GPU kernels, billing, or multi-region serving.
Current milestone
Measure one worker. We will test latency, throughput, queue time, token rate, and GPU use before adding tenants or more workers.
Milestone 1 result
Qwen3-1.7B served normal and streaming requests through vLLM on one NVIDIA L4. A warm stream returned its first byte in 0.333 seconds. The missing-GPU test failed as expected, and the final cleanup left no clusters or VM instances running.
The run also changed the design. L4 quota did not guarantee hardware capacity, so the working deployment moved from us-central1 to us-east1-c and from g2-standard-4 to g2-standard-8. Kubernetes also injected a VLLM_PORT value that stopped vLLM from starting. Disabling legacy Service environment variables fixed it.
Read the implementation, evidence, and ADRs in pull request 1.