Automate Blue-Green Deployments with a Kubernetes Operator and Istio

Running a blue-green deployment by hand is a good way to learn. It is not a good way to manage dozens of releases.

In this guide, you will put the new container image in a Kubernetes custom resource:

apiVersion: delivery.tejo.dev/v1alpha1
kind: BlueGreenDeployment
metadata:
  name: web
  namespace: delivery-demo
spec:
  image: traefik/whoami:v1.10.4

A Kubernetes controller sees the change and starts the release. It prepares the inactive color, waits until it is ready, runs a test Job, moves Istio traffic in steps, and records the result.

If the rollout or test fails, the controller sends all traffic back to the active color and stops. It keeps the failed version so that an operator can inspect it.

You will build this as a Kubernetes Operator with Kubebuilder and Go.

Architecture and state machine

The controller does not run one long script. Kubernetes may restart it at any time. Instead, each pass checks the current state, makes one safe change, updates the status, and returns. Kubernetes repeats this until the release is complete.

The controller mental model

Read this diagram from left to right:

• Declare: you say what should be true, such as spec.image: app:v2.
• Reconcile: the controller repeatedly observes the cluster, compares it with the specification, and makes a small correction.
• Converge: Deployments, validation Jobs, Istio routes, and status eventually match the requested release.

The arrow returning to reconciliation is important. The controller does not assume an action succeeded merely because it submitted an API request; it observes the result on a later pass.

The rollout phases are:

1. Preparing — create or update the inactive Deployment with the desired image.
2. WaitingForReady — wait until every candidate replica is available.
3. Validating — run a Kubernetes Job against the candidate-only preview route.
4. ShiftingTraffic — apply one Istio weight stage at a time.
5. Promoting — record the candidate color and image as stable.
6. Stable — no work until spec.image or spec.retryNonce changes.
7. Failed — route 100% to the active color and wait for a new desired generation.

Prerequisites

You need:

• a 64-bit macOS, Linux, or Windows machine; use WSL on Windows
• 12 GB of memory and 15 GB of free disk space recommended
• Docker
• Go 1.26
• kubectl
• kind
• Istio 1.30.x and istioctl
• Kubebuilder
• GNU Make and Git

You do not need a container registry account, a GitHub organization, a cloud account, or a Kubernetes cluster outside your laptop. The controller runs locally with make run, and the lab pulls public multi-architecture images directly from Docker Hub:

• traefik/whoami:v1.10.3 for Blue
• traefik/whoami:v1.10.4 for Green
• curlimages/curl:8.16.0 for validation

The manual Istio blue-green guide explains Docker, kubectl, kind, and Istio installation in detail. The complete checks and missing-tool paths are repeated here so this guide can stand alone.

Check existing tools

docker version
go version
kubectl version --client
kind version
istioctl version --remote=false
kubebuilder version
make --version
git --version

Keep any working installation. Install only missing or unsupported tools.

1. Install Docker

macOS: install and start Docker Desktop for Mac.

Windows: install Docker Desktop for Windows with the WSL 2 backend. Run the rest of the guide inside WSL.

Linux: follow the Docker Engine instructions for your distribution and start the service.

Verify the engine:

docker run --rm hello-world

2. Install Go 1.26

Download the installer or archive for your operating system from the official Go downloads. On Windows, install Go inside WSL rather than mixing Windows and Linux toolchains.

Open a new shell and verify:

go version
go env GOOS GOARCH

The version should be Go 1.26.x. Kubebuilder currently requires at least Go 1.24.6; using the current Go release also matches current controller-runtime scaffolding.

3. Install kubectl

macOS:

brew install kubectl

Windows host:

winget install -e --id Kubernetes.kubectl

For this guide, install it inside WSL using the Linux path as well.

Linux or WSL:

KUBECTL_ARCH=amd64
[ "$(uname -m)" = "aarch64" ] && KUBECTL_ARCH=arm64
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/${KUBECTL_ARCH}/kubectl"
chmod +x kubectl
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Verify with kubectl version --client.

4. Install kind

macOS: brew install kind

Windows host: winget install Kubernetes.kind

Linux or WSL:

KIND_VERSION=v0.32.0
KIND_ARCH=amd64
[ "$(uname -m)" = "aarch64" ] && KIND_ARCH=arm64
curl -Lo kind "https://kind.sigs.k8s.io/dl/${KIND_VERSION}/kind-linux-${KIND_ARCH}"
chmod +x kind
sudo mv kind /usr/local/bin/kind

Check the kind quick start for the latest stable version, then run kind version.

5. Install Istio

On macOS, Linux, or WSL:

curl -L https://istio.io/downloadIstio | ISTIO_VERSION=1.30.3 sh -
cd istio-1.30.3
export PATH="$PWD/bin:$PATH"
cd ..

Use an absolute path in your shell profile if you want istioctl to persist across terminal sessions. Verify with istioctl version --remote=false.

6. Install Kubebuilder

Kubebuilder officially supports macOS and Linux. Windows users should run this in WSL:

curl -L -o kubebuilder \
  "https://go.kubebuilder.io/dl/latest/$(go env GOOS)/$(go env GOARCH)"
chmod +x kubebuilder
sudo mv kubebuilder /usr/local/bin/
kubebuilder version

The generated project pins compatible Kubernetes libraries and helper tools. Keep those generated versions together rather than independently upgrading controller-runtime, client-go, and controller tools.

7. Install Make and Git

macOS: install Apple's Command Line Tools if these commands are missing:

xcode-select --install

Ubuntu/Debian/WSL:

sudo apt-get update
sudo apt-get install -y build-essential git curl

Fedora:

sudo dnf group install -y "Development Tools"
sudo dnf install -y git curl

1. Create the lab cluster and install Istio

kind create cluster \
  --name operator-lab \
  --image kindest/node:v1.35.0 \
  --wait 5m

istioctl install -y \
  --set profile=default \
  --set meshConfig.accessLogFile=/dev/stdout

kubectl rollout status deployment/istiod -n istio-system --timeout=3m
kubectl rollout status deployment/istio-ingressgateway -n istio-system --timeout=3m
istioctl verify-install

Create the application namespace with sidecar injection:

kubectl create namespace delivery-demo
kubectl label namespace delivery-demo istio-injection=enabled

All application resources stay in that namespace. The only cross-namespace call is the validation Job's request to Istio's ingress Service.

Every namespaced manifest below includes namespace: delivery-demo. Resources created by the controller always use the custom resource's metadata.namespace, so the same controller logic also works in a differently named namespace.

2. Scaffold the Operator

mkdir bluegreen-operator
cd bluegreen-operator

kubebuilder init \
  --domain tejo.dev \
  --repo example.com/bluegreen-operator

kubebuilder create api \
  --group delivery \
  --version v1alpha1 \
  --kind BlueGreenDeployment \
  --resource \
  --controller

Answer y when asked to create the resource and controller. Kubebuilder creates the API types, CRD generation markers, controller, tests, RBAC configuration, Makefile, and deployment manifests.

3. Design the custom resource API

Replace the spec and status declarations in api/v1alpha1/bluegreendeployment_types.go with these types. Keep Kubebuilder's generated package declaration, imports, root object types, and init() function.

package v1alpha1

import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

type TrafficStage struct {
	// +kubebuilder:validation:Minimum=0
	// +kubebuilder:validation:Maximum=100
	CandidateWeight int32 `json:"candidateWeight"`

	// +kubebuilder:validation:Minimum=0
	PauseSeconds int32 `json:"pauseSeconds,omitempty"`
}

type ValidationSpec struct {
	// Container image that contains curl.
	Image string `json:"image,omitempty"`

	// +kubebuilder:default="/"
	Path string `json:"path,omitempty"`
}

type BlueGreenDeploymentSpec struct {
	Image string `json:"image"`

	// Increment to retry the same failed image.
	// +kubebuilder:default=0
	RetryNonce int64 `json:"retryNonce,omitempty"`

	// +kubebuilder:validation:Minimum=1
	// +kubebuilder:default=2
	Replicas int32 `json:"replicas,omitempty"`

	// +kubebuilder:default=80
	ContainerPort int32 `json:"containerPort,omitempty"`

	// +kubebuilder:default="/"
	HealthPath string `json:"healthPath,omitempty"`

	Host string `json:"host"`
	Stages []TrafficStage `json:"stages,omitempty"`
	Validation ValidationSpec `json:"validation,omitempty"`
}

type BlueGreenDeploymentStatus struct {
	ObservedGeneration int64 `json:"observedGeneration,omitempty"`
	ActiveColor string `json:"activeColor,omitempty"`
	StableImage string `json:"stableImage,omitempty"`
	CandidateImage string `json:"candidateImage,omitempty"`
	Phase string `json:"phase,omitempty"`
	CurrentStage int32 `json:"currentStage,omitempty"`
	NextStageAt *metav1.Time `json:"nextStageAt,omitempty"`
	Conditions []metav1.Condition `json:"conditions,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:printcolumn:name="Phase",type=string,JSONPath=`.status.phase`
// +kubebuilder:printcolumn:name="Active",type=string,JSONPath=`.status.activeColor`
// +kubebuilder:printcolumn:name="Image",type=string,JSONPath=`.status.stableImage`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
type BlueGreenDeployment struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`
	Spec   BlueGreenDeploymentSpec   `json:"spec,omitempty"`
	Status BlueGreenDeploymentStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true
type BlueGreenDeploymentList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`
	Items []BlueGreenDeployment `json:"items"`
}

func init() {
	SchemeBuilder.Register(&BlueGreenDeployment{}, &BlueGreenDeploymentList{})
}

Generate the CRD and deepcopy methods:

make generate
make manifests

Inspect config/crd/bases/delivery.tejo.dev_bluegreendeployments.yaml. It should expose a /status subresource. Users write spec; the controller writes status with separate RBAC and optimistic concurrency.

4. Define the rules that must always hold

Before writing controller code, make the safety rules explicit:

• A Service selects both colors by app, never by version.
• The DestinationRule owns exactly two subsets: blue and green.
• The first VirtualService route is a candidate-only preview header.
• The second route owns production weights and always totals 100.
• The active Deployment is never mutated during candidate preparation.
• A failed rollout always routes 100% to the active color.
• A failed generation stays failed; reconciliation does not retry forever.
• Incrementing retryNonce creates a new generation and permits a controlled retry.
• Status is evidence, not the only source of truth; owned objects are re-read on every pass.

These invariants make repeated reconciliation safe.

5. Add the controller's permissions and imports

Open internal/controller/bluegreendeployment_controller.go. Use these RBAC markers above the reconciler:

// +kubebuilder:rbac:groups=delivery.tejo.dev,resources=bluegreendeployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=delivery.tejo.dev,resources=bluegreendeployments/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=delivery.tejo.dev,resources=bluegreendeployments/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=deployments,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=batch,resources=jobs,verbs=get;list;watch;create;delete
// +kubebuilder:rbac:groups="",resources=services,verbs=get;list;watch;create;update;patch
// +kubebuilder:rbac:groups=networking.istio.io,resources=virtualservices,verbs=get;list;watch;create;update;patch
// +kubebuilder:rbac:groups=networking.istio.io,resources=destinationrules,verbs=get;list;watch;create;update;patch

The controller uses typed Kubernetes objects and unstructured Istio objects. Unstructured access avoids pinning a second generated client library just to manage two CRDs.

import (
	"context"
	"fmt"
	"time"

	deliveryv1alpha1 "example.com/bluegreen-operator/api/v1alpha1"
	appsv1 "k8s.io/api/apps/v1"
	batchv1 "k8s.io/api/batch/v1"
	corev1 "k8s.io/api/core/v1"
	apierrors "k8s.io/apimachinery/pkg/api/errors"
	"k8s.io/apimachinery/pkg/api/meta"
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/schema"
	"k8s.io/apimachinery/pkg/types"
	"k8s.io/apimachinery/pkg/util/intstr"
	"k8s.io/client-go/tools/record"
	"k8s.io/utils/ptr"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
	"sigs.k8s.io/controller-runtime/pkg/controller/controllerutil"
)

type BlueGreenDeploymentReconciler struct {
	client.Client
	Scheme   *runtime.Scheme
	Recorder record.EventRecorder
}

6. Implement the state-driven reconcile loop

Use this structure for Reconcile. Each branch is short and can safely run again.

func (r *BlueGreenDeploymentReconciler) Reconcile(
	ctx context.Context,
	req ctrl.Request,
) (ctrl.Result, error) {
	var bg deliveryv1alpha1.BlueGreenDeployment
	if err := r.Get(ctx, req.NamespacedName, &bg); err != nil {
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	// Do not loop on a failed desired generation. A new image or retryNonce
	// changes metadata.generation and explicitly authorizes another attempt.
	if bg.Status.Phase == "Failed" &&
		bg.Status.ObservedGeneration == bg.Generation {
		return ctrl.Result{}, nil
	}

	if err := validateSpec(&bg); err != nil {
		return r.fail(ctx, &bg, "InvalidSpec", err.Error())
	}

	active := bg.Status.ActiveColor
	if active == "" {
		active = "blue"
	}
	candidate := opposite(active)

	if err := r.ensureService(ctx, &bg); err != nil {
		return ctrl.Result{}, err
	}
	if err := r.ensureDestinationRule(ctx, &bg); err != nil {
		return ctrl.Result{}, err
	}

	// First reconciliation bootstraps Blue directly as the stable release.
	if bg.Status.StableImage == "" {
		if err := r.ensureDeployment(ctx, &bg, active, bg.Spec.Image); err != nil {
			return ctrl.Result{}, err
		}
		if !r.deploymentReady(ctx, &bg, active) {
			return r.progress(ctx, &bg, "Preparing", active, 0, 5*time.Second)
		}
		if err := r.ensureVirtualService(ctx, &bg, active, opposite(active), 0); err != nil {
			return ctrl.Result{}, err
		}
		bg.Status.ActiveColor = active
		bg.Status.StableImage = bg.Spec.Image
		return r.complete(ctx, &bg, "Initial release is stable")
	}

	if bg.Spec.Image == bg.Status.StableImage &&
		bg.Status.ObservedGeneration == bg.Generation &&
		bg.Status.Phase == "Stable" {
		return ctrl.Result{}, nil
	}

	if bg.Status.ObservedGeneration != bg.Generation {
		bg.Status.CandidateImage = bg.Spec.Image
		bg.Status.CurrentStage = 0
		bg.Status.NextStageAt = nil
		bg.Status.Phase = "Preparing"
		bg.Status.ObservedGeneration = bg.Generation
		if err := r.Status().Update(ctx, &bg); err != nil {
			return ctrl.Result{}, err
		}
		return ctrl.Result{Requeue: true}, nil
	}

	if err := r.ensureDeployment(ctx, &bg, candidate, bg.Spec.Image); err != nil {
		return r.rollback(ctx, &bg, active, candidate, "CandidateCreateFailed", err.Error())
	}
	if failed, reason := r.deploymentFailed(ctx, &bg, candidate); failed {
		return r.rollback(ctx, &bg, active, candidate, "CandidateRolloutFailed", reason)
	}
	if !r.deploymentReady(ctx, &bg, candidate) {
		return r.progress(ctx, &bg, "WaitingForReady", candidate, 0, 5*time.Second)
	}

	if err := r.ensureVirtualService(ctx, &bg, active, candidate, 0); err != nil {
		return ctrl.Result{}, err
	}

	validation, err := r.ensureValidationJob(ctx, &bg, candidate)
	if err != nil {
		return ctrl.Result{}, err
	}
	if validation == "failed" {
		return r.rollback(ctx, &bg, active, candidate, "ValidationFailed", "Candidate validation Job failed")
	}
	if validation != "complete" {
		return r.progress(ctx, &bg, "Validating", candidate, 0, 3*time.Second)
	}

	stages := bg.Spec.Stages
	if len(stages) == 0 {
		stages = []deliveryv1alpha1.TrafficStage{
			{CandidateWeight: 10, PauseSeconds: 30},
			{CandidateWeight: 50, PauseSeconds: 30},
			{CandidateWeight: 100, PauseSeconds: 0},
		}
	}
	if bg.Status.NextStageAt != nil && time.Now().Before(bg.Status.NextStageAt.Time) {
		return ctrl.Result{RequeueAfter: time.Until(bg.Status.NextStageAt.Time)}, nil
	}
	if int(bg.Status.CurrentStage) < len(stages) {
		stage := stages[bg.Status.CurrentStage]
		if err := r.ensureVirtualService(ctx, &bg, active, candidate, stage.CandidateWeight); err != nil {
			return ctrl.Result{}, err
		}
		bg.Status.CurrentStage++
		bg.Status.Phase = "ShiftingTraffic"
		next := metav1.NewTime(time.Now().Add(time.Duration(stage.PauseSeconds) * time.Second))
		bg.Status.NextStageAt = &next
		if err := r.Status().Update(ctx, &bg); err != nil {
			return ctrl.Result{}, err
		}
		return ctrl.Result{RequeueAfter: time.Duration(stage.PauseSeconds) * time.Second}, nil
	}

	bg.Status.ActiveColor = candidate
	bg.Status.StableImage = bg.Spec.Image
	bg.Status.CandidateImage = ""
	bg.Status.NextStageAt = nil
	return r.complete(ctx, &bg, fmt.Sprintf("%s promoted", candidate))
}

candidateWeight remains correct across successive rollouts: Green is the first candidate, Blue is the next candidate, and both progress through 10, 50, and 100 percent. nextStageAt makes the pause durable. A status update may trigger an immediate reconcile, but the controller waits until that timestamp before advancing.

7. Build the owned Kubernetes resources

The Deployment helper uses deterministic names such as web-blue and web-green. CreateOrUpdate makes it idempotent.

func (r *BlueGreenDeploymentReconciler) ensureDeployment(
	ctx context.Context,
	bg *deliveryv1alpha1.BlueGreenDeployment,
	color, image string,
) error {
	healthPath := bg.Spec.HealthPath
	if healthPath == "" {
		healthPath = "/"
	}
	dep := &appsv1.Deployment{ObjectMeta: metav1.ObjectMeta{
		Name: bg.Name + "-" + color, Namespace: bg.Namespace,
	}}
	_, err := controllerutil.CreateOrUpdate(ctx, r.Client, dep, func() error {
		labels := map[string]string{"app": bg.Name, "version": color}
		dep.Labels = labels
		dep.Spec.Replicas = ptr.To(bg.Spec.Replicas)
		dep.Spec.ProgressDeadlineSeconds = ptr.To(int32(120))
		dep.Spec.Selector = &metav1.LabelSelector{MatchLabels: labels}
		dep.Spec.Template.ObjectMeta.Labels = labels
		dep.Spec.Template.Spec.Containers = []corev1.Container{{
			Name: "app", Image: image, ImagePullPolicy: corev1.PullIfNotPresent,
			Ports: []corev1.ContainerPort{{ContainerPort: bg.Spec.ContainerPort}},
			ReadinessProbe: &corev1.Probe{
				ProbeHandler: corev1.ProbeHandler{HTTPGet: &corev1.HTTPGetAction{
					Path: healthPath, Port: intstr.FromInt32(bg.Spec.ContainerPort),
				}},
				InitialDelaySeconds: 2, PeriodSeconds: 3,
			},
		}}
		return controllerutil.SetControllerReference(bg, dep, r.Scheme)
	})
	return err
}

func (r *BlueGreenDeploymentReconciler) ensureService(
	ctx context.Context,
	bg *deliveryv1alpha1.BlueGreenDeployment,
) error {
	svc := &corev1.Service{ObjectMeta: metav1.ObjectMeta{
		Name: bg.Name, Namespace: bg.Namespace,
	}}
	_, err := controllerutil.CreateOrUpdate(ctx, r.Client, svc, func() error {
		svc.Spec.Selector = map[string]string{"app": bg.Name}
		svc.Spec.Ports = []corev1.ServicePort{{
			Name: "http", Port: 80,
			TargetPort: intstr.FromInt32(bg.Spec.ContainerPort),
		}}
		return controllerutil.SetControllerReference(bg, svc, r.Scheme)
	})
	return err
}

Readiness and rollout failure are derived from the Deployment, not trusted from old status:

func (r *BlueGreenDeploymentReconciler) deploymentReady(
	ctx context.Context,
	bg *deliveryv1alpha1.BlueGreenDeployment,
	color string,
) bool {
	var dep appsv1.Deployment
	if r.Get(ctx, types.NamespacedName{
		Name: bg.Name + "-" + color, Namespace: bg.Namespace,
	}, &dep) != nil {
		return false
	}
	return dep.Status.ObservedGeneration == dep.Generation &&
		dep.Status.AvailableReplicas == bg.Spec.Replicas
}

func (r *BlueGreenDeploymentReconciler) deploymentFailed(
	ctx context.Context,
	bg *deliveryv1alpha1.BlueGreenDeployment,
	color string,
) (bool, string) {
	var dep appsv1.Deployment
	if err := r.Get(ctx, types.NamespacedName{
		Name: bg.Name + "-" + color, Namespace: bg.Namespace,
	}, &dep); err != nil {
		return false, ""
	}
	for _, condition := range dep.Status.Conditions {
		if condition.Type == appsv1.DeploymentProgressing &&
			condition.Status == corev1.ConditionFalse &&
			condition.Reason == "ProgressDeadlineExceeded" {
			return true, condition.Message
		}
	}
	return false, ""
}

8. Reconcile Istio resources

Create the DestinationRule as an unstructured object:

func (r *BlueGreenDeploymentReconciler) ensureDestinationRule(
	ctx context.Context,
	bg *deliveryv1alpha1.BlueGreenDeployment,
) error {
	host := fmt.Sprintf("%s.%s.svc.cluster.local", bg.Name, bg.Namespace)
	dr := &unstructured.Unstructured{Object: map[string]any{
		"apiVersion": "networking.istio.io/v1",
		"kind": "DestinationRule",
		"metadata": map[string]any{"name": bg.Name, "namespace": bg.Namespace},
	}}
	dr.SetGroupVersionKind(schemaGVK("DestinationRule"))
	_, err := controllerutil.CreateOrUpdate(ctx, r.Client, dr, func() error {
		dr.Object["spec"] = map[string]any{
			"host": host,
			"subsets": []any{
				map[string]any{"name": "blue", "labels": map[string]any{"version": "blue"}},
				map[string]any{"name": "green", "labels": map[string]any{"version": "green"}},
			},
		}
		return controllerutil.SetControllerReference(bg, dr, r.Scheme)
	})
	return err
}

Add this small GroupVersionKind helper:

func schemaGVK(kind string) schema.GroupVersionKind {
	return schema.GroupVersionKind{
		Group: "networking.istio.io", Version: "v1", Kind: kind,
	}
}

The VirtualService always includes the candidate preview route first and production weights second:

func (r *BlueGreenDeploymentReconciler) ensureVirtualService(
	ctx context.Context,
	bg *deliveryv1alpha1.BlueGreenDeployment,
	active, candidate string,
	candidateWeight int32,
) error {
	host := fmt.Sprintf("%s.%s.svc.cluster.local", bg.Name, bg.Namespace)
	vs := &unstructured.Unstructured{Object: map[string]any{
		"apiVersion": "networking.istio.io/v1",
		"kind": "VirtualService",
		"metadata": map[string]any{"name": bg.Name, "namespace": bg.Namespace},
	}}
	vs.SetGroupVersionKind(schemaGVK("VirtualService"))
	_, err := controllerutil.CreateOrUpdate(ctx, r.Client, vs, func() error {
		vs.Object["spec"] = map[string]any{
			"hosts": []any{bg.Spec.Host},
			// namespace/name makes the Gateway reference unambiguous.
			"gateways": []any{bg.Namespace + "/web"},
			"http": []any{
				map[string]any{
					"name": "candidate-preview",
					"match": []any{map[string]any{"headers": map[string]any{
						"x-release-preview": map[string]any{"exact": candidate},
					}}},
					"route": []any{destination(host, candidate, 100)},
				},
				map[string]any{
					"name": "production",
					"route": []any{
						destination(host, active, 100-candidateWeight),
						destination(host, candidate, candidateWeight),
					},
				},
			},
		}
		return controllerutil.SetControllerReference(bg, vs, r.Scheme)
	})
	return err
}

func destination(host, subset string, weight int32) map[string]any {
	return map[string]any{
		"destination": map[string]any{"host": host, "subset": subset},
		"weight": int64(weight),
	}
}

For the lab, create one shared ingress Gateway outside the controller:

apiVersion: networking.istio.io/v1
kind: Gateway
metadata:
  name: web
  namespace: delivery-demo
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - operator.local

Save it as config/samples/gateway.yaml.

9. Add the candidate validation Job

The validation Job calls the same ingress route a user would call but adds the private candidate header. In-cluster validation uses the ingress gateway Service.

func (r *BlueGreenDeploymentReconciler) ensureValidationJob(
	ctx context.Context,
	bg *deliveryv1alpha1.BlueGreenDeployment,
	candidate string,
) (string, error) {
	name := fmt.Sprintf("%s-validate-%d", bg.Name, bg.Generation)
	job := &batchv1.Job{}
	key := types.NamespacedName{Name: name, Namespace: bg.Namespace}
	if err := r.Get(ctx, key, job); err != nil {
		if !apierrors.IsNotFound(err) {
			return "", err
		}
		image := bg.Spec.Validation.Image
		if image == "" {
			image = "curlimages/curl:8.16.0"
		}
		path := bg.Spec.Validation.Path
		if path == "" {
			path = "/"
		}
		job = &batchv1.Job{
			ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: bg.Namespace},
			Spec: batchv1.JobSpec{
				BackoffLimit: ptr.To(int32(1)),
				ActiveDeadlineSeconds: ptr.To(int64(60)),
				Template: corev1.PodTemplateSpec{
					ObjectMeta: metav1.ObjectMeta{Annotations: map[string]string{
						"sidecar.istio.io/inject": "false",
					}},
					Spec: corev1.PodSpec{
					RestartPolicy: corev1.RestartPolicyNever,
					Containers: []corev1.Container{{
						Name: "validate", Image: image,
						Command: []string{"curl", "--fail", "--show-error", "--silent"},
						Args: []string{
							"-H", "Host: " + bg.Spec.Host,
							"-H", "x-release-preview: " + candidate,
							"http://istio-ingressgateway.istio-system.svc.cluster.local" + path,
						},
					}},
				}},
			},
		}
		if err := controllerutil.SetControllerReference(bg, job, r.Scheme); err != nil {
			return "", err
		}
		if err := r.Create(ctx, job); err != nil {
			return "", err
		}
		return "running", nil
	}
	if job.Status.Succeeded > 0 {
		return "complete", nil
	}
	if job.Status.Failed > 0 {
		return "failed", nil
	}
	return "running", nil
}

The generation in the Job name prevents a successful Job from a previous image from approving a new release. The pod template explicitly disables Istio sidecar injection: the Job reaches the ingress gateway through its cluster Service, and an injected long-running proxy could otherwise keep this short-lived Job from completing.

10. Save progress so rollback survives a restart

Status conditions give humans and automation a standard way to understand the rollout. Add these helpers:

func (r *BlueGreenDeploymentReconciler) progress(
	ctx context.Context,
	bg *deliveryv1alpha1.BlueGreenDeployment,
	phase, candidate string,
	weight int32,
	after time.Duration,
) (ctrl.Result, error) {
	if bg.Status.Phase == phase && bg.Status.CandidateImage == bg.Spec.Image {
		return ctrl.Result{RequeueAfter: after}, nil
	}
	bg.Status.Phase = phase
	bg.Status.CandidateImage = bg.Spec.Image
	setCondition(bg, "Progressing", metav1.ConditionTrue, phase,
		fmt.Sprintf("candidate=%s weight=%d", candidate, weight))
	if err := r.Status().Update(ctx, bg); err != nil {
		return ctrl.Result{}, err
	}
	return ctrl.Result{RequeueAfter: after}, nil
}

func (r *BlueGreenDeploymentReconciler) rollback(
	ctx context.Context,
	bg *deliveryv1alpha1.BlueGreenDeployment,
	active, candidate, reason, message string,
) (ctrl.Result, error) {
	if err := r.ensureVirtualService(ctx, bg, active, candidate, 0); err != nil {
		return ctrl.Result{}, err
	}
	bg.Status.Phase = "Failed"
	setCondition(bg, "Ready", metav1.ConditionFalse, reason, message)
	setCondition(bg, "RolledBack", metav1.ConditionTrue, reason,
		"Production traffic restored to "+active)
	r.Recorder.Event(bg, corev1.EventTypeWarning, "RolloutFailed", message)
	return ctrl.Result{}, r.Status().Update(ctx, bg)
}

func (r *BlueGreenDeploymentReconciler) complete(
	ctx context.Context,
	bg *deliveryv1alpha1.BlueGreenDeployment,
	message string,
) (ctrl.Result, error) {
	bg.Status.Phase = "Stable"
	bg.Status.ObservedGeneration = bg.Generation
	bg.Status.CandidateImage = ""
	bg.Status.NextStageAt = nil
	setCondition(bg, "Progressing", metav1.ConditionFalse, "Stable", message)
	setCondition(bg, "Ready", metav1.ConditionTrue, "Stable", message)
	r.Recorder.Event(bg, corev1.EventTypeNormal, "RolloutComplete", message)
	return ctrl.Result{}, r.Status().Update(ctx, bg)
}

func (r *BlueGreenDeploymentReconciler) fail(
	ctx context.Context,
	bg *deliveryv1alpha1.BlueGreenDeployment,
	reason, message string,
) (ctrl.Result, error) {
	bg.Status.Phase = "Failed"
	bg.Status.ObservedGeneration = bg.Generation
	setCondition(bg, "Ready", metav1.ConditionFalse, reason, message)
	return ctrl.Result{}, r.Status().Update(ctx, bg)
}

func setCondition(
	bg *deliveryv1alpha1.BlueGreenDeployment,
	typeName string,
	status metav1.ConditionStatus,
	reason, message string,
) {
	meta.SetStatusCondition(&bg.Status.Conditions, metav1.Condition{
		Type: typeName, Status: status, Reason: reason, Message: message,
		ObservedGeneration: bg.Generation,
	})
}

func opposite(color string) string {
	if color == "green" { return "blue" }
	return "green"
}

func validateSpec(bg *deliveryv1alpha1.BlueGreenDeployment) error {
	if bg.Spec.Image == "" || bg.Spec.Host == "" {
		return fmt.Errorf("spec.image and spec.host are required")
	}
	previous := int32(-1)
	for _, stage := range bg.Spec.Stages {
		if stage.CandidateWeight < previous {
			return fmt.Errorf("stages must be ordered by increasing candidateWeight")
		}
		previous = stage.CandidateWeight
	}
	if len(bg.Spec.Stages) > 0 && bg.Spec.Stages[len(bg.Spec.Stages)-1].CandidateWeight != 100 {
		return fmt.Errorf("the final stage must set candidateWeight to 100")
	}
	return nil
}

Finally, watch the owned resources and initialize the event recorder:

func (r *BlueGreenDeploymentReconciler) SetupWithManager(mgr ctrl.Manager) error {
	r.Recorder = mgr.GetEventRecorderFor("bluegreen-controller")
	return ctrl.NewControllerManagedBy(mgr).
		For(&deliveryv1alpha1.BlueGreenDeployment{}).
		Owns(&appsv1.Deployment{}).
		Owns(&batchv1.Job{}).
		Owns(&corev1.Service{}).
		Named("bluegreendeployment").
		Complete(r)
}

Run formatting, generation, and tests:

go fmt ./...
make generate manifests
make test

11. Install the CRD and run the controller

Install the generated CRD:

make install
kubectl get crd bluegreendeployments.delivery.tejo.dev

Run the controller locally against your current kubeconfig:

make run

Leave it running. This local mode shortens the development loop. Later, make docker-build docker-push deploy IMG=... installs the controller in-cluster.

Apply the shared Gateway from another terminal:

kubectl apply -f config/samples/gateway.yaml

12. Create the first stable release

The sample uses two real, public versions of Traefik Whoami. It listens on port 80 and returns 200 from /, needs no credentials, and publishes images for common laptop architectures.

Create config/samples/delivery_v1alpha1_bluegreendeployment.yaml:

apiVersion: delivery.tejo.dev/v1alpha1
kind: BlueGreenDeployment
metadata:
  name: web
  namespace: delivery-demo
spec:
  image: traefik/whoami:v1.10.3
  retryNonce: 0
  replicas: 2
  containerPort: 80
  healthPath: /
  host: operator.local
  validation:
    image: curlimages/curl:8.16.0
    path: /
  stages:
    - candidateWeight: 10
      pauseSeconds: 30
    - candidateWeight: 50
      pauseSeconds: 60
    - candidateWeight: 100
      pauseSeconds: 0

Apply it as written—there is no organization name or registry login to replace:

kubectl apply -f config/samples/delivery_v1alpha1_bluegreendeployment.yaml
kubectl get bluegreendeployment -n delivery-demo -w

The first image bootstraps Blue. Inspect the objects and events:

kubectl get deployments,pods,services,jobs -n delivery-demo
kubectl get virtualservice,destinationrule -n delivery-demo
kubectl describe bluegreendeployment web -n delivery-demo

Wait until the phase is Stable and ACTIVE is blue.

13. Deploy a new image automatically

Blue remains available through preparation and validation. Green receives production traffic only after its pre-traffic gates pass.

This is the only release action:

kubectl patch bluegreendeployment web -n delivery-demo \
  --type=merge \
  -p '{"spec":{"image":"traefik/whoami:v1.10.4"}}'

This changes only the desired image, from the public v1.10.3 image to public v1.10.4. Watch the rollout:

kubectl get bluegreendeployment web -n delivery-demo -w
kubectl get pods,jobs -n delivery-demo -w

The controller will:

1. update web-green to the new image
2. wait for all Green replicas
3. route preview-header traffic to Green
4. run the validation Job
5. shift production traffic through 10%, 50%, and 100% Green
6. set status.activeColor: green and status.stableImage to the new image

You can inspect status without parsing logs:

kubectl get bluegreendeployment web -n delivery-demo -o yaml
kubectl get events -n delivery-demo --sort-by=.lastTimestamp

14. Test automatic rollback

Every failed gate converges on the same safe routing state: 100% traffic to the active color, a failed status condition, and the candidate preserved for diagnosis.

Set an image that does not exist:

kubectl patch bluegreendeployment web -n delivery-demo \
  --type=merge \
  -p '{"spec":{"image":"traefik/whoami:this-tag-does-not-exist"}}'

The inactive Blue Deployment cannot become ready. After its progress deadline, the controller:

• leaves the currently active Green Deployment untouched
• restores 100% production traffic to Green
• sets phase to Failed
• records Ready=False and RolledBack=True
• stops retrying that generation

Inspect the evidence:

kubectl describe bluegreendeployment web -n delivery-demo
kubectl get virtualservice web -n delivery-demo -o yaml
kubectl get pods -n delivery-demo

To retry the same corrected tag, increment retryNonce:

kubectl patch bluegreendeployment web -n delivery-demo \
  --type=merge \
  -p '{"spec":{"image":"traefik/whoami:v1.10.4","retryNonce":1}}'

Changing spec creates a new Kubernetes generation, which is an explicit retry signal.

Before using this in production

This controller demonstrates the control loop, but time alone is not a production analysis gate. Before broad use, add:

• an analysis provider that queries error rate, latency, and business metrics at every stage
• a maximum stage deadline stored in status, not process memory
• admission validation for images, hosts, stages, and immutable fields
• image digest resolution and signature verification
• namespace-scoped or per-tenant RBAC
• a policy for concurrent spec changes during an active rollout
• leader election and multiple controller replicas
• finalizers only for external cleanup that garbage collection cannot handle
• Prometheus metrics for phase duration, promotions, and rollbacks
• EnvTest cases for restart recovery, stale generations, failed Jobs, and conflicts
• a compatibility contract for database and event-schema changes

Clean up

Stop make run with Ctrl+C, then remove the CRD and local cluster:

make uninstall
kind delete cluster --name operator-lab

Further reading

• Kubernetes custom resources and controllers
• Kubebuilder quick start
• Kubebuilder controller implementation
• Istio traffic management
• Istio VirtualService reference