Cluster API on OpenStack: Part 4 - Day-2 Operations

With clusters running in production, you’ll need to perform ongoing maintenance. CAPI’s immutable infrastructure model means changes like disk expansion require creating new templates and performing rolling updates.

This is part 4 of the Cluster API on OpenStack series:

1. Introduction and Architecture
2. Building CAPI Images
3. Deploying Workload Clusters
4. Day-2 Operations (this post)

Expanding Worker Disks

A common operational task is expanding the root disk on worker nodes. Since CAPI uses immutable infrastructure, you can’t resize disks in place. Instead, you create a new template with the larger disk and update the MachineDeployment.

Check Current Configuration

# Find your MachineDeployment
kubectl get machinedeployment

# See which template it references
kubectl get machinedeployment prod-cluster-md-0 \
  -o jsonpath='{.spec.template.spec.infrastructureRef.name}'

Export and Modify the Template

# Export current template
TEMPLATE=$(kubectl get machinedeployment prod-cluster-md-0 \
  -o jsonpath='{.spec.template.spec.infrastructureRef.name}')

kubectl get openstackmachinetemplate $TEMPLATE -o yaml > worker-template.yaml

Edit worker-template.yaml to add or increase rootVolume:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: OpenStackMachineTemplate
metadata:
  name: prod-cluster-md-0-50gb  # New name
spec:
  template:
    spec:
      flavor: m1.medium-disk
      image:
        filter:
          name: ubuntu-2404-kube-v1.32.0
      rootVolume:
        sizeGiB: 50           # Increased from 20
        volumeType: ceph-hdd  # Your Cinder volume type

Remove metadata that would prevent creation:

# Remove resourceVersion, uid, creationTimestamp
sed -i '/resourceVersion:/d' worker-template.yaml
sed -i '/uid:/d' worker-template.yaml
sed -i '/creationTimestamp:/d' worker-template.yaml

Apply and Update the Reference

# Create the new template
kubectl apply -f worker-template.yaml

# Verify it exists
kubectl get openstackmachinetemplates

# Patch the MachineDeployment to use it
kubectl patch machinedeployment prod-cluster-md-0 \
  --type=json \
  -p='[{"op": "replace", "path": "/spec/template/spec/infrastructureRef/name", "value": "prod-cluster-md-0-50gb"}]'

Watch the Rolling Update

kubectl get machines -w

CAPI will:

1. Create new machines with the updated template
2. Wait for them to become ready
3. Drain and delete old machines one at a time

The maxSurge and maxUnavailable settings on the MachineDeployment control the rollout pace.

Template Isolation: One Template Per Cluster

A word of caution on template management: avoid sharing templates between clusters. While it might seem efficient to have a single worker-50gb template used by multiple clusters, this creates several problems:

• Coupled lifecycles - Deleting cluster A’s templates breaks cluster B
• No independent updates - You can’t change disk size for one cluster without affecting others
• Accidental cascading changes - Updating “your” template triggers rolling updates everywhere

Instead, use a clear naming convention:

prod-cluster-md-0-50gb
staging-cluster-md-0-30gb
dev-cluster-md-0-20gb

The storage overhead is negligible—these are small Kubernetes objects—and the operational clarity is worth it.

Scaling Workers

Scaling is straightforward with MachineDeployments:

# Scale to 5 workers
kubectl scale machinedeployment prod-cluster-md-0 --replicas=5

# Or patch the spec
kubectl patch machinedeployment prod-cluster-md-0 \
  --type=merge \
  -p='{"spec":{"replicas":5}}'

Kubernetes Version Upgrades

Upgrades are a two-part change: bump the control plane version, then roll workers by pointing the MachineDeployment at a new template built from a matching image.

1) Prepare a New Worker Template

Create a new OpenStackMachineTemplate that references an image built for the target Kubernetes version (for example, v1.33.0).

# Identify the current worker template name
TEMPLATE=$(kubectl get machinedeployment prod-cluster-md-0 \
  -o jsonpath='{.spec.template.spec.infrastructureRef.name}')

# Export it so we can edit a copy
kubectl get openstackmachinetemplate "$TEMPLATE" -o yaml > worker-template-v1.33.0.yaml

Edit worker-template-v1.33.0.yaml:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: OpenStackMachineTemplate
metadata:
  name: prod-cluster-md-0-v1.33.0
spec:
  template:
    spec:
      image:
        filter:
          name: ubuntu-2404-kube-v1.33.0

Remove fields that prevent creation:

sed -i '/resourceVersion:/d' worker-template-v1.33.0.yaml
sed -i '/uid:/d' worker-template-v1.33.0.yaml
sed -i '/creationTimestamp:/d' worker-template-v1.33.0.yaml

Apply the new template:

kubectl apply -f worker-template-v1.33.0.yaml

2) Prepare a New Control Plane Template

The control plane also uses an OpenStackMachineTemplate, referenced by the KubeadmControlPlane. Create a new template that points at the target image.

# Identify the current control plane template name
CP_TEMPLATE=$(kubectl get kubeadmcontrolplane prod-cluster-control-plane \
  -o jsonpath='{.spec.machineTemplate.infrastructureRef.name}')

# Export it so we can edit a copy
kubectl get openstackmachinetemplate "$CP_TEMPLATE" -o yaml > controlplane-template-v1.33.0.yaml

Edit controlplane-template-v1.33.0.yaml:

apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
kind: OpenStackMachineTemplate
metadata:
  name: prod-cluster-control-plane-v1.33.0
spec:
  template:
    spec:
      image:
        filter:
          name: ubuntu-2404-kube-v1.33.0

Remove fields that prevent creation:

sed -i '/resourceVersion:/d' controlplane-template-v1.33.0.yaml
sed -i '/uid:/d' controlplane-template-v1.33.0.yaml
sed -i '/creationTimestamp:/d' controlplane-template-v1.33.0.yaml

Apply the new template:

kubectl apply -f controlplane-template-v1.33.0.yaml

3) Upgrade the Control Plane

Patch both the Kubernetes version and the control plane template reference. CAPI will roll the control plane nodes one by one.

kubectl patch kubeadmcontrolplane prod-cluster-control-plane \
  --type=merge \
  -p='{"spec":{"version":"v1.33.0","machineTemplate":{"infrastructureRef":{"name":"prod-cluster-control-plane-v1.33.0"}}}}'

4) Roll Workers to the New Template

Point the MachineDeployment at the new template to trigger a rolling update of worker nodes.

kubectl patch machinedeployment prod-cluster-md-0 \
  --type=json \
  -p='[{"op": "replace", "path": "/spec/template/spec/infrastructureRef/name", "value": "prod-cluster-md-0-v1.33.0"}]'

5) Watch the Upgrade

kubectl get machines -w

If you want to track control plane nodes specifically:

kubectl get kubeadmcontrolplane prod-cluster-control-plane -o wide

Conclusion

Cluster API transforms Kubernetes cluster management from imperative scripts to declarative resources. On OpenStack, this means:

• Version-controlled cluster definitions
• Automated rolling updates for Kubernetes upgrades
• Consistent day-2 operations across environments
• Natural integration with GitOps workflows

The learning curve is real—understanding the resource hierarchy takes time—but the operational benefits compound over time. Start with a simple workload cluster, get comfortable with the resource relationships, and gradually adopt more advanced patterns like ClusterClass for standardized configurations.

The combination of OpenStack’s infrastructure capabilities with CAPI’s declarative management creates a powerful platform for running Kubernetes at scale.