Running one AKS cluster by hand is manageable. Running six across dev, QA and production is where that approach quietly falls apart, because drift arrives faster than anyone reconciles it. GitOps fleet management solves the reconciliation problem by making the repository the only thing that describes a cluster, and the cluster the thing that catches up. This post covers the cluster IaC, the Flux layout that survives past one cluster, workload identity, and the day 2 operations that actually bite.
Architecture
GitHub or Azure Repos
main env/dev env/prod
| | |
| PR->merge gates |
v v v
┌──────────────────────────────────┐
│ Flux │
│ GitRepository + Kustomization │
│ Reconcile loop per cluster │
└──────────────────────────────────┘
^ ^ ^
| | |
┌────────────┴───┐ ┌─┴─────────────┐ ┌───────────────┐
│ Cluster Dev │ │ Cluster QA │ │ Cluster Prod │
│ MI + OIDC │ │ MI + OIDC │ │ MI + OIDC │
│ CNI + NP │ │ CNI + NP │ │ CNI + NP │
└──────┬─────────┘ └─────┬─────────┘ └──────┬────────┘
| | |
flux bootstrap flux bootstrap flux bootstrap
| | |
bootstrap repo path bootstrap repo path bootstrap repo path
clusters/dev clusters/qa clusters/prod
Notes: OIDC issuer and Workload Identity connect pods to Azure resources. CNI is Azure CNI or Cilium. NP means NetworkPolicy.
Cluster IaC
az CLI quickstart
# Resource group and ACR
az group create -n rg-aks-prod -l uksouth
az acr create -n acrprod123 -g rg-aks-prod --sku Premium
# AKS with managed identity, OIDC, workload identity, autoscaler
az aks create \
-g rg-aks-prod -n aks-prod \
--enable-managed-identity \
--enable-oidc-issuer --enable-workload-identity \
--network-plugin azure \
--node-count 3 --min-count 3 --max-count 10 \
--enable-cluster-autoscaler \
--auto-upgrade-channel stable
# Attach ACR pull
az aks update -g rg-aks-prod -n aks-prod --attach-acr acrprod123
# Add a user node pool for workloads
az aks nodepool add -g rg-aks-prod --cluster-name aks-prod \
-n userpool --mode User --node-count 3 \
--enable-cluster-autoscaler --min-count 3 --max-count 20
# Flux bootstrap against repo
az aks get-credentials -g rg-aks-prod -n aks-prod
flux bootstrap github \
--owner your-org --repository aks-gitops \
--branch env/prod --path clusters/prod --personal
Bicep sketch
param location string = resourceGroup().location
param clusterName string = 'aks-prod'
param acrId string
@description('Kubernetes minor to pin. Must sit inside the current AKS support window. List what your region offers with: az aks get-versions --location <region> --output table')
param kubernetesVersion string = '1.35'
resource aks 'Microsoft.ContainerService/managedClusters@2024-05-01' = {
name: clusterName
location: location
identity: { type: 'SystemAssigned' }
properties: {
dnsPrefix: '${clusterName}-dns'
oidcIssuerProfile: { enabled: true }
workloadIdentityProfile: { enabled: true }
kubernetesVersion: kubernetesVersion // pinned via parameter, not hardcoded
apiServerAccessProfile: {
enablePrivateCluster: true
}
autoScalerProfile: {
maxEmptyBulkDelete: '10'
scanInterval: '20s'
}
agentPoolProfiles: [
{
name: 'system'
mode: 'System'
count: 3
vmSize: 'Standard_D4s_v5'
osType: 'Linux'
enableAutoScaling: true
minCount: 3
maxCount: 10
orchestratorVersion: kubernetesVersion
}
]
networkProfile: {
networkPlugin: 'azure' // or 'none' with Cilium CNI addon
}
}
}
@description('User pool defined as child resource for surge and taints')
resource userpool 'Microsoft.ContainerService/managedClusters/agentPools@2024-05-01' = {
name: '${aks.name}/userpool'
properties: {
mode: 'User'
count: 3
vmSize: 'Standard_D4s_v5'
enableAutoScaling: true
minCount: 3
maxCount: 20
nodeTaints: [ 'workload=true:NoSchedule' ]
}
}
// ACR pull permission
resource acrRole 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
name: guid(aks.id, 'AcrPull')
scope: resourceId('Microsoft.ContainerRegistry/registries', split(acrId,'/')[8])
properties: {
roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', '7f951dda-4ed3-4680-a7ca-43fe172d538d')
principalId: aks.identity.principalId
principalType: 'ServicePrincipal'
}
}
Pin the version, but pin it to something that still exists. AKS supports a rolling window of three GA minor versions, the latest and the two before it, which Microsoft calls N-2. Checked on 26 September 2026, that window is 1.34, 1.35 and 1.36, and 1.29 left community support in March 2025. A sample pinned to a dead minor is worse than no pin at all, because it deploys cleanly and strands the cluster outside support from the first day. Take the current window from the AKS supported versions table, then confirm what your region actually offers with az aks get-versions --location <region> --output table before you set the parameter.
Workload Identity for a pod
apiVersion: v1
kind: ServiceAccount
metadata:
name: sa-blob
namespace: app
annotations:
azure.workload.identity/client-id: <user-assigned-mi-client-id>
azure.workload.identity/tenant-id: <tenant-id>
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
namespace: app
spec:
replicas: 3
selector: { matchLabels: { app: api } }
template:
metadata: { labels: { app: api } }
spec:
serviceAccountName: sa-blob
containers:
- name: api
image: acrprod123.azurecr.io/api:1.2.3
env:
- name: AZURE_CLIENT_ID
valueFrom:
fieldRef: { fieldPath: metadata.annotations['azure.workload.identity/client-id'] }
GitOps with Flux
Environment branches hold overlays. Flux reconciles per cluster.
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
name: gitops
namespace: flux-system
spec:
interval: 1m
url: https://github.com/your-org/aks-gitops
ref: { branch: env/prod }
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: platform
namespace: flux-system
spec:
interval: 1m
prune: true
sourceRef: { kind: GitRepository, name: gitops }
path: ./platform
wait: true
dependsOn: []
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: workloads
namespace: flux-system
spec:
interval: 1m
prune: true
sourceRef: { kind: GitRepository, name: gitops }
path: ./workloads/prod
dependsOn:
- name: platform
Repo structure example:
/cluster-bootstrap
/clusters/dev
/clusters/prod
/platform/cert-manager
/platform/ingress
/workloads/dev
/workloads/prod
The newsletter
Get the next Azure fix like this one by email
Subscribe and get the free 8-page Intune troubleshooting field guide (PDF), then new fixes about twice a month. From a Microsoft MVP.
GitOps fleet management across many clusters
GitOps fleet management is mostly a question of what you keep identical across clusters and what you deliberately allow to differ. Get that boundary wrong and you either copy the same YAML six times, or you build an abstraction nobody can debug when a reconcile fails at three in the morning.
- Namespaces vs clusters: use namespaces for teams with shared trust and soft isolation. The same separation question shows up at subscription level, which is the argument for Entra ID centric RBAC in an Azure landing zone. Use dedicated clusters for tenant isolation, regulatory boundaries, or conflicting add-ons.
- Placement: one repo, many clusters. Parameterize overlays by cluster labels. Flux supports kustomize substitutions and Json6902 patches.
- Multi tenancy: restrict RBAC to namespaces. Use network policies as a default deny.
NetworkPolicy baseline:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: app
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-dns-and-egress-gw
namespace: app
spec:
podSelector: {}
egress:
- to:
- namespaceSelector: { matchLabels: { kube-system: "true" } }
ports: [ { port: 53, protocol: UDP }, { port: 53, protocol: TCP } ]
- to:
- ipBlock: { cidr: 0.0.0.0/0 } # replace with egress IP or firewall
ports: [ { port: 443, protocol: TCP } ]
policyTypes: [Egress]
Day 2 operations
- Upgrades: pin minor in Bicep. Rotate node images regularly. Revisit the pin whenever the support window moves, which is roughly every four months.
# Control plane first
az aks upgrade -g rg-aks-prod -n aks-prod --control-plane-only --kubernetes-version 1.35
# Node images
az aks upgrade -g rg-aks-prod -n aks-prod --node-image-only
# Optional pool level
az aks nodepool upgrade -g rg-aks-prod --cluster-name aks-prod -n userpool
- Blue or green strategy: create a parallel user pool with taints and drain workloads by label. At the traffic layer, use Gateway API with weighted backends.
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: shop
spec:
parentRefs: [ { name: public-gw } ]
rules:
- backendRefs:
- name: shop-blue
weight: 90
- name: shop-green
weight: 10
- Backup and restore: Velero is portable. Snapshot volumes via CSI. For Azure native, use Azure Backup for AKS to protect PVs and workloads.
- Egress and Private Link: use a NAT Gateway or Azure Firewall for deterministic egress IPs. Disk choice matters here too, and Premium SSD v2 against ephemeral OS disks covers what node storage actually costs. Prefer Private Link to ACR, Key Vault, Storage. Lock down API server with private cluster and authorized subnets.
Observability
- Container Insights: enable via addon for node and control plane metrics and logs. Route to Log Analytics.
az aks enable-addons -g rg-aks-prod -n aks-prod -a monitoring
- Prometheus stack: kube-prometheus-stack via Flux. Record SLOs and alert on burn rates.
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: slo-api
namespace: monitoring
spec:
groups:
- name: api-latency
rules:
- record: slo:request_latency_seconds:rate5m
expr: histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="api"}[5m])))
- alert: APIHighErrorRate
expr: sum(rate(http_requests_total{job="api",code=~"5.."}[5m])) / sum(rate(http_requests_total{job="api"}[5m])) > 0.05
for: 10m
labels: { severity: page }
annotations:
summary: API error rate above 5 percent
Dashboards: surface SLO target, error budget remaining, and burn alerts. Grafana, Azure Managed Grafana, or vendor of choice are fine.
Security
- Secrets: use Secrets Store CSI with Azure provider. Mount secrets from Key Vault with Workload Identity.
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: kv-secrets
namespace: app
spec:
provider: azure
parameters:
useWorkloadIdentity: "true"
clientID: <user-assigned-mi-client-id>
keyvaultName: my-kv
objects: |
array:
- |
objectName: app-secret
objectType: secret
tenantId: <tenant-id>
- Image provenance: sign images in CI using Cosign. Store in ACR with OCI attestations. Enforce at admission.
- Policy: Gatekeeper or Kyverno. Example Kyverno verifyImages.
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: verify-signed-images
spec:
rules:
- name: require-cosign
match:
any:
- resources:
kinds: [Pod, Deployment]
verifyImages:
- image: "acrprod123.azurecr.io/*"
keyless:
issuer: "https://token.actions.githubusercontent.com"
subject: "repo:your-org/*:ref:refs/heads/main"
attestations:
- predicateType: "cosign.sigstore.dev/attestation/v1"
Gatekeeper example for required labels:
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
name: req-team-label
spec:
match:
kinds:
- apiGroups: [""]
kinds: ["Namespace"]
parameters:
labels: ["owner", "costCenter"]
Ops checklist
- Pull requests gate changes. Flux reconciles in minutes.
- Autoscaler and max surge tuned. Drains scripted during upgrades.
- Egress pinned to NAT IPs. Private Link for supply chain.
- SLOs define your paging. Policy prevents bad images and missing labels.
- Backups verified monthly. Restore playbook tested.
- Idle clusters still bill. What Azure actually charges for when a resource is stopped applies to node pools as much as to single VMs.
