AKS konnectivity-agent tunnel instability causing intermittent 503s on admin operations

Question

AKS konnectivity-agent tunnel instability causing intermittent 503s on admin operations

Bernard Kowalski 0

Cluster aks-asclepius-prod (resource group rg-aks, East US 2, K8s 1.35.1, private cluster with KMS etcd encryption) experiences persistent konnectivity tunnel failures.

Symptoms:

konnectivity-agent pods enter cannot connect once error loops, unable to re-establish tunnel to control plane proxy servers
rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: EOF"
Intermittent 503 Service Unavailable on kubectl exec, port-forward, and logs
Pod-to-pod application traffic is unaffected

Environment:

Single node: Standard_D4ps_v6 (ARM64)
KMS etcd encryption via public Key Vault (asclepius-infra-prod) — ruled out as cause (0 throttled requests, ~36ms avg latency)
Dev cluster (same region, also private, no KMS) shows identical EOF churn but functions correctly

Troubleshooting performed:

Rollout restart of konnectivity-agent — no improvement
az aks update --yes control plane reconcile — temporarily restores tunnel, but issue recurs

Ask: Investigate why konnectivity-agent pods intermittently fail to reconnect to control plane proxy servers, requiring manual control plane reconciliation to restore admin operations.

Jilakara Hemalatha 11,600 Reputation points Microsoft External Staff Moderator

2026-04-09T05:02:32.5233333+00:00
Hello Bernard,

Thank you for contacting us regarding the intermittent 503 errors observed during kubectl exec, port-forward, and logs on your AKS private cluster aks-asclepius-prod.

Based on the behavior described, it appears that the konnectivity-agent, responsible for managing the tunnel between the control plane and cluster nodes in private clusters, is intermittently losing its connection. This can result in temporary 503 errors for admin operations, while pod-to-pod traffic remains unaffected.

To help identify potential causes and gather additional information, we recommend the following checks:

Verify konnectivity-agent health and restarts
kubectl get pods -n kube-system -l app=konnectivity-agent -o wide
Look for repeated restarts or OOMKilled/CrashLoopBackOff events.

Check node and pod resource usage
kubectl top nodes
Memory or CPU pressure can affect the agent’s ability to maintain connections.

In parallel, please validate whether any networking controls in the outbound path could be intermittently blocking or resetting the control-plane-to-node administrative connectivity flow. In private clusters, tunnel connectivity issues commonly surface when NSGs/UDRs/firewalls or other network appliances intermittently disrupt the required connectivity, which then impacts kubectl exec/logs/port-forward.

To help us understand the environment better:

Are any konnectivity-agent pods being OOMKilled, or are nodes under memory pressure?

Do you have any network appliances (Azure Firewall, NAT Gateway) between the agents and the control plane?

Can you share the ladder configuration in the konnectivity-agent-autoscaler ConfigMap?

Are both clusters running the same agent image version and identical CPA/resource settings?

How frequently do you see the “error reading server preface: EOF” in the logs, and does it ever recover automatically?

Reference: https://dotnet.territoriali.olinfo.it/en-us/troubleshoot/azure/azure-kubernetes/connectivity/tunnel-connectivity-issues?

https://dotnet.territoriali.olinfo.it/en-us/troubleshoot/azure/azure-kubernetes/connectivity/error-from-server-error-dialing-backend-dial-tcp?

Share via

AKS konnectivity-agent tunnel instability causing intermittent 503s on admin operations

Your answer