10250 I/O timeouts error when running kubectl log command

Summary

A TCP 10250 i/o timeout in kubectl logs is often caused by blocked internal traffic between cluster nodes. Use this article to verify whether network security groups (NSGs) are blocking required node subnet traffic and restore log retrieval.

Connect to the AKS cluster

First, connect to your Azure Kubernetes Service (AKS) cluster by running the following command:

export RESOURCE_GROUP=<your-resource-group>
export CLUSTER_NAME=<your-cluster-name>

az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME

Symptoms

Tunnel functionalities, such as kubectl logs and code execution, work only for pods that are hosted on nodes on which tunnel service pods are deployed. Pods on other nodes that have no tunnel service pods cannot reach to the tunnel. When viewing the logs of these pods, you receive the following error message:

kubectl logs $POD_NAME

Results:

Error from server: Get "https://aks-agentpool-xxxxxxxxx-vmssxxxxxxxxx:10250/containerLogs/vsm-mba-prod/mba-api-app-xxxxxxxxxx/technosvc": dial tcp <IP-Address>:10250: i/o timeout

Solution

To resolve this issue, allow traffic on port 10250 as described in this article.