Troubleshooting And Debugging Kubernetes
Troubleshooting K8s - Common Issues in K8S - finding the reasons and Fixing Them#
Pod in ImagePullBackOff#
With an error message like: Back-off pulling image registry.co.za/repo/image_name:hash
A lot of tutorials say uou should use describe on the pod. This is wrong.
You cannot describe the pod if it is not running.
It will not be running as k8s cannot even get the image is should run.
The best place to start is to look at the events of a pod with:
kubectl get event -n portal --field-selector involvedObject.name=my-pod-zl6m6
You will get output like:
LAST SEEN TYPE REASON OBJECT MESSAGE
21m Normal BackOff pod/my-pod-zl6m6 Back-off pulling image "registry.co.za/repo/image_name:hash"
71s Warning Failed pod/my-pod-zl6m6 Error: ImagePullBackOff
but that is telling us what we already know.
If there is only 1 pod in the namespace it is easier to look at all events in the namespace:
kubectl get event -n portal
But there is no clear indication except for in rancher on a rare occasion it will show the actual error from the registry:
ErrImagePull: rpc error: code = Unknown desc = Error response from daemon: Get https://registry.co.za/v2/repo/image_name/manifests/1c9d75694a5c7a016da73d851b44737eb5bff0b8: Get https://core.harbor.domain/service/token?scope=repository%3Acrepo%2Fimage_name%3Apull&service=harbor-registry: dial tcp: lookup core.harbor.domain: no such host
In which case sometimes the registry gives errors…
A Pod is stuck in Pending State#
Here a MySQL pod has been stuck for 48 minutes in a Pending state
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
mysql-6q5hp 0/1 Pending 0 48m
Describe the pod:
$ kubectl describe pod mysql-6q5hp
Name: mysql-6q5hp
Namespace: default
Priority: 0
Node: <none>
Labels: app=mysql
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/mysql
Containers:
database:
Image: mysql
Port: 3306/TCP
Host Port: 0/TCP
Requests:
cpu: 1
memory: 2Gi
Liveness: tcp-socket :3306 delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
MYSQL_ROOT_PASSWORD: some-password-here
Mounts:
/var/lib/mysql from database (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-mbn5b (ro)
Conditions:
Type Status
PodScheduled False
Volumes:
database:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: database
ReadOnly: false
default-token-mbn5b:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-mbn5b
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> default-scheduler 0/1 nodes are available: 1 Insufficient memory.
Warning FailedScheduling <unknown> default-scheduler 0/1 nodes are available: 1 Insufficient memory.
Source: My pod stays pending
Pod Status shows a RunContainerError#
$ kubectl get pods -w
NAME READY STATUS RESTARTS AGE
nginx-fast-storage-wknfz 1/1 Running 0 128m
parallel-kcvxt 0/1 ContainerCreating 0 10s
parallel-lpttv 0/1 ContainerCreating 0 10s
parallel-ng5xd 0/1 RunContainerError 0 10s
parallel-pjwdz 0/1 ContainerCreating 0 10s
parallel-vnskl 0/1 ContainerCreating 0 10s
parallel-pjwdz 0/1 RunContainerError 0 11s
parallel-lpttv 0/1 RunContainerError 0 13s
parallel-vnskl 0/1 RunContainerError 0 16s
parallel-kcvxt 0/1 RunContainerError 0 19s
Check the events for any info
You can get the events causing the error:
kubectl get events --sort-by=.metadata.creationTimestamp
That said the error:
41m Normal Created pod/parallel-ng5xd Created container kuard
41m Warning Failed pod/parallel-ng5xd Error: failed to start container "kuard": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"--keygen-enable\": executable file not found in $PATH": unknown
View DNS Records#
kubectl get pods -n kube-system
Pod in CrashLoopBackoff shows secrets error in logs#
Check pod status
$ kubectl get po
NAME READY STATUS RESTARTS AGE
dashboard-demo-kubernetes-dashboard-59db4d65dc-dnbx8 0/1 CrashLoopBackOff 6 8m11s
Check the logs:
kubectl logs dashboard-demo-kubernetes-dashboard-59db4d65dc-dnbx8
...
Storing encryption key in a secret
panic: secrets is forbidden: User "system:serviceaccount:default:dashboard-demo-kubernetes-dashboard" cannot create resource "secrets" in API group "" in the namespace "kube-system"
Solution:
?