View log files of crashed pods in kubernetes

kubectl logs command only works if the pod is up and running. If they are not, you can use the kubectl events command.

kubectl get events -n <your_app_namespace> --sort-by='.metadata.creationTimestamp'

By default it does not sort the events, hence the --sort-by flag.


Assuming that your pod still exists:

kubectl logs <podname> --previous

$ kubectl logs -h
-p, --previous[=false]: If true, print the logs for the previous instance of the container in a pod if it exists.


In many cases, kubectl logs <podname> --previous is returning:

Error from server (BadRequest): previous terminated container "<container-name>" in pod "<pod-name>" not found

So you can try to check in the namespace's events (kubectl get events ..) like @alltej showed.

If you don't find the reason for the error with kubectl logs / get events and you can't view it with external logging tool I would suggest:

1 ) Check on which node that pod was running on with:

$kubectl get -n <namespace> pod <pod-name> -o=custom-columns=NAME:.metadata.name,STATUS:.status.phase,NODE:.spec.nodeName

NAME              STATUS    NODE
failed-pod-name   Pending   dns-of-node

(If you remove the <pod-name> you can see other pods in the namespace).

2 ) SSH into that node and:

A ) Search for the failed pod container name in /var/log/containers/ and dump its .log file and search for errors - in most of the cases the cause of error will be displayed there alongside with the actions / events that took place before the error.

B ) If previous step doesn't help try searching for latest System level errors by running:
sudo journalctl -u kubelet -n 100 --no-pager.

Tags:

Kubernetes