The Total Kubernetes Troubleshooting Guide

If you’re having issues with Kubernetes, we’ve got you covered with this total Kubernetes troubleshooting guide. Kubernetes is well known for its benefits, like automating the deployment, scaling and management of containerized applications. But you only get those benefits if it’s working correctly. Here’s how to troubleshoot the most common Kubernetes issues so you can get the full value out of your container orchestration platform.

Kubernetes troubleshooting: The basics

Kubernetes troubleshooting is the process of diagnosing and resolving issues in a Kubernetes cluster. Because Kubernetes encompasses complex systems that orchestrate containers across multiple nodes, problems can be in the Pods, Services, Ingress or several other places. 

Troubleshooting in Kubernetes helps identify and resolve issues that stem from misconfigurations, resource constraints, networking failures or unexpected changes in workloads. Not only does Kubernetes troubleshooting help you resolve issues, it also helps you prevent those issues from happening again, as well as helps you identify bottlenecks and possible security weaknesses in the process.

Benefits of Kubernetes troubleshooting 

Kubernetes troubleshooting is important because the faster you can troubleshoot an issue, the faster you can resolve it. Fast troubleshooting can prevent breaches from going further and can limit damage. There are several advantages to diagnosing an issue quickly, including:

• Less downtime on your applications
• Better customer service and a better customer experience because of more uptime and fewer service disruptions
• Enhanced security because anomalies are identified early and unauthorized access can be revoked quickly
• Increased compliance with data protection laws and regulations because breaches are prevented or limited
• More efficient scaling, as troubleshooting can identify low-performing and overworked containers and fix issues
• Better resource management and lower costs with optimized workloads because containers aren’t failing
• Smoother workflows between better coordinated containers

Understanding, Management and Prevention: The pillars of Kubernetes troubleshooting

Kubernetes troubleshooting is based on three key pillars: Understanding, Management and Prevention. These three pillars serve as best practices to guide you through the troubleshooting process in a thorough and sustainable approach.

Understanding

The first pillar of Kubernetes troubleshooting is understanding. You can’t fix a problem if you don’t understand what the cause is. This pillar involves both understanding what is going wrong (e.g., high pod restarts) as well as why it’s happening. 

However, understanding the problem is often easier said than done. Kubernetes can be complex and involve many clusters and systems. Diagnosing an issue can involve a lot of work, including reviewing recent changes to a cluster or pod to see what may have caused the problem. You may also need to review GitHub repositories, analyze logs and traces and  compare metrics like disk pressure and response times. Reviewing these records can give you clues as to what’s happening and what’s causing it. 

Helpful hint: It’s best practice to have dashboards for many of these metrics. Dashboards make it easier to identify anomalies and find issues faster! You can also use monitoring tools, observability tools, logging tools and live debugging tools to help you understand the problem. All in one tools like SUSE Cloud Observability come with out-of-the-box dashboards and a full platform to easily troubleshoot with features like time travel and service maps that easily visualize your clusters.

Management

Now that you’ve identified the problem and understand what’s causing it, it’s time to do something about it. The second pillar of Kubernetes troubleshooting is management, or actually solving the issue. You may need to restart pods, scale resources differently or update configurations.

There are several different approaches you can take to manage the Kubernetes issue. The approach you take can vary depending on the organization of your team, who’s responsible for the particular component and how urgent the problem is. Generally though, there are three ways to go about resolving Kubernetes issues. 

The first way is with an ad hoc solution. Ad hoc solutions are usually work-arounds. These solutions may be temporary. If they work long-term, they still may not always be best practice, but they hold well enough. If someone who understands the component well, they may know how to navigate the component well enough to implement a fix quickly that works without breaking anything else. 

Another way to approach Kubernetes issue management is with a manual runbook. A manual runbook is a sort of SOP. It’s a standardized, clear procedure that has been documented. With this approach, it’s clear what needs to be done, within what kind of timeframe it needs to be done and who should do it. 

Kubernetes troubleshooting can also be done with automated runbooks. An automated runbook is an automated process that is triggered when certain conditions happen. Depending on your enterprise, an automated runbook could be a script you’ve implemented, an Infrastructure as Code (IaC) template or a Kubernetes operator. When possible, it’s best to have automated runbooks as your default method for managing Kubernetes issues. Automated runbooks don’t make human errors, and they’re always available to fix issues immediately.

Prevention

Once an issue is fixed, you need to make sure it doesn’t happen again. That’s why the third pillar of troubleshooting Kubernetes is prevention. As part of continuous improvement, you always want to do what you can to prevent the problem from recurring. 

In addition to putting steps in place to prevent the specific issue from happening again, this is also a chance to implement general best practices. Consider using namespaces properly, implementing a centralized logging system and using comprehensive Kubernetes monitoring. Maybe your team needs to rethink how processes currently happen, or maybe there are alerts or fixes that can be automated.

Common challenges in Kubernetes troubleshooting

Kubernetes troubleshooting is tricky and is much easier said than done. Here are some of the common challenges of troubleshooting Kubernetes. The better you understand these challenges, the better you can overcome them. 

Challenge 1: Kubernetes is complicated

The first and most prominent challenge is that Kubernetes is complicated. Kubernetes can be complex and have many components, making it difficult to identify the source of a problem. The complexity of a highly distributed system with dependencies across multiple services requires a deep understanding of Kubernetes to be able to fix any issues. 

Adding to the complexity is Kubernetes’s ephemeral nature. Pods and containers are constantly being created, destroyed and rescheduled. This makes it difficult to track an issue or recreate the circumstances under which it occurred. 

Challenge 2: Pod scheduling and resource constraints 

Another challenge is pod scheduling and resource constraints. By default, Kubernetes does not have any pod resource limits, which can be a double-edged sword. If you want any kind of restraints on resource consumption, you’ll have to configure them, which takes a significant amount of work. However, with the correct pod scheduling and resource constraints, you can preserve your cluster’s stability. Configuring appropriate constraints can prevent pods from failing, prevent pods from getting stuck in a pending state, minimize the chances of resource overload and keep neighboring nodes healthy. 

Challenge 3: Network issues

Networking issues are a common source of problems for Kubernetes, but that doesn’t mean they’re easy to figure out. Kubernetes uses complex networking features, such as services, network policies and ingress controllers. Issues could arise at any of these layers — whether it’s misconfigured services, DNS resolution failures, network policies blocking traffic or problems with ingress controllers. If your pods aren’t communicating, there may be a misconfiguration in the network policies or with the service configurations around traffic routing. You may need to check multiple layers of your system to find out why your network isn’t working.

How to troubleshooting basic Kubernetes issues

Kubernetes can be complicated, but we’ve compiled some basic Kubernetes troubleshooting fixes for you. Use this as a checklist to go through to figure out what could be causing the issues. This can help you find the problem quickly, or at least rule out what isn’t causing the issue. 

CrashLoopBackOff

If a pod fails, Kubernetes automatically restarts it. Usually that’s a good thing — it ensures continuity. However, if a pod continues having problems and keeps failing and restarting quickly, it can consume excessive resources. That’s where CrashLoopBackOff comes in. This error state starts slowing down the pod’s restart loop. This prevents the pod from continuing to drain resources. It also gives admins more time to see and troubleshoot the issue. 

ImagePullBackOff

ImagePullBackOff is a useful troubleshooting tool that keeps pods from wasting resources trying to pull an image that isn’t working. Even if a pod can’t pull an image correctly, it will keep trying. This can put the pod in a bad cycle of trying over and over again. Sometimes the image isn’t working because there’s an incorrect image tag. Sometimes the image isn’t found in the registry, there’s a registry authentication failure or there was an image pull policy that was preventing the image from being pulled. Whatever the reason is, ImagePullBackOff slows down the time between image pulls. This limits the amount of resources a pod can consume trying to complete this task, as well as gives admins more time to identify and correct the problem. 

Exit Codes

One of the most useful troubleshooting fixes for Kubernetes is exit codes. Anytime a container is terminated, it automatically generates an exit code. Think of an exit code as the black box in an airplane. If the airplane crashes, the black box can give records as to why the crash may have happened. Exit codes for Kubernetes follow the standard Linux exit codes, making them useful for diagnosing pod failures. 

For example, an exit code of 0 means that a container was successfully terminated normally. However, an exit code of 139 indicates a segmentation fault, meaning that the application crashed due to an invalid memory reference. These codes help you uncover why the error happened and what you might be able to do to fix the underlying problem. 

Node Not Ready

NodeNotReady is a useful Kubernetes troubleshooting tool. If your Kubernetes system is conducting health checks on its nodes and a particular node doesn’t pass the health check or is unable to schedule new pods, Kubernetes labels it “NodeNotReady.” This code flags the node to admins and lets people know that the node is experiencing issues. The code also prevents the node from scheduling new pods until it is fixed, evicts existing pods and attempts to recover the node. 

CreateContainerConfigError

When a pod fails to create a container due to configuration issues, it creates the error CreateContainerConfigError. This error means that Kubernetes encountered misconfigurations while trying to set up the container runtime environment. There are several reasons why this could happen — including missing secrets, incorrect volume amounts or invalid environment variables — but whatever the error is, Kubernetes couldn’t resolve it on its own. This troubleshooting error prevents the container from starting up and wasting resources until a human can take a look at it and fix the issue. 

Kubernetes OOMKilled

If a Kubernetes container exceeds its memory limit, it automatically starts using Kubernetes OOMKilled. This error terminates the container. It’s a useful failsafe which prevents the rest of the node from completely running out of memory. In addition to keeping the rest of the node from crashing, it can help identify memory leaks in applications running inside containers.

Troubleshooting for Kubernetes clusters

Kubernetes clusters are complex, so it can be tough to troubleshoot issues with them. Ideally, you should have a dedicated observability platform if you’re going to production. Alternatively, you can use Kubernetes dashboard or tools such as Prometheus or Grafana set up to help you monitor issues and diagnose anything that’s going wrong. If you don’t have those or they aren’t providing enough information, here’s how to troubleshoot manually. 

If you’re having issues with your cluster, you may need to check all its components (control plane, nodes, pods, services, volumes and more). As you go through all the layers, here is a general checklist you can use to check for common issues:

1. Check cluster health by using kubectl cluster-info
2. Check node health by using kubectl get nodes
3. Check the status of your pods by using kubectl get pods –all-namespaces

If the issue is at the cluster level, you’ll likely find the problem in the control plane. If issues affect scheduling or management of the cluster, verify the status of your control plane components (API server, etcd, controller manager). Ensure that your Kubernetes version is compatible with other tools and services in your environment. Check the versions of your kubelet, kube-proxy and other components to avoid issues caused by version mismatches.

Another source of common cluster issues is networking problems. Start by checking service and endpoint configuration and reviewing network policies to make sure they aren’t blocking traffic between pods. 

If that doesn’t work, start reviewing logs of critical components. If issues are affecting scheduling or cluster state, check the logs of the control plane components. For example, the API server and scheduler logs are available in the Kubernetes master node. You can also use kubectl to describe events and resources, which may give you clues as to what’s going wrong. 

How to troubleshoot a Kubernetes application

If you’re having problems with your Kubernetes applications, here are some tips for troubleshooting. Many issues with applications stem from incorrect configuration in Kubernetes resources like deployments, configmaps and secrets. Your application might depend on environment variables, configmaps or secrets, so make sure that those are defined correctly.

To further investigate the issue, check the container logs. If a container is crashing, the logs often contain stack traces or application errors that reveal the root cause. If the application is a service, make sure it’s listening on the correct ports.

If your application is being killed due to memory issues (you’ll know because you’ll see OOMKilled), it might be exceeding the memory limit. Check the resource limits and requests in the pod definition. If a pod is consuming more memory or CPU than allocated, it might need higher resource limits, or the application may need optimization.

You also may need to check for application-specific issues. If you’re using monitoring tools like Prometheus or Datadog, check for application-specific metrics, like HTTP request errors and response times, to identify performance bottlenecks or failures.

How to troubleshoot your Kubernetes pod

To troubleshoot Kubernetes pods, start by checking the status of your pods. Look specifically for pods in CrashLoopBackOff, Pending, Error, Terminating or Completed states. This can help you identify or eliminate which pods are having issues. 

If you’ve identified which pod is struggling, use kubectl describe pod <pod-name> to get a more detailed error message and find out what’s wrong. That may give you some insights. If you suspect that the pod may be struggling due to memory issues, use kubectl top nodes to get more information. You can also use kubectl describe node <node-name> to check to see if the pod isn’t functioning due to unsatisfied node conditions, such as unavailable resources. If the pod is in a Pending state, it could be because Kubernetes can’t find an appropriate node for the pod. If your pod relies on persistent storage, ensure the PVC is bound and that the volume is correctly mounted.

It’s also worth checking network and connectivity issues between pods and services to see if that may be the source of the problem.

Best practices and tips for Kubernetes troubleshooting

Troubleshooting Kubernetes becomes much easier if you follow best practices. These guidelines make it easier to diagnose problems as well as prevent issues from arising in the first place. Here are some of the top best practices we recommend:

• Invest in observability platforms. Implement a centralized platform that can give you full visibility into your infrastructure 
• Set up automated alerts. This can help you catch issues as soon as they happen. 
• Set up a dashboard. Kubernetes dashboards are an underused yet highly informative tool for catching problems as well as generally improving the performance of your clusters. 
• Double-check configurations. It’s tough to hear, but a large portion of problems come from misconfigurations. It’s worth double-checking the configurations of your network policy, your ConfigMaps and your Secrets. 
• Try to reproduce issues. If you’re struggling to fix an issue, try to recreate it from scratch. Use Helm logs and GitHub repositories to properly recreate the scenario in which an issue happened.
• Check resource utilization. Insufficient resources can cause pods to fail or become unresponsive. Also, ensure that resource requests and limits are correctly set in your pod specs to avoid resource contention or over-provisioning. Not only does this help troubleshoot issues, it can also help you identify bottlenecks and inefficient areas.

Get more value from your Kubernetes with SUSE

No matter how complex your ecosystem is, Kubernetes troubleshooting is simple with SUSE Rancher Prime that comes with a full observability platform. SUSE provides enterprise-level support, automation and security to enhance Kubernetes performance, troubleshoot issues and streamline operations.

FAQs on Kubernetes troubleshooting

Troubleshooting Kubernetes problems can be tricky. Here are some of the most commonly asked questions related to Kubernetes troubleshooting — and our answers to guide you through the process. 

How do you check if a service is working in Kubernetes?

To check if a service is working in Kubernetes, start by checking the service status. This will let you know whether it’s created and exposed correctly. You can also verify service endpoints. If there are no endpoints or incorrect endpoints, the service won’t work as expected.

Another way to check if a service is working in Kubernetes is to test the connectivity. To test the connectivity and functionality of the service, you can use a simple curl or ping command from a pod that can access the service.

What are the biggest issues with Kubernetes?

One of the biggest issues with Kubernetes is how complex it is. It takes a lot to configure. Once you’ve gotten it all set up, it’s still a lot of work to keep up with upgrades and versioning. If you don’t configure it correctly or you don’t keep up with upgrades, you’re bound to have issues. 

Another major issue with Kubernetes is security. Kubernetes isn’t inherently insecure — in fact, it has very strong security features. However, any slight misconfiguration can lead to data breaches. If the network policies aren’t 100% correct, it can lead to unauthorized access. Ensuring container image security, enforcing security policies and managing secrets securely are critical best practices. 

Why do Kubernetes pods fail?

Kubernetes pods can fail for a variety of reasons. Some of the most common reasons Kubernetes pods fail are:

• Resource constraints
• Image pull failures
• Network issues
• Pod disruption
• Liveness and readiness probe failures
• Configuration errors
• Unhealthy dependencies