How to troubleshoot using the namespace of a container

When troubleshooting an issue, often a faithful reproduction and exact environment are needed. This can be a challenge in a containerized environment, where tools and a shell environment may not be easily available within containers of a Pod.

There are two approaches that can be taken:
 

By running a container in the same namespaces as another, it's possible to use that container for troubleshooting.

The sidecar container can be started using the same network and PID namespaces while attaching the same volumes:

• Set the ID or name of the container you wish to troubleshoot:

ID=<container ID or name>

• Run the sidecar container using the network, PID and volumes

docker run -it --net=container:$ID --pid=container:$ID --volumes-from=$ID alpine sh

• It is now possible to troubleshoot with commands from the alpine container, within the context of the container or Pod with the issue.

For example, if you were experiencing a network issue from this Pod, it is now possible to use tools available in the sidecar container to simulate the connection, view the network configuration and troubleshoot interactively.

Substitute the alpine container as needed with an image of your choice.

Note, this will attach the same volumes as the parent container, but the parent container read/write layers will not be accesible - to access the same container filesystem, see the nsenter example below.

Alternatively, you can use tools available on the host for the same use case with the nsenter command. The nsenter command is standard on most Linux distributions, for example on Ubuntu, it is provided by the util-linux package.

• Set the ID or name of the container you wish to troubleshoot:

ID=<container ID or name>

• Obtain the first process in the container (PID 1):

PID=$(docker inspect --format '{{ .State.Pid }}' $ID)
 

•  Run commands from the node within the network namespace context of the container/Pod with nsenter:

nsenter -n -t $PID <command>
 

• Run commands from the node within the context of all of the container/Pod namespaces with nsenter:

nsenter -a -t $PID <command>

For example, if troubleshooting a network issue, tools like tcpdump, curl, dig and mtr can be used to troubleshoot the issue interactively.

Note, the -a flag is available in recent versions of nsenter, if this does not succeed, use a flag for a specific namespace, check the nsenter --help output.