Managing Cluster in Kubernetes

.

Analyzing Cluster Nodes

• Kubernetes cluster nodes run Linux processes. To monitor these processes,
  generic Linux rules apply
  • Use systemctl status kubelet to get runtime information about the kubelet
  • Use log files in /var/log as well as journalctl output to get access to logs
• Generic node information is obtained through kubectl describe
• If the Metrics Server is installed, use kubectl top nodes to get a summary
  of CPU/memory usage on a node.

Analyzing Node State Commands

• Is -lrt /var/log
• journalctl
• systemctl status kubelet

Now, let’s reproduce a failure:

 

crictl command

• All Pods are started as containers on the nodes
• crictl is a generic tool that communicates to the container runtime to get
  information about running containers
• As such, it replaces generic tools like docker and podman
• To use it, a runtime-endpoint and image-endpoint need to be set
• The most convenient way to do so, is by defining the /etc/crictl.yaml file on
  the nodes where you want to run crictl

Using crictl

• List containers: sudo crictl ps
• List Pods that have been scheduled on this node: sudo crictl pods
• Inspect container configuration: sudo crictl inspect <name-or-id>
• Pull an image: sudo crictl pull <imagename>
• List images: sudo crictl images
• For more options, use crictl --help

 

Static Pods

• The kubelet systemd process is configured to run static Pods from the
  /etc/kubernetes/manifests directory
• On the control node, static Pods are an essential part of how Kubernetes
  works: systemd starts kubelet, and kubelet starts core Kubernetes services
  as static Pods
• Administrators can manually add static Pods if so desired, just copy a
  manifest file into the /etc/kubernetes/manifests directory and the kubelet
  process will pick it up
• To modify the path where Kubelet picks up the static Pods, edit
  staticPodPath in /var/lib/kubelet/config.yaml and use sudo systemctl
  restart kubelet to restart
• Never do this on the control node!

Running Static Pods

• kubectl run staticpod --image=nginx --dry-run=client -o yaml >
staticpod.yaml
• sudo cp staticpod.yaml /etc/kubernetes/manifests/
• kubectl get pods -o wide

That’s how it works. If you run a static pod the kubelet will automatically pick it up at the moment you create it.

Managing Node State

• kubectl cordon is used to mark a node as unschedulable
• kubectl drain is used to mark a node as unschedulable and
  remove all running Pods from it
  • Pods that have been started from a DaemonSet will not be
    removed while using kubectl drain, add --ignore-daemonsets to
    ignore that
  • Add --delete-emptydir-data to delete data from emptyDir Pod volumes.
• While using cordon or drain, a taint is set on the nodes
• Use kubectl uncordon to get the node back in a schedulable
  state

Managing Node State – Commands

• kubectl cordon worker2
• kubectl describe node worker2 # look for taints
• kubectl get nodes
• kubectl uncordon worker2

 

Managing Node Services

• The container runtime (often containerd) and kubelet are managed by the
  Linux systemd service manager
• Use systemctl status kubelet to check the current status of the kubelet
• To manually start it, use sudo systemctl start kubelet
• Notice that Pods that are scheduled on a node show as container processes in ps aux output. Don’t use Linux tools to manage Pods !

Managing Node Services Commands

• ps aux | grep kubelet
• ps aux | grep containerd
• systemctl status kubelet
• sudo systemctl stop kubelet
• sudo systemctl start kubelet

As we see even if we kill the kubelet process the systemt will pick it up. But if we would down the service by systemctl systemd won’t pick the kubelet process up. It only works for distaster not for intended operations.

 

Lab: Managing Static Pods

• On node worker1, run a static Pod with the name mypod, using an Nginx
  image and no further configuration
• Use the appropriate tools to verify that the static Pod has started successfully