{"id":5295,"date":"2023-11-11T17:43:01","date_gmt":"2023-11-11T16:43:01","guid":{"rendered":"http:\/\/miro.borodziuk.eu\/?p=5295"},"modified":"2025-05-19T15:07:36","modified_gmt":"2025-05-19T13:07:36","slug":"managing-cluster-in-kubernetes","status":"publish","type":"post","link":"http:\/\/miro.borodziuk.eu\/index.php\/2023\/11\/11\/managing-cluster-in-kubernetes\/","title":{"rendered":"Managing Cluster in Kubernetes"},"content":{"rendered":"<p>.<\/p>\n<p><!--more--><\/p>\n<p><span style=\"color: #3366ff;\">Analyzing Cluster Nodes<\/span><\/p>\n<ul>\n<li>Kubernetes cluster nodes run Linux processes. To monitor these processes,<br \/>\ngeneric Linux rules apply<\/p>\n<ul>\n<li>Use<code> systemctl status kubelet<\/code> to get runtime information about the kubelet<\/li>\n<li>Use log files in<code> \/var\/log<\/code> as well as journalctl output to get access to logs<\/li>\n<\/ul>\n<\/li>\n<li>Generic node information is obtained through kubectl describe<\/li>\n<li>If the Metrics Server is installed, use kubectl top nodes to get a summary<br \/>\nof CPU\/memory usage on a node.<\/li>\n<\/ul>\n<p><span style=\"color: #3366ff;\">Analyzing Node State Commands<\/span><\/p>\n<ul>\n<li><code>Is -lrt \/var\/log<\/code><\/li>\n<li><code>journalctl<\/code><\/li>\n<li><code>systemctl status kubelet<\/code><\/li>\n<\/ul>\n<p>Now, let&#8217;s reproduce a failure:<\/p>\n<pre class=\"lang:default decode:true \">[root@k8s ~]# kubectl get nodes\r\nNAME            STATUS   ROLES           AGE    VERSION\r\nk8s.example.pl   Ready    control-plane   3d3h   v1.28.3\r\n\r\n[root@k8s ~]# systemctl stop kubelet\r\n\r\n[root@k8s ~]# systemctl status kubelet\r\n\u25cf kubelet.service - kubelet: The Kubernetes Node Agent\r\n   Loaded: loaded (\/usr\/lib\/systemd\/system\/kubelet.service; disabled; vendor preset: disabled)\r\n  Drop-In: \/etc\/systemd\/system\/kubelet.service.d\r\n           \u2514\u250010-kubeadm.conf\r\n   Active: inactive (dead) since Sat 2024-02-03 13:55:06 EST; 8s ago\r\n     Docs: http:\/\/kubernetes.io\/docs\/\r\n  Process: 14821 ExecStart=\/var\/lib\/minikube\/binaries\/v1.28.3\/kubelet --bootstrap-kubeconfig=\/etc\/kubernetes\/bootstrap-kubelet.conf --config=\/var\/lib\/ku&gt;\r\n Main PID: 14821 (code=exited, status=0\/SUCCESS)\r\n\r\nlut 03 13:54:33 k8s.example.pl kubelet[14821]: E0203 13:54:33.646323   14821 pod_workers.go:1300] \"Error syncing pod, skipping\" err=\"failed to \\\"StartCon&gt;\r\nlut 03 13:54:45 k8s.example.pl kubelet[14821]: I0203 13:54:45.645915   14821 scope.go:117] \"RemoveContainer\" containerID=\"a31a7becd0cfd282ab55f4c39c573b4&gt;\r\nlut 03 13:54:45 k8s.example.pl kubelet[14821]: E0203 13:54:45.646441   14821 pod_workers.go:1300] \"Error syncing pod, skipping\" err=\"failed to \\\"StartCon&gt;\r\nlut 03 13:54:59 k8s.example.pl kubelet[14821]: E0203 13:54:59.729342   14821 desired_state_of_world_populator.go:320] \"Error processing volume\" err=\"erro&gt;\r\nlut 03 13:55:00 k8s.example.pl kubelet[14821]: I0203 13:55:00.646196   14821 scope.go:117] \"RemoveContainer\" containerID=\"a31a7becd0cfd282ab55f4c39c573b4&gt;\r\nlut 03 13:55:00 k8s.example.pl kubelet[14821]: E0203 13:55:00.646616   14821 pod_workers.go:1300] \"Error syncing pod, skipping\" err=\"failed to \\\"StartCon&gt;\r\nlut 03 13:55:06 k8s.example.pl systemd[1]: Stopping kubelet: The Kubernetes Node Agent...\r\nlut 03 13:55:06 k8s.example.pl kubelet[14821]: I0203 13:55:06.846437   14821 dynamic_cafile_content.go:171] \"Shutting down controller\" name=\"client-ca-bu&gt;\r\nlut 03 13:55:06 k8s.example.pl systemd[1]: kubelet.service: Succeeded.\r\nlut 03 13:55:06 k8s.example.pl systemd[1]: Stopped kubelet: The Kubernetes Node Agent.\r\n\r\n[root@k8s ~]# kubectl describe node k8s.example.pl\r\nName:               k8s.example.pl\r\nRoles:              control-plane\r\nLabels:             beta.kubernetes.io\/arch=amd64\r\n                    beta.kubernetes.io\/os=linux\r\n                    kubernetes.io\/arch=amd64\r\n                    kubernetes.io\/hostname=k8s.example.pl\r\n                    kubernetes.io\/os=linux\r\n                    minikube.k8s.io\/commit=8220a6eb95f0a4d75f7f2d7b14cef975f050512d\r\n                    minikube.k8s.io\/name=minikube\r\n                    minikube.k8s.io\/primary=true\r\n                    minikube.k8s.io\/updated_at=2024_01_31T10_03_27_0700\r\n                    minikube.k8s.io\/version=v1.32.0\r\n                    node-role.kubernetes.io\/control-plane=\r\n                    node.kubernetes.io\/exclude-from-external-load-balancers=\r\nAnnotations:        kubeadm.alpha.kubernetes.io\/cri-socket: unix:\/\/\/var\/run\/cri-dockerd.sock\r\n                    node.alpha.kubernetes.io\/ttl: 0\r\n                    volumes.kubernetes.io\/controller-managed-attach-detach: true\r\nCreationTimestamp:  Wed, 31 Jan 2024 10:03:23 -0500\r\nTaints:             node.kubernetes.io\/unreachable:NoSchedule\r\nUnschedulable:      false\r\nLease:\r\n  HolderIdentity:  k8s.example.pl\r\n  AcquireTime:     &lt;unset&gt;\r\n  RenewTime:       Sat, 03 Feb 2024 13:54:59 -0500\r\nConditions:\r\n  Type             Status    LastHeartbeatTime                 LastTransitionTime                Reason              Message\r\n  ----             ------    -----------------                 ------------------                ------              -------\r\n  MemoryPressure   Unknown   Sat, 03 Feb 2024 13:51:48 -0500   Sat, 03 Feb 2024 13:55:39 -0500   NodeStatusUnknown   Kubelet stopped posting node status.\r\n  DiskPressure     Unknown   Sat, 03 Feb 2024 13:51:48 -0500   Sat, 03 Feb 2024 13:55:39 -0500   NodeStatusUnknown   Kubelet stopped posting node status.\r\n  PIDPressure      Unknown   Sat, 03 Feb 2024 13:51:48 -0500   Sat, 03 Feb 2024 13:55:39 -0500   NodeStatusUnknown   Kubelet stopped posting node status.\r\n  Ready            Unknown   Sat, 03 Feb 2024 13:51:48 -0500   Sat, 03 Feb 2024 13:55:39 -0500   NodeStatusUnknown   Kubelet stopped posting node status.\r\nAddresses:\r\n  InternalIP:  172.30.9.24\r\n  Hostname:    k8s.example.pl\r\nCapacity:\r\n  cpu:                8\r\n  ephemeral-storage:  64177544Ki\r\n  hugepages-1Gi:      0\r\n  hugepages-2Mi:      0\r\n  memory:             16099960Ki\r\n  pods:               110\r\nAllocatable:\r\n  cpu:                8\r\n  ephemeral-storage:  64177544Ki\r\n  hugepages-1Gi:      0\r\n  hugepages-2Mi:      0\r\n  memory:             16099960Ki\r\n  pods:               110\r\nSystem Info:\r\n  Machine ID:                 0cc7c63085694b83adcd204eff748ff8\r\n  System UUID:                3e3ec47d-1fe1-b5b7-cbca-edd2da14db37\r\n  Boot ID:                    79a4e58f-5d2a-4f44-ad34-520bab9b01cc\r\n  Kernel Version:             4.18.0-500.el8.x86_64\r\n  OS Image:                   CentOS Stream 8\r\n  Operating System:           linux\r\n  Architecture:               amd64\r\n  Container Runtime Version:  docker:\/\/25.0.1\r\n  Kubelet Version:            v1.28.3\r\n  Kube-Proxy Version:         v1.28.3\r\nPodCIDR:                      10.244.0.0\/24\r\nPodCIDRs:                     10.244.0.0\/24\r\nNon-terminated Pods:          (35 in total)\r\n  Namespace                   Name                                          CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age\r\n  ---------                   ----                                          ------------  ----------  ---------------  -------------  ---\r\n  default                     apples-78656fd5db-4rpj7                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         20h\r\n  default                     apples-78656fd5db-qsm4x                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         20h\r\n  default                     apples-78656fd5db-t82tg                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         20h\r\n  default                     deploydaemon-zzllp                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         45h\r\n  default                     firstnginx-d8679d567-249g9                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d22h\r\n  default                     firstnginx-d8679d567-66c4s                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d22h\r\n  default                     firstnginx-d8679d567-72qbd                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d22h\r\n  default                     firstnginx-d8679d567-rhhlz                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d5h\r\n  default                     init-demo                                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d7h\r\n  default                     lab4-pod                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         28h\r\n  default                     morevol                                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         41h\r\n  default                     mydaemon-d4dcd                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         45h\r\n  default                     newdep-749c9b5675-2x9mb                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         21h\r\n  default                     nginxsvc-5f8b7d4f4d-dtrs7                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         22h\r\n  default                     pv-pod                                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         40h\r\n  default                     sleepy                                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d8h\r\n  default                     testpod                                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d23h\r\n  default                     two-containers                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d5h\r\n  default                     web-0                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d10h\r\n  default                     web-1                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         45h\r\n  default                     web-2                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         45h\r\n  default                     webserver-76d44586d-8gqhf                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         29h\r\n  default                     webshop-7f9fd49d4c-92nj2                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         25h\r\n  default                     webshop-7f9fd49d4c-kqllw                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         25h\r\n  default                     webshop-7f9fd49d4c-x2czc                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         25h\r\n  ingress-nginx               ingress-nginx-controller-6858749594-27tm9     100m (1%)     0 (0%)      90Mi (0%)        0 (0%)         22h\r\n  kube-system                 coredns-5dd5756b68-sgfkj                      100m (1%)     0 (0%)      70Mi (0%)        170Mi (1%)     3d3h\r\n  kube-system                 etcd-k8s.example.pl                            100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         3d3h\r\n  kube-system                 kube-apiserver-k8s.example.pl                  250m (3%)     0 (0%)      0 (0%)           0 (0%)         3d3h\r\n  kube-system                 kube-controller-manager-k8s.example.pl         200m (2%)     0 (0%)      0 (0%)           0 (0%)         3d3h\r\n  kube-system                 kube-proxy-5nmms                              0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d3h\r\n  kube-system                 kube-scheduler-k8s.example.pl                  100m (1%)     0 (0%)      0 (0%)           0 (0%)         3d3h\r\n  kube-system                 storage-provisioner                           0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d3h\r\n  kubernetes-dashboard        dashboard-metrics-scraper-7fd5cb4ddc-9ld5n    0 (0%)        0 (0%)      0 (0%)           0 (0%)         46h\r\n  kubernetes-dashboard        kubernetes-dashboard-8694d4445c-xjlsr         0 (0%)        0 (0%)      0 (0%)           0 (0%)         46h\r\nAllocated resources:\r\n  (Total limits may be over 100 percent, i.e., overcommitted.)\r\n  Resource           Requests    Limits\r\n  --------           --------    ------\r\n  cpu                850m (10%)  0 (0%)\r\n  memory             260Mi (1%)  170Mi (1%)\r\n  ephemeral-storage  0 (0%)      0 (0%)\r\n  hugepages-1Gi      0 (0%)      0 (0%)\r\n  hugepages-2Mi      0 (0%)      0 (0%)\r\nEvents:\r\n  Type    Reason        Age   From             Message\r\n  ----    ------        ----  ----             -------\r\n  Normal  NodeNotReady  74s   node-controller  Node k8s.example.pl status is now: NodeNotReady\r\n\r\n[root@k8s ~]# systemctl start kubelet\r\n\r\n[root@k8s ~]# kubectl describe node k8s.example.pl\r\nName:               k8s.example.pl\r\nRoles:              control-plane\r\nLabels:             beta.kubernetes.io\/arch=amd64\r\n                    beta.kubernetes.io\/os=linux\r\n                    kubernetes.io\/arch=amd64\r\n                    kubernetes.io\/hostname=k8s.example.pl\r\n                    kubernetes.io\/os=linux\r\n                    minikube.k8s.io\/commit=8220a6eb95f0a4d75f7f2d7b14cef975f050512d\r\n                    minikube.k8s.io\/name=minikube\r\n                    minikube.k8s.io\/primary=true\r\n                    minikube.k8s.io\/updated_at=2024_01_31T10_03_27_0700\r\n                    minikube.k8s.io\/version=v1.32.0\r\n                    node-role.kubernetes.io\/control-plane=\r\n                    node.kubernetes.io\/exclude-from-external-load-balancers=\r\nAnnotations:        kubeadm.alpha.kubernetes.io\/cri-socket: unix:\/\/\/var\/run\/cri-dockerd.sock\r\n                    node.alpha.kubernetes.io\/ttl: 0\r\n                    volumes.kubernetes.io\/controller-managed-attach-detach: true\r\nCreationTimestamp:  Wed, 31 Jan 2024 10:03:23 -0500\r\nTaints:             &lt;none&gt;\r\nUnschedulable:      false\r\nLease:\r\n  HolderIdentity:  k8s.example.pl\r\n  AcquireTime:     &lt;unset&gt;\r\n  RenewTime:       Sat, 03 Feb 2024 13:59:11 -0500\r\nConditions:\r\n  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message\r\n  ----             ------  -----------------                 ------------------                ------                       -------\r\n  MemoryPressure   False   Sat, 03 Feb 2024 13:59:11 -0500   Sat, 03 Feb 2024 13:59:11 -0500   KubeletHasSufficientMemory   kubelet has sufficient memory                                                                                     available\r\n  DiskPressure     False   Sat, 03 Feb 2024 13:59:11 -0500   Sat, 03 Feb 2024 13:59:11 -0500   KubeletHasNoDiskPressure     kubelet has no disk pressure\r\n  PIDPressure      False   Sat, 03 Feb 2024 13:59:11 -0500   Sat, 03 Feb 2024 13:59:11 -0500   KubeletHasSufficientPID      kubelet has sufficient PID av                                                                                    ailable\r\n  Ready            True    Sat, 03 Feb 2024 13:59:11 -0500   Sat, 03 Feb 2024 13:59:11 -0500   KubeletReady                 kubelet is posting ready stat                                                                                    us\r\nAddresses:\r\n  InternalIP:  172.30.9.24\r\n  Hostname:    k8s.example.pl\r\nCapacity:\r\n  cpu:                8\r\n  ephemeral-storage:  64177544Ki\r\n  hugepages-1Gi:      0\r\n  hugepages-2Mi:      0\r\n  memory:             16099960Ki\r\n  pods:               110\r\nAllocatable:\r\n  cpu:                8\r\n  ephemeral-storage:  64177544Ki\r\n  hugepages-1Gi:      0\r\n  hugepages-2Mi:      0\r\n  memory:             16099960Ki\r\n  pods:               110\r\nSystem Info:\r\n  Machine ID:                 0cc7c63085694b83adcd204eff748ff8\r\n  System UUID:                3e3ec47d-1fe1-b5b7-cbca-edd2da14db37\r\n  Boot ID:                    79a4e58f-5d2a-4f44-ad34-520bab9b01cc\r\n  Kernel Version:             4.18.0-500.el8.x86_64\r\n  OS Image:                   CentOS Stream 8\r\n  Operating System:           linux\r\n  Architecture:               amd64\r\n  Container Runtime Version:  docker:\/\/25.0.1\r\n  Kubelet Version:            v1.28.3\r\n  Kube-Proxy Version:         v1.28.3\r\nPodCIDR:                      10.244.0.0\/24\r\nPodCIDRs:                     10.244.0.0\/24\r\nNon-terminated Pods:          (35 in total)\r\n  Namespace                   Name                                          CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age\r\n  ---------                   ----                                          ------------  ----------  ---------------  -------------  ---\r\n  default                     apples-78656fd5db-4rpj7                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         21h\r\n  default                     apples-78656fd5db-qsm4x                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         21h\r\n  default                     apples-78656fd5db-t82tg                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         21h\r\n  default                     deploydaemon-zzllp                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         45h\r\n  default                     firstnginx-d8679d567-249g9                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d22h\r\n  default                     firstnginx-d8679d567-66c4s                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d22h\r\n  default                     firstnginx-d8679d567-72qbd                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d22h\r\n  default                     firstnginx-d8679d567-rhhlz                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d5h\r\n  default                     init-demo                                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d7h\r\n  default                     lab4-pod                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         28h\r\n  default                     morevol                                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         41h\r\n  default                     mydaemon-d4dcd                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         46h\r\n  default                     newdep-749c9b5675-2x9mb                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         21h\r\n  default                     nginxsvc-5f8b7d4f4d-dtrs7                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         22h\r\n  default                     pv-pod                                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         40h\r\n  default                     sleepy                                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d8h\r\n  default                     testpod                                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d23h\r\n  default                     two-containers                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d5h\r\n  default                     web-0                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d10h\r\n  default                     web-1                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         46h\r\n  default                     web-2                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         46h\r\n  default                     webserver-76d44586d-8gqhf                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         29h\r\n  default                     webshop-7f9fd49d4c-92nj2                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         25h\r\n  default                     webshop-7f9fd49d4c-kqllw                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         25h\r\n  default                     webshop-7f9fd49d4c-x2czc                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         25h\r\n  ingress-nginx               ingress-nginx-controller-6858749594-27tm9     100m (1%)     0 (0%)      90Mi (0%)        0 (0%)         22h\r\n  kube-system                 coredns-5dd5756b68-sgfkj                      100m (1%)     0 (0%)      70Mi (0%)        170Mi (1%)     3d3h\r\n  kube-system                 etcd-k8s.example.pl                            100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         3d3h\r\n  kube-system                 kube-apiserver-k8s.example.pl                  250m (3%)     0 (0%)      0 (0%)           0 (0%)         3d3h\r\n  kube-system                 kube-controller-manager-k8s.example.pl         200m (2%)     0 (0%)      0 (0%)           0 (0%)         3d3h\r\n  kube-system                 kube-proxy-5nmms                              0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d3h\r\n  kube-system                 kube-scheduler-k8s.example.pl                  100m (1%)     0 (0%)      0 (0%)           0 (0%)         3d3h\r\n  kube-system                 storage-provisioner                           0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d3h\r\n  kubernetes-dashboard        dashboard-metrics-scraper-7fd5cb4ddc-9ld5n    0 (0%)        0 (0%)      0 (0%)           0 (0%)         46h\r\n  kubernetes-dashboard        kubernetes-dashboard-8694d4445c-xjlsr         0 (0%)        0 (0%)      0 (0%)           0 (0%)         46h\r\nAllocated resources:\r\n  (Total limits may be over 100 percent, i.e., overcommitted.)\r\n  Resource           Requests    Limits\r\n  --------           --------    ------\r\n  cpu                850m (10%)  0 (0%)\r\n  memory             260Mi (1%)  170Mi (1%)\r\n  ephemeral-storage  0 (0%)      0 (0%)\r\n  hugepages-1Gi      0 (0%)      0 (0%)\r\n  hugepages-2Mi      0 (0%)      0 (0%)\r\nEvents:\r\n  Type    Reason                   Age              From             Message\r\n  ----    ------                   ----             ----             -------\r\n  Normal  NodeNotReady             3m36s            node-controller  Node k8s.example.pl status is now: NodeNotReady\r\n  Normal  Starting                 4s               kubelet          Starting kubelet.\r\n  Normal  NodeAllocatableEnforced  4s               kubelet          Updated Node Allocatable limit across pods\r\n  Normal  NodeReady                4s               kubelet          Node k8s.example.pl status is now: NodeReady\r\n  Normal  NodeHasSufficientMemory  3s (x2 over 4s)  kubelet          Node k8s.example.pl status is now: NodeHasSufficientMemory\r\n  Normal  NodeHasNoDiskPressure    3s (x2 over 4s)  kubelet          Node k8s.example.pl status is now: NodeHasNoDiskPressure\r\n  Normal  NodeHasSufficientPID     3s (x2 over 4s)  kubelet          Node k8s.example.pl status is now: NodeHasSufficientPID\r\n\r\n[root@k8s ~]# ls -lrt \/var\/log\r\nrazem 46576\r\ndrwxr-xr-x.  2 root   root          6 2021-08-31  glusterfs\r\ndrwx------.  2 root   root          6 2023-07-17  private\r\ndrwxr-xr-x.  3 root   root         21 2023-07-17  swtpm\r\ndrwxr-xr-x.  2 root   root       4096 2023-07-17  anaconda\r\n-rw-------.  1 root   root      54470 2023-07-18  boot.log-20230718\r\ndrwxr-xr-x.  2 root   root         23 09-04 12:17 tuned\r\n-rw-r--r--.  1 root   root    1048499 10-13 18:11 dnf.librepo.log.1\r\n-rw-------.  1 root   root          0 10-22 03:19 secure-20231029\r\n-rw-------.  1 root   root          0 10-22 03:19 maillog-20231029\r\n-rw-------.  1 root   root          0 10-22 03:19 spooler-20231029\r\n-rw-r--r--.  1 root   root       2700 10-29 02:00 hawkey.log-20231029\r\n-rw-------.  1 root   root      53654 10-29 02:00 messages-20231029\r\n-rw-------.  1 root   root      45594 10-29 03:01 cron-20231029\r\n-rw-------.  1 root   root          0 10-29 03:07 spooler-20231105\r\n-rw-------.  1 root   root          0 10-29 03:07 secure-20231105\r\n-rw-------.  1 root   root          0 10-29 03:07 maillog-20231105\r\n-rw-------.  1 root   root      46120 11-05 03:01 cron-20231105\r\n-rw-r--r--.  1 root   root       2580 11-05 03:09 hawkey.log-20231105\r\n-rw-------.  1 root   root      53957 11-05 03:09 messages-20231105\r\n-rw-------.  1 root   root          0 11-05 03:22 spooler-20231112\r\n-rw-------.  1 root   root          0 11-05 03:22 secure-20231112\r\n-rw-------.  1 root   root          0 11-05 03:22 maillog-20231112\r\ndrwx------.  2 root   root         23 11-06 03:13 audit\r\n-rw-r--r--.  1 root   root       2640 11-12 01:35 hawkey.log-20231112\r\n-rw-------.  1 root   root      53494 11-12 02:50 messages-20231112\r\n-rw-------.  1 root   root      45736 11-12 03:34 cron-20231112\r\n-rw-------.  1 root   root          0 11-12 03:34 spooler-20240131\r\n-rw-------.  1 root   root          0 11-12 03:34 maillog-20240131\r\ndrwx------.  3 root   root         18 12-12 13:03 libvirt\r\ndrwxr-x---.  2 sssd   sssd          6 01-13 11:33 sssd\r\ndrwxr-xr-x.  2 root   root          6 01-17 14:34 qemu-ga\r\ndrwx------.  3 root   root         17 01-18 06:29 samba\r\ndrwxr-x---.  2 chrony chrony        6 01-23 11:14 chrony\r\n-rw-r--r--.  1 root   root    1048519 01-31 08:35 dnf.log.4\r\n-rw-r--r--.  1 root   root     976665 01-31 08:45 dnf.log.3\r\n-rw-r--r--.  1 root   root     922460 01-31 08:49 dnf.log.2\r\n-rw-r--r--.  1 root   root     929588 01-31 08:50 dnf.log.1\r\n-rw-------.  1 root   root       4340 01-31 09:01 cron-20240131\r\n-rw-------.  1 root   root       7290 01-31 09:44 secure-20240131\r\n-rw-r--r--.  1 root   root       2040 01-31 09:47 hawkey.log-20240131\r\ndrwx------.  3 root   root         18 01-31 09:49 crio\r\n-rw-------.  1 root   root     269633 01-31 09:49 messages-20240131\r\n-rw-------.  1 root   root      18241 01-31 09:51 boot.log-20240131\r\n-rw-------.  1 root   root          0 01-31 09:51 spooler\r\n-rw-------.  1 root   root          0 01-31 09:51 maillog\r\n-rw-rw----.  1 root   utmp        384 01-31 10:06 btmp-20240201\r\n-rw-rw----.  1 root   utmp          0 02-01 03:25 btmp\r\n-rw-------.  1 root   root       6190 02-01 08:38 kdump.log\r\n-rw-r-----.  1 root   root      20742 02-01 15:47 firewalld\r\n-rw-------   1 root   root       9508 02-02 03:40 boot.log-20240202\r\n-rw-------.  1 root   root          0 02-02 03:40 boot.log\r\ndrwxr-xr-x. 37 root   root       4096 02-02 16:57 pods\r\n-rw-r--r--.  1 root   root       1740 02-03 11:40 hawkey.log\r\n-rw-------.  1 root   root      19604 02-03 13:01 cron\r\n-rw-r--r--.  1 root   root     497330 02-03 13:38 dnf.librepo.log\r\n-rw-r--r--.  1 root   root     182173 02-03 13:38 dnf.rpm.log\r\n-rw-r--r--.  1 root   root     785398 02-03 13:38 dnf.log\r\n-rw-rw-r--.  1 root   utmp      40320 02-03 13:52 wtmp\r\n-rw-rw-r--.  1 root   utmp     292584 02-03 13:52 lastlog\r\n-rw-------.  1 root   root     352561 02-03 13:52 secure\r\ndrwxr-xr-x.  2 root   root       8192 02-03 13:59 containers\r\n-rw-------.  1 root   root   38193765 02-03 13:59 messages\r\n\r\n[root@k8s ~]# journalctl -u kubelet\r\n-- Logs begin at Thu 2024-02-01 08:38:25 EST, end at Sat 2024-02-03 14:02:35 EST. --\r\nlut 01 08:40:55 k8s.example.pl systemd[1]: Started kubelet: The Kubernetes Node Agent.\r\nlut 01 08:40:55 k8s.example.pl kubelet[3174]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file sp&gt;\r\nlut 01 08:40:55 k8s.example.pl kubelet[3174]: I0201 08:40:55.958248    3174 server.go:467] \"Kubelet version\" kubeletVersion=\"v1.28.3\"\r\nlut 01 08:40:55 k8s.example.pl kubelet[3174]: I0201 08:40:55.958354    3174 server.go:469] \"Golang settings\" GOGC=\"\" GOMAXPROCS=\"\" GOTRACEBACK=\"\"\r\nlut 01 08:40:55 k8s.example.pl kubelet[3174]: I0201 08:40:55.958716    3174 server.go:895] \"Client rotation is on, will bootstrap in background\"\r\nlut 01 08:40:55 k8s.example.pl kubelet[3174]: I0201 08:40:55.965902    3174 certificate_store.go:130] Loading cert\/key pair from \"\/var\/lib\/kubelet\/pki\/k&gt;\r\nlut 01 08:40:55 k8s.example.pl kubelet[3174]: I0201 08:40:55.967590    3174 dynamic_cafile_content.go:157] \"Starting controller\" name=\"client-ca-bundle:&gt;\r\nlut 01 08:40:55 k8s.example.pl kubelet[3174]: E0201 08:40:55.970239    3174 certificate_manager.go:562] kubernetes.io\/kube-apiserver-client-kubelet: Fai&gt;\r\nlut 01 08:40:55 k8s.example.pl kubelet[3174]: I0201 08:40:55.992341    3174 server.go:725] \"--cgroups-per-qos enabled, but --cgroup-root was not specifi&gt;\r\n\r\n[root@k8s ~]# systemctl status kubelet\r\n\u25cf kubelet.service - kubelet: The Kubernetes Node Agent\r\n   Loaded: loaded (\/usr\/lib\/systemd\/system\/kubelet.service; disabled; vendor preset: disabled)\r\n  Drop-In: \/etc\/systemd\/system\/kubelet.service.d\r\n           \u2514\u250010-kubeadm.conf\r\n   Active: active (running) since Sat 2024-02-03 13:59:11 EST; 3min 50s ago\r\n     Docs: http:\/\/kubernetes.io\/docs\/\r\n Main PID: 555436 (kubelet)\r\n    Tasks: 16 (limit: 100376)\r\n   Memory: 56.0M\r\n   CGroup: \/system.slice\/kubelet.service\r\n           \u2514\u2500555436 \/var\/lib\/minikube\/binaries\/v1.28.3\/kubelet --bootstrap-kubeconfig=\/etc\/kubernetes\/bootstrap-kubelet.conf --config=\/var\/lib\/kubelet\/c&gt;\r\n\r\nlut 03 14:02:55 k8s.example.pl kubelet[555436]: E0203 14:02:55.687080  555436 kubelet.go:1907] \"Unable to attach or mount volumes for pod; skipping pod\" &gt;\r\nlut 03 14:02:55 k8s.example.pl kubelet[555436]: E0203 14:02:55.687181  555436 pod_workers.go:1300] \"Error syncing pod, skipping\" err=\"unmounted volumes=[&gt;\r\nlut 03 14:02:56 k8s.example.pl kubelet[555436]: E0203 14:02:56.435108  555436 desired_state_of_world_populator.go:320] \"Error processing volume\" err=\"err&gt;\r\nlut 03 14:02:56 k8s.example.pl kubelet[555436]: E0203 14:02:56.686939  555436 kubelet.go:1907] \"Unable to attach or mount volumes for pod; skipping pod\" &gt;\r\nlut 03 14:02:56 k8s.example.pl kubelet[555436]: E0203 14:02:56.687005  555436 pod_workers.go:1300] \"Error syncing pod, skipping\" err=\"unmounted volumes=[&gt;\r\nlut 03 14:02:57 k8s.example.pl kubelet[555436]: E0203 14:02:57.445006  555436 desired_state_of_world_populator.go:320] \"Error processing volume\" err=\"err&gt;\r\nlut 03 14:02:57 k8s.example.pl kubelet[555436]: E0203 14:02:57.686464  555436 kubelet.go:1907] \"Unable to attach or mount volumes for pod; skipping pod\" &gt;\r\nlut 03 14:02:57 k8s.example.pl kubelet[555436]: E0203 14:02:57.686526  555436 pod_workers.go:1300] \"Error syncing pod, skipping\" err=\"unmounted volumes=[&gt;\r\nlut 03 14:02:59 k8s.example.pl kubelet[555436]: I0203 14:02:59.386449  555436 scope.go:117] \"RemoveContainer\" containerID=\"e6e599d43e20022cb554472cf2e128&gt;\r\nlut 03 14:02:59 k8s.example.pl kubelet[555436]: E0203 14:02:59.386925  555436 pod_workers.go:1300] \"Error syncing pod, skipping\" err=\"failed to \\\"StartCo&gt;\r\n<\/pre>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\"><code>crictl<\/code> command<\/span><\/p>\n<ul>\n<li>All Pods are started as containers on the nodes<\/li>\n<li><code>crictl<\/code> is a generic tool that communicates to the container runtime to get<br \/>\ninformation about running containers<\/li>\n<li>As such, it replaces generic tools like docker and podman<\/li>\n<li>To use it, a runtime-endpoint and image-endpoint need to be set<\/li>\n<li>The most convenient way to do so, is by defining the <code>\/etc\/crictl.yaml<\/code> file on<br \/>\nthe nodes where you want to run <code>crictl<\/code><\/li>\n<\/ul>\n<p><span style=\"color: #3366ff;\">Using crictl<\/span><\/p>\n<ul>\n<li>List containers: <code>sudo crictl ps<\/code><\/li>\n<li><code><\/code>List Pods that have been scheduled on this node: <code>sudo crictl pods<\/code><\/li>\n<li><code><\/code>Inspect container configuration: <code>sudo crictl inspect &lt;name-or-id&gt;<\/code><\/li>\n<li><code><\/code>Pull an image:<code> sudo crictl pull &lt;imagename&gt;<\/code><\/li>\n<li>List images: <code>sudo crictl images<\/code><\/li>\n<li>For more options, <code>use crictl --help<\/code><\/li>\n<\/ul>\n<pre class=\"lang:default decode:true \">[root@k8s ~]# crictl ps\r\nCONTAINER           IMAGE                                                                                                              CREATED                                                                                                 STATE               NAME                        ATTEMPT             POD ID              POD\r\nbc6f6c3b226a9       busybox@sha256:6d9ac9237a84afe1516540f40a0fafdc86859b2141954b4d643af7066d598b74                                    2 minutes ago                                                                                           Running             busybox-container           272                 776a9d9f213bc       two-containers\r\n44bb58b75411b       busybox@sha256:6d9ac9237a84afe1516540f40a0fafdc86859b2141954b4d643af7066d598b74                                    26 minutes ago                                                                                          Running             sleepy                      46                  b598cba0e6d7f       sleepy\r\n04f6f10d3e46a       eeb6ee3f44bd0                                                                                                      59 minutes ago                                                                                          Running             centos2                     41                  4eb967073bfbd       morevol\r\nc8d89745f340c       eeb6ee3f44bd0                                                                                                      59 minutes ago                                                                                          Running             centos1                     41                  4eb967073bfbd       morevol\r\na65e79f99bba9       nginx@sha256:31754bca89a3afb25c04d6ecfa2d9671bc3972d8f4809ff855f7e35caa580de9                                      21 hours ago                                                                                            Running             nginx                       0                   9793a44f77d5a       apples-78656fd5db-qsm4x\r\n383fb11871cd9       nginx@sha256:31754bca89a3afb25c04d6ecfa2d9671bc3972d8f4809ff855f7e35caa580de9                                      21 hours ago                                                                                            Running             nginx                       0                   93f7305ee0d77       apples-78656fd5db-4rpj7\r\nbc4f405f4de6f       nginx@sha256:31754bca89a3afb25c04d6ecfa2d9671bc3972d8f4809ff855f7e35caa580de9                                      21 hours ago                                                                                            Running             nginx                       0                   58dbf2f78a5b6       apples-78656fd5db-t82tg\r\n464ab8ab35afc       gcr.io\/google-samples\/hello-app@sha256:7104356ed4e3476a96a23b96f8d7c04dfa7a1881aa97d66a76217f6bc8a370d0            22 hours ago                                                                                            Running             hello-app                   0                   64eac220b635b       newdep-749c9b5675-2x9mb\r\n3a65e1db97169       nginx@sha256:31754bca89a3afb25c04d6ecfa2d9671bc3972d8f4809ff855f7e35caa580de9                                      23 hours ago                                                                                            Running             nginx                       0                   0fd201ae2d934       nginxsvc-5f8b7d4f4d-dtrs7\r\n1a4722cbaaf94       registry.k8s.io\/ingress-nginx\/controller@sha256:1405cc613bd95b2c6edd8b2a152510ae91c7e62aea4698500d23b2145960ab9c   23 hours ago                                                                                            Running             controller                  0                   ea8cb6f3530f0       ingress-nginx-controller-6858749594-27tm9\r\n332cd7a3b2aa9       nginx@sha256:31754bca89a3afb25c04d6ecfa2d9671bc3972d8f4809ff855f7e35caa580de9                                      26 hours ago                                                                                            Running             nginx                       0                   8956ee62249ab       webshop-7f9fd49d4c-x2czc\r\ne136bd99527b9       nginx@sha256:31754bca89a3afb25c04d6ecfa2d9671bc3972d8f4809ff855f7e35caa580de9                                      26 hours ago                                                                                            Running             nginx                       0                   ae8bad2f8c457       webshop-7f9fd49d4c-92nj2\r\n2c1067f28073c       nginx@sha256:31754bca89a3afb25c04d6ecfa2d9671bc3972d8f4809ff855f7e35caa580de9                                      26 hours ago                                                                                            Running             nginx                       0                   746bebf244884       webshop-7f9fd49d4c-kqllw\r\nc6c00eece623f       nginx@sha256:31754bca89a3afb25c04d6ecfa2d9671bc3972d8f4809ff855f7e35caa580de9                                      29 hours ago                                                                                            Running             task-pv-container           0                   d39bb441ef944       lab4-pod\r\n036a4a1599a1a       nginx@sha256:31754bca89a3afb25c04d6ecfa2d9671bc3972d8f4809ff855f7e35caa580de9                                      30 hours ago                                                                                            Running             nginx                       0                   6b4abfa363771       webserver-76d44586d-8gqhf\r\nf7426897bdb2e       nginx@sha256:31754bca89a3afb25c04d6ecfa2d9671bc3972d8f4809ff855f7e35caa580de9                                      41 hours ago                                                                                            Running             pv-container                0                   a09dd92ff2186       pv-pod\r\n8dc188f5131a2       nginx@sha256:985224176778a8939b3869d3b9b9624ea9b3fe4eb1e9002c5f444d99ef034a9b                                      46 hours ago                                                                                            Running             nginx                       0                   ced6eefe01d16       deploydaemon-zzllp\r\n29bf2d747ac9a       nginx@sha256:985224176778a8939b3869d3b9b9624ea9b3fe4eb1e9002c5f444d99ef034a9b                                      46 hours ago                                                                                            Running             nginx                       0                   2a08474cc3d2a       init-demo\r\n...                                                                                               47 hours ago                                                                                            Running             kube-scheduler              1                   6e075c204c3c8       kube-scheduler-k8s.example.pl\r\n\r\n[root@k8s cka]# crictl pods\r\nPOD ID              CREATED             STATE               NAME                                         NAMESPACE              ATTEMPT             RUNTI                                                                                    ME\r\n93f7305ee0d77       22 hours ago        Ready               apples-78656fd5db-4rpj7                      default                0                   (defa                                                                                    ult)\r\n9793a44f77d5a       22 hours ago        Ready               apples-78656fd5db-qsm4x                      default                0                   (defa                                                                                    ult)\r\n58dbf2f78a5b6       22 hours ago        Ready               apples-78656fd5db-t82tg                      default                0                   (defa                                                                                    ult)\r\n64eac220b635b       23 hours ago        Ready               newdep-749c9b5675-2x9mb                      default                0                   (defa                                                                                    ult)\r\n0fd201ae2d934       24 hours ago        Ready               nginxsvc-5f8b7d4f4d-dtrs7                    default                0                   (defa                                                                                    ult)\r\nea8cb6f3530f0       24 hours ago        Ready               ingress-nginx-controller-6858749594-27tm9    ingress-nginx          0                   (defa                                                                                    ult)\r\nae8bad2f8c457       26 hours ago        Ready               webshop-7f9fd49d4c-92nj2                     default                0                   (defa                                                                                    ult)\r\n8956ee62249ab       26 hours ago        Ready               webshop-7f9fd49d4c-x2czc                     default                0                   (defa                                                                                    ult)\r\n746bebf244884       26 hours ago        Ready               webshop-7f9fd49d4c-kqllw                     default                0                   (defa                                                                                    ult)\r\nd39bb441ef944       29 hours ago        Ready               lab4-pod                                     default                0                   (defa                                                                                    ult)\r\n6b4abfa363771       30 hours ago        Ready               webserver-76d44586d-8gqhf                    default                0                   (defa                                                                                    ult)\r\na09dd92ff2186       41 hours ago        Ready               pv-pod                                       default                0                   (defa                                                                                    ult)\r\n4eb967073bfbd       42 hours ago        Ready               morevol                                      default                0                   (defa                                                                                    ult)\r\nced6eefe01d16       47 hours ago        Ready               deploydaemon-zzllp                           default                0                   (defa                                                                                    ult)\r\n11c7ed7ad36e9       47 hours ago        Ready               web-2                                        default                0                   (defa                                                                                    ult)\r\nb1be4ab59e2ca       47 hours ago        Ready               web-1                                        default                0                   (defa                                                                                    ult)\r\n15ef4cc356862       47 hours ago        Ready               web-0                                        default                0                   (defa                                                                                    ult)\r\n776a9d9f213bc       47 hours ago        Ready               two-containers                               default                0                   (defa                                                                                    ult)\r\n...\r\n\r\n[root@k8s cka]# crictl inspect cadb21950a794\r\n{\r\n  \"status\": {\r\n    \"id\": \"cadb21950a7944a7fccd11cdd28bfc0b243338638f970e83055f0f3cc0d4f104\",\r\n    \"metadata\": {\r\n      \"attempt\": 1,\r\n      \"name\": \"POD\"\r\n    },\r\n    \"state\": \"CONTAINER_RUNNING\",\r\n    \"createdAt\": \"2024-02-01T15:18:29.256532148-05:00\",\r\n    \"startedAt\": \"2024-02-01T15:18:29.524703813-05:00\",\r\n    \"finishedAt\": \"0001-01-01T00:00:00Z\",\r\n    \"exitCode\": 0,\r\n    \"image\": {\r\n      \"annotations\": {},\r\n      \"image\": \"registry.k8s.io\/pause:3.9\"\r\n    },\r\n    \"imageRef\": \"docker-pullable:\/\/registry.k8s.io\/pause@sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097\",\r\n    \"reason\": \"\",\r\n    \"message\": \"\",\r\n    \"labels\": {\r\n      \"component\": \"kube-controller-manager\",\r\n      \"io.kubernetes.pod.name\": \"kube-controller-manager-k8s.example.pl\",\r\n      \"io.kubernetes.pod.namespace\": \"kube-system\",\r\n      \"io.kubernetes.pod.uid\": \"5579bcc112143af09d2938747a302b57\",\r\n      \"tier\": \"control-plane\"\r\n    },\r\n    \"annotations\": {\r\n      \"kubernetes.io\/config.hash\": \"5579bcc112143af09d2938747a302b57\",\r\n      \"kubernetes.io\/config.seen\": \"2024-02-01T15:18:28.614844905-05:00\",\r\n      \"kubernetes.io\/config.source\": \"file\"\r\n    },\r\n    \"mounts\": [],\r\n    \"logPath\": \"\"\r\n  },\r\n  \"info\": {\r\n    \"sandboxID\": \"\",\r\n    \"pid\": 14988\r\n  }\r\n}\r\n\r\n[root@k8s cka]# crictl  images\r\nIMAGE                                                TAG                 IMAGE ID            SIZE\r\nbusybox                                              latest              3f57d9401f8d4       4.26MB\r\ncentos                                               7                   eeb6ee3f44bd0       204MB\r\ngcr.io\/google-samples\/hello-app                      2.0                 f59157bf39125       27.2MB\r\ngcr.io\/k8s-minikube\/storage-provisioner              v5                  6e38f40d628db       31.5MB\r\nk8s.gcr.io\/nginx-slim                                0.8                 18ea23a675dae       110MB\r\nkubernetesui\/dashboard                               &lt;none&gt;              07655ddf2eebe       246MB\r\nkubernetesui\/metrics-scraper                         &lt;none&gt;              115053965e86b       43.8MB\r\nnginx                                                latest              b690f5f0a2d53       187MB\r\nnginx                                                &lt;none&gt;              a8758716bb6aa       187MB\r\nregistry.k8s.io\/coredns\/coredns                      v1.10.1             ead0a4a53df89       53.6MB\r\nregistry.k8s.io\/etcd                                 3.5.9-0             73deb9a3f7025       294MB\r\nregistry.k8s.io\/ingress-nginx\/controller             &lt;none&gt;              2bdab7410148a       261MB\r\nregistry.k8s.io\/ingress-nginx\/kube-webhook-certgen   &lt;none&gt;              eb825d2bb76b9       53.6MB\r\nregistry.k8s.io\/kube-apiserver                       v1.28.3             5374347291230       126MB\r\nregistry.k8s.io\/kube-controller-manager              v1.28.3             10baa1ca17068       122MB\r\nregistry.k8s.io\/kube-proxy                           v1.28.3             bfc896cf80fba       73.1MB\r\nregistry.k8s.io\/kube-scheduler                       v1.28.3             6d1b4fd1b182d       60.1MB\r\nregistry.k8s.io\/pause                                3.9                 e6f1816883972       744kB\r\n\r\n[root@k8s cka]# crictl pull docker.io\/library\/mysql\r\nImage is up to date for mysql@sha256:d7c20c5ba268c558f4fac62977f8c7125bde0630ff8946b08dde44135ef40df3\r\n\r\n[root@k8s cka]# crictl --help\r\nNAME:\r\n   crictl - client for CRI\r\n\r\nUSAGE:\r\n   crictl [global options] command [command options] [arguments...]\r\n\r\nVERSION:\r\n   v1.24.1\r\n\r\nCOMMANDS:\r\n   attach              Attach to a running container\r\n   create              Create a new container\r\n   exec                Run a command in a running container\r\n   version             Display runtime version information\r\n   images, image, img  List images\r\n   inspect             Display the status of one or more containers\r\n   inspecti            Return the status of one or more images\r\n   imagefsinfo         Return image filesystem info\r\n   inspectp            Display the status of one or more pods\r\n   logs                Fetch the logs of a container\r\n   port-forward        Forward local port to a pod\r\n   ps                  List containers\r\n   pull                Pull an image from a registry\r\n   run                 Run a new container inside a sandbox\r\n   runp                Run a new pod\r\n   rm                  Remove one or more containers\r\n   rmi                 Remove one or more images\r\n   rmp                 Remove one or more pods\r\n   pods                List pods\r\n   start               Start one or more created containers\r\n...\r\n<\/pre>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Static Pods<\/span><\/p>\n<ul>\n<li>The kubelet systemd process is configured to run static Pods from the<br \/>\n<code>\/etc\/kubernetes\/manifests<\/code> directory<\/li>\n<li>On the control node, static Pods are an essential part of how Kubernetes<br \/>\nworks: systemd starts kubelet, and kubelet starts core Kubernetes services<br \/>\nas static Pods<\/li>\n<li>Administrators can manually add static Pods if so desired, just copy a<br \/>\nmanifest file into the <code>\/etc\/kubernetes\/manifests<\/code> directory and the kubelet<br \/>\nprocess will pick it up<\/li>\n<li>To modify the path where Kubelet picks up the static Pods, edit<br \/>\nstaticPodPath in<code> \/var\/lib\/kubelet\/config.yaml<\/code> and use sudo systemctl<br \/>\nrestart kubelet to restart<\/li>\n<li>Never do this on the control node!<\/li>\n<\/ul>\n<p><span style=\"color: #3366ff;\">Running Static Pods<\/span><\/p>\n<ul>\n<li><code>kubectl run staticpod --image=nginx --dry-run=client -o yaml &gt;<\/code><br \/>\n<code>staticpod.yaml<\/code><\/li>\n<li><code>sudo cp staticpod.yaml \/etc\/kubernetes\/manifests\/<\/code><\/li>\n<li><code>kubectl get pods -o wide<\/code><\/li>\n<\/ul>\n<pre class=\"lang:default mark:38 decode:true\">[root@k8s cka]# kubectl run staticpod --image=nginx --dry-run=client -o yaml\r\napiVersion: v1\r\nkind: Pod\r\nmetadata:\r\n  creationTimestamp: null\r\n  labels:\r\n    run: staticpod\r\n  name: staticpod\r\nspec:\r\n  containers:\r\n  - image: nginx\r\n    name: staticpod\r\n    resources: {}\r\n  dnsPolicy: ClusterFirst\r\n  restartPolicy: Always\r\nstatus: {}\r\n\r\n[root@k8s \/]# kubectl run staticpod --image=nginx --dry-run=client -o yaml &gt; \/etc\/kubernetes\/manifests\/staticpod.yaml\r\n\r\n[root@k8s \/]# kubectl get pods\r\nNAME                         READY   STATUS    RESTARTS          AGE\r\napples-78656fd5db-4rpj7      1\/1     Running   0                 23h\r\napples-78656fd5db-qsm4x      1\/1     Running   0                 23h\r\napples-78656fd5db-t82tg      1\/1     Running   0                 23h\r\ndeploydaemon-zzllp           1\/1     Running   0                 2d\r\nfirstnginx-d8679d567-249g9   1\/1     Running   0                 3d1h\r\nfirstnginx-d8679d567-66c4s   1\/1     Running   0                 3d1h\r\nfirstnginx-d8679d567-72qbd   1\/1     Running   0                 3d1h\r\nfirstnginx-d8679d567-rhhlz   1\/1     Running   0                 2d8h\r\ninit-demo                    1\/1     Running   0                 2d10h\r\nlab4-pod                     1\/1     Running   0                 30h\r\nmorevol                      2\/2     Running   86 (45m ago)      43h\r\nmydaemon-d4dcd               1\/1     Running   0                 2d\r\nnewdep-749c9b5675-2x9mb      1\/1     Running   0                 23h\r\nnginxsvc-5f8b7d4f4d-dtrs7    1\/1     Running   0                 24h\r\npv-pod                       1\/1     Running   0                 42h\r\nsleepy                       1\/1     Running   48 (11m ago)      2d10h\r\nstaticpod-k8s.netico.pl      1\/1     Running   0                 29s\r\ntestpod                      1\/1     Running   0                 3d1h\r\ntwo-containers               2\/2     Running   282 (7m30s ago)   2d7h\r\nweb-0                        1\/1     Running   0                 2d13h\r\nweb-1                        1\/1     Running   0                 2d\r\nweb-2                        1\/1     Running   0                 2d\r\nwebserver-76d44586d-8gqhf    1\/1     Running   0                 31h\r\nwebshop-7f9fd49d4c-92nj2     1\/1     Running   0                 27h\r\nwebshop-7f9fd49d4c-kqllw     1\/1     Running   0                 27h\r\nwebshop-7f9fd49d4c-x2czc     1\/1     Running   0                 27h\r\n<\/pre>\n<p>That&#8217;s how it works. If you run a static pod the kubelet will automatically pick it up at the moment you create it.<\/p>\n<p><span style=\"color: #3366ff;\">Managing Node State<\/span><\/p>\n<ul>\n<li><code>kubectl cordon<\/code> is used to mark a node as unschedulable<\/li>\n<li><code>kubectl drain<\/code> is used to mark a node as unschedulable and<br \/>\nremove all running Pods from it<\/p>\n<ul>\n<li>Pods that have been started from a DaemonSet will not be<br \/>\nremoved while using <code>kubectl drain<\/code>, add<code> --ignore-daemonsets<\/code> to<br \/>\nignore that<\/li>\n<li>Add <code>--delete-emptydir-data<\/code> to delete data from emptyDir Pod volumes.<\/li>\n<\/ul>\n<\/li>\n<li>While using cordon or drain, a taint is set on the nodes<\/li>\n<li>Use <code>kubectl uncordon<\/code> to get the node back in a schedulable<br \/>\nstate<\/li>\n<\/ul>\n<p><span style=\"color: #3366ff;\">Managing Node State &#8211; Commands<\/span><\/p>\n<ul>\n<li><code>kubectl cordon worker2<\/code><\/li>\n<li><code>kubectl describe node worker2 # look for taints<\/code><\/li>\n<li><code>kubectl get nodes<\/code><\/li>\n<li><code>kubectl uncordon worker2<\/code><\/li>\n<\/ul>\n<pre class=\"lang:default mark:59 decode:true\">[root@k8s \/]# kubectl get nodes\r\nNAME            STATUS   ROLES           AGE    VERSION\r\nk8s.example.pl   Ready    control-plane   3d6h   v1.28.3\r\n\r\n[root@k8s \/]# kubectl get pods\r\nNAME                         READY   STATUS    RESTARTS         AGE\r\napples-78656fd5db-4rpj7      1\/1     Running   0                23h\r\napples-78656fd5db-qsm4x      1\/1     Running   0                23h\r\napples-78656fd5db-t82tg      1\/1     Running   0                23h\r\ndeploydaemon-zzllp           1\/1     Running   0                2d\r\nfirstnginx-d8679d567-249g9   1\/1     Running   0                3d1h\r\nfirstnginx-d8679d567-66c4s   1\/1     Running   0                3d1h\r\nfirstnginx-d8679d567-72qbd   1\/1     Running   0                3d1h\r\nfirstnginx-d8679d567-rhhlz   1\/1     Running   0                2d8h\r\ninit-demo                    1\/1     Running   0                2d10h\r\nlab4-pod                     1\/1     Running   0                31h\r\nmorevol                      2\/2     Running   88 (11m ago)     44h\r\nmydaemon-d4dcd               1\/1     Running   0                2d\r\nnewdep-749c9b5675-2x9mb      1\/1     Running   0                24h\r\nnginxsvc-5f8b7d4f4d-dtrs7    1\/1     Running   0                25h\r\npv-pod                       1\/1     Running   0                43h\r\nsleepy                       1\/1     Running   48 (38m ago)     2d11h\r\ntestpod                      1\/1     Running   0                3d1h\r\ntwo-containers               2\/2     Running   285 (4m7s ago)   2d8h\r\nweb-0                        1\/1     Running   0                2d13h\r\nweb-1                        1\/1     Running   0                2d\r\nweb-2                        1\/1     Running   0                2d\r\nwebserver-76d44586d-8gqhf    1\/1     Running   0                32h\r\nwebshop-7f9fd49d4c-92nj2     1\/1     Running   0                27h\r\nwebshop-7f9fd49d4c-kqllw     1\/1     Running   0                27h\r\nwebshop-7f9fd49d4c-x2czc     1\/1     Running   0                27h\r\n\r\n[root@k8s \/]# kubectl cordon k8s.example.pl\r\nnode\/k8s.example.pl cordoned\r\n\r\n[root@k8s \/]# kubectl get nodes\r\nNAME            STATUS                     ROLES           AGE    VERSION\r\nk8s.example.pl   Ready,SchedulingDisabled   control-plane   3d6h   v1.28.3\r\n\r\n[root@k8s \/]# kubectl describe node k8s.example.pl\r\nName:               k8s.example.pl\r\nRoles:              control-plane\r\nLabels:             beta.kubernetes.io\/arch=amd64\r\n                    beta.kubernetes.io\/os=linux\r\n                    kubernetes.io\/arch=amd64\r\n                    kubernetes.io\/hostname=k8s.example.pl\r\n                    kubernetes.io\/os=linux\r\n                    minikube.k8s.io\/commit=8220a6eb95f0a4d75f7f2d7b14cef975f050512d\r\n                    minikube.k8s.io\/name=minikube\r\n                    minikube.k8s.io\/primary=true\r\n                    minikube.k8s.io\/updated_at=2024_01_31T10_03_27_0700\r\n                    minikube.k8s.io\/version=v1.32.0\r\n                    node-role.kubernetes.io\/control-plane=\r\n                    node.kubernetes.io\/exclude-from-external-load-balancers=\r\nAnnotations:        kubeadm.alpha.kubernetes.io\/cri-socket: unix:\/\/\/var\/run\/cri-dockerd.sock\r\n                    node.alpha.kubernetes.io\/ttl: 0\r\n                    volumes.kubernetes.io\/controller-managed-attach-detach: true\r\nCreationTimestamp:  Wed, 31 Jan 2024 10:03:23 -0500\r\nTaints:             node.kubernetes.io\/unschedulable:NoSchedule\r\nUnschedulable:      true\r\nLease:\r\n  HolderIdentity:  k8s.example.pl\r\n  AcquireTime:     &lt;unset&gt;\r\n  RenewTime:       Sat, 03 Feb 2024 16:42:05 -0500\r\nConditions:\r\n  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message\r\n  ----             ------  -----------------                 ------------------                ------                       -------\r\n  MemoryPressure   False   Sat, 03 Feb 2024 16:37:14 -0500   Sat, 03 Feb 2024 13:59:11 -0500   KubeletHasSufficientMemory   kubelet has sufficient memory                                                                                     available\r\n  DiskPressure     False   Sat, 03 Feb 2024 16:37:14 -0500   Sat, 03 Feb 2024 13:59:11 -0500   KubeletHasNoDiskPressure     kubelet has no disk pressure\r\n  PIDPressure      False   Sat, 03 Feb 2024 16:37:14 -0500   Sat, 03 Feb 2024 13:59:11 -0500   KubeletHasSufficientPID      kubelet has sufficient PID av                                                                                    ailable\r\n  Ready            True    Sat, 03 Feb 2024 16:37:14 -0500   Sat, 03 Feb 2024 13:59:11 -0500   KubeletReady                 kubelet is posting ready stat                                                                                    us\r\nAddresses:\r\n  InternalIP:  172.30.9.24\r\n  Hostname:    k8s.example.pl\r\nCapacity:\r\n  cpu:                8\r\n  ephemeral-storage:  64177544Ki\r\n  hugepages-1Gi:      0\r\n  hugepages-2Mi:      0\r\n  memory:             16099960Ki\r\n  pods:               110\r\nAllocatable:\r\n  cpu:                8\r\n  ephemeral-storage:  64177544Ki\r\n  hugepages-1Gi:      0\r\n  hugepages-2Mi:      0\r\n  memory:             16099960Ki\r\n  pods:               110\r\nSystem Info:\r\n  Machine ID:                 0cc7c63085694b83adcd204eff748ff8\r\n  System UUID:                3e3ec47d-1fe1-b5b7-cbca-edd2da14db37\r\n  Boot ID:                    79a4e58f-5d2a-4f44-ad34-520bab9b01cc\r\n  Kernel Version:             4.18.0-500.el8.x86_64\r\n  OS Image:                   CentOS Stream 8\r\n  Operating System:           linux\r\n  Architecture:               amd64\r\n  Container Runtime Version:  docker:\/\/25.0.1\r\n  Kubelet Version:            v1.28.3\r\n  Kube-Proxy Version:         v1.28.3\r\nPodCIDR:                      10.244.0.0\/24\r\nPodCIDRs:                     10.244.0.0\/24\r\nNon-terminated Pods:          (35 in total)\r\n  Namespace                   Name                                          CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age\r\n  ---------                   ----                                          ------------  ----------  ---------------  -------------  ---\r\n  default                     apples-78656fd5db-4rpj7                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         23h\r\n  default                     apples-78656fd5db-qsm4x                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         23h\r\n  default                     apples-78656fd5db-t82tg                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         23h\r\n  default                     deploydaemon-zzllp                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d\r\n  default                     firstnginx-d8679d567-249g9                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d1h\r\n  default                     firstnginx-d8679d567-66c4s                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d1h\r\n  default                     firstnginx-d8679d567-72qbd                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d1h\r\n  default                     firstnginx-d8679d567-rhhlz                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d8h\r\n  default                     init-demo                                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d10h\r\n  default                     lab4-pod                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         31h\r\n  default                     morevol                                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         44h\r\n  default                     mydaemon-d4dcd                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d\r\n  default                     newdep-749c9b5675-2x9mb                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         24h\r\n  default                     nginxsvc-5f8b7d4f4d-dtrs7                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         25h\r\n  default                     pv-pod                                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         43h\r\n  default                     sleepy                                        0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d11h\r\n  default                     testpod                                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d1h\r\n  default                     two-containers                                0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d8h\r\n  default                     web-0                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d13h\r\n  default                     web-1                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d\r\n  default                     web-2                                         0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d\r\n  default                     webserver-76d44586d-8gqhf                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         32h\r\n  default                     webshop-7f9fd49d4c-92nj2                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         27h\r\n  default                     webshop-7f9fd49d4c-kqllw                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         27h\r\n  default                     webshop-7f9fd49d4c-x2czc                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         27h\r\n  ingress-nginx               ingress-nginx-controller-6858749594-27tm9     100m (1%)     0 (0%)      90Mi (0%)        0 (0%)         25h\r\n  kube-system                 coredns-5dd5756b68-sgfkj                      100m (1%)     0 (0%)      70Mi (0%)        170Mi (1%)     3d6h\r\n  kube-system                 etcd-k8s.example.pl                            100m (1%)     0 (0%)      100Mi (0%)       0 (0%)         3d6h\r\n  kube-system                 kube-apiserver-k8s.example.pl                  250m (3%)     0 (0%)      0 (0%)           0 (0%)         3d6h\r\n  kube-system                 kube-controller-manager-k8s.example.pl         200m (2%)     0 (0%)      0 (0%)           0 (0%)         3d6h\r\n  kube-system                 kube-proxy-5nmms                              0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d6h\r\n  kube-system                 kube-scheduler-k8s.example.pl                  100m (1%)     0 (0%)      0 (0%)           0 (0%)         3d6h\r\n  kube-system                 storage-provisioner                           0 (0%)        0 (0%)      0 (0%)           0 (0%)         3d6h\r\n  kubernetes-dashboard        dashboard-metrics-scraper-7fd5cb4ddc-9ld5n    0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d1h\r\n  kubernetes-dashboard        kubernetes-dashboard-8694d4445c-xjlsr         0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d1h\r\nAllocated resources:\r\n  (Total limits may be over 100 percent, i.e., overcommitted.)\r\n  Resource           Requests    Limits\r\n  --------           --------    ------\r\n  cpu                850m (10%)  0 (0%)\r\n  memory             260Mi (1%)  170Mi (1%)\r\n  ephemeral-storage  0 (0%)      0 (0%)\r\n  hugepages-1Gi      0 (0%)      0 (0%)\r\n  hugepages-2Mi      0 (0%)      0 (0%)\r\nEvents:\r\n  Type    Reason              Age   From     Message\r\n  ----    ------              ----  ----     -------\r\n  Normal  NodeNotSchedulable  27s   kubelet  Node k8s.example.pl status is now: NodeNotSchedulable\r\n\r\n[root@k8s \/]# kubectl uncordon k8s.example.pl\r\nnode\/k8s.example.pl uncordoned\r\n\r\n[root@k8s \/]# kubectl get nodes\r\nNAME            STATUS   ROLES           AGE    VERSION\r\nk8s.example.pl   Ready    control-plane   3d6h   v1.28.3\r\n<\/pre>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Managing Node Services<\/span><\/p>\n<ul>\n<li>The container runtime (often containerd) and kubelet are managed by the<br \/>\nLinux systemd service manager<\/li>\n<li>Use <code>systemctl status kubelet<\/code> to check the current status of the kubelet<\/li>\n<li>To manually start it, use <code>sudo systemctl start kubelet<\/code><\/li>\n<li>Notice that Pods that are scheduled on a node show as container processes in <code>ps aux<\/code> output. Don&#8217;t use Linux tools to manage Pods !<\/li>\n<\/ul>\n<p><span style=\"color: #3366ff;\">Managing Node Services Commands<\/span><\/p>\n<ul>\n<li><code>ps aux | grep kubelet<\/code><\/li>\n<li><code>ps aux | grep containerd<\/code><\/li>\n<li><code>systemctl status kubelet<\/code><\/li>\n<li><code>sudo systemctl stop kubelet<\/code><\/li>\n<li><code>sudo systemctl start kubelet<\/code><\/li>\n<\/ul>\n<pre class=\"lang:default mark:36 decode:true\">[root@k8s \/]# ps aux | grep kubelet\r\nroot       15161  4.2  1.9 1042036 311412 ?      Ssl  lut01 127:51 kube-apiserver --advertise-address=172.30.9.24 --allow-privileged=true --authorization                                                                                    -mode=Node,RBAC --client-ca-file=\/var\/lib\/minikube\/certs\/ca.crt --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageCl                                                                                    ass,DefaultTolerationSeconds,NodeRestriction,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota --enable-bootstrap-token-auth=true --etcd-                                                                                    cafile=\/var\/lib\/minikube\/certs\/etcd\/ca.crt --etcd-certfile=\/var\/lib\/minikube\/certs\/apiserver-etcd-client.crt --etcd-keyfile=\/var\/lib\/minikube\/certs\/apise                                                                                    rver-etcd-client.key --etcd-servers=https:\/\/127.0.0.1:2379 --kubelet-client-certificate=\/var\/lib\/minikube\/certs\/apiserver-kubelet-client.crt --kubelet-cl                                                                                    ient-key=\/var\/lib\/minikube\/certs\/apiserver-kubelet-client.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --proxy-client-cert-file=\/                                                                                    var\/lib\/minikube\/certs\/front-proxy-client.crt --proxy-client-key-file=\/var\/lib\/minikube\/certs\/front-proxy-client.key --requestheader-allowed-names=front-                                                                                    proxy-client --requestheader-client-ca-file=\/var\/lib\/minikube\/certs\/front-proxy-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requesthead                                                                                    er-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=8443 --service-account-issuer=https:\/\/kubernetes.default.svc                                                                                    .cluster.local --service-account-key-file=\/var\/lib\/minikube\/certs\/sa.pub --service-account-signing-key-file=\/var\/lib\/minikube\/certs\/sa.key --service-clus                                                                                    ter-ip-range=10.96.0.0\/12 --tls-cert-file=\/var\/lib\/minikube\/certs\/apiserver.crt --tls-private-key-file=\/var\/lib\/minikube\/certs\/apiserver.key\r\nroot      555436  4.4  0.8 1938596 133496 ?      Ssl  13:59   8:08 \/var\/lib\/minikube\/binaries\/v1.28.3\/kubelet --bootstrap-kubeconfig=\/etc\/kubernetes\/boot                                                                                    strap-kubelet.conf --config=\/var\/lib\/kubelet\/config.yaml --container-runtime-endpoint=unix:\/\/\/var\/run\/cri-dockerd.sock --hostname-override=k8s.example.pl                                                                                     --kubeconfig=\/etc\/kubernetes\/kubelet.conf --node-ip=172.30.9.24\r\nroot      591406  0.0  0.0  12216  1168 pts\/0    S+   17:01   0:00 grep --color=auto kubelet\r\n\r\n[root@k8s \/]# ps aux | grep containerd\r\nroot       14187  4.3  0.9 3733916 153944 ?      Ssl  lut01 131:09 \/usr\/bin\/dockerd -H fd:\/\/ --containerd=\/run\/containerd\/containerd.sock\r\nroot       14899  0.0  0.1 720064 17276 ?        Sl   lut01   0:09 \/usr\/bin\/containerd-shim-runc-v2 -namespace moby -id 437b912e567ef421738d84dfea73f07f9                                                                                    854e00f9b5c142b5506db6d5ed580d1 -address \/run\/containerd\/containerd.sock\r\nroot       14900  0.0  0.0 720064 15620 ?        Sl   lut01   0:09 \/usr\/bin\/containerd-shim-runc-v2 -namespace moby -id cadb21950a7944a7fccd11cdd28bfc0b2                                                                                    43338638f970e83055f0f3cc0d4f104 -address \/run\/containerd\/containerd.sock\r\nroot       14915  0.0  0.0 720384 15992 ?        Sl   lut01   0:08 \/usr\/bin\/containerd-shim-runc-v2 -namespace moby -id 6e075c204c3c87f7f8340dccd39d5426a                                                                                    c3038dbbf5f54913d4919009e70834c -address \/run\/containerd\/containerd.sock\r\n...\r\n\r\n[root@k8s \/]# systemctl status kubelet\r\n\u25cf kubelet.service - kubelet: The Kubernetes Node Agent\r\n   Loaded: loaded (\/usr\/lib\/systemd\/system\/kubelet.service; disabled; vendor preset: disabled)\r\n  Drop-In: \/etc\/systemd\/system\/kubelet.service.d\r\n           \u2514\u250010-kubeadm.conf\r\n   Active: active (running) since Sat 2024-02-03 13:59:11 EST; 3h 3min ago\r\n     Docs: http:\/\/kubernetes.io\/docs\/\r\n Main PID: 555436 (kubelet)\r\n    Tasks: 16 (limit: 100376)\r\n   Memory: 70.8M\r\n   CGroup: \/system.slice\/kubelet.service\r\n           \u2514\u2500555436 \/var\/lib\/minikube\/binaries\/v1.28.3\/kubelet --bootstrap-kubeconfig=\/etc\/kubernetes\/bootstrap-kubelet.conf --config=\/var\/lib\/kubelet\/c&gt;\r\n\r\nlut 03 17:02:46 k8s.example.pl kubelet[555436]: E0203 17:02:46.407467  555436 desired_state_of_world_populator.go:320] \"Error processing volume\" err=\"err&gt;\r\n...\r\n[root@k8s \/]# systemctl cat kubelet\r\n# \/usr\/lib\/systemd\/system\/kubelet.service\r\n[Unit]\r\nDescription=kubelet: The Kubernetes Node Agent\r\nDocumentation=http:\/\/kubernetes.io\/docs\/\r\nStartLimitIntervalSec=0\r\n\r\n[Service]\r\nExecStart=\/var\/lib\/minikube\/binaries\/v1.28.3\/kubelet\r\nRestart=always\r\n# Tuned for local dev: faster than upstream default (10s), but slower than systemd default (100ms)\r\nRestartSec=600ms\r\n\r\n[Install]\r\nWantedBy=multi-user.target\r\n\r\n# \/etc\/systemd\/system\/kubelet.service.d\/10-kubeadm.conf\r\n[Unit]\r\nWants=docker.socket\r\n\r\n[Service]\r\nExecStart=\r\nExecStart=\/var\/lib\/minikube\/binaries\/v1.28.3\/kubelet --bootstrap-kubeconfig=\/etc\/kubernetes\/bootstrap-kubelet.conf --config=\/var\/lib\/kubelet\/config.yaml&gt;\r\n\r\n\r\n[root@k8s \/]# systemctl status kubelet\r\n\u25cf kubelet.service - kubelet: The Kubernetes Node Agent\r\n   Loaded: loaded (\/usr\/lib\/systemd\/system\/kubelet.service; disabled; vendor preset: disabled)\r\n  Drop-In: \/etc\/systemd\/system\/kubelet.service.d\r\n           \u2514\u250010-kubeadm.conf\r\n   Active: active (running) since Sat 2024-02-03 13:59:11 EST; 3h 7min ago\r\n     Docs: http:\/\/kubernetes.io\/docs\/\r\n Main PID: 555436 (kubelet)\r\n    Tasks: 16 (limit: 100376)\r\n   Memory: 70.4M\r\n   CGroup: \/system.slice\/kubelet.service\r\n           \u2514\u2500555436 \/var\/lib\/minikube\/binaries\/v1.28.3\/kubelet --bootstrap-kubeconfig=\/etc\/kubernetes\/bootstrap-kubelet.conf --config=\/var\/lib\/kubelet\/c&gt;\r\n\r\nlut 03 17:06:57 k8s.example.pl kubelet[555436]: W0203 17:06:57.828054  555436 reflector.go:458] vendor\/k8s.io\/client-go\/informers\/factory.go:150: watch o&gt;\r\nlut 03 17:06:58 k8s.example.pl kubelet[555436]: E0203 17:06:58.406624  555436 event.go:289] Unable to write event: '&amp;v1.Event{TypeMeta:v1.TypeMeta{Kind:\"&gt;\r\nlut 03 17:06:58 k8s.example.pl kubelet[555436]: E0203 17:06:58.449716  555436 desired_state_of_world_populator.go:320] \"Error processing volume\" err=\"err&gt;\r\nlut 03 17:06:58 k8s.example.pl kubelet[555436]: I0203 17:06:58.603442  555436 scope.go:117] \"RemoveContainer\" containerID=\"14c60aafe505b57dcec7dbcf2d50dd&gt;\r\nlut 03 17:06:58 k8s.example.pl kubelet[555436]: I0203 17:06:58.603939  555436 scope.go:117] \"RemoveContainer\" containerID=\"c739ddcefc6ada2c434f3e403abefb&gt;\r\nlut 03 17:06:58 k8s.example.pl kubelet[555436]: I0203 17:06:58.604584  555436 status_manager.go:853] \"Failed to get status for pod\" podUID=\"ea109a012b2a2&gt;\r\nlut 03 17:06:58 k8s.example.pl kubelet[555436]: E0203 17:06:58.687274  555436 kubelet.go:1907] \"Unable to attach or mount volumes for pod; skipping pod\" &gt;\r\nlut 03 17:06:58 k8s.example.pl kubelet[555436]: E0203 17:06:58.687341  555436 pod_workers.go:1300] \"Error syncing pod, skipping\" err=\"unmounted volumes=[&gt;\r\nlut 03 17:06:58 k8s.example.pl kubelet[555436]: W0203 17:06:58.745783  555436 reflector.go:535] vendor\/k8s.io\/client-go\/informers\/factory.go:150: failed &gt;\r\nlut 03 17:06:58 k8s.example.pl kubelet[555436]: E0203 17:06:58.745856  555436 reflector.go:147] vendor\/k8s.io\/client-go\/informers\/factory.go:150: Failed &gt;\r\n\r\n[root@k8s \/]# kill -9 15161\r\n\r\n[root@k8s \/]# systemctl status kubelet\r\n\u25cf kubelet.service - kubelet: The Kubernetes Node Agent\r\n   Loaded: loaded (\/usr\/lib\/systemd\/system\/kubelet.service; disabled; vendor preset: disabled)\r\n  Drop-In: \/etc\/systemd\/system\/kubelet.service.d\r\n           \u2514\u250010-kubeadm.conf\r\n   Active: active (running) since Sat 2024-02-03 13:59:11 EST; 3h 8min ago\r\n     Docs: http:\/\/kubernetes.io\/docs\/\r\n Main PID: 555436 (kubelet)\r\n    Tasks: 16 (limit: 100376)\r\n   Memory: 71.0M\r\n   CGroup: \/system.slice\/kubelet.service\r\n           \u2514\u2500555436 \/var\/lib\/minikube\/binaries\/v1.28.3\/kubelet --bootstrap-kubeconfig=\/etc\/kubernetes\/bootstrap-kubelet.conf --config=\/var\/lib\/kubelet\/c&gt;\r\n\r\nlut 03 17:07:00 k8s.example.pl kubelet[555436]: E0203 17:07:00.923579  555436 reflector.go:147] object-\"kube-system\"\/\"kube-root-ca.crt\": Failed to watch &gt;\r\nlut 03 17:07:01 k8s.example.pl kubelet[555436]: E0203 17:07:01.439804  555436 desired_state_of_world_populator.go:320] \"Error processing volume\" err=\"err&gt;\r\nlut 03 17:07:01 k8s.example.pl kubelet[555436]: E0203 17:07:01.688453  555436 kubelet.go:1907] \"Unable to attach or mount volumes for pod; skipping pod\" &gt;\r\nlut 03 17:07:01 k8s.example.pl kubelet[555436]: E0203 17:07:01.688506  555436 pod_workers.go:1300] \"Error syncing pod, skipping\" err=\"unmounted volumes=[&gt;\r\nlut 03 17:07:02 k8s.example.pl kubelet[555436]: I0203 17:07:02.386672  555436 scope.go:117] \"RemoveContainer\" containerID=\"e391b0bd09e900fcb6aa768fa23d0b&gt;\r\nlut 03 17:07:02 k8s.example.pl kubelet[555436]: E0203 17:07:02.387187  555436 pod_workers.go:1300] \"Error syncing pod, skipping\" err=\"failed to \\\"StartCo&gt;\r\nlut 03 17:07:08 k8s.example.pl kubelet[555436]: E0203 17:07:08.490599  555436 desired_state_of_world_populator.go:320] \"Error processing volume\" err=\"err&gt;\r\nlut 03 17:07:08 k8s.example.pl kubelet[555436]: E0203 17:07:08.687894  555436 kubelet.go:1907] \"Unable to attach or mount volumes for pod; skipping pod\" &gt;\r\nlut 03 17:07:08 k8s.example.pl kubelet[555436]: E0203 17:07:08.687958  555436 pod_workers.go:1300] \"Error syncing pod, skipping\" err=\"unmounted volumes=[&gt;\r\nlut 03 17:07:11 k8s.example.pl kubelet[555436]: E0203 17:07:11.331441  555436 file.go:187] \"Could not process manifest file\" err=\"\/etc\/kubernetes\/manifes&gt;\r\n<\/pre>\n<p>As we see even if we kill the kubelet process the systemt will pick it up. But if we would down the service by systemctl systemd won&#8217;t pick the kubelet process up. It only works for distaster not for intended operations.<\/p>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Lab: Managing Static Pods<\/span><\/p>\n<ul>\n<li>On node <em>worker1<\/em>, run a static Pod with the name mypod, using an Nginx<br \/>\nimage and no further configuration<\/li>\n<li>Use the appropriate tools to verify that the static Pod has started successfully<\/li>\n<\/ul>\n<pre class=\"lang:default decode:true\">[root@k8s ~]# kubectl run static --image=nginx --dry-run=client -o yaml\r\napiVersion: v1\r\nkind: Pod\r\nmetadata:\r\n  creationTimestamp: null\r\n  labels:\r\n    run: static\r\n  name: static\r\nspec:\r\n  containers:\r\n  - image: nginx\r\n    name: static\r\n    resources: {}\r\n  dnsPolicy: ClusterFirst\r\n  restartPolicy: Always\r\nstatus: {}\r\n\r\n[root@k8s ~]# ssh root@worker2\r\nssh: Could not resolve hostname worker2: Name or service not known\r\n\r\n[root@k8s ~]# cd \/etc\/kubernetes\/manifests\r\n\r\n[root@k8s manifests]# vi mystaticpod.yaml\r\n\r\n[root@k8s manifests]# cat mystaticpod.yaml\r\napiVersion: v1\r\nkind: Pod\r\nmetadata:\r\n  creationTimestamp: null\r\n  labels:\r\n    run: static\r\n  name: mystaticpod\r\nspec:\r\n  containers:\r\n  - image: nginx\r\n    name: mystaticpod\r\n    resources: {}\r\n  dnsPolicy: ClusterFirst\r\n  restartPolicy: Always\r\nstatus: {}\r\n\r\n[root@k8s manifests]# crictl pods\r\nPOD ID              CREATED              STATE               NAME                                         NAMESPACE              ATTEMPT             RUNTIME\r\n51c54def1614c       About a minute ago   Ready               mystaticpod-k8s.example.pl                    default                0                   (default)\r\n93f7305ee0d77       37 hours ago         Ready               apples-78656fd5db-4rpj7                      default                0                   (default)\r\n9793a44f77d5a       37 hours ago         Ready               apples-78656fd5db-qsm4x                      default                0                   (default)\r\n...\r\n\r\n[root@k8s manifests]# kubectl get pods\r\nNAME                         READY   STATUS    RESTARTS          AGE\r\napples-78656fd5db-4rpj7      1\/1     Running   0                 36h\r\napples-78656fd5db-qsm4x      1\/1     Running   0                 36h\r\napples-78656fd5db-t82tg      1\/1     Running   0                 36h\r\ndeploydaemon-zzllp           1\/1     Running   0                 2d13h\r\nfirstnginx-d8679d567-249g9   1\/1     Running   0                 3d14h\r\nfirstnginx-d8679d567-66c4s   1\/1     Running   0                 3d14h\r\nfirstnginx-d8679d567-72qbd   1\/1     Running   0                 3d14h\r\nfirstnginx-d8679d567-rhhlz   1\/1     Running   0                 2d21h\r\ninit-demo                    1\/1     Running   0                 2d23h\r\nlab4-pod                     1\/1     Running   0                 44h\r\nmorevol                      2\/2     Running   114 (7m50s ago)   2d9h\r\nmydaemon-d4dcd               1\/1     Running   0                 2d13h\r\nmystaticpod-k8s.example.pl    1\/1     Running   0                 97s\r\nnewdep-749c9b5675-2x9mb      1\/1     Running   0                 37h\r\n...\r\n<\/pre>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>.<\/p>\n","protected":false},"author":2,"featured_media":5955,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[99],"tags":[],"_links":{"self":[{"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/posts\/5295"}],"collection":[{"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/comments?post=5295"}],"version-history":[{"count":28,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/posts\/5295\/revisions"}],"predecessor-version":[{"id":5472,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/posts\/5295\/revisions\/5472"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/media\/5955"}],"wp:attachment":[{"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/media?parent=5295"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/categories?post=5295"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/tags?post=5295"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}