{"id":5015,"date":"2023-08-26T13:44:57","date_gmt":"2023-08-26T11:44:57","guid":{"rendered":"http:\/\/miro.borodziuk.eu\/?p=5015"},"modified":"2025-05-19T15:28:11","modified_gmt":"2025-05-19T13:28:11","slug":"managing-openshift-clusters","status":"publish","type":"post","link":"http:\/\/miro.borodziuk.eu\/index.php\/2023\/08\/26\/managing-openshift-clusters\/","title":{"rendered":"Managing Openshift Clusters"},"content":{"rendered":"<p><!--more--><\/p>\n<p><span style=\"color: #3366ff;\">Cluster Troubleshooting<\/span><\/p>\n<ul>\n<li>An OpenShift cluster has two focal areas for troubleshooting<\/li>\n<li>OpenShift operators are cluster applications that can be monitored and fixed<br \/>\nlike any other application that runs in OpenShift<\/li>\n<li>OpenShift nodes can be monitored individually<\/li>\n<li>Other problems may come from version mismatches<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Verifying Node Health<\/span><\/p>\n<ul>\n<li><code>oc get nodes<\/code> is a good first step to investigate current health of nodes\n<ul>\n<li>Anything other than Ready means that the node is dead to the control plane<\/li>\n<\/ul>\n<\/li>\n<li><code>oc adm top nodes<\/code> shows current node health, based on statistics gathered by the metrics server<\/li>\n<li><code>oc describe node<\/code> may be used to investigate recent events and resource usage\n<ul>\n<li><code>Events<\/code> shows an event log<\/li>\n<li><code>Allocated resources<\/code> gives an overview of allocated resources and requests<\/li>\n<li><code> Capacity<\/code> shows available capacity<\/li>\n<li><code>Non-terminated Pods<\/code> shows Pods currently being used<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<pre class=\"lang:default decode:true \">$ oc whoami\r\nsystem:admin\r\n\r\n$ oc get nodes\r\nNAME        STATUS    ROLES     AGE       VERSION\r\nlocalhost   Ready     &lt;none&gt;    7d        v1.11.0+d4cacc0\r\n\r\n$ oc adm top nodes\r\nError from server (NotFound): the server could not find the requested resource (get services https:heapster:)\r\n\r\n$ oc describe node | less\r\n\r\n$ oc describe node\r\nName:               localhost\r\nRoles:              &lt;none&gt;\r\nLabels:             beta.kubernetes.io\/arch=amd64\r\n                    beta.kubernetes.io\/os=linux\r\n                    disktype=nvme\r\n                    kubernetes.io\/hostname=localhost\r\nAnnotations:        volumes.kubernetes.io\/controller-managed-attach-detach=true\r\nCreationTimestamp:  Sat, 22 Jul 2023 20:54:32 +0200\r\nTaints:             &lt;none&gt;\r\nUnschedulable:      false\r\nConditions:\r\n  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message\r\n  ----             ------  -----------------                 ------------------                ------                       -------\r\n  OutOfDisk        False   Sun, 30 Jul 2023 15:33:30 +0200   Sat, 22 Jul 2023 20:54:22 +0200   KubeletHasSufficientDisk     kubelet has sufficient disk space available\r\n  MemoryPressure   False   Sun, 30 Jul 2023 15:33:30 +0200   Sat, 22 Jul 2023 20:54:22 +0200   KubeletHasSufficientMemory   kubelet has sufficient memory available\r\n  DiskPressure     False   Sun, 30 Jul 2023 15:33:30 +0200   Sat, 22 Jul 2023 20:54:22 +0200   KubeletHasNoDiskPressure     kubelet has no disk pressure\r\n  PIDPressure      False   Sun, 30 Jul 2023 15:33:30 +0200   Sat, 22 Jul 2023 20:54:22 +0200   KubeletHasSufficientPID      kubelet has sufficient PID available\r\n  Ready            True    Sun, 30 Jul 2023 15:33:30 +0200   Sat, 22 Jul 2023 20:54:22 +0200   KubeletReady                 kubelet is posting ready status\r\nAddresses:\r\n  InternalIP:  172.30.9.22\r\n  Hostname:    localhost\r\nCapacity:\r\n cpu:            8\r\n hugepages-1Gi:  0\r\n hugepages-2Mi:  0\r\n memory:         7981844Ki\r\n pods:           250\r\nAllocatable:\r\n cpu:            8\r\n hugepages-1Gi:  0\r\n hugepages-2Mi:  0\r\n memory:         7879444Ki\r\n pods:           250\r\nSystem Info:\r\n Machine ID:                     a37388a4746444f1b3f079f777748845\r\n System UUID:                    6099DE02-9EA8-C210-7553-A7697F2C302A\r\n Boot ID:                        7c076895-85e9-45ce-ae2c-8bbe7127be73\r\n Kernel Version:                 3.10.0-1160.92.1.el7.x86_64\r\n OS Image:                       CentOS Linux 7 (Core)\r\n Operating System:               linux\r\n Architecture:                   amd64\r\n Container Runtime Version:      docker:\/\/24.0.3\r\n Kubelet Version:                v1.11.0+d4cacc0\r\n Kube-Proxy Version:             v1.11.0+d4cacc0\r\nNon-terminated Pods:             (45 in total)\r\n  Namespace                      Name                                                       CPU Requests  CPU Limits  Memory Requests  Memory Limits\r\n  ---------                      ----                                                       ------------  ----------  ---------------  -------------\r\n  debug                          dnginx-88c7766dd-hlbtd                                     0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  default                        bitginx-1-jzk9r                                            0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  default                        busybox                                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  default                        docker-registry-1-ctgff                                    100m (1%)     0 (0%)      256Mi (3%)       0 (0%)\r\n  default                        lab4pod                                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  default                        linginx1-dc9f65f54-6zw8j                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  default                        linginx2-69bf6fc66b-mv6wx                                  0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  default                        nginx                                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  default                        nginx-cm                                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  default                        pv-pod                                                     0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  default                        router-1-k8zgt                                             100m (1%)     0 (0%)      256Mi (3%)       0 (0%)\r\n  default                        test1                                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  kube-dns                       kube-dns-t727w                                             0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  kube-proxy                     kube-proxy-cr7kh                                           0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  kube-system                    kube-controller-manager-localhost                          0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  kube-system                    kube-scheduler-localhost                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  kube-system                    master-api-localhost                                       0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  kube-system                    master-etcd-localhost                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  limits                         nee-597889d8c7-p6tc2                                       0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  love                           anti1                                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  love                           newpod-1-qgmpj                                             0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  love                           runonssd                                                   0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  network-security               nginxlab-1-bcgkt                                           0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  nodesel                        simple-6f55965d79-mklpc                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  nodesel                        simple-6f55965d79-q8pq9                                    0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  openshift-apiserver            openshift-apiserver-thwpd                                  0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  openshift-controller-manager   openshift-controller-manager-c9ms5                         0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  openshift-core-operators       openshift-service-cert-signer-operator-6d477f986b-jzcgw    0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  openshift-core-operators       openshift-web-console-operator-664b974ff5-px7gw            0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  openshift-service-cert-signer  apiservice-cabundle-injector-8ffbbb6dc-x9l4r               0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  openshift-service-cert-signer  service-serving-cert-signer-668c45d5f-lxvff                0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  openshift-web-console          webconsole-78f59b4bfb-qqv4p                                100m (1%)     0 (0%)      100Mi (1%)       0 (0%)\r\n  quota-test                     bitginx-84b698ff5c-h5j57                                   10m (0%)      50m (0%)    5Mi (0%)         20Mi (0%)\r\n  quota-test                     bitginx-84b698ff5c-qxwd5                                   10m (0%)      50m (0%)    5Mi (0%)         20Mi (0%)\r\n  quota-test                     bitginx-84b698ff5c-xfnp4                                   10m (0%)      50m (0%)    5Mi (0%)         20Mi (0%)\r\n  source-project                 nginx-access                                               0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  source-project                 nginx-noaccess                                             0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  target-project                 nginx-target-1-9kdn6                                       0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  template-project               anti1                                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  test-project                   nginxmany-5859c9dbb6-5xxr6                                 0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  test-project                   nginxmany-5859c9dbb6-6ljwm                                 0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  test-project                   nginxmany-5859c9dbb6-9lv6c                                 0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  test-project                   nginxmany-5859c9dbb6-dgr7k                                 0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  test-project                   nginxmany-5859c9dbb6-dl8wr                                 0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\n  test-project                   nginxmany-5859c9dbb6-hk2sm                                 0 (0%)        0 (0%)      0 (0%)           0 (0%)\r\nAllocated resources:\r\n  (Total limits may be over 100 percent, i.e., overcommitted.)\r\n  Resource  Requests    Limits\r\n  --------  --------    ------\r\n  cpu       330m (4%)   150m (1%)\r\n  memory    627Mi (8%)  60Mi (0%)\r\nEvents:     &lt;none&gt;\r\n<\/pre>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Monitoring Operators<\/span><\/p>\n<ul>\n<li>Operators are the programs that are responsible for starting the different components running in the cluster<\/li>\n<li>These components are started by operators as Daemonsets or Deployments<\/li>\n<li><code> oc get clusteroperators<\/code> shows the current status of operators\n<ul>\n<li>If an operator is in progressing state, it is currently being updated<\/li>\n<li>If in degraded state, something is wrong and further investigation is required<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Analyzing Operators<\/span><\/p>\n<ul>\n<li><code>ClusterOperator<\/code> resources are non-namespaced<\/li>\n<li>Each operator starts its resources in dedicated namespaces\n<ul>\n<li>Some operators use one namespace, some operators use more<\/li>\n<\/ul>\n<\/li>\n<li>If an operator shows a degraded status in <code>oc get co<\/code>, investigate resources running in its namespace and use common tools to check their status<\/li>\n<\/ul>\n<pre class=\"lang:default decode:true\">$ oc get co\r\nNAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE\r\nauthentication                             4.13.1    True        False         False      3d22h   \r\nbaremetal                                  4.13.1    True        False         False      263d    \r\ncloud-controller-manager                   4.13.1    True        False         False      263d    \r\ncloud-credential                           4.13.1    True        False         False      263d    \r\ncluster-autoscaler                         4.13.1    True        False         False      263d    \r\nconfig-operator                            4.13.1    True        False         False      263d    \r\nconsole                                    4.13.1    True        False         False      84d     \r\ncontrol-plane-machine-set                  4.13.1    True        False         False      190d    \r\ncsi-snapshot-controller                    4.13.1    True        False         False      263d    \r\ndns                                        4.13.1    True        False         False      133d    \r\netcd                                       4.13.1    True        False         False      133d    \r\nimage-registry                             4.13.1    True        False         False      190d    \r\ningress                                    4.13.1    True        False         False      59d     \r\ninsights                                   4.13.1    True        False         False      133d    \r\nkube-apiserver                             4.13.1    True        False         False      263d    \r\nkube-controller-manager                    4.13.1    True        False         False      263d    \r\nkube-scheduler                             4.13.1    True        False         False      263d    \r\nkube-storage-version-migrator              4.13.1    True        False         False      59d     \r\nmachine-api                                4.13.1    True        False         False      263d    \r\nmachine-approver                           4.13.1    True        False         False      263d    \r\nmachine-config                             4.13.1    True        False         False      41h     \r\nmarketplace                                4.13.1    True        False         False      263d    \r\nmonitoring                                 4.13.1    True        False         False      133d    \r\nnetwork                                    4.13.1    True        False         False      263d    \r\nnode-tuning                                4.13.1    True        False         False      59d     \r\nopenshift-apiserver                        4.13.1    True        False         False      10d     \r\nopenshift-controller-manager               4.13.1    True        False         False      190d    \r\nopenshift-samples                          4.13.1    True        False         False      59d     \r\noperator-lifecycle-manager                 4.13.1    True        False         False      263d    \r\noperator-lifecycle-manager-catalog         4.13.1    True        False         False      263d    \r\noperator-lifecycle-manager-packageserver   4.13.1    True        False         False      131d    \r\nservice-ca                                 4.13.1    True        False         False      263d    \r\nstorage                                    4.13.1    True        False         False      263d   \r\n\r\n\r\n$ oc get ns | grep monitoring\r\nError from server (Forbidden): namespaces is forbidden: User \"makarewicz-openshift\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope\r\n\r\n$ oc get ns                  \r\nError from server (Forbidden): namespaces is forbidden: User \"makarewicz-openshift\" cannot list resource \"namespaces\" in API group \"\" at the cluster scope\r\n\r\n$ oc get all -n openshift-monitoring\r\nError from server (Forbidden): pods is forbidden: User \"makarewicz-openshift\" cannot list resource \"pods\" in API group \"\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): replicationcontrollers is forbidden: User \"makarewicz-openshift\" cannot list resource \"replicationcontrollers\" in API group \"\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): services is forbidden: User \"makarewicz-openshift\" cannot list resource \"services\" in API group \"\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): daemonsets.apps is forbidden: User \"makarewicz-openshift\" cannot list resource \"daemonsets\" in API group \"apps\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): deployments.apps is forbidden: User \"makarewicz-openshift\" cannot list resource \"deployments\" in API group \"apps\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): replicasets.apps is forbidden: User \"makarewicz-openshift\" cannot list resource \"replicasets\" in API group \"apps\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): statefulsets.apps is forbidden: User \"makarewicz-openshift\" cannot list resource \"statefulsets\" in API group \"apps\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User \"makarewicz-openshift\" cannot list resource \"horizontalpodautoscalers\" in API group \"autoscaling\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): cronjobs.batch is forbidden: User \"makarewicz-openshift\" cannot list resource \"cronjobs\" in API group \"batch\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): jobs.batch is forbidden: User \"makarewicz-openshift\" cannot list resource \"jobs\" in API group \"batch\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): deploymentconfigs.apps.openshift.io is forbidden: User \"makarewicz-openshift\" cannot list resource \"deploymentconfigs\" in API group \"apps.openshift.io\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): buildconfigs.build.openshift.io is forbidden: User \"makarewicz-openshift\" cannot list resource \"buildconfigs\" in API group \"build.openshift.io\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): builds.build.openshift.io is forbidden: User \"makarewicz-openshift\" cannot list resource \"builds\" in API group \"build.openshift.io\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): imagestreams.image.openshift.io is forbidden: User \"makarewicz-openshift\" cannot list resource \"imagestreams\" in API group \"image.openshift.io\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): routes.route.openshift.io is forbidden: User \"makarewicz-openshift\" cannot list resource \"routes\" in API group \"route.openshift.io\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): brokers.eventing.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"brokers\" in API group \"eventing.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): triggers.eventing.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"triggers\" in API group \"eventing.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): eventtypes.eventing.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"eventtypes\" in API group \"eventing.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): kafkasinks.eventing.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"kafkasinks\" in API group \"eventing.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): sequences.flows.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"sequences\" in API group \"flows.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): parallels.flows.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"parallels\" in API group \"flows.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): subscriptions.messaging.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"subscriptions\" in API group \"messaging.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): inmemorychannels.messaging.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"inmemorychannels\" in API group \"messaging.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): channels.messaging.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"channels\" in API group \"messaging.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): kafkachannels.messaging.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"kafkachannels\" in API group \"messaging.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): configurations.serving.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"configurations\" in API group \"serving.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): routes.serving.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"routes\" in API group \"serving.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): services.serving.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"services\" in API group \"serving.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): revisions.serving.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"revisions\" in API group \"serving.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): domainmappings.serving.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"domainmappings\" in API group \"serving.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): pingsources.sources.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"pingsources\" in API group \"sources.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): sinkbindings.sources.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"sinkbindings\" in API group \"sources.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): apiserversources.sources.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"apiserversources\" in API group \"sources.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): containersources.sources.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"containersources\" in API group \"sources.knative.dev\" in the namespace \"openshift-monitoring\"\r\nError from server (Forbidden): kafkasources.sources.knative.dev is forbidden: User \"makarewicz-openshift\" cannot list resource \"kafkasources\" in API group \"sources.knative.dev\" in the namespace \"openshift-monitoring\"<\/pre>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Verifying Cluster Versions<\/span><\/p>\n<ul>\n<li><code>oc get clusterversion<\/code> shows details about the current version of the cluster that is used<\/li>\n<li><code>oc describe clusterversion<\/code> shows more details about versions of the different components<\/li>\n<li><code>oc version<\/code> shows OpenShift version, Kubernetes version, as well as client version<\/li>\n<\/ul>\n<pre class=\"lang:default decode:true \">$ oc get clusterversion\r\nNAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS\r\nversion   4.13.1    True        False         59d     Cluster version is 4.13.1\r\n\r\n\r\n$ oc describe clusterversion\r\nName:         version\r\nNamespace:    \r\nLabels:       &lt;none&gt;\r\nAnnotations:  &lt;none&gt;\r\nAPI Version:  config.openshift.io\/v1\r\nKind:         ClusterVersion\r\nMetadata:\r\n  Creation Timestamp:  2022-11-08T19:49:58Z\r\n  Generation:          12\r\n  Resource Version:    1072696093\r\n  UID:                 02fed286-c1d7-4b17-923d-54cb4de01696\r\nSpec:\r\n  Channel:     fast-4.13\r\n  Cluster ID:  8a263e14-10dc-4a52-925b-2d53054945a2\r\n  Desired Update:\r\n    Version:  4.13.1\r\nStatus:\r\n  Available Updates:\r\n    Channels:\r\n      candidate-4.13\r\n      candidate-4.14\r\n      fast-4.13\r\n    Image:    quay.io\/openshift-release-dev\/ocp-release@sha256:3ca57045e070978b38c36d4c98e188795a6cb4b128130f9c8d7a08b47c133aba\r\n    URL:      https:\/\/access.redhat.com\/errata\/RHSA-2023:4226\r\n    Version:  4.13.6\r\n    Channels:\r\n      candidate-4.13\r\n      candidate-4.14\r\n      fast-4.13\r\n      stable-4.13\r\n    Image:    quay.io\/openshift-release-dev\/ocp-release@sha256:af19e94813478382e36ae1fa2ae7bbbff1f903dded6180f4eb0624afe6fc6cd4\r\n    URL:      https:\/\/access.redhat.com\/errata\/RHSA-2023:4091\r\n    Version:  4.13.5\r\n    Channels:\r\n      candidate-4.13\r\n      candidate-4.14\r\n      fast-4.13\r\n      stable-4.13\r\n    Image:    quay.io\/openshift-release-dev\/ocp-release@sha256:e3fb8ace9881ae5428ae7f0ac93a51e3daa71fa215b5299cd3209e134cadfc9c\r\n    URL:      https:\/\/access.redhat.com\/errata\/RHSA-2023:3614\r\n    Version:  4.13.4\r\n    Channels:\r\n      candidate-4.13\r\n      candidate-4.14\r\n      fast-4.13\r\n      stable-4.13\r\n    Image:    quay.io\/openshift-release-dev\/ocp-release@sha256:bc9835804046aa844c874d2cc37387ec95fe7e87d8ce96129fba78d465c932fa\r\n    URL:      https:\/\/access.redhat.com\/errata\/RHSA-2023:3537\r\n    Version:  4.13.3\r\n    Channels:\r\n      candidate-4.13\r\n      candidate-4.14\r\n      fast-4.13\r\n      stable-4.13\r\n    Image:    quay.io\/openshift-release-dev\/ocp-release@sha256:6ef3cf4bed1970d547dce08a6e334b675d361b212427c4493151dcad6e093d27\r\n    URL:      https:\/\/access.redhat.com\/errata\/RHSA-2023:3367\r\n    Version:  4.13.2\r\n  Capabilities:\r\n    Enabled Capabilities:\r\n      CSISnapshot\r\n      Console\r\n      Insights\r\n      NodeTuning\r\n      Storage\r\n      baremetal\r\n      marketplace\r\n      openshift-samples\r\n    Known Capabilities:\r\n      CSISnapshot\r\n      Console\r\n      Insights\r\n      NodeTuning\r\n      Storage\r\n      baremetal\r\n      marketplace\r\n      openshift-samples\r\n  Conditional Updates:\r\n    Conditions:\r\n      Last Transition Time:  2023-07-30T14:52:52Z\r\n      Message:               The update is recommended, because none of the conditional update risks apply to this cluster.\r\n      Reason:                AsExpected\r\n      Status:                True\r\n      Type:                  Recommended\r\n    Release:\r\n      Channels:\r\n        candidate-4.13\r\n        candidate-4.14\r\n        fast-4.13\r\n        stable-4.13\r\n      Image:    quay.io\/openshift-release-dev\/ocp-release@sha256:e3fb8ace9881ae5428ae7f0ac93a51e3daa71fa215b5299cd3209e134cadfc9c\r\n      URL:      https:\/\/access.redhat.com\/errata\/RHSA-2023:3614\r\n      Version:  4.13.4\r\n    Risks:\r\n      Matching Rules:\r\n        Promql:\r\n          Promql:  group(network_attachment_definition_instances &gt; 0) or 0 * group(network_attachment_definition_instances)\r\n        Type:      PromQL\r\n      Message:     Upgrade can get stuck on clusters that have multiple network attachments.\r\n      Name:        MultiNetworkAttachmentsWhereaboutsVersion\r\n      URL:         https:\/\/access.redhat.com\/solutions\/7024726\r\n    Conditions:\r\n      Last Transition Time:  2023-07-30T14:52:52Z\r\n      Message:               The update is recommended, because none of the conditional update risks apply to this cluster.\r\n      Reason:                AsExpected\r\n      Status:                True\r\n      Type:                  Recommended\r\n    Release:\r\n      Channels:\r\n        candidate-4.13\r\n        candidate-4.14\r\n        fast-4.13\r\n        stable-4.13\r\n      Image:    quay.io\/openshift-release-dev\/ocp-release@sha256:bc9835804046aa844c874d2cc37387ec95fe7e87d8ce96129fba78d465c932fa\r\n      URL:      https:\/\/access.redhat.com\/errata\/RHSA-2023:3537\r\n      Version:  4.13.3\r\n    Risks:\r\n      Matching Rules:\r\n        Promql:\r\n          Promql:  group(network_attachment_definition_instances &gt; 0) or 0 * group(network_attachment_definition_instances)\r\n        Type:      PromQL\r\n      Message:     Upgrade can get stuck on clusters that have multiple network attachments.\r\n      Name:        MultiNetworkAttachmentsWhereaboutsVersion\r\n      URL:         https:\/\/access.redhat.com\/solutions\/7024726\r\n    Conditions:\r\n      Last Transition Time:  2023-07-30T14:52:52Z\r\n      Message:               The update is recommended, because none of the conditional update risks apply to this cluster.\r\n      Reason:                AsExpected\r\n      Status:                True\r\n      Type:                  Recommended\r\n    Release:\r\n      Channels:\r\n        candidate-4.13\r\n        candidate-4.14\r\n        fast-4.13\r\n        stable-4.13\r\n      Image:    quay.io\/openshift-release-dev\/ocp-release@sha256:6ef3cf4bed1970d547dce08a6e334b675d361b212427c4493151dcad6e093d27\r\n      URL:      https:\/\/access.redhat.com\/errata\/RHSA-2023:3367\r\n      Version:  4.13.2\r\n    Risks:\r\n      Matching Rules:\r\n        Promql:\r\n          Promql:  group(network_attachment_definition_instances &gt; 0) or 0 * group(network_attachment_definition_instances)\r\n        Type:      PromQL\r\n      Message:     Upgrade can get stuck on clusters that have multiple network attachments.\r\n      Name:        MultiNetworkAttachmentsWhereaboutsVersion\r\n      URL:         https:\/\/access.redhat.com\/solutions\/7024726\r\n  Conditions:\r\n    Last Transition Time:  2023-01-21T01:57:45Z\r\n    Status:                True\r\n    Type:                  RetrievedUpdates\r\n    Last Transition Time:  2023-05-31T23:04:38Z\r\n    Message:               Capabilities match configured spec\r\n    Reason:                AsExpected\r\n    Status:                False\r\n    Type:                  ImplicitlyEnabledCapabilities\r\n    Last Transition Time:  2023-05-31T23:04:26Z\r\n    Message:               Payload loaded version=\"4.13.1\" image=\"quay.io\/openshift-release-dev\/ocp-release@sha256:9c92b5ec203ee7f81626cc4e9f02086484056a76548961e5895916f136302b1f\" architecture=\"amd64\"\r\n    Reason:                PayloadLoaded\r\n    Status:                True\r\n    Type:                  ReleaseAccepted\r\n    Last Transition Time:  2022-11-08T20:11:43Z\r\n    Message:               Done applying 4.13.1\r\n    Status:                True\r\n    Type:                  Available\r\n    Last Transition Time:  2023-07-28T20:56:25Z\r\n    Status:                False\r\n    Type:                  Failing\r\n    Last Transition Time:  2023-06-01T00:29:13Z\r\n    Message:               Cluster version is 4.13.1\r\n    Status:                False\r\n    Type:                  Progressing\r\n    Last Transition Time:  2023-05-31T23:36:38Z\r\n    Message:               Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: openshift-operators\/openshift-pipelines-operator-rh.v1.10.2 is incompatible with OpenShift minor versions greater than 4.13\r\n    Reason:                IncompatibleOperatorsInstalled\r\n    Status:                False\r\n    Type:                  Upgradeable\r\n  Desired:\r\n    Channels:\r\n      candidate-4.13\r\n      candidate-4.14\r\n      fast-4.13\r\n      stable-4.13\r\n    Image:    quay.io\/openshift-release-dev\/ocp-release@sha256:9c92b5ec203ee7f81626cc4e9f02086484056a76548961e5895916f136302b1f\r\n    URL:      https:\/\/access.redhat.com\/errata\/RHSA-2023:3304\r\n    Version:  4.13.1\r\n  History:\r\n    Completion Time:    2023-06-01T00:29:13Z\r\n    Image:              quay.io\/openshift-release-dev\/ocp-release@sha256:9c92b5ec203ee7f81626cc4e9f02086484056a76548961e5895916f136302b1f\r\n    Started Time:       2023-05-31T23:04:26Z\r\n    State:              Completed\r\n    Verified:           true\r\n    Version:            4.13.1\r\n    Completion Time:    2023-05-27T07:59:25Z\r\n    Image:              quay.io\/openshift-release-dev\/ocp-release@sha256:7ca5f8aa44bbc537c5a985a523d87365eab3f6e72abc50b7be4caae741e093f4\r\n    Started Time:       2023-05-27T03:06:28Z\r\n    State:              Completed\r\n    Verified:           true\r\n    Version:            4.12.17\r\n    Completion Time:    2023-04-04T02:20:05Z\r\n    Image:              quay.io\/openshift-release-dev\/ocp-release@sha256:96bf74ce789ccb22391deea98e0c5050c41b67cc17defbb38089d32226dba0b8\r\n    Started Time:       2023-04-04T01:02:02Z\r\n    State:              Completed\r\n    Verified:           true\r\n    Version:            4.12.9\r\n    Completion Time:    2023-01-21T02:26:02Z\r\n    Image:              quay.io\/openshift-release-dev\/ocp-release@sha256:4c5a7e26d707780be6466ddc9591865beb2e3baa5556432d23e8d57966a2dd18\r\n    Started Time:       2023-01-21T01:05:32Z\r\n    State:              Completed\r\n    Verified:           true\r\n    Version:            4.12.0\r\n    Completion Time:    2022-11-22T19:11:59Z\r\n    Image:              quay.io\/openshift-release-dev\/ocp-release@sha256:9ffb17b909a4fdef5324ba45ec6dd282985dd49d25b933ea401873183ef20bf8\r\n    Started Time:       2022-11-22T18:15:57Z\r\n    State:              Completed\r\n    Verified:           true\r\n    Version:            4.11.13\r\n    Completion Time:    2022-11-08T20:11:43Z\r\n    Image:              quay.io\/openshift-release-dev\/ocp-release@sha256:1ce5676839bca4f389cdc1c3ddc1a78ab033d4c554453ca7ef61a23e34da0803\r\n    Started Time:       2022-11-08T19:50:00Z\r\n    State:              Completed\r\n    Verified:           false\r\n    Version:            4.11.3\r\n  Observed Generation:  12\r\n  Version Hash:         aIi9LgsoS9c=\r\nEvents:                 &lt;none&gt;\r\n\r\n\r\n$ oc version\r\nClient Version: 4.13.0-202303241616.p0.g92b1a3d.assembly.stream-92b1a3d\r\nKustomize Version: v4.5.7\r\nServer Version: 4.13.1\r\nKubernetes Version: v1.26.3+b404935<\/pre>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Understanding Nodes<\/span><\/p>\n<ul>\n<li>OpenShift worker nodes run <strong>CoreOS<\/strong><\/li>\n<li>CoreOS is a minimized operating system that is managed like a container\n<ul>\n<li>No direct modifications allowed<\/li>\n<\/ul>\n<\/li>\n<li>Most services on the CoreOS node run as containers\n<ul>\n<li>Investigate like any other container<\/li>\n<\/ul>\n<\/li>\n<li>Some services are managed by systemd\n<ul>\n<li><strong>CRI-o<\/strong> is the container engine that is required to run the containers<\/li>\n<li><strong>kubelet<\/strong> is the interface that allows the OpenShift cluster to schedule containers on top of the container engine<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Investigating Node Logs<\/span><\/p>\n<ul>\n<li><code>oc adm node-logs nodename<\/code> will show logs generated by a CoreOS node<\/li>\n<li><code>oc adm node-logs -u crio nodename<\/code> will show logs generated by the CRI-o service<\/li>\n<li><code>oc adm node-logs -u kubelet nodename<\/code> will show logs generated by the kubelet service<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Opening a Shell on a Node<\/span><\/p>\n<ul>\n<li>Opening a shell session to nodes in a managed full-stack automation OpenShift cluster is not always necessary, because of how the cluster is offered within the cloud<\/li>\n<li>Use <code>oc debug node\/nodename<\/code> to open a debug shell on a node\n<ul>\n<li>The debug shell mounts the node root file system at the \/host folder, which allows you to inspect files from the node<\/li>\n<li>To run host binaries, use <code>chroot \/host<\/code><\/li>\n<li>Notice that the host is running a minimal operating system and does not provide access to all Linux tools<\/li>\n<li>Use <code>systemctl status kubelet<\/code> or <code>systemctl status crio<\/code> to investigate status of these vital services<\/li>\n<li>Use <code>crictl ps<\/code> for low-level information about CRI-o containers<\/li>\n<\/ul>\n<\/li>\n<li>If the control plane is not running, you cannot use <code>oc debug node<\/code><\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Using Direct SSH Access<\/span><\/p>\n<ul>\n<li>You should not use direct SSH access<\/li>\n<li>If you want to do it anyway, look up the SSH keys stored on the client machine in some deployment scenarios<\/li>\n<li>On CRC, use <code>ssh -i ~\/.crc\/machines\/crc\/id_rsa coreos@$(crc ip)<\/code> to open a shell as user coreos<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Cluster Scaling<\/span><\/p>\n<ul>\n<li>Manual or automatic cluster scaling works through machine API<\/li>\n<li>Installing an additional worker node is not considered scaling!<\/li>\n<li>Machine API is a standard component that runs as an operator<\/li>\n<li>This operator provides controllers that interact with cluster resources<\/li>\n<li>In a full-stack automated environment, it communicates with the provider to take care of cluster scaling<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Machine API Custom Resources<\/span><\/p>\n<ul>\n<li>Machines are the compute units in the cluster<\/li>\n<li>MachineSets describe groups of machines, but not control plane nodes\n<ul>\n<li>MachineSets are to machines what replica sets are to Pods<\/li>\n<li>It includes labels that allow you to work with regions, zones, and instance types<\/li>\n<li>When deployed to public cloud, you&#8217;ll typically get one machine set per availability zone<\/li>\n<\/ul>\n<\/li>\n<li>MachineHealthChecks verify the health of a machine and take action if required<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Manually Scaling Machines<\/span><\/p>\n<ul>\n<li>Manually scaling the number of machines works in two ways:\n<ul>\n<li>Use <code>oc scale <\/code><\/li>\n<li>Change the number of replicas in the machine set resource<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Automatic scaling<\/span><\/p>\n<ul>\n<li>Automatic scaling in full-stack automation requires two custom resources:\n<ul>\n<li><code>MachineAutoscaler <\/code><\/li>\n<li><code>ClusterAutoscaler<\/code><\/li>\n<\/ul>\n<\/li>\n<li>The Machine API operator must be operational in order to configure any type of scaling<\/li>\n<li>The machine autoscaler automatically scales the number of replicas based on load<\/li>\n<li>The cluster autoscaler enforces limits for the entire cluster\n<ul>\n<li><code>MaxNodesTotal<\/code> sets maximum nodes<\/li>\n<li><code>MaxMemoryTotal<\/code> sets maximum memory<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Implementing AutoScaler<\/span><\/p>\n<ul>\n<li>To implement autoscaling, the following requirements must be met:\n<ul>\n<li>The cluster is deployed in full-stack automation<\/li>\n<li>There is a cluster autoscaler resource\n<ul>\n<li>Set <code>scaleDown<\/code> to <code>enabled: true<\/code> to allow for downscaling as well<\/li>\n<\/ul>\n<\/li>\n<li>At least one machine autoscaler resource exists<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Cluster Updates<\/span><\/p>\n<ul>\n<li>OpenShift 4.x offers Over-the-Air (OTA) upgrades<\/li>\n<li>The OTA software distribution system manages controller manifests, cluster roles and other resources necessary to update a cluster<\/li>\n<li>OTA is offered as a service on https:\/\/cloud.redhat.com; which provides a web interface to easily perform the update<\/li>\n<li>OTA requires the cluster to have a persistent connection to the Internet<\/li>\n<\/ul>\n<p><span style=\"color: #3366ff;\">How OTA Works<\/span><\/p>\n<ul>\n<li>Prometheus based telemetry is used to determine the update path<\/li>\n<li>Supported operators can be automatically updated<\/li>\n<li>Future versions will include Independent Software Vendor (ISV) operators to be updated in this way as well<\/li>\n<li>From the cloudiredhat.com interface, an update channel can be selected to determine the version of OpenShift to update to<\/li>\n<li>Notice that no support is offered to do rollback<\/li>\n<\/ul>\n<p><span style=\"color: #3366ff;\">How OTA Flow<\/span><\/p>\n<ul>\n<li>First, all operators need to be updated to the newer version<\/li>\n<li>Next, the CoreOS images can be updated\n<ul>\n<li>The node will first pull the new image<\/li>\n<li>Next, the image is written to disk<\/li>\n<li>Then the bootloader is changed to boot the new image<\/li>\n<li>To complete, the CoreOS machine reboots<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Manually Updating the Cluster<\/span><\/p>\n<ul>\n<li><code>oc get clusterversion<\/code> will show the current version<\/li>\n<li><code>oc adm upgrade<\/code> will show if an upgrade is available<\/li>\n<li><code>oc adm upgrade --to-latest=true<\/code> will upgrade to the latest version<\/li>\n<li><code>oc adm upgrade --to=version<\/code> will upgrade to a specific version<\/li>\n<li><code>oc get clusterversion<\/code> allows to verify the update<\/li>\n<li><code>oc get clusteroperators<\/code> will show if operators are in the right state<\/li>\n<li><code>oc describe clusterversion<\/code> will show an overview of past upgrades<\/li>\n<\/ul>\n<p>&nbsp;<\/p>\n<pre class=\"lang:default decode:true\">$ oc get clusterversion\r\nNAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS\r\nversion   4.13.1    True        False         59d     Cluster version is 4.13.1\r\nbash-4.4 ~ $ \r\n\r\n\r\n$ oc adm upgrade\r\nCluster version is 4.13.1\r\n\r\nUpgradeable=False\r\n\r\n  Reason: IncompatibleOperatorsInstalled\r\n  Message: Cluster operator operator-lifecycle-manager should not be upgraded between minor versions: ClusterServiceVersions blocking cluster upgrade: openshift-operators\/openshift-pipelines-operator-rh.v1.10.2 is incompatible with OpenShift minor versions greater than 4.13\r\n\r\nUpstream is unset, so the cluster will use an appropriate default.\r\nChannel: fast-4.13 (available channels: candidate-4.13, candidate-4.14, fast-4.13, stable-4.13)\r\n\r\nRecommended updates:\r\n\r\n  VERSION     IMAGE\r\n  4.13.6      quay.io\/openshift-release-dev\/ocp-release@sha256:3ca57045e070978b38c36d4c98e188795a6cb4b128130f9c8d7a08b47c133aba\r\n  4.13.5      quay.io\/openshift-release-dev\/ocp-release@sha256:af19e94813478382e36ae1fa2ae7bbbff1f903dded6180f4eb0624afe6fc6cd4\r\n  4.13.4      quay.io\/openshift-release-dev\/ocp-release@sha256:e3fb8ace9881ae5428ae7f0ac93a51e3daa71fa215b5299cd3209e134cadfc9c\r\n  4.13.3      quay.io\/openshift-release-dev\/ocp-release@sha256:bc9835804046aa844c874d2cc37387ec95fe7e87d8ce96129fba78d465c932fa\r\n  4.13.2      quay.io\/openshift-release-dev\/ocp-release@sha256:6ef3cf4bed1970d547dce08a6e334b675d361b212427c4493151dcad6e093d27\r\n\r\n\r\n$ oc adm upgrade --to-latest=true\r\nerror: Unable to upgrade: clusterversions.config.openshift.io \"version\" is forbidden: User \"makarewicz-openshift\" cannot patch resource \"clusterversions\" in API group \"config.openshift.io\" at the cluster scope\r\n\r\n\r\n$ oc get clusterversion\r\nNAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS\r\nversion   4.13.1    True        False         59d     Cluster version is 4.13.1<\/pre>\n<p>&nbsp;<\/p>\n<p><span style=\"color: #3366ff;\">Lab: Monitoring Cluster Health<\/span><\/p>\n<ul>\n<li>Use the appropriate tools to create a full cluster health report, and write the result of these commands to the file<code> \/tmp\/health.txt<\/code><\/li>\n<\/ul>\n<pre class=\"lang:default decode:true \">$ oc get nodes &gt; \/tmp\/health.txt\r\n$ oc describe nodes\r\nName:               localhost\r\nRoles:              &lt;none&gt;\r\nLabels:             beta.kubernetes.io\/arch=amd64\r\n                    beta.kubernetes.io\/os=linux\r\n                    disktype=nvme\r\n                    kubernetes.io\/hostname=localhost\r\nAnnotations:        volumes.kubernetes.io\/controller-managed-attach-detach=true\r\nCreationTimestamp:  Sat, 22 Jul 2023 20:54:32 +0200\r\nTaints:             &lt;none&gt;\r\nUnschedulable:      false\r\nConditions:\r\n  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message\r\n  ----             ------  -----------------                 ------------------                ------                       -------\r\n  OutOfDisk        False   Sun, 30 Jul 2023 20:16:27 +0200   Sat, 22 Jul 2023 20:54:22 +0200   KubeletHasSufficientDisk     kubelet has                   sufficient disk space available\r\n  MemoryPressure   False   Sun, 30 Jul 2023 20:16:27 +0200   Sat, 22 Jul 2023 20:54:22 +0200   KubeletHasSufficientMemory   kubelet has                   sufficient memory available\r\n  DiskPressure     False   Sun, 30 Jul 2023 20:16:27 +0200   Sat, 22 Jul 2023 20:54:22 +0200   KubeletHasNoDiskPressure     kubelet has                   no disk pressure\r\n  PIDPressure      False   Sun, 30 Jul 2023 20:16:27 +0200   Sat, 22 Jul 2023 20:54:22 +0200   KubeletHasSufficientPID      kubelet has                   sufficient PID available\r\n  Ready            True    Sun, 30 Jul 2023 20:16:27 +0200   Sat, 22 Jul 2023 20:54:22 +0200   KubeletReady                 kubelet is                   posting ready status\r\nAddresses:\r\n  InternalIP:  172.30.9.22\r\n  Hostname:    localhost\r\nCapacity:\r\n cpu:            8\r\n hugepages-1Gi:  0\r\n hugepages-2Mi:  0\r\n memory:         7981844Ki\r\n pods:           250\r\nAllocatable:\r\n cpu:            8\r\n hugepages-1Gi:  0\r\n hugepages-2Mi:  0\r\n memory:         7879444Ki\r\n pods:           250\r\nSystem Info:\r\n Machine ID:                     a37388a4746444f1b3f079f777748845\r\n System UUID:                    6099DE02-9EA8-C210-7553-A7697F2C302A\r\n Boot ID:                        7c076895-85e9-45ce-ae2c-8bbe7127be73\r\n Kernel Version:                 3.10.0-1160.92.1.el7.x86_64\r\n OS Image:                       CentOS Linux 7 (Core)\r\n Operating System:               linux\r\n Architecture:                   amd64\r\n Container Runtime Version:      docker:\/\/24.0.3\r\n Kubelet Version:                v1.11.0+d4cacc0\r\n Kube-Proxy Version:             v1.11.0+d4cacc0\r\nNon-terminated Pods:             (45 in total)\r\n  Namespace                      Name                                                       CPU Requests  CPU Limits  Memory Requests                    Memory Limits\r\n  ---------                      ----                                                       ------------  ----------  ---------------                    -------------\r\n  debug                          dnginx-88c7766dd-hlbtd                                     0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  default                        bitginx-1-jzk9r                                            0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  default                        busybox                                                    0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  default                        docker-registry-1-ctgff                                    100m (1%)     0 (0%)      256Mi (3%)                         0 (0%)\r\n  default                        lab4pod                                                    0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  default                        linginx1-dc9f65f54-6zw8j                                   0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  default                        linginx2-69bf6fc66b-mv6wx                                  0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  default                        nginx                                                      0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  default                        nginx-cm                                                   0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  default                        pv-pod                                                     0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  default                        router-1-k8zgt                                             100m (1%)     0 (0%)      256Mi (3%)                         0 (0%)\r\n  default                        test1                                                      0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  kube-dns                       kube-dns-t727w                                             0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  kube-proxy                     kube-proxy-cr7kh                                           0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  kube-system                    kube-controller-manager-localhost                          0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  kube-system                    kube-scheduler-localhost                                   0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  kube-system                    master-api-localhost                                       0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  kube-system                    master-etcd-localhost                                      0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  limits                         nee-597889d8c7-p6tc2                                       0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  love                           anti1                                                      0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  love                           newpod-1-qgmpj                                             0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  love                           runonssd                                                   0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  network-security               nginxlab-1-bcgkt                                           0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  nodesel                        simple-6f55965d79-mklpc                                    0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  nodesel                        simple-6f55965d79-q8pq9                                    0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  openshift-apiserver            openshift-apiserver-thwpd                                  0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  openshift-controller-manager   openshift-controller-manager-c9ms5                         0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  openshift-core-operators       openshift-service-cert-signer-operator-6d477f986b-jzcgw    0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  openshift-core-operators       openshift-web-console-operator-664b974ff5-px7gw            0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  openshift-service-cert-signer  apiservice-cabundle-injector-8ffbbb6dc-x9l4r               0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  openshift-service-cert-signer  service-serving-cert-signer-668c45d5f-lxvff                0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  openshift-web-console          webconsole-78f59b4bfb-qqv4p                                100m (1%)     0 (0%)      100Mi (1%)                         0 (0%)\r\n  quota-test                     bitginx-84b698ff5c-h5j57                                   10m (0%)      50m (0%)    5Mi (0%)                           20Mi (0%)\r\n  quota-test                     bitginx-84b698ff5c-qxwd5                                   10m (0%)      50m (0%)    5Mi (0%)                           20Mi (0%)\r\n  quota-test                     bitginx-84b698ff5c-xfnp4                                   10m (0%)      50m (0%)    5Mi (0%)                           20Mi (0%)\r\n  source-project                 nginx-access                                               0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  source-project                 nginx-noaccess                                             0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  target-project                 nginx-target-1-9kdn6                                       0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  template-project               anti1                                                      0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  test-project                   nginxmany-5859c9dbb6-5xxr6                                 0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  test-project                   nginxmany-5859c9dbb6-6ljwm                                 0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  test-project                   nginxmany-5859c9dbb6-9lv6c                                 0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  test-project                   nginxmany-5859c9dbb6-dgr7k                                 0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  test-project                   nginxmany-5859c9dbb6-dl8wr                                 0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\n  test-project                   nginxmany-5859c9dbb6-hk2sm                                 0 (0%)        0 (0%)      0 (0%)                             0 (0%)\r\nAllocated resources:\r\n  (Total limits may be over 100 percent, i.e., overcommitted.)\r\n  Resource  Requests    Limits\r\n  --------  --------    ------\r\n  cpu       330m (4%)   150m (1%)\r\n  memory    627Mi (8%)  60Mi (0%)\r\nEvents:     &lt;none&gt;\r\n\r\n$ oc describe nodes &gt;&gt; \/tmp\/health.txt \r\n$ oc get co &gt;&gt; \/tmp\/health.txt\r\n$ oc get clusterversion &gt;&gt; \/tmp\/health.txt\r\n<\/pre>\n","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":5958,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[93],"tags":[],"_links":{"self":[{"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/posts\/5015"}],"collection":[{"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/comments?post=5015"}],"version-history":[{"count":31,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/posts\/5015\/revisions"}],"predecessor-version":[{"id":5959,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/posts\/5015\/revisions\/5959"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/media\/5958"}],"wp:attachment":[{"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/media?parent=5015"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/categories?post=5015"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/miro.borodziuk.eu\/index.php\/wp-json\/wp\/v2\/tags?post=5015"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}