Podが"Pending"のまま起動しない原因(kubernetes)
見出し画像

Podが"Pending"のまま起動しない原因(kubernetes)

kujiraitakahiro


LinuCエヴァンジェリストの鯨井貴博@opensourcetechです。


はじめに


KubernetesでPodが"Pending"のまま起動しない原因に関するメモです。



事象の発生


以下のように、Deployment(Pod)を含むをapplyします。

kubeuser@kubemaster1:~$ kubectl apply -f nginx.yaml
deployment.apps/nginx created

kubeuser@kubemaster1:~$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-6c67f5ff6f-nrdvt   0/1     Pending   0          7s
nginx-6c67f5ff6f-t4sg8   0/1     Pending   0          7s

kubeuser@kubemaster1:~$ kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES
nginx-6c67f5ff6f-nrdvt   0/1     Pending   0          13s   <none>   <none>   <none>           <none>
nginx-6c67f5ff6f-t4sg8   0/1     Pending   0          13s   <none>   <none>   <none>           <none>

.
.
.
kubeuser@kubemaster1:~$ kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES
nginx-6c67f5ff6f-nrdvt   0/1     Pending   0          77s   <none>   <none>   <none>           <none>
nginx-6c67f5ff6f-t4sg8   0/1     Pending   0          77s   <none>   <none>   <none>           <none>


STATUSが"Pending"のまな起動してくる気配がありません。


Podが起動しない原因


Podが起動しない原因は、Worker Nodeにあります。

kubeuser@kubemaster1:~$ kubectl get nodes
NAME          STATUS   ROLES    AGE    VERSION
kubemaster1   Ready    master   6d1h   v1.18.0
kubemaster2   Ready    master   6d1h   v1.18.0
kubemaster3   Ready    master   6d     v1.18.0
kubeworker    Ready    <none>   6d1h   v1.18.0
kubeworker2   Ready    <none>   6d     v1.18.0


STATUS "Ready"となっており特に異常はないようですが、
実は、Worker Nodeではkubeletが停止しています。
kubernetesのクラスターでは、以下のようにMaster NodeのAPI-ServerとWorker Nodeのkubeletが通信をしていますが、kubeletが停止しているとこれが出来ずPodの配置(スケジューリング)が出来なくなることが原因です。


https://kubernetes.io/ja/docs/concepts/overview/components/

Worker Node(1台目)。

kubeuser@kubeworker:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor prese>
    Drop-In: /etc/systemd/system/kubelet.service.d
             mq10-kubeadm.conf
     Active: inactive (dead) since Tue 2021-12-07 13:12:25 UTC; 3s ago
       Docs: https://kubernetes.io/docs/home/
    Process: 19532 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET>
   Main PID: 19532 (code=exited, status=0/SUCCESS)

Dec 07 10:52:30 kubeworker kubelet[19532]: E1207 10:52:30.866416   19532 remote>
Dec 07 11:36:48 kubeworker kubelet[19532]: E1207 11:36:48.594091   19532 contro>
Dec 07 11:36:55 kubeworker kubelet[19532]: E1207 11:36:55.696722   19532 contro>
Dec 07 11:36:55 kubeworker kubelet[19532]: E1207 11:36:55.875900   19532 contro>
Dec 07 11:36:55 kubeworker kubelet[19532]: E1207 11:36:55.876512   19532 contro>
Dec 07 11:36:55 kubeworker kubelet[19532]: E1207 11:36:55.876845   19532 contro>
Dec 07 11:36:56 kubeworker kubelet[19532]: I1207 11:36:55.877071   19532 contro>
Dec 07 13:12:25 kubeworker systemd[1]: Stopping kubelet: The Kubernetes Node Ag>
Dec 07 13:12:25 kubeworker systemd[1]: kubelet.service: Succeeded.
Dec 07 13:12:25 kubeworker systemd[1]: Stopped kubelet: The Kubernetes Node Age>


Worker Node(2台目)。

kubeuser@kubeworker2:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor prese>
    Drop-In: /etc/systemd/system/kubelet.service.d
             mq10-kubeadm.conf
     Active: inactive (dead) since Tue 2021-12-07 13:12:56 UTC; 1s ago
       Docs: https://kubernetes.io/docs/home/
    Process: 661 ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_C>
   Main PID: 661 (code=exited, status=0/SUCCESS)

Dec 07 13:03:15 kubeworker2 kubelet[3396]: 2021-12-07 13:03:15.293 [INFO][3410]>
Dec 07 13:03:15 kubeworker2 kubelet[3396]: time="2021-12-07T13:03:15Z" level=in>
Dec 07 13:03:15 kubeworker2 kubelet[3396]: 2021-12-07 13:03:15.318 [INFO][3396]>
Dec 07 13:03:38 kubeworker2 kubelet[661]: E1207 13:03:38.903002     661 control>
Dec 07 13:03:38 kubeworker2 kubelet[661]: E1207 13:03:38.952191     661 kubelet>
Dec 07 13:03:42 kubeworker2 kubelet[661]: E1207 13:03:42.313131     661 control>
Dec 07 13:09:25 kubeworker2 kubelet[661]: E1207 13:09:25.525803     661 kubelet>
Dec 07 13:12:56 kubeworker2 systemd[1]: Stopping kubelet: The Kubernetes Node A>
Dec 07 13:12:56 kubeworker2 systemd[1]: kubelet.service: Succeeded.
Dec 07 13:12:56 kubeworker2 systemd[1]: Stopped kubelet: The Kubernetes Node Ag>

kubectl describe nodesでみても、"Kubelet stopped posting node status."と出てますね。

kubeuser@kubemaster1:~$ kubectl describe nodes kubeworker2
Name:               kubeworker2
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=kubeworker2
                    kubernetes.io/os=linux
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    projectcalico.org/IPv4Address: 192.168.1.254/24
                    projectcalico.org/IPv4IPIPTunnelAddr: 10.0.225.0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 01 Dec 2021 12:23:46 +0000
Taints:             node.kubernetes.io/unreachable:NoExecute
                    node.kubernetes.io/unreachable:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  kubeworker2
  AcquireTime:     <unset>
  RenewTime:       Tue, 07 Dec 2021 13:43:03 +0000
Conditions:
  Type                 Status    LastHeartbeatTime                 LastTransitionTime                Reason              Message
  ----                 ------    -----------------                 ------------------                ------              -------
  NetworkUnavailable   False     Tue, 07 Dec 2021 13:03:14 +0000   Tue, 07 Dec 2021 13:03:14 +0000   CalicoIsUp          Calico is running on this node
  MemoryPressure       Unknown   Tue, 07 Dec 2021 13:43:04 +0000   Tue, 07 Dec 2021 13:43:47 +0000   NodeStatusUnknown   Kubelet stopped posting node status.
  DiskPressure         Unknown   Tue, 07 Dec 2021 13:43:04 +0000   Tue, 07 Dec 2021 13:43:47 +0000   NodeStatusUnknown   Kubelet stopped posting node status.
  PIDPressure          Unknown   Tue, 07 Dec 2021 13:43:04 +0000   Tue, 07 Dec 2021 13:43:47 +0000   NodeStatusUnknown   Kubelet stopped posting node status.
  Ready                Unknown   Tue, 07 Dec 2021 13:43:04 +0000   Tue, 07 Dec 2021 13:43:47 +0000   NodeStatusUnknown   Kubelet stopped posting node status.
Addresses:
  InternalIP:  192.168.1.254
  Hostname:    kubeworker2
Capacity:
  cpu:                2
  ephemeral-storage:  20511312Ki
  hugepages-2Mi:      0
  memory:             2035140Ki
  pods:               110
Allocatable:
  cpu:                2
  ephemeral-storage:  18903225108
  hugepages-2Mi:      0
  memory:             1932740Ki
  pods:               110
System Info:
  Machine ID:                 7c474b3b662c452a98ea24d02d1871e9
  System UUID:                7c474b3b-662c-452a-98ea-24d02d1871e9
  Boot ID:                    d487a166-68fa-4da9-b89d-45cefb6bddc1
  Kernel Version:             5.4.0-91-generic
  OS Image:                   Ubuntu 20.04.3 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.7
  Kubelet Version:            v1.18.0
  Kube-Proxy Version:         v1.18.0
PodCIDR:                      10.0.1.0/24
PodCIDRs:                     10.0.1.0/24
Non-terminated Pods:          (2 in total)
  Namespace                   Name                 CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----                 ------------  ----------  ---------------  -------------  ---
  kube-system                 calico-node-lk26d    250m (12%)    0 (0%)      0 (0%)           0 (0%)         6d1h
  kube-system                 kube-proxy-xv78r     0 (0%)        0 (0%)      0 (0%)           0 (0%)         6d1h
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                250m (12%)  0 (0%)
  memory             0 (0%)      0 (0%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:
  Type     Reason                   Age                 From                     Message
  ----     ------                   ----                ----                     -------
  Normal   Starting                 44m                 kubelet, kubeworker2     Starting kubelet.
  Warning  ImageGCFailed            44m                 kubelet, kubeworker2     failed to get imageFs info: unable to find data in memory cache
  Normal   NodeAllocatableEnforced  44m                 kubelet, kubeworker2     Updated Node Allocatable limit across pods
  Normal   NodeHasSufficientMemory  44m (x2 over 44m)   kubelet, kubeworker2     Node kubeworker2 status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    44m (x2 over 44m)   kubelet, kubeworker2     Node kubeworker2 status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     44m (x2 over 44m)   kubelet, kubeworker2     Node kubeworker2 status is now: NodeHasSufficientPID
  Warning  Rebooted                 44m                 kubelet, kubeworker2     Node kubeworker2 has been rebooted, boot id: d487a166-68fa-4da9-b89d-45cefb6bddc1
  Normal   NodeReady                44m                 kubelet, kubeworker2     Node kubeworker2 status is now: NodeReady
  Normal   Starting                 43m                 kube-proxy, kubeworker2  Starting kube-proxy.
  Normal   Starting                 115s                kubelet, kubeworker2     Starting kubelet.
  Normal   NodeHasSufficientMemory  115s                kubelet, kubeworker2     Node kubeworker2 status is now: NodeHasSufficientMemory
  Normal   NodeHasSufficientPID     115s                kubelet, kubeworker2     Node kubeworker2 status is now: NodeHasSufficientPID
  Normal   NodeAllocatableEnforced  115s                kubelet, kubeworker2     Updated Node Allocatable limit across pods
  Normal   NodeHasNoDiskPressure    99s (x2 over 115s)  kubelet, kubeworker2     Node kubeworker2 status is now: NodeHasNoDiskPressure


参考のため、以下は正常時のもの。

kubeuser@kubemaster1:~$ kubectl describe nodes kubeworker2
Name:               kubeworker2
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=kubeworker2
                    kubernetes.io/os=linux
Annotations:        kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
                    node.alpha.kubernetes.io/ttl: 0
                    projectcalico.org/IPv4Address: 192.168.1.254/24
                    projectcalico.org/IPv4IPIPTunnelAddr: 10.0.225.0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 01 Dec 2021 12:23:46 +0000
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  kubeworker2
  AcquireTime:     <unset>
  RenewTime:       Tue, 07 Dec 2021 13:43:03 +0000
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Tue, 07 Dec 2021 13:03:14 +0000   Tue, 07 Dec 2021 13:03:14 +0000   CalicoIsUp                   Calico is running on this node
  MemoryPressure       False   Tue, 07 Dec 2021 13:43:04 +0000   Tue, 07 Dec 2021 13:43:04 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Tue, 07 Dec 2021 13:43:04 +0000   Tue, 07 Dec 2021 13:43:04 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Tue, 07 Dec 2021 13:43:04 +0000   Tue, 07 Dec 2021 13:43:04 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Tue, 07 Dec 2021 13:43:04 +0000   Tue, 07 Dec 2021 13:43:04 +0000   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  192.168.1.254
  Hostname:    kubeworker2
Capacity:
  cpu:                2
  ephemeral-storage:  20511312Ki
  hugepages-2Mi:      0
  memory:             2035140Ki
  pods:               110
Allocatable:
  cpu:                2
  ephemeral-storage:  18903225108
  hugepages-2Mi:      0
  memory:             1932740Ki
  pods:               110
System Info:
  Machine ID:                 7c474b3b662c452a98ea24d02d1871e9
  System UUID:                7c474b3b-662c-452a-98ea-24d02d1871e9
  Boot ID:                    d487a166-68fa-4da9-b89d-45cefb6bddc1
  Kernel Version:             5.4.0-91-generic
  OS Image:                   Ubuntu 20.04.3 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://20.10.7
  Kubelet Version:            v1.18.0
  Kube-Proxy Version:         v1.18.0
PodCIDR:                      10.0.1.0/24
PodCIDRs:                     10.0.1.0/24
Non-terminated Pods:          (2 in total)
  Namespace                   Name                 CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----                 ------------  ----------  ---------------  -------------  ---
  kube-system                 calico-node-lk26d    250m (12%)    0 (0%)      0 (0%)           0 (0%)         6d1h
  kube-system                 kube-proxy-xv78r     0 (0%)        0 (0%)      0 (0%)           0 (0%)         6d1h
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                250m (12%)  0 (0%)
  memory             0 (0%)      0 (0%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:
  Type     Reason                   Age                From                     Message
  ----     ------                   ----               ----                     -------
  Normal   Starting                 43m                kubelet, kubeworker2     Starting kubelet.
  Warning  ImageGCFailed            43m                kubelet, kubeworker2     failed to get imageFs info: unable to find data in memory cache
  Normal   NodeAllocatableEnforced  43m                kubelet, kubeworker2     Updated Node Allocatable limit across pods
  Normal   NodeHasSufficientMemory  42m (x2 over 43m)  kubelet, kubeworker2     Node kubeworker2 status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    42m (x2 over 43m)  kubelet, kubeworker2     Node kubeworker2 status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     42m (x2 over 43m)  kubelet, kubeworker2     Node kubeworker2 status is now: NodeHasSufficientPID
  Warning  Rebooted                 42m                kubelet, kubeworker2     Node kubeworker2 has been rebooted, boot id: d487a166-68fa-4da9-b89d-45cefb6bddc1
  Normal   NodeReady                42m                kubelet, kubeworker2     Node kubeworker2 status is now: NodeReady
  Normal   Starting                 42m                kube-proxy, kubeworker2  Starting kube-proxy.
  Normal   Starting                 44s                kubelet, kubeworker2     Starting kubelet.
  Normal   NodeHasSufficientMemory  44s                kubelet, kubeworker2     Node kubeworker2 status is now: NodeHasSufficientMemory
  Normal   NodeHasSufficientPID     44s                kubelet, kubeworker2     Node kubeworker2 status is now: NodeHasSufficientPID
  Normal   NodeAllocatableEnforced  44s                kubelet, kubeworker2     Updated Node Allocatable limit across pods
  Normal   NodeHasNoDiskPressure    28s (x2 over 44s)  kubelet, kubeworker2     Node kubeworker2 status is now: NodeHasNoDiskPressure


事象の復旧


Worker Nodeのkubeletを起動します。

ubeuser@kubeworker:~$ sudo systemctl start kubelet
kubeuser@kubeworker:~$ sudo systemctl status kubelet
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/lib/systemd/system/kubelet.service; enabled; vendor prese>
    Drop-In: /etc/systemd/system/kubelet.service.d
             mq10-kubeadm.conf
     Active: active (running) since Tue 2021-12-07 13:14:32 UTC; 2s ago
       Docs: https://kubernetes.io/docs/home/
   Main PID: 2055516 (kubelet)
      Tasks: 1 (limit: 2278)
     Memory: 3.8M
     CGroup: /system.slice/kubelet.service
             mq2055516 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/>

Dec 07 13:14:32 kubeworker systemd[1]: Started kubelet: The Kubernetes Node Age>


では、改めてDeploymentのapplyを実行します。

kubeuser@kubemaster1:~$ kubectl apply -f nginx.yaml
deployment.apps/nginx created

kubeuser@kubemaster1:~$ kubectl get all
NAME                         READY   STATUS    RESTARTS   AGE
pod/nginx-6c67f5ff6f-cc6x8   0/1     Pending   0          2s
pod/nginx-6c67f5ff6f-rkwt2   0/1     Pending   0          3s

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   6d1h

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx   0/2     2            0           9s

NAME                               DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-6c67f5ff6f   2         2         0       9s

kubeuser@kubemaster1:~$ kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE         NOMINATED NODE   READINESS GATES
nginx-6c67f5ff6f-cc6x8   1/1     Running   0          23s   10.0.42.15   kubeworker   <none>           <none>
nginx-6c67f5ff6f-rkwt2   1/1     Running   0          24s   10.0.42.14   kubeworker   <none>           <none>


今後は、無事に起動出来ました!



おわりに


・Worker Node構築時にkubeletが自動起動になっていない(systemctl enable kubeletの未実行) ・何かしらの理由でkubeletがダウンした などに起因して発生するとは思うので発生頻度は稀かもしれませんが、
このことを想定している(知っている)か、そうではないかではトラブルシュートに差が出てきそうですね。

この記事が気に入ったら、サポートをしてみませんか?
気軽にクリエイターの支援と、記事のオススメができます!
ありがとうございます!!!!!!!
kujiraitakahiro
Kubernetes・コンテナ技術・Linuxなどオープンソース技術を中心とした記事を書いています。