【2020年2月25日掲載】CSI Driver for Dell EMC Isilonを試す（テスト編）
サービスマネージメントグループの秋元です。
前回はCSI Driver for Dell EMC Isilonのインストールの流れをご紹介しました。
CSI Driver for Dell EMC Isilonを試す（インストール編）
今回はProduct Guideにある３つのテストの内、１番目のテストがどのような動作になるかご紹介したいと思います。
Test the CSI driver for Dell EMC Isilon
Test the CSI driver for Dell EMC Isilon
Test creating snapshots
Test restoring from a snapshot
Test the CSI driver for Dell EMC Isilon
このテストでは2つのPVC(PersistentVolumeClaim)とStatefulSetを作成し、StatefulSetから作成されるPod内のコンテナがPVCから作成されるPVをNFSマウントしていることを確認します。
1. test namespaceを作成
[root@node1 ~]# kubectl create namespace test
2. test用のディレクトリに移動
[root@node1 ~]# cd csi-isilon/test/helm/
3. 2つのボリュームをマウントするtest scriptを実行
[root@node1 helm]# sh ./starttest.sh -t 2vols -n test
NAME: 2vols
LAST DEPLOYED: Fri Jan 10 11:35:31 2020
NAMESPACE: default
STATUS: DEPLOYED
RESOURCES:
==> v1/PersistentVolumeClaim
NAME AGE
pvol0 0s
pvol1 0s
==> v1/Pod(related)
NAME AGE
isilontest-0 0s
==> v1/ServiceAccount
NAME AGE
isilontest 0s
==> v1/StatefulSet
NAME AGE
isilontest 0s
waiting 60 seconds on pod to initialize
Name: isilontest-0
Namespace: test
Priority: 0
PriorityClassName:
Node: node3/172.16.26.26
Start Time: Fri, 10 Jan 2020 11:35:51 +0900
Labels: app=isilontest
controller-revision-hash=isilontest-744dc88545
statefulset.kubernetes.io/pod-name=isilontest-0
Annotations:
Status: Running
IP: 172.16.26.26
Controlled By: StatefulSet/isilontest
Containers:
test:
Container ID: docker://a61daef38de38ed850a6864701465b8bb4fe34903879d88f519be02708ddf1c1
Image: docker.io/centos:latest
Image ID: docker-pullable://centos@sha256:f94c1d992c193b3dc09e297ffd54d8a4f1dc946c37cbeceb26d35ce1647f88d9
Port:
Host Port:
Command:
/bin/sleep
3600
State: Running
Started: Fri, 10 Jan 2020 11:36:02 +0900
Ready: True
Restart Count: 0
Environment:
Mounts:
/data0 from pvol0 (rw)
/data1 from pvol1 (rw)
/var/run/secrets/kubernetes.io/serviceaccount from isilontest-token-vm75m (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
pvol0:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: pvol0
ReadOnly: false
pvol1:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: pvol1
ReadOnly: false
isilontest-token-vm75m:
Type: Secret (a volume populated by a Secret)
SecretName: isilontest-token-vm75m
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 50s (x5 over 61s) default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
Normal Scheduled 41s default-scheduler Successfully assigned test/isilontest-0 to node3
Normal SuccessfulAttachVolume 41s attachdetach-controller AttachVolume.Attach succeeded for volume "k8s-df3e5dbb33"
Normal SuccessfulAttachVolume 41s attachdetach-controller AttachVolume.Attach succeeded for volume "k8s-df3ebd2c33"
Normal Pulling 34s kubelet, node3 Pulling image "docker.io/centos:latest"
Normal Pulled 30s kubelet, node3 Successfully pulled image "docker.io/centos:latest"
Normal Created 30s kubelet, node3 Created container test
Normal Started 30s kubelet, node3 Started container test
Name: isilontest-0
Namespace: test
Priority: 0
PriorityClassName:
Node: node3/172.16.26.26
Start Time: Fri, 10 Jan 2020 11:35:51 +0900
Labels: app=isilontest
controller-revision-hash=isilontest-744dc88545
statefulset.kubernetes.io/pod-name=isilontest-0
Annotations:
Status: Running
IP: 172.16.26.26
Controlled By: StatefulSet/isilontest
Containers:
test:
Container ID: docker://a61daef38de38ed850a6864701465b8bb4fe34903879d88f519be02708ddf1c1
Image: docker.io/centos:latest
Image ID: docker-pullable://centos@sha256:f94c1d992c193b3dc09e297ffd54d8a4f1dc946c37cbeceb26d35ce1647f88d9
Port:
Host Port:
Command:
/bin/sleep
3600
State: Running
Started: Fri, 10 Jan 2020 11:36:02 +0900
Ready: True
Restart Count: 0
Environment:
Mounts:
/data0 from pvol0 (rw)
/data1 from pvol1 (rw)
/var/run/secrets/kubernetes.io/serviceaccount from isilontest-token-vm75m (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
pvol0:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: pvol0
ReadOnly: false
pvol1:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: pvol1
ReadOnly: false
isilontest-token-vm75m:
Type: Secret (a volume populated by a Secret)
SecretName: isilontest-token-vm75m
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 50s (x5 over 61s) default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
Normal Scheduled 41s default-scheduler Successfully assigned test/isilontest-0 to node3
Normal SuccessfulAttachVolume 41s attachdetach-controller AttachVolume.Attach succeeded for volume "k8s-df3e5dbb33"
Normal SuccessfulAttachVolume 41s attachdetach-controller AttachVolume.Attach succeeded for volume "k8s-df3ebd2c33"
Normal Pulling 34s kubelet, node3 Pulling image "docker.io/centos:latest"
Normal Pulled 30s kubelet, node3 Successfully pulled image "docker.io/centos:latest"
Normal Created 30s kubelet, node3 Created container test
Normal Started 30s kubelet, node3 Started container test
172.16.27.157:/ifs/data/csi/k8s-df3ebd2c33 231972494336 21611520 223778394112 1% /data0
172.16.27.157:/ifs/data/csi/k8s-df3e5dbb33 231972494336 21611520 223778394112 1% /data1
172.16.27.157:/ifs/data/csi/k8s-df3ebd2c33 on /data0 type nfs (rw,relatime,vers=3,rsize=131072,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.27.157,mountvers=3,mountport=300,mountproto=udp,local_lock=none,addr=172.16.27.157)
172.16.27.157:/ifs/data/csi/k8s-df3e5dbb33 on /data1 type nfs (rw,relatime,vers=3,rsize=131072,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.16.27.157,mountvers=3,mountport=300,mountproto=udp,local_lock=none,addr=172.16.27.157)
[root@node1 helm]#
Kubernetes側の状態を確認していきます。
Pods
[root@node1 helm]# kubectl get pods -n test
NAME READY STATUS RESTARTS AGE
isilontest-0 1/1 Running 0 2m24s
[root@node1 helm]#
[root@node1 helm]# kubectl describe pods -n test isilontest-0
Name: isilontest-0
Namespace: test
Priority: 0
PriorityClassName:
Node: node3/172.16.26.26
Start Time: Fri, 10 Jan 2020 11:35:51 +0900
Labels: app=isilontest
controller-revision-hash=isilontest-744dc88545
statefulset.kubernetes.io/pod-name=isilontest-0
Annotations:
Status: Running
IP: 172.16.26.26
Controlled By: StatefulSet/isilontest
Containers:
test:
Container ID: docker://a61daef38de38ed850a6864701465b8bb4fe34903879d88f519be02708ddf1c1
Image: docker.io/centos:latest
Image ID: docker-pullable://centos@sha256:f94c1d992c193b3dc09e297ffd54d8a4f1dc946c37cbeceb26d35ce1647f88d9
Port:
Host Port:
Command:
/bin/sleep
3600
State: Running
Started: Fri, 10 Jan 2020 11:36:02 +0900
Ready: True
Restart Count: 0
Environment:
Mounts:
/data0 from pvol0 (rw)
/data1 from pvol1 (rw)
/var/run/secrets/kubernetes.io/serviceaccount from isilontest-token-vm75m (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
pvol0:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: pvol0
ReadOnly: false
pvol1:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: pvol1
ReadOnly: false
isilontest-token-vm75m:
Type: Secret (a volume populated by a Secret)
SecretName: isilontest-token-vm75m
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 2m50s (x5 over 3m1s) default-scheduler pod has unbound immediate PersistentVolumeClaims (repeated 2 times)
Normal Scheduled 2m41s default-scheduler Successfully assigned test/isilontest-0 to node3
Normal SuccessfulAttachVolume 2m41s attachdetach-controller AttachVolume.Attach succeeded for volume "k8s-df3e5dbb33"
Normal SuccessfulAttachVolume 2m41s attachdetach-controller AttachVolume.Attach succeeded for volume "k8s-df3ebd2c33"
Normal Pulling 2m34s kubelet, node3 Pulling image "docker.io/centos:latest"
Normal Pulled 2m30s kubelet, node3 Successfully pulled image "docker.io/centos:latest"
Normal Created 2m30s kubelet, node3 Created container test
Normal Started 2m30s kubelet, node3 Started container test
[root@node1 helm]#
PVC
[root@node1 helm]# kubectl get pvc -n test
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
pvol0 Bound k8s-df3ebd2c33 8Gi RWO isilon 3m43s
pvol1 Bound k8s-df3e5dbb33 12Gi RWO isilon 3m43s
[root@node1 helm]#
PV
[root@node1 helm]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
k8s-df3e5dbb33 12Gi RWO Delete Bound test/pvol1 isilon 4m36s
k8s-df3ebd2c33 8Gi RWO Delete Bound test/pvol0 isilon 4m46s
[root@node1 helm]#
Volumeのマウント状況を確認
[root@node1 helm]# kubectl exec -n test isilontest-0 df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 36805060 3090888 33714172 9% /
tmpfs 65536 0 65536 0% /dev
tmpfs 1940168 0 1940168 0% /sys/fs/cgroup
172.16.27.157:/ifs/data/csi/k8s-df3ebd2c33 231972494336 21612544 223778393088 1% /data0
172.16.27.157:/ifs/data/csi/k8s-df3e5dbb33 231972494336 21612544 223778393088 1% /data1
/dev/mapper/centos-root 36805060 3090888 33714172 9% /etc/hosts
shm 65536 0 65536 0% /dev/shm
tmpfs 1940168 12 1940156 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 1940168 0 1940168 0% /proc/acpi
tmpfs 1940168 0 1940168 0% /proc/scsi
tmpfs 1940168 0 1940168 0% /sys/firmware
[root@node1 helm]#
Isilon側
H400-1# ls -l /ifs/data/csi
total 64
drwxrwxrwx 2 root wheel 0 Jan 10 11:34 k8s-df3e5dbb33
drwxrwxrwx 2 root wheel 0 Jan 10 11:34 k8s-df3ebd2c33
H400-1#
H400-1#
H400-1# isi nfs exports list
ID Zone Paths Description
-------------------------------------------------------
1 System /ifs Default export
2 System /ifs/data/csi/k8s-df3ebd2c33 -
3 System /ifs/data/csi/k8s-df3e5dbb33 -
-------------------------------------------------------
Total: 3
H400-1#
H400-1# isi nfs exports view 2
ID: 2
Zone: System
Paths: /ifs/data/csi/k8s-df3ebd2c33
Description: -
Clients: 172.16.26.26
Root Clients: -
Read Only Clients: -
Read Write Clients: -
All Dirs: No
Block Size: 8.0k
Can Set Time: Yes
Case Insensitive: No
Case Preserving: Yes
Chown Restricted: No
Commit Asynchronous: No
Directory Transfer Size: 128.0k
Encoding: DEFAULT
Link Max: 32767
Map Lookup UID: No
Map Retry: Yes
Map Root
Enabled: True
User: nobody
Primary Group: -
Secondary Groups: -
Map Non Root
Enabled: False
User: nobody
Primary Group: -
Secondary Groups: -
Map Failure
Enabled: False
User: nobody
Primary Group: -
Secondary Groups: -
Map Full: Yes
Max File Size: 8192.00000P
Name Max Size: 255
No Truncate: No
Read Only: No
Readdirplus: Yes
Readdirplus Prefetch: 10
Return 32Bit File Ids: No
Read Transfer Max Size: 1.00M
Read Transfer Multiple: 512
Read Transfer Size: 128.0k
Security Type: unix
Setattr Asynchronous: No
Snapshot: -
Symlinks: Yes
Time Delta: 1.0 ns
Write Datasync Action: datasync
Write Datasync Reply: datasync
Write Filesync Action: filesync
Write Filesync Reply: filesync
Write Unstable Action: unstable
Write Unstable Reply: unstable
Write Transfer Max Size: 1.00M
Write Transfer Multiple: 512
Write Transfer Size: 512.0k
H400-1#
H400-1# isi nfs exports view 3
ID: 3
Zone: System
Paths: /ifs/data/csi/k8s-df3e5dbb33
Description: -
Clients: 172.16.26.26
Root Clients: -
Read Only Clients: -
Read Write Clients: -
All Dirs: No
Block Size: 8.0k
Can Set Time: Yes
Case Insensitive: No
Case Preserving: Yes
Chown Restricted: No
Commit Asynchronous: No
Directory Transfer Size: 128.0k
Encoding: DEFAULT
Link Max: 32767
Map Lookup UID: No
Map Retry: Yes
Map Root
Enabled: True
User: nobody
Primary Group: -
Secondary Groups: -
Map Non Root
Enabled: False
User: nobody
Primary Group: -
Secondary Groups: -
Map Failure
Enabled: False
User: nobody
Primary Group: -
Secondary Groups: -
Map Full: Yes
Max File Size: 8192.00000P
Name Max Size: 255
No Truncate: No
Read Only: No
Readdirplus: Yes
Readdirplus Prefetch: 10
Return 32Bit File Ids: No
Read Transfer Max Size: 1.00M
Read Transfer Multiple: 512
Read Transfer Size: 128.0k
Security Type: unix
Setattr Asynchronous: No
Snapshot: -
Symlinks: Yes
Time Delta: 1.0 ns
Write Datasync Action: datasync
Write Datasync Reply: datasync
Write Filesync Action: filesync
Write Filesync Reply: filesync
Write Unstable Action: unstable
Write Unstable Reply: unstable
Write Transfer Max Size: 1.00M
Write Transfer Multiple: 512
Write Transfer Size: 512.0k
H400-1#
Kubernetes上でのオペレーションだけでIsilon側にディレクトリが作成され、NFS exportが設定されることが確認できました。
データの永続性の確認
ファイルを作成して、podをscale –replicas=0 → 1で再作成して確認してみます。
[root@node1 helm]# kubectl exec -n test isilontest-0 -- touch /data0/aaa
[root@node1 helm]# kubectl exec -n test isilontest-0 -- ls -l /data0
total 24
-rw-r--r-- 1 nobody nobody 0 Jan 10 02:56 aaa
[root@node1 helm]# kubectl exec -n test isilontest-0 -- touch /data1/bbb
[root@node1 helm]# kubectl exec -n test isilontest-0 -- ls -l /data1
total 24
-rw-r--r-- 1 nobody nobody 0 Jan 10 02:57 bbb
[root@node1 helm]#
[root@node1 helm]# kubectl get pods -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
isilontest-0 1/1 Running 0 24m 172.16.26.26 node3
[root@node1 helm]# kubectl scale -n test statefulset isilontest --replicas=0
statefulset.apps/isilontest scaled
[root@node1 helm]# kubectl get pods -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
isilontest-0 1/1 Terminating 0 24m 172.16.26.26 node3
[root@node1 helm]# kubectl get pods -n test -o wide
No resources found.
[root@node1 helm]# kubectl scale -n test statefulset isilontest --replicas=1
statefulset.apps/isilontest scaled
[root@node1 helm]# kubectl get pods -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
isilontest-0 0/1 ContainerCreating 0 6s 172.16.26.26 node3
[root@node1 helm]# kubectl get pods -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
isilontest-0 1/1 Running 0 30s 172.16.26.26 node3
[root@node1 helm]#
ファイルに影響はありません。
[root@node1 helm]# kubectl exec -n test isilontest-0 -- ls -l /data0
total 24
-rw-r--r-- 1 nobody nobody 0 Jan 10 02:56 aaa
[root@node1 helm]# kubectl exec -n test isilontest-0 -- ls -l /data1
total 24
-rw-r--r-- 1 nobody nobody 0 Jan 10 02:57 bbb
[root@node1 helm]#
Isilon側のNFS exportも変化はありません。
H400-1# isi nfs exports list
ID Zone Paths Description
-------------------------------------------------------
1 System /ifs Default export
2 System /ifs/data/csi/k8s-df3ebd2c33 -
3 System /ifs/data/csi/k8s-df3e5dbb33 -
-------------------------------------------------------
Total: 3
H400-1#
では、podを別のnodeに移動させた場合はどうなるでしょうか。Isilon側のNFS exportにはclientのIP制限が入っているのでここも併せて確認します。
H400-1# isi nfs export view 2 | grep Client
Clients: 172.16.26.26
Root Clients: -
Read Only Clients: -
Read Write Clients: -
H400-1# isi nfs export view 3 | grep Client
Clients: 172.16.26.26
Root Clients: -
Read Only Clients: -
Read Write Clients: -
H400-1#
drainでpodを移動させます。
[root@node1 helm]# kubectl get node
NAME STATUS ROLES AGE VERSION
node1 Ready master 21d v1.14.10
node2 Ready 21d v1.14.10
node3 Ready 21d v1.14.10
[root@node1 helm]#
[root@node1 helm]# kubectl get pods -n test --show-labels
NAME READY STATUS RESTARTS AGE LABELS
isilontest-0 1/1 Running 0 10m app=isilontest,controller-revision-hash=isilontest-744dc88545,statefulset.kubernetes.io/pod-name=isilontest-0
[root@node1 helm]#
[root@node1 helm]# kubectl get pods -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
isilontest-0 1/1 Running 0 10m 172.16.26.26 node3
[root@node1 helm]#
[root@node1 helm]#
[root@node1 helm]# kubectl drain node3 --pod-selector='app=isilontest'
node/node3 cordoned
evicting pod "isilontest-0"
pod/isilontest-0 evicted
node/node3 evicted
[root@node1 helm]# kubectl get pods -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
isilontest-0 0/1 ContainerCreating 0 7s 172.16.26.25 node2
[root@node1 helm]# kubectl get pods -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
isilontest-0 1/1 Running 0 19s 172.16.26.25 node2
[root@node1 helm]#
[root@node1 helm]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
node1 Ready master 21d v1.14.10
node2 Ready 21d v1.14.10
node3 Ready,SchedulingDisabled 21d v1.14.10
[root@node1 helm]#
[root@node1 helm]# kubectl uncordon node3
node/node3 uncordoned
[root@node1 helm]#
[root@node1 helm]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
node1 Ready master 21d v1.14.10
node2 Ready 21d v1.14.10
node3 Ready 21d v1.14.10
[root@node1 helm]#
isilontest-0はnode2に移動したようです。Isilon側はどうでしょう。
H400-1# isi nfs exports list
ID Zone Paths Description
-------------------------------------------------------
1 System /ifs Default export
2 System /ifs/data/csi/k8s-df3ebd2c33 -
3 System /ifs/data/csi/k8s-df3e5dbb33 -
-------------------------------------------------------
Total: 3
H400-1# isi nfs export view 2 | grep Client
Clients: 172.16.26.25
Root Clients: -
Read Only Clients: -
Read Write Clients: -
H400-1# isi nfs export view 3 | grep Client
Clients: 172.16.26.25
Root Clients: -
Read Only Clients: -
Read Write Clients: -
H400-1#
NFS exportのclientのIP制限はnode2のIPに自動的に変更されています。
ファイルの永続性についても問題ありません。
[root@node1 helm]# kubectl exec -n test isilontest-0 -- ls -l /data0
total 24
-rw-r--r-- 1 nobody nobody 0 Jan 10 02:56 aaa
[root@node1 helm]# kubectl exec -n test isilontest-0 -- ls -l /data1
total 24
-rw-r--r-- 1 nobody nobody 0 Jan 10 02:57 bbb
[root@node1 helm]#
scale outさせた場合はどうなるでしょうか。見ていきます。
[root@node1 helm]# kubectl scale -n test statefulset isilontest --replicas=2
statefulset.apps/isilontest scaled
[root@node1 helm]# kubectl get pods -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
isilontest-0 1/1 Running 0 8m41s 172.16.26.25 node2
isilontest-1 0/1 ContainerCreating 0 83s 172.16.26.26 node3
[root@node1 helm]#
isilontest-1はContainerCreatingで止まってしまいます。describeを見てみます。
[root@node1 helm]# kubectl describe pods -n test isilontest-1
Name: isilontest-1
Namespace: test
Priority: 0
PriorityClassName:
Node: node3/172.16.26.26
Start Time: Fri, 10 Jan 2020 12:20:17 +0900
Labels: app=isilontest
controller-revision-hash=isilontest-744dc88545
statefulset.kubernetes.io/pod-name=isilontest-1
Annotations:
Status: Pending
IP: 172.16.26.26
Controlled By: StatefulSet/isilontest
Containers:
test:
Container ID:
Image: docker.io/centos:latest
Image ID:
Port:
Host Port:
Command:
/bin/sleep
3600
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment:
Mounts:
/data0 from pvol0 (rw)
/data1 from pvol1 (rw)
/var/run/secrets/kubernetes.io/serviceaccount from isilontest-token-vm75m (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
pvol0:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: pvol0
ReadOnly: false
pvol1:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: pvol1
ReadOnly: false
isilontest-token-vm75m:
Type: Secret (a volume populated by a Secret)
SecretName: isilontest-token-vm75m
Optional: false
QoS Class: BestEffort
Node-Selectors:
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 2m6s default-scheduler Successfully assigned test/isilontest-1 to node3
Warning FailedAttachVolume 2m6s attachdetach-controller Multi-Attach error for volume "k8s-df3ebd2c33" Volume is already used by pod(s) isilontest-0
Warning FailedAttachVolume 2m6s attachdetach-controller Multi-Attach error for volume "k8s-df3e5dbb33" Volume is already used by pod(s) isilontest-0
Warning FailedMount 3s kubelet, node3 Unable to mount volumes for pod "isilontest-1_test(203a17a7-3358-11ea-a609-0050569cf7a7)": timeout expired waiting for volumes to attach or mount for pod "test"/"isilontest-1". list of unmounted volumes=[pvol0 pvol1]. list of unattached volumes=[pvol0 pvol1 isilontest-token-vm75m]
[root@node1 helm]#
Multi-Attach errorの為マウントができないようです。これはPVCのACCESS MODESがRWO(ReadWriteOnce)で設定されている為です。
[root@node1 helm]# kubectl get pvc -A
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test pvol0 Bound k8s-df3ebd2c33 8Gi RWO isilon 48m
test pvol1 Bound k8s-df3e5dbb33 12Gi RWO isilon 48m
[root@node1 helm]#
ACCESS MODESをRWX(ReadWriteMany)に変更すると解消できそうです。対象となるHelm Chartは下記の2つの様です。
[root@node1 helm]# cat 2vols/templates/pvc0.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvol0
namespace: {{ .Values.namespace }}
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 8Gi
storageClassName: isilon
[root@node1 helm]# cat 2vols/templates/pvc1.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvol1
namespace: {{ .Values.namespace }}
spec:
accessModes:
- ReadWriteOnce
volumeMode: Filesystem
resources:
requests:
storage: 12Gi
storageClassName: isilon
[root@node1 helm]#
Helm Chartを変更して試してみます。変更の前に現在のテストを終了させ関連リソースを削除します。
[root@node1 helm]# sh ./stoptest.sh -t 2vols
release "2vols" deleted
NAME READY STATUS RESTARTS AGE
isilontest-0 1/1 Terminating 0 37m
waiting for persistent volumes to be cleaned up
No resources found.
deleting...
No resources found.
[root@node1 helm]#
[root@node1 helm]# kubectl get pods -n test
No resources found.
[root@node1 helm]#
[root@node1 helm]# kubectl get pvc -A
No resources found.
[root@node1 helm]#
[root@node1 helm]# kubectl get pv
No resources found.
[root@node1 helm]#
テストを終了させるとPod,PVC,PVが削除されます。
Helm Chart内のReadWriteOnceをReadWriteManyに変更します。
[root@node1 helm]# sed -i 's/ReadWriteOnce/ReadWriteMany/' 2vols/templates/pvc0.yaml
[root@node1 helm]# sed -i 's/ReadWriteOnce/ReadWriteMany/' 2vols/templates/pvc1.yaml
再度テストスクリプトを実行します。
[root@node1 helm]# sh ./starttest.sh -t 2vols -n test
[root@node1 helm]# kubectl get pods -n test
NAME READY STATUS RESTARTS AGE
isilontest-0 1/1 Running 0 81s
[root@node1 helm]# kubectl get pvc -A
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
test pvol0 Bound k8s-dfab9ff933 8Gi RWX isilon 85s
test pvol1 Bound k8s-dfabab2733 12Gi RWX isilon 85s
[root@node1 helm]# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
k8s-dfab9ff933 8Gi RWX Delete Bound test/pvol0 isilon 92s
k8s-dfabab2733 12Gi RWX Delete Bound test/pvol1 isilon 82s
[root@node1 helm]#
PVCのACCESS MODESがRWXに変更されました。
ファイルの状況を確認します。
[root@node1 helm]# kubectl exec -n test isilontest-0 -- ls -l /data0
total 0
[root@node1 helm]# kubectl exec -n test isilontest-0 -- ls -l /data1
total 0
[root@node1 helm]#
PVCのRECLAIM POLICYがDeleteの為Volumeは削除、再作成されファイルは残っていない状態です。
Isilon側もディレクトリ、NFS exportが削除、再作成されています。
H400-1# ls -l /ifs/data/csi
total 64
drwxrwxrwx 2 root wheel 0 Jan 10 12:52 k8s-dfab9ff933
drwxrwxrwx 2 root wheel 0 Jan 10 12:53 k8s-dfabab2733
H400-1# isi nfs export list
ID Zone Paths Description
-------------------------------------------------------
1 System /ifs Default export
4 System /ifs/data/csi/k8s-dfab9ff933 -
5 System /ifs/data/csi/k8s-dfabab2733 -
-------------------------------------------------------
Total: 3
H400-1#
ファイルを作成してpodをscale outさせてみます。
[root@node1 helm]# kubectl exec -n test isilontest-0 -- touch /data0/AAA
[root@node1 helm]# kubectl exec -n test isilontest-0 -- ls -l /data0
total 24
-rw-r--r-- 1 nobody nobody 0 Jan 10 04:00 AAA
[root@node1 helm]# kubectl exec -n test isilontest-0 -- touch /data1/BBB
[root@node1 helm]# kubectl exec -n test isilontest-0 -- ls -l /data1
total 24
-rw-r--r-- 1 nobody nobody 0 Jan 10 04:00 BBB
[root@node1 helm]#
[root@node1 helm]# kubectl get node
NAME STATUS ROLES AGE VERSION
node1 Ready master 21d v1.14.10
node2 Ready 21d v1.14.10
node3 Ready 21d v1.14.10
[root@node1 helm]#
[root@node1 helm]# kubectl get pods -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
isilontest-0 1/1 Running 0 8m27s 172.16.26.26 node3
[root@node1 helm]#
[root@node1 helm]# kubectl scale -n test statefulset isilontest --replicas=2
statefulset.apps/isilontest scaled
[root@node1 helm]#
[root@node1 helm]# kubectl get pods -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
isilontest-0 1/1 Running 0 8m48s 172.16.26.26 node3
isilontest-1 0/1 ContainerCreating 0 7s 172.16.26.25 node2
[root@node1 helm]# kubectl get pods -n test -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
isilontest-0 1/1 Running 0 9m9s 172.16.26.26 node3
isilontest-1 1/1 Running 0 28s 172.16.26.25 node2
[root@node1 helm]#
scale outして作成されたpod(isilontest-1)がRunningとなりました。ファイルを確認してみます。
[root@node1 helm]# kubectl exec -n test isilontest-1 -- ls -l /data0
total 24
-rw-r--r-- 1 nobody nobody 0 Jan 10 04:00 AAA
[root@node1 helm]# kubectl exec -n test isilontest-1 -- ls -l /data1
total 24
-rw-r--r-- 1 nobody nobody 0 Jan 10 04:00 BBB
[root@node1 helm]#
同じvolumeをマウントできています。
Isilon側はNFS exportのclientのIP制限にscale outで作成されたpodが稼働しているnodeのIPが追加されています。
H400-1# isi nfs export view 4 | grep Client
Clients: 172.16.26.25, 172.16.26.26
Root Clients: -
Read Only Clients: -
Read Write Clients: -
H400-1# isi nfs export view 5 | grep Client
Clients: 172.16.26.25, 172.16.26.26
Root Clients: -
Read Only Clients: -
Read Write Clients: -
H400-1#
まとめ
前回と今回の記事でCSI Driver for Dell EMC Isilonのインストールの流れと、付属のテストの動作をご紹介いたしました。このCSI Driverをインストールすることで、高可用性や拡張性を備えたDell EMC IsilonをKubernetesの永続性のあるストレージとして簡単に利用できるようになることが分かりました。CSI Driver for Dell EMC Isilonはすでにproduction-gradeとなっていますので、Dell EMC Isilonの新しい利用用途としてご検討頂ければと思います。
ブロードバンドタワーのエンジニアブログ『 Tower of Engineers 』で公開されていた記事をnoteに再投稿させていただきました。
過去(2020年2月25日掲載)のものではありますが、皆様に再び楽しんでいただけると嬉しいです。これからも、価値ある記事や興味深い内容を、noteでシェアさせていただく予定です。どうぞ楽しみにお待ちください。
X（旧Twitter）でも情報を発信しております。ぜひフォローお願いします！
