The Pod is stuck on ContainerCreating
The Pod is stuck on ContainerCreating
Kubernetes
I am running a blog that is deployed in the Kubernetes cluster. And as with every self-hosted application, it requires a bit of maintenance from time to time. Every application has bugs and those bugs get fixed at some point. It is good to update an application regularly to apply all the necessary fixes. It is really important as it reduces the likelihood of the application being exploited. Plus probably there are new features as well.
Rolling update
Not hesitating too much I updated Helm Charts and triggered an update.
❯ helm upgrade ghost bitnami/ghost --namespace ghost-test --version 19.1.67 -f values.ymlNew Pod creation has started.
❯ kubectl get pods
NAME READY STATUS RESTARTS AGE
ghost-7d4f67c998-bbfxg 1/1 Running 5 (40m ago) 44h
ghost-ff4b84cfd-hvwbz 0/1 ContainerCreating 0 44mBut my excitement fainted really quickly as I noticed that Pod got stuck in the ContainerCreating state indefinitely.
❯ kubectl get events
2m11s Normal Scheduled pod/ghost-ff4b84cfd-hvwbz Successfully assigned ghost-test/ghost-ff4b84cfd-hvwbz to vm0102
3s Warning FailedMount pod/ghost-ff4b84cfd-hvwbz MountVolume.MountDevice failed for volume "pvc-6cbd418d-b9bd-410c-90c5-822123c5df94" : rpc error: code = Internal desc = Volume pvc-6cbd418d-b9bd-410c-90c5-822123c5df94 still mounted on node vm0202
8s Warning FailedMount pod/ghost-ff4b84cfd-hvwbz Unable to attach or mount volumes: unmounted volumes=[ghost-data], unattached volumes=[kube-api-access-9cqkw ghost-data]: timed out waiting for the condition
2m11s Normal SuccessfulCreate replicaset/ghost-ff4b84cfd Created pod: ghost-ff4b84cfd-hvwbzIt turned out that the newly spawned Pod wanted to mount the volume that had been already mounted but on the older Pod.
❯ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-ghost-mysql-0 Bound pvc-4ed9e42c-920a-4032-b915-a114061eef31 8Gi RWO openebs-cstor-csi-default 45h
ghost Bound pvc-6cbd418d-b9bd-410c-90c5-822123c5df94 8Gi RWO openebs-cstor-csi-default 45hPrinting PersistentVolumeClaims showed the volumes were mounted in ReadWriteOnce (RWO) mode, so there was no way they could have been mounted on another node hosting the newly created Pod.
💡 My Kubernetes cluster has OpenEBS cStor installed and configured for Dynamic Volume provisioning.
One option would be not to scale the number of Pods during an update. So Terminate an old Pod and create a new one. But this would result in an outage of service for a while. So why not mount the volume in ReadWriteMany (RWX) mode?
The failed approach
I uninstalled ghost and then updated the values.yml file with the persistence section to force RWX.
❯ cat values.yml
ghostUsername: 'kuba'
ghostPassword: '12345abcde'
ghostEmail: 'jakub@slys.dev'
ghostBlogTitle: 'slys.dev'
ghostHost: 'blog.slys'
allowEmptyPassword: false
ghostSkipInstall: false
ingress:
enabled: true
hostname: 'blog.slys'
ingressClassName: 'nginx'
service:
type: 'ClusterIP'
startupProbe:
enabled: true
periodSeconds: 30
failureThreshold: 10
image:
debug: true
mysql:
auth:
password: '5S3MjHA0QQ'
rootPassword: 'k72H3TCqnA'
persistence:
accessModes:
- ReadWriteManyApplied helm again.
❯ helm upgrade --install ghost bitnami/ghost --namespace ghost-test --version 19.1.60 -f values.yml
Release "ghost" does not exist. Installing it now.
NAME: ghost
LAST DEPLOYED: Tue Feb 7 12:32:14 2023
NAMESPACE: ghost-test
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
CHART NAME: ghost
CHART VERSION: 19.1.60
APP VERSION: 5.31.0
** Please be patient while the chart is being deployed **
1. Get the Ghost URL and associate its hostname to your cluster external IP:
export CLUSTER_IP=$(minikube ip) # On Minikube.
Use: `kubectl cluster-info` on others K8s clusters echo "Ghost URL: http://blog.slys"
echo "$CLUSTER_IP blog.slys" | sudo tee -a /etc/hosts
2. Get your Ghost login credentials by running:
echo Email: jakub@slys.dev
echo Password: $(kubectl get secret --namespace ghost-test ghost -o jsonpath="{.data.ghost-password}" | base64 -d)Pods started to create. However, after a moment I realized that created Pod was stuck, again.
❯ kubectl get events
8s Normal Provisioning persistentvolumeclaim/ghost External provisioner is provisioning volume for claim "ghost-test/ghost"
2s Normal ExternalProvisioning persistentvolumeclaim/ghost waiting for a volume to be created, either by external provisioner "cstor.csi.openebs.io" or manually created by system administrator
8s Warning ProvisioningFailed persistentvolumeclaim/ghost failed to provision volume with StorageClass "openebs-cstor-csi-default": rpc error: code = InvalidArgument desc = only SINGLE_NODE_WRITER supported, unsupported access mode requested: MULTI_NODE_MULTI_WRITERPrinting events exposed an issue - the current provisioner is unable to satisfy the request.
Dynamic NFS Provisioner
The remedy to those stuck Pods was to install another dynamic volume provisioner in the cluster - the NFS provisioner.
❯ helm upgrade openebs openebs/openebs -n openebs \
--set cstor.enabled=true \
--set nfs-provisioner.enabled=true \
--namespace openebsAfter a while, a Pod with an NFS provisioner was up and running.
❯ kubectl get pods -n openebs
NAME READY STATUS RESTARTS AGE
openebs-nfs-provisioner-787d694555-8k72g 1/1 Running 0 24dAlso, a new StorageClass was installed.
❯ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
openebs-kernel-nfs openebs.io/nfsrwx Delete Immediate false 24dThe last bit was to update the values.yml file to point to the proper StorageClass.
❯ cat values.yml
...
persistence:
storageClass: 'openebs-kernel-nfs'
accessModes:
- ReadWriteManyAnd then install it again.
❯ helm install ghost bitnami/ghost --namespace ghost-test --create-namespace --version 19.1.60 -f values.ymlNow, volume was provisioned with desired Access Mode - RWX, at last!
❯ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-ghost-mysql-0 Bound pvc-647eb4c2-b113-4bde-bdf6-1276c83caad1 8Gi RWO openebs-cstor-csi-default 10m
ghost Bound pvc-c0f9610e-b4d1-4fa3-b06e-e451113b7568 8Gi RWX openebs-kernel-nfs 10mBut still, the Pod was stuck...
❯ kubectl get pods
NAME READY STATUS RESTARTS AGE
ghost-7d4f67c998-6cznw 0/1 Running 2 (108s ago) 4m58s
ghost-mysql-0 1/1 Running 1 (2m ago) 4m58sDescribing that miserable Pod exposed yet another issue.
❯ kubectl describe pod ghost-7d4f67c998-6cznw
Name: ghost-7d4f67c998-6cznw
Namespace: ghost-test
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 6m kubelet MountVolume.SetUp failed for volume "pvc-c0f9610e-b4d1-4fa3-b06e-e451113b7568" : mount failed: exit status 32 Mounting command: mount Mounting arguments: -t nfs 10.97.190.112:/ /var/lib/kubelet/pods/aa9e7549-c24d-4ca2-9d53-f63e98ef3a87/volumes/kubernetes.io~nfs/pvc-c0f9610e-b4d1-4fa3-b06e-e451113b7568 Output: mount.nfs: access denied by server while mounting 10.97.190.112:/
Normal Pulled 3m14s (x3 over 5m59s) kubelet Container image "docker.io/bitnami/ghost:5.31.0-debian-11-r0" already present on machineThe Pod doesn't have enough privileges to mount NFS resources.
Making it right
Due to security reasons, we don't want Pods to be running in privileged mode as the root user. So outstanding issue to crack was granting proper privileges.
I had to add an annotation in values.yml for OpenEBS to provision volume with correct access rights for the user 1001.
persistence:
storageClass: 'openebs-kernel-nfs'
accessModes:
- ReadWriteMany
annotations:
cas.openebs.io/config: |
- name: FilePermissions
data:
GID: '1001'
UID: '1001'
mode: "g+s"The last thing was to update the security context to run as the user 1001.
volumePermissions:
enabled: true
securityContext:
runAsUser: 1001
runAsGroup: 1001And voila! Pod was up and running and the volume was provisioned in RWX mode.
❯ kubectl get pods
NAME READY STATUS RESTARTS AGE
ghost-9bd7dbb8c-llc9p 1/1 Running 2 (4m52s ago) 8m1s
ghost-mysql-0 1/1 Running 1 (5m1s ago) 8m1s❯ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
data-ghost-mysql-0 Bound pvc-c63e0252-f30b-455c-9d08-585dbe69cb17 8Gi RWO openebs-cstor-csi-default 9m15s
ghost Bound pvc-0b6869e7-a258-43a5-b805-fa53e506d6e1 8Gi RWX openebs-kernel-nfs 9m15sNow was the time to test out a rolling update.
❯ helm upgrade ghost bitnami/ghost --namespace ghost-test --version 19.1.66 -f values.yml We could see as before, a new Pod was created, but now it didn't get stuck in the ContainerCreating state.
❯ kubectl get pods
NAME READY STATUS RESTARTS AGE
ghost-75559f9d44-zwt7m 0/1 Running 0 22s
ghost-9bd7dbb8c-llc9p 1/1 Running 2 (8m34s ago) 11m
ghost-mysql-0 0/1 Running 0 18sOnce the new Pod was initialized, the old one was Terminated.
❯ kubectl get pods
NAME READY STATUS RESTARTS AGE
ghost-75559f9d44-zwt7m 1/1 Running 1 (2m23s ago) 3m39s
ghost-mysql-0 1/1 Running 0 3m35sConclusion
RWX access mode can be very useful when deploying an application that is not designed to run natively in a cloud environment. In our example with the Ghost application, we can see that when performing an update, a new Pod is spawned. The new Pod wants to bind to the same PersistentVolume as an old Pod is bound to. In the case of RWX, it is pretty legal, so they will share the same volume. However, if RWO was selected, newly deployed would get stuck waiting to be bound, but that would never happen.
One last remark. Let's print PersistentVolumes.
❯ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-0b6869e7-a258-43a5-b805-fa53e506d6e1 8Gi RWX Delete Bound ghost-test/ghost openebs-kernel-nfs 18m pvc-c63e0252-f30b-455c-9d08-585dbe69cb17 8Gi RWO Delete Bound ghost-test/data-ghost-mysql-0 openebs-cstor-csi-default 18mWe can see that pvc-0b6869e7-a258-43a5-b805-fa53e506d6e1 comes from ghost-test/ghost. However, pvc-f469b7a5-8fda-4823-8162-8786fe7d22a1 comes from openebs/nfs-pvc-0b6869e7-a258-43a5-b805-fa53e506d6e1. That means that the NFS provisioner uses cStor underneath to provision a volume.