Magnolia home persistent volume is almost full
Symptom
A CustomerMagnoliaHomePersistentVolumeAlmostFull
alert is firing
CustomerMagnoliaHomePersistentVolumeAlmostFull alerts are sent to subscribers via email.
|
Observations
Here are the details on the alerts:
Alert: CustomerMagnoliaHomePersistentVolumeAlmostFull
Expression |
|
Delay |
|
Labels |
|
Annotations |
|
Determine current size of Magnolia home persistent volume
The name of the persistent volume claim and its namespace are contained in the alert, for example "database-prod-magnolia-helm-public-db-0".
You can find the details on the persistent volume by displaying the persistent volume claim using kubectl:
kubectl -n <namespace from alert> get pvc <persistent volume claim from alert>
Executing the above will list the size of the persistent volume:
$ kubectl -n prod get pvc magnolia-home-prod-magnolia-helm-public-0
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
magnolia-home-prod-magnolia-helm-public-0 Bound pvc-30bb25b4-289f-4029-946e-29197a5aea44 10Gi RWO gp2 228d
The value is listed in the CAPACITY column.
|
Solutions
This section provides a solution that should help resolve the issue in most cases.
Increase the capacity of the persistent volume
The space allocated to the persistent volume can be increased by editing the persistent volume in Rancher and by modifying the values of the Helm chart deploying Magnolia.
You can edit the persistent volume claim and increase the capacity in Rancher or with kubectl
.
Edit the persistent volume claim or the persistent volume associated with the claim and increase the size of the storage resource.
-
Get your desired namespace.
kubectl -n <namespace> get pvc <pvc-name>
-
Edit the PVC.
kubectl -n <namespace> edit pvc <pvc-name>
-
In the editor, adjust the
storage
.resources: requests: storage: 10Gi (1)
1 Increase the storage
property.The storage resource for the persistent volume claim is not permanent. It may be overridden when Magnolia is redeployed with the Helm chart.
Increase capacity via Helm chart
-
Set the Helm chart value
magnoliaPublic.persistence.size
to an appropriate value.The default value of
magnoliaPublic.persistence.size
is10Gi
. -
Redeploy Magnolia.
Delete the StatefulSet
Because StatefulSet
volume definitions cannot be updated in place, you must manually update them.
-
Run the
kubectl
command to delete the statefulset.kubectl delete statefulset <name> --cascade=orphan (1)
1 Important: The --cascade=orphan
flag ensures running pods are not deleted. -
Redeploy Magnolia.