Database persistent volume is filling

Symptom

A CustomerDatabasePersistentVolumeFillingUp alert is firing.

CustomerDatabasePersistentVolumeFillingUp alerts are sent to subscribers via email.

Observations

Here are the details on the alerts:

Alert: CustomerDatabasePersistentVolumeFillingUp

Expression

predict_linear(kubelet_volume_stats_available_bytes{persistentvolumeclaim=~"data.*"}[24h], 8 * 3600) < 0 and (kubelet_volume_stats_available_bytes{persistentvolumeclaim=~"data.*"} / kubelet_volume_stats_capacity_bytes{persistentvolumeclaim=~"data.*"}) * 100 < 15

Delay

30 minutes

Labels

team: customer

Annotations

  • summary

  • description

  • tenant

  • cluster_id

  • cluster_name

  • namespace

  • pvc

Determine current size of database persistent volume

The name of the database persistent volume claim and its namespace are contained in the alert, for example "database-prod-magnolia-helm-public-db-0".

You can find the details on the persistent volume by displaying the persistent volume claim using kubectl:

kubectl -n <namespace from alert> get pvc <persistent volume claim from alert>

Executing the above will list the size of the persistent volume shown in the CAPACITY column.

$ kubectl -n prod get pvc database-prod-magnolia-helm-public-db-0
NAME                                     STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
database-prod-magnolia-helm-public-db-0   Bound    pvc-e695274c-5794-4a78-afe0-e86b0efd40ed   10Gi       RWO            local-path     9d

Solutions

This section provides solutions that should help resolve the issue in most cases.

Increase the capacity of the database persistent volume

The space allocated to the Database persistent volume can be increased with a "hot fix" or by modifying the values of the Helm chart deploying Magnolia.

You can edit the persistent volume claim and increase the capacity in Rancher or with kubectl.

Edit the persistent volume claim or the persistent volume associated with the claim and increase the size of the storage resource.

  resources:
    requests:
      storage: 10Gi (1)
1 Increase the storage property.
The storage resource for the persistent volume claim is not permanent. It may be overridden when Magnolia is redeployedwith the Helm chart.

Increase capacity via Helm chart

  1. Set the Helm chart value db.persistence.size to an appropriate value.

    The default value of db.persistence.size is 10Gi.

  2. Redeploy Magnolia.

Feedback

PaaS

×

Location

This widget lets you know where you are on the docs site.

You are currently perusing through the Magnolia PaaS docs.

Main doc sections

DX Core Headless PaaS Legacy Cloud Incubator modules