Cluster filesystem filling up

Symptom

A CustomerClusterFilesystemFillingUp alert is firing. It indicates a filesystem on a node in one of your cluster(s) is almost full.

CustomerClusterFilesystemFillingUp alerts are sent to subscribers via email.

Observations

Here are the details on the alerts:

Alert: CustomerClusterFilesystemFillingUp

Expression

predict_linear(node_filesystem_avail_bytes[24h], 8 * 3600) < 0 and instance:node_filesystem_avail:ratio < 0.3

Delay

30 minutes

Labels

team: customer

Annotations

  • summary

  • description

  • tenant

  • cluster_id

  • cluster_name

  • instance

  • device

  • mountpoint

Determine the cluster node and filesystem filling up

The alert contains:

  • the cluster name/id (the k8s_cluster_name and k8s_cluster_id labels)

  • the cluster node (the instance label)

  • the device (the device label)

  • the filesystem (the mountpoint label)

It’s important to note the values of these labels.

Solutions

This section provides solutions that should help resolve the issue in most cases.

Increase device size for cluster

You can increase the size of the device used by the cluster node in AWS.

  1. Use the cluster node value (instance label) to find the node and the device (device label) running out of space.

  2. Increase the device size in the AWS EC2 web interface or AWS command line.

Contact the PaaS Helpdesk

If you have tried the above solutions(), and nothing has worked, reach out to us directly via the DX Cloud Helpdesk.

Feedback

PaaS

×

Location

This widget lets you know where you are on the docs site.

You are currently perusing through the Magnolia PaaS docs.

Main doc sections

DX Core Headless PaaS Legacy Cloud Incubator modules