Certificate is expiring

Symptom

A CustomerCertificateExpiring alert is firing.

CustomerCertificateExpiring alerts are are sent to subscribers via email.

Observations

Here are the details on the alerts:

Alert: CustomerCertificateExpiring

Expression

(avg(nginx_ingress_controller_ssl_expire_time_seconds{kubernetes_pod_name=".",namespace=".",ingress=~".*"}) by (tenant,host,k8s_cluster_name,k8s_cluster_id) - time()) / (24*60*60) < 14

Delay

0 minutes

Labels

team: customer

Annotations

  • host

  • summary

  • description

  • tenant

  • cluster_id

  • cluster_name

Determine the ingress(es) using the certificate

The alert notes the host certificate that is expiring (or has expired) and the cluster where the ingress is running. Find the ingress using the certificate on the cluster noted in the alert.

Here’s how to find the ingress using kubectl:

kubectl config use-context <cluster in alert>
kubectl get ingress -A | grep "<host in alert>"

Results

Note the namespace of the ingress in the first column and the ingress name in the second column.

$ kubectl get ingress -A | grep "www.example.com"
prod                 bl-example             <none>   www.example.com                                                             a3052a43267a91491a437.......a6503628baf.elb.eu-central-1.amazonaws.com   80, 443   255d
prod                 cm-acme-example  nginx    www.example.com                                                             a3052a43267a9491491a43.......58a6503628baf.elb.eu-central-1.amazonaws.com   80        8d

There are two ingresses using the certificate.

  • prod/bl-example is the nginx ingress actually using the certificate. This ingress actually uses the expiring certificate.

  • prod/cm-acme-example - is a temporary ingress used for an HTTP acme challenge (more on this later).

Determine certificate issuer used by the ingress

The ingress you found in the previous observation will show the certificate issuer in an annotation of the ingress.

You can display the annotations of the ingress as follows:

kubectl -n <namespace of the ingress> describe ingress <ingress>

or:

kubectl -n <namespace of the ingress> get ingress <ingress> -o yaml
cert-manager.io/cluster-issuer: letsencrypt-prod (1)
1 The certificate issuer - cert-manager.io/cluster-issuer.

Determine the renewal method of the cluster issuer

The "solver" configuration of the cluster issuer determines how the certificate is renewed.

You can display the solver configuration with:

kubectl -n cert-manager describe clusterissuer <clusterissuer>

or

kubectl -n cert-manager get clusterissuer <cluster issuer> -o yaml

In either case, look for the solver section in the spec:

    solvers:
    - http01:

or:

    solvers:
    - dns01:

Solutions

This section provides solutions that should help resolve the issue in most cases.

Renew certificate (dns01)

You can renew certificates directly from the Cockpit.

dns01 certificates can also be renewed with a DNS acme challenge. To renew the certificate using DNS, a TXT record for the domain _acme-challenge.<YOUR_DOMAIN> must be added to the DNS record.

Feedback

PaaS

×

Location

This widget lets you know where you are on the docs site.

You are currently perusing through the Magnolia PaaS docs.

Main doc sections

DX Core Headless PaaS Legacy Cloud Incubator modules