Troubleshoot backups and restores

You can troubleshoot failed backups or restores by checking the status of backup and restore objects, examining the jobs that executed them, and reviewing logs from the jobs and /or the pods created by the Jobs.

The overall troubleshooting workflow looks like this:

1. Check the object status. Use kubectl get ps-backup or kubectl get ps-restore to see the current state.
2. Review error details. Use kubectl describe to get the error message from the State Description field.
3. Examine job logs. Check the logs from the backup or restore job for detailed execution information.
4. Check source pod logs. For backups, review logs from the xtrabackup container in the source pod. For restores, view logs of the Pod created by the restore Job.
5. Verify configuration. Ensure storage configurations, credentials, and cluster settings are correct.

Refer to the following sections for details. For status field reference, see Custom resource statuses.

Backups

Check backup status

Start by checking the status of your backup objects:

kubectl get ps-backup -n <namespace>

This shows you all backup objects with their current state. The STATE column indicates whether a backup succeeded, failed, or is in progress.

Example output

NAME                                           STORAGE      DESTINATION                                                  STATE       COMPLETED   AGE
backup1                                        aws-s3       s3://operator-testing/ps-cluster1-2025-11-07-20:59:28-full   Succeeded   11m         11m
backup2                                        azure                                                                     Error                   10m
backup3                                        azure-blob   operator-testing/ps-cluster1-2025-11-07-21:00:14-full        Succeeded   10m         10m
backup4                                        gcp-cs       s3://operator-testing/ps-cluster1-2025-11-07-21:01:19-full   Succeeded   3m37s       9m29s
backup5                                        gcp-cs       s3://operator-testing/ps-cluster1-2025-11-07-21:07:20-full   Succeeded   3m17s       3m28s
cron-ps-cluster1-aws-s3-20251107211029-45srk   aws-s3       s3://operator-testing/ps-cluster1-2025-11-07-21:10:29-full   Succeeded

Check backup error details

When a backup fails, use kubectl describe to get detailed error information:

kubectl describe ps-backup <backup-name> -n <namespace>

The Status section contains the State and State Description fields that explain why the backup failed.

Example error output

Name:         backup2
Namespace:    default
...
Status:
  State:              Error
  State Description:  azure not found in spec.backup.storages in PerconaServerMySQL CustomResource
Events:               <none>

Common error scenarios include:

• Storage not found: The storage name specified in the backup doesn’t exist in the cluster configuration
• Authentication failures: Invalid credentials for accessing cloud storage
• Network issues: Problems connecting to the storage service
• Insufficient permissions: The backup job doesn’t have permission to write to the storage location

Check backup Jobs

When a backup runs, the Operator creates a Kubernetes Job to execute it. You can view these Jobs to see which backups are running or have completed:

kubectl get jobs -n <namespace>

Backup jobs follow this naming pattern:

• xb-<backup-name>-<storage-name> for on-demand backups
• xb-cron-<cluster-name>-<storage-name>-<timestamp>-<hash>-<storage-name> for scheduled backups

Example output

NAME                                                     STATUS     COMPLETIONS   DURATION   AGE
xb-backup1-aws-s3                                        Complete   1/1           14s        11m
xb-backup3-azure-blob                                    Complete   1/1           12s        10m
xb-backup4-gcp-cs                                        Complete   1/1           5m52s      9m45s
xb-backup5-gcp-cs                                        Complete   1/1           10s        3m43s
xb-cron-ps-cluster1-aws-s3-20251107211029-45srk-aws-s3   Complete   1/1           12s        35s

View backup job logs

To see detailed logs from a backup job, use the job name:

kubectl logs job/<job-name> -n <namespace>

Find the name using the steps from the Check backup Jobs section

For example, to view logs from the xb-backup1-aws-s3 job:

kubectl logs job/xb-backup1-aws-s3 -n <namespace>

These logs show the backup process execution, including Percona XtraBackup operations and any errors that occurred.

Example output

Defaulted container "xtrabackup" out of: xtrabackup, xtrabackup-init (init)
Trying to run backup backup1 on ps-cluster1-mysql-1.ps-cluster1-mysql.default
2025-11-07T20:59:33.889841-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --datadir=/var/lib/mysql 
2025-11-07T20:59:33.890011-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized client arguments: --backup=1 --stream=xbstream --safe-slave-backup=1 --slave-info=1 --target-dir=/backup/ --user=xtrabackup --password=* 
xtrabackup version 8.4.0-4 based on MySQL server 8.4.0 Linux (x86_64) (revision id: c584cb20)
2025-11-07T20:59:33.890062-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: xtrabackup, password: set, port: not set, socket: not set
2025-11-07T20:59:33.903741-00:00 0 [Note] [MY-011825] [Xtrabackup] Using server version 8.4.6-6
2025-11-07T20:59:33.934529-00:00 0 [Note] [MY-011825] [Xtrabackup] Not checking slave open temp tables for --safe-slave-backup because host is not a slave
2025-11-07T20:59:33.934783-00:00 0 [Note] [MY-011825] [Xtrabackup] Executing LOCK TABLES FOR BACKUP ...
2025-11-07T20:59:33.940450-00:00 0 [Note] [MY-011825] [Xtrabackup] uses posix_fadvise().
2025-11-07T20:59:33.940508-00:00 0 [Note] [MY-011825] [Xtrabackup] cd to /var/lib/mysql

You can also view logs from the Pod created by the Job:

kubectl logs <pod-name> -n <namespace>

To find the Pod name, list pods and look for pods with names matching your backup job:

kubectl get pods -n <namespace> | grep xb-

Find the backup source Pod

Each backup object contains information about which MySQL Pod was used as the backup source. You can retrieve this information using:

kubectl get ps-backup <backup-name> -o jsonpath='{.status.backupSource}' -n <namespace>

For example:

kubectl get ps-backup backup1 -o jsonpath='{.status.backupSource}'

This returns the fully qualified domain name (FQDN) of the source pod, such as ps-cluster1-mysql-1.ps-cluster1-mysql.default.

View backup logs from the source pod

The xtrabackup container in the source pod also contains backup logs. To view them, run:

kubectl logs <source-pod-name> -c xtrabackup -n <namespace>

For example, if the backup source is ps-cluster1-mysql-1:

kubectl logs ps-cluster1-mysql-1 -c xtrabackup -n <namespace>

These logs show the backup request received by the sidecar container, including the backup destination, storage type, and Percona XtraBackup commands executed.

Example output

2025-11-07T20:44:11Z    INFO    startServer     starting http server
2025-11-07T20:59:33Z    INFO    sidecar.create backup   Checking if backup exists       {"namespace": "default", "name": "backup1"}
2025-11-07T20:59:33Z    INFO    sidecar.create backup   Backup starting {"namespace": "default", "name": "backup1", "destination": "ps-cluster1-2025-11-07-20:59:28-full", "storage": "s3", "xtrabackupCmd": "/usr/bin/xtrabackup --backup --stream=xbstream --safe-slave-backup --slave-info --target-dir=/backup/ --user=xtrabackup ", "xbcloudCmd": "/usr/bin/xbcloud put --parallel=10 --curl-retriable-errors=7 --md5 --storage=s3 --s3-bucket=operator-testing --s3-region=us-east-1   ps-cluster1-2025-11-07-20:59:28-full"}
2025-11-07T20:59:33.889841-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized server arguments: --datadir=/var/lib/mysql 
2025-11-07T20:59:33.890011-00:00 0 [Note] [MY-011825] [Xtrabackup] recognized client arguments: --backup=1 --stream=xbstream --safe-slave-backup=1 --slave-info=1 --target-dir=/backup/ --user=xtrabackup --password=* 
xtrabackup version 8.4.0-4 based on MySQL server 8.4.0 Linux (x86_64) (revision id: c584cb20)
2025-11-07T20:59:33.890062-00:00 0 [Note] [MY-011825] [Xtrabackup] Connecting to MySQL server host: localhost, user: xtrabackup, password: set, port: not set, socket: not set
2025-11-07T20:59:33.903741-00:00 0 [Note] [MY-011825] [Xtrabackup] Using server version 8.4.6-6
2025-11-07T20:59:33.934529-00:00 0 [Note] [MY-011825] [Xtrabackup] Not checking slave open temp tables for --safe-slave-backup because host is not a slave
2025-11-07T20:59:33.934783-00:00 0 [Note] [MY-011825] [Xtrabackup] Executing LOCK TABLES FOR BACKUP ...

Restores

Check restore status

To check the status of restore operations, run:

kubectl get ps-restore -n <namespace>

This shows all restore objects with their current state.

Example output

NAME       STATE       AGE
restore1   Error       5m56s
restore2   Succeeded   5m37s

Check restore error details

When a restore fails, use the kubectl describe command to get detailed error information:

kubectl describe ps-restore <restore-name> -n <namespace>

The Status section contains the State and State Description fields that explain why the restore failed.

Example error output

Name:         restore1
Namespace:    default
...
Status:
  State:              Error
  State Description:  PerconaServerMySQLBackup backup11 in namespace default is not found
Events:               <none>

Example success output

Name:         restore2
Namespace:    default
...
Status:
  State:  Succeeded
Events:   <none>

Common restore error scenarios include:

• Backup not found: The backup specified in the restore doesn’t exist
• Storage access issues: Problems reading from the backup storage location
• Cluster state conflicts: The cluster is not in a state that allows restore
• Insufficient resources: Not enough disk space or memory to complete the restore

Check restore jobs

When a restore runs, the Operator creates a Kubernetes Job to execute it. View these Jobs:

kubectl get jobs -n <namespace>

Restore jobs follow the naming pattern xb-restore-<restore-name>.

Example output

NAME                                                     STATUS     COMPLETIONS   DURATION   AGE
xb-restore-restore2                                      Complete   1/1           23s        5m42s

View restore job logs

To see detailed logs from a restore Job, run:

kubectl logs job/<restore-job-name> -n <namespace>

For example:

kubectl logs job/xb-restore-restore2 -n <namespace>

Sample output

Defaulted container "xtrabackup" out of: xtrabackup, xtrabackup-init (init)
++ awk '/^xtrabackup version/{print $3}'
++ awk -F. '{print $1"."$2}'
++ xtrabackup --version
* XTRABACKUP_VERSION=8.4
* DATADIR=/var/lib/mysql
++ grep -c processor /proc/cpuinfo
* PARALLEL=4
* XBCLOUD_ARGS='--curl-retriable-errors=7 --parallel=4 '
* '[' -n true ']'
* [[ true == \f\a\l\s\e ]]

View logs from the Pod created by the restore Job

You can also view logs from the Pod created by the restore Job. To find the Pod name, list pods and look for pods with names matching your restore job:

kubectl get pods -n <namespace> | grep xb-restore

Then check the logs:

kubectl logs <restore-pod-name> -n <namespace>

Example output

Defaulted container "xtrabackup" out of: xtrabackup, xtrabackup-init (init)
++ awk '/^xtrabackup version/{print $3}'
++ awk -F. '{print $1"."$2}'
++ xtrabackup --version
+ XTRABACKUP_VERSION=8.4
+ DATADIR=/var/lib/mysql
++ grep -c processor /proc/cpuinfo
+ PARALLEL=4
+ XBCLOUD_ARGS='--curl-retriable-errors=7 --parallel=4 '
+ '[' -n true ']'
+ [[ true == \f\a\l\s\e ]]

Point-in-time recovery

Point-in-time recovery adds a Binlog Server Pod and a PITR Job (pitr-restore-<restore-name>) on top of the base restore Job (xb-restore-<restore-name>). Check both Jobs and the Binlog Server Pod, not only xb-restore-*.

Binlog Server is in CrashLoopBackOff

If the binlog bucket contains too many objects, Binlog Server can get stuck listing them and enter the CrashLoopBackOff state. Point-in-time recovery cannot proceed until the Pod is healthy.

Delete old binlog objects from the bucket so the object count drops. If you miss the binlog expiration window, restore becomes much harder. Monitor object count in the point-in-time recovery bucket and clean up regularly.

Restore fails after the base backup is already restored

Point-in-time recovery retries are not idempotent. If the PITR Job fails after the base backup is on disk, a retry does not restore that backup again to reset the state.

Set spec.backup.backoffLimit=0 in the cluster Custom Resource so the Job does not retry automatically. Fix the cause, then start a new restore from a known-good base backup.

Replication errors during binlog replay

Updates made by mysql-shell can produce errors such as Error_code: 1032 / HA_ERR_KEY_NOT_FOUND on mysql_innodb_cluster_metadata.instances.

If you choose to ignore SQL errors, add force: true under spec.pitr in the restore object. This passes --force to the MySQL client and silently ignores all SQL errors during replay, which can hide data loss. See Ignore SQL errors during binlog replay.

Encrypted binlogs fail to decrypt

The PITR Job cannot unwrap a binlog if the mounted keyring does not contain the KEK recorded in that file.

Typical causes:

• Restore to a new cluster without spec.pitr.keyringSecret
• Keys rotated into another Secret, but the restore still uses the cluster keyring
• A KEK removed from keyring.json

Check the PITR Job logs (pitr-restore-<restore-name>) for KEK / keyring errors. To copy the Secret and set keyringSecret, see Restore with encrypted binlogs. Lost keys cannot be recovered.

Operator user password changed after the backup

If the Operator user password in the live cluster differs from the password stored in the base backup, point-in-time recovery fails. Take a new full backup after you change that password, then restore from the new backup.

---

Last update: August 27, 2026
Created: August 27, 2026