-
Notifications
You must be signed in to change notification settings - Fork 138
Description
Documentation
- I acknowledge that I have read the relevant [documentation](https://github.com/mariadb-operator/mariadb-operator/tree/main/docs).
Describe the bug
When using the Backup CRD to create database backups with a Windows NFS server as storage, the backup job fails with an "Input/output error" when attempting to write to the backup file. This occurs because the default filename format includes colons (:
) in the timestamp format (%Y-%m-%dT%H:%M:%SZ
), which are not allowed in Windows filesystems.
In the job container, we can observe:
root@backup-manual-job-fixed-9z2fb:/# cat /backup/0-backup-target.txt
/backup/backup.2025-04-18T12:06:38Z.sql
root@backup-manual-job-fixed-9z2fb:/# $(cat /backup/0-backup-target.txt)
bash: /backup/backup.2025-04-18T12:06:38Z.sql: Input/output error
From the logs:
mariadb 💾 Exporting env
mariadb 💾 Writing target file: /backup/0-backup-target.txt
mariadb 💾 Taking backup: /backup/backup.2025-04-18T12:31:05Z.sql
mariadb bash: $(cat '/backup/0-backup-target.txt'): Input/output error
The underlying issue is that Windows filesystems do not allow colons in filenames, which makes the backup feature incompatible with Windows-based NFS servers.
Expected behaviour
The backup job should use a filename format that is compatible with all common filesystems, especially Windows filesystems which are frequently used for NFS servers in enterprise environments.
One solution would be to use a different date format that doesn't include colons, such as %Y-%m-%dT%H-%M-%SZ
(using hyphens instead of colons for hours, minutes, seconds).
Steps to reproduce the bug
- Set up a Windows-based NFS server as storage for backups
- Create a MariaDB instance using the operator
- Create a Backup resource pointing to this MariaDB instance with storage mounted from the Windows NFS server
- Observe the backup job failing with I/O errors when trying to write to a file with colons in the name
Debug information
- From the failing job pod, we see the command:
set -euo pipefail;echo 💾 Exporting env;export BACKUP_FILE=/backup/backup.$(date -u +'%Y-%m-%dT%H:%M:%SZ').sql;echo 💾 Writing target file: /backup/0-backup-target.txt;printf "${BACKUP_FILE}" > /backup/0-backup-target.txt;echo 💾 Taking backup: $(cat '/backup/0-backup-target.txt');mariadb-dump --user=${MARIADB_OPERATOR_USER} --password=${MARIADB_OPERATOR_PASSWORD} --host=myapp-db.test-jerome.svc.cluster.local --port=3306 --single-transaction --events --routines --all-databases --ssl --ssl-ca /etc/pki/ca.crt --ssl-cert /etc/pki/client.crt --ssl-key /etc/pki/client.key --ssl-verify-server-cert --verbose > $(cat '/backup/0-backup-target.txt')
When the backup job tries to write to this file, it fails with an I/O error because the filename contains colons which are invalid in Windows filesystems.
Environment details:
- Kubernetes version: 1.31.4
- Kubernetes distribution: RKE
- MariaDB Operator version: 0.38.0
- MariaDB Server version: [Version number]
- Install method: Helm
- Storage: Windows-based NFS server
Additional context
The issue is critical for environments using Windows-based NFS servers for storage, which is common in many enterprise setups. The current implementation makes the backup feature unusable in these environments.
The issue appears to be in the backup script's use of the date format. A simple change to the date format in the script would fix this issue:
Change:
export BACKUP_FILE=/backup/backup.$(date -u +'%Y-%m-%dT%H:%M:%SZ').sql
To:
export BACKUP_FILE=/backup/backup.$(date -u +'%Y-%m-%dT%H-%M-%SZ').sql
This would avoid the problematic colons in filenames while still maintaining ISO 8601 readability.
Proposed solution
Either:
- Change the default date format to use hyphens instead of colons, making it compatible with all common filesystems
- Add an option in the Backup CRD to configure the date format used in backup filenames, with a safe default that doesn't include characters problematic for filesystems (like colons)
This issue affects cross-platform compatibility and should be addressed to ensure the operator works in diverse storage environments.