TrilioVault: Effective Backup & Restoration of a Kubernetes Cluster
Learn how to effectively secure and restore your cluster using TrilioVault for Kubernetes (TVK). From configuration to scheduling backups and retention policies – enhance the security of your Kubernetes cluster and protect against data loss.
TrilioVault for Kubernetes (TVK) is a cloud-native solution that secures your application data and metadata by storing on-demand backups in an independent storage repository. With TVK, you can create full or incremental backups of your cluster and restore them in the event of data loss. Additionally, it enables migration from one cluster to another, execution of pre- and post-hooks for backup and restore operations, and the creation of backup plans. This blog post explains how to use TVK to protect your local Kubernetes cluster or a centron Kubernetes Service and safeguard your applications.
Before you begin, ensure that you have the necessary tools and resources: a ccenter account, a Kubernetes cluster, Kubectl, a storage location for your backups, Helm, and a TVK license.
Step 1 – Configuring the Kubernetes Cluster
Before installing TrilioVault, ensure your Kubernetes cluster is properly configured for smooth functionality. You’ll need to make sure the cluster supports the Container Storage Interface (CSI) and that the volumeSnapshot Custom Resource Definition (CRD) is installed.
To check if the CSI driver is installed and configured, run `kubectl get storageclass` and review the output. By default, the centron Managed Kubernetes Service is configured with the CSI driver `dobs.csi.centron.de`.
A successful installation of TrilioVault also requires the volumeSnapshot CRD. Verify this with `kubectl get crd | grep volumesnapshot`. Ensure that the CRD is supported with the required API versions, particularly `v1`.
Finally, confirm that the CRD supports the v1 API version by running `kubectl get crd volumesnapshots.snapshot.storage.k8s.io -o yaml` and reviewing the output. The supported versions should be listed under `storedVersions`.
After verifying your Kubernetes cluster configuration, you’re ready to install TrilioVault, covered in the next step.
Step 2 – Installing TrilioVault for Kubernetes
Once your Kubernetes cluster configuration is verified, it’s time to install TrilioVault. In this step, we’ll deploy TrilioVault on a local Kubernetes cluster and manage installations via Helm. Backup data will be stored in the previously created S3-compatible bucket.
There are several ways to install TrilioVault depending on the Kubernetes cluster distribution. In this tutorial, we’ll use Helm and the `triliovault-operator` chart to install TrilioVault.
First, add the TrilioVault Helm repository and list the available charts. Then, install TrilioVault for Kubernetes using Helm, using the default parameters from the TrilioVault Helm values file.
The installation includes the TVK Operator and TVK Manager. The Operator monitors the lifecycle of the application and performs automatic recovery operations if needed. The Manager manages various CRDs and their controllers, as well as a webhook server for validation and mutation of CRD instances.
After installation, verify the status of TrilioVault to ensure successful installation. You can do this via Helm or directly with kubectl. Then, confirm your TVK license installation, checking its status and details, such as expiration date and license scope.
Once your TVK license is successfully applied, you’re ready to explore the TVK web console to manage your backup and restoration operations.
Step 3 – Accessing the TVK Management Console
With TrilioVault successfully installed, it’s time to access the TVK management console to create and manage backups. This console offers a user-friendly interface for performing backup and restore operations, providing better visualization and inspection of TVK cluster objects compared to using kubectl and CRDs through the command line.
To access the TVK management console, you must first export your kubeconfig file for your Kubernetes cluster and port-forward the Ingress Controller service for TVK. After exporting your kubeconfig file, set up port-forwarding to access the console.
Once port-forwarding is set up, you can open the console in your web browser by navigating to the appropriate URL. When prompted to select the kubeconfig file, choose the file you created earlier.
Note that TVK uses the kubeconfig file for authentication but does not store user data. The console provides a variety of features to manage your backup and restore operations.
In the next step, you will define the storage backend for TrilioVault, referred to as the target.
Step 4 – Creating a TrilioVault Target for Backup Storage
TrilioVault needs information on where to store your backups, which is referred to as a target. The following target types are supported: S3 and NFS. This tutorial uses the S3 type. For more information, see the Backup Target section in the TVK documentation.
Each target requires credentials to the bucket, stored in a secret. In this step, you will create a TrilioVault target for backups and a secret to store your bucket credentials.
First, create the Kubernetes secret containing your S3 bucket credentials. Create a file named “trilio-s3-target-secret” and add the following code, replacing the highlighted values with your personal credentials:
apiVersion: v1
kind: Secret
metadata:
name: trilio-s3-target-secret
namespace: tvk
type: Opaque
stringData:
accessKey: Your_Bucket_Access_Key
secretKey: Your_Bucket_Secret_Key
Then, apply the manifest to create the secret:
kubectl apply -f trilio-s3-target-secret.yaml -n tvk
Next, create a manifest for the target. Create a new file named “trilio-s3-target.yaml” and add the following code, replacing the highlighted values for `bucketName`, `region`, and `url` with your centron bucket information:
apiVersion: triliovault.trilio.io/v1
kind: Target
metadata:
name: trilio-s3-target
namespace: tvk
spec:
type: ObjectStore
vendor: Other
enableBrowsing: true
objectStoreCredentials:
bucketName: Your_Bucket_Name
region: Your_Bucket_Region
url: https://nyc1.centronspaces.com
credentialSecret:
name: trilio-s3-target-secret
namespace: tvk
thresholdCapacity: 10Gi
Save and close the manifest file and apply it to create the target object:
kubectl apply -f trilio-s3-target.yaml -n tvk
After creating the target, verify its status by running the following command:
kubectl get target trilio-s3-target -n tvk
If the status is “Available,” the target is successfully configured. Otherwise, there may be an issue with the configuration process. In this case, you can check the target validator logs to identify and resolve the issue.
Step 5 – Backing Up and Restoring the Kubernetes Cluster
In this step, you will create a backup of your Kubernetes cluster, then delete the namespaces and use the backup to restore all important applications in those namespaces. You will perform a cluster recovery operation through the target location. The same workflow applies if you need to perform a cluster migration operation.
The main idea is to perform a full cluster backup by including all critical namespaces that contain your essential applications and configurations. This is not a complete cluster backup and restore but rather a multi-namespace backup and restore operation. In practice, this is all that’s needed as everything in Kubernetes is organized by namespaces.
### Creating the Kubernetes Cluster Backup
In this section, you’ll create a multi-namespace backup using a ClusterBackupPlan CRD, targeting all critical namespaces from your Kubernetes cluster.
To begin the cluster backup operation, create a ClusterBackupPlan that defines a set of resources to be backed up. The specification includes the backup schedule, backup target, and resources to back up. Resources can be defined as Helm releases, operators, or plain Kubernetes API resources.
Using your text editor, create a ClusterBackupPlan manifest file named “k8s-cluster-backup-plan.yaml” and add the following code block, which is a typical manifest targeting multiple namespaces:
apiVersion: triliovault.trilio.io/v1
kind: ClusterBackupPlan
metadata:
name: k8s-cluster-backup-plan
namespace: tvk
spec:
backupConfig:
target:
name: trilio-s3-target
namespace: tvk
backupComponents:
– namespace: wordpress
– namespace: mysqldb
– namespace: etcd
Ensure that the namespaces listed in `backupComponents` exist in the cluster. Note that `kube-system` (or other Kubernetes cluster-related namespaces) are typically not included unless there’s a special case that requires keeping settings at this level.
Save and close the file.
Now, create the ClusterBackupPlan resource with kubectl:
kubectl apply -f k8s-cluster-backup-plan.yaml
Your output will look like this:
Output
clusterbackupplan.triliovault.trilio.io/k8s-cluster-backup-plan created
Now inspect the status of the ClusterBackupPlan with kubectl:
kubectl get clusterbackupplan k8s-cluster-backup-plan -n tvk
The output will look similar to this:
Output
NAME TARGET … STATUS
k8s-cluster-backup-plan trilio-s3-target … Available
Check that the STATUS column is set to “Available.”
You can also view the status of the ClusterBackupPlan through the TVK Management Console. After logging in, select Backup & Recovery and then Backup Plans to view it.
At this point, you have created a ClusterBackupPlan. Next, you’ll create a ClusterBackup, which is a configuration that references the actual ClusterBackupPlan in `spec.clusterBackupPlan.name`. The ClusterBackupPlan remains the same; you can create multiple backups by updating it in multiple ClusterBackup manifest files.
Now create a ClusterBackup manifest file named “k8s-cluster-backup.yaml” and add the following code block:
apiVersion: triliovault.trilio.io/v1
kind: ClusterBackup
metadata:
name: k8s-cluster-backup
namespace: tvk
spec:
type: Full
clusterBackupPlan:
name: k8s-cluster-backup-plan
namespace: tvk
Save and close the file.
Finally, create the ClusterBackup resources with kubectl:
kubectl apply -f k8s-cluster-backup.yaml
Applying the ClusterBackup manifest will trigger the backup operation.
Your output will look like this:
Output
clusterbackup.triliovault.trilio.io/k8s-cluster-backup created
Now inspect the status of the ClusterBackup with kubectl:
kubectl get clusterbackup k8s-cluster-backup -n tvk
The output will look like this:
Output
NAME BACKUPPLAN BACKUP TYPE STATUS … PERCENTAGE COMPLETE
k8s-cluster-backup k8s-cluster-backup-plan Full Available … 100
Check that the STATUS column is set to “Available” and the PERCENTAGE COMPLETE is 100.
You can also view the status of the cluster backup through the TVK Management Console. From the main dashboard, select Monitoring and then TrilioVault Monitoring in the left panel.
It may take some time for the complete cluster backup to finish, depending on how many namespaces, associated resources, and data on the PVCs are involved in the operation. When the output appears as above, all critical application namespaces included in the backup plan have been successfully backed up.
In this section, you created a cluster backup. In the next sections, you will delete the namespaces and then restore them from the backup.
Step 6 – Scheduling Backups
Automatically creating backups on a schedule is a highly useful feature. It allows you to rewind and reset the system to an earlier working state if something goes wrong. By default, TrilioVault creates three scheduled policies: daily, weekly, and monthly.
In the TVK console, you can view the default policies under Backup & Recovery and then Schedules. Schedules can be used for BackupPlan or ClusterBackupPlan CRDs.
Create a manifest file named “scheduled-backup-every-5min.yaml” and add the following code, which is a typical custom Schedule Policy CRD:
apiVersion: triliovault.trilio.io/v1
kind: Policy
metadata:
name: scheduled-backup-every-5min
namespace: tvk
spec:
type: Schedule
scheduleConfig:
schedule:
– “*/5 * * * *” # Trigger every 5 minutes
This manifest creates a scheduled backup policy named “scheduled-backup-every-5min” in the “tvk” namespace. It will trigger a scheduled backup every five minutes depending on the BackupPlan objects.
After creating the manifest, you can use it to create the schedule policy:
kubectl apply -f scheduled-backup-every-5min.yaml
Your output will look like this:
Output
policy.triliovault.trilio.io/scheduled-backup-every-5min created
To apply the schedule policy, add it to a ClusterBackupPlan CRD. Open the ClusterBackupPlan CRD you created in Step 5 and add the highlighted lines:
apiVersion: triliovault.trilio.io/v1
kind: ClusterBackupPlan
metadata:
name: k8s-cluster-backup-plan
namespace: tvk
spec:
backupConfig:
target:
name: trilio-s3-target
namespace: tvk
schedulePolicy:
fullBackupPolicy:
name: scheduled-backup-every-5min
namespace: tvk
backupComponents:
– namespace: wordpress
– namespace: mysqldb
– namespace: etcd
The ClusterBackupPlan CRD references the previously defined policy CRD via the `spec.backupConfig.schedulePolicy` field. You can create separate policies for full or incremental backups, so the `fullBackupPolicy` or `incrementalBackupPolicy` can be specified in `spec`.
Save and close your file.
In this step, you have scheduled backups and added a schedule policy to a ClusterBackupPlan. The next step will cover how to set a retention policy for your backups.
Step 7 – Creating a Backup Retention Policy
In this step, you’ll create a backup retention policy to determine the frequency of your backups. Retention policies are important because storage is limited and can become expensive if too many objects are retained.
The retention policy allows you to set the number of backups to retain and the rhythm of deleting backups according to compliance requirements. The retention policy CRD provides a YAML specification to define the number of backups to keep by day, week, month, year, latest, etc.
TVK also has a default retention policy, which you can view in the TVK console under Backup & Recovery and then Retention Policies.
Retention policies can be used for BackupPlan or ClusterBackupPlan CRDs. Create a new file named “sample-retention-policy.yaml” and add the following lines:
apiVersion: triliovault.trilio.io/v1
kind: Policy
metadata:
name: sample-retention-policy
spec:
type: Retention
retentionConfig:
latest: 2
weekly: 1
dayOfWeek: Wednesday
monthly: 1
dateOfMonth: 15
monthOfYear: March
yearly: 1
This is a typical Policy manifest for the retention type. Here’s an explanation of the above configuration:
- `spec.type`: Defines the policy type: Retention or Schedule.
- `spec.retentionConfig`: Describes the retention configuration, such as the interval for backup retention and how many to keep.
- `spec.retentionConfig.latest`: Maximum number of latest backups to keep.
- `spec.retentionConfig.weekly`: Maximum number of weekly backups to retain.
- `spec.retentionConfig.dayOfWeek`: The day of the week for weekly backups to retain.
- `spec.retentionConfig.monthly`: Maximum number of monthly backups to retain.
- `spec.retentionConfig.dateOfMonth`: The day of the month to retain monthly backups.
- `spec.retentionConfig.monthOfYear`: The month to retain yearly backups.
- `spec.retentionConfig.yearly`: Maximum number of yearly backups to retain.
In the above retention policy configuration, the backup policy would retain one backup weekly on Wednesdays, once a month on the 15th, and once a year in March. The latest two backups will always be available.
The basic flow for creating a retention policy resource is the same as for scheduled backups. You need a BackupPlan or ClusterBackupPlan CRD to reference the retention policy, and then a Backup or ClusterBackup object to trigger the process.
To apply the retention policy, open your ClusterBackupPlan CRD and update it as follows:
apiVersion: triliovault.trilio.io/v1
kind: ClusterBackupPlan
metadata:
name: k8s-cluster-backup-plan
namespace: tvk
spec:
backupConfig:
target:
name: trilio-s3-target
namespace: tvk
retentionPolicy:
fullBackupPolicy:
name: sample-retention-policy
namespace: tvk
backupComponents:
– namespace: wordpress
– namespace: mysqldb
– namespace: etcd
The manifest uses a `retentionPolicy` field to reference the appropriate policy. You can have a backup plan that sets both types of policies to handle scheduled backups and retention strategies – Effective backup and restoration of a Kubernetes cluster with TrilioVault.
