Greengage DB backup and recovery. Part 2

In the previous article, we discussed an approach to back up a Greengage cluster that was intended to provide readers with some basic knowledge necessary to understand how to create backups and then restore from them. We also considered a utility implementing this approach in practice. However, all previous discussions assumed "ideal conditions," without considering possible failures or the influence of external factors. Now, we are ready to move on to the next step, which is to explore a number of scenarios that can occur in practice and that we should take into account when backing up a Greengage cluster.

First, let’s consider the practicality of creating a full backup of a cluster each time. It is clear that this can be time-consuming and resource-intensive (both for creating a backup and for storing it). But we also should not forget about the RPO (Recovery Point Objective), which actually means how much data may be lost in case of recovery. Therefore, the more frequently we can create restore points, the lower the RPO will be.

In the previous article, we created a restore point after completing a physical backup on all cluster segments. However, we are not prevented from creating additional restore points after that. What’s the benefit? We can also restore data from an existing backup and replay the WAL log to a new restore point. Yes, the WAL log will be larger, but our data will be more up-to-date. You can read more about creating restore points here.

However, this also has a downside: the longer the gap between the backup and the restore point, the longer the RTO (Recovery Time Objective) will be, since a larger WAL should be applied over the restored files. The idea is quite simple, but it is important to note the following: since restoring to a restore point requires a backup made earlier and WAL, deleting that backup will make it impossible to restore to a similar restore point. This means that we need to take this into account when deleting backups, which we will discuss later.

Next, let’s discuss handling backup interruptions. There can be a lot of reasons for this, but it’s important to handle such situations correctly. The first question is: if a backup was interrupted, do we need to start from scratch to restart it? After all, we might have a backup that was running for a long time before the interruption occurred (either intentionally or due to an error). And if it were possible to restart the backup using the data already available, this would significantly reduce both time and space (because it wouldn’t require deleting the data already received).

In pgbackrest (the utility discussed in the previous article as the base for the prototype of the ggbm backup solution), the backup command has a special --resume option to handle this situation. It does the following: if the previous backup run was of the same type and did not complete successfully, the next run will skip files that have already been backed up. Exceptions are files whose checksums do not match and those that are grouped into a bundle (see the --repo-bundle option).

The use of this approach in Greengage is complemented by the following nuances:

Taking all these details into account, we can implement a separate command to interrupt the backup, as well as handle an error during the backup on one of the segments, and automatically stop the backup process on the remaining cluster segments. Additionally, it becomes possible to restart the backup process.

Creating backups does not make much sense without a mechanism to delete them. Data in backups becomes outdated over time, not to mention it takes up space. But we also need to understand that deleting backup data is a very important operation. Therefore, it is necessary to consider different approaches and scenarios.

First, it’s worth considering the deletion mechanism built into pgbackrest and its applicability to Greengage. Specifically, let’s look at the retention policy setting. This mechanism allows configuring automatic deletion of backups after each successful completion of the backup command. There are two types of deletion available: by time or by quantity. The number of backups can only be configured for the full and diff types, while incremental backups are deleted along with the backups they depend on.

In addition, pgbackrest has the expire command that allows us to:

Overall, the functionality is quite flexible and automates the deletion process. However, if we look at the retention policy in pgbackrest from the Greengage side, we won’t be able to use it for the following reasons:

To illustrate the latter point, let’s look at this example: we executed the backup command with the full backup type. Then, we ran it again with the same backup type, but an error occurred on one of the segments (it’s possible the backup did not even start due to an error). We did not restart the backup, but instead created a new backup for the entire cluster, of the incr type this time. Now, for some segments, the number of full backups differs. This means that if we delete backups automatically in the future, we may lose part of the cluster backup.

So what should we do in this case? We’ll need to manage the deletion process outside of pgbackrest, disabling automatic deletion in the pgbackrest.conf configuration file:

The deletion algorithm itself may be similar to how it is implemented in pgbackrest:

The result of deleting a specific backup depends on whether automatic policy-based deletion is applied. And if so, whether it is before or after. It depends on what you consider the expected behavior in a given scenario.

To complete the picture, we would like to add that deleting a backup of the diff type will be roughly the same as deleting a backup of the full type, but with some nuances. Deleting an incremental backup, however, doesn’t need to be considered separately, since it will always be deleted along with the full or diff backups it depends on.

In Greengage, cluster availability is ensured by mirroring segments. Switching a primary segment to its mirror can occur either automatically (for example, if a primary segment fails) or as planned (when a segment host is stopped for maintenance). But what is backup?

The point is that these events on a cluster change the display of the cluster state in the gp_segment_configuration table. Imagine a situation: we created a backup, then shut down one of the cluster hosts (all primary segments on that host switched to their mirrors, and the host itself became unavailable over the network). And then we needed to restore from a previously created backup. However, the backup metadata only contains information about the hosts on which the primary segments were active at the time of the backup. To restore the cluster, it is desirable to know the latest state of the cluster topology.

The solution here is clear: we need to obtain the current cluster topology (this is especially important if the cluster is no longer accessible), then compare the metadata of each segment from the backup with its current location, and restore it on the host where it was last in the primary status. However, before starting the cluster, it is important not to forget that even though we’ve restored the cluster on the correct hosts, the state in gp_segment_configuration will not correspond to it. Therefore, first, we should start only the master and overwrite the cluster topology on it with a new state:

Then we can launch the entire cluster:

In addition to the above, a similar scenario is changing the cluster topology between an existing backup and a restore point created after. To account for this situation, it’s worth considering a restore point as another type of backup that refers to the previous physical backup. This requires storing the cluster topology for each restore point, which makes it possible to restore the cluster to the topology that existed when the restore point was created.

In this article, we’ve covered application scenarios that should definitely be considered in a Greengage cluster backup system. These include deleting and automatically deleting backups, taking into account changes in the cluster topology, and handling some errors. After implementing this functionality in the backup system, we can apply it to real cluster installations in a production environment. However, this does not mean we should stop there.

In the next article, we will explore even more complex backup scenarios, such as: