ggrebalance: Part 1. Shrink

To understand what ggrebalance does during shrink, it is useful to briefly review how gpexpand adds new segments (Figure 7). In the first stage, gpexpand prepares new segments: it locks the catalog for updates, preventing tables from being created on the old segment configuration, copies the coordinator backup to the new segment locations (the coordinator catalog is used as a template for new segments), starts the new segments, updates gp_segment_configuration (all subsystems implementing distributed operations rely on this relation for segment count and layout), releases the catalog lock, and stores the list of existing relations in a utility table based on a snapshot of pg_class across all databases.

Finally, it synchronizes the mirrors of the newly added segments.

In the second stage, data is redistributed using ALTER TABLE …​ EXPAND: for each table in the previously prepared list, a temporary relation with an updated distribution is created, the source table is scanned on the old segment set, and its data is redistributed across all segments into the temporary relation. To preserve the properties of the original relation while linking it to the redistributed data, ALTER TABLE swaps the relfilenode values of the new and original relations in pg_class, and then drops the temporary relation. This data redistribution method is known as the CTAS approach and is a reliable way to reorganize data in Greengage (users can redistribute tables using the ALTER TABLE …​ SET WITH (reorganize=true) command). However, the CTAS method has several drawbacks. First, relation data is duplicated — in large clusters, this leads to a significant increase in disk space usage. Second, creating a new relation increments the OID counter each time, which brings OID wraparound slightly closer. This is not critical in practice but still worth avoiding. You can illustrate the creation of an intermediate relation in the CTAS approach by manually creating a table on a smaller number of segments and running CTAS into a temporary table:

As shown in the example, the original table is scanned on the initial set of segments, after which tuples are redistributed across the entire cluster. After redistribution, relfilenode is swapped, and the temporary relation is dropped.

The third drawback is that at the end of ALTER TABLE …​ EXPAND, indexes are fully rebuilt. This introduces an additional I/O load and increases the operation time proportionally to the number and size of indexes.

Shrink, in turn, also follows the same redistribution principles: the target number of segments is lower than the current one, and redistribution must complete before gp_segment_configuration is updated. This means that during redistribution, the cluster still operates with the old number of segments, and all query routing mechanisms continue to function as before.

In ggrebalance, table shrink uses an alternative approach, which required enhancements to the DBMS core: data redistribution via a target INSERT, implemented through the newly introduced ALTER TABLE <name> REBALANCE <target_segment_count> command. This approach covers both expansion (using CTAS in release 1.0) and shrink, which is the focus of this article. When reducing the number of segments, instead of creating an intermediate relation, the INSERT approach streams data directly from the segments being removed to the remaining ones by executing a query. The planner is also extended to support the shrink case; outside ALTER TABLE …​ REBALANCE, the execution plan differs:

When implementing the ALTER TABLE …​ REBALANCE command, scheduler changes were introduced for the shrink case. First, the number of distribution segments on insert into the target relation was adjusted, creating a distribution conflict between the insert relation and the INSERT subplan. When this conflict is detected, the scheduler adds a Redistribute Motion. Second, to execute the scan of t1 directly on the shrink segments, the predicate gp_segment_id IN ({set of shrunk content IDs}) is added, explicitly indicating where the scan is performed. The first slice is executed on the segments being removed, while the root slice, where the insert is performed, is launched on the target segments. As a result, resolving redistribution on the INSERT side avoids unnecessary intermediate data duplication (except, of course, for tuples originating from the segments being removed) and eliminates index rebuilding.

The difference between the two approaches is clear even in a conceptual example. Assume a cluster with 6 segments is shrunk to 4. For a 600 GB table evenly distributed across 6 segments (~100 GB per segment), the CTAS approach would create a full temporary copy of the table during shrink (600 GB of additional data on disk). The INSERT approach moves only the rows from segments 4 and 5, that is, about 200 GB. Until the segments are removed from the cluster, this data exists in two copies, but overall resource consumption is significantly lower.

The previously mentioned gpshrink uses a simple imperative implementation approach (written as a single Python file, similar to gpexpand). It relies on straightforward if/elif/else control flow, where progress is essentially a numbered sequence of steps without formal validation of allowed transitions between them. ggrebalance, in turn, introduces more control over the shrink process. The shrink operation in ggrebalance is implemented as a deterministic finite-state machine (FSM) based on the transitions Python library, with different types of states: ephemeral states (not persisted anywhere, required for transition mechanics but irrelevant to the shrink logic itself), main operation states (persisted in the database and ordered), and rollback states. The library allows defining state transition rules and enforcing them during execution. It also provides a flexible system of hooks and callbacks that can be executed before or after transitions; they can be bound to specific states, define additional transition conditions, handle exceptions, and more. In other words, the library is well-suited for describing complex processes.

A simplified view of the architecture, without listing all states and transitions, is shown in Figure 8. ggrebalance is built around several nested finite-state machines; this design separates responsibilities across levels and persists each level’s state in the database, enabling recovery after interruptions.

The GGRebalanceMainSM and GGShrink classes define explicit state graphs and transitions between them. Each transition is stored in the service ggrebalance schema (Figure 9) in the postgres database. This ensures reentrancy: after an interruption, the utility reads the last committed state and resumes execution from the next step.

As an example, in a shrink scenario, the status recording may look as follows:

The finite-state machine includes several independent operation sequences: the main utility flow (orchestration of all operations), rollback of the main flow, the shrink flow (responsible only for shrink), rollback of the shrink flow, the segment balancing flow, and rollback of segment balancing. Cluster balancing and planning moves to achieve an even distribution of segments will be covered in the next parts of the series.

Next, we examine the key stages of the shrink flow in more detail.

Consider a practical example from Figure 6: a cluster of 3 hosts with 12 primary segments (4 per host), where the number of segments needs to be reduced to 9 and balanced to 3 segments per host. The mirroring strategy remains unchanged (grouped).

Important note: shrinking a cluster means removing segments whose content_id values correspond to the tail of the sequence of all content_id values after removing <target_segment_count> from it. Due to the specifics of Greengage hashing, an arbitrary segment cannot be removed — only segments from the end of the sequence. The command:

Before startup, ggrebalance ensures that no competing utilities (gpexpand, gprecoverseg, gpbackup, etc.) are running, verifies that all primary segments are available, and creates a PID file to prevent parallel launches. Next, the following steps are performed:

The sample output is as follows:

The theoretical advantage of the INSERT approach over CTAS is that the amount of data moved is proportional to the share of segments being removed, rather than the full size of the tables. To verify this claim, a series of experiments was conducted to evaluate shrink performance.

The first set of measurements compares the execution speed of ALTER TABLE …​ REBALANCE based on the INSERT logic with the same operation implemented via CTAS. A Greengage cluster consisting of 8 hosts and 64 primary segments was deployed in the cloud. The segment host specifications and cluster configuration are briefly summarized in Table 1.

The following scenarios were considered:

The cluster was populated with a TPC-DS dataset adapted for Greengage DB, with a total size of approximately 2 TB (generated with scale factor = 3000).

For a complete picture, we measured shrink execution time and disk space growth (disk amplification — the increase in occupied space relative to the logical volume of the data itself) for the following tables (all partitioned):

For each of the five tables, shrink was performed three times using both the CTAS and INSERT methods — 30 runs in total. After each shrink, the table was restored to its original number of segments using ALTER TABLE …​ REBALANCE <nsegs_origin>, which triggers CTAS-based redistribution; this means the post-shrink recovery step introduces the same noise into the measurement results. The following values were measured:

The theoretical estimate of disk bloat assumes that table data is evenly distributed across the cluster. For the CTAS method, a full duplicate of the relation is expected to exist at some point during execution. For INSERT, only a fraction of the data proportional to the number of segments being scanned is expected to be present.

Note that this is not full-scale load testing but a partial performance assessment using tools and resources available to regular developers. A detailed performance analysis on near-production data (~600 primary segments) will be published later.

Below are the measurement results.

Based on the obtained results, the following conclusions can be drawn. First, the CTAS "full table copy" hypothesis was confirmed experimentally. The bloat rate can be calculated both from relation size columns and from total cluster size values. Theory and practice match almost exactly. Minor deviations are explained by the host polling interval (segment sizes were collected every 10 seconds); the peak could have occurred between two measurements.

Second, it can be observed that INSERT is faster than CTAS (Table 5).

CTAS: growth is proportional to the target number of segments: 56, 32, 16. The smaller the target, the longer the redistribution takes; the total time consists of two I/O phases of roughly equal weight. For INSERT, the pattern is similar: execution time scales linearly with the amount of data moved, while network-based tuple redistribution contributes more significantly.

Thus, in terms of disk amplification, INSERT outperforms CTAS in all scenarios without exception. The difference is greatest for 64 → 56, where INSERT generates 7—​8 times less peak disk pressure than CTAS. In terms of execution time, INSERT outperforms CTAS for 64 → 56 and 64 → 32 across all tables by a wide margin. For 64 → 16, the advantage of INSERT decreases to 20—​30% on large tables and disappears on small uncompressed AO tables. This behavior is likely explained by the increase in generated WAL as the volume of redistributed data grows, since INSERT writes data row by row through the standard heap_insert / appendonly_insert paths, which, at wal_level >= replica, generate WAL for each block. When moving 50% of store_sales data (232 GB), this results in a significant amount of WAL traffic. On top of that, when inserting into an existing AO table, each segment must update rows in pg_aoseg — a system catalog that stores metadata for AO segment files. Under a high parallel insert load, contention on this catalog table becomes observable.

You can also estimate throughput in GB/s using the following formula:

CTAS at 64 → 56 achieves the highest throughput of 0.154 GB/s — because all 56 target segments write in parallel and write workload is relatively large. As the target number of segments decreases, throughput degrades. This is because the formula accounts only for the amount of data written to the new table, while the actual execution time is also determined by reading the full source table, which becomes the dominant phase during aggressive shrink. At 64 → 16, CTAS reads 100% of the data but writes only 25% — the remaining 75% of the read data is effectively redistributed across fewer segments. This imbalance between read and write workload explains the very low effective CTAS throughput under aggressive shrink conditions.

The experiments provide a quantitative basis for comparing two fundamentally different data redistribution strategies in a Greengage cluster. The measured results show that the INSERT approach consistently outperforms CTAS in terms of disk amplification in all scenarios, and in execution time under moderate shrink, but loses its advantage under aggressive shrink due to increased WAL generation and contention on AO storage metadata. The CTAS approach results in higher peak storage usage but demonstrates lower overhead under aggressive shrink conditions. These aspects are planned for further optimization in upcoming releases.

However, beyond performance considerations, ggrebalance also aims to ensure safe shrink operations without data loss by persistently tracking execution state throughout the process.

Thanks to the persistent finite-state machine, ggrebalance can safely resume after any interruption (network failure, insufficient disk space, SIGINT, and so on). The utility reads the last committed FSM state and compares it with the current cluster state:

At the table level, each entry in the service queue has a status: none — not processed, done — processed. On resume, the worker pool processes only unprocessed tables; already redistributed ones are skipped. This prevents duplicate work even after multiple interruptions during the SHRINK_TABLES step.

Let us explain the need to ensure reentrancy with an example. Suppose a shrink process was started (see Figure 6) and began redistributing data. At that time, a non-critical incident occurred in the Greengage cluster, after which it became necessary to restart the system and interrupt the shrink operation. ggrebalance itself can be interrupted by:

Suppose that the shrink process was interrupted in the following state:

The log shows that the t1 table was processed successfully, after which ggrebalance moved on to t2 but was interrupted before completing it. This is also reflected in the table_rebalance_status_detail table: t1 has the done status with a completion timestamp, while t2 remains in the none state, meaning its rebalance was never recorded as started. The rebalance_status table shows that the last persisted machine state is STATE_SHRINK_TABLES_STARTED — the FSM entered the table processing loop but did not complete it before the interruption. At first glance, it might seem sufficient to simply resume from the same state and continue redistributing t2. However, this is where one of the implementation details of the shrink operation becomes important: in the STATE_BACKUP_CATALOG_AND_UPDATE_TARGET_SEGMENT_COUNT_STARTED state, which precedes table redistribution, ggrebalance calls gp_toolkit.gp_set_rebalance_numsegments(target_count) — setting the target number of segments in the catalog. This parameter is global and not tied to a particular session. After the cluster is restarted, this parameter is reset to the default value — the full set of segments. As a result, if new tables are created between the interruption of shrink and its restart, they will be distributed across the full segment set rather than the target one. This creates a dangerous condition for shrink consistency and may lead to data loss on the segments being removed.

ggrebalance handles this and other edge cases by ensuring that segments are not removed until all tables have been redistributed. Instead of resuming directly from the interruption point, ggrebalance takes a step back: it restores rebalance_numsegments for concurrent DDL operations, rescans all databases, and rebuilds the redistribution queue by selecting tables that still satisfy the distribution condition. Without this logic, any cluster restart during shrink could turn into an incident: newly created tables would remain outside the operation, and the next ggrebalance run would either complete the shrink while ignoring them or require manual analysis of inconsistencies. In this respect, gpshrink remains vulnerable to data loss during concurrent cluster activity.

The shrink rollback operation (ggrebalance --rollback) is available only before gp_segment_configuration is updated — that is, no later than the STATE_SHRINK_TABLES_DONE state. Once the catalog has been updated, rollback becomes impossible, since the cluster already operates with the new number of segments. When rollback is still allowed, the rollback flow starts and performs the following steps:

Rollback itself is fully reentrant: each step is persisted in the same rollback state flow (states_rollback_flow), and rerunning ggrebalance --rollback correctly resumes an interrupted rollback operation. At the same time, tables already processed during rollback (their status has been reset to none) are not processed again. Thus, both edge-case scenarios — "the operation was interrupted, and I want to continue" and "the operation was interrupted, and I want to restore everything to its original state" — are handled by ggrebalance deterministically and without manual intervention in the cluster state.

Do not confuse shrink rollback with cluster-balancing rollback (the following parts describe segment movement between hosts in more detail). Future releases are also expected to support full shrink rollback via a reverse expand operation.