Rebalance and resize a cluster

In this topic, a balanced cluster means that:

Note that in this context, balance refers to the physical distribution of segments among cluster hosts, not the distribution of table rows or workload. Evenly distributed table rows do not guarantee balanced physical topology. For example, a cluster may become physically unbalanced after segment recovery or topology changes even if all tables remain evenly distributed.

The ggrebalance utility performs complex cluster topology changes while preserving balanced segment distribution and minimizing downtime.

To perform a topology change, ggrebalance generates and executes a rebalance plan based on the requested target topology. It is defined by utility options such as:

Using this information, ggrebalance determines how table data must be redistributed and which segment instances must be moved to different hosts.

Unlike gpexpand, ggrebalance automatically determines segment placement and does not require administrators to manually define a mapping between segments and hosts.

Whenever possible, ggrebalance performs operations in parallel to speed up execution.

The utility is stateful and reentrant. Its progress and metadata are stored in a service schema — ggrebalance — in the postgres database. This allows operations to be interrupted and resumed later. Depending on the execution stage, ggrebalance may also automatically roll back interrupted operations to return the cluster to its original state.

During some stages of rebalance, a temporary reduction of cluster availability is expected. For example, while a mirror segment is being moved, the corresponding primary segment temporarily operates without high availability protection. In order to move a primary segment (which is not supported directly), ggrebalance performs a switchover — a role swap during which the primary segment is stopped and restarted as a mirror. Because switchovers interrupt access to involved segments, ggrebalance requests explicit confirmation before performing each of them unless automatic confirmation is enabled.

This topic describes how to use ggrebalance to rebalance and resize a cluster in different maintenance scenarios.

For additional background information about the utility and its implementation, see this Greengage DB blog post: What has been going on with shrink and Iceberg connector.

ggrebalance supports the following operations on cluster topology:

Cluster shrink decreases the number of segments in the system. This operation may be useful when the cluster contains too many small segments relative to the actual amount of stored data. In such cases, segment management overhead may outweigh the benefits of additional parallelism.

Cluster shrink can also be combined with topology changes such as host removal or host replacement. See Combined topology changes for an example of such an operation.

When shrinking the cluster, ggrebalance redistributes table data across the remaining segments while preserving balanced segment placement. Because of this, the target number of segments must be distributable evenly across the cluster’s segment hosts. For example:

During shrink, ggrebalance removes segments with the highest content_id values and redistributes their data across the remaining segment set. As a result, the sizes of remaining segments increase, and table redistribution may require substantial temporary I/O and network traffic.

To decrease the number of segments while keeping the same set of hosts, specify the target segment count smaller than the current one in the -x/--target-segment-count option:

The operation may require user confirmation for certain steps, including switchovers. To approve switchovers automatically, use the -y/--approve-swap-roles option. To automatically select default options for all user interactions during the operation, use non-interactive mode.

When the operation is completed, ggrebalance prints its summary:

After the operation completes, proceed to Post-rebalance actions to finalize it.

Removing segment hosts may be useful when the cluster contains more hardware resources than required. For example, the amount of stored data is smaller than expected, or resource utilization on cluster hosts is consistently low.

To remove hosts while preserving the current number of segments:

Example:

When removing more than one host, separate them with commas:

The output should end with lines like the following:

After the operation completes, proceed to Post-rebalance actions to finalize the operation.

Adding segment hosts allows redistributing workload across more hardware resources while preserving the current number of segments. This may be useful when CPU or memory utilization is high on existing hosts, or when you want to reduce resource contention without changing table distribution.

To add hosts while preserving the current number of segments:

Example:

After successful completion, the utility prints a summary similar to the following:

After the operation completes, proceed to Post-rebalance actions to finalize it.

If the cluster uses custom segment data directory locations instead of the default /data1/primary/gpseg<N> and /data1/mirror/gpseg<N>, specify them in the -d/--target-datadirs option. Example:

The directories must exist on the hosts being added.

Replacing segment hosts allows migrating clusters to new hardware without changing the total number of hosts or segments. Typical use cases include replacing aging hardware, migrating the cluster to a new data center, or changing the hardware characteristics of the cluster.

To replace specific segment hosts, combine the options that remove and add segment hosts -R/--remove-hosts and -A/--add-hosts:

If the cluster uses custom segment data directory locations, specify them using the -d/--target-datadirs option.

After successful completion, the utility prints a summary similar to the following:

After the operation completes, proceed to Post-rebalance actions to finalize it.

When replacing many hosts at once, for example during full cluster migration, it is convenient to specify the complete target host list in a file. This file must list all target segment hosts one per line, for example:

Pass the file to ggrebalance using the --target-hosts-file option:

ggrebalance supports resumable rebalance sessions.

A rebalance operation may be interrupted and continued later without losing its progress. This allows administrators to execute topology changes gradually during scheduled maintenance windows or periods of low activity.

To run a limited rebalance session, use one of the following options:

Examples:

When the specified time limit is reached, ggrebalance stops the current session safely and preserves the rebalance state.

To continue an interrupted rebalance operation later, use one of the following methods:

To improve resource utilization and reduce execution time, ggrebalance performs some operations in parallel whenever possible. The utility provides two independent parallelism controls:

Example:

Higher parallelism values may reduce total rebalance time but can also increase CPU utilization, disk I/O load, and the impact on active queries. Choose parallelism settings according to available cluster resources and maintenance requirements.

During execution, ggrebalance may request user input or confirmation for potentially disruptive operations. For example, confirmation may be required before segment switchovers or rollbacks.

To avoid blocking execution while waiting for user input, enable non-interactive mode:

In non-interactive mode, ggrebalance automatically selects default responses for all prompts.

If only automatic approval of segment role swaps is required, use the -y/--approve-swap-roles option instead. Example:

This option automatically approves switchovers that temporarily interrupt access to affected segment pairs. It may be useful during planned maintenance windows where short service interruptions are acceptable.

The ggrebalance schema is automatically created in the postgres database when a new rebalance operation is created (ggrebalance call with the -x option). The schema contains tables and views describing rebalance progress and operation state:

By default, ggrebalance writes execution logs to files named ggrebalance_YYYYMMDD.log. The log files are stored in the gpAdminLogs subdirectory of the gpadmin user’s home directory on the coordinator host. You can change the ggrebalance log location using the -l/--log-dir option.

To increase log verbosity, use the --verbose option.

To suppress console output while continuing to write log files, use the --quiet option.

The ggrebalance schema is no longer needed after rebalance is complete. However, it is recommended to keep the schema until the new cluster topology and workload behavior are validated successfully. As long as the schema exists, rebalance metadata remains available and rollback operations are still possible.

To remove the schema, call ggrebalance with the -c/--clean option:

After topology changes or table redistribution, optimizer statistics may no longer accurately reflect the new data distribution. Collect fresh statistics to ensure that the optimizer generates efficient execution plans for the updated cluster layout.

You can collect statistics using the ANALYZE command or analyzedb utility. Alternatively, use the -a/--analyze option to make ggrebalance automatically collect statistics after completing rebalance operations. Example:

Keep in mind that this can significantly increase operation time for large data volumes.