ggrebalance: Part 3. Executing a rebalance operation

Moving a segment with data is a fundamentally hard problem. A Greengage segment is a full-fledged PostgreSQL instance. It serves its share of every distributed query, takes part in a two-phase commit, streams WAL continuously, and interacts with the FTS (Fault Tolerance Service) mechanism. Its address, port, and data path are stored in gp_segment_configuration, the table the dispatcher consults when routing every query. Any mismatch between the catalog and the segment’s actual state immediately breaks the cluster. Moving a segment can disrupt the cluster just as easily, so without careful preparation a move risks data loss, consistency violations, or query errors.

In theory, you could take the offline route: stop the whole cluster, copy PGDATA to the new hosts, update the gp_segment_configuration catalog in coordinator-only mode, and restart everything. It’s straightforward, but error-prone, and the downtime it causes is too long for production. ggrebalance instead moves segments online, relying on Greengage’s own capabilities: streaming replication, FTS probing, and role switching between primary and mirror.

Greengage itself does not provide a single tool for arbitrarily moving segments between hosts. ggrebalance combines two standard tools from the ecosystem, each covering half the job:

Move operations inevitably affect cluster availability and reliability, so this must be considered during planning. During a mirror move (gpmovemirrors), the primary segment keeps serving queries without interruption. The mirror, however, is temporarily unavailable during copying and the subsequent synchronization. This temporarily reduces fault tolerance: if the primary fails during this window, its data becomes unrecoverable.

Role switching works differently. The primary segment is stopped forcibly, and any transactions it was processing at that moment are aborted. FTS promotes the mirror to primary. That takes some time, and that time counts as downtime — though far shorter than the offline approach. ggrebalance provides the --replay-lag option, forwarded to gprecoverseg, which sets the maximum mirror lag allowed before triggering a switchover. It’s a way to trade operation speed against consistency guarantees. Because ggrebalance is reentrant, the role-switching stage can be postponed to a scheduled maintenance window, leaving the cluster in an intermediate but fully functional configuration.

Consider the process of balancing a cluster after a shrink (Figure 1).

The planner component described in the second article of the series builds a move plan that, once executed, results in an even distribution of segments across hosts. The plan looks like this:

The printed plan is logical: it shows which segments need to move where so the cluster becomes balanced. For moving a mirror, this representation is more than enough, since a mirror can be relocated to another host in a single step with one call to gpmovemirrors. Primary segments are a bit trickier — Greengage doesn’t support online migration of a primary segment without a service interruption. So ggrebalance moves a primary segment through a three-step scheme: promote the mirror to primary, move the former primary (now a mirror) to the target host, then revert to original roles. ggrebalance implements this sequence as follows (Figure 2):

Because pg_basebackup runs for a fairly long time during a gpmovemirrors call, executing plan steps strictly sequentially is impractical in production. So ggrebalance parallelizes move execution instead. The executor component converts the logical plan produced by the planner into a physical plan. At the execution level, the whole plan is divided into atomic steps of three kinds, implemented as subclasses of RebalanceStep and described in Table 1.

Steps of the same type that can run in parallel are grouped into batches. For example, when moving several primary segments, the implementation doesn’t interleave switchover and move steps (switchover1 → move1 → switchover_back1 → switchover2 → move2 → …​); instead, it groups steps of the same type together. That way, gprecoverseg runs once for a whole group of switchovers instead of once per segment. Operations within a batch run in parallel, and batch size can be controlled with the --parallel option.

Like the shrink operation, cluster balancing is implemented on top of a state machine. Explicit states and transitions give this design a few real advantages: the current stage is stored in persistent storage, execution is reproducible after a failure, and each stage’s logic stays cleanly separated. What makes RebalanceSM reentrant is that each state is responsible for exactly one action, and immediately persists the fact that it was entered. The state diagrams for the rebalance operation and its components are shown in Figures 3 and 4.

A rollback mechanism for moves is also provided. Rollback execution mirrors the forward rebalance flow, with only a few implementation and state-tracking nuances. The rollback flow deliberately converges on STATE_REBALANCE_EXECUTION_STARTED, which eliminates execution logic duplication. The rollback scenario is covered in more detail in the following sections.

As covered in the previous articles, every stage of what ggrebalance does — whether a shrink step or a cluster-balancing step — has a corresponding status, which Greengage stores in the ggrebalance service schema of the postgres database (Figure 5). The schema holds the saved plan (for recovery after an interruption), the current state of the main state machine and the shrink and rebalance state machines, and tables with detailed information about every stage of the work. Storing statuses directly in the DBMS was a deliberate choice, made to keep the utility’s work traceable and avoid losing information about completed operations.

For the balancing process, the ggrebalance.segment_move_steps table stores one of the following statuses for each planned move.

Switchover steps are deliberately created with APPROVE_REQUIRED status, as mentioned earlier — role switching briefly affects cluster behavior, and the operator has to consciously confirm readiness for it.

ggrebalance also logs extensively, the same way other cluster commands do. Both standard output and the log file in the gpAdminLogs directory contain detailed information about the progress of ggrebalance. Log verbosity can be controlled with two options: --verbose for detailed debug output on the Python utilities' work at each step, and --quiet to suppress normal logging.

Any long-running operation in a distributed system must assume it will be interrupted — by a network outage, a coordinator failure, or the maintenance window expiring. ggrebalance was designed around this assumption — every run of the utility restores its context from the service schema and resumes work from where the previous run stopped.

On every run, STATE_CHECK_PREVIOUS_RUN reads the last saved state from the schema and determines the next state to transition into so the interrupted process can resume. The mapping logic is as follows:

For RebalanceStepMoveMirror steps, ggrebalance performs a detailed analysis of every failed step on restart, based on the cluster’s actual state:

For switchover steps, the algorithm is much simpler: since gprecoverseg is idempotent, the operator is simply offered the choice to repeat the operation or roll it back.

Consider three example scenarios. The cluster used here has eight hosts, sdw-1 through sdw-8, with eight primaries on each, for a total of 64 segments. Mirrors are distributed across neighboring hosts using the grouped scheme. A rebalance runs after shrinking to 56 segments and decommissioning the sdw-8 host:

Rollback is triggered with the ggrebalance --rollback command (or -r). Rollback begins by reading the saved plan, checking which steps have already completed, and building a reverse plan. Three situations are possible:

When rollback starts, the original list of moves is transformed according to the following rules in on_enter_STATE_REBALANCE_ROLLBACK_PREPARE_MOVES_STARTED:

Once preparation is done, the steps are saved to the schema, and control passes to the standard rebalance state machine loop. No additional execution code is needed — rollback uses exactly the same infrastructure as the initial rebalance run. The rollback itself is fully reentrant: if it was interrupted, running ggrebalance --rollback again detects that the rollback flow has already started and resumes from where it stopped, without re-preparing the steps.

For example, rolling back the interrupted rebalance from scenario 1 in the previous section looks like this:

After the rollback completes, the ggrebalance schema is dropped, and the cluster returns to exactly the distribution it had before the original run. One warning, though: a shrink can’t be rolled back once it’s done, since the segments are already gone from the cluster.

Consider a scenario on our 8x8 cluster. Initially, there are 8 hosts, sdw-1 through sdw-8, with 8 primaries on each, using the grouped mirroring strategy. The scenario:

The entire workflow can be performed with a single command:

For clarity and to demonstrate some additional options, we’ll cover the scenario step by step:

Additional options:

Before covering the recommendations, it’s important to understand exactly what downtime ggrebalance can cause and what contributes to it. ggrebalance was designed to be "online", but that doesn’t mean zero downtime. As already mentioned when describing the data-transfer method, moving mirrors doesn’t affect availability at all, since mirrors don’t serve user queries. While gpmovemirrors copies data and replays the outstanding WAL, the primary continues operating normally. The only consequence is that the content has no backup copy during this time, which raises risk but doesn’t cause downtime.

Role switchover, on the other hand, causes some downtime. Every primary/mirror role switch via gprecoverseg -r introduces downtime that consists of:

In a test run, fully moving a primary segment (two switchovers plus two mirror moves) totaled about 20 minutes of unavailability per content, spread across the whole operation. Note that the --parallel option increases the number of concurrent switchovers.

Before starting, it’s a good idea to:

During the run:

After completion:

What to do in case of a failure:

This series of articles on ggrebalance covered the utility’s key aspects: its architecture and usage scenarios, the planner implementation, and the execution side. Taken together, that’s the full picture of a tool built for a specific operational problem: keeping the load evenly distributed as the topology continues to change.

ggrebalance covers something Greengage operations lacked before: a way to change cluster topology without taking the cluster offline. The approach relies on a custom planner, a shrink implementation with an optimized table-redistribution mechanism, and a segment-move executor that supports recovery after interruption and operation rollback, all combined with the standard gpmovemirrors/gprecoverseg utilities. That combination is what lets ggrebalance execute online, recover cleanly after interruptions, and roll back changes when needed.

How much this matters in practice is clear from a comparison with the alternatives: gpshrink (a shrink-only tool) and the manual approach using gpmovemirrors + gprecoverseg separately.

The manual approach covers only basic segment operations, and only if the operator takes on planning, state tracking, and failure recovery themselves. gpshrink automates data redistribution during a shrink but doesn’t address balancing the physical placement of segments across hosts. ggrebalance combines both tasks — shrink and rebalance — into a single managed process with the full set of properties needed for long-running operations in production.

Across these three articles, we’ve covered the main scenarios: shrink, balancing, adding hosts, and decommissioning them. But that’s only the first stage in the tool’s evolution — a few directions are next in line for further work.

The long-term goal is to make topology changes in a Greengage cluster as routine as running vacuum — no lengthy planning, no maintenance window, no manual oversight of every step.