GitHub

Enable cluster mirroring

Pavel Semyonov

This topic explains how to enable mirroring in a Greengage DB (based on Greenplum) cluster.

Greengage DB ensures cluster fault tolerance using the mirroring mechanism. Each cluster instance — segment or master — can be duplicated by a mirror that stores either the same data (segment mirror) or the system catalog (standby master). Mirrors automatically stay in sync with their corresponding primary instances. If a primary instance fails, its mirror takes over, preserving cluster integrity and availability.

A Greengage DB cluster can be initialized with mirroring from the start, as described in Initialize DBMS. If the cluster was initially set up without mirrors, you can add them later without interrupting service.

To add mirrors, the following utilities are used:

  • gpaddmirrors — add segment mirrors.

  • gpinitstandby — add a standby master.

Enable segment mirroring

Segment mirroring ensures data availability in the event of a segment failure. If a primary segment fails, the system automatically promotes its mirror to primary without requiring manual intervention.

Segment mirroring is applied to the entire cluster in a single operation. Running the gpaddmirrors utility on the master host creates mirrors for all segment instances and distributes them across cluster hosts according to the specified (or default) mirror configuration.

TIP

Initializing segment mirrors does not require a cluster shutdown. However, table data replication temporarily increases the load on segment hosts. To minimize performance impact, it is recommended to perform mirror initialization during off-peak hours.

Standard mirror configuration

Greengage DB automatically distributes mirror segments across the cluster’s segment hosts. By default, it applies the following mirror configuration:

  • Mirror hosts: all existing segment hosts.

  • Mirroring policy: grouped (mirrors of all primary segments from a given host are placed together on another host).

  • Mirror base port: primary segment base port plus 1000.

  • Data directories: specified during mirror creation.

To create mirror segments on existing cluster hosts:

  1. Log in to the master host as gpadmin.

  2. Run gpaddmirrors:

    $ gpaddmirrors

    Optionally, you can add utility options to adjust mirror configuration:

    • -p — specifies a port offset:

      $ gpaddmirrors -p 5000
      IMPORTANT

      Port offset is a number to add to primary segment port numbers. For example, if primary segments run on ports from 10000 to 10003, and -p value is 5000, mirror segments will occupy ports from 15000 to 15003.

      The default port offset is 1000.

    • -s — uses the spread mirroring policy:

      $ gpaddmirrors -s

      With spread mirroring, mirrors of primary segments from a single host are distributed across other segment hosts so that no two mirrors from the same host are placed on the same destination. This policy requires the cluster’s number of hosts to be greater than the number of segments per host.

    • -m — specifies a file with mirror data directories:

      $ gpaddmirrors -m mirror_dirs
      IMPORTANT

      Mirror data directories must be different from primary segment data directories.

      Each line in this file should contain an absolute path to a directory to store a mirror segment. The number of lines must match the number of primary segments per host.

      • Four mirror segments per host, all in the same directory:

        /data1/mirror
/data1/mirror
/data1/mirror
/data1/mirror

        All mirrors will be stored in /data1/mirror/gpseg<N> directories, where <N> is the segment identifier.

      • Separate location for each of the four mirror segments on a host:

        /data1/mirror1
/data1/mirror2
/data1/mirror3
/data1/mirror4

        Each of the specified directories will contain one nested segment directory, such as /data1/mirror1/gpseg<N>, /data1/mirror2/gpseg<M>, and so on.

  3. (Optional) If no file with data directories was specified in the gpaddmirrors call, enter paths to data directories for each mirror segment on cluster hosts one by one:

    Enter mirror segment data directory location 1 of 2 >

    Greengage DB prepares the mirror configuration based on the specified parameters and outputs the following details for each mirror:

    [INFO]:---------------------------------------------
[INFO]:-Mirror 1 of 8
[INFO]:---------------------------------------------
[INFO]:-   Primary instance host        = sdw1
[INFO]:-   Primary instance address     = sdw1
[INFO]:-   Primary instance directory   = /data1/primary/gpseg0
[INFO]:-   Primary instance port        = 10000
[INFO]:-   Mirror instance host         = sdw2
[INFO]:-   Mirror instance address      = sdw2
[INFO]:-   Mirror instance directory    = /data1/mirror/gpseg0
[INFO]:-   Mirror instance port         = 15000

  4. Enter y and press Enter to confirm the mirror creation:

    Continue with add mirrors procedure Yy|Nn (default=N):

    After a successful mirror creation, the following lines are shown:

    [INFO]:-******************************************************************
[INFO]:-Mirror segments have been added; data synchronization is in progress.
[INFO]:-Data synchronization will continue in the background.
[INFO]:-Use  gpstate -s  to check the resynchronization progress.
[INFO]:-******************************************************************

Custom mirror configuration

If the default mirror configuration does not meet your requirements, you can explicitly define each mirror segment’s parameters: host, port, and data directory. This allows you to place mirror segments on a separate set of hosts, reducing the load on existing hosts in case of a primary segment failure. However, this approach requires additional resources.

Before setting up segment mirrors on external hosts, prepare these hosts to run Greengage DB as described in the Initialize DBMS topic.

To create a custom mirror configuration, you need a file that specifies mirroring parameters such as hosts, ports, and directories. You can generate a template mirror configuration file interactively using the gpaddmirrors utility or prepare it manually using the structure described in this section.

To generate a template mirror configuration file:

  1. Log in to the master host as gpadmin.

  2. Run gpaddmirror specifying the name to assign to the generated file in the -o option:

    $ gpaddmirrors -o mirror_conf_file

    Optionally, define a port offset (-p option), a mirror location file (-m option), and a mirroring policy (-s for spread):

    $ gpaddmirrors -o mirror_conf_file -p 2000 -m mirror_dirs

  3. (Optional) If no mirror location file was specified, enter mirror data directories one by one when prompted:

    Enter mirror segment data directory location 1 of 2 >

    The utility informs about the file generation in a message like this:

    [INFO]:-Configuration file output to mirror_conf_file successfully.

The following template file is generated for a cluster with four segment hosts — sdw1, sdw2, sdw3, and sdw4 — using the default (grouped) mirroring policy.

0|sdw2|11000|/data1/mirror/gpseg0
1|sdw2|11001|/data1/mirror/gpseg1
2|sdw3|11000|/data1/mirror/gpseg2
3|sdw3|11001|/data1/mirror/gpseg3
4|sdw4|11000|/data1/mirror/gpseg4
5|sdw4|11001|/data1/mirror/gpseg5
6|sdw1|11000|/data1/mirror/gpseg6
7|sdw1|11001|/data1/mirror/gpseg7

The mirror configuration file consists of lines of the following structure:

content|address|port|datadir
Mirror configuration file structure
Field Description

content

A content identifier for a segment instance

address

The hostname used to access a particular segment instance on a segment host. This value may be the same as hostname on systems that do not have per-interface hostnames configured

port

The TCP port the database segment is using

datadir

A segment instance data directory

The generated file template contains the standard mirror configuration based on the specified parameters. Modify the file to align with your desired mirror configuration. For example, to place mirrors on different hosts, replace the existing segment host addresses with new host addresses. The file below creates segment mirrors on new hosts sdw5, sdw5, sdw7, and sdw8:

0|sdw5|11000|/data1/mirror/gpseg0
1|sdw5|11001|/data1/mirror/gpseg1
2|sdw6|11000|/data1/mirror/gpseg2
3|sdw6|11001|/data1/mirror/gpseg3
4|sdw7|11000|/data1/mirror/gpseg4
5|sdw7|11001|/data1/mirror/gpseg5
6|sdw8|11000|/data1/mirror/gpseg6
7|sdw8|11001|/data1/mirror/gpseg7

To create mirrors using the customized configuration file, use the -i option:

$ gpaddmirrors -i mirror_config_file

Enable master mirroring

Master mirroring enhances the fault tolerance of a Greengage DB cluster by introducing a standby master — a backup instance that maintains a synchronized copy of the system catalog. It serves as a warm standby: in the event of a master failure, the standby master can take over, ensuring continued operation.

IMPORTANT

Greengage DB does not automatically fail over to the standby master upon a master failure. To manually activate the standby master, run gpactivatestandby on its host. To learn more, see the Recover a failed master topic.

Add a standby master

If a cluster was initialized without a standby master, you can add one at any time on the running cluster.

Before initializing a standby master on a host, ensure that it is properly configured to run Greengage DB as described in Initialize DBMS.

To create a standby master on a prepared host:

  1. Log in to the active master host as gpadmin.

  2. Run gpinitstandby on the master host with the standby hostname in the -s option:

    $ gpinitstandby -s smdw

    By default, Greengage DB assigns the same port and data directory to the standby master as on the active master. You can customize these settings using the -P and -S options:

    $ gpinitstandby -s smdw -P 12000 -S /data2/master

    The gpadmin user must have read and write access to the specified standby master data directory.

    TIP

    Unless there are specific reasons to change them, it’s recommended to use the same port and data directory as the current master.

    The utility outputs the future standby master configuration as follows:

    [INFO]:------------------------------------------------------
[INFO]:-Greengage standby master initialization parameters
[INFO]:------------------------------------------------------
[INFO]:-Greengage master hostname               = mdw
[INFO]:-Greengage master data directory         = /data1/master/gpseg-1
[INFO]:-Greengage master port                   = 5432
[INFO]:-Greengage standby master hostname       = smdw
[INFO]:-Greengage standby master port           = 5432
[INFO]:-Greengage standby master data directory = /data1/master/gpseg-1
[INFO]:-Greengage update system catalog         = On

  3. Enter y to confirm the stanby master initialization and press Enter:

    Do you want to continue with standby master initialization? Yy|Nn (default=N):

    After a successful standby master initialization, the utility outputs the following line:

    [INFO]:-Successfully created standby master on smdw
NOTE

If for some reason you want to create a standby master on the current master host, specify only the port and data directory. They must be different from those used by the active master.

$ gpinitstandby -P 6432 -S /data2/master

Replace an existing standby master

If your cluster already has a standby master, and you want to move it to another host, follow these steps:

  1. Remove the existing standby master by running gpinitstandby with the -r option:

    $ gpinitstandby -r
    CAUTION

    After this step, the cluster will not be fault-tolerant until you initialize a new standby master.

  2. Initialize a new standby on a different host:

    $ gpinitstandby -s smdw2
Restart standby master

If the existing standby has stopped functioning for some reason, you can restart it with gpinitstandby -n:

$ gpinitstandby -n

Monitor mirroring state

This section lists the ways to check the cluster mirroring state.

gpstate utility

The gpstate utility has options that display information about mirrors in the cluster:

  • gpstate -m outputs the list of mirror segments and the mirroring policy used in the cluster:

    $ gpstate -m

    Example output:

    [INFO]:--------------------------------------------------------------
[INFO]:--Current GPDB mirror list and status
[INFO]:--Type = Group
[INFO]:--------------------------------------------------------------
[INFO]:-   Mirror    Datadir                Port    Status    Data Status
[INFO]:-   sdw2      /data1/mirror/gpseg0   11000   Passive   Synchronized
[INFO]:-   sdw2      /data1/mirror/gpseg1   11001   Passive   Synchronized
[INFO]:-   sdw1      /data1/mirror/gpseg2   11000   Passive   Synchronized
[INFO]:-   sdw1      /data1/mirror/gpseg3   11001   Passive   Synchronized
[INFO]:--------------------------------------------------------------

    The following values indicate that the mirror segments work in their configured roles and are synced with the master:

    • Status = Passive

    • Data Status = Synchronized

  • gpstate -f outputs the standby master information:

    $ gpstate -f

    Example output:

    [INFO]:-Standby master details
[INFO]:-----------------------
[INFO]:-   Standby address          = smdw
[INFO]:-   Standby data directory   = /data1/master/gpseg-1
[INFO]:-   Standby port             = 5432
[INFO]:-   Standby PID              = 1310
[INFO]:-   Standby status           = Standby host passive
[INFO]:--------------------------------------------------------------
[INFO]:--pg_stat_replication
[INFO]:--------------------------------------------------------------
[INFO]:--WAL Sender State: streaming
[INFO]:--Sync state: sync
[INFO]:--Sent Location: 0/141D6D18
[INFO]:--Flush Location: 0/141D6D18
[INFO]:--Replay Location: 0/141D6D18
[INFO]:--------------------------------------------------------------

    The following lines indicate that the standby master is working and synced with the master:

    • Standby status = Standby host passive

    • WAL Sender State: streaming

    • Sync state: sync

Other gpstate options, such as -s or -e, can be used to obtain more details about the mirroring state. To learn more about gpstate options, see Check cluster state.

gp_segment_configuration table

The gp_segment_configuration table stores information about cluster segments, including their roles: primary (role = 'p') and mirror (role = 'm').

SELECT * FROM gp_segment_configuration;

Query result:

 dbid | content | role | preferred_role | mode | status | port  | hostname | address |        datadir
------+---------+------+----------------+------+--------+-------+----------+---------+-----------------------
    1 |      -1 | p    | p              | n    | u      |  5432 |   mdw    |   mdw   | /data1/master/gpseg-1
    6 |      -1 | m    | m              | s    | u      | 10000 |   sdw1   |   sdw1  | /data1/primary/gpseg0
    2 |       0 | p    | p              | s    | u      | 10000 |   sdw2   |   sdw2  | /data1/primary/gpseg2
    7 |       0 | m    | m              | s    | u      | 10000 |   sdw3   |   sdw3  | /data1/primary/gpseg4
    3 |       1 | p    | p              | s    | u      | 10000 |   sdw4   |   sdw4  | /data1/primary/gpseg6
    8 |       1 | m    | m              | s    | u      | 10001 |   sdw1   |   sdw1  | /data1/primary/gpseg1
    4 |       2 | p    | p              | s    | u      | 10001 |   sdw2   |   sdw2  | /data1/primary/gpseg3
    9 |       2 | m    | m              | s    | u      | 10001 |   sdw3   |   sdw3  | /data1/primary/gpseg5
    5 |       3 | p    | p              | s    | u      | 10001 |   sdw4   |   sdw4  | /data1/primary/gpseg7
   10 |       3 | m    | m              | s    | u      |  5432 |   smdw   |   smdw  | /data1/master/gpseg-1
(10 rows)

Rows with content = -1 represent the active and standby masters:

SELECT * FROM gp_segment_configuration WHERE content = -1;

Query result:

 dbid | content | role | preferred_role | mode | status | port | hostname | address |        datadir
------+---------+------+----------------+------+--------+------+----------+---------+-----------------------
    1 |      -1 | p    | p              | n    | u      | 5432 |   mdw    |   mdw   | /data1/master/gpseg-1
    6 |      -1 | m    | m              | s    | u      | 5432 |   smdw   |   smdw  | /data1/master/gpseg-1
(2 rows)

If all instances are running in their preferred roles, the number of rows with role = preferred_role should match the total number of cluster instances.

SELECT COUNT(*) FROM gp_segment_configuration WHERE role = preferred_role;

Query result:

count
-------
    10
(1 row)

pg_stat_replication view

Mirroring state is also stored in the system catalog view pg_stat_replication. It stores information about system processes used for segment and master replication between primary and mirror instances.

Example query:

SELECT gp_segment_id, client_addr, client_port, state, sync_state, sync_error
FROM gp_stat_replication;

Query result:

 gp_segment_id | client_addr  | client_port |   state   | sync_state | sync_error
---------------+--------------+-------------+-----------+------------+------------
            -1 | 10.92.41.13  |       34318 | streaming | sync       | none
             3 | 10.92.41.158 |       26458 | streaming | sync       | none
             2 | 10.92.41.158 |       16108 | streaming | sync       | none
             1 | 10.92.42.185 |       53158 | streaming | sync       | none
             0 | 10.92.42.185 |       64946 | streaming | sync       | none
(5 rows)
Previous
Expand
Next
Check and recover segments
Open search page