Gfarm File System: Master-Slave Metadata Servers

documents > User's Manual > Master-Slave Metadata Servers - Tutorial

# Master-Slave Metadata Servers - Tutorial

English | Japanese

This document describes how to configure master-slave metadata servers.

## 1. Installation

Let host-a be a master gfmd and host-b be a slave gfmd. Install the Gfarm file system on both host-a and host-b. See INSTALL document for details.

## 2. Configuration

### 2.1 Master metadata server

Create a user '_gfarmmd' which is used as the privilege for inter-gfmd communication.

    # useradd -c "Gfarm gfmd" -m _gfarmmd

Configure the master metadata server by the root user. Add extra -r option to the the config-gfarm command which is described in the SETUP.en docuemnt to enable replicated metadata servers.

    # config-gfarm -r

The metadata journal file is stored in /var/gfarm-metadata/journal directory by default. The location can be changed by the -j option. If there is an existing metadata server configuration, use config-gfarm-update with the -r option to update the configuration after stopping the gfmd. For config-gfarm-update, the backend PostgreSQL should be running.

    install newer version of Gfarm
    # /etc/init.d/gfmd stop
    # config-gfarm-update -r --update
    # /etc/init.d/gfmd start

For now, the PostgreSQL backend database is only supported for the replicated metadata servers.

Register host-b as a slave metadata server.
The gfmdhost command has to be invoked by a Gfarm system administrator, who has "gfarmadm" group privilege. Default setting of Gfarm system administrator is the user who is specified by the -A option of the config-gfarm command.

    $ gfmdhost -c host-b
    $ gfmdhost -l
    + master -     m (default)    host-a 601
    - slave  sync  c (default)    host-b 601

Insert a metadb_server_list directive to list all metadata hostnames and ports in gfarm2.conf.

    metadb_server_list host-a:601 host-b:601

Dump the metadata by the root user.
If you specified --prefix or --l option to the config-gfarm command, you have to specify same options to the gfdump.postgresql command too.

    # gfdump.postgresql -d -f gfarm-pgsql.dmp

Create the shared secret key for a user '_gfarmmd' which is used for inter-gfmd communication.

    # su _gfarmmd
    $ gfkey -f -p 31536000

Note that the inter-gfmd communication only supports the shared secret authentication for now.

### 2.2 Slave metadata server

Create a user '_gfarmmd' and copy the shared secret key from the master metadata server for inter-gfmd communication.

    # useradd -c "Gfarm gfmd" -m _gfarmmd
    copy the shared secret key (~_gfarmmd/.gfarm_shared_key) from host-a

Configure a slave metadata server by the root user. To enable replicated metadata servers, execute config-gfarm with the -r option.

    # config-gfarm -r

Restore the metadata by the root user.
If you specified --prefix or --l option to the config-gfarm command, you have to specify same options to the gfdump.postgresql command too.

    # gfdump.postgresql -r -f gfarm-pgsql.dmp

NOTE: Currently, you cannot do "config-gfarm -r" on multiple slave metadata servers simultaneously, even if you'd like to configure multiple slave gfmd at once. Instead, you have to do "config-gfarm -r" and then "gfdump.postgresql -r" on each slave server, and then you can proceed to next server. Copy gfarm2.conf from the master metadata server in which the metadb_server_list directive is inserted. This is because some gfarm client commands like gfmdhost may be invoked on slave metadata servers as well.

### 2.3 gfsd and client

Copy gfarm2.conf from the master metadata server in which the metadb_server_list directive is inserted.

See SETUP document for further configuration.

If the gfsd configuration is already set up, restart gfsd by the root user.
on each gfsd host:

    # /etc/init.d/gfsd restart

Now, master-slave metadata servers are set up.

### 2.4 Confirmation

gfmdhost -l displays status of metadata servers. '+' means the slave metadata server is connected to the master metadata server.

    $ gfmdhost -l
    + master -     m (default)    host-a 601
    + slave  sync  c (default)    host-b 601

gfjournal displays the metadata journal. Check whether the metadata updates are successfully transferred to the slave metadata server.

    # gfjournal /var/gfarm-metadata/journal/0000000000.gmj
    records  seqnum(min/max)            record length(min/max/ave)
         11            63/          73          24/    118/     43

Update the metadata, for example, by 'gfmkdir tmp'. If 'max' of 'seqnum' on the master metadata server is same as it on the slave metadata server, the updates are transferred.

## 3. Upgrade a slave gfmd to a master gfmd

This section explains how to upgrade a slave gfmd to a master gfmd when the master gfmd fails.

Confirm that the master gfmd has been stopped. If it is still running, stop it by the root user on host-a.

host-a:

    # /etc/init.d/gfmd stop

Note that one of the following conditions should be met when a slave gfmd is upgraded to a master gfmd.

- The current master gfmd has been stopped.
- The current master gfmd could communicate to neither any slave gfmd, any gfsd, and any client.

If one of above conditions is not met, there is a possibility of inconsistency among the master gfmd and slave gfmds.

To upgrade, send USR1 signal to the slave gfmd on host-b.
host-b:

    # kill -USR1 <process id of gfmd>

By the way, there is a gfarm_zabbix package which has a feature to automatically upgrade a slave gfmd to a master gfmd, when a fault is detected on a master gfmd.

## 4. Restore the original master gfmd

This section describe how to restart old master gfmd as a new slave gfmd.

### 4.1. restarting gfmd as a slave Invoke the following command as root user to do so:
host-a:

    # /etc/init.d/gfmd slavestart

### 4.2. Confirmation

Check whether the gfmd correctly starts as a slave or not by the gfmdhost -l command:

    $ gfmdhost -l
    + slave  sync  c (default)    host-a 601
    + master -     m (default)    host-b 601

If "+" is displayed, the gfmd correctly starts as a slave.

If "x" is displayed, please proceed to "4.3. Manual synchronization".

### 4.3. Manual synchronization

If "x" is displayed, it means automatic synchronization is impossible, because the metadata changes while host-a was stopping were too much.
In that case, you have to copy the database on host-b to host-a by the following procedure:
dump metadata by root user on host-b:

    # gfdump.postgresql -d -f gfarm-pgsql.dmp

restore the metadata by root user on host-a:

    # gfdump.postgresql -r -f gfarm-pgsql.dmp

Then, try again from "4.2 Confirmation".