Replacing a Node
At some point, for various reasons, you might need to replace a node in your Riak cluster (which is different from recovering a failed node). Here is the recommended way to go about replacing a node.
Back up your data directory on the node in question. In this example scenario, we’ll call the node
riak4:sudo tar -czf riak_backup.tar.gz /var/lib/riak /etc/riakIf you have any unforeseen issues at any point in the node replacement process, you can restore the node’s data from this backup.
Download and install Riak on the new node you wish to bring into the cluster and have it replace the
riak4node. We’ll call the new noderiak7for the purpose of this example.Start the new
riak7node withriak start:riak startPlan the join of the new
riak7node to an existing node already participating in the cluster; for exampleriak0with theriak-admin cluster joincommand executed on the newriak7node:riak-admin cluster join riak0Plan the replacement of the existing
riak4node with the newriak7node using theriak-admin cluster replacecommand:riak-admin cluster replace riak4 riak7Single NodesIf a node is started singly using default settings (as, for example, you might do when you are building your first test environment), you will need to remove the ring files from the data directory after you edit `/etc/vm.args`. `riak-admin cluster replace` will not work as the node has not been joined to a cluster.Examine the proposed cluster changes with the
riak-admin cluster plancommand executed on the newriak7node:riak-admin cluster planIf the changes are correct, you can commit them with the
riak-admin cluster commitcommand:riak-admin cluster commitIf you need to clear the proposed plan and start over, use
riak-admin cluster clear:riak-admin cluster clear
Once you have successfully replaced the node, it should begin leaving
the cluster. You can check on ring readiness after replacing the node
with the riak-admin ringready
and riak-admin member-status
commands.
You’ll need to make sure that no other ring changes occur between the time when you start the new node and the ring settles with the new IP info.
The ring is considered settled when the new node reports true when you run
the riak-admin ringready command.
