Upgrading to Riak KV 2.2.3

Overview

You can upgrade one node or your whole cluster to Riak KV 2.2.3 by following the instructions below.

General Process

For every node in the cluster:

1. Stop Riak KV.
2. Back up the Riak /etc, /data, and /basho-patches directories.
3. Remove your /basho-patches directory.
4. Upgrade Riak KV.
   • If you are upgrading from OSS to EE, uninstall your OSS KV package before upgrading.
5. (Optional) If you would like to potentially downgrade at some point, update your advanced.config file to opt-out of the AAE updates.
6. If you’re upgrading from OSS to EE, apply your customized settings to vm.args and app.config
7. If you’re using MDC replication to clusters with versions less than 2.2.0, update your advanced.config file to over-ride the default bucket properties for compatibility.
8. Start Riak KV.
9. Verify Riak KV is running the upgraded version.
10. Wait for the riak_kv service to start.
11. Wait for any hinted handoffs to complete.

Before starting the rolling upgrade process on your cluster, check out the Upgrading Riak KV: Production Checklist, which covers details and questions to consider before upgrading.

Data File Format Changes

Riak KV 2.2 introduces on-disk data file format changes that can impact the upgrade/downgrade process:

• Changes to active anti-entropy related to inconsistent hashing.
• Upgrading to Solr 4.10.4 for Riak search.

Components That Complicate Downgrades

We do our best to make all features that change data formats on disk opt-in; however, some features may be introduced that we either believe are so important that we automatically opt-in users on upgrade or there is no way to provide direct backward compatibility. Downgrading environments with these features can require more effort or might not be possible.

• Automatic features alter the data format on disk, but are considered important enough for users to be automatically opted-in.
• Required features must be accepted as a part of the upgrade. Internal Solr version upgrades that change the data format on disk are an example of a required feature upgrade.
• One Way features, when enabled, will make a clean downgrade of a cluster impossible.

When Downgrading is No Longer an Option

If you decide to upgrade to version 2.2, you can still downgrade your cluster to an earlier version of Riak KV if you wish, unless you perform one of the following actions in your cluster:

• Enable LZ4 Compression in LevelDB
• Enable Global Expiration in LevelDB

If you use other new features, such as the HyperLogLog data type, you can still downgrade your cluster, but you will no longer be able to use those features or access data in new formats after the downgrade.

Upgrading process

1. Stop Riak KV on the node you are going to upgrade:

riak stop

2. Back up your /etc (app.config and vm.args), /data, and /basho-patches directories.

sudo tar -czf riak_backup.tar.gz /var/lib/riak /etc/riak /usr/lib64/riak/lib/basho-patches

sudo tar -czf riak_backup.tar.gz /var/lib/riak /etc/riak /usr/lib/riak/lib/basho-patches

3. Remove your /basho-patches directory:

sudo rm -rf /usr/lib64/riak/lib/basho-patches/*

sudo rm -rf /usr/lib/riak/lib/basho-patches*

4. Upgrade Riak KV:

sudo rpm -Uvh »riak_package_name«.rpm

sudo dpkg -i »riak_package_name«.deb

5.a. (Optional) If you would like to keep your AAE trees in a format that will facilitate downgrading, the capability override should be in the riak_kv proplist of the advanced.config file:

   {riak_kv, [
     {override_capability, [
       {object_hash_version, [{use, legacy}] }
     ]}
   ]}

5.b. (EE Only)If you are upgrading from Riak KV OSS to Riak KV EE, you must perform the following steps before moving on:

• A standard package uninstall should not have removed your data directories, but if it did, move your backup to where the data directory should be.
• Then copy any customizations from your backed-up vm.args to the riak_ee installed vm.args file (these files may be identical).
• The app.config file from riak_ee will be significantly different from your backed-up file. It will have many new sections along with the original ones. Copy the customizations from your original app.config file into the appropriate sections in the new one. Ensure that the following sections are present in app.config:
  • riak_core — the cluster_mgr setting must be present. See MDC v3 Configuration for more information.
  • riak_repl — See MDC v3 Configuration for more information.
  • riak_jmx — See JMX Monitoring for more information.
  • snmp — See SNMP for more information.

5.c. (EE Only with MDC)If you need to replicate to clusters with versions less than 2.2.0, the capability override for bucket properties should be in the riak_repl proplist of the advanced.config file:

   {riak_repl, [
     {override_capability, [
       {default_bucket_props_hash, [{use, [consistent, datatype, n_val, allow_mult, last_write_wins]}] }
     ]}
   ]}

Once all of the clusters have been upgraded to version 2.2.0 or greater, this override should be removed.

6. Restart Riak KV:

riak start

7. Verify that Riak KV is running the new version:

riak version

8. Wait for the riak_kv service to start:

riak-admin wait-for-service riak_kv »target_node«

• »target_node« is the node which you have just upgraded (e.g. riak@192.168.1.11)

9. Wait for any hinted handoff transfers to complete:

riak-admin transfers

• While the node was offline, other nodes may have accepted writes on its behalf. This data is transferred to the node when it becomes available.

10. Repeat the process for the remaining nodes in the cluster.

Riak Patches

After upgrading, you should ensure that any custom patches contained in the basho-patches directory are examined to determine their application to the upgraded version. You can find this information in the Release Notes.

If you find that patches no longer apply to the upgraded version, you should remove them from the basho-patches directory prior to operating the node in production.

The following lists locations of the basho-patches directory for each supported operating system:

• CentOS & RHEL Linux: /usr/lib64/riak/lib/basho-patches
• Debian & Ubuntu Linux: /usr/lib/riak/lib/basho-patches
• FreeBSD: /usr/local/lib/riak/lib/basho-patches
• SmartOS: /opt/local/lib/riak/lib/basho-patches
• Solaris 10: /opt/riak/lib/basho-patches

Riaknostic

It is a good idea to also verify some basic configuration and general health of the Riak KV node after upgrading by using Riak KV’s built-in diagnostic utility Riaknostic.

Ensure that Riak KV is running on the node, and issue the following command:

riak-admin diag

Make the recommended changes from the command output to ensure optimal node operation.