Errors & Messages

This is not a comprehensive listing of every error that Riak may encounter – screws fall out all of the time, the world is an imperfect place. This is an attempt at capturing the most common recent errors that users do encounter, as well as give some description to non critical error atoms which you may find in the logs.

Discovering the source of an error can take some detective work, since one error can cause a cascade of errors.

The tables in this document do not specify which logs these error messages may appear in. Depending upon your log configuration some may appear more often (i.e., if you set the log to debug), while others may output to your console (eg. if you tee’d your output or started as riak console).

You can optionally customize your log message format via the lager_default_formatter field under lager in app.config. If you do, your messages will look different from those shown in this document.

Finally, this document is organized to be able to lookup portions of a log message, since printing every variation would be a bit unwieldy. For example, this message:

12:34:27.999 [error] gen_server riak_core_capability terminated with reason:\
no function clause matching orddict:fetch('riak@192.168.2.81', []) line 72

Starts with a date (12:34:27.999), followed by the log severity ([error]), with a message formatted by lager (found in the Lager table below as gen_server Mod terminated with reason: Reason)

Lager Formats

Riak’s main logging mechanism is the project Lager, so it’s good to note some of the more common message formats. In almost every case the reasons for the error are described as variables, such as Reason of Mod (meaning the Erlang module which is generally the source of the error).

Riak does not format all error messages that it receives into human-readable sentences. However, It does output errors as objects.

The above example error message corresponds with the first message in this table, where the Erlang Mod value is riak_core_capability and the reason was an Erlang error: no function clause matching orddict:fetch('riak@192.168.2.81', []) line 72.

Error Atoms

Since Erlang programming support is a “happy path/fail fast” style, one of the more common error log strings you might encounter contain {error,{badmatch,{.... This is Erlang’s way of telling you that an unexpected value was assigned, so these errors can prefix the more descriptive parts. In this case, {error,{badmatch,{... prefixes the more interesting insufficient_vnodes_available error, which can be found in the riak_kv table later on in this document.

2012-01-13 02:30:37.015 [error] <0.116.0> webmachine error: path="/riak/contexts"\
{error,{error,{badmatch,{error,insufficient_vnodes_available}},\
[{riak_kv_wm_keylist,produce_bucket_body,2},{webmachine_resource,resource_call,3},\
{webmachine_resour,resource_call,1},{webmachine_decision_core,decision,1},\
{webmachine_decision_core,handle_request,2},\
{webmachine_mochiweb,loop,1},{mochiweb_http,headers,5}]}}

Erlang Errors

Although relatively rare once a Riak cluster is running in production, users new to Riak or Erlang occasionally encounter errors on initial installation. These spring from a setup Erlang does not expect, generally due to network, permission, or configuration problems.

Riak Errors and Messages

Many KV errors have prescriptive messages. For such cases we leave it to Riak to explain the correct course of action. For example, the map/reduce parse_input phase will respond like this when it encounters an invalid input:

For the remaining common error codes, they are often marked by Erlang atoms (and quite often wrapped within an {error,{badmatch,{... tuple, as described in the Error section above). This table lays out those terse error codes and related log messages, if they exist.

Riak Core

Riak Core is the underlying implementation for KV. These are errors originating from that framework, and can appear whether you use KV, Search, or any Core implementation.

Riak KV

Riak KV is the key/value implementation, generally just considered to be Riak proper. This is the source of most of the code, and consequently, most of the error messages.

Backend Errors

These errors tend to stem from server-based problems. Backends are sensitive to low or corrupt disk or memory resources, native code, and configuration differences between nodes. Conversely, a network issue is unlikely to affect a backend.

JavaScript

These are some errors related to JavaScript pre-commit functions, MapReduce functions, or simply the management of the pool of JavaScript VMs. If you do not use JavaScript, these should not be encountered. If they are, check your configuration for high *js_vm* values or as an epiphenomenon to a real issue, such as low resources.

MapReduce

These are possible errors logged by Riak’s MapReduce implementation, both legacy as well as Pipe. If you never use or call MapReduce, you should not run across these.

Specific messages

Although you can put together many error causes with the tables above, here are some common yet esoteric messages with known causes and solutions.

More

1. Ensure node inter-communication

   • Check riak-admin member-status and ensure the cluster is valid.
   • Check riak-admin ring-status and ensure the ring and vnodes are communicating as expected.
   • Ensure your machine does not have a firewall or other issue that prevents traffic to the remote node.
   • Your remote vm.args -setcookie must be the same value for every node in the cluster.
   • The vm.args -name value must not change after joining the node (unless you use riak-admin cluster replace).

2. Run LevelDB compaction

   1. find . -name "LOG" -exec grep -l 'Compaction error' {} \; (Finding one compaction error is interesting, more than one might be a strong indication of a hardware or OS bug)
   2. Stop Riak on the node: riak stop
   3. Start an Erlang session (do not start riak, we just want Erlang)

   4. From the Erlang console perform the following command to open the LevelDB database

      [application:set_env(eleveldb, Var, Val) || {Var, Val} <-
      [{max_open_files, 2000},
      {block_size, 1048576},
      {cache_size, 20*1024*1024*1024},
      {sync, false},
      {data_root, "/var/db/riak/leveldb"}]].
      

   5. For each of the corrupted LevelDB databases (found by find . -name "LOG" -exec | grep -l 'Compaction error' {} \;) run this command substituting in the proper vnode number.

      eleveldb:repair("/var/db/riak/leveldb/442446784738847563128068650529343492278651453440", []).
      

   6. When all have finished successfully you may restart the node: riak start

   7. Check for proper operation by looking at log files in /var/log/riak and in the LOG files in the effected LevelDB vnodes.