Riak数据库在短时间内失败 [英] Riak database fails after a short period

问题描述

我创建了一个简单的erlang应用程序，它定期收集所需的数据并将其放在一个riak数据库中。

当我启动我的应用程序它运行顺利..但经过一段时间后，它坚持PUT请求riak数据库变得太慢..它是从我的应用程序的日志：

  2013-06-26 12：44：09.090 [info]< 0.60.0> [16476 ms]中处理的数据
 2013-06-26 12：45：51.472 [info]< 0.60.0> [18793 ms]处理的数据
 ... 
 2013-06-26 12：57：28.138 [info]< 0.60.0> [15135 ms]中处理的数据
 2013-06-26 13：07：01.484 [info]< 0.60.0>以[488420 ms]处理的数据
 2013-06-26 14：03：11.561 [info]< 0.60.0>在[3370075 ms]处理的数据
  

在riak崩溃日志中，我可以看到很多消息，如

  2013-06-26 17:06:20 = CRASH REPORT ==== 
 crasher：
 initial调用：riak_kv_index_hashtree：init / 1 
 pid：< 0.13660.7> 
 registered_name：[] 
异常退出：{{badmatch，{error，{db_open，IO error：./data/anti_entropy/ 
 433883298582611803841718934712646521460354973696 / MANIFEST-000004：
不能分配内存}}}，[{hashtree，new_segment_store，2，
 [{file，src / hashtree.erl}，{line，499}]}，
 {hashtree，new， 2，[{file，src / hashtree.erl}，{line，215}]}，
 {riak_kv_index_hashtree，do_new_tree，2，
 [{file，src / riak_kv_index_hashtree.erl} ，
 {line，426}]}，{lists，foldl，3，[{file，lists.erl}，
 {line，1197}]}，{riak_kv_index_hashtree，
 init_trees，2，[{file，src / riak_kv_index_hashtree.erl}，
 {line，368}]}，{riak_kv_index_hashtree，init，1，
 [{file，src / riak_kv_index_hashtree.erl }，
 {line，225}]}，{gen_server，init_it，6，[{file，gen_server.erl}，{line，304}]}，
 {proc_lib，init_p_do_apply， 3，[{file，proc_lib.erl}，{line，227}]}]}，
 [{gen_server，init_it，6，[{file，gen_server.erl}，{line，328 }]}，
 {proc_lib，init_p_do_ apply，3，[{file，proc_lib.erl}，{line，227}]}}} 
祖先：[< 0.955.0>，riak_core_vnode_sup，riak_core_sup，< 0.129.0] 
消息：[] 
链接：[] 
字典：[] 
 trap_exit：false 
状态：运行
 heap_size：1597 
 stack_size： 24 
减少：593 
邻居：
  

我可以看到相同的行为Amazon AWS和本地虚拟机。我的VM是相当小的512-1024 mb .. AWS是微，所以它有相同的内存量。

目前没有集群。只有单个节点与Riak和我的应用程序运行。

我已经检查了riak文档，他们建议做的基本事情是增加ulimit和更新sysctl。所以，我的服务器ulimit显示：ulimit -n 65536 AND sysctl按推荐更新。

我尝试过bitcask和eleveldb，但结果是一样的。

目前，我无法弄清楚什么是坏的，为什么riak不能分配内存..
谢谢。

解决方案

1GB RAM对于Riak节点来说相当小，甚至更多，因为您也在运行您的应用程序。 Riak中的默认设置针对的是具有更多RAM和处理能力的环境，因此您需要调整默认的设置，以使其正常工作。以下是一些可能有帮助的指针：

1. 由于您只有一个节点，因此通过设置
   { anti_entropy，{off，[]}} 。


2. 减小戒指尺寸。这将限制您的扩展能力，但最有可能需要它才能使其工作。一个合适的起始值可能是16，但也可能低至8。


3. 更改默认的桶属性，使您具有 n_val ， r ， w ， dw 和 rw 全部设置为1，否则将会将每个记录的多个副本写入磁盘。当您扩展并添加更多节点时，这些将需要增加。


4. 由于比特币需要将所有密钥保存在内存中，因此使用 leveldb 作为后端。然而，您很可能需要显着减少写入缓冲区以及缓存的大小。您可能需要尝试找到合适的级别。


5. 由于这种环境最可能太小，无法继续运行mapreduce，您还可以设置 map_js_vm_count 和 reduce_js_vm_count 配置参数为0，以节省一些额外的内存。

I crerated a simple erlang application which periodically collects required data and puts it in a riak database.

As I start my application it runs smoothly.. but after a period of time it stucks as PUT requests to riak database becomes too slow.. It is logs from my app:

2013-06-26 12:44:09.090 [info] <0.60.0> data processed in [16476 ms]
2013-06-26 12:45:51.472 [info] <0.60.0> data processed in [18793 ms]
...
2013-06-26 12:57:28.138 [info] <0.60.0> data processed in [15135 ms]
2013-06-26 13:07:01.484 [info] <0.60.0> data processed in [488420 ms]
2013-06-26 14:03:11.561 [info] <0.60.0> data processed in [3370075 ms]

In riak crash logs I can see a lot of messages like

2013-06-26 17:06:20 =CRASH REPORT====
crasher:
initial call: riak_kv_index_hashtree:init/1
pid: <0.13660.7>
registered_name: []
exception exit: {{{badmatch,{error,{db_open,"IO error: ./data/anti_entropy/
    433883298582611803841718934712646521460354973696/MANIFEST-000004: 
    Cannot allocate memory"}}}, [{hashtree,new_segment_store,2,
    [{file,"src/hashtree.erl"},{line,499}]},
    {hashtree,new,2,[{file,"src/hashtree.erl"},{line,215}]},
    {riak_kv_index_hashtree,do_new_tree,2,
    [{file,"src/riak_kv_index_hashtree.erl"},
    {line,426}]},{lists,foldl,3,[{file,"lists.erl"},
    {line,1197}]},{riak_kv_index_hashtree,
    init_trees,2,[{file,"src/riak_kv_index_hashtree.erl"},
    {line,368}]},{riak_kv_index_hashtree,init,1,
    [{file,"src/riak_kv_index_hashtree.erl"},
    {line,225}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]},
    {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]},
    [{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},
    {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
ancestors: [<0.955.0>,riak_core_vnode_sup,riak_core_sup,<0.129.0>]
messages: []
links: []
dictionary: []
trap_exit: false
status: running
heap_size: 1597
stack_size: 24
reductions: 593
neighbours:

I can see the same behavior on Amazon AWS and local virtual machine. my VM's are quite small 512-1024 mb.. AWS is Micro, so it has the same amount of memory.

There are no cluster currently. Just single node with Riak and my app running on it.

I've checked riak documentation and basic things they recommend to do is to increase ulimit and to update sysctl. So, my server ulimit shows: ulimit -n 65536 AND sysctl updated as recommended.

I've tried bitcask and eleveldb, but result is the same.

Currently, I can't figure out what is broken and why riak Cannot allocate memory.. Thanks.

解决方案

1GB RAM is quite small for a Riak node, and even more so as you are also running your application there. The default settings in Riak are targeted at environments with considerably more RAM and processing power, so you will need to tweak the default settings in order to get it to work. Here are a few pointers that may help:

1. As you only have one node, disable AAE by setting {anti_entropy, {off, []}}.
2. Reduce the ring size. This will limit your ability to scale out, but is most likely required in order to get it to work. A suitable starting value could perhaps be 16, but possibly even as low as 8.
3. Change default bucket properties so that you have n_val, r, w, dw and rw all set to 1, as you otherwise will be writing multiple copies of every record to disk. These will need to be increased when you scale out and add more nodes.
4. As bitcask requires all keys to be kept in memory, it is probably a good idea to instead use leveldb as a backend. You will however most likely need to reduce the size of write buffers as well as the cache significantly. You may need to experiment to find a suitable level.
5. As this environment most likely is too small to be able to run mapreduce on anyway, you can also set the map_js_vm_count and reduce_js_vm_count configuration parameters to 0 in order to save some additional memory.

这篇关于Riak数据库在短时间内失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！