HBase - WAL 和 MemStore 有什么区别? [英] HBase - What's the difference between WAL and MemStore?

问题描述

我正在尝试了解 HBase 架构.我可以看到两个不同的术语用于同一目的.

Write Ahead Logs 和 Memstore，两者都用于存储尚未持久化到永久存储的新数据.>

WAL 和 MemStore 有什么区别?

更新:

WAL - 用于在服务器崩溃时恢复尚未持久化的数据.MemStore - 将更新存储在内存中作为排序的键值.

在将数据写入磁盘之前，似乎有很多重复的数据.

解决方案

WAL 用于恢复而不是用于数据重复.(

I am trying to understand the HBase architecture. I can see two different terms are used for same purpose.

Write Ahead Logs and Memstore, both are used to store new data that hasn't yet been persisted to permanent storage.

What's the difference between WAL and MemStore?

Update:

WAL - is used to recover not-yet-persisted data in case a server crashes. MemStore - stores updates in memory as Sorted Keyvalue.

It seems lot of duplication of data before writing the data to Disk.

解决方案

WAL is for recovery NOT for data duplication.(further see my answer here)

Pls go through below to understand more...

• A Hbase Store hosts a MemStore and 0 or more StoreFiles (HFiles). A Store corresponds to a column family for a table for a given region.

• The Write Ahead Log (WAL) records all changes to data in HBase, to file-based storage. if a RegionServer crashes or becomes unavailable before the MemStore is flushed, the WAL ensures that the changes to the data can be replayed.

• With a single WAL per RegionServer, the RegionServer must write to the WAL serially, because HDFS files must be sequential. This causes the WAL to be a performance bottleneck.

• WAL can be disabled to improve performance bottleneck. This is done by calling the Hbase client field

Mutation.writeToWAL(false)

General Note : Its general practice that while doing bulkloading data, WAL is disabled to get speed. But side effect is if you disable WAL you cant get back data to replay if in case any memory crashes.

More over if you use solr+ HBASE + LILY, i.e LILY Morphiline NRT indexes with hbase then it will work on WAL if you disable WAL for performance reasons, then Solr NRT indexing wont work. since Lily works on WAL.

please have a look at Hbase architecture section

这篇关于HBase - WAL 和 MemStore 有什么区别?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！