DSE - Cassandra：提交日志磁盘对性能的影响 [英] DSE - Cassandra : Commit Log Disk Impact on Performances
问题描述
我正在运行DSE 4.6.5群集（Cassandra 2.0.14.352）。
遵循datastax的指导原则，在每台机器上，我将数据目录与commitlog / saved caches目录分开：
- 快速启动
- 提交日志和已保存的缓存位于系统驱动器上：2 HDD RAID1
在执行密集写入时，使用OpsCenter监控磁盘，我看到第一个没有问题，但是我看到队列大小从更高（提交日志）平均大约300到400，尖峰高达700请求。当然，这些驱动器上的延迟也相当高...
这是否影响我的集群的性能？
您建议将提交日志和保存的缓存放在SSD上吗？
编辑 - 从节点之一添加tpstats：
[root @ dbc4〜]＃nodetool tpstats
池名活动挂起完成已阻止所有时间已阻止
ReadStage 0 0 15938 0 0
RequestResponseStage 0 0 154745533 0 0
MutationStage 1 0 306973172 0 0
ReadRepairStage 0 0 253 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 340298 0 0
CacheCleanupExecutor 0 0 0 0 0
MigrationStage 0 0 0 0 0
MemoryMeter 1 1 36284 0 0
FlushWriter 0 0 23419 0 996
ValidationExecutor 0 0 0 0 0
InternalResponseStage 0 0 0 0 0
AntiEntropyStage 0 0 0 0 0
MemtablePostFlusher 0 0 27007 0 0
MiscStage 0 0 0 0 0
PendingRangeCalculator 0 0 7 0 0
CompactionExecutor 8 10 7400 0 0
commitlog_archiver 0 0 0 0 0
HintedHandoff 0 1 222 0 0
消息类型已删除
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 49547
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
编辑2 - sar输出：
04:10:02 AM CPU％user％nice％system％iowait％steal％idle
04:10:02全部22.25 26.33 1.93 0.48 0.00 49.02
04:20 ：01 PM all 23.23 26.19 1.90 0.49 0.00 48.19
04:30:01 PM all 23.71 26.44 1.90 0.49 0.00 47.45
04:40:01 PM all 23.89 26.22 1.86 0.47 0.00 47.55
04： 50:01全部23.58 26.13 1.88 0.53 0.00 47.88
平均：全部21.60 26.12 1.71 0.56 0.00 50.01
使用OpsCenter监控磁盘，同时执行密集写入，我发现第一个没有问题，
Cassandra在内存（memtable）和commitlog（磁盘）上持续写入。
当memtable大小增加到一个阈值时，或者当您手动触发它时，Cassandra会将所有内容写入磁盘（清空memtables）。
要确保您的设置能够处理您的工作负载，您的memtables
nodetool刷新
。或者只是一个特定的键空间与
nodetool flush [keyspace] [columnfamilfy]
同时监视磁盘I / O。
如果您有高I / O等待您可以通过添加更多节点来共享工作负载，或者将数据驱动器切换到更高的吞吐量。
注意丢弃的突变是发送写入/提示的其他节点）和删除刷写器。
提交日志）平均大约300到400，尖峰高达700个请求。
这可能是您对commitlog的写入。
你的硬件服务任何其他的东西？是软件raid？你有交换禁用吗？
Cassandra最好单独工作:)所以是的，至少，把commitlog放在一个单独的（可以是更小的）磁盘上。
I'm running a DSE 4.6.5 Cluster (Cassandra 2.0.14.352). Following datastax's guidelines, on every machine, I separated the data directory from the commitlog/saved caches directories:
- data is on blazing fast drives
- commit log and saved caches are on the system drives : 2 HDD RAID1
Monitoring disks with OpsCenter while performing intensive writes, I see no issue with the first, however I see the queue size from the later (commit log) averaging around 300 to 400 with spikes up to 700 requests. Of course the latency is also fairly high on theses drives ...
Is this affecting, the performance of my cluster ? Would you recommend putting the commit log and saved cache on a SSD ? separated from the system disks ?
Thanks.
Edit - Adding tpstats from one of nodes :
[root@dbc4 ~]# nodetool tpstats
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 0 15938 0 0
RequestResponseStage 0 0 154745533 0 0
MutationStage 1 0 306973172 0 0
ReadRepairStage 0 0 253 0 0
ReplicateOnWriteStage 0 0 0 0 0
GossipStage 0 0 340298 0 0
CacheCleanupExecutor 0 0 0 0 0
MigrationStage 0 0 0 0 0
MemoryMeter 1 1 36284 0 0
FlushWriter 0 0 23419 0 996
ValidationExecutor 0 0 0 0 0
InternalResponseStage 0 0 0 0 0
AntiEntropyStage 0 0 0 0 0
MemtablePostFlusher 0 0 27007 0 0
MiscStage 0 0 0 0 0
PendingRangeCalculator 0 0 7 0 0
CompactionExecutor 8 10 7400 0 0
commitlog_archiver 0 0 0 0 0
HintedHandoff 0 1 222 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 0
PAGED_RANGE 0
BINARY 0
READ 0
MUTATION 49547
_TRACE 0
REQUEST_RESPONSE 0
COUNTER_MUTATION 0
Edit 2 - sar output :
04:10:02 AM CPU %user %nice %system %iowait %steal %idle
04:10:02 PM all 22.25 26.33 1.93 0.48 0.00 49.02
04:20:01 PM all 23.23 26.19 1.90 0.49 0.00 48.19
04:30:01 PM all 23.71 26.44 1.90 0.49 0.00 47.45
04:40:01 PM all 23.89 26.22 1.86 0.47 0.00 47.55
04:50:01 PM all 23.58 26.13 1.88 0.53 0.00 47.88
Average: all 21.60 26.12 1.71 0.56 0.00 50.01
Monitoring disks with OpsCenter while performing intensive writes, I see no issue with the first,
Cassandra persists writes in memory (memtable) and on the commitlog (disk).
When the memtable size grows to a threshold, or when you manually trigger it, Cassandra will write everything to disk (flush the memtables).
To make sure your setup is capable of handling your workload try to manually flush all your memtables
nodetool flush
on a node. Or just a specific keyspace with
nodetool flush [keyspace] [columnfamilfy]
At the same time monitor your disks I/O.
If you have high I/O wait you can either share the workload by adding more nodes, or switch the data drives to better one with higher throughput.
Keep an eye to dropped mutations (can be other nodes sending the writes/hints) and dropped flush-writer.
I see the queue size from the later (commit log) averaging around 300 to 400 with spikes up to 700 requests.
This will probably be your writes to the commitlog. Is your hardware serving any other thing? Is it software raid? Do you have swap disabled?
Cassandra works best alone :) So yes, put at least, the commitlog on a separate (can be smaller) disk.
这篇关于DSE - Cassandra：提交日志磁盘对性能的影响的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
