如何加快MongoDB插入/秒? [英] How to speed up MongoDB Inserts/sec?

问题描述

我正在尝试最大化每秒的插入次数.我目前大约有20k插入/秒.我的性能实际上是在降低我使用的更多线程和CPU的性能(我有16个可用内核).目前，在16核双处理器计算机上，2个线程每秒执行的操作比16个线程更多.关于什么是问题的任何想法?是因为我只使用一个mongod吗?是索引编制可能会使事情变慢吗?我需要使用分片吗?我想知道是否有一种分片的方法，而且还能使数据库保持上限...

I'm trying to maximize inserts per second. I currently get around 20k inserts/sec. My performance is actually degrading the more threads and CPU I use (I have 16 cores available). 2 threads currently do more per sec than 16 threads on a 16 core dual processor machine. Any ideas on what the problem is? Is it because I'm using only one mongod? Is it indexing that could be slowing things down? Do I need to use sharding? I wonder if there's a way to shard, but also keep the database capped...

约束:必须每秒处理约30万次插入，必须具有自限性(上限)，必须可以相对快速地进行查询

Constraints: must handle around 300k inserts/sec, must be self-limiting(capped), must be query-able relatively quickly

问题空间:必须处理大型手机公司的通话记录(大约30万次插入/秒)，并使这些通话记录可查询的时间尽可能长(例如一周)

Problem Space: must handle call records for a major cellphone company (around 300k inserts/sec) and make those call records query-able for as long as possible (a week, for instance)

#!/usr/bin/perl

use strict;
use warnings;
use threads;
use threads::shared;

use MongoDB;
use Time::HiRes;

my $conn = MongoDB::Connection->new;

my $db = $conn->tutorial;

my $users = $db->users;

my $cmd = Tie::IxHash->new(
    "create"    => "users",
    "capped"    => "boolean::true",
    "max"       => 10000000,
    );

$db->run_command($cmd);

my $idx = Tie::IxHash->new(
    "background"=> "boolean::true",
);
$users->ensure_index($idx);


my $myhash =
    {
        "name"  => "James",
        "age"   => 31,
        #    "likes" => [qw/Danielle biking food games/]
    };

my $j : shared = 0;

my $numthread = 2;  # how many threads to run

my @array;
for (1..100000) {
    push (@array, $myhash);
    $j++;
}

sub thInsert {
    #my @ids = $users->batch_insert(\@array);
    #$users->bulk_insert(\@array);
    $users->batch_insert(\@array);
}

my @threads;

my $timestart = Time::HiRes::time();
push @threads, threads->new(\&thInsert) for 1..$numthread;
$_->join foreach @threads; # wait for all threads to finish
print (($j*$numthread) . "\n");
my $timeend = Time::HiRes::time();

print( (($j*$numthread)/($timeend - $timestart)) . "\n");

$users->drop();
$db->drop();

推荐答案

尽管集合级别锁定有望很快到来.通过使用更多的线程，当线程在等待释放锁的过程中互相阻塞时，您可能会引入更多的并发问题.

Writes to MongoDB currently aquire a global write lock, although collection level locking is hopefully coming soon. By using more threads you're likely introducing more concurrency problems as the threads block eachother while they wait for the lock to be released.

索引也将使您变慢，以获得最佳插入性能，理想的是在加载数据后添加它们，但这并不总是可能的，例如，如果您使用唯一索引.

Indexes will also slow you down, to get the best insert performance it's ideal to add them after you've loaded your data, however this isn't always possible, for example if you're using a unique index.

要真正最大化写入性能，最好的选择是分片.当您在多台计算机上分配写操作时，这将为您提供更好的并发性和更高的磁盘I/O容量.

To really maximise write performance, your best bet is sharding. This'll give you a much better concurrency and higher disk I/O capacity as you distribute writes across several machines.

这篇关于如何加快MongoDB插入/秒?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！