如何才达到每秒10插入与Azure存储表 [英] How to achive more 10 inserts per second with azure storage tables
问题描述
我写简单WorkerRole,在表中添加测试数据。插入的code是这样的。
I write simple WorkerRole that add test data in to table. The code of inserts is like this.
var TableClient = this.StorageAccount.CreateCloudTableClient();
TableClient.CreateTableIfNotExist(TableName);
var Context = TableClient.GetDataServiceContext();
this.Context.AddObject(TableName, obj);
this.Context.SaveChanges();
这code为每个客户端请求的运行。我做1-30客户端线程测试。
我有各种尺寸的实例的各个计数许多改掉。我不知道我做错了,但我不能达到每秒10多插入。
如果有人知道如何加快速度,请告诉我。
谢谢
This code runs for each client requests. I do test with 1-30 client threads. I have many trys with various count of instances of various sizes. I don't know what I do wrong but I can't reach more 10 inserts per second. If someone know how to increase speed please advise me. Thanks
更新
- CreateTableIfNotExist的简化版,去掉差异使我插入的测试。
- 开关模式expect100Continue =假useNagleAlgorithm =false的制作时间短效果时,插入率跳30-40 IPS。但随后,在30秒后插入率回落至6 ips的50%超时。
推荐答案
为了加快速度,你应该使用批处理事务(实体集团交易),使您可以在单个请求中提交多达100个项目:
To speed things up you should use batch transactions (Entity Group Transactions), allowing you to commit up to 100 items within a single request:
foreach (var item in myItemsToAdd)
{
this.Context.AddObject(TableName, item);
}
this.Context.SaveChanges(SaveChangesOptions.Batch);
您可以用<一本结合起来href=\"http://msdn.microsoft.com/en-us/library/system.collections.concurrent.partitioner.create.aspx\">Partitioner.Create (+进行AsParallel)发送每批100个项目,以使事情变得真快不同的线程/内核的多个请求。
You can combine this with Partitioner.Create (+ AsParallel) to send multiple requests on different threads/cores per batch of 100 items to make things really fast.
但是这样做这一切之前,通过使用批处理事务的局限性阅读(100个项目,每交易1分区...)。
But before doing all of this, read through the limitations of using batch transactions (100 items, 1 partition per transaction, ...).
更新:
由于不能使用事务这里有一些小窍门。看看<一个href=\"http://social.msdn.microsoft.com/Forums/en-US/windowsazuredata/thread/d84ba34b-b0e0-4961-a167-bbe7618beb83\">this MSDN线程有关使用表存储时提高性能。我写了一些code向您展示的区别:
Since you can't use transactions here are some other tips. Take a look at this MSDN thread about improving performance when using table storage. I wrote some code to show you the difference:
private static void SequentialInserts(CloudTableClient client)
{
var context = client.GetDataServiceContext();
Trace.WriteLine("Starting sequential inserts.");
var stopwatch = new Stopwatch();
stopwatch.Start();
for (int i = 0; i < 1000; i++)
{
Trace.WriteLine(String.Format("Adding item {0}. Thread ID: {1}", i, Thread.CurrentThread.ManagedThreadId));
context.AddObject(TABLENAME, new MyEntity()
{
Date = DateTime.UtcNow,
PartitionKey = "Test",
RowKey = Guid.NewGuid().ToString(),
Text = String.Format("Item {0} - {1}", i, Guid.NewGuid().ToString())
});
context.SaveChanges();
}
stopwatch.Stop();
Trace.WriteLine("Done in: " + stopwatch.Elapsed.ToString());
}
所以,我第一次运行这个我得到以下的输出:
So, the first time I run this I get the following output:
Starting sequential inserts.
Adding item 0. Thread ID: 10
Adding item 1. Thread ID: 10
..
Adding item 999. Thread ID: 10
Done in: 00:03:39.9675521
这需要超过3分钟,增加1000个项目。现在,我改变了基于在MSDN论坛上提示的app.config(MAXCONNECTION应该是12 * CPU核心数):
It takes more than 3 minutes to add 1000 items. Now, I changed the app.config based on the tips on the MSDN forum (maxconnection should be 12 * number of CPU cores):
<system.net>
<settings>
<servicePointManager expect100Continue="false" useNagleAlgorithm="false"/>
</settings>
<connectionManagement>
<add address = "*" maxconnection = "48" />
</connectionManagement>
</system.net>
和重新运行应用程序之后,我得到这样的输出:
And after running the application again I get this output:
Starting sequential inserts.
Adding item 0. Thread ID: 10
Adding item 1. Thread ID: 10
..
Adding item 999. Thread ID: 10
Done in: 00:00:18.9342480
从3分钟至18秒。一有什么区别!但是,我们可以做得更好。下面是一些code将分别使用一个分区程序中的所有项目(刀片将并行发生):
From over 3 minutes to 18 seconds. What a difference! But we can do even better. Here is some code inserts all items using a Partitioner (inserts will happen in parallel):
private static void ParallelInserts(CloudTableClient client)
{
Trace.WriteLine("Starting parallel inserts.");
var stopwatch = new Stopwatch();
stopwatch.Start();
var partitioner = Partitioner.Create(0, 1000, 10);
var options = new ParallelOptions { MaxDegreeOfParallelism = 8 };
Parallel.ForEach(partitioner, options, range =>
{
var context = client.GetDataServiceContext();
for (int i = range.Item1; i < range.Item2; i++)
{
Trace.WriteLine(String.Format("Adding item {0}. Thread ID: {1}", i, Thread.CurrentThread.ManagedThreadId));
context.AddObject(TABLENAME, new MyEntity()
{
Date = DateTime.UtcNow,
PartitionKey = "Test",
RowKey = Guid.NewGuid().ToString(),
Text = String.Format("Item {0} - {1}", i, Guid.NewGuid().ToString())
});
context.SaveChanges();
}
});
stopwatch.Stop();
Trace.WriteLine("Done in: " + stopwatch.Elapsed.ToString());
}
和结果:
Starting parallel inserts.
Adding item 0. Thread ID: 10
Adding item 10. Thread ID: 18
Adding item 999. Thread ID: 16
..
Done in: 00:00:04.6041978
瞧,从3m39s我们下降到18岁,现在我们甚至降至 4S
这篇关于如何才达到每秒10插入与Azure存储表的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!