DynamoDB 本地糟糕的性能 [英] DynamoDB Local Terrible Performance

问题描述

DynamoDB local 对我的表执行单个 put 操作需要 100 多毫秒.文档说本地数据库的吞吐量被忽略，仅受硬盘/计算机速度的限制.

DynamoDB local is taking 100+ ms to perform a single put operation against my table. The docs say that throughput is ignored for local dbs, and is only limited by the speed of the hard disk/computer.

与 mongodb 相比，我的写入吞吐量比应有的慢 100 倍.我可以做些什么来加快写入本地 DynamoDB 的速度吗?

Compared to mongodb my write throughput is 100x slower than it should be. Is there something I can do to speed writes to DynamoDB local up?

我会尝试批量放置，但问题仍然存在.在这一点上，我需要数年时间才能输入我的(相当大的)测试数据.

I will try batch puts, but the problem still remains. At this point, it's going to take me years to input my (rather large) test data.

我使用 Clojure 和 Faraday 作为我的客户端 api，但已确认这不是瓶颈.

I'm using Clojure and Faraday as my client api, but have confirmed that is not the bottleneck.

我以 25 per 的速度实施了批次，这将总进度减慢了大约 25 倍 :).因此，即使是批量，我的每个项目的写入速度也约为 120 毫秒.

I've implemented batches at 25 per, which slowed down the total progress by about a factor of 25 :). So, even with batches, I'm getting a write speed of about 120 ms per item.

使用 Mongo，即使使用保守的 WriteConcern/ACKNOWLEDGED 标志，每个项目也会产生大约 250 微秒(快约 500 倍)，这甚至不需要发送批次.所以问题不是我的硬盘或操作系统.

Using Mongo, even with conservative WriteConcern/ACKNOWLEDGED flag, results in about 250 microseconds per item (~500x faster), and that's without even needing to send batches. So it's not my harddrive or OS that's the problem.

推荐答案

你问这个问题已经有一段时间了，但我还是想提供一些背景信息.

It's been a while since you asked your question but I still want to give some background information.

MongoDB 之所以如此之快，是因为它在将更新写入磁盘之前将其缓冲在内存中.您可以在官方 MongoDB 常见问题解答 这里

MongoDB is so fast because it buffers the updates in memory before writing them to disk. You can read more about this in the official MongoDB FAQ here

另一方面，DynamoDBLocal 非常慢，因为它在后台使用 SQLite 数据库(您可以通过使用 SQLite 浏览器打开 DynamoDBLocal 目录中的 *.db 文件来检查这一点)和与其他数据库相比，SQLite 的写入速度非常慢.更多信息这里.

DynamoDBLocal on the other hand is quite slow because it uses a SQLite database behind the scenes (You can check this by opening the *.db file in the DynamoDBLocal directory with a SQLite Browser) and SQLite has very slow write speeds compared to other databases. More info here.

DynamoDBLocal 只是一个用于本地开发的基本工具，它将 DynamoDB API 的子集包装在一个简单的 SQLite 数据库周围.当然，真正的 DynamoDB 不依赖 SQLite，而且速度更快.

DynamoDBLocal is just a basic tool for local development that wraps a subset of the DynamoDB API around a simple SQLite database. The real DynamoDB, of course, does not rely on SQLite and is much faster.

这篇关于DynamoDB 本地糟糕的性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！