以压缩格式存储mongodb数据 [英] Store mongodb data in compressed format

问题描述

我使用mongodb来存储网页的原始HTML数据使用scrapy框架。在一天的网络刮擦25GB磁盘空间被填满。是否有以压缩格式存储原始数据的方法。

I am using mongodb to store raw HTML data of web pages using scrapy framework. In one day of web scraping 25GB disk space is filled up. Is there a way to store raw data in compressed format.

推荐答案

没有内置的压缩​​。一些操作系统提供磁盘/文件压缩，但是如果你想要更多的控制，我建议你使用库使用你所使用的任何编程语言进行压缩，并手动控制压缩。

There's nothing built in for compression. Some operating systems offer disk/file compression, but if you want more control, I'd suggest you compress it using a library for whatever programming language you're using and manually control the compression.

例如，NodeJs为此提供了简单的方便方法： http://nodejs.org/api/zlib.html#zlib_examples

For example, NodeJs offers simple convenience methods for this: http://nodejs.org/api/zlib.html#zlib_examples

如果您选择切换到3.0的新存储引擎WiredTiger，您可以选择几种类型的压缩，如记录此处。当然，您需要测试生产工作负载中的此更改，以确定是否额外的CPU利用率值得接受压缩。

If you choose to switch to the new storage engine WiredTiger which ships with 3.0, you can choose between several types of compression as documented here. Of course, you'll want to test this change in production workloads to find if the additional CPU utilization is worth the compression received.

这篇关于以压缩格式存储mongodb数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！