下载具有多个进程的多个文件时，Gsutil会占用大量内存 [英] Gsutil uses a lot of memory when download multiple files with a lot of processes
问题描述
我需要使用gsutil下载多个文件，并且我注意到gsutil在下载多个文件时会占用大量内存. (下载三个每个都有9个进程的2G文件时，大约需要1-2 GB内存).有没有办法调整gsutil的记忆体使用量?这对我来说很重要，因为我在GKE中运行gsutil，如果使用过多的内存(超过限制)，容器将被杀死
I need to download multiple files with gsutil and I notices that gsutil uses a lot of memory when downloading multiple files. (Around 1-2 GB ram when download three 2G files with 9 processes each). Is there a way to tune memory usage of gsutil? This is kind of important to me because I am running gsutil in GKE, and a container will get killed if use too much memory (more than limit)
另一个问题:gsutil似乎无法在单个命令中下载具有相同名称的文件(一个将覆盖另一个?).因此，我没有使用-m选项.相反，我使用单个gsutil命令下载每个文件: gsutil -o"GSUtil:parallel_thread_count = 1" -o"GSUtil:sliced_object_download_component_size = 250M" -o"GSUtil:sliced_object_download_max_components = 9" -o"GSUtil:parallel_process_count = 9" cp存储桶/文件desFile
Another issue: it seems like gsutil can not download files with the same name in a single command (one will overwrite the other?). So I am not using the -m option. Instead I am downloading each file with a single gsutil command: gsutil -o "GSUtil:parallel_thread_count=1" -o "GSUtil:sliced_object_download_component_size=250M" -o "GSUtil:sliced_object_download_max_components=9" -o "GSUtil:parallel_process_count=9" cp bucket/file desFile
推荐答案
我确实测试过下载2GB文件，并更改-o"GSUtil:parallel_process_count = X"更改了Debian和Ubuntu上的内存消耗:
I did test download the 2GB file and changing -o "GSUtil:parallel_process_count=X" changes memory consumption on Debian and Ubuntu:
- 1个并行进程:85MB
- 5个并行进程:125MB
- 10个平行 进程:165MB
- 50个并行进程:310MB
- 1 parallel process: 85MB
- 5 parallel processes: 125MB
- 10 parallel processes: 165MB
- 50 paraller processes: 310MB
如果您在CentOS上使用gsutil在GKE上遇到内核恐慌问题容器映像，切换到Ubuntu映像应该会有所帮助.
If you have kernel panic issues on GKE using gsutil with CentOS container image, switching to Ubuntu image should help.
如果内存消耗太高，无法同时下载3个文件，则可以考虑仅使用1或2次下载.
If the memory consumption is too high for 3 files simultaneous download, you can consider using only 1 or 2 downloads.
这篇关于下载具有多个进程的多个文件时，Gsutil会占用大量内存的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
