dask计算不并行执行 [英] dask computation not executing in parallel

问题描述

我有一个json文件的目录，我试图转换为一个dask DataFrame并保存到castra。
有200个文件，在它们之间包含O（10 ** 7）个json记录。
代码很简单，主要是下面的教程示例。

I have a directory of json files that I am trying to convert to a dask DataFrame and save it to castra. There are 200 files containing O(10**7) json records between them. The code is very simple largely following tutorial examples.

import dask.dataframe as dd
import dask.bag as db
import json
txt = db.from_filenames('part-*.json')
js = txt.map(json.loads)
df = js.to_dataframe()
cs=df.to_castra("data.castra")

在32核机器上运行它，但代码只使用一个核心在100％。
我从文档的理解是这个代码并行执行。
为什么不是？
我误解了什么？

I am running it on a 32 core machine, but the code only utilizes one core at 100%. My understanding from the docs is that this code execute in parallel. Why is it not? Did I misunderstand something?

推荐答案

你的最终集合是一个dask dataframe，默认使用线程，必须明确告诉dask使用进程。

Your final collection is a dask dataframe, which uses threads by default, you will have to explicitly tell dask to use processes.

您可以全局执行

import dask
import dask.multiprocessing
dask.set_options(get=dask.multiprocessing.get)

或只在 to_castra 调用

df.to_castra("data.castra", get=dask.multiprocessing.get)

此外，只是作为一个警告，Castra主要是一个实验。它的速度非常快，但也不像HDF5那样成熟。

Also, just as a warning, Castra was mostly an experiment. It's decently fast but also not nearly mature as something like HDF5.

这篇关于dask计算不并行执行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！