Clojure的pmap函数会为URL提取操作生成多少个线程? [英] How many threads does Clojure's pmap function spawn for URL-fetching operations?
问题描述
pmap
函数的文档让我想知道,像通过Web提取XML提要的集合那样的效率如何。我不知道pmap会产生多少个并发的取指操作,最大值是多少。
The documentation on the pmap
function leaves me wondering how efficient it would be for something like fetching a collection of XML feeds over the web. I have no idea how many concurrent fetch operations pmap would spawn and what the maximum would be.
推荐答案
如果检查源,参见:
> (use 'clojure.repl)
> (source pmap)
(defn pmap
"Like map, except f is applied in parallel. Semi-lazy in that the
parallel computation stays ahead of the consumption, but doesn't
realize the entire result unless required. Only useful for
computationally intensive functions where the time of f dominates
the coordination overhead."
{:added "1.0"}
([f coll]
(let [n (+ 2 (.. Runtime getRuntime availableProcessors))
rets (map #(future (f %)) coll)
step (fn step [[x & xs :as vs] fs]
(lazy-seq
(if-let [s (seq fs)]
(cons (deref x) (step xs (rest s)))
(map deref vs))))]
(step rets (drop n rets))))
([f coll & colls]
(let [step (fn step [cs]
(lazy-seq
(let [ss (map seq cs)]
(when (every? identity ss)
(cons (map first ss) (step (map rest ss)))))))]
(pmap #(apply f %) (step (cons coll colls))))))
(+ 2(.. Runtime getRuntime availableProcessors))
是那里的主要线索。 pmap将抢先完成(+ 2个处理器)
的工作,并通过 future
异步运行它们。因此,如果您有2个核心,那么它将一次启动4个工作,并试图保持领先地位,但最大数量应为2 + n。
The (+ 2 (.. Runtime getRuntime availableProcessors))
is a big clue there. pmap will grab the first (+ 2 processors)
pieces of work and run them asynchronously via future
. So if you have 2 cores, it's going to launch 4 pieces of work at a time, trying to keep a bit ahead of you but the max should be 2+n.
未来
最终使用支持无限线程数量的代理I / O线程池。它会随着工作量的增长而增长,如果未使用线程,则会收缩。
future
ultimately uses the agent I/O thread pool which supports an unbounded number of threads. It will grow as work is thrown at it and shrink if threads are unused.
这篇关于Clojure的pmap函数会为URL提取操作生成多少个线程?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!