Clojure 的 pmap 函数为 URL 获取操作生成了多少个线程? [英] How many threads does Clojure's pmap function spawn for URL-fetching operations?

问题描述

关于 pmap 函数的文档让我想知道对于通过网络获取 XML 提要集合之类的事情会有多高效.我不知道 pmap 会产生多少并发提取操作以及最大值是多少.

The documentation on the pmap function leaves me wondering how efficient it would be for something like fetching a collection of XML feeds over the web. I have no idea how many concurrent fetch operations pmap would spawn and what the maximum would be.

推荐答案

如果你查看源码你会看到:

If you check the source you see:

> (use 'clojure.repl)
> (source pmap)
(defn pmap
  "Like map, except f is applied in parallel. Semi-lazy in that the
  parallel computation stays ahead of the consumption, but doesn't
  realize the entire result unless required. Only useful for
  computationally intensive functions where the time of f dominates
  the coordination overhead."
  {:added "1.0"}
  ([f coll]
   (let [n (+ 2 (.. Runtime getRuntime availableProcessors))
         rets (map #(future (f %)) coll)
         step (fn step [[x & xs :as vs] fs]
                (lazy-seq
                 (if-let [s (seq fs)]
                   (cons (deref x) (step xs (rest s)))
                   (map deref vs))))]
     (step rets (drop n rets))))
  ([f coll & colls]
   (let [step (fn step [cs]
                (lazy-seq
                 (let [ss (map seq cs)]
                   (when (every? identity ss)
                     (cons (map first ss) (step (map rest ss)))))))]
     (pmap #(apply f %) (step (cons coll colls))))))

(+ 2 (.. Runtime getRuntime availableProcessors)) 是一个重要的线索.pmap 将获取第一个 (+ 2 个处理器) 的工作片段并通过 future 异步运行它们.因此，如果您有 2 个内核，它将一次启动 4 个工作，试图保持领先于您，但最大值应该是 2+n.

The (+ 2 (.. Runtime getRuntime availableProcessors)) is a big clue there. pmap will grab the first (+ 2 processors) pieces of work and run them asynchronously via future. So if you have 2 cores, it's going to launch 4 pieces of work at a time, trying to keep a bit ahead of you but the max should be 2+n.

future 最终使用支持无限数量线程的代理 I/O 线程池.它会随着工作的增加而增长，如果线程未使用则收缩.

future ultimately uses the agent I/O thread pool which supports an unbounded number of threads. It will grow as work is thrown at it and shrink if threads are unused.

这篇关于Clojure 的 pmap 函数为 URL 获取操作生成了多少个线程?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！