Clojure的并行doseq [英] Parallel doseq for Clojure

问题描述

我没有在Clojure中使用多线程，所以不知道从哪里开始。

I haven't used multithreading in Clojure at all so am unsure where to start.

我有一个 doseq 其身体可以平行运行。我想要的是在那里总是有3个线程运行（留下1核心免费），评估身体并行，直到范围耗尽。没有共享的状态，没有什么复杂 - 相当于Python的多处理器就好了。

I have a doseq whose body can run in parallel. What I'd like is for there always to be 3 threads running (leaving 1 core free) that evaluate the body in parallel until the range is exhausted. There's no shared state, nothing complicated - the equivalent of Python's multiprocessing would be just fine.

像这样：

(dopar 3 [i (range 100)]
  ; repeated 100 times in 3 parallel threads...
  ...)

我应该从哪里开始？有这样的命令吗？标准包装？一个很好的参考？

Where should I start looking? Is there a command for this? A standard package? A good reference?

到目前为止，我发现了 pmap ，并且可以使用一次看起来像是每次使用32个 - 没有，源说​​2 +处理器数），但似乎这是一个基本的原始应该已经存在的地方。

So far I have found pmap, and could use that (how do I restrict to 3 at a time? looks like it uses 32 at a time - no, source says 2 + number of processors), but it seems like this is a basic primitive that should already exist somewhere.

澄清 ：我真的想控制线程的数量。我有一个进程是长期运行和使用大量的内存，所以创建一个大数量，希望事情工作OK不是一个好办法（示例使用了一个重要的chunk可用mem ）。

clarification: I really would like to control the number of threads. I have processes that are long-running and use a fair amount of memory, so creating a large number and hoping things work out OK isn't a good approach (example which uses a significant chunk available mem).

update ：开始编写一个宏，并且我需要一个信号量（或互斥量，或者一个原子我可以等待）。 Clojure中是否存在信号量？还是应该使用ThreadPoolExecutor？看起来奇怪，必须从Java这么多 - 我认为并发编程在Clojure应该是很容易...也许我在想这个完全错误的方式？嗯。代理？

update: Starting to write a macro that does this, and I need a semaphore (or a mutex, or an atom i can wait on). Do semaphores exist in Clojure? Or should I use a ThreadPoolExecutor? It seems odd to have to pull so much in from Java - I thought parallel programming in Clojure was supposed to be easy... Maybe I am thinking about this completely the wrong way? Hmmm. Agents?

推荐答案

pmap 实际上可以在大多数情况下正常工作使用一个线程池与您的机器有明显的线程数。我不会想尝试创建自己的机制来控制线程数，除非你有真正的基准证据，默认值导致一个问题。

pmap will actually work fine in most circumstances - it uses a thread pool with a sensible number of threads for your machine. I wouldn't bother trying to create your own mechanisms to control the number of threads unless you have real benchmark evidence that the defaults are causing a problem.

如果你真的想限制到最多三个线程，一个简单的方法是只使用pmap的范围的3个子集：

Having said that, if you really want to limit to a maximum of three threads, an easy approach is to just use pmap on 3 subsets of the range:

(defn split-equally [num coll] 
  "Split a collection into a vector of (as close as possible) equally sized parts"
  (loop [num num 
         parts []
         coll coll
         c (count coll)]
    (if (<= num 0)
      parts
      (let [t (quot (+ c num -1) num)]
        (recur (dec num) (conj parts (take t coll)) (drop t coll) (- c t)))))) 

(defmacro dopar [thread-count [sym coll] & body]
 `(doall (pmap 
    (fn [vals#]
      (doseq [~sym vals#]
        ~@body))  
    (split-equally ~thread-count ~coll))))

注意使用 doall ，这是强制评估 pmap （这是lazy）所需要的。

Note the use of doall, which is needed to force evaluation of the pmap (which is lazy).

这篇关于Clojure的并行doseq的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！