Clojure core.async.如何用go block里面的open懒洋洋地下载? [英] Clojure core.async. How to download lazily with go block inside with-open?

问题描述

这是我上一个问题的延续如何在clojure中按比例产生一个惰性序列?

It`s a continuation of my previous question How to produce a lazy sequence by portion in clojure?

我想按部就班从数据库下载数据.最初，我先下载前500行，然后发送请求以获取下500行，依此类推，直到从服务器接收到所有数据为止.

I want download data from a database by portions. Initially I download first 500 rows and then I send a request to fetch next 500 rows and so on until I receive all data from a server.

我写了代码:

(jdbc/atomic conn
 (with-open [cursor (jdbc/fetch-lazy conn [sql_query])]
   (let [lazyseq (jdbc/cursor->lazyseq cursor)
         counter (atom 1)]
     (swap! lazyseq_maps assoc :session_id {:get_next? (chan 1) :over_500 (chan 1) :data []})
     (>!! (:get_next? (:session_id @lazyseq_maps)) true)
     (go
       (doseq [row lazyseq]
         (swap! counter inc)
         (when (<! (:get_next? (:session_id @lazyseq_maps)))
           (swap! lazyseq_maps update-in [:session_id :data] conj row)
           (if (not= 0 (mod @counter 500))
             (>! (:get_next? (:session_id @lazyseq_maps)) true)
             (>! (:over_500 (:session_id @lazyseq_maps)) true))))
        ;
        (close! (:get_next? (:session_id @lazyseq_maps)))
        (close! (:over_500 (:session_id @lazyseq_maps)))
        (.close conn))

     (when (<!! (:over_500 (:session_id @lazyseq_maps))) {:message "over 500 rows"
                                                          :id :session_id
                                                          :data (:data (:session_id @lazyseq_maps))}))))

我在doseq周期的帮助下获取行.当doseq超过500行时，我停放循环(when (<! (:get_next? (:session_id @lazyseq_maps)))，并等待来自外部的信号来检索下一个500行.

I fetch rows with help of the doseq cycle. When doseq passed 500 rows I park the cycle (when (<! (:get_next? (:session_id @lazyseq_maps))) and wait for a signal from outside to retrieve next 500 rows.

但是这里我有一个问题.当我发送信号时，程序将引发错误结果集已关闭".即连接在开放范围之外关闭.但是我不明白为什么，因为go块放在了open范围内.您能帮我解决问题吗?

But here I have a problem. When I send the signal, the program throws error "Resultset is closed". I.e connection is closed outside with-open scope. But I don`t understand why, because go block is placed inside with-open scope. Can you help me solve the problem?

推荐答案

(go ...)立即返回，因此(with-open ...)也是如此.

(go ...) returns immediately, therefore, so does (with-open ...).

您可能想用另一种方法来做:

You may want to do it the other way around:

(go (with-open ...))

(go (with-open ...))

但是，请注意，此过程可能会在很长一段时间内保持数据库连接(稀缺资源！)，这可能是不希望的，并且与使用轻量级"线程的好处背道而驰，谢谢到go块.以下是一些可供考虑的替代方法:

However, do note that this process will hold on to a database connection (a scarce resource!) for a potentially very long time, which may not be desirable, and kind of goes against the benefit of having 'lightweight' threads thanks to go blocks. Here are some alternatives to consider:

• 也许您可以为每个批次重新打开数据库连接?
• 也许您可以急于将整个结果集流式传输到外部存储(例如AWS S3)，并让客户端对此进行轮询?

这篇关于Clojure core.async.如何用go block里面的open懒洋洋地下载?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！