为什么迭代所有Java邮件用户文件夹的速度非常慢? [英] Why iterating over all java mail user's folders very slow?

问题描述

   Properties props = System.getProperties();
   props.put("mail.imap.connectiontimeout",5000);
   Session session = Session.getInstance(props);
   Store store = session.getStore("imap");
   for(50K users){

        //login,password changed in loop  
        String[] folders = {"inbox", "f1", "f2", "f3", "spam"};
        store.connect(serverAddress, login + emailSuffix, password);
        for (int i = 0; i < folders.length; i++) {
                    Folder x = store.getFolder(folders[i]);
                    x.open(Folder.READ_ONLY);
                    System.out.println("folder " + folders[i] + " of " + login);
                    x.getUnreadMessageCount();
                    x.close(false);
        }
        store.close();
    }

我对所有连接使用相同的存储，请根据此答案在dovecot中将service_count更改为改善了imap-dovecot的性能，但我只看到了第一次迭代，并且在该代码挂起之后或长时间后未执行下一个system.out.

I'm using same store for all connections, changed service_count in dovecot according to this answer in order to improve imap-dovecot performance but I see only first iteration and after that code hangs or does next system.out after long time.

实际上，当我想从纯Java Mail迁移到某些自定义格式时，我需要获取所有用户的所有旧邮件+计算所有未读邮件.我什至没有设法只遍历所有用户和每个用户的文件夹，因为即使简单的store.connect在第一次迭代后也会挂起！

Actually, I need to grab all old messages of all users + count all unread messages as I want to migrate from pure Java Mail to some custom format. I didn;t manage even to just iterate over all users and folders for each user because even simple store.connect hangs after 1-st iteration!

我个人认为瓶颈是我的鸽舍配置，但它使用默认限制(1000个连接)，看起来不错.

I personally think that bottleneck is my dovecot config, but it uses default limits (1000 connections) which looks good.

我以某种方式改善了我的鸽舍或仅对所有用户连接了一次商店，或者以其他方式获取了所有用户的所有消息和所有用户的unreadMessagesCount?

My I somehow improve my dovecot or connect my store only once for all users or somehow fetch all messages of all users and unreadMessagesCount of all users in other way?

PS.编程方式的唯一替代方法是maildir中的一些bash脚本，它将从文件系统中读取每条消息并将其传递给其他部分，然后转换为我的自定义格式)，但是比Java难得多，因此难以解析smptp，解析来自文件名的标志等等.

PS. The only alternative to programmatic way is some bash script in maildir which whill read each message from file system and pass it to some rest which converts to my custom format) but it much more harder than Java it's too difficult to parse smptp, parse seen flags from file name and so on.

更新

我发现apache commons net imapclient运行得非常快.

I found apache commons net imapclient which works very fast.

<dependency>
    <groupId>commons-net</groupId>
    <artifactId>commons-net</artifactId>
    <version>3.3</version>
</dependency>

我的代码如下

IMAPClient client = new IMAPClient();
client.connect("localhost");
for(50K users){
    client.login(login + emailSuffix, password);
    for (int i = 0; i < folders.length; i++) {
        System.out.println(client.select("INBOX"); //prints true, it's ok
    }
}

1. 看起来它的连接速度比Java mailapi快，因为它可能 连接一次即可托管，然后每个用户登录.我可以在JavaMail API中以某种方式重复这种行为吗?
2. 如何与apache commons客户端抓取消息?所有方法都返回boolean或void，所以看起来就像只是服务器检查库，对吗?是否可以通过某种方式从imapclient获得有用的信息?

1. Looks like it connects faster than java mailapi because it may connect once to host and after that login for each user. May I somehow repeat such behavior in JavaMail API?
2. How may I grab messages with apache commons client? All methods return boolean or void, so it's looks like just server checking library am I right? Is it possible to somehow get useful info from imapclient?

推荐答案

通过简单地遍历文件系统(我具有maildir格式)，终于解决了我的问题.

Finally solved my problem by simple iterating over file system (I have maildir format).

我想Java Mail API在store.connect中为每个用户进行新的dovecot身份验证，而它应该只连接一次(使用dovecot身份验证)，然后为每个用户登录(使用dovecot imap登录).

这就是为什么我每次迭代都等待1分钟的原因-dovecot配置中的身份验证过程处于标准空闲状态.我不确定，但看起来是这样.

I guess Java Mail API makes new dovecot auth for each user in store.connect while it should just connect once (consume dovecot auth) and after that login for each user (consume dovecot imap-login). That's why I waited 1 minute for each iteration - it's std idle for auth process in dovecot config. I'm not sure but looks so.

Apache lib 工作迅速，但它只是测试库，用于ping服务器，检查连接和其他imap操作.它返回有关操作的布尔结果，但不返回有用的信息(

Apache lib works fast but it's just testing library for pinging server, checking connection and other imap operations. It returns boolean result about operations but not useful information(

这篇关于为什么迭代所有Java邮件用户文件夹的速度非常慢?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！