提高这种情况的表现的方法 [英] ways to improve the performance of this scenario

问题描述

我有一个地图，大量的数据被填充（大约30万条记录）

并重复如下，

  for（Map.Entry< String，List< Classobj>>条目：testMap 
 .entrySet（））{ 
 //发送电子邮件与地图键作为email'id 
 //电子邮件内容从列表中填充
 //执行一个sql更新到列与这里生成的dyanamic值与电子邮件'id 
} 
  

如上所述，我担心会出现的性能问题导致在内执行循环之前执行的上述操作。

更新：

该场景是。我正在迭代一个拥有大量数据的地图，

在迭代它时，我得到了用户标识，我必须对用户ID进行计算。例如，考虑 userid +一些常量，这应该在数据库表中更新。

，还应该添加到电子邮件内容以及我地图上的列表值

所以我认为批量更新是不可能的，我是否正确理解？

$ b我应该遵循这种做法吗？或者与任何其他想法一起出现

解决方案

由于两个原因，for循环需要时间。

1 ）个人电子邮件通过较少的运输连接改善运输连接

2）单独提交通过

所以理想的是处理两者，我建议做对于1000的批次，然后玩数字

示例

  int BATCH_SIZE = 1000 
 conn = DriverManager.getConnection（username，password）; 
 conn.setAutoCommit（false）; 
语句stmt = conn.createStatement（
 ResultSet.TYPE_SCROLL_INSENSITIVE，
 ResultSet.CONCUR_UPDATABLE）; 
 int count = 0; 
 
地图< String，String> emails_map = new HashMap（BATCH_SIZE）< String，String> ;; 
 for（Map.Entry< String，List< Classobj>>条目：testMap 
 .entrySet（））{
 String email = get_useremail（）; 
 String const_val = do_magic（）; //这是你如何计算一些常量
 String body = construct_body（）; 
 
 count ++; 
 String SQL =您的更新状态; 
 stmt.executeUpdate（SQL）; 
 emails_map.put（email，body）; //可以创建
 if（count％BATCH_SIZE == 0）{
 //提交所有转录
 conn.commit（）; 
 //批量发送电子邮件发送
 //http://stackoverflow.com/questions/13287515/how-to-send-bulk-mails-using-javax-mail-api-efficiently-can- we-use-reuse-auth 
 
 bulk_emails_send（emails_map）
} 
 
} 
 
 
 public void bulk_emails_send（Map< ; String，String> emails_map）{
 //通过你的设置获取默认的Session对象
 Session session = Session.getDefaultInstance（properties）; 
传输t = session.getTransport（）; 
 t.connect（）; 
尝试{
 for（String email_id in emails_map）{
 Message m = new MimeMessage（session）; 
 // add to，from，subject，body 
 m.saveChanges（）; 
 t.sendMessage（m，m.getAllRecipients（））; 
} 
} finally {
 t.close（）; 
} 
} 
  

I have a map with huge amount of data being populated (around 300,000 records in approx)

and iterating it as below ,

    for (Map.Entry<String, List<ClassOBj>> entry : testMap
                    .entrySet()) {
 // send email with map keys as email'id
 // email content is populated from the list
// Perform a sql update to the column with the dyanamic value generated here with the email'id
                                  }

As mentioned above , i am worried about the performance issues that will be caused of the above operation that is performed inside the for loop.

Update:

the scenario is . i am iterating a map which holds large amount of data,

On iterating it i am getting the userid's and i have to make a computation of the user id .For example, consider userid+some constants and this should be updated in the database table.

and also should be added to the email content along with list values from my map

so i thought batch updates are not possible, am i correct with my understanding ?

should i follow this approach ? or go with any alternate ideas

解决方案

The for loop is taking time due to two reasons.
1) Individual Email improve it by less Transport connection
2) Individual commits improve it by

So Ideal is to handle both, I would recommend do it for batch of 1000, then play with numbers

Example

int BATCH_SIZE = 1000
conn = DriverManager.getConnection("username","password");
conn.setAutoCommit(false);
Statement stmt = conn.createStatement(
        ResultSet.TYPE_SCROLL_INSENSITIVE,
        ResultSet.CONCUR_UPDATABLE);
int count = 0;

Map<String, String> emails_map = new HashMap(BATCH_SIZE)<String, String>;
for (Map.Entry<String, List<ClassOBj>> entry : testMap
        .entrySet()) {
    String email = get_useremail();
    String const_val = do_magic(); // this is how you are computing some constant
    String body = construct_body();

    count++;
    String SQL = "YOUR UPDATE STATEMENT";
    stmt.executeUpdate(SQL);  
    emails_map.put(email, body); // can create 
    if (count % BATCH_SIZE == 0) {
        //commits all transcations
        conn.commit();
        //bulk send emails sending 
        //http://stackoverflow.com/questions/13287515/how-to-send-bulk-mails-using-javax-mail-api-efficiently-can-we-use-reuse-auth

        bulk_emails_send(emails_map)
    }

}


public void bulk_emails_send(Map<String, String> emails_map) {
    // Get the default Session object through your setting
    Session session = Session.getDefaultInstance(properties);
    Transport t = session.getTransport();
    t.connect();
    try {
        for (String email_id in emails_map) {
            Message m = new MimeMessage(session);
            //add to, from , subject, body
            m.saveChanges();
            t.sendMessage(m, m.getAllRecipients());
        }
    } finally {
        t.close();
    }
}

这篇关于提高这种情况的表现的方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！