Hazelcast IMap的加载时间 [英] Loading time of Hazelcast IMap

问题描述

我在数据库中有100万个条目，我必须将所有内容都放入Hazelcast IMap中.

那么现在如何测试IMap的加载时间?

解决方案

您可以通过多种方式来加载地图.一旦有了执行加载的代码，就可以执行简单的操作，例如在加载之前和之后调用System.currentTimeMillis()来查看它花费了多长时间.

您可以使用Hazelcast提供的MapLoader功能在成员(集群)端进行加载，也可以编写自己的JDBC代码在客户端加载条目并将其添加到集群中.您还可以将Hazelcast Jet用作摄取/ETL引擎，如果要在加载时对数据进行过滤，扩充或数据转换，则该功能特别有用.

如果确实要从客户端进行加载，那么您真的要避免天真的方法，即遍历JDBC结果集并将条目一次放入映射中-对每个集群进行一次完整的集群往返一百万个条目将影响您的表现.在我最近所教的一个培训班的实验练习中，我将一个55000个条目的整个表加载到java.util.HashMap中，然后执行IMap.putAll()以单个操作将整个Map推入Hazelcast./p>

这是来自实验室的代码-它实际上是用相同的数据填充两个表(一个表具有索引，另一个没有索引，这使我们可以在以后进行查询和汇总的实验室中比较性能).它包含有关每个负载需要花费多长时间的简单时序信息-毫不奇怪，当我们为表建立索引时，它会变慢，但是对索引表的查询会更快.

https://github.com/hazelcast/training-courses/blob/master/Special_Projects/mod02-db-etl-lab/src/main/java/com/hztraining/solutions/PopulateCacheWithJDBCSolution.java

I have 1 Millions entries in the database and I have to put everything inside Hazelcast IMap.

So now how to test the loading time IMap?

解决方案

There are a number of ways you can load the map; once you have the code that does the loading you can do something simple like calling System.currentTimeMillis() before and after the load to see how long it's taking.

You can use the MapLoader functionality provided by Hazelcast to do loading on the member (cluster) side, or write your own JDBC code to load entries on the client side and add them to the cluster. You can also use Hazelcast Jet as a ingest/ETL engine, which is especially useful if you want to do filtering, enrichment, or data transformation of the data as it's being loaded.

If you do loading from the client side, you really want to avoid the naive approach of iterating over the JDBC result set and putting entries to the map one at a time - doing a full round-trip to the cluster for each of a million entries will kill your performance. In a lab exercise from a training class I recently taught, I loaded an entire table of 55000 entries into a java.util.HashMap, and then did a IMap.putAll() to push the entire Map into Hazelcast in a single operation.

Here's the code from the lab - it actually populates two tables with the same data (one has indexes, the other doesn't, which lets us compare performance in later labs that do queries and aggregations). It includes simple timing information on how long each load takes - not surprisingly, it's slower when we're building indexes for the tables, but then queries on the indexed table will be much faster.

https://github.com/hazelcast/training-courses/blob/master/Special_Projects/mod02-db-etl-lab/src/main/java/com/hztraining/solutions/PopulateCacheWithJDBCSolution.java

这篇关于Hazelcast IMap的加载时间的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！