在HIVE的JDBCTemplate结果上使用Java8流 [英] Use Java8 Stream on JDBCTemplate Results from HIVE

问题描述

我正在使用jdbcTemplate查询配置单元，然后将结果写入.csv文件.我基本上只是生成一个对象列表，然后蒸一下列表以将每个记录写入文件. 我希望将结果从配置单元返回时进行流处理，然后将其写入文件中，而不是等待整个处理然后再处理它.谁能指出我正确的方向?谢谢！

I am using jdbcTemplate to query hive then writing the results to a .csv file. I basically just generate a list of objects then steam the list to write each record to the file. I will like to stream the results as they coming back from hive and write it to the file instead of wait to get the whole thing then processing it. Can anyone pointing me to the right direction? Thanks!

private List<Avs> queryAvsData(String asSql) {
    List<Avs> llistAvs = new ArrayList<Avs>();
    List<Map<String, Object>> rows = hiveJdbcTemplate.queryForList(asSql);
    Iterator<Map<String, Object>> it = rows.iterator();
    while (it.hasNext()) {
        Map<String, Object> row = it.next();
        Avs laAvs = Avs.builder()
                .make((String) row.get("make"))
                .model((String) row.get("model"))
                .build();
        llistAvs.add(laAvs);
    }
    return llistAvs;
}

推荐答案

似乎没有内置解决方案，但是您可以做到.基本上，您将现有功能包装在迭代器中，然后使用拆分器将其转换为流. 这是一个博客关于这个主题的帖子:

It doesn't look like there's a built-in solution, but you can do it. Basically, you wrap the existing functionality in an iterator, and use a spliterator to turn it into a stream. Here's a blog post on the subject:

该代码实现了Spring的ResultSetExtractor接口，它是一个Single Abstract Method(SAM)接口，允许使用lambda表达式来实现它.

The code implements Spring’s ResultSetExtractor interface, which is a Single Abstract Method (SAM) interface, allowing the use of a lambda expression to implement it.

该实现将SQL ResultSet包装在迭代器中，使用Spliterators和StreamSupport实用工具类构造流，并将其应用于具有行集流并返回通用结果的Function.

The implementation wraps the SQL ResultSet in an iterator, constructs a stream using the Spliterators and StreamSupport utility classes, and applies that to a Function taking a stream of row sets and returning a generic result.

这篇关于在HIVE的JDBCTemplate结果上使用Java8流的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！