使用UDF中的“添加文件"读取添加到Hive资源的文件 [英] Read file that is added to Hive resource using Add File from UDF

问题描述

我想知道如何从Udf中读取使用 ADD FILE 添加的Hive资源?例如

I would like to know how can I read the Hive resources that is added using ADD FILE from Udf? e.g.

Hive > add file /users/temp/key.jks

是否可以用Java中的UDF读取此文件?在Udf中获取此文件的路径是什么?

Is it possible to read this file in UDF in Java? What will be the path to get this file in Udf?

谢谢大卫

推荐答案

一旦使用 ADD 命令将资源添加到会话中，Hive查询便可以通过其名称引用它(在map/reduce中)/transform子句)，并且资源在执行时在整个Hadoop集群上本地可用.Hive使用Hadoop的Distributed Cache在查询执行时将添加的资源分配给集群中的所有计算机.参见此处: HiveResources

Once a resource is added to a session using ADD command, Hive queries can refer to it by its name (in map/reduce/transform clauses) and the resource is available locally at execution time on the entire Hadoop cluster. Hive uses Hadoop's Distributed Cache to distribute the added resources to all the machines in the cluster at query execution time. See here: HiveResources

Hive中有 in_file(string str，string filename)函数-如果字符串str在filename中显示为整行，则返回true.您可以使用in_file源代码作为示例:

There is the in_file(string str, string filename) function in Hive - returns true if the string str appears as an entire line in filename. You can use in_file source code as an example: GenericUDFInFile.java

源代码中很少有方法:

  private BufferedReader getReaderFor(String filePath) throws HiveException {
    try {
      Path fullFilePath = FileSystems.getDefault().getPath(filePath);
      Path fileName = fullFilePath.getFileName();
      if (Files.exists(fileName)) {
        return Files.newBufferedReader(fileName, Charset.defaultCharset());
      }
      else
      if (Files.exists(fullFilePath)) {
        return Files.newBufferedReader(fullFilePath, Charset.defaultCharset());
      }
      else {
        throw new HiveException("Could not find \"" + fileName + "\" or \"" + fullFilePath + "\" in IN_FILE() UDF.");
      }
    }
    catch(IOException exception) {
      throw new HiveException(exception);
    }
  }

  private void loadFromFile(String filePath) throws HiveException {
    set = new HashSet<String>();
    BufferedReader reader = getReaderFor(filePath);
    try {
      String line;
      while((line = reader.readLine()) != null) {
        set.add(line);
      }
    } catch (Exception e) {
      throw new HiveException(e);
    }
    finally {
      IOUtils.closeStream(reader);
    }
}

这篇关于使用UDF中的“添加文件"读取添加到Hive资源的文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！