如何收集单行数据帧并将字段用作常量 [英] How to collect a single row dataframe and use fields as constants

问题描述

我正在尝试从 Hive 表中读取一行，并将其字段用作 Spark 应用程序其余部分中的常量.

I'm trying to read a single row from a Hive table and use its fields as constants in the rest of my Spark application.

object IniConstHive extends Serializable {
  val techTbl: DataFrame = spark.table("my_db.my_conf_table").orderBy(col("my_date").desc)
  val firstrow: Row = techTbl.head

  val my_firstfield: Double = firstrow.getAs[java.math.BigDecimal](0).doubleValue
  val my_secondfield: Double = firstrow.getAs[java.math.BigDecimal](1).doubleValue  
}

不幸的是，当我调用 IniConstHive.my_firstfield 时出现此异常:

Unfortunately, I get this exception when I call IniConstHive.my_firstfield:

Caused by: java.util.NoSuchElementException: None.get
        at scala.None$.get(Option.scala:347)
        at scala.None$.get(Option.scala:345)
        at org.apache.spark.storage.BlockInfoManager.releaseAllLocksForTask(BlockInfoManager.scala:343)
        at org.apache.spark.storage.BlockManager.releaseAllLocksForTask(BlockManager.scala:676)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:329)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

这是正确的做法吗?有没有更有效的方法来实现结果?

Is this the right approach to do this? Is there a more efficient way to acheive the result?

推荐答案

原来问题不在 IniConstHive 对象，而是在我调用它的地方.事实上，我试图在自定义 UDAF 中调用 IniConstHive.my_firstfield，它在执行程序中运行.出于这个原因，我的应用程序尝试在驱动程序外部实例化多个 SparkContext.

It turned out the problem was not in the IniConstHive object but the place where I called it. In fact, I was trying to call IniConstHive.my_firstfield inside a custom UDAF, which runs inside executors. For this reason my application tried to instantiate multiple SparkContext outside the driver.

我通过在驱动程序中调用 IniConstHive.my_firstfield 来解决，然后将结果作为参数传递给 UDAF 构造函数.

I solved by calling IniConstHive.my_firstfield in the driver and then passing the result as a parameter in the UDAF constructor.

这篇关于如何收集单行数据帧并将字段用作常量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！