外部表没有从火花流写入的地板文件中更新 [英] External Table not getting updated from parquet files written by spark streaming

问题描述

我正在使用Spark Stream将汇总的输出作为parquet文件写入使用SaveMode.Append的hdfs。我创建了一个外部表：

  CREATE TABLE如果不存在可汇总
 USING org.apache.spark.sql .parquet 
选项（
 pathhdfs：////
）; 
  

我有一个印象，就外部表而言，查询应该从新添加的parquet中获取数据也。但是，好像新写入的文件没有被拾取。

每次删除并重新创建表格都可以正常工作，但不是解决方案。

请建议我的表格还有来自较新文件的数据。

解决方案

您是否正在用火花阅读这些表格？
如果是这样的话，spark会缓存parquet表元数据（因为架构发现可能会很昂贵）。

为了克服这个问题，你有两个选择：

1. 将配置 spark.sql.parquet.cacheMetadata 设置为 false
2. 在查询之前刷新表： sqlContext.refreshTable（my_table）



详情请参阅： http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-metastore-parquet-table-conversion


I am using spark streaming to write the aggregated output as parquet files to the hdfs using SaveMode.Append. I have an external table created like :

CREATE TABLE if not exists rolluptable
USING org.apache.spark.sql.parquet
OPTIONS (
  path "hdfs:////"
);

I had an impression that in case of external table the queries should fetch the data from newly parquet added files also. But, seems like the newly written files are not being picked up.

Dropping and recreating the table every time works fine but not a solution.

Please suggest how can my table have the data from newer files also.

解决方案

Are you reading those tables with spark? if so, spark caches parquet tables metadata (since schema discovery can be expensive)

To overcome this, you have 2 options:

1. Set the config spark.sql.parquet.cacheMetadata to false
2. refresh the table before the query: sqlContext.refreshTable("my_table")

See here for more details: http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-metastore-parquet-table-conversion

这篇关于外部表没有从火花流写入的地板文件中更新的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！