将具有无效字符的嵌套字段从Spark 2导出到Parquet [英] Exporting nested fields with invalid characters from Spark 2 to Parquet

问题描述

我正在尝试使用spark 2.0.2将JSON文件转换为镶木地板.

I am trying to use spark 2.0.2 to convert a JSON file into parquet.

• JSON文件来自外部来源，因此该模式在到达之前无法更改.
• 该文件包含属性映射.在收到文件之前，属性名称未知.
• 属性名称包含无法在实木复合地板中使用的字符.

{
    "id" : 1,
    "name" : "test",
    "attributes" : {
        "name=attribute" : 10,
        "name=attribute with space" : 100,
        "name=something else" : 10
    }
}

空格和等号都不能用在实木复合地板中，出现以下错误:

Both the space and equals character can't be used in parquet, I get the following error:

 org.apache.spark.sql.AnalysisException: Attribute name "name=attribute" contains invalid character(s) among " ,;{}()\n\t=". Please use alias to rename it.; 

• 因为这些是嵌套字段，所以我不能使用别名重命名它们，这是真的吗?
• 我已尝试按照此处建议的方式重命名架构中的字段:
  • As these are nested fields I can't rename them using an alias, is this true?
  • I have tried renaming the fields within the schema as suggested here: How to rename fields in an DataFrame corresponding to nested JSON. This works for some files, However, I now get the following stackoverflow:

  
  java.lang.StackOverflowError 
  
  at scala.runtime.BoxesRunTime.boxToInteger(BoxesRunTime.java:65) 
  at org.apache.spark.scheduler.DAGScheduler.getCacheLocs(DAGScheduler.scala:258) 
  at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:1563) 
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply$mcVI$sp(DAGScheduler.scala:1579) 
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:1578) 
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:1578) 
  at scala.collection.immutable.List.foreach(List.scala:381) 
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:1578) 
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:1576) 
  at scala.collection.immutable.List.foreach(List.scala:381) 
  at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal(DAGScheduler.scala:1576) 
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply$mcVI$sp(DAGScheduler.scala:1579) 
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:1578) 
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2$$anonfun$apply$1.apply(DAGScheduler.scala:1578) 
  at scala.collection.immutable.List.foreach(List.scala:381) 
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:1578) 
  at org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$getPreferredLocsInternal$2.apply(DAGScheduler.scala:1576) 
  at scala.collection.immutable.List.foreach(List.scala:381) 
  ...
  repeat
  ...
  

  我要执行以下一项操作:

  I want to do one of the following:

  • 当我将数据加载到spark中时，从字段名称中剥离无效字符
  • 更改架构中的列名称而不会导致堆栈溢出
  • 以某种方式更改架构以加载原始数据，但在内部使用以下内容:

  {
      "id" : 1,
      "name" : "test",
      "attributes" : [
          {"key":"name=attribute", "value" : 10},
          {"key":"name=attribute with space", "value"  : 100},
          {"key":"name=something else", "value" : 10}
      ]
  }
  

  推荐答案

  我通过以下方式解决了这个问题:

  I solved the problem this way:

  df.toDF(df
      .schema
      .fieldNames
      .map(name => "[ ,;{}()\\n\\t=]+".r.replaceAllIn(name, "_")): _*)
  

  我用"_"替换了所有不正确的符号.

  where I replaced all incorrect symbols by "_".

  这篇关于将具有无效字符的嵌套字段从Spark 2导出到Parquet的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！