如何将生成的 Java 代码转储到标准输出? [英] How to dump generated Java code to stdout?

问题描述

在 Apache Spark 2.+ 上使用 DataFrames，有没有办法获取底层 RDD 并将生成的 Java 代码转储到控制台?

Using DataFrames on Apache Spark 2.+, is there a way to get the underlying RDDs and dump the generated Java code to the console?

推荐答案

这可以使用 QueryExecution.debug.codegen.该值可通过 .queryExecution 在 Dataframe/Dataset 上访问(这是一个开发者 API"，即不稳定，容易损坏，因此只能用于调试).这适用于 Spark 2.4.0，并且从代码来看，它似乎应该从 2.0.0(或更高版本)开始工作:

This can be done using QueryExecution.debug.codegen. This value is accessible on Dataframe/Dataset via .queryExecution (which is a "Developer API", that is, unstable, subject to breakage, and so should only be used for debugging). This works on Spark 2.4.0, and, from the code, seems like it should work since 2.0.0 (or more):

scala> val df = spark.range(1000)
df: org.apache.spark.sql.Dataset[Long] = [id: bigint]

scala> df.queryExecution.debug.codegen
Found 1 WholeStageCodegen subtrees.
== Subtree 1 / 1 ==
*(1) Range (0, 1000, step=1, splits=12)

Generated code:
/* 001 */ public Object generate(Object[] references) {
/* 002 */   return new GeneratedIteratorForCodegenStage1(references);
/* 003 */ }
/* 004 */
/* 005 */ // codegenStageId=1
/* 006 */ final class GeneratedIteratorForCodegenStage1 extends org.apache.spark.sql.execution.BufferedRowIterator {
/* 007 */   private Object[] references;
/* 008 */   private scala.collection.Iterator[] inputs;
/* 009 */   private boolean range_initRange_0;
/* 010 */   private long range_number_0;
/* 011 */   private TaskContext range_taskContext_0;
/* 012 */   private InputMetrics range_inputMetrics_0;
/* 013 */   private long range_batchEnd_0;
/* 014 */   private long range_numElementsTodo_0;
/* 015 */   private org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[] range_mutableStateArray_0 = new org.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter[1];

...

/* 104 */       ((org.apache.spark.sql.execution.metric.SQLMetric) references[0] /* numOutputRows */).add(range_nextBatchTodo_0);
/* 105 */       range_inputMetrics_0.incRecordsRead(range_nextBatchTodo_0);
/* 106 */
/* 107 */       range_batchEnd_0 += range_nextBatchTodo_0 * 1L;
/* 108 */     }
/* 109 */   }
/* 110 */
/* 111 */ }

这篇关于如何将生成的 Java 代码转储到标准输出?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！