使用Spark/java的ST_geomfromtext函数 [英] ST_geomfromtext function using Spark / java

问题描述

由于ST_GeomFromText不是org.apache.spark.sql.functions的一部分，因此它不会在内部识别它.我需要首先为此函数定义UDF.意味着我需要定义该函数的定义，然后将该火花注册为UDF，然后只有该函数可以使用.

Since the ST_GeomFromText is not the part of org.apache.spark.sql.functions so it will not recognise it internally.I need to first define the UDF for this function. means I need to define the definition of that function and then register that function with spark as UDF then only I can use this function.

我陷入了开始定义此功能的困境，将采用什么参数.

I got stuck in beginning to define this function, what parameters will take.

编辑

我使用的代码如下:

 sparkSession.udf().register("ST_GeomFromText", new UDF1<String, String>() {
        @Override
        public String call(String txt ) {
            return (new ST_GeomFromText(txt));
        }
    }, DataTypes.StringType);

我真的需要你的帮助.

谢谢

推荐答案

类似问题-

1. 使用Spark Java的Geopark图书馆
2. 使用Java从ResultSet到Spark数据帧
3. 使用Spark/Java的GeoSpark
4. 未定义函数:使用Spark/Java的"ST_GeomFromText"

1. GeoSpark librairy using Spark Java
2. From ResultSet to Spark dataframe using Java
3. GeoSpark using Spark / Java
4. Undefined function: 'ST_GeomFromText' Using Spark / Java

我认为，您尚未遵循 GeoSparkSQL -概述/#quick-start 彻底-

I think, you haven't followed the GeoSparkSQL-Overview/#quick-start thoroughly-

1. 根据快速入门，您需要将GeoSpark-core和GeoSparkSQL添加到项目POM.xml或build.sbt

<!-- Geo spark lib doc - https://datasystemslab.github.io/GeoSpark/api/sql/GeoSparkSQL-Overview/#quick-start-->
        <dependency>
            <groupId>org.datasyslab</groupId>
            <artifactId>geospark-sql_2.3</artifactId>
            <version>1.3.1</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/com.vividsolutions/jts -->
        <dependency>
            <groupId>com.vividsolutions</groupId>
            <artifactId>jts</artifactId>
            <version>1.13</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.datasyslab/geospark-viz -->
        <dependency>
            <groupId>org.datasyslab</groupId>
            <artifactId>geospark-viz_2.3</artifactId>
            <version>1.3.1</version>
        </dependency>
        <dependency>
            <groupId>org.datasyslab</groupId>
            <artifactId>geospark</artifactId>
            <version>1.3.1</version>
        </dependency>

1. 声明您的Spark会话

SparkSession sparkSession = SparkSession.builder()
                .config("spark.serializer", KryoSerializer.class.getName())
                .config("spark.kryo.registrator", GeoSparkKryoRegistrator.class.getName())
                .master("local[*]")
                .appName("myGeoSparkSQLdemo")
                .getOrCreate();

1. 注册从geospark-sql_2.3到sparkSession的所有功能，以便可以直接使用spark-sql

1. Register all the functions from geospark-sql_2.3 to the sparkSession so that it can be used directly spark-sql

// register all functions from geospark-sql_2.3 to sparkSession
GeoSparkSQLRegistrator.registerAll(sparkSession);

现在这是工作示例-

   SparkSession sparkSession = SparkSession.builder()
                .config("spark.serializer", KryoSerializer.class.getName())
                .config("spark.kryo.registrator", GeoSparkKryoRegistrator.class.getName())
                .master("local[*]")
                .appName("myGeoSparkSQLdemo")
                .getOrCreate();

        // register all functions from geospark-sql_2.3 to sparkSession
        GeoSparkSQLRegistrator.registerAll(sparkSession);
        try {
            System.out.println(sparkSession.catalog().getFunction("ST_Geomfromtext"));
            // Function[name='ST_GeomFromText', className='org.apache.spark.sql.geosparksql.expressions.ST_GeomFromText$', isTemporary='true']
        } catch (Exception e) {
            e.printStackTrace();
        }
        // https://datasystemslab.github.io/GeoSpark/api/sql/GeoSparkSQL-Function/
        Dataset<Row> dataframe = sparkSession.sql("select ST_GeomFromText('POINT(-7.07378166 33.826661)')");
        dataframe.show(false);
        dataframe.printSchema();
        /**
         * +---------------------------------------------+
         * |st_geomfromtext(POINT(-7.07378166 33.826661))|
         * +---------------------------------------------+
         * |POINT (-7.07378166 33.826661)                |
         * +---------------------------------------------+
         */

        // using longitude and latitude column from existing dataframe
        Dataset<Row> df = sparkSession.sql("select -7.07378166 as longitude, 33.826661 as latitude");
        df.withColumn("ST_Geomfromtext ",
                expr("ST_GeomFromText(CONCAT('POINT(',longitude,' ',latitude,')'))"))
        .show(false);
        /**
         * +-----------+---------+-----------------------------+
         * |longitude  |latitude |ST_Geomfromtext              |
         * +-----------+---------+-----------------------------+
         * |-7.07378166|33.826661|POINT (-7.07378166 33.826661)|
         * +-----------+---------+-----------------------------+
         */

这篇关于使用Spark/java的ST_geomfromtext函数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！