如何从 Zeppelin 加载 Flink Streaming 数据 [英] How to load Flink Streaming data from Zeppelin

问题描述

我开始在 Zeppelin 上使用 Flink，并尝试在流媒体中运行最简单的程序:wordcount.当我使用终端在本地模式下运行此代码时，它可以工作.

I'm starting to use Flink on Zeppelin and trying to run the simplest program in streaming: wordcount. When I run this code in local mode using the terminal, it works.

这是我的做法:https://ci.apache.org/projects/flink/flink-docs-release-1.2/quickstart/setup_quickstart.html

这是代码:

object SocketWindowWordCount {

  /** Main program method */
  def main(args: Array[String]) : Unit = {

    // the host and the port to connect to
    var hostname: String = "localhost"
    var port: Int = 0

    try {
      val params = ParameterTool.fromArgs(args)
      hostname = if (params.has("hostname")) params.get("hostname") else "localhost"va
      port = params.getInt("port")
    } catch {
      case e: Exception => {
        System.err.println("No port specified. Please run 'SocketWindowWordCount " +
          "--hostname <hostname> --port <port>', where hostname (localhost by default) and port " +
          "is the address of the text server")
        System.err.println("To start a simple text server, run 'netcat -l <port>' " +
          "and type the input text into the command line")
        return
      }
    }

    // get the execution environment
    val env: StreamExecutionEnvironment = StreamExecutionEnvironment.getExecutionEnvironment

    // get input data by connecting to the socket
    val text = env.socketTextStream(hostname, port, '\n')

    // parse the data, group it, window it, and aggregate the counts 
    val windowCounts = text
          .flatMap { w => w.split("\\s") }
          .map { w => WordWithCount(w, 1) }
          .keyBy("word")
          .timeWindow(Time.seconds(5))
          .sum("count")

    // print the results with a single thread, rather than in parallel
    windowCounts.print().setParallelism(1)

    env.execute("Socket Window WordCount")
  }

  /** Data type for words with count */
  case class WordWithCount(word: String, count: Long)
}

但是，当我尝试在 Zeppelin 上运行此代码时……我不知道该怎么做.我想我也需要在终端中打开一个套接字，但我不知道如何让 Flink 连接到这个端口.我应该在 Flink 上打开几个窗口吗?

However, when I try to run this code on Zeppelin... I do not how to do it. I guess I need to open a socket in terminal too but I do not how make that Flink connect with this port. Should I open several windows on Flink?

我可以总结中的问题:如何从Zeppelin加载flink流数据???

我需要在最后添加一些 Scala 方法，例如:

I need add some Scala method at the end like:

import scala.io.Source

val programRunning = Source.from"Socket"...?

预先感谢您的帮助！:)

Thanks in advance for your help! :)

推荐答案

在终端中，你可以使用

nc -lk 9999

建立一个绑定到端口 9999 的服务，该服务将把它在 stdin 上接收到的任何内容发送给它的客户端.

to establish a service bound to port 9999 that will send whatever it receives on stdin to its clients.

Apache Zeppelin:一种更友好的方式Flink 也可能有帮助.

这篇关于如何从 Zeppelin 加载 Flink Streaming 数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！