从Spark Streaming中的kafka消息中提取时间戳? [英] Extract the time stamp from kafka messages in spark streaming?

问题描述

尝试从kafka来源阅读.我想从收到的消息中提取时间戳以进行结构化火花流. kafka(版本0.10.0.0) Spark Streaming(版本2.0.1)

解决方案

spark.read
  .format("kafka")
  .option("kafka.bootstrap.servers", "your.server.com:9092")
  .option("subscribe", "your-topic")
  .load()
  .select($"timestamp", $"value")

字段时间戳"是您要寻找的.类型-java.sql.Timestamp.确保您连接到0.10 Kafka服务器.早期版本中没有时间戳. 此处描述的字段的完整列表- http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#creating-a-kafka-source-for-batch-queries

Trying to read from kafka source. I want to extract timestamp from message received to do structured spark streaming. kafka(version 0.10.0.0) spark streaming(version 2.0.1)

解决方案

spark.read
  .format("kafka")
  .option("kafka.bootstrap.servers", "your.server.com:9092")
  .option("subscribe", "your-topic")
  .load()
  .select($"timestamp", $"value")

Field "timestamp" is what you are looking for. Type - java.sql.Timestamp. Make sure that you are connecting to 0.10 Kafka server. There is no timestamp in earlier versions. Full list of fields described here - http://spark.apache.org/docs/latest/structured-streaming-kafka-integration.html#creating-a-kafka-source-for-batch-queries

这篇关于从Spark Streaming中的kafka消息中提取时间戳?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！