如何把数据从服务器到室壁运动流 [英] How to put data from server to Kinesis Stream

问题描述

我是新来的室壁运动。读出的文档，我发现我可以创建室壁运动流从生产者获取数据。然后，使用KCL将读取该数据流，以进一步处理。我知道如何通过implemeting IRecordProcessor写KCL应用。

I am new to Kinesis. Reading out the documentation i found i can create the Kinesis Stream to get data from Producer. Then using KCL will read this data from Stream to further processing. I understand how to write the KCL application by implemeting IRecordProcessor .

然而，第一个阶段是如何把数据室壁运动流依然不清晰。我们有一些AWS API，它确实需要实现这个功能。

However the very first stage as how to put data on Kinesis stream is still not clear to me. Do we have some AWS API which does need implementation to achieve this.

情景：我有一个服务器，它是contineously获取数据从文件夹中的各种来源。每个文件夹都包含该文本文件中包含furhter分析工作所需要的属性，其行。我要推这些数据室壁运动流。

Scenarios: I have an server which is contineously getting data from various sources in the folders. Each folder is containing the text file whose rows are containing the required attributes for furhter analytical work. i have to push all these data to Kinesis Stream.

我需要西港岛线被用来在室壁运动流code如下一些下面类putData方法

I need code something as below below class putData method wil be used to out in Kinesis stream

public class Put {

    AmazonKinesisClient kinesisClient;

    Put()
    {
        String accessKey = "My Access Key here" ;
        String secretKey = "My Secret Key here" ;
        AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
       kinesisClient = new AmazonKinesisClient(credentials);
       kinesisClient.setEndpoint("kinesis.us-east-1.amazonaws.com", "kinesis", "us-east-1");
        System.out.println("starting the Put Application");
    }

    public void putData(String fileContent,String session) throws Exception
    {
         final String myStreamName = "ClickStream";

            PutRecordRequest putRecordRequest = new PutRecordRequest();
            putRecordRequest.setStreamName(myStreamName);
            String putData = fileContent;
            putRecordRequest.setData(ByteBuffer.wrap(putData.getBytes()));
            putRecordRequest.setPartitionKey("session"+session);
            PutRecordResult putRecordResult = kinesisClient.putRecord(putRecordRequest);
            System.out.println("Successfully putrecord, partition key : " + putRecordRequest.getPartitionKey()
                    + ", ShardID : " + putRecordResult.getShardId());
            System.out.println(fileContent);
            System.out.println("Sequence Number: "+putRecordResult.getSequenceNumber());

            System.out.println("Data has been PUT successfully");


    }
}

不过从服务器的源文件夹，然后设计了什么，我应该使用调用putData以获得室壁运动流记录读取文件。我是否需要无限循环和阅读中的所有文件，然后做这做一些框架，这将更好地照顾容错能力，单点故障都做到这一点。任何帮助将大大AP preciated。

However reading file from the source folder from the server and then what design i should use to call putData to get the record on Kinesis stream. Do i need infinite loop and reading all files and then do this or some framework which will better do this with care of fault tolerance , single point of failure all . Any help would be greatly appreciated.

简单地说：我需要一个更好的方法将定期生成的数据，室壁运动流的数据以一定的时间间隔服务器生成。 谢谢

Briefly: I need a better technique to put regularly generated data to Kinesis Stream the data is generated at regular interval to server. Thanks

推荐答案

因此​​，看来你已经在使用... <一href="http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/AmazonKinesisClient.html" rel="nofollow">http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/AmazonKinesisClient.html

So it seems you are already using... http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/AmazonKinesisClient.html

您想要具体的方法如下。

Specific method you want is as follows.

您需要一个流名，录制和流密钥。 <一href="http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/model/PutRecordResult.html" rel="nofollow">http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/model/PutRecordResult.html

You need a stream name, record, and stream key. http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/kinesis/model/PutRecordResult.html

不过好像你有这一切？

您然后将需要运行的程序总是拖尾您的服务器的日志文件，当曾经有一个新的生产线也将推动这一点。

You would then need a program running always tailing your server log file and when ever there is a new line it will push this.

但你的数据将只能坐在他们24小时。然后，您需要一个客工计划消耗数据，并把它放在其他AWS资源。

But your data will only sit their for 24 hours. You then need a worker program to consume the data and place it in some other AWS resource.

这篇关于如何把数据从服务器到室壁运动流的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！