使用cloudformation模板将日志流化为弹性 [英] stream logs to elastic using cloudformation template

问题描述

Cloudtrail默认日志可以流到弹性搜索域，如下图所示。如何使用cloudformation模板实现此目标？

解决方案

更新：

如果您使用的是aws-cli，请看一下我的答案< a href = https://stackoverflow.com/questions/58062415/stream-logs-to-elastic-using-aws-cli>此处。

< hr>

嗯，经过几个小时的探索和阅读了大量文档，我终于成功创建了此模板。

设计师概述：

为了启用流日志到elasticsearch，我们需要创建以下资源：

1. lambda函数wi将日志从cloudwatch日志组转发到Elasticsearch。


2. 相关的IAM角色，从cloudwatch获取日志并插入到Elasticsearch。



3. 
   
   
   
   
   

   Cloudwatch日志：
   

   
   
   
   
   

   Cloudtrail default logs can be streamed to elasticsearch domain as shown in this image. How do I achieve this using cloudformation template?

   解决方案

   Update:

   If you are using aws-cli, take a look at my answer here.

   ---

   Well, after a few hours of exploring and reading a lot of documentation I finally succeeded to create this template.

   Designer Overview :

   In order to enable the stream logs to elasticsearch we need to create the following resources:

   1. The lambda function will forward the logs from cloudwatch log group to Elasticsearch.
   2. Relevant IAM Role to get logs from cloudwatch and insert to Elasticsearch.
   3. Lambda permission - The AWS::Lambda::Permission resource grants an AWS service or another account permission to use a function to allow the cloudwatch log group to trigger the lambda.
   4. Subscription Filter - The AWS::Logs::SubscriptionFilter resource specifies a subscription filter and associates it with the specified log group. Subscription filters allow you to subscribe to a real-time stream of log events and have them delivered to a specific destination.

   Template usage:

   1. Download LogsToElasticsearch.zip from my Github page.
   2. Update var endpoint = '${Elasticsearch_Endpoint}'; in index.js with your Elasticseatch url e.g - 'search-xxx-yyyy.eu-west-1.es.amazonaws.com';.
   3. Copy the zip file to s3 bucket which will be used in the template (LambdaArtifactBucketName).
   4. Fill relevant Parameters - you can find descriptions to each resource.

   Template YAML:

   AWSTemplateFormatVersion: 2010-09-09
   Description: Enable logs to elasticsearch
   Parameters:
     ElasticsearchDomainName:
       Description: Name of the Elasticsearch domain that you want to insert logs to
       Type: String
       Default: amitb-elastic-domain
     CloudwatchLogGroup:
       Description: Name of the log group you want to subscribe
       Type: String
       Default: /aws/eks/amitb-project/cluster
     LambdaName:
       Description: Name of the lambda function
       Type: String
       Default: amitb-cloudwatch-logs
     LambdaRole:
       Description: Name of the role used by the lambda function
       Type: String
       Default: amit-cloudwatch-logs-role
     LambdaArtifactBucketName:
       Description: The bucket where the lambda function located
       Type: String
       Default: amit-bucket
     LambdaArtifactName:
       Description: The name of the lambda zipped file
       Type: String
       Default: LogsToElasticsearch.zip
     VPC:
       Description: Choose which VPC the Lambda-functions should be deployed to
       Type: 'AWS::EC2::VPC::Id'
       Default: vpc-1111111
     Subnets:
       Description: Choose which subnets the Lambda-functions should be deployed to
       Type: 'List<AWS::EC2::Subnet::Id>'
       Default: 'subnet-123456789,subnet-123456456,subnet-123456741'
     SecurityGroup:
       Description: Select the Security Group to use for the Lambda-functions
       Type: 'List<AWS::EC2::SecurityGroup::Id>'
       Default: 'sg-2222222,sg-12345678'
   Resources:
     ExampleInvokePermission:
       Type: 'AWS::Lambda::Permission'
       DependsOn: ExampleLambdaFunction
       Properties:
         FunctionName:
           'Fn::GetAtt':
             - ExampleLambdaFunction
             - Arn
         Action: 'lambda:InvokeFunction'
         Principal: !Sub 'logs.${AWS::Region}.amazonaws.com'
         SourceArn: !Sub >-
           arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:${CloudwatchLogGroup}:*
         SourceAccount: !Ref 'AWS::AccountId'
     LambdaExecutionRole:
       Type: 'AWS::IAM::Role'
       Properties:
         RoleName: !Ref LambdaRole
         ManagedPolicyArns:
           - 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole'
         AssumeRolePolicyDocument:
           Version: 2012-10-17
           Statement:
             - Effect: Allow
               Principal:
                 Service:
                   - lambda.amazonaws.com
               Action:
                 - 'sts:AssumeRole'
         Path: /
         Policies:
           - PolicyName: lambda-to-es-via-vpc-policy
             PolicyDocument:
               Version: 2012-10-17
               Statement:
                 - Effect: Allow
                   Action:
                     - 'es:*'
                   Resource:
                     - !Sub >-
                       arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/${ElasticsearchDomainName}
           - PolicyName: logs-and-ec2-permissions
             PolicyDocument:
               Version: 2012-10-17
               Statement:
                 - Effect: Allow
                   Action:
                     - 'ec2:CreateNetworkInterface'
                     - 'ec2:DescribeNetworkInterfaces'
                     - 'ec2:DeleteNetworkInterface'
                     - 'logs:CreateLogGroup'
                     - 'logs:CreateLogStream'
                     - 'logs:PutLogEvents'
                   Resource: '*'
     ExampleLambdaFunction:
       Type: 'AWS::Lambda::Function'
       DependsOn: LambdaExecutionRole
       Properties:
         Code:
           S3Bucket: !Ref LambdaArtifactBucketName
           S3Key: !Ref LambdaArtifactName
         FunctionName: !Ref LambdaName
         Handler: !Sub '${LambdaName}.handler'
         Role:
           'Fn::GetAtt':
             - LambdaExecutionRole
             - Arn
         Runtime: nodejs8.10
         Timeout: '300'
         VpcConfig:
           SecurityGroupIds: !Ref SecurityGroup
           SubnetIds: !Ref Subnets
         MemorySize: 512
     SubscriptionFilter:
       Type: 'AWS::Logs::SubscriptionFilter'
       DependsOn: ExampleInvokePermission
       Properties:
         LogGroupName: !Ref CloudwatchLogGroup
         FilterPattern: '[host, ident, authuser, date, request, status, bytes]'
         DestinationArn:
           'Fn::GetAtt':
             - ExampleLambdaFunction
             - Arn
   

   Results:

   Cloudwatch log:

   Hope you find it helpfull.

   Update 02/09/2020:

   node.js 8.10 is now deprecated, you should use node.js 10 or 12.

   https://docs.aws.amazon.com/lambda/latest/dg/runtime-support-policy.html

   这篇关于使用cloudformation模板将日志流化为弹性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！