使用 cloudformation 模板将日志流式传输到弹性 [英] stream logs to elastic using cloudformation template
问题描述
Cloudtrail 默认日志可以流式传输到 elasticsearch 域,如此图所示.我如何使用 cloudformation 模板实现这一点?
更新:
如果您使用的是 aws-cli,请查看我的回答
为了启用流日志到elasticsearch,我们需要创建以下资源:
- lambda 函数会将来自 cloudwatch 日志组的日志转发到 Elasticsearch.
- 从 cloudwatch 获取日志并插入到 Elasticsearch 的相关 IAM 角色.
Cloudwatch 日志:
希望对您有所帮助.
2020 年 2 月 9 日更新:
node.js 8.10 现已弃用,您应该使用 node.js 10 或 12.
Cloudtrail default logs can be streamed to elasticsearch domain as shown in this image. How do I achieve this using cloudformation template?
解决方案Update:
If you are using aws-cli, take a look at my answer here.
Well, after a few hours of exploring and reading a lot of documentation I finally succeeded to create this template.
Designer Overview :
In order to enable the stream logs to elasticsearch we need to create the following resources:
- The lambda function will forward the logs from cloudwatch log group to Elasticsearch.
- Relevant IAM Role to get logs from cloudwatch and insert to Elasticsearch.
- Lambda permission -
The AWS::Lambda::Permission resource grants an AWS service or another account permission to use a function
to allow the cloudwatch log group to trigger the lambda. - Subscription Filter -
The AWS::Logs::SubscriptionFilter resource specifies a subscription filter and associates it with the specified log group. Subscription filters allow you to subscribe to a real-time stream of log events and have them delivered to a specific destination.
Template usage:
- Download LogsToElasticsearch.zip from my Github page.
- Update
var endpoint = '${Elasticsearch_Endpoint}';
in index.js with your Elasticseatch url e.g -'search-xxx-yyyy.eu-west-1.es.amazonaws.com';
. - Copy the zip file to s3 bucket which will be used in the template (LambdaArtifactBucketName).
- Fill relevant Parameters - you can find descriptions to each resource.
Template YAML:
AWSTemplateFormatVersion: 2010-09-09 Description: Enable logs to elasticsearch Parameters: ElasticsearchDomainName: Description: Name of the Elasticsearch domain that you want to insert logs to Type: String Default: amitb-elastic-domain CloudwatchLogGroup: Description: Name of the log group you want to subscribe Type: String Default: /aws/eks/amitb-project/cluster LambdaName: Description: Name of the lambda function Type: String Default: amitb-cloudwatch-logs LambdaRole: Description: Name of the role used by the lambda function Type: String Default: amit-cloudwatch-logs-role LambdaArtifactBucketName: Description: The bucket where the lambda function located Type: String Default: amit-bucket LambdaArtifactName: Description: The name of the lambda zipped file Type: String Default: LogsToElasticsearch.zip VPC: Description: Choose which VPC the Lambda-functions should be deployed to Type: 'AWS::EC2::VPC::Id' Default: vpc-1111111 Subnets: Description: Choose which subnets the Lambda-functions should be deployed to Type: 'List<AWS::EC2::Subnet::Id>' Default: 'subnet-123456789,subnet-123456456,subnet-123456741' SecurityGroup: Description: Select the Security Group to use for the Lambda-functions Type: 'List<AWS::EC2::SecurityGroup::Id>' Default: 'sg-2222222,sg-12345678' Resources: ExampleInvokePermission: Type: 'AWS::Lambda::Permission' DependsOn: ExampleLambdaFunction Properties: FunctionName: 'Fn::GetAtt': - ExampleLambdaFunction - Arn Action: 'lambda:InvokeFunction' Principal: !Sub 'logs.${AWS::Region}.amazonaws.com' SourceArn: !Sub >- arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:${CloudwatchLogGroup}:* SourceAccount: !Ref 'AWS::AccountId' LambdaExecutionRole: Type: 'AWS::IAM::Role' Properties: RoleName: !Ref LambdaRole ManagedPolicyArns: - 'arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole' AssumeRolePolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Principal: Service: - lambda.amazonaws.com Action: - 'sts:AssumeRole' Path: / Policies: - PolicyName: lambda-to-es-via-vpc-policy PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - 'es:*' Resource: - !Sub >- arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/${ElasticsearchDomainName} - PolicyName: logs-and-ec2-permissions PolicyDocument: Version: 2012-10-17 Statement: - Effect: Allow Action: - 'ec2:CreateNetworkInterface' - 'ec2:DescribeNetworkInterfaces' - 'ec2:DeleteNetworkInterface' - 'logs:CreateLogGroup' - 'logs:CreateLogStream' - 'logs:PutLogEvents' Resource: '*' ExampleLambdaFunction: Type: 'AWS::Lambda::Function' DependsOn: LambdaExecutionRole Properties: Code: S3Bucket: !Ref LambdaArtifactBucketName S3Key: !Ref LambdaArtifactName FunctionName: !Ref LambdaName Handler: !Sub '${LambdaName}.handler' Role: 'Fn::GetAtt': - LambdaExecutionRole - Arn Runtime: nodejs8.10 Timeout: '300' VpcConfig: SecurityGroupIds: !Ref SecurityGroup SubnetIds: !Ref Subnets MemorySize: 512 SubscriptionFilter: Type: 'AWS::Logs::SubscriptionFilter' DependsOn: ExampleInvokePermission Properties: LogGroupName: !Ref CloudwatchLogGroup FilterPattern: '[host, ident, authuser, date, request, status, bytes]' DestinationArn: 'Fn::GetAtt': - ExampleLambdaFunction - Arn
Results:
Cloudwatch log:
Hope you find it helpfull.
Update 02/09/2020:
node.js 8.10 is now deprecated, you should use node.js 10 or 12.
https://docs.aws.amazon.com/lambda/latest/dg/runtime-support-policy.html
这篇关于使用 cloudformation 模板将日志流式传输到弹性的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!