CloudWatch Event Notifications Using AWS Lambda

A primary use case for CloudWatchEvents is to keep track of changes across an AWS infrastructure. It currently supports events emitted across Auto Scaling groups, EC2, EBS, and various others. To do anything meaningful with these events, we need a way to consume them. AWS uses the term targets to refer to anything that wants to consume the events and supports AWS Lambda and several others.

In this article, we will see how we can set up an AWS Lambda function to consume events from CloudWatch. By the end of this article, we will have an AWS Lambda function that will post a notification to a Slack channel. However, since the mechanism will be generic, you should be able to customize it to your use cases. Let’s get started!

Setup and General Information

The code repository for this article can be found here. It has two subdirectories:

• functions, which has the source code for the Lambda functions.
• terraform, which has the infrastructure configuration.

To follow along, we will need:

• an AWS user account
• Terraform
• Python 3
• Bash/Powershell/utilities

AWS user account and CLI configuration

We will need an AWS account with the following IAM policies:

• AWSLambdaFullAccess
• IAMFullAccess

The AWS CLI as well as terraform will make use of standard AWS configuration with appropriate credentials set in a AWS profile.

Please note that you may be charged for trying out the demos.

Terraform

We will use Terraform to set up the entire infrastructure, including uploading our Lambda functions.

Python 3

Our Lambda functions will be written in Python 3, and we will be using Python in the scripts used to deploy and update our AWS Infrastructure.

Bash/Powershell/utilities

If you are on Linux/OS X, you will need bash to run the scripts. For Windows, you will need powershell. On Linux/OS X, we will use the zip command to create Lambda deployment artifacts.

An Overview of the Architecture

The overall architecture of the solution that we will build in this article will look as follows:

AWS CloudWatch event -> Lambda function invoked -> Notifications

We will focus on two events:

• EC2 State Change event: This event happens when a AWS EC2 instance changes state – a new EC2 instance is launched or an existing EC2 instance is terminated.
• CloudWatch Health event: A CloudWatch health event happens when there are health related infrastructural changes happening in your AWS account.

To make a CloudWatch event automatically trigger a Lambda function, we need to set up a cloudwatch rule. Regardless of the event we are handling or what we are doing with the event, our Lambda function that receives the event will have the same basic structure.

We will write our Lambda function in Python 3.6, and a fully working function looks as follows:

def handler(event, context):
    print(event)

The function name is handler, which takes two parameters: event and context. The event object has the payload for the event that triggers the Lambda function, and the context object has various metadata related to the specific event.

To see how we can accomplish all of the above, we will first implement a solution that will call a Lambda function whenever an EC2 instance changes state.

Demo: EC2 Instance Running Notification

The Lambda function for this notification looks as follows and is saved in the file main.py:

def handler(event, context):
    print(event)

Whenever it is called, it will print the event body to standard output, which will get automatically logged to AWS CloudWatch logs. We will discuss how we’ll upload our Lambda function shortly. First, let’s briefly go over the infrastructure setup for our Lambda function to be invoked.

The Terraform configuration can be found in the file ec2_state_change.tf. It defines the following key terraform resources:

aws_cloudwatch_event_rule

This defines the rule that we want our lambda function to be invoked for. The event_pattern for EC2 Instance state change is defined as:

"source": [ "aws.ec2" ],
"detail-type": [ "EC2 Instance State-change Notification" ]

aws_cloudwatch_event_target

Next, we define what is invoked once the event happens using this resource. The key parameters are:

target_id = "InvokeLambda"
arn       = "${aws_lambda_function.ec2_state_change.arn}"
}

The arn parameter specifies the Amazon Resource Name for the Lambda function.

aws_lambda_function

This resource registers the lambda function and has the following key parameters:

function_name = "ec2_state_change"
role          = "${aws_iam_role.ec2_state_change_lambda_iam.arn}"
handler       = "main.handler"
runtime       = "python3.6"

s3_bucket         = "aws-health-notif-demo-lambda-artifacts"
s3_key            = "ec2-state-change/src.zip"
s3_object_version = "${var.ec2_state_change_handler_version}"

The function_name above is an identifier for AWS and doesn’t have any relationship to the name of your function in your code. The Lambda function’s IAM role specified by another resource has a default sts:AssumeRole policy and a policy that allows our function logs to be pushed to CloudWatch.

The handler is of the form <python-module>.<function> and specfies the Python function name that you want to be invoked. runtime specifies the AWS Lambda runtime.

s3_bucket specifies the bucket in which our Lambda’s code will live, s3_key the key name for the Lambda code, and s3_object_version allows us to deploy a specific version of the above object.

ec2_state_change_cloudwatch

The last key resource that is defined allows CloudWatch to invoke our Lambda function and has the following parameters:

action        = "lambda:InvokeFunction"
function_name = "${aws_lambda_function.ec2_state_change.function_name}"
principal     = "events.amazonaws.com"
source_arn    = "${aws_cloudwatch_event_rule.ec2_state_change.arn}"

Uploading Lambda function

As we saw in the configuration for the Lambda function, the code for the Lambda function will live in S3. Hence, after every code change, we will update our code in S3 using the AWS CLI as follows. On Linux, this will look similar to:

# Create a .zip of src
$ pushd src
$ zip -r ../src.zip *
$ popd

$ aws s3 cp src.zip s3://aws-health-notif-demo-lambda-artifacts/ec2-state-change/src.zip

We can make the above execution part of a continuous integration pipeline.

Deploying the latest version of the code

Once our code has been uploaded to S3, we can then run terraform to update our Lambda function to use the new version of the code as follows:

$ version=$(aws s3api head-object --bucket aws-health-notif-demo-lambda-artifacts --key ec2-state-change/src.zip)
$ version=$(echo $version | python -c 'import json,sys; obj=json.load(sys.stdin); print(obj["VersionId"])')

# Deploy to demo environment
$ pushd ../../terraform/environments/demo
$ ./tf.bash cloudwatch_event_handlers apply -var ec2_state_change_handler_version=$version \
    -target=aws_lambda_function.ec2_state_change \
    -target=aws_lambda_permission.ec2_state_change_cloudwatch \
    -target=aws_cloudwatch_event_target.ec2_state_change \
    -target=aws_iam_role_policy.ec2_state_change_lambda_cloudwatch_logging
$ popd

Both of the above steps can be encapsulated in a single script that becomes a single entrypoint for creating the EC2 state change CloudWatch event handler as well as updating the Lambda function that handles it.

Running the demo

To set up the above Lambda function and all necessary infrastructure in an AWS account, we will just have to run the functions\ec2_state_change\deploy.bash or the functions\ec2_state_change\deploy.ps1 script. Once it has been done, if you create a new EC2 instance or stop/terminate an existing one, you will see CloudWatch logs as follows:

[2018-07-04T09:46:18+10:00] (2018/07/03/[$LATEST]aa226226b6b24a0cae83a948dcc29b95) START RequestId: 4798542c-7f1b-11e8-8493-836165a23514 Version: $LATEST

[2018-07-04T09:46:18+10:00] (2018/07/03/[$LATEST]aa226226b6b24a0cae83a948dcc29b95) {'version': '0', 'id': '73c10269-00a0-644d-b92b-820846bb19db', 'detail-type': 'EC2 Instance State-change Notification', 'source': 'aws.ec2', 'account': '033145145979', 'time': '2018-07-03T23:46:16Z', 'region': 'ap-southeast-2', 'resources': ['arn:aws:ec2:ap-southeast-2:033145145979:instance/i-0e1153ece20b77590'], 'detail': {'instance-id': 'i-0e1153ece20b77590', 'state': 'pending'}}

[2018-07-04T09:46:18+10:00] (2018/07/03/[$LATEST]aa226226b6b24a0cae83a948dcc29b95) END RequestId: 4798542c-7f1b-11e8-8493-836165a23514

Demo: AWS Health Events -> Slack

Next, we will write a Lambda function that will post AWS Health events to a Slack channel of your choice. First, we will configure an incoming webhook for our Slack channel. Please see this link to start off how you can add one for your channel. If you follow the setup through, you will have a webhook URL similar to https://hooks.slack.com/services/string/<string>/<string>. Beyond this stage, I will assume that we have this webhook URL.

Writing the Lambda function

The Lambda function will look as follows:

import os
import sys
import json

CWD = os.path.dirname(os.path.realpath(__file__))
sys.path.insert(0, os.path.join(CWD, "libs"))

import requests

def handler(event, context):

    WEBHOOK_URL = os.getenv("WEBHOOK_URL")
    if not WEBHOOK_URL:
        print("WEBHOOK_URL not defined or empty")
        return
    # see: https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/EventTypes.html#health-event-types for event structure 
    r = requests.post(
        WEBHOOK_URL,
        json = {'text': '*New AWS Health event* ```{0}```'.format(str(event))}
    )
    print(r)

Infrastructure configuration

The infrastructure configuration for this Lambda function is exactly the same as our previous one, except for the aws_cloudwatch_event_rule which defines the event_pattern as follows:

"source": [ "aws.health" ],
"detail-type": [ "AWS Health Event" ]

Deploying the Lambda function

Similar to our EC2 state change demo above, to deploy the above function, we will run the deployment script from the functions/health_event directory:

$ HEALTH_EVENT_WEBHOOK_URL="<your webhook url>" ./deploy.bash

This will create the necessary CloudWatch event rules and set up the targets in addition to deploying the Lambda function.

Once everything has completed, you can invoke the Lambda function directly using the AWS CLI:

$ aws lambda invoke --invocation-type RequestResponse --function-name health_event --log-type Tail --payload '{"message":"hello"}' outfile.txt

You should see a Slack message in your configured channel:

Summary

In this article, we learned how we can set up Lambda functions as targets for CloudWatch events. The Slack notification demo works but can be improved in a couple of ways:

• The webhook URL can be encrypted at rest using AWS Secrets Manager.
• The notification can be made richer by being able to process the incoming messages.

The repository used for this article is available here.

The following resources should be helpful for learning more:

• CloudWatch events
• Guide for writing Lambda functions in Python
• Terraform guide for AWS

Do you want to know how to develop your skillset to become a Java Rockstar?

Subscribe to our newsletter to start Rocking right now!

To get you started we give you our best selling eBooks for FREE!

1. JPA Mini Book

2. JVM Troubleshooting Guide

3. JUnit Tutorial for Unit Testing

4. Java Annotations Tutorial

5. Java Interview Questions

6. Spring Interview Questions

7. Android UI Design

and many more ....

I agree to the Terms and Privacy Policy

Sign up

Thank you!

We will contact you soon.