Retrieve logs from AWS Cloud-watch using Python

Amazon Web Services provides many services for developers and end users. Amazon cloudwatch is a platform where the logs of you application can be stored and visualized according to your need. Even though Amazon provides you with various types of services, there could be instances where you have to retrieve the logs from cloudwatch to your application. Here we are going to discuss about how to retrieve the logs from cloud watch to your python application.

To retrieve the logs from cloud watch using Python, you need to use the SDK provided by Amazon for python users, which is Boto3. Before starting with Boto3, you have to configure your credentials. First, you should have AWS CLI installed. Then you can use this following command to start the AWS and configure the credentials.

aws configure

Next you will have to input the AWS access key ID followed by the AWS Secret access key. After that you'll have to input the default region name as well.

If not, this process could be done manually as well. You have to open the file “~/.aws/config” and add the following credentials and save.

[default]
aws_access_key_id = **********************
aws_secret_access_key = **********************

These credentials you provide could be from an SSO user profile, or an IAM user profile. But depending on the application that you are going to use these logs, the credential types that you require could vary. A SSO profile credentials will be expiring within a small amount of time. So, if your application needs to run for a while, you cannot use the SSO credentials. So what you require is IAM accounts credentials. The IAM account credentials will not expire until manually removed.

Now we are all set from the AWS end. We have to use Boto3 along with python to retrieve the logs. Will start by installing Boto3.

pip install boto3

Everything related to the environment are properly configured now. Now we can start coding our application. Start your favorite python IDE and create a new project.

Now let's start by importing boto3 and creating a client.

import boto3

client = boto3.client('logs', region_name='us-east-1')

Now we have successfully created a client. Hereafter we can use that client and retrieve the logs from cloud watch. There are many APIs provided for us to do all actions related to Amazon cloud watch. Here we are going to use the filter log events API to filter and get the logs.

In these APIs, some parameters are mandatory and some parameters are optional. In the filter log events API, the log group name is mandatory and others are optional. Here is a sample request :

response = client.filter_log_events(
    logGroupName='string',
    logStreamNames=[
        'string',
    ],
    logStreamNamePrefix='string',
    startTime=123,
    endTime=123,
    filterPattern='string',
    nextToken='string',
    limit=123,
    interleaved=True|False
)

The logGroupName parameter contains the name of the group which could be strictly identified by its name. But the log stream name could be dynamic. So we can use the logStreamNamePrefix parameter to filter the log streams by a prefix. Also we can use filterPattern, which means the API will look for the word or phrase that we provide through the filterPattern parameter, in the logs.

Next you can simply print the response to see the logs that you retrieved from Amazon cloud watch.

print(response)

Great, now we have had a walk-through on how to retrieve the logs from cloud watch. Still there are some important things that you should keep in mind.

When you are retrieving logs, the API will filter from either 10000 logs or logs worth of 1 MB. So when you receive the response, it will not be the complete response on some instances. The easiest way to identify if you have gotten the complete response is to check for the next token in the response end. The next token will be the last parameter of the response.

You will receive the next token only when the filtering is not done completely. So you have to send the same request again along with the next token as a parameter to start from the point where it stopped filtering. You will have to repeat this process until you receive a response without a next token. So, when you finally receive a response without the next token, you can make sure that you have received the complete response.

Yaay, Now we know how to retrieve logs from AWS Cloud Watch. If you require details about other available APIs for boto3, refer to the available services in the Boto3 documentation.

Kudos.