Monitoring and Logging Solution for different systems using Cloud watch, Graphana and Loki

Grafana is multi-platform open source analytics and interactive visualization software available since 2014. It provides charts, graphs, and alerts for the web when connected to supported data sources. It is expandable through a plug-in system. End users can create complex monitoring dashboards using interactive query builders.

We have displayed LOG management using ELK in one of the previous post. This post will be showing how to install Graphana , add the Loki shipper endpoint to create serverless Lambda and use Cloudwatch to ingest logs from different systems.

Please note , Focus will mainly on creating the bare bone architecture of the whole platform rather than using Graphana or its scalability.

Installation of Graphana

We have used Bitnami cloud launcher portal to host a single instance graphana portal. This will help us creating a running graphana instance and provide us the admin credentials with URL to login.

This will help us create a small instance of graphana and when we hit the url with default port(3000) in the browser with the host ip of the machine. We will receive the homepage of graphana.

Next we will consider hosting the Logging stash open source library(Loki Shipper) in the EC2 environment in order to ingest the logs from Cloud Watch components

Hosting Loki and Promtail

There is an excellent blog from Loki host loki in AWS cloud. We have followed the same tutorials and created a AMI of the instance to run in an ASG under a EIP which will ingest logs from CW Subscription filter in a scalable way. Once you set up the loki as an scalable solution in cloud. We can create a subscription filter with Lambda for ingesting the logs.

Lambda for subscription filter

Once the loki shipper endpoint is obtained, we need to create a lambda function which will be triggered everytime any logs are being put into the subscription group. We can customise the lamda to add different tags for loki shippers to be successfully recognised in the graphana monitoring dashboard.

Here is the open source github tutorial which helps us to host the lamba function : loki shipper lambda – This requires the following details :

  • AWS Account
  • AWC CLI User with the following permissions:
    • Lambda full Access
    • IAM read access
    • CloudWatch logs full access
    • CloudWatch events full access
  • A IAM Role with name “loki-shipper” for the loki-shipper lambda function with the following permissions
    • CloudWatch logs full access

Once the setup is complete, We have achieved a full integration to automate logs movement from Cloudwatch logs group to graphana. The following dashboard shows the lambda logs flowing in real time to graphana

Conclusion

We have showcased the architecture and different pieces which comes together to give us single source of dashboard for monitoring and logging. Here are some of the links which can be used for reference :

Be the first to comment

Leave a Reply

Your email address will not be published.


*