In the current Cloud Ecosystem, Where Reactive Programming is the new Buzz Word for developing any business use case. We shall demonstrate an Architecture which shows the system to system messages or email when we have any event driven invocation(upload any object to S3 in this case).
We will be familiarized with three integral components in this post. Simple Notification Service(SNS) ,Simple Queue Service(SQS) and Simple Email Service(SES).
Other Components which are used in the architecture.
- Cognito User Pool
- API Gateway
Let’s look at the high level architecture. When a client uploads an object to the configured S3 bucket, an S3 event notification will fire towards SNS and an trigger for Lambda Function is executed. SNS publishing the event inside the respective topic. There will be a subscribers for that topic: An SQS queue. The Triggered Lambda Function stores the signed Downloadable URL in the DynamoDB.
The SQS queue stores the event for asynchronous processing, e.g. thumbnail generation or image classification. A Different Lambda Function is configured for the SQS queue which gets invoked when we have incoming message. Within the scope of this blog post we are not going to discuss the asynchronous processing part. Due to the decoupling of publishing and subscribing with SNS we are free to add more consumers for the events later.
In our case we are going to send the events to SNS and then allow interested applications to subscribe. This is referred to as the messaging fanout pattern. Instead of sending events directly to all parties, by using SNS as an intermediate broker we decouple publishing and subscription.
SNS is a simple pub/sub service which organizes around topics. A topic groups together messages of the same type which might be of interest to a set of subscribers. In case of a new message being published to a topic, SNS will notify all subscribers. You can configure delivery policies including configuration of maximum receive rates and retry delays.
We will also subscribe an SQS queue to the topic, storing the events for asynchronous processing by, e.g., another Lambda function or a long running polling service. The next section explains how to implement the architecture.
Most of the coding is done in Node JS which is very lightweight and configurable.
We will create an S3 Bucket which will configurable to send notification using Lambda to SNS Topic and also store the data to DynamoDB.
The above implementation shows sends the details to SNS topic and also storing the url in dynamodb table. There are other SNS topics which it currently subscribes but those are not part of this post.
Event driven Implementation
The Above components are also configured by reactive architecture. This pattern is also called Fanning out Architecture. All the below configurations are part of this architecture which needs to done only one time and will be scalable automatically.
S3 Object Notification
We need to add the above mentioned Lambda function in the Properties Tab and Events Settings. All the other tabs and security can also be configured but not in scope for this post.
SNS is a fully managed push notification service that lets you send individual messages or to fan-out messages to large numbers of recipients. Amazon SNS makes it simple and cost effective to send push notifications to mobile device users, email recipients or even send messages to other distributed services.
We need to create an SNS Topic and add a SQS Subscription to the topic. Since, This is an SQS Topic which we will be publishing, so there will be no need for verification of the endpoint.
Save the ARN of the topic and add the same in the above mentioned Lambda function. The Protocol of the Subscription should be SQS and in the later part, I will demonstrate how to configure the same. We need to be in the Same Region as the Bucket to get the triggered notification.
SQS refers to a message queuing service for reliably communicating among distributed software components and microservices – at any scale
We have created a simple Queue in SQS which consumes the messages and a different lambda gets triggered based on the incoming message.
We need save the ARN of the topic and add it in the above SNS configurations. There are also other parameters which can be used to customize and orchestrate the queue.
The Lambda function will be configured in the Triggers tab and we need to confirm the queue to receive the SNS Messages in QUEUE ACTIONS(Not in the Picture) drop down.
The Above picture should that Lambda is configured for Getting the Messages in the Queue and send the same to the Email using SES for dedicated Users.
The Above lambda shows implementation of getting the contents from SQS Queue and Sending Emails. The Emails are being fetched from a dedicated User Pool in Cognito Service. We are getting the details of the users by making an AWS SDK invocation inside the lambda asynchronously. The Scope of Configuring the Cognito Service is not part of this post.We will show the configuration of Simple Email Service(SES) used in the above implementation. The Open Source NodeMailer JS library is used to send the email by using simple SMTP protocol.
Amazon Simple Email Service (Amazon SES) is a cost-effective email service built on the reliable and scalable infrastructure that Amazon.com developed to serve its own customer base.
We will verify an email Address by adding the same and confirming the same through the auto generated email send by AWS as part of the managed service. This Email Address will be now added in the TO field of the Lambda function for sending emails.
SMTP Settings for the SES Service: We need to configure the SMTP credentials in order to send the email from the functions. The Credentails can be obtained in the following tab.
Server Name, Port and Authentication Credentials (which will be one time generated) needs to be added in the lambda functions.
Note:Amazon doesn’t grant unlimited SES usage to new users instead they are placed in Amazon sandbox mode. In sandbox, the users have full access to all Amazon SES sending methods, however, the From address and To addresses must be verified once and there is a limit of 200messages per 24 hour period.
To remove the restriction on recipient addresses and increase your sending limits, you can opt out from sandbox mode to do this — follow the step in below mentioned link
Finally, we are done!
We can also add other subscribers in the SNS topic such as automated notification in a Slack Channel or Saving the details to other Storage. But overall, this architecture should serve the purpose for having more customization as per the business implementation.
We have seen how it is possible to implement an event processing pipeline with potentially multiple consumers and producers using fully managed building blocks like SNS, SQS, and Lambda. SNS provides pub/sub functionality to decouple producers and consumers, while SQS gives us the ability to process events asynchronously.