1. AWS Messaging Services
AWS Messaging services enable similar and different software systems in the cloud to communicate and exchange information asynchronously. These Software systems can be incompatible and in different languages, and also could be on different platforms. AWS messaging services are highly available, highly scalable and highly reliable. AWS Supports multiple messaging services for different types of use cases. In this Article, we’ll discuss various AWS messaging services and picking the right messaging service for depending on the application requirements.
Messaging services allow different software systems, often incompatible and developed in different programming languages on different platforms to communicate and exchange information.
2. Amazon SQS (Simple Queue Service)
Amazon SQS is a distributed queuing webservice for exchanging messages between software systems asynchronously. SQS is used to develop large scale distributed systems with decoupled components without worrying about creating and managing the queue. SQS comes in two flavors, Standard and FIFO (First-In-First-Out). FIFO queue maintains the order of the messages that are produced and consumed and it provides exactly-once delivery mechanism. Standard queue attempts to preserve the order but is flexible, and it provides at least once delivery. The message in Standard queue can be delivered more than once and standard Queue offers high throughput.
As the volume of your requests grows, Amazon SQS is scaled horizontally.
Messages in SQS are encrypted with Server-side encryption and are secure during
their lifetime in Queue.
If your application needs faster messages processing, but don’t care about the message order or duplicate delivery of the message, Standard queue is the best option. For guaranteed order and exactly once delivery, use FIFO queue. When a consumer receives the message and processes it, the message remains in the queue until they are removed explicitly or message retention time is expired. Amazon SQS sets visibility timeout on message (30 sec by default and Max 12 hrs) to prevent other consumers from processing the same message again.
SQS is best useful in cases of one to one messaging, sending emails, form and file processing.
2.1 SQS Limitations
- Message size is limited to 256 KB
- By default the messages in SQS will be deleted after 4 days, but this can be extended up to 14 days
- In Standard SQS, the message could be delivered mre than once
2.2 SQS Best Practices
- Depending on how long it takes to process a message, extend the message’s visibility timeout to the max time it takes to process and delete the message.
- If you don’t know how long it takes to process a message from SQS, create heartbeat for consumer process and extend the visibility timeout accordingly.
- Long polling helps reduce the Amazon SQS cost. While implementing long polling, use one thread per queue, instead of one thread for all the queues. Using one thread for all the queues might cause your application to become unable to process messages available in other queues.
- Use Unique message Ids to prevent the same message being processed by the consumer.
3. Amazon SNS (Simple Notification Service)
Amazon SNS is a fully managed Publish/subscribe asynchronous push notification system where message producer publishes a message and multiple subscribers receive messages. Pub/Sub messaging model is used to decouple applications and to increase the application’s performance, scalability and reliability. Messages pushed to SNS will be persisted across multiple availability zones to prevent loss of messages. SNS sends a message with subscription URL to all subscribers and the subscribers need to confirm the subscription to receive the message.
SNS allows to send messages to multiple services including SQS, Lambda, HTTP(s), Email, SMS etc.. Amazon SNS can be used in combination with SQS, so the message published to SNS will be sent to each SQS and consumer of each SQS will receive the message.
Amazon SNS provides encrypted topics to protect your messages from unauthorized and anonymous access. When a message is published to the encrypted topic, the messages are immediately encrypted using AES-GCM algorithm and a Customer Master Key issued with Key Management Service. The messages are decrypted as they are delivered to subscribing endpoints. Before deleting SNS topic, make sure all the subscriptions for the topic are deleted first. Using SNS, a message can be send to a variety of platforms such as iOS, Android, Java, Node Js, Python, .NET etc SDKs from AWS.
SNS is best suited for use cases such as event notification, publishing message to multiple subscribers, sending dedicated message to each of the SQS Queue.
3.1 SNS Message
Message pushed to SNS topic will have the following parts attached to it.
- MessageId – Unique Id of the message
- Timestamp – Message creation timestamp
- TopicArn – Topic to which the message is published
- Type – Type of delivery
- UnsubscribeUrl – Link to unsubscribe from this topic
- Message – Message Payload
- Subject – Subject field
- Signature – Base64 encoded signature of the message
- Signature version – Version of SNS signature
3.2 SNS Limitations
- The message from SNS may be delivered more than once
- In case of subscriber failure, message may be delivered for a configured number of times.
4. Amazon MQ
Amazon MQ is a managed messaging service for Apache ActiveMQ. AWS
manages the provisioning, setup, patching, and maintenance of ActiveMQ. Existing
applications can be easily migrated and connect to Amazon MQ using standard APIs
and protocols such as JMS, NMS, AMQP, STOMP, MQTT and WebSocket.
Amazon MQ provided high availability and durability of messages. Latency in Amazon MQ is very less (single digit millisecond) between message producer and consumer. Messages placed in Amazon MQ are always stored redundantly across multiple Availability Zones. Amazon MQ supports distributed transactions. Broker name should be unique in an AWS account and there is a limit of 200GB storage capacity per broker.
A Lambda function can be triggered when a message is placed on Amazon MQ. In a hybrid cloud environment, an ActiveMQ in On-premises data center can be integrated with Amazon MQ on the cloud to exchange messages. Refer to this documentation for connecting your Java application with Amazon MQ and also for creating message producer and consumer.
4.1 Best Practices of Amazon MQ
- Amazon MQ encrypts messages at rest and in transit using its securely managed encryption keys. It is highly recommended to use client encryption as well.
- To optimize Amazon MQ for slow consumers, set the concurrentStoreAndDispatchQueues attribute to false.
- Use right broker instance type for better throughput
- In the case of multi-producer and multi-consumer applications, it can be highly inefficient to create a large number of connections on demand for multiple producers. Use connection pooling instead of using PooledConnectionFactory class. Don’t use PooledConnectionFactory on message consumers.
4.2 Configuring Connection pool (from AWS Documentation)
// Create a connection factory. final ActiveMQConnectionFactory connectionFactory = new ActiveMQConnectionFactory(wireLevelEndpoint); // Pass the username and password. connectionFactory.setUserName(activeMqUsername); connectionFactory.setPassword(activeMqPassword); // Create a pooled connection factory. final PooledConnectionFactory pooledConnectionFactory = new PooledConnectionFactory(); pooledConnectionFactory.setConnectionFactory(connectionFactory); pooledConnectionFactory.setMaxConnections(10); // Establish a connection for the producer. final Connection producerConnection = pooledConnectionFactory.createConnection(); producerConnection.start();
4.3 Amazon MQ Limitations
- Number of brokers per broker instance type, per AWS account is limited to 20
- Number of security groups limited to per broker is 5
- Storage capacity per broker is limited to 200GB
5. Amazon Kinesis
Amazon Kinesis is a fully managed web service for processing big data in real time. Kinesis is helpful in collecting, processing and analyzing real-time streaming data to get insights into the information. Using Kinesis the data can be analyzed as it arrives and respond instantly. Using Kinesis the Video, audio, application logs, website click streams, and IoT telemetry can be analyzed. Amazon Kinesis is highly scalable and can handle any amount of streaming data and process data from hundreds and thousands of sources with little latency. Kinesis is capable of processing terabytes of data per hour from various data streams. Kinesis keeps the order of the messages.
Amazon Kinesis is best applicable in use cases such as collecting and analyzing sensor data such as temperature, humidity sensors etc. It is also useful in ingesting all market data reliably and build real-time analytics.
5.1 Kinesis Limitations
- Data blob size is limited to 1 MB
- By default message is deleted from the stream after 24 hrs, but can be extended up to 7 days
6. Amazon Pinpoint
Amazon Pinpoint is a fully managed marketing and analytics service
hosted on Amazon Web Services. Using Amazon Pinpoint marketing team can get
insights to strengthen campaigns and determine how to effectively communicate
with the customer base. Using Amazon Pinpoint, a marketer can define a campaign
schedule to send targeted messages to app end users.
Amazon Pinpoint can be integrated with Mobile and Web applications
to get insights into how the customers are interacting with your apps. It also
helps in sending targeted email, SMS, push notifications, and voice messages
and also track how the customers respond to your messages- i.e Number of
messages delivered, opened or clicked. This is useful for sending promotional
or educational content to engage and retain your users.
Some of the use cases where Amazon Pinpoint is useful are:
- Aggregation of data from all Campaigns
- Demographic charts that provide info on the
devices on which your app is installed - Analytics about user engagement and campaign
metrics - Analytics about application usage time and customer
retention - Revenue charts that provide details on customer
purchases and revenue generated by it - User authentication analytics
7. Amazon IoT Broker
AWS IoT message broker is a managed high throughput publisher/subscriber
broker service that enables the sending and receiving of messages to and from
AWS IoT. Client sends a message to AWS IoT to a topic. This message broker
sends message to all the registered clients for that topic. Using AWS IoT, the
messages between all of your IoT devices can be transmitted securely with low
latency. Based on the message volume, the message broker can be scaled automatically.
AWS IoT supports one-to-one messaging or one-to-million broadcast notification
systems. More AWS IoT features are listed in AWS Documentation.
Hi ,
Nice Article but In your example 4. Mq you mentioned triggering a lambda but MQ dosent provide out of the box event based integration with lambda so you have to long poll which is not supported by lambda.
So have you tried running it on lambda or just local system ?
Regards,
Sachin