AWS SQS as Messaging Queue In Spring Boot

Messaging queues are a basic de-facto when it comes in handling asynchronous operations. We will try to achieve communication using AWS SQS in Spring Boot

What is a Messaging Queue?

A Messaging Queue can be thought of as a container which holds messages in either ordered or un-ordered manner. It does the basic push and pop operations on the messages.

Note, a message queue is different than message brokers. Message queue present in market is ActiveMQ, RabbitMQ, etc, whereas Apache Kafka is a message broker.

Why do we need a Messaging Queue?

Messaging queues help in establishing asynchronous communication between services. Services can communicate with the queue for publishing and consuming messages instead of each other.

What is AWS SQS?

AWS SQS (Simple Queue Service) is an AWS hosted and managed queue service which provides all the MQ (Message Queue) capabilities to decouple micro-services, distributed systems and serverless applications.

Internally, SQS uses elastic-mq to perform all the operations

SQS provides two types of messaging queues :

1. FIFO Queue (First In First Out Order)
2. Standard Queue (Best Effort Ordering)

Basic Attributes of an SQS Queue :

Naming: A FIFO queue can be identified with the name having an extension of “.fifo”, else it is a Standard Queue

Visibility Timeout: This is the length of time that a message received from a queue (by one consumer) will not be visible to the other message consumers. This can be from 0 seconds to max 12 hours.

Basically this means, once a message has been popped out of the queue, it will be visible to the consumer only who requested it and will not be visible for the configured time to anyone else. Once, the time period is completed the message goes back to the queue.

Message retention period: This is the time period till which the message will be retained in the queue. This can be between 1 minute to max 4 days.

Say, we send a message to the queue, then the message will be retained in the queue for the configured retain period and after that it will be automatically deleted.

Delivery delay: This is the time period which states that after you make a send message call to queue, the message will be physically present after a certain delay. This can be configured to be in between 0 seconds to 15 minutes.

Maximum message size: This is self explanatory. The message size can be configured in between 1 KB to 256 KB.

Receive message wait time: This is the maximum amount of time that polling or consumer will wait for messages to become available to receive. The time period is configurable between 0 seconds to 20 seconds

Content Based Deduplication: This is an optional configuration only for FIFO queues. If this is enabled, it makes message de-duplication optional.

We will keep this out of scope for this blog.

Dead Letter Queue: This is one of the features AWS SQS provides which helps in moving trouble-some messages automatically.

Let’s say, a message in the queue is unable to get processed due to some faulty message content and keeps on failing. This message can be moved into a separate queue after some particular retries and can be debugged separately.

We can define a DLQ similar to creating a Standard / FIFO queue and assign it to the primary queue as a DLQ and define the number of receives after which messages should be moved from primary queue to DLQ.

Messages in Queue:

Both FIFO / Standard Queue messages have few common properties -

Body: Here the message content is sent

Message Attributes: Optional Key-Value pairs can be sent along-with the message body

Delivery Delay: This can be configured only while producing Standard Queue messages.

Message group ID: This is a property only for FIFO messages and has to be provided some value. Messages are ordered in FIFO queues based on group ids.

Message deduplication ID: This is also specific to FIFO messages only. This is optional, if Content-Based Deduplication is enabled.

In SQS, if the same message with same group id is coming within an interval of 5 mins to the queue, the message does not get stored.

So, message deduplication values help in differentiating between the messages. If content-based deduplication is enabled, it will ignore if no value present.

Code Sample

In this code sample, we used the below configurations :

1. Java 8
2. Spring Boot 2.3.5
3. AWS Account (https://aws.amazon.com/sqs/)
4. Gradle

So, for this example, let us see the basic gradle dependencies required. You can either create a sample spring boot project from spring-initializer or setup manually.

So, “build.gradle” should look like this below

Now, let us have the configuration ready to communicate with AWS services.

In Spring Boot, it is pretty easy to setup a SQSClient. We have to define few property values.

So, our “application.yml” should look something like this -:

In the above, we define the IAM programmatic keys and the region to communicate with the SQS.

We have asked specifically here to pick the credentials from the yml file, so the instance-profile and use-default-aws-credentials-chain has been set to false.

After this, in our main application class, we add the EnableSQS annotation and exclude the ContextInstanceDataAutoConfiguration from autoloading.

Our code is now ready for communication with AWS SQS. We can now use the “AmazonSQSAsync” client for performing all the operations with our AWS SQS.

Let us go ahead and create the queues for further operations.

So, If we notice above there are 2 additional attributes between creating a FIFO queue and Standard Queue.

In the FIFO Queue, we have enabled the ContentBasedDeduplication. We will see how it behaves later.

When a queue is created, the CreateQueueResult holds the queue url through which we can communicate to send or receive messages.

Now we can publish and receive messages with the queues we just created. Spring Boot provides QueueMessagingTemplate for publishing and consuming messages from SQS. In this blog, we will use that for our purpose.

We have to define the QueueMessagingTemplate bean for using it.

Now, above code sends messages to a FIFO queue and one to the Standard Queue. We notice that for sending message to the FIFO queue we need few headers to avoid sending failures.

We can also send message attributes in the headers as key-value pairs along with the message.

Now, we will consume the messages sent to the queue and it will be an active listener method. We will use the SQSListener annotation provided by Spring Boot on our listener method.

Here, we have added listeners each for FIFO queue and Standard Queue. As soon as a message is available in the Queues, it will be consumed by these listeners.

Note, there is a deletion policy mentioned which has been set to “Never”. It means once, the acknowledgement is manually done then only the message will be removed else it will return back to the queue after the visibility timeout happens for the message.

We have different set of Deletion policies which we can use for separate use-cases.

Finally, let us see how we can delete the queues.

Advantages of using AWS SQS as Messaging Queue

1. All Message Communication happens through HTTP
2. Security, Availability and Customisation becomes easier
3. DLQs can be configured for faulty message processing
4. AWS Lambda functions can be integrated with SQS
5. Responsibility of managing queue service is handed to cloud and focus can be pertained for communication
6. Faster message processing and option of choosing between ordered and non-ordered queues

Conclusion

We noticed how we can use AWS SQS as a messaging queue and how to use it using spring boot. It is pretty easy to configure and spring boot provides out-of-box features to support the same.

The full code is available here.