AWS SDK Memory Leak
I have an application on which AWS SQS is used as a messaging system. Producer pushes the message to queue and consumer poll for the messages.
Problem
While analyzing heap memory we notice that there is a continuous increase in heap memory size.
After analyzing heap memory from heap dump what I found out is software.amazon.AWS-SDK.http.apache.internal.conn.IdleConnectionReaper$ReaperTask this thread is consuming about 79% of heap memory.
From Eclipse MAT “org.apache.http.impl.conn.PoolingHttpClientConnectionManager”, created by software.amazon.AWS-SDK.http.apache.internal.conn.IdleConnectionReaper$ReaperTask occupy 258,715,456 (79.96%) bytes. These instances are referenced from one instance of “java.util.concurrent.ConcurrentHashMap$Node[]”, loaded by “<system class loader>”, which occupies 562,608 (0.17%) bytes.
Solution
IdleConnectionReaper is a daemon thread that periodically checks connection pools for idle connections. This class helps to close the connection that is sitting in the HTTP connection pool for a long time. Since each client has its own connection pool resource. we should either shut down the client once a task is completed or we can create a single instance client and use them. If we create a new client for each request it will eventually end up creating multiple connection pools, which if not closes properly, will create a memory leak.
Problematic code:
private void consumeMessage(String queueName) {
while (true) {
SqsClient client = SqsClient.create();ReceiveMessageRequest receiveMessageRequest =
ReceiveMessageRequest
.builder()
.queueUrl(queueUrl)
.maxNumberOfMessages(1)
.waitTimeSeconds(20)
.build();List<Message> messages = client.receiveMessage(receiveMessageRequest).messages();//handle messages
}
}
The problem with this code is, it is creating an Sqs client for each request and not properly closing them which results in the creation of multiple connection pools.
Depending on the application what we can do is either create a single instance SQS client or close the client after each request. The solution I go with is creating a single instance client since this method is thread-safe as well as I can eliminate the cost associated with creating a client for each request.
New Code
public class PollingService {
private static SqsClient client;
private void consume(String queueName) {
while (true) {
if (client == null){
client = SqsClient.create();
}
ReceiveMessageRequest request = ReceiveMessageRequest.builder()
.queueUrl(queueUrl)
.maxNumberOfMessages(1)
.waitTimeSeconds(20)
.build();
List<Message> messages = client.receiveMessage(request).messages();
//handle messages
}
}
}Tools used to analyze heap memory: VisualVm and Eclipse MAT