Pre-create the shutdown Thread #86
Conversation
…allocated, in order to prevent an OOM when allocating new native thread
…nFactory) in order to have only 1 of it per ConnectionFactory, managing the shutdown of potentially multiple ChannelManagers. The ScheduledThreadPoolExecutor ensures a Thread is ready WAITING for a Task to be submitted.
|
@YannRobert thank you for your time. I feel the real solution is to bump Tomcat and OS limits so that your JVM can allocate new threads or use fewer connections (and more channels). One (temporary) shutdown thread per connection doesn't sound too bad. The suggested implementation complicates things quite a bit for a pretty minor gain. I'd first explore the option with using the higher limit/fewer connections and only modify the client if that is absolutely impossible. |
|
On some environment, like on public clouds providers, you cannot really fine tune OS limits and application resources. In Cloud Foundry Public Clouds for instance, you will get the following limits for a 1 GB memory application. When this is sufficient for the application to run and handle normal workload, you don't want to go to an upper plan (to get more resources) just so that the application can shutdown without an Error. As for the number of connections, it is already really low, and cannot reasonably be lowered more without affecting throughput. |
|
@YannRobert so you run on |
|
@michaelklishin I run on both |
|
Got it. The beauty of portability ;) |
* optimize maven dependencies format (rabbitmq#86)
On some condition, we can get the following Error in the application logs
Investigations shows that the Thread being created is a Thread that will have to shutdown stuff in RabbitMQ Client.
And that the condition for this to occur, is the tomcat container being shutdown by Cloud Foundry.
It appears that the ThreadFactory cannot create new Native Threads while the container is being shutdown. At the same time, the application tries to shutdown allocated resources and running processes. So when RabbitMQ client tries to shutdown, it creates a new Thread that will do those shutdown tasks.
But as a new Native Thread cannot be created, the Threads fails to start, and the shutdown task is not executed.
As the condtion for this to occur is that the application and JVM is shutting down, this may sound like it's not a real issue. In fact, I am concerned by the
OutOfMemoryErrorappearing in the logs. Logs are monitored and trigger alerts, so you don't want to see this in your logs unless it's a real problem.As a proof of concept, I patched the RabbitMQ client to create the shutdown Thread as soon at the ConnectionFactory is created : ie a ThreadPool of size 1 is created with the ConnectionFactory, so that the native Thread is created and is ready to execute any task submited to it.
It turned out that this patch did fix the problem: there is no OOM Error at application shutdown anymore.
I don't know for sure if the fix is good enough for every cases.
Like, this patch makes a new Thread waiting for the whole life of the ConnectionFactory. As if you have multiple ConnectionFactories in your app, you have the same number of ThreadPools waiting for the shutdown task.
I guess a typical application would only have only 2 ConnectionFactories anyways (1 for consumers and 1 for producers) so this may not be a big overhead. And maybe we can rework the code to ensure we only use as much as 1 ThreadPool for every ConnectionFactories in an application ?
Hoping it's a good start anyways.
Please review and comment.