Potential deadlock in Subscriber #1855
Description
@davidtorres @garrettjonesgoogle
A lot of our code uses callback without specifying an executor. PollingSubscriberConnection is an example. IIUC, if we don't specify one, the library uses "directExecutor". Is this just the same executor we create the channel with? I think that it is.
If I'm right, this is dangerous. If we receive a bunch of messages, pullers will block on FlowController, waiting for messages to get processed. Messages can't get processed because all the threads in the pool are blocked on FlowController, resulting in a deadlock. I don't think this can happen in default config since we have 5 threads for each puller, but if user provides their own executor, all bets are off.
My current idea: Only use the executor for RPCs. Additionally ask for numWorkerThreads and create a new executor for processing messages. Assuming that the user-provided MessageReceiver cannot directly cause more tasks to be registered to the pullers, we should be able to avoid deadlocks.
What do you think?