Unexpected AMQP close-reason, initiated by Peer, code=504, text='CHANNEL_ERROR - expected 'channel.open' #1835

XCiteForever · 2025-05-16T12:47:45Z

Describe the bug

Endless sending of messages to a non-existent exchange when PublisherConfirmationsEnabled = false or PublisherConfirmationsEnabled = true and PublisherConfirmationTrackingEnabled = false.
Attached is a sample code that causes the error. Is this normal behavior or not?
And why?

At
PublisherConfirmationsEnabled = true
PublisherConfirmationTrackingEnabled = true
It works correctly

RabbitMQ.Client.Exceptions.OperationInterruptedException: The AMQP operation was interrupted: AMQP close-reason, initiated by Peer, code=504, text='CHANNEL_ERROR - expected 'channel.open'', classId=60, methodId=40

Reproduction steps

Run Code
See log

Expected behavior

Works stably

Additional context

    static async Task Main(String[] args)
    {
        var factory = new ConnectionFactory
        {
            HostName = "localhost",
            Port = 5672,
            UserName = "guest",
            Password = "guest"
        };

        var connection = await factory.CreateConnectionAsync();
        connection.ConnectionShutdownAsync += Connection_ConnectionShutdownAsync;

        var options = new CreateChannelOptions(false, false);
        var json = "{ \"A\" : \"X\"}";
        var body = Encoding.UTF8.GetBytes(json);
        var bp = new BasicProperties
        {
            ContentType = MediaTypeNames.Application.Json,
            ContentEncoding = "UTF-8",
        };

        IChannel channel = await connection.CreateChannelAsync(options);
        channel.ChannelShutdownAsync += Channel_ChannelShutdownAsync;

        while (true)
        {
            try
            {
                await channel.BasicPublishAsync("NotExistsExchange", String.Empty, true, bp, body);
                Console.WriteLine("send");
            }
            catch (AlreadyClosedException)
            {
                Console.WriteLine("not send");
            }
            
            if (channel.IsClosed)
            {
                channel = await connection.CreateChannelAsync(options);  // The error occurs here
                Console.WriteLine("Recreate");
                channel.ChannelShutdownAsync += Channel_ChannelShutdownAsync;
            }
        }
    }

    private static async Task Connection_ConnectionShutdownAsync(object sender, RabbitMQ.Client.Events.ShutdownEventArgs @event)
    {
        Console.WriteLine("Connection: " + @event.ReplyCode);
    }

    private static async Task Channel_ChannelShutdownAsync(object sender, RabbitMQ.Client.Events.ShutdownEventArgs @event)
    {
        Console.WriteLine("Channel: " + @event.ReplyCode);
    }
}

The text was updated successfully, but these errors were encountered:

XCiteForever · 2025-05-16T13:01:02Z

Some log RabbitMQ

2025-05-16 15:50:38 2025-05-16 12:50:38.294140+00:00 [error] <0.102182.0> Channel error on connection <0.101966.0> (172.22.0.1:39782 -> 172.22.0.2:5672, vhost: '/', user: 'guest'), channel 1:
2025-05-16 15:50:38 2025-05-16 12:50:38.294140+00:00 [error] <0.102182.0> operation basic.publish caused a channel exception not_found: no exchange 'NotExistsExchange' in vhost '/'
2025-05-16 15:50:38 2025-05-16 12:50:38.343681+00:00 [error] <0.102189.0> Channel error on connection <0.101966.0> (172.22.0.1:39782 -> 172.22.0.2:5672, vhost: '/', user: 'guest'), channel 1:
2025-05-16 15:50:38 2025-05-16 12:50:38.343681+00:00 [error] <0.102189.0> operation basic.publish caused a channel exception not_found: no exchange 'NotExistsExchange' in vhost '/'
2025-05-16 15:50:38 2025-05-16 12:50:38.391125+00:00 [error] <0.101966.0> Error on AMQP connection <0.101966.0> (172.22.0.1:39782 -> 172.22.0.2:5672, vhost: '/', user: 'guest', state: running), channel 1:
2025-05-16 15:50:38 2025-05-16 12:50:38.391125+00:00 [error] <0.101966.0> operation basic.publish caused a connection exception channel_error: "expected 'channel.open'"
2025-05-16 15:50:38 2025-05-16 12:50:38.393535+00:00 [info] <0.101966.0> closing AMQP connection <0.101966.0> (172.22.0.1:39782 -> 172.22.0.2:5672, vhost: '/', user: 'guest')

lukebakken · 2025-05-16T15:13:08Z

Hello, thanks for this report and for code to reproduce the issue.

I created a complete project here: https://github.com/lukebakken/rabbitmq-dotnet-client-1835

Here is the problem:

The code publishes as fast as possible, and does not wait for confirmations from the server.
Since the code publishes to a non-existent exchange, with mandatory: true, every publish results in a channel exception.
When RabbitMQ closes the channel due to the exception, it must send channel.close back to the client application, which, remember, is publishing as fast as possible.
Sometimes, this channel.close message makes it back at just the right time to initiate channel closure in this library BEFORE the next publish.
However, if a basic.publish is sent after RabbitMQ closes the channel, but before the channel.close message makes it back to this client, you will see the exception reported.

I'm not sure if it's possible to "do better" in this scenario. It is always possible that, without using publisher confirmations, a client application could send basic.publish to a closed channel.

I'm going to remove the "bug" label for now while I investigate.

Gsantomaggio · 2025-05-27T12:33:18Z

Ok, I can reproduce the problem too. Will start looking.

Gsantomaggio · 2025-05-28T06:39:07Z

@lukebakken I don't think this is a client issue.
The client's asynchronous nature requires some external synchronisation in this case, which can be achieved through a publish confirm or a wait, as shown in the code below.

My opinion:
Attempting to reconnect a channel too quickly results in an application error. All our reconnection implementation provides a delay during the reconnection.

I would not change the client to deal with a non-real use case like this.

The code like this is enough to solve the problem.

using RabbitMQ.Client;
using RabbitMQ.Client.Events;
using RabbitMQ.Client.Exceptions;
using System.Net.Mime;
using System.Text;

var factory = new ConnectionFactory
{
    HostName = "localhost",
    Port = 5672,
    UserName = "guest",
    Password = "guest"
};

var connection = await factory.CreateConnectionAsync();
connection.ConnectionShutdownAsync += Connection_ConnectionShutdownAsync;

var options = new CreateChannelOptions(false, false);
var json = "{ \"A\" : \"X\"}";
var body = Encoding.UTF8.GetBytes(json);
var bp = new BasicProperties
{
    ContentType = MediaTypeNames.Application.Json,
    ContentEncoding = "UTF-8",
};

IChannel channel = await connection.CreateChannelAsync(options);
channel.ChannelShutdownAsync += Channel_ChannelShutdownAsync;

while (true)
{
    try
    {
        await channel.BasicPublishAsync("NotExistsExchange", String.Empty, true, bp, body);
        Console.WriteLine("after BasicPublishAsync");
    }
    catch (AlreadyClosedException)
    {
        Console.WriteLine("caught AlreadyClosedException");
    }

    if (channel.IsClosed)
    {
        int retry = 0;
        while (retry < 3)
        {
            Console.WriteLine("channel.IsClosed is true, retrying...");
            await Task.Delay(1000);
            Console.WriteLine("channel.IsClosed is true");
            try
            {
                if (!connection.IsOpen)
                {
                    Console.WriteLine("connection.IsOpen is false, retrying...");
                    connection.ConnectionShutdownAsync -= Connection_ConnectionShutdownAsync;
                    connection = await factory.CreateConnectionAsync();
                    connection.ConnectionShutdownAsync += Connection_ConnectionShutdownAsync;
                }

                channel.ChannelShutdownAsync -= Channel_ChannelShutdownAsync;
                channel = await connection.CreateChannelAsync(options); // The error occurs here
                channel.ChannelShutdownAsync += Channel_ChannelShutdownAsync;
                Console.WriteLine("channel recreated successfully");
                break;
            }
            catch (Exception e)
            {
                Console.WriteLine("Exception during channel recreation: " + e.Message);
            }
            
            retry++;
        }
    }
}

static Task Connection_ConnectionShutdownAsync(object sender, ShutdownEventArgs @event)
{
    Console.WriteLine("Connection shutdown: " + @event.ReplyCode);
    return Task.CompletedTask;
}

static Task Channel_ChannelShutdownAsync(object sender, ShutdownEventArgs @event)
{
    Console.WriteLine("Channel shutdown: " + @event.ReplyCode);
    return Task.CompletedTask;
}

michaelklishin · 2025-05-28T07:42:46Z

Attempting to reconnect a channel too quickly results in an application error. All our
reconnection implementation provides a delay during the reconnection.

(I'm mentioning this for the readers who are not core team members, not insulting Gabriele's intelligence ;)) All RabbitMQ clients maintained by our team perform connection recovery after a delay.

Are you suggesting a small delay between channel recovery? That's a reasonable idea but with a large number of channels — whether we recommend it or not, there is no shortage of such apps — can result in significant recovery delays.

Or is your recommendation against such a delay?

Gsantomaggio · 2025-05-28T09:01:21Z

Are you suggesting a small delay between channel recovery?

Yes. I suggest adding a delay (a back-off would be better), as we do for other clients. I am thinking of the stream clients experience.
We have a back-off pattern, ex:
Stream .NET Client
stream Golang client
Java stream client

That's my opinion, based on the experiences of other clients, but I am open to discussing

Gsantomaggio · 2025-06-04T14:46:56Z

I will close the issue. Feel free to reopen in case needed.

XCiteForever added the bug label May 16, 2025

lukebakken self-assigned this May 16, 2025

lukebakken added this to the 7.2.0 milestone May 16, 2025

lukebakken removed the bug label May 16, 2025

lukebakken assigned Gsantomaggio May 16, 2025

Gsantomaggio closed this as completed Jun 4, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Unexpected AMQP close-reason, initiated by Peer, code=504, text='CHANNEL_ERROR - expected 'channel.open' #1835

Unexpected AMQP close-reason, initiated by Peer, code=504, text='CHANNEL_ERROR - expected 'channel.open' #1835

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Unexpected AMQP close-reason, initiated by Peer, code=504, text='CHANNEL_ERROR - expected 'channel.open' #1835

Unexpected AMQP close-reason, initiated by Peer, code=504, text='CHANNEL_ERROR - expected 'channel.open' #1835

Comments

Uh oh!

Describe the bug

Reproduction steps

Expected behavior

Additional context

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!