SSL Authentication Error connecting to Amazon Aurora #576

comagnan · 2018-10-29T21:23:00Z

This happens randomly on the first call that opens a connection to the database with connection.Open(). The inner exception states "The message received was unexpected or badly formatted".

Here is the full stack trace.

MySql.Data.MySqlClient.MySqlException: SSL Authentication Error
 ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception.
 ---> System.ComponentModel.Win32Exception: The message received was unexpected or badly formatted
   --- End of inner exception stack trace ---
   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, Exception exception)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
   at MySqlConnector.Core.ServerSession.<InitSslAsync>d__74.MoveNext()
   --- End of inner exception stack trace ---
   at MySqlConnector.Core.ServerSession.<InitSslAsync>d__74.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MySqlConnector.Core.ServerSession.<ConnectAsync>d__58.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MySqlConnector.Core.ConnectionPool.<GetSessionAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at MySqlConnector.Core.ConnectionPool.<GetSessionAsync>d__10.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at MySql.Data.MySqlClient.MySqlConnection.<CreateSessionAsync>d__91.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at MySql.Data.MySqlClient.MySqlConnection.<OpenAsync>d__25.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at MySql.Data.MySqlClient.MySqlConnection.Open()

Over the hundreds of connections opened automatically during testing of MySqlConnector over the weekend, connections have failed to open roughly 90 times. In the majority of cases the library works as expected. When it fails, it's always on the first attempt at opening a connection since starting the process (then the process exits).

========
Some information about the setup:

MySqlConnector version: 0.46.0
Windows version: 10.0.14393.2430
.net version: 4.6.1
mySQL version: 5.6.10 (Aurora)
Code is running inside of Docker containers (Enterprise Edition 18.03.1-ee-3)

The text was updated successfully, but these errors were encountered:

bgrainger · 2018-10-30T03:35:18Z

Windows version: 10.0.14393.2430

Are you running on Windows 10 client or Windows Server?

Code is running inside of Docker containers (Enterprise Edition 18.03.1-ee-3)

Are these Windows Containers?

Would it be possible to get a packet capture of one of the failed connections?

bgrainger · 2018-10-30T03:43:18Z

This error has been previously reported with Aurora: #428

Also reported on Stack Overflow (also with Aurora); in this case the poster was using Connector/NET, but the System.Net.Security.SslState is extremely similar: https://stackoverflow.com/questions/46518563/net-mysql-error-a-call-to-sspi-failed-message-received-was-unexpected

The common factor here seems to be Amazon Aurora. The SO poster adds:

We do not receive this error in any other of our environments, which we are not running Aurora and simply just running MySQL 5.7.17 on Amazon RDS.

The evidence suggests a problem with Amazon Aurora, but I'm not sure how to “prove” that (or how to file a bug report for Amazon).

comagnan · 2018-10-30T14:30:18Z

The OS is Windows Server 2016, and these are Windows Containers. I'll see if I can get packet captures.

jefraim · 2019-02-01T05:54:13Z

@comagnan , were you able to resolve this issue?

Something similar also happens to our platform at random:

System.Data.Entity.Core.EntityException: The underlying provider failed on Open.
 ---> System.Security.Authentication.AuthenticationException: A call to SSPI failed, see inner exception.
 ---> System.ComponentModel.Win32Exception: The message received was unexpected or badly formatted
   --- End of inner exception stack trace ---
   at MySql.Data.Common.Ssl.StartSSL(Stream& baseStream, Encoding encoding, String connectionString)
   at MySql.Data.MySqlClient.NativeDriver.Open()
   at MySql.Data.MySqlClient.Driver.Open()
   at MySql.Data.MySqlClient.Driver.Create(MySqlConnectionStringBuilder settings)
   at MySql.Data.MySqlClient.MySqlPool.CreateNewPooledConnection()
   at MySql.Data.MySqlClient.MySqlPool.GetPooledConnection()
   at MySql.Data.MySqlClient.MySqlPool.TryToGetDriver()
   at MySql.Data.MySqlClient.MySqlPool.GetConnection()
   at MySql.Data.MySqlClient.MySqlConnection.Open()
   at System.Data.Common.DbConnection.OpenAsync(CancellationToken cancellationToken)
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Data.Entity.Core.EntityClient.EntityConnection.<OpenAsync>d__8.MoveNext()
   --- End of inner exception stack trace ---
   at System.Data.Entity.Core.EntityClient.EntityConnection.<OpenAsync>d__8.MoveNext()

bgrainger · 2019-02-01T06:38:49Z

8000

@jefraim Are you also using Amazon Aurora?

jefraim · 2019-02-01T08:56:22Z

@jefraim Are you also using Amazon Aurora?

We're using Amazon RDS MySQL,

bgrainger · 2019-02-01T14:37:38Z

@jefraim I just noticed that (from your call stack) you appear to be using Oracle's Connector/NET. This repo is for MySqlConnector, an OSS replacement for that library. To report a problem with that library, go to https://bugs.mysql.com/.

You could try switching to MySqlConnector https://www.nuget.org/packages/MySqlConnector/ if you like, but it's unlikely to help with this particular problem. It does fix a lot of other bugs, however: https://mysql-net.github.io/MySqlConnector/tutorials/migrating-from-connector-net/#fixed-bugs

mikebollandajw · 2019-02-07T11:13:56Z

We are also experiencing this with Amazon Aurora and Oracle Connector/Net

I will go to the relevent bug location to report this

https://bugs.mysql.com/bug.php?id=94240

MikeAlhayek · 2019-05-21T00:21:09Z

I am having the same problem. Here is my LINQ statement

Context.Integrations.Where(x => x.IsActive == isActive.Value).Include(x => x.Mappings)
                                                        .First();

I randomly get the following error

Error Message: Authentication failed, see inner exception.

Inner Error Message: The message or signature supplied for verification has been altered

Here is the stack trace

Stack Trace:    at MySql.Data.Common.Ssl.StartSSL(Stream& baseStream, Encoding encoding, String connectionString)
   at MySql.Data.MySqlClient.NativeDriver.Open()
   at MySql.Data.MySqlClient.Driver.Open()
   at MySql.Data.MySqlClient.Driver.Create(MySqlConnectionStringBuilder settings)
   at MySql.Data.MySqlClient.MySqlPool.CreateNewPooledConnection()
   at MySql.Data.MySqlClient.MySqlPool.GetPooledConnection()
   at MySql.Data.MySqlClient.MySqlPool.TryToGetDriver()
   at MySql.Data.MySqlClient.MySqlPool.GetConnection()
   at MySql.Data.MySqlClient.MySqlConnection.Open()
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.OpenDbConnection(Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Storage.RelationalConnection.Open(Boolean errorsExpected)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.BufferlessMoveNext(DbContext _, Boolean buffer)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryingEnumerable`1.Enumerator.MoveNext()
   at System.Linq.Enumerable.SelectEnumerableIterator`2.MoveNext()
   at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ResultEnumerable`1.GetEnumerator()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider._TrackEntities[TOut,TIn](IEnumerable`1 results, QueryContext queryContext, IList`1 entityTrackingInfos, IList`1 entityAccessors)+MoveNext()
   at Microsoft.EntityFrameworkCore.Query.Internal.LinqOperatorProvider.ExceptionInterceptor`1.EnumeratorExceptionInterceptor.MoveNext()
   at System.Linq.Enumerable.TryGetFirst[TSource](IEnumerable`1 source, Boolean& found)
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source)
   at Microsoft.EntityFrameworkCore.Query.Internal.QueryCompiler.<>c__DisplayClass15_1`1.<CompileQueryCore>b__0(QueryContext qc)
   at System.Linq.Queryable.First[TSource](IQueryable`1 source)

bgrainger · 2019-05-21T02:32:47Z

@CrestApps You are using Connector/NET, created by Oracle. Their bug reporting site is https://bugs.mysql.com/.

Are you also using Amazon Aurora as the MySQL server? That appears to be the common factor in these bug reports.

comagnan · 2019-05-21T12:52:58Z

While I am unfortunately not in a position to give packet captures after all, I wanted to give some more information I found while comparing Connector/NET and MySqlConnector.

SSL errors can be thrown by both connectors, as we can see above. The cause for the exception does appear to be tightly related to Aurora, as I was not able to reproduce it with a MySql docker container.
There is however a difference in behavior between the two connectors (which is not necessarily a bug). Connector/NET does not wrap the AuthenticationException in a MySqlException, while MySqlConnector does.

These errors are temporary, so I ended up simply adding a retry on MySqlExceptions caused by AuthenticationException.

mikebollandajw · 2019-05-28T07:38:56Z

Does anyone have a link to the aurora/oracle ticket related to this?

donaddon · 2019-11-15T16:26:11Z

We are seeing the same issue connecting on first start of a new service process to an Amazon Aurora MySQL DB. Anyone have the link to the Oracle bug?

bgrainger · 2019-11-15T18:05:56Z

See above: #576 (comment)

https://bugs.mysql.com/bug.php?id=94240

kaplandani · 2020-05-02T16:52:13Z

it happens also on mysql running on windows server (not related to amazon)

AbrahamWilton · 2021-07-23T06:08:12Z

it happens also on mysql running on windows server (not related to amazon)

In windows i have the same error, but resetting iis server . that's resolve my problem

charlesgardiner · 2021-07-26T17:36:13Z

I am experiencing this issue with AWS Aurora. Since this issue is 3 years old, I imagine a fix is not coming.

Has anyone found a work around when running against AWS Aurora?

bgrainger · 2021-07-26T18:46:11Z

It's a good idea to assume that the network is unreliable and that MySqlConnection.Open may fail for various reasons. So putting it in a retry loop (Polly is a common way) will handle this and other transient failures. That's the best "workaround" I can suggest right now.

charlesgardiner · 2021-07-27T18:02:07Z

@bgrainger Thanks, I appreciate your quick feedback.

bgrainger · 2022-12-23T18:54:50Z

All indications are that this is a problem on Aurora's end. There's not much MySqlConnector can do to work around it.

Recommendation is that clients retry opening a connection several times (e.g., with Polly) to recover from intermittent connection errors like this.

bgrainger added the waiting for answer Needs more information from the bug reporter label Nov 18, 2018

bgrainger mentioned this issue Mar 19, 2019

SSL Authentication Error: The message or signature supplied for verification has been altered #621

Closed

mguinness mentioned this issue Apr 4, 2019

SSL Authentication Error PomeloFoundation/Pomelo.EntityFrameworkCore.MySql#771

Closed

bgrainger mentioned this issue May 17, 2019

SSL Authentication Error - A call to SSPI failed, see inner exception. #428

Closed

bgrainger changed the title ~~Random SSL Authentication Error - The message received was unexpected or badly formatted~~ SSL Authentication Error connecting to Amazon Aurora May 28, 2019

mguinness mentioned this issue Apr 3, 2020

Command Timeout expired PomeloFoundation/Pomelo.EntityFrameworkCore.MySql#749

Closed

bgrainger removed the waiting for answer Needs more information from the bug reporter label Dec 23, 2022

bgrainger closed this as not planned Won't fix, can't repro, duplicate, stale Dec 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SSL Authentication Error connecting to Amazon Aurora #576

SSL Authentication Error connecting to Amazon Aurora #576

SSL Authentication Error connecting to Amazon Aurora #576

SSL Authentication Error connecting to Amazon Aurora #576

Comments