8000 Datastore emulator memory Leak · Issue #582 · googleapis/python-datastore · GitHub
[go: up one dir, main page]
Skip to content

Datastore emulator memory Leak #582

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
guibeira-evite opened this issue Jan 2, 2025 · 0 comments
Open

Datastore emulator memory Leak #582

guibeira-evite opened this issue Jan 2, 2025 · 0 comments
Assignees
@daniel-sanche
Labels
api: datastore Issues related to the googleapis/python-datastore API.

Comments

@guibeira-evite
Copy link

Environment details

  • OS type and version: macOS Sonoma 14.4.1 arm64
  • Python version: 3.12
  • google-cloud-datastore 2.20.1

After some time of running the datastore emulator, I see a gigantic memory usage and the emulator stops working with the following error in the application logs:

my-app  |   File "/usr/local/lib/python3.12/site-packages/google/cloud/datastore/batch.py", line 389, in __exit__
my-app  |     self.commit()
my-app  |   File "/usr/local/lib/python3.12/site-packages/google/cloud/datastore/transaction.py", line 339, in commit
my-app  |     super(Transaction, self).commit(**kwargs)
my-app  |   File "/usr/local/lib/python3.12/site-packages/google/cloud/datastore/batch.py", line 359, in commit
my-app  |     self._commit(retry=retry, timeout=timeout)
my-app  |   File "/usr/local/lib/python3.12/site-packages/google/cloud/datastore/batch.py", line 321, in _commit
my-app  |     commit_response_pb = self._client._datastore_api.commit(
my-app  |                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
my-app  |   File "/usr/local/lib/python3.12/site-packages/google/cloud/datastore_v1/services/datastore/client.py", line 1226, in commit
my-app  |     response = rpc(
my-app  |                ^^^^
my-app  |   File "/usr/local/lib/python3.12/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
my-app  |     return wrapped_func(*args, **kwargs)
my-app  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
my-app  |   File "/usr/local/lib/python3.12/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
my-app  |     return func(*args, **kwargs)
my-app  |            ^^^^^^^^^^^^^^^^^^^^^
my-app  |   File "/usr/local/lib/python3.12/site-packages/google/api_core/grpc_helpers.py", line 78, in error_remapped_callable
my-app  |     raise exceptions.from_grpc_error(exc) from exc
my-app  | google.api_core.exceptions.Unknown: None

After some research, I found that the emulator is using a lot of memory, and I think that this is the reason for the error.

Here is the docker total memory usage:

NAME                                  CPU %     MEM USAGE / LIMIT     MEM %     NET I/O         BLOCK I/O        PIDS
local-v2-datastore-emulator-1   92.70%    12.44GiB / 20.24GiB   61.48%    127MB / 8.2MB   84.2MB / 954MB   87

Datastore emulator logs

datastore-emulator-1  | [datastore] INFO: Detected HTTP/2 connection.
datastore-emulator-1  | [datastore] Jan 02, 2025 8:58:09 PM io.netty.util.ResourceLeakDetector reportTracedLeak
datastore-emulator-1  | [datastore] SEVERE: LEAK: ByteBuf.release() was not called before it's garbage-collected. See https://netty.io/wiki/reference-counted-objects.html for more information.
datastore-emulator-1  | [datastore] Recent access records:
datastore-emulator-1  | [datastore] Created at:
datastore-emulator-1  | [datastore]     io.netty.buffer.AbstractByteBufAllocator.compositeDirectBuffer(AbstractByteBufAllocator.java:224)
datastore-emulator-1  | [datastore]     io.netty.buffer.AbstractByteBufAllocator.compositeBuffer(AbstractByteBufAllocator.java:202)
datastore-emulator-1  | [datastore]     io.netty.handler.codec.MessageAggregator.decode(MessageAggregator.java:269)
datastore-emulator-1  | [datastore]     io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
datastore-emulator-1  | [datastore]     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
datastore-emulator-1  | [datastore]     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
datastore-emulator-1  | [datastore]     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
datastore-emulator-1  | [datastore]     io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
datastore-emulator-1  | [datastore]     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
datastore-emulator-1  | [datastore]     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
datastore-emulator-1  | [datastore]     io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
datastore-emulator-1  | [datastore]     io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
datastore-emulator-1  | [datastore]     io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
datastore-emulator-1  | [datastore]     io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
datastore-emulator-1  | [datastore]     io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:425)
datastore-emulator-1  | [datastore]     io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
datastore-emulator-1  | [datastore]     io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
datastore-emulator-1  | [datastore]     io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
datastore-emulator-1  | [datastore]     io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
datastore-emulator-1  | [datastore]     io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
datastore-emulator-1  | [datastore]     io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
datastore-emulator-1  | [datastore]     java.base/java.lang.Thread.run(Thread.java:840)
datastore-emulator-1  | [datastore] Jan 02, 2025 8:58:12 PM io.gapi.emulators.netty.HttpVersionRoutingHandler channelRead
datastore-emulator-1  | [datastore] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "grpc-default-executor-12"
datastore-emulator-1  | [datastore]
datastore-emulator-1  | [datastore] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "LocalDatastoreService-4"
datastore-emulator-1  | [datastore] Exception in thread "LocalDatastoreService-9" Exception in thread "nioEventLoopGroup-3-18" Exception in thread "LocalDatastoreService-1" java.lang.OutOfMemoryError: Java heap space
datastore-emulator-1  | [datastore] Exception in thread "LocalDatastoreService-2" java.lang.OutOfMemoryError: Java heap space
datastore-emulator-1  | [datastore] java.lang.OutOfMemoryError: Java heap space
datastore-emulator-1  | [datastore] java.lang.OutOfMemoryError: Java heap space
datastore-emulator-1  | [datastore] Exception in thread "LocalDatastoreService-6" java.lang.OutOfMemoryError: Java heap space
datastore-emulator-1  | [datastore] Jan 02, 2025 9:06:14 PM io.grpc.Context$CancellableContext$1CancelOnExpiration run
datastore-emulator-1  | [datastore] SEVERE: Cancel threw an exception, which should not happen
datastore-emulator-1  | [datastore] java.lang.OutOfMemoryError: Java heap space
datastore-emulator-1  | [datastore]
datastore-emulator-1  | [datastore] Exception in thread "LocalDatastoreService-0" java.lang.OutOfMemoryError: Java heap space
datastore-emulator-1  | [datastore] Exception in thread "grpc-default-executor-13" java.lang.OutOfMemoryError: Java heap space
datastore-emulator-1  | [datastore] Exception in thread "nioEventLoopGroup-3-2" java.lang.OutOfMemoryError: Java heap space
datastore-emulator-1  | [datastore] Jan 02, 2025 9:06:14 PM io.netty.channel.DefaultChannelPipeline onUnhandledInboundException
datastore-emulator-1  | [datastore] WARNING: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
datastore-emulator-1  | [datastore] java.lang.OutOfMemoryError: Java heap space
datastore-emulator-1  | [datastore]
datastore-emulator-1  | [datastore] Jan 02, 2025 9:06:14 PM io.grpc.netty.NettyServerTransport notifyTerminated
datastore-emulator-1  | [datastore] INFO: Transport failed
datastore-emulator-1  | [datastore] io.netty.handler.codec.http2.Http2Exception: Frame length: 6648621 exceeds maximum: 16384
datastore-emulator-1  | [datastore]     at io.netty.handler.codec.http2.Http2Exception.connectionError(Http2Exception.java:109)
datastore-emulator-1  | [datastore]     at io.netty.handler.codec.http2.DefaultHttp2FrameReader.processHeaderState(DefaultHttp2FrameReader.java:186)
datastore-emulator-1  | [datastore]     at io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:147)
datastore-emulator-1  | [datastore]     at io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
datastore-emulator-1  | [datastore]     at io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173)
datastore-emulator-1  | [datastore]     at io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393)
datastore-emulator-1  | [datastore]     at io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453)
datastore-emulator-1  | [datastore]     at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
datastore-emulator-1  | [datastore]     at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
datastore-emulator-1  | [datastore]     at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
datastore-emulator-1  | [datastore]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
datastore-emulator-1  | [datastore]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
datastore-emulator-1  | [datastore]     at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
datastore-emulator-1  | [datastore]     at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
datastore-emulator-1  | [datastore]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
datastore-emulator-1  | [datastore]     at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
datastore-emulator-1  | [datastore]     at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
datastore-emulator-1  | [datastore]     at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
datastore-emulator-1  | [datastore]     at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
datastore-emulator-1  | [datastore]     at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
datastore-emulator-1  | [datastore]     at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
datastore-emulator-1  | [datastore]     at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
datastore-emulator-1  | [datastore]     at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
datastore-emulator-1  | [datastore]     at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
datastore-emulator-1  | [datastore]     at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
datastore-emulator-1  | [datastore]     at java.base/java.lang.Thread.run(Thread.java:840)
datastore-emulator-1  | [datastore]
datastore-emulator-1  | [datastore] Exception in thread "grpc-default-executor-20" java.lang.OutOfMemoryError: Java heap space
datastore-emulator-1  | [datastore] Exception in thread "grpc-default-executor-19" java.lang.OutOfMemoryError: Java heap space

After some tests i increase the heap memory using this env:

  • JAVA_TOOL_OPTIONS="-Xms4G -Xmx1 7900 2G"

But even increasing the heap memory to 22gb, it's not enough to run the emulator without the error.
@product-auto-label product-auto-label bot added the api: datastore Issues related to the googleapis/python-datastore API. label Jan 2, 2025
@igorbernstein2 igorbernstein2 removed their assignment Apr 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@daniel-sanche daniel-sanche

Labels
api: datastore Issues related to the googleapis/python-datastore API.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@daniel-sanche @igorbernstein2 @guibeira-evite
0