10000 [BUG] another coredump for version 3.4.6-1 · Issue #9302 · arangodb/arangodb · GitHub
[go: up one dir, main page]
Skip to content

[BUG] another coredump for version 3.4.6-1 #9302

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
daoxian opened this issue Jun 20, 2019 · 28 comments
Closed

[BUG] another coredump for version 3.4.6-1 #9302

daoxian opened this issue Jun 20, 2019 · 28 comments
Assignees
@graetzer @goedderz
Labels
1 Bug 3 Core

Comments

@daoxian
Copy link
daoxian commented Jun 20, 2019

3.4.6-1 , CentOS-7.4, Single mode on metal machine.

Failed to read a valid object file image from memory.
Core was generated by `/usr/sbin/arangod -c /home/admin/arangodb/data_3.4.5_singlemode_partition/singl'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000001ef472c in malloc_mutex_lock (mutex=0x40e3740 <je_arenas_lock>, tsdn=0x0) at include/jemalloc/internal/mutex.h:205
205     include/jemalloc/internal/mutex.h: No such file or directory.
(gdb) bt
#0  0x0000000001ef472c in malloc_mutex_lock (mutex=0x40e3740 <je_arenas_lock>, tsdn=0x0) at include/jemalloc/internal/mutex.h:205
#1  je_arena_init (extent_hooks=<optimized out>, ind=0, tsdn=0x0) at src/jemalloc.c:360
#2  malloc_init_hard_a0_locked () at src/jemalloc.c:1270
Cannot access memory at address 0x7f3ca2d973a8
@daoxian
Copy link
Author
daoxian commented Jun 20, 2019

Here's yet another coredump:

Core was generated by `/usr/sbin/arangod -c /home/admin/arangodb/data_3.4.5_singlemode_partition/singl'.
Program terminated with signal 11, Segmentation fault.
bt#0  getCollectionIdLocal (name=..., this=0x7f3a0cc1f880) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:69
69      /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp: No such file or directory.
(gdb) bt
#0  getCollectionIdLocal (name=..., this=0x7f3a0cc1f880) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:69
#1  arangodb::CollectionNameResolver::getCollectionId (this=0x7f3a0cc1f880, name=...) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:159
#2  0x00000000009273c0 in arangodb::SingleCollectionTransaction::SingleCollectionTransaction (this=0x7f3a49a7c800, transactionContext=..., name=...,
    accessType=arangodb::AccessMode::WRITE) at /work/ArangoDB/arangod/Utils/SingleCollectionTransaction.cpp:58
#3  0x0000000000e728c9 in arangodb::pregel::GraphStore<arangodb::pregel::HITSValue, signed char>::_storeVertices (this=0x7f3a0e626620, globalShards=..., it=...)
    at /work/ArangoDB/arangod/Pregel/GraphStore.cpp:591
#4  0x0000000000e72d53 in arangodb::pregel::GraphStore<arangodb::pregel::HITSValue, signed char>::storeResults(arangodb::pregel::WorkerConfig*, std::function<void ()>)::{lambda(bool)#1}::operator()(bool) const (__closure=0x7f39658a59c0, cancelled=false) at /work/ArangoDB/arangod/Pregel/GraphStore.cpp:664
#5  0x00000000008bdc5c in operator() (__args#0=<optimized out>, this=0x7f3a4a59de48) at /usr/include/c++/6.4.0/functional:2127
#6  operator() (__closure=0x7f3a4a59de40) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:327
#7  asio_handler_invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/handler_invoke_hook.hpp:68
#8  invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()>, arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (context=...,
    function=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_invoke_helpers.hpp:37
#9  complete<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (this=<synthetic pointer>, handler=..., function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_work.hpp:81
#10 asio::detail::completion_handler<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> >::do_complete(void *, asio::detail::operation *, const asio::error_code &, std::size_t) (owner=<optimized out>, base=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/completion_handler.hpp:69
#11 0x00000000008c5e43 in complete (bytes_transferred=0, ec=..., owner=0x7f45c70cee00, this=0x7f395d3c1970)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/scheduler_operation.hpp:39
#12 do_run_one (ec=..., this_thread=..., lock=..., this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:400
#13 run_one (ec=..., this=0x7f45c70cee00) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:174
#14 run_one (this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/io_context.ipp:76
#15 arangodb::SchedulerThread::run (this=0x7f3b76004540) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:212
#16 0x0000000001204499 in runMe (this=0x7f3b76004540) at /work/ArangoDB/lib/Basics/Thread.cpp:347
#17 arangodb::Thread::startThread (arg=0x7f3b76004540) at /work/ArangoDB/lib/Basics/Thread.cpp:82
#18 0x00000000012c4559 in ThreadStarter (data=0x7f3b6ed34b30) at /work/ArangoDB/lib/Basics/threads-posix.cpp:71
#19 0x00000000027d1a08 in start (p=0x7f3a4a5a0ae0) at src/thread/pthread_create.c:150
#20 0x00000000027d2fe9 in __clone () at src/thread/x86_64/clone.s:21
Backtrace stopped: frame did not save the PC

@daoxian
Copy link
Author
daoxian commented Jun 20, 2019

And another:

#0  atomic_load_p (mo=atomic_memory_order_acquire, a=<optimized out>) at include/jemalloc/internal/atomic.h:55
#1  extent_arena_get (extent=0x0) at include/jemalloc/internal/extent_inlines.h:49
#2  je_large_dalloc (tsdn=0x7f5b6570af98, extent=0x0) at src/large.c:347
#3  0x00000000010b6ef3 in deallocate (this=0x7f5c6ed0c968, __p=<optimized out>) at /usr/include/c++/6.4.0/ext/new_allocator.h:110
#4  deallocate (__a=..., __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/6.4.0/bits/alloc_traits.h:462
#5  _M_deallocate (this=0x7f5c6ed0c968, __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/6.4.0/bits/stl_vector.h:178
#6  std::vector<arangodb::pregel::SenderMessage<double>, std::allocator<arangodb::pregel::SenderMessage<double> > >::_M_range_insert<__gnu_cxx::__normal_iterator<arangodb::pregel::SenderMessage<double> const*, std::vector<arangodb::pregel::SenderMessage<double>, std::allocator<arangodb::pregel::SenderMessage<double> > > > > (this=0x7f5c6ed0c968, __position=...,
    __first=..., __last=...) at /usr/include/c++/6.4.0/bits/vector.tcc:685
#7  0x00000000010b7289 in _M_insert_dispatch<__gnu_cxx::__normal_iterator<arangodb::pregel::SenderMessage<double> const*, std::vector<arangodb::pregel::SenderMessage<double>, std::allocator<arangodb::pregel::SenderMessage<double> > > > > (__last=..., __first=..., __pos=..., this=<optimized out>) at /usr/include/c++/6.4.0/bits/stl_vector.h:1375
#8  insert<__gnu_cxx::__normal_iterator<arangodb::pregel::SenderMessage<double> const*, std::vector<arangodb::pregel::SenderMessage<double>, std::allocator<arangodb::pregel::SenderMessage<double> > > > > (__last=..., __first=..., __position=..., this=<optimized out>) at /usr/include/c++/6.4.0/bits/stl_vector.h:1100
#9  arangodb::pregel::ArrayInCache<arangodb::pregel::SenderMessage<double> >::mergeCache (this=0x7f5c3e609d00, config=..., otherCache=0x7f5c3e643580)
    at /work/ArangoDB/arangod/Pregel/IncomingCache.cpp:151
#10 0x0000000000ea0eb7 in arangodb::pregel::Worker<arangodb::pregel::HITSValue, signed char, arangodb::pregel::SenderMessage<double> >::_processVertices (this=0x7f5cc5072b10,
    threadId=<optimized out>, vertexIterator=...) at /work/ArangoDB/arangod/Pregel/Worker.cpp:418
#11 0x0000000000eb1710 in arangodb::pregel::Worker<arangodb::pregel::HITSValue, signed char, arangodb::pregel::SenderMessage<double> >::_startProcessing()::{lambda(bool)#1}::operator()(bool) const (__closure=0x7f584c79c520) at /work/ArangoDB/arangod/Pregel/Worker.cpp:335
#12 0x00000000008bdc5c in operator() (__args#0=<optimized out>, this=0x7f5b65709e48) at /usr/include/c++/6.4.0/functional:2127
#13 operator() (__closure=0x7f5b65709e40) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:327
#14 asio_handler_invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/handler_invoke_hook.hpp:68
#15 invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()>, arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (context=...,
    function=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_invoke_helpers.hpp:37
#16 complete<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (this=<synthetic pointer>, handler=..., function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_work.hpp:81
#17 asio::detail::completion_handler<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> >::do_complete(void *, asio::detail::operation *, const asio::error_code &, std::size_t) (owner=<optimized out>, base=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/completion_handler.hpp:69
#18 0x00000000008c5e43 in complete (bytes_transferred=0, ec=..., owner=0x7f69b74e7300, this=0x7f5c7d1038b0)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/scheduler_operation.hpp:39
#19 do_run_one (ec=..., this_thread=..., lock=..., this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:400
#20 run_one (ec=..., this=0x7f69b74e7300) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:174
#21 run_one (this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/io_context.ipp:76
#22 arangodb::SchedulerThread::run (this=0x7f69820c1150) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:212
#23 0x0000000001204499 in runMe (this=0x7f69820c1150) at /work/ArangoDB/lib/Basics/Thread.cpp:347
#24 arangodb::Thread::startThread (arg=0x7f69820c1150) at /work/ArangoDB/lib/Basics/Thread.cpp:82
#25 0x00000000012c4559 in ThreadStarter (data=0x7f6981fb1090) at /work/ArangoDB/lib/Basics/threads-posix.cpp:71
#26 0x00000000027d1a08 in start (p=0x7f5b6570cae0) at src/thread/pthread_create.c:150
#27 0x00000000027d2fe9 in __clone () at src/thread/x86_64/clone.s:21
Backtrace stopped: frame did not save the PC

@daoxian
Copy link
Author
daoxian commented Jun 21, 2019 
Core was generated by `/usr/sbin/arangod -c /home/admin/arangodb/data_3.4.5_singlemode_partition/singl'.
Program terminated with signal 11, Segmentation fault.
#0  0x000000000110cb09 in readIntegerNonEmpty<unsigned long> (length=<optimized out>, start=0x0) at /work/ArangoDB/3rdParty/velocypack/include/velocypack/velocypack-common.h:225
225     /work/ArangoDB/3rdParty/velocypack/include/velocypack/velocypack-common.h: No such file or directory.
(gdb) bt
#0  0x000000000110cb09 in readIntegerNonEmpty<unsigned long> (length=<optimized out>, start=0x0) at /work/ArangoDB/3rdParty/velocypack/include/velocypack/velocypack-common.h:225
#1  byteSize (this=<synthetic pointer>) at /work/ArangoDB/3rdParty/velocypack/include/velocypack/Slice.h:777
#2  addInternal<arangodb::velocypack::Slice> (sub=..., attrLength=<optimized out>, attrName=0x7ec1c807df10 "H,\016\003", this=0x7f412e5320e0)
    at /work/ArangoDB/3rdParty/velocypack/include/velocypack/Builder.h:681
#3  addInternal<arangodb::velocypack::Slice> (sub=..., attrName=..., this=0x7f412e5320e0) at /work/ArangoDB/3rdParty/velocypack/include/velocypack/Builder.h:645
#4  add (sub=..., attrName=..., this=0x7f412e5320e0) at /work/ArangoDB/3rdParty/velocypack/include/velocypack/Builder.h:377
#5  (anonymous namespace)::handleProjections (node=<optimized out>, trxPtr=<optimized out>, slice=..., b=..., useRawDocumentPointers=<optimized out>)
    at /work/ArangoDB/arangod/Aql/DocumentProducingBlock.cpp:59
#6  0x000000000106710e in operator+= (__n=3910498560, this=<synthetic pointer>) at /usr/include/c++/6.4.0/bits/stl_deque.h:218
#7  operator+ (__n=3910498560, this=0x7eb24c484310) at /usr/include/c++/6.4.0/bits/stl_deque.h:228
#8  operator[] (__n=3910498560, this=0x7eb24c484310) at /usr/include/c++/6.4.0/bits/stl_deque.h:244
#9  operator[] (__n=3910498560, this=0x7eb24c484300) at /usr/include/c++/6.4.0/bits/stl_deque.h:1367
#10 operator() (b=..., a=..., this=<synthetic pointer>) at /work/ArangoDB/arangod/Aql/SortBlock.cpp:45
#11 operator()<__gnu_cxx::__normal_iterator<std::pair<unsigned int, unsigned int>*, std::vector<std::pair<unsigned int, unsigned int> > >, const std::pair<unsigned int, unsigned int> >
    (__val=..., __it=..., this=<synthetic pointer>) at /usr/include/c++/6.4.0/bits/predefined_ops.h:147
#12 __lower_bound<__gnu_cxx::__normal_iterator<std::pair<unsigned int, unsigned int>*, std::vector<std::pair<unsigned int, unsigned int> > >, std::pair<unsigned int, unsigned int>, __gnu_cxx::__ops::_Iter_comp_val<(anonymous namespace)::OurLessThan> > (__comp=..., __val=..., __last=..., __first=...) at /usr/include/c++/6.4.0/bits/stl_algobase.h:959
#13 std::__merge_adaptive<__gnu_cxx::__normal_iterator<std::pair<unsigned int, unsigned int>*, std::vector<std::pair<unsigned 
8000
int, unsigned int> > >, long, std::pair<unsigned int, unsigned int>*, __gnu_cxx::__ops::_Iter_comp_iter<(anonymous namespace)::OurLessThan> > (__first=..., __middle=..., __last=..., __len1=139919698076146, __len2=139304249083264,
    __buffer=0x7eb24c483e60, __buffer_size=139324492732352, __comp=...) at /usr/include/c++/6.4.0/bits/stl_algo.h:2441
#14 0x000000000106ee05 in LoggerStream (this=0x2) at /work/ArangoDB/lib/Logger/LoggerStream.h:43
#15 arangodb::aql::TraversalBlock::shutdown (this=<optimized out>, errorCode=<optimized out>) at /work/ArangoDB/arangod/Aql/TraversalBlock.cpp:248
#16 0x00000000010e28bb in arangodb::SocketSslTcp::sslHandshake (this=0x7eb24c4848b0) at /work/ArangoDB/arangod/Scheduler/SocketSslTcp.cpp:79
#17 0x00000000010e3256 in system_category () at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/error_code.ipp:198
#18 get_system_category () at /work/ArangoDB/3rdParty/asio/1.12/include/asio/error.hpp:228
#19 error_wrapper<int> (ec=..., return_value=-792) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/socket_ops.ipp:85
#20 shutdown (ec=..., what=1, s=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/socket_ops.ipp:459
#21 shutdown (this=<optimized out>, impl=..., ec=..., what=asio::socket_base::shutdown_send) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/reactive_socket_service.hpp:210
#22 shutdown (ec=..., what=asio::socket_base::shutdown_send, this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/basic_socket.hpp:1622
#23 arangodb::SocketSslTcp::shutdownSend (this=<optimized out>, ec=...) at /work/ArangoDB/arangod/Scheduler/SocketSslTcp.h:94
#24 0x00000000010d3a86 in _M_capacity (this=<optimized out>, __capacity=<optimized out>) at /usr/include/c++/6.4.0/bits/basic_string.h:160
#25 _M_construct<char const*> (__end=0xfd643e0c63c3 <Address 0xfd643e0c63c3 out of bounds>, __beg=0x7eb1f1c41773 "", this=0x7eb24c484b90)
    at /usr/include/c++/6.4.0/bits/basic_string.tcc:220
#26 _M_construct_aux<char const*> (__end=0xfd643e0c63c3 <Address 0xfd643e0c63c3 out of bounds>, __beg=0x7eb1f1c41773 "", this=0x7eb24c484b90)
    at /usr/include/c++/6.4.0/bits/basic_string.h:196
#27 _M_construct<char const*> (__end=0xfd643e0c63c3 <Address 0xfd643e0c63c3 out of bounds>, __beg=0x7eb1f1c41773 "", this=0x7eb24c484b90)
    at /usr/include/c++/6.4.0/bits/basic_string.h:215
#28 basic_string (__a=..., __n=<optimized out>, __s=0x7eb1f1c41773 "", this=0x7eb24c484b90) at /usr/include/c++/6.4.0/bits/basic_string.h:447
#29 copyString (this=<synthetic pointer>) at /work/ArangoDB/3rdParty/velocypack/include/velocypack/Slice.h:684
#30 arangodb::RestCollectionHandler::<lambda(const std::shared_ptr<arangodb::LogicalCollection>&)>::operator()(const std::shared_ptr<arangodb::LogicalCollection> &) const (
    __closure=0x7eb24c484c50, coll=...) at /work/ArangoDB/arangod/RestHandler/RestCollectionHandler.cpp:423
#31 0x00000000010d3ca0 in ?? () at /usr/include/c++/6.4.0/bits/basic_string.h:180
#32 0x00007f41580eabe0 in ?? ()
#33 0x00000000010e2d62 in arangodb::SocketSslTcp::sslHandshake (this=0x2710) at /work/ArangoDB/arangod/Scheduler/SocketSslTcp.cpp:57
---Type <return> to continue, or q <return> to quit---
#34 0x00000000010d4f93 in do_use_service (owner=<optimized out>, factory=0x73a9c0 <asio::detail::service_registry::create<asio::detail::scheduler, asio::execution_context>(void*)>,
    key=<synthetic pointer>, this=0x7f49d4ffd640) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.ipp:147
#35 use_service<asio::detail::scheduler> (this=0x7f49d4ffd640) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:29
#36 use_service<asio::detail::scheduler> (e=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/execution_context.hpp:32
#37 epoll_reactor (ctx=..., this=0x7eb214044b20) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/epoll_reactor.ipp:39
#38 create<asio::detail::epoll_reactor, asio::io_context> (owner=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:86
#39 do_use_service (owner=<optimized out>, factory=<optimized out>, key=<synthetic pointer>, this=0x7f49df214000)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.ipp:131
#40 use_service<asio::detail::epoll_reactor> (owner=..., this=0x7f49df214000) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:38
#41 use_service<asio::detail::epoll_reactor> (ioc=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/io_context.hpp:38
#42 reactive_socket_service_base (io_context=..., this=0x7eb24c484c28) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/reactive_socket_service_base.ipp:33
#43 reactive_socket_service (io_context=..., this=0x7eb24c484c00) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/reactive_socket_service.hpp:78
#44 asio::detail::service_registry::create<asio::detail::reactive_socket_service<asio::ip::tcp>, asio::io_context> (owner=<optimized out>)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:86
#45 0x00000000010d5245 in unlock (this=0x7f46fb2fbb60) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/posix_mutex.hpp:57
#46 unlock (this=<synthetic pointer>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/scoped_lock.hpp:71
#47 do_use_service (owner=0x7f49e0a94e00, factory=0x73b060 <asio::detail::service_registry::create<asio::detail::epoll_reactor, asio::execution_context>(void*)>,
    key=<synthetic pointer>, this=0x7f49e0a94e00) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.ipp:130
#48 use_service<asio::detail::epoll_reactor> (this=0x7f49e0a94e00) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:29
#49 use_service<asio::detail::epoll_reactor> (e=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/execution_context.hpp:32
#50 init_task (this=0x31cfdb8 <ssl3_ciphers+3192>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:131
#51 init_task (this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/epoll_reactor.ipp:147
#52 reactive_socket_service_base (io_context=..., this=0x45c) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/reactive_socket_service_base.ipp:35
#53 reactive_socket_service (io_context=..., this=0x434) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/reactive_socket_service.hpp:78
#54 asio::detail::service_registry::create<asio::detail::reactive_socket_service<asio::ip::tcp>, asio::io_context> (owner=<optimized out>)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:86
#55 0x00000000010d543b in throw_error (location=0x2980711 "mutex", err=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/throw_error.hpp:41
#56 posix_mutex (this=0x0) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/posix_mutex.ipp:36
#57 conditionally_enabled_mutex (enabled=144, this=0x7eb24c484d80) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/conditionally_enabled_mutex.hpp:108
#58 epoll_reactor (ctx=..., this=0x7eb24c484d50) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/epoll_reactor.ipp:46
#59 create<asio::detail::epoll_reactor, asio::io_context> (owner=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:86
#60 do_use_service (owner=<optimized out>, factory=<optimized out>, key=<synthetic pointer>, this=0xffffffffffffffff)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.ipp:131
#61 use_service<asio::detail::epoll_reactor> (owner=..., this=0xffffffffffffffff) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:38
#62 use_service<asio::detail::epoll_reactor> (ioc=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/io_context.hpp:38
#63 reactive_socket_service_base (io_context=..., this=0x7eb24c484d68) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/reactive_socket_service_base.ipp:33
#64 reactive_socket_service (io_context=..., this=0x7eb24c484d40) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/reactive_socket_service.hpp:78
#65 asio::detail::service_registry::create<asio::detail::reactive_socket_service<asio::ip::tcp>, asio::io_context> (owner=<optimized out>)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/service_registry.hpp:86
#66 0x00000000008c83ad in unlock (this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/posix_static_mutex.hpp:49
#67 ~scoped_lock (this=<optimized out>, __in_chrg=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/scoped_lock.hpp:53
#68 add (ec=..., signal_number=15, impl=..., this=0x7f49df214000) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/signal_set_service.ipp:256
#69 add (signal_number=15, this=0x7eb24c484d90) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/signal_set.hpp:256
#70 arangodb::SchedulerFeature::buildControlCHandler (this=0x7eb24c484eb0) at /work/ArangoDB/arangod/Scheduler/SchedulerFeature.cpp:335
---Type <return> to continue, or q <return> to quit---
#71 0x00000000008c2e5b in operator|= (__i=9223372036854775808, this=0x7f4031c9d308) at /usr/include/c++/6.4.0/bits/atomic_base.h:336
#72 setStopping (this=0x7f4031c9d300) at /work/ArangoDB/arangod/Scheduler/Scheduler.h:112
#73 arangodb::rest::Scheduler::beginShutdown (this=0x7f4031c9d300) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:636
#74 0x00000000011f7709 in arangodb::basics::StringUtils::replace (sourceStr=..., fromStr=..., toStr=...) at /work/ArangoDB/lib/Basics/StringUtils.cpp:1144
#75 0x00000000012b76e9 in ~MutexLocker (this=<optimized out>, __in_chrg=<optimized out>) at /work/ArangoDB/lib/Basics/MutexLocker.h:103
#76 arangodb::ScriptLoader::defineScript (this=0x0, name=..., script=...) at /work/ArangoDB/lib/Utilities/ScriptLoader.cpp:84
#77 0x00000000027c5a9c in tan (x=<unavailable>) at src/math/tan.c:50
#78 0x00000000027c707d in getnameinfo (sa=0x7ec1c807df10, sl=503, node=0x2 <Address 0x2 out of bounds>, nodelen=1, serv=0x520b <Address 0x520b out of bounds>, servlen=3776104640,
    flags=0) at src/network/getnameinfo.c:138
#79 0x0000000000000000 in ?? ()

@daoxian
Copy link
Author
daoxian commented Jun 21, 2019

Could you plz analyze and fix these crashes ?

@mpoeter mpoeter mentioned this issue Jun 21, 2019
Closed
@goedderz
Copy link
Member

Hi @daoxian,

I've had a look at the last backtrace (#9302 (comment)), and it looks broken. Would it be possible for you to provide us with the coredump?

Best,

Tobias

@daoxian
Copy link
Author
daoxian commented Jun 21, 2019

Hi @daoxian,

I've had a look at the last backtrace (#9302 (comment)), and it looks broken. Would it be possible for you to provide us with the coredump?

Best,

Tobias

Thank you for your quick response. Do you mean the coredump files?

@goedderz
Copy link
Member

Yes, exactly.

@markuspf markuspf mentioned this issue Jun 21, 2019
Merged
9 tasks
@daoxian
Copy link
Author
daoxian commented Jun 21, 2019

Yes, exactly.

OK, wait for hours please, for the files are very large.

@daoxian
Copy link
Author
daoxian commented Jun 24, 2019 •
edited
Loading

Yes, exactly.

@goedderz I've uploaded relating coredump files:

https://test-bucket-duplicate.oss-cn-hangzhou.aliyuncs.com/core.222.tgz?OSSAccessKeyId=LTAI5u7ihxRjz5Yc&Expires=1563935585&Signature=GqX84VeOv0QkAG6DaqqQDuK5tT4%3D

https://test-bucket-duplicate.oss-cn-hangzhou.aliyuncs.com/core.49.tgz?OSSAccessKeyId=LTAI5u7ihxRjz5Yc&Expires=1563935615&Signature=9H7JThkwYyU8iLcR1q5%2FIvSU678%3D

https://test-bucket-duplicate.oss-cn-hangzhou.aliyuncs.com/core.158.tgz?OSSAccessKeyId=LTAI5u7ihxRjz5Yc&Expires=1563935627&Signature=91MUOi9pDS0Tlr270Fi6nosGKDM%3D

Thanks!

@goedderz
Copy link
Member

Hi @daoxian, thanks a lot for sharing. I am currently downloading the files. We will have a look and then come back to you.

@goedderz goedderz self-assigned this Jun 24, 2019
@goedderz
Copy link
Member

Hi @daoxian,
gdb says that two of the core files are truncated (namely core.11626 and core.1301). These are the sizes in bytes of the files I got - are these consistent with yours?

703720357888 core.11626
868124520448 core.1301
547165097984 core.66237
102406086656 core.70242

I'm also calculating SHA256 sums, but this is going to take a few hours.

@goedderz
Copy link
Member

Independent of that, the backtraces of all 4 look like they point to the same problem, which is the same as in #9301, as @mpoeter already pointed out. @graetzer will look into it.

@daoxian
Copy link
Author
daoxian commented Jun 26, 2019

Hi @daoxian,
gdb says that two of the core files are truncated (namely core.11626 and core.1301). These are the sizes in bytes of the files I got - are these consistent with yours?

703720357888 core.11626
868124520448 core.1301
547165097984 core.66237
102406086656 core.70242

I'm also calculating SHA256 sums, but this is going to take a few hours.

Yes they're the same size as those on my local machine.

@daoxian
Copy link
Author
daoxian commented Jun 26, 2019 •
edited
Loading

@goedderz Everyday I can see new coredump files. Two of them listed as follows:

Core was generated by `/usr/sbin/arangod -c /home/admin/arangodb/data_3.4.6_singlemode_partition/singl'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000000000b0a79e in arangodb::RocksDBIndex::blackListKey (this=this@entry=0x7f6647cd7710,
    data=0x7f52a5526077 "site_shikoku_np_co_jp__2Fen/6753de861d88c1e6f3c5835033162d1e48a9bf5155e651352e279c0f2K_Y3LDEQK-AnGversionG6224570Fvertexxfdf341d57f9cc7fcad17c30860df63a9ea7594c92b52da52c4b017ddDpathC/enKdomain_namePs"..., len=<optimized out>) at /work/ArangoDB/arangod/RocksDBEngine/RocksDBIndex.cpp:289
289     /work/ArangoDB/arangod/RocksDBEngine/RocksDBIndex.cpp: No such file or directory.
(gdb) bt
#0  0x0000000000b0a79e in arangodb::RocksDBIndex::blackListKey (this=this@entry=0x7f6647cd7710,
    data=0x7f52a5526077 "site_shikoku_np_co_jp__2Fen/6753de861d88c1e6f3c5835033162d1e48a9bf5155e651352e279c0f2K_Y3LDEQK-AnGversionG6224570Fvertexxfdf341d57f9cc7fcad17c30860df63a9ea7594c92b52da52c4b017ddDpathC/enKdomain_namePs"..., len=<optimized out>) at /work/ArangoDB/arangod/RocksDBEngine/RocksDBIndex.cpp:289
#1  0x0000000000fcf9e6 in blackListKey (ref=..., this=0x7f6647cd7710) at /work/ArangoDB/arangod/RocksDBEngine/RocksDBIndex.h:147
#2  arangodb::RocksDBEdgeIndex::insertInternal (this=0x7f6647cd7710, trx=0x7f5517bee260, mthd=0x7f54cab0d1e0, documentId=..., doc=..., mode=arangodb::Index::normal)
    at /work/ArangoDB/arangod/RocksDBEngine/RocksDBEdgeIndex.cpp:605
#3  0x0000000000fb3e21 in arangodb::RocksDBCollection::insertDocument (this=this@entry=0x7f6649b85800, trx=trx@entry=0x7f5517bee260, documentId=..., doc=..., options=...)
    at /work/ArangoDB/arangod/RocksDBEngine/RocksDBCollection.cpp:1296
#4  0x0000000000fb4060 in arangodb::RocksDBCollection::insert(arangodb::transaction::Methods*, arangodb::velocypack::Slice, arangodb::ManagedDocumentResult&, arangodb::OperationOptions&, unsigned long&, bool, unsigned long&, arangodb::KeyLockInfo*, std::function<arangodb::Result ()>) (this=0x7f6649b85800, trx=0x7f5517bee260, slice=..., mdr=..., options=...,
    resultMarkerTick=@0x7f5517bed3f0: 0, revisionId=@0x7f5517bed3f8: 1637397464475500713, callbackDuringLock=...) at /work/ArangoDB/arangod/RocksDBEngine/RocksDBCollection.cpp:797
#5  0x00000000009dfa85 in arangodb::LogicalCollection::insert(arangodb::transaction::Methods*, arangodb::velocypack::Slice, arangodb::ManagedDocumentResult&, arangodb::OperationOptions&, unsigned long&, bool, unsigned long&, arangodb::KeyLockInfo*, std::function<arangodb::Result ()>) (this=<optimized out>, trx=<optimized out>, slice=..., slice@entry=..., result=...,
    options=..., resultMarkerTick=@0x7f5517bed3f0: 0, lock=false, revisionId=@0x7f5517bed3f8: 1637397464475500713, keyLockInfo=0x7f5517bed760, callbackDuringLock=...)
    at /work/ArangoDB/arangod/VocBase/LogicalCollection.cpp:934
#6  0x0000000000911fd0 in arangodb::transaction::Methods::<lambda(arangodb::transaction::Methods::VPackSlice)>::operator()(arangodb::transaction::Methods::VPackSlice) const (
    __closure=__closure@entry=0x7f5517bed7b0, value=...) at /work/ArangoDB/arangod/Transaction/Methods.cpp:1787
#7  0x0000000000917838 in arangodb::transaction::Methods::insertLocal (this=this@entry=0x7f5517bee260, collectionName=..., value=..., options=...)
    at /work/ArangoDB/arangod/Transaction/Methods.cpp:1849
#8  0x0000000000917b7b in arangodb::transaction::Methods::insert (this=this@entry=0x7f5517bee260, collectionName=..., value=..., options=...)
    at /work/ArangoDB/arangod/Transaction/Methods.cpp:1602
#9  0x00000000007fe8da in arangodb::RestImportHandler::performImport (this=this@entry=0x7f55393f2400, trx=..., result=..., collectionName=..., babies=..., complete=false, opOptions=...)
    at /work/ArangoDB/arangod/RestHandler/RestImportHandler.cpp:884
#10 0x0000000000800482 in arangodb::RestImportHandler::createFromJson (this=this@entry=0x7f55393f2400, type=...) at /work/ArangoDB/arangod/RestHandler/RestImportHandler.cpp:511
#11 0x000000000080362e in arangodb::RestImportHandler::execute (this=0x7f55393f2400) at /work/ArangoDB/arangod/RestHandler/RestImportHandler.cpp:101
#12 0x00000000006fda5e in arangodb::rest::RestHandler::executeEngine (this=this@entry=0x7f55393f2400, isContinue=isContinue@entry=false)
    at /work/ArangoDB/arangod/GeneralServer/RestHandler.cpp:427
#13 0x00000000006fe92a in arangodb::rest::RestHandler::runHandlerStateMachine (this=this@entry=0x7f55393f2400) at /work/ArangoDB/arangod/GeneralServer/RestHandler.cpp:269
#14 0x0000000001113f41 in runHandler (cb=<error reading variable: access outside bounds of object referenced via synthetic pointer>, this=<optimized out>)
    at /work/ArangoDB/arangod/GeneralServer/RestHandler.h:75
#15 arangodb::rest::GeneralCommTask::handleRequest (this=<optimized out>, doLock=<optimized out>, handler=...) at /work/ArangoDB/arangod/GeneralServer/
8000
GeneralCommTask.cpp:482
#16 0x000000000111402d in operator() (isDirect=<optimized out>, __closure=<optimized out>) at /work/ArangoDB/arangod/GeneralServer/GeneralCommTask.cpp:450
#17 std::_Function_handler<void(bool), arangodb::rest::GeneralCommTask::handleRequestSync(std::shared_ptr<arangodb::rest::RestHandler>)::<lambda(bool)> >::_M_invoke(const std::_Any_data &, <unknown type in /usr/lib/debug/usr/sbin/arangod.debug, CU 0x4b59ce, DIE 0x4b59e2>) (__functor=..., __args#0=<optimized out>) at /usr/include/c++/6.4.0/functional:1731
#18 0x00000000008bdc5c in operator() (__args#0=<optimized out>, this=0x7f5517beee48) at /usr/include/c++/6.4.0/functional:2127
#19 operator() (__closure=0x7f5517beee40) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:327
#20 asio_handler_invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/handler_invoke_hook.hpp:68
#21 invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()>, arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (context=...,
    function=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_invoke_helpers.hpp:37
#22 complete<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (this=<synthetic pointer>, handler=..., function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_work.hpp:81
#23 asio::detail::completion_handler<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> >::do_complete(void *, asio::detail::operation *, const asio::error_code &, std::size_t) (owner=<optimized out>, base=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/completion_handler.hpp:69
---Type <return> to continue, or q <return> to quit---
#24 0x00000000008c5e43 in complete (bytes_transferred=0, ec=..., owner=0x7f7720bb7e00, this=0x7f5539414ec0)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/scheduler_operation.hpp:39
#25 do_run_one (ec=..., this_thread=..., lock=..., this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:400
#26 run_one (ec=..., this=0x7f7720bb7e00) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:174
#27 run_one (this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/io_context.ipp:76
#28 arangodb::SchedulerThread::run (this=0x7f58b61dec70) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:212
#29 0x0000000001204499 in runMe (this=0x7f58b61dec70) at /work/ArangoDB/lib/Basics/Thread.cpp:347
#30 arangodb::Thread::startThread (arg=0x7f58b61dec70) at /work/ArangoDB/lib/Basics/Thread.cpp:82
#31 0x00000000012c4559 in ThreadStarter (data=0x7f63ee5104a0) at /work/ArangoDB/lib/Basics/threads-posix.cpp:71
#32 0x00000000027d1a08 in start (p=0x7f5517bf1ae0) at src/thread/pthread_create.c:150
#33 0x00000000027d2fe9 in __clone () at src/thread/x86_64/clone.s:21
Backtrace stopped: frame did not save the PC

Core was generated by `/usr/sbin/arangod -c /home/admin/arangodb/data_3.4.6_singlemode_partition/singl'.
Program terminated with signal 11, Segmentation fault.
#0  getCollectionIdLocal (name=..., this=0x7f4ca5f3e280) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:69
69      /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp: No such file or directory.
(gdb) bt
#0  getCollectionIdLocal (name=..., this=0x7f4ca5f3e280) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:69
#1  arangodb::CollectionNameResolver::getCollectionId (this=0x7f4ca5f3e280, name=...) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:159
#2  0x00000000009273c0 in arangodb::SingleCollectionTransaction::SingleCollectionTransaction (this=0x7f4c97cd4980, transactionContext=..., name=...,
    accessType=arangodb::AccessMode::WRITE) at /work/ArangoDB/arangod/Utils/SingleCollectionTransaction.cpp:58
#3  0x0000000000e728c9 in arangodb::pregel::GraphStore<arangodb::pregel::HITSValue, signed char>::_storeVertices (this=0x7f9adb3e16e0, globalShards=..., it=...)
    at /work/ArangoDB/arangod/Pregel/GraphStore.cpp:591
#4  0x0000000000e72d53 in arangodb::pregel::GraphStore<arangodb::pregel::HITSValue, signed char>::storeResults(arangodb::pregel::WorkerConfig*, std::function<void ()>)::{lambda(bool)#1}::operator()(bool) const (__closure=0x7f4ca5e24d40, cancelled=false) at /work/ArangoDB/arangod/Pregel/GraphStore.cpp:664
#5  0x00000000008bdc5c in operator() (__args#0=<optimized out>, this=0x7f4777b7de48) at /usr/include/c++/6.4.0/functional:2127
#6  operator() (__closure=0x7f4777b7de40) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:327
#7  asio_handler_invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/handler_invoke_hook.hpp:68
#8  invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()>, arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (context=...,
    function=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_invoke_helpers.hpp:37
#9  complete<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (this=<synthetic pointer>, handler=..., function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_work.hpp:81
#10 asio::detail::completion_handler<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> >::do_complete(void *, asio::detail::operation *, const asio::error_code &, std::size_t) (owner=<optimized out>, base=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/completion_handler.hpp:69
#11 0x00000000008c5e43 in complete (bytes_transferred=0, ec=..., owner=0x7fba1ce8ae00, this=0x7f4acc270890)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/scheduler_operation.hpp:39
#12 do_run_one (ec=..., this_thread=..., lock=..., this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:400
#13 run_one (ec=..., this=0x7fba1ce8ae00) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:174
#14 run_one (this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/io_context.ipp:76
#15 arangodb::SchedulerThread::run (this=0x7fa6eac217a0) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:212
#16 0x0000000001204499 in runMe (this=0x7fa6eac217a0) at /work/ArangoDB/lib/Basics/Thread.cpp:347
#17 arangodb::Thread::startThread (arg=0x7fa6eac217a0) at /work/ArangoDB/lib/Basics/Thread.cpp:82
#18 0x00000000012c4559 in ThreadStarter (data=0x7fa6eabf3d80) at /work/ArangoDB/lib/Basics/threads-posix.cpp:71
#19 0x00000000027d1a08 in start (p=0x7f4777b80ae0) at src/thread/pthread_create.c:150
#20 0x00000000027d2fe9 in __clone () at src/thread/x86_64/clone.s:21
Backtrace stopped: frame did not save the PC

If you need coredump files, I can also supply.

@goedderz
Copy link
Member

Thanks a lot. For now we're fine and probably have enough information to fix it, otherwise we will ask you for more.

Best,

Tobias

@daoxian
Copy link
Author
daoxian commented Jun 26, 2019 •
edited
Loading

Thanks， and please let me know when this issue gets fixed

@graetzer
Copy link
Contributor

now I am curious did you enable the collection cache ?

@daoxian
Copy link
Author
daoxian commented Jun 27, 2019

now I am curious did you enable the collection cache ?

No, the startup cmd is:
arangodb --starter.mode=single --server.storage-engine=rocksdb --starter.data-dir=/home/admin/arangodb/data_3.4.6_singlemode_partition --all.rocksdb.block-cache-size=1000000000 --all.rocksdb.enforce-block-cache-size-limit=true --all.rocksdb.total-write-buffer-size=1000000000 --all.rocksdb.write-buffer-size=100000000 --all.rocksdb.max-write-buffer-number=100000 --all.rocksdb.min-write-buffer-number-to-merge=1 --all.rocksdb.max-total-wal-size=83886080 --all.server.maximal-queue-size=40960000 --all.server.prio1-size=1048576000

@daoxian
Copy link
Author
daoxian commented Jun 27, 2019

Btw, there're over 10,000 tables in one database. And all the tables are created dynamically.

@daoxian
Copy link
Author
daoxian commented Jun 28, 2019 •
edited
Loading

now I am curious did you enable the collection cache ?

Hi @graetzer, I think I've found the coredump reason: After bulk import, I called collection.unload() method, while pregel on this collection is running. This is related to the coredump info "getCollectionIdLocal".

@daoxian
Copy link
Author
daoxian commented Jul 1, 2019 •
edited
Loading

Hi @graetzer , I think you've fixed that, which version or git branch should I use to verify this fix ?

This was referenced Jul 1, 2019
Merged
Merged
@daoxian
Copy link
Author
daoxian commented Jul 2, 2019

Thanks for your patch. But I've found new coredumps after disable unload. Maybe my previous guess was wrong.

Core was generated by `/usr/sbin/arangod -c /home/admin/arangodb/data_3.4.6_singlemode_partition/singl'.
Program terminated with signal 11, Segmentation fault.
#0  getCollectionIdLocal (name=..., this=0x7ef6cc930780) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:69
69      /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp: No such file or directory.
(gdb) bt
#0  getCollectionIdLocal (name=..., this=0x7ef6cc930780) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:69
#1  arangodb::CollectionNameResolver::getCollectionId (this=0x7ef6cc930780, name=...) at /work/ArangoDB/arangod/Utils/CollectionNameResolver.cpp:159
#2  0x00000000009273c0 in arangodb::SingleCollectionTransaction::SingleCollectionTransaction (this=0x7ef701207f80, transactionContext=..., name=...,
    accessType=arangodb::AccessMode::WRITE) at /work/ArangoDB/arangod/Utils/SingleCollectionTransaction.cpp:58
#3  0x0000000000e728c9 in arangodb::pregel::GraphStore<arangodb::pregel::HITSValue, signed char>::_storeVertices (this=0x7ef8379f2120, globalShards=..., it=...)
    at /work/ArangoDB/arangod/Pregel/GraphStore.cpp:591
#4  0x0000000000e72d53 in arangodb::pregel::GraphStore<arangodb::pregel::HITSValue, signed char>::storeResults(arangodb::pregel::WorkerConfig*, std::function<void ()>)::{lambda(bool)#1}::operator()(bool) const (__closure=0x7ef70120b7c0, cancelled=false) at /work/ArangoDB/arangod/Pregel/GraphStore.cpp:664
#5  0x00000000008bdc5c in operator() (__args#0=<optimized out>, this=0x7ea3779dbe48) at /usr/include/c++/6.4.0/functional:2127
#6  operator() (__closure=0x7ea3779dbe40) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:327
#7  asio_handler_invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/handler_invoke_hook.hpp:68
#8  invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()>, arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (context=...,
    function=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_invoke_helpers.hpp:37
#9  complete<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (this=<synthetic pointer>, handler=..., function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_work.hpp:81
#10 asio::detail::completion_handler<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> >::do_complete(void *, asio::detail::operation *, const asio::error_code &, std::size_t) (owner=<optimized out>, base=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/completion_handler.hpp:69
#11 0x00000000008c5e43 in complete (bytes_transferred=0, ec=..., owner=0x7f2847e63e00, this=0x7ef6f9a13a40)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/scheduler_operation.hpp:39
#12 do_run_one (ec=..., this_thread=..., lock=..., this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:400
#13 run_one (ec=..., this=0x7f2847e63e00) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:174
#14 run_one (this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/io_context.ipp:76
#15 arangodb::SchedulerThread::run (this=0x7efe72040db0) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:212
#16 0x0000000001204499 in runMe (this=0x7efe72040db0) at /work/ArangoDB/lib/Basics/Thread.cpp:347
#17 arangodb::Thread::startThread (arg=0x7efe72040db0) at /work/ArangoDB/lib/Basics/Thread.cpp:82
#18 0x00000000012c4559 in ThreadStarter (data=0x7f1514fc9b80) at /work/ArangoDB/lib/Basics/threads-posix.cpp:71
#19 0x00000000027d1a08 in start (p=0x7ea3779deae0) at src/thread/pthread_create.c:150
#20 0x00000000027d2fe9 in __clone () at src/thread/x86_64/clone.s:21

@graetzer
Copy link
Contributor
graetzer commented Jul 2, 2019

is there a chance you are using empty edge collection names ?

@daoxian
Copy link
Author
daoxian commented Jul 3, 2019 •
edited
Loading

is there a chance you are using empty edge collection names ?

Do you mean an edge collection that doen't exists or an empty edge collection containing no edges? An empty edge collection is very likely to be used in my codes, this is a common case. And when an exception occurs while creating a collection, there IS a chance that the subsequent codes using a graph whose edge collection doesn't exist.

@daoxian
Copy link
Author
daoxian commented Jul 3, 2019

Btw, can vertex or edge collection be modified (adding or removing vertex/edge) when pregel is running on that graph?

@daoxian
Copy link
Author
daoxian commented Jul 4, 2019 •
edited
Loading

I've tried skipping empty edge collections and broken graph, still coredump:

Core was generated by `/usr/sbin/arangod -c /home/admin/arangodb/data_3.4.6_singlemode_partition/singl'.
Program terminated with signal 11, Segmentation fault.
#0  atomic_load_p (mo=atomic_memory_order_acquire, a=<optimized out>) at include/jemalloc/internal/atomic.h:55
55      include/jemalloc/internal/atomic.h: No such file or directory.
(gdb) bt
#0  atomic_load_p (mo=atomic_memory_order_acquire, a=<optimized out>) at include/jemalloc/internal/atomic.h:55
#1  extent_arena_get (extent=0x0) at include/jemalloc/internal/extent_inlines.h:49
#2  je_large_dalloc (tsdn=0x7f961a882f98, extent=0x0) at src/large.c:347
#3  0x00000000010b6ef3 in deallocate (this=0x7f9816750a48, __p=<optimized out>) at /usr/include/c++/6.4.0/ext/new_allocator.h:110
#4  deallocate (__a=..., __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/6.4.0/bits/alloc_traits.h:462
#5  _M_deallocate (this=0x7f9816750a48, __n=<optimized out>, __p=<optimized out>) at /usr/include/c++/6.4.0/bits/stl_vector.h:178
#6  std::vector<arangodb::pregel::SenderMessage<double>, std::allocator<arangodb::pregel::SenderMessage<double> > >::_M_range_insert<__gnu_cxx::__normal_iterator<arangodb::pregel::SenderMessage<double> const*, std::vector<arangodb::pregel::SenderMessage<double>, std::allocator<arangodb::pregel::SenderMessage<double> > > > > (this=0x7f9816750a48, __position=...,
    __first=..., __last=...) at /usr/include/c++/6.4.0/bits/vector.tcc:685
#7  0x00000000010b7289 in _M_insert_dispatch<__gnu_cxx::__normal_iterator<arangodb::pregel::SenderMessage<double> const*, std::vector<arangodb::pregel::SenderMessage<double>, std::allocator<arangodb::pregel::SenderMessage<double> > > > > (__last=..., __first=..., __pos=..., this=<optimized out>) at /usr/include/c++/6.4.0/bits/stl_vector.h:1375
#8  insert<__gnu_cxx::__normal_iterator<arangodb::pregel::SenderMessage<double> const*, std::vector<arangodb::pregel::SenderMessage<double>, std::allocator<arangodb::pregel::SenderMessage<double> > > > > (__last=..., __first=..., __position=..., this=<optimized out>) at /usr/include/c++/6.4.0/bits/stl_vector.h:1100
#9  arangodb::pregel::ArrayInCache<arangodb::pregel::SenderMessage<double> >::mergeCache (this=0x7f9b13091480, config=..., otherCache=0x7f9b13091b80)
    at /work/ArangoDB/arangod/Pregel/IncomingCache.cpp:151
#10 0x0000000000ea0eb7 in arangodb::pregel::Worker<arangodb::pregel::HITSValue, signed char, arangodb::pregel::SenderMessage<double> >::_processVertices (this=0x7f9a85dbf010,
    threadId=<optimized out>, vertexIterator=...) at /work/ArangoDB/arangod/Pregel/Worker.cpp:418
#11 0x0000000000eb1710 in arangodb::pregel::Worker<arangodb::pregel::HITSValue, signed char, arangodb::pregel::SenderMessage<double> >::_startProcessing()::{lambda(bool)#1}::operator()(bool) const (__closure=0x7f93fc36f140) at /work/ArangoDB/arangod/Pregel/Worker.cpp:335
#12 0x00000000008bdc5c in operator() (__args#0=<optimized out>, this=0x7f961a881e48) at /usr/include/c++/6.4.0/functional:2127
#13 operator() (__closure=0x7f961a881e40) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:327
#14 asio_handler_invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/handler_invoke_hook.hpp:68
#15 invoke<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()>, arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> > (context=...,
    function=...) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_invoke_helpers.hpp:37
#16 complete<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()&
F438
gt; > (this=<synthetic pointer>, handler=..., function=...)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/handler_work.hpp:81
#17 asio::detail::completion_handler<arangodb::rest::Scheduler::post(std::function<void(bool)>, bool)::<lambda()> >::do_complete(void *, asio::detail::operation *, const asio::error_code &, std::size_t) (owner=<optimized out>, base=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/completion_handler.hpp:69
#18 0x00000000008c5e43 in complete (bytes_transferred=0, ec=..., owner=0x7fd2f854de00, this=0x7f975ce6d950)
    at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/scheduler_operation.hpp:39
#19 do_run_one (ec=..., this_thread=..., lock=..., this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:400
#20 run_one (ec=..., this=0x7fd2f854de00) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/detail/impl/scheduler.ipp:174
#21 run_one (this=<optimized out>) at /work/ArangoDB/3rdParty/asio/1.12/include/asio/impl/io_context.ipp:76
#22 arangodb::SchedulerThread::run (this=0x7fbfbd103bd0) at /work/ArangoDB/arangod/Scheduler/Scheduler.cpp:212
#23 0x0000000001204499 in runMe (this=0x7fbfbd103bd0) at /work/ArangoDB/lib/Basics/Thread.cpp:347
#24 arangodb::Thread::startThread (arg=0x7fbfbd103bd0) at /work/ArangoDB/lib/Basics/Thread.cpp:82
#25 0x00000000012c4559 in ThreadStarter (data=0x7fbfc1b3e130) at /work/ArangoDB/lib/Basics/threads-posix.cpp:71
#26 0x00000000027d1a08 in start (p=0x7f961a884ae0) at src/thread/pthread_create.c:150
#27 0x00000000027d2fe9 in __clone () at src/thread/x86_64/clone.s:21
Backtrace stopped: frame did not save the PC

Look at the coredump files, I don't know how long can I bear all the stuff, maybe switching to a alternative db is a good choise.
image

@mpoeter
Copy link
Contributor
mpoeter commented Jul 4, 2019

This is yet another occurrence of the same issue as in #9302 (comment) and #9301 (comment)

As already mentioned, this looks very much like a race on ArrayInCache::_shardMap to me (or more specifically on the entries of that map). It seems that ArrayInCache<M>::_set, ArrayInCache<M>::clear and ArrayInCache<M>::erase modify the _shardMap or entries of that map without acquiring the _bucketLocker. I am not familiar enough with the code to judge whether this is indeed a problem, or if other synchronization mechanisms are used somewhere else, but maybe someone from the ArangoDB team can take a look at it.

@daoxian daoxian mentioned this issue Jul 9, 2019
Open
@sourabhedake-mm
Copy link

@dothebart Is there any update on the core dump mentioned in - #9302 (comment)

I am facing a similar core in the prod -

(gdb) bt 
#0  0x0000556d1d209d0a in arangodb::velocypack::readIntegerNonEmpty<unsigned long> (start=0x7f364e208ae8 <error: Cannot access memory at address 0x7f364e208ae8>, length=4)
    at /c8db/3rdParty/velocypack/include/velocypack/velocypack-common.h:230
#1  0x0000556d1d209622 in arangodb::velocypack::Slice::getNthOffset (this=0x7f35ee9f5820, index=0) at /c8db/3rdParty/velocypack/src/Slice.cpp:495
#2  0x0000556d1be88709 in arangodb::velocypack::Slice::getNthKeyUntranslated (this=0x7f35ee9f5820, index=0) at /c8db/3rdParty/velocypack/include/velocypack/Slice.h:1080
#3  0x0000556d1bee4ea9 in arangodb::velocypack::ObjectIterator::operator* (this=0x7f35ee9f5820) at /c8db/3rdParty/velocypack/include/velocypack/Iterator.h:242
#4  0x0000556d1e8d222e in rice::needStrip (document=...) at /c8db/arangod/Rice/Rice.cpp:514
#5  0x0000556d1e8d22a9 in rice::strip (document=..., didStrip=0x7f35ee9f5a67, copyIfNotStrip=false) at /c8db/arangod/Rice/Rice.cpp:524
#6  0x0000556d1c78c3e7 in arangodb::RocksDBAllIndexIterator::nextDocument(std::function<void (arangodb::LocalDocumentId const&, arangodb::velocypack::Slice)> const&, unsigned long, bool) (this=0x7f358729c520, 
    cb=..., limit=1000, raw=false) at /c8db/arangod/RocksDBEngine/RocksDBIterators.cpp:128
#7  0x0000556d1c5d3f55 in arangodb::OperationCursor::nextDocument(std::function<void (arangodb::LocalDocumentId const&, arangodb::velocypack::Slice)> const&, unsigned long, bool) (this=0x7f353c7f0960, 
    callback=..., batchSize=1000, raw=false) at /c8db/arangod/Utils/OperationCursor.cpp:86
#8  0x0000556d1d00b60f in arangodb::aql::EnumerateCollectionExecutor::produceRows (this=0x7f35b28586a8, output=...) at /c8db/arangod/Aql/EnumerateCollectionExecutor.cpp:148
#9  0x0000556d1c9f9f0a in arangodb::aql::ExecutionBlockImpl<arangodb::aql::EnumerateCollectionExecutor>::getSomeWithoutTrace (this=0x7f35b2858400, atMost=1000) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:169
#10 0x0000556d1c9f9a47 in arangodb::aql::ExecutionBlockImpl<arangodb::aql::EnumerateCollectionExecutor>::getSome (this=0x7f35b2858400, atMost=1000) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:119
#11 0x0000556d1cfdf462 in arangodb::aql::DependencyProxy<true>::prefetchBlock (this=0x7f35f9608a08, atMost=1000) at /c8db/arangod/Aql/DependencyProxy.cpp:35
#12 0x0000556d1cfdf1c8 in arangodb::aql::DependencyProxy<true>::fetchBlockForPassthrough (this=0x7f35f9608a08, atMost=1000) at /c8db/arangod/Aql/DependencyProxy.cpp:211
#13 0x0000556d1cb87d73 in arangodb::aql::SingleRowFetcher<true>::fetchBlockForPassthrough (this=0x7f35f9608ae8, atMost=1000) at /c8db/arangod/Aql/SingleRowFetcher.cpp:66
#14 0x0000556d1ca2062c in arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0>::fetchBlockForPassthrough (this=0x7f35f9608bb0, atMost=1000) at /c8db/arangod/Aql/CalculationExecutor.h:114
#15 0x0000556d1ca19bc9 in arangodb::aql::RequestWrappedBlock<(arangodb::aql::RequestWrappedBlockVariant)1>::run<arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0> > (executor=..., 
    engine=..., nrItems=1000, nrRegs=2) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:572
#16 0x0000556d1c9f5b5a in arangodb::aql::ExecutionBlockImpl<arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0> >::requestWrappedBlock (this=0x7f35f9608900, nrItems=1000, nrRegs=2)
    at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:677
#17 0x0000556d1c9f578c in arangodb::aql::ExecutionBlockImpl<arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0> >::getSomeWithoutTrace (this=0x7f35f9608900, atMost=1000)
    at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:144
#18 0x0000556d1c9f5461 in arangodb::aql::ExecutionBlockImpl<arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0> >::getSome (this=0x7f35f9608900, atMost=1000)
    at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:119
#19 0x0000556d1cfdfd62 in arangodb::aql::DependencyProxy<false>::prefetchBlock (this=0x7f3601fe2008, atMost=1000) at /c8db/arangod/Aql/DependencyProxy.cpp:35
#20 0x0000556d1cfdf702 in arangodb::aql::DependencyProxy<false>::fetchBlock (this=0x7f3601fe2008, atMost=1000) at /c8db/arangod/Aql/DependencyProxy.cpp:78
#21 0x0000556d1cb87be0 in arangodb::aql::SingleRowFetcher<false>::fetchBlock (this=0x7f3601fe20e8, atMost=1000) at /c8db/arangod/Aql/SingleRowFetcher.cpp:52
#22 0x0000556d1ca16a29 in arangodb::aql::SingleRowFetcher<false>::preFetchNumberOfRows (this=0x7f3601fe20e8, atMost=1000) at /c8db/arangod/Aql/SingleRowFetcher.h:98
#23 0x0000556d1ca743de in arangodb::aql::FilterExecutor::expectedNumberOfRows (this=0x7f3601fe2170, atMost=1000) at /c8db/arangod/Aql/FilterExecutor.cpp:94
#24 0x0000556d1ca1b30a in arangodb::aql::RequestWrappedBlock<(arangodb::aql::RequestWrappedBlockVariant)2>::run<arangodb::aql::FilterExecutor> (executor=..., engine=..., nrItems=1000, nrRegs=2)
    at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:629
#25 0x0000556d1c9fb99c in arangodb::aql::ExecutionBlockImpl<arangodb::aql::FilterExecutor>::requestWrappedBlock (this=0x7f3601fe1f00, nrItems=1000, nrRegs=2) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:677
#26 0x0000556d1c9fb594 in arangodb::aql::ExecutionBlockImpl<arangodb::aql::FilterExecutor>::getSomeWithoutTrace (this=0x7f3601fe1f00, atMost=1000) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:144
#27 0x0000556d1c9fb301 in arangodb::aql::ExecutionBlockImpl<arangodb::aql::FilterExecutor>::skipSome (this=0x7f3601fe1f00, atMost=1000) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:320
#28 0x0000556d1cfdfc04 in arangodb::aql::DependencyProxy<false>::skipSome (this=0x7f35f9608408, toSkip=1000000000) at /c8db/arangod/Aql/DependencyProxy.cpp:178
#29 0x0000556d1cb87b1a in arangodb::aql::SingleRowFetcher<false>::skipRows (this=0x7f35f96084e8, atMost=1000000000) at /c8db/arangod/Aql/SingleRowFetcher.cpp:74
#30 0x0000556d1c9f4808 in arangodb::aql::CountCollectExecutor::produceRows (this=0x7f35f9608570, output=...) at /c8db/arangod/Aql/CountCollectExecutor.h:106
#31 0x0000556d1c9f8786 in arangodb::aql::ExecutionBlockImpl<arangodb::aql::CountCollectExecutor>::getSomeWithoutTrace (this=0x7f35f9608300, atMost=1000) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:169
#32 0x0000556d1c9f826b in arangodb::aql::ExecutionBlockImpl<arangodb::aql::CountCollectExecutor>::getSome (this=0x7f35f9608300, atMost=1000) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:119
#33 0x0000556d1cfdf462 in arangodb::aql::DependencyProxy<true>::prefetchBlock (this=0x7f35f9608708, atMost=1000) at /c8db/arangod/Aql/DependencyProxy.cpp:35
#34 0x0000556d1cfdf1c8 in arangodb::aql::DependencyProxy<true>::fetchBlockForPassthrough (this=0x7f35f9608708, atMost=1000) at /c8db/arangod/Aql/DependencyProxy.cpp:211
#35 0x0000556d1cb87d73 in arangodb::aql::SingleRowFetcher<true>::fetchBlockForPassthrough (this=0x7f35f96087e8, atMost=1000) at /c8db/arangod/Aql/SingleRowFetcher.cpp:66
#36 0x0000556d1ca2062c in arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0>::fetchBlockForPassthrough (this=0x7f35f96088b0, atMost=1000) at /c8db/arangod/Aql/CalculationExecutor.h:114
#37 0x0000556d1ca19bc9 in arangodb::aql::RequestWrappedBlock<(arangodb::aql::RequestWrappedBlockVariant)1>::run<arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0> > (executor=..., 
    engine=..., nrItems=1000, nrRegs=4) at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:572
#38 0x0000556d1c9f5b5a in arangodb::aql::ExecutionBlockImpl<arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0> >::requestWrappedBlock (this=0x7f35f9608600, nrItems=1000, nrRegs=4)
    at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:677
#39 0x0000556d1c9f578c in arangodb::aql::ExecutionBlockImpl<arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0> >::getSomeWithoutTrace (this=0x7f35f9608600, atMost=1000)
    at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:144
#40 0x0000556d1c9f5461 in arangodb::aql::ExecutionBlockImpl<arangodb::aql::CalculationExecutor<(arangodb::aql::CalculationType)0> >::getSome (this=0x7f35f9608600, atMost=1000)
    at /c8db/arangod/Aql/ExecutionBlockImpl.cpp:119
#41 0x0000556d1ca3d946 in arangodb::aql::ExecutionBlockImpl<arangodb::aql::IdExecutor<void> >::getSome (this=0x7f35f961a3c0, atMost=1000) at /c8db/arangod/Aql/IdExecutor.h:92
--Type <RET> for more, q to quit, c to continue without paging--c
#42 0x0000556d1ca279f6 in arangodb::aql::ExecutionEngine::getSome (this=0x7f35b2853300, atMost=1000) at /c8db/arangod/Aql/ExecutionEngine.cpp:519
#43 0x0000556d1c18f9fd in arangodb::aql::Query::execute (this=0x7f35ee9f7b00, registry=0x7f360f5d4640, queryResult=...) at /c8db/arangod/Aql/Query.cpp:746
#44 0x0000556d1c191bfd in arangodb::aql::Query::executeSync (this=0x7f35ee9f7b00, registry=0x7f360f5d4640) at /c8db/arangod/Aql/Query.cpp:1027
#45 0x0000556d1c02e00e in c8db_db::queryFetch (aql="FOR entry IN _collections FILTER entry.tenant == \"qa_amol_test.com\" AND entry.type == 2 AND SUBSTRING(entry.topic, 0, 1) != \"_\" COLLECT WITH COUNT INTO count RETURN {\"count\": count}", vars=std::shared_ptr<arangodb::velocypack::Builder> (empty) = {...}, vocbase=0x7f35c7d82200) at /c8db/arangod/C8DBFeatures/query_helper.cpp:84
#46 0x0000556d1c02d4eb in c8db_db::queryFetchSingle (aql="FOR entry IN _collections FILTER entry.tenant == \"qa_amol_test.com\" AND entry.type == 2 AND SUBSTRING(entry.topic, 0, 1) != \"_\" COLLECT WITH COUNT INTO count RETURN {\"count\": count}", vars=std::shared_ptr<arangodb::velocypack::Builder> (empty) = {...}, builder=..., vocbase=0x7f35c7d82200) at /c8db/arangod/C8DBFeatures/query_helper.cpp:51
#47 0x0000556d1c52f10e in getCount (c8ql="FOR entry IN _collections FILTER entry.tenant == \"qa_amol_test.com\" AND entry.type == 2 AND SUBSTRING(entry.topic, 0, 1) != \"_\" COLLECT WITH COUNT INTO count RETURN {\"count\": count}", tenant="qa_amol_test.com", database="QATest23_24") at /c8db/arangod/Statistics/StatisticsWorker.cpp:262
#48 0x0000556d1c53a626 in arangodb::StatisticsWorker::updateDatabaseStats (this=0x7f360f451500, tenant="qa_amol_test.com", database="QATest23_24") at /c8db/arangod/Statistics/StatisticsWorker.cpp:1592
#49 0x0000556d1c5372cc in arangodb::StatisticsWorker::updateTenantStatistics (this=0x7f360f451500) at /c8db/arangod/Statistics/StatisticsWorker.cpp:1252
#50 0x0000556d1c53be05 in arangodb::StatisticsWorker::run (this=0x7f360f451500) at /c8db/arangod/Statistics/StatisticsWorker.cpp:1691
#51 0x0000556d1d2684ab in arangodb::Thread::runMe (this=0x7f360f451500) at /c8db/lib/Basics/Thread.cpp:317
#52 0x0000556d1d265f8b in arangodb::Thread::startThread (arg=0x7f360f451500) at /c8db/lib/Basics/Thread.cpp:96
#53 0x0000556d1d30d3fa in ThreadStarter (data=0x7f360e22b330) at /c8db/lib/Basics/threads-posix.cpp:78

@sourabhedake-mm sourabhedake-mm mentioned this issue Jan 12, 2023
Open
@goedderz goedderz closed this as not planned Won't fix, can't repro, duplicate, stale May 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@graetzer graetzer

@goedderz goedderz

Labels
1 Bug 3 Core
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@graetzer @dothebart @mpoeter @maxkernbach @goedderz @daoxian @sourabhedake-mm
0