8000 Write rocksdb error, db change to read-only mode · Issue #10711 · arangodb/arangodb · GitHub
[go: up one dir, main page]
Skip to content

Write rocksdb error, db change to read-only mode #10711

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
dongdaogua 8000 ng opened this issue Dec 16, 2019 · 3 comments
Closed

Write rocksdb error, db change to read-only mode #10711

dongdaoguang opened this issue Dec 16, 2019 · 3 comments
Labels
1 Analyzing 2 User Abandoned Resolution 3 RocksDB Storage engine related

Comments

@dongdaoguang
Copy link

My Environment

  • ArangoDB Version: 3.4.5
  • Storage Engine: RocksDB
  • Deployment Mode: Single Server
  • Deployment Strategy: ArangoDB Starter
  • Configuration:
  • Infrastructure:
  • Operating System: CentOS release 6.9 (Final)
  • Total RAM in your machine: 128Gb
  • Disks in use: SSD
  • Used Package: Debian

Component, Query & Data

error log

2019-12-02T09:30:37Z [59857] INFO {authentication} Jwt secret not specified, generating...
2019-12-02T09:30:37Z [59857] INFO using storage engine rocksdb
2019-12-02T09:30:37Z [59857] INFO {syscall} file-descriptors (nofiles) hard limit is 655350, soft limit is 131072
2019-12-02T09:30:37Z [59857] INFO {authentication} Authentication is turned on (system only), authentication for unix sockets is turned on
2019-12-02T09:30:37Z [59857] WARNING found existing lockfile '/var/lib/arangodb3/LOCK' of previous process with pid 439705, and that process seems to be still running
2019-12-02T09:30:37Z [59857] FATAL database is locked by process 439705; please stop it first and check that the lockfile '/var/lib/arangodb3/LOCK' goes away. If you are sure no other arangod process is running, please remove the lockfile '/var/lib/arangodb3/LOCK' and try again
2019-12-02T09:36:27Z [59955] INFO {authentication} Jwt secret not specified, generating...
2019-12-02T09:36:27Z [59955] INFO using storage engine rocksdb
2019-12-02T09:36:27Z [59955] INFO {syscall} file-descriptors (nofiles) hard limit is 655350, soft limit is 131072
2019-12-02T09:36:27Z [59955] INFO {authentication} Authentication is turned on (system only), authentication for unix sockets is turned on
2019-12-02T09:36:42Z [60116] INFO ArangoDB 3.4.5 [linux] 64bit, using jemalloc, build tags/v3.4.5-0-g648fbb8191, VPack 0.1.33, RocksDB 5.16.0, ICU 58.1, V8 5.7.492.77, OpenSSL 1.1.0j 20 Nov 2018
2019-12-02T09:36:42Z [60116] INFO detected operating system: Linux version 2.6.32-754.12.1.el6.x86_64 (mockbuild@x86-01.bsys.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-23) (GCC) ) #1 SMP Tue Apr 9 14:52:26 UTC 2019
2019-12-02T09:36:42Z [60116] WARNING {memory} It is recommended to set NUMA to interleaved.
2019-12-02T09:36:42Z [60116] WARNING {memory} put 'numactl --interleave=all' in front of your command
2019-12-02T09:36:42Z [60116] INFO {authentication} Jwt secret not specified, generating...
2019-12-02T09:36:42Z [60116] INFO using storage engine rocksdb
2019-12-02T09:36:42Z [60116] INFO {cluster} Starting up with role SINGLE
2019-12-02T09:36:42Z [60116] INFO {syscall} file-descriptors (nofiles) hard limit is 655350, soft limit is 131072
2019-12-02T09:36:42Z [60116] INFO {authentication} Authentication is turned on (system only), authentication for unix sockets is turned on
2019-12-02T09:36:46Z [60116] INFO using endpoint 'http+tcp://0.0.0.0:8529' for non-encrypted requests
2019-12-02T09:36:48Z [60116] INFO ArangoDB (version 3.4.5 [linux]) is ready for business. Have fun!
2019-12-02T09:38:14Z [60116] WARNING Failed to update Foxx store from GitHub.
2019-12-02T09:38:50Z [60116] WARNING Failed to update Foxx store from GitHub.
2019-12-02T09:39:28Z [60116] WARNING Failed to update Foxx store from GitHub.
2019-12-02T10:42:30Z [60116] ERROR {rocksdb} RocksDB encountered a background error during a memtable operation; The database will be put in read-only mode, and subsequent write errors are likely
2019-12-02T10:42:32Z [60116] WARNING {engines} could not sync metadata for collection 'album'
2019-12-02T10:42:32Z [60116] WARNING {engines} background settings sync failed: Corruption: two consecutive TryAgain in WriteBatch handler; this is either a software bug or data corruption.
2019-12-02T10:42:34Z [60116] ERROR {rocksdb} rocksdb: [/work/ArangoDB/3rdParty/rocksdb/v5.16.X/db/db_impl_compaction_flush.cc:1520] Waiting after background compaction error: Corruption: two consecutive TryAgain in WriteBatch handler; this is either a software bug or data corruption., Accumulated background error counts: 1
2019-12-02T10:42:39Z [60116] WARNING {statistics} could not commit stats to _statisticsRaw: Corruption: two consecutive TryAgain in WriteBatch handler; this is either a software bug or data corruption.
2019-12-02T10:42:39Z [60116] WARNING {statistics} could not commit stats to _statistics: Corruption: two consecutive TryAgain in WriteBatch handler; this is either a software bug or data corruption.
2019-12-02T10:42:49Z [60116] WARNING {statistics} could not commit stats to _statisticsRaw: Corruption: two consecutive TryAgain in WriteBatch handler; this is either a software bug or data corruption.
2019-12-02T10:42:59Z [60116] WARNING {statistics} could not commit stats to _statisticsRaw: Corruption: two consecutive TryAgain in WriteBatch handler; this is either a software bug or data corruption.
@OmarAyo OmarAyo changed the title Wirte rocksdb error, db change to read-only mode Write rocksdb error, db change to read-only mode Dec 18, 2019
@OmarAyo OmarAyo added 1 Analyzing 3 RocksDB Storage engine related labels Dec 18, 2019
@maxkernbach
Copy link
Contributor

Hi @dongdaoguang,

The log is reporting that the database is locked by another process:

2019-12-02T09:30:37Z [59857] WARNING found existing lockfile '/var/lib/arangodb3/LOCK' of previous process with pid 439705, and that process seems to be still running
2019-12-02T09:30:37Z [59857] FATAL database is locked by process 439705; please stop it first and check that the lockfile '/var/lib/arangodb3/LOCK' goes away. If you are sure no other arangod process is running, please remove the lockfile '/var/lib/arangodb3/LOCK' and try again

In case you did not already react to this error, the following might help:

Since you are running a single instance, only one arangod process should be running. You can check whether other arangod processes are running by executing ps auxw | grep arangod. If this is the case, kill the locking process, remove the lockfile located at /var/lib/arangodb3/LOCK and restart your instance.

@maxkernbach
Copy link
Contributor

Hi @dongdaoguang,

Did you have a chance to read my previous message? Was it helpful?

@maxkernbach
Copy link
Contributor

Hi @dongdaoguang,

Since we have not heard back from you in a while, I am closing this issue for now.
In case the reported error persists, please add a comment and we will reopen this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
1 Analyzing 2 User Abandoned Resolution 3 RocksDB Storage engine related
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@dongdaoguang @maxkernbach @OmarAyo
0