8000 unable to initialize RocksDB engine: Corruption: SST file is ahead of WALs · Issue #9757 · arangodb/arangodb · GitHub
[go: up one dir, main page]
Skip to content

unable to initialize RocksDB engine: Corruption: SST file is ahead of WALs #9757

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
rahulserver opened this issue Aug 19, 2019 · 6 comments
Open

unable to initialize RocksDB engine: Corruption: SST file is ahead of WALs #9757

rahulserver opened this issue Aug 19, 2019 · 6 comments
Labels
1 Analyzing 2 User Abandoned Resolution 3 RocksDB Storage engine related 4 Windows workaround available

Comments

@rahulserver
Copy link

My Environment

  • ArangoDB Version: 3.4.7
  • Storage Engine: RocksDB
  • Deployment Mode: Single Server
  • Deployment Strategy: arangod from windows cmd
  • Configuration:
  • Infrastructure: windows pc
  • Operating System: windows 10
  • Total RAM in your machine: 16Gb
  • Disks in use: SSD(C:) and HDD(D&E)
  • Used Package:

I Stopped arangod from cmd using ctrl+c.
Then after sometime when I try to start it using arangod, I get this trace:

C:\Users\r\Downloads\ArangoDB3-3.4.7_win64\ArangoDB3-3.4.7_win64\usr\bin>arangod
2019-08-19T16:10:15Z [1252] INFO ArangoDB 3.4.7 [win64] 64bit, using build tags/v3.4.7-0-ga1c85830e5, VPack 0.1.33, RocksDB 5.16.0, ICU 58.1, V8 5.7.492.77, OpenSSL 1.1.0j  20 Nov 2018
2019-08-19T16:10:15Z [1252] INFO {authentication} Jwt secret not specified, generating...
2019-08-19T16:10:15Z [1252] INFO using storage engine rocksdb
2019-08-19T16:10:15Z [1252] INFO {cluster} Starting up with role SINGLE
2019-08-19T16:10:15Z [1252] INFO {authentication} Authentication is turned on (system only)
2019-08-19T16:10:15Z [1252] FATAL {startup} unable to initialize RocksDB engine: Corruption: SST file is ahead of WALs

This is the first time I am using arangodb, so not sure what do I do to resolve this.
@OmarAyo OmarAyo added 1 Analyzing 3 RocksDB Storage engine related 4 Windows labels Aug 21, 2019
@maxkernbach
Copy link
Contributor

Hi @rahulserver,

Sorry for the late reply.

Whenever this error occurs, it is recommended to add the following WAL startup options:

--wal.ignore-logfile-errors <boolean>                       ignore logfile errors. this will read recoverable data from corrupted logfiles but ignore any unrecoverable data (default: false)
--wal.ignore-recovery-errors <boolean>                      continue recovery even if re-applying operations fails (default: false)

and restart the DBservers (the single instance in your case).

Once the deployment is up and running, you can remove those options again.

@psoKnight
Copy link

ArangoDB Version: 3.7.5
Storage Engine: RocksDB
Deployment Mode: Cluster
Deployment Strategy: Linux

How do we deal with this in the latest release with “ArangoDB Starter”。
image

@psoKnight
Copy link

2021-01-09T04:14:47Z [704] INFO [3844e] {authentication} Authentication is turned on (system only), authentication for unix sockets is turned on
2021-01-09T04:14:51Z [704] FATAL [fe3df] {startup} unable to initialize RocksDB engine: Corruption: SST file is ahead of WALs
2021-01-09T04:14:52Z [841] INFO [e52b0] ArangoDB 3.7.3 [linux] 64bit, using jemalloc, build tags/v3.7.3-0-g8ad8edb71f, VPack 0.1.33, RocksDB 6.8.0, ICU 64.2, V8 7.9.317, OpenSSL 1.1.1g 21 Apr 2020
2021-01-09T04:14:52Z [841] INFO [75ddc] detected operating system: Linux version 3.10.0-957.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Thu Nov 8 23:39:32 UTC 2018
2021-01-09T04:14:52Z [841] INFO [25362] {memory} Available physical memory: 33331818496 bytes, available cores: 8
2021-01-09T04:14:52Z [841] INFO [144fe] using storage engine 'rocksdb'
2021-01-09T04:14:52Z [841] INFO [3bb7d] {cluster} Starting up with role PRIMARY
2021-01-09T04:14:52Z [841] INFO [6ea38] using endpoint 'http+tcp://[::]:8530' for non-encrypted requests
2021-01-09T04:14:52Z [841] INFO [a1c60] {syscall} file-descriptors (nofiles) hard limit is 8192, soft limit is 8192
2021-01-09T04:14:52Z [841] INFO [3844e] {authentication} Authentication is turned on (system only), authentication for unix sockets is turned on
2021-01-09T04:14:56Z [841] FATAL [fe3df] {startup} unable to initialize RocksDB engine: Corruption: SST file is ahead of WALs
2021-01-09T04:14:56Z [979] INFO [e52b0] ArangoDB 3.7.3 [linux] 64bit, using jemalloc, build tags/v3.7.3-0-g8ad8edb71f, VPack 0.1.33, RocksDB 6.8.0, ICU 64.2, V8 7.9.317, OpenSSL 1.1.1g 21 Apr 2020
2021-01-09T04:14:56Z [979] INFO [75ddc] detected operating system: Linux version 3.10.0-957.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-36) (GCC) ) #1 SMP Thu Nov 8 23:39:32 UTC 2018
2021-01-09T04:14:56Z [979] INFO [25362] {memory} Available physical memory: 33331818496 bytes, available cores: 8
2021-01-09T04:14:56Z [979] INFO [144fe] using storage engine 'rocksdb'
2021-01-09T04:14:56Z [979] INFO [3bb7d] {cluster} Starting up with role PRIMARY
2021-01-09T04:14:56Z [979] INFO [6ea38] using endpoint 'http+tcp://[::]:8530' for non-encrypted requests
2021-01-09T04:14:56Z [979] INFO [a1c60] {syscall} file-descriptors (nofiles) hard limit is 8192, soft limit is 8192
2021-01-09T04:14:56Z [979] INFO [3844e] {authentication} Authentication is turned on (system only), authentication for unix sockets is turned on
2021-01-09T04:15:00Z [979] FATAL [fe3df] {startup} unable to initialize RocksDB engine: Corruption: SST file is ahead of WALs

@psoKnight
Copy link

@maxkernbach

@psoKnight
Copy link

Looking forward to your reply. Thx.
A784

@dothebart dothebart reopened this Jan 11, 2021
@dothebart
Copy link
Contributor

Hi,
Please don't use old closed issues; messages may easily be overlooked there. Instead open a new issue, and reference the issue you feel is related to your issue.
You can add the referenced options to the starter via the pass through mechanism; for this case only the db-servers need this option - so use --dbservers.... .

@mrinc mrinc mentioned this issue Mar 16, 2021
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
1 Analyzing 2 User Abandoned Resolution 3 RocksDB Storage engine related 4 Windows workaround available
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@dothebart @rahulserver @Simran-B @maxkernbach @OmarAyo @psoKnight
0