8000 unable to initialize RocksDB engine: Corruption: SST file is ahead of WALs · Issue #13728 · arangodb/arangodb · GitHub
[go: up one dir, main page]
Skip to content

unable to initialize RocksDB engine: Corruption: SST file is ahead of WALs #13728

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mrinc opened this issue Mar 16, 2021 · 2 comments
Closed

unable to initialize RocksDB engine: Corruption: SST file is ahead of WALs #13728

mrinc opened this issue Mar 16, 2021 · 2 comments
Labels
1 Analyzing 2 Solved Resolution 3 RocksDB Storage engine related

Comments

@mrinc
Copy link
mrinc commented Mar 16, 2021 •
edited
Loading

My Environment

  • ArangoDB Version: 3.7.3
  • Storage Engine: Volume on Docker - underlying is glusterFS
  • Deployment Mode: Single server
  • Deployment Strategy: ArangoDB Starter in Docker
  • Configuration: Default
  • Infrastructure: Own
  • Operating System: CentOS7
  • Total RAM in your machine: 16Gb
  • Disks in use: SSD
  • Used Package: arangodb/arangodb:3.7.3

Component, Query & Data

Affected feature:
Startup

Steps to reproduce

  1. Start docker container

Problem:
[132] FATAL [fe3df] {startup} unable to initialize RocksDB engine: Corruption: SST file is ahead of WALs

Expected result:
It working?

Notes:

https://www.arangodb.com/docs/stable/programs-arangod-wal.html#ignore-logfile-errors
^ no longer valid
I did have a server issue which I think shutdown the DB in an unsafe manner to cause this.

Reference:

#9757

Startup logs:

docker run -ti -v /mnt/gluster/db:/var/lib/arangodb3 arangodb/arangodb:3.7.3@sha256:b30060778565c982f5d04fdf7cad451c87bd12b059d8c498b5b0b0eb06ac46b0 /bin/sh
/ # arangod --wal.ignore-logfile-errors=true --wal.ignore-recovery-errors=true
2021-03-16T20:12:19Z [8] WARNING {startup} obsolete option '--wal.ignore-logfile-errors' used in configuration. setting this option will not have any effect.
2021-03-16T20:12:19Z [8] WARNING {startup} obsolete option '--wal.ignore-recovery-errors' used in configuration. setting this option will not have any effect.
2021-03-16T20:12:19Z [8] INFO [e52b0] ArangoDB 3.7.3 [linux] 64bit, using jemalloc, build tags/v3.7.3-0-g8ad8edb71f4, VPack 0.1.33, RocksDB 6.8.0, ICU 64.2, V8 7.9.317, OpenSSL 1.1.1g  21 Apr 2020
2021-03-16T20:12:19Z [8] INFO [75ddc] detected operating system: Linux version 4.18.0-240.10.1.el8_3.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)) #1 SMP Mon Jan 18 17:05:51 UTC 2021
2021-03-16T20:12:19Z [8] INFO [25362] {memory} Available physical memory: 16458944512 bytes, available cores: 4
2021-03-16T20:12:19Z [8] WARNING [118b0] {memory} maximum number of memory mappings per process is 65530, which seems too low. it is recommended to set it to at least 256000
2021-03-16T20:12:19Z [8] WARNING [49528] {memory} execute 'sudo sysctl -w "vm.max_map_count=256000"'
2021-03-16T20:12:19Z [8] WARNING [e8b68] {memory} /sys/kernel/mm/transparent_hugepage/enabled is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2021-03-16T20:12:19Z [8] WARNING [f3108] {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/enabled"'
2021-03-16T20:12:19Z [8] WARNING [f3108] {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/defrag"'
2021-03-16T20:12:19Z [8] INFO [43396] {authentication} Jwt secret not specified, generating...
2021-03-16T20:12:19Z [8] INFO [144fe] using storage engine 'rocksdb'
2021-03-16T20:12:19Z [8] INFO [3bb7d] {cluster} Starting up with role SINGLE
2021-03-16T20:12:19Z [8] INFO [6ea38] using endpoint 'http+tcp://0.0.0.0:8529' for non-encrypted requests
2021-03-16T20:12:19Z [8] INFO [a1c60] {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2021-03-16T20:12:19Z [8] INFO [3844e] {authentication} Authentication is turned on (system only), authentication for unix sockets is turned on
2021-03-16T20:12:19Z [8] FATAL [fe3df] {startup} unable to initialize RocksDB engine: Corruption: SST file is ahead of WALs
/ # arangod --wal.ignore-logfile-errors true --wal.ignore-recovery-errors true
2021-03-16T20:13:10Z [132] WARNING {startup} obsolete option '--wal.ignore-logfile-errors' used in configuration. setting this option will not have any effect.
2021-03-16T20:13:10Z [132] WARNING {startup} obsolete option '--wal.ignore-recovery-errors' used in configuration. setting this option will not have any effect.
2021-03-16T20:13:10Z [132] INFO [e52b0] ArangoDB 3.7.3 [linux] 64bit, using jemalloc, build tags/v3.7.3-0-g8ad8edb71f4, VPack 0.1.33, RocksDB 6.8.0, ICU 64.2, V8 7.9.317, OpenSSL 1.1.1g  21 Apr 2020
2021-03-16T20:13:10Z [132] INFO [75ddc] detected operating system: Linux version 4.18.0-240.10.1.el8_3.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 8.3.1 20191121 (Red Hat 8.3.1-5) (GCC)) #1 SMP Mon Jan 18 17:05:51 UTC 2021
2021-03-16T20:13:10Z [132] INFO [25362] {memory} Available physical memory: 16458944512 bytes, available cores: 4
2021-03-16T20:13:10Z [132] WARNING [118b0] {memory} maximum number of memory mappings per process is 65530, which seems too low. it is recommended to set it to at least 256000
2021-03-16T20:13:10Z [132] WARNING [49528] {memory} execute 'sudo sysctl -w "vm.max_map_count=256000"'
2021-03-16T20:13:10Z [132] WARNING [e8b68] {memory} /sys/kernel/mm/transparent_hugepage/enabled is set to 'always'. It is recommended to set it to a value of 'never' or 'madvise'
2021-03-16T20:13:10Z [132] WARNING [f3108] {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/enabled"'
2021-03-16T20:13:10Z [132] WARNING [f3108] {memory} execute 'sudo bash -c "echo madvise > /sys/kernel/mm/transparent_hugepage/defrag"'
2021-03-16T20:13:10Z [132] INFO [43396] {authentication} Jwt secret not specified, generating...
2021-03-16T20:13:10Z [132] INFO [144fe] using storage engine 'rocksdb'
2021-03-16T20:13:10Z [132] INFO [3bb7d] {cluster} Starting up with role SINGLE
2021-03-16T20:13:10Z [132] INFO [6ea38] using endpoint 'http+tcp://0.0.0.0:8529' for non-encrypted requests
2021-03-16T20:13:10Z [132] INFO [a1c60] {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2021-03-16T20:13:10Z [132] INFO [3844e] {authentication} Authentication is turned on (system only), authentication for unix sockets is turned on
2021-03-16T20:13:10Z [132] FATAL [fe3df] {startup} unable to initialize RocksDB engine: Corruption: SST file is ahead of WALs
@maxkernbach
Copy link
Contributor
maxkernbach commented Mar 17, 2021 •
edited
Loading

Hi @mrinc,

please start the affected server with the option --rocksdb.wal-recovery-skip-corrupted true, which is ava 8000 ilable in 3.7 and let us know whether this resolves the issue.

arangod --help-. | grep corrupted
--rocksdb.wal-recovery-skip-corrupted <boolean>              skip corrupted records in WAL recovery (default: false)

@mrinc
Copy link
Author
mrinc commented Mar 17, 2021

Looks good, all back up and running

Thanks @maxkernbach

@mrinc mrinc closed this as completed Mar 17, 2021
@maxkernbach maxkernbach added 2 Solved Resolution and removed Waiting User Reply labels Mar 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
1 Analyzing 2 Solved Resolution 3 RocksDB Storage engine related
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mrinc @maxkernbach
0