English | 简体中文
NutsDB is a simple, fast, embeddable and persistent key/value store written in pure Go.
It supports fully serializable transactions and many data structures such as list、set、sorted set. All operations happen inside a Tx. Tx represents a transaction, which can be read-only or read-write. Read-only transactions can read values for a given bucket and a given key or iterate over a set of key-value pairs. Read-write transactions can read, update and delete keys from the DB.
We can learn more about NutsDB in details on the documents site of NutsDB: NutsDB Documents
- v1.0.0 release, see for details: https://github.com/nutsdb/nutsdb/releases/tag/v1.0.0
- v0.14.3 release, see for details: https://github.com/nutsdb/nutsdb/releases/tag/v0.14.3
- v0.14.2 release, see for details: https://github.com/nutsdb/nutsdb/releases/tag/v0.14.2
- v0.14.1 release, see for details: https://github.com/nutsdb/nutsdb/releases/tag/v0.14.1
📢 Note: Starting from v0.9.0, defaultSegmentSize in DefaultOptions has been adjusted from 8MB to 256MB. The original value is the default value, which needs to be manually changed to 8MB, otherwise the original data will not be parsed. The reason for the size adjustment here is that there is a cache for file descriptors starting from v0.9.0 (detail see #164 ), so users need to look at the number of fds they use on the server, which can be set manually. If you have any questions, you can open an issue.
After nutsdb v1.0.0, due to changes in the underlying data storage protocol, the data of the old version is not compatible. Please rewrite it before using the new version. And the current Bucket needs to be created manually. Please see the Bucket usage documentation for details.
Welcome contributions to NutsDB.
To start using NutsDB, first needs Go installed (version 1.18+ is required). and run go get:
go get -u github.com/nutsdb/nutsdb
To open your database, use the nutsdb.Open() function,with the appropriate options.The Dir , EntryIdxMode and SegmentSize options are must be specified by the client. About options see here for detail.
package main
import (
"log"
"github.com/nutsdb/nutsdb"
)
func main() {
// Open the database located in the /tmp/nutsdb directory.
// It will be created if it doesn't exist.
db, err := nutsdb.Open(
nutsdb.DefaultOptions,
nutsdb.WithDir("/tmp/nutsdb"),
)
if err != nil {
log.Fatal(err)
}
defer db.Close()
...
}NutsDB introduces Merge V2 and HintFile features to significantly improve database performance, especially for startup time and memory efficiency during large-scale compaction operations.
Merge V2 is a high-performance compaction algorithm that optimizes memory usage while maintaining data consistency. It's designed to handle large-scale data compaction efficiently.
- Memory Efficiency: Reduces memory usage by ~65% during merge operations (from ~145 bytes/entry to ~50 bytes/entry)
- Concurrent Safety: Allows concurrent writes during merge operations
- Atomic Operations: Ensures data consistency through atomic commit and rollback
- Large-Scale Support: Optimized for handling 10GB+ datasets efficiently
- Preparation Phase: Enumerates files and validates merge state
- Rewrite Phase: Processes data files and rewrites valid entries to new merge files
- Commit Phase: Updates in-memory indexes and writes hint files
- Finalization: Ensures all data is persisted and cleans up old files
Merge V2 uses negative FileIDs for merge files to ensure correct processing order:
- Merge files: Negative IDs (starting from
math.MinInt64) - Normal data files: Positive IDs (starting from 0)
- Processing order: Merge files are always processed before normal files during index rebuild
// Merge V2 is enabled by default and works automatically
// You can configure merge behavior through options
options := nutsdb.DefaultOptions
options.SegmentSize = 256 * 1024 * 1024 // 256MB segments
db, err := nutsdb.Open(options, nutsdb.WithDir("/tmp/nutsdb"))HintFile is an index persistence feature that dramatically reduces database startup time by maintaining persistent indexes of key metadata.
- Fast Startup: Eliminates full data file scans during database recovery
- Automatic Fallback: Gracefully falls back to traditional scanning if hint files are missing or corrupted
- Transparent Operation: Works seamlessly with existing data structures
- Configurable: Can be enabled/disabled via configuration
- During Merge: Hint files are automatically created alongside merge data files
- During Startup: Database loads hint files to reconstruct in-memory indexes instantly
- Fallback Mechanism: If hint files are unavailable, the system falls back to scanning data files
Each hint entry contains:
- Bucket ID, Key metadata, Value size
- Timestamp, TTL, and operation flags
- File ID and data position for direct access
- Key data for lookup operations
// Basic configuration with both features enabled
options := nutsdb.DefaultOptions
options.EnableHintFile = true
options.EnableMergeV2 = true
options.SegmentSize = 256 * 1024 * 1024 // 256MB segments
// Or use option functions
db, err := nutsdb.Open(
nutsdb.DefaultOptions,
nutsdb.WithDir("/tmp/nutsdb"),
nutsdb.WithEnableHintFile(true),
nutsdb.WithEnableMergeV2(true),
)| Option | Type | Default | Description |
|---|---|---|---|
EnableHintFile |
bool | false |
Enable/disable hint file feature |
EnableMergeV2 |
bool | false |
Enable/disable Merge V2 algorithm |
SegmentSize |
int64 | 256MB |
Size of data files before rotation |
- Traditional Merge: ~145 bytes per entry
- Merge V2: ~50 bytes per entry
- Savings: ~65% memory reduction for large datasets
- With HintFile: Index reconstruction in milliseconds
- Without HintFile: Full data file scanning (seconds to minutes)
- Fallback: Automatic degradation if hint files are unavailable
Performance testing with Intel Core i5-14600KF on large datasets:
| Configuration | Total Time | Memory Usage | Allocations | Avg Restart | Merge Time |
|---|---|---|---|---|---|
| Merge V1 + No HintFile | 4.29s | 2.32GB | 40.4M | 4.27s | 12.86s |
| Merge V1 + HintFile | 1.69s | 1.56GB | 25.4M | 1.64s | 16.37s |
| Merge V2 + HintFile | 1.87s | 1.56GB | 25.4M | 1.83s | 8.63s |
- HintFile Impact: 61.6% faster startup time (4.27s → 1.64s)
- Memory Optimization: 32.9% reduction in memory allocation with HintFile
- Merge V2 Efficiency: 47.3% faster merge operations (16.37s → 8.63s)
- Overall Performance: Best balance with Merge V2 + HintFile combination
- Large Datasets: Essential for databases with millions of entries
- Frequent Restarts: Critical for applications requiring fast restarts
- Memory-Constrained Environments: Reduces peak memory usage during maintenance
- High Availability: Enables quick database recovery after failures
- Enable HintFile for production workloads with large datasets
- Monitor Disk Space - Hint files add additional storage overhead
- Regular Merges - Schedule merges during low-traffic periods
- Backup Strategy - Include hint files in your backup process
// Step 1: Backup your database
// Step 2: Enable HintFile and perform a merge to create hint files
options := nutsdb.DefaultOptions
options.EnableHintFile = true
options.EnableMergeV2 = true
db, err := nutsdb.Open(options)
if err != nil {
log.Fatal(err)
}
// Step 3: Perform a merge to generate hint files
if err := db.Merge(); err != nil {
log.Fatal(err)
}
// Step 4: Restart database - hint files will be used for fast startup
db.Close()
db, err = nutsdb.Open(options)// Simply set EnableHintFile to false
options := nutsdb.DefaultOptions
options.EnableHintFile = false
options.EnableMergeV2 = true // Keep Merge V2 if desired
// Hint files will be ignored but not automatically deleted
// You can manually delete .hint files if you want to reclaim space-
HintFile Corruption
Symptom: Database startup fails or takes long time Solution: NutsDB automatically falls back to scanning data files -
Merge V2 Memory Usage
Symptom: High memory usage during merge operations Solution: Reduce SegmentSize or disable HintFile temporarily -
Slow Startup with Large Dataset
Symptom: Database takes minutes to start Solution: Enable HintFile and perform a manual merge
// Check database health
db, err := nutsdb.Open(options)
if err != nil {
// Try disabling HintFile if corruption suspected
options.EnableHintFile = false
db, err = nutsdb.Open(options)
}
// Force rebuild hint files
if err := db.Merge(); err != nil {
log.Printf("Merge failed: %v", err)
}- Backward Compatible: Works with existing NutsDB databases
- Configurable: HintFile can be disabled if not needed
- Data Structures: Supports all NutsDB data structures (BTree, Set, List, SortedSet)
- Graceful Degradation: Automatic fallback when hint files are unavailable
Buckets
Pairs
Iterator
Data Structures
Database Options
Advanced Features
Comparison
Benchmark
Thank you for considering contributing to NutsDB! The contribution guide can be found in the CONTRIBUTING for details on submitting patches and the contribution workflow.
This package is inspired by the following:
The NutsDB is open-sourced software licensed under the Apache 2.0 license.