MohammedDeveloper
diff --git a/‎CHANGELOG
Lines changed: 2 additions & 0 deletions b/‎CHANGELOG
Lines changed: 2 additions & 0 deletions
diff --git a/‎Documentation/Books/Manual/Administration/Configuration/Compaction.md
Lines changed: 116 additions & 0 deletions b/‎Documentation/Books/Manual/Administration/Configuration/Compaction.md
Lines changed: 116 additions & 0 deletions
diff --git a/‎Documentation/Books/Manual/SUMMARY.md
Lines changed: 1 addition & 0 deletions b/‎Documentation/Books/Manual/SUMMARY.md
Lines changed: 1 addition & 0 deletions
diff --git a/‎arangod/MMFiles/CMakeLists.txt
Lines changed: 1 addition & 0 deletions b/‎arangod/MMFiles/CMakeLists.txt
Lines changed: 1 addition & 0 deletions
diff --git a/‎arangod/MMFiles/MMFilesCollection.cpp
Lines changed: 23 additions & 15 deletions b/‎arangod/MMFiles/MMFilesCollection.cpp
Lines changed: 23 additions & 15 deletions
diff --git a/‎arangod/MMFiles/MMFilesCompactionFeature.cpp
Lines changed: 122 additions & 0 deletions b/‎arangod/MMFiles/MMFilesCompactionFeature.cpp
Lines changed: 122 additions & 0 deletions
@@ -1,6 +1,8 @@
 devel
 -----
 
+* make the mmfiles compactor configurable
+
 * leader-follower replication catchup code has been rewritten in C++
 
 * early stage AQL optimization now also uses the C++ implementations of
 
@@ -0,0 +1,116 @@
+MMFiles Compaction options
+==========================
+
+The ArangoDB MMFiles storage engine will run a compaction over data files.
+
+ArangoDB writes Documents in the WAL file. Once they have been sealed in the wal file,
+the collector may copy them into a per collection journal file. 
+
+Once journal files fill up, they're sealed to become data files.
+
+=> one collection may have documents in the WAL logs, its journal file, and an arbitrary 
+number of data files.
+
+If a collection is loaded, each of these files are opened (thus use a file handle) and 
+are mmap'ed. Since file handles and memory mapped files are also a sparse resource, 
+that number should be kept low.
+
+Once you update or remove documents from data files (or already did while it was the journal file)
+these documents are marked as 'dead' with a deletion marker.
+
+Over time the number of dead documents may rise, and we don't want to use the previously mentioned
+resources, plus the disk space should be given back to the system.
+Thus several journal files can be combined to one, ommitting the dead documents.
+
+Combining several of these data files into one is called compaction. The compaction process reads
+the alive documents from the original data files, and writes them into new data file.
+
+Once that is done, the memory mappings to the old data files is released, and the files are erased.
+
+Since the compaction locks the collection, and also uses I/O resources, its carefully configurable
+under which conditions the system should perform which amount of these compaction jobs:
+
+ArangoDB spawns one compactor thread per database. The settings below vary in scope. 
+
+
+# Activity control
+
+The activity control parameters alter the behaviour in terms of scan / execution frequency
+of the compaction.
+
+Sleep interval between two compaction runs (in seconds):
+`--compaction.db-sleep-time`
+The number of seconds the collector thread will wait between two attempts to search for 
+compactable data files of collections in one Database. If the compactor has actually executed work,
+a subsequent lookup is done. Scope: Database.
+
+Minimum sleep time between two compaction runs (in seconds):
+`--compaction.min-interval`
+When an actual compaction was executed for one collection, we wait for this time
+before we execute the compaction on this collection again.
+This is here to let eventually piled up user load be worked out. Scope: collection.
+
+
+# Source data files
+These parameters control which data files are taken into account for a compaction run.
+You can specify several criteria which each off may be sufficcient alone.
+
+The scan over the data files belonging to one collection is executed from oldest
+data file to newest; if files qualify for a compaction they may be merged with newer files
+(containing younger documents)
+
+Scope: Collection level, some are influenced by collection settings.
+
+minimal filesize threshold original data files have to be below for a compaction:
+`--compaction.min-small-data-file-size`
+This is the threshold which controls below which minimum total size a data file
+will always be taken into account for the compaction.
+
+Minimum unused count of documents in a datafile:
+`--compaction.dead-documents-threshold`
+Data files will often contain dead documents. This parameter specifies their
+top most accetpeable count until the data file qualifies for compaction.
+
+How many bytes of the source data file are allowed to be unused at most:
+`--compaction.dead-size-threshold`
+The dead data size varies along with the size of your documents.
+If you have many big documents, this threshold may hit before the document count threshold.
+
+How many percent of the source data file should be unused at least:
+`--compaction.dead-size-percent-threshold`
+since the size of the documents may vary this threshold works on the 
+percentage of the dead documents size. Thus, if you have many huge
+dead documents, this threshold kicks in earlier. 
+
+To name an example with numbers, if the data file contains 800 kbytes of alive
+and 400 kbytes of dead documents, the share of the dead documents is:
+
+`400 / (400 + 800) = 33 %`.
+
+If this value if higher than the specified threshold, the data file will be compacted.
+
+
+# Compacted target files
+Once data files of a collection are qualified for a compaction run, these parameters control 
+how many data files are merged into one, (or even one source data file may be compacted into
+one smaller target data file)
+
+Scope: Collection level, some are influenced by collection settings.
+
+Maximum number of files to merge to one file:
+`--compaction.dest-max-files`
+How many data files (at most) we may merge into one resulting data file during one compaction run.
+
+How large the resulting file may be in comparison to the collections `database.maximal-journal-size` setting:
+`--compaction.dest-max-file-size-factor`
+In ArangoDB you can configure a default `journal filesize` globally and override it on a per collection level.
+This value controls the size of collected data files relative to the configured journal file size of the collection in question.
+
+A factor of 3 means that the maximum filesize of the compacted file is
+3 times the size of the maximum collection journal file size.
+
+how large may the compaction result file become:
+`--compaction.dest-max-result-file-size`
+next to the factor above, a totally maximum allowed filesize in bytes may be specified.
+This will overrule all previous parameters. 
+
@@ -162,6 +162,7 @@
     * [Logging Options](Administration/Configuration/Logging.md)
     * [General Options](Administration/Configuration/GeneralArangod.md)
     * [Write-Ahead Log Options](Administration/Configuration/Wal.md)
+    * [Compaction Options](Administration/Configuration/Compaction.md)
     * [Cluster Options](Administration/Configuration/Cluster.md)
     * [RocksDB Engine Options](Administration/Configuration/RocksDB.md)
     * [Hash Cache Options](Administration/Configuration/Cache.md)
 
@@ -13,6 +13,7 @@ set(MMFILES_SOURCES
   MMFiles/MMFilesCollectionExport.cpp
   MMFiles/MMFilesCollectionKeys.cpp
   MMFiles/MMFilesCollectorThread.cpp
+  MMFiles/MMFilesCompactionFeature.cpp
   MMFiles/MMFilesCompactorThread.cpp
   MMFiles/MMFilesDatafile.cpp
   MMFiles/MMFilesDatafileStatistics.cpp
 
@@ -1242,12 +1242,6 @@ void MMFilesCollection::figuresSpecific(
     strftime(&lastCompactionStampString[0], sizeof(lastCompactionStampString),
              "%Y-%m-%dT%H:%M:%SZ", &tb);
   }
-
-  builder->add("compactionStatus", VPackValue(VPackValueType::Object));
-  builder->add("message", VPackValue(lastCompactionStatus));
-  builder->add("time", VPackValue(&lastCompactionStampString[0]));
-  builder->close();  // compactionStatus
-
   builder->add("documentReferences",
                VPackValue(_ditches.numMMFilesDocumentMMFilesDitches()));
 
@@ -1257,17 +1251,31 @@ void MMFilesCollection::figuresSpecific(
 
   // add datafile statistics
   MMFilesDatafileStatisticsContainer dfi = _datafileStatistics.all();
+  MMFilesDatafileStatistics::CompactionStats stats = _datafileStatistics.getStats();
 
-  builder->add("alive", VPackValue(VPackValueType::Object));
-  builder->add("count", VPackValue(dfi.numberAlive));
-  builder->add("size", VPackValue(dfi.sizeAlive));
-  builder->close();  // alive
+  builder->add("alive", VPackValue(VPackValueType::Object)); {
+    builder->add("count", VPackValue(dfi.numberAlive));
+    builder->add("size", VPackValue(dfi.sizeAlive));
+    builder->close();  // alive
+  }
 
-  builder->add("dead", VPackValue(VPackValueType::Object));
-  builder->add("count", VPackValue(dfi.numberDead));
-  builder->add("size", VPackValue(dfi.sizeDead));
-  builder->add("deletion", VPackValue(dfi.numberDeletions));
-  builder->close();  // dead
+  builder->add("dead", VPackValue(VPackValueType::Object)); {
+    builder->add("count", VPackValue(dfi.numberDead));
+    builder->add("size", VPackValue(dfi.sizeDead));
+    builder->add("deletion", VPackValue(dfi.numberDeletions));
+    builder->close();  // dead
+  }
+
+  builder->add("compactionStatus", VPackValue(VPackValueType::Object)); {
+    builder->add("message", VPackValue(lastCompactionStatus));
+    builder->add("time", VPackValue(&lastCompactionStampString[0]));
+
+    builder->add("count", VPackValue(stats._compactionCount));
+    builder->add("filesCombined", VPackValue(stats._filesCombined));
+    builder->add("bytesRead", VPackValue(stats._compactionBytesRead));
+    builder->add("bytesWritten", VPackValue(stats._compactionBytesWritten));
+    builder->close();  // compactionStatus
+  }
 
   // add file statistics
   READ_LOCKER(readLocker, _filesLock);
 
@@ -0,0 +1,122 @@
+////////////////////////////////////////////////////////////////////////////////
+/// DISCLAIMER
+///
+/// Copyright 2014-2016 ArangoDB GmbH, Cologne, Germany
+/// Copyright 2004-2014 triAGENS GmbH, Cologne, Germany
+///
+/// Licensed under the Apache License, Version 2.0 (the "License");
+/// you may not use this file except in compliance with the License.
+/// You may obtain a copy of the License at
+///
+///     http://www.apache.org/licenses/LICENSE-2.0
+///
+/// Unless required by applicable law or agreed to in writing, software
+/// distributed under the License is distributed on an "AS IS" BASIS,
+/// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+/// See the License for the specific language governing permissions and
+/// limitations under the License.
+///
+/// Copyright holder is ArangoDB GmbH, Cologne, Germany
+///
+/// @author Wilfried Goesgens
+////////////////////////////////////////////////////////////////////////////////
+
+#include "MMFilesCompactionFeature.h"
+
+#include "Basics/Exceptions.h"
+#include "Logger/Logger.h"
+#include "ProgramOptions/ProgramOptions.h"
+#include "ProgramOptions/Section.h"
+#include "MMFiles/MMFilesLogfileManager.h"
+#include "RestServer/DatabaseFeature.h"
+
+using namespace arangodb;
+using namespace arangodb::application_features;
+using namespace arangodb::basics;
+using namespace arangodb::options;
+
+MMFilesCompactionFeature* MMFilesCompactionFeature::COMPACTOR = nullptr;
+
+MMFilesCompactionFeature::MMFilesCompactionFeature(ApplicationServer* server)
+  : ApplicationFeature(server, "MMFilesCompaction"),
+    _compactionSleepTime(1),
+    _compactionCollectionInterval(10.0), 
+    _maxFiles(3),
+    _maxSizeFactor(3),
+    _smallDatafileSize(128 * 1024),
+    _maxResultFilesize(128 * 1024 * 1024),
+    _deadNumberThreshold(16384),
+    _deadSizeThreshold(128 * 1024),
+    _deadShare(0.1) { 
+  setOptional(true);
+  requiresElevatedPrivileges(false);
+  onlyEnabledWith("MMFilesEngine");
+  MMFilesCompactionFeature::COMPACTOR = this;
+}
+
+
+void MMFilesCompactionFeature::collectOptions(std::shared_ptr<options::ProgramOptions> options) {
+  options->addSection("compaction", "Configure the MMFiles compactor thread");
+  
+  options->addOption("--compaction.db-sleep-time", "sleep interval between two compaction runs (in s)",
+                     new UInt64Parameter(&_compactionSleepTime));
+  
+  options->addOption("--compaction.min-interval", "minimum sleep time between two compaction runs (in s)",
+                     new DoubleParameter(&_compactionCollectionInterval));
+
+
+  options->addOption("--compaction.min-small-data-file-size", "minimal filesize threshhold original data files have to be below for a compaction",
+                     new UInt64Parameter(&_smallDatafileSize));
+  
+  options->addOption("--compaction.dead-documents-threshold", "minimum unused count of documents in a datafile",
+                     new UInt64Parameter(&_deadNumberThreshold));
+  
+  options->addOption("--compaction.dead-size-threshold", "how many bytes of the source data file are allowed to be unused at most",
+                     new UInt64Parameter(&_deadSizeThreshold));
+  
+  options->addOption("--compaction.dead-size-percent-threshold", "how many percent of the source datafile should be unused at least",
+                     new DoubleParameter(&_deadShare));
+
+
+  options->addOption("--compaction.max-files", "Maximum number of files to merge to one file",
+                     new UInt64Parameter(&_maxFiles));
+
+  options->addOption("--compaction.max-result-file-size", "how large may the compaction result file become (in bytes)",
+                     new UInt64Parameter(&_maxResultFilesize));
+
+  options->addOption("--compaction.max-file-size-factor", "how large the resulting file may be in comparison to the collections '--database.maximal-journal-size' setting",
+                     new UInt64Parameter(&_maxSizeFactor));
+}
+
+void MMFilesCompactionFeature::validateOptions(std::shared_ptr<options::ProgramOptions> options) {
+  if (_deadNumberThreshold < 1024) {
+    LOG_TOPIC(WARN, Logger::COMPACTOR)
+      << "compaction.dead-documents-threshold should be at least 1024.";
+    _deadNumberThreshold = 1024;
+  }
+  
+  if (_deadSizeThreshold < 10240) {
+    LOG_TOPIC(WARN, Logger::COMPACTOR)
+      << "compaction.dead-size-threshold should be at least 10k.";
+    _deadNumberThreshold = 10240;
+  }
+
+  if (_deadShare < 0.001) {
+    LOG_TOPIC(WARN, Logger::COMPACTOR)
+      << "compaction.dead-size-percent-threshold should be at least 0.001%.";
+    _deadShare = 0.01;
+  }
+
+  if (_maxResultFilesize < TRI_JOURNAL_MINIMAL_SIZE) {
+    LOG_TOPIC(WARN, Logger::COMPACTOR)
+      << "compaction.max-result-file-size should be at least: " << TRI_JOURNAL_MINIMAL_SIZE;
+    _maxResultFilesize = TRI_JOURNAL_MINIMAL_SIZE;
+  }
+
+  if (_maxSizeFactor < 1) {
+    LOG_TOPIC(WARN, Logger::COMPACTOR)
+      << "compaction.max-file-size-factor should be at least: 1";
+    _maxSizeFactor = 1;
+  }
+
+}