10000 Bug fix/fixes 0609 by jsteemann · Pull Request #3227 · arangodb/arangodb · GitHub
[go: up one dir, main page]

Skip to content

Bug fix/fixes 0609 #3227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 55 commits into from
Sep 13, 2017
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
c080632
do not use V8 variant of AQL functions in early optimization stage wh…
jsteemann Sep 6, 2017
358356c
added missing space in warning message
jsteemann Sep 6, 2017
b23b85f
make line-counting more efficient
jsteemann Sep 6, 2017
512f549
removed unused includes
jsteemann Sep 6, 2017
b6ebf9d
remove duplicate V8 context number control variables
jsteemann Sep 6, 2017
03ed4f8
warn when more than 90% of max mappings are in use
jsteemann Sep 6, 2017
efb3a17
added C++ variant of replication catchup
jsteemann Sep 6, 2017
cdfc90c
speed up test
jsteemann Sep 6, 2017
a6d1923
speed up test
jsteemann Sep 6, 2017
2b72b90
speed up test
jsteemann Sep 7, 2017
db9bb73
make ctor explicit
jsteemann Sep 7, 2017
7965c94
V8 macro cleanup, StringBuffer optimizations
jsteemann Sep 7, 2017
d2bbd0d
Merge branch 'devel' of https://github.com/arangodb/arangodb into bug…
jsteemann Sep 7, 2017
4fe6779
fix a deadlock
jsteemann Sep 7, 2017
e57e2ef
added `--log.role` option
jsteemann Sep 7, 2017
7a0151e
updated CHANGELOG
jsteemann Sep 7, 2017
024327a
Merge branch 'devel' of https://github.com/arangodb/arangodb into bug…
jsteemann Sep 7, 2017
92f398f
removed useless vector for authentication...
jsteemann Sep 7, 2017
cd03efb
removed non-existing scheduler.threads option from config
jsteemann Sep 7, 2017
9bc0bab
removed several occurrences of TRI_CORE_MEM_ZONE
jsteemann Sep 7, 2017
17be850
Merge branch 'devel' of https://github.com/arangodb/arangodb into bug…
jsteemann Sep 7, 2017
967f0e0
jslint
jsteemann Sep 7, 2017
e13bbd5
fix include order
jsteemann Sep 7, 2017
90b5915
Merge branch 'devel' of https://github.com/arangodb/arangodb into bug…
jsteemann Sep 7, 2017
86e29db
remove JavaScript-based replication remainders
jsteemann Sep 7, 2017
5c4aa5a
use correct database
jsteemann Sep 7, 2017
44fdadb
some fixes for replication
jsteemann Sep 7, 2017
d2f86e8
removed useless __FILE__, __LINE__ invocations
jsteemann Sep 8, 2017
f0cd8a2
micro optimizations
jsteemann Sep 8, 2017
2d3dc23
joyful colorized logs
jsteemann Sep 8, 2017
88bccfe
Merge branch 'devel' of https://github.com/arangodb/arangodb into bug…
jsteemann Sep 8, 2017
fbc66bd
fix typo
jsteemann Sep 8, 2017
95f13a6
port some of the non-yet-ready cluster optimization rules
jsteemann Sep 8, 2017
3bda7d0
updated CHANGELOG
jsteemann Sep 8, 2017
6140c3e
allow a priority V8 context
jsteemann Sep 8, 2017
a163c3f
use functions to generate V8 strings, so we can add error reporting t…
jsteemann Sep 8, 2017
3620043
fix assertion failure
jsteemann Sep 11, 2017
229b90d
Merge branch 'devel' of https://github.com/arangodb/arangodb into bug…
jsteemann Sep 11, 2017
5b217e7
updated CHANGELOG
jsteemann Sep 12, 2017
fff754c
updated CHANGELOG
jsteemann Sep 12, 2017
e3a1931
remove TRI_CORE_MEM_ZONE
jsteemann Sep 12, 2017
a6f674b
Merge branch 'devel' of https://github.com/arangodb/arangodb into bug…
jsteemann Sep 12, 2017
63c24f1
try to fix Windows errors & warnings
jsteemann Sep 12, 2017
f8a4832
cleanup
jsteemann Sep 12, 2017
ce160c7
Merge branch 'devel' of https://github.com/arangodb/arangodb into bug…
jsteemann Sep 12, 2017
aae4846
removed memory zones altogether
jsteemann Sep 13, 2017
6f1086c
Merge branch 'devel' of https://github.com/arangodb/arangodb into bug…
jsteemann Sep 13, 2017
4e7ba81
Merge branch 'devel' of https://github.com/arangodb/arangodb into bug…
jsteemann Sep 13, 2017
ecd866f
fix Visual Studio warnings
jsteemann Sep 13, 2017
f2ffc27
fix MSVC compile error
jsteemann Sep 13, 2017
ca29268
micro optimizations
jsteemann Sep 13, 2017
55a8c65
try to fix compile error
jsteemann Sep 13, 2017
125d126
revert some potential performance improvements. does not compile ever…
jsteemann Sep 13, 2017
f4e86b9
wait dynamically
jsteemann Sep 13, 2017
a556dcb
exclude system collections from collection tests
jsteemann Sep 13, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
some fixes for replication
  • Loading branch information
jsteemann committed Sep 7, 2017
commit 44fdadb79e4bf38d07b826a8f85e3a28b6e5afcb
47 changes: 20 additions & 27 deletions arangod/Replication/ContinuousSyncer.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -511,13 +511,13 @@ int ContinuousSyncer::processDocument(TRI_replication_operation_e type,
return TRI_ERROR_REPLICATION_INVALID_RESPONSE;
}

VPackBuilder builder;
builder.openObject();
builder.add(StaticStrings::KeyString, key);
builder.add(StaticStrings::RevString, rev);
builder.close();
_documentBuilder.clear();
_documentBuilder.openObject();
_documentBuilder.add(StaticStrings::KeyString, key);
_documentBuilder.add(StaticStrings::RevString, rev);
_documentBuilder.close();

VPackSlice const old = builder.slice();
VPackSlice const old = _documentBuilder.slice();

// extract "tid"
std::string const transactionId = VelocyPackHelper::getStringValue(slice, "tid", "");
Expand Down Expand Up @@ -881,6 +881,8 @@ int ContinuousSyncer::applyLog(SimpleHttpResult* response,

// buffer must end with a NUL byte
TRI_ASSERT(*end == '\0');

auto builder = std::make_shared<VPackBuilder>();

while (p < end) {
char const* q = strchr(p, '\n');
Expand All @@ -901,8 +903,7 @@ int ContinuousSyncer::applyLog(SimpleHttpResult* response,

processedMarkers++;

auto builder = std::make_shared<VPackBuilder>();

builder->clear();
try {
VPackParser parser(builder);
parser.parse(p, static_cast<size_t>(q - p));
Expand All @@ -927,7 +928,8 @@ int ContinuousSyncer::applyLog(SimpleHttpResult* response,

//LOG_TOPIC(ERR, Logger::FIXME) << slice.toJson();

if (skipMarker(firstRegularTick, slice)) {
if (!_transientApplierState &&
skipMarker(firstRegularTick, slice)) {
// entry is skipped
res = TRI_ERROR_NO_ERROR;
skipped = true;
Expand Down Expand Up @@ -1530,8 +1532,7 @@ int ContinuousSyncer::syncCollectionFinalize(std::string& errorMsg,
"/logger-follow?chunkSize=" + _chunkSize +
"&from=" + StringUtils::itoa(fetchTick) +
"&serverId=" + _localServerIdString +
"&collection=" + StringUtils::urlEncode(collectionName) +
"&includeSystem=true";
"&collection=" + StringUtils::urlEncode(collectionName);

// send request
std::string const progress =
Expand Down Expand Up @@ -1560,32 +1561,24 @@ int ContinuousSyncer::syncCollectionFinalize(std::string& errorMsg,
return TRI_ERROR_REPLICATION_MASTER_ERROR;
}

bool checkMore = false;
if (response->getHttpReturnCode() == 204) {
// No content: this means we are done
return TRI_ERROR_NO_ERROR;
}

bool found;
std::string header =
response->getHeaderField(TRI_REPLICATION_HEADER_CHECKMORE, found);

bool checkMore = false;
if (found) {
checkMore = StringUtils::boolean(header);
} else {
errorMsg = "got invalid response from master at " +
std::string(_masterInfo._endpoint) +
": required header is missing";
return TRI_ERROR_REPLICATION_INVALID_RESPONSE;
}
}

header =
response->getHeaderField(TRI_REPLICATION_HEADER_LASTINCLUDED, found);
if (found) {
TRI_voc_tick_t lastIncludedTick = StringUtils::uint64(header);

if (lastIncludedTick > fetchTick) {
fetchTick = lastIncludedTick;
} else {
// we got the same tick again, this indicates we're at the end
checkMore = false;
}
fetchTick = StringUtils::uint64(header);
} else {
errorMsg = "got invalid response from master at " +
std::string(_masterInfo._endpoint) +
Expand All @@ -1601,7 +1594,7 @@ int ContinuousSyncer::syncCollectionFinalize(std::string& errorMsg,
return res;
}

if (!checkMore || processedMarkers == 0) {
if (!checkMore) { // || processedMarkers == 0) { // TODO: check if we need this!
// done!
return TRI_ERROR_NO_ERROR;
}
Expand Down
7 changes: 6 additions & 1 deletion arangod/Replication/ContinuousSyncer.h
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,11 @@

#include "Basics/Common.h"
#include "Replication/Syncer.h"
#include "VocBase/replication-applier.h"

#include <velocypack/Builder.h>

struct TRI_vocbase_t;
class TRI_replication_applier_t;

namespace arangodb {

Expand Down Expand Up @@ -182,6 +184,9 @@ class ContinuousSyncer : public Syncer {
/// @brief which transactions were open and need to be treated specially
std::unordered_map<TRI_voc_tid_t, ReplicationTransaction*>
_ongoingTransactions;

/// @brief recycled builder for repeated document creation
arangodb::velocypack::Builder _documentBuilder;
};
}

Expand Down
5 changes: 4 additions & 1 deletion js/server/modules/@arangodb/cluster.js
Original file line number Diff line number Diff line change
Expand Up @@ -589,7 +589,9 @@ function synchronizeOneShard (database, shard, planId, leader) {
database,
collection: shard,
leaderId: leader,
from: sy.lastLogTick
from: sy.lastLogTick,
reques 57AE tTimeout: 60,
connectTimeout: 60
});

try {
Expand All @@ -599,6 +601,7 @@ function synchronizeOneShard (database, shard, planId, leader) {
throw err4;
}
} catch (err3) {
ok = false;
console.topic('heartbeat=error', 'synchronizeOneshard: exception in',
'syncCollectionFinalize:', err3);
}
Expand Down
CAE
0