arangodb · ansoboleva · Jun 29, 2022 · May 17, 2022 · May 19, 2022 · May 20, 2022
diff --git a/3.10/http/general.md b/3.10/http/general.md
@@ -611,3 +611,41 @@ receiving Coordinator will also check the header on its own.
 Apart from that, the header will not be included in cluster-internal requests
 executed by the Coordinator, e.g. when the Coordinator issues sub-requests
 to DB-Servers or Agency instances.
+
+Responding to liveliness probes
+-------------------------------
+
+The HTTP REST interface of arangod instances can optionally be started early or late 
+during the startup process, depending on what is preferred. ArangoDB versions
+before 3.10 only allowed starting the interface late, after recovery was finished.
+
+By default, the HTTP REST interface of an instance is opened late during the startup
+sequence, as in all previous versions. With this configuration, an instance will respond 
+with HTTP 503 (Service unavailable) until all REST APIs are available and usable.
+
+Since ArangoDB 3.10, the HTTP REST interface can optionally be opened early in the 
+startup sequence by setting the startup option `--server.early-connections` to `true`. 
+This configuration allows an instance to respond to a limited set of REST APIs 
+during startup, even during recovery. This can be useful because the recovery procedure 
+can take time proportional to the amount of data to recover.
+
+When the `--server.early-connections` option is set to `true`, the instance will be
+able to respond to requests to the following REST APIs e
8000
arly on:
+
+- GET `/_api/version` and `/_admin/version`: these APIs return the server version 
+  number, but can also be used as a lifeliness probe, to check if the instance is
+  responding to incoming HTTP requests.
+- GET `/_admin/status`: this API returns information about the instance's status, now
+  also including recovery progress and information about which server feature is
+  currently starting.
+
+During the early startup phase, all other APIs than the ones listed above will be 
+responded to with an HTTP response code 503, so that callers can see that the instance 
+is not fully ready. In addition the `maintenance` attribute in the response to GET
+`/_admin/status` requests can be checked for general instance readiness.
+
+If authentication is used, then only JWT authentication can be used during the early 
+startup phase. Incoming requests relying on other authentication mechanisms that 
+require access to the database data (e.g. HTTP basic authentication) will also be 
+responded to with HTTP 503 errors, even if correct credentials are used. This is
+because access to the database data is not possible early during the startup.
diff --git a/3.10/release-notes-api-changes310.md b/3.10/release-notes-api-changes310.md
@@ -11,6 +11,64 @@ integrations for ArangoDB 3.10.
 
 ## HTTP RESTful API
 
+### Behavior changes
+
+The HTTP interface of arangod instances can optionally be started earlier during
+the startup process than in previous versions of ArangoDB. This is useful for
+responding to ping probes from monitoring tools before the instance has fully started.
+
+By default, the HTTP interface is opened at the same point during the startup
+sequence as in previous versions, but it can optionally be opened earlier by setting 
+the new startup option `--server.early-connections` to `true`. This will
+open the HTTP interface early in the startup sequence, so that the instance can respond
+to a limited set of REST APIs even during recovery. This can be useful because the 
+recovery procedure can take time proportional to the amount of data to recover.
+
+When the `--server.early-connections` option is set to `true`, the instance will be
+able to respond to requests to the following APIs early on:
+
+- GET `/_api/version` and `/_admin/version`: these APIs return the server version 
+  number, but can also be used as a lifeliness probe, to check if the instance is
+  responding to incoming HTTP requests.
+- GET `/_admin/status`: this API returns information about the instance's status, now
+  also including recovery progress and information about which server feature is
+  currently starting.
+
+If the `--server.early-connections` option is set to `true`, these APIs will return
+HTTP 200 responses during the startup. All other APIs than the ones listed above will be 
+responded to with an HTTP response code 503, so that callers can see that the instance
+is not fully ready.
+
+If authentication is used, then only JWT authentication can be used during the early 
+startup phase. Incoming requests relying on other authentication mechanisms that 
+require access to the database data (e.g. HTTP basic authentication) will also be 
+responded to with HTTP 503 errors, even if correct credentials are used. This is
+because access to the database data is not possible early during the startup.
+
+The GET `/_admin/status` API now also returns startup and recovery information. This
+can be used to determine the instance's progress during startup. The new `progress`
+attribute will be returned inside the `serverInfo` object with the following subattributes:
+
+- `phase`: name of the lifecycle phase the instance is currently in. Normally one of
+  "in prepare", "in start", "in wait", "in shutdown", "in stop" or "in unprepare".
+- `feature`: internal name of the feature that is currently being prepared, started, 
+   stopped or unprepared.
+- `recoveryTick`: current recovery sequence number value if the instance is currently in
+  recovery. If the instance is already past the recovery, this attribute will contain 
+  the last handled recovery sequence number.
+
+The exact values of these attributes should not be relied on, i.e. client applications
+should not check for any exact values in them. Feature and phase names are subject to
+change between different versions of ArangoDB. 
+The progress attributes can still be used to determine if the instance has made progress
+between two calls: if `phase`, feature` and `recoveryTick` don't change, then there hasn't
+been progress. Note that this is only true if the instance is still in startup. Once the
+instance has fully started and has opened the complete REST interface, the values in the
+`progress` attribute are expected to not change until shutdown.
+
+Note that the `maintenance` attribute in responses to GET `/_admin/status` can still be 
+used to determine if the instance is fully available for arbitrary requests.
+
 ### Privilege changes
 
 ### Endpoint return value changes
@@ -46,6 +104,31 @@ The following HTTP APIs are affected:
 * `/_api/wal/tail`
 * `/_api/replication/logger-follow`
 
+
+The GET `/_admin/status` API now also returns startup and recovery information. This
+can be used to determine the instance's progress during startup. The new `progress`
+attribute will be returned inside the `serverInfo` object with the following subattributes:
+
+- `phase`: name of the lifecycle phase the instance is currently in. Normally one of
+  "in prepare", "in start", "in wait", "in shutdown", "in stop" or "in unprepare".
+- `feature`: internal name of the feature that is currently being prepared, started, 
+   stopped or unprepared.
+- `recoveryTick`: current recovery sequence number value if the instance is currently in
+  recovery. If the instance is already past the recovery, this attribute will contain 
+  the last handled recovery sequence number.
+
+The exact values of these attributes should not be relied on, i.e. client applications
+should not check for any exact values in them. Feature and phase names are subject to
+change between different versions of ArangoDB. 
+The progress attributes can still be used to determine if the instance has made progress
+between two calls: if `phase`, feature` and `recoveryTick` don't change, then there hasn't
+been progress. Note that this is only true if the instance is still in startup. Once the
+instance has fully started and has opened the complete REST interface, the values in the
+`progress` attribute are expected to not change until shutdown.
+
+Note that the `maintenance` attribute in responses to GET `/_admin/status` can still be 
+used to determine if the instance is fully available for arbitrary requests.
+
 #### Cursor API
 
 The cursor API can now return additional statistics values in its `stats` subattribute:

diff --git a/3.10/release-notes-new-features310.md b/3.10/release-notes-new-features310.md
@@ -208,6 +208,40 @@ This feature is only available in the Enterprise Edition.
 Server options
 --------------
 
+### Responses early during instance startup
+
+The HTTP interface of arangod instances can optionally be started earlier during
+the startup process, so that ping probes from monitoring tools can already be 
+responded to when the instance has not fully started.
+
+By default, the HTTP interface is opened at the same point during the startup
+sequence as in previous versions, but it can optionally be opened earlier by setting 
+the new startup option `--server.early-connections` to `true`. This will
+open the HTTP interface early in the startup sequence, so that the instance can respond
+to a limited set of REST APIs even during recovery. This can be useful because the 
+recovery procedure can take time proportional to the amount of data to recover.
+
+When the `--server.early-connections` option is set to `true`, the instance will be
+able to respond to requests to the following APIs early on:
+
+- GET `/_api/version` and `/_admin/version`: these APIs return the server version 
+  number, but can also be used as a lifeliness probe, to check if the instance is
+  responding to incoming HTTP requests.
+- GET `/_admin/status`: this API returns information about the instance's status, now
+  also including recovery progress and information about which server feature is
+  currently starting.
+
+During the early startup phase, all other APIs than the ones listed above will be 
+responded to with an HTTP response code 503, so that callers can see that the instance 
+is not fully ready. In addition the `maintenance` attribute in the response to GET
+`/_admin/status` requests can be checked for general instance readiness.
+
+If authentication is used, then only JWT authentication can be used during the early 
+startup phase. Incoming requests relying on other authentication mechanisms that 
+require access to the database data (e.g. HTTP basic authentication) will also be 
+responded to with HTTP 503 errors, even if correct credentials are used. This is
+because access to the database data is not possible early during the startup.
+
 ### RocksDB startup options
 
 The default value of the `--rocksdb.cache-index-and-filter-blocks` startup option was changed
@@ -346,6 +380,6 @@ The bundled version of the Boost library has been upgraded from 1.71.0 to 1.78.0
 
 The bundled version of the immer library has been upgraded from 0.6.2 to 0.7.0.
 
-The bundled version of the jemalloc library has been upgraded from 5.2.1-dev to 5.2.1-RC.
+The bundled version of the jemalloc library has been upgraded from 5.2.1-dev to 5.3.0.
 
 The bundled version of the zlib library has been upgraded from 1.2.11 to 1.2.12.